diff --git a/AGENTS.md b/AGENTS.md index a7d2eec3..20165847 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,9 +37,11 @@ player's unless a section says otherwise. packages/player/ @missing-elements/h5p-offline-player — the published package src/ h5p-offline-player.ts the custom element — the only public entry point - plan.ts `./plan` -> dist/h5p-plan.js: the player's library rules (`planPack`, `librariesNamedIn`, - `addonPatterns`, `textMatches`, `dependenciesIn`) for the packages that write what it - plays; not covered by docs/compatibility.md, so they pin the player exactly (#106) + plan.ts `./plan` -> dist/h5p-plan.js: the player's library rules (`planPack`, `ownLibraries`, + `librariesNamedIn`, `addonPatterns`, `textMatches`, `dependenciesIn`, `reachableFolder`, + `walkDependencies` and the folder helpers) for the packages that write what it plays, + h5p-author and embed; not covered by docs/compatibility.md, so they pin the player + exactly (#106, #109) jobs-worker-handle.ts starts the Jobs worker from the first script the page's CSP allows, holding jobs until it runs user-data-store.ts the learner's saved state for `resume`, in a database of the page's own shadow.css the element's own box, imported `?inline` so Vite minifies it into the bundle @@ -63,6 +65,10 @@ packages/player/ @missing-elements/h5p-offline-player — the published content-libraries.ts which libraries `h5p.json`, a `library.json` and content's parameters name, and an addon's patterns: out of package-reader.ts on 2026-11-02, so that the plan and `./plan` carry no zip.js (#106) + library-folders.ts which folder serves a library (`reachableFolder`, `resolveLibraryFolder`), what a + manifest names that nothing carries, and `walkDependencies`, the walk through what + the libraries need in turn, which the reader's check and embed's SCORM writer both + run: out of package-reader.ts on 2026-11-02, for `./plan` (#109) pack-index.ts generated by sync-h5p-assets from packages/libraries/parts/index.json: the pinned pack's index, which the worker plans from and checks each part against metadata.ts `h5p.json`'s metadata in the shape the runtime reads, held to the core's licence table @@ -223,11 +229,13 @@ packages/embed/ @missing-elements/h5p-embed — the embed page as a st lib/build.mjs buildSite(): copies the page, the player's dist/, the runtime's dist/ as frame-assets/ and the library pack; writes config.js, the policy into the page, the headers for five kinds of host and NOTICE.txt - lib/scorm.mjs buildScorm(), `--scorm`: one .h5p, the player and the runtime as a SCORM 1.2 - zip, refused when the package does not carry its libraries + lib/scorm.mjs buildScorm(), `--scorm`: one .h5p, the player, the runtime and the parts of the + pack it needs as a SCORM 1.2 zip, planned with the player's `./plan`; refused for + a library neither the package nor the pack carries scorm/ the SCORM package's launch page (index.html, its title, language and build filled in by the writer) and scorm.js, its bridge to the LMS's API bin/h5p-embed.mjs the command + scripts/ check-player-published.mjs, the publish guard for the player it pins exactly (#109) tests/ build.test.ts, no browser; site.test.ts, a written site on 127.0.0.1 framed by a page on localhost, as deployed; scorm.test.ts, the zip in Node and the launch page under a stand-in for an LMS's player page in @@ -2287,19 +2295,21 @@ one row each; an LMS added here gets a row there. LMS's origin by design, so the package's own code runs, as any SCORM package's does. An export without its libraries, from h5p.com or h5p.org, gets the parts of the pack it needs, in the zip (the owner's call, the same day: those exports are what many teachers have). The writer runs the - player's planner as the launch page has the element run it, filling gaps only: `planParts` is - `planPack` with `refresh` off and nothing attached before, and `librariesNamedIn`, - `addonPatterns`, `textMatches` and `ownLibraries` are copies of the player's. Keep them in step - until #109 has the writer import them from the player's `./plan`, as h5p-author does. + player's own planner as the launch page has the element run it, filling gaps only: `planPack` + from the player's `./plan` with `refresh` off and nothing provided, over `ownLibraries`, + `dependenciesIn` and `librariesNamedIn` from the same module. Until #109 (2026-11-02) those were + copies kept in step by hand, which nothing checked; since then embed pins the player exactly, as + h5p-author does, and `scripts/check-player-published.mjs` refuses to publish until the pinned + version is on npm with `./plan`. It writes those parts into `libraries/`, with an `index.json` that lists only them, and the page names the folder with `libraries="pack"`, `pack-base` and `refresh-libraries="off"`. The element then plans the same parts and finds each in the zip, and plays a package's own libraries as exported. Because the index lists only what the zip carries, a plan that ever came out different could only fall short: a gap the element refuses by name, or an addon left out, never a request for a part the zip lacks. What neither the package nor the pack supplies is refused when the zip - is written, by the walk the player makes (`servingFolder` mirrors `reachableFolder`, the walk - `missingDependencies`), run over the package without the folders a part replaces and over the - parts. The walk starts from the sub-content the parameters name as well, which the player found + is written, by the walk the player makes (`walkDependencies` over `reachableFolder`, the same + functions `PackageReader.checkDependencies` runs), run over the package without the folders a + part replaces and over the parts. The walk starts from the sub-content the parameters name as well, which the player found missing only when the runtime reached it until #85, and since then finds at index only where a source has the library's major. So a gap shows up when the teacher writes the zip, not when a learner opens it. Measured: the demo's formulas quiz, a stripped export, took 11 parts, diff --git a/docs/compatibility.md b/docs/compatibility.md index 47147eb4..6546c5b9 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -134,8 +134,8 @@ warning can come in any release. - Exports and files this page does not name: a package's other exports, the rest of the player's `dist/`, the files inside the runtime's. Nor anything reached by a path a package does not export. Among them the player's `./plan`, its library rules for the packages of this - repository that write what it plays, which depend on an exact version of the player for that - reason. + repository that write what it plays, h5p-author and the embed command, which depend on an exact + version of the player for that reason. - The demo site, including `/app/` and `/embed` there, and this repository's scripts and workflows. diff --git a/packages/embed/lib/scorm.mjs b/packages/embed/lib/scorm.mjs index 0b7fb1a3..a2050f73 100644 --- a/packages/embed/lib/scorm.mjs +++ b/packages/embed/lib/scorm.mjs @@ -3,8 +3,9 @@ import { createReadStream, createWriteStream, openAsBlob } from 'node:fs' import { readdir, readFile, rename, rm, stat } from 'node:fs/promises' import { basename, dirname, extname, join, resolve, sep } from 'node:path' import { Writable } from 'node:stream' -import { fileURLToPath } from 'node:url' +import { fileURLToPath, pathToFileURL } from 'node:url' import { BlobReader, TextReader, TextWriter, ZipReader, ZipWriter, configure } from '@zip.js/zip.js' +import { dependenciesIn, indexLibraryFolders, librariesNamedIn, ownLibraries, planPack, reachableFolder, walkDependencies } from '@missing-elements/h5p-offline-player/plan' import { EmbedError, PLAYER_FILES, installed } from './build.mjs' import { crc32 } from './crc32.mjs' @@ -34,266 +35,22 @@ const PACKAGE = 'package.h5p' const LIBRARIES = 'libraries' /** - * A dependency as `h5p.json` and every `library.json` list them, or nothing for an entry that is - * not one. - * @param {unknown} json - * @returns {{ machineName: string, major: number, minor: number }[]} - */ -function dependenciesIn(json) { - const list = /** @type {{ preloadedDependencies?: unknown }} */ (json ?? {}).preloadedDependencies - if (!Array.isArray(list)) return [] - return list.flatMap((dependency) => { - const major = Number(dependency?.majorVersion) - const minor = Number(dependency?.minorVersion) - return typeof dependency?.machineName === 'string' && Number.isInteger(major) && Number.isInteger(minor) - ? [{ machineName: dependency.machineName, major, minor }] - : [] - }) -} - -/** - * The folder that serves a dependency, by the player's own rule: its versioned or bare name when - * that folder has a `library.json`, else the lowest versioned folder of the same major with a - * minor at least as high (`reachableFolder` in the player's `package-reader.ts`). - * @internal - * @param {{ machineName: string, major: number, minor: number }} dependency - * @param {ReadonlySet} names the archive's entry names - * @returns {string | undefined} - */ -export function servingFolder({ machineName, major, minor }, names) { - const own = [`${machineName}-${major}.${minor}`, machineName].find((folder) => names.has(`${folder}/library.json`)) - if (own) return own - let best - for (const name of names) { - const match = /^(.+)-(\d+)\.(\d+)\/library\.json$/.exec(name) - if (!match || match[1] !== machineName || Number(match[2]) !== major || Number(match[3]) < minor) continue - if (!best || Number(match[3]) < best.minor) best = { folder: `${match[1]}-${match[2]}.${match[3]}`, minor: Number(match[3]) } - } - return best?.folder -} - -/** - * What the content asks for, and what those libraries need in turn, that no folder in `names` - * serves: the player refuses such a package when it opens it, and the deep walk is the one it makes - * before the runtime boots (`missingDependencies`). The sub-content the parameters name counts - * too, which the player finds missing only when the runtime reaches it. - * @param {{ machineName: string, major: number, minor: number }[]} roots - * @param {ReadonlySet} names - * @param {(name: string) => Promise} read a `library.json` among `names`, parsed - * @returns {Promise<{ machineName: string, major: number, minor: number }[]>} - */ -async function missingLibraries(roots, names, read) { - /** @type {Map} */ - const missing = new Map() - const seen = new Set() - let level = roots - while (level.length > 0) { - const next = [] - for (const dependency of level) { - const key = `${dependency.machineName} ${dependency.major}.${dependency.minor}` - if (seen.has(key)) continue - seen.add(key) - const folder = servingFolder(dependency, names) - if (!folder) missing.set(key, dependency) - else next.push(...dependenciesIn(await read(`${folder}/library.json`))) - } - level = next - } - return [...missing].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)).map(([, dependency]) => dependency) -} - -/* - * The library pack's part of the work: which of its parts the package needs. These follow the - * player's own functions, named on each, so that the element, planning from the zip's index, - * asks for exactly the parts the zip carries; keep them in step. - */ - -/** - * The sub-content libraries the parameters name, by each object's `library` string, as - * `"H5P.MultiChoice 1.16"`: what a full export's `h5p.json` lists and a stripped one's does not. - * The player's `librariesNamedIn`. - * @param {unknown} params - * @returns {{ machineName: string, major: number, minor: number }[]} - */ -function librariesNamedIn(params) { - /** @type {Map} */ - const found = new Map() - /** @param {unknown} value @param {number} depth */ - const walk = (value, depth) => { - if (depth > 64 || !value || typeof value !== 'object') return - if (Array.isArray(value)) { - for (const item of value) walk(item, depth + 1) - return - } - const library = /** @type {{ library?: unknown }} */ (value).library - if (typeof library === 'string') { - const match = /^([A-Za-z0-9_.-]+) (\d+)\.(\d+)$/.exec(library) - if (match && !found.has(match[1])) found.set(match[1], { machineName: match[1], major: Number(match[2]), minor: Number(match[3]) }) - } - for (const child of Object.values(value)) walk(child, depth + 1) - } - walk(params, 0) - return [...found.values()] -} - -/** - * An addon's patterns, under `addTo.content.types[].text.regex`, PHP's `/…/flags` made - * JavaScript's; one JavaScript cannot read, or with a flag it has no counterpart for, is left out. - * The player's `addonPatterns`. - * @param {unknown} addTo - * @returns {RegExp[]} - */ -function addonPatterns(addTo) { - const types = /** @type {{ content?: { types?: unknown } } | null | undefined} */ (addTo)?.content?.types - if (!Array.isArray(types)) return [] - const patterns = [] - for (const type of types) { - const source = type?.text?.regex - if (typeof source !== 'string' || source.length < 2) continue - const delimiter = source[0] - if (/[\p{L}\p{N}\\\s([{<]/u.test(delimiter)) continue - const end = source.lastIndexOf(delimiter) - const flags = source.slice(end + 1) - if (end < 1 || /[^imsu]/.test(flags)) continue - try { - patterns.push(new RegExp(source.slice(1, end), flags)) - } catch { - // PCRE that JavaScript does not read. - } - } - return patterns -} - -/** - * Whether a string anywhere in the parameters matches one of the patterns. The player's - * `textMatches`. - * @param {unknown} params - * @param {RegExp[]} patterns - */ -function textMatches(params, patterns) { - /** @param {unknown} value @param {number} depth @returns {boolean} */ - const walk = (value, depth) => { - if (typeof value === 'string') return patterns.some((pattern) => pattern.test(value)) - if (depth > 64 || !value || typeof value !== 'object') return false - return Object.values(value).some((child) => walk(child, depth + 1)) - } - return patterns.length > 0 && walk(params, 0) -} - -/** - * The library folders the package carries, each as its `library.json` says: the player's - * `ownLibraries` once every `library.json` is at hand. One that does not parse, or a folder named - * for another library, is left out. - * @param {Iterable} names - * @param {(name: string) => Promise} read - */ -async function ownLibraries(names, read) { - /** @param {unknown} part */ - const version = (part) => (typeof part === 'number' && Number.isInteger(part) && part >= 0 ? part : typeof part === 'string' && /^\d+$/.test(part) ? Number(part) : null) - const libraries = [] - for (const name of names) { - const slash = name.indexOf('/') - if (slash < 0 || name.slice(slash + 1) !== 'library.json') continue - const folder = name.slice(0, slash) - const library = /** @type {{ machineName?: unknown, majorVersion?: unknown, minorVersion?: unknown } | null} */ (await read(name).catch(() => null)) - const major = version(library?.majorVersion) - const minor = version(library?.minorVersion) - const machineName = library?.machineName - if (typeof machineName !== 'string' || major === null || minor === null) continue - const versioned = /^(.+)-(\d+)\.(\d+)$/.exec(folder) - if (versioned ? versioned[1] !== machineName || Number(versioned[2]) !== major : folder !== machineName) continue - libraries.push({ folder, machineName, major, minor, dependencies: dependenciesIn(library) }) - } - return libraries -} - -/** - * The library pack's index, its `parts/index.json`, as the plan reads it. `entry` is the library's - * own record, which the zip's index repeats. + * The library pack's index, its `parts/index.json`, as the plan takes it: each library's record, + * which the zip's index repeats as `record`, with the part's file as `url`. A record the index got + * wrong is left out. * @param {unknown} index + * @param {string} packDir the folder holding the index and the parts */ -function packLibraries(index) { +function packLibraries(index, packDir) { const list = /** @type {{ libraries?: unknown } | null} */ (index)?.libraries - return (Array.isArray(list) ? list : []).flatMap((entry) => { - const { folder, machineName, majorVersion, minorVersion, sha256, addTo } = entry ?? {} - return typeof folder === 'string' && typeof machineName === 'string' && Number.isInteger(majorVersion) && Number.isInteger(minorVersion) && typeof sha256 === 'string' - ? [{ folder, machineName, major: Number(majorVersion), minor: Number(minorVersion), sha256, dependencies: dependenciesIn(entry), addTo, entry: /** @type {object} */ (entry) }] - : [] + return (Array.isArray(list) ? list : []).flatMap((record) => { + const { folder, machineName, majorVersion, minorVersion, patchVersion, size, sha256 } = record ?? {} + if (typeof folder !== 'string' || typeof machineName !== 'string' || ![majorVersion, minorVersion, patchVersion, size].every(Number.isInteger) || typeof sha256 !== 'string') return [] + const library = /** @type {import('@missing-elements/h5p-offline-player/plan').PackLibrary} */ (record) + return [{ ...library, url: pathToFileURL(join(packDir, `${folder}.h5p`)).href, record: library }] }) } -/** - * The parts of the library pack the package needs, and the folders of its own they replace: the - * player's `planPack` (`sw/pack-plan.ts`) as the launch page has the element run it, filling gaps - * with nothing attached before (`refresh-libraries="off"`). - * - What the content asks for, and the package carries at that major and at no lower minor, plays - * from the package. Anything else comes from the pack, which has one version of each major, and - * so does whatever a part needs; an older minor the package carries gives way to the part. - * - Once anything comes from the pack, so do the addons whose pattern matches a string in the - * parameters: H5P.MathDisplay, for LaTeX. - * @param {object} input - * @param {Awaited>} input.own - * @param {{ machineName: string, major: number, minor: number }[]} input.roots what `h5p.json` - * declares and the parameters name as sub-content - * @param {unknown} input.params - * @param {ReturnType} input.pack - */ -function planParts({ own, roots, params, pack }) { - const byLibrary = new Map(pack.map((library) => [`${library.machineName} ${library.major}`, library])) - /** @type {Map[number]>} */ - const planned = new Map() - /** @type {string[]} */ - const superseded = [] - let carried = own - const queue = [...roots] - const seen = new Set() - - /** - * @param {{ machineName: string, major: number, minor: number }} version - * @param {{ machineName: string, major: number, minor: number }} dependency - */ - const satisfies = (version, dependency) => - version.machineName === dependency.machineName && version.major === dependency.major && version.minor >= dependency.minor - /** @param {ReturnType[number]} library */ - const plan = (library) => { - if (planned.has(library.folder)) return - planned.set(library.folder, library) - queue.push(...library.dependencies) - } - const follow = () => { - for (let next = queue.shift(); next; next = queue.shift()) { - const dependency = next - const key = `${dependency.machineName} ${dependency.major}.${dependency.minor}` - if (seen.has(key)) continue - seen.add(key) - const ownVersion = carried.find((library) => satisfies(library, dependency)) - if (ownVersion) { - queue.push(...ownVersion.dependencies) - continue - } - if ([...planned.values()].some((library) => satisfies(library, dependency))) continue - const candidate = byLibrary.get(`${dependency.machineName} ${dependency.major}`) - if (!candidate || candidate.minor < dependency.minor) continue - const older = carried.find((library) => library.machineName === dependency.machineName && library.major === dependency.major) - if (older) { - carried = carried.filter((library) => library !== older) - superseded.push(older.folder) - } - plan(candidate) - } - } - - follow() - if (planned.size > 0) { - for (const addon of pack) { - if (addon.addTo === undefined || carried.some((library) => library.machineName === addon.machineName)) continue - if (textMatches(params, addonPatterns(addon.addTo))) plan(addon) - } - follow() - } - return { parts: [...planned.values()].sort((a, b) => (a.folder < b.folder ? -1 : a.folder > b.folder ? 1 : 0)), superseded } -} - /** @param {string} text */ const escapeXml = (text) => text.replace(/[&<>"']/g, (char) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[char] ?? char) @@ -443,13 +200,13 @@ export async function buildScorm({ h5p, out, force = false }) { if (targetInfo && !force) throw new EmbedError(`${out} exists. Pass --force to replace it.`) const packDir = dirname(installed('@missing-elements/h5p-libraries/parts/index.json', 'Reinstall this package.')) - const pack = packLibraries(JSON.parse(await readFile(join(packDir, 'index.json'), 'utf8'))) + const pack = packLibraries(JSON.parse(await readFile(join(packDir, 'index.json'), 'utf8')), packDir) // What the package declares and carries, and what the pack adds, read before anything is written. /** @type {any} `h5p.json`, parsed */ let manifest - /** @type {ReturnType} */ - let plan + /** @type {typeof pack} the parts of the pack the zip carries, by folder */ + let parts let reader try { reader = new ZipReader(new BlobReader(await openAsBlob(source))) @@ -473,22 +230,39 @@ export async function buildScorm({ h5p, out, force = false }) { throw new EmbedError(`${basename(h5p)}: h5p.json names no libraries, so there is nothing to play.`) } const params = await readJson('content/content.json') - const roots = [...dependenciesIn(manifest), ...librariesNamedIn(params)] - plan = planParts({ own: await ownLibraries(byName.keys(), readJson), roots, params, pack }) + /** @param {{ machineName: string, majorVersion: string | number, minorVersion: string | number }} dependency */ + const numbered = ({ machineName, majorVersion, minorVersion }) => ({ machineName, majorVersion: Number(majorVersion), minorVersion: Number(minorVersion) }) + const roots = [...dependenciesIn(manifest), ...librariesNamedIn(params)].map(numbered) + const folders = [...byName.keys()].flatMap((name) => { + const match = /^([^/]+)\/library\.json$/.exec(name) + return match ? [match[1]] : [] + }) + // The player's own plan, as the launch page has the element run it: gaps filled from the pack, + // nothing provided before, no refresh (`refresh-libraries="off"`). + const plan = planPack({ own: await ownLibraries(folders, (folder) => readJson(`${folder}/library.json`)), provided: [], roots, params, pack, refresh: false, fill: true }) + const byFolder = new Map(pack.map((library) => [library.folder, library])) + parts = (plan?.parts ?? []).map((part) => /** @type {typeof pack[number]} */ (byFolder.get(part.folder))).sort((a, b) => (a.folder < b.folder ? -1 : a.folder > b.folder ? 1 : 0)) // Held to its libraries as the player holds it once the parts are attached: the folders they // replace withheld, and each part's `library.json` naming what its record in the index does. - const parts = new Map(plan.parts.map((part) => [`${part.folder}/library.json`, part.entry])) - const withheld = new Set(plan.superseded) + const records = new Map(parts.map((part) => [`${part.folder}/library.json`, part.record])) + const withheld = new Set(plan?.superseded ?? []) const names = new Set([...byName.keys()].filter((name) => !withheld.has(name.split('/')[0]))) - for (const name of parts.keys()) names.add(name) - const missing = await missingLibraries(roots, names, async (name) => parts.get(name) ?? readJson(name)) + for (const name of records.keys()) names.add(name) + const available = indexLibraryFolders(names) + const { missing } = await walkDependencies( + roots, + (dependency) => reachableFolder(dependency, names, available), + async (folder) => records.get(`${folder}/library.json`) ?? readJson(`${folder}/library.json`) + ) if (missing.length > 0) { - const packed = new Map(pack.map((library) => [`${library.machineName} ${library.major}`, library])) - const listed = missing.map(({ machineName, major, minor }) => { - const older = packed.get(`${machineName} ${major}`) - return `${machineName} ${major}.${minor}${older ? ` (the pack has ${older.major}.${older.minor})` : ''}` - }) + const packed = new Map(pack.map((library) => [`${library.machineName} ${library.majorVersion}`, library])) + const listed = missing + .map(({ machineName, majorVersion, minorVersion }) => { + const older = packed.get(`${machineName} ${Number(majorVersion)}`) + return `${machineName} ${majorVersion}.${minorVersion}${older ? ` (the pack has ${older.majorVersion}.${older.minorVersion})` : ''}` + }) + .sort() throw new EmbedError( `${basename(h5p)} needs libraries that neither it nor the library pack carries: ${listed.join(', ')}. ` + 'Export the package with its libraries, as Lumi and the H5P plugins of Moodle, WordPress and Drupal do.' @@ -506,12 +280,12 @@ export async function buildScorm({ h5p, out, force = false }) { const title = named || basename(h5p, extname(h5p)) // The learner's saved place is kept against the libraries too, as the player keeps it against // the revision: a zip written with others starts learners over, as a refreshed package does. - const hashes = [await sha256(source), ...plan.parts.map((part) => part.sha256)] + const hashes = [await sha256(source), ...parts.map((part) => part.sha256)] const build = (hashes.length === 1 ? hashes[0] : createHash('sha256').update(hashes.join('\n')).digest('hex')).slice(0, 16) const player = dirname(installed('@missing-elements/h5p-offline-player/dist/h5p-player.js', 'Run `pnpm build` in the workspace, or reinstall this package.')) const runtime = dirname(installed('@missing-elements/h5p-runtime/dist/h5p.css', 'Run `pnpm build` in the workspace, or reinstall this package.')) const version = (await readFile(join(player, 'VERSION'), 'utf8').catch(() => 'unknown')).trim() - const packVersion = plan.parts.length > 0 ? String(JSON.parse(await readFile(join(dirname(packDir), 'package.json'), 'utf8')).version) : null + const packVersion = parts.length > 0 ? String(JSON.parse(await readFile(join(dirname(packDir), 'package.json'), 'utf8')).version) : null const page = (await readFile(join(SCORM, 'index.html'), 'utf8')) .replace('%LANG%', languageTag(manifest.defaultLanguage)) @@ -531,8 +305,8 @@ export async function buildScorm({ h5p, out, force = false }) { [PACKAGE, { file: source }] ] if (packVersion) { - files.push([`${LIBRARIES}/index.json`, { text: `${JSON.stringify({ libraries: plan.parts.map((part) => part.entry) }, null, 2)}\n` }]) - for (const part of plan.parts) files.push([`${LIBRARIES}/${part.folder}.h5p`, { file: join(packDir, `${part.folder}.h5p`) }]) + files.push([`${LIBRARIES}/index.json`, { text: `${JSON.stringify({ libraries: parts.map((part) => part.record) }, null, 2)}\n` }]) + for (const part of parts) files.push([`${LIBRARIES}/${part.folder}.h5p`, { file: join(packDir, `${part.folder}.h5p`) }]) files.push(['libraries.txt', { file: join(dirname(packDir), 'libraries.txt') }]) } files.push(['NOTICE.txt', { text: notice({ version, source: basename(h5p), pack: packVersion }) }]) @@ -566,5 +340,5 @@ export async function buildScorm({ h5p, out, force = false }) { throw error } - return { out: target, title, version, libraries: plan.parts.map((part) => part.folder), build, size: (await stat(target)).size } + return { out: target, title, version, libraries: parts.map((part) => part.folder), build, size: (await stat(target)).size } } diff --git a/packages/embed/package.json b/packages/embed/package.json index c40b7ffe..e1f56a21 100644 --- a/packages/embed/package.json +++ b/packages/embed/package.json @@ -61,12 +61,12 @@ "test": "pnpm --filter @missing-elements/h5p-offline-player build && vitest run", "typecheck": "pnpm types && tsc -p tsconfig.json", "prepack": "node -e \"require('node:fs').copyFileSync('../../LICENSE', 'LICENSE')\" && pnpm types", - "prepublishOnly": "node -e \"if (!/^pnpm\\//.test(process.env.npm_config_user_agent || '')) { console.error('Publish with pnpm publish: npm would ship the workspace:^ dependency ranges as they are, and the package would not install.'); process.exit(1) }\" && pnpm typecheck && pnpm test", - "types": "node -e \"require('node:fs').rmSync('types', { recursive: true, force: true })\" && tsc -p tsconfig.types.json" + "prepublishOnly": "node -e \"if (!/^pnpm\\//.test(process.env.npm_config_user_agent || '')) { console.error('Publish with pnpm publish: npm would ship the workspace: dependency ranges as they are, and the package would not install.'); process.exit(1) }\" && node scripts/check-player-published.mjs && pnpm typecheck && pnpm test", + "types": "pnpm --filter @missing-elements/h5p-offline-player exec tsc -b && node -e \"require('node:fs').rmSync('types', { recursive: true, force: true })\" && tsc -p tsconfig.types.json" }, "dependencies": { "@missing-elements/h5p-libraries": "workspace:^", - "@missing-elements/h5p-offline-player": "workspace:^", + "@missing-elements/h5p-offline-player": "workspace:*", "@missing-elements/h5p-runtime": "workspace:^", "@zip.js/zip.js": "^2.24.0" }, diff --git a/packages/embed/scripts/check-player-published.mjs b/packages/embed/scripts/check-player-published.mjs new file mode 100644 index 00000000..cbef3385 --- /dev/null +++ b/packages/embed/scripts/check-player-published.mjs @@ -0,0 +1,21 @@ +/** + * Refuses to publish the embed package while the player it pins is not on npm, or is there without + * `./plan`, which the SCORM writer imports. The dependency is `workspace:*` since #109, published + * as the exact version packages/player has, because `./plan` is not covered by + * docs/compatibility.md: a release made before the player's would not install, and one pinned to a + * player published before `./plan` existed would install and fail on its first import. + */ +import { readFile } from 'node:fs/promises' +import { resolve } from 'node:path' + +const { version } = JSON.parse(await readFile(resolve(import.meta.dirname, '..', '..', 'player', 'package.json'), 'utf8')) +const response = await fetch(`https://registry.npmjs.org/@missing-elements%2fh5p-offline-player/${version}`) +if (!response.ok) { + console.error(`[player] @missing-elements/h5p-offline-player@${version} is not on npm (${response.status}). Publish it from packages/player first: the embed package pins it, for its ./plan.`) + process.exit(1) +} +if (!(await response.json()).exports?.['./plan']) { + console.error(`[player] @missing-elements/h5p-offline-player@${version} on npm does not export ./plan, which the SCORM writer imports: bump the player's version, publish it, and publish the embed package against that.`) + process.exit(1) +} +console.log(`[player] @missing-elements/h5p-offline-player@${version} is published, and exports ./plan`) diff --git a/packages/embed/tests/package-types.ts b/packages/embed/tests/package-types.ts index 91efc285..0e418fc7 100644 --- a/packages/embed/tests/package-types.ts +++ b/packages/embed/tests/package-types.ts @@ -10,8 +10,6 @@ import { pack } from '@missing-elements/h5p-libraries' import { runtime } from '@missing-elements/h5p-runtime' // @ts-expect-error a helper the tests use, not part of the package's API (#60) import { parseOrigin } from '@missing-elements/h5p-embed' -// @ts-expect-error the same for the SCORM writer's -import { servingFolder } from '@missing-elements/h5p-embed/scorm' export async function asACallerWould(): Promise { const site: BuiltSite = await buildSite({ out: 'public/player', packages: ['https://cdn.example.org'], defaultLibraries: 'pack' }) @@ -30,6 +28,6 @@ export async function asACallerWould(): Promise { // @ts-expect-error the package is named `h5p` await buildScorm({ package: 'course.h5p', out: 'course.scorm.zip' }) - void [hosts, build, fromPack, parseOrigin, servingFolder, new EmbedError('refused') instanceof Error] + void [hosts, build, fromPack, parseOrigin, new EmbedError('refused') instanceof Error] return site.csp } diff --git a/packages/embed/tests/scorm.test.ts b/packages/embed/tests/scorm.test.ts index c0b5d663..72ec0c8d 100644 --- a/packages/embed/tests/scorm.test.ts +++ b/packages/embed/tests/scorm.test.ts @@ -12,7 +12,7 @@ import { BlobReader, BlobWriter, TextWriter, Uint8ArrayReader, Uint8ArrayWriter, import { chromium, type Browser, type BrowserContext, type Frame, type Page } from 'playwright' import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { EmbedError } from '../lib/build.mjs' -import { buildScorm, servingFolder } from '../lib/scorm.mjs' +import { buildScorm } from '../lib/scorm.mjs' /** * The SCORM package `--scorm` writes: what is in the zip and what is refused, in Node; and in @@ -197,14 +197,6 @@ describe('the package', () => { expect(existsSync(out)).toBe(false) }) - it('takes a newer minor of the same major, and only that, as the player serves libraries', () => { - const names = new Set(['H5P.Text-1.3/library.json', 'H5P.Text-1.5/library.json', 'H5P.Text-2.0/library.json', 'Bare/library.json']) - expect(servingFolder({ machineName: 'H5P.Text', major: 1, minor: 1 }, names)).toBe('H5P.Text-1.3') - expect(servingFolder({ machineName: 'H5P.Text', major: 1, minor: 4 }, names)).toBe('H5P.Text-1.5') - expect(servingFolder({ machineName: 'H5P.Text', major: 1, minor: 6 }, names)).toBeUndefined() - expect(servingFolder({ machineName: 'Bare', major: 3, minor: 0 }, names)).toBe('Bare') - }) - it('refuses what is not a package, and a zip already there unless forced', async () => { const notZip = join(dir, 'not.h5p') await writeFile(notZip, 'not a zip') diff --git a/packages/player/src/plan.ts b/packages/player/src/plan.ts index 3e8efae9..19534282 100644 --- a/packages/player/src/plan.ts +++ b/packages/player/src/plan.ts @@ -1,12 +1,24 @@ /** * `@missing-elements/h5p-offline-player/plan`: the player's rules for the libraries a package plays * with, for the packages that write what it plays, so that they run the player's own rules rather - * than copies of them: which libraries content names, an addon's patterns and what they match, and - * which parts of the library pack a package needs. Pure and free of zip.js, for Node and a page - * alike. Not named in docs/compatibility.md, so it may change in any release: a package that - * imports it depends on an exact version of the player. + * than copies of them: which libraries content names, an addon's patterns and what they match, + * which folder serves a library and the walk through what the libraries need in turn, what a + * package carries of its libraries, and which parts of the library pack it needs. Pure and free of + * zip.js, for Node and a page alike. Not named in docs/compatibility.md, so it may change in any + * release: a package that imports it depends on an exact version of the player (h5p-author since + * #106, embed since #109). */ -export { addonPatterns, dependenciesIn, librariesNamedIn, textMatches, type LibraryDependency } from './shared/content-libraries.js' -export { planPack, type OwnLibrary, type PlanInput, type ProvidedLibrary } from './sw/pack-plan.js' +export { addonPatterns, dependenciesIn, librariesNamedIn, textMatches, type LibraryDependency, type PackageManifest } from './shared/content-libraries.js' +export { + findMissingLibraries, + indexLibraryFolders, + libraryFolderNames, + parseLibraryFolder, + reachableFolder, + resolveLibraryFolder, + walkDependencies, + type LibraryFolder +} from './shared/library-folders.js' +export { ownLibraries, planPack, type OwnLibrary, type PlanInput, type ProvidedLibrary } from './sw/pack-plan.js' export type { PackDependency, PackLibrary, PackPart, PackPlan, RefreshedLibrary } from './shared/protocol.js' diff --git a/packages/player/src/shared/content-libraries.ts b/packages/player/src/shared/content-libraries.ts index ac3c4e11..a4b1f578 100644 --- a/packages/player/src/shared/content-libraries.ts +++ b/packages/player/src/shared/content-libraries.ts @@ -11,6 +11,13 @@ export interface LibraryDependency { minorVersion: string | number } +/** The parts of `h5p.json` the player needs. Everything in it comes from an untrusted archive. */ +export interface PackageManifest { + title?: string + mainLibrary?: string + preloadedDependencies?: LibraryDependency[] +} + /** * Every library a content's parameters name, as sub-content does it: an object whose `library` * is `" ."`, anywhere in the tree. In archive order of discovery, diff --git a/packages/player/src/shared/library-folders.ts b/packages/player/src/shared/library-folders.ts new file mode 100644 index 00000000..49fb5d57 --- /dev/null +++ b/packages/player/src/shared/library-folders.ts @@ -0,0 +1,178 @@ +/** + * Which folder serves a library, and the walk through what the libraries need in turn. Pure and + * free of zip.js, like `content-libraries.ts`, so that the reader, the plan and the packages that + * write what the player plays, through `./plan`, decide by one rule. Out of package-reader.ts on + * 2026-11-02 (#109); `bootReady`, `mergedManifest`, `get` and the missing-library checks there + * still decide by `reachableFolder`. + */ + +import { dependenciesIn, type LibraryDependency, type PackageManifest } from './content-libraries.js' + +export interface LibraryFolder { + machineName: string + major: number + minor: number + folder: string +} + +/** Splits `H5P.Text-1.1` into its parts, or returns `null` for a folder that is not a library. */ +export function parseLibraryFolder(folder: string): LibraryFolder | null { + const match = /^(.+)-(\d+)\.(\d+)$/.exec(folder) + if (!match) return null + return { + machineName: match[1], + major: Number(match[2]), + minor: Number(match[3]), + folder + } +} + +/** Every library folder in a set of entry names, grouped by machine name. */ +export function indexLibraryFolders(entryNames: Iterable): Map { + const byName = new Map() + + for (const name of entryNames) { + const slash = name.indexOf('/') + if (slash < 0 || name.slice(slash + 1) !== 'library.json') continue + + const parsed = parseLibraryFolder(name.slice(0, slash)) + if (!parsed) continue + + const existing = byName.get(parsed.machineName) + if (existing) existing.push(parsed) + else byName.set(parsed.machineName, [parsed]) + } + + return byName +} + +/** + * The two folder names a library may be stored under: the versioned one, and the bare machine + * name older packages use. Each is tried in that order, as h5p-standalone tried them. + */ +export function libraryFolderNames(dependency: LibraryDependency): string[] { + return [ + `${dependency.machineName}-${dependency.majorVersion}.${dependency.minorVersion}`, + dependency.machineName + ] +} + +/** + * Picks the folder that satisfies a dependency: the same major version, and a minor at least as + * high as the one asked for. H5P bumps the major for a change content cannot follow, and gives a + * minor that changed the shape of a library's parameters a step in its `upgrades.js`; the frame + * runs those steps on content written for an older minor before the runtime starts + * (`content-upgrade.ts` in the runtime package), as the platforms do when they install a newer one. + * + * This matters whenever libraries come from somewhere other than the package: the library pack ships + * the current `H5P.Text-1.1` while content authored earlier asks for `H5P.Text-1.0`, and without + * this every such request is a 404. + * + * The lowest satisfying version wins, as the closest to what the content was written against. + * + * With `downgrade`, a dependency nothing satisfies is served by the newest older minor of the same + * major: content made on h5p.com after a minor bump the library pack has not caught up with (#85). + * Three bumps in four add fields the older code ignores; the fourth moved some, which the older + * code then does not find. The element asks for it only once every library source has been tried + * for the version named, and never for an archive still arriving, which may yet carry it. + */ +export function resolveLibraryFolder( + available: Map, + dependency: { machineName: string; major: number; minor: number }, + downgrade = false +): string | undefined { + const sameMajor = (available.get(dependency.machineName) ?? []).filter((candidate) => candidate.major === dependency.major) + const satisfying = sameMajor.filter((candidate) => candidate.minor >= dependency.minor).sort((a, b) => a.minor - b.minor) + if (satisfying.length > 0 || !downgrade) return satisfying[0]?.folder + + return [...sameMajor].sort((a, b) => b.minor - a.minor)[0]?.folder +} + +/** + * The folder a dependency is served from: its name as written, or the bare machine name + * h5p-standalone falls back to, when that folder's `library.json` is among `names`; else a + * compatible newer minor of the same major from `available`, the rule `get()` serves library + * files by, or with `downgrade` an older one. `bootReady`, `mergedManifest` and + * `findMissingLibraries` all decide by this, so it lives once; so does the SCORM writer, for the + * zip it writes (#109). + */ +export function reachableFolder( + dependency: LibraryDependency, + names: ReadonlySet, + available: Map, + downgrade = false +): string | undefined { + return ( + libraryFolderNames(dependency).find((folder) => names.has(`${folder}/library.json`)) ?? + resolveLibraryFolder( + available, + { + machineName: dependency.machineName, + major: Number(dependency.majorVersion), + minor: Number(dependency.minorVersion) + }, + downgrade + ) + ) +} + +/** + * The dependencies `h5p.json` declares that the archive does not actually carry. Pure, so the + * rule can be checked without building a zip. + */ +export function findMissingLibraries( + entryNames: ReadonlySet, + manifest: PackageManifest, + downgrade = false +): LibraryDependency[] { + const dependencies = manifest.preloadedDependencies + if (!Array.isArray(dependencies)) return [] + + const available = indexLibraryFolders(entryNames) + + return dependencies.filter((dependency) => reachableFolder(dependency, entryNames, available, downgrade) === undefined) +} + +/** + * The walk the player makes before the runtime boots (`PackageReader.checkDependencies`), and the + * SCORM writer before it writes a zip: from `roots`, what `h5p.json` declares and the parameters + * name, through each library's own `preloadedDependencies`, a level of the tree at a time, so a + * server can answer each level's `library.json` reads from its cache. `locate` says what serves a + * dependency, by `reachableFolder` or a reader's lookup, or `undefined` for nothing; `read` gives + * the `library.json` of what it located, parsed. Returns what nothing served, each dependency once + * by the folder it would be in, and what was served with its `library.json`, for the caller to look + * at: the reader reports the versions served below the minor asked. A `read` that throws ends the + * walk with its error: counted as having no dependencies, the walk would pass a tree it did not see. + */ +export async function walkDependencies( + roots: Iterable, + locate: (dependency: LibraryDependency) => T | undefined, + read: (located: T, dependency: LibraryDependency) => Promise +): Promise<{ missing: LibraryDependency[]; found: Array<{ dependency: LibraryDependency; located: T; library: unknown }> }> { + const missing: LibraryDependency[] = [] + const found: Array<{ dependency: LibraryDependency; located: T; library: unknown }> = [] + const seen = new Set() + let level = [...roots] + + while (level.length > 0) { + const next: LibraryDependency[] = [] + await Promise.all( + level.map(async (dependency) => { + const key = libraryFolderNames(dependency)[0] + if (seen.has(key)) return + seen.add(key) + const located = locate(dependency) + if (located === undefined) { + missing.push(dependency) + return + } + const library = await read(located, dependency) + found.push({ dependency, located, library }) + next.push(...dependenciesIn(library)) + }) + ) + level = next + } + + return { missing, found } +} diff --git a/packages/player/src/sw/library-load.ts b/packages/player/src/sw/library-load.ts index 9794da20..fd721c5c 100644 --- a/packages/player/src/sw/library-load.ts +++ b/packages/player/src/sw/library-load.ts @@ -1,6 +1,5 @@ import type { LibraryLoad } from '../shared/protocol.js' -import { dependenciesIn, type LibraryDependency } from '../shared/content-libraries.js' -import type { PackageManifest } from './package-reader.js' +import { dependenciesIn, type LibraryDependency, type PackageManifest } from '../shared/content-libraries.js' /** * What the frame loads, and in what order: every library the content needs, its scripts and its diff --git a/packages/player/src/sw/mount.ts b/packages/player/src/sw/mount.ts index 3e485cb4..6344d457 100644 --- a/packages/player/src/sw/mount.ts +++ b/packages/player/src/sw/mount.ts @@ -42,7 +42,8 @@ import { contentRange, parseRange, type ByteRange } from '../shared/range.js' import { openSource } from '../shared/source.js' import * as db from '../shared/idb.js' import { dependenciesIn } from '../shared/content-libraries.js' -import { type IndexedEntry, type LocatedEntry, PackageReader, explainMissingLibraries, libraryFolderNames, locationOf } from './package-reader.js' +import { libraryFolderNames } from '../shared/library-folders.js' +import { type IndexedEntry, type LocatedEntry, PackageReader, explainMissingLibraries, locationOf } from './package-reader.js' import { libraryLoad, walkLibraries } from './library-load.js' import { STORED } from '../shared/local-header.js' import { CONTENT_CHECK_HEADER, buildFrameDocument, createNonce } from './frame-document.js' diff --git a/packages/player/src/sw/pack-plan.ts b/packages/player/src/sw/pack-plan.ts index cde09250..adc5844d 100644 --- a/packages/player/src/sw/pack-plan.ts +++ b/packages/player/src/sw/pack-plan.ts @@ -1,5 +1,6 @@ import type { PackDependency, PackLibrary, PackPart, PackPlan, RefreshedLibrary } from '../shared/protocol.js' -import { addonPatterns, textMatches } from '../shared/content-libraries.js' +import { addonPatterns, dependenciesIn, textMatches } from '../shared/content-libraries.js' +import { parseLibraryFolder } from '../shared/library-folders.js' /** * Which parts of the library pack a package needs, and which of its own library folders the @@ -54,6 +55,54 @@ export interface ProvidedLibrary { minor: number } +/** + * What a package carries of its libraries, for the plan, from its library folders: the top-level + * folders holding a `library.json`. With `read`, each folder's `library.json` gives the version, + * patch included, and the dependencies; without it — a `library.json` read is two ranged requests + * over HTTP until the warm has cached it — the folder's name gives the version and the rest is + * left unknown, which leaves the library as it is and its dependencies to the next look; a folder + * whose name gives no version, an unversioned one, is the library at a version unknown, which is + * how the runtime takes it too, falling back to the bare name. A `library.json` that does not + * parse, or a folder named for another library, is left out. `PackageReader.ownLibraries` and the + * SCORM writer both take what a package carries through here (#109). + */ +export async function ownLibraries(folders: Iterable, read?: (folder: string) => Promise): Promise { + const version = (part: unknown) => (typeof part === 'number' && Number.isInteger(part) && part >= 0 ? part : typeof part === 'string' && /^\d+$/.test(part) ? Number(part) : null) + const libraries: OwnLibrary[] = [] + for (const folder of folders) { + const parsed = parseLibraryFolder(folder) + if (!read) { + libraries.push( + parsed + ? { folder, machineName: parsed.machineName, major: parsed.major, minor: parsed.minor, patch: null, dependencies: null } + : { folder, machineName: folder, major: null, minor: null, patch: null, dependencies: null } + ) + continue + } + const library = (await Promise.resolve() + .then(() => read(folder)) + .catch(() => null)) as Record | null + const major = version(library?.majorVersion) + const minor = version(library?.minorVersion) + const machineName = library?.machineName + if (typeof machineName !== 'string' || major === null || minor === null) continue + if (parsed ? parsed.machineName !== machineName || parsed.major !== major : folder !== machineName) continue + libraries.push({ + folder, + machineName, + major, + minor, + patch: version(library?.patchVersion) ?? 0, + dependencies: dependenciesIn(library).map(({ machineName: name, majorVersion, minorVersion }) => ({ + machineName: name, + majorVersion: Number(majorVersion), + minorVersion: Number(minorVersion) + })) + }) + } + return libraries +} + export interface PlanInput { own: OwnLibrary[] provided: ProvidedLibrary[] diff --git a/packages/player/src/sw/package-reader.ts b/packages/player/src/sw/package-reader.ts index 7d4a1b0b..09562d3b 100644 --- a/packages/player/src/sw/package-reader.ts +++ b/packages/player/src/sw/package-reader.ts @@ -19,8 +19,9 @@ import { deferredStream, inflated } from '../shared/stream-utils.js' import type { ForwardEntry, ForwardIndexSnapshot } from '../shared/forward-index.js' import { indexFingerprint, librariesFingerprint, revisionOf } from '../shared/revision.js' import { contentMetadata, type ContentMetadata } from '../shared/metadata.js' -import { addonPatterns, dependenciesIn, librariesNamedIn, textMatches, type LibraryDependency } from '../shared/content-libraries.js' -import type { OwnLibrary } from './pack-plan.js' +import { addonPatterns, dependenciesIn, librariesNamedIn, textMatches, type LibraryDependency, type PackageManifest } from '../shared/content-libraries.js' +import { findMissingLibraries, indexLibraryFolders, libraryFolderNames, parseLibraryFolder, reachableFolder, resolveLibraryFolder, walkDependencies, type LibraryFolder } from '../shared/library-folders.js' +import { ownLibraries as carriedLibraries, type OwnLibrary } from './pack-plan.js' /** * The package reader: a zip central directory turned into a name → entry index, plus the decision @@ -36,13 +37,6 @@ export type Strategy = /** Large and deflated (method 8): the Jobs worker inflates into chunks, served progressively. */ | { kind: 'chunked' } -/** The parts of `h5p.json` the player needs. Everything in it comes from an untrusted archive. */ -export interface PackageManifest { - title?: string - mainLibrary?: string - preloadedDependencies?: LibraryDependency[] -} - /** A library that names, in its `library.json`'s `addTo`, the content the platforms add it to. */ export interface Addon extends LibraryDependency { patterns: RegExp[] @@ -365,48 +359,12 @@ export class PackageReader { } /** - * What the package carries of its libraries, for the pack's plan. With `read`, each folder's - * `library.json` gives the version, patch included, and the dependencies; without it — a - * `library.json` read would be two ranged requests — the folder's name gives the version and - * the rest is left unknown, which leaves the library as it is and its dependencies to the next - * look; a folder whose name gives no version, an unversioned one, is the library at a version - * unknown, which is how the runtime takes it too, falling back to the bare name. A - * `library.json` that does not parse, or a folder named for another library, is left out. + * What the package carries of its libraries, for the pack's plan: the plan's `ownLibraries` over + * this archive's library folders, reading each folder's `library.json` through `read` when the + * caller can afford to — over HTTP each read is two ranged requests until the warm has cached it. */ async ownLibraries(read?: (located: LocatedEntry) => Promise): Promise { - const libraries: OwnLibrary[] = [] - for (const folder of this.ownLibraryFolders()) { - if (!read) { - const parsed = parseLibraryFolder(folder) - libraries.push( - parsed - ? { folder, machineName: parsed.machineName, major: parsed.major, minor: parsed.minor, patch: null, dependencies: null } - : { folder, machineName: folder, major: null, minor: null, patch: null, dependencies: null } - ) - continue - } - const library = (await read({ reader: this, entry: this.entries.get(`${folder}/library.json`)! }).catch(() => null)) as Record | null - const version = (part: unknown) => (typeof part === 'number' && Number.isInteger(part) && part >= 0 ? part : typeof part === 'string' && /^\d+$/.test(part) ? Number(part) : null) - const major = version(library?.majorVersion) - const minor = version(library?.minorVersion) - const machineName = library?.machineName - if (typeof machineName !== 'string' || major === null || minor === null) continue - const parsed = parseLibraryFolder(folder) - if (parsed ? parsed.machineName !== machineName || parsed.major !== major : folder !== machineName) continue - libraries.push({ - folder, - machineName, - major, - minor, - patch: version(library?.patchVersion) ?? 0, - dependencies: dependenciesIn(library).map(({ machineName: name, majorVersion, minorVersion }) => ({ - machineName: name, - majorVersion: Number(majorVersion), - minorVersion: Number(minorVersion) - })) - }) - } - return libraries + return carriedLibraries(this.ownLibraryFolders(), read && ((folder) => read({ reader: this, entry: this.entries.get(`${folder}/library.json`)! }))) } /** The library folders the attached archives provide, a part's or a bundle's. */ @@ -634,49 +592,40 @@ export class PackageReader { read: (located: LocatedEntry) => Promise, offered: ReadonlyArray<{ machineName: string; majorVersion: number }> = [] ): Promise<{ missing: MissingLibraries | null; downgraded: DowngradedLibrary[] }> { - const missing: string[] = [] - const downgraded = new Map() - const seen = new Set() const available = this.availableLibraries() const sourced = ({ machineName, majorVersion }: LibraryDependency) => (available.get(machineName) ?? []).some(({ major }) => major === Number(majorVersion)) || offered.some((library) => library.machineName === machineName && library.majorVersion === Number(majorVersion)) - let level = [...dependenciesIn(manifest), ...(await this.contentLibraries()).filter(sourced)] - - while (level.length > 0) { - const next: LibraryDependency[] = [] - await Promise.all( - level.map(async (dependency) => { - const folders = libraryFolderNames(dependency) - if (seen.has(folders[0])) return - seen.add(folders[0]) - const located = folders.map((folder) => this.get(`${folder}/library.json`)).find(Boolean) - if (!located) { - missing.push(folders[0]) - return - } - let library: unknown - try { - library = await read(located) - } catch (error) { - // Counted as having no dependencies, the walk would pass a tree it did not see, and the - // runtime would die on the same file with "Unexpected end of JSON input" and no name. - const reason = error instanceof Error ? error.message : String(error) - throw new PlayerError('bad-archive', `${folders[0]}/library.json could not be read: ${reason}`, { cause: error }) - } - const older = olderThanAsked(dependency, located, library) - if (older) downgraded.set(`${older.library} ${older.from}`, older) - next.push(...dependenciesIn(library)) - }) - ) - level = next - } + const roots = [...dependenciesIn(manifest), ...(await this.contentLibraries()).filter(sourced)] + + // The walk is `walkDependencies`, which the SCORM writer runs over a zip too (#109); what it + // locates here goes through `get`, so an attached archive and a compatible newer minor count. + const { missing, found } = await walkDependencies( + roots, + (dependency) => libraryFolderNames(dependency).map((folder) => this.get(`${folder}/library.json`)).find(Boolean), + async (located, dependency) => { + try { + return await read(located) + } catch (error) { + // Counted as having no dependencies, the walk would pass a tree it did not see, and the + // runtime would die on the same file with "Unexpected end of JSON input" and no name. + const reason = error instanceof Error ? error.message : String(error) + throw new PlayerError('bad-archive', `${libraryFolderNames(dependency)[0]}/library.json could not be read: ${reason}`, { cause: error }) + } + } + ) + const downgraded = new Map() + for (const { dependency, located, library } of found) { + const older = olderThanAsked(dependency, located, library) + if (older) downgraded.set(`${older.library} ${older.from}`, older) + } const listed = [...downgraded.values()].sort((a, b) => a.library.localeCompare(b.library) || a.from.localeCompare(b.from)) if (missing.length === 0) return { missing: null, downgraded: listed } - const carried = this.withheld.size > 0 && missing.every((folder) => this.withheld.has(`${folder}/library.json`)) + const folders = missing.map((dependency) => libraryFolderNames(dependency)[0]).sort() + const carried = this.withheld.size > 0 && folders.every((folder) => this.withheld.has(`${folder}/library.json`)) return { - missing: { mainLibrary: manifest.mainLibrary, folders: missing.sort(), all: false, ...(carried ? { carried } : {}) }, + missing: { mainLibrary: manifest.mainLibrary, folders, all: false, ...(carried ? { carried } : {}) }, downgraded: listed } } @@ -1049,130 +998,6 @@ function olderThanAsked(dependency: LibraryDependency, located: LocatedEntry, li return { library: dependency.machineName, from: `${wanted.major}.${wanted.minor}`, to: `${served.major}.${served.minor}.${patchVersion}` } } -/** - * The two folder names a library may be stored under: the versioned one, and the bare machine - * name older packages use. Each is tried in that order, as h5p-standalone tried them. - */ -export function libraryFolderNames(dependency: LibraryDependency): string[] { - return [ - `${dependency.machineName}-${dependency.majorVersion}.${dependency.minorVersion}`, - dependency.machineName - ] -} - -/** - * The folder a dependency is served from: its name as written, or the bare machine name - * h5p-standalone falls back to, when that folder's `library.json` is among `names`; else a - * compatible newer minor of the same major from `available`, the rule `get()` serves library - * files by, or with `downgrade` an older one. `bootReady`, `mergedManifest` and - * `findMissingLibraries` all decide by this, so it lives once. - */ -function reachableFolder( - dependency: LibraryDependency, - names: ReadonlySet, - available: Map, - downgrade = false -): string | undefined { - return ( - libraryFolderNames(dependency).find((folder) => names.has(`${folder}/library.json`)) ?? - resolveLibraryFolder( - available, - { - machineName: dependency.machineName, - major: Number(dependency.majorVersion), - minor: Number(dependency.minorVersion) - }, - downgrade - ) - ) -} - -export interface LibraryFolder { - machineName: string - major: number - minor: number - folder: string -} - -/** Splits `H5P.Text-1.1` into its parts, or returns `null` for a folder that is not a library. */ -export function parseLibraryFolder(folder: string): LibraryFolder | null { - const match = /^(.+)-(\d+)\.(\d+)$/.exec(folder) - if (!match) return null - return { - machineName: match[1], - major: Number(match[2]), - minor: Number(match[3]), - folder - } -} - -/** Every library folder in a set of entry names, grouped by machine name. */ -export function indexLibraryFolders(entryNames: Iterable): Map { - const byName = new Map() - - for (const name of entryNames) { - const slash = name.indexOf('/') - if (slash < 0 || name.slice(slash + 1) !== 'library.json') continue - - const parsed = parseLibraryFolder(name.slice(0, slash)) - if (!parsed) continue - - const existing = byName.get(parsed.machineName) - if (existing) existing.push(parsed) - else byName.set(parsed.machineName, [parsed]) - } - - return byName -} - -/** - * Picks the folder that satisfies a dependency: the same major version, and a minor at least as - * high as the one asked for. H5P bumps the major for a change content cannot follow, and gives a - * minor that changed the shape of a library's parameters a step in its `upgrades.js`; the frame - * runs those steps on content written for an older minor before the runtime starts - * (`content-upgrade.ts` in the runtime package), as the platforms do when they install a newer one. - * - * This matters whenever libraries come from somewhere other than the package: the library pack ships - * the current `H5P.Text-1.1` while content authored earlier asks for `H5P.Text-1.0`, and without - * this every such request is a 404. - * - * The lowest satisfying version wins, as the closest to what the content was written against. - * - * With `downgrade`, a dependency nothing satisfies is served by the newest older minor of the same - * major: content made on h5p.com after a minor bump the library pack has not caught up with (#85). - * Three bumps in four add fields the older code ignores; the fourth moved some, which the older - * code then does not find. The element asks for it only once every library source has been tried - * for the version named, and never for an archive still arriving, which may yet carry it. - */ -export function resolveLibraryFolder( - available: Map, - dependency: { machineName: string; major: number; minor: number }, - downgrade = false -): string | undefined { - const sameMajor = (available.get(dependency.machineName) ?? []).filter((candidate) => candidate.major === dependency.major) - const satisfying = sameMajor.filter((candidate) => candidate.minor >= dependency.minor).sort((a, b) => a.minor - b.minor) - if (satisfying.length > 0 || !downgrade) return satisfying[0]?.folder - - return [...sameMajor].sort((a, b) => b.minor - a.minor)[0]?.folder -} - -/** - * The dependencies `h5p.json` declares that the archive does not actually carry. Pure, so the - * rule can be checked without building a zip. - */ -export function findMissingLibraries( - entryNames: ReadonlySet, - manifest: PackageManifest, - downgrade = false -): LibraryDependency[] { - const dependencies = manifest.preloadedDependencies - if (!Array.isArray(dependencies)) return [] - - const available = indexLibraryFolders(entryNames) - - return dependencies.filter((dependency) => reachableFolder(dependency, entryNames, available, downgrade) === undefined) -} - /** * What a package declares but does not carry, or `null` when it carries everything. * diff --git a/packages/player/tests/package-types.ts b/packages/player/tests/package-types.ts index 91487af6..1ed4f373 100644 --- a/packages/player/tests/package-types.ts +++ b/packages/player/tests/package-types.ts @@ -8,7 +8,7 @@ */ import { H5PPlayerElement, type PackageMetadata, type PlayerErrorDetail, type ReadyDetail } from '@missing-elements/h5p-offline-player' import { mountH5P } from '@missing-elements/h5p-offline-player/sw' -import { librariesNamedIn, planPack, type PackPlan } from '@missing-elements/h5p-offline-player/plan' +import { librariesNamedIn, ownLibraries, planPack, reachableFolder, walkDependencies, type PackPlan } from '@missing-elements/h5p-offline-player/plan' export function asACallerWould(worker: ServiceWorkerGlobalScope): string[] { const player: H5PPlayerElement | null = document.querySelector('h5p-player') @@ -19,6 +19,11 @@ export function asACallerWould(worker: ServiceWorkerGlobalScope): string[] { const plan: PackPlan | null = planPack({ own: [], provided: [], roots, params: {}, pack: [], refresh: false, fill: true }) // @ts-expect-error the parts are a list const parts: PackPlan['parts'] = 'H5P.Blanks-1.14' + // The folder rule and the walk, as embed's SCORM writer runs them over a zip. + const names = new Set(['H5P.Blanks-1.14/library.json']) + const folder: string | undefined = reachableFolder(roots[0], names, new Map()) + const walked: Promise<{ missing: { machineName: string }[] }> = walkDependencies(roots, (dependency) => reachableFolder(dependency, names, new Map()), async (located: string) => ({ located })) + const own: Promise<{ folder: string; major: number | null }[]> = ownLibraries(['H5P.Blanks-1.14']) // @ts-expect-error not an error code const code: PlayerErrorDetail['code'] = 'offline' @@ -29,6 +34,6 @@ export function asACallerWould(worker: ServiceWorkerGlobalScope): string[] { // @ts-expect-error the title is a string const title: PackageMetadata['title'] = 1 - void [player, code, folders, source, title, plan, parts] + void [player, code, folders, source, title, plan, parts, folder, walked, own] return player?.libraryBundle?.refreshed?.map(({ library, to }) => `${library} ${to}`) ?? [] } diff --git a/packages/player/tests/unit/libraries.test.ts b/packages/player/tests/unit/libraries.test.ts index 8bd1c74f..dd770492 100644 --- a/packages/player/tests/unit/libraries.test.ts +++ b/packages/player/tests/unit/libraries.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest' -import { explainMissingLibraries, findMissingLibraries, servedWhenUntrusted } from '../../src/sw/package-reader' +import { findMissingLibraries } from '../../src/shared/library-folders' +import { explainMissingLibraries, servedWhenUntrusted } from '../../src/sw/package-reader' /** * An export that carries content but not the libraries it names is the single most common broken diff --git a/packages/player/tests/unit/library-versions.test.ts b/packages/player/tests/unit/library-versions.test.ts index 21bba9ce..85382b61 100644 --- a/packages/player/tests/unit/library-versions.test.ts +++ b/packages/player/tests/unit/library-versions.test.ts @@ -3,8 +3,12 @@ import { findMissingLibraries, indexLibraryFolders, parseLibraryFolder, - resolveLibraryFolder -} from '../../src/sw/package-reader' + reachableFolder, + resolveLibraryFolder, + walkDependencies +} from '../../src/shared/library-folders' +import type { LibraryDependency } from '../../src/shared/content-libraries' +import { ownLibraries } from '../../src/sw/pack-plan' /** * H5P's compatibility rule: a library satisfies a dependency when the machine name and major @@ -113,3 +117,82 @@ describe('findMissingLibraries with compatible versions', () => { expect(findMissingLibraries(names, manifest, true)).toEqual([]) }) }) + +describe('reachableFolder', () => { + // The SCORM writer's own `servingFolder` until #109, now the one rule it and the reader share: + // the folder as named or the bare name, else the lowest newer minor of the same major, never + // another major, and with `downgrade` the newest older minor. + const names = new Set(['H5P.Text-1.3/library.json', 'H5P.Text-1.5/library.json', 'H5P.Text-2.0/library.json', 'Bare/library.json']) + const available = indexLibraryFolders(names) + + it('serves the folder named, or the bare name, before any other', () => { + expect(reachableFolder({ machineName: 'H5P.Text', majorVersion: 1, minorVersion: 3 }, names, available)).toBe('H5P.Text-1.3') + expect(reachableFolder({ machineName: 'Bare', majorVersion: '3', minorVersion: '0' }, names, available)).toBe('Bare') + }) + + it('takes the lowest newer minor of the same major, and only that', () => { + expect(reachableFolder({ machineName: 'H5P.Text', majorVersion: 1, minorVersion: 1 }, names, available)).toBe('H5P.Text-1.3') + expect(reachableFolder({ machineName: 'H5P.Text', majorVersion: 1, minorVersion: 4 }, names, available)).toBe('H5P.Text-1.5') + expect(reachableFolder({ machineName: 'H5P.Text', majorVersion: 1, minorVersion: 6 }, names, available)).toBeUndefined() + expect(reachableFolder({ machineName: 'H5P.Text', majorVersion: 1, minorVersion: 6 }, names, available, true)).toBe('H5P.Text-1.5') + }) +}) + +describe('walkDependencies', () => { + const tree: Record = { + 'H5P.Column-1.22': { preloadedDependencies: [{ machineName: 'H5P.TrueFalse', majorVersion: 1, minorVersion: 8 }, { machineName: 'H5P.Text', majorVersion: 1, minorVersion: 1 }] }, + 'H5P.TrueFalse-1.8': { preloadedDependencies: [{ machineName: 'H5P.Question', majorVersion: '1', minorVersion: '5' }, { machineName: 'H5P.Text', majorVersion: 1, minorVersion: 1 }] }, + 'H5P.Text-1.1': {}, + 'H5P.Question-1.5': { preloadedDependencies: [{ machineName: 'FontAwesome', majorVersion: 4, minorVersion: 5 }] } + } + const names = new Set(Object.keys(tree).map((folder) => `${folder}/library.json`)) + const available = indexLibraryFolders(names) + const locate = (dependency: LibraryDependency) => reachableFolder(dependency, names, available) + const roots = [{ machineName: 'H5P.Column', majorVersion: 1, minorVersion: 22 }] + + it('walks the tree a level at a time, reads each library.json once, and names what nothing serves', async () => { + const reads: string[] = [] + const { missing, found } = await walkDependencies(roots, locate, async (folder) => { + reads.push(folder) + return tree[folder] + }) + expect(missing).toEqual([{ machineName: 'FontAwesome', majorVersion: 4, minorVersion: 5 }]) + expect(found.map(({ located }) => located).sort()).toEqual(['H5P.Column-1.22', 'H5P.Question-1.5', 'H5P.Text-1.1', 'H5P.TrueFalse-1.8']) + expect(reads.sort()).toEqual(['H5P.Column-1.22', 'H5P.Question-1.5', 'H5P.Text-1.1', 'H5P.TrueFalse-1.8']) + }) + + it('ends with the error of a read that fails, rather than passing a tree it did not see', async () => { + await expect( + walkDependencies(roots, locate, async () => { + throw new Error('not JSON') + }) + ).rejects.toThrow('not JSON') + }) +}) + +describe('ownLibraries', () => { + const files: Record = { + 'H5P.Text-1.1': { machineName: 'H5P.Text', majorVersion: 1, minorVersion: 1, patchVersion: 7, preloadedDependencies: [{ machineName: 'H5P.Base', majorVersion: '1', minorVersion: '0' }] }, + Bare: { machineName: 'Bare', majorVersion: '2', minorVersion: '3' }, + 'H5P.Other-1.0': { machineName: 'H5P.Mismatch', majorVersion: 1, minorVersion: 0 }, + 'Broken-1.0': null + } + const read = async (folder: string) => { + if (files[folder] === null) throw new Error('not JSON') + return files[folder] + } + + it("reads each folder's library.json for its version, patch and dependencies, leaving out one that does not parse or names another library", async () => { + expect(await ownLibraries(Object.keys(files), read)).toEqual([ + { folder: 'H5P.Text-1.1', machineName: 'H5P.Text', major: 1, minor: 1, patch: 7, dependencies: [{ machineName: 'H5P.Base', majorVersion: 1, minorVersion: 0 }] }, + { folder: 'Bare', machineName: 'Bare', major: 2, minor: 3, patch: 0, dependencies: [] } + ]) + }) + + it("without a read, takes the version from a versioned folder's name and leaves the rest unknown", async () => { + expect(await ownLibraries(['H5P.Text-1.1', 'Bare'])).toEqual([ + { folder: 'H5P.Text-1.1', machineName: 'H5P.Text', major: 1, minor: 1, patch: null, dependencies: null }, + { folder: 'Bare', machineName: 'Bare', major: null, minor: null, patch: null, dependencies: null } + ]) + }) +}) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b0312d1a..b1819892 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -135,7 +135,7 @@ importers: specifier: workspace:^ version: link:../libraries '@missing-elements/h5p-offline-player': - specifier: workspace:^ + specifier: workspace:* version: link:../player '@missing-elements/h5p-runtime': specifier: workspace:^