diff --git a/AGENTS.md b/AGENTS.md index c8a29458..55d37959 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,21 +1,19 @@ # AGENTS.md -Working notes for this repository. This file is the index: what the project is, how to work in -it, and which file holds each area's rules. [`design/`](design/) holds the rules in full, with the -reasons and the measurements behind them, one file per area; together they are the authority on -the design, and where they and the code disagree, one of them is wrong. **Before changing code in -an area, read its file in `design/`.** A name in italics, *like this*, is a rule or a section -there: grep `design/` for it. The README and `docs/` are written for people integrating the -packages. +The index for working in this repository. [`design/`](design/) holds the rules in full, one file +per area, with the reasons and the measurements behind them; it is the authority on the design, and +where it and the code disagree, one of them is wrong. **Before changing code in an area, read its +file in `design/`:** its rules are the bold sentences there, and its opening says what it +covers. A name in italics, *like this*, is a rule or a section there: grep `design/` for it. The +README and `docs/` are written for people integrating the packages. ## What this is -Self-hosted H5P without an H5P server: a browser-only player, shipped as one web component, -that a host puts on infrastructure it controls. That is the product's main benefit, and the -documents lead with it. It plays an arbitrary `.h5p` archive -fetched from a URL or picked from disk, with no server-side code and no extraction step: the zip -is read in place and a Service Worker serves its entries to the H5P runtime over a virtual file -server. +Self-hosted H5P without an H5P server: a browser-only player, shipped as one web component, that a +host puts on infrastructure it controls. That is the product's main benefit, and the documents lead +with it. It plays an arbitrary `.h5p` archive fetched from a URL or picked from disk, with no +server-side code and no extraction step: the zip is read in place and a Service Worker serves its +entries to the H5P runtime over a virtual file server. The hard constraints that shape every file here: @@ -30,7 +28,9 @@ The hard constraints that shape every file here: ## The three contexts -Most bugs here come from forgetting which context a line runs in. +Most bugs here come from forgetting which context a line runs in. The Service Worker is stateless +by assumption, and asks a page for what only a page can do through the frame: +[`design/serving.md`](design/serving.md). | Context | Lives as long as | Can | Cannot | |---|---|---|---| @@ -38,33 +38,40 @@ Most bugs here come from forgetting which context a line runs in. | Service Worker | One event, unpredictably | Serve requests, read IndexedDB and the chunk store | Run long work, spawn a Worker, hold state across events | | Jobs worker | The tab | Download, inflate, take Web Locks | Intercept requests, reach the DOM | -The Service Worker is **stateless by assumption**. Any handler may be the first one after a -restart, so it rebuilds readers from the `packages` table on demand. When it needs something only -a page can do, it messages the frame client (`event.clientId`), the frame relays to its parent -element, and the element acts. That relay is why `frame-document.ts` has a `message` listener. - -## Where things are - -A pnpm workspace of eleven parts: nine packages and two apps. Eight of the packages are published -to npm; the ninth, the VS Code extension, is a `.vsix`. In `design/`, `src/…` and `scripts/…` mean -the player's unless a file says otherwise. [`design/layout.md`](design/layout.md) maps every file. - -| Part | What it is | -|---|---| -| `packages/player/` | `@missing-elements/h5p-offline-player`, MIT: the `` element, the Service Worker (`src/sw/`), the Jobs worker (`src/jobs/`), what the three contexts share (`src/shared/`), and `./plan`, its library rules for author and embed | -| `packages/runtime/` | `@missing-elements/h5p-runtime`, GPL-3.0: the H5P core vendored at a commit, and the boot script that runs in the frame | -| `packages/libraries/` | `@missing-elements/h5p-libraries`: the library pack, one archive per library, checked against the H5P hub every week | -| `packages/normalize/` | `@missing-elements/h5p-normalize`: rewrites a package so it streams | -| `packages/verify/` | `@missing-elements/h5p-verify`: plays a package headless and reports, or opens it for a person | -| `packages/author/` | `@missing-elements/h5p-author`: a playable package of any content type from a short source | -| `packages/cmi5/` | `@missing-elements/h5p-cmi5`: a page holding the element as a cmi5 assignable unit | -| `packages/embed/` | `@missing-elements/h5p-embed`: the embed site for a player domain, and a package as a SCORM zip | -| `packages/vscode/` | H5P Preview, the VS Code extension | -| `apps/demo/` | the dev server for the whole repository, and the hosted site and its installable app | -| `apps/skills/` | the agent skills' test project | -| `skills/` | the agent skills, at the root, where `npx skills add` looks | -| `docs/` | the README's longer sections, for integrators | -| `design/` | the design rules in full | +## Where things are, and whose rules + +A pnpm workspace of nine packages and two apps. [`design/layout.md`](design/layout.md) maps every +file; in `design/`, `src/…` and `scripts/…` mean the player's unless a file says otherwise. + +| Part | What it is | Rules | +|---|---|---| +| `packages/player/` | the `` element, MIT: the Service Worker in `src/sw/`, the Jobs worker in `src/jobs/`, what they share in `src/shared/` | by area, below | +| `packages/runtime/` | the H5P core, vendored, and the frame's boot script, GPL-3.0 | [frame](design/frame.md) | +| `packages/libraries/` | the library pack, one archive per library | [libraries](design/libraries.md) | +| `packages/normalize/` | rewrites a package so it streams | [normalize](design/normalize.md) | +| `packages/verify/` | plays a package headless and reports | [verify](design/verify.md) | +| `packages/author/`, `skills/h5p-author/` | a playable package of any content type from a short source | [author](design/author.md) | +| `packages/cmi5/` | the element as a cmi5 assignable unit | [cmi5](design/cmi5.md) | +| `packages/embed/`, `skills/h5p-to-scorm/` | the embed site for a player domain, and a package as a SCORM zip | [embed](design/embed.md), [scorm](design/scorm.md) | +| `packages/vscode/` | H5P Preview, the VS Code extension | [vscode](design/vscode.md) | +| `apps/demo/`, `vercel.json`, `api/` | the dev server for the whole repository, the hosted site and its app | [demo](design/demo.md) | +| `apps/skills/`, `skills/` | the agent skills, and the project they are tested from | [testing](design/testing.md) | + +The player, by area: + +- [Serving archives](design/serving.md): the Service Worker's file server, how a source is read, the warm. +- [Storage and the Jobs worker](design/storage.md): the chunk store, eviction and quota, the Jobs worker. +- [Libraries](design/libraries.md): which libraries a package plays with, the pack, refresh, addons. +- [The frame and the runtime](design/frame.md): the frame document, the boot route, the runtime package. +- [Results and saved state](design/results.md): statements and `revision`, `finished`, `resume`. +- [The element](design/element.md): loads, the worker's registration and upgrade, the events, the box. +- [Untrusted packages and security](design/security.md): entry names, `untrusted`, the content check, the frame's policy and messages. + +Across the repository: build shape, types, publishing and Node versions (22 LTS or 24 LTS) in +[`design/build.md`](design/build.md); tests and CI in [`design/testing.md`](design/testing.md); +licensing and names in [`design/licensing.md`](design/licensing.md), where the player is MIT and +carries no GPL code, reaching `@missing-elements/h5p-runtime` only over HTTP and `postMessage`; what +is not built yet in [`design/scope.md`](design/scope.md). ## Commands @@ -84,67 +91,7 @@ pnpm verify course.h5p # play a package in a headless browser and re pnpm author build source.json course.h5p # after pnpm build, or the player's build:plan ``` -Every other command (the demo's content, images and screenshots, the library pack, translations, -the core's sync, the sweep, the VS Code extension) and what to know when running them, such as why -`pnpm test` runs one package at a time: [`design/commands.md`](design/commands.md). - +Every other command, and what to know when running them: [`design/commands.md`](design/commands.md). **What is tested**, by the owner's rule: the packages and their source, and of the demo app only the installable app; the demo's pages are checked by hand, and the skills are tested from -`apps/skills`: [`design/testing.md`](design/testing.md). Node 22 LTS or 24 LTS: -[`design/build.md`](design/build.md). - -## Design rules, by area - -Read the area's file before changing code in it: its rules are the bold sentences there. - -- **Serving archives**, [`design/serving.md`](design/serving.md): before changing the Service - Worker's file server (`src/sw/mount.ts`, `package-reader.ts`, `routes.ts`, `watermark-wait.ts`), - how a source is read (`src/shared/source.ts`, `block-cache.ts`, `source-reader.ts`, - `span-stream.ts`, `resilient-stream.ts`, `range.ts`, `local-header.ts`, `forward-index.ts`, - `mime.ts`) or the warm (`src/jobs/warm.ts`). -- **Storage and the Jobs worker**, [`design/storage.md`](design/storage.md): before changing the - chunk store, eviction or quota (`src/shared/chunk-store.ts`, `eviction.ts`, `locks.ts`, - `storage.ts`, `idb.ts`) or the Jobs worker (`src/jobs/`). -- **Libraries**, [`design/libraries.md`](design/libraries.md): before changing which libraries a - package plays with: the library check, the `libraries` sources, the pack and its parts, refresh, - downgrade, addons, the merged manifest (`src/sw/pack-plan.ts`, `src/shared/library-folders.ts`, - `content-libraries.ts`, `pack.ts`, `src/plan.ts`, the element's `index()`, `packages/libraries/`). -- **The frame and the runtime**, [`design/frame.md`](design/frame.md): before changing the frame - document or the boot route (`src/sw/frame-document.ts`, `library-load.ts`) or the runtime package - (`packages/runtime/`: the boot script, upgrades, translations). -- **Results and saved state**, [`design/results.md`](design/results.md): before changing what - reaches the host: statements and `revision`, `finished`, `resume` (`src/shared/revision.ts`, - `src/user-data-store.ts`, `packages/runtime/src/finished.ts`). -- **The element**, [`design/element.md`](design/element.md): before changing - `src/h5p-offline-player.ts`: loads, the worker's registration and upgrade, the events, the - element's box and its height. -- **Untrusted packages and security**, [`design/security.md`](design/security.md): before changing - entry names, `untrusted`, the content check (`packages/runtime/src/content-check.ts`), the frame's - policy or the messages between the frame and the element. - -## The other packages, the site and the rest - -- **The normalizer** (`packages/normalize/`): [`design/normalize.md`](design/normalize.md). -- **The verifier** (`packages/verify/`): [`design/verify.md`](design/verify.md). -- **The author package** (`packages/author/`, `skills/h5p-author/`): - [`design/author.md`](design/author.md). -- **The VS Code extension** (`packages/vscode/`): [`design/vscode.md`](design/vscode.md). -- **The SCORM package** (`packages/embed/lib/scorm*.mjs`, `packages/embed/scorm/`, the SCORM page, - `skills/h5p-to-scorm/`): [`design/scorm.md`](design/scorm.md). -- **The embed site** (`packages/embed/` but its SCORM side, the demo's `/embed`): - [`design/embed.md`](design/embed.md). -- **cmi5** (`packages/cmi5/`, the demo's cmi5 page, CATAPULT): [`design/cmi5.md`](design/cmi5.md). -- **The hosted demo and the app** (`apps/demo/`, `vercel.json`, `api/`): - [`design/demo.md`](design/demo.md). It compiles the element from the player's source, not from - `dist/`, as [`design/layout.md`](design/layout.md) explains. -- **Build shape, types and publishing** (the artefacts and how each is built, the plain-JS packages' - declarations, Node versions): [`design/build.md`](design/build.md). -- **Tests and CI** (the engines, CI's browsers, the fixtures, the sweep): - [`design/testing.md`](design/testing.md). -- **Licensing and names**: [`design/licensing.md`](design/licensing.md). The player is MIT and - carries no GPL code: the H5P core and the boot script are `@missing-elements/h5p-runtime`, - GPL-3.0, reached only over HTTP and `postMessage`, and every shipped file with third-party code - opens with its notice. A name that leads with "H5P" says who made it ("… by Missing Elements"), as - H5P Group's trademark policy asks. -- **Scope** (what differs from the original architecture document, and what is not built yet): - [`design/scope.md`](design/scope.md). +`apps/skills`. diff --git a/design/author.md b/design/author.md index b1295956..d89aa983 100644 --- a/design/author.md +++ b/design/author.md @@ -15,7 +15,7 @@ already holds text to the content check's rules, from the pack an untrusted chec `apps/skills` plays every example untrusted too. Measured on a five-question quiz, at four characters a token: `content.json` was about 3,470 tokens to write by hand, after reading 3,900 to 9,500 of a type's semantics; the source is 1,000 to 1,800 tokens, written from a type file of 200 -to 3,000. Thirty things the code does not say by itself: +to 3,000. Thirty-two things the code does not say by itself: - **One generator, three ways to run it.** The third, `saved`, checks content as H5P stores it (*Saved content is checked as the build checks what it writes*, below). `ContentGenerator` was the sweep's @@ -111,6 +111,66 @@ to 3,000. Thirty things the code does not say by itself: normalizer's `normalizeArchive`, so it is stored and laid out to stream, an mp4's index first. An addon whose pattern the text matches, H5P.MathDisplay for LaTeX, is carried as a site with it installed exports it. +- **A build that acts on what a model wrote can be confined** (2026-11-17, #220, for the local MCP + server of #219). The command line reads any path a source names and fetches any address, which is + right for a person's own command. `buildPackage` takes `within`, a list of the folders the output + and every local file a source names must be inside, judged on real paths, so `..` and a link out + are refused; one folder given alone, as a string, is refused, since it would be read letter by + letter, `/` among them, and confine nothing. It is judged before the file is looked for, and a + confined build's problems name no path but the one written: the first cut said "outside" of a + file that was there and "no such file" of one that was not, so a model could map the disk around + the folders, and it named every absolute path, the user's name with it. A file the build cannot + read is a problem of the source's as well, found where the file is looked for, and not a failure + halfway through that names its path. `check` builds and writes no package (`build --check`), + running the same check as a build, `checkContent` being `composePackage`'s, and takes no output; + a media file named by web address is still downloaded, to a temporary folder removed after, as + the check reads its bytes, and a video on YouTube, Vimeo, Panopto or Echo360 stays an address, as + in a build. `publicFetch`, passed as `fetch`, refuses every network that is not the + public internet on every hop, through `net.BlockList`, which matches an IPv4 address however IPv6 + writes it: the first cut checked dotted text, and a URL writes `[::ffff:127.0.0.1]` as + `[::ffff:7f00:1]`, which reached a server on loopback. The local IPv4 ranges are refused behind + NAT64's prefix and in 6to4 too, which a gateway carries to an IPv4 network, and with them the IPv6 + ranges no web host is on: IPv4-translated, Teredo, benchmarking, documentation, discard-only and + segment routing's. A name that does not resolve is refused in the words a local one is, so a + refusal does not tell which names a local network has. A redirect to another origin drops + `Authorization`, `Proxy-Authorization` and `Cookie`, as `fetch` does, and the signal stops a + lookup, which takes none of its own. It resolves a name and then asks, so it guards against a + model's mistake or a planted address, not a hostile network. Nor is `within` a guard against + another process racing on the folders, swapping a path for a link between its check and its use: + one that can write there already acts as the user, and Node has no directory handles to hold a + path while it is used. +- **The outline is the content for a review, read through the semantics** (2026-11-17, #220). Every + result has `outline`, all of it, made when it is first read, so a build that never reads it never + runs it. `outlineOf(libraries, content)` takes its arguments as `checkContent` does, refuses them + the other way round rather than answer with nothing, and cuts at a `limit` only where one is + given. It writes the text fields a learner reads that hold more than their default, which in + loaded semantics is the content language's, so interface texts drop out in any language; a rich + text's blocks and line breaks apart, by ` / `, and a tag inside a word taken out with nothing in + its place (`H2O`); a ✓ on a `correct` item, a `correct` select's answer, a score range + before its feedback in a RangeList. Settings are left out, and with them a group or a list the + editor does not draw (Game Map's stage positions) and a text no label names, which a widget of the + type's own sets (Find the Hotspot's figure, Virtual Tour's positions): by a scan of every library + in the pack, the only unlabelled texts but a list's items and those `TYPES` labels. What a few + types say only by position or number is that one table, `TYPES`: Single Choice Set's and + Summary's first item is the right one, as their semantics say in words alone; slides, chapters, + questions and a scenario's content are numbered; a question's title the build gave + (`questionTitle`, the set's own label and the number, shared with its layout) is not written; a + scenario's links name those numbers, a question's own link left out, since its alternatives lead; + an Interactive Video's interactions and bookmarks start with their times, a bookmark named as + one. The first cut read keys, not semantics: it dropped every number ("1969" as an answer), hid + True/False's answer, gave a Question Set 29 lines of which 23 were interface texts, ran a heading + into its paragraph, and showed no slide, link or time. The tags a text holds are taken out by + `stripTags`, one function in `text.mjs` for the generator, the layouts, `schema` and the outline, + which reads markup in the content check's pieces (`PIECE`, as `markupText` checks a field): a tag + runs from a `<` before a letter, `/` or `!` to the first `>`, and each is a space, never nothing, + which is what CodeQL asks of a tag remover (alert #46 on the first cut). A second cut read quotes + in a tag as a browser does, and took the apostrophe of an unquoted attribute (`href=it's`) for the + start of a quoted one, running a link's tag into the text after it. Read as the check reads it, a + `<` inside an attribute no longer cuts a tag short and a `<` that starts none is text: the one + change to what a build writes without the options, its result having `outline` besides, where the + generator's check for an empty field and the layouts' measures had read part of a tag as text, or + a text as a tag. The outline splits rich text at its tags (`tagsIn`) and plain text at its line + breaks before it reads entities, so no text, an encoded one included, can pass for a break. - **The pack is pinned.** The dependency is `workspace:*`, published as the exact version packages/libraries has, so a release builds with the libraries it was tested with and one command needs no project set up; `--libraries` names another `parts/` folder. Each part is checked diff --git a/design/layout.md b/design/layout.md index f028378a..9fc59700 100644 --- a/design/layout.md +++ b/design/layout.md @@ -178,13 +178,17 @@ packages/author/ @missing-elements/h5p-author — a playable .h5p of an libraries and content written into a zip.js writer lib/node.mjs what only Node does, and the one module that imports `node:`: the pack read from disk, a source's files beside it, downloads in a temporary folder, - the package written to a path and normalized (#108) + the package written to a path and normalized (#108); a build confined to + folders (`within`) and to the public internet (`publicFetch`, #220) lib/install.mjs what an H5P site's upload check wants with the libraries that play, at the exact versions (`installClosure`), written after the content; the check itself, `missingForInstall`, for apps/skills and a tool beside the build (#146) lib/check.mjs `checkContent`: content in the form H5P saves it checked, the layouts' `after` run, for a tool that edits a saved package and for the build, which runs it last on what it writes (#107); no Node import of its own + lib/outline.mjs `outlineOf`: a content as plain lines for a review, read through its + libraries' semantics, every build result's `outline` (#220); no Node + import of its own lib/libraries.mjs `loadLibraries`: the libraries content brings in, loaded and localized as the build loads them, for the build and for a tool beside it; `packageLibraries`, a package's own over the pack's (#107); no Node @@ -212,7 +216,8 @@ packages/author/ @missing-elements/h5p-author — a playable .h5p of an carries and git does not (#172, #178) tests/ node tests: the generator, text and translations, layouts and schema, the loader, builds with media from disk and a local server, the command, the - quiz format, the check of saved content, the demo's packages among it, and + quiz format, the check of saved content, the demo's packages among it, a + confined build, the guarded fetch and the outline (confine.test.ts), and the page's build (page.test.ts, against Node's); browser/ the page's build in Chromium, its packages played by the verifier in Node through a Vitest command (`playUntrusted`, vitest.config.ts); fixtures/ the quiz format's diff --git a/design/serving.md b/design/serving.md index 5a2faa0d..2af55247 100644 --- a/design/serving.md +++ b/design/serving.md @@ -4,6 +4,13 @@ How a package's bytes reach the H5P runtime: the Service Worker's virtual file s ## Invariants +- **The Service Worker is stateless by assumption.** Any handler may be the first one after a + restart, so it rebuilds readers from the `packages` table on demand. When it needs something only + a page can do, it posts to the frame client (`event.clientId`, `postToClient` in `src/sw/mount.ts`), + the frame's boot script relays that to its parent element (`packages/runtime/src/frame-boot.ts`), + and the element acts (`handleWorkerRequest`). With no client to answer, as for `need-file` after a + restart, it posts to every window it controls, and only the element whose package it names acts. + The element also hears the worker directly where the page itself is controlled. - **`SourceReader` overrides `createReadable`.** It hands the span to `streamSpan`, which is also what the Jobs worker reads an entry through. Without it zip.js falls back to walking an entry through `readUint8Array` in 64 kB steps. Over a picked `File` each step is a free `slice()`; diff --git a/packages/author/README.md b/packages/author/README.md index a0b62714..99a9f9ea 100644 --- a/packages/author/README.md +++ b/packages/author/README.md @@ -168,6 +168,7 @@ Set's 2.6 MB, a container's up to 8.4 MB. `--all ` writes one per conten - `--libraries ` or `--libraries=`: another copy of the pack's `parts/` folder, in place of the pinned one. +- `build --check `: the problems and notes of a source, with no package written. - `-h`, `--help`; `-v`, `--version`: this package's version and the pack's. ## API @@ -175,12 +176,38 @@ Set's 2.6 MB, a container's up to 8.4 MB. `--all ` writes one per conten ```js import { buildPackage, schemaOf, SourceError } from '@missing-elements/h5p-author' -const { output, manifest, content, libraries, install, translations, notes, media } = +const { output, manifest, content, libraries, install, translations, notes, media, outline } = await buildPackage(source, 'rivers.h5p', { base: process.cwd() }) ``` -`buildPackage(source, output, { libraries, base, fetch })` refuses an existing output, and throws a -`SourceError` whose `problems` are `{ path, message }`; `libraries` are the folders the package +`buildPackage(source, output, { libraries, base, fetch, within, check })` refuses an existing output, +and throws a `SourceError` whose `problems` are `{ path, message }`; with `check` it builds and +writes no package and takes no output (`buildPackage(source, undefined, { check: true })`), though a +media file named by web address is still downloaded, to a temporary folder removed after; a video on +YouTube, Vimeo, Panopto or Echo360 stays an address, as in a build. For a build run +on what a model wrote: `within` is a list of the folders the output and every local file the source +names must be inside, on their real paths, and its problems then name no path but the one the +source wrote. It does not cover a file named by web address, which is downloaded with the `fetch` +given, the global one by default: pass `fetch: publicFetch`, a `fetch` that refuses an address on a +local network however it is written, redirects included. A result's `outline` is the content as +plain lines for a review, made when it is first read: what the author wrote, the right answers +marked, slides, chapters and questions numbered, a scenario's links and a video's times written, +interface texts and settings left out. `outlineOf(libraries, content, { limit })` makes one from +what `loadLibraries` loaded for a content and the content, `{ type, params }`, as `checkContent` +takes them, cut at `limit` lines where one is given. + +```js +import { readFile } from 'node:fs/promises' +import { join, resolve } from 'node:path' +import { buildPackage, publicFetch } from '@missing-elements/h5p-author' + +// A build run on what a model wrote: files only from `folder`, nothing from a local network. +const folder = resolve('work') +const source = JSON.parse(await readFile(join(folder, 'rivers.json'), 'utf8')) +const { outline } = await buildPackage(source, join(folder, 'rivers.h5p'), { base: folder, within: [folder], fetch: publicFetch }) +``` + +A result's `libraries` are the folders the package plays with, `install` those carried for an H5P site's upload check. `missingForInstall(libraries)` is that check, over a package's `library.json` files: what a site with no library installed would refuse it for, empty for a build. `schemaOf(machineName, { markdown, all, diff --git a/packages/author/bin/h5p-author.mjs b/packages/author/bin/h5p-author.mjs index 50126b2b..ba257624 100755 --- a/packages/author/bin/h5p-author.mjs +++ b/packages/author/bin/h5p-author.mjs @@ -7,13 +7,15 @@ import { packVersion } from '../lib/node.mjs' const USAGE = `Usage: h5p-author build [--libraries ] + h5p-author build --check [--libraries ] h5p-author schema [--markdown] [--all] [--libraries ] h5p-author bundle [--libraries ] h5p-author bundle --all [--libraries ] build Makes a playable .h5p from a source: { "type", "title", "language", "license", "authors", "params" }, or a quiz in the h5p-author skill's quiz format. Writes only - when the whole source is right, and never over an existing file. + when the whole source is right, and never over an existing file; --check lists the + problems and notes and writes no package. schema Prints a content type's fields as a source writes them: --markdown as a skill's type file carries them, --all with the interface texts. bundle Writes the libraries a build of one content type reads, as a zip of a parts/ @@ -57,6 +59,7 @@ function parse(argv) { allowPositionals: true, options: { libraries: { type: 'string' }, + check: { type: 'boolean' }, markdown: { type: 'boolean' }, all: { type: 'boolean' }, help: { type: 'boolean', short: 'h' }, @@ -82,8 +85,9 @@ try { for (const flag of Object.keys(flags)) if (!['markdown', 'all', 'libraries'].includes(flag)) throw new Error(`schema does not take --${flag}`) console.log(await schemaOf(rest[0], { markdown: Boolean(flags.markdown), all: Boolean(flags.all), libraries: /** @type {string | undefined} */ (flags.libraries) })) } else if (command === 'build') { - if (rest.length !== 2) throw new Error('build takes a source and an output: h5p-author build quiz.json quiz.h5p') - for (const flag of Object.keys(flags)) if (flag !== 'libraries') throw new Error(`build does not take --${flag}`) + const check = Boolean(flags.check) + if (check ? rest.length !== 1 : rest.length !== 2) throw new Error(check ? 'build --check takes a source: h5p-author build --check quiz.json' : 'build takes a source and an output: h5p-author build quiz.json quiz.h5p') + for (const flag of Object.keys(flags)) if (flag !== 'libraries' && flag !== 'check') throw new Error(`build does not take --${flag}`) const sourcePath = resolve(rest[0]) let source try { @@ -91,10 +95,10 @@ try { } catch (error) { throw new Error(`${rest[0]} is not a readable JSON file: ${error instanceof Error ? error.message : error}`) } - const result = await buildPackage(source, rest[1], { base: dirname(sourcePath), libraries: /** @type {string | undefined} */ (flags.libraries) }) + const result = await buildPackage(source, check ? undefined : rest[1], { base: dirname(sourcePath), libraries: /** @type {string | undefined} */ (flags.libraries), check }) const from = flags.libraries ? `the pack in ${flags.libraries}` : `the library pack ${await packVersion()}` const install = result.install.length ? `, ${result.install.length} more for installing on an H5P site` : '' - console.log(`Built ${result.output}: ${result.manifest.mainLibrary}, ${result.libraries.length} libraries from ${from}${install}${result.media.length ? `, ${result.media.length} media file${result.media.length === 1 ? '' : 's'}` : ''}.`) + console.log(`${check ? 'Checked, no package written' : `Built ${result.output}`}: ${result.manifest.mainLibrary}, ${result.libraries.length} libraries from ${from}${install}${result.media.length ? `, ${result.media.length} media file${result.media.length === 1 ? '' : 's'}` : ''}.`) const language = String(result.manifest.language) if (!/^en(?:-|$)/i.test(language)) { for (const { library, translation, strings, english } of result.translations) { @@ -107,7 +111,7 @@ try { } } for (const { path, message } of result.notes) console.warn(`Note: ${path} ${message}`) - console.log('Not verified: run npx @missing-elements/h5p-verify@latest on it and look at its screenshot before handing it over.') + if (!check) console.log('Not verified: run npx @missing-elements/h5p-verify@latest on it and look at its screenshot before handing it over.') } else if (command === 'bundle') { for (const flag of Object.keys(flags)) if (flag !== 'all' && flag !== 'libraries') throw new Error(`bundle does not take --${flag}`) const libraries = /** @type {string | undefined} */ (flags.libraries) diff --git a/packages/author/lib/author.mjs b/packages/author/lib/author.mjs index 5d579d10..ee5c3385 100644 --- a/packages/author/lib/author.mjs +++ b/packages/author/lib/author.mjs @@ -16,7 +16,7 @@ import { schemaLines, schemaMarkdown } from './schema.mjs' import { LAYOUTS } from './types.mjs' export * from './core.mjs' -export { buildPackage, openPack, writeBundle, writeBundles } from './node.mjs' +export { buildPackage, openPack, publicFetch, writeBundle, writeBundles } from './node.mjs' /** * @typedef {import('./node.mjs').BuildResult} BuildResult diff --git a/packages/author/lib/browser.mjs b/packages/author/lib/browser.mjs index 9d988276..c6594909 100644 --- a/packages/author/lib/browser.mjs +++ b/packages/author/lib/browser.mjs @@ -27,9 +27,10 @@ export * from './core.mjs' * library with its part's URL * @typedef {{ * blob: Blob, manifest: Record, content: Record, libraries: string[], install: string[], - * translations: import('./translate.mjs').TranslationReport[], notes: import('./build.mjs').Problem[], media: string[] + * translations: import('./translate.mjs').TranslationReport[], notes: import('./build.mjs').Problem[], media: string[], outline: string * }} BuildResult the package, and what is worth telling about it: `libraries`, the folders it plays - * with; `install`, those it carries for an H5P site's upload check + * with; `install`, those it carries for an H5P site's upload check; `outline`, the content as text + * for a review (`outlineOf`) * @typedef {{ libraries: Pack | PackIndex | string | URL, files?: Map | Record, fetch?: typeof fetch }} BuildOptions * `libraries`, a pack `openPack` opened, which a page keeps for its builds, or what to open one * from; `files`, the files a source names, by the names it gives them; `fetch`, for media named by @@ -177,7 +178,8 @@ export async function buildPackage(input, options) { install: [...built.install.keys()], translations: built.translations, notes: built.notes, - media: media.files.map((item) => item.name) + media: media.files.map((item) => item.name), + get outline() { return built.outline } } } finally { if (pack !== libraries) await pack.close() diff --git a/packages/author/lib/build.mjs b/packages/author/lib/build.mjs index 9e127240..2b91e102 100644 --- a/packages/author/lib/build.mjs +++ b/packages/author/lib/build.mjs @@ -7,6 +7,7 @@ import { loadLibraries } from './libraries.mjs' import { CC_VERSIONS, LICENSES, ROLES } from './metadata.mjs' import { machineNameOf, NAMING_KEYS, namingKey, nearestNames } from './names.mjs' import { quizProblems, quizSource } from './quiz.mjs' +import { outlineOf } from './outline.mjs' import { SourceError } from './source-error.mjs' import { escapeText } from './text.mjs' import { LAYOUTS, placesOf } from './types.mjs' @@ -264,8 +265,9 @@ async function closure(pack, main, content) { * @param {import('./media.mjs').MediaStore} media * @param {Problem[]} [said] the problems `sourceOf` found in the source's own fields, which come * first among the content's: all of them in one round (#174) - * @returns {Promise<{ manifest: Record, content: Record, selected: Map, install: Map, translations: TranslationReport[], notes: Problem[] }>} - * `selected`, the libraries the package plays with; `install`, those an H5P site's upload check wants with them + * @returns {Promise<{ manifest: Record, content: Record, selected: Map, install: Map, translations: TranslationReport[], notes: Problem[], outline: string }>} + * `selected`, the libraries the package plays with; `install`, those an H5P site's upload check wants with them; `outline`, the + * content as text for a review (`outlineOf`), made when it is first read * @internal */ export async function composePackage(source, pack, media, said = []) { @@ -361,7 +363,14 @@ export async function composePackage(source, pack, media, said = []) { machineName: part.library.machineName, majorVersion: String(part.library.majorVersion), minorVersion: String(part.library.minorVersion) })) } - return { manifest, content, selected, install, translations, notes } + /** @type {string | undefined} */ + let outline + return { + manifest, content, selected, install, translations, notes, + // Made when it is first read: it is for a review, which only some callers make, and a build + // that never reads it never runs it. + get outline() { return (outline ??= outlineOf({ entries, main: mainLibrary }, { params: content })) } + } } /** diff --git a/packages/author/lib/core.mjs b/packages/author/lib/core.mjs index a81a6f53..b43de7af 100644 --- a/packages/author/lib/core.mjs +++ b/packages/author/lib/core.mjs @@ -8,6 +8,7 @@ export { bundleLibraries, contentTypes } from './bundle.mjs' export { checkContent } from './check.mjs' +export { outlineOf } from './outline.mjs' export { ContentGenerator, librariesIn, libraryString } from './generator.mjs' export { missingForInstall } from './install.mjs' export { loadLibraries, packageLibraries } from './libraries.mjs' diff --git a/packages/author/lib/generator.mjs b/packages/author/lib/generator.mjs index ab9b3ffc..7fe6745f 100644 --- a/packages/author/lib/generator.mjs +++ b/packages/author/lib/generator.mjs @@ -1,7 +1,7 @@ import { checkMetadata } from './metadata.mjs' import { machineNameOf, namingKey } from './names.mjs' import { isInterfaceText } from './schema.mjs' -import { allowedTags, markupText, plainText, takesMarkup } from './text.mjs' +import { allowedTags, markupText, plainText, stripTags, takesMarkup } from './text.mjs' /** * Content for a content type, made from its own `semantics.json` as H5P's editor saves a form: @@ -1138,11 +1138,6 @@ function fieldNames(fields) { return written.length && written.length < fields.length ? `${names(written)}, and the interface texts` : names(fields) } -/** @param {string} html */ -function stripTags(html) { - return html.replace(/<[^<>]*>/g, ' ').replace(/\s+/g, ' ').trim() -} - /** @param {any} object @param {string[]} keys */ function omit(object, keys) { return Object.fromEntries(Object.entries(object).filter(([key]) => !keys.includes(key))) diff --git a/packages/author/lib/node.mjs b/packages/author/lib/node.mjs index 855194f7..69c9a707 100644 --- a/packages/author/lib/node.mjs +++ b/packages/author/lib/node.mjs @@ -1,6 +1,9 @@ import { constants, createReadStream, createWriteStream } from 'node:fs' -import { copyFile, lstat, mkdir, mkdtemp, open, readFile, rm, stat } from 'node:fs/promises' -import { basename, dirname, join, resolve } from 'node:path' +import { copyFile, lstat, mkdir, mkdtemp, open, readFile, realpath, rm, stat } from 'node:fs/promises' +import { lookup } from 'node:dns/promises' +import { tmpdir } from 'node:os' +import { BlockList, isIP } from 'node:net' +import { basename, dirname, isAbsolute, join, relative, resolve, sep } from 'node:path' import { Readable, Writable } from 'node:stream' import { fileURLToPath, pathToFileURL } from 'node:url' import { Uint8ArrayReader, ZipWriter } from '@zip.js/zip.js' @@ -23,12 +26,18 @@ import { SourceError } from './source-error.mjs' * @typedef {import('./translate.mjs').TranslationReport} TranslationReport * @typedef {{ * output: string, manifest: Record, content: Record, - * libraries: string[], install: string[], translations: TranslationReport[], notes: Problem[], media: string[] + * libraries: string[], install: string[], translations: TranslationReport[], notes: Problem[], media: string[], outline: string * }} BuildResult what was written, and what is worth telling about it: `libraries`, the folders the - * package plays with; `install`, those carried for an H5P site's upload check - * @typedef {{ libraries?: string, base?: string, fetch?: typeof fetch }} BuildOptions + * package plays with; `install`, those carried for an H5P site's upload check; `outline`, the + * content as text for a review (`outlineOf`) + * @typedef {{ libraries?: string, base?: string, fetch?: typeof fetch, within?: string[], check?: boolean }} BuildOptions * `libraries`, a library pack's parts folder in place of the pinned pack; `base`, the folder a - * source's file paths are read from; `fetch`, for media named by web address + * source's file paths are read from; `fetch`, for media named by web address; `within`, the + * folders the output and every local file a source names must be inside, on their real paths, a + * list even of one; `check`, to build and write no package: no output is needed or looked at, and + * the result's `output` is empty, but a media file named by web address is still downloaded, to a + * temporary folder removed after, as the build checks it; a video on YouTube, Vimeo, Panopto or + * Echo360 stays an address, as in a build * @typedef {{ output: string, type: string, parts: string[], size: number }} BundleResult * `type`, the content type with the version bundled, `H5P.QuestionSet 1.21`; `parts`, the * folders of the parts it holds; `size`, the zip's in bytes @@ -94,21 +103,30 @@ export async function packVersion() { /** * A source's files on disk: beside the source, by their paths from `base`; downloads kept in * `temporary`, numbered. - * @param {{ base: string, temporary: string }} folders + * @param {{ base: string, temporary: string, within?: string[] }} folders `within`, the folders a file must be inside * @returns {import('./media.mjs').MediaAccess} * @internal */ -export function nodeMedia({ base, temporary }) { +export function nodeMedia({ base, temporary, within }) { let downloads = 0 return { what: 'a file beside the source', name: (source) => basename(resolve(base, source)), async local(source) { const from = resolve(base, source) + // Judged before the file is looked for, so a path outside is answered the same whether + // or not something is there, and a confined build names no path but the one written. + if (within && !(await inside(from, within))) return { missing: 'is outside the folders this build may read' } + const found = await stat(from).catch(() => null) + if (!found?.isFile()) return { missing: within ? 'no such file in the folders this build may read' : `no such file beside the source (${from})` } + // Opened once here, so a file the build cannot read is a problem of the source's, said with + // the rest, and not a failure halfway through the build that names its whole path. try { - if ((await stat(from)).isFile()) return { file: from } - } catch {} - return { missing: `no such file beside the source (${from})` } + await (await open(from, 'r')).close() + } catch { + return { missing: within ? 'cannot be read' : `cannot be read (${from})` } + } + return { file: from } }, async keep(body, extension) { downloads += 1 @@ -130,6 +148,169 @@ export function nodeMedia({ base, temporary }) { } } +/** + * Whether a path, as the file system resolves it, a link followed, is in one of the folders. + * A path that does not exist yet is judged by its nearest folder that does. + * @param {string} path + * @param {string[]} folders + */ +export async function inside(path, folders) { + let real = resolve(path) + const tail = [] + for (;;) { + try { + real = join(await realpath(real), ...tail.reverse()) + break + } catch { + const parent = dirname(real) + if (parent === real) return false + tail.push(basename(real)) + real = parent + } + } + for (const folder of folders) { + const root = await realpath(resolve(folder)).catch(() => resolve(folder)) + const from = relative(root, real) + // `..notes.png` is a name in the folder; only `..` itself, or a path through it, leaves it. + if (!isAbsolute(from) && from !== '..' && !from.startsWith(`..${sep}`)) return true + } + return false +} + +/** + * Every network that is not the public internet, as `BlockList` reads it, which matches an IPv4 + * address however it is written: `::ffff:127.0.0.1` and `::ffff:7f00:1` are 127.0.0.1, and a URL + * writes the first as the second. IPv4: this network, private, carrier-grade NAT, loopback, + * link-local, IETF protocol assignments, documentation (TEST-NET-1 to 3), benchmarking, multicast + * and reserved; and each of those as IPv6 carries it to an IPv4 network, behind NAT64's well-known + * prefix (`64:ff9b::/96`, RFC 6052), which a NAT64 gateway translates, and in 6to4 (`2002::/16`). + * IPv6: the unspecified and loopback addresses with the IPv4-compatible ones (`::/96`), the + * IPv4-translated ones (`::ffff:0:0:0/96`, RFC 2765), NAT64's local-use prefix (`64:ff9b:1::/48`, + * RFC 8215), discard-only (`100::/64`), Teredo (`2001::/32`), which carries an IPv4 address, + * benchmarking (`2001:2::/48`), documentation (`2001:db8::/32`, `3fff::/20`), segment routing's + * (`5f00::/16`), unique-local, link-local, site-local and multicast. + */ +const LOCAL_NETWORKS = new BlockList() +for (const [address, prefix] of /** @type {[string, number][]} */ ([ + ['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8], ['169.254.0.0', 16], ['172.16.0.0', 12], + ['192.0.0.0', 24], ['192.0.2.0', 24], ['192.168.0.0', 16], ['198.18.0.0', 15], ['198.51.100.0', 24], ['203.0.113.0', 24], + ['224.0.0.0', 3] +])) { + LOCAL_NETWORKS.addSubnet(address, prefix, 'ipv4') + const [a, b, c, d] = address.split('.').map(Number) + const high = ((a << 8) | b).toString(16) + const low = ((c << 8) | d).toString(16) + LOCAL_NETWORKS.addSubnet(`64:ff9b::${high}:${low}`, 96 + prefix, 'ipv6') + LOCAL_NETWORKS.addSubnet(`2002:${high}:${low}::`, 16 + prefix, 'ipv6') +} +for (const [address, prefix] of /** @type {[string, number][]} */ ([ + ['::', 96], ['::ffff:0:0:0', 96], ['64:ff9b:1::', 48], ['100::', 64], ['2001::', 32], ['2001:2::', 48], ['2001:db8::', 32], ['3fff::', 20], + ['5f00::', 16], ['fc00::', 7], ['fe80::', 10], ['fec0::', 10], ['ff00::', 8] +])) { + LOCAL_NETWORKS.addSubnet(address, prefix, 'ipv6') +} + +/** + * Whether an IP address is on no public network (`LOCAL_NETWORKS`). Text that is no address is + * taken as one, so a check it cannot read refuses. + * @param {string} address + */ +export function isPrivateAddress(address) { + const family = isIP(address) + return family === 0 || LOCAL_NETWORKS.check(address, family === 6 ? 'ipv6' : 'ipv4') +} + +/** + * @typedef {(input: string | URL | Request, init?: RequestInit) => Promise} Fetch + * @typedef {(host: string, options: { all: true }) => Promise<{ address: string }[]>} Lookup + */ + +/** What a request carries for its own site, which a redirect to another does not, as `fetch` drops them. */ +const CREDENTIALS = ['authorization', 'proxy-authorization', 'cookie'] + +/** + * A promise that settles as the one given does, or rejects with the signal's reason once it + * aborts: a lookup takes no signal of its own, so a resolver that hangs would hold a build past + * its download's idle limit. + * @template T + * @param {Promise} promise + * @param {AbortSignal | null | undefined} signal + * @returns {Promise} + */ +function abortable(promise, signal) { + if (!signal) return promise + if (signal.aborted) return Promise.reject(signal.reason) + return new Promise((resolvePromise, reject) => { + const abort = () => reject(signal.reason) + signal.addEventListener('abort', abort, { once: true }) + promise.then(resolvePromise, reject).finally(() => signal.removeEventListener('abort', abort)) + }) +} + +/** + * A `fetch` that refuses an address on a local network (`isPrivateAddress`), on every redirect it + * follows, over the resolver and `fetch` given: the tests' seam. `publicFetch` is this over the + * system's. As `fetch` does, a redirect to another origin drops what the request carried for its + * own: `Authorization`, `Proxy-Authorization` and `Cookie`. + * @param {{ lookup?: Lookup, fetch?: Fetch }} [over] + * @returns {Fetch} + * @internal + */ +export function guardedFetch({ lookup: resolveHost = lookup, fetch: send = (input, init) => fetch(input, init) } = {}) { + return async function guarded(input, init = {}) { + const request = input instanceof Request ? input : undefined + let url = new URL(request ? request.url : String(input)) + const method = (init.method ?? request?.method ?? 'GET').toUpperCase() + if (method !== 'GET' && method !== 'HEAD') throw new TypeError(`fetches media, by GET or HEAD, not by ${method}`) + const mode = init.redirect ?? request?.redirect ?? 'follow' + const headers = new Headers(init.headers ?? request?.headers) + const signal = init.signal ?? request?.signal + const options = { ...init, method, headers, signal, redirect: /** @type {const} */ ('manual') } + for (let hops = 0; hops <= 10; hops++) { + if (!/^https?:$/.test(url.protocol)) throw new TypeError(`${url.protocol} addresses are not fetched`) + const host = url.hostname.replace(/^\[|\]$/g, '') + // An aborted request asks no resolver. + signal?.throwIfAborted() + /** @type {{ address: string }[]} */ + let addresses = [] + if (isIP(host)) addresses = [{ address: host }] + else { + try { + addresses = await abortable(resolveHost(host, { all: true }), signal) + } catch (error) { + if (signal?.aborted) throw error + } + } + // A name that does not resolve is answered as a local one is, so a refusal does not tell + // which names a local network has. + if (!addresses.length || addresses.some(({ address }) => isPrivateAddress(address))) throw new TypeError(`${url.hostname} is not on the public internet, the only place a build here fetches from`) + const response = await send(url, options) + const next = response.headers.get('location') + if (response.status < 300 || response.status >= 400 || !next || mode === 'manual') { + Object.defineProperty(response, 'url', { value: url.href }) + return response + } + await response.body?.cancel() + if (mode === 'error') throw new TypeError(`${url.href} redirects, and redirects were refused`) + const target = new URL(next, url) + if (target.origin !== url.origin) for (const name of CREDENTIALS) headers.delete(name) + url = target + } + throw new TypeError('more than 10 redirects') + } +} + +/** + * A `fetch` for media that refuses an address on a local network, on every redirect it follows: + * for a build that acts on what a model wrote, where the command line's own may reach any host. + * A build uses it only where it is passed as `fetch`. It looks a name up and then asks, so a host + * that answers differently the second time is not covered: it is a guard against a model's + * mistake or a planted address, not against a hostile network. A name that does not resolve is + * refused in the words a local one is, so what it refuses does not tell which names exist on a + * local network. GET and HEAD only, as media is fetched. + */ +export const publicFetch = guardedFetch() + /** * The refusal of an output that exists, with the file system's code, so a caller tells it from * a source's problems. @@ -141,17 +322,39 @@ function exists(output, what = 'a build') { } /** - * Builds a package from a source and writes it to `output`, which must not exist yet. + * `within` as given, checked: a list of folders, or none. One folder given alone, as a string, + * would be read as a list of its characters, `/` among them, which confines nothing. + * @param {unknown} within + * @returns {string[] | undefined} + */ +function foldersOf(within) { + if (within === undefined) return undefined + if (!Array.isArray(within) || !within.every((folder) => typeof folder === 'string' && folder.trim())) { + throw new TypeError('within takes a list of folders: [folder]') + } + return within +} + +/** + * Builds a package from a source and writes it to `output`, which must not exist yet; with + * `check`, builds it and writes none, and takes no `output`. * @param {unknown} input the source: `{ type, title, language, license, authors, params }`, or a quiz in the published quiz format - * @param {string} output + * @param {string | undefined} output the package's path; `undefined` with `check` * @param {BuildOptions} [options] * @returns {Promise} */ export async function buildPackage(input, output, options = {}) { - output = resolve(output) - // Before any work, and again as the package lands, for a file that appears in between. - if (await lstat(output).then(() => true, () => false)) throw exists(output) - if (!(await stat(dirname(output)).catch(() => null))?.isDirectory()) throw new Error(`${dirname(output)} is not a folder to write the package in`) + const { check = false } = options + const within = foldersOf(options.within) + if (!check && (typeof output !== 'string' || !output.trim())) throw new TypeError('a build takes the path to write its package to, and a check (check: true) none') + // A check writes nothing, so where a package would go is not looked at. + const target = check ? '' : resolve(/** @type {string} */ (output)) + if (!check) { + if (within && !(await inside(target, within))) throw new Error(`${target} is outside the folders this build may write in`) + // Before any work, and again as the package lands, for a file that appears in between. + if (await lstat(target).then(() => true, () => false)) throw exists(target) + if (!(await stat(dirname(target)).catch(() => null))?.isDirectory()) throw new Error(`${dirname(target)} is not a folder to write the package in`) + } const pack = await openPack(options.libraries) let temporary try { @@ -159,9 +362,22 @@ export async function buildPackage(input, output, options = {}) { // source whose own fields have problems is checked in the same round (#174). const read = sourceOf(input, typeNames(pack)) if (!read.typed) throw new SourceError(read.problems) - temporary = await mkdtemp(join(dirname(output), '.h5p-author-')) - const media = new MediaStore(nodeMedia({ base: options.base ?? process.cwd(), temporary }), { fetch: options.fetch }) + temporary = await mkdtemp(join(check ? tmpdir() : dirname(target), '.h5p-author-')) + const media = new MediaStore(nodeMedia({ base: options.base ?? process.cwd(), temporary, within }), { fetch: options.fetch }) const built = await composePackage(read.source, pack, media, read.problems) + if (check) { + return { + output: '', + manifest: built.manifest, + content: built.content, + libraries: [...built.selected.keys()], + install: [...built.install.keys()], + translations: built.translations, + notes: built.notes, + media: media.files.map((item) => item.name), + get outline() { return built.outline } + } + } const staged = join(temporary, 'package.h5p') const file = await open(staged, 'wx') try { @@ -184,18 +400,19 @@ export async function buildPackage(input, output, options = {}) { await normalizeArchive({ input: staged, output: final, last: [...built.install.keys()] }) } // Unlike a hard link, an exclusive copy also works on FAT/exFAT destinations. - await copyFile(final, output, constants.COPYFILE_EXCL).catch((error) => { - throw error?.code === 'EEXIST' ? exists(output) : error + await copyFile(final, target, constants.COPYFILE_EXCL).catch((error) => { + throw error?.code === 'EEXIST' ? exists(target) : error }) return { - output, + output: target, manifest: built.manifest, content: built.content, libraries: [...built.selected.keys()], install: [...built.install.keys()], translations: built.translations, notes: built.notes, - media: media.files.map((item) => item.name) + media: media.files.map((item) => item.name), + get outline() { return built.outline } } } finally { await pack.close() diff --git a/packages/author/lib/outline.mjs b/packages/author/lib/outline.mjs new file mode 100644 index 00000000..234c6380 --- /dev/null +++ b/packages/author/lib/outline.mjs @@ -0,0 +1,299 @@ +import { isInterfaceText } from './schema.mjs' +import { decodeEntities, editorWidget, tagsIn, takesMarkup } from './text.mjs' +import { questionTitle } from './types.mjs' + +/** + * A built content as plain text, for a review before a package is handed over: what an author + * wrote, one line each, indented by the list it sits in, read through the libraries' semantics. + * Only text a learner reads is written: a text field the editor draws as a text box, a textarea or + * rich text, which is not an interface text (`isInterfaceText`) and holds more than its default, + * which in loaded semantics is the content language's. Where a text breaks, a paragraph, a line, a + * list item, ` / ` keeps the parts apart. Settings are left out: selects, numbers, booleans, + * colours, what the editor does not draw (`widget: 'none'`), and a text with no label that is not + * an item's own, which a widget of the type's own sets, a figure or a position. What says which + * answer is right is written: a ✓ before an item whose `correct` is true, and the answer a + * `correct` select names; and a score range before its feedback, in a list the editor draws as + * H5P's RangeList. Sub-content gets its own library's semantics and its title where an author gave + * one; media are named by kind and path. What a few types say only by position or by number is + * written out as `TYPES` says. + */ + +/** @param {unknown} value @returns {value is Record} */ +const isObject = (value) => value !== null && typeof value === 'object' && !Array.isArray(value) + +/** The widgets that draw text an author writes and a learner reads. */ +const PROSE = new Set(['text', 'textarea', 'html']) + +/** Fields whose text is the item itself, written without a label. */ +const UNLABELLED = new Set(['text', 'question']) + +/** + * What a few types say only by position or by number, written out. Paths are where a list or a + * field sits in the type's parameters, as `a.b[].c`. + * - `right`: lists whose first item is the right one, which each type's semantics say in words + * alone: "Alternatives - first alternative is the correct one" (Single Choice Set), "the first + * statement is correct" (Summary). + * - `items`: lists whose items get a numbered header, as their editors show them one at a time: + * slides, chapters, questions, a scenario's content, which its links name by that number. + * - `titles`: lists whose items the build titles where an author did not (`questionTitle`): such a + * title is the build's, and not written. + * - `times`: lists whose items start with the time the video shows them: an interaction's + * `duration`, a bookmark's or an end screen's `time`. + * - `links`: a scenario's `nextContentId`, the index of the content a choice or a content leads + * to, negative for the end; and `endScreenScore`, the score an end gives. + * - `labels`: a label for a field whose semantics give none. + * @type {Record, titles?: Record string>, + * times?: Record, links?: boolean, labels?: Record + * }>} + */ +const TYPES = { + 'H5P.SingleChoiceSet': { right: ['choices[].answers'] }, + 'H5P.Summary': { right: ['summaries[].summary'] }, + 'H5P.QuestionSet': { items: { questions: 'Question' }, titles: { questions: questionTitle } }, + 'H5P.InteractiveBook': { items: { chapters: 'Chapter' } }, + 'H5P.CoursePresentation': { items: { 'presentation.slides': 'Slide' }, labels: { 'presentation.slides[].keywords[].main': 'Keyword' } }, + 'H5P.BranchingScenario': { items: { 'branchingScenario.content': 'Content', 'branchingScenario.endScreens': 'End screen' }, links: true }, + 'H5P.BranchingQuestion': { links: true }, + 'H5P.InteractiveVideo': { + times: { 'interactiveVideo.assets.interactions': 'duration', 'interactiveVideo.assets.bookmarks': 'time', 'interactiveVideo.assets.endscreens': 'time' }, + labels: { 'interactiveVideo.assets.bookmarks[].label': 'Bookmark', 'interactiveVideo.assets.endscreens[].label': 'End screen' } + } +} + +/** The kinds of file field, as a line names them. */ +const MEDIA = /** @type {Record} */ ({ image: 'Image', video: 'Video', audio: 'Audio', file: 'File' }) + +/** + * A tag where rich text breaks: a line break, a rule, and the start or end of a block, a paragraph, + * a heading, a list or its item, a table's row or cell. + */ +const BREAK = /^<\/?\s*(?:p|div|h[1-6]|ul|ol|li|dl|dt|dd|pre|blockquote|table|thead|tbody|tfoot|tr|td|th|caption|figure|figcaption|section|article|header|footer|aside|nav|br|hr)\b/i + +/** A tag that sits in a line without parting the text around it: `H2O` is one word. */ +const INLINE = /^<\/?\s*(?:a|abbr|b|bdi|bdo|cite|code|data|del|dfn|em|font|i|ins|kbd|mark|q|s|samp|small|span|strike|strong|sub|sup|time|tt|u|var)\b/i + +/** + * Stored text as a learner reads it, its parts ` / ` apart. Rich text is split where a block starts + * or ends or a line breaks; a tag inside a line is taken out with nothing in its place, any other + * with a space; tags are read as `stripTags` reads them. Plain text is split at its line breaks, + * `\r` alone among them. Entities are read in each part after the split, so no text, an encoded + * one included, can pass for a break. + * @param {string} stored + * @param {boolean} markup whether the field holds markup + */ +function readable(stored, markup) { + /** @type {string[]} */ + let parts + if (markup) { + parts = [''] + let at = 0 + for (const tag of tagsIn(stored)) { + const index = tag.index ?? 0 + parts[parts.length - 1] += `${stored.slice(at, index)}${INLINE.test(tag[0]) ? '' : ' '}` + at = index + tag[0].length + if (BREAK.test(tag[0])) parts.push('') + } + parts[parts.length - 1] += stored.slice(at) + } else { + parts = stored.split(/\r\n?|\n/) + } + return parts.map((part) => decodeEntities(part).replace(/\s+/g, ' ').trim()).filter(Boolean).join(' / ') +} + +/** A time in a video, `m:ss`, with tenths where it has them. @param {number} seconds */ +function clock(seconds) { + const tenths = Math.round(seconds * 10) + const whole = Math.floor(tenths / 10) + return `${Math.floor(whole / 60)}:${String(whole % 60).padStart(2, '0')}${tenths % 10 ? `.${tenths % 10}` : ''}` +} + +/** + * @typedef {{ library: { machineName: string, majorVersion: number, minorVersion: number }, semantics: any[] | null }} OutlineEntry + */ + +/** + * The content as plain lines, in the order `checkContent` takes its arguments. + * @param {{ entries: Map, main: OutlineEntry }} libraries what `loadLibraries` + * loaded for the content: the content type's semantics, and each sub-content's by its library + * @param {{ params: unknown }} content the content as `checkContent` takes it, `{ type, params }`: + * its `params`, a package's `content.json`, are read + * @param {{ limit?: number }} [options] `limit`, the most lines written, the rest counted in a last + * line; all of them where none is given + * @returns {string} + */ +export function outlineOf(libraries, content, { limit = Infinity } = {}) { + if (!isObject(libraries) || !(libraries.entries instanceof Map) || !isObject(libraries.main) || !isObject(content) || !('params' in content)) { + throw new TypeError('outlineOf takes what loadLibraries loaded and the content, { type, params }, as checkContent does') + } + const { entries, main } = libraries + /** @type {string[]} */ + const lines = [] + let more = 0 + // A mark for the next line: ✓, a score range, or a time. + let pending = '' + // The title the build gave the next sub-content, where its type titles one an author did not. + let given = '' + /** Each library's semantics as the walk reaches it, which `TYPES`' titles are read from. @type {Map} */ + const semanticsOf = new Map() + + /** @param {number} depth @param {string} text */ + const emit = (depth, text) => { + if (lines.length >= limit) more += 1 + else lines.push(`${' '.repeat(depth)}${pending}${text}`) + pending = '' + } + + /** A sub-content's library, by the string its value names, at that version where the pack has it. @param {string} named */ + const libraryOf = (named) => { + const [name, version = ''] = named.trim().split(/\s+/) + const [major, minor] = version.split('.').map(Number) + /** @type {OutlineEntry | undefined} */ + let other + for (const entry of entries.values()) { + if (entry.library.machineName !== name || !entry.semantics) continue + if (entry.library.majorVersion === major && entry.library.minorVersion === minor) return entry + other ??= entry + } + return other + } + + /** + * @param {any[]} fields + * @param {unknown} params + * @param {number} depth + * @param {string} type the library the fields are of + * @param {string} path where they sit in its parameters, as `a.b[].c`; empty for a library's own + * @param {boolean} [bare] their text written without labels: a score range's feedback + */ + const walkFields = (fields, params, depth, type, path, bare = false) => { + if (!isObject(params)) return + if (!path) semanticsOf.set(type, fields) + for (const field of fields) walk(field, params[field.name], depth, type, path ? `${path}.${field.name}` : field.name, bare, params) + } + + /** + * The mark a list's item starts with: ✓ for the right one, or the time a video shows it. + * @param {string} type @param {string} path the list's @param {unknown} item @param {number} index + */ + const markOf = (type, path, item, index) => { + const rules = TYPES[type] + if (index === 0 && rules?.right?.includes(path)) return '✓ ' + const time = rules?.times?.[path] + if (!time || !isObject(item)) return '' + if (time === 'time') return typeof item.time === 'number' ? `${clock(item.time)} ` : '' + const { from, to } = isObject(item.duration) ? item.duration : {} + return typeof from === 'number' && typeof to === 'number' ? `${clock(from)}–${clock(to)} ` : '' + } + + /** + * @param {any} field + * @param {unknown} value + * @param {number} depth + * @param {string} type + * @param {string} path + * @param {boolean} bare whether its text is written without a label: a list's item, a score range's feedback + * @param {Record} [holder] the parameters the value is one of + */ + const walk = (field, value, depth, type, path, bare, holder) => { + if (value === undefined || value === null || isInterfaceText(field)) return + // A group or a list the editor does not draw holds settings, whatever they are: Game Map's + // stage positions. A number it does not draw can still be read, as a scenario's links are. + if (field.widget === 'none' && (field.type === 'group' || field.type === 'list')) return + switch (field.type) { + case 'text': { + if (typeof value !== 'string' || !PROSE.has(editorWidget(field))) return + // A label as plain text too, where a library writes one with an entity or a tag, and + // without a colon of its own, as a line puts one after it. + const label = (TYPES[type]?.labels?.[path] ?? (typeof field.label === 'string' ? readable(field.label, true) : '')).replace(/\s*:\s*$/, '') + // A text with no label that is not an item's own is set by a widget of the type's own: + // Find the Hotspot's figure, Virtual Tour's positions. + if (!label && !bare && !UNLABELLED.has(field.name)) return + const markup = takesMarkup(field) + const text = readable(value, markup) + if (!text || (typeof field.default === 'string' && text === readable(field.default, markup))) return + emit(depth, bare || UNLABELLED.has(field.name) || !label ? text : `${label}: ${text}`) + return + } + case 'select': + if (field.name === 'correct') { + const option = (field.options ?? []).find((/** @type {any} */ each) => String(each?.value) === String(value)) + emit(depth, `Correct answer: ${option?.label ?? value}`) + } + return + case 'number': + if (!TYPES[type]?.links || typeof value !== 'number') return + // Where a scenario goes is its shape, default or not; a score only where an author gave one. + // A question leads where its alternatives do, and its content's own link is not read. + if (field.name === 'nextContentId') { + const library = isObject(holder?.type) && typeof holder.type.library === 'string' ? holder.type.library.trim().split(/\s+/)[0] : '' + if (library !== 'H5P.BranchingQuestion') emit(depth, value < 0 ? 'Next: the end' : `Next: Content ${value + 1}`) + } + else if (field.name === 'endScreenScore' && value !== field.default) emit(depth, `Score: ${value}`) + return + case 'group': { + const children = field.fields ?? [] + // A group of one field is stored as that field's value, as the editor stores it. + if (children.length === 1 && !field.isSubContent) return walk(children[0], value, depth, type, path, bare, holder) + if (!isObject(value)) return + const marked = children.some((/** @type {any} */ child) => child.name === 'correct' && child.type === 'boolean') && value.correct === true + if (marked) pending = '✓ ' + // A list's item that is a group writes its fields with their labels. + walkFields(children, value, depth, type, path) + // A mark the group did not use is not passed on to the next line. + if (marked) pending = '' + return + } + case 'list': { + if (!Array.isArray(value) || !field.field) return + const header = TYPES[type]?.items?.[path] + const titled = TYPES[type]?.titles?.[path] + const ranges = (field.widgets ?? []).some((/** @type {any} */ widget) => widget?.name === 'RangeList') && field.field.type === 'group' + value.forEach((item, index) => { + let at = depth + 1 + if (header) { + pending = '' + emit(at, `${header} ${index + 1}`) + at += 1 + } + if (ranges && isObject(item) && typeof item.from === 'number' && typeof item.to === 'number') { + pending = `${item.from}–${item.to}%: ` + walkFields(field.field.fields ?? [], item, at, type, `${path}[]`, true) + } else { + pending = markOf(type, path, item, index) + given = titled ? readable(titled(semanticsOf.get(type) ?? [], index), false) : '' + walk(field.field, item, at, type, `${path}[]`, true) + } + pending = '' + given = '' + }) + return + } + case 'library': { + // Taken by the sub-content it was given for, so none inside it reads it. + const builds = given + given = '' + if (!isObject(value) || typeof value.library !== 'string') return + // Its title, where an author gave one: not the content type's name, nor a title the build + // gave it. + const title = typeof value.metadata?.title === 'string' ? readable(value.metadata.title, false) : '' + const typeName = typeof value.metadata?.contentType === 'string' ? readable(value.metadata.contentType, false) : '' + if (title && title !== typeName && title !== builds) emit(depth, `Title: ${title}`) + const entry = libraryOf(value.library) + if (entry?.semantics) walkFields(entry.semantics, value.params, depth, entry.library.machineName, '') + return + } + default: { + const kind = MEDIA[field.type] + if (!kind) return + const files = Array.isArray(value) ? value : [value] + const paths = files.map((file) => (isObject(file) && typeof file.path === 'string' ? file.path : '')).filter(Boolean) + if (paths.length) emit(depth, `${kind}: ${paths.join(', ')}`) + } + } + } + + if (main.semantics) walkFields(main.semantics, content.params, 0, main.library.machineName, '') + if (more) lines.push(`… ${more} more line${more === 1 ? '' : 's'}`) + return lines.join('\n') +} diff --git a/packages/author/lib/schema.mjs b/packages/author/lib/schema.mjs index 77aa69fb..f730de49 100644 --- a/packages/author/lib/schema.mjs +++ b/packages/author/lib/schema.mjs @@ -1,4 +1,4 @@ -import { editorWidget, takesMarkup } from './text.mjs' +import { decodeEntities, editorWidget, stripTags, takesMarkup } from './text.mjs' /** * A content type's fields as an author writes them, from its `semantics.json`: names, kinds, @@ -29,7 +29,7 @@ export function isInterfaceText(field) { /** @param {string} html */ function plain(html) { - return html.replace(/<\/li>/g, '. ').replace(/<[^<>]*>/g, ' ').replace(/ /g, ' ').replace(/"/g, '"').replace(/'|'/g, "'").replace(/>/g, '>').replace(/</g, '<').replace(/&/g, '&').replace(/\s+/g, ' ').replace(/\.\s*\./g, '.').trim() + return decodeEntities(stripTags(html.replace(/<\/li>/g, '. '))).replace(/\s+/g, ' ').replace(/\.\s*\./g, '.').trim() } /** @param {unknown} value */ diff --git a/packages/author/lib/text.mjs b/packages/author/lib/text.mjs index 5a8b9192..f438f7bd 100644 --- a/packages/author/lib/text.mjs +++ b/packages/author/lib/text.mjs @@ -19,6 +19,53 @@ export function escapeText(value) { return value.replace(BARE_AMPERSAND, '&').replaceAll('<', '<').replaceAll('>', '>').replaceAll('"', '"').replaceAll("'", ''') } +/** + * A tag, a comment, or a lone `<` or `>`: the pieces the content check reads markup in. A tag runs + * from a `<` before a letter, `/` or `!` to the first `>`. A comment left open runs to the end, as a + * browser reads it, so its end is looked for once and not again from every `|$)|<[^>]*(>|$)|>)/g + +/** + * Text without its tags and comments, read as the content check splits markup (`PIECE`): each a + * space, never nothing, so taking one out cannot join what was around it into another; a lone `<` + * or `>` kept as text; every run of space one, trimmed. Entities are left as they are + * (`decodeEntities` reads them). + * @param {string} html + */ +export function stripTags(html) { + return html.replace(PIECE, (piece) => (piece.length > 1 ? ' ' : piece)).replace(/\s+/g, ' ').trim() +} + +/** + * The tags and comments in a markup text, as `stripTags` reads them, each with its `index`. + * @param {string} html + * @returns {Generator} + */ +export function* tagsIn(html) { + for (const piece of html.matchAll(PIECE)) if (piece[0].length > 1) yield piece +} + +/** The named entities stored text holds: those `escapeText` writes, and a few an author types. */ +const NAMED = /** @type {Record} */ ({ + amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', shy: '­', ndash: '–', mdash: '—', hellip: '…', + lsquo: '‘', rsquo: '’', ldquo: '“', rdquo: '”', laquo: '«', raquo: '»', bull: '•', middot: '·', deg: '°', times: '×', + copy: '©', reg: '®', trade: '™', euro: '€', pound: '£', yen: '¥', cent: '¢', sect: '§', para: '¶', frac12: '½' +}) + +/** + * Entities read, in one pass, so `&lt;` is `<` and not `<`: numeric ones, and the named + * ones above. An entity it does not know is left as written. + * @param {string} text + */ +export function decodeEntities(text) { + return text.replace(/&(?:#(\d{1,7})|#[xX]([0-9a-fA-F]{1,6})|([a-zA-Z][a-zA-Z0-9]*));/g, (entity, decimal, hex, name) => { + if (name) return Object.hasOwn(NAMED, name) ? NAMED[name] : entity + const code = decimal ? Number(decimal) : Number.parseInt(hex, 16) + return code > 0 && code <= 0x10ffff && (code < 0xd800 || code > 0xdfff) ? String.fromCodePoint(code) : entity + }) +} + /** * The widget H5P's editor draws a field with: its `widget`, or for a ShowWhen, the widget the * ShowWhen draws inside it, else the field's type (h5peditor.js `getWidgetName`, H5PEditor.ShowWhen). @@ -99,12 +146,6 @@ function stylePatterns(field) { return patterns.map((pattern) => new RegExp(pattern, 'i')) } -/** - * A tag, a comment, or a lone `<` or `>`: the pieces the content check reads markup in. A comment - * left open runs to the end, as a browser reads it, so its end is looked for once and not again - * from every `|$)|<[^>]*(>|$)|>)/g const ATTRIBUTE = /([a-zA-Z_:][-a-zA-Z0-9_:.]*)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g const SAFE_URL = /^(?:https?:|mailto:|ftp:|[^:]*$)/i diff --git a/packages/author/lib/types.mjs b/packages/author/lib/types.mjs index 7bcc8705..bb6ba4a4 100644 --- a/packages/author/lib/types.mjs +++ b/packages/author/lib/types.mjs @@ -1,4 +1,5 @@ import { newId } from './generator.mjs' +import { stripTags } from './text.mjs' /** * What a content type needs beyond its semantics. H5P's editor has a widget of its own for some @@ -49,7 +50,7 @@ const machineName = (library) => (typeof library === 'string' ? library.trim().s /** Text without its tags and entities, for measuring. @param {string} html */ function textOf(html) { - return html.replace(/<[^<>]*>/g, ' ').replace(/&[a-zA-Z#0-9]+;/g, '_').replace(/\s+/g, ' ').trim() + return stripTags(html).replace(/&[a-zA-Z#0-9]+;/g, '_') } /** A field's semantics, found by a path of names. @param {any[]} semantics @param {string[]} names */ @@ -67,6 +68,17 @@ function fieldAt(semantics, names) { /* ---------------------------------------------------------------- Question Set */ +/** + * The title a Question Set gives a question that has none: the set's own label for a question, in + * the content's language, and its number. The outline leaves it out, as the build's and not an + * author's. + * @param {any[]} semantics the set's, as loaded for the content + * @param {number} index + */ +export function questionTitle(semantics, index) { + return `${fieldAt(semantics, ['texts', 'questionLabel'])?.default ?? 'Question'} ${index + 1}` +} + /** * A question with no title is named as the set labels it itself, in the content's language: its * result screen lists each question by this title. @@ -74,9 +86,8 @@ function fieldAt(semantics, names) { */ const questionSet = { after(params, path, context) { - const label = fieldAt(context.semantics, ['texts', 'questionLabel'])?.default ?? 'Question' ;(params.questions ?? []).forEach((/** @type {any} */ question, /** @type {number} */ i) => { - if (isObject(question?.metadata) && question.metadata.title === undefined) question.metadata.title = `${label} ${i + 1}` + if (isObject(question?.metadata) && question.metadata.title === undefined) question.metadata.title = questionTitle(context.semantics, i) }) } } diff --git a/packages/author/tests/confine.test.ts b/packages/author/tests/confine.test.ts new file mode 100644 index 00000000..e643fbf4 --- /dev/null +++ b/packages/author/tests/confine.test.ts @@ -0,0 +1,385 @@ +import { execFile } from 'node:child_process' +import { chmod, mkdir, mkdtemp, readdir, rm, symlink, writeFile } from 'node:fs/promises' +import { createServer, type Server } from 'node:http' +import type { AddressInfo } from 'node:net' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' +import { promisify } from 'node:util' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { buildPackage, loadLibraries, openPack, outlineOf, publicFetch, SourceError } from '../lib/author.mjs' +import { guardedFetch, inside, isPrivateAddress } from '../lib/node.mjs' +import { mp4, png } from './helpers/media' + +const exec = promisify(execFile) +const bin = resolve(import.meta.dirname, '../bin/h5p-author.mjs') +const META = { title: 'Rivers', language: 'en', license: 'CC BY', authors: [{ name: 'Ann Author', role: 'Author' }] } +const withImage = (path: string) => ({ ...META, type: 'H5P.Column', params: { content: [{ content: { library: 'H5P.Image', params: { file: { path }, alt: 'A river' } } }] } }) +const question = { ...META, type: 'H5P.MultiChoice', params: { question: 'Capital of France?', answers: [{ text: 'Paris', correct: true }, { text: 'Lyon', correct: false }] } } +let dir: string +let allowed: string +let outside: string + +beforeAll(async () => { + dir = await mkdtemp(join(tmpdir(), 'h5p-author-confine-')) + allowed = join(dir, 'allowed') + outside = join(dir, 'outside') + await mkdir(allowed) + await mkdir(outside) + await writeFile(join(allowed, 'in.png'), png(8, 8)) + await writeFile(join(allowed, '..notes.png'), png(8, 8)) + await writeFile(join(allowed, 'clip.mp4'), mp4(true)) + await writeFile(join(outside, 'out.png'), png(8, 8)) + await symlink(join(outside, 'out.png'), join(allowed, 'link.png')) +}) +afterAll(async () => { await rm(dir, { recursive: true, force: true }) }) + +/** A source's outline, from a build that writes nothing. */ +async function outline(source: Record) { + return (await buildPackage({ ...META, ...source }, undefined, { check: true })).outline +} + +describe('within', () => { + it('builds with a file and an output inside the folders, a name that starts with two dots among them', async () => { + expect((await buildPackage(withImage('in.png'), join(allowed, 'a.h5p'), { base: allowed, within: [allowed] })).media).toHaveLength(1) + expect((await buildPackage(withImage('..notes.png'), join(allowed, 'a2.h5p'), { base: allowed, within: [allowed] })).media).toHaveLength(1) + }) + + it('refuses a file outside them, by .. and by a link, and an output outside', async () => { + await expect(buildPackage(withImage('../outside/out.png'), join(allowed, 'b.h5p'), { base: allowed, within: [allowed] })).rejects.toThrow(SourceError) + await expect(buildPackage(withImage('link.png'), join(allowed, 'c.h5p'), { base: allowed, within: [allowed] })).rejects.toThrow(/outside the folders/) + await expect(buildPackage(question, join(outside, 'd.h5p'), { within: [allowed] })).rejects.toThrow(/outside the folders this build may write in/) + expect(await readdir(outside)).toEqual(['out.png']) + }) + + it('answers a path outside the folders the same whether or not a file is there, and names no absolute path', async () => { + const at = 'params.content[0].content.params.file.path' + for (const path of ['../outside/out.png', '../outside/missing.png']) { + const error = await buildPackage(withImage(path), join(allowed, 'e.h5p'), { base: allowed, within: [allowed] }).catch((caught) => caught) + expect(error.problems).toEqual([{ path: at, message: `${path}: is outside the folders this build may read` }]) + } + const error = await buildPackage(withImage('nothing.png'), join(allowed, 'e.h5p'), { base: allowed, within: [allowed] }).catch((caught) => caught) + expect(error.problems).toEqual([{ path: at, message: 'nothing.png: no such file in the folders this build may read' }]) + }) + + it('judges a path that does not exist yet by its nearest folder, and a folder is not its sibling of the same prefix', async () => { + expect(await inside(join(allowed, 'new', 'x.h5p'), [allowed])).toBe(true) + expect(await inside(join(allowed, '..notes.png'), [allowed])).toBe(true) + expect(await inside(join(allowed, '..', 'outside', 'out.png'), [allowed])).toBe(false) + expect(await inside(`${allowed}-other/x.h5p`, [allowed])).toBe(false) + }) + + it('takes a list of folders, and refuses one folder given alone, which would confine nothing', async () => { + await expect(buildPackage(withImage('in.png'), join(allowed, 'h.h5p'), { base: allowed, within: allowed as unknown as string[] })).rejects.toThrow(/within takes a list of folders/) + expect(await readdir(allowed)).not.toContain('h.h5p') + }) + + it.skipIf(process.platform === 'win32' || process.getuid?.() === 0)('says a file it cannot read is a problem of the source, in a check too, and names no path', async () => { + await writeFile(join(allowed, 'locked.mp4'), mp4(true)) + await chmod(join(allowed, 'locked.mp4'), 0o000) + try { + const video = { ...META, type: 'H5P.Column', params: { content: [{ content: { library: 'H5P.Video', params: { sources: [{ path: 'locked.mp4' }] } } }] } } + for (const output of [undefined, join(allowed, 'i.h5p')]) { + const error = await buildPackage(video, output, { base: allowed, within: [allowed], check: !output }).catch((caught) => caught) + expect(error.problems, String(output)).toEqual([{ path: 'params.content[0].content.params.sources[0].path', message: 'locked.mp4: cannot be read' }]) + } + } finally { + await chmod(join(allowed, 'locked.mp4'), 0o644) + } + }) +}) + +describe('check', () => { + it('takes no output and writes no package, and reports the problems of a wrong source as a build does', async () => { + const result = await buildPackage(question, undefined, { check: true }) + expect(result.output).toBe('') + expect(result.manifest.mainLibrary).toBe('H5P.MultiChoice') + await expect(buildPackage({ ...question, title: undefined }, undefined, { check: true })).rejects.toThrow(SourceError) + await expect(buildPackage(question, undefined)).rejects.toThrow(/takes the path to write its package to/) + }) + + it('looks at no output: a folder that is not there, outside the folders, is no matter to a check', async () => { + expect((await buildPackage(question, join(outside, 'no', 'such', 'x.h5p'), { check: true, within: [allowed] })).output).toBe('') + }) + + it('is build --check on the command line, with one argument and no file written', async () => { + const source = join(allowed, 'q.json') + await writeFile(source, JSON.stringify(question)) + const before = await readdir(allowed) + const { stdout } = await exec('node', [bin, 'build', '--check', source]) + expect(stdout).toMatch(/Checked, no package written: H5P.MultiChoice/) + expect(await readdir(allowed)).toEqual(before) + await expect(exec('node', [bin, 'build', '--check', source, 'x.h5p'])).rejects.toThrow(/build --check takes a source/) + }) +}) + +describe('publicFetch', () => { + let server: Server + let port: number + let requests = 0 + beforeAll(async () => { + server = createServer((request, response) => { + requests += 1 + response.writeHead(200).end('local') + }) + await new Promise((resolveListen) => server.listen(0, '127.0.0.1', resolveListen)) + port = (server.address() as AddressInfo).port + }) + afterAll(() => { server.close() }) + + it('refuses a local address however it is written, a URL writing ::ffff:127.0.0.1 as ::ffff:7f00:1, and asks nothing of it', async () => { + for (const host of ['127.0.0.1', 'localhost', '[::1]', '[::ffff:127.0.0.1]', '[::ffff:7f00:1]', '[::127.0.0.1]', '2130706433', '0x7f.1']) { + await expect(publicFetch(`http://${host}:${port}/`), host).rejects.toThrow(/not on the public internet/) + } + await expect(publicFetch('file:///etc/passwd')).rejects.toThrow(/not fetched/) + await expect(publicFetch('https://example.com/', { method: 'POST' })).rejects.toThrow(/GET or HEAD/) + expect(requests).toBe(0) + }) + + it('knows the local ranges, in IPv4 and IPv6, and an IPv4 address written as IPv6', () => { + for (const address of ['10.0.0.1', '172.16.5.5', '172.31.0.1', '192.168.1.1', '169.254.169.254', '127.0.0.1', '100.64.0.1', '0.0.0.0', '192.0.0.1', '198.18.0.1', '224.0.0.1', '255.255.255.255', + '::', '::1', '0:0:0:0:0:0:0:1', '::7f00:1', 'fd00::1', 'fe80::1', 'fec0::1', 'ff02::1', '::ffff:10.0.0.1', '::ffff:a00:1', '::ffff:7f00:1', + '64:ff9b::7f00:1', '64:ff9b::a00:5', '64:ff9b::a9fe:a9fe', '64:ff9b::c0a8:101', '64:ff9b:1::808:808', '2002:a00:1::1', '2002:7f00:1::', + '192.0.2.1', '198.51.100.7', '203.0.113.9', '2001:db8::1', + '64:ff9b::c000:201', '2002:c000:201::', '64:ff9b::c633:6407', '2002:c633:6407::', '64:ff9b::cb00:7109', '2002:cb00:7109::', + '::ffff:0:7f00:1', '::ffff:0:127.0.0.1', '100::1', '2001:0:4136:e378:8000:63bf:3fff:fdd2', '2001:2::1', '3fff::1', '5f00::1', 'not an address']) { + expect(isPrivateAddress(address), address).toBe(true) + } + for (const address of ['8.8.8.8', '172.32.0.1', '100.128.0.1', '2606:4700::1111', '2001:4860:4860::8888', '::ffff:808:808', '::ffff:8.8.8.8', '64:ff9b::808:808', '2002:808:808::1']) { + expect(isPrivateAddress(address), address).toBe(false) + } + }) + + /** A web of hosts: `public.test` and `cdn.test` resolve to public addresses, `inside.test` to a local one. */ + const web = (answer: (url: URL) => Response) => { + const asked: string[] = [] + const fetch = guardedFetch({ + lookup: async (host) => [{ address: host === 'inside.test' ? '10.1.2.3' : '93.184.216.34' }], + fetch: async (input) => { + const url = new URL(String(input)) + asked.push(url.href) + return answer(url) + } + }) + return { fetch, asked } + } + const redirect = (location: string) => new Response(null, { status: 302, headers: { location } }) + + it('refuses a redirect to a local address, by name and by number, without asking it', async () => { + for (const location of ['http://inside.test/secret', 'http://[::ffff:127.0.0.1]/secret', 'http://169.254.169.254/latest']) { + const { fetch, asked } = web((url) => (url.pathname === '/hop' ? redirect(location) : new Response('reached'))) + await expect(fetch('https://public.test/hop'), location).rejects.toThrow(/not on the public internet/) + expect(asked).toEqual(['https://public.test/hop']) + } + }) + + it('refuses a name that does not resolve in the words it refuses a local one, so a refusal names no local host', async () => { + const fetch = guardedFetch({ + lookup: async (host) => { + if (host === 'inside.test') return [{ address: '10.1.2.3' }] + throw Object.assign(new Error(`getaddrinfo ENOTFOUND ${host}`), { code: 'ENOTFOUND' }) + }, + fetch: async () => new Response('') + }) + const said = (host: string) => fetch(`https://${host}/`).then(() => 'fetched', (error) => error.message.replace(host, 'HOST')) + expect(await said('inside.test')).toBe('HOST is not on the public internet, the only place a build here fetches from') + expect(await said('missing.test')).toBe(await said('inside.test')) + }) + + it('follows a redirect between public hosts, keeps the request\'s headers, and names the address it ended at', async () => { + let header: string | null = null + const { fetch, asked } = web((url) => (url.hostname === 'public.test' ? redirect('https://cdn.test/clip.mp4') : new Response('video'))) + const answered = await fetch(new Request('https://public.test/clip', { headers: { 'x-test': 'kept' } })) + expect(await answered.text()).toBe('video') + expect(answered.url).toBe('https://cdn.test/clip.mp4') + expect(asked).toEqual(['https://public.test/clip', 'https://cdn.test/clip.mp4']) + const seen = guardedFetch({ lookup: async () => [{ address: '93.184.216.34' }], fetch: async (_, init) => { header = new Headers(init?.headers).get('x-test'); return new Response('') } }) + await seen(new Request('https://public.test/', { headers: { 'x-test': 'kept' } })) + expect(header).toBe('kept') + }) + + it('drops what a request carries for its own site on a redirect to another, and keeps it on its own', async () => { + const seen: [string, string | null, string | null, string | null][] = [] + const fetch = guardedFetch({ + lookup: async () => [{ address: '93.184.216.34' }], + fetch: async (input, init) => { + const url = new URL(String(input)) + const headers = new Headers(init?.headers) + seen.push([url.href, headers.get('authorization'), headers.get('cookie'), headers.get('accept')]) + return url.pathname === '/a' ? redirect('/b') : url.pathname === '/b' ? redirect('https://other.test/c') : new Response('ok') + } + }) + await fetch('https://first.test/a', { headers: { authorization: 'Bearer x', cookie: 'k=v', accept: 'video/*' } }) + expect(seen).toEqual([ + ['https://first.test/a', 'Bearer x', 'k=v', 'video/*'], + ['https://first.test/b', 'Bearer x', 'k=v', 'video/*'], + ['https://other.test/c', null, null, 'video/*'] + ]) + }) + + it('asks no resolver for a request that is already aborted', async () => { + let asked = false + const fetch = guardedFetch({ lookup: async () => { asked = true; return [{ address: '93.184.216.34' }] }, fetch: async () => new Response('') }) + await expect(fetch('https://slow.test/', { signal: AbortSignal.abort(new Error('stopped')) })).rejects.toThrow('stopped') + expect(asked).toBe(false) + }) + + it('stops waiting for a name when the signal aborts, as a download\'s idle limit does', async () => { + const hanging = guardedFetch({ lookup: () => new Promise(() => {}), fetch: async () => new Response('') }) + const controller = new AbortController() + const fetching = hanging('https://slow.test/', { signal: controller.signal }) + controller.abort(new Error('nothing arrived for 60 s')) + await expect(fetching).rejects.toThrow('nothing arrived for 60 s') + }) + + it('keeps a build it is given to off a local network, and asks nothing there', async () => { + const before = requests + const error = await buildPackage(withImage(`http://127.0.0.1:${port}/river.png`), join(allowed, 'g.h5p'), { base: allowed, within: [allowed], fetch: publicFetch }).catch((caught) => caught) + expect(error).toBeInstanceOf(SourceError) + expect(error.message).toMatch(/could not be downloaded: 127\.0\.0\.1 is not on the public internet/) + expect(requests).toBe(before) + }) + + it('answers a redirect as asked: manual returns it, error refuses it, and more than ten are refused', async () => { + const { fetch } = web(() => redirect('https://cdn.test/next')) + expect((await fetch('https://public.test/', { redirect: 'manual' })).status).toBe(302) + await expect(fetch('https://public.test/', { redirect: 'error' })).rejects.toThrow(/redirects were refused/) + await expect(fetch('https://public.test/')).rejects.toThrow(/more than 10 redirects/) + }) +}) + +describe('outline', () => { + it('writes the question, its answers with the right one marked, numbers included, and the feedback by its label', async () => { + expect(await outline({ + type: 'H5P.MultiChoice', + params: { question: 'When did Apollo 11 land?', answers: [{ text: '1969', correct: true, tipsAndFeedback: { chosenFeedback: 'Yes, in July.' } }, { text: '1972', correct: false }] } + })).toBe('When did Apollo 11 land?\n ✓ 1969\n Message displayed if answer is selected: Yes, in July.\n 1972') + }) + + it('names the right answer where a select holds it, or where a type takes the first item as the right one', async () => { + expect(await outline({ type: 'H5P.TrueFalse', params: { question: 'The Rhine flows into the North Sea.', correct: 'true' } })) + .toBe('The Rhine flows into the North Sea.\nCorrect answer: True') + expect(await outline({ type: 'H5P.SingleChoiceSet', params: { choices: [{ question: 'Which sea does the Danube reach?', answers: ['Black Sea', 'North Sea'] }] } })) + .toBe(' Which sea does the Danube reach?\n ✓ Black Sea\n North Sea') + expect(await outline({ type: 'H5P.Summary', params: { summaries: [{ summary: ['Rivers flow downhill.', 'Rivers flow uphill.'] }] } })) + .toBe(' ✓ Rivers flow downhill.\n Rivers flow uphill.') + }) + + it('leaves out interface texts, settings and the titles the build gives in any language, numbers questions, and writes the score ranges of a RangeList', async () => { + const quiz = { + type: 'H5P.QuestionSet', + params: { + questions: [ + { library: 'H5P.TrueFalse', params: { question: 'Water is wet.', correct: 'true' }, metadata: { title: 'Water' } }, + // Titled by the build, "Question 2" or "Frage 2": the set's own label, and not written. + { library: 'H5P.TrueFalse', params: { question: 'Ice is cold.', correct: 'true' } } + ], + endGame: { overallFeedback: [{ from: 0, to: 49, feedback: 'Try again.' }, { from: 50, to: 100, feedback: 'Passed.' }] } + } + } + const expected = [ + ' Question 1', ' Title: Water', ' Water is wet.', ' Correct answer: True', + ' Question 2', ' Ice is cold.', ' Correct answer: True', + ' 0–49%: Try again.', ' 50–100%: Passed.' + ].join('\n') + expect(await outline(quiz)).toBe(expected) + expect(await outline({ ...quiz, language: 'de' })).toBe(expected) + }) + + it('keeps a text\'s paragraphs, lines and list items apart, and numbers slides, with each slide\'s keyword', async () => { + const text = (html: string) => ({ action: { library: 'H5P.AdvancedText', params: { text: html } } }) + expect(await outline({ + type: 'H5P.CoursePresentation', + params: { presentation: { slides: [ + { elements: [text('

Rivers

The Volga is the longest.

')], keywords: [{ main: 'Rivers' }] }, + { elements: [text('

Line one
Line two

  • Danube
  • Rhine
')] } + ] } } + })).toBe(' Slide 1\n Rivers / The Volga is the longest.\n Keyword: Rivers\n Slide 2\n Line one / Line two / Danube / Rhine') + }) + + it('writes when a video shows each interaction and bookmark, names a bookmark as one, and names the video', async () => { + expect((await buildPackage({ + ...META, + type: 'H5P.InteractiveVideo', + params: { interactiveVideo: { + video: { files: [{ path: 'clip.mp4' }] }, + assets: { interactions: [{ duration: { from: 2.5, to: 66 }, label: 'Where does it go?', action: { library: 'H5P.Text', params: { text: 'To the sea.' } } }], bookmarks: [{ time: 7, label: 'Question' }] } + } } + }, undefined, { check: true, base: allowed })).outline).toBe('Video: videos/clip.mp4\n 0:02.5–1:06 Label: Where does it go?\n To the sea.\n 0:07 Bookmark: Question') + }) + + it('writes where each choice and content of a scenario leads, but not a question\'s own link, and the scores an author gave', async () => { + const scenario = await outline({ + type: 'H5P.BranchingScenario', + params: { branchingScenario: { + title: 'River', + startScreen: { startScreenTitle: 'River', startScreenSubtitle: 'Choose.' }, + content: [ + { type: { library: 'H5P.BranchingQuestion', params: { branchingQuestion: { question: 'Cross how?', alternatives: [{ text: 'Wade', nextContentId: 1 }, { text: 'Bridge', nextContentId: -1 }] } } } }, + { type: { library: 'H5P.AdvancedText', params: { text: 'Swept away.' } }, nextContentId: -1, feedback: { title: 'Wet', endScreenScore: 0 } } + ], + endScreens: [{ endScreenTitle: 'The end' }] + } } + }) + // One block, line after line: no link of the question's own between its last choice and the next content. + expect(scenario).toContain([ + ' Content 1', ' Cross how?', ' Wade', ' Next: Content 2', ' Bridge', ' Next: the end', + ' Content 2', ' Swept away.', ' Next: the end', ' Feedback title: Wet', ' Score: 0' + ].join('\n')) + expect(scenario.endsWith(' Score: 0')).toBe(true) + }) + + it('numbers chapters, writes an author\'s title but not the content type\'s, and names a picture with its alternative text', async () => { + expect(await outline({ + type: 'H5P.InteractiveBook', + params: { chapters: [{ library: 'H5P.Column', metadata: { title: 'Sources' }, params: { content: [{ content: { library: 'H5P.AdvancedText', params: { text: 'A river begins at a spring.' }, metadata: { title: 'Text' } } }] } }] } + })).toBe(' Chapter 1\n Title: Sources\n A river begins at a spring.') + expect((await buildPackage(withImage('in.png'), undefined, { check: true, base: allowed })).outline).toBe(' Image: images/in.png\n Alternative text: A river') + }) + + /** A content type of the test's own, with these fields. */ + const own = (semantics: unknown[]) => ({ entries: new Map(), main: { library: { machineName: 'Test.Type', majorVersion: 1, minorVersion: 0 }, semantics } }) + + it('reads text as a learner does: every line break and block, a tag inside a word taken out whole, an encoded character as itself, a label as plain text', () => { + const libraries = own([{ name: 'note', type: 'text', label: 'Tom & Jerry' }, { name: 'text', type: 'text', widget: 'html', tags: ['p', 'a', 'em', 'sub'] }]) + const params = { + note: 'one\rtwo\r\nthree\nfour', + text: '

ab

H2O and a link, said.

text

para' + } + expect(outlineOf(libraries, { params })).toBe('Tom & Jerry: one / two / three / four\nab / H2O and a link, said. / text / para') + }) + + it('leaves out what the editor does not draw and a text no label names, and writes a label without a colon of its own', () => { + const libraries = own([ + // Game Map's stage positions, Find the Hotspot's figure: set by a widget of the type's own. + { name: 'telemetry', type: 'group', widget: 'none', fields: [{ name: 'x', type: 'text' }, { name: 'y', type: 'text' }] }, + { name: 'figure', type: 'text' }, + { name: 'empty', type: 'text', label: 'Feedback if the user selects an empty spot:' }, + { name: 'items', type: 'list', field: { name: 'item', type: 'text' } } + ]) + expect(outlineOf(libraries, { params: { telemetry: { x: '15', y: '20' }, figure: 'circle', empty: 'Look again.', items: ['one', 'two'] } })) + .toBe('Feedback if the user selects an empty spot: Look again.\n one\n two') + }) + + it('writes every line where no limit is given, and counts what a limit cuts off', () => { + const libraries = own([{ name: 'items', type: 'list', field: { name: 'item', type: 'text' } }]) + const params = { items: Array.from({ length: 450 }, (_, index) => `Item ${index + 1}`) } + expect(outlineOf(libraries, { params }).split('\n')).toHaveLength(450) + expect(outlineOf(libraries, { params }, { limit: 2 })).toBe(' Item 1\n Item 2\n… 448 more lines') + }) + + it('is outlineOf over what loadLibraries loaded and the content, as checkContent takes them, made only when a result\'s outline is read', async () => { + const result = await buildPackage(question, undefined, { check: true }) + expect(typeof Object.getOwnPropertyDescriptor(result, 'outline')?.get).toBe('function') + const pack = await openPack() + try { + const content = { type: 'H5P.MultiChoice', language: 'en', params: question.params } + const libraries = await loadLibraries(pack, content) + expect(outlineOf(libraries, { ...content, params: result.content })).toBe(result.outline) + // The other way round, or the parameters alone, is refused, not an empty outline. + expect(() => outlineOf(result.content as never, libraries as never)).toThrow(/as checkContent does/) + expect(() => outlineOf(libraries, result.content as never)).toThrow(/as checkContent does/) + } finally { + await pack.close() + } + }) +}) diff --git a/packages/author/tests/text.test.ts b/packages/author/tests/text.test.ts index f69630a3..da145585 100644 --- a/packages/author/tests/text.test.ts +++ b/packages/author/tests/text.test.ts @@ -1,8 +1,22 @@ import { describe, expect, it } from 'vitest' -import { allowedTags, blockTag, editorWidget, escapeText, markupText, plainText, takesMarkup } from '../lib/text.mjs' +import { allowedTags, blockTag, decodeEntities, editorWidget, escapeText, markupText, plainText, stripTags, takesMarkup } from '../lib/text.mjs' import { translateDefaults, translationFiles } from '../lib/translate.mjs' describe('text as H5P stores it', () => { + it('takes tags and comments out as the content check splits markup, each a space, so what was around one never joins into a tag', () => { + expect(stripTags('

One

two three

')).toBe('One two three') + // An apostrophe in an attribute, quoted or not, is no matter: a tag ends at its first `>`. + expect(stripTags('one and two')).toBe('one and two') + expect(stripTags('link 5 < 6 > 4 ab')).toBe('link 5 < 6 > 4 a b') + expect(stripTags('ipt>alert(1)')).toBe('ipt>alert(1)') + }) + + it('reads entities in one pass: numeric, the named ones stored text holds, and leaves others as written', () => { + expect(decodeEntities('Tom & Jerry's "x" <b> / — ’')).toBe('Tom & Jerry\'s "x" / — ’') + expect(decodeEntities('&lt; is not <')).toBe('< is not <') + expect(decodeEntities('&unknown; � � �')).toBe('&unknown; � � �') + }) + it('escapes plain text as htmlspecialchars does, keeping entities already there', () => { expect(escapeText(`Tom & Jerry's ""`)).toBe('Tom & Jerry's "<b>"') expect(escapeText('AT&T © — & more')).toBe('AT&T © — & more')