move loc build tooling to ui-core

loc-tools/README.md

@@ -0,0 +1,123 @@
+# Localization tooling
+
+Build-time localization machinery shared by the microbit-apps stack. A consumer
+depends on `@microbit-apps/ui-core` and drives this tooling through its API and
+two bins. The runtime side (`ui.loc` and the `_loc` namespace) lives in
+`loc.ts`; this directory is the Node-side build tooling only.
+
+## Layout
+
+- `catalogs.mjs` -- layer discovery and per-language catalog merge.
+- `fonts.mjs` -- font glyph-coverage parsing.
+- `extract.mjs` -- source-string extraction from `ui.loc(...)` call sites.
+- `charsets.mjs` -- generic per-language field data (resolve + validate).
+- `emit.mjs` -- `loc.g.ts` generation and default/restore content.
+- `gen.mjs` -- the `runLocGen` / `runCoverageMode` orchestrator.
+- `index.mjs` -- the public API (imported as `@microbit-apps/ui-core/loc`).
+- `bin/loc-strings.mjs`, `bin/loc-gen.mjs` -- the CLI entry points.
+
+## Bins
+
+- `loc-strings [srcLang]` -- regenerate the calling project's
+  `locales/<srcLang>.json` (default `en`) from the display strings passed to
+  `ui.loc` / `ui.locf` / `ui.locc` in the files listed in `pxt.json`. Runs in
+  the current working directory.
+- `loc-gen [--coverage] [--verbose] [--keep] [lang ...]` -- per-language build
+  for simple projects. Optional `loc.config.json` (see below). Projects that
+  need app-specific hooks call `runLocGen` from their own wrapper instead.
+
+## Layer discovery
+
+Catalog layers are derived from the dependency graph, never hard-coded. For a
+project rooted at `root`:
+
+1. Walk the project's `pxt.json` dependencies transitively. Every dependency
+   module resolved under `pxt_modules/<name>/` that ships a `locales/`
+   directory becomes a layer, ordered so a library appears before any library
+   that depends on it.
+2. The project's own `locales/` is the final layer, named `app`.
+
+Later layers win on merge, so the app overrides its libraries and a library
+overrides its own dependencies. The same discovery drives translation catalogs
+(`<lang>.json`) and charset data (`charsets.json`).
+
+## Charsets
+
+Any layer may ship `locales/charsets.json` carrying per-language field data:
+
+```json
+{
+  "es": { "alphabetLower": "abc...", "alphabetUpper": "ABC...", "symbols": "+-*/" },
+  "fr": { "accentsLower": "àâä...", "accentsUpper": "ÀÂÄ..." }
+}
+```
+
+Each field is an arbitrary string. The tooling is agnostic: it never interprets
+field names. Language resolution per layer is the exact language key, else the
+base-language prefix (`es` serves `es-ES`). Fields merge across layers per
+language, the app layer winning.
+
+Resolved fields are hard-validated (a failure is an error, never a silent drop):
+
+- a `<x>Lower` / `<x>Upper` field pair, when both are present, must have equal
+  code-point length;
+- no field may repeat a code point within itself;
+- every code point of every field must exist in the build font's coverage.
+
+Each resolved field is emitted into the generated `loc.g.ts` as a `_loc` member
+assignment (`<field> = <json string>` inside `namespace _loc`). The library that
+owns a field declares the matching `_loc` member (via namespace reopening in
+that library's own source); this tooling only writes the assignment.
+
+## runLocGen options
+
+`runLocGen(options, langArgs)` runs the per-language build; `runCoverageMode`
+runs the coverage report. Defaults make a bare consumer work with only `root`.
+
+- `root` (required) -- project directory.
+- `srcLang` -- source language (default `"en"`); its build ships no table.
+- `langs` -- explicit language list; otherwise `langArgs`, otherwise the source
+  language plus every discovered language.
+- `reservedLangNames` -- non-language basenames in `locales/` to skip during
+  auto-discovery (the source language and `charsets` are always skipped).
+- `font` -- build font for glyph and charset validation (default `"font8"`).
+- `textPath` -- font glyph-table source (default the display-shield `text.ts`).
+- `hexDir`, `hexName(lang)` -- output directory and per-language file name
+  (default `<root>/assets/hex` and `<pxt name>.<lang>.hex`).
+- `defaultLocG` -- exact content restored to `loc.g.ts` after a run and written
+  for the source build (required for a build run).
+- `locGPath` -- the generated file (default `<root>/loc.g.ts`).
+- `smallFontContext` -- `{ font, strings }`: source strings validated against a
+  second (small) font instead of the build font, dropped when it cannot render
+  them. `strings` is an array, a Set, or a function returning either.
+- `exclude` -- `{ label, keys(merged) }`: source strings to drop from device
+  tables (they ship by another route). `label` names the count in the summary.
+- `extraInventory` -- extra coverage inventory beyond the layer source
+  catalogs: entries of `{ path, mapper(catalog) }` or precomputed string arrays.
+- `postMerge(lang, merged)` -- called after merge, before drops, for
+  app-specific side outputs. `merged` is `null` for the source language.
+- `coverage`, `verbose`, `keep` -- flags mirrored from the CLI.
+
+## Thin-wrapper pattern
+
+Apps with app-specific behavior wrap the API. The wrapper owns everything the
+generic tool should not know about, and stays thin:
+
+```js
+import { runLocGen } from "@microbit-apps/ui-core/loc"
+
+runLocGen(
+    {
+        root: ROOT,
+        defaultLocG: DEFAULT_LOC_G,
+        hexName: lang => `myapp.${lang}.hex`,
+        // app-specific hooks: exclude, smallFontContext, extraInventory, postMerge
+    },
+    process.argv.slice(2).filter(a => !a.startsWith("--")),
+)
+```
+
+A project with no such needs skips the wrapper entirely and uses the `loc-gen`
+bin, optionally with a `loc.config.json` setting `srcLang`, `font`, `hexName`
+(a pattern with a `<lang>` placeholder), `reservedLangNames`, `textPath`, or
+`hexDir`.

loc-tools/bin/loc-gen.mjs

@@ -0,0 +1,45 @@
+#!/usr/bin/env node
+// Thin per-language build CLI for simple consumers. Run from the project root.
+//
+// Configuration is optional: with no loc.config.json a bare project builds the
+// source language plus every locales/<lang>.json using stock defaults. A
+// loc.config.json (JSON) may set any of: srcLang, font, hexName (a pattern with
+// a "<lang>" placeholder), reservedLangNames, textPath, hexDir. Apps needing
+// app-specific hooks (extra inventory, side outputs, exclusions) call the
+// runLocGen API from their own wrapper instead.
+//
+// Flags: --coverage [--verbose], --keep, and positional language names.
+
+import { existsSync, readFileSync } from "node:fs"
+import { join } from "node:path"
+import { runLocGen } from "../gen.mjs"
+import { GENERIC_DEFAULT_LOC_G } from "../emit.mjs"
+
+const root = process.cwd()
+const argv = process.argv.slice(2)
+const coverage = argv.indexOf("--coverage") >= 0
+const verbose = argv.indexOf("--verbose") >= 0
+const keep = argv.indexOf("--keep") >= 0
+const positional = argv.filter(a => !a.startsWith("--"))
+
+let config = {}
+const configPath = join(root, "loc.config.json")
+if (existsSync(configPath)) config = JSON.parse(readFileSync(configPath, "utf8"))
+
+const options = {
+    root,
+    srcLang: config.srcLang,
+    font: config.font,
+    textPath: config.textPath ? join(root, config.textPath) : undefined,
+    hexDir: config.hexDir ? join(root, config.hexDir) : undefined,
+    hexName: config.hexName ? lang => config.hexName.replace("<lang>", lang) : undefined,
+    reservedLangNames: config.reservedLangNames,
+    // The CLI always uses the stock default; a custom loc.g.ts default is an
+    // API-level concern (pass defaultLocG to runLocGen from a wrapper).
+    defaultLocG: GENERIC_DEFAULT_LOC_G,
+    coverage,
+    verbose,
+    keep,
+}
+
+runLocGen(options, positional)

loc-tools/bin/loc-strings.mjs

@@ -0,0 +1,8 @@
+#!/usr/bin/env node
+// Regenerate the calling project's source catalog (locales/<srcLang>.json) from
+// its listed .ts files. Run from the project root.
+
+import { runLocStrings } from "../extract.mjs"
+
+const srcLang = process.argv[2] || "en"
+runLocStrings(process.cwd(), srcLang)

loc-tools/catalogs.mjs

@@ -0,0 +1,63 @@
+// Translation catalog layer discovery and merge.
+//
+// A build's translations come from ordered layers: each dependency library
+// that ships a locales/ directory, in transitive dependency order (a library
+// before any library that depends on it), then the consuming project's own
+// locales/ last. Later layers win on merge, so the app overrides its
+// libraries and a library overrides its own dependencies.
+
+import { existsSync, readFileSync } from "node:fs"
+import { join } from "node:path"
+
+// Parse a JSON catalog file, or null when it does not exist.
+export function readCatalog(path) {
+    if (!existsSync(path)) return null
+    return JSON.parse(readFileSync(path, "utf8"))
+}
+
+// Dependency names declared in a pxt.json, or [] when the file is absent.
+function readDeps(pxtJsonPath) {
+    if (!existsSync(pxtJsonPath)) return []
+    const pxt = JSON.parse(readFileSync(pxtJsonPath, "utf8"))
+    return Object.keys(pxt.dependencies || {})
+}
+
+// The ordered locale layers for a project rooted at `root`: every dependency
+// module (resolved under pxt_modules) that carries a locales/ directory, in
+// transitive dependency order, followed by the project's own locales/ as the
+// final, highest-priority layer named "app". Each layer is { name, localesDir }.
+export function discoverLayers(root) {
+    const modulesDir = join(root, "pxt_modules")
+    const seen = new Set()
+    const layers = []
+    const visit = name => {
+        if (seen.has(name)) return
+        seen.add(name)
+        const modDir = join(modulesDir, name)
+        for (const dep of readDeps(join(modDir, "pxt.json"))) visit(dep)
+        const localesDir = join(modDir, "locales")
+        if (existsSync(localesDir)) layers.push({ name, localesDir })
+    }
+    for (const dep of readDeps(join(root, "pxt.json"))) visit(dep)
+    layers.push({ name: "app", localesDir: join(root, "locales") })
+    return layers
+}
+
+// Merge the per-language catalog of every layer into one table, later layers
+// winning. Returns the merged table and per-layer info (count is null when the
+// layer has no file for the language).
+export function mergeLayers(layers, lang) {
+    const merged = {}
+    const info = []
+    for (const layer of layers) {
+        const path = join(layer.localesDir, lang + ".json")
+        const cat = readCatalog(path)
+        if (cat === null) {
+            info.push({ name: layer.name, path, count: null })
+            continue
+        }
+        for (const k of Object.keys(cat)) merged[k] = cat[k]
+        info.push({ name: layer.name, path, count: Object.keys(cat).length })
+    }
+    return { merged, layers: info }
+}

loc-tools/charsets.mjs

@@ -0,0 +1,85 @@
+// Generic per-language field data.
+//
+// Any layer (discovered exactly like translation catalogs) may ship
+// locales/charsets.json:
+//
+//   { "<lang>": { "<fieldName>": "<string>", ... }, ... }
+//
+// Each field is a string of characters that a consuming library treats as
+// per-language configuration -- for example a keyboard alphabet. The tooling
+// is agnostic: it never interprets field names, only merges, validates, and
+// emits them as `_loc.<fieldName>` assignments (see emit.mjs). The library that
+// owns a field declares the matching `_loc` member.
+//
+// Language resolution per layer: the exact language key, else the base-language
+// prefix (so "es" serves "es-ES"). Fields merge across layers per language,
+// the app layer winning.
+
+import { join } from "node:path"
+import { readCatalog } from "./catalogs.mjs"
+import { toHexCp } from "./fonts.mjs"
+
+// The base-language prefix of a locale tag ("es" for "es-ES", "fr" for "fr").
+function baseLang(lang) {
+    const i = lang.indexOf("-")
+    return i < 0 ? lang : lang.slice(0, i)
+}
+
+// Resolve the merged field set for a language across all layers. Returns an
+// object mapping field name to its resolved string (empty when no layer ships
+// data for the language).
+export function resolveCharsets(layers, lang) {
+    const merged = {}
+    for (const layer of layers) {
+        const data = readCatalog(join(layer.localesDir, "charsets.json"))
+        if (!data) continue
+        const entry = data[lang] || data[baseLang(lang)]
+        if (!entry) continue
+        for (const field of Object.keys(entry)) merged[field] = entry[field]
+    }
+    return merged
+}
+
+// Validate resolved field data. These are hard errors, never silent drops:
+//   - a `<x>Lower`/`<x>Upper` field pair (both present) must have equal
+//     code-point length;
+//   - no field may repeat a code point within itself;
+//   - every code point of every field must be present in the build font's
+//     coverage.
+export function validateCharsets(fields, fontCoverage, lang) {
+    for (const [name, str] of Object.entries(fields)) {
+        const seen = new Set()
+        for (const ch of str) {
+            const cp = ch.codePointAt(0)
+            if (seen.has(cp))
+                throw new Error(
+                    `charsets: ${lang} field ${JSON.stringify(name)}: duplicate character ${ch} ${toHexCp(cp)}`,
+                )
+            seen.add(cp)
+        }
+    }
+
+    for (const name of Object.keys(fields)) {
+        if (!name.endsWith("Lower")) continue
+        const upper = name.slice(0, -"Lower".length) + "Upper"
+        if (fields[upper] === undefined) continue
+        const lowerLen = Array.from(fields[name]).length
+        const upperLen = Array.from(fields[upper]).length
+        if (lowerLen !== upperLen)
+            throw new Error(
+                `charsets: ${lang} pair ${JSON.stringify(name)}/${JSON.stringify(upper)}: ` +
+                    `code-point length ${lowerLen} != ${upperLen}`,
+            )
+    }
+
+    for (const [name, str] of Object.entries(fields)) {
+        for (const ch of str) {
+            const cp = ch.codePointAt(0)
+            if (!fontCoverage.has(cp))
+                throw new Error(
+                    `charsets: ${lang} field ${JSON.stringify(name)}: character ${ch} ${toHexCp(cp)} ` +
+                        `not in build font coverage`,
+                )
+        }
+    }
+}

loc-tools/emit.mjs

@@ -0,0 +1,55 @@
+// Generation and restore of the app's loc.g.ts.
+//
+// loc.g.ts holds one shipped language's translation table (and any per-language
+// field data) for a per-language build. It is a transient build intermediate:
+// the generator writes a language's content, builds, and restores the default
+// state afterward. The default state assigns nothing, so a vanilla build falls
+// back to the source strings.
+
+import { writeFileSync } from "node:fs"
+
+// A generic assign-nothing loc.g.ts, used when a consumer does not supply its
+// own default content. The runtime falls back to the source strings, so a
+// vanilla build produces the source-string image with no localization tooling
+// involved. A consumer whose build depends on loc.g.ts being listed first in
+// pxt.json (for example to fix font-capture ordering) supplies its own text.
+export const GENERIC_DEFAULT_LOC_G = `// Generated placeholder for a per-language build. In this default state it
+// assigns nothing: the runtime falls back to the source strings.
+//
+// This file must exist because it is listed in pxt.json "files". A per-language
+// build writes a translation table here, builds, and restores this default
+// state afterward. Only the default state belongs in a commit.
+`
+
+// Emit loc.g.ts content for one language. `table` is the id-to-string catalog
+// (emitted only when non-empty); `charsets` maps field names to their resolved
+// strings (each emitted as a `_loc` member assignment). Both live inside one
+// reopened `namespace _loc` block.
+export function emitLocG(lang, table, charsets) {
+    const keys = Object.keys(table).sort()
+    const fields = Object.keys(charsets).sort()
+    const lines = []
+    lines.push("// Generated for a per-language build. Holds the one shipped language's")
+    lines.push("// translation table and per-language field data.")
+    lines.push("//")
+    lines.push("// Transient build intermediate: the generator restores the default state of")
+    lines.push("// this file after the run. Do not commit this generated content.")
+    lines.push("// lang: " + lang)
+    lines.push("namespace _loc {")
+    if (keys.length > 0) {
+        const entries = keys.map(k => "        " + JSON.stringify(k) + ": " + JSON.stringify(table[k]))
+        lines.push("    table = {")
+        lines.push(entries.join(",\n"))
+        lines.push("    }")
+    }
+    for (const field of fields) {
+        lines.push("    " + field + " = " + JSON.stringify(charsets[field]))
+    }
+    lines.push("}")
+    return lines.join("\n") + "\n"
+}
+
+// Write the default (assign-nothing) state to loc.g.ts.
+export function writeDefault(locGPath, defaultLocG) {
+    writeFileSync(locGPath, defaultLocG)
+}