[MaxGhenis]

Closes the gaps #105 left in scripts/build_manifest.py and the dashboard's build hooks: its follow-up items 1 to 3, plus the diff-size problem #106 hit. Draft until CI passes on Linux and an independent review agrees.

1. A missing PyYAML stops the build

• Without PyYAML, main() prints how to run the script through uv and exits 1 before it writes anything. parse_frontmatter raises the same message for direct callers.
• I deleted the subset parser (parse_yaml_subset). Every frontmatter block in the repo's skills, agents, commands and analyses parses as a YAML mapping, so nothing needs it, and its output broke the Analyses page. The regex scan for metadata.category in load_skills existed only for that path, and no skill used it, so it went too.
• Frontmatter that isn't valid YAML, or isn't a mapping, now stops the build with a FrontmatterError that names the file. Before, it dropped silently to the subset parser.
• The builder reads every input as UTF-8 (read_utf8) and writes the manifest as UTF-8 with \n line endings. Before, every read used the locale's encoding: under a Latin-9 locale (LC_ALL=en_US.ISO8859-15) on Python 3.11, this PR before the review fixes stopped on skills/policyengine/SKILL.md with "unacceptable character #x0080", and main's builder exits 0 after corrupting 5 of the 6 analyses (numbers turn into strings in 4, and all 5 lose their country) and finding 323 overlaps instead of 326 (on main at a8d02cf). Python 3.15 runs in UTF-8 mode by default, so it wasn't affected. Both versions now match the committed manifest under that locale.
• In dashboard/package.json, manifest runs uv run python scripts/build_manifest.py. manifest:safe runs the same command and keeps the committed manifest when the build can't run.

Before this PR, /usr/bin/python3 scripts/build_manifest.py (macOS's Python 3.9.6, no PyYAML) exited 0. It wrote "child_poverty_pct_change": "-34.8" and "country": null into every analysis. Now:

build_manifest.py needs PyYAML, and /Applications/Xcode.app/Contents/Developer/usr/bin/python3 can't import it.
Run the script through uv, which installs the repo's dependencies:
    uv run python scripts/build_manifest.py
The script wrote nothing.

I ran the package.json scripts through bun, which runs the same script strings:

Case	Result
manifest, uv on PATH	builds through uv
manifest, no uv on PATH	uv: command not found, exit 127, manifest untouched
manifest:safe, no uv on PATH	keeps the committed manifest, exit 0
manifest:safe in a copy of dashboard/ alone	keeps the committed manifest, exit 0
builder under /usr/bin/python3	exit 1 with the message above, manifest untouched

2. CI checks that the manifest is fresh

• --check builds the manifest without writing it and compares it with the file on disk, with generated_at blanked in both. On a mismatch it prints the sections that differ, the first 200 diff lines and the command that fixes it, then exits 1. When the file on disk isn't valid JSON, or holds JSON that isn't an object, it says so instead of raising a traceback.
• A new manifest job in .github/workflows/test.yml runs --check on ubuntu-latest under Python 3.11 and 3.15, the oldest and newest Python the repo supports. The existing test job already runs the suite under 3.15. The job installs dependencies in its own step, so a failed install shows as that step failing rather than as a stale manifest, and its error annotation says the log gives the reason.
• I couldn't run Linux here (colima's Docker context is down), so I took the platform out of the question instead of waiting to see what Linux does. compute_overlaps now takes IDF from the decimal module's correctly rounded ln, not math.log, which rounds however the platform's C library does. On macOS, math.log misses the correctly rounded value of ln((n + 1) / (df + 1)) for 44 of the 79,799 pairs with 2 ≤ n < 400 and 1 ≤ df ≤ n. So a manifest built on a Mac could fail the Linux check whenever the artifact count hit one of those ratios. None of them occurs at today's 80 artifacts, so no score changes.
• CI confirms it. On Ubuntu 24.04 (x86_64, glibc), both manifest jobs report that the manifest I built on macOS matches a fresh build: under CPython 3.11.18 and 3.15.2 at the head, and under 3.11.18 and 3.14.10 at the first push. In a fresh clone, editing one skill without rebuilding makes the same command exit 1 and list artifacts, overlaps as the sections that differ.
• On macOS (arm64), --check matches under Python 3.11.15, 3.12.14, 3.13.13, 3.14.7 and 3.15.2, and under 3.11.15 and 3.15.2 with LC_ALL=en_US.ISO8859-15.
• shared_terms now break exact ties by term, so their order never depends on where a word first appears in a text. On main's current corpus (d34fd48) that reorders 7 of the 324 lists; in 3 of them a different term makes the 8-term cut.

3. A test guards fsum

test_overlaps_never_use_builtin_sum swaps the built-in sum(), as build_manifest sees it, for one that returns 1% more for floats, then checks that compute_overlaps returns the same pairs. A manifest comparison would catch a return to sum() only on Python 3.11, where sum([0.1] * 10) gives 0.9999999999999999 (3.12 on gives 1.0). This test fails on every version. test_overlaps_never_use_the_c_librarys_log does the same for math.log.

Mutation check: each of these edits to the builder makes its test fail:

• sum() for the norm
• sum() for the score
• math.log for IDF
• dropping the term tie-break
• sorting pairs by score
• swallowing bad YAML
• a --check that ignores content
• removing both PyYAML guards

Removing only main()'s PyYAML check changes nothing a user sees, since parse_frontmatter still raises the same message, so that test still passes. test_check_fails_cleanly_on_a_malformed_manifest fails against this PR's first push, whose --check raised AttributeError on a manifest holding [].

Two tests came from the review:

• test_builder_never_uses_the_locale_encoding runs the build and --check with -X warn_default_encoding -W error::EncodingWarning, so a read or write that leaves the encoding out fails it under every Python version, 3.15 included. It fails if read_utf8 drops its encoding, if one input goes back to read_text(), or if the write drops its encoding.
• test_bad_frontmatter_makes_the_script_exit_1_and_write_nothing fails if main() returns 0 after a FrontmatterError. CI's --check and the dashboard's predev hook rely on that exit status.

4. Overlap diffs shrink

• compute_overlaps no longer sorts pairs by score. They follow artifact order: by a, then by b.
• dashboard/src/data.ts exports overlapsByScore, a stable sort by score, and Drawer.tsx (first 8) and Duplicates.tsx (first 50) read it. A stable sort keeps tied scores in artifact order, the order the old manifest gave them, so both views list the same pairs in the same order as before. The shared terms they print change for 4 pairs, because of the term tie-break in section 2: the Drawer's first 5 for program-reviewer and backdate-program, and the Duplicates page's first 6 for that pair and for dashboard-planner and dashboard-scaffold, prior-scores-finder and prior-scores, and prior-analysis and prior-scores. I checked:
  • the whole list: on main's current corpus (d34fd48), a stable sort of this PR's 324 pairs by score reproduces main's builder's order exactly, across 31 adjacent ties (and on a8d02cf, 326 pairs across 26 ties). Both views take their first 8 or 50 from that list, so they match too;
  • the Duplicates raw view: 29 same-kind pairs at 30% or more, checked in the browser at the first push;
  • the policyengine skill's drawer: the same 8 pairs in the same order, three of them tied at 23%, checked in the browser at the first push.
• I rebuilt Correct how the data skill says the US default got its 57,240 households #106's edit both ways: with pairs ordered by score it changes 1,084 manifest lines, and in artifact order 242.
• Stop the data skill steering agents to microimpute and uk-data #108 and Require every reported number to come from a model run #92 edited skills, an agent and every bundle without rebuilding the manifest, so this PR's rebuild also carries those edits: 5 changed artifacts, one new and two dropped overlap pairs, 282 moved scores, and changes to bundles_raw and repo_coverage. Built from main's current corpus, main's builder and this branch's give the same 324 pairs and the same scores, and differ only in overlaps: their order and 7 shared_terms lists. The manifest diff has 3,908 insertions and 3,929 deletions, nearly all from reordering overlaps once.

5. Left out: the 6,000-character cap in infer_target_repos

47 of the 80 artifacts have bodies longer than 6,000 characters. Dropping the cap adds 67 artifact-to-repo links across 26 artifacts (66 before #108). It would also make a separate bug worse: aliases match as substrings. "SSI" matches inside "missing" in 17 artifacts (policyengine-us), "PIP" inside "pipeline" in 11 (policyengine-uk), and "GIS" inside "registry" or "legislative" in others (policyengine-canada). Word-boundary aliases on their own remove 64 links. Both changes move repo_coverage enough to need their own review, so they belong in a follow-up: #109 takes them up.

Invariants

• A rebuild with no content change rewrites only generated_at, whatever the hash seed (tests), the Python version from 3.11 to 3.15 (macOS runs of each; CI checks 3.11 and 3.15 on Linux), the platform between macOS and Linux (CI on Linux against the manifest I built on macOS), or the locale (the EncodingWarning test, and macOS runs under Latin-9).
• The builder either writes a manifest built from fully parsed YAML, or writes nothing and exits non-zero (tests for missing PyYAML and for bad frontmatter).
• --check never writes.
• overlaps lists each pair once, a before b, in artifact order.
• A pair's score and shared_terms depend only on the words in the corpus, not on the order of artifacts or of words within a text (Hypothesis property test).
• Overlap scores don't depend on how the built-in sum() or the C library's log rounds.

Tests

• uv run pytest --ignore=tests/test_skill_examples.py: 224 passed (Python 3.14.7, macOS). CI's test job runs the same suite under CPython 3.15.2 on Linux.
• tests/test_build_manifest.py: 21 passed under Python 3.11.15 too.
• bun run build (tsc -b && vite build) passes.
• In the built dashboard, the Analyses page renders its numbers, and the console shows no errors.

Other PRs that touch the manifest

Rebase onto main and regenerate with uv run python scripts/build_manifest.py rather than hand-merging; this PR did that on top of #108 and again on top of #92. The open PRs #94, #95, #100, #109 to #112 and #116 all change files that feed the manifest, so once this lands, the new manifest check will fail each of them until it does the same.

axiom: n/a: tooling (dashboard manifest builder, hooks and CI), no policy change.

🤖 Generated with Claude Code