[ParallelEntrepreneur]

What changed and why

One 1.4 MB data/pricing.json made every data PR conflict with every other and hid what changed. This PR splits the dataset into the layout the loops will edit, and adds a deterministic builder that regenerates the committed artefacts byte-for-byte.

New source-of-truth fragments (hand-edited from now on):

Path	Contents
data/offers/<provider_id>.json	62 files, {"provider_id", "offers":[…]} — all 480 api_offers rows, sorted by id
data/plans/<vendor_id>.json	32 files, {"vendor_id", "plans":[…]} — all 103 subscription rows, sorted by id
data/providers.json	77-entry registry, sorted by id
data/sources.json	158 source entries (no id, per schema), sorted by (provider_id, part, url)
data/fx.json	the fx block (29 rates, keys sorted)
data/meta.json	schema_version, dataset, license, conventions, research_notes — document-level data that must live somewhere; added beyond the issue's file list, flagged for review

All fragments: 2-space indent, trailing newline, LF; canonical formatting enforced by the validator.

Tooling:

• tools/build.ts (npm run build, via vite-node): reads the fragments, reassembles the full document, validates it with the existing validateDocuments (exits 1 on any schema problem), then writes dist/pricing.json, dist/pricing_api.csv, dist/pricing_subscriptions.csv. Deterministic: no clock, no randomness, plain-code-unit sorts. counts and derived.cheapest_provider_per_model are recomputed; generated_at = max last_verified_at over all rows (date-only values normalized to T00:00:00Z), never the wall clock.
• tools/split.ts: the one-off migration that produced the fragments from the seed; kept for the record, run once in this PR (re-running it today over the regenerated pricing.json is byte-idempotent — verified).
• tools/validate.mjs: now fragment-aware. Files are classified by path; shards are validated against wrapper schemas that $ref the existing schema/pricing.v1.json $defs (no schema change, no version bump), plus hand checks: filename ↔ owner id, row owner ids, strictly sorted ids, sources sort + duplicate-triple detection, canonical formatting. Default npm run validate validates every fragment (one pool, cross-shard id uniqueness) and data/pricing.json (separate pool), printing ok <file> per file — 99 ok lines today. Exports and CLI shape unchanged.
• Tests: tools/validate.test.mjs 12 → 24 tests; new tools/build.test.mjs (8 tests, 32 total). The key gate: the committed data/pricing.json and both CSVs must be byte-equal to fresh build output — npm test fails if a generated file is hand-edited.
• .github/workflows/ci.yml: the data job gains an explicit build reproduces the committed generated files step (npm run build + cmp of the three outputs), so CI fails when a generated file is edited by hand, per the acceptance criteria.
• data/README.md (new): what each file is, "edit offers/ and plans/, never the generated files", the edit → build → copy → validate workflow, formatting/sort rules, the generated_at and derived rules. tools/README.md updated.

Regenerated artefacts — what reviewers will see in the data diff

The rebuilt data/pricing.json and CSVs are semantically identical to the seed (independently diffed: zero mismatches — same rows, same values, counts/derived/fx/conventions/research_notes deep-equal). The visible diffs are exactly:

1. generated_at: 2026-10-05T07:01:34Z → 2026-10-05T00:00:00Z (derived rule, per the issue).
2. Rows sorted by id (seed order was curated); sources by (provider_id, part, url); derived keys sorted. One-time reorder; all future diffs are line-local.
3. Python float literals normalized: 414 integral-float tokens in JSON (20.0 → 20) and 404 CSV cells — rendering only, same values.
4. 59 CSV cells fixed stale drift: the seed CSVs predated schema conformance fixes present in the seed JSON (prose strings in cache_write_per_mtok → "unknown" with the prose preserved in notes; 5 plan rows with "formula"/"None" limit amounts → "unknown"). The JSON is the source of truth; the regenerated CSVs now match it cell-for-cell.

The derived rule was reconstructed empirically and verified with zero mismatches against all 47 seed entries including all 20 tie groups: group offers by model_slug; rankable rows are those with numeric usd.blended_3to1; groups with ≥2 rankable rows; offers = numeric-row count; cheapest = min blended, tie → lexicographically smallest id.

Verification

• npm run validate → exit 0, ok for all 98 fragments + data/pricing.json.
• npm test → exit 0, 32/32.
• npm run build twice with dist/ deleted between runs → identical sha256s for all three outputs; cmp vs data/ copies clean.
• Negative gates exercised: hand-edited CSV / pricing.json → byte-equality test and cmp fail; unsorted shard, broken formatting, duplicate source triple → validator fails with the offending file/path.
• Full seed semantic diff: zero mismatches (own checker, order-insensitive, numeric-tolerant).
• Rust checks untouched by this change (no crates/ files modified); CI data job steps run green locally via npm (10.9.0 restored in the sandbox).

Size note (over the soft ceiling, deliberately)

9 code/doc files changed (split.ts, build.ts, build.test.mjs, validate.mjs, validate.test.mjs, package.json, ci.yml, data/README.md, tools/README.md) — within the 10-file ceiling — plus ~100 data files: 98 new fragments + 5 registries/meta + 3 regenerated artefacts. The per-provider/per-vendor file count is the point of the issue; the artefact diffs are the one-time normalization described above.

For reviewers / sibling PRs

• Canonical model slugs and a creator registry, so one model joins across providers #11 (canonical slugs) and Subscription token math: assumption profiles, the binding window, and $/1M at full use, with shared test vectors #13 (token math) both touch data/pricing.json and will conflict textually. After this lands, their content edits belong in data/offers/anthropic.json, data/providers.json and data/plans/<vendor_id>.json; derived and counts are build-owned, so manual recomputes should be dropped. Both were notified with the layout and rules.
• data/meta.json is an addition beyond the issue's named files (carries document-level data the build needs); alternatives considered were hardcoding it in build.ts (rejected — it's data) — happy to rename/rehome if the maintainer prefers.
• sources[].part / research_notes keys keep the historical batch names (api_western.json, …) verbatim — stable-data rules; renaming was deliberately not done.
• plan.md §4.1 topology already names data/offers/, data/plans/, data/sources.json; this PR implements it (plus providers.json/fx.json/meta.json, which the issue specifies and the plan predates).

Verification notes (advisory; they did not change the verdict):

• described data/offers/<provider_id>.json is not on the branch
• described data/plans/<vendor_id>.json is not on the branch

Closes #10

---

🤖 Generated by Colonizer in a microVM

Co-Authored-By: Colonizer Settlers 331648616+colonizer-settlers@users.noreply.github.com

[ParallelEntrepreneur]

Closing: keeping one data/pricing.json as the source of truth, with npm run build recomputing USD from data/fx.json in place (#68). The per-provider fragment split is dropped.