[anth-volk]

Fixes #1341

Merge only after PolicyEngine/calibration-diagnostics#255 (which reads schema 9) is merged and deployed to production. #255 is stacked on calibration-diagnostics#253; both deploy production when merged. A Microcosm merge deploys nothing, but the next release a build publishes would be schema 9, and the dashboard rejects schema 9 until #255 is live ("Unsupported calibration diagnostics schema_version 9"). Also ready first, or knowingly accepted: microcosm-collector's planned content handler for calibration-diagnostics submissions (not written yet; intake today checks only role, size and SHA-256) must accept schema 9.

Base: #1298's branch

This targets calibration-diagnostics-json-schema (#1298, head aa25c0926), not main, because the JSON Schema has to be regenerated for schema 9: #1298 publishes a JSON Schema generated from CalibrationDiagnosticsV8 for consumers that check the file without importing Microcosm (the collector), and that schema refuses schema 9 (schema_version const 8, closed objects). This PR adds a schema 9 file beside it with the same generator. #1298's branch already contains main at 954257d, including #1336, which this PR needs (see "Tests").

Because the base is not main, CI does not run on this PR (deliberately, to limit runners). Local checks are listed below. Once #1298 merges, GitHub retargets this PR to main and CI runs.

Problem

Schema 8 copies every provider, category, geography and dimension label onto every target row. Microcosm checks that the facts inside one target agree and, since #1336, that a US registry gives each geography ID one label; nothing checks dimension values across targets, and other countries only warn. Published files therefore give one identifier two labels:

• US national populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z: us:statutes/26/62#adjusted_gross_income under_1 (106 targets / 2), all (47 / 53), age 85_plus (51 / 1). The current US feed pin (Chronicle 51aa40f) still produces all three.
• US ACS local …-766b7c7-…: 51 states "Maine" / "ME" (fixed in the producer by Give each US geography one label across target sources #1336).
• UK (measured from the dashboard's published trees, whose target detail rows carry each row's schema 8 hierarchy): Chronicle's local facts carry the area as a dimension named geography whose value ID is the lower-case GSS code, labelled with each publisher's name. The October 2026 dense release microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe gives 88 areas two names (61 E06 unitary authorities "Hartlepool UA" / "Hartlepool", 22 Welsh W06, one each E08, S12, E12, plus constituencies E14001581 "Weston-Super-Mare" / "Weston-super-Mare" and W07000112 "Ynys Mon" / "Ynys Môn"); the October national releases give one ("Yorkshire And The Humber" / "Yorkshire and The Humber"). The geography nodes themselves have one label per ID in every current UK release.

Design: diagnostics schema 9

Schema 9 (CalibrationDiagnosticsV9) is schema 8 with each hierarchy label declared once in a top-level hierarchy_catalogs object and referred to by ID from the rows. Every other field is schema 8's.

{
  "schema_version": 9,
  "hierarchy_catalogs": {
    "providers":   [{"id": "obr", "label": "Office for Budget Responsibility"}],
    "categories":  [{"id": "obr.efo_receipts", "label": "Economic and fiscal outlook receipts", "provider_id": "obr"}],
    "geographies": [{"id": "K02000001", "label": "United Kingdom", "level": "country"}],
    "dimensions":  [{"id": "obr.efo_line", "label": "Economic and fiscal outlook line",
                     "values": [{"id": "income_tax", "label": "Income tax"}]}]
  },
  "targets": [{
    "name": "obr.income_tax@2025", "target_name": "obr.income_tax", "...": "every other schema 8 field",
    "hierarchy": {
      "category_id": "obr.efo_receipts",
      "geography_id": "K02000001",
      "dimensions": [{"id": "obr.efo_line", "value_id": "income_tax"}],
      "target": {"id": "obr.income_tax", "label": "Income tax receipts"}
    }
  }]
}

• Geographies are keyed by ID with one label and one level (each geography ID has one level in every published US file and UK tree).
• Dimensions list their values; a row's value_id is looked up under its own dimension. A row's dimensions keep producer order.
• Providers and categories move into catalogs too. Their labels have never varied, but once the row shape changes it costs nothing, the file then mirrors the country contracts' authoring catalogs (hierarchy.providers, hierarchy.categories), and a category names its provider once, so a category under two providers cannot be written. A row names only its category.
• The target's own label stays on the row; a target is one row.
• Catalogs are arrays, so a repeated ID stays visible (a JSON object keeps the last duplicate key silently).
• The model refuses an ID listed twice, a category whose provider is not listed, and a row that refers to an unlisted category, geography, dimension or value (looked up under its own dimension), or lists a dimension twice.

Change

• Models (microcosm.diagnostics.schema): CalibrationDiagnosticsV9, TargetDiagnosticV9, HierarchyCatalogs and its entries, CatalogHierarchyReference. CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION is now 9; producers keep assembling schema 8 (CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION); HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS is {8, 9}; parse_calibration_diagnostics reads 6 to 9. CalibrationDiagnosticsV8 is unchanged (its count checks moved into a shared function; the committed v8 JSON Schema regenerates byte for byte).
• Catalogs (microcosm.diagnostics.catalogs): catalog_calibration_diagnostics builds the catalogs from the targets, in first-use order, and raises HierarchyLabelConflictError listing every identifier the rows label two ways (or a geography placed at two levels), each spelling with its first target, unless a HierarchyLabelChoices names the label. A choice applies only where the rows disagree, and every disagreement it settles is logged as a warning naming the identifier, the spellings and the label written. expand_calibration_diagnostics gives back schema 8.
• Writer: publishes schema 9; a refusal returns DiagnosticsWriteFailure with the new code label_conflict, writes nothing and logs every conflict (the outcome message is shortened to 500 characters). As before, a diagnostics failure does not fail the dataset build; the release manifest records failed.
• Label choices (microcosm.calibrate.hierarchy_labels), applied by the calibrate writer and the transport diagnostics kernel from the registry's country (UK, US and Belgium; see the next three sections).
• Other producers: the UK writer (write_uk_calibration_diagnostics), the transport diagnostics kernel and the UK graph's final validation kernel publish the same schema 9 document; the two kernels raise on a conflict, as they already raise on an invalid schema 8 document.
• Readers: the release contract treats 8 and 9 alike wherever it handled the current version, and an available declaration may name 8 or 9 (failed: expected 8 or 9). The UK local scorer and the UK dense assembler accept both.
• JSON Schema: calibration_diagnostics.v9.schema.json is generated from CalibrationDiagnosticsV9, ships in the wheel beside the v8 file (checked with uv build --wheel), and lists the model rules it cannot express (CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS: schema 8's, with the row-hierarchy rule replaced by catalog-identity and reference rules). tools/generate_calibration_diagnostics_json_schema.py writes or checks both files, or one with --schema-version.
• Docs: docs/calibration-target-hierarchy.md (new schema 9 section), the agent guide, the transport kernel doc, the root, diagnostics, calibrate and data READMEs, and changelog.d/1341-diagnostics-schema-9-catalogs.changed.md.

What refusal means for the UK, and the handling

A catalog holds one label, so without handling the writer would refuse today's UK files: the dense release (88 values) and the national releases (1 value). UK builds would still finish, but their releases would carry calibration_diagnostics: failed, and UK release workflows stop when diagnostics are unavailable.

The least risky handling chooses the area's label from what Microcosm already treats as authoritative, only for the geography dimension and only where the rows disagree:

1. the payload's own geography label for that code (the label the dashboard already shows on the area's geography node);
2. else UK_GEOGRAPHY_ID_TO_LABEL (the UK, nations and English regions);
3. else the ONS April 2023 names in local_authority_names.json (added by the UK build's writers).

Checked on the published UK trees: all 88 dense-release values and the national one resolve; in all 89 the file's geography label equals one of the spellings, and for the 87 that the catalogs cover (86 dense, 1 national) it equals the catalog's label (for example "Hartlepool", "Kingston upon Hull, City of", "Yorkshire and The Humber"); the two constituencies, which no Microcosm catalog covers, take the file's own geography label ("Weston-super-Mare", "Ynys Môn"). The September 2026 local candidates (staging, before Microcosm gave each geography one label) still have two constituency geographies with two labels and would be refused; no current build produces them.

A UK geography (not a dimension value) with two labels is resolved only by the catalogs; a constituency there is refused, because Microcosm has no constituency name catalog.

The three US values

The current US feed pin still produces two spellings of three values, so a refusing writer would stop US national diagnostics. Each gets the label chosen in review on 2026-10-21; a US feed re-pin being prepared brings the same labels from Chronicle, after which these choices apply only to older feeds. As with Chronicle's REVIEWED_VALUE_LABEL_VARIANTS (chronicle#377), each choice covers only the two spellings reviewed, so a third spelling is refused, and each use is logged as a warning:

Value	Spellings in release 0fb05b6	Written as
us:statutes/26/62#adjusted_gross_income = under_1	"No adjusted gross income" (106), "No adjusted gross income and deficit" (2; chronicle#377 fixes the source)	"No adjusted gross income"
us:statutes/26/62#adjusted_gross_income = all	"All returns" (47), "All returns, total" (53)	"All returns, total"
age = 85_plus	"Age 85 plus" (51), "Age 85 and over" (1)	"Age 85 plus"

Real-file check: the public 0fb05b6 file is refused without choices (3 conflicts) and written as schema 9 with them, with one warning listing the three. The calibration dashboard (calibration-diagnostics#255) builds the same 23 tree files from that schema 9 file as from the schema 8 file with those three labels applied. The public 766b7c7 file is refused (its 51 state conflicts are what #1336 fixed in the producer).

The Belgian value

Each Belgian Chronicle package that the Belgian target contract (be/target_references.json) reads states its national figure as the group-by value measure = country_total and labels it with its own measure name. Microcosm copies that label into each target's hierarchy (ledger_targets._inherited_dimensions), so a Belgian calibration gives one value up to four labels. Without a choice, the writer refuses it, and because Belgium builds through the transport graph, whose diagnostics step (diagnostics.calibration@1) raises on a refusal, a Belgian build would have failed, not merely recorded failed diagnostics as an earlier version of this description said.

The value is now written as "Country total": the label chosen in review on 2026-10-22, and the label Chronicle's canonical register gives it (chronicle#377). Like the US choices, it applies only where the rows disagree, covers exactly the four package spellings (Chronicle registers the same four as reviewed variants, chronicle#265), so another spelling is still refused, and every use is logged as a warning:

Chronicle package (at 6d74e42)	Microcosm target	Role	Label for measure = country_total
spf_finances/pit_2023	spf_finances_pit_total	calibration	"Belgium PIT federal and local tax before withholding"
onss/contributions_2024	onss_employee_contribution_total	calibration	"Worker-borne personal social security contributions"
onem_rva/unemployment_2024	onem_unemployment_caseload	calibration	"Complete unemployment benefit recipients"
nbb/national_accounts_household_disposable_income_2024	nbb_household_disposable_income	validation	"Household disposable income gross"

All four are written as "Country total". The two published Belgian releases carry no schema 8 hierarchy, and no Belgian build was run here; the failure and the fix are shown by tests that write a Belgian-style file through the calibrate writer, which takes its choices the same way the transport kernel does (hierarchy_label_choices(registry country)).

Other countries and new variants

Country	Choice	Values covered
UK	a geography dimension value takes its area's label	88 dense-release areas, 1 national value
US	one reviewed label per value	AGI under_1, AGI all, age 85_plus
Belgium	"Country total"	measure = country_total

• New UK variants: Chronicle also registers UK spellings such as sex = female ("Female", "Females", "female") and property_type values. No current UK release has them on one value ID (checked in the published trees), so nothing is chosen for them; a build that brings two of them together is refused and its warning names them.
• That refusal is the point of schema 9: a new disagreement stops the diagnostics write with every conflicting spelling named, instead of reaching the dashboard.

Tests (red, then green)

Commit	What	Result
b08c64607	Tests only: test_hierarchy_catalogs.py (new), test_json_schema_v9.py (new), test_hierarchy_label_choices.py (new), and changes to the schema, JSON Schema, calibrate, UK diagnostics, transport kernel, US fiscal-refresh, UK scorer, dense assembler and release contract tests; the JSON Schema test helper picks the committed schema by the document's version.	In the five changed modules that import: 21 failed, 465 passed, 8 skipped. Failures: the scorer refuses and the dense assembler rewrites a schema 9 candidate, the US fiscal-refresh writer writes schema 8, the v9 JSON Schema functions and the tool's --schema-version do not exist. The other 7 changed modules fail to import the new names (ImportError).
eb758cc51	The implementation and docs; no test file changed	All the above pass (below)
a5945f9ec	Test only: test_hierarchy_label_choices.py gains the Belgian choice, a Belgian-style file written with "Country total" and one warning, and a fifth, unreviewed spelling refused; the tests of countries without choices stop using be	That file: 3 failed, 9 passed. Failures: hierarchy_label_choices("be") is empty; the Belgian file is refused (failed, not available); the unreviewed spelling is refused without the "(a spelling its reviewed list does not include)" marker, because no Belgian choice exists
5c6cb3116	The Belgian choice in microcosm.calibrate.hierarchy_labels, the hierarchy doc and the changelog fragment; no test file changed	That file 12 passed; with the writer's other test files (test_diagnostics.py, test_hierarchy_catalogs.py, test_transport_terminal_kernels.py) 96 passed, 0 failed

Existing test expectations changed in the red commit, on purpose: the written file is schema 9 while the in-memory payload stays schema 8; a non-finite value added to an already-validated model is refused as validation_error (the writer validates the schema 9 document it builds) instead of serialization_error; the contract's message says "publishes version 9".

At 5c6cb3116: ruff check . clean; ruff format --check clean on the two changed Python files. The rest of this list was run at eb758cc51.

At eb758cc51, locally (CI does not run on this base):

• every test file that touches the diagnostics code paths (41 files, found by searching the test tree for the writer, parser, version constant, JSON Schema helper and label choices): 2,263 passed, 13 skipped, 0 failed;
• ruff check . clean; ruff format --check clean on the changed files; tools/ci_test_plan.py verify ok; uv lock --check ok (uv.lock unchanged); tools/generate_calibration_diagnostics_json_schema.py --check current for both files.

On #1298's previous head (c3244bfb4, before it merged main with #1336), two ACS local tests failed here: the ACS local tool still labelled state population targets "CA" beside the IRS SOI label for the same state, which schema 9 refuses. With #1336 in the base they pass. So this PR must not land on a base without #1336.

Environment variables

No workflow, deployment or configuration change, and no new environment variable is read.

Notes

• Releases already published keep their schema 8 files; readers here and in the dashboard read both.
• Unpushed local commits were amended while implementing (test corrections and two follow-ups from the coordinator's notes: reviewed spellings, the chosen US labels and the warning log). The pushed history is two red-then-green pairs (schema 9, then the Belgian choice), and each red commit is test-only.
• Nothing was merged, marked ready or dispatched.

🤖 Generated with Claude Code