[anth-volk]

Fixes #3967

Summary

Catalog format 2 stores, publishes and serves the metadata the app's v3 workspace reads from GET /{country}/metadata that the v2 catalog lacked. With #3969 this completes the plan's item 1E-1 (v2 metadata gaps for v3); the gap list is below.

• Model versions: entities (key, plural, label, description, is_person, roles), basic_inputs, and catalog_format.
• Variables: value_type (float, int, bool, str, date or Enum), definition_period, unit, module_name, is_input_variable, hidden_input, possible_value_labels.
• Parameters: metadata_label (the label in the parameter's own metadata, without PolicyEngine.py's generated fallback), period, economy, household.
• Parameter nodes: the bracket schedules (Core ParameterScale nodes), which PolicyEngine.py omits: 866 US and 41 UK, almost all with labels v3 shows as schedule names.

The metadata routes (/v2/variables, /v2/parameters, /v2/parameters/children, /v2/tax-benefit-models/by-country, /v2/tax-benefit-model-versions) and /v2/model-catalog return the new fields; the model-catalog document format number becomes 2, so its ETag changes.

Where the extractor reads them (decision for review)

PolicyEngine.py's public records (6.7.5, and 6.8.0) do not carry these fields. The extractor reads them from the country system PolicyEngine.py built its model from, through the model version's _load_system hook, by attribute. It imports neither PolicyEngine Core nor a country package, so api.catalog.extractor-import-isolation still holds, but _load_system is not part of PolicyEngine.py's public interface: a release that removes it makes extraction fail with an error (the seed job would stop the deploy). The alternative is to publish these fields in PolicyEngine.py's public Variable, Parameter and TaxBenefitModelVersion records, release it, and read them from there; that change is outside this repository. Values the extractor reads are validated as strictly as the existing fields (type checks, length bounds, enum labels must match the possible values, every public variable and parameter must exist in the country system).

Published versions are never rewritten

The generated revision adds nullable columns only. Publication still inserts a version only when it is absent. A version published before format 2 has catalog_format null, keeps null values, and later publications of it compare only the fields it was published with (no bracket-schedule nodes, no format 2 columns). So the first deploy succeeds with the running version unchanged, and the next PolicyEngine.py version published (for example through the automatic bundle-update PRs) carries the new fields. Until then the routes return them as null, which the app's adapter handles with v1's defaults.

Gap list: what v3 reads from GET /{country}/metadata and its v2 source

Found by following every import from the (workspace) route group in calculator-app and reading each metadata access, then comparing with the v2 routes.

v3 reads	v2 before this stack	Now
The whole parameter set with all values, at start-up (US: 100,276 government parameters, 1,174,586 values)	Paged at 500 per request	/v2/model-catalog (#3969)
Parameter node labels (breadcrumbs, tree folders, schedule and state names)	One tree level per request (/v2/parameters/children)	Nodes in /v2/model-catalog (#3969)
Bracket-schedule node labels	Absent	Added here
Parameter economy and household flags (which parameters search and the tree offer)	Absent	Added here
Parameter label as v1 shows it (own metadata label, else the last path segment)	Only PolicyEngine.py's generated label, which differs for 91% of US leaves	metadata_label here; the app applies v1's fallback
Parameter period	Absent	Added here
Parameter values by start date	Paged /v2/parameter-values; the earliest start is 0001-01-01 (v1: 0000-01-01)	Map in /v2/model-catalog; the app restores v1's spelling
Parameter path, unit, description	Present	Unchanged
Variable name, label, entity, documentation, default, adds, subtracts	Present	Unchanged
Variable value type (enum or not; int, float, bool, date)	data_type is str for enums	value_type here
Variable definition period, unit, module, input flag, hidden-input flag	Absent	Added here
Enum value labels	Names only	possible_value_labels here
Entities (plural, label, is_person; roles are not read)	Absent	Model version entities here
Basic inputs	Absent	Model version basic_inputs here
Regions: name, label, type; state abbreviation and name for congressional districts	/v2/regions (code, label, region_type, state_code, state_name)	Unchanged; the app maps them and leaves out places, which v1 never listed
Years, current-law ID	Model version metadata_time_periods, current_law_id	Unchanged
version (readiness)	policyengine_version	Unchanged; see below

Not filled here (each needs a decision or belongs to another group):

1. UK constituencies and local authorities. v2 has 5 UK regions (national and four countries); v1 lists 650 constituencies and 360 local authorities. v1 names them by name (constituency/Aldershot), while PolicyEngine.py names them by code (constituency/E14…) and builds them only from lookups in a private Cloud Storage bucket. v1's country/ni is country/northern_ireland in v2. This changes which region strings economy reports send, so it belongs with the v2 economy work (2A, 3A).
2. US SPM availability (spm.available). v3 requires it before a US household calculation; it depends on the certified SPM artifact, not the catalog. The v2 household calculation path (2B-2, 3B-2) should publish it.
3. National region labels. v1 says "the US" and "the UK"; v2 says "United States" and "United Kingdom".
4. Variables whose adds or subtracts name a parameter (237 US, 4 UK). PolicyEngine.py resolves the parameter to its 2025 list; v1 gave the parameter path, which v3 evaluates for the household's year.
5. version. v1 gave the country package version (policyengine-us 2.2.1); v2 gives the PolicyEngine.py version (6.7.5). v3 compares it with a report's model version to offer a rerun; that comparison moves with v2 report provenance (3A-3).
6. Not read by v3 and not filled: variableModules, modelled_policies, economy_options.datasets, variable category and indexInModule, parameter indexInModule.

Measured parity

The real 6.7.5 catalog was published to a disposable Postgres, served through the v2 routes, converted by the app's adapter (PolicyEngine/policyengine-app-v2#1347) and compared field by field with v1's documents:

• US parameters: labels 100,275 of 100,276 equal (the exception has an explicit null label in v1, which v3 displays the same way); descriptions, units, periods, economy and household flags 100% equal; values equal except two parameters whose source contains 2021-06-31, stored as 2021-06-30; all 29,310 node labels equal; the only missing v1 nodes are 8 empty ones (__pycache__ and similar).
• US variables: label, entity, documentation, value type, definition period, unit, module, input flag, hidden-input flag and default 100% equal; enum labels 6,165 of 6,167 (integer labels arrive as strings); adds 5,930 of 6,167 (point 4 above).
• Entities, basic inputs, current law and years equal (entity descriptions are trimmed of surrounding whitespace).
• US regions 487 of 488 identical (the national label). UK: the same results, and 3 of 1,015 regions (point 1).

Alembic

• Revision 72fab86fa97d ("store catalog presentation metadata") after fbb7488ebe8a, generated with uv run alembic -c alembic-v2.ini revision --autogenerate and only formatted. It adds 14 nullable columns and drops them on downgrade. V2_ALEMBIC_HEAD_REVISION is updated.
• Checked against a disposable Postgres: upgrade to head, alembic check (no drift), downgrade one revision, upgrade again; tests/integration/test_alembic_v2_lifecycle.py passes.
• Coordination: groups 1F (1F-2, countries table) and 1G (1G-2, Drop the empty simulation-domain tables and retired association columns from the v2 schema #3961) also add v2 revisions. Only one may be open against the chain head at a time: whichever merges second rebases and regenerates its revision on the new head (updating down_revision, V2_ALEMBIC_HEAD_REVISION and the revision-chain test).

API contract change

Additive: new optional fields on MetadataVariable, MetadataParameterSummary and MetadataModelVersionDetail, and new MetadataEntity and MetadataEntityRole schemas. Nothing calls them until the app's switch (plan item 4A-1); PolicyEngine/policyengine-app-v2#1346 and #1347 add the unused app code.

What stays on the v1 routes

Everything. GET /{country}/metadata and its callers are unchanged.

Testing

• tests/unit/v2/test_catalog_extraction.py: presentation metadata read through the country system; parameters without flags default to economy and household as v1 does; enum labels follow the possible values; bracket schedules are added; a model without its country system, records missing from it, enum labels that do not match the possible values, and flags that are not booleans stop extraction.
• tests/unit/v2/test_metadata_service.py, test_metadata_catalog.py: the routes and the catalog document return the new fields, and versions published before format 2 read as null; the fields a built document serves match the field list recorded for its document format, which the ETag names (this fails if a field changes without a new format).
• tests/unit/v2/test_catalog_publication.py, test_alembic_v2.py: both comparison modes build matching column lists; the revision is generated, reversible and the chain head.
• Disposable Postgres (tests/integration/test_v2_catalog_publication.py): new versions store the fields and schedules and republish cleanly; a version published before format 2 is compared and left unchanged (this test fails if publication compares the new fields for such a version); a format 2 version with a changed flag is refused.
• RUN_V2_CATALOG_COMPATIBILITY=1 RUN_V2_CATALOG_PUBLICATION_QUALIFICATION=1: the installed-catalog test (parameter nodes now 30,213, including the 907 schedules) and the full publication qualification pass.
• Results: uv run pytest tests/unit/v2 tests/contract tests/unit/test_migration_contract_artifacts.py -q: 882 passed; v2 Postgres integration suites: 49 passed; mypy, ruff format --check ., ruff check ., run_quality_guards.py passed; export_migration_contracts.py: no changes.
• After the review fixes (Serve a whole v2 metadata catalog version in one response (GET /v2/model-catalog) #3969's shared query model, and the tests and row construction added here): the same unit command plus tests/unit/test_query_parameters.py tests/unit/test_asgi_factory.py: 968 passed; test_alembic_v2_lifecycle.py, test_v2_catalog_publication.py, test_v2_metadata_routes.py and test_v2_user_identity_persistence.py against a disposable Postgres: 42 passed; mypy, ruff format, ruff check, run_quality_guards.py passed; export_migration_contracts.py: no changes.

Merge order and deployment

• Stack: Let a saved policy be repointed to a v2 policy (PATCH /v2/user-policies policy_id) #3968 → Serve a whole v2 metadata catalog version in one response (GET /v2/model-catalog) #3969 → Store the metadata v3 reads from /{country}/metadata in the v2 catalog #3970 (this). Merge bottom-up.
• Merging deploys the migration and runs the existing seed job (seed-v2-database.yml, called from push.yml for staging and production). No new environment variables or secrets: the seed job keeps using V2_DATA_WRITE_DATABASE_URL and the migration job its existing URL; the workflows are unchanged.
• No operator step is needed. The new fields appear for the first PolicyEngine.py version published after deploy.

Possible conflicts

• The Alembic chain with 1F-2 and 1G-2 (above).
• The open ruff refactor PRs (Enable ruff structure and import rules (INP, PLC, SLF, N, A, FAST, E) #3920, Enable ruff argument, idiom and pytest-style rules (ARG, B, C4, FBT, PERF, PLW, PT, PTH, RUF, SIM and others) #3924, Refactor the economy and calculation services within ruff's complexity limits #3932–Refactor tests, scripts and the rules tool, then select every ruff rule (ALL) #3934) may touch the catalog extraction and publication modules.
• The automatic bundle-update PRs (Update PolicyEngine bundle to 6.7.6 #3954, Update PolicyEngine bundle to 6.8.0 #3957) change the installed-catalog counts asserted in tests/integration/test_v2_catalog_installed.py.

🤖 Generated with Claude Code