Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2)

.github/workflows/v2-integration-check.yml

@@ -73,8 +73,8 @@ jobs:
         run: uv run python scripts/reconcile_v2_privileges.py apply --create-missing-roles
       - name: Verify the managed v2 roles hold their declared privileges
         run: uv run python scripts/reconcile_v2_privileges.py verify
-      - name: Verify the installed PolicyEngine.py catalog interface
-        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py
+      - name: Verify the installed PolicyEngine.py catalog interface and its parity with v1 metadata
+        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py tests/integration/test_v2_catalog_metadata_parity.py
         env:
           RUN_V2_CATALOG_COMPATIBILITY: "1"
       - name: Test v2 metadata publication and resource routes

changelog.d/v2plan-1f-115-catalog-metadata-fields.added.md

@@ -0,0 +1 @@
+Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records: each model version's entities and basic inputs; each variable's input and hidden-input flags, module, the parameter path its adds or subtracts names, its enum values' labels, its unit and its definition period; each parameter's economy and household flags. Bracket schedules are stored as parameter nodes with their stored label. A generated revision adds the columns as nullable, so catalog versions published earlier keep them empty and are compared without them when republished. An opt-in parity test, run by the v2 integration check, compares every new field for the US and the UK with /{country}/metadata. Each country's default dataset now comes from the installed PolicyEngine.py release's manifest; the Public API no longer fixes its own (populace_us_2024 and microcosm_uk_2024_25), and publication stops if the release names no default or does not certify it.

docs/migration/stage-9-v2-metadata.md

@@ -99,6 +99,34 @@ metadata time-period options are stored on the model-version snapshot.
 PolicyEngine Core and country-package versions are derived diagnostic evidence
 and are not separately persisted.
 
+Catalog format 2 adds the metadata the app's v3 workspace reads, taken only
+from PolicyEngine.py's public records, which carry it from the release that
+publishes entities, basic inputs and these flags:
+
+| Record | Fields |
+| --- | --- |
+| Model version | `catalog_format` (2), `entities` (`key`, `plural`, `is_person`, in the package's order), `basic_inputs` (the household builder's default fields) |
+| Variable | `is_input_variable`, `hidden_input`, `module_name`, `adds_parameter_path` and `subtracts_parameter_path` (the parameter whose dated value lists the variables summed; `adds` and `subtracts` keep the list PolicyEngine.py resolved), `enum_values` (`value` and declared `label`, in order), `unit` (for example `currency-USD`) and `definition_period` (`year`, `month` or `eternity`), which the household builder and breakdown read |
+| Parameter | `economy`, `household` |
+
+Bracket schedules are parameter nodes with their stored label, or none when the
+package stores none. An empty basic-input list is stored as empty. Extraction
+stops before reading any record if the installed PolicyEngine.py does not
+publish these fields, and names each missing field: 6.7.5 and the locked 6.10.0
+publish only the variables' unit and definition period. It also stops if the
+model has other than one person entity, names a basic input or a parameter path
+that is not published, gives enum labels that do not match the variable's
+possible values or are not JSON scalars, or gives a flag that is not a boolean.
+An opt-in test (`tests/integration/test_v2_catalog_metadata_parity.py`, run by
+the v2 integration check) compares every format 2 field for the US and the UK
+with what `/{country}/metadata` builds.
+
+Publication never rewrites a stored version. A version published before
+format 2 has a null `catalog_format` and null format 2 fields, and later
+publications of the same PolicyEngine.py version compare it without those
+fields. The first PolicyEngine.py version published after format 2 deploys
+carries them.
+
 Parameter extraction requires the PolicyEngine.py public history contract from
 5.0.4 and later: values are ordered from oldest to newest, bounded `end_date`
 values are inclusive and fall one day before the following `start_date`, and
@@ -111,16 +139,20 @@ Regions come from the Public API's checked-in reference files, one per country
 (`policyengine_api/data/v2/catalog/region_reference_files/`), not from
 PolicyEngine.py's region registry. Each row lists a region's code, label, type,
 parent, US state, the value that selects its households, and the v1 identifier
-it replaces. A scoping rule per country and region type gives each region its
-dataset from the certified data release and its filter:
+it replaces. Each country's default dataset is the one the installed
+PolicyEngine.py release names in its manifest (`default_dataset`); the Public
+API sets none of its own. Publication stops if the manifest names no default,
+if the release does not certify it, or if the release routes national regions
+to another dataset. A scoping rule per country and region type gives each
+region its dataset from the certified data release and its filter:
 
 | Regions | Dataset | Filter |
 | --- | --- | --- |
-| US national | the release's national dataset (`populace_us_2024`) | none |
+| US national | the release's default dataset (`populace_us_2024` in 6.10.0) | none |
 | US states and congressional districts | the release's state and district dataset | `row_filter` on `state_fips` or `congressional_district_geoid` |
-| UK national | the release's national dataset (`microcosm_uk_2024_25`) | none |
-| UK nations | the release's national dataset | `row_filter` on `region` |
-| UK constituencies and local authorities | the release's local-area dataset (`microcosm_uk_2024_25_dense`), which carries their codes | `row_filter` on `constituency_code_oa` or `la_code_oa`, value the ONS code |
+| UK national | the release's default dataset (`microcosm_uk_2024_25` in 6.10.0) | none |
+| UK nations | the release's default dataset | `row_filter` on `region` |
+| UK constituencies and local authorities | the release's local-area dataset (`microcosm_uk_2024_25_dense` in 6.10.0), which carries their codes | `row_filter` on `constituency_code_oa` or `la_code_oa`, value the ONS code |
 
 If the release names no dataset for a US state or district, those regions use
 the default dataset and the command emits one summary warning with the affected
@@ -137,8 +169,8 @@ household columns. It does not check that a column holds rows for every
 region's value.
 
 Review the fallback and omitted-region summaries on every new PolicyEngine.py
-release. A changed count may identify an upstream catalog change that needs a
-reviewed dataset selection even when publication otherwise succeeds.
+release. A changed count may identify an upstream change to the certified data
+release even when publication otherwise succeeds.
 
 A catalog version is identified by its country and PolicyEngine.py version and
 never changes once published. When the command republishes a stored version

migrations/v2/versions/f5c1d9b4685b_store_catalog_format_2_metadata.py

@@ -0,0 +1,94 @@
+"""store catalog format 2 metadata
+
+Revision ID: f5c1d9b4685b
+Revises: fa7fa1035444
+Create Date: 2026-10-26 20:13:40.733924
+Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+
+
+revision: str = "f5c1d9b4685b"
+down_revision: Union[str, None] = "fa7fa1035444"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column("parameters", sa.Column("economy", sa.Boolean(), nullable=True))
+    op.add_column("parameters", sa.Column("household", sa.Boolean(), nullable=True))
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("catalog_format", sa.Integer(), nullable=True),
+    )
+    op.add_column(
+        "tax_benefit_model_versions", sa.Column("entities", sa.JSON(), nullable=True)
+    )
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("basic_inputs", sa.JSON(), nullable=True),
+    )
+    op.add_column(
+        "variables", sa.Column("is_input_variable", sa.Boolean(), nullable=True)
+    )
+    op.add_column("variables", sa.Column("hidden_input", sa.Boolean(), nullable=True))
+    op.add_column(
+        "variables",
+        sa.Column(
+            "module_name", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=True
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "adds_parameter_path",
+            sqlmodel.sql.sqltypes.AutoString(length=512),
+            nullable=True,
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "subtracts_parameter_path",
+            sqlmodel.sql.sqltypes.AutoString(length=512),
+            nullable=True,
+        ),
+    )
+    op.add_column("variables", sa.Column("enum_values", sa.JSON(), nullable=True))
+    op.add_column(
+        "variables",
+        sa.Column("unit", sqlmodel.sql.sqltypes.AutoString(length=128), nullable=True),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "definition_period",
+            sqlmodel.sql.sqltypes.AutoString(length=32),
+            nullable=True,
+        ),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("variables", "definition_period")
+    op.drop_column("variables", "unit")
+    op.drop_column("variables", "enum_values")
+    op.drop_column("variables", "subtracts_parameter_path")
+    op.drop_column("variables", "adds_parameter_path")
+    op.drop_column("variables", "module_name")
+    op.drop_column("variables", "hidden_input")
+    op.drop_column("variables", "is_input_variable")
+    op.drop_column("tax_benefit_model_versions", "basic_inputs")
+    op.drop_column("tax_benefit_model_versions", "entities")
+    op.drop_column("tax_benefit_model_versions", "catalog_format")
+    op.drop_column("parameters", "household")
+    op.drop_column("parameters", "economy")
+    # ### end Alembic commands ###