v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch by anth-volk · Pull Request #1441 · PolicyEngine/policyengine-app-v2

anth-volk · 2026-10-27T13:58:32Z

Stacked on #1444 (v3 parameters on demand, the generated Public API client and V3_API_ROUTES).

What changes

On the v2 routes (V3_API_ROUTES = 'v2'), v3's household builder, its variable picker, the household breakdown and the views that name variables now work from GET /v2/model-catalog/variables instead of v1's whole metadata.

Background load: every variable of the catalog version loads once in the background after a v3 page renders: on the workspace pages through WorkspaceLayout, and on the report and bill report pages. The variables go into the metadata store in the shape v1's metadata gave them, so the existing builder, picker and breakdown code reads them unchanged.
Caching and model catalog loads: the load is keyed by the model catalog load, as the parameters are. A new load asks for the variables again, and an earlier load's cache entry is removed, so only the current load's variables stay cached. Nothing is requested when the store already holds them, or when the store's model catalog is for another country than the page's. Nothing is kept in localStorage.
Parts named by a parameter: the household breakdown (VariableArithmetic) takes a parameter-named adds or subtracts list for the report's year from the dated values the route sends (part_parameter_values), instead of from the whole parameter list.
Waiting and failure: views that read the variables wait for them, with no time limit, and say when loading failed, with Try again. Those views are:
- the report's household section;
- the budgetary impact by programme chart and its CSV, whose download stays unavailable, with the reason as its tooltip and accessible description;
- calibration matches and the geography fit;
- changes that list variables, in the provision list and the report's changes.
Old catalog versions: a catalog version published before catalog format 2 fails with a clear message, rather than giving variables without input flags, enum labels or parts.

Field conversion and parity with v1's metadata

Each catalog variable becomes the record v1's GET /{country}/metadata gave, checked against records serialized from .py's public Variable records in the catalog's shape:

Enums: .py publishes every Enum variable with data type str and its values. A variable with enum values is v1's Enum, with possibleValues {value, label} and labels kept as declared.
Defaults: as v1 gave them: a value for numbers, yes-or-no values and enums (the default member's name), none for text or date variables.
Unit and definition period: read as v1 gave them, from catalog format 2. A variable without a unit formats as a plain number and is "Number" in the picker; a period that is neither a year nor a month gets no "per" suffix, as on v1.
Parts named by a parameter: a parameter-named adds or subtracts keeps the parameter's path, as on v1.
Other fields: label (the name when there is none), entity, documentation, input and hidden flags, and module.
Dated values: these and the parameters' values share one converter, which checks each value where it arrives.

Merged unused

Every new path is chosen when its module loads, from V3_API_ROUTES, which stays 'v1' until the switch. With it off:

the variables hooks report 'ready' at once;
the waiting wrapper shows its content;
nothing is requested;
every view renders as before.

Tests cover both settings.

Note for whoever makes the switch

The other segment's PR #1392 adds a v2-only standalone household report and a builder panel that read state.metadata.variables directly, without V3VariablesGate or useV3Variables. The background load fills the store, so they do get the variables. But until the variables arrive they will show empty fields, and nothing at all if the load fails. When the switch makes that page reachable, it will need the same wait.

Environment variables

None read or added. No workflow changed.

What this waits for before it can merge

v3: load metadata on demand from the Public API's v2 routes, unused until the switch #1444, the PR below it.
The Public API PRs it reads from, deployed: Serve v2 metadata in parts loaded on demand, cacheable per catalog version policyengine-api#4042, #4046 and #4049, and the stack below them. That stack waits for the .py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701.
Review.

Tests and evidence

GitHub Actions runs on v2plan/* branches are cancelled at the user's request, so the evidence is local, on Node 26.10.0. Every commit was written test-first: its new tests failed for the reason stated in the commit, then passed. The exceptions are the commits marked as refactors or test-only.

On the final commit (9b986d87):

app suite: 6,139 passed, 10 skipped (501 files);
website tests: 483 passed;
make typecheck and make lint clean;
bun run check (type check, ESLint, Prettier) and bun run stylelint clean;
bun run public-api:check: the generated types match openapi.json.

Review

Three review rounds:

Round 1: one major bug. Enum inputs showed a free-text box, because the catalog's str data type was copied as the value type. Also quality findings: the store filled only when a reader mounted; duplicated converters; typing; test placement; documentation.
Round 2: one minor bug. The programme CSV listed raw names while the variables loaded or after they failed. Also quality findings: report pages did not start the load; earlier loads stayed cached; duplicated query code; test organisation; a real parameter-named variable was untested.
Round 3: two minor bugs. The variables could be downloaded again after their cache entry expired, and the CSV's unavailable reason could not be reached. Also quality findings: status names; catalog versions before format 2; test sections and fixtures.

All are fixed in this branch.

🤖 Generated with Claude Code

On the v2 routes the metadata store starts with no variables. After a v3 page renders, every variable of the catalog version now loads once from `GET /v2/model-catalog/variables`, in the shape v1's metadata gave each variable, so the household builder, its variable picker and the breakdown read them unchanged. The dated values of each parameter a variable's adds or subtracts names come with them and join the store as well. The response is cached per catalog version for the session, and a response for another country or version is dropped. `useV3Variables` says whether they are in, for views that wait for them. On the v1 routes (still the default), nothing changes and nothing is requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… from v2's part values Where a variable's adds or subtracts names a parameter, the household breakdown (`VariableArithmetic`) used the list that parameter holds in the report's year, read from the whole parameter list. On the v2 routes there is no whole parameter list, so it now reads the dated values the variables response carries for that parameter, and the whole parameter list on v1, as before. A parameter with no values lists nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On the v2 routes the variables load in the background, so the views that read them now show a placeholder until they are in, with no time limit, and a failure says so and asks again: - the report's household section (the builder, its variable picker and the breakdown), behind `V3VariablesGate`; - the budgetary impact by programme chart, whose programme names are variable labels; - calibration matches and the geography fit, whose target names are; - a reform's changes that list variables, in the provision list and in the report's changes. The household builder's default fields and entities come from `/v2/model-catalog` through the metadata store, and a test pins that they, and the variable picker's entered, hidden and enum fields, read correctly from the v2 variables. On the v1 routes (still the default) every view shows as before and nothing is requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The Public API's catalog format 2 now stores each variable's unit and definition period, and `/v2/model-catalog/variables` serves both (null for catalog versions published earlier). `fetchCatalogVariables` now maps them into v1's variable shape (`unit`, `definitionPeriod`), so on the v2 routes the household breakdown formats currencies and rates, and the variable picker says "per year" or "per month", as on v1. A null unit reads as v1's variables without a unit do: a plain number, and "Number" in the picker. A null period gives no "per" suffix, as v1 shows for a period that is neither a year nor a month. The generated client on this branch predates the two fields, so a small, documented local extension of its variable type carries them until the client is regenerated from the final Public API branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… failed load in every view that waits On the rebase onto 1.19's final branch: - The variables load is now keyed by the model catalog load (`v2LoadId`), as 1.19 keys parameters, so a new load (a retry, or a newer catalog) empties the store and asks for the variables again instead of refilling it from an earlier load's response; a response asked for under an earlier load is dropped. - The local extension of the generated variable type is gone: the client regenerated from the final Public API branch carries each variable's unit and definition period, and its other catalog format 2 fields are required and nullable. - Every view that waits for the variables on the v2 routes now also says when loading them failed, with Try again: the provision list and the report's changes (where a change lists variables), and calibration matches and the geography fit. The household section and the programme chart already did, through `V3VariablesGate`. `useV3Variables` reports this as `failed`, only ever on the v2 routes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The catalog stores each variable's Python type name, and .py publishes every Enum variable with data type "str" and its values. The conversion copied that into v1's `valueType`, so on the v2 routes every enum input (the default fields `state_name`, `region`, `brma`, `local_authority` among them) showed a free-text box instead of a list, and the picker called them "Number". A variable with enum values is now v1's "Enum". v1 also gave a default only for numbers, yes-or-no values and enums (the default value's name), none for a text or date variable; the conversion now does the same. Tests use records serialized from .py's public `Variable` records in the catalog's shape, and the small fixture's enum uses data type "str" as the catalog does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g of another country The background load filled only the query cache, so the metadata store got the variables only once a view using `useV3Variables` mounted; a view that reads `state.metadata.variables` directly saw none. The background load now puts them in the store when they arrive, under the model catalog load they were asked for (a failure stays in the query, where waiting views report it). Both the background load and `useV3Variables` now also skip a store whose model catalog is for another country than the page's, which a client-side country change leaves until the new catalog loads, instead of starting a download that would be dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…takes Parameters (`catalogParameters`) and the parts of variables (`catalogVariables`) each had their own copy of the converter from v2's dated values to v1's, with its earliest-date constants; both now use `v1DatedValues` in `api/publicApi/datedValues`, which checks each value where it arrives with 1.19's `parameterValueFromApi`, so a malformed part value also fails there, naming its parameter. The variables request takes its country through `publicApiCountryId`, like its siblings, instead of a cast, and `catalogVariablesLoaded` takes `V1ShapedVariable` records. The existing tests cover both converters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…und variables load - Tests import from `@test-utils`. The breakdown test's metadata fixtures move under the fixtures folder. The builder and list tests are split to mirror the modules they cover (`metadataUtils`, `VariableResolver`, `VariablePickerDialog`, `householdValues`, `ProvisionList`, `ReportChanges`), with the store seeding they share in a fixture. - The `v3ApiRoutes` and `modelCatalog` comments and the country ID architecture document describe the background variables load, and the comment on `MetadataState.v2LoadId` says a new load empties the variables too. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The UK "Budgetary impact by programme" card built its CSV outside the waiting chart, so on the v2 routes a CSV downloaded while the variables loaded, or after they failed, named programmes by raw variable name. The card's CSV button now shows disabled until the variables are in, with "Loading programme names" or "Could not load the programme names" as its tooltip (`ReportChartCard`'s new `csvUnavailableReason`); on v1, and once the variables are in, it downloads as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The background variables load ran only from `WorkspaceLayout`, which the report and bill report pages do not use, so on those pages the download started only when a view that reads the variables mounted. Both pages now start it when they render, like the workspace pages; on v1 it does nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e the views one state to read - `useV3Variables` and the background load now build their query from one helper (`catalogVariablesQuery`), whose fetch also puts the variables in the store under the load they were asked for, so the key, the options, the country check and the store update live in one place. - The query's `gcTime` is now finite, and the background load removes an earlier model catalog load's query when a new load starts, so only the current load's variables stay cached for the session. - `V3VariablesStatus` is one `state` ('ready', 'waiting' or 'failed'), which the lists, the calibration sections, the programme CSV and `V3VariablesGate` all read; on v1 it is always 'ready', so the waiting wrapper is one component and shows its content at once there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The real-shape fixture now also holds `above_the_line_deductions`, as the catalog publishes it, and the dated values of `gov.irs.ald.deductions` that name its parts, in the route's date format. A breakdown test on it checks that a 2017 report adds `domestic_production_ald` and a 2018 report does not (it passes, and fails when the breakdown ignores the part values). - The `catalogVariablesLoaded` reducer cases sit with `catalogParametersLoaded`'s in the metadata reducer's v2 tests. - The variables route path, the list provision and a fail-once-then-answer route are shared fixtures, and the country-change test waits for the query client to be idle instead of a fixed 50 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he store lacks them The variables query kept its entry for five minutes, and the background load fetches without an observer, so on a page with no reader the entry went after five minutes and the next mount downloaded the whole response again, although the store already held that load's variables. The current load's entry is now kept for the session (an earlier load's entry is still removed when a new load starts), and neither the background load nor `useV3Variables` asks for the variables when the store already holds those of the current load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The CSV button showed why it was unavailable only in a tooltip on a `disabled` button, which cannot take focus or open its tooltip, so the reason was never shown. While unavailable the button now stays focusable with `aria-disabled`, does nothing when clicked, opens its tooltip on hover or focus, and carries the reason ("Loading programme names" or "Could not load the programme names") as its accessible description. An available CSV button renders as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…bles status as its siblings do - `fetchCatalogVariables` now fails, naming the catalog version, when the response's `catalog_format` is not 2: such a version publishes no input flags, enum labels, modules or parameter-named parts, so v3's builder and breakdown would otherwise get variables with none of them and show no inputs, no dropdowns and wrong parts lists. Views report it as a failed load, and `V3VariablesGate` shows the message. - `V3VariablesStatus` uses the same field and values as `V3ParametersStatus` and `V3SearchCatalog`: `status`, one of 'ready', 'loading' and 'error'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ixtures and request checks - `v1DatedValues` has its own tests: real dated values keyed by the day each starts, Python's earliest date written as v1 wrote it, no values, and a value that is not a parameter value failing with the parameter's name. - The breakdown and parts tests have Given, When and Then sections; their parts values, and the catalog test's unlabelled variable, are fixtures. - The report and bill report page tests check that the variables request is sent, not only that the background load hook ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-27T13:58:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-calculator-next	Ready	Preview	Oct 27, 2026 1:58pm UTC
policyengine-website	Ready	Preview	Oct 27, 2026 1:58pm UTC

anth-volk and others added 17 commits October 27, 2026 16:52

vercel Bot deployed to Preview – policyengine-calculator-next October 27, 2026 13:58

View deployment

vercel Bot deployed to Preview – policyengine-website October 27, 2026 13:58

View deployment

PavelMakarchuk mentioned this pull request Oct 27, 2026

v3 validation: one related score per measure and year, calibration headlines first #1442

Merged

anth-volk added this pull request to stack #1445 October 27, 2026 14:06

Navigation Menu

Repository navigation

v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch - #1441

v3 variables on the v2 routes: the household builder and breakdown from /v2/model-catalog/variables, unused until the switch#1441
anth-volk wants to merge 17 commits into
v2plan/1f-119-v3-parameters-on-demandfrom
v2plan/1f-120-v3-variables-on-v2

anth-volk commented Oct 27, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

anth-volk commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Field conversion and parity with v1's metadata

Merged unused

Note for whoever makes the switch

Environment variables

What this waits for before it can merge

Tests and evidence

Review

Uh oh!

vercel Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

anth-volk commented Oct 27, 2026 •

edited

Loading

vercel Bot commented Oct 27, 2026 •

edited

Loading