Derive property_wealth from its uprated components instead of saving the WAS total

[MaxGhenis]

Draft — stacked on policyengine-uk-data#501, which must merge first. This branch tracks was-wealth-split and must be restacked whenever that branch moves, without rewriting #501's history. Release only in a batched uk-data release after the companion model changes specified in docs/property-wealth-model-changes.md. No model PR is opened by this work.

A saved property_wealth column overrides the model aggregate and lacks temporal uprating. This PR removes that override while preserving the complete WAS gross-property definition. It adopts #501's treatment of second homes and buy-to-let as one other-residential component; exporting buy-to-let again would double-count it.

Confirmed source coverage

The WAS round-8 derived-variable specification, worksheet DV_Spec_Property_R8, rows 49–59, defines:

DVPropertyR8 = DVhValueR8 + DVHseValR8_sum + DVBltValR8_sum
             + DVBlDValR8_sum + DVLUKValR8_sum + DVLOSValR8_sum
             + DVOprValR8_sum

DVPropertyR8 is gross, not mortgage-adjusted net property wealth. Row 45 separately defines HpropWR8 = DVPropertyR8 - HmortGR8; see also the user guide, p. 39.

Seven source terms map to six saved components:

Source	Stored component	Treatment
DVhValueR8	main_residence_value	Main residence
DVHseValR8_sum	other_residential_property_value	Second UK houses, combined with buy-to-let by #501
DVBltValR8_sum	other_residential_property_value	UK buy-to-let, combined with second homes by #501; counted once
DVBlDValR8_sum	non_residential_property_value	Other UK buildings
DVLUKValR8_sum	owned_land	UK land-only holdings; included in worldwide property/total wealth and counted once in household land
DVLOSValR8_sum	overseas_property_value	Overseas land and property; included in worldwide wealth, excluded from UK land/LVT
DVOprValR8_sum	other_property_value	Other real estate; separate component, classification question below

The R8 dictionaries, UProp2R8, category 5, include overseas dwellings and time-shares. The questionnaire, pp. 197–199 and 205–206, provides the category and valuation questions. The former non_uk_land reference column did not preserve these assets because it was absent from the imputed outputs. #501 already retains buy-to-let; #543 completes overseas and other-real-estate mapping and the direct-land aggregate treatment.

Data changes and retained #501 design

• Retain Split pensions, shares, trusts, ISAs and secured debt out of WAS wealth #501's second_homes_value + buy_to_let_value construction of other_residential_property_value. buy_to_let_value is an internal survey intermediate, with no separate buy-to-let model export.
• Retain Split pensions, shares, trusts, ISAs and secured debt out of WAS wealth #501's combined second-home/buy-to-let other_residential_property_secured_debt, the other secured-debt inputs, asset-before-debt chain order, debt zeroing, pension/share split, derived corporate_wealth and independent per-target quantile seeds.
• Append only overseas_property_value and other_property_value to the existing imputation targets, preserving Split pensions, shares, trusts, ISAs and secured debt out of WAS wealth #501's existing target order and seeds. The expanded target metadata invalidates incompatible cached models and requires retraining. Retrained draws have not been compared with production data in this workspace.
• Keep DVPropertyR8 in its sequential conditioning slot, but never save that draw as a dataset column. Remove an inherited saved property_wealth total too. Independent component predictions have no enforced identity to the conditioning draw; the model derives the final aggregate from the components.
• Normalise documented -9/-8 property missing codes and missing values to zero before combining second homes and buy-to-let. Reject other negative gross amounts with aggregate-only diagnostics. Zero-filled source reconciliation is a diagnostic, not a claim that incomplete source cases reconcile.
• Give all six saved property inputs GDP-per-capita temporal uprating. The two new materialisation-table rows match the existing property inputs. Regional HPI adjustment remains main residence only.
• Require a compatible installed model before imputation. Validate component availability, aggregate membership, the specified GDP indices, and land-only, overseas-only and mixed domestic-property land behavior.

Required model companion and dependencies

The property companion must stack on #501's two open model companions:
policyengine-uk#1974 (wealth-share-components) and
#1969 (uc-capital-sale-expenses).
Preserve their share routing and secured-debt valuation. The only new property model inputs are overseas_property_value and other_property_value, registered as household stocks with both variable-level and dataset-YAML GDP-per-capita uprating.

The exact files, formulas and tests are specified in the companion document:

residential_property_value = (
    main_residence_value + other_residential_property_value
)
property_wealth = (
    residential_property_value + non_residential_property_value
    + owned_land + overseas_property_value + other_property_value
)
household_land_value = (
    property_wealth - owned_land - overseas_property_value
) * regional_intensity + owned_land

residential_property_value keeps its existing two-term formula: #501 puts buy-to-let into the other-residential term already. total_wealth receives direct land, overseas property and other real estate through property_wealth, with no separate additions. Gross property wealth remains separate from #1969's countable-capital valuation, which deducts sale expenses and identified secured debt.

No companion model implementation or model dependency/version bump is claimed here. The locked model deliberately fails the build guard until the property companion is released and the data dependency is updated. The synthetic model tests use a local Reform representing the required contract; passing them does not establish that the installed production model has changed.

Consumers and calibration

The companion document separates the historical 2.93.0 file:line audit from current behavior checked on the model companion branches. Property wealth feeds household land/LVT, total wealth, wealth deciles and wealth-tax reforms. Benefit-capital rules and domestic purchase/SDLT proxies read components directly.

The property companion must add overseas_property_value and other_property_value exactly once to both the countable-capital and sale-expense source lists for UC, Housing Benefit, Income Support, income-based JSA, income-related ESA and Pension Credit. Preserve existing source names, effective dates, reported-capital behavior, household-to-benefit-unit allocation and broad-asset/disregard limitations. Retain #1969's source-specific deduction of sale expenses before secured debt and its zero floor. Overseas/other property have no newly exported secured-debt inputs, so unidentified debt on those holdings cannot be deducted by this mapping.

The current UC unearned-income formula has no property-income gate to extend. Actual interest, dividends and rent remain excluded at every capital level; capital enters through tariff income. The existing additional-residential purchase/SDLT stock proxy already reads other_residential_property_value, so #501 includes domestic buy-to-let once automatically. Keep overseas and unclassified other real estate outside that dwelling-purchase proxy.

Removing only the aggregate column has no direct effect on component-based benefit, purchase or high-value council-tax formulas at fixed weights. Completing category mapping and routing can change capital and purchase calculations. Recalibration can change weighted aggregates even for formulas without a direct property-wealth dependency.

No calibration control values, target weights or objective terms change here. National and regional land columns change when the model calculates land from the completed components. Evaluated losses and fitted constituency/local-authority weights can consequently change; fitting is not guaranteed to eliminate every discrepancy.

Base-year source reconciliation

Licensed WAS reconciliation: NOT RUN. This workspace has no licensed R8 household TAB file. Run the aggregate-only measurement before release:

uv run --no-sync python tools/reconcile_was_property.py \
  --household-tab /licensed/was_round_8_hhold_eul_may_2025_230525.tab

The tool independently transcribes all seven source fields before QRF imputation or HPI adjustment. It reports the historical pre-#501 three-term baseline, #501's four-term baseline including buy-to-let, the remaining contributions, the full seven-term residual, and the six mapped model-input sums. Buy-to-let appears once in the grouped other-residential input.

Reports include weighted/unweighted sums, absolute residuals, discrepancy counts, category contributions, households owning only omitted/remaining categories, missing/sentinel counts and invalid-weight treatment. Complete cases and a separately labelled zero-filled diagnostic are both reported. No records or household identifiers are emitted. Earlier comparisons of independently imputed release columns are not raw WAS source reconciliation and are not used to quantify this fix.

Invariants and validation

• All seven synthetic source categories survive mapping/storage, including singleton owners; second homes and buy-to-let become one other-residential input.
• Complete synthetic source cases reconcile to the gross source total before independent prediction. Buy-to-let is counted exactly once.
• No conditioning-total draw or inherited saved aggregate survives as a property_wealth override, even when the draw differs arbitrarily from the component sum.
• Gross property wealth equals the six saved-input sum; with other wealth zero, total wealth equals that property sum. Main residence is a lower bound for non-negative components.
• UK land-only holdings contribute once to property/total wealth and once to household land. Explicit overseas-only holdings contribute to worldwide wealth and zero to UK land/LVT.
• New inputs survive materialisation and tested annual projections; consistent growth preserves the aggregate identity. Companion tests must also verify monthly stock handling.
• Split pensions, shares, trusts, ISAs and secured debt out of WAS wealth #501's pension/share/debt identities, independent seeds and chain order remain intact. Hypothesis property tests are retained.

Local results: 81 passed, 8 data-dependent skips. Each file ran individually in the foreground with pytest.main(["-n", "0", "-q", FILE]), using Python 3.13 and a workspace dependency overlay constrained by the lockfile. The runner reused loaded dependencies and reloaded the changed source and test modules between commands.

Touched test file	Result
test_property_wealth_components.py	51 passed, 8 skipped (enhanced FRS absent)
test_was_property_reconciliation.py	18 passed
test_regional_property_uprating.py	6 passed, including a fresh-source rerun
test_student_loan_balance.py	6 passed

Ruff formatting and lint checks passed for all seven touched Python files; git diff --check passed. Preloaded third-party plugins emitted assertion-rewrite warnings; test modules were imported afresh. The synthetic model checks apply the local Reform contract, not a released property companion.

17 mutation variants produced the expected test failures, with byte-for-byte restoration verified by SHA-256. They covered omitted/doubled/re-exported buy-to-let; bypassed raw residential-source cleaning; omitted overseas/other mapping; restored conditioning-total persistence; retained inherited totals; missing direct-land/overseas subtraction; separate buy-to-let grouping or an omitted raw term in reconciliation; missing rows in each uprating table; accidental HPI scaling of direct land; and disabled aggregate/uprating compatibility guards. The HPI check initially reused a cached function; loading its current source fixed that validation issue and the rerun failed as expected. No licensed records were used.

Only touched test files are run, one file per foreground command, with at most two pytest workers and OMP_NUM_THREADS no greater than four. Whole-suite, folder and multi-file runs are skipped under the shared-host rules. Data-dependent built-dataset checks require a rebuilt dataset and the released companion model; synthetic Reform tests establish only the specified contract.

Reviewer questions

1. Worldwide wealth versus domestic property: chosen here is the WAS worldwide gross property_wealth, with explicit overseas exclusion in land. An alternative is domestic property_wealth plus a separate overseas addition in total_wealth; that diverges from WAS's property definition and requires updating every relevant wealth consumer.
2. UK land routing: chosen here is inclusion in gross property wealth and subtraction from the intensity base before adding owned_land once. Alternatively, exclude it from property wealth and add it explicitly to total wealth, preserving the current land formula. That also diverges from the seven-term source definition.
3. Other real estate geography: WAS's other-real-estate category does not separately identify geography or residential status. The proposed formula retains the existing regional-intensity approximation for this component; treating all of it as domestic is an assumption. Reviewers may instead exclude it from UK land pending classification, or recover a geography split from more detailed licensed source information. The explicit overseas category is excluded under either choice.
4. UC land capital: five of the six capital-source lists already include owned_land; UC's list currently omits it. The proposed companion preserves that existing UC treatment. Correcting it requires a deliberate separate model decision.

Release boundary

Merge #501 first, then #543. Release notes must state full seven-category source coverage in six saved components; the inherited #501 buy-to-let treatment; affected wealth, land, decile and nonzero reform outputs; explicit companion capital/purchase routing; unchanged controls/objective terms with potentially changed losses/weights; main-residence-only regional HPI adjustment; and no migration of existing published releases.

Remaining gates: licensed source reconciliation; release of the property companion on the open #1974/#1969 model stack; data dependency/lock update; production rebuild/recalibration; fitted-weight comparisons; production distributional/LVT-impact measurements; and resolution of the reviewer classification questions. Existing published datasets retain their stored-total overrides. Keep this PR a draft until the batched release is authorised.

axiom: n/a: data asset mapping and microsimulation accounting; companion model routing is specified separately.

🤖 Generated with Claude Code

[juaristi22]

Review of af64d3757c2648e820219faee96186d3d7cd9a63

Review status: PARTIAL
Review severity: COMMENT
Confirmed critical count: 0

No confirmed code or policy defects were found in the changed behavior and affected dependencies. The persistence filter follows the locked model's property-wealth definition, leaves sequential predictions and other saved outputs unchanged, and remains effective through calibration and export. One material source-coverage question remains unresolved.

Evidence gap: property asset coverage

At wealth.py:289–290, omitting the saved total activates a sum of DVhvalueR8, DVHseValR8_sum and DVBlDValR8_sum (source mapping at lines 102–104). The unresolved question is whether those source fields cover every property category intended for downstream total wealth, and what property categories the discarded DVPropertyR8 previously carried. ONS methodology, table 1 includes other land/property in household gross property wealth; the round-8 guide, physical page 39 separately explains net property wealth. Neither establishes the exact field-level mapping. In locked UK 2.93.0, owned_land denotes land-only plots and enters household_land_value, but does not enter total_wealth. The missing source definitions therefore leave unresolved whether activating the sum loses a category from total wealth, or changes overlap in the land calculation; this matters for wealth deciles and nonzero wealth/LVT reforms. Obtain the round-8 derived-variable specification for the four property fields and DVLUKValR8_sum to resolve this. This is not a confirmed omission/double-counting bug, and not merely a failure to corroborate background PR narrative about whether DVPropertyR8 is net or gross.

Validation

Verified CI run 37410613766 at the reviewed head: the TESTING=1 dataset build succeeded, followed by 642 passed, 3 skipped, 1 xfailed. All 11 property-component tests (including all 8 built-dataset checks), 6 regional tests, 6 student-loan tests and 8 land-target tests passed. Lint, changelog, documentation and bundle-manifest checks also passed.

Local tests were not run because the available environments lacked the required dependencies; none were installed. The locked policyengine-uk 2.93.0 source was independently verified against its published wheel. The arithmetic and persistence tests do not resolve the source-field coverage question above.

Scope and limitations

• Filtering occurs after model.predict, so this edit preserves the sequential predictor set and later draws. The newly calculated total follows the existing model definition and its uprated components.
• National and regional land targets are unchanged; their model contributions change as disclosed. No full production recalibration was run, so effects on production weights, distributional results and published LVT estimates remain unmeasured. This is a release-impact limitation, not a separate confirmed defect.
• This PR changes future builds. Existing published datasets retain the saved column; no migration or defensive fix for those releases is claimed.
• The unknown gross/net definition of DVPropertyR8 alone does not establish a defect. The unresolved question is the coverage of the newly activated aggregate.

Primary evidence: ONS wealth methodology, WAS round-8 guide, and policyengine-uk 2.93.0. The exact WAS derived-variable specification was not obtained.

[MaxGhenis]

@juaristi22 your coverage question was right, and it was a blocker. WAS round 8 defines gross property as seven terms (DV_Spec_Property_R8, rows 49–59): DVhValueR8 + DVHseValR8_sum + DVBltValR8_sum + DVBlDValR8_sum + DVLUKValR8_sum + DVLOSValR8_sum + DVOprValR8_sum. The first version kept only three of them, so dropping the saved total would have lost UK buy-to-let, UK land, overseas land and property, and other real estate.

At 0c7cf0a (rebased on main), all seven survive as separately uprated components. owned_land stays the UK land-only term, overseas and other real estate become new components, and the WAS total stays as a conditioning draw but is never saved. The description has the mapping table, invariants, test results and four classification questions for you and Vahid. The two main ones: whether property_wealth should be the worldwide WAS gross total with overseas excluded only from UK land, and how to treat the geography of 'other real estate'.

The data side now needs a companion policyengine-uk change. That's docs/property-wealth-model-changes.md in this PR: new component variables, the property_wealth sum, and household_land_value counting UK land once and overseas property never. A draft PR for it is being built now and will be linked here. The licensed-data reconciliation (tools/reconcile_was_property.py) hasn't been run; it needs the R8 household file.

[MaxGhenis]

Overlap: #501 (WAS wealth split, draft) also edits wealth.py and the uprating tables, and folds buy-to-let into other_residential_property_value; this PR made it a separate component. I've asked #501's owner to agree an order. The proposal is that #501 lands first and this PR rebases onto it, adopting #501's buy-to-let treatment and keeping only overseas property, other real estate and the persistence fix. The companion policyengine-uk PR is on hold until that's settled, so it follows the final design.