Remove the UK QRF tail-concentration gate (#1276)

[juaristi22]

Removes the UK QRF tail-concentration gate (uk_qrf_tail_concentration) from the national release-cut battery. #1276 asks for a replacement that measures what the gate was meant to.

Closes #1277, which asked to re-arm this gate; its independent-draw point now lives in #1276's replacement request.

Why

On the UK surface the gate's top-100 share could only bite on a column with fewer than 100 independent draws whose CGT-clone copies took it past the 104-row floor. So its verdict measured how little mass those copies carry, not whether an imputation broadcast a donor tail. At the armed 0.99947 threshold, the #462 incident it was built for (89% of a column in 100 records) would have passed. The module docstring already noted that no single threshold was both incumbent-compatible and incident-catching.

On 14 October the gate blocked certification of the national candidate twice, neither time for a data-quality reason: once on clone-copy arithmetic, and once on an exclusion that #1256's new draws made stale. The evidence and the request for a replacement are on #1276.

Changes

• Spec and runtime:

  • remove the entry from uk/gates.json and from UK_NATIONAL_GATE_SCOPE;
  • remove the UK binding, UKQRFTailConcentrationPolicy, uk_qrf_tail_concentration_gate, the reviewed-exclusion loader and constants;
  • remove qrf_tail_reviewed_exclusions.json and its country_package.json listing.

• Data contract:

  • drop the gate's threshold pins, detail schema, legacy stage, entry mappings and validation block, and remove it from the release-cut part's entry set;
  • re-pin the release-cut part digests and the full-battery policy, manifest and fingerprint.

  The spine and calibration-seam part digests are unchanged, so existing spine and seam reports stay valid.

• Kept: uk_qrf_tail_concentration_columns and tools/measure_uk_weighted_integrity_baselines.py stay as the measurement surface for the replacement. The shared tail_concentration_gate and the US QRF tail register are untouched.

• Tests and docs: the gate's tests go; tests that used it as an example of a generic exclusion behaviour now use the input-mass or degenerate-surface gates. One doc table row and the doc's count of re-minted names are updated, and there is a removed changelog fragment.

• Left as history: national_staging_build_record.json, spine_swap_signed_differences.json, experiments/ and older changelog fragments still mention the gate.

Verification

At df9d13a09, rebased onto main 71b9b8b52 (#1271):

• Re-pins: UK: policyengine-uk 2.130.1 and the fourth batch of #1095 ports, with the release's weight year, identifiers and birth months fixed #1271 moved the spec, so the release-cut part and full-battery digests are re-pinned on top of it. The spine and calibration-seam part digests are still unchanged.
• Touched surface (every test file this PR changes, plus every test importing a touched module, tool or resource or loading the UK spec): 86 files and 2,475 tests. All pass except test_engine_is_reported_unavailable_without_the_uk_extra, which fails only because the local environment has the UK extra installed; it fails the same way on main there.
• Integration test: not run. It builds the synthetic spine, whose gate scope and digests this PR doesn't touch.
• python3 tools/ci_test_plan.py verify: ok. ruff check passes on every changed file.
• Certification (before the rebase): I certified the 14 October national candidate (built at main 121fd4dd6, attempt uk-frs-calibration-attempt-20261014T214443Z-d12ff9b3) with this branch's certifier. It was shippable: true: spine 37/37, calibration seam 7/7, release cut 18/18, rule 1 passed. Local assembly validated it against the branch's contract, and nothing was published. Since the rebase moved the release-cut pins and UK: policyengine-uk 2.130.1 and the fourth batch of #1095 ports, with the release's weight year, identifiers and birth months fixed #1271 locked policyengine-uk 2.130.1, a cut for publication should be rebuilt and certified on main after this merges.

Notes

• Until UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276's replacement lands, no release-blocking check covers the tails of imputed columns. The national release runbook now has an interim manual check: compare each candidate's SPI stage-1 columns with the promoted cut, using the kept baselines tool.
• validate_release_dir runs only when a release is assembled, preflighted or published, never when one is loaded. So the new pins don't affect releases that are already published.
• Main's contract still pins the old release-cut digests, so a cut certified under this branch can be published only from this branch or after it merges.

🤖 Generated with Claude Code

[vahid-ahmadi]

Fine to merge. The removal is justified, but it leaves the SPI imputed columns with no release-blocking check, so #1276 needs to be owned as real work, not a nice-to-have.

@juaristi22, review of 8915469 against main 121fd4d.

Is the removal justified?

Yes, on the evidence in the PR and in #1276.

• The gate's own module docstring said in August that no single top-share threshold was both incumbent-compatible and incident-catching. At the armed threshold it would have passed the Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462 signature it was built for. A release-blocking gate that cannot fail its own incident, and that fails on clone-copy arithmetic instead, costs more than it protects.
• It was a quality gate, not disclosure control. It measured how a column's weighted mass is spread, not whether drawn values reproduce donor records. So removing it does not weaken disclosure control for released files. Note that this is "unchanged", not "covered elsewhere": the disclosure-safe donor bounds in uk_support cover the WAS, LCFS, ETB, NTS and UC-deduction outputs, not the SPI stage-1 draws.
• If anything, removal reduces exposure. The removed gate's details emitted per-column carrier_counts and thin_columns, which for the thin SPI columns are small cells. terminal_gates.json is validated as part of the release directory, so if it ships with a public release those counts were going out with it.
• Keeping uk_qrf_tail_concentration_columns and the baselines tool as the measurement surface for the replacement is the right call.

What it leaves uncovered (the PR body says this; I checked the extent):

• uk_input_mass_parity is the only remaining terminal check on imputed-column mass. Its reference's own scope note (uk/gates.json, efrs-post-calibration) says it carries no admin-restored mass in the SPI-channel-exclusive columns, and the 0.0 floor skips zero-mass reference columns. So the hmrc_spi_* stage-1 columns now have no release-blocking check on mass or shape at all.
• For the columns it does cover, the tolerance is a relative 4.52 (uk/gates.json, uk_input_mass_parity). A Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462-style inflation of a few times the reference total would pass.
• uk_weight_ratio, uk_weight_ess and the stage-health gates look at weights and stage receipts, not at how an imputed column's mass is spread.

So the #1256 incapacity-benefit drop described in #1276 would now go unseen. That is acceptable for a short gap, but the acceptance cases in #1276 (the #462 replay must fail, the #1256 drop must fail, clone arithmetic must pass) should be what the replacement is held to.

Findings

1. should: Re-arm the UK QRF tail-concentration gate on independent draws #1277 ("Re-arm the UK QRF tail-concentration gate on independent draws") is still open and asks for work on the gate this PR deletes. changelog.d/1124-uk-qrf-tail-surface.changed.md:3 also still points readers to it for the now-deleted exclusions. Please close Re-arm the UK QRF tail-concentration gate on independent draws #1277 as superseded by UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276 (or fold its independent-draw point into UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276) when this merges, so nobody re-arms a gate that no longer exists.
2. should: until the replacement lands, consider a manual check on each national candidate, using the kept uk_qrf_tail_concentration_columns surface (or the baselines tool) to compare weighted prevalence and weighted totals of the SPI stage-1 columns with the previous candidate. It costs little and would catch a Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256-style drop in the meantime. Your call on whether that is worth writing into the certification checklist.
3. should: when the replacement is designed, keep raw carrier or independent-draw counts out of committed gate notes and out of any report that ships with a release. The removed uk/gates.json notes carried small per-column carrier counts and a source row count, and the removed gate's details carried the same kind of counts (see above). Shares, ratios and pass or fail are enough for the record.

Clean removal

Checked at the head commit:

• uk/gates.json entry, UK_NATIONAL_GATE_SCOPE, the UK binding, UKQRFTailConcentrationPolicy, the gate function, the exclusion loader and constants, qrf_tail_reviewed_exclusions.json and its country_package.json entry are all gone.
• The data contract drops the threshold pins, detail schema, entry mappings, legacy stage and validation block, and removes the gate from the release-cut scope. Release-cut and full-battery digests are re-pinned; spine and seam part digests are unchanged.
• git grep for uk_qrf_tail, UKQRFTail, UK_QRF_TAIL, reviewed_qrf_tail, qrf_tail_reviewed_exclusions and _evaluate_tail_concentration finds only the kept measurement function, its test and the baselines tool, plus the two history files the PR body names. Other qrf_tail and tail_concentration hits are the shared gate or the US line, which are untouched.
• The count-bearing notes on the old gate entry are gone from the spec at this head. They stay in git history, which can't be helped.
• Helpers the removed evaluator used (_in_line, _with_out_of_line, _export_dropped_columns_from_spine, legacy_name) are still used elsewhere, so nothing is left dead.

What I checked

• Read UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276 (body and your comment), the PR body and the full diff.
• Fresh uv sync --all-packages --locked in a separate worktree.
• ruff check .: passes. ruff format --check flags signed_differences.py, but it is unformatted on main too and CI does not run the format check.
• tools/ci_test_plan.py verify and tools/graph_acceptance_burndown.py --verify: both ok.
• pytest on every touched test file, the UK certification, assembly, gate and contract test files, and all of packages/microcosm-data/tests: all passed (exit 0).
• CI on this head is green.
• The PR adds no new behaviour tests (it removes the gate's tests and repoints a few generic ones), so there was no revert-and-fail check to run.
• Not checked: the certification of the 14 October candidate under this branch, since that needs licensed data. I also did not check whether any already-published UK release is re-validated against the new contract.

🤖 Generated with Claude Code

[juaristi22]

Thanks @vahid-ahmadi. Rebased onto main 71b9b8b52 (#1271) and addressed at df9d13a09:

1. Re-arm the UK QRF tail-concentration gate on independent draws #1277. The PR body now says "Closes Re-arm the UK QRF tail-concentration gate on independent draws #1277", so it closes as superseded when this merges. Its independent-draw point is part of UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276's replacement request (note on #1277). changelog.d/1124-uk-qrf-tail-surface.changed.md now points to UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276 for the removed exclusions instead of Re-arm the UK QRF tail-concentration gate on independent draws #1277.
2. Interim check. Added to the national release runbook (docs/uk-national-release-assembly-runbook-806.md, §2, after certification). Run tools/measure_uk_weighted_integrity_baselines.py on the candidate and the promoted cut together, compare each SPI stage-1 column's weighted total and carrier count, and explain any column that moves by more than half before assembly. The output stays local as licensed output, and the tool suppresses counts below 10.
3. Counts in the replacement. Added to UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276 as a requirement (comment): no raw carrier or draw counts in committed notes or release-shipped reports; shares, ratios and pass or fail only.

On your unchecked item: validate_release_dir is called only from the publication path (microcosm.data.release), the two assembly tools and tools/preflight_uk_local_release_candidate.py, never from the loader. So already-published UK releases aren't re-validated against the new pins.

After the rebase, #1271 moved the spec, so the release-cut part and full-battery digests are re-pinned again; the spine and calibration-seam digests are still unchanged. Touched surface (every changed test file, plus every test importing a touched module or tool, or loading the UK spec): 86 files and 2,475 tests. All pass except the environment-only test_engine_is_reported_unavailable_without_the_uk_extra, which fails on main too when the UK extra is installed.

🤖 Generated with Claude Code