Skip to content

Dashboard: withdraw the provisional expected saving after the binding C2 failure - #48

Merged
MaxGhenis merged 3 commits into
mainfrom
dashboard-withdrawn-ev
Oct 28, 2026
Merged

MaxGhenis merged 3 commits into
mainfrom
dashboard-withdrawn-ev

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Max said go (d2146, 28 Oct): merge on the gates below. This changes a published page. The explainer post went live with the matching note at 20:52Z (PolicyEngine/policyengine-app-v2#1473, d2147), so until this merges the dashboard still shows the average the post has withdrawn.

The binding C2 score on post-Budget data (#45, docs/C2_BINDING.md) failed: test A's floor-frequency bias is 0.267 against a limit of 0.25. The floor frequency is the share of years with earnings growth strictly below 2.5% (ts_backtest.py, METHOD); the model predicted 44.6% against 17.9% realised. Test A gained a seventh forecast (March 2022) once September's CPI completed its window; the dry run had six. Under Max's d955(c), the expected value then falls back automatically to (a): scenarios only, no expected value. The rebuild on the Budget's inputs, booked from 29 October, writes that file itself, but takes about a day of Microcosm compute (REBUILD.md's runtime paragraph). Until it lands, the live dashboard is the provisional pre-Budget (c) release. That release leads with the expected saving and its middle-80% range.

The explainer post is getting the matching note in PolicyEngine/policyengine-app-v2#1473. This PR's notice uses the same facts, order and wording, except two deliberate differences: the notice quotes no withdrawn figure, and the model's paths stay as labelled illustrations, which d955(a) lists (the post drops them).

What changes on the page

  • A dated notice at the top: "Updated 28 October 2026". It says what the pre-Budget release showed, that the model failed a test we had registered before the Budget (linking Binding C2 re-score on October 2026 Budget inputs #45), and what was withdrawn. It adds what the pre-Budget release uses (forecast, PolicyEngine UK version and survey, read from the file) and what the rebuild uses: PolicyEngine UK 2.151.0, the Microcosm UK dataset (engine.py's binding bundle, d2011), and the OBR's October 2026 forecast to 2030 with its March 2026 long-term assumptions after (d2124).

  • No expected value anywhere. withBindingOutcome() (dataHelpers.js) takes the provisional (c) file and:

    • drops expected_value;
    • records the (c)→(a) fallback the way the binding build does (uncertainty_ruling, expected_value_omission.reason);
    • leaves every other block identical (tested by reference).

    The page then uses its existing, tested scenarios-only rendering:

    • the scenario envelope is the headline: central forecast, OBR long-term path, the ±0.5pt earnings mean-path scenarios and the last-decade replay, all labelled "not a probability interval";
    • the expected saving, its middle-80% range, the share-losing average and the histograms are gone.
  • Path names. Without an expected value, the model's paths are scenarios (d955(a): "model paths labelled as scenarios"). They become:

    • "Model path: a random draw";
    • "Model path: a middle-sized gap";
    • "Model path: a large gap".

    "Typical", "high-saving" and the per-path percentile are gone, and the paths tab says these are illustrations, not a typical or likely path. The names follow the expected value alone (pathsAreScenarios), whatever the ruling, so they also apply to the rebuild's scenarios-only file.

  • No typical path or average anywhere. The Distributional impact tab's poverty card no longer gives "the typical path" as context, and its footer no longer points to an average over all paths. The "What each path is" list no longer places the scenario paths at the median, the 90th percentile or a percentile band.

  • Methodology tab:

    • it shows the omission reason;
    • it labels the C2 scores it shows (from the file) as the dry run's, adding that, rescored on 28 October with a seventh past forecast, the model failed.

Any other file, a sample included, passes through unchanged. Remove BINDING_C2_OUTCOME, withBindingOutcome and BindingOutcomeNotice with the rebuild, whose file records the outcome itself.

Checks

  • New BindingOutcome.test.jsx (13 tests):
    • passthrough for passing and failing binding files, a file with no expected value, a sample and a provisional sample;
    • path names that follow the expected value alone, even under ruling (c);
    • every tab's whole text, collapsed <details> footers included, scanned for typical, percentile, median and average-over-paths wording. Only four phrases are allowed: the envelope's earnings scenarios, the past-years check score, and the two lines saying no path is typical and no average is shown;
    • the transform's exact changes, with every other block kept by reference;
    • idempotence and no mutation of the committed object;
    • scenario path names;
    • on every tab: the notice once, no withdrawn figure, no "provisional until", no typical or percentile path names, and nothing broken or "unavailable";
    • the "What each path is" list on every tab uses the scenario names;
    • the reason states the statistic as earnings growth below 2.5%, never "the floor setting the rise", and its numbers are consistent (bias = predicted − realised, above the limit);
    • the notice's inputs and rebuild sentences, and no "Budget's figures";
    • the method tab's omission, dry-run and rescore labels;
    • no notice on the committed file itself or on a sample.
  • Four existing tests in Steps.test.jsx now follow the file's path names, and whether it has an expected value, instead of hard-coding "90th percentile" or the typical-path context. One new test checks that a file without an expected value names no typical path.
  • Test runs:
    • bun run test: 188 passed, 4 skipped.
    • TRIPLE_LOCK_RESULT_FIXTURE=c2-pass: 174 passed.
    • TRIPLE_LOCK_RESULT_FIXTURE=c2-fail: 1 failure plus the ruling-(b) suite. Both already fail on main in that mode and are untouched here; they belong to the rebuild.
  • Mutation check. Each of these makes the new tests fail: the old floor wording, an inconsistent bias, "Budget's figures" in the notice, a hard-coded random path name, a dropped model version, the old method-tab line, a transformed sample, the typical-path context, the footer's average, the percentile footer, and path names that require ruling (a).
  • Review. Sol at 588b696 requested changes on five points. 101f3ba fixed the floor wording; 563db6f fixed the other four. A re-review at 563db6f is running (20261028-172859-pr48-review-r2).
  • Build and lint. bun run lint shows the same 2 warnings as main. bun run build passes.
  • Browser check. I checked the page in a local next start of this branch: the notice, the Budget-impact scenario envelope, the paths tab names and the methodology text. There were no console errors.

Merge gates

  1. Binding C2 re-score on October 2026 Budget inputs #45 merged: the notice states its result.
  2. An independent review approves at the head.
  3. CI green. Max's go is given (d2146).

🤖 Generated with Claude Code

… C2 failure

The binding C2 score on the Budget's inputs (#45) failed, so ruling (c)
falls back to (a): no expected value, scenarios only. Until the rebuild
replaces it, the committed file is still the provisional pre-Budget (c)
release, which carries the expected value.

withBindingOutcome() presents that file as the fallback requires: it
drops expected_value and records the (c) to (a) fallback the way the
binding build does, so the page uses its existing scenarios-only
rendering. Every other figure is the file's own. A dated notice says
what the page showed, why it was withdrawn and what it shows now, and
the method tab labels the scores it shows as the dry run's.

Without an expected value the model's paths are scenarios, so they are
named for their gap ("Model path: a large gap") with no typical or
percentile claim. That also applies to the rebuild's scenarios-only file.

Any other file is unchanged; delete the transform and notice with the
rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
uk-triple-lock Ready Ready Preview Oct 28, 2026 9:29pm UTC

Request Review

…post's note

The reason string said the model overpredicts "the 2.5% floor setting the rise". C2's
floor frequency is the share of years with earnings growth strictly below 2.5%
(ts_backtest.py, METHOD), distinct from the older both-inputs-below-floor diagnostic.
It now says so, as the explainer post does.

The notice no longer says the model failed "on the Budget's figures", which implied the
Budget caused it. The rescore added a seventh past forecast once September's CPI
completed the March 2022 window. The notice and method tab now say that and use the
post's words. The second paragraph names what the pre-Budget release used (read from
the file) and what the rebuild uses (engine.py's binding bundle and d2124's forecast
mix). The "What each path is" list now uses the file's path names for the random path
too. The misplaced isPreBudget docstring is back on its function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… expected value

The Sol review at 588b696 found four gaps in the withdrawal (its fifth finding, the
floor wording, was fixed in 101f3ba):

- Distributional impact still described "the typical path" in the poverty card, and
  its footer pointed readers to an average over all paths. Without an expected value,
  neither appears now.
- The "What each path is" list described the scenario paths by median, 90th
  percentile and percentile band. It now says they were picked for the size of
  their gap.
- The transform changed a provisional sample. Samples now pass through untouched.
- Path names required ruling (a) as well as no expected value. They now follow the
  expected value alone (`pathsAreScenarios`), whatever the ruling. The envelope
  rendering still uses `isScenariosOnly`.

Each tab's whole text, collapsed <details> footers included, is now scanned for
typical, percentile, median and average-over-paths wording. Only four phrases are
allowed: the envelope's earnings scenarios, the past-years check score, and the two
lines saying no path is typical and no average is shown. Each fix, reverted alone,
fails a test. Two Steps tests now follow whether the file has an expected value, so
c2-fail mode fails only where main already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge audit for head 563db6f99c941d94079b8d6c195c6d2e32c773f5 (base main at b4c56c2, after #45 merged).

  • Decision: d2146, decided by Max on 28 October (~21:00Z): merge once its Sol reviews approve, CI is green and Binding C2 re-score on October 2026 Budget inputs #45 has merged. Binding C2 re-score on October 2026 Budget inputs #45 merged at 22:22Z (b4c56c2).
  • Reviews:
    • GPT-6.1 Sol review at 588b696 asked for changes (five blockers); 563db6f fixes them.
    • GPT-6.1 Sol r2 at 563db6f: APPROVE, job 20261028-172859-pr48-review-r2, verdict at ~/reviews/uk-triple-lock-2026-09-29/release-night/pr48/review-sol-r2.md. Its one non-blocking note is the allowlist granularity in BindingOutcome.test.jsx. That code goes away when Binding rebuild on the October 2026 Budget inputs (scenarios only) #49 deletes the withdrawal notice after the binding rebuild, so it isn't changed here.
  • CI: on the merge with the new main, gh pr checks 48 exits 0. Pipeline run 40780494920 passed (31 min) and the Dashboard run 40780494918 passed.
  • mergeable is MERGEABLE; there are no CHANGES_REQUESTED reviews; the PR isn't a draft.

Merged with --match-head-commit 563db6f.

@MaxGhenis
MaxGhenis merged commit da346e3 into main Oct 28, 2026
6 checks passed
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
…fstat, locale-free months

The hard review of 125bb99..1f0a18e approved, with one P3, three P4s and
four notes (~/reviews/uk-triple-lock-2026-09-29/release-night/pr49/review-sol-delta5.md).

1. (P3) _release finds each directory the write made under its parent
   and name, so one that was itself moved or renamed stays behind, empty.
   1f0a18e's docstring and commit message claimed otherwise. The
   docstrings of _release and write_interim now say what the cleanup
   reaches and what it misses.
2. (P4) _open_directory now reads a new directory's identity before it
   dups the parent handle, so a failed fstat leaves no descriptor open.
3. (P4) The _release docstring now names all three ways the identity
   check can remove another process's empty directory: a swap between the
   mkdir and the open after it, a swap between the check and the rmdir,
   and a file system that reuses a removed directory's identity.
4. (P4) The C2 panel says "only" only when the models missed some final
   years. A new test covers the guard that drops the published-earnings
   sentence when the rows scored different numbers of years; mutant D6
   survived without it.
5. (Note) write_interim releases the directories it made even when
   closing its directory handle raises.
6. (Note) _month_order reads the English month names itself instead of
   through strptime, so a process-wide LC_TIME can no longer stop the
   results file's assembly after every job has run.
7. (Note) BindingOutcomeNotice keeps its prefix match. It shows only on
   the pre-Budget file, whose id it knows, and the binding release
   deletes it with the rest of #48's withdrawal code.
8. (Note) The build call site's comment now says the pre-Budget file
   keeps the historical note (METHOD_LIMITATIONS). 1f0a18e's message
   called that "the note that release published", but the published
   release is a provisional (c) file.

Tests: the full suite (2431 passed) fails only the four stale-file
assertions the build clears; the dashboard passes 235 tests, lint and
build. A test catches each of the five new mutants
(dup before fstat, release skipped on a failed close, strptime months,
"only" always, no scored guard).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
The binding file records the C2 outcome itself, so the page no longer
needs withBindingOutcome, BindingOutcomeNotice or BINDING_C2_OUTCOME.
Rendering the interim file showed what those had hidden:

- The Budget tab and the Method tab printed the file's own reason,
  "failed the frozen C2 statutory screen; automatic d955(a) scenario-only
  fallback: A/suspended: |floor_bias| 0.267 exceeds 0.25", after "The
  original primary failed C2, so the build automatically takes (a)".
- The Method tab's headline said coverage fell "short of the pass mark".
  Coverage passed; the floor-frequency bias failed (docs/C2_BINDING.md).

getC2FailureText now words the failure from the file's own scores, and
the headline states the counts without judging them. A failure the page
cannot word gives only "The forecast model failed its test on past
forecasts, so only scenarios are shown"; the page never prints the
file's reason.

Ten tests and one suite pinned the pre-Budget file and would have failed
on the binding commit. They now run on a file of the right shape, or
only on the pre-Budget file they describe. ScenariosOnly.test.jsx keeps
the withdrawn page's checks on a failed-C2 file, and adds the wording's.

The suite passes on the committed file, on data/interim/scenarios-first.json
in its place, and in both fixture modes. This must land with the binding
results: on the pre-Budget file the page would show the expected value again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 563db6f9 Deployed Oct 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant