Skip to content

Fix broken source and research links on the study site - #52

Draft
MaxGhenis wants to merge 3 commits into
mainfrom
fix-broken-links
Draft

MaxGhenis wants to merge 3 commits into
mainfrom
fix-broken-links

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the broken links in the uprating viewer (/research, Interactive Uprating Explorer), in Approach, and in one UK example card.

Uprating viewer source links

The viewer built every "View source" link as github.com/PolicyEngine/policyengine-us/blob/master/policyengine_us/parameters/<file_path>. Of the 317 links, 24 returned 404:

policyengine-us names main its default branch, and GitHub redirects master links there.

#3 added public/data/uprating_analysis.csv in October 2025 without recording which policyengine-us release it describes. When #3 merged, policyengine-us's main stood at release 1.413.0 (commit 971e191, 9 October 2025, the 1.412.0 → 1.413.0 version bump). At that commit every file the table names exists, and 315 of its 317 rows match their file's uprating. The other two, social_security_dependents and social_security_survivors, list gov.ssa.uprating. PolicyEngine/policyengine-us#4054 commented out that line in both files on 2 March 2024, and PolicyEngine/policyengine-us#8220 replaced it with calibration.gov.irs.soi.social_security on 4 May 2026. At 1.413.0 both sit on the input list in tools/default_uprating.py, so add_default_uprating grows them with calibration.gov.cbo.income_by_source.adjusted_gross_income instead.

Against main at 61d9e85 (9 November 2026), 65 rows don't match:

So each link now opens its file at commit 971e191. All 317 resolve for good, and each file shows its uprating as it stood in that release. src/data/policyengineUsSnapshot.json holds the pin, and src/utils/policyengineUsSource.js builds the URLs.

The intro

The intro read "Explore the 294 parameters and 23 variables in PolicyEngine-US that grow over time", as if the table listed all of them. At 1.413.0, 306 parameter files carry an uprating key and 60 variable files set uprating, and add_default_uprating grows 71 more input variables with CBO's AGI projection. The table lists 294 parameters and 23 variables, two of them from that default group.

The table leaves out employment_income, self_employment_income, qualified_dividend_income, long_term_capital_gains and taxable_pension_income, among others. Each of those names an IRS SOI target (calibration.gov.irs.soi.*) as its uprating, and that target's own metadata names a CBO income-by-source series. The table also leaves out 12 rate schedules in California, Maine, Massachusetts and Rhode Island. PyYAML's safe_load rejects those 12 and no other uprating file at 1.413.0: seven carry a year-0000 date, and five Maine files fail to scan.

The intro now reads "Explore 294 parameters and 23 variables that PolicyEngine-US 1.413.0 (October 2025) grows over time in line with economic indicators. Each source link opens the file as it stood in that release." It takes the counts from the table, and the release and month from the snapshot file.

Research links

policyengine.org/uk/research/autumn-budget-2024-policy-reform-impacts returns 404, and no post has ever had that slug. policyengine-app-v2 builds slugs from post filenames, and neither its posts.json nor its main history contains that slug. The slug entered this repo in e41149d (2026-02-10), and #28 reused it for the UK example card.

Every other link in both components returns 200. cbo.gov answers curl with 403 (bot protection), but its page loads in a browser.

The check

  • tests/fixtures/policyengine-us-files.txt lists all 6,352 files under policyengine_us/parameters and policyengine_us/variables at the pinned commit. scripts/snapshot-policyengine-us-files.mjs writes it from GitHub's API, and its list matches what git ls-tree -r --name-only 971e191 gives for both folders. The script also reads the commit's version (from pyproject.toml) and date, and writes both into the snapshot file and the list's header. .gitattributes marks the list generated, so GitHub folds it in diffs.
  • src/utils/policyengineUsSource.test.js checks offline that every CSV row's link opens a listed file. It also checks that:
    • parameters go to .yaml files under parameters/ and variables to .py files under variables/;
    • every URL pins the commit the list records;
    • the snapshot's version and date match the list's header;
    • a row with no file or an unknown type gets no link.
  • src/components/UpratingViewer.test.jsx renders the viewer on the real CSV and opens every category. It requires each card to show one link for each item it lists (up to 50) and each link to open its own row's file. It also checks that a variable (household_weight) links to its Python file.
  • vitest.config.mjs runs the suite in America/Los_Angeles. There, UTC midnight on the 1st falls in the prior month, so the release-month test fails if the formatter drops its UTC time zone. A precondition in that test fails if the suite stops running west of UTC.

At least one test fails when:

  • the URL builder sends variables back to parameters/;
  • it links main instead of the commit;
  • the pin moves and nobody regenerates the list;
  • the snapshot file keeps a stale version or date;
  • a row points at a file that doesn't exist;
  • the viewer shows links only for a card's first item or only for parameters, or rotates links between items;
  • the release-month formatter drops its UTC time zone, even with TZ=UTC in the environment;
  • the config stops setting its time zone while the environment sets TZ=UTC.

All 317 pinned URLs return 200 from github.com. On a local static build, the viewer shows the new intro and 153 links across its 18 categories, one for each item a card shows. Each link pins 971e191 and opens its own item's file.

To move to a newer release:

  1. Regenerate the CSV against it.
  2. Set commit in src/data/policyengineUsSnapshot.json.
  3. Run bun scripts/snapshot-policyengine-us-files.mjs. It fills in version and date.
  4. Run bun run test:run.

Invariants

  • Every source link the viewer renders opens a file present at the pinned commit. The unit test covers all 317 rows.
  • Each item a card shows carries exactly one link, which opens that item's own file.
  • Every link pins a full 40-character commit, the same one the file list records. The release and date the intro names belong to that commit.
  • Parameters link .yaml files under policyengine_us/parameters; variables link .py files under policyengine_us/variables.
  • A row with no file or an unknown type renders no link.
  • The release month the intro names doesn't depend on the reader's time zone.

Not in this PR

  • The table itself. It lists part of what 1.413.0 grows, and its two Social Security rows name an index that release doesn't apply. StochasticForecasting.jsx hard-codes "318 uprating parameters" and the category counts above the viewer, while the CSV holds 317 rows. Regenerating the table takes its own PR, with a committed script that reads each value's effective uprating, defaults included, and stops on any file it can't parse.
  • UK card images. The UK cards reuse US screenshots; this card keeps the "AI-written policy reports" image it had.
  • "Used by the UK government": d2686 asks Max what should back it.

Overlap with #50

#50 edits Approach.jsx and ExampleProjects.jsx in other hunks, so this PR rebases onto main after #50 merges.

Review

An independent Opus review of 585fc49 requested changes, and the next two commits answer each finding:

  • The No 10 link: the claim now links nothing, and d2686 asks Max what it should cite.
  • The intro's implied completeness: the intro drops "the" and names the release, and this body lists what the table leaves out.
  • "All 317 rows match": 315 do, and this body says why the other two don't.
  • A time-zone guard that couldn't fail in CI: the suite now runs west of UTC, and a precondition fails if it stops.
  • A per-card check that a first-item-only mutant passed: the test now requires one link per item, each to its own file.
  • Unchecked version and date: the script reads both from GitHub, and a test checks them against the list's header.

A second Opus review, at 3c54031, approved. This body carries the three text fixes it asked for: the Fellowship article, the count of default-uprated inputs (71, not 69), and a scoped follow-up for the table.

axiom: n/a: study-site links and tests; no policy change.

🤖 Generated with Claude Code

The uprating viewer linked every row to policyengine-us's master branch
under policyengine_us/parameters, so all 23 variable links pointed at the
wrong folder, and one parameter file has since moved. Every row in
public/data/uprating_analysis.csv matches policyengine-us 1.413.0 (commit
971e191, 9 October 2025): the file exists and shows the uprating the row
reports. The links now open that commit, which keeps all 317 resolving
whatever the repository moves later. The intro names the release and
takes its counts from the table.

tests/fixtures/policyengine-us-files.txt lists every file under the
parameters and variables folders at that commit, written from GitHub's
tree API by scripts/snapshot-policyengine-us-files.mjs. Vitest checks
every row's link against it offline, and a render test checks the links
the viewer shows.

No post ever had the slug autumn-budget-2024-policy-reform-impacts.
"UK government" in Approach now cites the No 10 Downing Street post, and
the UK example card shows the combined Autumn Budget 2025 analysis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai-inequality Ready Ready Preview Nov 9, 2026 7:38pm UTC

Request Review

The Approach card's "UK government" no longer links a post. Both posts
that could back it, No 10 and the HM Treasury pilot, credit Nikhil
Woodruff, and gov.uk's record says HMT does not currently use
PolicyEngine. Decision d2686 asks Max what the claim should cite.

The uprating viewer's intro now says it lists some of the parameters and
variables that PolicyEngine-US 1.413.0 grows over time. At that release,
306 parameter files and 60 variable files set uprating, and the model
gives 69 more input variables a default; the table lists 294 parameters
and 23 variables.

The snapshot script reads the pinned commit's version and date from
GitHub and writes them into the snapshot file and the file list's header,
and a test checks that the two agree. Vitest now runs in
America/Los_Angeles, so the release-month test fails if the formatter
drops its UTC time zone. The viewer test checks that each item a card
shows carries one link, to its own row's file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Explore 294 parameters and 23 variables that PolicyEngine-US 1.413.0
grows over time" claims only what the table holds. It drops the "the"
that said the table lists all of them, without an "of the" that points
readers at the rest. "In line with" replaces the dangling "based on",
and the link sentence says each file opens as it stood in that release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 3c54031b Deployed Nov 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant