Skip to content

Refuse to send an edition whose analyses lack a fresh comparables search - #29

Draft
MaxGhenis wants to merge 6 commits into
mainfrom
feat/comparables-freshness
Draft

MaxGhenis wants to merge 6 commits into
mainfrom
feat/comparables-freshness

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Max, 2026-10-21: "also like even if we've already published something, we should do a fresh comp review before big pushes like newsletters". This is the promotion-time half of the pre-publish comparables gate (PolicyEngine/policyengine-scorecard#208, PolicyEngine/policyengine-app-v2#1380).

What changed

  • Merge upstream/main and retain its hosting-link lint and upload checks alongside the comparables gate.
  • Pin the test matrix to Ubuntu 24.04 so setup-python can install the supported Python 3.8 and 3.9 versions.
  • Preserve NUL-delimited changed filenames through CI selection, including Unicode, tabs and newlines.
  • Recognize external primary record pages and complete browser-equivalent URLs; keep shared deployments scoped by path and selected country.
  • Reject malformed index URLs, registry items and unresolved or noncanonical route-map entries before matching links. Add regressions for every R1–R7 counterexample.

The rule

Every policyengine.org analysis an edition links must have, in PolicyEngine/policyengine-scorecard:

  • a comparables search dated on the push date or in the scorecard's promotion_window_days before it (7 by default, counting both ends; the window is Max's call, queued);
  • that compared the results files the analysis reads now (the search's results_pins equal the record's current_pins);
  • and no gap found by then still pending (a pending gap with no found_on counts as found).

An analysis here is an apps.json app listed with research or tagged policy, or any post, under the rules the scorecard's index publishes (entry_rules). Only a post that is plainly not an analysis (tagged org, event, giving and so on, and none of the analysis tags) may skip the check by declaring no_estimate; policy wins over it, and an app never may. A page that has left the registry but still has a scorecard record is held to that record. The rule applies to pushes on or after 2026-10-26.

Where links lead

Read from policyengine-app-v2's route map (website/src/data/analysisRoutes.json, PolicyEngine/policyengine-app-v2#1380), built from the site's evaluated rewrites and redirects: every path, redirect (tracker, CliffWatch, OBBBA aliases, country-wide zone routes) and deployment host (Vercel, Modal, GitHub Pages narrowed by project path, a deployment shared by two countries narrowed by path and country query/default) that leads to a research app. And from the scorecard's records: each analysis's page and deployments, so a dashboard taken off the site, like the childcare one, is held to its record on every link, its own Vercel URL included. Links are normalised first: any scheme or none, www., case, ports, HTML entities and percent-encoding, dot segments, a trailing dot on the host, trailing punctuation; posts by slug under any country, through /research/, /blog/ or a dated slug. Mailchimp tracking links and link shorteners are refused. For shared deployments, source URLs in the same pinned registry disambiguate the route map's candidates; they never add routes. Ambiguous destinations are refused. Every source (registries, route map, index) is read at the commit its main branch points to, so a cached copy is never read.

Where it is enforced

  • CI: the Fresh comparables for promoted analyses job checks the editions a change adds or modifies under editions/ (on a push, the pushed range; a new branch against the empty tree), each on its own date, so an edition already sent is not re-judged against later data. A changed edition must be named for its date, a social post outside editions/ fails, and a base commit the checkout cannot read fails rather than passing. It downloads its sources only when an edition is due; a failed download, malformed index or registry, or inconsistent route map fails the job.
  • upload-newsletter, for new drafts and --campaign-id updates: refuses an undated file, judges the edition on the later of its date and today, and refuses a stale one before calling Mailchimp.
  • /create-newsletter, /upload-draft, CLAUDE.md: the pre-send step, and CLAUDE.md's draft-update recipe now goes through upload-newsletter --campaign-id rather than the content endpoint.

The check confirms the search is fresh against current results pins and no pending gap remains; it does not put the comparable in the edition.

The 20 October UK edition fails it, run with the cutover moved back, app-v2's current registries and scorecard#208's index:

FAIL 2026-10-20-uk.html
  2026-10-20-uk.html: policyengine.org/uk/middle-east-war-living-standards has no comparables record in the scorecard; run the comparables review (analysis-uk-middle-east-war-living-standards) before sending
  2026-10-20-uk.html: policyengine.org/uk/childcare-100k-threshold: no comparables search between 2026-10-13 and 2026-10-20 (last: none); run a prepromotion search before sending

The Scottish Budget social post (2026-01-27-uk-scottish-budget-social.md), which links without a scheme, fails too: its dashboard has no record.

Tests

  • Full suites on Python 3.8.20 and 3.11.15: 346 passed on each, including workflow filename selection, all R1–R7 counterexamples, source validation and the existing hosting-link checks.
  • GitHub CI passes the newsletter lint, comparables job and Python 3.8–3.11 matrix at 7427224.
  • Black and ruff pass for src/, tests/ and the new selection helper. The newsletter HTML lint passes.
  • The review's immutable app-v2 and scorecard artifacts validate, and 16 end-to-end review cases pass against them.
  • Freshness still agrees with all 96 shared scorecard cases; sources remain pinned to the commits their main branches resolve to for each run.

Order

This merges after scorecard#208, whose index carries the window, pins, entry rules and deployments, and after app-v2#1380, which publishes the route map. At verification, app-v2 main (e413ba8549f53d3c994148f35603ea33dc2200de) still lacks website/src/data/analysisRoutes.json; the gate continues to fail closed until that artifact is published.

🤖 Generated with Claude Code

Max, 2026-10-21: "even if we've already published something, we should do a
fresh comp review before big pushes like newsletters". Every policyengine.org
analysis an edition dated on or after 2026-10-26 links must have a comparables
search in PolicyEngine/policyengine-scorecard within its promotion window and
no gap still pending. A new CI job and upload-newsletter both enforce it;
/create-newsletter and /upload-draft say how to satisfy it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Links are recognised in every common form (no scheme, any case, escaped,
blog/ and dated slugs, the site's redirects, the apps' own hosts), and
tracking links and shorteners are refused. Posts are found by slug under any
country. Every analysis is checked unless it may declare no estimate. The
index's shape is validated, and fresh now requires the search to have
compared the current results, tested against the scorecard's own cases.
Uploads refuse undated files and judge on the later of the edition's date
and today; CI checks only the editions a change adds or modifies. The draft
update recipe goes through upload-newsletter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits October 22, 2026 13:58
…in CI

Links are matched against policyengine-app-v2's analysisRoutes.json (every
path, redirect and deployment host that leads to a research app) and the
scorecard's records (each analysis's page and deployments), so tracker
paths, country-wide zone routes, non-Vercel hosts, GitHub Pages projects and
a delisted dashboard's own URL are all checked. URLs are normalised (dot
segments, a trailing dot on the host). The index and route map are validated
strictly, and every source is read at the commit its main branch points to.
CI fails on an unreadable base, compares a new branch with the empty tree,
and refuses social posts outside editions/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the hosting-link lint and upload checks alongside the comparables gate.

Co-Authored-By: Codex <noreply@openai.com>
Preserve changed filenames through CI, resolve complete destination URLs, and scope shared deployments by path and country. Validate all consumed source artifacts so malformed data cannot remove coverage, with regressions for every round-three finding.

Co-Authored-By: Codex <noreply@openai.com>
Pin the test runner to Ubuntu 24.04 because ubuntu-latest now resolves to 26.04, where setup-python cannot install Python 3.8 or 3.9.

Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant