Skip to content

Add the pre-publish comparables review skill and wire it into publishing - #100

Draft
MaxGhenis wants to merge 8 commits into
mainfrom
feat/comparables-review
Draft

MaxGhenis wants to merge 8 commits into
mainfrom
feat/comparables-review

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Every PolicyEngine analysis needs a comparables review before publication and a fresh review before promotion. This PR supplies the operating procedure, background search brief and command wiring.

What changed

  • Merge origin/main, retain its 4.2.0 bundles and added skills alongside comparables wiring, and regenerate the dashboard manifest with the deterministic builder. The catalog now has 26 skills, 36 agents, 21 commands and eight bundles.
  • Document which checks the companion tooling actually enforces. Require manual route/alias resolution and source review because asset heuristics, absolute redirects, unknown links and semantic validator evasions leave gaps.
  • Complete scorecard deployments and actual published metadata after deployment, regenerate the index/feeds, merge the follow-up before registration/promotion, and explicitly verify direct URLs resolve through scorecard metadata alone.
  • Rebuild app-v2's route map for every research registry addition/source change, including iframe-only tools. Cover tracker scaffolding through the supported app path and carry the publication-date follow-up through the bill-review merge.
  • Apply promotion freshness to all PolicyEngine analyses, including calculators, deployment URLs and local drafts. Resolve the record explicitly; distinguish first prepublish/backfill searches and later prepromotion searches.
  • Preserve unexplained records in source-only draft PRs. Qualify the report limitation for existing explains_gap: false annotations and defer generated builds/merge until validation passes.

The skill and command remain in complete, analysis-tools, dashboard-builder and app-development. Companion repositories are unchanged.

Companion PRs and merge order

  1. Record every comparable external estimate before an analysis is published policyengine-scorecard#208: records, producer lists, validation, index and CLI.
  2. Require a comparables record on research entries after the cutover policyengine-app-v2#1380: registry block and publication gate.
  3. Refuse to send an edition whose analyses lack a fresh comparables search newsletters#29: freshness checks before sending recognized analysis links.
  4. This PR last, so the tooling it documents is on main first.

Remaining companion defects include semantic scorecard evasions and the existing-annotation report failure, unregistered HTML/zone routes, omitted absolute redirect aliases and unrecognized delisted deployments when metadata is empty. The procedure describes these limits and supplies manual checks; it does not claim they are mechanically fixed. Required checks, rulesets, CODEOWNERS enforcement and cutover choices remain policy decisions.

Verification

  • uv run pytest --ignore=tests/test_skill_examples.py -q: 632 passed, six skipped; five failures came from a test-only macOS temp wrapper that incorrectly prefixed explicit paths. uv run pytest tests/test_review_program_snapshot.py tests/test_codex_command_skills.py -q passed all 15 tests after restricting that wrapper to argument-free mktemp, verifying all 637 tests across the full run and focused retry. Every run used its own scratch TMPDIR and --basetemp, one worker; scratch was removed afterwards.
  • uv run python scripts/build_claude_wrapper.py --source-root . --output-root <scratch>: passed; packaged skill, brief and command match the sources.
  • uv run python scripts/build_manifest.py --check: passed after regeneration.
  • bun run build in dashboard/: passed.
  • Skill frontmatter validation passed for both edited skills. Companion helper counterexamples and direct-URL recognition checks were exercised read-only.

After merge, refresh app-v2's AI agents stats (26 skills, 36 agents, 21 commands, eight bundles): PolicyEngine/policyengine-app-v2#1397.

policyengine-comparables-review is the operating procedure for the gate
policyengine-app-v2 CI enforces: search every producer on the scorecard's
list, stage what is found under the harvest rules, compare with the
analysis's own figures, and explain every gap over the threshold. Ships a
Subfleet research brief and /comparables-review, and makes the review a
required step in /deploy-dashboard, /new-tool, /create-dashboard,
report-logger drafts and the writing checklist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 5 commits October 21, 2026 11:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-21: "even if we've already published something, we should do a
fresh comp review before big pushes like newsletters". Adds the Before a push
section and the prepromotion search kind to the comparables review, and makes
the freshness check a required first step of /generate-content and the
content orchestrator, and of the writing checklist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The background job gets a writable workspace and gh push access, and the
brief covers every search kind, pushes a draft PR even with an unexplained
gap, and writes a pending cause that cites its issue. The skill lists the
record's fields, the evasions never to use, a reviewer checklist, US sign
conventions and fiscal years, the calculator path, and the commands as CI
runs them. A pending gap blocks a push. Dashboards are reviewed before they
are public. The brief is renamed search-brief.md, and the catalog manifest
is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
basis_note and program_note, the price base, unstated basis or prices,
percent figures, range dashes, one-sided bounds, and hashing the documents
annotations cite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/new-tool and /deploy-dashboard review before anything is public, wire the
zone through appZoneRoutes.ts, and use a registry template app-v2 accepts.
The tracker route is reviewed too, as a tracker record. The skill states the
listing-day, banner and proxied-page rules, the deployments field, record's
latest-search default, report's tolerance of unexplained gaps, the Python
3.12 suite, and that window totals are not compared yet. The brief repins
after superseding estimates. The catalog manifest is rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Heads-up: #107 merged into main (8315e46). It adds a CI job, manifest, that fails when dashboard/src/data/manifest.json differs from a fresh build. This PR changes files the manifest reads, so its next CI run will fail that job until the branch is rebased and the manifest rebuilt. Please don't hand-merge the manifest; run:

git fetch origin && git rebase origin/main
uv run python scripts/build_manifest.py
uv run python scripts/build_manifest.py --check

Then commit the rebuilt manifest. The builder now needs PyYAML, which uv run installs, and it stops on frontmatter that isn't valid YAML.

MaxGhenis and others added 2 commits November 14, 2026 00:52
Keep main's 4.2.0 bundles and added skills alongside the comparables wiring, and rebuild the manifest for the merged sources.

Co-Authored-By: Codex <noreply@openai.com>
Document known enforcement gaps, complete deployment and tracker metadata handoffs, regenerate iframe route maps and cover calculator, deployment and local-file promotion. Preserve unexplained drafts without running index builds that reject them.

Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant