Skip to content

Infer target repos from whole-token mentions in the whole body - #109

Draft
MaxGhenis wants to merge 1 commit into
mainfrom
repo-coverage
Draft

MaxGhenis wants to merge 1 commit into
mainfrom
repo-coverage

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes how scripts/build_manifest.py decides which PolicyEngine repos each skill, agent and command targets: the follow-up that #107's section 5 left out. infer_target_repos fills each artifact's target_repos in dashboard/src/data/manifest.json, and repo_coverage and some registry owners follow from it. It matched aliases as case-insensitive substrings, matched a repo name inside a longer token, and read only the first 6,000 characters of a body. This PR removes 107 of its 267 artifact-to-repo links and adds 62.

#107 merged, so this PR now targets main: I rebased it onto 8315e46 and rebuilt the manifest, and every number below compares main's manifest at 8315e46 with this branch's. It stays a draft until an independent review agrees.

1. The rule

RepoMatcher reads an artifact's name, description, whole body and category from left to right. At each position it takes the longest term that stands there as a whole token, records that term's repo, and carries on after it. The terms are each repo name in the inventory, its import name (policyengine_us) and the aliases in REPO_ALIASES.

  • Whole tokens. A letter, digit or underscore on either side makes a different token, so "SSI" in "missing" and "PIP" in "pipeline" don't count.
  • Slugs. Repo names, and aliases with a hyphen or a dot in them ("app-v2", "api.policyengine.org"), are slugs. A hyphen on either side, or a dot before, makes a longer slug or a hostname, so "policyengine-model-development" doesn't name policyengine-model and "v2.api.policyengine.org" doesn't name policyengine-api. A dot after a slug ends it, so "policyengine-us.git" and a sentence-ending "policyengine-us." count.
  • Words. Every other alias is a word or a phrase, and a hyphen leaves it whole: "SNAP-eligible" and "non-SSI" count.
  • Longest first. "policyengine-us-data" names policyengine-us-data and never policyengine-us. "api.policyengine.org" names policyengine-api, not the web app.
  • Case. A term in capitals counts only in capitals; every other term counts in any case (section 2).
  • Order. The result lists the repos named outright in inventory order, then the repos that only an alias implies, in REPO_ALIASES order, each once. The old function used the same order.

The old function ran one regex per repo and one substring test per alias. The new one runs str.find per term and checks the characters on either side, so it never builds a regex from repo names. A full build takes about 2 seconds here.

2. Decisions

Case. An acronym in lowercase is usually code or another word: uv pip install, foo.cpp, OBBBA's 2030 "snap-back", a ctc filter tag. So a term written in capitals (PIP, SNAP, CRM) counts only in capitals. I measured the alternative on main's 80 artifacts: counting the capitals aliases in any case adds 7 links. Four of them come from pip and tag policyengine-uk, one comes from the ri-ctc-calculator-backend service name and tags policyengine-us, and two are arguable (the parameter file gov/irs/credits/ctc.yaml in reference-validator's example table, and the ctc/eitc filter tags in /prior-analysis). Dotted model paths such as tax_unit.eitc and gov.usda.snap appear only in artifacts that target policyengine-us anyway, so capitals-only drops none of those links.

Every other term, repo names included, counts in any case. GitHub resolves repo names in any case (gh api repos/policyengine/aca-calc returns PolicyEngine/ACA-Calc), prose capitalizes them, and trigger lists write phrases in lowercase ("universal credit"). The old function matched repo names only as written; ignoring case adds 3 links, all real: "a new PolicyEngine-US state benefit" in encode-policy-v2, and aca-calc in /create-dashboard and /new-tool.

The cap. Gone. A cap at any position breaks "adding text never removes a repo", because text added near the top pushes a later mention past it. 47 of the 80 artifacts have bodies longer than 6,000 characters. Capped at 6,000 characters, the new rule would find 57 fewer links:

  • 37 are new links that only text past the old cut supplies, the row of that name in section 3. 34 are plain mentions of the repo, such as policyengine-household-api in the cloud-costs skill's table of past incidents. The other 3 come from the limits in section 6.
  • 6 are new links that section 3 credits to the web-app aliases or api.policyengine.org, whose only mention also sits past the cut.
  • The old function had the other 14 too, through a match in the first 6,000 characters that this PR drops or remaps.

The alias table.

  • policyengine.org and app-v2 now point at policyengine-app-v2, not the legacy policyengine-app. The policyengine-app skill describes app-v2 as "the site served at policyengine.org".
  • api.policyengine.org now points at policyengine-api. Before, the substring policyengine.org inside it tagged the legacy web app.
  • "frontend" no longer counts. The generic word tagged three dashboard artifacts as the legacy app.

Registry owners. owner_for_artifact now treats the country-models category as country-model work. Two agents in that category, model-worker and review-worker, owed their only country-model link to a false match, so without this change they would move to the generic owner.

3. What changes

Only target_repos, registry_owner and the per-repo counts in repo_coverage change. Links between artifacts and repos, by repo kind:

Repo kind Before After Added Removed
country-model 129 77 5 57
platform 78 91 39 26
other 26 9 2 19
interactive-instance 13 17 8 4
library 7 7 0 0
research-platform 7 8 2 1
internal 3 4 1 0
data-pipeline 2 7 5 0
long-lived-tool 2 2 0 0
All 267 222 62 107

69 of the 80 artifacts gain or lose a link. A 70th, pr-pusher, keeps its repos in a new order: its policyengine-us link now comes from the repo's name, not an alias, so it sorts with the named repos. Why each link went:

Removed because Links Examples
an alias sat inside a longer word 55 "SSI" in missing, guessing, permissions, Accessibility, classifies; "PIP" in pipeline, piped; "GIS" in registry, registered, legislative, legislature; "SNAP" in snapshot
a repo name sat inside a longer token 33 policyengine-model in policyengine-model-development (18); policyengine-app in policyengine-app-v2/... (7); atlas in policyengine-atlas-pbif-grant.md; poverty-dashboard in child-poverty-dashboard
the web-app aliases now name policyengine-app-v2 14 "embeds in policyengine.org", api.policyengine.org/calculate
"frontend" no longer counts 3 "Wires frontend components to backend"
an acronym in lowercase 2 "bare pip", "non-uv pip installs"
Added because Links Examples
the mention sat past the old 6,000-character cut 37 policyengine-household-api and policyengine-api in the cloud-costs skill's table of past incidents; state-legislative-tracker in microsim-runner
the web-app aliases now name policyengine-app-v2 16 "Next.js multi-zones behind policyengine.org"
api.policyengine.org now names policyengine-api 5 "call api.policyengine.org" in microsim-runner
an import name 2 'policyengine_uk' : 'policyengine_us' in dashboard-integrator
a repo name in another case 2 aca-calc

By repo:

Repo Kind Before After Added Removed
policyengine-us country-model 73 49 1 25
policyengine-app platform 33 10 1 24
policyengine-model other 18 0 0 18
policyengine-canada country-model 25 8 1 18
policyengine-uk country-model 31 20 3 14
atlas interactive-instance 1 0 0 1
ctc-calculator interactive-instance 2 1 0 1
marriage other 3 2 0 1
policyengine-api-v2 platform 4 3 0 1
poverty-dashboard interactive-instance 1 0 0 1
uk-autumn-budget-dashboard interactive-instance 1 0 0 1
CRM other 3 4 1 0
calibration-diagnostics research-platform 7 8 2 1
newsletters internal 1 2 1 0
policyengine-household-api platform 0 1 1 0
snap-bbce-repeal other 0 1 1 0
uk-spring-statement-2026 interactive-instance 0 1 1 0
ACA-Calc interactive-instance 0 2 2 0
policyengine-claude platform 1 3 2 0
policyengine-uk-data data-pipeline 0 2 2 0
south-carolina-2026-tax-changes interactive-instance 2 4 2 0
policyengine-core platform 7 10 3 0
policyengine-us-data data-pipeline 2 5 3 0
policyengine.py platform 3 7 4 0
state-legislative-tracker interactive-instance 1 4 3 0
policyengine-api platform 12 17 6 1
policyengine-skills platform 2 7 5 0
policyengine-app-v2 platform 13 30 17 0
  • Repos that any artifact targets: 39 → 40. Five join (ACA-Calc, policyengine-household-api, policyengine-uk-data, snap-bbce-repeal, uk-spring-statement-2026). Four leave (atlas, policyengine-model, poverty-dashboard, uk-autumn-budget-dashboard), and in each case the only link came from a longer token.
  • Tooling-relevant repos that no artifact targets: 35 → 33.
  • Registry owners: /prior-analysis and /prior-scores move from "Country model engineering" to "PolicyEngine". Their only country-model links came from "pipeline", "registry", "not-possible" and the ctc/eitc filter tags.

The Catalog's repo filter and chips, the drawer's "Target repos" list and the Workflows page's repo chips read target_repos. The drawer, the Catalog and the Cleanup page show registry_owner. No page reads repo_coverage yet.

4. Invariants

These hold for every input, and the tests check each one:

  • A term with a letter, digit or underscore on either side never counts. Nor does a slug with a hyphen on either side or a dot before it (Hypothesis, any generated term).
  • A term between common delimiters (spaces, brackets, backticks, quotes, slashes, sentence punctuation, .git, 's) always counts (Hypothesis).
  • A hyphen on either side leaves a word alias whole (Hypothesis).
  • A term in capitals counts only as written; every other term counts in lowercase, capitals, title case or swapped case (Hypothesis).
  • Inserting lines never removes a repo, and inserting lines that name no repo changes nothing. The generated lines include one longer than the old 6,000-character cut (Hypothesis).
  • The order of lines never changes the result (Hypothesis).
  • The matcher agrees with reference_targets, a plain character-by-character scan of the same rule. The tests check this for the real table, and for random small tables whose terms overlap, nest, repeat and start one another (Hypothesis, two differential tests).
  • Every repo name in the inventory, and every import name, targets exactly its own repo. Every alias targets a known repo (exhaustive over the inventory).
  • A table with an empty alias, or an alias that repeats a repo name, an import name or another alias in any case, raises ValueError.
  • The result lists each repo once: named repos in inventory order, then alias-only repos in REPO_ALIASES order.

5. Tests

  • tests/test_target_repos.py: 64 tests, 8 of them Hypothesis properties, in under 30 seconds. Most examples come from this repo's own skills, agents and commands, with at least one for each kind of false positive in section 3.
  • I ran every property with 4,000 examples, not the default 100 or 300. All passed.
  • Mutation check. Each of these 18 edits to the matcher makes at least two tests fail, at least one of them a fixed example:
    • dropping the left boundary, or the right one;
    • letting a hyphen follow a slug, or a dot precede one;
    • treating every alias as a word, or as a slug;
    • letting capitals terms ignore case, making every term exact, or making repo names exact;
    • dropping the merge, letting the shortest term win, or listing alias-only repos first;
    • dropping import names, restoring the 6,000-character cap, or restoring the old alias table;
    • accepting duplicate aliases, matching only a term's first occurrence, or restoring substring aliases.
  • uv run pytest --ignore=tests/test_skill_examples.py: 288 passed under Python 3.15.2 and under 3.11.15 (macOS), at this head after the rebase. The stress and mutation runs above ran there too.
  • uv run python scripts/build_manifest.py --check passes under Python 3.11 and 3.15.

6. Limits this PR leaves

  • Repo names that are common words count wherever the word stands alone: strategy, marriage, newsletters, atlas, pulse, mortality. Reference-validator's "News articles/newsletters" now tags the newsletters repo. The old function had the same problem.
  • Skills and agents that share a repo's name. "See the policyengine-app skill" tags the legacy policyengine-app repo, though that skill covers app-v2. A mention of the calibration-diagnostics agent tags the calibration-diagnostics repo.
  • Research posts. A policyengine.org/us/research/... URL tags policyengine-app-v2, which serves the posts, even when the artifact only reads them.
  • v2.api.policyengine.org, the API v2 alpha, has no alias.

Other PRs that touch the manifest

Rebase onto main and regenerate with uv run python scripts/build_manifest.py rather than hand-merging the manifest.

axiom: n/a: tooling (dashboard manifest builder), no policy change.

🤖 Generated with Claude Code

infer_target_repos matched aliases as case-insensitive substrings, so
"SSI" in "missing", "PIP" in "pipeline" and "GIS" in "registry" tagged
the country models. It matched a repo name inside a longer token, so
policyengine-app-v2 tagged policyengine-app and the skill name
policyengine-model-development tagged policyengine-model. And it read
only the first 6,000 characters of a body.

Now a repo name, its import name or an alias counts only where it stands
as a whole token, and at each position the longest term wins. A term in
capitals (PIP, CRM) counts only in capitals; every other term counts in
any case, as GitHub resolves repo names. The matcher reads the whole
body. The web-app aliases point at policyengine-app-v2,
api.policyengine.org points at policyengine-api, and "frontend" no
longer counts. The registry owner rule treats the country-models
category as country-model work, so two agents that lost false links keep
their owner.

target_repos links: 267 -> 222 (+62 / -107) across 69 artifacts,
against main at 8315e46.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis reopened this Nov 11, 2026
@MaxGhenis
MaxGhenis changed the base branch from manifest-pyyaml to main November 11, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant