Repository navigation
[Review] grade PR risk, list decisions for a human, flag deletions, check every card before posting - #2243
Conversation
Code metrics gate: PASSEDJudged: the 14 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
Thresholds and sources: |
chaojhou
left a comment
There was a problem hiding this comment.
PR #2243 -- [Review] grade PR risk, list decisions for a human, flag deletions, check every card before posting
What it does: Changes the review-pr skill.
risk.pygrades a PR's changed paths low/medium/high intorisk.txt, withreferences/tiers.mdas its only data.- Cards gain
Risk:,Decisions for a human:(required at medium and high) and an optionalTrim in this PR:list. - Card validation moves from
pr-review-publish.shintocheck-card.sh, shared by CI and local runs. Publish re-grades risk from GitHub's file list, so the review job cannot lower the grade. references/subtract.mdasks which tests and mechanisms need to exist, and SKILL.md is condensed.
First review at 711e857. Since then #2227 (d5f1bde) has landed on main, and this PR now conflicts with it in SKILL.md and fetch.sh (re-review.md and rules.md need the same rebase).
Blocking issues: 3
-
[V5]
coverage (Python 3.10), a required check, fails at this head [verified]
Problem: diff-cover reports.claude/skills/review-pr/risk.py (60.0%): Missing lines 32,67-69,73-79,83-97against the 80% floor (job 122131982627). Those areRisk.__str__,is_test,gradeandmain, the code that does the grading.
Impact: the PR cannot merge while a required check is red.
Action: make the tests' runs of these functions count toward coverage, or add the missing cases, then confirm the diff-coverage job passes. -
[free:stale-requirements-row]
references/tiers.mdgradesrequirements/*, whichmainhas deleted, so the PR's own test fails once merged [verified]
Problem:.claude/skills/review-pr/references/tiers.md:85listsrequirements/*in the high row "dependency pins and gate limits (rule X7)". #2237 (606476a, onmainsince this PR's merge base 9010db3) deletedrequirements/app.inandrequirements/app.lock, so no tracked path matches the pattern.scripts/tests/test_review_pr_risk.py:28-38(test_every_row_of_the_tables_names_a_tracked_file) fails on the merge of this head intomain. CI passed only because it ran on the old base.
Impact: merging turnsmain's test run red. The body's claim that every table row matches a tracked file is false against the branch it merges into.
Action: after rebasing, droprequirements/*from that row, or point it at what now holds the pins (pyproject.tomlalready sits there), and re-run the test on the rebased tree. -
[free:rename-graded-by-new-path] A rename is graded only by its destination, so moving a high-risk file out of a high-risk location lowers the floor [verified]
Problem: fetch.sh's numstat awk takes theb/path ofdiff --git(.claude/skills/review-pr/fetch.sh:117). A pure rename has no---/+++lines, so only the new path reachesfiles.txt.pr-review-publish.sh:44lists--jq '.[].filename'and dropsprevious_filename.risk.pytherefore never sees the source path of a rename. For example,git mv src/hyperloom/orchestrator/loop/coordinator.py src/hyperloom/orchestrator/core/coordinator.py, or moving.github/workflows/lint.ymlunderdocs/, grades medium or low on both graders.check-card.shthen accepts a card that never names the Tier 1 file or the removed CI workflow among the decisions for a human.
Impact: the case the floor exists for, a high-risk path removed or moved, is graded below its risk on both graders.
Action: grade both ends of a rename. Emit the old path in fetch.sh (fromrename fromor thea/side), and use--jq '.[] | .filename, (.previous_filename // empty)'in publish. Add a rename case to the tests.
Checked:
- risk.py, tiers.md (53 rules) against
git ls-treeat head and on origin/main, and fetch.sh's numstat awk. - check-card.sh against the checks removed from pr-review-publish.sh (moved unchanged apart from comments, plus the new Skill/Model checks).
- pr-review.yml wiring: an issue_comment trigger, and publish runs main's scripts with the write token, not PR-head code.
- A sentence-by-sentence audit of SKILL.md at 9010db3 against head and the references it links: no instruction dropped; factual counts only.
git merge-treeagainst #2227 and againstmainafter d5f1bde.
Ran:
- risk.py on the last 20 merges and on edge inputs (CRLF, empty, overlapping rows, tests-only, renames).
- check-card.sh under Linux bash on accepted and rejected cards: risk floor, raise marker, decision coverage including directories, CRLF.
- The PR's risk test on the merge tree with origin/main: the stale-row failure.
- Two independent skeptics per finding; two candidates were dropped (#2227 overlap as non-blocking, a Git-Bash-only hang).
Skill: 56bad557b6ff | Model: claude-opus-5-5 | Base: 9010db3 | Head: 711e857
711e857 to
10f10a2
Compare
chaojhou
left a comment
There was a problem hiding this comment.
PR #2243 -- [Review] grade PR risk, list decisions for a human, flag deletions, check every card before posting
What it does: Changes the review-pr skill and its CI publisher.
risk.pygrades a PR's changed paths, including a rename's old path, low, medium or high intorisk.txt. Its only data isreferences/tiers.md: Tier 1 is high, Tier 2 medium, and a new risk table grades the rest.- Cards gain
Risk:,Decisions for a human:(required at medium and high) and an optionalTrim in this PR:list. - Card validation moves from
pr-review-publish.shintocheck-card.sh, which CI and local runs share. The publisher re-grades risk from GitHub's file list. references/subtract.mdasks which tests and mechanisms need to exist, and SKILL.md is condensed.
Re-reviewed at 10f10a2, rebased onto 0198aa7. All three earlier findings are fixed. Two new ones come from the risk table.
Blocking issues: 2
-
[free:risk-table-row-stale-on-main] The PR's own tracked-file test fails once it is merged, because a Tier 2 row names a file
mainhas since moved [verified]
Problem:.claude/skills/review-pr/references/tiers.md:26nameskernelforge/config.py. #2282 (05e67da, onmainsince this PR's merge base) moved it tosrc/kernelforge/config/__init__.py, and that move's script did not rewrite.claude/.scripts/tests/test_review_pr_risk.py::test_every_row_of_the_tables_names_a_tracked_file, which this PR adds, fails on the trial merge with origin/main 3abe484:assert ['*/kernelforge/config.py'] == []. CI passed only because it ran on the older base. This is the same shape as the earlierrequirements/*finding.
Impact: merging turnsmain's test run red, and changes tosrc/kernelforge/config/grade only through thesrc/*row.
Action: rebase onto currentmain, change the row tokernelforge/config/, and re-run the test on the rebased tree. -
[free:shipped-skills-graded-low] The risk table grades the demo skills the wheel installs as "prose and metadata nothing executes" [verified]
Problem: the low rowexamples/*(.claude/skills/review-pr/references/tiers.md:93) covers four files thatpyproject.toml:229-243installs as agent skills under.agents/,.claude/and.cursor/skills, chained from/hyperloom-setup:examples/hyperloom-qwen3-8b-3h/SKILL.mdexamples/hyperloom-qwen3-14b-fp8-12h/SKILL.mdexamples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.mdexamples/hyperloom-custom-advanced/SKILL.md
The same table grades that class medium:
src/*"the prompts and skills agents load at runtime" (line 92), andexamples/skills/*(line 91).risk.pyonexamples/hyperloom-qwen3-8b-3h/SKILL.mdprintsrisk: lowwith "prose and metadata nothing executes".
Impact: a PR that changes the instructions an installed agent skill follows gets a low floor.check-card.shand the publisher then accept a card with noDecisions for a human:, which is the case the floor exists to prevent, and the row's own stated reason is false for these files.
Action: grade the shipped example skills medium, for exampleexamples/*/SKILL.mdin a medium row above theexamples/*low row. Add one of them to the per-path cases intest_review_pr_risk.py.
Checked:
- Earlier findings:
- V5 fixed:
coverage (Python 3.10)and(3.11)pass at this head. - free:stale-requirements-row fixed:
requirements/*is gone fromtiers.md. - free:rename-graded-by-new-path fixed:
fetch.sh:133addsrename frompaths, andpr-review-publish.sh:44lists.previous_filename.
- V5 fixed:
- Every check
main'spr-review-publish.shmakes is still made, bycheck-card.shor the publisher, and none is weaker. The publish job runs only default-branch scripts, includingrisk.pyandtiers.mdfrom its sparse checkout, and grades risk from GitHub's paginated file list. - SKILL.md and
re-review.mdonmaincompared sentence by sentence with the merged text: no instruction is dropped or changed in meaning. SKILL.md is 305 lines against its 365 budget. - Every artifact and argument the new text names matches what the scripts produce and read.
rules.mdchanges one prose sentence of V2, and names no new command.
Ran:
- The skill as it is on
main(272461950d5f), against this PR. test_review_pr_risk.pyin a worktree of the trial merge with origin/main 3abe484: 1 failed (finding 1), 5 passed.risk.pyon edge inputs (empty, CRLF, tests-only, workflows,pyproject.toml, renames, docs-only), on all tracked files and on the last 15 commits onmain.check-card.shunder Git Bash on hand-made cards: risk floor, raise marker, decision coverage, trim list.- git merge-tree against origin/main: clean.
- CI at 10f10a2: green except readthedocs, which this PR's paths cannot affect.
- Three area readers, with two skeptics on each finding: two candidates were dropped.
- SKIPPED: the shell-script tests (
test_pr_review_publish.py,test_review_pr_check_card.py,test_review_pr_fetch.py) locally, because the harness passes Windows paths to bash on this host (exit 127, onmaintoo). They pass in CI at the head, and no commit onmainsince the merge base touches those scripts.
Skill: 272461950d5f | Model: claude-opus-5-5 | Prior: V5 fixed, free:stale-requirements-row fixed, free:rename-graded-by-new-path fixed | Base: 0198aa7 | Head: 10f10a2
…st what the author can delete A fix that answers each finding with one more guard grows the tree without converging, and the review never asked whether an added test pins anything another test does not, or whether the mechanism a fix patches should exist at all. references/subtract.md, read from Step 5's test-falsifies and Step 6: an added test that repeats another, pins plumbing, or re-pins one contract at each layer is a Contract violation under "Test the contract, not the plumbing"; on a fix, deleting or changing existing code comes before adding; a guard added where a deletion would do is a finding. Dead ends that were already there, inside what the diff touches, go on the card as an optional "Trim in this PR:" list that only ever removes code and never counts as blocking. Action lines prefer deletion. pr-review-publish.sh refuses a malformed trim list (empty, more than five lines, a line without a place or evidence, or two lists); the CI contract allows the list. Co-authored-by: Cursor <cursoragent@cursor.com>
SKILL.md goes from 365 lines and 3709 words to 286 lines and about 3180, re-review.md and subtract.md by about a tenth, with every requirement, artifact, format, PR citation and prohibition kept: an independent sentence-by-sentence comparison of old against new left only wording changes. The gain comes from merging what was said twice: "Links out" joins the output contract, the evidence threshold, the Scenario and Layer conditions and the CI pointer join the card contract's list, Step 6's lines become one list, and the sub-agent checks one paragraph. Step 1b no longer hardcodes the rule count, which every added rule made stale. No behaviour changes: a docs-only change to the skill's instructions. Co-authored-by: Cursor <cursoragent@cursor.com>
…st decide, and check every card before it is posted A human approving a PR after a clean card had nothing on the card telling them what needed their judgement, and cards posted from local runs -- nearly all of them -- reached the PR without any of the checks pr-review-publish.sh applies to the bot's. risk.py grades the changed paths low, medium or high, with references/tiers.md as its only data: a Tier 1 backbone file is high and a Tier 2 one medium; the new risk table there gives the gates, the reviewer's own files, CI, pins, install scripts and baseline configs high, the authoring contract, runtime-read docs and src/ medium, prose low; an unmatched path is medium, a test low. fetch.sh writes risk.txt. On the last 400 merged PRs it grades 177 high, 219 medium and 4 low. The card gains a Risk line, which may only rise above risk.txt and then says "(raised from <level>)", and, unless the risk is low, a "Decisions for a human:" list naming every operator- observable change and every high-risk path; a medium card with nothing to decide says so. check-card.sh holds a card to the whole card contract -- artifacts, re-review settlement, heading, risk and decisions, findings, trim list, Checked line -- and is what both CI and a local run call before posting. pr-review-publish.sh keeps only what is CI's own: the review job's result, a clean skill and a known model matching skill.txt, the PR still open at the reviewed head. It grades the risk itself from the PR's files on GitHub, so nothing the review job wrote can lower the floor. The publish job's sparse checkout adds the skill directory for that. Locally the skill now posts card.md as check-card.sh passed it, and says why it never approves: an approval from a local run is the operator's code-owner approval of a model's verdict. The bot still never approves, whatever the risk: this is shadow mode, the risk only shows the human reviewer where to look. Observable effect: every card carries "Risk:" and, for medium and high risk, "Decisions for a human:"; the bot posts "no verdict applied" for a card without them, with a Risk below the paths' grade, or with a high-risk path no decision names. Co-authored-by: Cursor <cursoragent@cursor.com>
Both graders saw only a rename's destination: fetch.sh fed risk.py files.txt, built from the ` b/` side of each diff header, and pr-review-publish.sh listed `.filename` alone. Moving a file out of a high-risk location, a baseline config into another directory or a workflow out of .github/, graded by where it landed. A deletion already grades by the path it removes; a rename removes one too. fetch.sh now grades files.txt plus the diff's `rename from` paths, and publish lists each file's `previous_filename` beside its name, so a renamed high-risk source sets the floor and must be named among the decisions for a human. files.txt, the input to rule selection, is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
The risk table graded all of examples/ low, as prose and metadata nothing executes, yet pyproject.toml installs four of its SKILL.md files as agent skills under .agents/, .claude/ and .cursor/skills, chained from /hyperloom-setup. A PR changing the instructions an installed skill follows got a low floor, and with it no "Decisions for a human:". examples/*/SKILL.md joins the medium row beside examples/skills/*, which already graded the federated skill that way; the rest of examples/ stays low. Co-authored-by: Cursor <cursoragent@cursor.com>
10f10a2 to
70e4fdd
Compare
Superseded by the re-review at 70e4fdd: both findings fixed.
PR #2243 -- [Review] grade PR risk, list decisions for a human, flag deletions, check every card before postingWhat it does: Changes the review-pr skill and its CI publisher.
Re-reviewed at 70e4fdd, rebased onto 356c7a2. Both findings from the last card are fixed. Blocking issues: none Checked:
Ran:
Skill: 272461950d5f | Model: claude-opus-5-5 | Prior: free:risk-table-row-stale-on-main fixed, free:shipped-skills-graded-low fixed | Base: 356c7a2 | Head: 70e4fdd LGTM |
chaojhou
left a comment
There was a problem hiding this comment.
Approved per the review above (user consent 2026-10-29).
main 20d8b20 into dev e81281d. No file conflicted. main's six commits since the last sync: - 20d8b20 bring ten functions to at most five parameters (#2312) - e32bd47 kernelforge: mirror the source tree in tests, split the loop and orchestration tests (#2311) - 83811d5 orchestrator: drop test-only knobs and unread fields in measurement, loop and GEAK (#2210) - 8d68eef [Review] grade PR risk, list decisions for a human, flag deletions (#2243) - c9b2d43 install: hold click below 8.3 only for Ray 2.44.1's Sentinel import failure (#2309) - f782665 code-metrics: count *args and **kwargs in the parameter-count dimension (#2310) Co-authored-by: Cursor <cursoragent@cursor.com>
Description: five standalone commits on top of review-pr: block on a premise that does not reproduce, a failure that survives, or a fix that patches the symptom #2227; please Rebase and merge.
references/subtract.md).Trim in this PR:list.risk.pygrades the changed paths low, medium or high intorisk.txt. Its only data isreferences/tiers.md: Tier 1 is high, Tier 2 medium, and a new risk table grades the rest.Risk:may only rise above that grade, and then says(raised from …).Decisions for a human:, naming every operator-visible change and every high-risk path.check-card.shholds a card to the whole card contract. CI and local runs both call it.pr-review-publish.shkeeps only the CI provenance checks. It re-grades risk from the PR's files on GitHub, so the review job cannot lower the grade.card.mdonly after the check passes, and never approve.rename fromin the diff, andprevious_filenamefrom GitHub. Moving a file out of a high-risk location no longer lowers the floor.examples/*/SKILL.md, whichpyproject.tomlinstalls as agent skills, is graded medium instead of low.On the last 400 merged PRs,
risk.pygrades 177 high, 219 medium and 4 low.Linked issue(s): none
Tests: added/updated? commands run? (what to test; if a test was replaced, where its assertions live now):
test_review_pr_risk.py: every table row matches a tracked file; per-path grades, including an installed example skill; a PR takes the grade of its riskiest path. It runsrisk.pyin-process, so diff coverage counts it (100%).test_review_pr_check_card.py: the risk floor, the raise marker, and decision coverage, including directories.test_pr_review_publish.py: the trim list, and risk graded from GitHub rather than fromrisk.txt.test_review_pr_fetch.py:risk.txt, including a renamed file.pytest scripts/tests: 769 passed, 0 failed.Size/complexity triggers crossed: split, or why the shape is right: none. 0 production Python lines (size/XS). New functions are at most cyclomatic 7 and cognitive 11.
If this simplifies or refactors: which mechanism was removed, which contract was preserved, and the tests that show it: card validation moves verbatim from
pr-review-publish.shintocheck-card.sh, so CI and local runs share one check. The existing publish tests still pass with their messages unchanged.Observable effect: what an operator will now see, and what the old behaviour cost them (if none, why the change is unobservable):
Risk:and, unless the risk is low,Decisions for a human:. A card may also carryTrim in this PR:.Breaking changes: yes/no (details if yes): no. The bot reviews with the default branch's skill, so its own cards comply once this merges.
PR addresses single concern: yes/no (details if no): no. Five standalone commits, kept together for two reasons. The risk commit needs the line budget the wording commit frees. The last two commits correct the risk commit's input and table.
Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: n/a