Repository navigation
Fix category guidance for Security finding Triage - #2024
Draft
JacobStephens2 wants to merge 2 commits into
Draft
JacobStephens2 wants to merge 2 commits into
JacobStephens2 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Existing-issue Triage could copy a Security finding's factory marker into its recommended category, causing publication to fail and leave
agent-failed. Supply actual repository category choices using publication's existing exclusions, and explain that a valid recommendation is required even when an existing category will be preserved.Closes #2018
Seams under test: the
thirdshift triageCLI through fake GitHub and Harness adapters, including the session prompt, publication labels/comments, and recommendation-only writes.Evidence
Execution evidence
Exact regression command:
cargo test --test triage_issue security_finding_triage_uses_repository_categories_instead_of_its_factory_marker -- --exact --nocaptureBefore: exit 101; the CLI exited 1 with
the Triage category must be a repository category label, not a triage state or another factory label. Labels were["security-finding", "ready-for-human", "agent-failed"]; comments werenull. Result:FAILED. 0 passed; 1 failed.After: exit 0;
ok. 1 passed; 0 failed. The regression exercises findings both with and without an existing category, in publishing and--recommend-onlymodes. Publishing posts exactly one comment with no failure label; recommendation-only leaves fake GitHub state unchanged.Cause confirmed: before changing production code, changing only the simulated Harness response from
security-findingtobugpassed all four combinations. The missing prompt guidance led to an invalid recommendation; the category validator correctly rejected it. The deterministic Harness regression checks this handoff and publication, rather than depending on a live model's response.cargo test --test triage_issue --test triage_spec_regressions: 14 passed, two subprocess entry points intentionally ignored. Coverage includes actual custom category names with JSON escaping, all factory-label exclusions, rejecting invalid categories even when an existing category will be preserved, and private investigation/questionnaire/failure text staying out of email.UPDATE_PROMPTS=1 cargo test prompts_page, thencargo test prompts_page: 14 prompt tests passed in each run. Manual regenerated withnpm run docs --prefix site.Final validation:
NEXTEST_TEST_THREADS=8 scripts/affected-tests mainis running on f8fd91b. The previous version passed all 68 site checks before its superseded Rust validation was interrupted for the review fix.Visual evidence
Paired captures use the same open sections and scroll anchors. The prompt and documentation describe category choices; they contain no private finding write-up.
Existing-issue prompt catalogue
1440×900 — Existing-issue prompt catalogue
390×844 — Existing-issue prompt catalogue
User manual
1440×900 — User manual
390×844 — User manual
Architecture: Triage run
1440×900 — Architecture: Triage run
390×844 — Architecture: Triage run
How it works: Triage
1440×900 — How it works: Triage
390×844 — How it works: Triage
Home: Triage workflow
1440×900 — Home: Triage workflow
390×844 — Home: Triage workflow
Review coverage
Standards: one finding raised and resolved; all 12 changed files inspected. Spec: zero initial findings, with the small follow-up under review. No generated-output exclusions or unread changed files.
Unaddressed findings
Standards
None. The duplicated instruction finding was addressed in f8fd91b and confirmed resolved by its reviewer.
Spec
None.
Merge Danger
Door: two-way
Revert restores the previous prompt guidance; no migration or stored-data change.
Blast Radius: Triage
Existing-issue prompts gain category choices. Report publication uses the same category exclusions through the shared helper.