Skip to content

Fix category guidance for Security finding Triage - #2024

Draft
JacobStephens2 wants to merge 2 commits into
mainfrom
issue-2018
Draft

JacobStephens2 wants to merge 2 commits into
mainfrom
issue-2018

Conversation

@JacobStephens2

Copy link
Copy Markdown
Owner

Summary

Existing-issue Triage could copy a Security finding's factory marker into its recommended category, causing publication to fail and leave agent-failed. Supply actual repository category choices using publication's existing exclusions, and explain that a valid recommendation is required even when an existing category will be preserved.

existing-issue Triage
  repository labels → exclude factory markers and comma-containing names
  session prompt → choose a valid repository category
  publication → validate again, preserve the finding marker and any existing category

Closes #2018

Seams under test: the thirdshift triage CLI through fake GitHub and Harness adapters, including the session prompt, publication labels/comments, and recommendation-only writes.

Evidence

Execution evidence

Exact regression command:

cargo test --test triage_issue security_finding_triage_uses_repository_categories_instead_of_its_factory_marker -- --exact --nocapture
  • Before: exit 101; the CLI exited 1 with the Triage category must be a repository category label, not a triage state or another factory label. Labels were ["security-finding", "ready-for-human", "agent-failed"]; comments were null. Result: FAILED. 0 passed; 1 failed.

  • After: exit 0; ok. 1 passed; 0 failed. The regression exercises findings both with and without an existing category, in publishing and --recommend-only modes. Publishing posts exactly one comment with no failure label; recommendation-only leaves fake GitHub state unchanged.

  • Cause confirmed: before changing production code, changing only the simulated Harness response from security-finding to bug passed all four combinations. The missing prompt guidance led to an invalid recommendation; the category validator correctly rejected it. The deterministic Harness regression checks this handoff and publication, rather than depending on a live model's response.

  • cargo test --test triage_issue --test triage_spec_regressions: 14 passed, two subprocess entry points intentionally ignored. Coverage includes actual custom category names with JSON escaping, all factory-label exclusions, rejecting invalid categories even when an existing category will be preserved, and private investigation/questionnaire/failure text staying out of email.

  • UPDATE_PROMPTS=1 cargo test prompts_page, then cargo test prompts_page: 14 prompt tests passed in each run. Manual regenerated with npm run docs --prefix site.

  • Final validation: NEXTEST_TEST_THREADS=8 scripts/affected-tests main is running on f8fd91b. The previous version passed all 68 site checks before its superseded Rust validation was interrupted for the review fix.

Visual evidence

Paired captures use the same open sections and scroll anchors. The prompt and documentation describe category choices; they contain no private finding write-up.

Existing-issue prompt catalogue

1440×900 — Existing-issue prompt catalogue

Before

After

390×844 — Existing-issue prompt catalogue

Before

After

User manual

1440×900 — User manual

Before

After

390×844 — User manual

Before

After

Architecture: Triage run

1440×900 — Architecture: Triage run

Before

After

390×844 — Architecture: Triage run

Before

After

How it works: Triage

1440×900 — How it works: Triage

Before

After

390×844 — How it works: Triage

Before

After

Home: Triage workflow

1440×900 — Home: Triage workflow

Before

After

390×844 — Home: Triage workflow

Before

After

Review coverage

Standards: one finding raised and resolved; all 12 changed files inspected. Spec: zero initial findings, with the small follow-up under review. No generated-output exclusions or unread changed files.

Unaddressed findings

Standards

None. The duplicated instruction finding was addressed in f8fd91b and confirmed resolved by its reviewer.

Spec

None.

Merge Danger

Door: two-way

Revert restores the previous prompt guidance; no migration or stored-data change.

Blast Radius: Triage

Existing-issue prompts gain category choices. Report publication uses the same category exclusions through the shared helper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A Triage of an existing Security finding fails: security-finding is rejected as its category

1 participant