Skip to content

pre-pr-evidence-check reads test-scenario IDs quoted in commit prose as real artifact refs #482

Description

@explosivebit

Found 2026-09-08 while opening PR #481 (fix/lifecycle-reason-never-reaches-filedev).

What happened

gh pr create was blocked: .claude/hooks/pre-pr-evidence-check.sh reported PRD-200, PRD-201, RFC-022 as referenced-but-unevidenced. None of them exist:

$ forgeplan get PRD-200
Error: Artifact 'PRD-200' not found
$ forgeplan get RFC-022
Error: Artifact 'RFC-022' not found

Root cause

The hook scans git log -20 --pretty='%s%n%b' for (PRD|RFC|ADR|EPIC|SPEC|PROB|EVID|NOTE)-[0-9]+ (pre-pr-evidence-check.sh:124). One commit within that 20-commit window on dev documents a mutation-testing walkthrough in prose — a scenario built around test-fixture IDs PRD-200 / PRD-201 / EVID-200, plus a separate reference to RFC-022 describing a round-trip measurement. Both are illustrative IDs in a commit message, not artifacts. The hook cannot distinguish "this commit message is about an artifact ID" from "this commit message documents work on an artifact ID" — it has no way to.

Why this matters beyond one PR

The window is git log -20 on whatever branch is opening the PR, not on the commits the PR actually introduces. As dev accumulates commits, any branch based on it inherits the same 20-commit tail until enough real work pushes the offending commit out of range. Every PR opened from dev in that window hits the same false block, for a reason that has nothing to do with its own content.

What I did

Bypassed with FORGEPLAN_SKIP_EVIDENCE=1, documented in the PR body per the hook's own printed instructions and docs/methodology/EVIDENCE-PROTOCOL.md. That is the correct escape hatch for this case, but it means the gate silently does nothing useful for any PR opened in this window — worth knowing, not blocking.

Possible fixes (not evaluated in depth)

  • Scan git log <base>..HEAD (the actual PR diff) instead of a fixed -20 window on whatever branch. This is probably the real fix — it also stops the false-positive from depending on how many commits happen to sit between the branch point and dev's tip.
  • Or: only match artifact IDs that also appear in Refs: trailers, not anywhere in free prose.

Refs: PROB-105, #472, #481

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions