Skip to content

Claim provenance is collected but never gated on: a comment-derived claim counts the same as a spec-derived one #116

Description

@raccioly

Summary

testguard claims already distinguishes spec-derived from comment-derived claims, but nothing else in the tool treats them differently — so a claim backed by a written spec and a claim backed by a code comment produce identical green.

Observed

6 claims in <repo>/testguard.claims.json — 0 carry a @claim annotation in source

  FE-PERM-WILDCARD               high     comment   2 faults ...
  FE-PERM-REPLY-GATE             high     spec      1 fault  ...
  FE-SECRET-OTHER-USER-PASSWORD  critical spec      2 faults ...
  FE-INBOX-SYSTEM-MESSAGE-INERT  high     spec      2 faults ...
  FE-INBOX-MONOTONIC             medium   comment   1 fault  ...
  FE-INBOX-UPDATE-NO-BLANK       medium   comment   1 fault  ...

Three spec, three comment. probe reported 9 killed, 0 unproven with no reference to that split, and status did not mention it at all.

Why the distinction is load-bearing

A comment-sourced claim is materially weaker evidence than a spec-sourced one, because a comment is code that doesn't run. It was written by the same author, at the same moment, with the same misunderstanding as the implementation it sits above. If the author misunderstood the requirement, the comment records the misunderstanding faithfully and the claim inherits it.

A spec/doc/bug-sourced claim has an independent origin — it was written before the code, or by someone else, or by production reality. That independence is the entire epistemic value of the claim.

Treating both as "a claim" flattens the strongest signal the tool already collects.

Proposal

  1. Report the split in status and probe, not only in claims:
    proven: 9/9 faults killed across 6 claims
    provenance: 3 spec · 3 comment · 0 bug   ⚠ 3 claims have no source outside the code they guard
    
  2. Allow gating on it. A --min-provenance spec flag (or a config key) lets a team say "comment-only claims don't count toward our gate" without deleting them.
  3. Extend the vocabulary beyond spec / comment. Useful kinds, roughly in descending strength: incident (a real production failure), bug (a fix commit), spec, doc, review (raised by a human reviewer), comment, inferred (scaffold's default when nothing was supplied). incident and bug are the strongest possible provenance and there is currently no way to express them.
  4. Record provenance in the evidence file so it survives into CI artefacts and can be trended. "We converted 4 comment-claims to bug-claims this quarter" is a real maturity signal.
  5. Relatedly: claims reports "0 carry a @claim annotation in source" as a flat fact with no severity and no next action — see the separate issue on annotation drift.

Acceptance

  • The provenance split is visible without running claims specifically.
  • A team can gate on provenance strength.
  • incident / bug provenance is expressible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    claim-qualityWhat a claim is, where it comes from, and whether its evidence is independentfield-reportRaised from running TestGuard on a real codebase

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions