Skip to content

MEASURED, NOT WORTH DOING: check_stub false positives (HTML-comment guidance, self-reference) #476

Description

@explosivebit

This issue is closed on arrival. It records two things that look like defects, were measured, and turned out not to be worth fixing. It exists so the next person who notices the same smell finds the numbers instead of re-deriving them.

Measured 2026-09-08 on dev at 551ddb9, across all 465 files in .forgeplan/.


1. check_stub counts markers inside the template's own HTML comments

The smell. check_stub scans for phrase markers without stripping HTML comments. The PRD template contains <!-- FORMAT: "[Actor] can [capability]" -->, so the rule counts the instruction that tells you what not to leave unfilled. This is the same latent class as the NFR-adjective bug fixed in #472 — a rule flagging the guidance that explains the rule.

The measurement. Scored every artifact twice — as-is, and with HTML comments stripped:

artifacts scanned:            465
flagged as stub now:           10
flagged if comments stripped:  10
verdict would change on:        0

Zero verdicts change. Every flagged artifact is flagged for reasons beyond the comments. The bug is real in the code and has no effect on the corpus.

Why not fix it anyway. A fix here is unverifiable against this corpus — the test would assert behaviour identical to today's. Fixing it would produce a green diff that proves nothing, which is exactly the failure class #472 was about. Revisit only if the measurement changes (see the trigger below).


2. PROB-105 trips the very rule it describes

forgeplan validate PROB-105 warns no-stub-content: Body appears to be unfilled template (3 markers found).

The cause: the problem report quotes the marker list verbatim while explaining that the marker list is the defect —

Its twelve PHRASE_MARKERS are all PRD prose — "Что мы строим и почему это важно", "What we are building and why", [Actor] can [capability].

A document about the detector looks like what the detector hunts.

Why not fix. Both available fixes are worse than the warning:

  • loosening the detector to ignore quoted markers weakens a rule that works;
  • rewriting PROB-105 to describe the markers instead of quoting them removes exactly the detail that makes the report verifiable.

It is a Should, not a block, on one self-referential document. Left as is.


Corpus state, for reference

10 of 465 artifacts trip no-stub-content:

Count Status What
8 deprecated / superseded pre-fill-era PRDs with literal [Actor] can [capability] rows — true positives, and history
1 active → now deprecated PROB-105, the self-reference case above
1 draft RFC-012, not examined

Trigger to reopen

Re-run the measurement and reopen if either becomes true:

  • stripping HTML comments would change the verdict on one or more artifacts (i.e. an artifact appears whose only stub markers live in comments), or
  • a second non-self-referential artifact is flagged that a human agrees is not a stub.

"later" is not a trigger. Re-run the script in this issue's history against the corpus at the time.

Refs: #472, prob-105, EVID-171

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    measured-not-plannedChecked with numbers, decided not to do. Closed on arrival; body carries the trigger to reopen.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions