[PROB-105] The SPEC validator passed empty templates and blocked real specs - #472
Merged
Merged
Conversation
…real specs Measured on 0.36.0 before writing any code, because issue #450's premise did not survive contact with this repository. An untouched SPEC template — body still `{METHOD} /v1/{resource}` — validated with 0 errors and 0 warnings, and once any evidence pack was linked it activated at R_eff 1.00. A complete behavioural spec with two requirements and two GIVEN/WHEN/THEN scenarios failed a MUST rule and could not activate at all. The validator waved through a document saying nothing and blocked one carrying a full test oracle. Two independent causes. check_stub, the gate whose whole job is catching unfilled templates, knows twelve phrases and all twelve are PRD prose ("Что мы строим и почему это важно", "[Actor] can [capability]"). None appears in the SPEC template. It also counted placeholders but capped that at +1 against a threshold of 3, so fifteen unfilled slots weighed the same as one. The count now scales, with the threshold taken from the corpus rather than taste: SPEC template 15 placeholders, PRD template 5, the six real SPECs 0-3. A line-count test was measured and rejected — the untouched template has 25 non-empty lines under `## API Contracts`, because placeholder JSON is still lines. spec-contracts accepted only API Contracts / Data Models / Contracts, all structural, so `## Requirements` + `#### Scenario` had no route through the MUST. The kernel was mandating one methodology's shape and rejecting the other — the reverse of what #450 reports. It now accepts a behavioural contract too. The rule still demands a contract; it stopped demanding one particular form. Also: the stub gate's remediation told every kind to "Fill MUST sections (Problem, Goals, FR)", meaningless for a SPEC or an ADR. The SPEC path was unreachable until now so the wrong advice was never printed; it names the sections of the kind in hand. New rules. spec-requirement-has-scenario (Should) — issue #450 narrowed to internal consistency: a spec that opens a `### Requirement` must close it with a `#### Scenario`. The blanket form was measured first and rejected — none of the six SPECs here uses those headings, so it would have fired on 6 of 6, none defective. Conditional means silent on a structural spec, firing only on a half-authored behavioural one. prd-nfr-exist and prd-nfr-measurable (both Should) — issue #449. PRD carried 24 validator rules and none for non-functional requirements, while extract_nfr_section already existed, called from exactly one place: the tech-leakage check. The subjective-adjective blacklist reads like a list of NFRs — scalable, robust, efficient, responsive, fast — and was only ever applied to the FR section. Across the 69 PRDs here: 2 hits in FR (checked), 15 in NFR (not). All 15 sit inside the template's own `<!-- BAD: "System should be fast and responsive" -->` guidance, so the rule strips non-prose first. Without that it produces 15 findings out of 15 false, closable only by deleting the template's instructions. Verified: 0 findings across all 69 PRDs, while a hand-written "NFR-001: The export is fast and the service is robust" still produces three with line numbers. Should, not Must, for all three: 30 of 69 PRDs have no NFR section, and turning them red at once is how a rule gets ignored rather than obeyed. Four rule-count tests failed and were updated rather than silenced. They sum named groups with comments instead of asserting a bare number, so updating one means explaining where the new term came from. rules_for_spec_returns_base_plus_3 was renamed to _plus_4 rather than edited in place — a name carrying a number goes stale silently. Nine new tests. Two mutation checks: remove the comment strip and nfr_adjectives_inside_html_comments_are_not_flagged fails; remove the placeholder scaling and the untouched template is back to PASS 0/0. Tests were run per crate, not with --workspace: a full workspace build does not fit in the free space on this machine and twice reported "run did not happen" with ld errno=28. core 2239 passed / 3 failed (17 binaries), cli 807/0 (58), mcp 274/0 (19) — 94 binaries total, matching what --workspace produces. The 3 are #454 in git::tests, which this diff does not touch. Refs: PROB-105, #449, #450
Context is large enough that this needs to be readable from a fresh start. Everything the document points at is committed, which is the correction to the last handoff written in this repo: that one named a scratchpad path that did not exist here, because the files were in a session temp directory tied to a session id. It handed off work that was about to evaporate while reporting it ready — the same defect class the release it accompanied was about. Carries the measurements rather than the conclusions: the placeholder counts that set the stub threshold (template 15, PRD template 5, real specs 0-3), the 2-vs-15 FR/NFR adjective split and why all 15 were false, the per-crate test results and why --workspace does not fit on this machine, and the two mutation checks. Also records what NOT to re-litigate — the rejected shape whitelist, the blanket #450 rule, Must instead of Should — and flags the uncommitted Cargo.toml and .cargo/ work belonging to a second session in this worktree, so nobody sweeps it into a commit. Refs: PROB-105, #449, #450
…l template The SPEC template is API-first and is the only signal an author has about what a spec looks like. Now that the validator accepts a behavioural contract, the template says so. The existing stub test pins a hand-abridged copy of the template, so it cannot notice the real file changing -- adding this comment proved the gap. A new test reads the file the binary actually ships (include_str!, same path as template::engine) and asserts two things: it still reads as a stub, and the guidance comment does not register as a heading. The comment names `## Requirements` inside backticks; if section_exists ever started matching those, an empty template would satisfy spec-contracts through its own comment. Both assertions mutation-checked: MANY_PLACEHOLDERS = 100 kills the first, appending a real `## Requirements` heading kills the second. Also records the #449 / #450 / PROB-105 work in CHANGELOG, and corrects the handoff's stale "not yet pushed" status. Refs: prob-105 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 4 mutation checks Records what was measured rather than what was reported: the before/after validator behaviour on the template and on a behavioural spec, the corpus numbers that set the placeholder threshold (SPEC template 15, PRD template 5, real SPECs 0-3), and the 2-vs-15 FR/NFR adjective split with all 15 inside the template's own BAD-example guidance. Four mutation checks, because a green test on broken code proves nothing. Also states this pack's limits: it certifies behaviour on this repo's corpus only, the FR-side rule still carries the same latent bug, and the 4 git::tests failures are the known #454 flake, not health. PROB-105 R_eff 0.00 -> 1.00; both artifacts activated. Refs: prob-105, EVID-171 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 8, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #450. Closes #449. Refs prob-105.
The finding that reframed both issues
Issue #450 asks for a rule enforcing "every
### Requirementhas a#### Scenario",on the grounds that the invariant is declared in prose and checked nowhere.
Measured before implementing, the premise does not hold here — and the truth is
worse than the report.
PASS — 0 error(s), 0 warning(s)→ activates at R_eff 1.00activaterefusesx [MUST] spec-contracts→ FAIL, cannot activatespec-summary+spec-contractsCore did not lack a rule about scenarios. It had a MUST pointing against them,
while waving through documents that said nothing. That is why the marketplace TDD
flow grew its own gate (
tdd-plannerHARD RULE 1): core actively rejected theshape TDD needs.
Three fixes, none of which teaches the kernel a methodology
check_stubcan see a SPEC. Its twelve phrase markers are all PRD prose,and the placeholder signal was capped at
+1against a threshold of 3 — fifteenunfilled slots weighed the same as one. The count scales now.
Threshold from the corpus, not taste: SPEC template 15 placeholders, PRD
template 5, the six real SPECs 0–3. A line-count test was measured and
rejected — the untouched template has 25 non-empty lines under
## API Contracts,because placeholder JSON is still lines.
spec-contractsaccepts a behavioural contract. It still demands acontract; it stopped demanding one particular form.
## Requirements,## Contract,## Behavioral Contractjoin the structural headings.spec-requirement-has-scenario(Should) — validate: нет правила «каждый Requirement в SPEC имеет Scenario» — инвариант держит только TDD-потребитель #450, conditional. Silent on astructural spec, fires only on a half-authored behavioural one. The blanket form
was measured: it fires on 6 of 6 SPECs here, none defective.
Plus: the stub gate's remediation told every kind to "Fill MUST sections (Problem,
Goals, FR)". It now names the sections of the kind in hand.
#449 — NFR rules, and the trap in them
prd-nfr-existandprd-nfr-measurable, both Should.extract_nfr_sectionalreadyexisted, called from exactly one place — the tech-leakage check — so the validator
could find the section and asked nothing about its contents.
The subjective-adjective blacklist reads like a list of NFRs (
scalable,robust,efficient,responsive,fast) and was only ever applied to the FR section.Across the 69 PRDs: 2 hits in FR (checked), 15 in NFR (not).
All 15 sit inside the template's own guidance —
<!-- BAD: "System should be fast and responsive" -->. A rule flagging them wouldbe unclosable: the only fix is deleting the instructions. The rule strips non-prose
first. Verified 0 findings across all 69 PRDs, while hand-written vague prose
still produces findings with line numbers.
Template + a test that reads it
The SPEC template is API-first and is the only signal an author has about what a
spec looks like. It now carries a short comment saying both contract shapes are
legitimate.
The existing stub test pins a hand-abridged copy of the template, so it cannot
notice the real file changing. A new test reads the file the binary actually ships
(
include_str!, same path astemplate::engine) and asserts it still reads as astub, and that the new guidance comment does not register as a heading — it names
## Requirementsinside backticks, and ifsection_existsever started matchingthose, an empty template would satisfy
spec-contractsthrough its own comment.Verification
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p forgeplan-core --features test-helpers --no-fail-fastcargo test -p forgeplan --no-fail-fastcargo test -p forgeplan-mcp --no-fail-fast3320 passed, 4 failed, 94 test binaries — the binary count matches what
--workspaceproduces, so the per-crate split skipped nothing. The core failuresare all in
git::tests(#454); they pass serially (51/51,--test-threads=1) and this diff does not touch that module.forgeplan-coreneeds
--features test-helpersbecause the*_for_testhelpers are gated andnormally arrive through
forgeplan-mcp's dev-dependency.Mutation checks — each fix reverted, the matching test had to fail:
nfr_adjectives_inside_html_comments_are_not_flaggedFAILEDPASS — 0 error(s), 0 warning(s)MANY_PLACEHOLDERS = 100→ the new shipped-template test FAILED## Requirementsheading to the template → the same test FAILED on its second assertionFour pre-existing rule-count tests failed and were updated, not silenced. They
sum named groups with comments rather than asserting a bare number, so updating one
means explaining where the new term came from.
rules_for_spec_returns_base_plus_3was renamed to
_plus_4rather than edited in place — a test name carrying anumber goes stale silently.
Not done
check_measurability_adjectives(FR side) still lacks the non-prose strip.Latent, not firing, deliberately out of scope.
spec; the fix is re-filing it, not loosening a rule.
Full reasoning, rejected alternatives, and environment notes:
docs/handoff/spec-validator-and-nfr-rules.md.Evidence: EVID-171 (
verdict: supports, CL3,evidence_type: test, withgit provenance), linked
informsto PROB-105 — R_eff 0.00 → 1.00.