Skip to content

[PROB-105] The SPEC validator passed empty templates and blocked real specs - #472

Merged
explosivebit merged 4 commits into
devfrom
fix/spec-validator-inverted
Sep 8, 2026
Merged

explosivebit merged 4 commits into
devfrom
fix/spec-validator-inverted

Conversation

@explosivebit

Copy link
Copy Markdown
Contributor

Closes #450. Closes #449. Refs prob-105.

The finding that reframed both issues

Issue #450 asks for a rule enforcing "every ### Requirement has a #### Scenario",
on the grounds that the invariant is declared in prose and checked nowhere.

Measured before implementing, the premise does not hold here — and the truth is
worse than the report.

Input Before After
untouched SPEC template PASS — 0 error(s), 0 warning(s) → activates at R_eff 1.00 stub warning; activate refuses
behavioural spec, 2 requirements + 2 GIVEN/WHEN/THEN scenarios x [MUST] spec-contracts → FAIL, cannot activate PASS
SPEC-002 … SPEC-006 pass unchanged
SPEC-001 fails spec-summary + spec-contracts unchanged — it is a document about writing specs, filed as a spec

Core did not lack a rule about scenarios. It had a MUST pointing against them,
while waving through documents that said nothing. That is why the marketplace TDD
flow grew its own gate (tdd-planner HARD RULE 1): core actively rejected the
shape TDD needs.

Three fixes, none of which teaches the kernel a methodology

  1. check_stub can see a SPEC. Its twelve phrase markers are all PRD prose,
    and the placeholder signal was capped at +1 against a threshold of 3 — fifteen
    unfilled slots weighed the same as one. The count scales now.

    Threshold from the corpus, not taste: SPEC template 15 placeholders, PRD
    template 5, the six real SPECs 0–3. A line-count test was measured and
    rejected — the untouched template has 25 non-empty lines under ## API Contracts,
    because placeholder JSON is still lines.

  2. spec-contracts accepts a behavioural contract. It still demands a
    contract; it stopped demanding one particular form. ## Requirements,
    ## Contract, ## Behavioral Contract join the structural headings.

  3. spec-requirement-has-scenario (Should) — validate: нет правила «каждый Requirement в SPEC имеет Scenario» — инвариант держит только TDD-потребитель #450, conditional. Silent on a
    structural spec, fires only on a half-authored behavioural one. The blanket form
    was measured: it fires on 6 of 6 SPECs here, none defective.

Plus: the stub gate's remediation told every kind to "Fill MUST sections (Problem,
Goals, FR)". It now names the sections of the kind in hand.

#449 — NFR rules, and the trap in them

prd-nfr-exist and prd-nfr-measurable, both Should. extract_nfr_section already
existed, called from exactly one place — the tech-leakage check — so the validator
could find the section and asked nothing about its contents.

The subjective-adjective blacklist reads like a list of NFRs (scalable, robust,
efficient, responsive, fast) and was only ever applied to the FR section.
Across the 69 PRDs: 2 hits in FR (checked), 15 in NFR (not).

All 15 sit inside the template's own guidance
<!-- BAD: "System should be fast and responsive" -->. A rule flagging them would
be unclosable: the only fix is deleting the instructions. The rule strips non-prose
first. Verified 0 findings across all 69 PRDs, while hand-written vague prose
still produces findings with line numbers.

check_measurability_adjectives, the FR-side rule that shipped long ago, does
not strip and carries the same latent bug. It has never fired only because the
template's BAD examples happen to live under the NFR heading. Left alone — noted
in the handoff so the next person does not rediscover it.

Template + a test that reads it

The SPEC template is API-first and is the only signal an author has about what a
spec looks like. It now carries a short comment saying both contract shapes are
legitimate.

The existing stub test pins a hand-abridged copy of the template, so it cannot
notice the real file changing. A new test reads the file the binary actually ships
(include_str!, same path as template::engine) and asserts it still reads as a
stub, and that the new guidance comment does not register as a heading — it names
## Requirements inside backticks, and if section_exists ever started matching
those, an empty template would satisfy spec-contracts through its own comment.

Verification

Gate Result
cargo fmt --all -- --check exit 0
cargo clippy --workspace --all-targets -- -D warnings exit 0, 0 warnings
cargo test -p forgeplan-core --features test-helpers --no-fail-fast 2239 passed, 4 failed (17 binaries)
cargo test -p forgeplan --no-fail-fast 807 passed, 0 failed (58 binaries)
cargo test -p forgeplan-mcp --no-fail-fast 274 passed, 0 failed (19 binaries)

3320 passed, 4 failed, 94 test binaries — the binary count matches what
--workspace produces, so the per-crate split skipped nothing. The core failures
are all in git::tests (#454); they pass serially (51/51,
--test-threads=1) and this diff does not touch that module. forgeplan-core
needs --features test-helpers because the *_for_test helpers are gated and
normally arrive through forgeplan-mcp's dev-dependency.

Mutation checks — each fix reverted, the matching test had to fail:

  • remove the comment strip → nfr_adjectives_inside_html_comments_are_not_flagged FAILED
  • remove the placeholder scaling → untouched template back to PASS — 0 error(s), 0 warning(s)
  • MANY_PLACEHOLDERS = 100 → the new shipped-template test FAILED
  • add a real ## Requirements heading to the template → the same test FAILED on its second assertion

Four pre-existing rule-count tests failed and were updated, not silenced. They
sum named groups with comments rather than asserting a bare number, so updating one
means explaining where the new term came from. rules_for_spec_returns_base_plus_3
was renamed to _plus_4 rather than edited in place — a test name carrying a
number goes stale silently.

Not done

  • check_measurability_adjectives (FR side) still lacks the non-prose strip.
    Latent, not firing, deliberately out of scope.
  • SPEC-001 still fails validation. It is a document about writing specs, filed as a
    spec; the fix is re-filing it, not loosening a rule.

Full reasoning, rejected alternatives, and environment notes:
docs/handoff/spec-validator-and-nfr-rules.md.

Evidence: EVID-171 (verdict: supports, CL3, evidence_type: test, with
git provenance), linked informs to PROB-105 — R_eff 0.00 → 1.00.

explosivebit and others added 4 commits September 7, 2026 18:03
…real specs

Measured on 0.36.0 before writing any code, because issue #450's premise did not
survive contact with this repository.

An untouched SPEC template — body still `{METHOD} /v1/{resource}` — validated
with 0 errors and 0 warnings, and once any evidence pack was linked it activated
at R_eff 1.00. A complete behavioural spec with two requirements and two
GIVEN/WHEN/THEN scenarios failed a MUST rule and could not activate at all. The
validator waved through a document saying nothing and blocked one carrying a full
test oracle.

Two independent causes.

check_stub, the gate whose whole job is catching unfilled templates, knows twelve
phrases and all twelve are PRD prose ("Что мы строим и почему это важно",
"[Actor] can [capability]"). None appears in the SPEC template. It also counted
placeholders but capped that at +1 against a threshold of 3, so fifteen unfilled
slots weighed the same as one. The count now scales, with the threshold taken
from the corpus rather than taste: SPEC template 15 placeholders, PRD template 5,
the six real SPECs 0-3. A line-count test was measured and rejected — the
untouched template has 25 non-empty lines under `## API Contracts`, because
placeholder JSON is still lines.

spec-contracts accepted only API Contracts / Data Models / Contracts, all
structural, so `## Requirements` + `#### Scenario` had no route through the MUST.
The kernel was mandating one methodology's shape and rejecting the other — the
reverse of what #450 reports. It now accepts a behavioural contract too. The rule
still demands a contract; it stopped demanding one particular form.

Also: the stub gate's remediation told every kind to "Fill MUST sections
(Problem, Goals, FR)", meaningless for a SPEC or an ADR. The SPEC path was
unreachable until now so the wrong advice was never printed; it names the
sections of the kind in hand.

New rules.

spec-requirement-has-scenario (Should) — issue #450 narrowed to internal
consistency: a spec that opens a `### Requirement` must close it with a
`#### Scenario`. The blanket form was measured first and rejected — none of the
six SPECs here uses those headings, so it would have fired on 6 of 6, none
defective. Conditional means silent on a structural spec, firing only on a
half-authored behavioural one.

prd-nfr-exist and prd-nfr-measurable (both Should) — issue #449. PRD carried 24
validator rules and none for non-functional requirements, while
extract_nfr_section already existed, called from exactly one place: the
tech-leakage check. The subjective-adjective blacklist reads like a list of NFRs
— scalable, robust, efficient, responsive, fast — and was only ever applied to
the FR section. Across the 69 PRDs here: 2 hits in FR (checked), 15 in NFR (not).

All 15 sit inside the template's own `<!-- BAD: "System should be fast and
responsive" -->` guidance, so the rule strips non-prose first. Without that it
produces 15 findings out of 15 false, closable only by deleting the template's
instructions. Verified: 0 findings across all 69 PRDs, while a hand-written
"NFR-001: The export is fast and the service is robust" still produces three with
line numbers.

Should, not Must, for all three: 30 of 69 PRDs have no NFR section, and turning
them red at once is how a rule gets ignored rather than obeyed.

Four rule-count tests failed and were updated rather than silenced. They sum
named groups with comments instead of asserting a bare number, so updating one
means explaining where the new term came from. rules_for_spec_returns_base_plus_3
was renamed to _plus_4 rather than edited in place — a name carrying a number
goes stale silently.

Nine new tests. Two mutation checks: remove the comment strip and
nfr_adjectives_inside_html_comments_are_not_flagged fails; remove the placeholder
scaling and the untouched template is back to PASS 0/0.

Tests were run per crate, not with --workspace: a full workspace build does not
fit in the free space on this machine and twice reported "run did not happen"
with ld errno=28. core 2239 passed / 3 failed (17 binaries), cli 807/0 (58), mcp
274/0 (19) — 94 binaries total, matching what --workspace produces. The 3 are
#454 in git::tests, which this diff does not touch.

Refs: PROB-105, #449, #450
Context is large enough that this needs to be readable from a fresh start.

Everything the document points at is committed, which is the correction to the
last handoff written in this repo: that one named a scratchpad path that did not
exist here, because the files were in a session temp directory tied to a session
id. It handed off work that was about to evaporate while reporting it ready —
the same defect class the release it accompanied was about.

Carries the measurements rather than the conclusions: the placeholder counts that
set the stub threshold (template 15, PRD template 5, real specs 0-3), the 2-vs-15
FR/NFR adjective split and why all 15 were false, the per-crate test results and
why --workspace does not fit on this machine, and the two mutation checks.

Also records what NOT to re-litigate — the rejected shape whitelist, the blanket
#450 rule, Must instead of Should — and flags the uncommitted Cargo.toml and
.cargo/ work belonging to a second session in this worktree, so nobody sweeps it
into a commit.

Refs: PROB-105, #449, #450
…l template

The SPEC template is API-first and is the only signal an author has about
what a spec looks like. Now that the validator accepts a behavioural
contract, the template says so.

The existing stub test pins a hand-abridged copy of the template, so it
cannot notice the real file changing -- adding this comment proved the
gap. A new test reads the file the binary actually ships (include_str!,
same path as template::engine) and asserts two things: it still reads as
a stub, and the guidance comment does not register as a heading. The
comment names `## Requirements` inside backticks; if section_exists ever
started matching those, an empty template would satisfy spec-contracts
through its own comment.

Both assertions mutation-checked: MANY_PLACEHOLDERS = 100 kills the
first, appending a real `## Requirements` heading kills the second.

Also records the #449 / #450 / PROB-105 work in CHANGELOG, and corrects
the handoff's stale "not yet pushed" status.

Refs: prob-105

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 4 mutation checks

Records what was measured rather than what was reported: the before/after
validator behaviour on the template and on a behavioural spec, the corpus
numbers that set the placeholder threshold (SPEC template 15, PRD template
5, real SPECs 0-3), and the 2-vs-15 FR/NFR adjective split with all 15
inside the template's own BAD-example guidance.

Four mutation checks, because a green test on broken code proves nothing.
Also states this pack's limits: it certifies behaviour on this repo's
corpus only, the FR-side rule still carries the same latent bug, and the
4 git::tests failures are the known #454 flake, not health.

PROB-105 R_eff 0.00 -> 1.00; both artifacts activated.

Refs: prob-105, EVID-171

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant