[PROB-105] The SPEC validator passed empty templates and blocked real specs

.forgeplan/evidence/EVID-171-prob-105-spec-validator-inversion-fixed-3320-tests-4-mutation-checks.md

@@ -0,0 +1,108 @@
+---
+depth: tactical
+id: EVID-171
+kind: evidence
+links:
+- target: PROB-105
+  relation: informs
+status: active
+title: 'PROB-105: SPEC validator inversion fixed — 3320 tests, 4 mutation checks'
+---
+
+---
+assigned_number: 171
+predicted_number: 171
+slug: evid-prob-105-spec-validator-inversion-fixed-3320-tests-4-mutation-checks
+---
+
+# EVID-171: the SPEC validator inversion, measured before and after
+
+## Summary
+
+PROB-105 claimed the SPEC validator was inverted: it passed empty templates and
+blocked real behavioural specs. Both halves reproduced on 0.36.0 with the shipped
+binary, and both are closed by `daa103b` + `c8e7bd3` on `fix/spec-validator-inverted`.
+
+## What was measured
+
+| Input | Before | After |
+|---|---|---|
+| untouched `forgeplan new spec` template | `PASS — 0 error(s), 0 warning(s)`, activates at R_eff 1.00 | stub warning; `activate` refuses |
+| behavioural spec, 2 requirements + 2 GIVEN/WHEN/THEN scenarios | `x [MUST] spec-contracts` → FAIL, cannot activate | PASS |
+| SPEC-002 … SPEC-006 | pass | unchanged |
+| SPEC-001 | fails `spec-summary` + `spec-contracts` | unchanged — a document *about* writing specs, filed as a spec |
+
+Corpus measurements that set the threshold, rather than taste:
+
+- placeholders: SPEC template **15**, PRD template **5**, the six real SPECs **0–3**.
+  `MANY_PLACEHOLDERS = 8` sits in the gap with room on both sides.
+- a line-count test was measured and rejected: the untouched template has 25
+  non-empty lines under `## API Contracts`, because placeholder JSON is still lines.
+- subjective-adjective hits across the 69 PRDs: **2** in FR (already checked),
+  **15** in NFR (never checked). All 15 sit inside the template's own
+  `<!-- BAD: "System should be fast and responsive" -->` guidance, so a rule
+  flagging them would be unclosable. With the non-prose strip: **0 findings across
+  all 69 PRDs**, while hand-written vague prose still produces findings with line
+  numbers.
+- the blanket #450 rule was measured before being rejected: fires on 6 of 6 SPECs
+  here, none defective. The shipped rule is conditional.
+
+## Test run
+
+Per crate — a full `cargo test --workspace` does not fit in this machine's free
+space (`ld: write() failed, errno=28` twice at ~300 MB free).
+
+| Command | Result |
+|---|---|
+| `cargo fmt --all -- --check` | exit 0 |
+| `cargo clippy --workspace --all-targets -- -D warnings` | exit 0, 0 warnings |
+| `cargo test -p forgeplan-core --features test-helpers --no-fail-fast` | 2239 passed, 4 failed (17 binaries) |
+| `cargo test -p forgeplan --no-fail-fast` | 807 passed, 0 failed (58 binaries) |
+| `cargo test -p forgeplan-mcp --no-fail-fast` | 274 passed, 0 failed (19 binaries) |
+
+**3320 passed, 4 failed, 94 test binaries.** The binary count matches what
+`--workspace` produces, so the per-crate split skipped nothing. The 4 failures are
+all `git::tests` (#454): they pass serially (51/51 under `--test-threads=1`) and
+this diff does not touch that module.
+
+## Mutation checks
+
+Each fix reverted; the matching test had to fail. A test that passes on broken
+code proves nothing, so each was checked rather than assumed.
+
+| Mutation | Test | Result |
+|---|---|---|
+| remove the HTML-comment strip | `nfr_adjectives_inside_html_comments_are_not_flagged` | FAILED |
+| remove the placeholder scaling | untouched template back to `PASS — 0 error(s), 0 warning(s)` | reproduced |
+| `MANY_PLACEHOLDERS = 100` | `the_shipped_spec_template_is_a_stub_and_declares_no_sections` | FAILED |
+| append a real `## Requirements` heading to the template | same test, second assertion | FAILED |
+
+Four pre-existing rule-count tests failed and were updated, not silenced. They sum
+named groups with comments rather than asserting a bare number, so updating one
+means explaining where the new term came from. `rules_for_spec_returns_base_plus_3`
+was renamed to `_plus_4` rather than edited in place — a test name carrying a number
+goes stale silently.
+
+## Limits of this pack
+
+- It certifies the validator's behaviour on this repository's corpus (69 PRDs, 6
+  SPECs) and on the shipped template. It says nothing about corpora with other
+  spec conventions.
+- `check_measurability_adjectives` (the FR-side rule) still lacks the non-prose
+  strip and carries the same latent bug. It has never fired only because the
+  template's BAD examples happen to live under the NFR heading. Out of scope here,
+  and recorded so it is not rediscovered.
+- The 4 `git::tests` failures are a known pre-existing flake (#454), not evidence
+  of health in that module.
+
+## Structured Fields
+
+verdict: supports
+congruence_level: 3
+evidence_type: test
+base_sha: 5e9b9a9f
+result_sha: c8e7bd3
+changed_paths: crates/forgeplan-core/src/validation/rules.rs, crates/forgeplan-core/src/validation/checks.rs, crates/forgeplan-core/src/lifecycle/mod.rs, templates/spec/_TEMPLATE.md, CHANGELOG.md, docs/handoff/spec-validator-and-nfr-rules.md
+
+
+

.forgeplan/prds/PRD-086-the-trust-layer-reports-values-it-never-computed.md

@@ -141,3 +141,4 @@ GitHub: #325, #392, #393, #330.
 
 
 
+

.forgeplan/problems/PROB-105-the-spec-validator-passes-empty-templates-and-blocks-real-behavioural-specs.md

@@ -0,0 +1,113 @@
+---
+depth: tactical
+id: PROB-105
+kind: problem
+links:
+- target: PRD-086
+  relation: informs
+status: active
+title: The SPEC validator passes empty templates and blocks real behavioural specs
+---
+
+---
+assigned_number: 105
+context: '{grouping tag}'
+created: 2026-09-07
+predicted_number: 105
+slug: prob-the-spec-validator-passes-empty-templates-and-blocks-real-behavioural-specs
+---
+
+# PROB-105: the SPEC validator is inverted
+
+## Signal
+
+Measured on 0.36.0 with the shipped binary, in a fresh workspace.
+
+**An untouched SPEC template validates clean and activates with perfect trust.**
+
+```
+$ forgeplan new spec "Widget sync API"      # 110-line template, nothing edited
+$ forgeplan validate SPEC-001
+  Result: PASS -- 0 error(s), 0 warning(s)
+Next: forgeplan activate SPEC-001
+```
+
+Link any evidence pack and it activates: `Status: active`, `R_eff: 1.00`. The
+body is still `{METHOD} /v1/{resource}` and "Что специфицируется. Одно
+предложение."
+
+**A complete behavioural spec fails a MUST and cannot activate.**
+
+```
+$ forgeplan validate SPEC-003    # 2 × ### Requirement, 2 × #### Scenario, GIVEN/WHEN/THEN
+  x [MUST] spec-contracts: Missing '## API Contracts' or '## Data Models' section
+  Result: FAIL -- 1 error(s)
+```
+
+So the validator waves through a document that says nothing and blocks one that
+carries a full test oracle.
+
+## Why each half happens
+
+**The block.** `check_spec_contracts` looks for the literal headings
+`API Contracts` or `Data Models` (rules.rs, `section_exists`). `## Requirements`
+and `## Behavioral Contract` are not in `expand_aliases`, so a behavioural spec
+has no route through a MUST rule. The kernel already encodes one methodology's
+shape as mandatory — which is the opposite of what issue #450 alleges.
+
+**The pass.** `check_stub` is the gate that exists to catch unfilled templates.
+Its twelve `PHRASE_MARKERS` are all PRD prose — "Что мы строим и почему это
+важно", "What we are building and why", `[Actor] can [capability]`. None appears
+in the SPEC template. It also counts placeholders, but caps that signal at +1
+while the threshold is 3, so fifteen unfilled placeholders count the same as one.
+
+The stub gate is structurally incapable of seeing a SPEC.
+
+## The measurement that gives a shape-agnostic fix
+
+Single-brace placeholders, counted across the corpus:
+
+| Document | Placeholders |
+|---|---|
+| SPEC template, untouched | **15** |
+| SPEC-001 … SPEC-006 (real) | 0–3 |
+| PRD template (which the gate *does* catch) | 5 |
+
+Clean separation with a wide margin, and it asks nothing about methodology: not
+"is this Gherkin or API", but "is this still a form to fill in".
+
+A line-count test would NOT work — the untouched template has 25 non-empty lines
+under `## API Contracts`, because placeholder JSON is still lines.
+
+## What this explains
+
+The marketplace TDD flow grew its own gate (`tdd-planner` HARD RULE 1 refusing to
+plan against a spec with no `#### Scenario`) because core actively rejects the
+shape it needs. Issue #450 read that as "core has no rule"; the truth is core has
+a MUST pointing the other way.
+
+## Fix
+
+Three changes, none of which teaches the kernel a methodology:
+
+1. **`check_stub` learns to see a SPEC** — scale the placeholder signal instead of
+   capping it at +1. Catches the template, silent on all six real specs.
+2. **`spec-contracts` accepts a behavioural route** — `Requirements`,
+   `Contract`, `Behavioral Contract` alongside the API-shaped headings. The rule
+   still demands *a* contract; it stops demanding one particular form of it.
+3. **Conditional consistency rule (#450, literally)** — if a spec opens
+   `### Requirement`, each must carry a `#### Scenario`. Vacuous on all six of
+   ours (none uses those headings); fires on a half-authored behavioural spec,
+   which is the real defect. Should, not Must.
+
+## Related
+
+| Artifact | Relation |
+|---|---|
+| PRD-086 | informs |
+
+GitHub: #450 (its premise is inverted — see above), #449 (separate).
+
+
+
+

CHANGELOG.md

@@ -89,6 +89,55 @@ corresponding sprint evidence under `.forgeplan/evidence/`.
   The progress line also counted every record instead of the ones being
   encoded (`Embedding 1 of 427`, not `Embedding 427`).
 
+- **The SPEC validator passed empty templates and blocked real specs** (#450,
+  PROB-105). An untouched `forgeplan new spec` template — every field still a
+  placeholder — validated `PASS — 0 error(s), 0 warning(s)` and activated at
+  **R_eff 1.00**. A spec with two requirements and two GIVEN/WHEN/THEN
+  scenarios failed the MUST rule `spec-contracts`, so it could not activate at
+  all. The kernel had a MUST pointing *against* behavioural specs while waving
+  through documents that said nothing.
+
+  Two causes. `check_stub`'s twelve phrase markers are all PRD prose, and its
+  placeholder signal was capped at `+1` against a threshold of 3 — fifteen
+  unfilled slots weighed the same as one. The count scales now; the threshold
+  comes from the corpus (SPEC template **15** placeholders, PRD template
+  **5**, the six real SPECs **0–3**), not from taste. And `spec-contracts`
+  demanded one particular contract shape; it still demands *a* contract, but
+  `## Requirements`, `## Contract`, and `## Behavioral Contract` now count
+  alongside `## API` and `## Data Model`.
+
+  This is why the marketplace TDD flow grew its own scenario gate: core was
+  rejecting the shape TDD needs.
+
+- **The stub gate told every artifact kind to fill a PRD's sections.** Its
+  remediation line read `Fill MUST sections (Problem, Goals, FR)` whether the
+  artifact was a SPEC, an ADR, or an Epic. It now names the sections of the
+  kind in hand.
+
+### Added
+
+- **`spec-requirement-has-scenario`** (Should, #450). Fires only on a spec
+  that already writes requirements behaviourally and leaves one without a
+  scenario. Deliberately conditional: the blanket form was measured against
+  this repository and fired on 6 of 6 SPECs, none of them defective.
+
+- **`prd-nfr-exist` and `prd-nfr-measurable`** (Should, #449). The PRD had 24
+  validator rules and none about non-functional requirements. `extract_nfr_section`
+  already existed and was called from exactly one place — the tech-leakage
+  check — so the validator could find the NFR section and asked nothing about
+  its contents.
+
+  Both are Should, not Must: 30 of 69 PRDs here have no NFR section at all.
+  `prd-nfr-measurable` strips non-prose before scanning, which is the whole
+  difficulty — the subjective-adjective list (`scalable`, `robust`,
+  `efficient`, `responsive`) reads like a list of NFRs and had only ever been
+  applied to the FR section. Across the 69 PRDs it matches **2** places in FR
+  and **15** in NFR, and all 15 sit inside the template's own
+  `<!-- BAD: "System should be fast and responsive" -->` guidance. A rule
+  flagging those would be unclosable — the only fix would be deleting the
+  instructions. Verified: **0 findings across all 69 PRDs**, while
+  hand-written vague prose still produces findings with line numbers.
+
 ### Internal
 
 - ADR-025 (orchestration sits above ForgePlan; per-surface dispositions) and

crates/forgeplan-core/src/lifecycle/mod.rs

@@ -191,8 +191,21 @@ pub async fn collect_activation_gates(
     let stub_check = crate::validation::rules::check_stub(&record.body, &record.frontmatter_map());
     let stub_ok = stub_check.is_none();
     let stub_msg = stub_check.map(|msg| {
+        // PROB-105. The remediation used to name Problem / Goals / FR for every
+        // kind — sections a SPEC does not have and an ADR does not have either.
+        // Until this release the SPEC case was unreachable (the gate could not
+        // see a SPEC at all), so the wrong advice was never printed; now that it
+        // can fire, telling a spec author to fill in a Problem section would send
+        // them looking for something that is not in their template.
+        let sections = match record.kind.to_ascii_lowercase().as_str() {
+            "spec" => "Summary, a contract section (API Contracts / Data Models / Requirements)",
+            "adr" => "Context, Decision, Consequences",
+            "rfc" => "Summary, Motivation, Options Considered, Proposed Direction",
+            "epic" => "Vision, Goals, Children",
+            _ => "Problem, Goals, Functional Requirements",
+        };
         format!(
-            "{msg} → Fill MUST sections (Problem, Goals, FR) before activating. \
+            "{msg} → Replace the template placeholders and fill {sections} before activating. \
              See PRD-043 FR-003 for stub detection rules."
         )
     });