Skip to content

score: a refuting leaf pack is told to add structured fields it already has #488

Description

@explosivebit

Found immediately after v0.37.0 shipped, while re-scoring the graph with the new binary. Introduced by the leaf-EVID scoring work in that release (#325), so this is fresh, not inherited.

Reproducer

```
$ forgeplan score EVID-146

EVID-146 — PROB-078 refuted: MCP read-after-write correct across 7 tests incl real-binary subprocess
Leaf evidence — scored on its own structured fields. R_eff = 0.00

✗ Structured fields missing or undeclared
→ forgeplan get EVID-146 — add verdict: and congruence_level: under ## Structured Fields

• Leaf evidence scored on its own fields: Refutes CL3 = 0.00
```

The output contradicts itself two lines apart. The hint says the structured fields are missing; the factor line below it reports having read them (Refutes CL3). They are present:

```

Structured Fields

verdict: refutes
congruence_level: 3
evidence_type: test
```

Cause

crates/forgeplan-cli/src/commands/score.rs:324 gates the hint on the score alone:

```rust
if report.r_eff <= 0.1 {
ui::error_hint("Structured fields missing or undeclared", ...)
```

That conflates two different reasons for a low leaf score:

  • fields absent → CL0 → 0.1. The hint is correct and useful.
  • verdict: refutes → 0.00 by design. The fields are present and correct; a refuting pack should score zero. The hint is wrong, and it asks the author to add something already there.

Why it matters more than a cosmetic wording bug

The scoring is right — only the advice is wrong. But following it leads nowhere: you open the file, the fields are there, and the tool keeps saying they aren't. That is the same shape as the defects v0.37.0 was about — a surface reporting something that is not true — and it now lives inside the fix for one of them.

Fix direction

Gate the hint on why the score is low, not on the number. The scorer already knows: it distinguishes Refutes CL3 = 0.00 from a CL0 fallback in the factor it emits. A refuting pack at 0.00 needs no hint at all, or a different one explaining that a refutation scoring zero is the intended reading.

Test that would catch it

Assert on the hint text for a leaf pack with verdict: refutes + congruence_level: 3 — not just on its score. The score was already correct; only the advice regressed, so a score-only assertion passes over this.

Refs: #325, PRD-086, v0.37.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions