Finding
Canonical planning and state mark Phase 3 complete at 14.7× the CPU baseline, but the executable benchmark does not implement the declared protocol and cannot fail the gate. The plan requires a seed-fixed 256-sentence batch, 10 warmups, 30 measured iterations, and a hard ≥10× pass condition. The code cycles 15 sentences to 32, performs one warmup and five iterations, compares batched candidate throughput to a sequential baseline, and merely prints a warning below 10×. Missing, malformed, or unparseable baseline data also returns success, and no CI stage runs the opt-in benchmark.
Evidence
- Canonical
projects/logismos/phases/03-stella/PLAN.md:59-72,557-566 defines the 256-sentence/10+30/hard-gate protocol.
- Canonical
projects/logismos/STATE.md:31-54,77-84 declares the phase and ≥10× gate complete.
crates/embed/benches/stella_throughput.rs:7-16,26-29,58-64 is env-opt-in and exits successfully when skipped.crates/embed/benches/stella_throughput.rs:98-139 runs the 32×5 protocol and treats a sub-threshold result as a warning.phases/03-stella/golden/baseline_cpu.py:59-79 measures one sentence at a time..kanon-ci.toml:25-42 runs no benchmark or performance assertion.
Why this matters
The strongest quantitative completion claim in the phase is a manually observed number, not a replayable gate. It can remain “complete” when the baseline disappears, parsing breaks, performance regresses below 10×, or the benchmark protocol changes.
Desired correction
Encode the benchmark protocol and environment in one typed manifest consumed by both baseline and candidate runners. Use the same pinned inputs, token-length distribution, batch/concurrency policy, timing boundary, warmup/iteration counts, model revision, thread affinity, and host identity; report distributions and uncertainty rather than one ratio. Make missing evidence and <10× terminal failures on the designated performance runner, publish a content-addressed result artifact, and derive the planning/state claim from that artifact.
Finding
Canonical planning and state mark Phase 3 complete at 14.7× the CPU baseline, but the executable benchmark does not implement the declared protocol and cannot fail the gate. The plan requires a seed-fixed 256-sentence batch, 10 warmups, 30 measured iterations, and a hard ≥10× pass condition. The code cycles 15 sentences to 32, performs one warmup and five iterations, compares batched candidate throughput to a sequential baseline, and merely prints a warning below 10×. Missing, malformed, or unparseable baseline data also returns success, and no CI stage runs the opt-in benchmark.
Evidence
projects/logismos/phases/03-stella/PLAN.md:59-72,557-566defines the 256-sentence/10+30/hard-gate protocol.projects/logismos/STATE.md:31-54,77-84declares the phase and ≥10× gate complete.crates/embed/benches/stella_throughput.rs:7-16,26-29,58-64is env-opt-in and exits successfully when skipped.crates/embed/benches/stella_throughput.rs:98-139runs the 32×5 protocol and treats a sub-threshold result as a warning.phases/03-stella/golden/baseline_cpu.py:59-79measures one sentence at a time..kanon-ci.toml:25-42runs no benchmark or performance assertion.Why this matters
The strongest quantitative completion claim in the phase is a manually observed number, not a replayable gate. It can remain “complete” when the baseline disappears, parsing breaks, performance regresses below 10×, or the benchmark protocol changes.
Desired correction
Encode the benchmark protocol and environment in one typed manifest consumed by both baseline and candidate runners. Use the same pinned inputs, token-length distribution, batch/concurrency policy, timing boundary, warmup/iteration counts, model revision, thread affinity, and host identity; report distributions and uncertainty rather than one ratio. Make missing evidence and <10× terminal failures on the designated performance runner, publish a content-addressed result artifact, and derive the planning/state claim from that artifact.