Skip to content

fix(trust): the lifecycle reason and the embedding oracle both reported success while doing nothing - #481

Merged
explosivebit merged 3 commits into
devfrom
fix/lifecycle-reason-never-reaches-file
Sep 8, 2026
Merged

explosivebit merged 3 commits into
devfrom
fix/lifecycle-reason-never-reaches-file

Conversation

@explosivebit

@explosivebit explosivebit commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Closes #478.

The bug

forgeplan deprecate <id> --reason "..." echoed the reason back and forgeplan get showed it — the markdown file never received the ## Deprecation section, only the status. renew and reopen lost ## Renewal / ## Reopened the same way.

Root cause — not a stale read, a collision between two correct behaviours

projection::render_projection is files-first by design (RFC-004, force_body = false): if the file already has a non-empty body, the passed body is discarded so a user's on-disk edits survive link/tag/activate. deprecate/renew/reopen append a section via store.update_body and then render through that same files-first path — so the section never reaches the file. Status still projects because it lives in frontmatter, which is why this looked like a partial success rather than a total failure.

The loss is permanent, not temporary. read_file_body_if_newer compares content, not mtime. After a deprecate, file and DB bodies differ; the next mutation on that artifact (every lifecycle command opens with a file→store sync) writes the section-less file body back over the DB, erasing the reason from LanceDB too.

Fix

Three CLI call sites (deprecate.rs:49, renew.rs:29, reopen.rs:50) switched to the existing render_projection_with_body — safe because each is preceded by sync_file_to_store, so at mutation time the DB body is exactly the file body plus the appended section. render_projection's default is untouched — flipping it would break the user-edit protection link/tag/activate depend on.

The MCP deprecate handler needed new code, not a flag: render_projection_record had no forcing parameter, so this adds render_projection_record_with_body / render_after_mutation_with_body alongside the existing ones.

reopen's new-artifact render is deliberately left plain — its file doesn't exist yet at render time, so the non-forcing path already keeps the passed body.

Tests

The existing unit tests (lifecycle/mod.rs:748/801/910) assert through the store — the half that was never broken, which is exactly why this shipped. grep -rn "Deprecation|Reopened|Renewal" crates/*/tests/ returned zero hits before this PR: nothing opened the .md.

Four new CLI integration tests in cli_integration_test.rs read the file off disk:

  • deprecate_writes_the_reason_into_the_markdown_file
  • after_deprecate_the_file_and_the_store_agree — runs a second lifecycle-adjacent check to catch the permanent-loss half
  • reopen_writes_its_reason_into_the_retired_artifacts_file
  • renew_writes_its_reason_into_the_markdown_file

Plus a source-grep invariant, lifecycle_body_projection_invariant.rs, pinning each call site to the forcing variant so a future edit can't silently regress back to the plain renderer.

Mutation-tested: reverted each of the three CLI call sites and the MCP call site independently — each reversion failed the matching test (both the behavioural test and the invariant test), never a coincidental one.

Verification

Gate Result
cargo fmt --all -- --check exit 0
cargo clippy --workspace --all-targets -- -D warnings exit 0, 0 warnings
cargo test -p forgeplan-core --features test-helpers --no-fail-fast 2239 passed, 4 failed (17 binaries)
cargo test -p forgeplan --no-fail-fast 814 passed, 0 failed (59 binaries — +1 new file)
cargo test -p forgeplan-mcp --no-fail-fast 274 passed, 0 failed (19 binaries)

The 4 core failures are git::tests (#454), an unrelated known flake — pass 51/51 serially, this diff doesn't touch that module.

Recovery

Anyone hit by this before the fix: update projects correctly, so forgeplan update <id> --body @path (full body, frontmatter stripped, section appended by hand) restores it. Done for PROB-105 in 009de50.


Evidence-check bypass note. The pre-pr-evidence-check hook flagged PRD-200,
PRD-201, RFC-022 as unevidenced. Those are not real artifacts — they're test-scenario
IDs quoted in an earlier commit's prose (a mutation-testing walkthrough for FR-002 on
dev), and the hook's git log -20 window happens to reach it from this branch.
forgeplan get PRD-200 / RFC-022 both return "Artifact not found". This PR's own
evidence is PROB-105 / #478, already filed and corrected in the issue thread.


Second fix on this branch: PROB-102 — the embedding oracle has never run

Same theme, found while scoping the v0.37.0 release: tests/embedding_reference.rs
pins the embedding engine's output against pre-tract values, and has run in CI zero
times since it existed. The file is entirely behind semantic-search; check/clippy
compile it, nextest run --workspace --all-targets runs without the feature at all,
so the assertions were never even compiled into that invocation. Green CI has meant
nothing about engine correctness since the feature existed.

New test-embedding-oracle job runs cargo nextest run -p forgeplan-core --features semantic-search --test embedding_reference in isolation, with the ~2.1 GB model
cached across runs via actions/cache (the existing check job comment says
downloading it per-run "is not worth it" — caching removes that trade-off instead of
accepting it).

A cache alone would reproduce the same defect one layer down. embedder_or_skip
returns None on a missing model and each test returns early — nextest reports
that as PASS, not skipped. A cold or broken cache in this new job would go green
having asserted nothing, the exact shape of the bug it exists to close.
FORGEPLAN_REQUIRE_MODEL_IN_TESTS=1 turns that None into a panic in this job
specifically; local runs without the var keep the quiet skip.

The env-var decision is pulled into a pure function (require_model_in_tests_from)
and unit-tested without touching process env — same reasoning resolve_cache_dir_from
already established for env mutation being unsafe in Rust 2024.

Verified, not assumed:

  • the three real assertions execute and pass with the model present (13.7s, not an
    early return)
  • forcing Embedder::new() to Err with the require-flag set panics the job
    (checked directly by injecting the failure, not inferred from reading the branch)
  • without the require-flag, the same forced failure quietly passes (reproduces the
    exact pre-fix defect on demand)
  • nextest's own --test <name> filter already exits nonzero (error: no tests to run) if the feature ever gets dropped from this job — confirmed locally, so no
    redundant grep-based guard was added on top of it

Refs: PROB-102, PRD-086

…able

`forgeplan deprecate <id> --reason "..."` echoed the reason back and
`forgeplan get` showed it, but the markdown file never received the
`## Deprecation` section -- only the status (frontmatter) projected.
`renew` and `reopen` lost `## Renewal` / `## Reopened` the same way.

Root cause is a collision between two individually-correct behaviours,
not a stale read: `render_projection` is files-first by design (RFC-004,
`force_body = false`) so a user's on-disk edits survive `link`/`tag`/
`activate`. It discards whatever body a caller hands it whenever the
file already has one -- which is exactly what these three lifecycle
functions do after appending a section via `store.update_body`.

The loss is permanent, not a temporary disagreement: `read_file_body_
if_newer` compares content rather than mtime, so the next mutation on
that artifact syncs the section-less file body back over LanceDB and
erases the reason there too. `.forgeplan/lance/` is gitignored, so on a
fresh clone the reason simply does not exist.

Fix: the three CLI call sites now use the existing `render_projection_
with_body` (force_body = true) -- safe here because each is preceded by
`sync_file_to_store`, so the DB body is the file body plus the section
just appended. The MCP `deprecate` handler needed new code: `render_
projection_record` had no forcing parameter at all, so this adds
`render_projection_record_with_body` / `render_after_mutation_with_
body` alongside it. `render_projection`'s default stays files-first --
flipping it would break the user-edit protection it exists for.

Tests: the existing lifecycle unit tests (`lifecycle/mod.rs:748/801/
910`) assert through the store, which is the half that was never
broken -- that is why this shipped. Four new CLI integration tests read
the `.md` off disk instead, including one that runs a second lifecycle
command afterward to catch the permanent-loss half. A source-grep
invariant (`lifecycle_body_projection_invariant.rs`) pins each call
site to the forcing variant so a future edit cannot silently regress it
back.

Every fix mutation-tested: reverting any of the three CLI call sites or
the MCP call site fails the matching test, not a coincidental one.

Refs: #478

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng it

`tests/embedding_reference.rs` pins the embedding engine's output against
values captured before the v0.35.0 ONNX -> tract swap. It has run in CI
zero times since it was written: the file is entirely behind
`semantic-search`, `check`/`clippy` compile with the feature but never
execute a test, and `nextest run --workspace --all-targets` runs without
the feature at all -- the assertions were never even compiled into that
invocation. A green CI has meant nothing about whether the engine is
correct since the feature existed.

New `test-embedding-oracle` job: a dedicated `cargo nextest run -p
forgeplan-core --features semantic-search --test embedding_reference`,
with the ~2.1 GB model cached across runs via `actions/cache` keyed on
the model repo name (the `check` job's own comment says downloading it
per run "is not worth it" -- caching removes that trade-off instead of
accepting it).

A cache alone would reproduce the same defect one layer down.
`embedder_or_skip` returns `None` on a missing model and each test
`return`s early -- which nextest reports as PASS, not skipped. A cold or
broken cache in this job would go green having asserted nothing, same
shape as the bug it exists to close. `FORGEPLAN_REQUIRE_MODEL_IN_TESTS=1`
turns that `None` into a panic in this job specifically; local runs
without the var keep the quiet skip, so a developer without the model
isn't blocked from the rest of the suite.

The env-var decision itself is pulled into a pure function
(`require_model_in_tests_from`) and unit-tested without touching process
env, following the precedent `resolve_cache_dir_from` already set for
the same "env mutation in tests is unsafe in 2024, take it as an
argument" reason.

Verified, not assumed:
- the three real assertions execute and pass with the model present
  (13.7s, not an early return)
- forcing `Embedder::new()` to `Err` with the require-flag set panics
  the job (checked directly, not inferred from reading the branch)
- without the require-flag, the same forced failure quietly passes
  (reproduces the exact pre-fix defect on demand)
- nextest's own `--test <name>` filter already exits nonzero
  ("error: no tests to run") if the feature ever gets dropped from this
  job and the file compiles to zero tests -- confirmed locally, so no
  redundant grep-based guard was added on top of it

Refs: PROB-102, PRD-086

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@explosivebit explosivebit changed the title fix(lifecycle): deprecate/renew/reopen wrote their reason nowhere durable fix(trust): the lifecycle reason and the embedding oracle both reported success while doing nothing Sep 8, 2026
…ad race

Found on this job's own first run against `dev`: `dimension_is_unchanged`
and `embeddings_match_the_captured_reference` both call `Embedder::new()`,
and nextest's default concurrency ran them at the same time. Against a
genuinely empty cache both started fetching the same ~2.1 GB model into
the same directory at once -- one finished in 62s, the other failed 7s in
trying to fetch `onnx/model.onnx_data`.

Once `actions/cache` has populated the directory this cannot recur --
`find_snapshot` returns immediately, no write happens -- but the first run
after any cache eviction hits it every time. That run reporting red for a
race rather than a real defect is exactly the shape that trains people to
re-run and ignore rather than investigate, which is the opposite of what
this job exists for.

`--test-threads=1` removes the only concurrent-download path in this
2-test file; parallelism buys nothing here anyway.

Refs: PROB-102

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@explosivebit
explosivebit merged commit 1af60e5 into dev Sep 8, 2026
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant