fix(trust): the lifecycle reason and the embedding oracle both reported success while doing nothing - #481
Merged
Conversation
…able `forgeplan deprecate <id> --reason "..."` echoed the reason back and `forgeplan get` showed it, but the markdown file never received the `## Deprecation` section -- only the status (frontmatter) projected. `renew` and `reopen` lost `## Renewal` / `## Reopened` the same way. Root cause is a collision between two individually-correct behaviours, not a stale read: `render_projection` is files-first by design (RFC-004, `force_body = false`) so a user's on-disk edits survive `link`/`tag`/ `activate`. It discards whatever body a caller hands it whenever the file already has one -- which is exactly what these three lifecycle functions do after appending a section via `store.update_body`. The loss is permanent, not a temporary disagreement: `read_file_body_ if_newer` compares content rather than mtime, so the next mutation on that artifact syncs the section-less file body back over LanceDB and erases the reason there too. `.forgeplan/lance/` is gitignored, so on a fresh clone the reason simply does not exist. Fix: the three CLI call sites now use the existing `render_projection_ with_body` (force_body = true) -- safe here because each is preceded by `sync_file_to_store`, so the DB body is the file body plus the section just appended. The MCP `deprecate` handler needed new code: `render_ projection_record` had no forcing parameter at all, so this adds `render_projection_record_with_body` / `render_after_mutation_with_ body` alongside it. `render_projection`'s default stays files-first -- flipping it would break the user-edit protection it exists for. Tests: the existing lifecycle unit tests (`lifecycle/mod.rs:748/801/ 910`) assert through the store, which is the half that was never broken -- that is why this shipped. Four new CLI integration tests read the `.md` off disk instead, including one that runs a second lifecycle command afterward to catch the permanent-loss half. A source-grep invariant (`lifecycle_body_projection_invariant.rs`) pins each call site to the forcing variant so a future edit cannot silently regress it back. Every fix mutation-tested: reverting any of the three CLI call sites or the MCP call site fails the matching test, not a coincidental one. Refs: #478 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng it
`tests/embedding_reference.rs` pins the embedding engine's output against
values captured before the v0.35.0 ONNX -> tract swap. It has run in CI
zero times since it was written: the file is entirely behind
`semantic-search`, `check`/`clippy` compile with the feature but never
execute a test, and `nextest run --workspace --all-targets` runs without
the feature at all -- the assertions were never even compiled into that
invocation. A green CI has meant nothing about whether the engine is
correct since the feature existed.
New `test-embedding-oracle` job: a dedicated `cargo nextest run -p
forgeplan-core --features semantic-search --test embedding_reference`,
with the ~2.1 GB model cached across runs via `actions/cache` keyed on
the model repo name (the `check` job's own comment says downloading it
per run "is not worth it" -- caching removes that trade-off instead of
accepting it).
A cache alone would reproduce the same defect one layer down.
`embedder_or_skip` returns `None` on a missing model and each test
`return`s early -- which nextest reports as PASS, not skipped. A cold or
broken cache in this job would go green having asserted nothing, same
shape as the bug it exists to close. `FORGEPLAN_REQUIRE_MODEL_IN_TESTS=1`
turns that `None` into a panic in this job specifically; local runs
without the var keep the quiet skip, so a developer without the model
isn't blocked from the rest of the suite.
The env-var decision itself is pulled into a pure function
(`require_model_in_tests_from`) and unit-tested without touching process
env, following the precedent `resolve_cache_dir_from` already set for
the same "env mutation in tests is unsafe in 2024, take it as an
argument" reason.
Verified, not assumed:
- the three real assertions execute and pass with the model present
(13.7s, not an early return)
- forcing `Embedder::new()` to `Err` with the require-flag set panics
the job (checked directly, not inferred from reading the branch)
- without the require-flag, the same forced failure quietly passes
(reproduces the exact pre-fix defect on demand)
- nextest's own `--test <name>` filter already exits nonzero
("error: no tests to run") if the feature ever gets dropped from this
job and the file compiles to zero tests -- confirmed locally, so no
redundant grep-based guard was added on top of it
Refs: PROB-102, PRD-086
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ad race Found on this job's own first run against `dev`: `dimension_is_unchanged` and `embeddings_match_the_captured_reference` both call `Embedder::new()`, and nextest's default concurrency ran them at the same time. Against a genuinely empty cache both started fetching the same ~2.1 GB model into the same directory at once -- one finished in 62s, the other failed 7s in trying to fetch `onnx/model.onnx_data`. Once `actions/cache` has populated the directory this cannot recur -- `find_snapshot` returns immediately, no write happens -- but the first run after any cache eviction hits it every time. That run reporting red for a race rather than a real defect is exactly the shape that trains people to re-run and ignore rather than investigate, which is the opposite of what this job exists for. `--test-threads=1` removes the only concurrent-download path in this 2-test file; parallelism buys nothing here anyway. Refs: PROB-102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #478.
The bug
forgeplan deprecate <id> --reason "..."echoed the reason back andforgeplan getshowed it — the markdown file never received the## Deprecationsection, only the status.renewandreopenlost## Renewal/## Reopenedthe same way.Root cause — not a stale read, a collision between two correct behaviours
projection::render_projectionis files-first by design (RFC-004,force_body = false): if the file already has a non-empty body, the passed body is discarded so a user's on-disk edits survivelink/tag/activate.deprecate/renew/reopenappend a section viastore.update_bodyand then render through that same files-first path — so the section never reaches the file. Status still projects because it lives in frontmatter, which is why this looked like a partial success rather than a total failure.The loss is permanent, not temporary.
read_file_body_if_newercompares content, not mtime. After a deprecate, file and DB bodies differ; the next mutation on that artifact (every lifecycle command opens with a file→store sync) writes the section-less file body back over the DB, erasing the reason from LanceDB too.Fix
Three CLI call sites (
deprecate.rs:49,renew.rs:29,reopen.rs:50) switched to the existingrender_projection_with_body— safe because each is preceded bysync_file_to_store, so at mutation time the DB body is exactly the file body plus the appended section.render_projection's default is untouched — flipping it would break the user-edit protectionlink/tag/activatedepend on.The MCP
deprecatehandler needed new code, not a flag:render_projection_recordhad no forcing parameter, so this addsrender_projection_record_with_body/render_after_mutation_with_bodyalongside the existing ones.reopen's new-artifact render is deliberately left plain — its file doesn't exist yet at render time, so the non-forcing path already keeps the passed body.Tests
The existing unit tests (
lifecycle/mod.rs:748/801/910) assert through the store — the half that was never broken, which is exactly why this shipped.grep -rn "Deprecation|Reopened|Renewal" crates/*/tests/returned zero hits before this PR: nothing opened the.md.Four new CLI integration tests in
cli_integration_test.rsread the file off disk:deprecate_writes_the_reason_into_the_markdown_fileafter_deprecate_the_file_and_the_store_agree— runs a second lifecycle-adjacent check to catch the permanent-loss halfreopen_writes_its_reason_into_the_retired_artifacts_filerenew_writes_its_reason_into_the_markdown_filePlus a source-grep invariant,
lifecycle_body_projection_invariant.rs, pinning each call site to the forcing variant so a future edit can't silently regress back to the plain renderer.Mutation-tested: reverted each of the three CLI call sites and the MCP call site independently — each reversion failed the matching test (both the behavioural test and the invariant test), never a coincidental one.
Verification
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p forgeplan-core --features test-helpers --no-fail-fastcargo test -p forgeplan --no-fail-fastcargo test -p forgeplan-mcp --no-fail-fastThe 4 core failures are
git::tests(#454), an unrelated known flake — pass 51/51 serially, this diff doesn't touch that module.Recovery
Anyone hit by this before the fix:
updateprojects correctly, soforgeplan update <id> --body @path(full body, frontmatter stripped, section appended by hand) restores it. Done for PROB-105 in009de50.Evidence-check bypass note. The
pre-pr-evidence-checkhook flagged PRD-200,PRD-201, RFC-022 as unevidenced. Those are not real artifacts — they're test-scenario
IDs quoted in an earlier commit's prose (a mutation-testing walkthrough for FR-002 on
dev), and the hook'sgit log -20window happens to reach it from this branch.forgeplan get PRD-200/RFC-022both return "Artifact not found". This PR's ownevidence is PROB-105 / #478, already filed and corrected in the issue thread.
Second fix on this branch: PROB-102 — the embedding oracle has never run
Same theme, found while scoping the v0.37.0 release:
tests/embedding_reference.rspins the embedding engine's output against pre-tract values, and has run in CI zero
times since it existed. The file is entirely behind
semantic-search;check/clippycompile it,
nextest run --workspace --all-targetsruns without the feature at all,so the assertions were never even compiled into that invocation. Green CI has meant
nothing about engine correctness since the feature existed.
New
test-embedding-oraclejob runscargo nextest run -p forgeplan-core --features semantic-search --test embedding_referencein isolation, with the ~2.1 GB modelcached across runs via
actions/cache(the existingcheckjob comment saysdownloading it per-run "is not worth it" — caching removes that trade-off instead of
accepting it).
A cache alone would reproduce the same defect one layer down.
embedder_or_skipreturns
Noneon a missing model and each testreturns early — nextest reportsthat as PASS, not skipped. A cold or broken cache in this new job would go green
having asserted nothing, the exact shape of the bug it exists to close.
FORGEPLAN_REQUIRE_MODEL_IN_TESTS=1turns thatNoneinto a panic in this jobspecifically; local runs without the var keep the quiet skip.
The env-var decision is pulled into a pure function (
require_model_in_tests_from)and unit-tested without touching process env — same reasoning
resolve_cache_dir_fromalready established for env mutation being
unsafein Rust 2024.Verified, not assumed:
early return)
Embedder::new()toErrwith the require-flag set panics the job(checked directly by injecting the failure, not inferred from reading the branch)
exact pre-fix defect on demand)
--test <name>filter already exits nonzero (error: no tests to run) if the feature ever gets dropped from this job — confirmed locally, so noredundant grep-based guard was added on top of it
Refs: PROB-102, PRD-086