Repository navigation
fix(code-index): replay published generations over older heads - #2927
Conversation
e0fb1eb (#2842) made retained graph-head recovery treat any head on another generation as an in-flight successor that owns the head, and stopped replaying that retained manifest for good. When the durable publication pointer names the retained manifest, the head is older: the manifest is the newest publication and its graph never seated. A stale ignored-dependency admission that loses to a competing reconcile leaves exactly that state, so the competing generation never served. Recovery now reads the publication pointer and resumes the replay when it names the retained manifest. The #2797 case, a stale retained owner behind a newer pointer, keeps the skip. The production change is the one in #2897, applied unchanged. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| Err(error) | ||
| if graph_head_belongs_to_another_generation(&error) | ||
| && retained_names_active_publication => | ||
| { |
There was a problem hiding this comment.
🔴 Newer graph head overwritten by stale replay
If publication advances during recover_verified_head, retained_names_active_publication remains true for the old generation. Its sealed replay can then replace the newer graph head while the pointer names the newer generation.
Learn more
The worker reads the durable text publication pointer while holding the scheduler mutex, then releases it before awaiting graph-head recovery. Another publisher can commit a successor and seat its graph during that await. A generation mismatch is then interpreted using the obsolete pointer snapshot, and the retained generation reaches publish_sealed_graph, which publishes using the graph store's current head rather than requiring that this generation still owns the text pointer. The graph can therefore regress behind the active text publication.
Example: The pointer initially names generation G2 and the graph head names G1. While G2's head recovery is awaiting, another publisher advances both to G3. G2's recovery sees G3 but the captured boolean remains true; G2 publishes its graph over G3 instead of deferring to G3.
Recommended fix: Revalidate active publication ownership at the point of replay admission and fence the graph-head commit against subsequent pointer changes. A second unchecked pointer read before the asynchronous graph publish only narrows, rather than closes, this race.
Was this helpful? React with 👍 or 👎 to provide feedback.
ScriptedAlchemy
changed the title
On master,
competing_reconcile_during_graph_activation_preserves_the_incumbentin daemon_suite fails every run. It times out after about 5.7 s with "serving generation ... did not seat: deadline has elapsed". The same failure shows up in the CI root-journeys lane.The breaking commit is e0fb1eb, "fix(code-index): wait out a held store lock on follow-ups", which landed in merge 1db05cb (#2842). I bisected master's first-parent history. 60fce0d and c6c2359 pass. 1db05cb, 555db0e, 06edb2f, 3cd1bb9, aa6af67, 10e84bf and 952a2fa all fail the same way. The later code-index commits (#2840 layered refresh, #2910, the page-segment commits) are not involved, because the failure is already present before each of them.
Root cause. e0fb1eb added a verified-head recovery arm in
registry/mount.rs. When the graph head belongs to a different generation than the retained manifest, the arm reads that as "an in-flight successor owns the head". It setsgraph_head_conflict_generation, and every later retained pass of that manifest skips withgraph_head_belongs_to_another_generation. In this test, a stale ignored-dependency admission loses to a competing reconcile. The publication pointer then names the competing generation 3, the retained manifest is generation 3, and the graph head is still the incumbent generation 1. The head is older, not newer, but the arm cannot tell the two cases apart. Generation 3 is never replayed and never seats. The debug log of the failing run showscode_index_graph_head_recovery_generation_conflictonce, thengraph_head_belongs_to_another_generationskips on every pass until the deadline.The fix reads the durable publication pointer during recovery. Graph heads only follow the pointer, and the pointer only moves forward. So when the pointer names the retained manifest, a head on any other generation is older, and recovery falls through to the normal sealed replay. The #2797 case is unchanged: there the worker holds a stale retained owner while the pointer already names a newer successor, and the skip still applies. With the fix, the same test logs
code_index_graph_head_recovery_resumes_publicationand generation 3 seats.This production change is the one in open PR #2897 (Devin, "resume a successor's graph seat after restart"), applied unchanged to
graph_activation.rs,registry.rsandregistry/mount.rs. So #2897 does fix this failure. Its restart test is the reason it has not landed yet, and that test is not included here. The hunks are identical, so #2897 should merge cleanly on top of this PR, or it can be reduced to its test. No test was changed or loosened here: the existing daemon_suite test is the deterministic regression check, failing before and passing after.Verification, run on this branch:
cargo test -p tracedecay --features test-helpers --test daemon_suite competing_reconcile_during_graph_activation: passed 3 of 3 runs in a row (it failed on master).cargo test -p tracedecay --features test-helpers --test daemon_suite code_index_ignored_dependencies_test: 21 passed, 0 failed.cargo test -p tracedecay-code-index-runtime --lib: 569 passed, 0 failed, 1 ignored.cargo clippy -p tracedecay-code-index-runtime --all-targets -- -D warnings: clean.cargo fmt --check: clean.Local verification (land-open-prs)
Verified on the branch merged with
origin/master(fa11c68af4).daemon_suite competing_reconcile_during_graph_activationpasses 3/3.serving generation … 00000003 … did not seat: deadline has elapsed.daemon_suite code_index_ignored_dependencies_test: 21/21.tracedecay-code-index-runtimelib: 569 passed (1 ignored). clippy-D warnings(all targets) and fmt are clean.