Skip to content

fix(code-index): resume a successor's graph seat after restart - #2897

Closed
devin-ai-integration[bot] wants to merge 2 commits into
masterfrom
devin/1790910874-resume-interrupted-refresh
Closed

devin-ai-integration[bot] wants to merge 2 commits into
masterfrom
devin/1790910874-resume-interrupted-refresh

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • If the daemon gets SIGTERM after a successor generation's publication pointer moves but before its graph seats, the restart now resumes that same generation. Its text continues from the durable staging cursor, its graph publication resumes from the sealed segments, and it reaches current. Before this change, graph serving stayed pending and status stayed warming/indexing with no end.
  • The restart redoes only unfinished work. It seals no new generation and re-extracts nothing, and text files that were already committed are not rebuilt.

Motivation

Refs #1226 (restart after SIGTERM mid-refresh rebuilds or never finishes). This builds on current master, after #2851 withdrew the parent-carried lexical artifact path. Nothing from that path comes back.

Root cause, from a kill/restart probe and the daemon log:

  1. The successor's text artifact already resumes per page batch. With a kill at bulk_commit 22/98, the restarted daemon continues at 22/98, so text receipts are already durable per batch.
  2. The batch that never resumes is the successor's graph seat. At startup, verified-head recovery (registry/mount.rs, added in e0fb1eb for code-index: store-lock contention spins follow-up reconciles; graph-head conflict forces full rebuild #2797) compares the retained manifest (the successor) with the durable graph head (still the older generation). It hits code_graph.recover_verified_snapshot_from_head.generation and reads the mismatch as "an in-flight successor owns the head". It then sets graph_head_conflict_generation, and every later retained pass skips with graph_head_belongs_to_another_generation. Nothing else ever publishes that generation's graph:
WARN ... event="code_index_graph_head_recovery_generation_conflict" ... `code_graph.recover_verified_snapshot_from_head.generation`
DEBUG ... event="code_index_graph_seat_skipped" reason="graph_head_belongs_to_another_generation"   (repeats; code_graph_serving=pending for >60 s)

To tell the two cases apart: a graph head is published only for a generation that the durable publication pointer has already named, and the pointer only moves forward. So if the retained manifest is the pointer's generation, a head on any other generation is older, which means this generation's own graph publication was interrupted. The #2797 case is different: the worker holds a stale retained owner while the pointer already names a newer successor. That case keeps the existing skip.

Changes

  • code_index_scheduler/graph_activation.rs: adds CodeIndexWorktreeSchedulerV1::names_active_publication(&CodeGenerationId) -> Result<bool, CodeIndexSchedulerErrorV1>, which reads the durable publication pointer. Errors stay typed.
  • code_index_scheduler/registry/mount.rs: the retained-recovery spawn_blocking returns (replay_binding, retained_names_active_publication). A new arm comes before the existing conflict arm:
    Err(e) if graph_head_belongs_to_another_generation(&e) && retained_names_active_publication
        => info!(code_index_graph_head_recovery_resumes_publication); fall through to the existing sealed-segment replay/publish
    Err(e) if graph_head_belongs_to_another_generation(&e)
        => unchanged #2797 skip (a newer generation owns the pointer)
    
  • registry.rs: doc comment on graph_head_belongs_to_another_generation changed so it no longer overstates what the conflict means.
  • tests/daemon_suite/indexing_lifecycle_test.rs: new test sigterm_after_successor_publication_resumes_its_unfinished_text_and_graph. The journey:
    1. Index a real git fixture, then install a 96-file batch and inject an overflow.
    2. Wait until the successor is the durable latest generation with partly applied text (0 < completed_files < total_files).
    3. SIGTERM the daemon and restart it.
    4. Assert that no restart progress for that generation reports fewer completed files than before the kill (finished files are not redone).
    5. Assert that the generation that becomes current is the same successor (nothing was re-extracted or resealed), graph serving is ready, a batch symbol (cancellation_probe_0095_127) resolves, and the original lifecycle_main_symbol still resolves from src/lib.rs.

No timeout was raised and no test bound was loosened (the existing RECEIPT_TIMEOUT is used). SQLite mmap is unchanged.

Test plan

Red before the fix (CLI built from unmodified production code): the new test failed after 47.40 s with restart did not make the retained successor current. Status showed code_graph_serving.state: "pending" while the successor's text was at 98/98.

Green after the fix:

Command Result
cargo fmt --all -- --check pass
cargo clippy -p tracedecay-code-index-runtime --all-targets -- -D warnings pass
cargo clippy -p tracedecay --all-targets --features test-helpers,test-transport -- -D warnings pass
cargo build -p tracedecay-cli --bin tracedecay pass
cargo test -p tracedecay --test daemon_suite --features test-helpers,test-transport --no-run pass
target/debug/deps/daemon_suite-<hash> --exact indexing_lifecycle_test::sigterm_after_successor_publication_resumes_its_unfinished_text_and_graph 1 passed; 0 failed, 3 of 3 runs
target/debug/deps/daemon_suite-<hash> --exact indexing_lifecycle_test::mounted_incremental_lifecycle_preserves_only_complete_compatible_generations 1 passed; 0 failed
cargo test -p tracedecay-code-index-runtime --lib code_index_scheduler::registry 52 passed; 0 failed; 513 filtered out
  • cargo nextest run --workspace --no-fail-fast passes (not run locally; only the affected suites above were run)
  • cargo clippy has no new warnings (touched crates, -D warnings)
  • Tested with a real daemon kill/restart journey (above)

Not covered here: a kill that lands before the pointer moves (during extraction or seal). In that case nothing about the successor is durable yet (staged segments are temporary files that get removed), so the restart rebuilds that batch from the retained complete generation, as it did before this change.

Checklist

  • CHANGELOG.md updated (generated from conventional commits)
  • No secrets, credentials, or .env files included
  • Breaking changes documented (none)

Link to Devin session: https://app.devin.ai/sessions/c35915f46ae44002baff5733673cb8bf
Open in Devin Desktop: https://app.devin.ai/desktop/session/c35915f46ae44002baff5733673cb8bf?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

Local verification (land-devin-prs): not landed

Rebased on master 6f99af4; CLI built from each variant (cargo build -p tracedecay-cli --bin tracedecay, TRACEDECAY_TEST_BIN set), daemon_suite --exact indexing_lifecycle_test::sigterm_after_successor_publication_resumes_its_unfinished_text_and_graph:

  • master production + this test: 5/5 FAILED, all at the precondition successor never exposed partially applied text progress (never reaches the restart).
  • this PR: 4/4 FAILED at the same precondition.

The final status in the failure shows the successor already current, progress.completed_files: 98 of 98 after ~2.5 s, so no 25 ms status poll observed 0 < completed < total on this host. The test races the product for an intermediate state and is not falsifiable as written: the fix has no fail-before/pass-after evidence here. It needs a deterministic interruption point for the successor's text (for example the restart test reaching the same state through the real overflow/cancel path at a receipt it can wait on, rather than a progress window) before it can land.

A SIGTERM after text publication could leave the graph head on the prior generation. Distinguish an interrupted active generation from a stale retained manifest by checking the durable publication pointer.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 5ca1e00

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

ScriptedAlchemy added a commit that referenced this pull request Oct 2, 2026
…2927)

e0fb1eb (#2842) made retained graph-head recovery treat any head on
another generation as an in-flight successor that owns the head, and
stopped replaying that retained manifest for good. When the durable
publication pointer names the retained manifest, the head is older: the
manifest is the newest publication and its graph never seated. A stale
ignored-dependency admission that loses to a competing reconcile leaves
exactly that state, so the competing generation never served.

Recovery now reads the publication pointer and resumes the replay when
it names the retained manifest. The #2797 case, a stale retained owner
behind a newer pointer, keeps the skip. The production change is the
one in #2897, applied unchanged.

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Closing as superseded by #2927, which merged with the same fix: replaying the published generation over an older head so a successor resumes its graph seat after restart. Thanks for the original diagnosis and repro; #2927 builds on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant