fix(code-index): diagnose graph-rebuild timeout and withhold a lagging seat

[ScriptedAlchemy]

Summary

• Diagnosis of graph_rebuild_status_test::background_refresh_and_reopen_report_only_servable_generations, which fails both nextest attempts at the 90s receipt (crates/tracedecay/tests/transport_acceptance_suite/graph_rebuild_status_test.rs:176).
• The page-limit tweak in 7bdc33d3cb is not the failure. Both receipts have truncation counters at 0.
• The two attempts do not share disk state. They share one cause — search keeps serving the predecessor while status advertised the replacement text owner — and they stop at different points on that path.
• This PR withholds terminal freshness until the serving seat matches the advertised text generation. It does not make the 90s journey green. The serving swap still waits out corpus-sized graph activation / clone backfill.

Targets codex/tracedecay-total-redesign-plan-reopened (PR #707). Does not touch main/master.

Evidence

Linux root-transport nextest report from CI run 35258386278 (commit 11ab3edc20, after the limit=3 tweak). Artifact nextest-junit-Linux-root-transport. Both attempts panic at line 176 (timed out waiting for current generation). Temp roots differ (/tmp/.tmpXiBdAX vs /tmp/.tmpOM98kG).

	Attempt 1 (96.7s)	Attempt 2 (92.2s)
Freshness	current / complete / fresh	stale / partial_source_verification / verifying
Search	never returned (search=null; completed tracedecay_search stayed at 2)	returned; served …00000001, latest …00000002; results: []
Graph bind	not reached	verified-code-graph-generation-mismatch (retryable)
Clone index	backfilling 0/2305	backfilling 1024/2305
Graph census	last_complete_stale, edge_count: 0, symbol_count: 98305	current, same counts
code_graph_serving	ready	ready
Progress	ready, 769/769 files, rebuild_in_flight: false	same
Extra	graph_superseded_replay_retirement_failed: parent /tmp/.tmpXiBdAX/profile/projects/proj_94c986a0cfdce935 missing (os error 2), stage publish, ~1s before the panic	warning absent

symbol_count 98305 is 768*128+1. edge_count 0 is collect_edge_evidence over functions that do not call each other (crates/tracedecay-code-index/src/production/mod.rs:2168), not a missing projector. Rejected as a cause.

Ranked causes

1. Serving seat lags the text owner that status advertises (shared cause). A publish writes the replacement text owner into worker_text_generation before the serving swap (crates/tracedecay-code-index-runtime/src/code_index_scheduler/registry/mount.rs:1059-1087). Freshness identity is taken from that text owner (serving_reads.rs, dashboard_text_freshness_identity). dashboard_generation_is_ready treated ready text lanes plus CodeGraphServingReadinessV1::Ready as terminal, without comparing seat ids. Search executes the serving slot and, when lanes are stale, reports CodeIndexLaneStatusV1::Stale (query_runtime.rs:385-388). The verified graph is already the head, so bind_verified_graph_to_search returns verified-code-graph-generation-mismatch (crates/tracedecay-mcp/src/handlers/graph/search_evidence.rs:61-76). Attempt 2 did this on ~32 searches. Attempt 1 only became current as the 90s window expired, so the search await was cancelled (search=null). This PR stops calling that split current. It does not move the seat.

2. The swap is skipped while graph activation or text projection is unfinished (why 90s is not enough). Retryable graph activation clears the seat candidate (mount.rs:1641-1671: result = Ok((Err(error), None, None)), “stays unseated until the scheduled retry”). An Unfinished text projection also clears latest so the swap predicate fails (mount.rs:1775-1786). Clone-fingerprint backfill is deliberately not started until the seat matches the text owner (mount.rs:590-611). The fixture is 768 files × 128 functions: 2305 clone pages, ~645–699 MiB, completed_lexical_units 455672760. Attempt 1 never left page 0; attempt 2 reached 1024/2305 and still served the predecessor. Not fixed here.

3. Mismatch is a typed symptom, not a second root. race_primary_search_with_graph waits for the graph on a sparse page (search.rs:238-244, should_check_external_import_hint is true when result_count < limit). Binding then refuses a newer graph. Empty results is because refresh_probe_0000_000 is not in generation 1, not because MCP truncated the page.

4. Page-limit / MCP truncation — rejected. 7bdc33d3cb dropped limit from 20 to 3. This CI run has irreversible_truncation_total: 0, store_attempts: 0, bytes_before_truncation_total: 0. Attempt 2 returned a full JSON body.

5. Missing graph-db parent — attempt 1 only, not the timeout. retire_superseded_replays logs and swallows GraphDbError::Unavailable (crates/tracedecay-store-runtime/src/session_registry/code_graph.rs:594-601; resolve at crates/tracedecay-graph-db/src/registry/path.rs:64-69). Different temp dir, absent from attempt 2, timestamped at teardown. Do not treat it as shared side effect. Nextest retries = 1 starts a new process (.config/nextest.toml).

Error taxonomy

Code-graph projection (CodeGraphProjectionError, graph_projection.rs:71)

Variant	Retryable activation (reconcile.rs:291)	Read mapping (graph-query/src/projection.rs:256)	MCP reason (map_code_graph_read_runtime_error)
Contract	no	InvalidRequest	code-graph-invalid-request
GenerationMismatch	no	Stale	code-graph-stale (retryable)
Cancelled	yes	Cancelled	code-graph-cancelled
BudgetExhausted	yes; resident-memory budget is a refusal (is_graph_activation_refusal)	BudgetExhausted	code-graph-budget-exhausted
DeadlineExceeded	yes	TimedOut	code-graph-timed-out (retryable)
Conflict	yes, unless the same conflict repeats (activation_conflict_context)	Unavailable	code-graph-unavailable (retryable)
ProjectionMismatch / RecoveredGenerationMismatch	no	corrupt / stale	code-graph-corrupt / code-graph-stale
ResetRequired	no	ResetRequired	reset-required authority, not retryable
Corrupt	no	Corrupt	code-graph-corrupt
Unavailable	yes; also the swallowed retirement error	Unavailable	code-graph-unavailable
DurabilityUncertain / Closed	Closed is retryable	Unavailable	code-graph-unavailable

GraphDbError (graph-db/src/error.rs:88) maps into those variants in CodeGraphProjectionError::from. Extra graph-db variants SourceCommitmentsUnavailable and SealedRevisionIncompatible become Unavailable and are not retryable activation.

Search / status surfaces (what the 90s test actually sees)

Surface	Where	What the receipt hit
verified-code-graph-generation-mismatch	search_evidence.rs:67, used by handle_search (search.rs:294-296) and the context path (search.rs:814)	attempt 2
verified-code-graph-read-unavailable	search_evidence.rs:50 when primary search finishes first and the graph is not required	not this receipt
freshness current / warming / stale / parked	handlers/info/status.rs:451	attempt 1 current; attempt 2 stale + warning “verifies source freshness”
retrieval last_complete_stale + source_verification	status.rs:284-286	attempt 2
ServingSwapOutcomeV1 Seated / SeatedStale / Superseded / Offered	registry.rs decide	swap never installed the head within the window
GraphActivationGateV1 Activate / AlreadyServing / UnchangedGraph / PendingAttemptSpent	registry.rs:272	activation can finish on the text handle while the slot stays old
CLI	tracedecay-cli/src/status_cmd.rs decodes the same graph_statistics + code_index_freshness; commands/index.rs calls tracedecay_admin_sync	no separate rebuild error enum
Test	transport_acceptance_suite/graph_rebuild_status_test.rs	the failing journey. fact_store/curation/tests.rs graph_rebuild_rejects_a_dangling_canonical_link_event is a different store and not this timeout

Daemon path is the scheduler worker in registry/mount.rs, entered by tracedecay_admin_sync (status: queued in the receipt). MCP search is handle_search. There is no separate CLI graph-rebuild subcommand.

Test plan

• scripts/require-exact-test.sh cargo test -p tracedecay-code-index-runtime --lib -- graph_rebuild_split_is_not_terminal_freshness dashboard_ready_requires_text_and_graph_lane_owners — 2 passed.
• background_refresh_and_reopen_report_only_servable_generations was not re-run. CI already shows it red at 90s both attempts. This PR does not install the serving seat, so that journey is still expected to time out. Do not raise RECEIPT_TIMEOUT.
• Workspace nextest / clippy not run (cold compile of this crate only).

Next fix PRs (not this change)

1. Seat text without waiting out graph activation. mount.rs:1641-1671 drops the swap candidate on retryable graph activation. Exact/lexical should seat, and graph activation should retry beside that seat. Today the comment at 1582 says the swap still installs, then the error arm deletes the candidate.
2. Do not treat an unfinished clone-fingerprint successor as “text projection unfinished” if exact/lexical owners are already ready (mount.rs:1704-1718 states that intent; 1775 still clears the seat). The 2305-page backfill is the wall-clock cost. Measure it before changing any budget.
3. After (1), keep bind_verified_graph_to_search strict. If the seat is current, the mismatch stops happening. Do not special-case the probe query.
4. Separately, graph_superseded_replay_retirement_failed on a missing projects/proj_<prefix> directory is a swallowed Unavailable. Confirm whether that parent is never created or is dropped during shutdown. Do not conflate it with the timeout.

Checklist

• No secrets
• CHANGELOG not updated (diagnosis + freshness verdict; not a user-facing release note)
• Breaking: status can stay warming while the text owner is ready and the serving slot is still the predecessor. That is the corrected contract.

  

[changeset-bot]

⚠️ No Changeset found

Latest commit: 5cf3482

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[github-actions]

Performance Comparison codex/tracedecay-total-redesign-plan-reopened → cursor/graph-rebuild-failure-taxonomy-f5c7

Total Elapsed Time: 5.44s → 5.12s (-5.8%)
CPU Baseline: 61.62µs → 59.16µs (-4.0%)
Benchmark ID: index-bench-timing

timing - Execution duration of functions.

+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| Function                                 | Calls                      | Avg                              | P95                              | Total                            | % Total                      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| tracedecay-index-bench                   | 1 → 1 (+0.0%)              | 5.44s → 5.12s (-5.9%)            | 5.44s → 5.12s (-5.9%)            | 5.44s → 5.12s (-5.9%)            | 100.00% → 100.00% (+0.0%)    |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.sqlite              | 5 → 5 (+0.0%)              | 297.36ms → 289.33ms (-2.7%)      | 336.33ms → 319.82ms (-4.9%)      | 1.49s → 1.45s (-2.7%)            | 27.34% → 28.25% (+3.3%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| domain.canonical.sha256                  | 72891 → 72891 (+0.0%)      | 20.32µs → 18.87µs (-7.1%)        | 64.67µs → 62.43µs (-3.5%)        | 1.48s → 1.38s (-6.8%)            | 27.23% → 26.87% (-1.3%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.workers.install               | 30 → 30 (+0.0%)            | 51.12ms → 43.77ms (-14.4%)       | 157.16ms → 115.54ms (-26.5%) 🚀  | 1.53s → 1.31s (-14.4%)           | 28.20% → 25.64% (-9.1%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.finalization.advance_wake | 28 → 28 (+0.0%)            | 33.83ms → 33.46ms (-1.1%)        | 252.31ms → 246.94ms (-2.1%)      | 947.22ms → 936.74ms (-1.1%)      | 17.42% → 18.29% (+5.0%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.postings            | 5 → 5 (+0.0%)              | 165.66ms → 160.49ms (-3.1%)      | 187.17ms → 181.27ms (-3.2%)      | 828.31ms → 802.45ms (-3.1%)      | 15.23% → 15.67% (+2.9%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.build.and_publish             | 2 → 2 (+0.0%)              | 326.22ms → 321.45ms (-1.5%)      | 600.83ms → 590.87ms (-1.7%)      | 652.45ms → 642.91ms (-1.5%)      | 12.00% → 12.56% (+4.7%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.extract.parser_artifact       | 276 → 276 (+0.0%)          | 2.28ms → 2.29ms (+0.4%)          | 3.72ms → 3.72ms (+0.0%)          | 629.59ms → 631.07ms (+0.2%)      | 11.58% → 12.32% (+6.4%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.parallel_prepare    | 5 → 5 (+0.0%)              | 133.64ms → 106.55ms (-20.3%) 🚀  | 157.16ms → 115.54ms (-26.5%) 🚀  | 668.19ms → 532.73ms (-20.3%) 🚀  | 12.29% → 10.40% (-15.4%)     |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| domain.canonical.json_bytes              | 206681 → 206681 (+0.0%)    | 3.04µs → 2.44µs (-19.7%)         | 2.65µs → 2.58µs (-2.6%)          | 628.22ms → 504.14ms (-19.8%)     | 11.55% → 9.85% (-14.7%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.postings.ngram_rows | 5 → 5 (+0.0%)              | 93.68ms → 91.83ms (-2.0%)        | 105.25ms → 103.55ms (-1.6%)      | 468.41ms → 459.13ms (-2.0%)      | 8.61% → 8.97% (+4.2%)        |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.restore.file_admit            | 260 → 260 (+0.0%)          | 2.30ms → 1.73ms (-24.8%) 🚀      | 5.58ms → 2.87ms (-48.6%) 🚀      | 598.35ms → 450.81ms (-24.7%) 🚀  | 11.00% → 8.80% (-20.0%)      |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.build.materialize_full        | 1 → 1 (+0.0%)              | 413.10ms → 406.84ms (-1.5%)      | 413.14ms → 406.85ms (-1.5%)      | 413.10ms → 406.84ms (-1.5%)      | 7.60% → 7.95% (+4.6%)        |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| code_index.collect.materialize_full      | 1 → 1 (+0.0%)              | 408.22ms → 402.30ms (-1.5%)      | 408.42ms → 402.39ms (-1.5%)      | 408.22ms → 402.30ms (-1.5%)      | 7.51% → 7.86% (+4.7%)        |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.commit              | 5 → 5 (+0.0%)              | 79.65ms → 78.73ms (-1.2%)        | 93.39ms → 90.96ms (-2.6%)        | 398.23ms → 393.65ms (-1.2%)      | 7.32% → 7.69% (+5.1%)        |
+------------------------------------------+----------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+

---

Generated with hotpath-rs