fix(sessions)!: defer Codex import, search newest day first - #2441
Merged
Merged
Conversation
|
cursor
Bot
force-pushed
the
cursor/codex-catchup-walk-335c
branch
from
September 28, 2026 08:49
4d14e35 to
b812d26
Compare
ScriptedAlchemy
marked this pull request as ready for review
September 28, 2026 08:56
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Shared catch-up charged every rollout file as directory work and then shrank that walk to the JSONL parse-slot count, so a large Codex home listed older days for minutes before emitting any session. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
sessions import waited for the historical cycle it had just queued. On a large Codex home that cycle does not finish inside the existing deadline, so the command exited timed out while admission was still moving. BREAKING CHANGE: A transcript import whose catch-up is still pending finishes as partial coverage with remaining work, instead of timed_out. Git sync still requires its own pass to finish. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
The project Codex pass wrote a coverage cursor for every out-of-scope rollout before it returned, so projection could not activate the newest day until that sweep finished. Yield once an in-scope window is admitted, and finish its projection backlog before the next history window. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
cursor
Bot
force-pushed
the
cursor/codex-catchup-walk-335c
branch
from
September 28, 2026 09:31
b812d26 to
283b48e
Compare
A newest-day yield is retryable backpressure. The worker treated that as a no-progress backoff and opened the older-day sweep after one projection slice, leaving the newest day on a building generation. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
Codex catch-up persists observations without ingest counters, so the yield is a no-progress backoff. Publish the projection backlog before that backoff opens the older-day sweep. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
A history wake that arrives during the newest-day window was starting the older-day sweep on the next pass. While that window is unpublished, ignore the wake and keep projecting. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
cursor
Bot
changed the title
An import read the worker's serving state before its wake, so a converged profile reported `completed` with no remaining work while rollouts written since the last pass were still unadmitted. Scheduling now marks history pending first and every hand-off is partial with one deferred unit per store; blocked or missing workers stay failures. A project Codex pass that persisted in-scope frames now yields at the next day directory instead of at the next out-of-scope rollout. On an interleaved corpus the old rule ended nearly every pass after a few rollouts. Removes the waiter's leftover progress counters, history sequence, and membership pre-read.
A per-day yield ended every pass of a project whose history is all in scope, so the discovery frontier never committed and each pass re-read the days before it. The pass now yields only at a day directory that opens with an out-of-project rollout after in-scope frames persisted; mixed days and in-scope days stay in one pass.
The history pass yields after in-scope frames persist so that a newest-first page does not cursor every older out-of-scope day before the admitted window is searchable. Any persisted frames set persisted_day, including appends to a rollout that was already cursored. With an active Codex session writing to today's rollout between passes, every pass ended at the first older out-of-scope day with the frontier uncommittable, so the hub redelivered the same page and older in-scope days were never reached. Only a rollout opened for the first time this pass now arms the yield. A resumed tail already has its earlier window projected, so its appends no longer end the pass.
A 1s wall-clock bound on the import settling is a real-time assertion that a 4-vCPU CI runner under load can miss. 10s still proves the request returned the settled catch-up instead of consuming its 60s deadline.
A deferred import can expose a locally fresh but empty projection while its historical worker still has sources to ingest. Treat that zero as stale until catch-up settles, and keep an end-to-end assertion on the first empty search response.
ScriptedAlchemy
changed the title
…s-sessions-2441 # Conflicts: # crates/tracedecay-cli/tests/core_cli_suite/tool_surface_transport_test.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sessions importhands historical catch-up to the refresh workers and returns a typed partial receipt: one deferred unit per store scope. It never reportscompletedfor work it has not done, and it no longer ends astimed_outwhile the daemon keeps admitting. Blocked or missing workers stay failures.stale, notcomplete_zero/fresh(sessions: message search reports complete_zero/fresh during catch-up #2512).Review changes in this round
4179085161raised the import hand-off test bound from 1s to 10s. I replaced the wall-clock bound entirely. The fixture runs no worker loop, so an import that waited for its pass could only end at the 60s deadline astimed_out. The tests now assert the literalpartialtermination and that the history wake is still queued.d09b67592c(CI Windows target) is already on master, so it adds nothing to the net diff.a846b169f9(sessions: message search reports complete_zero/fresh during catch-up #2512) now has a deterministic test,session_retrieval::tests::empty_answer_is_stale_until_historical_catch_up_is_current. The same empty store answersCompleteZerobehind aCurrentserving port andStalebehind historical convergence. Witha846b169f9'sadmitted.rshunk reverted, it fails withCompleteZero { ..watermarks all 0.., freshness: Fresh }, the exact sessions: message search reports complete_zero/fresh during catch-up #2512 answer.core_cli_suite::tool_surface_transport_test::application_surface_primitive_tools_resolve_the_working_directory_projectcalledsource_outlineright afterinit.initreturns before the code graph serves; master loses this race too. Measuredinit-to-served: master 0.44s, 0.12s, 0.11s, with the first call unavailable in run 1; this branch 0.32s, 0.29s, 0.28s. Master's fix(daemon): serve cold-daemon reads once their owner is ready #2612 fixed the same race independently withawait_graph_ready, and this branch takes that version.Evidence (local; debug CLI; isolated
HOMEand XDG dirs; one daemon capped withsystemd-run --user --scope -p MemoryMax=6G)9000 synthetic Codex rollouts over 90 days, newest day in the project, current head:
sessions import: exit 0 after 0.15s,session import scheduled (...); historical catch-up has remaining work 2no messages matched ... (status: stale ...)2150c91ed8: import exit 1 after 35.6s, first hit 402.6s)On a 3000-rollout corpus where the project is interleaved within each day, the previous round measured 1510 raw messages by about 610s. Master reached 853 in the same time with no search hit.
Tests that fail on master (with only the test code applied) and pass here:
session_sync::import_admission_testsnewest_day_messages_are_durable_before_older_days_are_openeda_mixed_day_is_admitted_in_one_pass_before_older_days_are_openedsession_import_accepts_deferred_catch_up_without_treating_it_as_failureempty_answer_is_stale_until_historical_catch_up_is_current, shown to fail without the fixSuites on the merged head:
session_suite440/440,transcript_ingest_suite193/193, fullmcp_suite607/607application_suite66/66,core_cli_suite167/167core_cli_suite169/169, session-filteredmcp_suite29/29 andsession_suite108/108.cargo clippy -D warnings(sessions, session-runtime, cli, tracedecay with test features) passes, andcargo fmt --checkpasses.Fixes #2512
Fixes #2390
BREAKING CHANGE:
sessions importschedules historical catch-up and finishes as partial coverage with one deferred unit per store scope; it no longer waits for the pass or reportscompleted/timed_out. An empty session search during historical convergence answersstaleinstead ofcomplete_zero.