Repository navigation
Codex discovery re-offer costs about 28 ms per unchanged rollout in project scope, and profile scope re-offers on every restart #4374
Description
Summary
Measured while verifying PR #4366 on the shipped CLI, with 2,010 Codex rollout files (7.9 MB) and one registered project.
- Project scope: a full re-offer of unchanged rollouts takes about 62 s, about 28 ms per file. Admission itself totals under 1 s for 4,020 file reads. The rest is spent between files, and no tracing span covers that gap.
- Profile scope: the same re-offer takes about 4.7 s, about 1.3 ms per file.
- Profile scope re-offers on every restart. It offers all 2,010 files on each daemon restart, reading 0 bytes.
masterat e2be4c3 does the same (2,010 files in one restart, 7 in another). The project scope settles and re-offers nothing.
PR #4366 deliberately triggers one re-offer of every rollout after the upgrade. With this per-file cost, that one-time pass grows with corpus size times the number of registered projects.
Next step
Instrument the project-scope discovery loop per AGENTS.md (TRACEDECAY_SPAN_TIMINGS=1) to find what the 28 ms per file is spent on. Separately, find why the profile-scope completion claim does not match after a restart.
Activity
devin-ai-integration commentedon Oct 20, 2026
Profile-scope half: fixed in #4380.
Root cause: after a restart, the consumer that registers second (usually profile) joins a hub whose shared scan has already started, so
CodexDiscoveryHub::registerputs it inReplaymode. Replay delivery started from positionNoneand never compared the consumer's durable complete frontier withindex.frontier, so every indexed rollout was redelivered and then skipped by the source ledger (0 bytes read). Registration order decides which consumer is second, which explains why one restart re-offered 2,010 files and another only 7. Reproduced with a hub-level test: 21/21 rollouts re-offered before the fix, 0 after, and a newly added rollout is still delivered.Project-scope 28 ms/file: still open, not measured yet. From reading the code, the per-file path in
run_codex(project_provider.rs) isSourceSweep::observe_file→ (sometimes)codex_rollout_project_membership→try_admit_codex_jsonl_observations_for_project_window. Only the admission reads are covered by the spans quoted in the issue. A cost of about 28 ms per file looks like a durable store commit per rollout (cursor or convergence write) rather than CPU, but this is a hypothesis. The next step is aTRACEDECAY_SPAN_TIMINGS=1daemon run with spans aroundobserve_fileand the per-file store writes.Written by Devin
perf(sessions): avoid empty JSONL replay scope work
fix(sessions): make empty-page bypass Codex-specific
fix(sessions): require exact no-op cursor position
perf(sessions): skip Codex scope resolution for empty pages
ScriptedAlchemy commentedon Nov 8, 2026
Published a focused source fix and behavioral regressions in #4807, commit
408fdb7ec143e31e6ced01b75a2bcff1eca847ed(Refs #4374, not complete acceptance).The authoritative JSONL scan could return an empty native-frame page, yet Codex still dispatched blocking scope preparation and resolved/revalidated Git identities for the cwd. There is no frame to consume that verdict. The fix skips only that provider-state work; cursor checks, skipped-range coverage, pending rewrite completion and cancellation remain in the canonical admission path. No empty-page scope verdict is cached. Added
sessions.observation.codex.resolve_scopefor span-timing attribution.Two new production-path regressions use real Git/JSONL inputs and the existing in-memory admission fixture: 16 sources split between project/profile stores, 96 re-offers with a cold then warm matcher cache, zero resolver/capture work and stable accepted identities; then actual appends crossing out of and back into the project, source identity/scope verification, convergence and cancellation. These are authored assertions, not yet measured execution results.
Exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442355355 (queued when checked). The local authoring environment has no checkout/Bazel, so no Rust pass or speedup is claimed. Require nonzero Bazel execution of
//crates/tracedecay-sessions:unit_test, existing cache/rewrite/discovery regressions, independent review, and comparable affected-host 2,010-rollout traces. This does not remove the durable pending-rewrite read speculatively, and does not establish that it was the remaining 28 ms/file bottleneck.The patch changes only Codex admission and its new test module, with no file overlap with #4793. Other branches and held draft #4776 were untouched. Keep this issue open until the remaining requirements are demonstrated on master.
fix(sessions): release reoffer probe lock before async reads
style(sessions): match pinned Codex reoffer test formatting
test(sessions): consolidate Codex cold-reoffer regressions
chore(sessions): retain Codex scope-resolution timings
ScriptedAlchemy commentedon Nov 8, 2026
Consolidated the two overlapping #4374 implementation PRs without rewriting their histories. #4806 is now the sole active review surface: nonforce, exact-old-head commits
3386b3cand08b01b9import #4807's exact formatted 16-rollout/96-reoffer Git-scope and provenance regression blob and preserve its scope-resolution timing span. #4806 also retains its existing Codex-only empty-initializer bypass and rewrite-completion/cross-provider safety tests. The redundant alternative #4807 was closed unmerged, branch/history retained. No peer/held draft or release files touched.Exact-head CI for #4806 is queued; no Bazel Rust run, reviewer approval or 2,010-rollout affected-host before/after benchmark is claimed. This remains Refs #4374, not full acceptance; keep the issue open and prohibit merging any held Kev/decision-model drafts or #4776.
test(sessions): refuse failed Codex no-op rewrite checks
style(sessions): fix rewrite refusal regression formatting
chore(sessions): isolate Codex re-offer review from scheduler tests
perf(sessions): skip empty JSONL scope resolution on re-offer
ScriptedAlchemy commentedon Nov 9, 2026
Merged the focused, independently source-reviewed Codex re-offer repair from #4806 into
masterasc231fb7219f8(Refs #4374, partial work; not Fixes).The current-head CI run 42540841569 completed successfully: Linux fleet 183/183 Bazel targets passed (23 executed, remainder Bazel-cached), Windows, Clippy, repository/rustfmt gates, feature gates and dashboard passed. The sessions unit-test target was an exact-input Bazel cache hit from its prior passing 40.2-second execution; this is not a new independent local run. macOS was skipped in that hosted workflow. The source/test diff is four Codex/session files, and no file overlap exists with #4793 or #4815. Protected draft #4776, other held Kev/decision-model work and release #4763 were not changed.
The merged change removes empty-page Codex scope initialization while retaining exact cursor/prefix and rewrite-completion authority. Nonvacuous regressions exercise 96 real Git/JSONL project/profile re-offers across cold/warm caches with zero extra identity resolutions, changed-scope appends, source/provenance stability, cancellation, unopted-in providers, and retryable rewrite-completion failures. This measures eliminated calls, not the original 28 ms/file affected-host latency.
Issue #4374 remains open. The missing acceptance evidence is comparable
TRACEDECAY_SPAN_TIMINGS=1before/after measurement on the original 2,010-rollout corpus (project and profile, restart behavior, plus durable cursor/rewrite store costs). The hosted macOS job was skipped and this execution environment has no checkout/Bazel to independently reproduce it. No claim of full 2,010-rollout speedup or complete master acceptance is made.ScriptedAlchemy commentedon Nov 9, 2026
Refs #4374 — post-merge master CI blocked by external Bazel download (2026-11-09 UTC)
I re-read issue #4374 and all four comments, current
master(nowbe2e1349c63f), rootAGENTS.md(full tree confirms no nestedAGENTS.md), merged #4806's four-file diff/tests/review, and related #4793/#4815 ownership and protected #4776/#4763 states.New post-merge CI observation: the master CI run 42556702004 on merge commit
c231fb7finished failure, although #4806's pre-merge exact-head PR CI 42540841569 passed. In the master Linux-fleet job 128992084463, Bazel analysis failed at 2026-11-09 05:47 UTC with HTTP 504 Gateway Time-out fetching pinnedhermetic-launcherrunfiles-stub-x86_64-linuxfrom the GitHubbinaries-20261025release. Bazel reported build/analysis errors for downstream targets, not failed Codex Rust assertions. The aggregate Linux job failed accordingly. This is a recurrence of an external fetch failure seen earlier on #4806 and cleared by its prior retry; it is not evidence of a #4374 source regression. Repository gates, Clippy, Windows, Dashboard and feature gates passed in this master run; macOS was skipped.The subsequent newer master CI run 42566883615 on
be2e1349c63fis already queued/in progress, so I did not add duplicate CI pressure with a retry of the older master run. No source file was changed. #4806 remains merged, #4374 remains open: the original 2,010-rollout same-host before/after project/profile timings, explicit rewrite-completion/store costs and restart result remain unmeasured. Its merged nonvacuous regression quantifies zero unnecessary identity resolutions, not the 28 ms/file wall-clock elimination. No local checkout/Bazel or affected host corpus was accessible to independently run the benchmark; no Cargo used. Other owners' work, protected drafts, release #4763 and CI policy remain untouched.ScriptedAlchemy commentedon Nov 9, 2026
Refs #4374 — new master CI blocker isolated from Codex sessions (2026-11-09 UTC)
Re-read the full issue discussion, current
master(cf54e0322bf5), rootAGENTS.mdand full repository tree (no nestedAGENTS.md), and the merged #4806 source/tests/review and related owner branches. #4806 is merged atc231fb7219f8; #4374 is still open. Its source/test scope remains four sessions files.New CI finding: master run 42566883615, on
be2e1349c63f, reports 182/183 Linux Bazel tests passed. I downloaded the originalbazel-test-logs-Linux-fleetartifact and inspectedcrates/tracedecay/mcp_suite/test.log: 674 MCP tests passed, one failed atcrates/tracedecay/tests/mcp_suite/mcp_handler_test/feedback_diagnostics_test.rs:285infeedback_diagnostics_returns_the_published_cycle_for_its_minted_handle(cycle["returned_findings"]was0, expected1). This is an MCP feedback-diagnostics failure, not an observed Codex re-offer assertion; existing feedback owner #4423 / draft #4817 must retain its separate scope. The same artifact records//crates/tracedecay-sessions:unit_test667 passed, 0 failed, in 40.03 s. The Dashboard job failed separately during Bazel analysis because GitHub again returned HTTP 504 for the pinnedhermetic-launcherbinary, not due to an attributed Rust assertion.The newer
masterrun 42583025434, exactcf54e0322bf5, is already pending; do not add duplicate CI pressure. No source edit is justified in #4374 by the unrelated MCP failure or external fetch timeout. No Bazel executable/checkout or original affected-host rollout corpus is available in this execution environment.#4374 remains open: the original same-host 2,010-rollout before/after project/profile/restart timing, including durable cursor/rewrite-store cost attribution, is still missing; zero re-offer identity lookups in the merged behavior regression do not prove the full 28 ms/file wall-clock saving. No issue closure, protected draft, release #4763, peer branch, CI policy, Cargo, or production/infra change.
ScriptedAlchemy commentedon Nov 9, 2026
Refs #4374 — fresh master CI blocker, isolated from Codex (2026-11-09 UTC)
Re-read the complete issue discussion, current
master(d6acbc49f791), rootAGENTS.mdand the full 7,919-entry repository tree (no nested AGENTS.md), accepted merged #4806 source/regressions, #4380/#4706, new merged #4848, reviews, and the separately owned #4793 / held draft #4776 / read-only release #4763. No change or ownership collision warrants a #4374 source patch.New CI observation: master run 42656801013 on
0497fd830868passed 182/183 Linux Bazel test targets. I retrieved its published Linux fleet diagnostics artifact and inspecteddaemon_suite/test.log: 35 daemon tests passed, one failed,store_shutdown_checkpoint_test::stop_during_project_open_truncates_project_store_walsatcrates/tracedecay/tests/daemon_suite/store_shutdown_checkpoint_test.rs:182. It foundsessions.dbWAL size 3,197,152 bytes instead of 0 after shutdown. This is a daemon/store shutdown issue, previously tracked among load-sensitive tests in #4175, not an observed Codex admission failure. Do not weaken the assertion or alter the separately owned store shutdown area as part of #4374.In the same archived Bazel output,
//crates/tracedecay-sessions:unit_testcompleted 671 passed, 0 failed, in 40.88s. It includes the merged #4374 cold-cache / repeated re-offer / provenance / rewrite-failure regressions and newer #4848 Codex directory witnesses. Repository gates, Clippy, dashboard, feature gates and other Linux targets passed. Windows is still running. NewermasterCI 42667032687 ond6acbc49f791is pending, so no duplicate CI was launched and no latest-head green is claimed.#4374 stays open: still missing comparable
TRACEDECAY_SPAN_TIMINGS=1affected-host 2,010-rollout before/after project/profile/restart measurements, including convergence-check, scope/admission and durable cursor/rewrite-store timings. The confirmed zero redundant Git identity lookups in the merged regression is not proof that all 28 ms/file disappeared. This environment has neither the original corpus nor a checkout/Bazel for an independent native run. No Cargo, CI edits, other-owner branch writes, model/draft/release mutations, merge, or closure.