Repository navigation
perf(sessions): bound flat Codex directory entry traversal - #4857
ScriptedAlchemy wants to merge 4 commits into
Conversation
Refs #4756 A flat archived directory with many .zst/non-JSONL entries previously consumed its entire ReadDir iterator in one structural scan turn, outside the directory-work budget. Charge directory entries independently of child directory traversal while retaining the iterator across turns. Add a real archived-directory regression requiring the first scan to yield in the directory phase, then converge on exactly the eligible JSONL path. Partial performance repair only; durable restart continuation and affected host CPU/write-latency acceptance remain outstanding.
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Refs #4756 Keep unsettled directory name witnesses scoped to traversable child buckets and JSONL candidate paths. Compressed rollouts and other ignored file churn must not independently invalidate the candidate witness. Cover ignored additions and renames, candidate admission, and child bucket discovery with a real-filesystem regression.
Refs #4756 — follow-up Codex discovery witness repair (partial)Published Cause and change: Even after limiting structural Nonvacuous filesystem regression: 128 ignored Detached exact diff is one commit ahead, two files, +46/-1; independently checked the diff and branch head. Exact-head Bazel CI run 42769750119 was queued at inspection; there is no local checkout or Bazel executable, so no Rust test pass or incident-scale performance measurement is claimed. No Cargo used. This is not |
Refs #4756 Apply the two exact formatting hunks emitted by Bazel CI; preserve every behavioral test assertion.
|
Refs #4756 — CI repair published as The exact preceding-head Bazel run #42769750119 failed New exact-head hosted Bazel run #42775257282 is queued. No Rust test pass, durable mid-directory restart-continuation proof, or incident-scale CPU/SQLite write-latency acceptance is claimed. This remains |
Refs #4756 — eliminate eager full-archive name enumeration (partial)Published Root cause: the recently bounded structural Source fix: obtain the native directory identity only; retain an incremental, constant-memory, order-independent candidate-name digest on the same New real-filesystem regression: create 128 ignored Review: independent GitHub compare verifies one commit with only This specifically removes the eager unbounded name-scan before bounded initial discovery; idle unsettled-directory rechecks, durable mid-directory restart continuation, and measured 4-hour CPU / SQLite write latency acceptance are not yet proved. Keep #4756 open, retain |
Refs #4756 — partial Codex directory-work bound (not Fixes)
Root cause and focused correction
The initial
CodexScanPhase::Directoriesphase charged queued and discovered subdirectories against its per-turn budget, but did not charge ordinary directory entries read fromReadDir. A flat~/.codex/archived_sessionsdirectory containing many.zstor other non-directory names could therefore be fully enumerated by one scheduler call, before the separately bounded file-candidate phase. The reported corpus includes 1,385 Codex.zstentries; those entries are not accepted as JSONL transcripts by this reader, so their enumeration should not monopolize a tick.The source now counts every entry consumed from the structural
ReadDiriterator against its ownmax_files × (MAX_SCAN_DEPTH+2)work window. When the limit is reached, the existing process-owned iterator remains intact for the next call. Child-directory structural work and the candidate-file phase retain their independent existing limits. There is no new format, scheduler, durable storage, or SQL writer behavior.Nonvacuous behavioral regression
A real filesystem fixture creates 64 ignored
.zstentries and one admitted.jsonlin a flat archived Codex directory, then runsmax_files=1discovery. The first call must yield inside the directory phase with its live iterator retained, rather than consuming the archive and entering the file phase in one call; the previous implementation fails this assertion. Repeated bounded calls must converge on exactly the eligible JSONL file and earn the complete frontier, with an empty subsequent idle pass. No other test or assertion was removed.Verification and boundaries
AGENTS.mdand recursive master tree (no nested AGENTS.md), original issue and both issue comments, existing perf(sessions): bound projection queue-head scans per write window #4793/perf(sessions): isolate settled Codex scan optimization #4848/perf(sessions): skip settled Codex directory fingerprints #4815 owner boundaries and perf(sessions): bound projection queue-head scans per write window #4793 CI/reviews.835291a, directly parented to master2f9d659. Independently inspected the exact detached Git diff. Published to this separate issue branch byexpected_sha=2f9d659,force=false. perf(sessions): bound projection queue-head scans per write window #4793 and concurrent branches were not modified.bazel test //crates/tracedecay-sessions:unit_test --test_output=errors, repository rustfmt and Clippy, and applicable exact-head hosted CI before merge. No Cargo used.ReadDiriterator still does not survive daemon restart. This is not proof of incident-scale CPU reductions, 125–160 ms SQLite write latency elimination, or durable mid-directory restart continuation. Keep sessions: re-import after store reset holds ~1.7 cores for 4+ hours; profile_sessions writes take 125–160 ms; restart repeats the import #4756 open; noFixes #4756or merge hold removal.Kev/decision-model PRs and mixed #4776 remain DRAFT / never merge or auto-merge; release #4763 read-only. No production deployment, infrastructure, tags, credentials, Kody/Copilot/delegation, weakened CI, other owner edits or schedule changes.