Skip to content

fix(storage): prevent session recovery lock-order deadlocks - #1336

Merged
XingYu-Zhong merged 1 commit into
KunAgent:developfrom
whitelonng:codex/fix-1335-session-deadlock
Sep 22, 2026
Merged

XingYu-Zhong merged 1 commit into
KunAgent:developfrom
whitelonng:codex/fix-1335-session-deadlock

Conversation

@whitelonng

@whitelonng whitelonng commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary / 概要

Fixes #1335.

Prevent a single conversation from permanently blocking its item/event RPCs when live-item recovery overlaps event append, compaction, or replacement. The thread queue is now acquired before JSONL leases and event-index mutation, and event append releases the event-index lock before waiting for the usage queue.

Why / 背景

Cold live-item recovery reads events while holding the thread queue. Previously, a replacement could hold a file lease while waiting for that queue, and event append could hold the event-index lock while waiting for it. Cold usage replay introduced another cycle: usage recovery waited for the event index while append held that index and waited for usage recovery. These cycles match the issue's stalled per-thread data path while health checks and other conversations remain responsive.

Changes / 变更

  • Apply one lock order to item reads/writes, fallback pagination, item compaction, event trimming, and usage compaction commits.
  • Keep event append and recordAppend in the same event-index critical section, then release it before updating the usage index.
  • Add 12 deterministic regression cases covering live recovery against append/trim/compaction, pagination behind replacement, and cold usage queries concurrent with append. Verify recovered content, unrelated-thread responsiveness, and no duplicate usage accounting after reopening.

Media / 截图或录屏

Actual Electron desktop after cold restart, showing the event-recovered COLD_RECOVERY_OK marker. The history is a synthetic fixture; the desktop, Service Manager, Runtime, HTTP/SSE, and filesystem are real. This screenshot demonstrates recovered UI content; request completion and regression tests provide the concurrency evidence.

Desktop history recovered after restart

Additional desktop interaction check

The reasoning selector remained usable after recovery (Off, High, and Ultra). This is a supplementary UI check, not evidence for the lock-order fix.

Reasoning selector remains usable

Screenshots are hosted on a separate evidence branch in the fork; this PR contains only the seven source/test files.

Tests / 测试

  • Relevant storage tests: 21 files / 166 tests passed.
  • Manager, usage, recorder, and HTTP usage integration tests: 6 files / 67 tests passed.
  • A fresh independent review reran 7 overlapping test files / 80 tests successfully; these are not added to the 233-test total above.
  • npm run typecheck, npm run build (including build:kun), npm run check:file-lines, and git diff --check passed.
  • macOS desktop smoke: 9,243,186-byte messages file, 72,402,902-byte events file, and a nonempty live checkpoint. Concurrent reads/append/trim completed in 322 ms, and 336 ms after quitting and cold-starting all owned processes. Thread snapshots, usage queries, and SSE replay completed; the recovery marker survived event trimming and restart. Owned processes exited cleanly.
Targeted test commands (run inside kun/)
node scripts/run-tests.mjs src/adapters/file/file-session src/adapters/file/jsonl-file-access.test.ts src/adapters/file/session-compaction-scheduler.test.ts tests/file-session --maxWorkers=4
node scripts/run-tests.mjs src/manager/service-manager-usage.test.ts src/manager/shared-data-store.test.ts src/manager/remote-data-stores.test.ts src/services/runtime-event-recorder.transient.test.ts src/services/usage-history.test.ts src/server/routes/usage.test.ts --maxWorkers=4

Validation / 验证

  • Contributor confirms the Contributor License Agreement.
  • Full npm run test (limitations below).
  • npm run typecheck
  • npm run build
  • Built Electron desktop smoke with real Manager/Runtime processes.
  • Logic change: regression tests added.

Notes / 备注

  • The full-suite attempt was not green: 11 self-update cases failed because the copied Homebrew Node executable could not locate libnode.141.dylib; one unrelated in-memory room integration case timed out under load and passed in isolation. The top-level run stopped before the renderer test phase. The 233 targeted tests above passed after the final fix.
  • No Windows host or original reporter dataset was available. Upstream-model smoke encountered authentication/request-budget failures and is not claimed as passing.
  • This change addresses the reproduced lock cycles. Watchdogs, server-side request budgets, and a repair CLI suggested in the issue remain outside its scope.

Serialize item/event/usage mutations on the per-thread queue before
taking JSONL leases or event-index mutations, release the event-index
section before usage work, and keep append + recordAppend atomic.

Also fail fast on same-path read/replacement reentry in the JSONL
access coordinator so a caller cannot silently self-deadlock.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@XingYu-Zhong
XingYu-Zhong force-pushed the codex/fix-1335-session-deadlock branch from 688a84f to 1e754e5 Compare September 21, 2026 01:53
@XingYu-Zhong

Copy link
Copy Markdown
Collaborator

Maintainer follow-up: pushed an amendment adding fail-fast guards to JsonlFileAccessCoordinator for the same bug class this PR fixes:

  • withReplacement(path) called inside an active withRead(path) scope now throws immediately instead of self-deadlocking on its own reader count.
  • acquireRead/withRead(path) called inside an active withReplacement(path) now throws instead of queueing behind its own replacement.
  • Nested withReplacement(path) on the same path is now reentrant via AsyncLocalStorage, matching withRead/withIndexMutation semantics.

Cross-path ordering is unchanged. Added 4 regression tests in jsonl-file-access.test.ts; locally verified: 188 file-adapter tests + 12 lock-order tests + tsc --noEmit all pass.

One process note for whoever merges: the body says Fixes #1335, which will auto-close the issue. The issue also asks for a data-plane RPC timeout budget, a compaction watchdog, and a repair CLI — none of which are in this PR. Consider filing those as follow-ups (or keeping #1335 open) before merge.

@XingYu-Zhong
XingYu-Zhong merged commit 86c4fa5 into KunAgent:develop Sep 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants