Skip to content

fix: auto-suspend oversized bot sessions - #722

Draft
Barrierml wants to merge 1 commit into
deepcoldy:masterfrom
Barrierml:fix/session-rss-auto-suspend
Draft

fix: auto-suspend oversized bot sessions#722
Barrierml wants to merge 1 commit into
deepcoldy:masterfrom
Barrierml:fix/session-rss-auto-suspend

Conversation

@Barrierml

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in per-bot maxSessionRssMiB guard for runaway session memory
  • periodically suspend idle, resumable sessions whose worker+CLI process tree exceeds the configured RSS budget
  • expose the field through bot config parsing, /config, and dashboard bot payloads

Verification

  • pnpm exec tsc --noEmit
  • pnpm vitest run --project unit test/session-rss-guard.test.ts test/bot-registry.test.ts test/bot-config-store.test.ts test/dashboard-bot-payload.test.ts test/idle-worker-sweeper.test.ts test/api-only-cli-gate.behavior.test.ts test/worker-terminal-read-auth.integration.test.ts
  • pnpm build

Note: pnpm test was first run before rebuilding dist/, causing the built-CLI gate test to read stale artifacts. After pnpm build, the previously failing built-CLI tests passed when rerun.

Add an opt-in per-bot maxSessionRssMiB guard that periodically samples live session process trees and suspends idle, resumable sessions when a single session exceeds its configured RSS budget. This gives operators a way to contain runaway CLI memory use before the host reaches Docker or global OOM.

Co-authored-by: TRAE CLI <noreply@bytedance.com>
@Barrierml
Barrierml requested a review from deepcoldy as a code owner August 4, 2026 03:57

@deepcoldy deepcoldy left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对抗复审结论:请求修改。adopt / restore 本身没有发现可触发的正确性 bug,但 attestation 覆盖存在 1 处 blocker。

[P2 / blocker] Herdr 的 CLI 内存长期漏算;首个 worker 还可能把共享 Herdr host 错算给单个会话

位置:src/core/session-rss-guard.ts:65-67

这里把会话归因定义为 worker 子树 + localProcessAttestation.cliPid 子树。这个前提在 managed Herdr 上不成立:

  • HerdrBackend.getChildPid() 只是返回 this.cliPid ?? nullsrc/adapters/backend/herdr-backend.ts:704-706),但 managed Herdr 的 spawn 路径从未给 cliPid 赋值。
  • 因此 worker 在 src/worker.ts:9317-9318 上报的是不含 CLI pid 的 attestation;后面的 25 次 late retry 仍调用同一个永远为 null 的 getter,约 3 秒后停止。
  • managed Herdr agent 跑在机器共享的 botmux Herdr host 下。host 已存在时,它不是当前 worker 的后代,所以 guard 永远只算 worker Node,自身 CLI 即使远超阈值也不会休眠。
  • 反过来,若当前 worker 恰好是首次创建共享 host 的那个 worker,detached/unref 并不会立刻改变 PPID;worker 子树会暂时包含共享 host 及其中其它 session 的 agent,可能把多会话内存全算到一个逻辑会话上并休眠错误对象。worker/daemon 重启后 host reparent,又退化成永久漏算。

所以 “attest 尚未上报” 不只是一个短窗口:Herdr 是稳定可复现的永久缺口。Zellij 也有同类 fail-open 风险:CLI pid 若在约 3 秒预算后才可解析,之后不再补 attestation,后续每轮都漏算;即使最终能在 3 秒内解析,期间碰到 60 秒 timer 也会漏掉当轮。

建议合并前:

  1. 对每个允许 RSS auto-suspend 的 detached backend,保证拿到绑定当前 worker generation + procStart 的精确 CLI/agent 根 pid;Herdr 应从当前 managed agent/pane 的前台进程解析,而不是依赖 worker 子树。
  2. late pid resolution 不应在固定 3 秒后永久放弃;至少在 idle/guard sweep 时继续重试并可观测地报告 attribution unavailable。
  3. 增加 Herdr(CLI 在 worker 树外且 attestation 缺失)、Zellij 超过 3 秒才解析、缺失 cliProcStart/PID reuse 的测试。

adopt / restore 复核

当前“不查 ds.session.adoptedFrom”确实尚未形成可触发 bug:startAdoptSession 同时写 runtime/persisted marker;restoreActiveSessions 从 persisted marker 构造 ds.adoptedFrom 后才 forkAdoptWorker;而 forkAdoptWorker 自身没有 runtime marker 会直接 throw,live worker 建立后还有 initConfig.adoptMode 第二层。因此生产路径下 live adopt worker 不会只剩 persisted marker。不过建议与 idle-worker-sweeper 对齐补上 ds.session.adoptedFrom,作为低成本防御纵深,并加回归测试。

验证:

  • pnpm exec vitest run --project unit test/session-rss-guard.test.ts test/idle-worker-sweeper.test.ts test/session-lifecycle-start.test.ts → 78/78
  • pnpm exec vitest run test/herdr-backend.e2e.ts → 9/9
  • pnpm build → 通过

@xu4wang

xu4wang commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in per-bot maxSessionRssMiB guard for runaway session memory
  • periodically suspend idle, resumable sessions whose worker+CLI process tree exceeds the configured RSS budget
  • expose the field through bot config parsing, /config, and dashboard bot payloads

Verification

  • pnpm exec tsc --noEmit
  • pnpm vitest run --project unit test/session-rss-guard.test.ts test/bot-registry.test.ts test/bot-config-store.test.ts test/dashboard-bot-payload.test.ts test/idle-worker-sweeper.test.ts test/api-only-cli-gate.behavior.test.ts test/worker-terminal-read-auth.integration.test.ts
  • pnpm build

Note: pnpm test was first run before rebuilding dist/, causing the built-CLI gate test to read stale artifacts. After pnpm build, the previously failing built-CLI tests passed when rerun.

这个会用在哪一个场景呢? 如果是为了节约内存的话,只要是idle状态的,都可以suspend释放内存(后续无损resume),不需要考虑占用RSS大小。 似乎根据idle的时间去释放更好一些?

@deepcoldy

Copy link
Copy Markdown
Owner

确实感觉现在的自动30个(可配置)的自动释放策略基本就足够了,agent做大项目时就是会多使用一些内存,到内存上限自动清理的另一个前提是:需要能精准判断好agent没有放在后台的定时任务(这个目前还没做过特殊判断)

@deepcoldy
deepcoldy marked this pull request as draft August 7, 2026 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants