Fix Docker OOM reporting and queue startup pacing

[konard]

Closes #2015

Summary

• Parse oomKilled from start-command $ --status output, including links-notation, JSON, and Docker-like State.OOMKilled payloads.
• Treat oomKilled true as terminal oom-killed before backend liveness probing, using exit code 137 when start-command has no better exit code.
• Add a two-minute Docker backend-gone grace period so a missing container alone does not immediately produce a false killed Telegram notification while status/log footer data catches up.
• Add localized killed-session copy for English, Russian, Chinese, and Hindi so Telegram no longer leaks telegram.work_session_killed.
• Pin Docker images to start-command@0.30.3, which includes the merged/released upstream fix from Fix detached Docker OOM status reconciliation link-foundation/start#149.
• Enforce a minimum 10-minute startup interval globally across tool queues and return only the oldest startable queued task per consumer pass, so cleared restrictions and immediate-start backlogs cannot launch in bursts.
• Cap CPU/RAM/disk cache freshness at one minute while leaving API cache TTLs unchanged.
• Preserve the issue investigation, raw logs, extracted summaries, screenshots, requirements, upstream links, and source links under docs/case-studies/issue-2015.

Investigation Findings

• Task 5ff719b9-9d2d-4479-b124-c4b8bda61dd0 was a false killed notification: the start-command log later recorded Exit Code: 0 at 2026-07-04 15:09:11.338, while Telegram had already reported the session as killed.
• Tasks 1e9e7513-edd7-43a2-b143-169cfd794af6 and d90880d4-aa05-4145-ac02-7542eea2041a exposed Docker oomKilled true while top-level status remained executing; Hive Mind now handles this downstream, and upstream start#148/start#149 released the matching start-command fix.
• Host resource pressure was present in the preserved bot logs, including CPU saturation and disk warnings, but it does not explain every symptom: task 1 completed successfully, while tasks 2 and 3 carried explicit Docker OOM markers.
• Queue startup pacing was also a reliability gap: per-tool intervals and immediate multi-task starts could start a backlog too quickly after restrictions lifted, before host metrics had time to settle.

Screenshots / Evidence

Tests

• node tests/test-issue-2015-oom-killed-status.mjs
• node tests/test-issue-2015-queue-stability.mjs
• node tests/test-issue-1927-killed-detection.mjs
• node tests/test-issue-1927-completion-labeling.mjs
• node tests/test-telegram-ui-i18n.mjs
• node tests/test-i18n.mjs
• node tests/test-issue-1686-log-command.mjs
• node tests/queue-config.test.mjs
• node tests/solve-queue-tool-tracking.test.mjs
• node tests/solve-queue.test.mjs
• npm run lint
• npm run format:check
• npm run check:duplication
• node --check src/solve.mjs
• node --check src/hive.mjs
• bash scripts/check-mjs-syntax.sh
• bash scripts/check-file-line-limits.sh
• node tests/docs-validation.mjs
• node tests/test-docs-language-sync.mjs
• npm test (305 selected test files)
• npx secretlint --no-gitignore --secretlintrcJSON '{"rules":[{"id":"@secretlint/secretlint-rule-preset-recommend"}]}' "docs/case-studies/issue-2015/**/*"

[konard]

Working session summary

Done. PR 2016 is updated, pushed, marked ready, and CI is green: #2016

Implemented:

• Parse oomKilled from $ --status output.
• Treat Docker oomKilled true as terminal oom-killed with exit code 137 fallback.
• Add a Docker backend-gone grace period to avoid false killed notifications.
• Fix missing localized telegram.work_session_killed copy.
• Add regression coverage in tests/test-issue-2015-oom-killed-status.mjs.
• Preserve the full investigation under docs/case-studies/issue-2015.
• Open upstream follow-up: $ --status can remain executing while oomKilled=true for detached docker sessions link-foundation/start#148

Verification:

• Local targeted tests, npm run lint, npm run format:check, npm test, and secret scan passed.
• GitHub Actions run 28742402058 on commit efcbcce93c70fc5497cfbd0c59f63924d8401a3d completed successfully.
• Working tree is clean.

Sources used: Docker Engine API State.OOMKilled reference, GitHub Checks API docs, and GitHub Workflow Runs docs.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Model: GPT-5.5
• Provider: OpenAI
• Public pricing estimate: $26.008301

📊 Context and tokens usage:

• 921.2K / 200K (461%) input tokens, 58.7K / 128K (46%) output tokens

Total: (921.2K + 14.2M cached) input tokens, 58.7K output tokens, $26.008301 cost

🤖 Models used:

• Tool: OpenAI Codex
• Requested: gpt-5.5
• Thinking level: max (~31999 tokens)
• Model: GPT-5.5 (gpt-5.5)

📎 Log file uploaded as Gist (15861KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

We can now use link-foundation/start#149 (new version was released)

We should double check all the details about of #2015, and fix all not yet fixed false positives, false negatives and negatives from the logs.

We also need to set 10 minutes as minimum interval between tasks startup, as when restrictions of queue are lifted (for any threshold) we executing bulk of tasks too fast, that leads to secondary mass killing of tasks.

Also when no limits are reached, we usually allowed immediate execution of multiple tasks, now we should queue them with same 10 minutes interval.

So the system have time to update metrics. Also we need to make sure we never cache CPU, RAM, disk usage for more than 1 minute, so we always use the most fresh details possible. API caching stays, as it can hit rate limits overwise.

Double check that our queue system and monitoring on all levels is correct and is the most robust possible, so we guarantee maximum reliability and stability.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-07-05T18:33:32.286Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

Follow-up complete

Addressed the latest review comment on PR #2016:

• Updated Docker images to start-command@0.30.3 after upstream start#149 was merged/released.
• Enforced a minimum 10-minute startup interval globally across tool queues and limited each consumer pass to the oldest startable queued task.
• Capped CPU/RAM/disk cache TTL at 1 minute while leaving API cache TTLs unchanged.
• Added tests/test-issue-2015-queue-stability.mjs and updated queue/default tests and configuration docs.

Verification:

• Local lint, format, duplication, syntax, docs, targeted issue tests, and npm test all passed.
• GitHub Actions run 28751492404 passed on head f83bef60abcd2cd73a5f3cbef2aa60fe0bf56e05.
• PR is ready for review and merge state is clean.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Model: GPT-5.5
• Provider: OpenAI
• Public pricing estimate: $16.740563

📊 Context and tokens usage:

• 487.7K / 200K (244%) input tokens, 38.1K / 128K (30%) output tokens

Total: (487.7K + 10.1M cached) input tokens, 38.1K output tokens, $16.740563 cost

🤖 Models used:

• Tool: OpenAI Codex
• Requested: gpt-5.5
• Thinking level: max (~31999 tokens)
• Model: GPT-5.5 (gpt-5.5)

📎 Log file uploaded as Gist (9214KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag