fix(cron): retry ACP pipe-broken deaths via typed check - #7593
fix(cron): retry ACP pipe-broken deaths via typed check#7593aniruddhaadak80 wants to merge 1 commit into
Conversation
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
There was a problem hiding this comment.
Pull request overview
Updates the Slack gateway’s cron failure handling so ACP process-death retries trigger on the typed AcpProcessDied exception (covering the common “pipe broken” failure mode), while preserving the existing substring-based fallback for older/untyped AcpError messages.
Changes:
- Prefer
isinstance(exc, AcpProcessDied)when deciding whether to perform the one-shot “reset session + retry” path. - Keep the legacy
"not running"/"process exited"substring checks as a fallback arm forAcpError.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| isinstance(exc, AcpProcessDied) | ||
| or ( | ||
| isinstance(exc, AcpError) | ||
| and ("not running" in exc_msg or "process exited" in exc_msg) | ||
| ) |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
The cron failure handler retried ACP process death only when the message contained 'not running' or 'process exited'. The common broken-pipe signature 'ACP process pipe broken: <cause>' carries neither substring, so the retry never fired and a recoverable child death counted toward consecutive_failures toward auto-pause. Test the typed signal AcpProcessDied first, keeping the substring fallback for older spellings that do not raise the typed error. The import already exists at the decision site. Fixes kirodotdev#7395.
5e16a97 to
a9f6a6a
Compare
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
1 similar comment
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
Thanks for this — the fix is right, but #7396 makes the same production change and additionally ships regression tests, so we're consolidating on that one to avoid two PRs racing the same line. Both PRs replace the cron retry guard in The difference: #7396 also adds Neither has landed yet — verified on if (
isinstance(exc, AcpError)
and ("not running" in exc_msg or "process exited" in exc_msg)
and not getattr(job, "_acp_retried", False)
and self.sessions is not None
):( Closing as a duplicate of #7396. Please do follow that PR — and if you'd rather your version be the one that lands, say so and we'll reopen this instead. |
Problem / Motivation
Cron's ACP-death retry at
src/kiro_crew/slack/gateway.py:4769string-matchedexcfor"not running"or"process exited". The common death is a broken pipe on the next write atsrc/kiro_crew/acp/client.py:3831raisingAcpProcessDied("ACP process pipe broken: <cause>"), which contains neither substring. The retry never fired.Why it matters
A recoverable one-off child death was recorded as a hard failure;
consecutive_failuresmarched toward auto-pause at 5, silently disabling a healthy job. Degradation is silent because the retry looks present in code.What changed (motivation → approach → change)
Symptom (retry skipped for pipe-broken) → root cause (string match vs typed signal) → fix: test the typed signal first.
isinstance(exc, AcpProcessDied)(already imported atgateway.py:46) now gates the retry, with the substring check kept as fallback for older spellings. Oneorarm:isinstance(AcpProcessDied) or (isinstance(AcpError) and substring), guarded by_acp_retriedandsessions is not None.Tests
_send_request/_send_response/_send_error) withBrokenPipeError→AcpProcessDied; verifiedshould_retrytrue before (false) after (true).AcpErrorwith old substrings still retries; otherAcpErrorand plainExceptiondo not.isort,flake8,black --target-version py310clean;mypyskipped (gateway.py huge, change is 9 lines type-safe).Manual verification
N/A — unit coverage sufficient. Cron retry path is covered by existing
record_failure/auto_pausedlogic; manual would require killing the ACP child and observing a secondstream_and_collectcall.Screenshots / video
N/A — no UI change.
Related Issues
Fixes #7395
Checklist
fix(cron): retry ACP pipe-broken deaths via typed check