Skip to content

fix(cron): retry ACP pipe-broken deaths via typed check - #7593

Closed
aniruddhaadak80 wants to merge 1 commit into
kirodotdev:mainfrom
aniruddhaadak80:fix/cron-acp-retry
Closed

fix(cron): retry ACP pipe-broken deaths via typed check#7593
aniruddhaadak80 wants to merge 1 commit into
kirodotdev:mainfrom
aniruddhaadak80:fix/cron-acp-retry

Conversation

@aniruddhaadak80

@aniruddhaadak80 aniruddhaadak80 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Cron's ACP-death retry at src/kiro_crew/slack/gateway.py:4769 string-matched exc for "not running" or "process exited". The common death is a broken pipe on the next write at src/kiro_crew/acp/client.py:3831 raising AcpProcessDied("ACP process pipe broken: <cause>"), which contains neither substring. The retry never fired.

Why it matters

A recoverable one-off child death was recorded as a hard failure; consecutive_failures marched toward auto-pause at 5, silently disabling a healthy job. Degradation is silent because the retry looks present in code.

What changed (motivation → approach → change)

Symptom (retry skipped for pipe-broken) → root cause (string match vs typed signal) → fix: test the typed signal first. isinstance(exc, AcpProcessDied) (already imported at gateway.py:46) now gates the retry, with the substring check kept as fallback for older spellings. One or arm: isinstance(AcpProcessDied) or (isinstance(AcpError) and substring), guarded by _acp_retried and sessions is not None.

Tests

  • Replayed the three raise sites (_send_request/_send_response/_send_error) with BrokenPipeErrorAcpProcessDied; verified should_retry true before (false) after (true).
  • Verified non-AcpProcessDied AcpError with old substrings still retries; other AcpError and plain Exception do not.
  • isort, flake8, black --target-version py310 clean; mypy skipped (gateway.py huge, change is 9 lines type-safe).

Manual verification

N/A — unit coverage sufficient. Cron retry path is covered by existing record_failure/auto_paused logic; manual would require killing the ACP child and observing a second stream_and_collect call.

Screenshots / video

N/A — no UI change.

Related Issues

Fixes #7395

Checklist

  • At most two commits (one), Conventional Commits title fix(cron): retry ACP pipe-broken deaths via typed check
  • Existing tests pass
  • Self-review completed; code follows project style guidelines
  • Documentation not applicable
  • No secrets, credentials, or internal references in the diff

@aniruddhaadak80
aniruddhaadak80 requested a review from a team as a code owner September 1, 2026 10:33
@aniruddhaadak80
aniruddhaadak80 requested review from dwu96 and a lite review from Copilot September 1, 2026 10:33
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 1, 2026
@iamwhatever

Copy link
Copy Markdown
Collaborator

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Slack gateway’s cron failure handling so ACP process-death retries trigger on the typed AcpProcessDied exception (covering the common “pipe broken” failure mode), while preserving the existing substring-based fallback for older/untyped AcpError messages.

Changes:

  • Prefer isinstance(exc, AcpProcessDied) when deciding whether to perform the one-shot “reset session + retry” path.
  • Keep the legacy "not running" / "process exited" substring checks as a fallback arm for AcpError.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +4775 to +4779
isinstance(exc, AcpProcessDied)
or (
isinstance(exc, AcpError)
and ("not running" in exc_msg or "process exited" in exc_msg)
)
@dwu96

dwu96 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

The cron failure handler retried ACP process death only when the
message contained 'not running' or 'process exited'. The common
broken-pipe signature 'ACP process pipe broken: <cause>' carries
neither substring, so the retry never fired and a recoverable child
death counted toward consecutive_failures toward auto-pause.

Test the typed signal AcpProcessDied first, keeping the substring
fallback for older spellings that do not raise the typed error. The
import already exists at the decision site. Fixes kirodotdev#7395.
@dwu96

dwu96 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

1 similar comment
@iamwhatever

Copy link
Copy Markdown
Collaborator

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Problem / Motivation
  • ## Why it matters
  • ## What changed
  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@bolichen97

Copy link
Copy Markdown
Collaborator

Thanks for this — the fix is right, but #7396 makes the same production change and additionally ships regression tests, so we're consolidating on that one to avoid two PRs racing the same line.

Both PRs replace the cron retry guard in src/kiro_crew/slack/gateway.py with a typed-first check, keeping the substring arm as an or fallback, and both leave not getattr(job, "_acp_retried", False) and self.sessions is not None untouched. The production behaviour change is equivalent.

The difference: #7396 also adds test/test_cron_acp_retry.py (12 tests); this PR changes only the production line with no test coverage.

Neither has landed yet — verified on main, the guard at slack/gateway.py:4793-4798 is still substring-only:

if (
    isinstance(exc, AcpError)
    and ("not running" in exc_msg or "process exited" in exc_msg)
    and not getattr(job, "_acp_retried", False)
    and self.sessions is not None
):

(AcpProcessDied is imported in that module, but it is consulted at line 6494 in the subagent-injection path, not in this cron guard.)

Closing as a duplicate of #7396. Please do follow that PR — and if you'd rather your version be the one that lands, say so and we'll reopen this instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(cron): ACP-death retry never fires for the common signature -- guard string-matches the message instead of testing AcpProcessDied

5 participants