Skip to content

feat(cron): name the job a loop-stall interrupted, and stop it re-running - #8279

Closed
bolichen97 wants to merge 1 commit into
kirodotdev:mainfrom
bolichen97:feat/loop-stall-job-attribution
Closed

feat(cron): name the job a loop-stall interrupted, and stop it re-running#8279
bolichen97 wants to merge 1 commit into
kirodotdev:mainfrom
bolichen97:feat/loop-stall-job-attribution

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Problem / Motivation

A gateway the loop-stall watchdog hard-exits runs no finally. The cron run in flight writes no last_run_ts and no history row, so the store shows the job as never fired; on the next boot it is due again the moment the timer arms, it fires, and it stalls again. A user hit exactly this as an hourly crash. kirocrew doctor showed the wedged stack but could not say which job it was, so he paused the wrong one (a script cron that never touches the model, and so could not have produced that stack).

Why it matters

Without a name, the operator's only options are to pause every cron or to read a faulthandler stack. Without a breaker, a single bad input becomes a crash loop that repeats on every boot until someone guesses right. This PR makes the crash self-limiting and self-explaining; #8277 fixes the gate that produced this particular stall.

What changed (motivation → approach → change)

Two additions and one reader, all off the event loop.

  • cron_inflight — a run writes <data home>/cron-running/<job id>.json (job_id, name, started_at, pid) when it starts executing and clears it on every finally path. A marker whose PID is dead is exactly "this job was in flight when that gateway died": no schedule inference, no timestamp heuristics. Writes are asyncio.to_thread and best-effort (a marker failure never fails a run); reads are size-bounded; job ids that would leave the directory are refused.
  • stall_attribution — reads the newest stack-bearing dump, names the surface from the outermost recognised frame of the wedged thread (a cron turn passes through slack/gateway.py, so a top-down match would call it Slack), names the permission-gate frame when the stall is inside it, and joins abandoned markers to the dump by PID (and started_at ≤ dump mtime). One match names the job; several name candidates and no job; none says so; a chat/channel surface is named and implicates no job. It never guesses. crash_dump_store gains two public readers (dump_owner_pid, dump_wedged_frames) for it.
  • Breaker in CronService.start() — runs on a worker before _arm_timer(). Only when the surface is cron and exactly one abandoned marker carries the dump's PID does it park that job auto_paused (enabled=False, last_status="error", a last_error naming the dump and kirocrew cron resume <id>), persisted under the store lock and SEL-audited as cron_auto_pause / auto_paused_loop_stall. The dump name is claimed in cron-running/.loop-stall-breaker so one crash pauses its job once: a dump stays on disk for a week and a job the operator resumed must not be re-paused on the next boot. Abandoned markers are swept after being read, whatever the verdict.
  • kirocrew doctor — under the dump's stack prints an attribution: block: stuck-in frame, surface, job, recommended: kirocrew cron pause <id>, and the job's current pause state from crons.json (job_pause_state_from_disk, no gateway needed). An attributed job is added to the issues summary. The boot notification carries the same lines.

Sample doctor output on a synthesized data home shaped like the field host:

Loop-stall Crash Dumps
  last dump:   ⚠️  loopstall-20260903T000000Z.txt (3.0h ago)
  MainThread stuck at:
    Timeout (0:00:25)!
    Thread 0x00007f0000000002 (most recent call first):
      File ".../kiro_crew/security.py", line 7969 in is_sensitive_bash_command
      File ".../kiro_crew/llm_helpers.py", line 2058 in _resolve_permission
      ...
  attribution:
    stuck in the tool permission gate (security.py:7969 is_sensitive_bash_command)
    the gateway (PID 4200001) was executing cron job 'twb-refresh' (59c321a9) when the watchdog terminated it
    recommended: kirocrew cron pause 59c321a9
    job is currently enabled

Trust boundary note: cron-running/ is not a fenced path. A forged marker could at most make the breaker pause one job (audited, with a visible reason naming the dump) and only when a cron-surface dump exists; it cannot enable, run or alter a job.

Specs: docs/system-specs/modules/learn-cron-dashboard.md, docs/system-specs/modules/cli.md.

Tests

  • test/test_cron_loop_stall_breaker.py — the breaker test seeds an overdue strict every job (its last_run_ts two periods back, as a killed run leaves it) and asserts it did not fire after start(); with the breaker call removed the job fires on the first tick (['<id>'] == []), which is the crash loop. A resumed job is not re-paused on a second boot; a chat-surface dump and a two-candidate dump pause nothing; no dump is a no-op; the pause is SEL-audited once. Markers are present during a run (read from inside the callback) and gone after normal and raising completion.
  • test/test_stall_attribution.py — marker round trip, id validation, unparseable/oversize/tmp files skipped, abandoned-vs-live owners; surface classification for cron (through the Slack module), dashboard chat, Slack, unknown, Windows frame paths; attribution for single/multiple/no marker, PID mismatch, a marker newer than the dump, a live owner; describe() lines; newest-dump selection past a header-only file.
  • test/stall_dump_helpers.py — synthesized dumps in faulthandler's format behind the store's real header.

test/test_cron.py::TestComputeNextRunTs::test_cron_schedule fails on untouched origin/main (timezone-dependent) and is unrelated.

Manual verification

Ran kirocrew doctor against a temp KIROCREW_HOME holding a crons.json, a cron-running marker and a dump in the store's format (output above). Existing cron, crash-dump store, cautious-boot and doctor suites pass.

Related Issues

Companion to #8277 (linear gate) and #8278 (title-tier off-loop scan). Independently mergeable.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…ning

A gateway the loop-stall watchdog hard-exits runs no `finally`. The
cron run in flight therefore writes no last_run_ts and no history row:
the store shows the job as never fired, it is due again the moment the
next boot arms the timer, it fires, and it stalls again. A user hit
this as an hourly crash, and `kirocrew doctor` could show the stack but
not the job, so he paused the wrong one.

Two additions, one reader:

* cron_inflight: a run drops `<data home>/cron-running/<job id>.json`
  (job_id, name, started_at, pid) when it starts executing and clears
  it on every `finally` path. A marker whose PID is dead is exactly
  "this job was in flight when that gateway died" -- no schedule
  inference. Writes are off-loop and best-effort.

* stall_attribution: reads the newest stack-bearing dump, names the
  surface from the OUTERMOST recognised frame of the wedged thread (a
  cron turn passes through the Slack gateway module, so top-down would
  misname it), names the permission-gate frame, and joins abandoned
  markers to the dump by PID. One match names the job; several name
  candidates and no job; none says so. It never guesses.

* CronService.start() runs the breaker on a worker BEFORE the timer is
  armed: a cron-surface dump plus exactly one matching marker parks
  that job auto_paused with a last_error naming the dump and the
  `kirocrew cron resume <id>` command, persisted under the store lock
  and SEL-audited (auto_paused_loop_stall). Claimed once per dump so a
  resumed job is not re-paused on the next boot while the dump is
  still on disk. Ambiguous evidence pauses nothing.

* `kirocrew doctor` prints the attribution under the dump's stack --
  stuck-in frame, surface, job, `recommended: kirocrew cron pause <id>`,
  and the job's current pause state from crons.json -- with no gateway
  running. The boot notification carries the same lines.

Tests: the breaker test seeds an OVERDUE strict job and asserts it did
not fire after start(); with the breaker call removed the job fires on
the first tick (['<id>'] == []), which is the crash loop. Markers are
asserted present during a run and gone after normal and raising
completion. Attribution covers single/multiple/no marker, PID mismatch,
a marker newer than the dump, a live owner, chat/slack/unknown
surfaces, and Windows frame paths. test_cron.py::test_cron_schedule
fails on untouched origin/main (timezone-dependent) and is unrelated.
@bolichen97

Copy link
Copy Markdown
Collaborator Author

Superseded by #8282, which carries this change together with its two companions as one PR.

@bolichen97 bolichen97 closed this Sep 3, 2026
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant