feat(cron): name the job a loop-stall interrupted, and stop it re-running - #8279
Closed
bolichen97 wants to merge 1 commit into
Closed
feat(cron): name the job a loop-stall interrupted, and stop it re-running#8279bolichen97 wants to merge 1 commit into
bolichen97 wants to merge 1 commit into
Conversation
…ning A gateway the loop-stall watchdog hard-exits runs no `finally`. The cron run in flight therefore writes no last_run_ts and no history row: the store shows the job as never fired, it is due again the moment the next boot arms the timer, it fires, and it stalls again. A user hit this as an hourly crash, and `kirocrew doctor` could show the stack but not the job, so he paused the wrong one. Two additions, one reader: * cron_inflight: a run drops `<data home>/cron-running/<job id>.json` (job_id, name, started_at, pid) when it starts executing and clears it on every `finally` path. A marker whose PID is dead is exactly "this job was in flight when that gateway died" -- no schedule inference. Writes are off-loop and best-effort. * stall_attribution: reads the newest stack-bearing dump, names the surface from the OUTERMOST recognised frame of the wedged thread (a cron turn passes through the Slack gateway module, so top-down would misname it), names the permission-gate frame, and joins abandoned markers to the dump by PID. One match names the job; several name candidates and no job; none says so. It never guesses. * CronService.start() runs the breaker on a worker BEFORE the timer is armed: a cron-surface dump plus exactly one matching marker parks that job auto_paused with a last_error naming the dump and the `kirocrew cron resume <id>` command, persisted under the store lock and SEL-audited (auto_paused_loop_stall). Claimed once per dump so a resumed job is not re-paused on the next boot while the dump is still on disk. Ambiguous evidence pauses nothing. * `kirocrew doctor` prints the attribution under the dump's stack -- stuck-in frame, surface, job, `recommended: kirocrew cron pause <id>`, and the job's current pause state from crons.json -- with no gateway running. The boot notification carries the same lines. Tests: the breaker test seeds an OVERDUE strict job and asserts it did not fire after start(); with the breaker call removed the job fires on the first tick (['<id>'] == []), which is the crash loop. Markers are asserted present during a run and gone after normal and raising completion. Attribution covers single/multiple/no marker, PID mismatch, a marker newer than the dump, a live owner, chat/slack/unknown surfaces, and Windows frame paths. test_cron.py::test_cron_schedule fails on untouched origin/main (timezone-dependent) and is unrelated.
Collaborator
Author
|
Superseded by #8282, which carries this change together with its two companions as one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / Motivation
A gateway the loop-stall watchdog hard-exits runs no
finally. The cron run in flight writes nolast_run_tsand no history row, so the store shows the job as never fired; on the next boot it is due again the moment the timer arms, it fires, and it stalls again. A user hit exactly this as an hourly crash.kirocrew doctorshowed the wedged stack but could not say which job it was, so he paused the wrong one (ascriptcron that never touches the model, and so could not have produced that stack).Why it matters
Without a name, the operator's only options are to pause every cron or to read a faulthandler stack. Without a breaker, a single bad input becomes a crash loop that repeats on every boot until someone guesses right. This PR makes the crash self-limiting and self-explaining; #8277 fixes the gate that produced this particular stall.
What changed (motivation → approach → change)
Two additions and one reader, all off the event loop.
cron_inflight— a run writes<data home>/cron-running/<job id>.json(job_id,name,started_at,pid) when it starts executing and clears it on everyfinallypath. A marker whose PID is dead is exactly "this job was in flight when that gateway died": no schedule inference, no timestamp heuristics. Writes areasyncio.to_threadand best-effort (a marker failure never fails a run); reads are size-bounded; job ids that would leave the directory are refused.stall_attribution— reads the newest stack-bearing dump, names the surface from the outermost recognised frame of the wedged thread (a cron turn passes throughslack/gateway.py, so a top-down match would call it Slack), names the permission-gate frame when the stall is inside it, and joins abandoned markers to the dump by PID (andstarted_at ≤ dump mtime). One match names the job; several name candidates and no job; none says so; a chat/channel surface is named and implicates no job. It never guesses.crash_dump_storegains two public readers (dump_owner_pid,dump_wedged_frames) for it.CronService.start()— runs on a worker before_arm_timer(). Only when the surface is cron and exactly one abandoned marker carries the dump's PID does it park that jobauto_paused(enabled=False,last_status="error", alast_errornaming the dump andkirocrew cron resume <id>), persisted under the store lock and SEL-audited ascron_auto_pause/auto_paused_loop_stall. The dump name is claimed incron-running/.loop-stall-breakerso one crash pauses its job once: a dump stays on disk for a week and a job the operator resumed must not be re-paused on the next boot. Abandoned markers are swept after being read, whatever the verdict.kirocrew doctor— under the dump's stack prints anattribution:block: stuck-in frame, surface, job,recommended: kirocrew cron pause <id>, and the job's current pause state fromcrons.json(job_pause_state_from_disk, no gateway needed). An attributed job is added to the issues summary. The boot notification carries the same lines.Sample doctor output on a synthesized data home shaped like the field host:
Trust boundary note:
cron-running/is not a fenced path. A forged marker could at most make the breaker pause one job (audited, with a visible reason naming the dump) and only when a cron-surface dump exists; it cannot enable, run or alter a job.Specs:
docs/system-specs/modules/learn-cron-dashboard.md,docs/system-specs/modules/cli.md.Tests
test/test_cron_loop_stall_breaker.py— the breaker test seeds an overdue stricteveryjob (itslast_run_tstwo periods back, as a killed run leaves it) and asserts it did not fire afterstart(); with the breaker call removed the job fires on the first tick (['<id>'] == []), which is the crash loop. A resumed job is not re-paused on a second boot; a chat-surface dump and a two-candidate dump pause nothing; no dump is a no-op; the pause is SEL-audited once. Markers are present during a run (read from inside the callback) and gone after normal and raising completion.test/test_stall_attribution.py— marker round trip, id validation, unparseable/oversize/tmp files skipped, abandoned-vs-live owners; surface classification for cron (through the Slack module), dashboard chat, Slack, unknown, Windows frame paths; attribution for single/multiple/no marker, PID mismatch, a marker newer than the dump, a live owner;describe()lines; newest-dump selection past a header-only file.test/stall_dump_helpers.py— synthesized dumps in faulthandler's format behind the store's real header.test/test_cron.py::TestComputeNextRunTs::test_cron_schedulefails on untouchedorigin/main(timezone-dependent) and is unrelated.Manual verification
Ran
kirocrew doctoragainst a tempKIROCREW_HOMEholding acrons.json, acron-runningmarker and a dump in the store's format (output above). Existing cron, crash-dump store, cautious-boot and doctor suites pass.Related Issues
Companion to #8277 (linear gate) and #8278 (title-tier off-loop scan). Independently mergeable.
Checklist