Skip to content

feat(jobs): resume jobs after a restart - #99

Merged
buzz-code merged 1 commit into
mainfrom
claude/jobs-survive-restart
Nov 10, 2026
Merged

buzz-code merged 1 commit into
mainfrom
claude/jobs-survive-restart

Conversation

@buzz-code

Copy link
Copy Markdown
Owner

Requested by הדסה · project thread

Problem

  • A job running when its worker was killed stayed running forever. Nothing released it.
  • A graceful shutdown only stopped the timer, so the running job ended up the same way.

Change

  • Lock heartbeat: while a handler runs, the worker renews lockedAt every JOB_LOCK_TTL_MS / 4. Before each claim it puts running jobs whose lock is older than JOB_LOCK_TTL_MS (default 10 min) back to pending (JobService.releaseStale).
  • Graceful shutdown (onModuleDestroy, now async): the worker claims nothing more and aborts the new JobContext.signal (the handler's optional 2nd argument). It then waits up to JOB_SHUTDOWN_WAIT_MS (default 25s, which fits a 40s stop_grace_period).
    • A handler that throws after the abort goes back to pending without using up an attempt (JobService.release).
    • Claimed jobs that haven't started are handed back.
  • This is additive: existing handlers ignore the new argument and behave as before. No CONSUMER_CHANGES row is needed.

Consumer: buzz-code/dnd-management-nra#180 (generate_audio moves onto jobs).

Tests

  • New specs cover stale release, heartbeat, abort+release on shutdown, completing during the wait, and handing back unstarted jobs.
  • utils/jobs: 50/50 pass (run from the dnd consumer).

Generated by Claude Code

A job running when its worker was killed stayed 'running' forever, and a
graceful shutdown just stopped the timer, leaving the running job to the same
fate.

- The worker renews a running job's lock while its handler runs, and before
  claiming releases running jobs whose lock is older than JOB_LOCK_TTL_MS
  (10 min) back to pending.
- On shutdown it claims nothing more, aborts the handler's new
  JobContext.signal and waits up to JOB_SHUTDOWN_WAIT_MS (25s). A handler that
  stops on the signal (throws) goes back to pending without using an attempt;
  claimed jobs not yet started are handed back.

Handlers that ignore the new optional argument behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GTictazJTxnndn2m4LbtH
@buzz-code buzz-code self-assigned this Nov 10, 2026
@buzz-code
buzz-code merged commit df80f42 into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants