Skip to content

fix(worker): heartbeat attempts until durable finalization - #649

Closed
hardbyte wants to merge 3 commits into
mainfrom
brian/heartbeat-until-finalized
Closed

hardbyte wants to merge 3 commits into
mainfrom
brian/heartbeat-until-finalized

Conversation

@hardbyte

@hardbyte hardbyte commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

A handler can return while its completion write is still queued or blocked. Removing that attempt from the heartbeat registry lets rescue run the job again before its first completion commits.

Keep the attempt registered through finalization and release it with a drop guard created before the finalizer-capacity wait. Finalizing attempts stop flushing progress. Heartbeats skip held closer locks, use a fresh statement snapshot to check closure evidence, and cannot recreate state for a closed claim. Legacy lease and canonical progress updates skip locked rows; skipped progress remains eligible for retry.

Validation: The stalled-finalizer regression uses a separate trigger gate after completion owns all of its attempt locks, then holds it for four five-second staleness windows. The identical fixture fails with duplicate executions before the heartbeat-retention fix and passes afterward on PostgreSQL 17 and 18, retaining durable-completion and exact-execution assertions. A measured WAL-sync delay let the earlier one-second fixture reach 1.315s of heartbeat age under contention; the new margin keeps that capacity limit separate from the retention invariant.

Required formatting, offline all-target/all-feature Clippy (-D warnings), and workspace build pass. Full CI is temporarily deferred while the approved release merge checks run.

No schema migration.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@hardbyte hardbyte added the full-ci Run the full CI matrix (Python build+test, E2E) on this PR label Nov 6, 2026

hardbyte commented Nov 7, 2026

Copy link
Copy Markdown
Owner Author

Parked under the branch policy now recorded in #383. Fixes land on release/0.6.6 first, and main takes one bundled forward-port after each 0.6 release. Part of this change may already be on main through #664's finalization heartbeat guard. The forward-port will reconcile the rest or close this PR as superseded.


Generated by Claude Code

@hardbyte

hardbyte commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

Superseded: main already keeps finalizing attempts in the heartbeat registry until completion commits (FinalizationHeartbeatGuard).

@hardbyte hardbyte closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full CI matrix (Python build+test, E2E) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant