Repository navigation
Conversation
The executor removed an attempt from the heartbeat set and released its worker slot as soon as the handler returned, then queued its finalize on the completion batcher. When finalization was slower than claiming, the dispatcher kept claiming and the queue of finished, unheartbeated attempts grew without bound. Those that waited past heartbeat_staleness were rescued and run again. Admit at most the completion pipeline's depth (2 x batch size x shards, and at least one per worker) of attempts to finalization. Past that, a finished attempt keeps its worker slot and heartbeat until there is room, which stops the dispatcher claiming. This is the bound main already has.
Address compact claims by their batch ID and item index, validating the job, attempt and receipt before closing. Avoid expanding every retained compact batch for each metadata or retry completion flush, and avoid locking immutable shared batch rows. Preserve per-attempt advisory serialization, row-local prelocking and the fresh deciding snapshot. A 50,000-claim recovery rehearsal with heartbeat protection timed out waiting 120 seconds for retries on PostgreSQL 17 and 18 before this change. With keyed completion, the same full rehearsal passes in 116.35s and 123.30s total. A new regression rejects a wrong item index, completes while the immutable batch row is locked elsewhere, and checks retained tags and metadata. Validation: cargo fmt --all; SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings; SQLX_OFFLINE=true cargo build --workspace; eight focused metadata, stale-completion, cancellation, idempotence and finalization-bound tests plus the shutdown-waits test on PostgreSQL 17 and 18.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
hardbyte
added
the
full-ci
This was referenced Nov 7, 2026
Merged
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #648. Backport of #649; no schema migration.
A finished handler can wait behind a blocked completion write long enough to be rescued and executed again. Keep its attempt in the heartbeat registry until durable finalization finishes, including while waiting for a finalizer permit. A drop guard removes it on completion or abandonment. Stop flushing its progress after the handler returns.
Compact receipt heartbeats address exact batch items and skip attempts whose closer lock is held. Take locks before the separate closure-check/upsert statement so a late snapshot cannot revive a completed attempt. Legacy lease and canonical progress writes skip locked rows; partial progress passes remain eligible for retry.
Validation: The stalled-finalizer regression uses a separate trigger gate after completion owns all of its attempt locks, then holds it for four five-second staleness windows. The identical fixture fails with duplicate executions before the heartbeat-retention fix and passes afterward on PostgreSQL 17 and 18, retaining durable-completion and exact-execution assertions. A measured WAL-sync delay let the earlier one-second fixture reach 1.315s of heartbeat age under contention; the new margin keeps that capacity limit separate from the retention invariant.
Reconciled with the merged release fixes at
65c62f85, preserving tracked shutdown and both live-ready point reads. Required formatting, offline all-target/all-feature Clippy (-D warnings), and workspace build pass again. Nine focused runtime regressions pass on each of PostgreSQL 17 and 18, covering the twenty-second finalizer hold, skipped closers, retryable partial progress, shutdown, metadata completion and point reads. Full CI runs for the pushed head; it is still pending.Combined #647/#648 recovery test also passes on both PG17/18 after #648's indexed metadata-completion fix: 50k abandoned claims drained, ring rotation resumes, and all45k retried jobs durably complete with exactly one recovery execution and no live-owner rescues.
The matched persona soak is a separate release gate: PostgreSQL 18 passed, while PostgreSQL 17 prune contention and the assert-enabled run remain unresolved. Earlier strict recovery passes do not clear those results.