Repository navigation
Conversation
The executor removed an attempt from the heartbeat set and released its worker slot as soon as the handler returned, then queued its finalize on the completion batcher. When finalization was slower than claiming, the dispatcher kept claiming and the queue of finished, unheartbeated attempts grew without bound. Those that waited past heartbeat_staleness were rescued and run again. Admit at most the completion pipeline's depth (2 x batch size x shards, and at least one per worker) of attempts to finalization. Past that, a finished attempt keeps its worker slot and heartbeat until there is room, which stops the dispatcher claiming. This is the bound main already has.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
hardbyte
added
the
full-ci
ChatGPT Codex Connector
Address compact claims by their batch ID and item index, validating the job, attempt and receipt before closing. Avoid expanding every retained compact batch for each metadata or retry completion flush, and avoid locking immutable shared batch rows. Preserve per-attempt advisory serialization, row-local prelocking and the fresh deciding snapshot. A 50,000-claim recovery rehearsal with heartbeat protection timed out waiting 120 seconds for retries on PostgreSQL 17 and 18 before this change. With keyed completion, the same full rehearsal passes in 116.35s and 123.30s total. A new regression rejects a wrong item index, completes while the immutable batch row is locked elsewhere, and checks retained tags and metadata. Validation: cargo fmt --all; SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings; SQLX_OFFLINE=true cargo build --workspace; eight focused metadata, stale-completion, cancellation, idempotence and finalization-bound tests plus the shutdown-waits test on PostgreSQL 17 and 18.
hardbyte
changed the title
This was referenced Nov 6, 2026
This was referenced Nov 7, 2026
Merged
This was referenced Nov 8, 2026
JosiahBull
added a commit
to JosiahBull/awa
that referenced
this pull request
Nov 8, 2026
Carry the complete fix from hardbyte#648 onto the current 0.6 release branch, preserving heartbeat ownership and lock-contention retries. Add coverage for shutdown while finalizer capacity is saturated. Co-authored-by: Brian Thorne <brian@hardbyte.nz>
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finished handlers could outpace committed completions, leaving an unbounded backlog. On 0.6, metadata and retry completions also expanded every retained compact batch on each flush.
Bound attempts waiting for finalization to
max(runtime worker capacity, 2 × completion batch size × completion shard count). A finished handler retains its worker slot and heartbeat while waiting for capacity. Compact metadata completions address the batch ID and one-based item index, validate the exact job/run lease/receipt, and avoid locking the shared immutable batch. The PostgreSQL 18 prelock and fresh deciding snapshot remain.Reconciled with the merged release fixes at
65c62f85, preserving #643's tracked graceful-shutdown finalizers. No migration or public API change. #650 supplies heartbeat protection after admission to finalization; it is still needed when a completion flush exceeds heartbeat staleness.Validation of the reconciled tree:
Full CI runs for the pushed head. This PR remains open pending the broader release gates; the matched PostgreSQL 17 and assert-enabled persona soaks have unresolved failures.