Repository navigation
fix(worker): bound finalization backlog and key metadata completions (0.6) - #695
Merged
Merged
Conversation
Carry the complete fix from #648 onto the current 0.6 release branch, preserving heartbeat ownership and lock-contention retries. Add coverage for shutdown while finalizer capacity is saturated. Co-authored-by: Brian Thorne <brian@hardbyte.nz>
…stic Exercise the same batch writer from its private test module with explicit distinct microsecond run_at values. Preserve the 9,000-row insertion, segment-count and claim assertions without relying on host clock resolution.
…d-finalization-0.6 # Conflicts: # CHANGELOG.md # awa-model/src/queue_storage.rs # awa/tests/queue_storage_runtime_test.rs
A finished handler waiting for finalizer capacity could win a permit released after shutdown cancelled its wait, and spawn a finalizer that outlives shutdown. The wait now checks cancellation first.
The stall test now runs for both completion paths: receipt claims, stalled by the completion advisory locks, and materialized leases, stalled by holding done_entries in SHARE mode. With the finalizer bound disabled both fail.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Add Carrot credits or activate Agent usage billing to use Autofix |
hardbyte
added
the
full-ci
Every attempt with lifecycle hooks spawned a detached task for Started, and its outcome hooks ran after the finalizer released its slot, so hooks slower than job throughput accumulated tasks each holding a copy of the job without slowing claiming. The Started task is now awaited inside the finalizer, before the outcome hooks, and the slot is held until both return. Maintenance's Rescued and duplicate Cancelled events stay detached so hooks cannot hold up rescue, but at most 1,024 are outstanding; further events are dropped with a warning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries #686 by @JosiahBull onto the current
release/0.6.6, so CI runs in this repository, and supersedes #686 and #648. Jo's two commits are unchanged.Summary
When handlers finish faster than completions commit, detached finalizers pile up without bound, each holding its job's payload. Under a 120 s completion stall that reached about 77k pending attempts and 1 GB of worker memory. Finalizer admission is now bounded to
max(worker capacity, 2 × completion batch size × completion shards). A finished handler keeps its worker slot while it waits, so the runtime stops claiming instead of growing; under the same stall it holds about 1,068 attempts, with memory 55 → 65 MB and no measurable throughput change. It also completes retried and metadata-carrying compact receipt claims by exact claim batch and item index.Added on top:
release/0.6.6. The metadata-completion CTE keeps this PR's exact-index lookup in place of thejob_idsGIN probe the branch gained in fix(queue-storage): index compact claim batches by job id (0.6) #684; the index still servesload_job, heartbeat, rescue and the other by-job lookups.Startedtask is awaited inside its finalizer before the outcome hooks, and the slot is held until both return, so hooks slower than job throughput slow claiming instead of accumulating tasks that each hold the job (reproduced: 116k tasks and +1.78 GB in 10 s with a blocking hook, throughput unchanged). Maintenance'sRescuedand duplicateCancelledevents stay detached so hooks cannot hold up rescue, capped at 1,024 outstanding with further events dropped and warned.test_queue_storage_slow_lifecycle_hook_holds_back_claimingfails without the change.done_entriesinSHAREmode). With the bound disabled both fail.Review notes
AWA_COMPLETION_BATCH_SIZEto shrink the bound in proportion.Testing
Format and clippy clean;
awa-workerunit tests,lifecycle_hook_test, and the stall, slow-hook, shutdown, metadata-completion andclaim_batch_job_indexintegration tests pass on PG17.