Repository navigation
Reconcile #648 with current 0.6 release fixes - #685
Closed
JosiahBull wants to merge 83 commits into
Closed
JosiahBull wants to merge 83 commits into
JosiahBull wants to merge 83 commits into
Conversation
Mia, Tom, Priya, Sam, Alex and a chaos agent run concurrently against one disposable database on the queue-storage engine. An oracle checks duplicate terminal rows, terminal-and-live overlap, unexplained cancels, stalls, promotion lag, lost jobs, cron double-fires and server crashes, and a failed run writes a reproducer. scripts/persona-soak.sh runs the engine matrix in private containers.
… handlers, document the new personas
…h recovery from the statistics reset
…, exit the Python process with its parent
…te, resume paused queues reliably, treat unfinished backlog as a finding
Parent jobs batch-insert unique child jobs into their own queue from concurrent handlers, and some replay a batch with an already-claimed unique key to exercise the unique-violation fallback to row-by-row inserts.
Changes enqueue_shards on two hot queues under live load while producers, promotion and claimers run as non-superuser roles: a schema owner, a migrator role with default privileges, and two application roles in a runtime group role. Covers prepare_queue before and without a raise, runtimes and producer handles that disagree about the cached shard count, and new lanes first created by the maintenance leader's role. The oracle gains per-shard claim cursor and drain checks, ordering_key FIFO within a settled window, execution counts across shard changes, and a scan of the server log for permission-denied and missing-lane errors with the role, lane, sequence owner and statement.
A separate process claims thousands of jobs on a zero-deadline queue and a deadline queue, heartbeats them, and is killed with SIGKILL again and again. After each kill the oracle times the rescue backlog until it clears, watches the claim ring, checks that no job is run again while its owner is alive, and checks attempts and the dead-letter queue against max_attempts. scripts/persona-soak-against.sh runs the harness against another revision of awa, and persona-soak.sh can run a prebuilt harness binary.
The abandon persona now counts the open claims of the running process against the handlers it has started, so a rescue from under a live process is caught even when the job is not run again.
… pass durations AWA_SOAK_HEARTBEAT_SECONDS and AWA_SOAK_STALENESS_SECONDS set the heartbeat interval and staleness of every runtime, including the claiming process, defaulting to the harness's 2 s and 12 s (the library's are 30 s and 90 s). The abandon persona records the duration of each heartbeat pass of the live process, read from the timestamps the heartbeats write, against the number of handlers it holds.
…ious pass's newest heartbeat
Prepare 0.6.14-alpha.1
Fix claim aging race and retained body hydration on 0.6
Fix admin recovery in larger installed ready rings
Keep busy retained recovery from blocking unrelated batches (0.6)
…sient lock contention on 0.6 Retry worker finalization after transient lock contention on 0.6
…-repair-0.6 fix(queue-storage): refresh SET-only lane repairs on 0.6
Promotion set a transaction-local 50 ms lock_timeout before taking its lane locks and kept it through pg_notify and COMMIT. A transaction that issued NOTIFY takes PostgreSQL's notify-queue lock at COMMIT, shared by every notifying transaction, so under concurrent enqueues the commit could time out. Promotion then rolled back the whole lane batch and reported it as a contended lane, delaying due scheduled and retryable jobs. Restore the caller's lock_timeout after the lane's inserts and before the NOTIFY. A lock timeout at COMMIT is now an error, logged per lane and counted by awa.maintenance.promote.commit_lock_timeouts, instead of lane contention.
…iling it (0.6) Hydrating a deleted lease fails when two retained ready bodies of one attempt rank equally. In heartbeat, deadline and receipt rescue that error rolled back the whole rescue transaction, and because the stale attempt sorts first it did so on every tick, stopping rescue for every other attempt in the schema. Rescue now treats such an attempt like one whose claim heads are busy: its savepoint rolls back, it is excluded from the pass, and an error is logged each tick. It stays running until its bodies are repaired; other attempts are rescued. Point reads and direct transitions still fail explicitly on an ambiguous body.
Promotion moved to one transaction per lane, each issuing NOTIFY. A notifying transaction takes PostgreSQL's notify-queue lock at COMMIT, which every notifying transaction in the cluster serialises on, so each lane transaction waited there while holding its lane's locks. Producers on the lane queued behind it, and before the previous commit the wait ran under the 50 ms lane lock timeout. Promotion now notifies once per batch after its lanes commit, outside any lane transaction. The wakeup is a hint claimers also poll for, so a failed NOTIFY is logged and the promotion stands. This supersedes the lock_timeout restore and the commit-timeout counter, which no longer have anything to guard.
--sql took precedence and nothing was extracted, so a runner following the external-runner steps applied SQL it never wrote to disk. The upgrade guide now gives the extract command on its own.
… (0.6) The worker re-reads a completed or parked job with load_job to build its lifecycle event. That read runs after the transition commits, but a failure made the whole outcome an error: the job was logged as failed to complete, its event and metrics were dropped, and a lock-timeout error there made finalization retry the committed transition. A failed re-read now logs a warning and the event is built from the worker's copy, as when the job is no longer found.
…sion The persona soak harness uses the ci_timing helper module, so soaking an older tag failed to compile without it.
With a lifecycle hook registered, every completion re-read the job with load_job to build its Completed event. load_job now reads from one REPEATABLE READ snapshot across several savepoint legs, so this added about 24 round trips per completion and dominated hook-enabled workloads. A completed attempt changes only its state, finalized_at and progress, so the event is built from the job the worker already holds.
…nd completion events (0.6) fix(queue-storage): promotion notify, rescue ambiguity, completion events (0.6)
* chore: prepare 0.6.14 release metadata and notes Bump the Rust and Python packages from 0.6.14-alpha.1 to 0.6.14 and turn the alpha notes into the 0.6.14 release section, including the promotion, rescue, lifecycle-event and migrate CLI fixes since the alpha. * docs: rewrite the 0.6.14 release notes for operators upgrading from 0.6.13 * docs: flow the 0.6.14 release notes and changed docs onto single lines The release workflow publishes the CHANGELOG section as the GitHub release body, where hard-wrapped lines render as line breaks. * docs: align substrate, troubleshooting, security and model mapping with 0.6.14 Fix the substrate doc's stale lane-provisioning precondition and its broken link to the renamed security section. Cover the two new 'still running' cases in troubleshooting, note the definer provisioning functions in the security guide, update the TLA+ code mapping for per-lane promotion, rescue deferral and aging fences, and record that rescue liveness excludes ambiguous legacy bodies.
Preserve finalizer admission backpressure together with the release branch heartbeat ownership and contention retries. Retain both sets of queue-storage regressions, move pending changes to Unreleased, and cover shutdown of a saturated finalizer budget for both storage engines.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Collaborator
Author
|
Superseded by #686, which contains the complete fix directly against release/0.6.6. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reconciles #648 with
release/0.6.6ata8febdc0. Merge this intobrian/fix-finalize-backpressure-0.6to update #648 and resolve its release-branch conflicts.Preserves bounded finalizer admission together with the release branch's heartbeat ownership and finalization contention retries. Keeps both sets of queue-storage regressions, moves the pending changes to Unreleased, and adds a regression proving that a saturated finalizer budget retains worker/heartbeat ownership until shutdown cancellation releases the waiter, for both storage engines.
Validation on Rust stable and a local PostgreSQL 18 instance:
test_queue_storage_enqueue_batch_writing_more_segments_than_the_old_bind_limitexpects over 8,191 segments but produced 4,479. It also fails on unmodifiedrelease/0.6.6ata8febdc0with 4,100 segments.PostgreSQL 17 and the persona soaks have not been rerun for this reconciled head.