Repository navigation
test(persona-soak): claim-and-abandon persona (SIGKILLed process with a large claim backlog) - #644
Conversation
Mia, Tom, Priya, Sam, Alex and a chaos agent run concurrently against one disposable database on the queue-storage engine. An oracle checks duplicate terminal rows, terminal-and-live overlap, unexplained cancels, stalls, promotion lag, lost jobs, cron double-fires and server crashes, and a failed run writes a reproducer. scripts/persona-soak.sh runs the engine matrix in private containers.
… handlers, document the new personas
…h recovery from the statistics reset
…, exit the Python process with its parent
…te, resume paused queues reliably, treat unfinished backlog as a finding
Parent jobs batch-insert unique child jobs into their own queue from concurrent handlers, and some replay a batch with an already-claimed unique key to exercise the unique-violation fallback to row-by-row inserts.
Changes enqueue_shards on two hot queues under live load while producers, promotion and claimers run as non-superuser roles: a schema owner, a migrator role with default privileges, and two application roles in a runtime group role. Covers prepare_queue before and without a raise, runtimes and producer handles that disagree about the cached shard count, and new lanes first created by the maintenance leader's role. The oracle gains per-shard claim cursor and drain checks, ordering_key FIFO within a settled window, execution counts across shard changes, and a scan of the server log for permission-denied and missing-lane errors with the role, lane, sequence owner and statement.
A separate process claims thousands of jobs on a zero-deadline queue and a deadline queue, heartbeats them, and is killed with SIGKILL again and again. After each kill the oracle times the rescue backlog until it clears, watches the claim ring, checks that no job is run again while its owner is alive, and checks attempts and the dead-letter queue against max_attempts. scripts/persona-soak-against.sh runs the harness against another revision of awa, and persona-soak.sh can run a prebuilt harness binary.
The abandon persona now counts the open claims of the running process against the handlers it has started, so a rescue from under a live process is caught even when the job is not run again.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
… pass durations AWA_SOAK_HEARTBEAT_SECONDS and AWA_SOAK_STALENESS_SECONDS set the heartbeat interval and staleness of every runtime, including the claiming process, defaulting to the harness's 2 s and 12 s (the library's are 30 s and 90 s). The abandon persona records the duration of each heartbeat pass of the live process, read from the timestamps the heartbeats write, against the number of handlers it holds.
…ious pass's newest heartbeat
hardbyte
added
the
full-ci
ChatGPT Codex Connector
Retain the existing soak scenarios and oracle controls on the current release code. Add an optional shared-creator grant profile for older-version comparisons and clarify terminal-observation and retained-session evidence.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Adds a claim-and-abandon persona to the multi-runtime soak for large abandoned-claim backlogs (#627), including the earlier #624/#639 harness work. The branch now incorporates the merged release fixes through #660.
A separate copy of the test binary claims jobs on compact and row-local claim queues, holds their handlers with heartbeats running, and is killed with SIGKILL. Replacement processes reclaim the jobs; a surviving runtime leads maintenance and serves a canary. After the last kill, an in-process runtime finishes the remaining work. A quarter of jobs have max_attempts=2 so repeated kills also exercise exhaustion.
The oracle checks per-kill rescue bounds, claim-ring recovery, rescue of a live owner, consecutive attempts, and DLQ limits. It retains execution records by process generation so a retry is allowed only after the previous owner died. Backlog samples, heartbeat pass timing, and per-kill tables go into the report.
A missing-claims sample is classified as rescue of a live owner only while the same process generation remains live after the database read. A read overlapping that generation's kill or replacement yields INCONCLUSIVE when its snapshot cannot be placed before the signal; earlier proven live-owner loss remains a failure. Successful signaling and death publication share the observation fence. Failed signaling or reaping makes the evidence incomplete; a failed signal records no death boundary.
Suspected claim-ring stalls also collect supplemental lock evidence into claim-ring-locks.jsonl. Captures begin only after an existing episode reaches half its architectural bound; four captures are reserved for actual violations, with at most eight attempts per run. The catalog-only observer records the incoming relations, holder and waiter PIDs, lock modes, activity, blocker edges and server timestamps. It caps retained locks and blocker lookups, uses 250ms statement and 100ms lock timeouts within a 750ms client deadline, and explicitly reports truncated, missing or privilege-limited evidence. These are dynamic snapshots, not a complete historical blocking graph. The capture does not change oracle verdicts or time bounds.
The claim-ring probe retains compact and row-local receipt identities after they materialize into running or callback-waiting leases. Running leases use authoritative attempt heartbeats; callback waits use callback_timeout_at and ignore the original heartbeat and job deadline. Durable closure and terminal records still remove the protector. Queue backlog probes retain their existing running-only semantics, and the 120s bound is unchanged.
Handler return is execution evidence. Lifecycle hooks are best-effort, so ordinary jobs that returned without a callback are also checked against committed terminal rows for their exact (job_id, run_lease). The observer caps explicit pending attempts and results at 100,000 rows and retains its two-second statement timeout; those caps do not promise a fixed physical scan count. NO_EVENTS workloads keep their optimized path: the same bulk observer checks only the immutable job_id % 64 = 0 sample every second. Reports count sampled, confirmed, and excluded jobs; the other 63/64 returned NO_EVENTS jobs have no per-job terminal-commit guarantee. Missing proof after pruning, failed queries, or exceeded caps produce INCONCLUSIVE instead of a pass or an unsupported loss claim. Observed durable proof survives later pruning.
Observed database outcome flags belong to the greatest proven run lease. A higher durable or lifecycle proof clears older DB outcome flags, unordered older rows cannot restore them, and conflicting states at the same lease remain visible. This keeps a historical cancellation from becoming the reported outcome of a newer completed attempt. Genuine cancellation of the current attempt still fails its existing expectation rules. Unexpected-outcome records now retain bounded persona, queue, flags, proof and execution leases, event counters and operator/drain timing context for diagnosis.
Python migration probes use a unique application name and observe idle transactions only after the child reports that migration and its startup queries completed. The pool limit applies throughout migration; every backend remains counted regardless of age. Missing readiness samples, failed session queries or probe timeouts make the result INCONCLUSIVE. A closed-only sample cannot stand in for a live idle-pool sample. Retained sessions prove close failure only after successful process exit and a CLOSED marker emitted after client.close() returns. Forced exits or missing markers retain the session count as incomplete evidence. The existing two-second post-exit budget covers both session queries and delayed stdout marker consumption; failed final queries invalidate retained-session proof without hiding pool-limit or post-ready idle-transaction failures. Five phase and real-database regressions pass on PostgreSQL 17 and 18, including healthy migration transactions, a real post-ready idle transaction, retained sessions and observer failure. Replaying the former unphased rule rejects the healthy transaction on both majors.
Python workers register their stop-production signal before migration starts and remember it through startup. A process restarted during drain starts workers with producers disabled. Liveness checks continue during drain, outstanding migration probes cannot delay the stop signal, and unfinished probes or failed restart/evidence checks make the result INCONCLUSIVE. The drain deadline is unchanged. Two actual-script process regressions fail before the fix and pass on CPython 3.13 and 3.14; three Rust lifecycle checks cover process recovery, failed spawning and a blocked probe.
Assertion waits and heartbeat staleness use ci_timing helpers. Load duration, measurement windows, kill schedules, and workload rates remain unscaled.
Independent scale knobs: AWA_SOAK_ABANDON_CLAIMS (default 4000), AWA_SOAK_ABANDON_KILL_SECONDS, AWA_SOAK_ABANDON_KILLS, AWA_SOAK_ABANDON_BOUND_SECONDS (default 120), and AWA_SOAK_RESCUE_INTERVAL_SECONDS. scripts/persona-soak-against.sh runs this harness against another revision; scripts/persona-soak.sh accepts SOAK_TEST_BIN for a prebuilt executable.
Container cleanup removes anonymous PostgreSQL volumes for successfully created containers only. A failed docker run does not register ownership or remove an existing same-name container; a stubbed collision check demonstrates the old failure and corrected behavior.
Earlier validation: required workspace fmt, offline clippy with warnings denied, and build passed. Two materialized-claim regressions fail with the original Slot query on PostgreSQL 17 and 18 and pass after the fix; both compact and row-local receipts cover running expiry, authoritative heartbeat refresh, callback waiting beyond its original deadline, callback timeout, durable completion, and another slot. All 30 database, timing, oracle and process lifecycle checks passed on both majors, and the two Python process regressions also pass. Six additional terminal-proof regressions cover a committed non-sampled eventful job without a callback, unfinalized work, proof pruned before observation, query failure, exact-attempt fencing, and unchanged NO_EVENTS sampling. The positive case is RED with the old observer on both majors after independently proving the database commit; the final ten terminal-proof checks are GREEN on both. Four injected oracle regressions passed on PostgreSQL 17 and 18: returned handler without finalization, failed final lookup, pruned unobserved completion, and observed completion retained across pruning. Ten timing-helper tests passed. Thirty-second Priya-only smoke passed on both majors: PG17 5,950 jobs with 92/92 sampled durable proofs; PG18 5,537 jobs with 86/86. These are harness checks, not bounded-rescue acceptance under mixed load.
Current temporal-evidence validation: required workspace fmt, offline Clippy with warnings denied, and build passed. Six deterministic controls pass without a database. Restoring only the old classification makes the kill-overlap and prior-loss controls fail logically, while still-live loss and unrelated-generation controls continue detecting live-owner rescues. Direct signal controls verify successful publication and that a failed signal leaves the generation live without a death timestamp. Historical mixed-load failures remain failure evidence; this follow-up does not establish bounded-rescue acceptance.
Supplemental lock-capture validation: required workspace fmt, offline Clippy with warnings denied, and build passed. All five focused controls pass on PostgreSQL 17 and 18: a real AccessShare holder and AccessExclusive waiter expose the expected relation, PID, mode and blocker edge; evidence clears after release; unresolved relations, query failure, output caps, restricted secondary activity and stage budgets stay explicit. The six temporal-evidence controls also remain green. Historical mixed-load verdicts are unchanged.
Current terminal-state validation: required workspace fmt, offline Clippy with warnings denied, and build passed. Five deterministic controls pass. Restoring only the old ledger method bodies makes both old-cancel/new-completion orderings and the lifecycle-completion sequence fail the actual final classifier; the newer-cancellation and same-lease-conflict controls remain passing controls that require the cancellation violation. Historical mixed-load reports, including the unexplained job50176 cancellation, retain their original verdicts; this follow-up does not identify that job's missing per-attempt context or establish acceptance under mixed load.
Current close-evidence validation: required workspace fmt, offline Clippy with warnings denied, and build passed. Eight deterministic controls pass. Restoring only the old verdict methods and premature post-exit observation inside the same helper API makes four controls fail: forced exit during a read, success without CLOSED, delayed stdout consumption, and a positive session count followed by query failure. The actual asynchronous settling helper is exercised directly. Successful CLOSED with retained sessions still fails, and pool-limit and ready idle-transaction checks remain independent. No historical report is reclassified and no time bound is changed.
Backlog-probe correlation validation: the closure-range membership subquery keeps the existing slot and receipt predicates correlated with OFFSET 0. In the retained post-run database, the former slot-only anti-join evaluated 32,183,226 rejected range pairs; the correlated variant preserves the same stable projection for all 822 holders. These are NEWSTATE plan and result controls, not the original slow backend executor plan or mixed-load acceptance. Required workspace formatting, all-target/all-feature Clippy with warnings denied, and build pass. All eight existing abandonment controls pass on PostgreSQL 17 and 18, including materialized running and callback-waiting receipts; workload rates, oracle rules and time bounds are unchanged.
Earlier v0.6.13 measurements used 25,000 claims per kill, four kills 300s apart, a 1500s load, and a 120s bound:
Mixed runs overlapped previous backlogs and did not quiesce within a 900s drain; they also observed rescue of live claims. Those measurements predate the sampled terminal-proof checks above and are failure baselines for the rescue fix.
Refs #639, #627.
Current integration: both Rust workspaces pass formatting, offline all-target/all-feature Clippy with warnings denied, and workspace builds. All 38 harness/script files match the frozen comparison harness; production source matches the current release branch. Existing Persona soak documentation and the shutdown-finalization coverage row are preserved.
An optional AWA_SOAK_SHARED_CREATOR_GRANTS=1 setup shares application-created objects when comparing older binaries. It requires schema CREATE and defaults off; leave it off for application-role privilege regression coverage. This does not change library provisioning behavior. Full-size mixed-load acceptance and the lock-pressure release gate remain separate from the harness regression checks.