[hardbyte]

Review prerequisite

Merge this branch only with #644, including the corrected shared oracle at 412419f864567c29386d9651cad93b82f6ddd997. This head still uses the earlier oracle. The corrected harness requires durable attempt-specific terminal proof, treats failed final lookups as inconclusive, accounts for materialized claims, and handles Python startup/drain restarts. It correlates migration sessions with probe phases, fences reads crossing process kills, keeps terminal observations tied to the greatest proven lease, and requires successful CLOSED proof before reporting a retained-session leak. Ambiguous or missing evidence remains inconclusive; still-live losses and proven conflicts remain failures.

Historical results below predate these corrections and need validation with the corrected harness before release acceptance. Those reports have not been recomputed or retrospectively cleared.

What this adds

A persona-driven soak harness with an invariant oracle, for validating a release candidate and as the base for later readiness work. Eleven personas run concurrently against one disposable database on the queue-storage engine, the way several applications share an awa schema.

• awa/tests/persona_soak_test.rs is one ignored test binary; the harness is in awa/tests/persona_soak/ (config, connection budget, ledger, oracle, report, one module per persona) and awa/tests/persona_soak/python/soak_python.py.
• scripts/persona-soak.sh runs the engine matrix in private Docker containers (postgres:17, postgres:18, an assert-enabled -O0 PostgreSQL 18 image) on separate ports and removes them afterwards.
• docs/test-plan.md has a "Persona soak" section with the variables.

Personas

Persona	Workload
Mia, ML pipeline	0.1-60 s jobs, per-job deadlines, progress, retries into the DLQ, handler cancels, latency spikes
Tom, multi-tenant	many queues at a low rate, unique jobs, scheduled jobs, cron on two runtimes, priorities with aging
Priya, firehose	one hot queue, 100k-row insert_many bursts (above the bind limit), COPY batches, tiny jobs, three runtimes
Sam, ordered streams	4 enqueue shards with 64 ordering keys, a 4-stripe queue (queue_stripe_count), scheduled jobs, a rate-limited queue
Alex, operator	pause/resume, drain, cancel and bulk cancel, move and reprioritise batch operations, DLQ retry and purge, the admin read queries, rolling restarts
Chaos	kills a runtime or the Python process without a shutdown handshake, terminates the maintenance leader's connection and random runtime connections, exhausts a pool, holds an idle-in-transaction session
Watchers	dashboard tabs polling overview, queues, counts, timeseries, runtime and storage at the UI's own interval, through the real awa serve server
Support engineer	job search by kind, queue, state and tag, before_id paging, job detail, DLQ, runtime, cron and kind pages
UI operator	the UI's mutation endpoints: queue and cron pause/resume, cancel, bulk cancel, DLQ retry and purge, batch operations
Impatient user	requests abandoned after a few milliseconds and bursts of refreshes, sampling the server's database sessions
Python users	a separate awa-python process: single inserts, a 6,500-row transactional insert_many, COPY, enqueues in transactions that sometimes roll back; async workers (complete, raise, RetryAfter, Snooze, progress, deadline overrun, a handler that ignores its deadline) on queues one of which is shared with a Rust runtime; a periodic migrate-then-query probe

Invariants

Checked while the load runs and again after it stops, against a client-side ledger (terminal rows are pruned by ring rotation within seconds, so the database alone cannot answer):

• more than one terminal row for a job attempt, a job reported completed twice, a job terminal and still available, deferred or leased
• a cancel nobody asked for, an operator cancel that overwrote a completion, a handler starting after its operator cancel committed
• stall: claimable jobs, idle workers, a claim cursor that does not move; promotion stall: scheduled jobs overdue
• a lost job (enqueued, never terminal once the system is quiet, not removed by an operator), a cron fire that produced two jobs
• a transaction the Python application rolled back that left a job in the store; sessions stranded by migrate()
• an admin-server query outliving its statement timeout; sessions still running after every UI client stopped
• a server crash: pg_postmaster_start_time(), crash-recovery detected from the statistics reset time, and terminated by signal, TRAP:, server process (PID and similar lines in the server log

Slow or failing admin endpoints and latency that grows with history or backlog are reported as non-fatal findings. A failed run writes report.md, report.json and reproducer.md (seed, personas, violating rows, timeline, server log excerpt).

Running it

AWA_SOAK_DATABASE_URL=postgres://postgres:pw@localhost:5432/awa_soak_run AWA_SOAK_SECONDS=900 \
  cargo test --release -p awa --test persona_soak_test -- --ignored --nocapture

scripts/persona-soak.sh [pg17|pg18|pg18-cassert ...]

The UI personas need cargo build --release -p awa-cli; the Python persona needs a virtualenv built with uv venv awa-python/.venv && uv pip install maturin && (cd awa-python && maturin develop --release). The script does both. AWA_SOAK_PERSONAS, AWA_SOAK_SECONDS, AWA_SOAK_SEED, AWA_SOAK_SCALE, SOAK_PG_CPUS and the oracle thresholds are described in docs/test-plan.md.

Results

All runs on the queue-storage engine, load phase length in the first column, seed and persona set as shown. Containers were capped at 12 CPUs for the 75-minute runs and the final PostgreSQL 17 run; the host has 24.

Run	Base	Personas	Jobs in ledger	Result
PG17, 15 min, scale 1	pre-#614/#617/#621	6	1,074,891	one promotion_stall (3 jobs, 10.2 s) right after a graceful restart of the leader: the leader handover gap (leader_election_interval, 10 s). The harness did not widen its window after restarts; fixed
PG18, 15 min, scale 1	pre-#614/#617/#621	6	1,079,424	one lost_job (48 firehose jobs). Harness race: a producer overwrote the ledger entry of jobs a worker had already run; fixed. Not reproduced since, with forensics enabled
PG18, 15 min, scale 1	3c35b7f	11	1,076,154	cron "double fire" (manual UI triggers carry no fire time), a queue left paused by a failed resume, an under-sized Python deadline queue; all harness issues, fixed. promotion_stall x3 (15-19 s) while the admin queries held the container at 1,839% CPU
PG17, 15 min, scale 0.5	3c35b7f	11	601,506	pass, no violations
PG18 release, 75 min, scale 0.5	3c35b7f	11	2,590,760	no duplicate terminal rows, lost jobs, stalls, unexplained cancels or crashes. promotion_stall x31 (overdue up to 124 s), see below
PG18 assert-enabled -O0, 75 min, scale 0.1	3c35b7f	11	712,569	no duplicate terminal rows, lost jobs, stalls, unexplained cancels, assertion failures, terminated by signal lines or crash recovery. promotion_stall x24 (up to 140 s) on a server at 1,190% CPU

The assert-enabled run used the instrumented awa-pg18-cassert:18.6-O0-epqlog5 image, which logs every run-time partition-pruning setup with its query text (about 190 MB/s under this workload). scripts/persona-soak.sh filters those messages through a FIFO for that image and defaults to the uninstrumented awa-pg18-cassert:18.6-O0 build, which needs no filter.

75-minute run activity (PG18 release): 2.0M firehose jobs and 25 100k-row bursts, 376k Python jobs (56 batch inserts of 6,500 rows, 3,395 rolled-back enqueues, none visible afterwards), 1,851 cron fires, 37 runtime kills, 19 Python kills, 28 leader terminations, 405 terminated connections, 19 pool exhaustions, 16 rolling restarts, 33 queue pauses. Peak 225 connections of 400, database 3.7 GB at the end, container average 714% CPU (peak 1,221%) and 9 GB memory.

Observations on AWA

These are not invariant violations; they are what the UI and operator personas saw.

• GET /api/kinds calls admin::job_kind_overviews uncached. Its current_jobs query takes about 2 s with fresh planner statistics and did not finish in 120 s after the schema's statistics were removed. Under mixed load it ran past 150 s in every 15-minute run. The other overview endpoints degrade with backlog and retained history too: with the PG18 75-minute run's firehose backlog, /api/queues, /api/stats, /api/stats/timeseries and /api/jobs had median latencies of 2.5-7 s and p99 of 38-41 s. Each slow poll starts another query, so the pool fills and the server returns 500s.
• GET /api/jobs?tag=... did not answer within a 45 s client timeout against a large table.
• A UI request abandoned by the client keeps its database query running until the statement timeout (23 of 30 s in the smoke run), holding a pool connection.
• Promotion lagged beyond the leader election interval in the 75-minute runs, in episodes of up to 124 s on PG18 release and 140 s on the -O0 server. The episodes follow leader terminations and rolling restarts of the leader but also occur outside chaos windows; both runs had the container at its CPU limit for long stretches. Not isolated further here.
• A Python handler that ignores its deadline frees its worker slot while the coroutine keeps running, as documented. The 75-minute runs did not show the process running more handlers than its 30 configured slots (peak 30).

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn