[hardbyte]

Review prerequisite

Merge this branch only with #644, including the corrected shared oracle at 412419f864567c29386d9651cad93b82f6ddd997. This head still uses the earlier oracle. The corrected harness requires durable attempt-specific terminal proof, treats failed final lookups as inconclusive, accounts for materialized claims, and handles Python startup/drain restarts. It correlates migration sessions with probe phases, fences reads crossing process kills, keeps terminal observations tied to the greatest proven lease, and requires successful CLOSED proof before reporting a retained-session leak. Ambiguous or missing evidence remains inconclusive; still-live losses and proven conflicts remain failures.

Historical results below predate these corrections and need validation with the corrected harness before release acceptance. Those reports have not been recomputed or retrospectively cleared.

Refs #624. This stacks on that PR (and on the fan-out persona commit), so the two commits to review are the shard operator persona and the follow-up that adds lane-race and repeat-execution reporting. It will be rebased once #624 lands. It is a separate PR so the persona can be reviewed on its own.

What this adds

A shard operator persona for the persona soak. An operator changes awa.queue_meta.enqueue_shards on two hot queues while producers, promotion (scheduled jobs and retries) and claimers run:

• each queue is taken 1 -> 4 -> 8, down to 2 and back to 1, repeatedly;
• some raises run prepare_queue first (as the schema owner or as the migrator role), some do not;
• some runtimes and producer QueueStorage handles are restarted before others, so the cached shard counts disagree across the fleet for a while, and the rest are restarted later (QueueStorage::reset() truncates the store, so a restart is the only way to drop the cache under load);
• producers use insert_many, insert_many_copy and QueueStorage::enqueue_params_batch, with plain, scheduled, retrying and ordering_key jobs;
• for some raises that skip prepare_queue, the producers are held until the maintenance leader's runtime has created the new lanes, so a different role is the next to use them.

Roles

When the persona is selected the soak runs its actors as non-superuser roles, granted as in docs/security.md:

• awa_soak_owner owns the database and schema, runs migrations and prepare_queue, and changes enqueue_shards;
• awa_soak_migrator is a member of the owner role with default privileges on tables, sequences and functions;
• awa_soak_app_a and awa_soak_app_b are members of awa_soak_runtime, with no default privileges of their own. They run the persona's producers and runtimes; the leader is one of the persona's runtimes (the persona ends other personas' leadership and its runtimes retry election every 100 ms).

AWA_SOAK_SHARDOPS_SCHEMA_CREATE=0 withholds schema CREATE and makes the operator provision every lane before raising a count (the restricted-runtime procedure). With the default (1) lanes are created on the fly. In a mixed run, default privileges for the superuser keep the other personas' objects usable by the application roles.

Oracle additions

• every job enqueued before, during and after a change runs and completes exactly once; a repeat after a stale-heartbeat rescue is reported as a finding, any other repeat is a violation;
• every shard drains, including shards above a lowered count once every runtime has observed it;
• per-shard claim cursors keep advancing while their shard has backlog and workers are idle (per-shard lag from ADR-025, with unclaimed rows counted exactly because aborted transactions leave sequence gaps);
• ordering_key jobs start in order per key within a window in which every runtime had observed the shard count (a key legitimately moves shard across a change);
• no permission-denied, missing-lane or lane-creation-race error from any role. The scan reads the server log, so scripts/persona-soak.sh now sets log_line_prefix to include %u@%a; each violation names the role, lane, sequence owner and statement.

Results

Seed 42, mia,tom,priya,sam,alex,shardops, AWA_SOAK_SCALE=0.2, 25 minute load, on postgres:17 and postgres:18.

Code	pg17	pg18
v0.6.13	fails: permission_denied from both application roles, claim cursors frozen on the affected queue, shards above the count never drain	fails the same way
#638 (SECURITY DEFINER lane provisioning)	no permission, lane or shard violations over 163k jobs and 25 shard changes	no permission, lane or shard violations over 162k jobs

On v0.6.13 the first denial is within a second of the new lanes being created: the leader's role owns the new sequences, and the other role's producers fail with permission denied for sequence queue_enqueue_seq_*. Reads of lane depth and claims fail too, so the whole queue stops claiming, not only the new shard. A raise that runs prepare_queue first is clean. With AWA_SOAK_SHARDOPS_SCHEMA_CREATE=0 and the persona alone, 10 minute runs pass on both engines.

The same cause as a few statements, with a runtime group role granted as in docs/security.md plus schema CREATE:

-- as application role A: the first to use shard 1 of queue q
INSERT INTO awa.queue_lanes (queue, priority) VALUES ('q', 2) ON CONFLICT DO NOTHING;
INSERT INTO awa.queue_enqueue_heads (queue, priority, enqueue_shard) VALUES ('q', 2, 1);
INSERT INTO awa.queue_claim_heads (queue, priority, enqueue_shard) VALUES ('q', 2, 1);

-- as application role B, same group
BEGIN;
SELECT awa.reserve_enqueue_seq('q', 2::smallint, 1::smallint, 1);
-- ERROR:  permission denied for sequence queue_enqueue_seq_...
ROLLBACK;

Also seen

• One pg_class_relname_nsp_index unique violation from insert_many_copy on v0.6.13 (pg17): two sessions creating the same lane sequence at once. The scan now reports it as lane_creation_race. Not seen on fix(queue-storage): create queue lanes as the schema owner (0.6) #638.
• Graceful runtime restarts occasionally leave a job that already ran to be rescued on a stale heartbeat and run again (about 1 in 10 restarts at this load, all engines and runs). Reported as a finding, not a violation. Not investigated.
• Under the mixed load, two transient promotion stalls of 10-18 s per engine on fix(queue-storage): create queue lanes as the schema owner (0.6) #638, and none in the persona-only runs. Not investigated.

Test plan

• cargo clippy -p awa --test persona_soak_test -- -D warnings, cargo fmt --all --check
• persona alone with AWA_SOAK_SHARDOPS_SCHEMA_CREATE=0, 10 minutes, pg17 and pg18: pass
• mixed run on v0.6.13 and on fix(queue-storage): create queue lanes as the schema owner (0.6) #638, 25 minutes, pg17 and pg18, as above

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn