Automatic enqueue-shard scaling for queues that accept partitioned FIFO #719

Problem

A queue whose producers contend on its enqueue head needs more enqueue shards (ADR-025). Today an operator has to notice the contention, pick a shard count, and change it by hand with SQL against awa.queue_meta (#670). Nothing raises the count when contention appears or lowers it when it goes away.

The decision is harder than it looks:

• It changes the ordering contract. S > 1 means partitioned FIFO instead of strict FIFO per (queue, priority), and any change remaps hash(ordering_key) % S, so per-key order is broken while old rows drain. Automatic changes are only acceptable on queues that have opted into partitioned FIFO.
• No signal measures the thing shards fix. ADR-025 sized shards from producer lock waits on the enqueue lane (pg_stat_activity Lock waits). The runtime records awa.enqueue.duration and awa.enqueue.batch_size, but no lane lock-wait time; lane contention is measured only in tests (awa/tests/enqueue_lane_contention_test.rs). Depth and lag (awa.queue.depth, awa.queue.lag) show a queue falling behind, but shards don't help a queue whose bottleneck is workers.
• Running Rust producers wouldn't see the change. QueueStorage::enqueue_shards_cache is filled once and cleared only by reset() (Supported API to change a queue's enqueue_shards, picked up by running producers #670 gap 2).

Prior art

• Temporal has the closest design: matching.numTaskqueueReadPartitions / WritePartitions in dynamic config, plus an experimental partitionScaleManager that runs in shadow mode by default (it logs and emits metrics without acting). It scales on add rate and backlog, with lower thresholds for scaling down, caps the rate of change, and keeps write partitions at or below read partitions so shrinking drains before reads stop.
• Oban Pro DynamicScaler scales nodes, not partitions, with a 120 s cooldown to stop thrashing.
• Kafka can add partitions but never remove them, and doesn't redistribute keys. Pulsar Key_Shared rebalances consumers across hash ranges, not partitions.
• No Postgres-backed job queue we looked at (Oban, River, GoodJob, Solid Queue, pg-boss, Graphile Worker, PGMQ, Procrastinate) changes partition or shard counts automatically.

Awa already has one automatic controller of this kind: queue_claimer_state.target_claimers chooses 1, 2 or 4 claimers per queue from the available backlog with hysteresis (desired_queue_claimer_target, queue_storage.rs), coordinated across the fleet through a conditional upsert. #380 proposes extending it. Shard scaling should reuse its shape and differ in one place: claimers never change ordering, and shards do.

Proposal

Prerequisites.

• Supported API to change a queue's enqueue_shards, picked up by running producers #670's atomic primitive: provision the lanes, then upsert queue_meta, in one transaction.
• A bounded producer refresh, so a change reaches running producers.

Signal.

• Add a producer metric for time spent waiting on the lane reservation lock, per queue: awa.enqueue.lane_lock_wait. It is useful on its own for manual tuning.
• The controller can't read OTel metrics, and producers that enqueue through the SQL insert functions don't publish runtime snapshots. So it needs a database-side signal too: the leader samples pg_locks for ungranted waits on lane advisory keys, which also catches non-Awa producers.
• A queue is a candidate to grow when lane lock waits are a meaningful share of enqueue time over a window. A queue is a candidate to shrink when they have been negligible for a long window.

Opt-in. Per queue, alongside the static count. For example queue_meta.enqueue_shards_max (NULL means manual), set through the #670 command (awa storage set-enqueue-shards --queue q --auto --max 8) and the UI. Opting in records that the queue accepts partitioned FIFO.

Controller. A new maintenance-leader branch next to queue_stats, registered in the branch inventory:

• moves at most one step at a time (×2 or ÷2) within [1, max];
• keeps a cooldown after each change, so remapped keys settle;
• grows only after the lanes for the new count are provisioned;
• shrinks only when the lanes ≥ the new count have no ready rows. Claiming has no shard < S predicate, so shrinking is already safe; waiting for the drain is what restores strict per-shard FIFO.

Shadow mode first. As with Temporal: by default the controller records what it would have done (metric and decision log) without changing queue_meta. Enforcement is a second opt-in once shadow data looks sane.

Operator surfaces.

• CLI: awa storage set-enqueue-shards --auto --max <n> | --manual <n>, and a show that prints the current count, mode, last decision and its reason.
• Admin UI: a sharding panel on the queue page with the current count, mode and max, per-shard ready depth and lag, the lane lock-wait signal, and recent decisions. Writes go through the same guarded routes as Supported API to change a queue's enqueue_shards, picked up by running producers #670.
• Applied state: producers report the shard count they currently use in runtime snapshots, so the UI can show "N of M producers on S = 4". The 0.6 runtime-controls observation protocol is the model.

Experiment gate. As in #380: run steady, bursty and skewed producer-contention shapes against static S = 1/2/4/8. Ship enforcement only if the controller doesn't oscillate and comes within a stated margin of the best static setting. Otherwise ship the lock-wait metric and documented guidance, and close this.

Done when

• The lane lock-wait metric exists, is documented and appears on the Grafana dashboard.
• Shadow-mode decisions are visible in the CLI and UI.
• With enforcement enabled, tests on PostgreSQL 17 and 18 cover:
  • growing under contention;
  • shrinking after it subsides;
  • no enqueue landing on an unprovisioned lane;
  • rows in shards above a lowered count draining;
  • running producers picking up the change.
• docs/configuration.md and ADR-025 (or a new ADR) describe the auto mode and its ordering consequences.

Depends on #670. Related: ADR-025, ADR-031, #380, #677, #543.