Repository navigation
Open
Description
Changing a queue's enqueue_shards today means hand-writing SQL against awa.queue_meta, and the change does not reach running producers until they restart. Operators fill the gap with ad-hoc SQL against lane internals (head tables, ensure_lane_sequences, lane advisory-lock keys), which is brittle across releases.
Gaps
- No supported operation sets the shard count. The documented path is an
INSERT … ON CONFLICTupsert intoawa.queue_meta(configuration).QueueStorage::prepare_queue/awa storage prepare-queueprovisions lanes but does not changequeue_meta, so provisioning and switching are two steps an operator has to order correctly. The upsert also needs the physical queue names when queue striping is in use, whichprepare_queuealready handles for its own step. - Running producers keep the shard count they first read.
QueueStorage::enqueue_shards_cacheis filled lazily byshard_for_enqueueand cleared only byreset(). After a change, Rust producers keep enqueueing with the old count until the process restarts, while enqueues through the SQL insert functions readqueue_metaper statement and switch at commit. Raising and lowering both behave this way. The docs say lowering "is safe at any time" but don't mention the restart. - On 0.6.x before the
definer_lane_provisioningpatch, raising the count without provisioning first is unsafe with more than one runtime role. The first producer or maintenance leader to touch a new shard creates its lane sequences as its own role, without grants for the other runtime roles. v053 on 0.7 and the 0.6.14 patch fix this; the API below should not rely on callers knowing it.
Proposal
- A Rust API, e.g.
QueueStorage::set_enqueue_shards(pool, queue, shards), taking the logical queue name. In one transaction it validates1..=64, provisions every priority × shard lane for each physical queue when raising, and upsertsqueue_metafor each physical queue. No producer can then see the new count before its lanes exist. Lowering needs no provisioning. - A CLI command for the same operation, e.g.
awa storage set-enqueue-shards --queue <q> --shards <n>(or--setonprepare-queue), and a Python binding ifprepare_queueis exposed there. - Running producers pick up the new count within a bounded interval without a restart. That means a refresh with a TTL, or reusing an existing periodic refresh, as long as it adds no per-enqueue query. If that isn't practical, document the restart requirement for both raising and lowering.
Done when
- Tests on PostgreSQL 17 and 18 cover:
- an atomic raise, where no enqueue lands on an unprovisioned shard;
- idempotent re-runs;
- lowering;
- a running producer switching shard count without a restart (or the documented limitation);
- striped queues.
docs/configuration.md, the CHANGELOG and theawa-operationsskill describe the API in place of the raw upsert.- Backported to the 0.6 line.
Metadata
Metadata
Assignees
Labels
No labels
Activity
hardbyte commentedon Nov 11, 2026
Additions after looking at operator control of sharding across the CLI and admin UI. #719 (automatic shard scaling) depends on this primitive.
Admin UI, not just CLI.
PUT /api/queues/{queue}/enqueue-shards. It should call the sameset_enqueue_shardsand be guarded byrequire_writable.admin.rsexposes S or per-shard depth today. Theawa.enqueue.shardattribute onawa.job.claimedis the only per-shard signal.Reporting what producers actually use.
shardopspersona soak (test(persona-soak): shard operator persona (live enqueue_shards changes under restricted roles) #639) changes 1→4→8→2→1 with fleet caches out of sync. But the operator can't tell when the change has taken effect.showand the UI can say "N of M producers on S = 4".queue_metaper statement, so only RustQueueStorageproducers and promotion need reporting.Producer refresh. Reuse the ADR-038 override refresh (~10 s, one single-row read per claimer per cadence) to expire
enqueue_shards_cache, instead of adding a TTL path of its own. On 0.6, which lacks ADR-038, a TTL is still needed.Permissions.
queue_metais writable by any runtime credential. The documented runtime role has DML on everyawatable.halt_runtime_scope, would let the CLI and UI change S without handing out runtime credentials.Docs fix.
docs/configuration.mdsays "Lowering the value is safe at any time." It's safe for correctness. It doesn't take effect in running Rust producers until they restart, and strict FIFO returns only once rows in shards ≥ S have drained. ADR-025 states the restart caveat; the configuration page doesn't.Prior art.
test_queue_storage_lowering_enqueue_shards_drains_existing_rows, is worth stating as a strength.Generated by Claude Code