Skip to content

Supported API to change a queue's enqueue_shards, picked up by running producers #670

Open

Description

@hardbyte

Changing a queue's enqueue_shards today means hand-writing SQL against awa.queue_meta, and the change does not reach running producers until they restart. Operators fill the gap with ad-hoc SQL against lane internals (head tables, ensure_lane_sequences, lane advisory-lock keys), which is brittle across releases.

Gaps

  1. No supported operation sets the shard count. The documented path is an INSERT … ON CONFLICT upsert into awa.queue_meta (configuration). QueueStorage::prepare_queue / awa storage prepare-queue provisions lanes but does not change queue_meta, so provisioning and switching are two steps an operator has to order correctly. The upsert also needs the physical queue names when queue striping is in use, which prepare_queue already handles for its own step.
  2. Running producers keep the shard count they first read. QueueStorage::enqueue_shards_cache is filled lazily by shard_for_enqueue and cleared only by reset(). After a change, Rust producers keep enqueueing with the old count until the process restarts, while enqueues through the SQL insert functions read queue_meta per statement and switch at commit. Raising and lowering both behave this way. The docs say lowering "is safe at any time" but don't mention the restart.
  3. On 0.6.x before the definer_lane_provisioning patch, raising the count without provisioning first is unsafe with more than one runtime role. The first producer or maintenance leader to touch a new shard creates its lane sequences as its own role, without grants for the other runtime roles. v053 on 0.7 and the 0.6.14 patch fix this; the API below should not rely on callers knowing it.

Proposal

  • A Rust API, e.g. QueueStorage::set_enqueue_shards(pool, queue, shards), taking the logical queue name. In one transaction it validates 1..=64, provisions every priority × shard lane for each physical queue when raising, and upserts queue_meta for each physical queue. No producer can then see the new count before its lanes exist. Lowering needs no provisioning.
  • A CLI command for the same operation, e.g. awa storage set-enqueue-shards --queue <q> --shards <n> (or --set on prepare-queue), and a Python binding if prepare_queue is exposed there.
  • Running producers pick up the new count within a bounded interval without a restart. That means a refresh with a TTL, or reusing an existing periodic refresh, as long as it adds no per-enqueue query. If that isn't practical, document the restart requirement for both raising and lowering.

Done when

  • Tests on PostgreSQL 17 and 18 cover:
    • an atomic raise, where no enqueue lands on an unprovisioned shard;
    • idempotent re-runs;
    • lowering;
    • a running producer switching shard count without a restart (or the documented limitation);
    • striped queues.
  • docs/configuration.md, the CHANGELOG and the awa-operations skill describe the API in place of the raw upsert.
  • Backported to the 0.6 line.

Activity

  1. hardbyte commented on Nov 11, 2026

    @hardbyte
    OwnerAuthor

    Additions after looking at operator control of sharding across the CLI and admin UI. #719 (automatic shard scaling) depends on this primitive.

    Admin UI, not just CLI.

    • Add a write route next to pause, resume and drain, for example PUT /api/queues/{queue}/enqueue-shards. It should call the same set_enqueue_shards and be guarded by require_writable.
    • The queue page should show the current S, per-shard ready depth and lag, and the ordering consequence ("raising this makes the queue partitioned FIFO") before confirming.
    • Nothing in admin.rs exposes S or per-shard depth today. The awa.enqueue.shard attribute on awa.job.claimed is the only per-shard signal.

    Reporting what producers actually use.

    • Raising or lowering S is correctness-safe while producers disagree. The shardops persona soak (test(persona-soak): shard operator persona (live enqueue_shards changes under restricted roles) #639) changes 1→4→8→2→1 with fleet caches out of sync. But the operator can't tell when the change has taken effect.
    • Each runtime could publish the shard count it is using per queue in its runtime snapshot, so show and the UI can say "N of M producers on S = 4".
    • SQL-function producers read queue_meta per statement, so only Rust QueueStorage producers and promotion need reporting.

    Producer refresh. Reuse the ADR-038 override refresh (~10 s, one single-row read per claimer per cadence) to expire enqueue_shards_cache, instead of adding a TTL path of its own. On 0.6, which lacks ADR-038, a TTL is still needed.

    Permissions.

    Docs fix. docs/configuration.md says "Lowering the value is safe at any time." It's safe for correctness. It doesn't take effect in running Rust producers until they restart, and strict FIFO returns only once rows in shards ≥ S have drained. ADR-025 states the restart caveat; the configuration page doesn't.

    Prior art.

    • Temporal changes task-queue partition counts through dynamic config. Its documented order for scaling down is to lower write partitions first, drain, then lower read partitions. That maps to Awa as "lower S, wait for shards ≥ S to drain, then report done".
    • Kafka can't shrink partition counts at all. Awa's lowering-is-safe property, enforced by test_queue_storage_lowering_enqueue_shards_drains_existing_rows, is worth stating as a strength.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions