Admin UI: is every queue keeping up? (queue history, health API, limited-by)

CHANGELOG.md

@@ -6,7 +6,7 @@ Notable changes between releases. Detailed migration notes for storage transitio
 
 ### Upgrading
 
-This release adds migrations v044 to v050. Run `awa migrate` as the migrator
+This release adds migrations v044 to v051. Run `awa migrate` as the migrator
 role; the [0.6→0.7 upgrade guide](docs/upgrade-0.6-to-0.7.md) has the details
 for each step.
 
@@ -73,6 +73,22 @@ for each step.
   `awa.migration_lock_key()` expose the runner's advisory lock key.
 - Python prints a diagnostic to stderr if native shutdown has stalled for five
   seconds.
+- **The admin overview answers "is it keeping up?" (v051,
+  [#543](https://github.com/hardbyte/awa/pull/543)).** The maintenance leader
+  writes one row per queue per minute to `awa.queue_history` (depth by state,
+  oldest wait, and claimed/completed/failed/retried/rate-limited counts) and
+  one fleet row to `awa.fleet_history` (live and healthy instances, leader,
+  running versions), keeping 48 hours by default. Runtimes publish the
+  per-queue outcome counters this needs in their existing snapshots. The
+  overview, Queues page and queue detail classify each queue as not being
+  worked, falling behind, waiting long, paused, draining, steady or idle,
+  say what is holding waiting work back (no workers, rate limit, worker
+  capacity), show measured completion and arrival rates and an hour of
+  history, and mark rollouts, leader changes and lost instances on the chart.
+  `GET /api/health/queues` returns the same verdicts for alerting
+  (`?fail_on=critical` or `warning` answers 503 at that level).
+  Configure retention with `ClientBuilder::queue_history_retention` or
+  Python's `queue_history_retention_hours`; zero stops sampling.
 
 ### Changed
 

README.md

@@ -4,7 +4,7 @@
 
 Awa (Māori: river) fills the gap between Postgres event queues that are too narrow for real job-queue behavior and language-specific job frameworks (River, Oban, Sidekiq) that couple you to one ecosystem. If you run Rust or Python (or both) on Postgres and want priorities, cron, DLQ, and transactional enqueue without Redis or RabbitMQ, Awa is built for you.
 
-![AWA Web UI — Jobs (dark mode)](https://raw.githubusercontent.com/hardbyte/awa/main/docs/images/awa-ui-dark.png)
+![Awa admin overview during an incident: one queue is not being worked, another is falling behind its rate limit, with recurring failures and fleet changes alongside](https://raw.githubusercontent.com/hardbyte/awa/main/docs/images/awa-ui-overview-incident-dark.png)
 
 ## Core Concepts
 
@@ -49,7 +49,7 @@ Awa (Māori: river) fills the gap between Postgres event queues that are too nar
 - **Operator descriptors** — code-declared queue and job-kind names/descriptions with stale/drift visibility in the UI.
 - **Postgres-only** — one dependency you already have; no Redis, no RabbitMQ, no separate scheduler.
 
-![AWA Web UI — Queue detail (dark mode)](https://raw.githubusercontent.com/hardbyte/awa/main/docs/images/awa-ui-queue-detail-dark.png)
+![Awa admin overview during recovery: three queues draining their backlog with an estimate of when all will be clear, after a hotfix rollout shown in the fleet changes](https://raw.githubusercontent.com/hardbyte/awa/main/docs/images/awa-ui-overview-recovery-dark.png)
 
 ## Correctness
 

awa-model/migrations/v051_queue_history.sql

@@ -0,0 +1,66 @@
+-- v051: per-minute queue and fleet history for the admin overview
+--
+-- Job tables record current state, not how a backlog moved, so the admin UI
+-- could not tell a queue that is draining from one falling behind. The
+-- maintenance leader now writes one row per queue per minute (depth by state,
+-- oldest available wait, and claim/completion/failure deltas differenced from
+-- the per-queue counters runtimes publish in their snapshots) plus one fleet
+-- row per minute (instance health, leader, running versions). The UI reads
+-- these instead of sampling in each `awa serve` process.
+--
+-- Both tables are additive and written only by the elected maintenance
+-- leader, never by triggers or the claim/finalize path, so ADR-045 does not
+-- apply and enqueue/claim cost is unchanged. Rows older than the configured
+-- retention (default 48 hours) are deleted by the same leader.
+--
+-- N-1 binaries: never read or write these tables, and a 0.7.x leader simply
+-- does not populate them; the UI then reports no history. Counter columns are
+-- NULL for a minute when any live runtime serving the queue predates the
+-- counters, so partial sums are never presented as totals.
+--
+-- Locks and time: two CREATE TABLE IF NOT EXISTS on new relations; no
+-- existing table is touched. Effectively instant.
+-- Runtime roles need SELECT, INSERT and DELETE on both tables; the default
+-- privileges in docs/security/database-roles.md grant that.
+-- Tracking: https://github.com/hardbyte/awa/pull/543
+
+CREATE TABLE IF NOT EXISTS awa.queue_history (
+    queue              TEXT        NOT NULL,
+    bucket             TIMESTAMPTZ NOT NULL,
+    available          BIGINT      NOT NULL,
+    running            BIGINT      NOT NULL,
+    scheduled          BIGINT      NOT NULL,
+    retryable          BIGINT      NOT NULL,
+    failed             BIGINT      NOT NULL,
+    waiting_external   BIGINT      NOT NULL,
+    lag_seconds        DOUBLE PRECISION,
+    paused             BOOLEAN     NOT NULL DEFAULT FALSE,
+    -- Deltas since the previous sample. NULL when unknown: the first sample
+    -- after a leader change, or a live runtime without counters.
+    claimed            BIGINT,
+    completed          BIGINT,
+    failed_attempts    BIGINT,
+    retried            BIGINT,
+    rate_limited       BIGINT,
+    claim_wait_ms      BIGINT,
+    -- Live runtimes serving the queue and their summed in-flight count.
+    live_workers       INT         NOT NULL DEFAULT 0,
+    capacity           BIGINT,
+    PRIMARY KEY (queue, bucket)
+);
+
+CREATE INDEX IF NOT EXISTS idx_awa_queue_history_bucket
+    ON awa.queue_history (bucket);
+
+CREATE TABLE IF NOT EXISTS awa.fleet_history (
+    bucket             TIMESTAMPTZ PRIMARY KEY,
+    live_instances     INT         NOT NULL,
+    healthy_instances  INT         NOT NULL,
+    stale_instances    INT         NOT NULL,
+    leader_instance_id UUID,
+    versions           TEXT[]      NOT NULL DEFAULT ARRAY[]::TEXT[]
+);
+
+INSERT INTO awa.schema_version (version, description)
+VALUES (51, 'Per-minute queue and fleet history')
+ON CONFLICT (version) DO NOTHING;

awa-model/src/admin.rs

@@ -1691,6 +1691,36 @@ pub struct QueueRuntimeSnapshot {
     pub in_flight: u32,
     pub overflow_held: Option<u32>,
     pub config: QueueRuntimeConfigSnapshot,
+    /// Cumulative outcome counters since this instance started (#543).
+    /// `None` from runtimes that predate them, so readers can tell "not
+    /// reported" from zero.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub counters: Option<QueueOutcomeCounters>,
+}
+
+/// Monotonic per-queue counters a runtime publishes with each snapshot.
+///
+/// They reset when the instance restarts; consumers difference consecutive
+/// readings of the same `instance_id` and treat a decrease as a reset.
+#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
+pub struct QueueOutcomeCounters {
+    /// Jobs this instance claimed from the queue.
+    pub claimed: u64,
+    pub completed: u64,
+    /// Attempts that ended the job as failed (terminal).
+    pub failed: u64,
+    /// Attempts that failed and were scheduled for another try.
+    pub retried: u64,
+    /// Dispatch polls skipped because the queue's rate limit had no tokens.
+    pub rate_limited: u64,
+    /// Sum over claimed jobs of the time between becoming runnable and
+    /// being claimed, in milliseconds. Divide by `claimed` for a mean.
+    pub claim_wait_ms: u64,
+    /// Attempts that parked waiting for an external callback. Whatever
+    /// resolves the callback may run in another process, so the eventual
+    /// outcome is not counted here.
+    #[serde(default)]
+    pub parked: u64,
 }
 
 #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)]

awa-model/src/lib.rs

@@ -13,6 +13,7 @@ pub mod job;
 pub mod kind;
 pub mod migrations;
 pub mod partitioned_queue;
+pub mod queue_history;
 pub mod queue_storage;
 pub mod reschedule;
 pub mod storage;

awa-model/src/migrations.rs

@@ -5,7 +5,7 @@ use sqlx::{Connection, PgPool};
 use tracing::{info, warn};
 
 /// Current schema version.
-pub const CURRENT_VERSION: i32 = 50;
+pub const CURRENT_VERSION: i32 = 51;
 
 /// Migrations that require an exclusive (no-live-runtime) upgrade window.
 ///
@@ -290,6 +290,11 @@ const MIGRATIONS: &[(i32, &str, &[&str])] = &[
         "Drop the redundant lane index on ready_entries partitions",
         &[V23_UP, V50_UP],
     ),
+    (
+        51,
+        "Per-minute queue and fleet history",
+        &[SCHEMA_DISCOVERY_UP, V51_UP],
+    ),
 ];
 
 const V1_UP: &str = include_str!("../migrations/v001_canonical_schema.sql");
@@ -348,6 +353,7 @@ const V47_UP: &str = include_str!("../migrations/v047_replica_identity.sql");
 const V48_UP: &str = include_str!("../migrations/v048_claim_cursor_catch_up.sql");
 const V49_UP: &str = include_str!("../migrations/v049_prepared_lane_sequences.sql");
 const V50_UP: &str = include_str!("../migrations/v050_drop_ready_lane_shard_index.sql");
+const V51_UP: &str = include_str!("../migrations/v051_queue_history.sql");
 
 /// Old version numbers from pre-0.4 releases that used V3/V4/V5 numbering.
 /// Also tolerates the unreleased inline-V6 branch numbering used during review.