[hardbyte]

The overview now answers one question first: is every queue keeping up? Stacked on #543.

Normal operations

Incident

Queue detail and mobile during the incident

Recovery

The screenshots come from the synthetic Playwright fixture (e2e/operations-fixture.ts), which drives all three phases. queue_health_scenario_test.rs runs the same story against a real database.

What changed

Workers. Each worker runtime reports per-queue outcome counters in its existing snapshot: claimed, completed, failed, retried, rate-limited, claim wait and parked-for-callback. The field is optional and additive, so N−1 binaries ignore it.

History (migration v051). The maintenance leader writes one row per queue per minute to awa.queue_history, and one fleet row to awa.fleet_history.

• Counter deltas are NULL (unknown), never zero, when a restart, a new runtime, an N−1 runtime without counters, or a sample more than 15s off the minute makes them unknowable.
• Completed, failed and retried are also unknown for a minute in which a callback could have resolved one of the queue's jobs, since callbacks resolve in whichever process receives them.
• The default retention is 48h. Set it with ClientBuilder::queue_history_retention or Python's queue_history_retention_hours; 0 disables sampling.

Assessment on the server. awa-ui/src/assessment.rs turns that history into a verdict for each queue, plus a "limited by" diagnosis: paused, no workers, rate limit, worker capacity, or no visible limit.

• Verdicts: stalled, behind, slow, paused, draining, steady, idle, or unknown.
• The UI and the new endpoints both read this one assessment:
  • GET /api/health/queues?fail_on=critical|warning returns 503 at the threshold, so it works as a probe or alert target.
  • GET /api/stats/queue-history?minutes=N returns the series, plus fleet events (version rollouts, leader changes, lost instances).

UI.

• The overview leads with the verdict, the total backlog and its trend, and the measured finishing and arrival rates. Fleet changes show as markers on the sparkline.
• The queue list shows only what needs attention or is recovering. The queues and queue-detail pages reuse the same assessment.
• The queues table fits at 1440px. Its controls are icon buttons pinned to the right edge, Capacity appears where there is room, and DLQ-off and config-mismatch flags sit beside the queue name.
• Gaps in history render as gaps, not zeros.
• The README images are replaced with the incident and recovery views.

Related open PRs

• perf(queue-storage): per-slot claim functions (v051) and heartbeat by claim identity #515, perf(queue-storage): enqueue a ready batch in one round trip (v056) #538 also claim migrations v051/v052. Whichever lands second renumbers. A version collision fails loudly. A gap would fail silently through the newer-schema guard, so this PR stays at v051 rather than skipping ahead.
• fix(worker): fixed-grid heartbeat ticks, reject a zero heartbeat interval #526 will conflict trivially with this PR in the JobExecutor::new test constructors (one extra argument).
• docs(upgrade): revoke a running 0.6 fleet's schema CREATE without a pause #541 has no overlap: no change to default privileges, and the v051 tables inherit existing grants.
• Redesign admin overview around queue pressure and operator actions #543 is the base. Add an interactive job recovery model to the docs #542 and the dependabot PRs are unrelated. Fix weekly receipt chaos contention and benchmark isolation #544 is already merged to main.

Testing

CI on this stacked PR runs only lint, build, the browser suite and the wheel smoke test: the PR-check selector skips the Rust and Python test shards for a non-main base. Everything below was run locally on Postgres 16.

• cargo test --workspace:
  • All new tests pass:
    • worker counters in snapshots;
    • the leader writing history, with a prune round trip;
    • the health and history endpoints;
    • the normal → incident → recovery → cleared DB scenario;
    • assessment and history unit tests, including off-minute intervals and callback-resolved outcomes.
  • migration_test (69/69), schema_discovery_owner_test, awa-model and awa-cli pass.
  • Unrelated to this change: chaos_suite_test::test_finalize_retries_after_terminated_finalization_backends fails on main too when it runs after the adapter/benchmark/cel tests on a shared DB (it leaves the storage canonical), and passes on a fresh DB.
• Frontend:
  • vitest: 54 tests;
  • synthetic Playwright suite: 10 tests, including the three-phase walkthrough that produced these screenshots;
  • DB-backed Playwright suite: 81 tests, on a fresh database against awa serve.
• Python: the bindings build; test_start_config.py and test_descriptors.py pass (31 tests, including the new queue_history_retention_hours validation test).

🤖 Generated with Claude Code

https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7c8a42c4-7716-42b3-bf13-be0df35f9863

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-25T23:20:25.742907Z	2eb1fa7	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2eb1fa7ab4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".