Repository navigation
Conversation
reserve_enqueue_seq read the lane's sequence name only after calling ensure_lane_sequences, which on every reservation computed two sequence names, ran two to_regclass lookups and two no-op UPDATEs of the lane heads. It now reads the name first and provisions only a lane row that has none. A multi-row range was reserved with nextval followed by setval, two sequence WAL records; it is now one setval under the lane lock that every writer of the sequence already holds, and a one-row reservation is a bare nextval. The definition moves into awa.refresh_enqueue_functions, called by the installer and by the v052 tail for every discovered substrate. Signature, lock key and return value are unchanged, so N-1 binaries and insert_job_compat call it as before. v050 and v051 are reserved for parallel branches.
The reserve path probed EXISTS on queue_enqueue_heads for a cached lane before reserving, but advance_enqueue_head already reads the head row and repairs a stale cache entry when it is missing, so a cached lane now goes straight to the reservation. The cancel paths that call ensure_lane without reserving keep the probe. The enqueue cursor read fetched the ring authority row on every call. Once a store has read ledger authority it reads the ledger cursor directly: the flip to ledger is one-way. prepare_schema and reset clear the observation.
A ready-only enqueue made ten round trips: BEGIN, job ids, the lane
probe, the reservation, the ring authority and cursor reads, the ready
and segment inserts, NOTIFY and COMMIT. v052 adds
{schema}.enqueue_ready_rows, which for each lane in the caller's lock
order provisions a missing head with the same lane lock and inserts as
ensure_lane_inserts, reserves through reserve_enqueue_seq, then reads
ring_cursor('queue'), appends the ready rows before their segments
(NULL job ids take the job id sequence) and notifies.
enqueue_params_copy and enqueue_params_batch call it in autocommit when
every row is ready, none takes a uniqueness claim and the batch is below
the COPY threshold: one round trip and one transaction. Inside a caller
transaction (uniqueness claims, deferred rows, enqueue_runtime_rows) it
replaces the per-lane ensure and reserve, the cursor reads and both
inserts. Batches of 2,048 rows or more keep COPY. The store probes once
for an executable function and keeps the multi-statement path on a
schema without it.
The one-write reservation read the sequence through a dynamic statement built per call. pg_sequence_last_value and setval take the sequence as a regclass, so the read and the write are static; sequence_next_value still covers a sequence that has not been called yet. Server time per 20-row reservation on PostgreSQL 17: 107 us on main, 55 us with the dynamic read, 30 us now; WAL per reservation stays at one sequence record.
…equences Counts ensure_lane_sequences calls inside a reservation with transaction-level function statistics: none for a named lane, one for a lane row whose sequence name is NULL. Fails against the pre-v052 reservation, which provisioned on every call.
…d version The newer-schema guard test deleted the newest version row and expected CURRENT_VERSION - 1. Versions are not contiguous (v008 is skipped and v050/v051 are reserved), so it now expects the highest listed version below CURRENT_VERSION.
…ck' into brian/enqueue-fold
The enqueue cursor read skipped the authority row once the store had seen ledger, an observation that could come from a caller's transaction that flipped the authority and rolled back. The cursor is now one statement that branches on the authority row inline: the same single round trip, no cache, and only the table privileges the old path used. The enqueue_ready_rows probe was also permanent. A missing function is now probed again after a minute, so a runtime started before v052 picks it up without a restart, and a call that fails with undefined_function or insufficient_privilege forgets the observation so the next enqueue re-probes and falls back. The probe also requires EXECUTE on ring_cursor, which the function calls and which is revoked from PUBLIC.
…ck' into brian/enqueue-fold
…nqueue-fold # Conflicts: # CHANGELOG.md # awa-model/src/migrations.rs
…den the reserve helper v052 sits above v050 in the runner, which only applies versions above the highest recorded, so the branch now carries #527's v050 and cannot land first. The upgrade guide no longer implies binary-first is safe for health-gated workers: /readyz waits for the schema to reach the binary's version. It also describes the current probe (re-probed after a minute, EXECUTE on ring_cursor required, a lost function failing one call), and the changelog distinguishes enqueue_params_batch from the COPY entry point's 2,048-row threshold. reserve_enqueue_seq rejects a NULL count, which returned a start without advancing the sequence. The lane-cache rollback test also runs with the function disabled, since only the multi-statement path fills that cache. The least-privilege test covers a runtime denied ring_cursor EXECUTE and EXECUTE revoked after use.
Rewrites the v052 changelog entry from the user's side while resolving the changelog conflict.
# Conflicts: # CHANGELOG.md
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
A single snapshot cannot say whether a backlog is growing, and retained created_at cannot stand in for an arrival rate: on the queue-storage substrate compacted terminal rows fall back to finalized_at once their ready entry rotates. So `awa serve` now samples queue depth itself. Server (awa-ui): - QueueHistory keeps an in-memory window (default 60m at 15s) of per-queue available/running/lag, fed by the cached queue_overviews read and by a background sampler that holds only a weak reference and stops when the router is dropped. Served bucketed per minute at /api/stats/queue-history with nulls for gaps. - RouterConfig / router_with_config expose the setting; awa serve adds --queue-history-minutes (0 disables) and --queue-history-interval. Frontend: - Each queue is classified as not being worked, falling behind, long wait, paused, draining (with time to clear), steady or idle from its snapshot plus a least-squares fit over the last 5 minutes. - The overview leads with one verdict, the total waiting count and its last hour, then lists only queues needing attention. Failures are grouped by kind, queue and error; workers show only problems. - One dense QueueFlowTable serves Overview and Queues, switching to cards by container width rather than viewport. The Queues page gets back capacity/rate limit, DLQ policy and descriptor flags, and keeps search, sort and filter in the URL. - Built from the existing Card/Badge/Heading primitives; the page-specific CSS block is gone. Tests: Rust unit and DB-backed sampler tests, vitest coverage for the trend and verdict logic, synthetic browser scenarios rewritten around a stalled / falling-behind / draining / paused story, and the DB-backed Playwright specs updated. Removes the committed screenshots of the previous design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
…r-side The in-memory sampler in `awa serve` could not survive restarts, differed between UI replicas, and could only see depth. Replace it with history the maintenance leader writes, plus measured rates from the workers. Workers (awa-worker): - Per-queue cumulative counters (claimed, completed, failed, retried, rate-limited polls, claim wait since run_at) published in the existing runtime snapshot as an optional `counters` field. Older readers ignore it; older writers leave it absent. Schema v051 (additive, no existing table touched): - awa.queue_history: one row per queue per minute with depth by state, oldest wait, paused, live workers, capacity, and counter deltas (NULL when unknown rather than guessed). - awa.fleet_history: one row per minute with live/healthy/stale instances, leader and running versions. Maintenance leader: - Samples once a minute in a task beside the maintenance loop (so a slow depth read cannot delay rescue or promotion), differences counters per instance with restart and leader-change handling, and prunes past `queue_history_retention` (default 48h; Rust builder and Python `queue_history_retention_hours`; zero stops sampling). awa-ui: - assessment.rs classifies every queue (stalled, behind, slow, paused, draining, steady, idle), says what limits it (no workers, rate limit, worker capacity, spare capacity), and derives arrival and completion rates. One source of truth for the UI and GET /api/health/queues (`?fail_on=critical|warning` answers 503 for plain HTTP alerting). - /api/stats/queue-history serves per-minute series and fleet events (rollouts, leader changes, lost instances) from the new tables. - The in-memory sampler and its CLI flags are gone. Frontend: consumes the server report; shows measured rates, a "Limited by" column, fleet-change markers on the backlog chart and a recent fleet changes list; queue detail shows its own rates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
… v053 - Renumber the history migration to v053: open PRs #515 and #538 claim v051 and v052, and the runner applies only versions above the current one, so this must land after both (noted at the migration list). - Add a "Recovering" verdict: when nothing needs attention but a queue is draining a backlog that takes two minutes or more, say so and when it will be clear, instead of a bare "Keeping up". - Keep draining queues in the overview's focus list so recovery is visible, titled "Recovering (n)" when nothing else needs attention. - Trim copy that narrated the screen: the eyebrow over the verdict, the history coverage line (now shown only when trends are missing), card descriptions, and the Queues footnote (its definition moved to the Waiting column's tooltip). - Tests: a unit test walking one queue through normal, incident, recovery and clear; a database-backed test of the same phases through /api/health/queues with real job rows; synthetic browser scenarios for each phase that also capture review screenshots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
Renumbering to v053 left a gap the runner cannot see past: it applies only versions above the current one, so a database that applied v053 would silently skip v051/v052 when the open PRs claiming them (#515, #538) land, and "one version behind" stopped meaning v052. Keeping v051 turns the collision into a merge conflict in migrations.rs for whichever PR lands second, which forces a deliberate renumber. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
The README showed the pre-0.6 top-nav Jobs and queue detail pages. Use the overview during an incident (hero) and during recovery, in dark mode at 2x like the images they replace. A new opt-in spec regenerates them from the synthetic fleet: AWA_README_IMAGES_DIR=../../docs/images npm run test:operations -- readme-images Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
Every pending bundle must refuse to migrate a database whose registered queue-storage substrate another role could inject code into. v051 does not touch queue-storage schemas, so like v024, v025, v031 and v035 it carries only the discovery bootstrap (and its preflight) ahead of its own step, in built-in runs and exported bundles alike. schema_discovery_owner_test caught the gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
"Spare capacity" read as a cause next to a stalled queue; "No visible limit" says what the diagnosis actually found. The focus list already includes draining queues, so its title counts them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
…mmit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
…by leaders, report lock skips - Analyse only the known ring parents, not every partitioned table in the schema, so a substrate sharing a schema with application tables never analyses them. - A leader that resolved canonical at start but holds the queue-storage runtime as standby refreshes the queue-storage schema once routing flips to it, checked on every pass. - Take the parent's SHARE UPDATE EXCLUSIVE lock before ANALYZE (SKIP_LOCKED), so a conflicting lock on the parent fails the call with lock_not_available (recorded as skipped_lock) instead of skipping every partition and reporting analyzed. Correct the documentation of what SKIP_LOCKED covers. - Link the pull request from the migration header. - Tests: refresh on election before the first interval, both rejection guards of the helper from the runtime role, and a locked parent.
…t the batch function insert_many and insert_many_copy used awa.insert_jobs_compat_batch unconditionally, so a binary running ahead of migration v053 failed every batch insert. They now look for the function in the catalog and use the per-row path until it exists. The per-row statement asks the catalog as it inserts, the process keeps the answer, and a call that finds a cached function gone fails once and resets it. insert_many_skip_unique_conflicts needs the function and reports a schema without it. Tests run a binary against a schema without the function, through the migration appearing and the function disappearing again, and mix producers on the per-row and batch functions on one lane.
…scovery preflight The fail-closed test rewound the schema by the newest migration and expected the rerun to hit the substrate-ownership preflight. A migration that touches no custom substrate, such as the batch insert function, carries no preflight, so the test now picks the newest migration that does.
A lane's enqueue and claim sequences were created by whichever role first touched the lane, through SECURITY INVOKER helpers, so only that role and the roles its default privileges named could use them. Another runtime role then failed with "permission denied for sequence" on enqueue, claim or promotion. Each queue-storage schema now has two SECURITY DEFINER functions owned by the schema owner, provision_lane and provision_lane_sequences. The lane helpers create a missing sequence through them, and each new sequence gets the privileges of the head table it backs. ensure_lane_inserts calls provision_lane when it exists and is executable, inside a savepoint, and otherwise writes the lane rows as before. v053 refreshes the helpers and repairs existing lane sequences one at a time, reassigning and granting only what the migrating role may and reporting the rest. awa.lane_sequence_problems and the new `awa storage lane-sequences` command list what remains with the statements to run; awa.repair_lane_sequences and `--repair` rerun the repair.
…vision_lane Queue storage takes any queue name, including an empty one, and any smallint priority, so the provisioning functions only reject a null argument or a shard outside 0-63.
…may analyse - Revoke EXECUTE on awa.analyze_queue_storage_relation from PUBLIC. The SHARE UPDATE EXCLUSIVE lock it takes lasts until the caller's transaction ends, so any role could hold it open and block ring prune. The migration grants EXECUTE to the roles that already hold INSERT, UPDATE and DELETE on the awa head tables, and the function requires the calling session user to hold those privileges on the target schema. - Give the function its own lock_timeout. - Accept only the two head tables and the thirteen ring parents inside the function, not any partitioned table of the schema. - Probe for the function once per pass, so a binary ahead of the schema makes one catalog query instead of fifteen failing calls and server ERROR lines. Pass-level outcomes use a new "pass" metric scope. - Correct the header, which claimed the function takes no lock. - Tests: a role without EXECUTE and a role without head-table privileges are denied, the migration grants EXECUTE to existing grantees, non-ring tables are rejected, and a missing helper costs one probe and no calls.
…ded lock wait v053 reassigned lane sequences inside the migration transaction. ALTER SEQUENCE ... OWNER takes the sequence's ACCESS EXCLUSIVE lock until commit, with no lock_timeout, so an open enqueue on one lane could stall the whole migration and every repaired lane waited for the upgrade to commit. v053 now only refreshes the helpers. After it commits, awa migrate repairs each lane sequence with its own statement through awa.repair_lane_sequence(schema, sequence), which has lock_timeout = 300ms; on 55P03 the sequence is left unchanged and reported as busy. `awa storage lane-sequences --repair` uses the same path, and the set-returning awa.repair_lane_sequences(schema) is gone. Reassigning needs the current owner's privileges, which the schema owner lacks for sequences the migration role or an app role created. A migration role connected with role set to the owner therefore could not reassign its own sequences. The repair now retries such a sequence as the login role, lane_sequence_problems reports repair_roles (the login roles that can repair each sequence), and leftovers are reported with the next step to take. The docs state which role to run the repair as.
Consume a departing worker's final published counters once and preserve unknown coverage when a participant disappears. Attribute striped job outcomes to the originating dispatcher's counter handle while keeping physical job identities intact. Keep receiver-only callback rates unknown in the overview so its hourly fallback still applies. Validation: required Rust formatting/Clippy/workspace build;14 history unit tests; striped complete/retry/fail/park regression RED then GREEN on PostgreSQL17/18; existing history cases on both majors;55 frontend tests, frontend typecheck/build and operations skill validation.
hardbyte
added
the
full-ci
ChatGPT Codex Connector
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The admin overview uses per-minute queue pressure and fleet history to show whether waiting work is draining, falling behind, or has no worker. Queue drilldowns keep state counts, rolling-hour completions, worker capacity, pause/drain actions, and mobile navigation distinct from measured history.
The maintenance leader records queue and fleet samples once a minute in additive migration v058, with configurable retention (48 hours by default). This branch preserves the migration chain #637 (v053) → #631 (v054) → #635 (v055) → #538 (v056) → #602 (v057) before appending history; existing main v051 claim functions and v052 job output remain unchanged. Merge those prerequisites first.
History corrections:
Counter history measures published snapshots; a process can lose its unpublished tail. Leader changes, missing coverage, and callback outcome intervals remain explicit gaps. Current job-state time series are not presented as queue-depth history.
Validation:
full-ciis enabled for this updated head; remote matrix results are pending.Enqueue lane-order fixture
The PG18
restshard exposed an inherited #538 fixture assumption: the v056 folded enqueue correctly skipped a provisioning gate on already prepared lanes, so the test waited after its enqueue had committed. The canonical fix is included through #602. It gates the second reservation lock and requires each intended backend PID to wait on the exact expected key; the 20-second bound, result ordering and accounting assertions stay unchanged.Original-fixture RED and deliberately reversed-order negative controls were preserved on PostgreSQL 17 and 18. The corrected seven cases pass on both majors at this stacked head, with required fmt/Clippy/build. Queue history remains v058 and product behavior is unchanged. The original failed CI job remains recorded; updated-head full CI is pending.
This head also includes the reviewed transaction-wide prune timeout handling from #642 through #635. Lock contention after child locks now rolls back the whole maintenance transaction and returns
Blocked; rollup stops at the first blocked slot. No other #642 ancestry was imported. Both Rust workspaces passed required format, all-target/all-feature Clippy, and builds; all seven lane-order cases and both post-child-lock rollback controls passed on PostgreSQL 17 and 18. The existing v058 migration and lane fixture are preserved. Fresh exact-head full CI is pending.