Repository navigation
Conversation
… analyses Autovacuum does not analyse partitioned tables, so the thirteen ring parents have no inherited statistics, and the two head tables (queue_claim_heads, queue_enqueue_heads) never reach the autoanalyze threshold, so the planner prices lane joins against them at its defaults. The maintenance leader now runs ANALYZE on them when it is elected and every five minutes (ClientBuilder::statistics_refresh_interval, Python statistics_refresh_interval_ms; zero disables). One relation per transaction, with a bounded lock_timeout and statement_timeout, partitioned parents with SKIP_LOCKED, in a task beside the leader loop so a slow ANALYZE never delays rotate, prune or promote ticks. Migration v053 adds awa.analyze_queue_storage_relation(), a SECURITY DEFINER function so a runtime role that does not own the tables can refresh statistics without a MAINTAIN grant. A missing function or an owner that may not analyse the tables is logged once and recorded in awa.maintenance.analyze.attempts without affecting job processing.
…maintenance Many concurrent producers batch-inserting into one queue lane serialise on the per-lane reservation lock, which a transaction holds until it commits. The new enqueue_lane_contention_test reports per-lane throughput and batch latency percentiles at high producer concurrency, with and without unique keys, across enqueue_shards, with a unique-conflict replay mix, and optionally drains every shard afterwards. Its nightly gate runs sixteen producers on one lane (and on four shards), checks lane integrity, and asserts contention-scaled floors; the rate and p99 regression thresholds live in benchmarks/baseline.json. enqueue_lane_maintenance_coupling_test measures a leader-shaped loop (promotion and heartbeat rescue on one task) with and without producers saturating a lane, and with a lane lock held for a fixed time: promotion batch latency, promotion lag on an unrelated queue, and how late rescue starts.
insert_many and insert_many_copy called awa.insert_job_compat once per row. The function takes the (queue, priority, shard) lane lock in reserve_enqueue_seq on the batch's first row, and the lock lasts to commit, so it was held while every remaining row was processed. A 185-row batch held it for about 40 ms on a local database, and producers on one lane queued behind whole batches: the lane's ceiling was roughly 4,000 jobs/s at any producer count, and a producer's call took (producers x 40 ms). Migration v053 adds awa.insert_jobs_compat_batch. It allocates job ids, claims unique keys in key order, registers new lanes and builds the rows before it reserves each lane's sequence range once, in (queue, priority, shard) order, then writes the ready rows and segments in two statements. It proposes head rows only for lanes that do not exist yet, because the head tables' row trigger takes the lane lock before the conflict check. insert_many and insert_many_copy call it. insert_many_copy now skips unique conflicts in one statement instead of one savepoint per row, and insert_many_skip_unique_conflicts exposes the same behaviour for callers that replay work: after a unique conflict they inserted the batch again one job at a time, taking the lane lock once per job. The lane-order test now parks a batch on one lane and checks which lane locks it holds, since job ids no longer follow lock order.
…by leaders, report lock skips - Analyse only the known ring parents, not every partitioned table in the schema, so a substrate sharing a schema with application tables never analyses them. - A leader that resolved canonical at start but holds the queue-storage runtime as standby refreshes the queue-storage schema once routing flips to it, checked on every pass. - Take the parent's SHARE UPDATE EXCLUSIVE lock before ANALYZE (SKIP_LOCKED), so a conflicting lock on the parent fails the call with lock_not_available (recorded as skipped_lock) instead of skipping every partition and reporting analyzed. Correct the documentation of what SKIP_LOCKED covers. - Link the pull request from the migration header. - Tests: refresh on election before the first interval, both rejection guards of the helper from the runtime role, and a locked parent.
…t the batch function insert_many and insert_many_copy used awa.insert_jobs_compat_batch unconditionally, so a binary running ahead of migration v053 failed every batch insert. They now look for the function in the catalog and use the per-row path until it exists. The per-row statement asks the catalog as it inserts, the process keeps the answer, and a call that finds a cached function gone fails once and resets it. insert_many_skip_unique_conflicts needs the function and reports a schema without it. Tests run a binary against a schema without the function, through the migration appearing and the function disappearing again, and mix producers on the per-row and batch functions on one lane.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 39 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
💡 Codex Review Align unique-key ordering with the N-1 insert path
During an overlapping rollout, the N-1 AGENTS.md reference: AGENTS.md:L134-L134 Lines 913 to 914 in c824b05 Keep COPY fallback recoverable after cached presence
When the process has observed v053 on one database and then uses AGENTS.md reference: AGENTS.md:L117-L117 Line 49 in c824b05 Update the jobs skill for the new skip API
This exports a new public conflict-skipping API, but AGENTS.md reference: AGENTS.md:L163-L164 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
…scovery preflight The fail-closed test rewound the schema by the newest migration and expected the rerun to hit the substrate-ownership preflight. A migration that touches no custom substrate, such as the batch insert function, carries no preflight, so the test now picks the newest migration that does.
A lane's enqueue and claim sequences were created by whichever role first touched the lane, through SECURITY INVOKER helpers, so only that role and the roles its default privileges named could use them. Another runtime role then failed with "permission denied for sequence" on enqueue, claim or promotion. Each queue-storage schema now has two SECURITY DEFINER functions owned by the schema owner, provision_lane and provision_lane_sequences. The lane helpers create a missing sequence through them, and each new sequence gets the privileges of the head table it backs. ensure_lane_inserts calls provision_lane when it exists and is executable, inside a savepoint, and otherwise writes the lane rows as before. v053 refreshes the helpers and repairs existing lane sequences one at a time, reassigning and granting only what the migrating role may and reporting the rest. awa.lane_sequence_problems and the new `awa storage lane-sequences` command list what remains with the statements to run; awa.repair_lane_sequences and `--repair` rerun the repair.
…vision_lane Queue storage takes any queue name, including an empty one, and any smallint priority, so the provisioning functions only reject a null argument or a shard outside 0-63.
…may analyse - Revoke EXECUTE on awa.analyze_queue_storage_relation from PUBLIC. The SHARE UPDATE EXCLUSIVE lock it takes lasts until the caller's transaction ends, so any role could hold it open and block ring prune. The migration grants EXECUTE to the roles that already hold INSERT, UPDATE and DELETE on the awa head tables, and the function requires the calling session user to hold those privileges on the target schema. - Give the function its own lock_timeout. - Accept only the two head tables and the thirteen ring parents inside the function, not any partitioned table of the schema. - Probe for the function once per pass, so a binary ahead of the schema makes one catalog query instead of fifteen failing calls and server ERROR lines. Pass-level outcomes use a new "pass" metric scope. - Correct the header, which claimed the function takes no lock. - Tests: a role without EXECUTE and a role without head-table privileges are denied, the migration grants EXECUTE to existing grantees, non-ring tables are rejected, and a missing helper costs one probe and no calls.
…ded lock wait v053 reassigned lane sequences inside the migration transaction. ALTER SEQUENCE ... OWNER takes the sequence's ACCESS EXCLUSIVE lock until commit, with no lock_timeout, so an open enqueue on one lane could stall the whole migration and every repaired lane waited for the upgrade to commit. v053 now only refreshes the helpers. After it commits, awa migrate repairs each lane sequence with its own statement through awa.repair_lane_sequence(schema, sequence), which has lock_timeout = 300ms; on 55P03 the sequence is left unchanged and reported as busy. `awa storage lane-sequences --repair` uses the same path, and the set-returning awa.repair_lane_sequences(schema) is gone. Reassigning needs the current owner's privileges, which the schema owner lacks for sequences the migration role or an app role created. A migration role connected with role set to the owner therefore could not reassign its own sequences. The repair now retries such a sequence as the login role, lane_sequence_problems reports repair_roles (the login roles that can repair each sequence), and leftovers are reported with the next step to take. The docs state which role to run the repair as.
hardbyte
changed the title
hardbyte
added
the
full-ci
This was referenced Nov 6, 2026
21 tasks
Owner
Author
|
Superseded by #690 on release/0.6.6; main receives it through the 0.6.15 forward-port. Changes merge into the 0.6 line first and reach main only by forward-port. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queue-storage compatibility inserts take a transaction-scoped lane lock on the first ready row and hold it until commit. Processing a large batch row by row therefore blocks other producers and promotion while the remaining bookkeeping runs.
Migration v055 adds
awa.insert_jobs_compat_batch. It prepares job IDs, unique claims and lane metadata before reserving sequence ranges in canonical lane order, then writes ready rows and segments in batches. Input order, transactional enqueue semantics and the lane-lock protocol shared with older binaries are preserved. Unique-key or application-lock cycles can still occur; the canonical lane order prevents opposing lane-lock acquisition.The optimized paths are:
insert_many_copy, including Python COPY: probes its own connection on every call and uses the batch function when installed; older schemas retain the per-row fallback. Unique conflicts are skipped on both paths.insert_many_skip_unique_conflicts: explicitly requires v055 and returns inserted rows in input order, with already-claimed keys absent. A repeated key within the batch keeps its first job.Generic
insert_manyretains its existingPgExecutorbound and one-statement per-row path on every supported schema. It cannot probe separately without consuming that executor. There is no process-wide schema cache: using a migrated database cannot select unavailable SQL for a subsequent call against an older database and abort its caller's transaction. A normal unique conflict still fails the whole generic batch with SQLSTATE23505.Prerequisites: #637 (lane ownership, v053) → #631 (planner statistics, v054) → this PR (batch insert, v055). Their commits are merged into this branch, with the three migrations registered separately. External runners apply the generated v054-to-v055 bundle. This remains main-only; no 0.6 backport is included.
Validation on the final stacked head:
42883. It alternates current/older/current databases through caller-owned transactions and verifies application writes, input order, duplicate skipping, commit and rollback.cargo fmt --all, offline clippy with all targets/features and warnings denied, and offline workspace build pass. Both portable skills validate; public API docs, upgrade notes, troubleshooting and CHANGELOG describe the actual paths.Promotion isolation, lane-lock wait metrics and runner-baseline calibration remain separate work.
Independent CI follow-up at
934d0f65: the statistics privilege regression now explicitly replays migration v054. Using the latest migration stopped exercising its grant SQL once v055 was added; both PostgreSQL CI legs exposed the stale test assumption. The statistics grant implementation is unchanged. All nine statistics tests pass on PostgreSQL 17 and 18, and the required fmt/Clippy/build checks pass. Fresh full CI: https://github.com/hardbyte/awa/actions/runs/42210128661.Prune lock-timeout handling
The prune CI suite exposed a lock timeout after the explicit child-table locks. Its existing fixture already retries
Blocked, but the transaction returned a raw55P03error. This branch now includes the reviewed #642 handling: a contention error anywhere in a queue/lease/claim prune or terminal-count rollup rolls back the whole transaction and reportsBlocked; a rollup pass stops at its first blocked slot.Both Rust workspaces pass fmt, all-target/all-feature Clippy and workspace builds. The original 27 prune cases plus the added post-child-lock regression all pass on the PostgreSQL 18.3 CI image (28/28). Both deterministic prune/rollup rollback controls pass on PostgreSQL 17 and 18. The v055 enqueue function and executor/cache behavior are preserved. Updated-head full CI is pending; the original prune CI failure remains recorded.