Queue storage: QueueStorage batch enqueue and per-row inserts can deadlock when creating a new lane #692

On queue storage, a QueueStorage batch enqueue and a per-row insert (insert, insert_many, insert_many_copy, or a 0.6.14 producer) can deadlock when both create the same brand-new (queue, priority) lane and also touch a second lane.

Cause. The two paths take the lane's provisioning lock and its queue_lanes row in opposite orders:

• QueueStorage::ensure_lane_inserts takes the provisioning advisory lock awa:<queue>:<priority>:<shard> first, then inserts the queue_lanes row.
• insert_job_compat (and the batch function from perf(insert): shorten lane lock holds for COPY and skip-conflict batches (0.6) #690) insert the queue_lanes row first, and take the same advisory lock afterwards, in the head-table trigger that provisions the lane sequences.

So one transaction can hold the advisory lock and wait on the other's uncommitted queue_lanes row, while the other holds that row and waits on the advisory lock. PostgreSQL aborts one side with 40P01.

Reproduction. Two concurrent transactions, each enqueueing into lanes A < B, where A is a (queue, priority) no producer has used before:

• QueueStorage::enqueue_params_batch against insert_many or insert_many_copy deadlocked in 25–30 of 30 iterations.
• The emulated QueueStorage path against per-row inserts did the same on [new A, new B] (30/30) and [new A, existing B] (29/30).
• Every same-path control pair, and every shape where A already exists, gave 0/30.

It only bites the first use of a queue or priority. Lanes that already exist are unaffected.

Fix direction. Give both paths one order. For example, ensure_lane_inserts could insert the queue_lanes row before taking the provisioning lock, matching the trigger path. Add a regression test that runs the two producers concurrently on [new A, new B].

Resolution — 2026-11-11

Fixed and released in v0.6.16 through #702, specifically c8ca4d3b. QueueStorage::ensure_lane_inserts now inserts the queue_lanes row before acquiring the provisioning advisory lock, matching the SQL insert paths.

The release includes deterministic new-A/new-B concurrency regressions for QueueStorage batches against single-row inserts, insert_many and COPY, plus reverse-order and same-path controls. This reconciliation verifies the fixing commits and tests are contained in the release tag; it does not claim a new execution of those suites.

Older QueueStorage binaries retain the old lock order during a rolling upgrade. Upgrade those producers, pre-provision lanes with the fixed version, or retry the whole failed transaction as described in the upgrade guide. Closing this issue as released.