Summary
The outbox relay's FOR UPDATE SKIP LOCKED claim is not exclusive across replicas. Two replicas can claim and publish the same rows.
Cause
claimBatch runs the SELECT ... FOR UPDATE SKIP LOCKED inside WithTx (internal/pg/outbox.go:63-89), which is pgx.BeginFunc (internal/pg/outbox.go:21) and therefore commits when the function returns. Row locks release at that commit. The publish loop and MarkPublished in RelayBatch (internal/pg/outbox.go:91) then run with no lock held.
SKIP LOCKED protects only the duration of the SELECT, which is not what the pattern is meant to guarantee.
Impact
Currently latent — the production outbox is drained (315 rows, 0 unpublished), so batches are small and the window is short. It becomes real under backlog: the deployment runs 3 replicas, each on a 5s relay ticker, and a large batch can take far longer than 5s to publish.
Blast radius is bounded by Hatchet's ConcurrencyKeySite serialisation, so the failure mode is duplicate work rather than corruption. The code still reads as exclusive and is not.
Fix direction
- Hold the transaction open across publish +
MarkPublished, or
- Add an explicit claim column (
claimed_at / claimed_by) with a lease and reclaim-on-expiry, which also survives a replica dying mid-batch.
A test asserting two concurrent relays never publish the same row would pin this.
Summary
The outbox relay's
FOR UPDATE SKIP LOCKEDclaim is not exclusive across replicas. Two replicas can claim and publish the same rows.Cause
claimBatchruns theSELECT ... FOR UPDATE SKIP LOCKEDinsideWithTx(internal/pg/outbox.go:63-89), which ispgx.BeginFunc(internal/pg/outbox.go:21) and therefore commits when the function returns. Row locks release at that commit. The publish loop andMarkPublishedinRelayBatch(internal/pg/outbox.go:91) then run with no lock held.SKIP LOCKEDprotects only the duration of the SELECT, which is not what the pattern is meant to guarantee.Impact
Currently latent — the production outbox is drained (315 rows, 0 unpublished), so batches are small and the window is short. It becomes real under backlog: the deployment runs 3 replicas, each on a 5s relay ticker, and a large batch can take far longer than 5s to publish.
Blast radius is bounded by Hatchet's
ConcurrencyKeySiteserialisation, so the failure mode is duplicate work rather than corruption. The code still reads as exclusive and is not.Fix direction
MarkPublished, orclaimed_at/claimed_by) with a lease and reclaim-on-expiry, which also survives a replica dying mid-batch.A test asserting two concurrent relays never publish the same row would pin this.