Skip to content

postgres_cdc: fail the snapshot barrier on nack (CON-504) - #4685

Closed
squiidz wants to merge 1 commit into
mainfrom
con-504-postgres-nack-gate
Closed

postgres_cdc: fail the snapshot barrier on nack (CON-504)#4685
squiidz wants to merge 1 commit into
mainfrom
con-504-postgres-nack-gate

Conversation

@squiidz

@squiidz squiidz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Part of CON-504 (CDC at-least-once / ack-gated progress).

Follow-up to #4584: the snapshot ack barrier treated a nack exactly like an ack. The ackFn ignored its error argument, so with auto_replay_nacks: false a rejected snapshot batch released the barrier, the replication slot was promoted, and the rejected rows were skipped forever on restart. A nacked batch also resolved its checkpoint slot, letting later acks acknowledge LSNs past undelivered streaming rows.

This applies the pattern hardened through the review rounds on #4675 and #4677:

  • A nack never resolves the checkpoint: nothing can be acknowledged past the rejected batch's rows, and an error log identifies the pinned position and the stall-once-checkpoint_limit-fills consequence so operators can attribute it.
  • A nacked snapshot batch fails the promotion barrier: the input restarts without promoting the slot and the snapshot re-runs.
  • The gate error resets per connection attempt, so one nack fails only the attempt it belongs to instead of livelocking reconnects.

Proof of Work

  • TestIntegrationPostgresSnapshotNackFailsBarrier: the consumer nacks the first snapshot batch (auto_replay_nacks: false), asserting the slot is not promoted while the snapshot is rejected; the input then self-heals — re-runs the snapshot, delivers every row, and only then promotes. Verified it fails against the pre-fix code: the rejected rows are never redelivered (the snapshot should have re-run and re-delivered every row after the nack, 5m timeout).
  • The full postgresql integration suite (13 tests, including the postgres_cdc: wait for snapshot acks before promoting the replication… #4584 crash barrier test) passes.
  • Unit tests green under -race; lint and docs clean.

…over rejected rows

The snapshot ack barrier (#4584) treated a nack exactly like an ack: the
ackFn ignored its error argument, so with auto_replay_nacks disabled a
rejected snapshot batch released the barrier, the replication slot was
promoted, and the rejected rows were skipped forever on restart. A nacked
batch also resolved its checkpoint slot, letting later acks acknowledge
LSNs past undelivered streaming rows.

A nack now never resolves the checkpoint (nothing can be acknowledged past
its rows, with an error log identifying the pinned position), and a nacked
snapshot batch fails the promotion barrier: the input restarts without
promoting the slot and the snapshot re-runs. The gate error resets per
connection attempt so one nack cannot livelock reconnects. Same pattern as
the review-hardened mssqlserver (#4677) and oracledb (#4675) gates.
Comment on lines +466 to +467
// The input outlives reconnects: clear any nack recorded by a previous
// snapshot attempt so the promotion barrier judges only this run.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels somewhat of a redundant comment.

iamAuthEnabled bool
}

func (p *pgStreamInput) recordSnapshotNack(err error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth moving recordSnapshotNack, snapshotNackError and resetSnapshotGate to the bottom of the file? Keeping the important, relevant or public ones (Connect, ReadBatch and Close) at the top?

@squiidz

squiidz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this PR's premise misread the framework contract. auto_replay_nacks: false is a documented opt-in to DROP rejected messages ("If set to false these messages will instead be deleted" — the field's own description), typically because failures are routed to a DLQ which acks. Failing the promotion barrier and pinning the checkpoint on nack contradicts that contract and produces permanent backpressure / re-snapshot loops for data the user explicitly chose to discard. The original #4584 behavior (a nack releases the barrier and the checkpoint advances) is correct. The nack-pinning changes made on the other CON-504 PRs during review are being unwound for the same reason.

@squiidz squiidz closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants