TD-6395: Guard checkpoint writes on ownership, not lease_count

[tkusumo]

Problem

order_service logs recurring :update_checkpoint_failed on its Kinesis consumers in prod (assessment-event-prod, cognitive-test-event-prod, external-assessment-event-prod) — ~1–3 per 10 min, with heavier bursts during deploy-time lease rebalancing. In the failures lease_owner == current_shard_owner, so it's a checkpoint conditional-write failure against the lease table, not a lease-ownership mismatch. A failed checkpoint doesn't advance the shard, so messages can be re-delivered (at-least-once).

Ticket: TD-6395

Root cause

The Ecto adapter's update_checkpoint did a read (get_shard_lease) then a conditional UPDATE via update_shard_lease/3, whose WHERE clause (build_where_clause/2) gates on lease_count. The Lease process renews every 30s (same owner), incrementing lease_count. When a renewal landed between the checkpoint's read and write, the UPDATE matched 0 rows → {:error, :update_unsuccessful} → :update_checkpoint_failed, even though this worker still held the lease.

That explains every symptom: same owner (renewal doesn't change owner); intermittent and low-rate (narrow read→write window); worse during rebalancing (extra lease_count churn from take_lease); redelivery with no loss.

The lease_count guard is correct for renew_lease/take_lease (optimistic concurrency) but wrong for checkpointing, which only needs to confirm the lease is still owned. The Dynamo adapter already gets this right — its update_checkpoint conditions only on lease_owner.

Fix

Give checkpoint its own ShardLeases.update_checkpoint/3 that writes in a single UPDATE ... RETURNING guarded on ownership only (shard_id + app_name + stream_name + lease_owner), never lease_count. This removes the read→write race window entirely and matches the Dynamo adapter's semantics.

• A lease genuinely taken by another worker still matches 0 rows and surfaces :update_checkpoint_failed (correct).
• renew_lease / take_lease keep their lease_count optimistic lock — untouched.

Tests

Added a describe "update_checkpoint ownership guard" block (the existing fake Repo can't distinguish the WHERE clause, so two small purpose-built stub repos were added):

• guards on lease_owner, not lease_count — captures the generated query and asserts the WHERE contains lease_owner and not lease_count (fails against the old code).
• lost lease → :update_checkpoint_failed — a repo returning {0, []} still maps to the correct error.

Full non-integration suite: 51 tests, 0 failures.

Notes

• Independent of PR TD-6376: Improve lease management: fix lease races and rework rebalancing to per-worker #49 (TD-6376); that PR left update_checkpoint's path unchanged, so this applies cleanly alongside it.
• The ticket assumed a DynamoDB lease table; the :update_checkpoint_failed atom is Ecto-only, so order_service's lease table is Postgres. Same class of issue, different store — worth correcting on the ticket.

🤖 Generated with Claude Code