feat(messaging): handle an inbound platform event once (CHOO-2626)

[petr-sandbox]

Fourth in the multi-tenancy Phase 4 stack, on top of #476.

Slack allows three seconds to acknowledge an event and re-sends whatever it does not get an answer to. The payload of a retry is byte-identical to the original, so nothing below the route could tell it from a second person saying the same words — and the visible failure is an agent replying twice to one question in a customer's channel. This is not an edge case for a distributed app; it is the ordinary consequence of one slow room, and it has to land before a live Slack app points at this deployment.

What it adds

messaging_event_receipts, one row per inbound event the platform numbers, plus the claim/handle/prune store around it and the X-Slack-Retry-Num plumbing to read a delivery's attempt count.

The decisions worth arguing with

The row is written before the work, not after. The unique index is what arbitrates: two retries in flight together both reach the insert and exactly one survives it. Recording afterwards would order the two the wrong way round — both would dispatch, and the duplicate would be detected once it no longer mattered.

The claim is committed before the dispatch. An uncommitted claim still holds the index entry, so a concurrent retry's insert would block on it rather than fail — and stay blocked for the length of an agent's turn. Committing straight away turns that wait into the immediate refusal it should be.

That ordering chooses at-most-once, and I want that stated rather than buried. An event claimed by a process that then dies is not retried, because the platform has already been told 200 and the table says the event is taken. It is not a new loss — the route has acknowledged before handling since it was written, because the deadline is shorter than a turn — so the event was already unrecoverable at that point. What this adds is handled_at, which makes the loss visible: a claimed receipt that never completed is a real event that reached nobody, and it can be found.

Uniqueness is (tenant_id, platform, external_event_id), not deployment-wide. The two protect equally — a Slack event id is unique in Slack's own namespace and a workspace belongs to one tenant — so the tenant-local index is the one to prefer: it keeps one customer's ids out of another's namespace and guarantees a conflict is always with a row the inserting tenant can actually see.

Deduplication sits after resolve, not before it. Two consequences, both wanted. The table is ordinary RLS-scoped and needs no new SECURITY DEFINER exemption, so db/tenant_lookup.py's closed list stays closed. And it composes correctly with the 503 a restarting bridge already answers: that path writes no receipt, so the retry Slack sends in response is handled rather than dropped. There is a test for exactly that.

I deviated from the plan I gave you earlier. The note said "restructure _inbound to verify → 200 → resolve/dedupe/dispatch in the background". On implementation I kept resolve before the 200, because moving it after would lose the WebhookBridgeUnavailable → 503 → Slack-retries property above. resolve is two indexed reads and comfortably inside the three seconds. Happy to be talked out of it.

Only numbered envelopes are claimed. Slack numbers Events API deliveries and retries only those; a slash command and an interaction arrive once with no id, so they dispatch unclaimed. A missing id means "the platform does not retry this", never "this was not checked" — and refusing an unnumbered event for having nothing to deduplicate by would drop a real message.

X-Slack-Retry-Num is a hint, never a decision. It is outside the signature, which covers the body and the timestamp, so a forged or garbled value degrades to zero. Its one use is a logger.warning: a run of retries is the only signal this deployment gets that its own acknowledgements are arriving too late.

Pruning is opportunistic, following the role_leases precedent. There is no row-deleting janitor anywhere in this backend, and inventing one for a single table would be a larger change than the table deserves. The sweep runs in the post-dispatch transaction, off the claim's critical path.

Follow-up worth its own decision

messaging_install_states, invitations and agent_sessions all grow without bound for the same reason — every "expiry" in this backend is a read-time predicate and nothing deletes rows. Not fixed here; worth a ticket.

Testing

Full suite green (3413 passed). New coverage: a retry of the same event_id dispatches once; a retry arriving mid-turn loses to the delivery in flight (deterministic, via a gated adapter); an unnumbered event dispatches every time and writes no receipt; one tenant's event ids do not block another's; handled_at is set on completion; a receipt past its retention is pruned by the next event; an unknown workspace writes no receipt; a bridge that was down still takes the retry. Plus the Slack-installer parsing tests for event_id and the retry header.

🤖 Generated with Claude Code