[slopfix] test(universaldb): postgres leader failover + abort resolver task on driver drop

[MasterPtato]

No description provided.

[MasterPtato]

Stack for rivet-dev/rivet

• #5552 [slopfix] fix(universaldb): match fdb conflict semantics in …
• #5551 [slopfix] fix(universaldb,util,pegboard-envoy): index dedup …
• #5398 feat(kitchen-sink): add churnDb actor and seed driver for co …
• #5397 chore: Clean up
• #5382 [slopfix] docs(self-hosting): correct postgres config format …
• #5356 [slopfix] fix(pegboard): decrement kv_storage_used metric on …
• #5354 [slopfix] refactor(universaldb,ups): single/multi-node udb o …
• #5353 [slopfix] perf(universaldb): fold postgres drain_batch into …
• #5342 [SLOP(claude-opus-4-8-high)] perf(universaldb): batch leader …
• #5341 [slopfix] docs(self-hosting): promote postgres from experime …
• #5340 [slopfix] fix(self-host): repair dev compose generation, kit …
• #5337 [SLOP(claude-opus-4-8-high)] feat(universaldb): graceful pos …
• #5336 [slopfix] test(universaldb): postgres leader failover + abor … 👈
• #5335 [SLOP(claude-opus-4-8-high)] feat(ups): table-backed postgre …
• #5329 [SLOP(claude-opus-4-8-high)] feat(universaldb): postgres lea …
• #5327 [SLOP(claude-opus-4-8)] feat(pegboard-envoy): rate-limit env …
• #5326 [SLOP(claude-opus-4-8)] feat(util): add rate limiter primiti …
• #5322 [SLOP(claude-opus-4-8)] feat(rivetkit-core): add serde durat …
• #5321 Add observation for serde_bare
• #5320 Add observation and ser/de metrics for serde calls
• #5314 [SLOP(claude-opus-4-8-medium)] chore: add profiling cargo pr …
• #5306 [SLOP(claude-opus-4-8)] fix(kitchen-sink): serve built front …
• main

Get stack: forklift get 5336
Push local edits: forklift submit
Merge when ready: forklift merge 5336

_{change tvmntvqo}

[claude]

Code Review

Re-reviewed the current diff (base: stack/slop-claude-opus-4-8-high-feat-ups-table-backed-postgres-transport-with-coalesced-doorbell-xutxrzrv). Summary: adds a Postgres leader-failover integration test for universaldb, aborts the resolver's JoinHandle in Drop so a dead leader stops renewing its lease, migrates universalpubsub's Postgres driver from per-subscriber heartbeats to a single per-node heartbeat + registry, and adds a no-responders fast-fail path for request/reply. The core resolver_handle.abort() fix and the failover test are solid, well-commented additions.

---

1. engine/packages/perf/src/lib.rs — breaks an existing CI test

The Drop for PerfMeasure log was demoted from tracing::warn! to tracing::debug!. engine/packages/perf/tests/perf_measure.rs::perf_drop_without_finish_warns_and_does_not_record (line 201) asserts Level::WARN for exactly this event and will now fail - the RecordingLayer has no level filter, so the event is still captured but at Debug, not Warn. (The other Level::WARN assertion at line 169 is for a different macro path - perf_finish!'s slow-threshold warning - and is unaffected.)

Beyond the broken test, demoting this hides violations of the CLAUDE.md invariant that every PerfMeasure must end with perf_finish!/perf_abandon!, making silently-discarded measurements harder to catch in production logs (which typically run at info).

---

2. engine/packages/universalpubsub/src/driver/postgres/mod.rs — extra DB round-trip on every request-reply publish

has_responders() runs unconditionally on every publish(..., Some(reply_subject)), adding a two-subquery Postgres round-trip to the most latency-sensitive publish path (actor action dispatch, pegboard coordination, etc.) before the actual insert. Consider caching live-subject state briefly, or checking only after a first timeout/attempt rather than on the hot path.

---

3. has_responders queue-subscriber branch matches on hash only, not full subject

The broadcast branch correctly guards with s.subject_hash = $1 AND s.subject = $2, but the queue branch only has s.subject_hash = $1 since ups_queue_subs stores no raw subject string. A hash collision between two different subjects would make has_responders return true for a subject with no real queue subscriber, so the request publishes a message that then sits unclaimed until GC instead of failing fast. Low probability with a 64-bit hash, but the asymmetry with the broadcast branch is worth closing by adding a subject column to ups_queue_subs.

---

4. engine/packages/universaldb/tests/failover.rs — hardcoded 4s sleep before first connection attempt

tokio::time::sleep(Duration::from_secs(4)).await is the only guard between docker_config.start() and the first connect_raw/make_db. On a loaded CI runner Postgres may not be accepting connections yet, and make_db will fail with the failure misattributed to a product regression. This same PR fixes exactly this class of flakiness in scripts/run/engine-postgres.sh by switching from a fixed wait / nc -z to polling pg_isready - worth applying the same retry-loop pattern here instead of a fixed sleep.

---

5. tokio::spawn in Drop for subscriber cleanup may not run at shutdown

Both PostgresSubscriber::drop and (pre-existing) PostgresQueueSubscriber spawn a task to DELETE their registry row. If the Tokio runtime is already shutting down when the drop fires, that spawned future may never be polled, leaving a stale ups_subs/ups_queue_subs row until node-level GC catches it (up to NODE_TTL_SECS = 30s). Given GC coverage this is probably fine as best-effort cleanup, but a short comment noting the trade-off would help future readers.

---

6. Diff bundles changes unrelated to the stated scope

Beyond the universaldb/universalpubsub work described in the title, this diff also: removes ApiPublic config entirely (config/api_public.rs, config-schema.json, test-deps/datacenter.rs), removes pegboard-gateway2's shutdown_watcher task and its SHUTDOWN_IN_FLIGHT_ABORTED_TOTAL metric, and downgrades several one-time startup logs from info to debug (metrics-server, both gateway shared_state.rs files). None of this is mentioned in the commit message or PR title. The shutdown_watcher/metric removal in particular is a real observability regression (loses visibility into in-flight requests abandoned on pod shutdown), not just a logging tweak - worth confirming these are intentional carry-overs from an earlier stack PR rather than an accidental revert against main (main currently still has both ApiPublic and the shutdown watcher).