fix(moq-pub-mmtp): backport reconnect-with-backoff onto pinned live-edge baseline 21006781 (BLO-26173)

[allyblockcast]

Issue: https://paperclip.blockcast.net/BLO/issues/BLO-26173

Why a backport instead of re-pinning to main

Production blockcastd-moq-pub-mmtp (the live public Solana shred publisher) crash-loops on relay session timeout: it propagates the error out of main, exits 1, and relies on restartPolicy: Always. Most recent occurrence 2026-08-22 — restartCount: 11 in 34 minutes, every one exitCode: 1.

The fix already exists on main (#76 + #78), but production is not on main and deliberately so. onprem-k8s/production-values.yaml:1221 pins the pub/relay pair to moq-rs 21006781 — "the four-MTU oldest-drop queue bound on the prior production protocol baseline… without the unrelated broad moq-rs upgrade" — authored by Omar Ramadan on 2026-08-06. main is 165 commits ahead of that pin and moves exactly the surface the pin exists to freeze:

file	churn across the gap
moq-transport/src/session/request_id.rs	+40 −17
moq-transport/src/session/draft19.rs	+9 −40
moq-transport/src/session/mod.rs	+157 −1
moq-transport/tests/draft19_wire.rs	−51

A mismatched pub reingests the shred outage (session error: invalid request ID crashloop). So this PR ports the resilience fix onto the pinned baseline rather than dragging the protocol migration in behind it.

Base branch release/pinned-live-edge-21006781 was created at that exact pinned commit so this renders as a reviewable 3-file diff.

Scope gate

$ git diff --stat 21006781..HEAD -- moq-transport/
(empty)
$ git diff --name-only 21006781..HEAD | grep -v '^moq-pub-mmtp/'
(none)

Strictly moq-pub-mmtp/**. The relay image is untouched, so the pub/relay pair stays matched.

What changed

• Relay session is established inside a retry loop with exponential backoff (--reconnect-backoff-min-ms 500 → --reconnect-backoff-max-ms 30000) instead of exiting.
• The input source is opened once and held across every reconnect. For --mmtp-input=udp this is what keeps the SSM (S,G) join alive — the join is a property of the socket, not of the moq-transport session. (AC BLO-8047 §B1: raw-passthrough fragmentation contract (stacked on #1) #2.)
• Packet ingest moves from a third spawned task into the session select! so the source is owned by main across reconnects. Ingest stays a single ordered flow, so datagram.rs's monotonic group_id assignment is unchanged and wire output is byte-identical.
• A session that outlives the backoff ceiling clears the consecutive-failure streak (fix(moq-pub-mmtp): reset reconnect backoff after a healthy session; unbreak test build #78), so a publisher that has dropped N times over its lifetime doesn't stall the full 30s on every later blip.
• Clean stdin EOF still exits 0 rather than reconnecting forever.

Deliberately NOT backported

• moq_pub_session_reconnect_total counter. metrics is not a dependency of moq-pub-mmtp at this baseline and metrics_endpoint.rs does not exist here — adding them pulls in the whole observability stack this backport exists to avoid. Reconnects are observable via structured task= / attempt= / backoff_ms= log fields instead, which satisfies BLO-26173 AC BLO-8047 §B4: G6 byte-diff vs libmoq — wire formats diverge by design (no code change) #4 ("counter and/or log at info").
• fix(moq-pub-mmtp): reset reconnect backoff after a healthy session; unbreak test build #78's moq-transport/src/serve/subgroup.rs binary-search change. Pure read-path optimization, unrelated to reconnect, and touching moq-transport is precisely what the pinned pair forbids.

Verification status — please read

cargo build/test were not run locally: the agent workspace has a Rust toolchain but no C linker (cc not found) and no root to install one. This PR is pushed so CI performs the first real compile. Static checks that were done: cargo fmt --check clean; bail! import retained (still used at 5 sites); connect() confirmed to return anyhow::Result so no useless_conversion.

Do not merge on a red or absent check. The tests added here (backoff_duration_doubles_and_caps, healthy_session_resets_the_reconnect_streak, reconnect_sleeps_the_current_attempt_then_increments, the two stdin next_input_event cases) are the unit-level gate; the 48h restartCount-flat criterion is only verifiable post-deploy.

Not in this PR

The production re-pin. That is kkroo's call once an image exists from this branch — see BLO-26173 and onprem-k8s#2560.

[allyblockcast]

🔗 Paperclip issue: BLO-26173

[allyblockcast]

🔗 Paperclip issue: BLO-26173

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: aca78a0

Critical Issues (0)

Important Issues (1)

• [native-codex] moq-pub-mmtp/src/main.rs:210 — the stdin frame reader is cancellation-unsafe inside this select!: next_input_event calls framing::read_one_frame, which consumes the prefix with read and then uses read_exact. If session_task or publish_namespace_task completes while the stdin future is between those reads, select! drops that future; the already-consumed prefix bytes are lost, and the next reconnect starts at the payload or a later frame. That produces a framing error and exits instead of reconnecting. Keep one persistent frame-read future/state across session attempts, or make input ingestion a dedicated task/channel whose read future is never cancelled; add a test that forces a task completion while a frame is only partially available.

Suggestions (0)

Strengths

• The UDP socket is opened once, preserving the multicast join across relay reconnects.
• Reconnect backoff and streak reset behavior are factored into small, directly tested helpers.
• The PR explicitly expands CI coverage to the release/** backport lane.

Recommended Action

1. Fix the Important issue before merge.
2. Re-run the package tests and CI on the pinned-baseline branch.
3. Verify reconnect behavior with a fragmented stdin frame and a concurrent session termination.

[allyblockcast]

Reviewer notes — the two questions this change should be interrogated on

1. Group IDs restart at 0 on every reconnect. Is that a regression?

No — it is bit-for-bit the behavior production already has today, just without the downtime.

Each reconnect rebuilds Tracks, so build_datagram_state re-enters at DatagramState::new(name, 0, datagrams) and group IDs restart from 0. That looks alarming until you note what it replaces: today the publisher exits(1) on session loss and restartPolicy: Always restarts the process, which re-enters the exact same constructor with the exact same 0.

So:

	today (exit + pod restart)	this PR (in-process reconnect)
group_id after a session loss	restarts at 0	restarts at 0
frequency	once per session loss	once per session loss
SSM (S,G) join	torn down and re-joined	held
downtime	pod schedule + QUIC + re-join	one backoff interval

Identical in kind, strictly better in duration. This PR does not introduce a new group-id discontinuity; it removes the outage wrapped around the existing one.

Production confirms the router in play is the datagram one, so this is the relevant path:

INFO moq_pub_mmtp: built publisher router from catalog router="datagram"

If a reviewer wants group IDs to survive a reconnect, that is a separate and larger change (the counter would have to outlive Tracks), and it should not ride along here.

2. select! cancellation safety of next_input_event

tokio::select! drops the losing branch's future. Worth being explicit about what that means per input mode:

• UDP (--mmtp-input=udp) — safe. UdpSocket::recv_from is cancellation-safe; a dropped pending recv leaves the datagram in the socket buffer. This is the production path — the live shred publisher runs --mmtp-input udp --mmtp-udp-source 69.25.95.57.
• stdin — not cancellation-safe. framing::read_one_frame reads a 4-byte length prefix then the body; dropping between the two would desync the frame stream.

The stdin exposure is real but narrow: the future is only dropped when session_task or publish_namespace_task completes, i.e. when the session is being torn down anyway, and stdin is the dev/test path, not a deployed one. I am flagging rather than fixing it because upstream #76 has identical semantics, and silently diverging from upstream in a backport is worse than inheriting a known, documented property. If reviewers want it hardened, the fix is to make read_one_frame resumable, and it belongs upstream on main first so the two do not drift.

Verification status

cargo build/test were not runnable in the authoring environment (Rust toolchain present, but no C linker and no root to install one — cargo build dies in proc-macro2's build script). CI on this PR is the first real compile. That is also why this PR carries the pr.yml trigger commit: without it a release/**-based PR gets zero check runs, which is precisely how 54d855f shipped a tree that failed cargo test -p moq-pub-mmtp.

Please do not merge on a red or absent build.