Demote per-5s datagram-loss WARN to debug, export dropped-total as a metric

[allyblockcast]

Summary

moq-pub-mmtp's startup catalog-track log line was being crowded out of the retained production log window by the moq_transport::session::subscribed loss WARN, which fires ~every 5s under sustained loss. That line is already rate-limited at the source, so matching production's RUST_LOG filter to staging's (BLO-22347) cannot suppress a line already emitted at WARN — it only quiets moq_transport's INFO chatter.

This PR implements the fallback AC-2 option from BLO-22347: demote the loss line so it stops dominating default-filter log output, and export the drop count as a durable Prometheus counter instead of only a log line.

• moq-transport/src/session/subscribed.rs: keep the existing 5s rate-limiting for the log line, but demote it from tracing::warn! to tracing::debug!. Every observed loss delta (not just the once-per-5s sampled delta) is now also recorded into a new moq_pub_mmtp_dropped_datagrams_total counter via the metrics crate facade, so the counter reflects real-time loss independent of the log's sampling cadence.
• moq-pub-mmtp: add the metrics-prometheus feature and a new metrics_endpoint module (mirrors moq-relay-ietf's always-compiled facade + optional Prometheus-exporter split). Activated by MOQ_PUB_METRICS_ADDR, the same env-var-only runtime-gating pattern the existing MOQ_PUB_PROFILE_ADDR profiling endpoint uses — a binary built with the feature compiled in stays inert until an operator opts in, and there's no exporter-crash path (install failure logs a warning and the publisher keeps running).

Scope note

This does not wire the Helm chart port/Service/ServiceMonitor or the Grafana panel — that's chart-repo work tracked separately in Blockcast/magma against the same issue.

Paperclip issue: https://paperclip.blockcast.net/BLO/issues/BLO-22882 (blocks BLO-22347 → BLO-21204)

Blast radius

This touches moq-transport session code — the hot path for a live public Solana shred publisher (moq-pub-mmtp in production). The datapath change is a rename (warn→debug) plus one counter increment per loop iteration where loss grew; no control-flow change. The new metrics exporter is off by default (feature-gated + env-var-gated) and fails soft (logs + continues) if bind fails, so it cannot regress publisher availability. Do not merge/deploy to production without a named approver, per the parent issue's blast-radius note.

Test plan

• CI: cargo test --verbose, cargo clippy --no-deps, cargo fmt --check, cargo machete (this repo's pr.yml)
• cargo build -p moq-relay-ietf -p moq-pub-mmtp -p moq-sub-raw + moq-pub-mmtp/tests/shred-datagram-e2e.sh still passes (no log-text or behavior dependency on the old WARN line)
• Manual, post-merge: build with --features metrics-prometheus, run with MOQ_PUB_METRICS_ADDR=127.0.0.1:9091, confirm curl :9091/metrics exposes moq_pub_mmtp_dropped_datagrams_total and it increases during induced loss
• Manual, post-merge: confirm the loss line no longer appears at RUST_LOG=info,quinn=warn,moq_transport=warn,... (shipped production filter) and only appears at RUST_LOG=...,moq_transport=debug

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-21204
🔗 Paperclip issue: BLO-22882
🔗 Paperclip issue: BLO-22347

[allyblockcast]

🔗 Paperclip issue: BLO-21204
🔗 Paperclip issue: BLO-22882
🔗 Paperclip issue: BLO-22347

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 4130bee

Critical Issues (0)

Important Issues (1)

• [code] moq-transport/src/session/subscribed.rs:671 — moq_pub_mmtp_dropped_datagrams_total only increments dropped_total - reported_dropped, which covers ring-superseded datagrams but excludes skipped_too_large. The latter is incremented at line 740 after this accounting block; its subsequent iteration sets loss_grew, but still records a zero dropped_delta. This leaves the advertised counter at zero for a stream that only drops over-MTU payloads.
  • Increment the counter for both deltas, preferably at each loss source, or include skipped_too_large - reported_too_large in the increment before advancing the reported totals.

Suggestions (1)

• [tests] moq-transport/src/session/subscribed.rs:665 — Add a focused metrics-recorder test covering ring loss and over-MTU loss so the exported counter's documented aggregate semantics cannot regress.

Strengths

• The exporter is feature- and environment-gated, and exporter installation failure preserves publisher availability.
• Demoting the rate-limited diagnostic while retaining structured debug context is a reasonable way to reduce production log pressure.

Recommended Action

1. Address the Important counter-accounting issue before merge.
2. Add the focused metric coverage this cycle.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 0993329

Critical Issues (0)

Important Issues (1)

• [code, gstack/review, native-codex] moq-transport/src/session/subscribed.rs:666 — The new counter only increments dropped_total - reported_dropped, which covers ring-superseded datagrams but not the skipped_too_large path. When an over-MTU datagram is skipped at line 740, the next loop sees loss_grew, records an increment of zero, and advances reported_too_large; that skip is consequently never exported. This contradicts both the metric description and the previous loss log, which include over-MTU skips.
  • Increment the metric by both newly ring-dropped and newly over-MTU-skipped datagrams, and add coverage for the over-MTU path.

Suggestions (0)

Strengths

• The exporter is explicitly runtime-gated and installed before the publisher session starts.
• Docker builds enable the exporter across the default and profiling image variants.

Recommended Action

1. Address the Important issue before merge.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 7a6568c

Prior Findings Dispositioned (2)

• prior:4130bee important 1 — fixed — moq-transport/src/session/subscribed.rs:54 — The metric delta now sums ring-superseded and over-MTU deltas before incrementing the counter.
• prior:0993329 important 1 — fixed — moq-transport/src/session/subscribed.rs:54 — The same aggregate calculation includes skipped_too_large - reported_too_large; the focused over-MTU test at moq-transport/src/session/subscribed.rs:908 verifies it.

Critical Issues (0)

Important Issues (1)

• [code, errors, native-codex] moq-transport/src/session/subscribed.rs:759 — An over-MTU payload increments skipped_too_large and immediately continues, while record_datagram_loss_metric runs only at the start of the next loop iteration. If this is the final datagram, the stream ends without another accounting pass, so the advertised total permanently omits that drop.
  • Record the over-MTU loss at the skip site, or flush outstanding skipped_too_large loss after the read loop; add an end-of-stream coverage case.

Suggestions (0)

Strengths

• The counter now correctly aggregates ring-superseded and over-MTU loss, with focused unit coverage for both deltas.
• The exporter remains explicitly runtime-gated and does not take down the publisher when installation fails.

Recommended Action

1. Fix the final-datagram accounting gap before merge.