Skip to content

feat(moq): add profile-negotiation observability for BLO-22255 - #74

Merged
kkroo merged 3 commits into
mainfrom
blo-22255-canary-and-metrics-observability
Aug 10, 2026
Merged

feat(moq): add profile-negotiation observability for BLO-22255#74
kkroo merged 3 commits into
mainfrom
blo-22255-canary-and-metrics-observability

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Aug 9, 2026

Copy link
Copy Markdown

Summary

Platform/SRE side of BLO-22255 (deploy Blockcast-profile producers + convergence receipts). moq-native-ietf already records moq_negotiation_total on every connect attempt (added alongside the Blockcast01 profile in #65/#66), but no producer binary ever installed a Prometheus recorder to serve it — the metric existed but was unobservable. This PR closes that gap without touching negotiation/enforcement logic:

  • moq-pub-mmtp: add --metrics-addr behind an opt-in metrics-prometheus cargo feature, mirroring moq-relay-ietf's existing exporter wiring exactly (same PrometheusBuilder pattern, same feature-off warning path). With the flag unset or the feature disabled at build time, behavior and the shipped binary are unchanged — this only adds an optional listener.
  • moq-canary (new binary): connects under moqt-blockcast-01, issues a SUBSCRIBE, and decodes SUBSCRIBE_OK key 0x40 via TrackReader::history_window() on a timed loop. Each cycle opens a fresh connection and reconnects, so a long-running instance is evidence of repeated negotiation, not one lucky session. Exposes moq_canary_probe_total{outcome=...} / moq_canary_history_window_groups behind the same opt-in exporter pattern (--metrics-addr + metrics-prometheus feature). Also supports --once for CI/manual smoke checks.

Both together give BLO-22255's verifying signal a path to exist: moq_negotiation_total{outcome="selected",required="moqt-blockcast-01"} from the publisher side, and a decoded-0x40-was-nonzero receipt from the canary side.

Not in scope here (tracked separately): the submodule-pointer bump (BLO-23504), the actual --metrics-addr deployment/scrape config, and any production re-pin — those are deliberate, separately-approved steps per the rollout record.

Test plan

  • cargo generate-lockfile-free minimal resolution checked locally (no cc/linker available in this sandbox, so full cargo build/test could not run here — relying on CI)
  • cargo fmt --edition 2021 --check clean on all new/changed files
  • CI: cargo test --verbose (covers new moq-canary unit tests + existing moq-pub-mmtp CLI tests)
  • CI: cargo clippy --no-deps
  • CI: cargo machete (verifies no unused deps — metrics-exporter-prometheus is cfg-gated in both crates, matching the existing moq-relay-ietf precedent)

moq-native-ietf already records moq_negotiation_total on every connect
attempt (added with the Blockcast01 profile in #65/#66), but no producer
binary installed a recorder to serve it, so the metric was invisible.

- moq-pub-mmtp: add --metrics-addr behind an opt-in metrics-prometheus
  feature, mirroring moq-relay-ietf's existing exporter wiring. With the
  flag unset or the feature off, behavior is unchanged.
- moq-canary: new binary that connects under moqt-blockcast-01, subscribes,
  and decodes SUBSCRIBE_OK key 0x40 via TrackReader::history_window() on a
  timed loop, reconnecting every cycle so a long-running instance is
  evidence of repeated negotiation rather than one lucky session. Exposes
  moq_canary_probe_total / moq_canary_history_window_groups behind the same
  opt-in exporter pattern.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@allyblockcast

allyblockcast Bot commented Aug 9, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22255
🔗 Paperclip issue: BLO-23504

1 similar comment
@allyblockcast

allyblockcast Bot commented Aug 9, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22255
🔗 Paperclip issue: BLO-23504

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9b62a5f

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

  • The canary creates a fresh negotiated connection for every cycle and fails closed when the required history-window receipt is absent.
  • The Prometheus listener is opt-in and follows the existing relay exporter pattern, preserving the default binaries' behavior.
  • CLI defaults and failure-label classification have focused unit coverage.

Recommended Action

  1. Merge once CI completes.

…EUSE

- moq-canary/src/main.rs: bind the Subscribe handle from subscribe_open()
  to a named variable instead of discarding it. Subscribe unsubscribes on
  Drop, so the previous code tore the subscription down immediately after
  issuing it (and failed to compile under -D unused-must-use).
- Dockerfile: the manifest-only pre-copy stage used for dependency-layer
  caching didn't know about the new moq-canary workspace member, so the
  first `cargo build --release` failed with "failed to load manifest for
  workspace member `/build/moq-canary`". Add its Cargo.toml copy, stub
  src dir, and stub main.rs alongside the other binary-only crates.
- REUSE.toml: add moq-canary/** to the Blockcast-fork-additions annotation
  block (same pattern as moq-pub-mmtp/**) so the new crate's files pick up
  copyright attribution without per-file SPDX headers.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@allyblockcast

allyblockcast Bot commented Aug 9, 2026

Copy link
Copy Markdown
Author

Fixed the three CI failures from 9b62a5f in c113f4f:

  • build: moq-canary/src/main.rs:125 discarded the Subscribe handle returned by subscribe_open() as a bare statement. Subscribe unsubscribes on Drop, so this wasn't just a -D unused-must-use lint failure — the canary was tearing its own subscription down immediately after opening it, before it could ever read the history window back off track_reader. Bound it to let _subscribe = ... so it lives through the read.
  • heap-profile-image: the Dockerfile's manifest-only pre-copy stage (used to cache the dependency layer before COPY . ./) didn't know about the new moq-canary workspace member, so the first cargo build --release failed with failed to load manifest for workspace member /build/moq-canary. Added its Cargo.toml copy + stub src/main.rs, same treatment as the other binary-only crates (moq-pub, moq-sub-raw, etc.).
  • reuse: added moq-canary/** to the Blockcast-fork-additions annotation in REUSE.toml, same pattern already used for moq-pub-mmtp/**.

CI is re-running on c113f4f.

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: c113f4f

Critical Issues (0)

Important Issues (1)

  • [native-codex] moq-canary/src/main.rs:56 — A timeout drops the probe_once future before it reaches the session_task.abort() at line 157. Dropping a Tokio JoinHandle detaches rather than cancels it, so any hung probe leaves session.run() and its connection alive; repeated failures accumulate background sessions despite the canary's bounded-cycle contract.
    • Run the probe/session under an owned task or cleanup guard and explicitly abort and await it on the timeout path before starting the next cycle.

Suggestions (0)

Strengths

  • The canary uses a fresh negotiation attempt per normal cycle and exposes a small, bounded metric-label set.
  • The Prometheus listener remains opt-in and follows the established relay pattern.

Recommended Action

  1. Address the timed-out session cleanup before merge.

@kkroo
kkroo merged commit 6d35e63 into main Aug 10, 2026
3 checks passed
@kkroo
kkroo deleted the blo-22255-canary-and-metrics-observability branch August 10, 2026 02:50

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: e2711a4

Prior Findings Dispositioned (1)

  • prior:c113f4 important 1 — fixed — moq-canary/src/main.rs:355 — The timeout now owns the probe in a JoinHandle, then explicitly aborts and awaits it before returning, so a timed-out session cannot detach into a later cycle.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

  • The new canary creates a fresh profile-negotiated connection for each probe and treats a missing history-window receipt as a failed probe.
  • Prometheus export remains opt-in in both binaries and preserves default runtime behavior.
  • The timeout regression has focused coverage that verifies the aborted task is dropped before the cycle continues.

Recommended Action

  1. Merge once the required repository checks are satisfied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant