feat(moq): add profile-negotiation observability for BLO-22255 - #74
Merged
Conversation
moq-native-ietf already records moq_negotiation_total on every connect attempt (added with the Blockcast01 profile in #65/#66), but no producer binary installed a recorder to serve it, so the metric was invisible. - moq-pub-mmtp: add --metrics-addr behind an opt-in metrics-prometheus feature, mirroring moq-relay-ietf's existing exporter wiring. With the flag unset or the feature off, behavior is unchanged. - moq-canary: new binary that connects under moqt-blockcast-01, subscribes, and decodes SUBSCRIBE_OK key 0x40 via TrackReader::history_window() on a timed loop, reconnecting every cycle so a long-running instance is evidence of repeated negotiation rather than one lucky session. Exposes moq_canary_probe_total / moq_canary_history_window_groups behind the same opt-in exporter pattern. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
1 similar comment
Author
Author
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9b62a5f
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The canary creates a fresh negotiated connection for every cycle and fails closed when the required history-window receipt is absent.
- The Prometheus listener is opt-in and follows the existing relay exporter pattern, preserving the default binaries' behavior.
- CLI defaults and failure-label classification have focused unit coverage.
Recommended Action
- Merge once CI completes.
…EUSE - moq-canary/src/main.rs: bind the Subscribe handle from subscribe_open() to a named variable instead of discarding it. Subscribe unsubscribes on Drop, so the previous code tore the subscription down immediately after issuing it (and failed to compile under -D unused-must-use). - Dockerfile: the manifest-only pre-copy stage used for dependency-layer caching didn't know about the new moq-canary workspace member, so the first `cargo build --release` failed with "failed to load manifest for workspace member `/build/moq-canary`". Add its Cargo.toml copy, stub src dir, and stub main.rs alongside the other binary-only crates. - REUSE.toml: add moq-canary/** to the Blockcast-fork-additions annotation block (same pattern as moq-pub-mmtp/**) so the new crate's files pick up copyright attribution without per-file SPDX headers. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Author
|
Fixed the three CI failures from
CI is re-running on |
Author
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: c113f4f
Critical Issues (0)
Important Issues (1)
- [native-codex]
moq-canary/src/main.rs:56— A timeout drops theprobe_oncefuture before it reaches thesession_task.abort()at line 157. Dropping a TokioJoinHandledetaches rather than cancels it, so any hung probe leavessession.run()and its connection alive; repeated failures accumulate background sessions despite the canary's bounded-cycle contract.- Run the probe/session under an owned task or cleanup guard and explicitly abort and await it on the timeout path before starting the next cycle.
Suggestions (0)
Strengths
- The canary uses a fresh negotiation attempt per normal cycle and exposes a small, bounded metric-label set.
- The Prometheus listener remains opt-in and follows the established relay pattern.
Recommended Action
- Address the timed-out session cleanup before merge.
Author
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: e2711a4
Prior Findings Dispositioned (1)
- prior:c113f4 important 1 — fixed —
moq-canary/src/main.rs:355— The timeout now owns the probe in aJoinHandle, then explicitly aborts and awaits it before returning, so a timed-out session cannot detach into a later cycle.
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The new canary creates a fresh profile-negotiated connection for each probe and treats a missing history-window receipt as a failed probe.
- Prometheus export remains opt-in in both binaries and preserves default runtime behavior.
- The timeout regression has focused coverage that verifies the aborted task is dropped before the cycle continues.
Recommended Action
- Merge once the required repository checks are satisfied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Platform/SRE side of BLO-22255 (deploy Blockcast-profile producers + convergence receipts).
moq-native-ietfalready recordsmoq_negotiation_totalon every connect attempt (added alongside theBlockcast01profile in #65/#66), but no producer binary ever installed a Prometheus recorder to serve it — the metric existed but was unobservable. This PR closes that gap without touching negotiation/enforcement logic:--metrics-addrbehind an opt-inmetrics-prometheuscargo feature, mirroringmoq-relay-ietf's existing exporter wiring exactly (samePrometheusBuilderpattern, same feature-off warning path). With the flag unset or the feature disabled at build time, behavior and the shipped binary are unchanged — this only adds an optional listener.moqt-blockcast-01, issues aSUBSCRIBE, and decodesSUBSCRIBE_OKkey0x40viaTrackReader::history_window()on a timed loop. Each cycle opens a fresh connection and reconnects, so a long-running instance is evidence of repeated negotiation, not one lucky session. Exposesmoq_canary_probe_total{outcome=...}/moq_canary_history_window_groupsbehind the same opt-in exporter pattern (--metrics-addr+metrics-prometheusfeature). Also supports--oncefor CI/manual smoke checks.Both together give BLO-22255's verifying signal a path to exist:
moq_negotiation_total{outcome="selected",required="moqt-blockcast-01"}from the publisher side, and a decoded-0x40-was-nonzero receipt from the canary side.Not in scope here (tracked separately): the submodule-pointer bump (BLO-23504), the actual
--metrics-addrdeployment/scrape config, and any production re-pin — those are deliberate, separately-approved steps per the rollout record.Test plan
cargo generate-lockfile-free minimal resolution checked locally (nocc/linker available in this sandbox, so fullcargo build/testcould not run here — relying on CI)cargo fmt --edition 2021 --checkclean on all new/changed filescargo test --verbose(covers newmoq-canaryunit tests + existingmoq-pub-mmtpCLI tests)cargo clippy --no-depscargo machete(verifies no unused deps —metrics-exporter-prometheusis cfg-gated in both crates, matching the existingmoq-relay-ietfprecedent)