stats: per-track counters exposed at /metrics/track

[afrind]

Per-(track, iothread) TrackStats live in the stats tier, owned by a TrackStatsRegistry that binds one TrackStatsCollector per data-plane thread and merges them on demand — the same split as StatsRegistry/MoQStatsCollector. A TrackStatsFilter counts on the relay ingest chain and on each downstream subscriber, including the PUBLISH fan-out path. No read path dereferences a registry forwarder, so counting behaves identically in single-thread, relay-exec and local-forwarder modes.

GET /metrics/track?service=&namespace=&track=&limit= reports subscribers, groups/subgroups/objects/datagrams/bytes in both directions, and publish-start/last-object timestamps. Rates are left to Prometheus. A match wider than the limit is rejected rather than truncated: an arbitrary F14-ordered subset would reshuffle between scrapes.

Namespaces and track names are safeName()-encoded in labels and expected in the same form on the query, so a scraped value pastes back into a query and two tracks cannot collapse onto one label set.

Groups are counted against an LRU window of the 3 most recent group IDs, so subgroups of concurrently-open groups do not inflate the count.

---

This change is 

[gmarzot]

@gmarzot reviewed 29 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on afrind).

[gmarzot]

claude is happy w/ it it at least and it will populate real data in a dashboard now, but without a way to iterate all (the wrong model anyway) the production deploy utility is limited. I am thinking about a more complete scalable push model ... M4 perhaps

[afrind]

@afrind made 23 comments.
Reviewable status: 18 of 29 files reviewed, 2 unresolved discussions (waiting on gmarzot).

---

docs/config.md line 436 at r2 (raw file):

  #   alpn: [h2, "http/1.1"]
  # track_metrics_enabled: true    # per-track counting and /metrics/track
  # track_metrics_limit: 10        # default tracks per /metrics/track scrape

track_metrics_endpoint_default_limit

---

docs/config.md line 437 at r2 (raw file):

  # track_metrics_enabled: true    # per-track counting and /metrics/track
  # track_metrics_limit: 10        # default tracks per /metrics/track scrape
  # track_metrics_max_limit: 1000  # ceiling on the ?limit= parameter

track_metrics_endpoint_max_limit

---

docs/metrics.md line 222 at r2 (raw file):

`conf.2eexample.2ecom-room.201`.

Because the encoding is injective, two tracks can never collapse onto one label

Remove the "Because the encoding is injective, " bit

---

docs/metrics.md line 246 at r2 (raw file):

concurrently-open groups can arrive interleaved without inflating the count. A
group revisited after 3 *other* groups have been seen is counted again — the
window bounds per-filter state, and MoQ group IDs advance, so this only shows up

and MOQT group IDs usually advance and high concurrency is not expected,

---

docs/metrics.md line 249 at r2 (raw file):

in pathological interleaving.

Byte counters measure **object payload bytes**, not wire bytes: no MOQT headers,

I'll file a follow up issue to report header bytes.

---

docs/metrics.md line 254 at r2 (raw file):

Sent counters are summed across subscribers, so one ingested object fanned out to
three subscribers increments `objects_sent` by 3.

Should note that "sent" here means "passed to transport" and e.g. and objects that are not sent due to stop_sending, delivery timeout, etc aren't reflected in the counts.

---

docs/metrics.md line 258 at r2 (raw file):

### Bitrate and object rate

Rates are not computed server-side — derive them from the counters:

Do we need to say that we don't support rates? This section could be removed.

---

src/MoqxRelay.cpp line 2616 at r2 (raw file):

    const {
  TrackMatch match;
  // forEachName, not forEach: EntryView copies two shared_ptrs per entry, which

This comment is not helpful. remove

---

src/MoqxRelayContext.cpp line 177 at r2 (raw file):

}

void MoqxRelayContext::initThreadStats(folly::IOThreadPoolExecutor& ioExecutor, bool trackStats) {

call this initTrackStats

---

src/MoqxRelayContext.cpp line 328 at r2 (raw file):

  // Each surviving service costs one executor hop per thread below.
  std::erase_if(services, [](const Service& s) { return s.match.keys.empty(); });

Do this as an if in the loop below, omitting an entry in countTasks?

---

src/MoqxRelayContext.cpp line 338 at r2 (raw file):

  auto counters = co_await folly::coro::collectAllRange(std::move(countTasks));
  for (size_t i = 0; i < services.size(); ++i) {
    for (const auto& ftn : services[i].match.keys) {

Why are we scanning keys here instead of just iterating counters and pushing back service it->first, it->second?

---

src/SubscriptionRegistry.h line 182 at r2 (raw file):

  void forEach(folly::FunctionRef<void(const EntryView&)> fn) const;

  // Cheaper than forEach for callers that only need names: EntryView copies a

This comment is more or less obvious, remove?

---

src/admin/TrackMetricsHandler.h line 19 at r2 (raw file):

class AdminServer;

struct TrackMetricsLimits {

this is sort of the same as the conig struct? Is it a normal pattern to include it here?

---

src/admin/TrackMetricsHandler.cpp line 31 at r2 (raw file):

namespace {

// Service names come from config, so they only need Prometheus quoting; track

Do other handlers do this and if so should there be a metrics utility header for it?

---

src/admin/TrackMetricsHandler.cpp line 68 at r2 (raw file):

};

std::unique_ptr<folly::IOBuf> formatPrometheus(const MoqxRelayContext::TrackMetricsResult& result) {

Same comment here about the helper?

---

src/admin/TrackMetricsHandler.cpp line 193 at r2 (raw file):

}

void sendError(proxygen::ResponseHandler* downstream, int status, const std::string& message) {

This almost certainly belongs in a helper

---

src/admin/TrackMetricsHandler.cpp line 237 at r2 (raw file):

        auto nsPrefix = parseSafeNamespace(nsParam);
        if (!nsPrefix) {
          sendError(downstream, 400, "namespace is not in the MoQT safe name form\n");

MOQT not MoQT

---

src/config/ConfigResolver.cpp line 1204 at r2 (raw file):

    }
    if (auto limit = adminOptional->track_metrics_limit.value()) {
      resolved.trackMetricsLimit = *limit;

Need a verifier that default is < max

---

src/relay/TrackStatsFilter.cpp line 106 at r2 (raw file):

TrackStatsFilter::~TrackStatsFilter() {
  bool egress = direction_ == Direction::Egress;
  if (stats_->onOwnerThread()) {

What are the scenarios where this is destroyed in the wrong thread? Can we rewrite with a lambda and if (onOnwer) fn else exec->add(fn)

---

src/stats/TrackStats.h line 33 at r2 (raw file):

// so every field must have a meaningful merge.
struct TrackCounters {
  uint64_t groupsReceived{0};

Should we make a single struct of these counter values and hold in received and sent? I think that could alleviate some conditionals we have above?

---

src/stats/TrackStats.h line 178 at r2 (raw file):

  friend class TrackStats;

  struct Slot {

Slot seems like an undescriptive name, pick a better one

---

src/stats/TrackStats.h line 204 at r2 (raw file):

};

inline folly::Executor* TrackStats::owningExec() const {

If there was a way to guarantee destruction on the right thread, I find it preferable to tracking this.

---

src/stats/TrackStatsRegistry.h line 36 at r2 (raw file):

  // For callers already running on the executor's thread, where dispatching
  // and waiting would deadlock.
  void bindHere(folly::Executor* exec);

This seems to only be used in tests -- is there a way to use bindAll instead and remove this API, merge addCollector into bindAll and remove the dup check?

[afrind]

@afrind made 16 comments and resolved 2 discussions.
Reviewable status: 4 of 32 files reviewed, 1 unresolved discussion (waiting on gmarzot).

---

docs/config.md line 436 at r2 (raw file):

Previously, afrind wrote…

track_metrics_endpoint_default_limit

renamed

---

docs/config.md line 437 at r2 (raw file):

Previously, afrind wrote…

track_metrics_endpoint_max_limit

renamed

---

docs/metrics.md line 222 at r2 (raw file):

Previously, afrind wrote…

Remove the "Because the encoding is injective, " bit

removed

---

src/MoqxRelayContext.cpp line 177 at r2 (raw file):

Previously, afrind wrote…

call this initTrackStats

Oops, I meant the boolean param should be "initTrackStats" - the fn should still be initThreadStatsCollectors

---

src/MoqxRelayContext.cpp line 338 at r2 (raw file):

Previously, afrind wrote…

Why are we scanning keys here instead of just iterating counters and pushing back service it->first, it->second?

removed key scan

---

src/admin/TrackMetricsHandler.h line 19 at r2 (raw file):

Previously, afrind wrote…

this is sort of the same as the conig struct? Is it a normal pattern to include it here?

decided to keep the struct

---

src/admin/TrackMetricsHandler.cpp line 31 at r2 (raw file):

Previously, afrind wrote…

Do other handlers do this and if so should there be a metrics utility header for it?

Moved to utilties

---

src/admin/TrackMetricsHandler.cpp line 68 at r2 (raw file):

Previously, afrind wrote…

Same comment here about the helper?

moved to helper

---

src/admin/TrackMetricsHandler.cpp line 193 at r2 (raw file):

Previously, afrind wrote…

This almost certainly belongs in a helper

Added PrometheusWriter

---

src/admin/TrackMetricsHandler.cpp line 237 at r2 (raw file):

Previously, afrind wrote…

MOQT not MoQT

Fixed - MOQT

---

src/config/ConfigResolver.cpp line 1204 at r2 (raw file):

Previously, afrind wrote…

Need a verifier that default is < max

Added check

---

src/relay/TrackStatsFilter.cpp line 106 at r2 (raw file):

Previously, afrind wrote…

What are the scenarios where this is destroyed in the wrong thread? Can we rewrite with a lambda and if (onOnwer) fn else exec->add(fn)

Used a lambda, but decided not to chase the cross-threadness

---

src/stats/TrackStats.h line 33 at r2 (raw file):

Previously, afrind wrote…

Should we make a single struct of these counter values and hold in received and sent? I think that could alleviate some conditionals we have above?

Added DirectionCounters

---

src/stats/TrackStats.h line 178 at r2 (raw file):

Previously, afrind wrote…

Slot seems like an undescriptive name, pick a better one

Slot -> TrackRef

---

src/stats/TrackStats.h line 204 at r2 (raw file):

Previously, afrind wrote…

If there was a way to guarantee destruction on the right thread, I find it preferable to tracking this.

Punted for now

---

src/stats/TrackStatsRegistry.h line 36 at r2 (raw file):

Previously, afrind wrote…

This seems to only be used in tests -- is there a way to use bindAll instead and remove this API, merge addCollector into bindAll and remove the dup check?

removed "bindHere" by scoping the tests

[afrind]

@afrind made 1 comment and resolved 1 discussion.
Reviewable status: 4 of 32 files reviewed, all discussions resolved (waiting on gmarzot).

---

src/MoqxRelayContext.cpp line 177 at r2 (raw file):

Previously, afrind wrote…

Oops, I meant the boolean param should be "initTrackStats" - the fn should still be initThreadStatsCollectors

fixed

[michalhosna]

@michalhosna made 3 comments.
Reviewable status: 4 of 32 files reviewed, 2 unresolved discussions (waiting on afrind and gmarzot).

---

docs/config.md line 451 at r3 (raw file):

guard rail, not a selection rule: a query matching more tracks than the limit is
rejected rather than truncated, because an arbitrary subset would give
Prometheus a series set that reshuffles between scrapes.

Nit: I think this documentation is relatively confusing without knowing the query structure. Maybe point to the full docs?

---

src/admin/TrackMetricsHandler.cpp line 215 at r3 (raw file):

            return;
          }
          limit = std::min(*parsed, limits.maxLimit);

If maxLimit ==0, this gets set to 0 i.e. always returns 400? Is that correct?
Why is there the std::max<size_t>(1, above? What's the reason of 1 floor there?

Should maxLimit = 0 have special meaning? Like unlimited?

---

src/config/Config.h line 230 at r3 (raw file):

  folly::SocketAddress address;
  std::optional<TlsConfig> tls;
  bool trackMetricsEnabled{true};

Why true by default? Isn't the collection somewhat expensive?

If you do not know about this feature, it seems much more dangerous to leave it enabled than useful to leave it enabled. If you do know about this feature, you don't care about default.

Or, as I like explicit over implicit, require decision in config file and have no default, but that's probably too harsh here

[afrind]

@afrind made 3 comments.
Reviewable status: 4 of 32 files reviewed, 2 unresolved discussions (waiting on gmarzot and michalhosna).

---

docs/config.md line 451 at r3 (raw file):

Previously, michalhosna (Michal Hošna) wrote…

Nit: I think this documentation is relatively confusing without knowing the query structure. Maybe point to the full docs?

ok

---

src/admin/TrackMetricsHandler.cpp line 215 at r3 (raw file):

Previously, michalhosna (Michal Hošna) wrote…

If maxLimit ==0, this gets set to 0 i.e. always returns 400? Is that correct?
Why is there the std::max<size_t>(1, above? What's the reason of 1 floor there?

Should maxLimit = 0 have special meaning? Like unlimited?

I don't think we want to allow unlimited. I'll add a config check that maxLimit > 0 when enabled=True

---

src/config/Config.h line 230 at r3 (raw file):

Previously, michalhosna (Michal Hošna) wrote…

Why true by default? Isn't the collection somewhat expensive?

If you do not know about this feature, it seems much more dangerous to leave it enabled than useful to leave it enabled. If you do know about this feature, you don't care about default.

Or, as I like explicit over implicit, require decision in config file and have no default, but that's probably too harsh here

Sure we can default to false.

[afrind]

@afrind made 1 comment and resolved 1 discussion.
Reviewable status: 3 of 33 files reviewed, 1 unresolved discussion (waiting on gmarzot and michalhosna).

---

src/admin/TrackMetricsHandler.cpp line 215 at r3 (raw file):

Previously, afrind wrote…

I don't think we want to allow unlimited. I'll add a config check that maxLimit > 0 when enabled=True

I removed the floor of 1, and 400 anyone who requests > max.

[michalhosna]

@michalhosna resolved 1 discussion.
Reviewable status: 3 of 33 files reviewed, all discussions resolved (waiting on gmarzot).