Skip to content

feat(examples): metrics and StreamMetricExporter in the OTel example - #90

Merged
nhobin219 merged 2 commits into
mainfrom
otel-metrics
Oct 3, 2026
Merged

nhobin219 merged 2 commits into
mainfrom
otel-metrics

Conversation

@nhobin219

Copy link
Copy Markdown
Owner

Adds metrics to the OTel example: an exporter that publishes OTel metrics to a stream, metrics from the two services, re-export to OTLP, and the dashboard's Metrics tab.

What

  • examples/otel/metrics.py:
    • StreamMetricExporter, an OTel MetricExporter that publishes each collection to a stream, one row per data point.
    • Every kind the Python SDK produces: sums, gauges, explicit-bucket and exponential histograms.
    • Numbers keep OTLP's int/double split (value_int / value_double), so an int64 counter never rounds through a double.
    • Exemplars are kept, as trace and span ids, so a metric leads to the trace it measured.
    • Delta temporality for counters and histograms, the SDK's delta preference; up-down counters stay cumulative. Each row is then what happened in its own interval, so a window's total is a sum(). otel-gui charts sum rates correctly for either temporality.
    • Non-finite doubles are stored as null, as in logs.
  • services.py: checkout counts shop.orders by outcome. Both services record http.server.request.duration (OTel's semantic convention) inside their server spans, so exemplars link to traces. Exported every second.
  • export.py: regroups metric rows every half second into MetricsData (resource → scope → metric), handed to OTel's OTLPMetricExporter. Metrics have no batch processor to hand a point to.
  • demo.py: a third stream. The one-shot run reports orders by outcome, mean request duration per service, and the failed order's exemplar, which is the failed request's trace id.
  • gui.py: otel-gui 2.1.0 → 3.0.0, the first release that accepts metrics (2.1.0 answers /v1/metrics with 501).
    • Same asset names. The SHA-256 is still checked against the published one.
    • warm() now also warms metrics. The lazy-load crash is unchanged in 3.0.0: with all three signals' first requests at once, a fresh otel-gui died 5 times in 5. Warming in turn: 0 of 5. 3.0.0 loads the metrics decoder into the same shared protobuf root (initProtobufMetrics).
  • Docs: examples/README.md and the CHANGELOG.

Second commit: the one-shot demo runs without a maintainer

This matches the migration demo. Profiling showed the one-shot demo's time going to its five maintainer processes:

  • They did no maintenance. They were still opening the logs when the run ended, then were SIGTERMed mid-recover(), printing tracebacks.
  • demo.main took 3–11 s with them, about 1 s without. The OTel test file went from 14.5 s to 4 s; each demo test is now under 1 s.
  • just demo otel's broker keeps them.

Kept as a separate commit so it can be reviewed or dropped on its own.

Tests

  • Exact round trip: every metric kind, from a real SDK reader → rows → metrics_data, encodes to identical OTLP with OTel's own encoder, exemplars included.
  • Row shape, edge cases and the demo:
    • Row shape per kind.
    • A non-finite double is stored as null.
    • The temporality preference.
    • The demo's totals, asserted as totals, since row counts depend on how many one-second collections a run spans.
    • The exemplar → failed trace link.
  • The OTLP receiver test now decodes /v1/metrics too.
  • Falsified: six breaks, each caught by at least one test.
    • Exemplars dropped; bucket counts reversed; counters cumulative; the order counted outside its span; ints stored as doubles; non-finite values kept.
    • The bucket check first passed under reversal because its counts were a palindrome, [0, 1, 0]; the test data is now asymmetric.
  • End to end: just demo otel ran with otel-gui 3.0.0, and otel-gui's own API listed shop.orders and http.server.request.duration for both services. No errors in any process or in otel-gui's log.

Found, not fixed here

A maintainer that gets SIGTERM while still opening its logs raises KeyboardInterrupt inside litelink's recover(), mid-SQLite. That's the cannot rollback - no transaction is active message. It's pre-existing, and separate from this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FsSDkeb5rVAxA1FSmKKfQi

nhobin219 and others added 2 commits October 3, 2026 21:46
examples/otel/metrics.py publishes OTel metric collections to a
stream, one row per data point. It covers sums, gauges and both
histogram kinds, keeps exemplars, and asks for delta temporality for
counters and histograms so a window's total is a sum().

The services count orders by outcome and record
http.server.request.duration inside their spans. export.py regroups
metric rows into MetricsData for OTel's OTLP metric exporter.
just demo otel moves to otel-gui 3.0.0, the first release to accept
metrics, and warms its metrics decoder too: the lazy-load race that
crashes it is unchanged in 3.0.0 (5 of 5 fresh instances died with all
three signals at once; none after warming).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FsSDkeb5rVAxA1FSmKKfQi
As the migration demo does: a run this short never fills a log enough
to seal it. Its five maintainer processes were still opening the logs
when the run ended, so they did nothing but cost time and print
tracebacks on SIGTERM. demo.main took 3-11 s with them and about 1 s
without, and the OTel test file went from 14.5 s to 4 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FsSDkeb5rVAxA1FSmKKfQi
@nhobin219
nhobin219 merged commit cee2aff into main Oct 3, 2026
8 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant