Skip to content

feat(otel): add exemplar support for metrics-to-traces linking#23

Open
jilio wants to merge 1 commit into
mainfrom
feature/otel-exemplars
Open

feat(otel): add exemplar support for metrics-to-traces linking#23
jilio wants to merge 1 commit into
mainfrom
feature/otel-exemplars

Conversation

@jilio
Copy link
Copy Markdown
Owner

@jilio jilio commented Nov 8, 2025

Summary

Enabled OpenTelemetry exemplars on histogram metrics to create clickable links from metrics to distributed traces in Grafana.

Changes

Configuration

  • Configured Grafana Prometheus datasource with exemplar trace destinations
  • Added Jaeger as a datasource in Grafana for trace visualization
  • Linked exemplars to use Jaeger for trace lookup

Dashboard Enhancements

  • Enabled exemplar display on p95 Handler Duration by Event Type panel
  • Enabled exemplar display on p95 Async Handler Duration by Event Type panel
  • Enabled exemplar display on p95 Persist Duration by Event Type panel

Documentation

  • Added comments explaining exemplar support in histogram recording code

How It Works

OpenTelemetry automatically attaches trace context (trace ID and span ID) to histogram metric samples when recording occurs within an active span. In Grafana, these exemplars appear as clickable points on the metric graphs that link directly to the corresponding trace in Jaeger.

Benefits

  • Direct correlation between metrics and traces
  • Click-through debugging from high-latency metrics to detailed traces
  • Root cause analysis made easier by linking performance anomalies to specific trace executions

Testing

  • All existing tests pass with 100% coverage maintained
  • No code changes to the core observability logic required
  • Exemplars work automatically with existing span instrumentation

Screenshots

Once deployed, users will see:

  1. Small dots/markers on duration graphs representing exemplar samples
  2. Clicking these markers opens the corresponding trace in Jaeger
  3. Easy navigation from "this event was slow" to "here's exactly why"

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Enabled OpenTelemetry exemplars on histogram metrics to create clickable
links from metrics to distributed traces in Grafana.

Changes:
- Added documentation to histogram recording explaining exemplar support
- Configured Grafana Prometheus datasource with exemplar trace destinations
- Added Jaeger as a datasource in Grafana for trace visualization
- Enabled exemplar display on all p95 duration panels in dashboard
  - p95 Handler Duration by Event Type
  - p95 Async Handler Duration by Event Type
  - p95 Persist Duration by Event Type

Exemplars automatically attach trace context (trace ID and span ID) to
histogram samples when there's an active span in the context. This allows
users to:
1. See high-latency samples directly on metric graphs
2. Click on exemplar points to jump to the corresponding trace in Jaeger
3. Debug performance issues by correlating metrics with detailed traces

All existing tests pass with 100% coverage maintained.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant