Skip to content

Add native OTLP export marker to trace chunks - #6390

Draft
mhlidd wants to merge 1 commit into
masterfrom
matthew.li/otlp-adoption-markers
Draft

mhlidd wants to merge 1 commit into
masterfrom
matthew.li/otlp-adoption-markers

Conversation

@mhlidd

@mhlidd mhlidd commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Tags the first span of each trace chunk with _dd.sdk.otlp_export: "false":

  • The new constant Tracing::Metadata::Ext::TAG_SDK_OTLP_EXPORT holds the key.
  • TraceFormatter#tag_sdk_otlp_export! sets it, inside the existing if first_span block, between _dd.tags.process and the git tags.

The marker follows the same first-span rules as process tags. Both the HTTP transport and the native (libdatadog) transport go through TraceFormatter.format!, so both carry it. The native transport's C extension (meta_iter_cb in ext/libdatadog_api/trace_exporter.c) copies every string meta entry with no allowlist, so no C change is needed.

Motivation:

This is part of the "Measuring Adoption of Datadog SDKs in OTel Mode (Traces)" RFC.

Today the backend infers which spans came from Datadog SDKs exporting OTLP at ingest time, using telemetry.sdk.name and span_source. That's unreliable: the OpenTelemetry Collector's datadogreceiver hardcodes telemetry.sdk.name:Datadog on native payloads it converts. So tracers now declare their export mode on the wire. "true" goes on the OTLP resource when exporting over OTLP, and "false" goes on the first span of each chunk for native export. An absent marker means an older tracer.

Ruby has no OTLP trace export today (the startup log reports otlp_traces_export_enabled: false), so the value is always "false", and the OTLP-only datadog.sdk.semantics resource attribute doesn't apply.

Related changes:

Change log entry

None. This is an internal _dd.* tag with no customer-visible behavior change.

Additional Notes:

  • AI-generated: this change was written with AI assistance and reviewed before submission.
  • No conditions: the tag is unconditional and doesn't guard against a user tag with the same key, matching the other tracers.
  • Chunks without a root span: the marker is still set. find_root_span falls back to the last span, and only a chunk with no spans returns early.

How to test the change?

  • New specs in spec/datadog/tracing/transport/trace_formatter_spec.rb cover:
    • the marker on the first span in the no-root-span, missing-root-span and root-span cases
    • multiple chunks
    • absence on non-first spans
  • Local results (Ruby 3.2.2):
    • bundle exec rspec spec/datadog/tracing/transport/: 343 examples, 0 failures, 6 pending (the pending ones need TEST_DATADOG_INTEGRATION=1)
    • native transport conformance and transport specs: 65 examples, 0 failures
    • tracer, writer and workers specs: 286 examples, 0 failures
  • Linters and types, run in the tracer-4.0 image: rubocop reports no offenses, and steep check on the changed lib files reports no errors.

🤖 Generated with Claude Code

Tag the first span of each chunk with _dd.sdk.otlp_export=false, next to
_dd.tags.process, so the backend can tell native-exported spans from spans
that reached it through the datadogreceiver instead of inferring it at
ingest time. Ruby has no OTLP trace export, so the value is always false.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd mhlidd added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Sep 25, 2026
@dd-octo-sts dd-octo-sts Bot added the tracing label Sep 25, 2026
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Tests

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 90.38% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 84fbf2e | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Sep 25, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-09-25 19:29:44

Comparing candidate commit 84fbf2e in PR branch matthew.li/otlp-adoption-markers with baseline commit de0abfe in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 1 performance regressions! Performance is the same for 51 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:profiling - Allocations ()

  • 🟥 throughput [-225909.277op/s; -183120.380op/s] or [-6.989%; -5.665%]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant