Skip to content

feat(tracing): add OTLP adoption markers to trace payloads - #10482

Open
mhlidd wants to merge 5 commits into
masterfrom
matthew.li/otlp-adoption-markers
Open

mhlidd wants to merge 5 commits into
masterfrom
matthew.li/otlp-adoption-markers

Conversation

@mhlidd

@mhlidd mhlidd commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds two wire markers that let the backend attribute ingested spans to the SDK's export path and semantic conventions:

  • _dd.sdk.otlp_export:
    • Native export: "false" goes into the meta of the first span of each chunk. It's written in extractChunkTags during span formatting, next to _dd.tags.process, so the v0.4/v0.5 encoders are unchanged. The SpanProcessor skips the marker when OTEL_TRACES_EXPORTER=otlp.
    • OTLP export: "true" is added as a resource attribute in buildResourceAttributes.
  • datadog.sdk.semantics, an OTLP resource attribute: "otel" when DD_TRACE_OTEL_SEMANTICS_ENABLED is on, "datadog" otherwise.

Both resource attributes are set after the global tags are merged, so a global tag can't override them. Matching dd-trace-java, span tags with either key are not stripped from OTLP span attributes.

Motivation

Today, "how many customers run Datadog SDKs emitting OTLP" is answered by dd.trace.intake.ingested_spans, tagged with sdk_name/span_source. Those tags are inferred at ingest time rather than declared by the tracer. The datadogreceiver hardcodes telemetry.sdk.name:Datadog on native payloads it converts, so a native payload that only passed through a Collector looks the same as one a Datadog SDK exported over OTLP. And span_source records which endpoint the payload reached, not the tracer's export mode.

Declaring both modes on the wire lets the backend resolve them into explicit tags and measure adoption directly. Related changes: DataDog/dd-trace-java#12516, DataDog/dd-trace-dotnet#9310 and DataDog/dd-trace-go#5436.

Additional Notes

  • Scope: the native marker follows dd-trace-js's existing process-tags convention and the RFC's wording ("the first span in the chunk"). That means one span per chunk. dd-trace-java writes it once per payload in its mappers. dd-trace-go also uses per-chunk scope for v0.4.
  • Which exporters get it: "false" means native Datadog export, so the only check is OTEL_TRACES_EXPORTER !== 'otlp'. Test Optimization and Electron override otlp and use their own exporters. Test Optimization sends to the citestcycle intake, whose spans are excluded from dd.trace.intake.ingested_spans, and Electron sends to the browser intake. So with otlp set they simply get no marker, and intake counts them as unknown export mode; they never get a wrong value.
  • Earlier approach: a previous revision wrote the marker in the v0.4/v0.5 encoders once per payload, which needed a per-payload flag and a copy of the first span's meta. It was replaced by this simpler approach; the encoders and their tests are back to master.
  • No v1 format: dd-trace-js only has v0.4 and v0.5 encoders, so no payload-level attribute variant is needed.
  • Coordinated change: this PR emits the markers. The Agent hoists _dd.sdk.otlp_export into the outgoing payload, logs-backend propagates it into chunk tags, and dd-go resolves it into the datadog.sdk.otlp_export metric tag. The wire marker and the metric tag are named differently on purpose.
  • Tests:
    • span_format.spec.js: the marker is on the first span of each chunk only.
    • span_processor.spec.js: the marker is requested for the first span of each chunk, including after a partial flush, and not for OTLP export.
    • opentelemetry/traces.spec.js: the resource markers, the semantics switch, and that global tags can't override them.
    • Results: the specs above, the encoder specs and the exporter specs pass. eslint --max-warnings 0 on changed files passes. An earlier revision ran the full npm run test:trace:core suite, where the only failure was process-tags.spec.js:104, which fails locally only because the worktree directory isn't named dd-trace-js; the full suite hasn't been re-run on the final gating change.

🤖 Generated with Claude Code

Declare the export mode and semantics mode on the wire so the backend can
measure OTLP adoption instead of inferring it at ingest time. Native v0.4/v0.5
payloads carry _dd.sdk.otlp_export=false once per payload; OTLP exports carry
_dd.sdk.otlp_export=true and datadog.sdk.semantics as resource attributes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd mhlidd added semver-minor AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos labels Sep 24, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 9.21 MB
Deduped: 9.95 MB
No deduping: 9.95 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 513.61 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.77%. Comparing base (13d864b) to head (e7eb884).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10482   +/-   ##
=======================================
  Coverage   98.77%   98.77%           
=======================================
  Files        1047     1047           
  Lines      166704   166755   +51     
  Branches       74       74           
=======================================
+ Hits       164662   164713   +51     
  Misses       2042     2042           
Flag Coverage Δ
ai-guard 63.49% <100.00%> (-0.02%) ⬇️
apm-capabilities 63.18% <100.00%> (+<0.01%) ⬆️
apm-integrations 81.17% <100.00%> (-0.01%) ⬇️
appsec 57.98% <100.00%> (-0.02%) ⬇️
debugger 69.92% <100.00%> (-0.01%) ⬇️
instrumentation 54.79% <100.00%> (-0.02%) ⬇️
llmobs 80.11% <100.00%> (-0.05%) ⬇️
master-coverage 98.77% <100.00%> (?)
openfeature 66.04% <100.00%> (-0.02%) ⬇️
platform 68.76% <100.00%> (+<0.01%) ⬆️
profiling 66.38% <100.00%> (-0.02%) ⬇️
serverless 65.12% <100.00%> (-0.02%) ⬇️
test-optimization 81.97% <100.00%> (-0.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 2 tests - 2 passed on retry View in Datadog

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.22% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e7eb884 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-09-24 20:39:29

Comparing candidate commit e7eb884 in PR branch matthew.li/otlp-adoption-markers with baseline commit 13d864b in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2303 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-with-attacks-20

  • unstable max_rss_usage [-13.810MB; +4.229MB] or [-8.076%; +2.473%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable max_rss_usage [-16.374MB; +27.097MB] or [-7.029%; +11.632%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-24

  • unstable max_rss_usage [-30.118MB; +7.819MB] or [-9.108%; +2.365%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable max_rss_usage [-11.216MB; +6.859MB] or [-6.855%; +4.192%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-26

  • unstable execution_time [-347.006ms; +484.227ms] or [-4.843%; +6.759%]
  • unstable throughput [-69367.443op/s; +50602.728op/s] or [-6.188%; +4.514%]

scenario:plugin-graphql-long-with-depth-on-max-26

  • unstable cpu_user_time [-633.335ms; +373.168ms] or [-7.892%; +4.650%]
  • unstable execution_time [-648.482ms; +380.652ms] or [-7.928%; +4.654%]
  • unstable max_rss_usage [-37.143MB; +22.933MB] or [-14.485%; +8.943%]
  • unstable throughput [-11.119op/s; +18.744op/s] or [-4.946%; +8.337%]

mhlidd and others added 3 commits September 24, 2026 15:41
Move _dd.sdk.otlp_export=false out of the v0.4/v0.5 encoders and into span
formatting, next to _dd.tags.process, so it is written on the first span of
each chunk without copying meta or tracking payload state. It is skipped
when traces are exported over OTLP, which the tracer already decides when it
picks the exporter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ilter

Compute whether traces go out over OTLP in SpanProcessor from config, next
to process tags, instead of widening its constructor. Stop stripping the
adoption marker keys from OTLP span attributes, matching dd-trace-java; the
resource values are still set after global tags so they cannot be overridden.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Test Optimization and Electron overrides send spans to intakes that do
not feed the ingestion metric, so the extra conditions add nothing that
intake uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Preserve the native marker against user-tag overwrites and make the semantics test environment-independent.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Adds OTLP export and semantic-convention markers to native and OTLP trace payloads.

Changes:

  • Adds native markers to the first span of each chunk.
  • Adds OTLP resource attributes for export mode and semantics.
  • Adds processor, formatting, and resource-marker tests.
File Description
packages/​dd-trace/​test/​span_processor.spec.js Tests marker selection and chunk behavior.
packages/​dd-trace/​test/​span_format.spec.js Tests native marker placement.
packages/​dd-trace/​test/​opentelemetry/​traces.spec.js Tests OTLP resource markers and precedence.
packages/​dd-trace/​src/​span_processor.js Selects marker behavior by exporter.
packages/​dd-trace/​src/​span_format.js Emits native export markers.
packages/​dd-trace/​src/​opentelemetry/​trace/​index.js Adds OTLP resource attributes.
packages/​dd-trace/​src/​constants.js Defines marker keys.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/dd-trace/src/span_format.js
Comment thread packages/dd-trace/test/opentelemetry/traces.spec.js
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd
mhlidd marked this pull request as ready for review September 24, 2026 20:28
@mhlidd
mhlidd requested review from a team as code owners September 24, 2026 20:28
@mhlidd
mhlidd requested review from BridgeAR and removed request for a team September 24, 2026 20:28

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7eb884d11

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/span_format.js

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos semver-minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants