Skip to content

Introducing Tags/OTLP Resource Attributes for OTLP Adoption Metrics - #12516

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
masterfrom
mhlidd/otlp_adoption_metrics
Sep 24, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
masterfrom
mhlidd/otlp_adoption_metrics

Conversation

@mhlidd

@mhlidd mhlidd commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds two markers that let the backend attribute ingested spans to the SDK's export path and semantics:

  • _dd.sdk.otlp_export payload marker (TraceMapper.SDK_OTLP_EXPORT) — written once per payload rather than on every span, since the value is uniform for the whole payload:
    • TraceMapperV0_4 / TraceMapperV0_5: written into the meta of the first span of the first non-empty chunk, gated on firstSpanInPayload (the same way _dd.tags.process is handled).
    • TraceMapperV1: written as a payload-level attribute (header field 10) alongside _dd.tags.process.
    • The value is "true"/"false" from Config.isOtlpTracesExportEnabled() (true when writerType is OtlpWriter or a MultiWriter containing it), snapshotted once per mapper via TraceMapper.otlpExportMarker(Config).
    • OTLP export path: OtlpResourceAttributes.traceResourceAttributes always emits _dd.sdk.otlp_export=true as a resource attribute, since reaching that encoder means the payload is leaving over OTLP.
  • datadog.sdk.semantics OTLP resource attribute — contributed by OtlpResourceAttributes.traceResourceAttributes on the OTLP export path, "otel" when Config.isTraceOtelSemanticsEnabled() is set and "datadog" otherwise.

Both keys are added to IGNORED_GLOBAL_TAGS in OtlpResourceAttributes so a global tag of the same name is not echoed into the resource.

Tests:

  • TraceMapperV04PayloadTest / TraceMapperV05PayloadTest: marker appears only on the first span of the first non-empty chunk, and is "true" when also exporting over OTLP.
  • TraceMapperV1PayloadTest: marker present in payload attributes, "true" when also exporting over OTLP (V1PayloadReader extended to read it).
  • OtlpResourceJsonTest / OtlpResourceProtoTest: trace resource attributes carry the OTLP export marker, and datadog.sdk.semantics switches to "otel" when enabled.
  • DDAgentApiTest, DDAgentWriterCombinedTest, DDSpanSerializationTest updated for the extra meta entry on the first span.

Motivation

We have no direct signal for how many spans reach the intake via the SDK's OTLP export path versus the native Datadog path, nor which semantic convention the SDK was configured with. Carrying both as markers on the wire lets the backend resolve them into tags on dd.trace.intake.ingested_spans (surfaced as datadog.sdk.otlp_export) and measure OTLP export adoption for traces directly, instead of inferring it.

Additional Notes

  • This is one of four coordinated changes. This PR emits the marker; datadog-agent hoists it into the Agent's outgoing payload (from the first span's meta into TracerPayload.Tags for v0.4/v0.5, and from the payload Attributes for V1, which is the default from Agent 7.84); logs-backend propagates it into chunk tags; dd-go resolves it into the datadog.sdk.otlp_export tag on dd.trace.intake.ingested_spans.
  • The metric tag datadog.sdk.otlp_export and the wire marker _dd.sdk.otlp_export are deliberately named differently — same concept, two layers. They should not be conflated when reading dashboards or backend code.
  • An earlier iteration stamped the marker on every span via CoreTracer.withTracerTags. That was dropped in favor of payload scope to avoid per-span wire overhead; the marker is no longer part of the tracer-defined span tag bundle.
  • Because the marker only exists on the wire and is never stored as a span tag, its key is deliberately not declared in the public DDTags API (dd-trace-api). It lives in dd-trace-core instead: TraceMapper.SDK_OTLP_EXPORT for the Agent mappers, and a private SDK_OTLP_EXPORT_KEY in OtlpResourceAttributes (next to _dd.stats_computed), so the otlp package does not depend on the ddagent writer package.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code

@datadog-datadog-prod-us1-2

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.81 s 14.73 s [-0.3%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent 13.59 s 13.79 s [-2.5%; -0.5%] (maybe better)
startup:petclinic:appsec:Agent 16.96 s 16.80 s [+0.1%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent 16.78 s 16.92 s [-1.6%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 16.85 s 16.80 s [-1.1%; +1.7%] (no difference)
startup:petclinic:sca:Agent 16.37 s 16.53 s [-5.2%; +3.2%] (no difference)
startup:petclinic:tracing:Agent 16.04 s 16.24 s [-2.1%; -0.3%] (maybe better)

Commit: 06e13575 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@mhlidd mhlidd added comp: telemetry Telemetry tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements labels Sep 16, 2026
@mhlidd mhlidd changed the title introducing new tags/otlp resource attributes for otlp adoption metrics Introducing Tags/OTLP Resource Attributes for OTLP Adoption Metrics Sep 16, 2026
@mhlidd
mhlidd force-pushed the mhlidd/otlp_adoption_metrics branch from a325006 to aab708f Compare September 16, 2026 18:38
@mhlidd
mhlidd marked this pull request as ready for review September 17, 2026 17:28
@mhlidd
mhlidd requested review from a team as code owners September 17, 2026 17:28
@mhlidd
mhlidd requested review from claponcet, jordan-wong, manuel-alvarez-alvarez and mcculls and removed request for a team September 17, 2026 17:28

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

A custom default span tag map removes _dd.sdk.otlp_export. The adoption metric then misses spans from these tracer configurations.

Open Bits AI session

🤖 Datadog Autotest · Commit aab708f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would strongly prefer this be solved at the serialization rather than adding a tag directly to the spans.

@mhlidd

mhlidd commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

@dougqh We do want the tag to come from the SDK as a source of truth. Instead of adding the tag to every span by default, WDYT about adding it as a OTLP Resource Attributes for OTLP export mode, for v1 add it as a trace-level tag, and fallback to adding a span tag for v0.4 and v0.5 on each span at serialization time? This should improve performance from the current approach and maintain our ability to have the tag come directly from the SDK

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OTLP part looks fine to me, so approving from that perspective.

However I'm hesitant about the need to have the _dd.sdk.otlp_export tag on all spans. If we added it to the trace resource attributes then potentially we could decorate the spans on ingestion of the OTLP payload and save on bandwidth.

So I would like to see @dougqh's comment addressed and resolved before merging (specifically the introduction of a new span tag vs just sending the same information once in the OTLP trace payload)

@mhlidd
mhlidd marked this pull request as draft September 21, 2026 16:51
@mhlidd
mhlidd requested a review from dougqh September 22, 2026 21:00

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

For v0.4 and v0.5 payloads, a dropped oversized trace can suppress the next marker. A same-name span tag can also replace the canonical value.

Open Bits AI session

🤖 Bits Code Review · Commit e3a4b8e · @DataDog review to ask questions

Comment thread dd-trace-core/src/test/java/datadog/trace/core/DDSpanSerializationTest.java Outdated

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Thanks for changing the approach.

@mhlidd
mhlidd requested review from amarziali and mcculls September 23, 2026 20:21

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Echoing what @dougqh said, the new approach reads much better - thanks!

@mhlidd
mhlidd enabled auto-merge September 24, 2026 14:38
@mhlidd
mhlidd disabled auto-merge September 24, 2026 17:01
@mhlidd
mhlidd enabled auto-merge September 24, 2026 17:02
@mhlidd
mhlidd disabled auto-merge September 24, 2026 17:02
@mhlidd
mhlidd added this pull request to the merge queue Sep 24, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-09-24 17:03:03 UTC ℹ️ Start processing command /merge


2026-09-24 17:03:08 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-09-24 18:51:11 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 24, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 361b6f0 into master Sep 24, 2026
605 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the mhlidd/otlp_adoption_metrics branch September 24, 2026 18:51
@github-actions github-actions Bot added this to the 1.67.0 milestone Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: telemetry Telemetry tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants