Introducing Tags/OTLP Resource Attributes for OTLP Adoption Metrics - #12516
Conversation
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
a325006 to
aab708f
Compare
There was a problem hiding this comment.
A custom default span tag map removes _dd.sdk.otlp_export. The adoption metric then misses spans from these tracer configurations.
🤖 Datadog Autotest · Commit aab708f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
dougqh
left a comment
There was a problem hiding this comment.
I would strongly prefer this be solved at the serialization rather than adding a tag directly to the spans.
|
@dougqh We do want the tag to come from the SDK as a source of truth. Instead of adding the tag to every span by default, WDYT about adding it as a OTLP Resource Attributes for OTLP export mode, for v1 add it as a trace-level tag, and fallback to adding a span tag for v0.4 and v0.5 on each span at serialization time? This should improve performance from the current approach and maintain our ability to have the tag come directly from the SDK |
mcculls
left a comment
There was a problem hiding this comment.
The OTLP part looks fine to me, so approving from that perspective.
However I'm hesitant about the need to have the _dd.sdk.otlp_export tag on all spans. If we added it to the trace resource attributes then potentially we could decorate the spans on ingestion of the OTLP payload and save on bandwidth.
So I would like to see @dougqh's comment addressed and resolved before merging (specifically the introduction of a new span tag vs just sending the same information once in the OTLP trace payload)
dougqh
left a comment
There was a problem hiding this comment.
This looks good to me. Thanks for changing the approach.
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What Does This Do
Adds two markers that let the backend attribute ingested spans to the SDK's export path and semantics:
_dd.sdk.otlp_exportpayload marker (TraceMapper.SDK_OTLP_EXPORT) — written once per payload rather than on every span, since the value is uniform for the whole payload:TraceMapperV0_4/TraceMapperV0_5: written into the meta of the first span of the first non-empty chunk, gated onfirstSpanInPayload(the same way_dd.tags.processis handled).TraceMapperV1: written as a payload-level attribute (header field 10) alongside_dd.tags.process."true"/"false"fromConfig.isOtlpTracesExportEnabled()(true whenwriterTypeisOtlpWriteror aMultiWritercontaining it), snapshotted once per mapper viaTraceMapper.otlpExportMarker(Config).OtlpResourceAttributes.traceResourceAttributesalways emits_dd.sdk.otlp_export=trueas a resource attribute, since reaching that encoder means the payload is leaving over OTLP.datadog.sdk.semanticsOTLP resource attribute — contributed byOtlpResourceAttributes.traceResourceAttributeson the OTLP export path,"otel"whenConfig.isTraceOtelSemanticsEnabled()is set and"datadog"otherwise.Both keys are added to
IGNORED_GLOBAL_TAGSinOtlpResourceAttributesso a global tag of the same name is not echoed into the resource.Tests:
TraceMapperV04PayloadTest/TraceMapperV05PayloadTest: marker appears only on the first span of the first non-empty chunk, and is"true"when also exporting over OTLP.TraceMapperV1PayloadTest: marker present in payload attributes,"true"when also exporting over OTLP (V1PayloadReaderextended to read it).OtlpResourceJsonTest/OtlpResourceProtoTest: trace resource attributes carry the OTLP export marker, anddatadog.sdk.semanticsswitches to"otel"when enabled.DDAgentApiTest,DDAgentWriterCombinedTest,DDSpanSerializationTestupdated for the extra meta entry on the first span.Motivation
We have no direct signal for how many spans reach the intake via the SDK's OTLP export path versus the native Datadog path, nor which semantic convention the SDK was configured with. Carrying both as markers on the wire lets the backend resolve them into tags on
dd.trace.intake.ingested_spans(surfaced asdatadog.sdk.otlp_export) and measure OTLP export adoption for traces directly, instead of inferring it.Additional Notes
datadog-agenthoists it into the Agent's outgoing payload (from the first span's meta intoTracerPayload.Tagsfor v0.4/v0.5, and from the payloadAttributesfor V1, which is the default from Agent 7.84);logs-backendpropagates it into chunk tags;dd-goresolves it into thedatadog.sdk.otlp_exporttag ondd.trace.intake.ingested_spans.datadog.sdk.otlp_exportand the wire marker_dd.sdk.otlp_exportare deliberately named differently — same concept, two layers. They should not be conflated when reading dashboards or backend code.CoreTracer.withTracerTags. That was dropped in favor of payload scope to avoid per-span wire overhead; the marker is no longer part of the tracer-defined span tag bundle.DDTagsAPI (dd-trace-api). It lives indd-trace-coreinstead:TraceMapper.SDK_OTLP_EXPORTfor the Agent mappers, and a privateSDK_OTLP_EXPORT_KEYinOtlpResourceAttributes(next to_dd.stats_computed), so theotlppackage does not depend on theddagentwriter package.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
🤖 Generated with Claude Code