Skip to content

feat(sampling): Support for OpenTelemetry consistent tracestate sampling #12397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MilanGarnier wants to merge 12 commits into from
Draft

feat(sampling): Support for OpenTelemetry consistent tracestate sampling #12397

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e833052
feat(core): support OTel tracestate sampling
MilanGarnier Aug 27, 2026
4006f90
clearer determination of ratelimiting and move th/rv computation in h…
MilanGarnier Sep 3, 2026
3955ca7
centralize values used for testing
MilanGarnier Sep 3, 2026
d6dc9d0
fix(propagation): align otel sampling state behavior
MilanGarnier Sep 3, 2026
c51c895
test(sampling): restore deterministic sampler table test
MilanGarnier Sep 4, 2026
61bcb3f
perf(propagation): compact OTel trace state representation
MilanGarnier Sep 4, 2026
ea27b06
perf(propagation): pre-size OTel tracestate builder
MilanGarnier Sep 4, 2026
bf6f450
test(propagation): pass otel member size to parser
MilanGarnier Sep 7, 2026
d1dfda5
refactor(propagation): clarify otel tracestate parsing
MilanGarnier Sep 7, 2026
95af86c
fix(propagation): preserve OTel state on limiter demotion
MilanGarnier Sep 8, 2026
a3f5a89
test(propagation): align limiter demotion expectation
MilanGarnier Sep 8, 2026
a0176c9
fix(sampling): keep empty agent rates uninitialized
MilanGarnier Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 25 additions & 17 deletions dd-trace-core/src/main/java/datadog/trace/common/sampling/RateByServiceTraceSampler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,20 +58,16 @@ public <T extends CoreSpan<T>> void setSamplingPriority(final T span) {

final RateSamplersByEnvAndService rates = serviceRates;
RateSampler sampler = rates.getSampler(env, serviceName);

if (sampler.sample(span)) {
span.setSamplingPriority(
PrioritySampling.SAMPLER_KEEP,
SAMPLING_AGENT_RATE,
sampler.getSampleRate(),
SamplingMechanism.AGENT_RATE);
} else {
span.setSamplingPriority(
PrioritySampling.SAMPLER_DROP,
SAMPLING_AGENT_RATE,
sampler.getSampleRate(),
SamplingMechanism.AGENT_RATE);
}
boolean sampled = sampler.sample(span);
int samplingPriority = sampled ? PrioritySampling.SAMPLER_KEEP : PrioritySampling.SAMPLER_DROP;

Boolean probabilitySamplingResult = rates.hasAgentRates() ? sampled : null;
span.setSamplingPriority(
samplingPriority,
SAMPLING_AGENT_RATE,
sampler.getSampleRate(),
SamplingMechanism.AGENT_RATE,
probabilitySamplingResult);
}

private <T extends CoreSpan<T>> String getSpanEnv(final T span) {
Expand Down Expand Up @@ -117,10 +113,12 @@ public void onResponse(
new TreeMap<>(String::compareToIgnoreCase);

RateSampler fallbackSampler = RateSamplersByEnvAndService.DEFAULT_SAMPLER;
boolean hasAgentRates = false;
for (final Map.Entry<String, Number> entry : newServiceRates.entrySet()) {
if (entry.getValue() == null) {
continue;
}
hasAgentRates = true;
double rate = entry.getValue().doubleValue();

EnvAndService envAndService = EnvAndService.fromString(entry.getKey());
Expand Down Expand Up @@ -161,7 +159,8 @@ public void onResponse(
if (canIncrease && anyCapped) {
lastCappedNanos = now;
}
serviceRates = new RateSamplersByEnvAndService(updatedEnvServiceRates, fallbackSampler);
serviceRates =
new RateSamplersByEnvAndService(updatedEnvServiceRates, fallbackSampler, hasAgentRates);
}

private static RateSampler createRateSampler(final double sampleRate) {
Expand All @@ -183,15 +182,24 @@ private static final class RateSamplersByEnvAndService {

private final Map<String, TreeMap<String, RateSampler>> envServiceRates;
private final RateSampler fallbackSampler;
// Whether this snapshot contains at least one non-null Agent-provided rate.
private final boolean hasAgentRates;

RateSamplersByEnvAndService() {
this(Collections.emptyMap(), DEFAULT_SAMPLER);
this(Collections.emptyMap(), DEFAULT_SAMPLER, false);
}

RateSamplersByEnvAndService(
Map<String, TreeMap<String, RateSampler>> envServiceRates, RateSampler fallbackSampler) {
Map<String, TreeMap<String, RateSampler>> envServiceRates,
RateSampler fallbackSampler,
boolean hasAgentRates) {
this.envServiceRates = envServiceRates;
this.fallbackSampler = fallbackSampler;
this.hasAgentRates = hasAgentRates;
}

boolean hasAgentRates() {
return hasAgentRates;
}

RateSampler getFallbackSampler() {
Expand Down
31 changes: 12 additions & 19 deletions dd-trace-core/src/main/java/datadog/trace/common/sampling/RuleBasedTraceSampler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,28 +146,21 @@ public <T extends CoreSpan<T>> void setSamplingPriority(final T span) {
if (matchedRule == null) {
fallbackSampler.setSamplingPriority(span);
} else {
if (matchedRule.sample(span)) {
if (rateLimiter.tryAcquire()) {
span.setSamplingPriority(
PrioritySampling.USER_KEEP,
SAMPLING_RULE_RATE,
matchedRule.getSampler().getSampleRate(),
matchedRule.getMechanism());
} else {
span.setSamplingPriority(
PrioritySampling.USER_DROP,
SAMPLING_RULE_RATE,
matchedRule.getSampler().getSampleRate(),
matchedRule.getMechanism());
}
boolean sampled = matchedRule.sample(span);
int samplingPriority;
if (sampled) {
samplingPriority =
rateLimiter.tryAcquire() ? PrioritySampling.USER_KEEP : PrioritySampling.USER_DROP;
span.setMetric(SAMPLING_LIMIT_RATE, rateLimit);
} else {
span.setSamplingPriority(
PrioritySampling.USER_DROP,
SAMPLING_RULE_RATE,
matchedRule.getSampler().getSampleRate(),
matchedRule.getMechanism());
samplingPriority = PrioritySampling.USER_DROP;
}
span.setSamplingPriority(
samplingPriority,
SAMPLING_RULE_RATE,
matchedRule.getSampler().getSampleRate(),
matchedRule.getMechanism(),
Boolean.valueOf(sampled));
}
}
}
6 changes: 5 additions & 1 deletion dd-trace-core/src/main/java/datadog/trace/core/CoreSpan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,11 @@ default void processTagsAndBaggageWithStructuredLinks(
T setSamplingPriority(int samplingPriority, int samplingMechanism);

T setSamplingPriority(
int samplingPriority, CharSequence rate, double sampleRate, int samplingMechanism);
int samplingPriority,
CharSequence rate,
double sampleRate,
int samplingMechanism,
Boolean probabilitySamplingResult);

T setSpanSamplingPriority(double rate, int limit);

Expand Down
22 changes: 21 additions & 1 deletion dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -646,6 +646,25 @@ public final DDSpan setSamplingPriority(final int newPriority, int samplingMecha

@Override
public DDSpan setSamplingPriority(
int samplingPriority,
CharSequence rate,
double sampleRate,
int samplingMechanism,
Boolean probabilitySamplingResult) {
if (setSamplingPriorityWithRate(samplingPriority, rate, sampleRate, samplingMechanism)
&& probabilitySamplingResult != null) {
context
.getPropagationTags()
.updateOtelTraceState(
getTraceId().toLong(),
sampleRate,
probabilitySamplingResult.booleanValue(),
samplingPriority);
}
return this;
}

private boolean setSamplingPriorityWithRate(
int samplingPriority, CharSequence rate, double sampleRate, int samplingMechanism) {
if (context.setSamplingPriority(samplingPriority, samplingMechanism)) {
setMetric(rate, sampleRate);
Expand All @@ -655,8 +674,9 @@ public DDSpan setSamplingPriority(
|| samplingMechanism == SamplingMechanism.REMOTE_ADAPTIVE_RULE) {
context.getPropagationTags().updateKnuthSamplingRate(sampleRate);
}
return true;
}
return this;
return false;
}

@Override
Expand Down
8 changes: 7 additions & 1 deletion dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,13 @@ public DDSpanContext(
setOrigin(origin);
}
if (samplingPriority != PrioritySampling.UNSET) {
setSamplingPriority(samplingPriority, SamplingMechanism.UNKNOWN);
if (this.propagationTags.getSamplingPriority() == samplingPriority) {

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve OTel state for compound Datadog extraction

With the default DATADOG, TRACECONTEXT extraction order, the first Datadog context owns the sampling priority while applyTraceContextToFirstContext copies only the later W3C tracestate. Its PropagationTags priority therefore remains UNSET, so this comparison fails and the UNKNOWN update reaches removeForNonProbabilityDecision(), deleting the inherited th even when the Datadog and W3C decisions agree. Dual headers emitted by one instrumented service consequently lose consistent-sampling state at the next Java hop; avoid reapplying extracted priorities as UNKNOWN or synchronize the merged propagation-tags priority first.

Useful? React with 👍 / 👎.

// Extractors already applied this priority to the propagation tags. Initialize the local
// field without reapplying the decision as an unknown local override.
SAMPLING_PRIORITY_UPDATER.set(this, samplingPriority);
} else {
setSamplingPriority(samplingPriority, SamplingMechanism.UNKNOWN);
}
}
setTag(PARENT_ID, this.propagationTags.getLastParentId());
}
Expand Down
3 changes: 3 additions & 0 deletions dd-trace-core/src/main/java/datadog/trace/core/propagation/PropagationTags.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ public interface Factory {

public abstract void forceKeep(int samplingMechanism);

public abstract void updateOtelTraceState(
long traceIdLowOrderBits, double sampleRate, boolean sampled, int samplingPriority);

public abstract int getSamplingPriority();

public abstract void updateTraceOrigin(CharSequence origin);
Expand Down
Loading