Wait for Reactor Kafka test pipelines to finish by amarziali · Pull Request #12501 · DataDog/dd-trace-java

amarziali · 2026-09-15T10:25:41Z

What Does This Do

Makes the Reactor Kafka test wait for all 100 offset commits before stopping the receiver.

Previously, take(100) could cancel the receiver as soon as the records were delivered, while asynchronous commits were still queued.

The test now:

Waits for the producer pipeline with blockLast().
Counts successful offset commit completions.
Disposes the receiver after all 100 commits complete.
Verifies the expected traces and parent relationships.

Cleanup also disposes the receiver and closes the Kafka sender.

sequenceDiagram
    participant Test
    participant Producer
    participant Kafka
    participant Receiver
    participant Commits as Commit latch
    participant Writer as Trace writer

    Test->>Receiver: subscribe
    Test->>Producer: send 100 records
    Producer->>Kafka: publish records

    loop 100 records
        Kafka->>Receiver: deliver record
        Receiver->>Kafka: commit offset
        Kafka-->>Receiver: commit completes
        Receiver->>Commits: count down
        Receiver->>Writer: publish consume trace
    end

    Producer-->>Test: blockLast completes
    Commits-->>Test: all 100 commits completed
    Test->>Receiver: dispose subscription
    Test->>Writer: verify 200 traces

Motivation

Trace publication is not a completion signal for the surrounding reactive work. The test could therefore finish while Reactor still owned active continuations or offset commits.

Waiting for successful commit completions makes the receiver lifecycle explicit and removes that race.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

chatgpt-codex-connector · 2026-09-15T10:27:06Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-15T10:29:33.699614Z	`9f1e243`	PR opened
🔒 Security Review	✅ Completed 2026-09-15T10:29:11.701395Z	`9f1e243`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-prod-us1-3

The receiver can close after it emits record 100 and before queued commit work ends. This keeps the test race that the change aims to remove.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 9f1e243 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

dd-octo-sts · 2026-09-15T10:48:41Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.91 s	14.73 s	[+0.3%; +2.2%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.69 s	13.72 s	[-1.0%; +0.7%] (no difference)
startup:petclinic:appsec:Agent	16.98 s	16.14 s	[+0.7%; +9.7%] (maybe worse)
startup:petclinic:iast:Agent	16.97 s	16.89 s	[-0.2%; +1.1%] (no difference)
startup:petclinic:profiling:Agent	16.57 s	16.95 s	[-3.4%; -1.0%] (significantly better)
startup:petclinic:sca:Agent	16.83 s	16.75 s	[-0.4%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.08 s	16.06 s	[-0.9%; +1.2%] (no difference)

Commit: 8281aadd · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

pr-commenter · 2026-09-15T11:05:48Z

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/diag-fix-kafka-reactor
git_commit_date	1789430290	1789473886
git_commit_sha	`7be11f6`	`8281aad`

See matching parameters

	Baseline	Candidate
ci_job_date	1789475036	1789475036
ci_job_id	2044279857	2044279857
ci_pipeline_id	137518031	137518031
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-4sl2wi19 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-4sl2wi19 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

amarziali · 2026-09-15T11:14:43Z

@DataDog review

datadog-prod-us1-3

The new take(100) stage can stop the Kafka receiver before queued offset commits finish. This race can make the test fail or time out.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 297eee1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

amarziali · 2026-09-15T12:26:34Z

@DataDog review

datadog-prod-us1-3

More details

The test waits until all producer sends and offset commits finish. It then stops the receiver and closes the sender.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 8281aad · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

pr-commenter · 2026-09-15T12:57:54Z

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/diag-fix-kafka-reactor
git_commit_date	1789430290	1789473886
git_commit_sha	`7be11f6`	`8281aad`

See matching parameters

	Baseline	Candidate
ci_job_date	1789475075	1789475075
ci_job_id	2044279860	2044279860
ci_pipeline_id	137518031	137518031
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-tvs8txbi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-tvs8txbi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	unsure [-11211.327op/s; -328.835op/s] or [-4.428%; -0.130%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

vandonr

thanks

amarziali · 2026-09-17T09:15:46Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-09-17T09:15:49Z

View all feedbacks in Devflow UI.

2026-09-17 09:15:49 UTC ℹ️ Start processing command /merge

2026-09-17 09:15:54 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-09-17 10:29:22 UTC ℹ️ MergeQueue: This merge request was merged

Wait for Reactor Kafka test pipelines to finish

9f1e243

amarziali added type: bug fix Bug fix comp: testing Testing tag: no release notes Changes to exclude from release notes labels Sep 15, 2026

amarziali requested review from a team as code owners September 15, 2026 10:26

amarziali requested review from vandonr and removed request for a team September 15, 2026 10:26

amarziali mentioned this pull request Sep 15, 2026

Detect scope and continuation leaks in instrumentation tests #12458

Draft

datadog-prod-us1-3 Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread ...agent/instrumentation/kafka/kafka-clients-0.11/src/test/groovy/KafkaReactorForkedTest.groovy Outdated

This comment has been minimized.

Sign in to view

Clarify Reactor Kafka completion ordering

297eee1

datadog-prod-us1-3 Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread ...agent/instrumentation/kafka/kafka-clients-0.11/src/test/groovy/KafkaReactorForkedTest.groovy Outdated

Wait for Reactor Kafka commits before stopping receiver

8281aad

datadog-prod-us1-3 Bot reviewed Sep 15, 2026

View reviewed changes

vandonr approved these changes Sep 17, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d Bot merged commit 7a2a675 into master Sep 17, 2026
610 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the andrea.marziali/diag-fix-kafka-reactor branch September 17, 2026 10:29

github-actions Bot added this to the 1.67.0 milestone Sep 17, 2026

Conversation

amarziali commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

chatgpt-codex-connector Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-prod-us1-3 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

pr-commenter Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kafka / producer-benchmark

Parameters

Summary

Uh oh!

amarziali commented Sep 15, 2026

Uh oh!

datadog-prod-us1-3 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amarziali commented Sep 15, 2026

Uh oh!

datadog-prod-us1-3 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Sep 15, 2026

Kafka / consumer-benchmark

Parameters

Summary

Uh oh!

vandonr left a comment

Choose a reason for hiding this comment

Uh oh!

amarziali commented Sep 17, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

amarziali commented Sep 15, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 15, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 15, 2026 •

edited

Loading

pr-commenter Bot commented Sep 15, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Sep 17, 2026 •

edited

Loading