[IGNORE] performance/flaky-benchmarks-monitoring by igoragoli · Pull Request #12580 · DataDog/dd-trace-java

igoragoli · 2026-09-20T15:48:08Z

Automation branch for flaky benchmarks monitoring. See docs.

dd-octo-sts · 2026-09-20T17:10:49Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass
Load	🟢 pass
DaCapo	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Variant	Metric	Candidate	master	Δ (95% CI of mean)
dacapo:biojava:tracing	datadog	execution_time	13.46 s	13.55 s	[-2.8%; +1.6%] (no difference)
dacapo:tomcat:tracing	datadog	execution_time	1.72 ms	1.73 ms	[-2.1%; +1.7%] (no difference)
startup:insecure-bank:iast:Agent	datadog	execution_time	14.07 s	14.01 s	[-0.4%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	datadog	execution_time	12.99 s	13.04 s	[-1.2%; +0.4%] (no difference)
webserver--insecure-bank--tracing--high_load	load	agg_http_req_duration_p50	808.07 µs	793.60 µs	[-1.4%; +5.1%] (no difference)
startup:petclinic:appsec:Agent	datadog	execution_time	17.66 s	17.43 s	[+0.5%; +2.2%] (maybe worse)
startup:petclinic:iast:Agent	datadog	execution_time	17.64 s	17.66 s	[-1.1%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	datadog	execution_time	17.59 s	17.55 s	[-1.0%; +1.4%] (no difference)
startup:petclinic:sca:Agent	datadog	execution_time	17.71 s	17.61 s	[-0.4%; +1.5%] (no difference)
startup:petclinic:tracing:Agent	datadog	execution_time	16.73 s	16.72 s	[-1.1%; +1.2%] (no difference)
webserver--spring-petclinic--sca--high_load	load	agg_http_req_duration_p50	25.11 ms	25.54 ms	[-7.8%; +4.5%] (unstable)
webserver--spring-petclinic--tracing--high_load	load	agg_http_req_duration_p50	24.26 ms	25.14 ms	[-9.8%; +2.7%] (unstable)

Commit: 73995b96 · CI Pipeline · Benchmarking Platform UI

pr-commenter · 2026-09-20T17:27:51Z

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_date	1790088897	1790090342
git_commit_sha	`be7c7c5`	`73995b9`

See matching parameters

	Baseline	Candidate
ci_job_date	1790117870	1790117870
ci_job_id	2070139654	2070139654
ci_pipeline_id	139343605	139343605
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-dvjm6wsc 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-dvjm6wsc 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

pr-commenter · 2026-09-20T17:40:17Z

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_date	1790088897	1790090342
git_commit_sha	`be7c7c5`	`73995b9`

See matching parameters

	Baseline	Candidate
ci_job_date	1790117877	1790117877
ci_job_id	2070139526	2070139526
ci_pipeline_id	139343601	139343601
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-34xt12rg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-34xt12rg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

pr-commenter · 2026-09-21T22:08:48Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1790117419	1790117736
end_time	2026-09-22T22:51:49	2026-09-22T22:57:06
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_sha	`8402b9b`	`73995b9`
start_time	2026-09-22T22:50:20	2026-09-22T22:55:37

See matching parameters

	Baseline	Candidate
ci_job_id	2070139667	2070139667
ci_pipeline_id	139343602	139343602
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
git_commit_date	1790090342	1790090342

Summary

Found 0 performance improvements and 5 performance regressions! Performance is the same for 5 metrics, 5 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	worse [+469.228µs; +565.498µs] or [+6.229%; +7.507%]	worse [+480.459µs; +577.653µs] or [+6.328%; +7.609%]	worse [+471.779µs; +570.897µs] or [+6.179%; +7.477%]	worse [+461.044µs; +579.450µs] or [+5.963%; +7.494%]	worse [-8.997op/s; -6.841op/s] or [-6.928%; -5.267%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-19.679µs; +16.805µs] or [-5.978%; +5.105%]	unstable [-33.294µs; +25.490µs] or [-9.167%; +7.018%]	unstable [-42.875µs; +33.515µs] or [-11.354%; +8.875%]	unstable [-49.839µs; +149.148µs] or [-4.540%; +13.586%]	same
scenario:basic	same	same	same	unstable [-26.889µs; +208.692µs] or [-2.975%; +23.093%]	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (363.193 µs) : 332, 394
.   : milestone, 363,
basic (336.429 µs) : 329, 344
.   : milestone, 336,
loop (7.592 ms) : 7585, 7600
.   : milestone, 7592,
section candidate
noprobe (359.291 µs) : 335, 383
.   : milestone, 359,
basic (341.869 µs) : 335, 349
.   : milestone, 342,
loop (8.121 ms) : 8058, 8185
.   : milestone, 8121,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	363.193 µs [331.901 µs, 394.485 µs]
basic	336.429 µs [329.017 µs, 343.842 µs]
loop	7.592 ms [7.585 ms, 7.6 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	359.291 µs [335.23 µs, 383.353 µs]
basic	341.869 µs [335.077 µs, 348.662 µs]
loop	8.121 ms [8.058 ms, 8.185 ms]

Generated by stability-generate-patch from benchmarking-platform-tools patch-specs/dd-trace-java.yml. Base: master @ 3723256. Do not edit by hand -- this branch is force-pushed on each refresh.

igoragoli force-pushed the performance/flaky-benchmarks-monitoring branch from 88fbaea to 623d388 Compare September 20, 2026 16:04

igoragoli force-pushed the performance/flaky-benchmarks-monitoring branch from 623d388 to 054c861 Compare September 21, 2026 21:48

igoragoli force-pushed the performance/flaky-benchmarks-monitoring branch 2 times, most recently from 5955793 to e9bad03 Compare September 21, 2026 23:32

igoragoli changed the title ~~performance/flaky-benchmarks-monitoring~~ [IGNORE] performance/flaky-benchmarks-monitoring Sep 22, 2026

igoragoli force-pushed the performance/flaky-benchmarks-monitoring branch from e9bad03 to 73995b9 Compare September 22, 2026 15:19

ci: gate pipeline to benchmark jobs for stability monitoring

13160e7

Generated by stability-generate-patch from benchmarking-platform-tools patch-specs/dd-trace-java.yml. Base: master @ 3723256. Do not edit by hand -- this branch is force-pushed on each refresh.

igoragoli force-pushed the performance/flaky-benchmarks-monitoring branch from 73995b9 to 13160e7 Compare September 23, 2026 12:22

Navigation Menu

[IGNORE] performance/flaky-benchmarks-monitoring - #12580

[IGNORE] performance/flaky-benchmarks-monitoring#12580
igoragoli wants to merge 1 commit into
masterfrom
performance/flaky-benchmarks-monitoring

igoragoli commented Sep 20, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Sep 21, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

igoragoli commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

pr-commenter Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kafka / producer-benchmark

Parameters

Summary

Uh oh!

pr-commenter Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kafka / consumer-benchmark

Parameters

Summary

Uh oh!

pr-commenter Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

igoragoli commented Sep 20, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 20, 2026 •

edited

Loading

pr-commenter Bot commented Sep 20, 2026 •

edited

Loading

pr-commenter Bot commented Sep 20, 2026 •

edited

Loading

pr-commenter Bot commented Sep 21, 2026 •

edited

Loading