Skip to content

Improve diagnostics for hung-test dump collection - #12607

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
alexeyk/hanged-test-dump-diagnostics
Sep 23, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
alexeyk/hanged-test-dump-diagnostics

Conversation

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor

What Does This Do

Add concise start/completion messages for dump commands, including the task, command, output destination, and elapsed time. Report command and directory failures with stack traces, warn when no test executor is found, and continue with subsequent dump commands after a command failure.

On interruption, stop the diagnostic subprocess and preserve the interrupt status. Close process-discovery streams after use.

Motivation

The existing collection-start message does not identify which command is still running. Errors lose their stack traces, and a failed heap-dump command aborts collection before thread dumps can be attempted. These gaps make missing dumps difficult to diagnose.

Additional Notes

Validation: all 4 tests passed in DumpHangedTestIntegrationTest, covering normal collection, no collection for a short test, directory-creation failure, and a simulated heap-dump failure followed by successful thread-dump attempts.

./gradlew :buildSrc:test -PrunBuildSrcTests --tests datadog.gradle.plugin.dump.DumpHangedTestIntegrationTest

The simulated failure reports java.io.IOException: Process failed with exit code 7 with a DumpHangedTestPlugin.runCmd stack frame; the test then verifies both per-process and all-JVM thread-dump output. This demonstrates that one failed command no longer discards the remaining diagnostic attempts.

Dump scheduling, heap-first ordering, and command wait times remain unchanged. Collection still depends on the per-task timer; this change does not capture dumps before a separate job-level deadline.

Related plugin configuration refactor: #11846.

Contributor Checklist

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added type: bug fix Bug fix comp: testing Testing tag: no release notes Changes to exclude from release notes tag: diagnostics Diagnostics related changes tag: ai generated Largely based on code generated by an AI or LLM labels Sep 23, 2026
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Sep 23, 2026
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD marked this pull request as ready for review September 23, 2026 02:42
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested a review from a team as a code owner September 23, 2026 02:42
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested review from bric3 and removed request for a team September 23, 2026 02:42
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T02:45:29.699709Z a042160 Draft marked ready
🔒 Security Review ✅ Completed 2026-09-23T02:46:48.957585Z a042160 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

A failed heap-dump command now logs its stack trace and does not stop later thread-dump commands. The interruption path stops the child process and keeps the interrupt status.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit a042160 · @DataDog review to ask questions

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.28% (-0.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a042160 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.93 s 14.05 s [-1.7%; -0.0%] (maybe better)
startup:insecure-bank:tracing:Agent 12.94 s 13.04 s [-1.5%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 17.04 s 17.01 s [-1.0%; +1.3%] (no difference)
startup:petclinic:iast:Agent 17.03 s 17.05 s [-1.1%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 16.77 s 16.75 s [-1.2%; +1.5%] (no difference)
startup:petclinic:sca:Agent 16.99 s 16.74 s [+0.6%; +2.4%] (maybe worse)
startup:petclinic:tracing:Agent 16.08 s 16.16 s [-1.5%; +0.6%] (no difference)

Commit: a0421605 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@bric3 bric3 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


On a related note, I discovered this week-end that JUnit 5 has a similar mechanism for thread dumps, e.g. (junit-platform.properties)

# Native SWT calls may ignore interruption; dump all threads before CI kills the JVM.
junit.jupiter.execution.timeout.threaddump.enabled=true
# Class-level @Timeout does not cover Display/Shell setup and disposal.
junit.jupiter.execution.timeout.lifecycle.method.default=30 s
junit.jupiter.execution.timeout.thread.mode.default=same_thread

Which of course won't help us since most of our tests are in Groovy / Spock.

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor Author

/merge -f --reason "This is build fix, tested on PR, no need to run MQ"

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-09-23 14:35:32 UTC ℹ️ Start processing command /merge -f --reason "This is build fix, tested on PR, no need to run MQ"


2026-09-23 14:35:36 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 0s (p90).


2026-09-23 14:35:47 UTC ℹ️ MergeQueue: This merge request was merged

Warning

This change was merged without running any pre merge CI checks

Reason: This is build fix, tested on PR, no need to run MQ

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 75c6d58 into master Sep 23, 2026
616 of 621 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the alexeyk/hanged-test-dump-diagnostics branch September 23, 2026 14:35
@github-actions github-actions Bot added this to the 1.67.0 milestone Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing tag: ai generated Largely based on code generated by an AI or LLM tag: diagnostics Diagnostics related changes tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants