Skip to content

Collect IBM JVM hang dumps with test reports - #12638

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
alexeyk/collect-ibm-javacores
Sep 25, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
alexeyk/collect-ibm-javacores

Conversation

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor

What Does This Do

  • Detect IBM/OpenJ9 from the JVM launcher selected for each Gradle Test task.
  • Set IBM_JAVACOREDIR to the same task dump directory collected with test reports.
  • Reuse the launcher decision for kill -3 thread dumps and HotSpot-only heap dump suppression.
  • Add integration coverage that performs a real IBM javacore capture when an IBM JDK is installed.
  • Update the dump investigation guide with the new artifact location.

Motivation

When an IBM test task timed out, kill -3 successfully produced a javacore, but the JVM wrote it outside the module dump directory, typically under /tmp. The CI report collector copies build/dumps, so the diagnostic artifact was unavailable after the job.

The previous /ibm8 executable-path check was also narrower than the Gradle toolchain configuration and could diverge from the JVM selected for the test task.

Additional Notes

Failure evidence from the timeout showed the JVM reporting a successful javacore write under /tmp/javacore...txt, while no javacore appeared in the collected module reports. The selected launcher is now the single source of truth, and IBM_JAVACOREDIR is inherited by the Gradle test worker and its child processes.

Validation:

  • ./gradlew -PrunBuildSrcTests=true :buildSrc:test --tests datadog.gradle.plugin.dump.DumpHangedTestIntegrationTest --rerun-tasks --no-build-cache — 8 tests passed, including a real IBM JVM javacore capture.
  • ./gradlew :buildSrc:spotlessCheck --no-build-cache

Relevant precedent: #10646 introduced IBM8 thread-dump collection; #12619 established thread-before-heap dump ordering.

OpenJ9 documents IBM_JAVACOREDIR as the Java dump destination: https://eclipse.dev/openj9/docs/env_var/#java-dump-options

Contributor Checklist

Jira ticket: N/A

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added type: bug fix Bug fix tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling tag: ibm IBM JVM related run-tests: ibm8 Run tests using IBM 8 JDK tag: ai generated Largely based on code generated by an AI or LLM labels Sep 24, 2026
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Sep 24, 2026
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD marked this pull request as ready for review September 24, 2026 21:14
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested review from a team as code owners September 24, 2026 21:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T21:19:37.848219Z fcb6357 Draft marked ready
🔒 Security Review ✅ Completed 2026-09-24T21:17:52.080225Z fcb6357 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sarahchen6 sarahchen6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcb6357586

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-official

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.79 s 14.64 s [-0.1%; +2.1%] (no difference)
startup:insecure-bank:tracing:Agent 13.62 s 13.69 s [-1.3%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.05 s 16.98 s [-0.6%; +1.4%] (no difference)
startup:petclinic:iast:Agent 16.32 s 17.01 s [-8.2%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.70 s 16.83 s [-2.0%; +0.4%] (no difference)
startup:petclinic:sca:Agent 16.97 s 16.62 s [+1.1%; +3.1%] (significantly worse)
startup:petclinic:tracing:Agent 16.15 s 16.13 s [-0.7%; +1.0%] (no difference)

Commit: fcb63575 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

An IBM task's dump timer applies its JVM choice to all discovered test processes. This sends SIGQUIT to native children and can stop processes in otherwise healthy parallel tests.

Open Bits AI session

🤖 Bits Code Review · Commit fcb6357 · @DataDog review to ask questions

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor Author

/merge -f --reason "It is a build change, tested in PR, no need to run MQ"

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-09-25 01:57:11 UTC ℹ️ Start processing command /merge -f --reason "It is a build change, tested in PR, no need to run MQ"


2026-09-25 01:57:16 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 0s (p90).


2026-09-25 01:57:26 UTC ℹ️ MergeQueue: This merge request was merged

Warning

This change was merged without running any pre merge CI checks

Reason: It is a build change, tested in PR, no need to run MQ

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 038e0f8 into master Sep 25, 2026
617 of 622 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the alexeyk/collect-ibm-javacores branch September 25, 2026 01:57
@github-actions github-actions Bot added this to the 1.67.0 milestone Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling run-tests: ibm8 Run tests using IBM 8 JDK tag: ai generated Largely based on code generated by an AI or LLM tag: ibm IBM JVM related tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants