Skip to content

Fix flaky WAFModuleHandleReloadRaceTest wait assertion - #13133

Open
cgcote wants to merge 1 commit into
masterfrom
dd/bits/fix-flaky-waf-handle-reload-race-test-7f3k
Open

cgcote wants to merge 1 commit into
masterfrom
dd/bits/fix-flaky-waf-handle-reload-race-test-7f3k

Conversation

@cgcote

@cgcote cgcote commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Fixes a test-only race in WAFModuleHandleReloadRaceTest.waitsForContextCreationBeforeClosingTheHandle. The test now asserts the "blocked in initOrUpdateWafHandle" state that its wait loop actually observed. It no longer samples the update thread's state a second time. Production code is unchanged.

Motivation

Flaky Test Management reports this test failing about 0.35% of the time on JDK 8, IBM8 and Semeru17 (amd64 and arm64). It failed at the assertTrue(blockedInHandleReplacement(updateThread)) line on master and in the merge queue, costing about 7.4h of CI across 7 pipelines.

The wait loop exits as soon as blockedInHandleReplacement(updateThread) returns true. The assertion then called the same method again, which re-reads Thread.getState() and the stack. The update thread is parked in StampedLock.writeLock() while the evaluation thread holds the read lock. It can briefly be non-WAITING, for example after a spurious wakeup and before it re-parks. A sample taken in that window fails the test, even though the loop had already seen the thread blocked and update.isDone() is still false. The locking in WAFModule.initOrUpdateWafHandle is not at fault.

Additional Notes

  • The test is just as strict as before. If the loop times out without ever seeing the update thread blocked, blocked stays false and the test fails. All other assertions are unchanged, including the checks that the reload hasn't finished and the handle stays online during context creation.
  • Red/green check: I used a temporary probe that was not committed. It unparks the update thread after the wait loop to simulate a spurious wakeup, then waits until the thread leaves WAITING. Over 300 repetitions each, the old re-sampling assertion failed 141 times and the new assertion passed every time.
  • With this fix, all 8 tests in WAFModuleHandleReloadRaceTest pass locally on JDK 21. I couldn't reproduce the original flake locally, since it fails only about 0.35% of the time on other JVMs.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code (link removed)

NOTE: some AI-generated links and images were removed to ensure safety.


PR by Bits - View session in Datadog • Open in Slack

Comment @DataDog to request changes

waitsForContextCreationBeforeClosingTheHandle sampled the update
thread's state twice: once in the wait loop and again in the assertion.
A parked StampedLock writer can briefly leave WAITING between samples
(for example on a spurious wakeup, before it re-parks), so the second
check could fail even though the loop had already seen the thread
blocked in initOrUpdateWafHandle. Assert the value the loop observed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: cgcote <5115638+cgcote@users.noreply.github.com>
@datadog-prod-us1-4

Copy link
Copy Markdown

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

@datadog-prod-us1-4

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.83 s 14.59 s [+1.0%; +2.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.58 s 13.62 s [-1.0%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.24 s 17.04 s [+0.4%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent 17.08 s 17.17 s [-1.4%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.91 s 16.78 s [-0.5%; +1.9%] (no difference)
startup:petclinic:sca:Agent 17.23 s 17.20 s [-0.6%; +0.9%] (no difference)
startup:petclinic:tracing:Agent 16.27 s 16.38 s [-1.5%; +0.2%] (no difference)

Commit: fccad2cc · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@cgcote
cgcote marked this pull request as ready for review October 30, 2026 11:53
@cgcote
cgcote requested a review from a team as a code owner October 30, 2026 11:53
@cgcote
cgcote requested review from claponcet and jandro996 October 30, 2026 11:53
@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@dd-octo-sts dd-octo-sts Bot added tag: ai generated Largely based on code generated by an AI or LLM and removed Bits AI labels Oct 30, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T11:54:32.084212Z fccad2c Draft marked ready
🔒 Security Review ✅ Completed 2026-10-30T11:55:22.576921Z fccad2c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Preserving the wait loop’s observed blocked state removes the second-sample race while retaining the timeout, reload-completion, and handle-liveness assertions. No reportable regression was identified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit fccad2c

@cgcote cgcote added the comp: asm waf Application Security Management (WAF) label Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) tag: ai generated Largely based on code generated by an AI or LLM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants