Skip to content

Add block_failure telemetry for Tomcat blocking enforcement failures - #12493

Open
jandro996 wants to merge 12 commits into
masterfrom
block-telemetry-2
Open

jandro996 wants to merge 12 commits into
masterfrom
block-telemetry-2

Conversation

@jandro996

@jandro996 jandro996 commented Sep 15, 2026

Copy link
Copy Markdown
Member

What Does This Do

  • Reports block_failure on the appsec.waf.requests metric when Tomcat's blocking-commit path fails to enforce a decided block, mirroring the Netty implementation (Add block-outcome telemetry for Netty blocking enforcement failures #12316).
  • Adds BlockFailureReporter.tryCommitAndReport(RequestContext, Flow.Action.RequestBlockingAction) in tomcat-common as a shared choke point: commits the blocking response via the registered BlockResponseFunction and, if the commit fails (returns false), reports failure through AppSecContext.reportBlockFailure() (resolved from RequestContext.getData(RequestContextSlot.APPSEC)). Exceptions thrown by the commit attempt are deliberately propagated (not converted into a false return), so the calling advice's suppress = Throwable.class aborts the whole advice and skips its blocking success-path side effects; such exception-based failures are a known telemetry gap shared with the Netty implementation.
  • Wires all Tomcat/GlassFish blocking-commit call sites (5.5/6.0/7.0) through this shared helper: CommitActionInstrumentation (5.5, 7.0), ParsedBodyParametersInstrumentation (5.5, 6.0), ParsePartsInstrumentation (7.0), GlassFishBlockingHelper/GlassFishMultipartInstrumentation (7.0), and TomcatServerInstrumentation (5.5 base).
  • Wires TomcatBlockingHelper.commitBlockingResponse()'s genuinely reachable failure path (GET_OUTPUT_STREAM == null) to also report block_failure, without changing its existing "already committed" / "write threw" true-return semantics (deliberately out of scope).
  • Guards GlassFishBlockingHelper.tryBlock()'s Servlet API fallback branch so reportBlockFailure() only fires when a response was actually available to commit through (i.e. a commit was genuinely attempted), not on the brf == null / fallbackResp == null case where nothing was attempted at all.

Split of TomcatBlockingHelper

  • Extracted tryCommitAndReport()/reportBlockFailure(RequestContext) into a new class, BlockFailureReporter (tomcat-common), that has no dependency on org.apache.catalina.* types.
  • TomcatBlockingHelper keeps only the catalina-typed methods (commitBlockingResponse(), start(), tryWriteWithOutputStream/Writer, the private reportBlockFailure(Request)).
  • Reason: helperClassNames() injects a helper's entire class bytecode, and Gradle's muzzle task validates every reference in that bytecode against each declared library version group. Before the split, instrumentations whose target library range does not guarantee org.apache.catalina.connector.Request/Response presence (e.g. some tomcat-appsec-5.5/6.0/7.0 groups) failed muzzle just from listing TomcatBlockingHelper, even though their advice only ever called the catalina-independent tryCommitAndReport/reportBlockFailure(RequestContext) methods.
  • RequestInstrumentation/ResponseInstrumentation (tomcat-5.5) also needed BlockFailureReporter added to helperClassNames(), since they list TomcatBlockingHelper transitively (via TomcatDecorator) and TomcatBlockingHelper's own reportBlockFailure(Request) still delegates to BlockFailureReporter.reportBlockFailure(reqCtx).

Motivation

APPSEC blocking-outcome telemetry currently only exists for Netty (#12316). This extends coverage to Tomcat so that a WAF-decided block that silently fails to actually block the request (e.g. response already committed, BlockResponseFunction throwing) is observable via the block_failure tag on appsec.waf.requests, instead of looking like a successful block.

Additional Notes

  • tomcat-common's new test dependencies (tomcat:catalina/tomcat-coyote/tomcat-util, javax.servlet:servlet-api:2.4, JUnit 5, Mockito) required regenerating gradle.lockfile for that module (tomcat-appsec-6.0's lockfile needs no change: its new implementation project(...) dependency adds no external dependency to that module's classpath).
  • Fixed a pre-existing bug in the spring-webmvc test fixtures (TestSpringBlockResponseFunction): it was unconditionally returning true/false instead of delegating to the previously registered BlockResponseFunction when RequestContextHolder is not yet populated by Spring, which clobbered Tomcat's real BlockResponseFunction in latestDepTest runs and surfaced as 10 unrelated test failures once tryCommitAndReport started being exercised on that path.

Contributor Checklist

Jira ticket: APPSEC-62696

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

Report block_failure on appsec.waf.requests when Tomcat's blocking-commit
path fails to enforce a decided block, mirroring the Netty implementation
(#12316). Adds a shared TomcatBlockingHelper.tryCommitAndReport choke point
used by all Tomcat/GlassFish blocking-commit call sites (5.5/6.0/7.0), and
guards it against exceptions thrown by the registered BlockResponseFunction.
…lemetry

Extract tryCommitAndReport()/reportBlockFailure(RequestContext) into a new
catalina-independent BlockFailureReporter class in tomcat-common, since
helperClassNames() injects the whole class and muzzle validates all its
references against every declared library version - including the
catalina-independent code path that doesn't need it.
@jandro996 jandro996 added type: feature Enhancements and improvements comp: asm waf Application Security Management (WAF) inst: others All other instrumentations labels Sep 15, 2026
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-15T20:20:59.237999Z 6fc0248 Draft marked ready
🔒 Security Review Completed 2026-09-15T20:23:26.239841Z 6fc0248 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: b6b1546544

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: b6b1546544

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Sep 15, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 68.82% (+9.72%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d0b8dcf | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.72 s 14.75 s [-1.2%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent 13.57 s 13.65 s [-1.2%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.00 s 16.92 s [-0.6%; +1.5%] (no difference)
startup:petclinic:iast:Agent 16.93 s 16.96 s [-1.0%; +0.7%] (no difference)
startup:petclinic:profiling:Agent 16.62 s 16.86 s [-2.6%; -0.2%] (maybe better)
startup:petclinic:sca:Agent 16.92 s 16.55 s [+1.3%; +3.2%] (significantly worse)
startup:petclinic:tracing:Agent 16.19 s 16.06 s [-0.3%; +1.9%] (no difference)

Commit: d0b8dcfa · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…Function

TestSpringBlockResponseFunction now delegates to the previously registered
BlockResponseFunction (Tomcat's own, in production tests) instead of
returning true/false unconditionally when RequestContextHolder is not yet
populated by Spring, fixing 10 latestDepTest failures exposed by the
tryCommitAndReport refactor.
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: f3cbd0a658

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f3cbd0a658

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…fallback response

Guard the Servlet API fallback branch of tryBlock() so reportBlockFailure()
only fires when a response was actually available to commit through (i.e. a
commit was genuinely attempted). Previously it reported unconditionally
whenever the fallback commit failed, including the brf == null / fallbackResp
== null case where nothing was attempted at all.
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f34f16feea

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f34f16feea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ailureReporter

Remove the catch(Exception) around brf.tryCommitBlockingResponse() in
tryCommitAndReport() so it matches the Netty reference implementation
(PR #12316), which only branches on the boolean return value.

Several call sites (CommitActionInstrumentation in 5.5/7.0,
ParsePartsInstrumentation, ParsedBodyParametersInstrumentation) rely on
their enclosing advice's suppress = Throwable.class to abort the whole
method when the commit throws, gating success-path side effects (closing
the connection, injecting a BlockingException, marking the segment
effectively blocked). Swallowing the exception inside tryCommitAndReport
made those side effects run unconditionally even when nothing was
actually committed to the client. Removing the catch restores the
pre-existing control flow; exception-based commit failures are now a
known gap not reported to block_failure telemetry, same as Netty.

Addresses Codex review comment on PR #12493.
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 1e718e1484

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1e718e1484

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jandro996
jandro996 marked this pull request as ready for review September 15, 2026 20:16
@jandro996
jandro996 requested a review from a team as a code owner September 15, 2026 20:16
@jandro996
jandro996 requested a review from a team as a code owner September 15, 2026 20:16
@jandro996
jandro996 requested review from AlexeyKuznetsov-DD and jordan-wong and removed request for a team September 15, 2026 20:16

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

All changed Tomcat and GlassFish commit paths use the shared failure reporter. Each affected instrumentation includes the required helper.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 6fc0248 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

testImplementation group: 'tomcat', name: 'tomcat-util', version: tomcatVersion
testImplementation group: 'javax.servlet', name: 'servlet-api', version: '2.4'
testImplementation libs.bundles.junit5
testImplementation libs.bundles.mockito

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since build files changed, please regenerate lock files too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the advice! Fixed in ad61aa5


dependencies {
compileOnly group: 'org.apache.tomcat', name: 'coyote', version: '6.0.53'
implementation project(':dd-java-agent:instrumentation:tomcat:tomcat-common')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, probably lock files should be regenerated.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regenerated but no-op diff for this file, it's a project dependency

Regenerate the lock file to reflect the testImplementation
dependencies added in TODO-9 (catalina/coyote/util, servlet-api,
JUnit 5, Mockito) for the sun.misc.Unsafe-based test. No change
needed for tomcat-appsec-6.0's lockfile: its new project(':...:tomcat-common')
dependency does not add any external dependency to its classpath.

Addresses review comments from AlexeyKuznetsov-DD on PR #12493.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) inst: others All other instrumentations type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants