Skip to content

Tolerate servlet responses without Servlet 3.0 header accessors (quick fix) - #12691

Draft
dougqh wants to merge 1 commit into
masterfrom
dougqh/servlet3-response-headers-unsupported
Draft

dougqh wants to merge 1 commit into
masterfrom
dougqh/servlet3-response-headers-unsupported

Conversation

@dougqh

@dougqh dougqh commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Makes the servlet 3.0 response-header reader tolerate response classes that do not implement the Servlet 3.0 header accessors.

HttpServletExtractAdapter.Response called HttpServletResponse.getHeaderNames() / getHeader(...) on every response. A response class compiled against Servlet 2.5 (for example an old wrapper bundled in a webapp) does not implement them, so the call throws AbstractMethodError. It escaped through HttpServerDecorator.callIGCallbackResponseAndHeaders, was caught and reported as "Failed to decorate span on response", and stopped the response-header callbacks on every response of that class.

Now Response.forEachKey:

  • Healthy path unchanged: reads the headers directly; the only added cost is a plain static boolean read (anyUnsupported).
  • First failure of a class: catches AbstractMethodError from the two accessor calls only, sets the flag, and marks that response class unsupported in a ClassValue.
  • After the flag is set: a per-class probe (getMethod plus Modifier.isAbstract) decides whether a class supports header access, so other bad classes never throw.
  • Fallback: for an unsupported class it unwraps ServletResponseWrapper (bounded depth) and reads headers from the nearest delegate that supports them; if there is none, it visits no headers.

The new nested HeaderAccessors helper is added to both helper-class lists that inject HttpServletExtractAdapter (Servlet3Instrumentation and AsyncContextInstrumentation).

Adds HttpServletExtractAdapterTest (JUnit 5, 6 cases), using an ASM-generated response class that implements HttpServletResponse but none of its methods, so the real AbstractMethodError is exercised.

Motivation

Instrumentation telemetry shows this at HttpServletExtractAdapter$Response.getHeaderNames:48: about 10.5k events a day, steady at 400-500 an hour, on tracers 1.58 to 1.66, mostly Java 8 services (a few pauth/pentitlements/poauth-style services, azkaban-*, wc-commerce-*). Reports aggregate repeats, so the real throw count is higher.

Muzzle cannot catch this: it verifies that HttpServletResponse.getHeaderNames() exists in the API, and it does. The failure is that a particular runtime implementation does not honor it.

Additional Notes

For the AppSec/IAST reviewers, please look at this:

  • In callIGCallbackResponseAndHeaders the responseStarted(status) callback fires before the header visit, so it was not affected. What was skipped is the response-header callbacks and igKeyClassifier.done() (responseHeaderDone).
  • With this change, for a response with no usable accessors the visitor returns without visiting, so done() still runs with no headers (like an empty header set). The servlet 2.x decorator instead returns a null responseGetter() ("There is no way to access the headers"), which skips the header callbacks and done() entirely. I went with "empty headers" so responseHeaderDone still fires. Happy to switch to the 2.x behavior if AppSec prefers it.
  • The catch is deliberately limited to the accessor calls, not the classifier.accept callback: an AbstractMethodError thrown by callback code says nothing about the response class and must not mark it unsupported (there is a test for this). If getHeader fails part-way through, it stops instead of retrying, to avoid re-emitting keys.

Notes:

  • The Jakarta servlet modules cannot hit this (a Servlet 5 response cannot be compiled against a 2.5 API), and Liberty has its own adapter copies, so only this module changes.
  • Whether the failing classes in the wild are ServletResponseWrapper subclasses (so the unwrap fallback helps) is a guess: the telemetry only shows our adapter frames.
  • The flag is a plain (non-volatile) field on purpose: it only goes false to true, and a stale read just takes the direct path and hits the handled catch. No benchmark is included.
  • Related idea captured separately as APMLP-1890 (a global tripped flag guarding a per-key store).

Contributor Checklist

Jira ticket: N/A

🤖 Generated with Claude Code

HttpServletExtractAdapter.Response called HttpServletResponse.getHeaderNames()
on every response. A response class compiled against Servlet 2.5 (for example
an old wrapper) does not implement it, so the call threw AbstractMethodError
and aborted the AppSec/IAST response-header callbacks on every response of
that class.

Read directly until a class fails, then remember unsupported classes in a
ClassValue (probing for an abstract interface method) so they are skipped
without throwing, and read headers from a wrapped delegate when there is one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@dougqh dougqh added type: bug fix Bug fix tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM inst:servlet Servlet instrumentation labels Sep 29, 2026
@datadog-datadog-prod-us1-2

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 68.82% (+9.58%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5a1c21e | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.95 s [-0.4%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.94 s 13.03 s [-1.4%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 16.96 s 16.92 s [-1.0%; +1.4%] (no difference)
startup:petclinic:iast:Agent 16.82 s 16.98 s [-1.7%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 16.79 s 16.24 s [-0.9%; +7.7%] (no difference)
startup:petclinic:sca:Agent 16.46 s 16.38 s [-5.7%; +6.7%] (unstable)
startup:petclinic:tracing:Agent 16.10 s 16.20 s [-1.5%; +0.4%] (no difference)

Commit: 5a1c21e5 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the title Tolerate servlet responses without Servlet 3.0 header accessors Tolerate servlet responses without Servlet 3.0 header accessors (quick fix) Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst:servlet Servlet instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant