Fix trace and StatsD connections over Windows named pipes by Lewis-E · Pull Request #12898 · DataDog/dd-trace-java

Lewis-E · 2026-10-15T21:05:31Z

What Does This Do

This fixes the tracer's connections to the Datadog Agent over Windows named pipes. Currently, Java apps on Azure Functions will fight over a shared UDP connection, leading to missing traces and metrics (SVLS-8247). Additionally, we're covering edge cases around Named Pipe connections, as they behave differently than UDP. During Azure Functions restarts, the underlying infrastructure (Azure App Servives, IIS) can have overlapping instances, which can cause incorrect connections and long failure states.

Traces, stats, remote config, and telemetry (HTTP over the trace agent pipe), commits 4 and 5:

NamedPipeSocketFactory opens a new pipe handle for every socket instead of returning one shared NamedPipeSocket per pipe. Like .NET's NamedPipeClientStream.Connect, it retries opening a busy or missing pipe for up to 500 ms, the .NET default DD_TRACE_PIPE_TIMEOUT_MS.
OkHttp clients that use a named pipe don't keep idle connections, so each request opens and closes its own pipe connection. For named pipes this replaces the maxRunningRequests connection pool, while the dispatcher limits stay as they are.

Tracer metrics (StatsD), commits 1 to 3 and 6:

The StatsD connection warns, instead of logging at debug, when it gives up connecting.
The StatsD connection keeps retrying its first connection forever, doubling the delay from 10 s to a 60 s cap. Before, it gave up after 20 attempts. It warns on the first failure and again at most every 5 minutes while the failure lasts.
Over a named pipe, the StatsD connection recreates its client once writes have kept failing for at least 30 s: 3 or more errors, each less than 60 s after the previous one. The UDP and Unix socket clients already send to their address on every write or reconnect themselves, so they are left alone.
DDAgentStatsDConnection uses port 0 when the config sets a named pipe and no port. Before, it threw NullPointerException.

Motivation

Customers on two Azure products hit these bugs.

Azure Functions with named pipes (SVLS-8247): Java tracing over a named pipe for Functions on a shared Windows plan need per-function pipes.

Azure App Service overlapping restarts (SVLS-9057, escalation SLES-2826): during an overlapped IIS recycle, the old and new worker share the DogStatsD pipe

Additional Notes

A Windows pipe server can create several instances of one pipe name. The .NET tracer opens a new pipe stream for every request to the same trace agent and serverless-compat pipes, and it doesn't hang. The StatsD client opens its own pipe, with a different name, so the shared HTTP socket never protected it.
The port fix fills in 0 only when no port is configured, so setups that set both a pipe and a port behave as before.
NamedPipeSocketFactory has a package-private File constructor so that its test can use a regular file as the pipe on any OS.

Tests:

NamedPipeSocketFactoryTest covers a new socket per connection
DDAgentStatsDConnectionTest covers the retry delay, the write-failure streak, and the named-pipe port
A full E2E test with these changes and the related PRs for the dogstatsd client and compat layer changes on Azure Functions passed with all expected trace/metric/request counts as expected, including under burst conditions.

#12899 (App Service agent supervision and logging) is stacked on this branch.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: SVLS-8247 SVLS-9057

dd-octo-sts · 2026-10-15T21:33:03Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.02 s	14.01 s	[-0.6%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent	12.94 s	13.05 s	[-1.6%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent	17.17 s	17.07 s	[-0.2%; +1.4%] (no difference)
startup:petclinic:iast:Agent	17.06 s	17.16 s	[-1.5%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	16.61 s	16.94 s	[-2.9%; -1.0%] (significantly better)
startup:petclinic:sca:Agent	17.12 s	17.09 s	[-0.6%; +0.9%] (no difference)
startup:petclinic:tracing:Agent	16.20 s	16.22 s	[-1.0%; +0.8%] (no difference)

Commit: 0c39b526 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

When the StatsD connection ran out of connect retries it logged at debug and stopped for the life of the JVM, so default customer logs showed no sign that tracer metrics were never sent. Warn instead, with the StatsD address, the number of attempts, and the last error.

When the DogStatsD pipe or port was missing at startup, the StatsD connection retried 20 times, 10 seconds apart, then gave up for the life of the JVM, so a DogStatsD that came up later was never used. In Azure App Service this happens when the tracer starts before DogStatsD listens. - Retry forever, doubling the delay from 10 seconds to a cap of 60 seconds. Drop the jitter on retries so the delay stays within the cap. - Warn on the first failure, and again at most every 5 minutes once the delay reaches the cap, so a lasting failure stays visible without debug logs. Log other attempts at debug, and log once at info when a connection succeeds after failures.

chatgpt-codex-connector · 2026-10-29T21:00:28Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-29T21:03:32.865758Z	`2254f7d`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-datadog-prod-us1-2

The pipe-recovery trigger also counts generic StatsD reporting exceptions, so recurring non-transport errors can repeatedly tear down a healthy shared connection. Recovery should be driven by pipe write failures.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 2254f7d}

The StatsD client opens its named pipe once and never reopens it. After the DogStatsD it was writing to went away, every write failed for the life of the JVM, even once a new DogStatsD served the same pipe name. In an Azure App Service overlapping restart, tracer metrics were still missing 50 minutes after the old worker's DogStatsD exited. - Track the client's own send failures, for named pipes only. They use a separate error handler, so errors reported through StatsDClient.error() are still logged and counted by handle() but never trigger a recreate. The UDP and Unix datagram clients send to their address on every write, and the Unix stream client reconnects itself, so recreating them would only drop queued metrics, for example while an overloaded agent fills the socket buffer. - Errors more than 60 seconds apart start a new streak; a streak of at least 3 errors spanning at least 30 seconds triggers a recreate. The 60 second gap is longer than the 30 second tracer health metrics flush, so a dead pipe keeps one streak going, while brief bursts and isolated timeouts under load do not trigger it. - Recreate on the task scheduler, not the client's own thread: close the client and reconnect through the existing connect path, which retries with backoff if the pipe is still missing. - Warn on recreate, rate limited to once every 5 minutes.

NamedPipeSocketFactory kept one NamedPipeSocket per pipe and returned it to every OkHttp connection (#3493). NamedPipeSocket ignores read timeouts, so when OkHttp checks a pooled connection's health it blocks on an unbounded pipe read, and every other agent request (traces, stats, remote config, telemetry) waits behind it on the shared handle. On Azure Functions with DD_TRACE_PIPE_NAME set, the tracer queued spans but never sent or failed a trace upload until the JVM restarted. Open a new pipe handle for each socket, as the .NET tracer does for each request. #3493 shared the handle because a named pipe was thought to allow only one client at a time; that change was untested on Windows, and the .NET tracer's per-request pipe connections show the agent's pipe server accepts several clients. The agent's pipe servers create their next pipe instance only after a client connects, so a request that opens the pipe in between fails with "All pipe instances are busy". Like .NET's NamedPipeClientStream.Connect, retry opening the pipe for up to 500 ms, the .NET tracer's default DD_TRACE_PIPE_TIMEOUT_MS. Java reports a busy pipe, a missing pipe, and other open failures with the same exception, so all of them are retried.

NamedPipeSocket can't time out a read, so when OkHttp checks a pooled connection's health before reusing it, the check can block forever on the pipe. Give named-pipe clients a connection pool that keeps no idle connections, so each request opens and closes its own pipe connection, as the .NET tracer does. For named pipes this replaces the maxRunningRequests connection pool; the dispatcher limits are unchanged.

When a DogStatsD named pipe was set and no port was configured, DDAgentStatsDConnection left the port null, and building the client threw NullPointerException on .port(port). Tracer health metrics and JMXFetch JVM metrics were never sent over a pipe. Use port 0 when a named pipe is set and no port is configured. The client connects to the pipe and ignores the port; port 0 also selects the same packet size and socket settings the tracer uses for Unix domain sockets.

Lewis-E · 2026-10-30T16:45:52Z

@DataDog review

datadog-datadog-prod-us1-2

More details

Windows pipe contention and overlapping Azure worker restarts remain unverified at runtime; confidence in the connection recovery changes is limited to source inspection.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 0c39b52}

datadog-datadog-prod-us1-2

More details

Windows named-pipe contention and recovery during overlapping restarts remain unverified at runtime. The completed static review identified no reportable defect.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 0c39b52}

Lewis-E added type: bug fix Bug fix comp: core Tracer core comp: metrics Metrics labels Oct 15, 2026

This was referenced Oct 15, 2026

[SVLS-8245] Give each Windows Azure Function its own named pipes DataDog/datadog-serverless-compat-java#43

Draft

Recover Azure App Service agents after overlapping restarts #12899

Draft

This comment has been minimized.

Sign in to view

Lewis-E mentioned this pull request Oct 16, 2026

Read dd.dogstatsd.pipe.name for the default named pipe DataDog/java-dogstatsd-client#344

Merged

Lewis-E added 2 commits October 29, 2026 15:45

Lewis-E force-pushed the lewis.lewis/java-named-pipe-statsd-fixes branch 2 times, most recently from 1de72a7 to 2254f7d Compare October 29, 2026 20:50

Lewis-E marked this pull request as ready for review October 29, 2026 21:00

Lewis-E requested review from a team as code owners October 29, 2026 21:00

Lewis-E requested review from ValentinZakharov and mabdinur and removed request for a team October 29, 2026 21:00

datadog-datadog-prod-us1-2 Bot reviewed Oct 29, 2026

View reviewed changes

Comment thread ...s/metrics/metrics-lib/src/main/java/datadog/metrics/impl/statsd/DDAgentStatsDConnection.java Resolved

Lewis-E added 4 commits October 30, 2026 12:44

Lewis-E force-pushed the lewis.lewis/java-named-pipe-statsd-fixes branch from 2254f7d to 0c39b52 Compare October 30, 2026 16:45

Lewis-E requested a review from a team as a code owner October 30, 2026 16:45

Lewis-E requested review from bric3 and removed request for a team October 30, 2026 16:45

datadog-datadog-prod-us1-2 Bot reviewed Oct 30, 2026

View reviewed changes

Navigation Menu

Repository navigation

Fix trace and StatsD connections over Windows named pipes - #12898

Fix trace and StatsD connections over Windows named pipes#12898
Lewis-E wants to merge 6 commits into
masterfrom
lewis.lewis/java-named-pipe-statsd-fixes

Lewis-E commented Oct 15, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

Uh oh!

Lewis-E commented Oct 30, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Lewis-E commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Tests:

Contributor Checklist

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

chatgpt-codex-connector Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Lewis-E commented Oct 30, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Lewis-E commented Oct 15, 2026 •

edited

Loading

dd-octo-sts Bot commented Oct 15, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 29, 2026 •

edited

Loading