Skip to content

Expose OpenTelemetry metrics shutdown through MeterProvider - #12317

Open
bm1549 wants to merge 9 commits into
masterfrom
brian.marks/otel-metrics-lifecycle
Open

Expose OpenTelemetry metrics shutdown through MeterProvider#12317
bm1549 wants to merge 9 commits into
masterfrom
brian.marks/otel-metrics-lifecycle

Conversation

@bm1549

@bm1549 bm1549 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Adds a Datadog lifecycle extension to the OpenTelemetry MeterProvider returned by GlobalOpenTelemetry:

MeterProvider meterProvider = GlobalOpenTelemetry.get().getMeterProvider();
DatadogMeterProvider datadogMeterProvider = (DatadogMeterProvider) meterProvider;
CompletableResultCode result = datadogMeterProvider.shutdown().join(10, TimeUnit.SECONDS);

shutdown() performs a final export and stops the Datadog OpenTelemetry metrics pipeline. Java does not expose a public forceFlush() extension. DatadogMeterProvider is independent of the OpenTelemetry SDK SdkMeterProvider.

Repeated calls observe the first shutdown result through isolated result views. A disabled or unavailable pipeline succeeds as a no-op. Existing internal metric flushing remains unchanged.

Motivation

Short-lived Java processes need a supported way to wait for pending custom metrics before exit. Exposing shutdown through the underlying meter provider keeps usage close to the OpenTelemetry API.

Tests

  • Datadog metrics API unit tests, Javadoc, and SpotBugs
  • OpenTelemetry 1.47 forked lifecycle test
  • Spotless formatting checks

Contributor Checklist

  • Follow the repository title and labeling conventions
  • Add focused deterministic tests
  • Keep the public API shutdown-only
  • Use merge queue after approval

Jira ticket: N/A

@datadog-official

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 13.96 s [-0.8%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 12.93 s 13.03 s [-1.5%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 16.93 s 16.31 s [-0.5%; +8.1%] (no difference)
startup:petclinic:iast:Agent 16.29 s 16.89 s [-7.6%; +0.5%] (no difference)
startup:petclinic:profiling:Agent 16.25 s 16.55 s [-6.1%; +2.5%] (no difference)
startup:petclinic:sca:Agent 16.84 s 16.69 s [-0.1%; +2.0%] (no difference)
startup:petclinic:tracing:Agent 15.88 s 16.06 s [-2.3%; +0.0%] (no difference)

Commit: dc3deb99 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh

dougqh commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Note from Claude (reviewing on behalf of @dougqh): not a blocker, but worth thinking through before this lands.

Neither OtlpMetricsService.forceFlush() nor shutdown() puts a deadline on the returned CompletableFuture<Boolean>. The only bound in the chain is the transport's own retry policy:

  • OtlpHttpSender retries via HttpRetryPolicy.Factory(5, 100, 2.0, true) — up to 5 retries, and the trailing true means InterruptedIOException (covers SocketTimeoutException) is retried, not just hard connection failures.
  • Each attempt is bounded by otlp.metrics.timeout, which defaults to 10s.
  • Worst case under a degraded (not dead) network — connects but reads stall — that's roughly 6 attempts × ~10s + backoff ≈ 60+ seconds before export()/finishShutdown() returns and the future completes.

For the automatic path (CoreTracer#close()) this is harmless today, since it calls shutdown() fire-and-forget without awaiting the future.

But it cuts against the PR's own motivation: a short-lived job calling OpenTelemetryMetrics.shutdown().join()/.get() to make sure metrics leave before the process exits could block up to ~a minute under a merely-slow network, which is a rough trade for something meant to bound a short-lived job's exit.

Worth considering wrapping the flush/shutdown future with orTimeout(...)/completeOnTimeout(...) (or using a tighter/no-retry policy specifically for the shutdown-triggered final export) so callers of the new public API get a predictable upper bound independent of the transport's retry behavior.

@mabdinur
mabdinur requested a review from mhlidd September 1, 2026 20:45
@mabdinur
mabdinur marked this pull request as ready for review September 1, 2026 21:57
@mabdinur
mabdinur requested review from a team as code owners September 1, 2026 21:57
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T21:04:45.940673Z dc3deb9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The tracer shutdown path starts the final export on a daemon thread but does not wait for it. The DDTracer wrapper also returns unavailable for both new lifecycle calls instead of forwarding them.

Open Bits AI session

🤖 Datadog Autotest · Commit 1b4200c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread dd-trace-api/src/main/java/datadog/trace/api/internal/InternalTracer.java Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

OtlpMetricsService.INSTANCE.shutdown();

P1 Badge Await metrics shutdown from CoreTracer.close

When CoreTracer.close() runs, especially from dd-tracer-shutdown-hook, this call now only enqueues finishShutdown and discards its future. The exporter uses a daemon AgentThreadFactory, so the JVM may terminate after the hook returns while the final send is still running; manual close likewise returns before sender.shutdown() completes. The previous implementation closed the sender synchronously, so wait for this future with a bounded timeout before returning.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dd-trace-api/src/main/java/datadog/trace/api/internal/InternalTracer.java Outdated
@mabdinur
mabdinur force-pushed the brian.marks/otel-metrics-lifecycle branch from 1b4200c to 7e11776 Compare September 1, 2026 23:25
@mabdinur

mabdinur commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@dougqh On the deadline question: the future intentionally has no built-in timeout. Callers can bound their wait with get(timeout, unit), which preserves Java 8 support and does not imply that the in-flight transport was cancelled. CoreTracer.close() now applies a 2.5-second bound.

@mabdinur mabdinur left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Datadog lifecycle implementation, addressed feedback, and focused tests. CI is green.

@mhlidd
mhlidd requested a review from mcculls September 2, 2026 19:05
Comment thread dd-trace-api/src/main/java/datadog/trace/api/metrics/OpenTelemetryMetrics.java Outdated
Comment thread dd-trace-api/src/main/java/datadog/trace/api/internal/InternalTracer.java Outdated
Comment thread dd-trace-api/src/main/java/datadog/trace/api/metrics/OpenTelemetryMetrics.java Outdated
Comment thread dd-trace-api/src/main/java/datadog/trace/api/metrics/OpenTelemetryMetrics.java Outdated

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid using CompletableFuture in a public trace/metrics API

I've suggested some alternatives in other comments

Comment thread dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated
@bm1549
bm1549 requested a review from a team as a code owner September 4, 2026 16:10
@bm1549
bm1549 requested review from vandonr and removed request for a team September 4, 2026 16:10
@bm1549 bm1549 changed the title Add OpenTelemetry metrics lifecycle controls Expose OpenTelemetry metrics shutdown through MeterProvider Sep 4, 2026
@bm1549
bm1549 marked this pull request as draft September 4, 2026 16:18

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

A callback on one shutdown result can block completion of later shutdown results. This behavior breaks the repeated-call result contract.

Open Bits AI session

🤖 Datadog Autotest · Commit cd8ab82 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@mabdinur
mabdinur requested review from mcculls and mhlidd September 8, 2026 20:39
@mabdinur
mabdinur marked this pull request as ready for review September 8, 2026 20:58

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The shutdown path returns separate views of one shared result. It exports the last metrics before it stops the sender and executor.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit dc3deb9 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: metrics Metrics inst: opentelemetry OpenTelemetry instrumentation tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants