feat(llmobs): emit gen_ai.* APM tags when LLMObs is disabled - #10300
feat(llmobs): emit gen_ai.* APM tags when LLMObs is disabled#10300mz1119 wants to merge 2 commits into
Conversation
The gen_ai.* attributes only reached the APM span through the LLMObs span
processor, which runs on span finish and only for spans the tagger registered.
With DD_LLMOBS_ENABLED unset, `LLMObsPlugin.configure` unsubscribed the whole
LLMObs layer, so APM-only users got no searchable model, provider, application,
conversation or token usage at all.
Keep the LLMObs plugins subscribed when only LLMObs itself is off, and run a
reduced path in that mode:
start emits gen_ai.operation.name, gen_ai.request.model,
gen_ai.provider.name, gen_ai.application.name and
gen_ai.conversation.id from the span register options every
integration already builds
asyncEnd emits gen_ai.usage.* from the new `getGenAiApmUsageMetrics` hook
An integration opt-out (`tracer.use(<name>, { llmobs: false })`) still disables
the layer entirely.
The register options cover the scalars for every integration from one call site,
so no per-integration tag code is needed; usage has to be read off the response,
so it is per integration as in dd-trace-py: openai, anthropic, google-genai,
vertexai, and bedrock (from its response headers, since the body is only parsed
for the LLMObs payload). The vercel-ai, langchain and claude-agent-sdk usage
paths are entangled with payload construction and report scalars only for now.
The auxiliary subscribers that aggregate streamed chunks stay inert while LLMObs
is disabled, so the reduced path costs one small object and a few setTag calls
per LLM call and never the message-body work.
`setGenAiApmTags` moves into llmobs/gen-ai-tags.js, shared by the span processor
and the plugin base so both paths normalize a facet value the same way, and the
tagger's camelCase metric mapping becomes METRIC_KEY_ALIASES so the disabled
path can accept the spelling integrations extract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Overall package sizeSelf size: 8.89 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 4770cc4 | Docs | View more details | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## max.zhang/llmobs-gen-ai-apm-tags #10300 +/- ##
====================================================================
+ Coverage 98.68% 98.73% +0.05%
====================================================================
Files 1006 1009 +3
Lines 158274 158869 +595
Branches 74 74
====================================================================
+ Hits 156188 156856 +668
+ Misses 2086 2013 -73
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 807380d523
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-09-11 18:47:33 Comparing candidate commit 4770cc4 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2305 metrics, 8 unstable metrics.
|
The reduced path read everything from the span register options at start, so any value an integration only resolves later was missing or wrong: - Vercel AI's dd-telemetry path carries the model in the span attributes, not in the register options, so model-backed spans reported the `custom` fallback. - Anthropic and google-genai build the streamed response in their own chunk subscriber, which the reduced path had inert, so streamed calls reported no `gen_ai.usage.*` at all. - The claude-agent-sdk query span learns its session id from the resolved stream, so the root agent span never got `gen_ai.conversation.id`. - A claude-agent-sdk tool span that launches a subagent is promoted from `tool` to `agent` once the nested step records its id, which the reduced path skipped, so those wrappers reported `gen_ai.operation.name=tool`. Generalize the end hook from `getGenAiApmUsageMetrics` to `getGenAiApmEndTags`, returning any `gen_ai.*` value resolved once the operation finished, including a corrected span kind. `updateGenAiApmTags` writes only the fields present, so a late update cannot re-default a model the start path already resolved. The two chunk subscribers now accumulate token usage while LLMObs is disabled and nothing else, leaving `ctx.result` to the LLMObs path since the google-genai tracing plugin reads it for `google_genai.response.model`. Anthropic's usage merge is shared with the enabled aggregation rather than duplicated. Reported by Codex on #10300. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4770cc466b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| getGenAiApmEndTags (ctx) { | ||
| const response = ctx.result?.data | ||
| return { metrics: response && this._extractMetrics(response) } |
There was a problem hiding this comment.
When LLMObs is disabled and OpenAI returns a model name different from the request value (for example, an alias resolves to a dated model version), this end hook updates only usage, so gen_ai.request.model retains the less accurate request model. The enabled path explicitly replaces it with response.model at lines 110–113; include that value in these end tags so the new APM-only path reports the same resolved model.
Useful? React with 👍 / 👎.
| spanKind: 'llm', | ||
| modelName: request.params.modelId.toLowerCase(), | ||
| modelProvider: 'amazon_bedrock', | ||
| metrics: extractTokens({ tokensFromHeaders, usage: {} }), |
There was a problem hiding this comment.
When LLMObs is disabled for Bedrock converse or converseStream, this always passes an empty body-usage object and relies on token-count headers. The recorded Converse responses contain usage only in response.usage or the stream's metadata.usage event and do not carry the x-amzn-bedrock-*-token-count headers, so these spans emit every gen_ai.usage.* metric as zero. Preserve lightweight Converse stream usage and use the already-deserialized response usage for non-stream calls.
Useful? React with 👍 / 👎.
| processedKey = REASONING_OUTPUT_TOKENS_METRIC_KEY | ||
| break | ||
| } | ||
| const processedKey = METRIC_KEY_ALIASES[key] ?? key |
There was a problem hiding this comment.
When a public LLMObs.annotate() call supplies a valid custom metric named constructor, toString, valueOf, or __proto__, this lookup resolves an inherited property from Object.prototype instead of falling back to the original metric name. The resulting event stores the value under a coerced function/object string rather than the user-provided key, whereas the previous switch preserved arbitrary custom names; use an own-property check or a null-prototype alias map.
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Emits the
gen_ai.*APM attributes when LLM Observability is disabled.LLMObsPlugin.configureunsubscribed the whole LLMObs layer whenDD_LLMOBS_ENABLEDwas off, so there was no hook left to tag from. It now stays subscribed in that case and runs a reduced path:start→gen_ai.operation.name,gen_ai.request.model,gen_ai.provider.name,gen_ai.application.name,gen_ai.conversation.id, from the span register options every integration already builds.asyncEnd→ any value an integration can only resolve once the operation finished (token usage, a session id the response carries, a corrected span kind), from a new per-integrationgetGenAiApmEndTagshook.An integration opt-out (
tracer.use(<name>, { llmobs: false })) still disables the layer entirely.Motivation
Stacked on #10261, which only covers the LLMObs-enabled path (the span processor tags on finish, and only for registered spans). APM-only users got no searchable model, provider, application, conversation or token usage at all. Matches DataDog/dd-trace-py#20083 and DataDog/dd-trace-java#12450.
Additional Notes
Merge after #10261.
setTagcalls per LLM call. The subscribers that aggregate streamed chunks accumulate token usage and nothing else, so no message-body work happens andctx.resultis left to the LLMObs path.setGenAiApmTagsmoved tollmobs/gen-ai-tags.js, shared by the span processor and the plugin base so both paths normalize a facet value identically.updateGenAiApmTagswrites only the fields present, so a late value cannot re-default a model the start path already resolved. The tagger's camelCase metric switch becameMETRIC_KEY_ALIASESso the disabled path accepts the spelling integrations extract.🤖 Generated with Claude Code