Skip to content

skill(apm-integrations): Message Queues guidance - #12293

Open
jordan-wong wants to merge 6 commits into
masterfrom
skill/messaging-per-message-span-pattern-v2
Open

jordan-wong wants to merge 6 commits into
masterfrom
skill/messaging-per-message-span-pattern-v2

Conversation

@jordan-wong

@jordan-wong jordan-wong commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Changes

Adds two rules to .agents/skills/apm-integrations/references/advice-class.md:

  1. Per-item span pattern for batch-consume ops — a call returning a batch (e.g. a broker poll returning N records) should wrap the returned Iterable/Iterator/List so each item gets its own span, not the batch call itself.
  2. Callback-overload span-finish rule — if one advice matches both a sync method and a callback-taking sibling (e.g. send(Message) / send(Message, CompletionListener)), it must finish the span from the callback when present, not unconditionally at exit — otherwise the async variant's span is truncated and its errors are lost.

Source

From a blind-regen research cycle on JMS (#11170) and kafka-clients (#12361): the batch pattern is already correct in kafka-clients-0.11 but undocumented; the callback bug was observed live in the JMS regen.

Toolkit research context

apm-instrumentation-toolkit Java eval-research: blind-regenerates existing instrumentation to find toolkit-guidance gaps, encodes fixes here. Docs-only.

…sume operations

When a client API returns a batch of items from one call (e.g. a message
broker's poll returning N records), spanning the batch call itself prevents
attaching per-item follow-on work to the item that triggered it. Document
the wrap-the-iterable pattern already used by kafka-clients (TracingIterable/
TracingIterator/TracingList/TracingListIterator) as the standard for any
future batch-consume instrumentation, not just messaging libraries.
@dd-octo-sts

dd-octo-sts Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.79 s 14.73 s [-0.3%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 13.68 s 13.72 s [-0.9%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 16.98 s 16.91 s [-0.6%; +1.5%] (no difference)
startup:petclinic:iast:Agent 16.87 s 16.97 s [-1.3%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.67 s 16.20 s [-1.6%; +7.4%] (no difference)
startup:petclinic:sca:Agent 16.91 s 16.72 s [+0.3%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent 16.14 s 16.13 s [-1.0%; +1.1%] (no difference)

Commit: 6ae5db83 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@jordan-wong
jordan-wong marked this pull request as ready for review September 1, 2026 13:02
@jordan-wong
jordan-wong requested a review from a team as a code owner September 1, 2026 13:02
@jordan-wong
jordan-wong requested review from jandro996 and removed request for a team September 1, 2026 13:02
@dd-octo-sts

dd-octo-sts Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T13:08:43.324334Z 50ff311 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 50ff311c27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .agents/skills/apm-integrations/references/advice-class.md Outdated
Comment thread .agents/skills/apm-integrations/references/advice-class.md Outdated
Comment thread .agents/skills/apm-integrations/references/advice-class.md Outdated
Comment thread .agents/skills/apm-integrations/references/advice-class.md

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The guidance can remove required client-operation spans from search and bulk integrations. It also gives incorrect timeout behavior for abandoned root iterations in the default legacy context manager.

Open Bits AI session

🤖 Datadog Autotest · Commit 50ff311 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread .agents/skills/apm-integrations/references/advice-class.md Outdated
Comment thread .agents/skills/apm-integrations/references/advice-class.md Outdated
@jordan-wong
jordan-wong removed the request for review from jandro996 September 9, 2026 14:13
@jordan-wong jordan-wong changed the title skill(apm-integrations): document per-item span pattern for batch-consume ops skill(apm-integrations): Message Queues - document per-item span pattern for batch-consume ops Sep 9, 2026
…er covers a callback overload too

If a single advice's method matcher covers both a synchronous overload and
a sibling overload that accepts a completion callback, unconditionally
finishing the span in the exit advice truncates the callback-based call's
span duration and drops any error the callback would report — the span
finishes when the submitting call returns, not when the operation actually
completes. Document the fix: branch on whether the callback argument is
present, and finish the span from the wrapped callback instead of the exit
advice for that call.
@datadog-prod-us1-4

This comment has been minimized.

…idance

Addresses 4 review findings on the "one span per item" rule:

- Scope it to independent-consume domains (message/job queues) only.
  Search and bulk client operations return many elements from ONE
  outbound call and must stay a single span (see
  Elasticsearch7RestClientInstrumentation) -- applying the per-item
  pattern there would misattribute application work and produce
  thousands of spans for one request.
- Add context extraction to the example: each item's span must be
  started from that item's own propagated context (e.g. its headers),
  not the caller's active context -- items in a batch can come from
  different producers. The prior example omitted this entirely.
- Correct the abandoned-iteration claim: under the default legacy
  context manager, activateNext's root iteration scopes ARE
  force-finished by a background cleaner after
  trace.scope.iteration.keep.alive (default 30s) -- the prior text
  said no such timeout exists.
- Note that a List-implementing wrapper (mirroring TracingList) does
  not preserve equals/hashCode -- callers relying on List equality
  need a different wrapper shape or explicit overrides.
@jordan-wong jordan-wong added inst: kafka Kafka instrumentation inst: jms JMS instrumentation labels Sep 23, 2026
@jordan-wong jordan-wong changed the title skill(apm-integrations): Message Queues - document per-item span pattern for batch-consume ops skill(apm-integrations): Message Queues guidance Sep 24, 2026
@jordan-wong
jordan-wong requested a review from mcculls September 24, 2026 11:09

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and clear

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: jms JMS instrumentation inst: kafka Kafka instrumentation type: documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants