Skip to content

Add AWS resource ARN tags to AWS SDK client spans - #12598

Open
pedramsafaei wants to merge 1 commit into
DataDog:masterfrom
pedramsafaei:pedramsafaei/aws-sdk-arn-span-tags
Open

pedramsafaei wants to merge 1 commit into
DataDog:masterfrom
pedramsafaei:pedramsafaei/aws-sdk-arn-span-tags

Conversation

@pedramsafaei

@pedramsafaei pedramsafaei commented Sep 22, 2026 •

Copy link
Copy Markdown

What Does This Do

Tags AWS SDK v1 and v2 client spans with the complete resource identity that is already present in the request, instead of only the short name:

Service Request field New span tags Existing tag kept
SNS TopicArn / TargetArn aws.topic.arn, aws.sns.topic_arn aws.topic.name, topicname
Kinesis StreamARN aws.stream.arn aws.stream.name, streamname
Step Functions stateMachineArn aws.state_machine.arn, statemachinearn –
Step Functions executionArn aws.execution.arn –
Lambda FunctionName aws.function.name, functionname –

No additional network calls are made; every value is read from the request object the instrumentation already inspects. Existing tags, peer.service derivation and Data Streams Monitoring behaviour are unchanged. Nothing is added when the request does not carry the field.

The plain-name variants (aws.sns.topic_arn, statemachinearn, functionname) are the tags dd-trace-py and dd-trace-js already emit for the same calls, so one query now works across the three tracers.

Implementation:

  • internal-api: new constants in InstrumentationTags.
  • aws-java-sdk-2.2: AwsSdkClientDecorator.onSdkRequest reads the fields via SdkRequest.getValueForField. getValueForField matches the API model's member name verbatim, and the Step Functions model declares its members in lowerCamelCase (unlike SNS TopicArn or DynamoDB TableName), so those two fields are read as stateMachineArn and executionArn.
  • aws-java-sdk-1.11: GetterAccess gains getStateMachineArn, getExecutionArn and getFunctionName method handles, resolved the same way as the existing getTableName; AwsSdkClientDecorator.onRequest uses them. The SNS and Kinesis branches tag the ARN they already read.

Motivation

peer.service and the *name tags identify which topic, stream, state machine or function a span talks to, but not whose. Two accounts can each own a payments state machine or a notifications topic, and the short name is ambiguous the moment a service crosses an account or Region boundary. The ARN removes that ambiguity and carries partition, Region and account with it.

This matters for anyone building account-aware dependency maps or joining APM spans with the AWS integration metrics (which are tagged by ARN-derived dimensions), and for cross-account debugging where the short name alone does not say which resource was hit. The tracer already had the ARN in hand for SNS, Kinesis and Step Functions and was discarding it.

The Java tracer was also the odd one out: the Python and JS tracers already emit aws.sns.topic_arn and statemachinearn, and Java emitted neither.

Additional Notes

Testing

Unit and instrumentation tests, all green locally on this branch:

  • aws-java-sdk-2.2: test (41 + 41 across the V0/V1 forks), LegacyAws2ClientForkedTest (23), dsmTest (14). New rows for Step Functions StartExecution / DescribeExecution and Lambda Invoke; SNS rows now assert both ARN tags; the Kinesis DSM tests assert aws.stream.arn.
  • aws-java-sdk-1.11: test (25), forkedTest (25 + 17). Same new rows; the test_before_1_11_106 configuration pins aws-java-sdk-core to 1.11.0 because aws-java-sdk-stepfunctions (a new test dependency) only exists from 1.11.63 and would otherwise pull a newer core than the pinned clients.
  • spotlessApply clean on the touched modules.

How the end-to-end test was run

  • Environment: an EKS cluster (managed node group, EC2 nodes) running the Datadog Helm chart with the node Agent's trace receiver; a real DynamoDB table, Kinesis stream, SNS topic, Step Functions state machine, Lambda function, S3 bucket and RDS for PostgreSQL instance in the same account; IRSA credentials on the workload's service account.
  • Workload: a small Java 21 application (AWS SDK for Java v2 2.29.52, PostgreSQL JDBC 42.7.4) that once per cycle calls SNS Publish, Kinesis PutRecord, Step Functions StartExecution + DescribeExecution, Lambda Invoke, DynamoDB GetItem (by name and by ARN), S3 GetObject and a JDBC SELECT 1, all inside one traced method so the client spans share a trace.
  • Two deployments of the same container image, differing only in the dd-java-agent.jar copied into it: the released 1.66.0 tracer (before) and the jar built from this branch with ./gradlew :dd-java-agent:shadowJar (after). Same node pool, same DD_ENV, distinct DD_VERSION so the two runs are separable in Span Search.
  • Verification: the Datadog Span Search API (/api/v2/spans/events/search) was queried for each run's client spans and the per-span attributes compared key by key; the raw responses are retained. Where the change enables a join to the AWS integration metrics, the corresponding /api/v1/query was run against the same org and the returned scope recorded.

Live before/after on a real EKS workload

A Java 21 workload using AWS SDK v2 (sns, kinesis, sfn, lambda, dynamodb) was run on an EKS cluster under IRSA, once with the stock dd-java-agent 1.66.0 and once with the jar built from this branch (dcb9e6a20c), same image, same node pool, same fixtures. Client spans as returned by the Span Search API (account masked):

Sns.Publish
  before: aws.topic.name=datadog-context-trace-fixture  topicname=datadog-context-trace-fixture
  after : + aws.topic.arn=arn:aws:sns:us-east-1:123456789012:datadog-context-trace-fixture
          + aws.sns.topic_arn=arn:aws:sns:us-east-1:123456789012:datadog-context-trace-fixture

Kinesis.PutRecord
  before: aws.stream.name=DatadogContextTraceFixture-FixtureStream-aiWeiLPpX1jU
  after : + aws.stream.arn=arn:aws:kinesis:us-east-1:123456789012:stream/DatadogContextTraceFixture-FixtureStream-aiWeiLPpX1jU

Sfn.StartExecution
  before: (no resource tags; peer.hostname=states.us-east-1.amazonaws.com only)
  after : + aws.state_machine.arn=arn:aws:states:us-east-1:123456789012:stateMachine:datadog-context-trace-fixture
          + statemachinearn=arn:aws:states:us-east-1:123456789012:stateMachine:datadog-context-trace-fixture

Sfn.DescribeExecution
  before: (no resource tags)
  after : + aws.execution.arn=arn:aws:states:us-east-1:123456789012:execution:datadog-context-trace-fixture:965b1f0d-...

Lambda.Invoke
  before: (no resource tags)
  after : + aws.function.name=datadog-context-normal-fixture
          + functionname=datadog-context-normal-fixture

DynamoDb.GetItem
  before == after (aws.table.name / tablename unchanged, as expected)

Design notes for reviewers

  • Duplicate keys (aws.topic.arn + aws.sns.topic_arn, aws.state_machine.arn + statemachinearn, aws.function.name + functionname) follow the existing convention in this decorator (aws.topic.name + topicname, aws.stream.name + streamname). The namespaced key matches Java's own style; the plain key matches the other tracers. Happy to drop either half if you would rather not carry both.
  • Lambda FunctionName may be a name, a partial ARN or a full ARN; it is tagged verbatim. The SDK v1 decorator only reads getFunctionName when the service is AWSLambda, and the v2 decorator when the service name is lambda, so unrelated request types with a FunctionName member are not affected.
  • Kinesis StreamARN was already read for Data Streams Monitoring; this only additionally sets it on the span, so DSM pathway keys are unchanged (the existing DSM tests still pass with the added tag).
  • EventBridge PutEvents was considered but its bus identity lives inside each entry (EventBusName per PutEventsRequestEntry), not on the request, so it is left for a follow-up if there is interest.

Contributor Checklist

Suggested labels (cannot set as an external contributor): inst: aws sdk, type: feature.

@pedramsafaei
pedramsafaei marked this pull request as ready for review September 23, 2026 16:03
@pedramsafaei
pedramsafaei requested review from a team as code owners September 23, 2026 16:03
@pedramsafaei
pedramsafaei requested review from mhlidd, sarahchen6 and ygree and removed request for a team September 23, 2026 16:03

@mhlidd mhlidd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from an SDK Capabilities POV

String topicName = null;
String topicArn = access.getTopicArn(originalRequest);
if (null != topicArn) {
span.setTag(InstrumentationTags.AWS_TOPIC_ARN, topicArn);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we skip v1 since it was EOLed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to keep at least part of v1 if you are open to it. v1 is EOL from AWS but the tracer still ships and tests this instrumentation, and the users still on it are the ones most likely to have cross-account SNS topics where the bare name is ambiguous. The SNS and Kinesis half of the v1 change is a few lines with no new dependencies and the decorator already reads the ARN and discards it. The Step Functions and Lambda half is what pulled in the stepfunctions test dependency and the core pin, so if the concern is that surface I am happy to drop those two from v1 and keep SNS and Kinesis. But anyways if the team's stance is absolutely no new features on v1 at all, tell me and I will drop v1 entirely here and in #12602.

request.getValueForField("TableName", String.class).ifPresent(name -> setTableName(span, name));

// Step Functions. The service model uses lowerCamelCase member names.
stringField(request, "stateMachineArn", "StateMachineArn")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to check both camelCase and PascalCase? Isn't the model generator supposed to normalize them to PascalCase?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generator does not normalize the member name so getValueForField is generated as an equals chain against the model's member name verbatim. Step Functions' API model defines its members in lowerCamelCase, so sfn's StartExecutionRequest.getValueForField matches "stateMachineArn" (checked in 2.2.0 and 2.15.35), whereas SNS matches "TopicArn" and DynamoDB "TableName". Querying "StateMachineArn" returns empty on sfn, which is why the lowerCamel spelling is there. You are right that probing both is unnecessary though, since the member name comes from the API model and does not vary by SDK version. I will drop the stringField helper and query "stateMachineArn" / "executionArn" directly with a comment on the casing.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. stringField is gone, the two fields are read directly as stateMachineArn / executionArn with a comment on why the casing differs from SNS and DynamoDB.

@ygree ygree left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a functional perspective, the change looks good, but the code footprint can be reduced by removing unnecessary parts, see details in the comments.

return Optional.empty();
}

private static void setTopicArn(AgentSpan span, String arn) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these three two-line methods, which were only used once, be inlined instead?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed. Inlined all three as each had exactly one call site. Left setStreamName and setQueueName as methods since they have two callers each.

@ygree ygree left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IDM approved. There are no blockers, but the code change footprint could likely be reduced.

The AWS SDK v1 and v2 instrumentations already read several resource
identifiers out of the request to derive peer.service, but only emitted
the short names. The ARN itself, which carries the partition, Region and
owning account, was dropped even though it was already in hand.

This tags AWS SDK client spans with the complete identity present in the
request:

- SNS: aws.topic.arn and aws.sns.topic_arn (full TopicArn / TargetArn),
  in addition to the existing aws.topic.name. dd-trace-py and dd-trace-js
  already emit aws.sns.topic_arn, so the same query now works across
  tracers.
- Kinesis: aws.stream.arn (StreamARN). The value was already read for
  Data Streams Monitoring but never set on the span.
- Step Functions: aws.state_machine.arn and statemachinearn
  (StateMachineArn) plus aws.execution.arn (ExecutionArn). The plain
  statemachinearn name matches the tag dd-trace-py and dd-trace-js emit.
- Lambda: aws.function.name and functionname (FunctionName) so Invoke
  spans name their target the same way the Python and JS tracers do.

No additional network calls are made; every value is a request field.
Existing tags are unchanged and peer.service derivation is untouched.

SDK v2 Step Functions members are lowerCamelCase in the service model
(getValueForField matches the model member name verbatim), so the
Step Functions fields are read as stateMachineArn and executionArn.
SDK v1 gains getStateMachineArn,
getExecutionArn and getFunctionName method handles on GetterAccess.

Tests cover both SDK versions for SNS ARN tags, Step Functions
StartExecution and DescribeExecution, and Lambda Invoke, and the
Kinesis DSM tests now assert aws.stream.arn.
@pedramsafaei
pedramsafaei force-pushed the pedramsafaei/aws-sdk-arn-span-tags branch from dcb9e6a to 78c738e Compare September 24, 2026 07:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants