Add AWS resource ARN tags to AWS SDK client spans - #12598
pedramsafaei wants to merge 1 commit into
Conversation
mhlidd
left a comment
There was a problem hiding this comment.
LGTM from an SDK Capabilities POV
| String topicName = null; | ||
| String topicArn = access.getTopicArn(originalRequest); | ||
| if (null != topicArn) { | ||
| span.setTag(InstrumentationTags.AWS_TOPIC_ARN, topicArn); |
There was a problem hiding this comment.
Should we skip v1 since it was EOLed?
There was a problem hiding this comment.
There was a problem hiding this comment.
I would like to keep at least part of v1 if you are open to it. v1 is EOL from AWS but the tracer still ships and tests this instrumentation, and the users still on it are the ones most likely to have cross-account SNS topics where the bare name is ambiguous. The SNS and Kinesis half of the v1 change is a few lines with no new dependencies and the decorator already reads the ARN and discards it. The Step Functions and Lambda half is what pulled in the stepfunctions test dependency and the core pin, so if the concern is that surface I am happy to drop those two from v1 and keep SNS and Kinesis. But anyways if the team's stance is absolutely no new features on v1 at all, tell me and I will drop v1 entirely here and in #12602.
| request.getValueForField("TableName", String.class).ifPresent(name -> setTableName(span, name)); | ||
|
|
||
| // Step Functions. The service model uses lowerCamelCase member names. | ||
| stringField(request, "stateMachineArn", "StateMachineArn") |
There was a problem hiding this comment.
Why do we need to check both camelCase and PascalCase? Isn't the model generator supposed to normalize them to PascalCase?
There was a problem hiding this comment.
The generator does not normalize the member name so getValueForField is generated as an equals chain against the model's member name verbatim. Step Functions' API model defines its members in lowerCamelCase, so sfn's StartExecutionRequest.getValueForField matches "stateMachineArn" (checked in 2.2.0 and 2.15.35), whereas SNS matches "TopicArn" and DynamoDB "TableName". Querying "StateMachineArn" returns empty on sfn, which is why the lowerCamel spelling is there. You are right that probing both is unnecessary though, since the member name comes from the API model and does not vary by SDK version. I will drop the stringField helper and query "stateMachineArn" / "executionArn" directly with a comment on the casing.
There was a problem hiding this comment.
Done. stringField is gone, the two fields are read directly as stateMachineArn / executionArn with a comment on why the casing differs from SNS and DynamoDB.
ygree
left a comment
There was a problem hiding this comment.
From a functional perspective, the change looks good, but the code footprint can be reduced by removing unnecessary parts, see details in the comments.
| return Optional.empty(); | ||
| } | ||
|
|
||
| private static void setTopicArn(AgentSpan span, String arn) { |
There was a problem hiding this comment.
Should these three two-line methods, which were only used once, be inlined instead?
There was a problem hiding this comment.
Yes, agreed. Inlined all three as each had exactly one call site. Left setStreamName and setQueueName as methods since they have two callers each.
ygree
left a comment
There was a problem hiding this comment.
IDM approved. There are no blockers, but the code change footprint could likely be reduced.
The AWS SDK v1 and v2 instrumentations already read several resource identifiers out of the request to derive peer.service, but only emitted the short names. The ARN itself, which carries the partition, Region and owning account, was dropped even though it was already in hand. This tags AWS SDK client spans with the complete identity present in the request: - SNS: aws.topic.arn and aws.sns.topic_arn (full TopicArn / TargetArn), in addition to the existing aws.topic.name. dd-trace-py and dd-trace-js already emit aws.sns.topic_arn, so the same query now works across tracers. - Kinesis: aws.stream.arn (StreamARN). The value was already read for Data Streams Monitoring but never set on the span. - Step Functions: aws.state_machine.arn and statemachinearn (StateMachineArn) plus aws.execution.arn (ExecutionArn). The plain statemachinearn name matches the tag dd-trace-py and dd-trace-js emit. - Lambda: aws.function.name and functionname (FunctionName) so Invoke spans name their target the same way the Python and JS tracers do. No additional network calls are made; every value is a request field. Existing tags are unchanged and peer.service derivation is untouched. SDK v2 Step Functions members are lowerCamelCase in the service model (getValueForField matches the model member name verbatim), so the Step Functions fields are read as stateMachineArn and executionArn. SDK v1 gains getStateMachineArn, getExecutionArn and getFunctionName method handles on GetterAccess. Tests cover both SDK versions for SNS ARN tags, Step Functions StartExecution and DescribeExecution, and Lambda Invoke, and the Kinesis DSM tests now assert aws.stream.arn.
dcb9e6a to
78c738e
Compare
What Does This Do
Tags AWS SDK v1 and v2 client spans with the complete resource identity that is already present in the request, instead of only the short name:
TopicArn/TargetArnaws.topic.arn,aws.sns.topic_arnaws.topic.name,topicnameStreamARNaws.stream.arnaws.stream.name,streamnamestateMachineArnaws.state_machine.arn,statemachinearnexecutionArnaws.execution.arnFunctionNameaws.function.name,functionnameNo additional network calls are made; every value is read from the request object the instrumentation already inspects. Existing tags,
peer.servicederivation and Data Streams Monitoring behaviour are unchanged. Nothing is added when the request does not carry the field.The plain-name variants (
aws.sns.topic_arn,statemachinearn,functionname) are the tagsdd-trace-pyanddd-trace-jsalready emit for the same calls, so one query now works across the three tracers.Implementation:
internal-api: new constants inInstrumentationTags.aws-java-sdk-2.2:AwsSdkClientDecorator.onSdkRequestreads the fields viaSdkRequest.getValueForField.getValueForFieldmatches the API model's member name verbatim, and the Step Functions model declares its members in lowerCamelCase (unlike SNSTopicArnor DynamoDBTableName), so those two fields are read asstateMachineArnandexecutionArn.aws-java-sdk-1.11:GetterAccessgainsgetStateMachineArn,getExecutionArnandgetFunctionNamemethod handles, resolved the same way as the existinggetTableName;AwsSdkClientDecorator.onRequestuses them. The SNS and Kinesis branches tag the ARN they already read.Motivation
peer.serviceand the*nametags identify which topic, stream, state machine or function a span talks to, but not whose. Two accounts can each own apaymentsstate machine or anotificationstopic, and the short name is ambiguous the moment a service crosses an account or Region boundary. The ARN removes that ambiguity and carries partition, Region and account with it.This matters for anyone building account-aware dependency maps or joining APM spans with the AWS integration metrics (which are tagged by ARN-derived dimensions), and for cross-account debugging where the short name alone does not say which resource was hit. The tracer already had the ARN in hand for SNS, Kinesis and Step Functions and was discarding it.
The Java tracer was also the odd one out: the Python and JS tracers already emit
aws.sns.topic_arnandstatemachinearn, and Java emitted neither.Additional Notes
Testing
Unit and instrumentation tests, all green locally on this branch:
aws-java-sdk-2.2:test(41 + 41 across the V0/V1 forks),LegacyAws2ClientForkedTest(23),dsmTest(14). New rows for Step FunctionsStartExecution/DescribeExecutionand LambdaInvoke; SNS rows now assert both ARN tags; the Kinesis DSM tests assertaws.stream.arn.aws-java-sdk-1.11:test(25),forkedTest(25 + 17). Same new rows; thetest_before_1_11_106configuration pinsaws-java-sdk-coreto 1.11.0 becauseaws-java-sdk-stepfunctions(a new test dependency) only exists from 1.11.63 and would otherwise pull a newer core than the pinned clients.spotlessApplyclean on the touched modules.How the end-to-end test was run
Publish, KinesisPutRecord, Step FunctionsStartExecution+DescribeExecution, LambdaInvoke, DynamoDBGetItem(by name and by ARN), S3GetObjectand a JDBCSELECT 1, all inside one traced method so the client spans share a trace.dd-java-agent.jarcopied into it: the released 1.66.0 tracer (before) and the jar built from this branch with./gradlew :dd-java-agent:shadowJar(after). Same node pool, sameDD_ENV, distinctDD_VERSIONso the two runs are separable in Span Search./api/v2/spans/events/search) was queried for each run's client spans and the per-span attributes compared key by key; the raw responses are retained. Where the change enables a join to the AWS integration metrics, the corresponding/api/v1/querywas run against the same org and the returned scope recorded.Live before/after on a real EKS workload
A Java 21 workload using AWS SDK v2 (
sns,kinesis,sfn,lambda,dynamodb) was run on an EKS cluster under IRSA, once with the stockdd-java-agent1.66.0 and once with the jar built from this branch (dcb9e6a20c), same image, same node pool, same fixtures. Client spans as returned by the Span Search API (account masked):Design notes for reviewers
aws.topic.arn+aws.sns.topic_arn,aws.state_machine.arn+statemachinearn,aws.function.name+functionname) follow the existing convention in this decorator (aws.topic.name+topicname,aws.stream.name+streamname). The namespaced key matches Java's own style; the plain key matches the other tracers. Happy to drop either half if you would rather not carry both.FunctionNamemay be a name, a partial ARN or a full ARN; it is tagged verbatim. The SDK v1 decorator only readsgetFunctionNamewhen the service isAWSLambda, and the v2 decorator when the service name islambda, so unrelated request types with aFunctionNamemember are not affected.StreamARNwas already read for Data Streams Monitoring; this only additionally sets it on the span, so DSM pathway keys are unchanged (the existing DSM tests still pass with the added tag).PutEventswas considered but its bus identity lives inside each entry (EventBusNameperPutEventsRequestEntry), not on the request, so it is left for a follow-up if there is interest.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labels — suggestedinst: aws sdk,type: feature(external contributor, cannot set labels)close,fix, or any linking keywords when referencing an issueSuggested labels (cannot set as an external contributor):
inst: aws sdk,type: feature.