Tag DynamoDB and S3 client spans with the owning AWS account

[pedramsafaei]

What Does This Do

Tags DynamoDB and S3 client spans (AWS SDK v1 and v2) with the AWS account that owns the addressed resource, using only information the tracer already holds. No additional network calls.

Case	New tags	Source of the account
DynamoDB, TableName is an ARN	aws_account, aws.table.arn; aws.table.name / tablename / peer.service now carry the bare table name	the ARN
DynamoDB, bare TableName (SDK v2)	aws_account, synthesized aws.table.arn	the credentials signing the request
S3, ExpectedBucketOwner set and the request succeeds	aws_account	the request field, applied from the successful response

Why the bare-name case is sound: DynamoDB documents that "If you only provide the table name parameter instead of a complete ARN, the API operation will be performed on the table in the account to which the requestor belongs" (Cross-account access with resource-based policies). Cross-account access requires either the full ARN as TableName (account is then in the request) or credentials from a role in the owning account (account is then in the credentials). Either way the account owning the signing credentials is the table owner whenever the name is bare.

Why ExpectedBucketOwner is sound: S3 rejects the request with 403 when it does not match the bucket owner, so on a successful response it is authoritative. The tag is applied from the response hook (v2: carried in an ExecutionAttribute from onSdkRequest to onSdkResponse, set only on a 2xx; v1: set in afterResponse, which does not run for errors), so a rejected owner is never tagged on the error span.

Where the caller account comes from on SDK v2: AwsSignerExecutionAttribute.AWS_CREDENTIALS is read in onSdkRequest, and AwsCredentialsIdentity.accountId() (added in SDK 2.26, populated by the STS, SSO, profile, process and container credential providers) is called through a reflective, per-class cached MethodHandle so the instrumentation still loads against the 2.2.0 floor. For older SDKs whose credentials carry no account, the account can be decoded from the access key ID (it is encoded in the trailing base32 characters). That encoding is not part of the documented AWS API surface, so it is opt-in behind dd.trace.aws.account.from.access.key.enabled / DD_TRACE_AWS_ACCOUNT_FROM_ACCESS_KEY_ENABLED, default false, registered in metadata/supported-configurations.json.

SDK v1 does not expose the signing credentials to request handlers (and HandlerContextKey does not exist at the 1.11.0 floor), so v1 covers the ARN and ExpectedBucketOwner cases only.

The tag name aws_account is what dd-trace-py already emits for the SNS and SQS account it derives from TopicArn and QueueUrl, and it is the dimension tag on the aws.dynamodb.* integration metrics, so a span and the metrics for the same table join without translation. It is also on the allow list of the Agent's credit card obfuscator (see below).

Implementation:

• aws-java-common: new AwsArn value type that parses an ARN once (partition, service, Region, account, resource) and owns the DynamoDB table-name extraction (table/<name>/index/... and other sub-resources stripped), plus AwsAccountIdentity (Region to partition mapping, table ARN assembly, access key decode). Both are pure java.lang with no agent dependencies. The SDK's own software.amazon.awssdk.arns.Arn is not used because the arns module is not a dependency of aws-core at the 2.2.0 floor these instrumentations compile against, and SDK v1 has no equivalent. Both SDK modules take aws-java-common as an implementation project(...) dependency and register the classes in helperClassNames() so they are injected alongside the decorators, the same pattern as elasticsearch-common and kafka-common. (Initially placed in agent-bootstrap; moved here per review, since bootstrap is loaded for every JVM and this is AWS-specific.)
• aws-java-sdk-2.2: onDynamoDbTable replaces the plain setTableName call; callerAccount / credentialsAccountId read the credentials; ExpectedBucketOwner handled in the S3 branch.
• aws-java-sdk-1.11: GetterAccess gains getExpectedBucketOwner; the decorator handles the ARN form of TableName and ExpectedBucketOwner.
• internal-api / dd-trace-api: tag constants, the new config flag.

Motivation

aws.table.name says which table, not whose. Two accounts can each own an orders table in the same Region, and the span alone cannot tell them apart, which breaks any account-aware dependency map and any join to the AWS integration metrics (which are keyed by account). The ARN and the signing credentials already carry the answer; the tracer was discarding it.

There is also a small correctness fix in the ARN case: today a TableName given as an ARN is emitted verbatim as aws.table.name, tablename and peer.service, so the same table produces different peer.service values depending on how the caller addressed it.

Additional Notes

Testing

• AwsArnTest and AwsAccountIdentityTest (aws-java-common, JUnit 5, 30 + 43 cases): ARN parsing across partitions and resource shapes, malformed ARNs, twelve-digit account validation, DynamoDB table-name extraction (plain, /index/, /stream/, non-table resources), partition mapping (commercial, cn, gov, iso, iso-b, iso-e, iso-f, eusc), table ARN assembly, the base32 decoder against the RFC 4648 foobar vector (upper and lower case, sub-range, alphabet rejects), account extraction from a hand-built byte vector, and the end-to-end access key decode against synthetically encoded keys plus malformed shapes.
• Aws2ClientTest (forkedTest, 37 + 37 across the V0/V1 naming forks + 23 legacy): new feature methods for an ARN TableName (asserts the exact tag set including aws.table.arn, aws_account and the bare name in peer.service) and for a bare name with credentials that carry no account (asserts no account tags are invented).
• S3BucketOwnerForkedTest (JUnit 5, 4 cases, in the payloadTaggingTest source set because its S3 model, 2.18.40, has ExpectedBucketOwner; the base suite pins S3 2.2.0 which predates the field): owner set and accepted, owner absent, owner malformed, owner rejected with 403 (no aws_account on the error span).
• AWS1ClientTest (24 + 24 forked + 17 legacy): new row with an ARN TableName.
• spotlessApply clean on all touched modules.

How the end-to-end test was run

• Environment: an EKS cluster (managed node group, EC2 nodes) running the Datadog Helm chart with the node Agent's trace receiver; a real DynamoDB table, Kinesis stream, SNS topic, Step Functions state machine, Lambda function, S3 bucket and RDS for PostgreSQL instance in the same account; IRSA credentials on the workload's service account.
• Workload: a small Java 21 application (AWS SDK for Java v2 2.29.52, PostgreSQL JDBC 42.7.4) that once per cycle calls SNS Publish, Kinesis PutRecord, Step Functions StartExecution + DescribeExecution, Lambda Invoke, DynamoDB GetItem (by name and by ARN), S3 GetObject and a JDBC SELECT 1, all inside one traced method so the client spans share a trace.
• Two deployments of the same container image, differing only in the dd-java-agent.jar copied into it: the released 1.66.0 tracer (before) and the jar built from this branch with ./gradlew :dd-java-agent:shadowJar (after). Same node pool, same DD_ENV, distinct DD_VERSION so the two runs are separable in Span Search.
• Verification: the Datadog Span Search API (/api/v2/spans/events/search) was queried for each run's client spans and the per-span attributes compared key by key; the raw responses are retained. Where the change enables a join to the AWS integration metrics, the corresponding /api/v1/query was run against the same org and the returned scope recorded.

Live before/after on a real EKS workload

A Java 21 workload using AWS SDK v2 2.29.52 under IRSA on an EKS cluster ran GetItem against a real table twice per cycle (once by bare name, once by ARN) and GetObject with ExpectedBucketOwner against a real bucket, once with the stock dd-java-agent 1.66.0 and once with the jar from this branch, same image, same node pool. Spans as returned by the Span Search API (account masked):

DynamoDb.GetItem, bare TableName, IRSA credentials
  before: aws.table.name=datadog-context-trace-fixture
  after : aws.table.name=datadog-context-trace-fixture
          + aws_account=123456789012
          + aws.table.arn=arn:aws:dynamodb:us-east-1:123456789012:table/datadog-context-trace-fixture
  (account came from AwsCredentialsIdentity.accountId(), which the STS web identity provider populates;
   the access key fallback stayed disabled)

DynamoDb.GetItem, TableName given as the ARN
  before: aws.table.name=arn:aws:dynamodb:us-east-1:123456789012:table/datadog-context-trace-fixture
          tablename=arn:aws:dynamodb:us-east-1:123456789012:table/datadog-context-trace-fixture
  after : aws.table.name=datadog-context-trace-fixture
          tablename=datadog-context-trace-fixture
          + aws.table.arn=arn:aws:dynamodb:us-east-1:123456789012:table/datadog-context-trace-fixture
          + aws_account=123456789012

S3.GetObject with ExpectedBucketOwner
  before: aws.bucket.name=<bucket>
  after : aws.bucket.name=<bucket>  + aws_account=123456789012

The join this enables, against the same org's AWS integration metrics:

sum:aws.dynamodb.successful_request_latency{tablename:datadog-context-trace-fixture} by {tablename,aws_account,region}
  -> scope: aws_account:123456789012,region:us-east-1,tablename:datadog-context-trace-fixture

Obfuscator note for reviewers

The first live run also emitted the S3 owner under a second key, aws.bucket.owner, and it arrived at Datadog as ?. The Agent's credit card obfuscator redacts any 12-digit value whose key is not on its allow list; aws_account is on that list, aws.bucket.owner was not. This PR therefore emits the account under aws_account only. Anyone adding a new account-bearing key in the tracer should expect the same behaviour (see also DataDog/datadog-agent#56242, which adds cloud.account.id and aws.account.id to that allow list).

Design notes

• The DynamoDB enrichment is gated on the DynamoDB service name. Other request models that also carry a TableName member (Timestream, Keyspaces, Athena) keep the pre-existing plain aws.table.name / tablename tags and never receive an account or a synthesized DynamoDB ARN.
• The access key decode only accepts keys in the account-encoding format (issued since late March 2019, top bit of the decoded 48-bit value set, equivalently fifth character Q or later). Older keys carry no account and are rejected rather than decoded into an arbitrary value; AKIAIOSFODNN7EXAMPLE from the AWS docs is such a key and is a test case.
• aws_account is only set where the account is provable: an ARN in the request, credentials whose account is asserted by the SDK, or an owner the caller asserted and S3 enforces. A bare S3 bucket name is deliberately not attributed to the caller, since bucket policies allow cross-account access by name.
• Kinesis StreamName and Lambda FunctionName follow the same "bare name resolves in the requestor's account" rule and could get the same treatment; left for a follow-up to keep this change reviewable.
• If reviewers would rather not carry the access key decode at all, it is isolated in AwsAccountIdentity.accountFromAccessKeyId plus the config flag and can be dropped without touching the rest.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels — suggested inst: aws sdk, inst: aws dynamodb, inst: aws s3, type: feature (external contributor, cannot set labels)
• Avoid using close, fix, or any linking keywords when referencing an issue
• Update the CODEOWNERS file on source file addition, migration, or deletion — the new files sit under dd-java-agent/instrumentation/aws-java/ (aws-java-common/ and aws-java-sdk-2.2/src/payloadTaggingTest/java/), covered by the existing @DataDog/apm-serverless directory entry
• Update public documentation with any new configuration flags or behaviors — one new opt-in flag, DD_TRACE_AWS_ACCOUNT_FROM_ACCESS_KEY_ENABLED, registered in metadata/supported-configurations.json; happy to add the public docs entry once naming is agreed
• Once approved, use merge queue to merge the PR

Suggested labels (cannot set as an external contributor): inst: aws sdk, inst: aws dynamodb, inst: aws s3, type: feature.

[mhlidd]

LGTM from SDK Capabilities POV

[PerfectSlayer]

Looking good from platform POV. Thanks for the follow up changes!

[vandonr]

Looks like we could use AWS SDK provided ARN parser for a good chunk of the work here, any reason to reimplement it ourselves here ?
https://docs.aws.amazon.com/java/api/latest/software/amazon/awssdk/arns/Arn.html

Also, if we want to keep our own implem, I think we should parse it once in an object. Here we end up checking several times if an ARN is valid for instance when we retrieve multiple values.

[pedramsafaei]

The arns module is not a dependency of aws-core at the 2.2.0 floor this instrumentation compiles against (checked the 2.2.0 POM), and SDK v1 has no equivalent, so depending on it would break loading on older v2 and be unusable from the v1 module. Agreed on parsing once though pushed a new change that replaces the per-field helpers with an AwsArn value type parsed in a single pass, and both decorators call parse once.

[vandonr]

doing the substring once with beginning and end would be more efficient (it'd save an intermediary string)

[pedramsafaei]

Done. The extraction moved into AwsArn.dynamoDbTableName() and does one substring(begin, end).

[vandonr]

isn't it weird that if we found that the tableName is an ARN but the resource doesn't start with "table/" then we don't save it in the "AWS_TABLE_ARN" tag ?

[pedramsafaei]

no it is intentional, but it deserved to be explicit. An ARN whose resource is not a table (a backup or export ARN, or a mistaken value) is not a table ARN, so tagging it as aws.table.arn would be wrong so in that case the raw value stays as the table name exactly as before this PR. In the new commit I pushed that decision lives in AwsArn.dynamoDbTableName() returning null, with a comment at both call sites.

[vandonr]

if it's authoritative when set, why do we need to check if the returned value is an account ID ?

[pedramsafaei]

Fair, the comment was misleading. It is authoritative for requests S3 accepts, but the span is tagged before the response is known, and S3 rejects a malformed owner (400) or a wrong one (403). Without the check a request the application got wrong would stamp aws_account=not-an-account on its error span. Kept the check and reworded the comment in the new commit the "malformed owner" test should cover it.

[vandonr]

looks like this duplicated part could go in the common helper

[pedramsafaei]

done

[vandonr]

new tests should be written in java using jUnit, not in groovy (except if strongly interconnected with existing tests, but I don't think it's the case here ?)

[pedramsafaei]

Done. Now the only Groovy changes are new rows in the existing Aws2ClientTest and AWS1ClientTest.

[vandonr]

why not with a dot ?

Suggested change

	public static final String AWS_ACCOUNT = "aws_account";
	public static final String AWS_ACCOUNT = "aws.account";

[pedramsafaei]

Three reasons, all about the join. aws_account is the exact dimension tag on the aws.dynamodb.* and aws.s3.* integration metrics, so span and metric join without a rename and it is the key dd-trace-py already emits for the SNS and SQS account and also it is on the Agent credit-card obfuscator's allow list, whereas any other key carrying a 12-digit value is rewritten to ? at intake today (that happened live with aws.bucket.owner, see the obfuscator note in the description). aws.account would arrive as ? until DataDog/datadog-agent#56242 lands. Happy to also emit aws.account once that merges if you want the dotted form.

[vandonr]

ok looks good (2 minor comments)
I'll need to duplicate this PR under my own account to be able to run the CI, I'll take care of this next week :)

[vandonr]

I think it'd be nicer to untangle the base 32 parsing from actually getting the access key ID

[pedramsafaei]

Done

[vandonr]

I think we can remove this as it's only used in tests, and checking AwsArn.parse(...) != null is actually a stricter check

[pedramsafaei]

Agreed, done