Tag database client spans with the identity encoded in RDS endpoint hostnames

[pedramsafaei]

What Does This Do

Database client spans against Amazon RDS currently carry only peer.hostname, for example orders-db.c9akciq32bzq.us-east-1.rds.amazonaws.com. That hostname deterministically encodes the DB instance (or Aurora cluster, custom endpoint, or RDS Proxy) identifier and the Region, but nothing exposes them as tags.

DatabaseClientDecorator.onConnection now parses the hostname whenever it matches an RDS endpoint and adds:

Tag	Value	Set for
aws.rds.identifier	leading hostname label	every RDS endpoint type
aws.rds.endpoint_type	instance, cluster, cluster-ro, cluster-custom, proxy	every RDS endpoint type
aws.region	Region encoded in the hostname	every RDS endpoint type
dbinstanceidentifier	instance identifier	instance endpoints only
dbclusteridentifier	cluster identifier	cluster and cluster-ro endpoints only

dbinstanceidentifier and dbclusteridentifier are spelled exactly like the dimension tags on the aws.rds.* integration metrics, so an APM span and the RDS integration (and Database Monitoring) join on the same key without any hostname parsing on the query side.

The instance or cluster identifier is only claimed when the endpoint type proves it. A cluster endpoint does not name the writer instance behind it, and a custom endpoint or proxy name is not a cluster identifier, so those types get aws.rds.identifier and aws.rds.endpoint_type only.

Implementation:

• New AwsRdsEndpoint in agent-bootstrap parses both endpoint shapes, <id>.<hash>.<region>.rds.amazonaws.com and the China partition <id>.<hash>.rds.<region>.amazonaws.com.cn, tolerating a trailing dot or :port, case-insensitively. The hash label prefix (cluster-, cluster-ro-, cluster-custom-, proxy-, or none) determines the endpoint type. The hash and Region labels are validated so unrelated hostnames that happen to end in rds.amazonaws.com are rejected. Hostnames that do not end in .amazonaws.com or .amazonaws.com.cn (an O(1) case-insensitive suffix check that tolerates the same trailing . and :port as the parser) return before the cache, so non-AWS database hosts, and lookalikes with labels appended after .amazonaws.com, never evict RDS entries or pay for the label split. Plausible hostnames are then parsed once and cached in a 16-entry DDCache (applications talk to a handful of RDS databases).
• DatabaseClientDecorator calls it right after setting peer.hostname, in a protected onRdsEndpoint hook. Because this is the shared base decorator, every database client integration (JDBC, R2DBC, Vert.x SQL, MongoDB, and so on) picks it up, and non-RDS hostnames are left exactly as before.
• New constants in InstrumentationTags.

Motivation

Correlating a slow query span with the RDS instance it hit means matching peer.hostname against the instance endpoint by hand, and building any account- or Region-aware view of database dependencies means re-implementing the hostname parse in every dashboard or downstream consumer. The identity is already in the hostname; the tracer should surface it once, in the same vocabulary the RDS integration uses.

A hostname cannot say which account owns the instance, so this intentionally stops at identifier, endpoint type and Region. Joining on dbinstanceidentifier plus region against the aws.rds.* metrics supplies aws_account, which is why matching the metric dimension names exactly matters.

Additional Notes

Testing

• AwsRdsEndpointTest (new, JUnit 5, 47 cases): instance, cluster, cluster reader, custom and proxy endpoints; upper-case, trailing dot and :port inputs; GovCloud, ISO and China Regions; and 15 negative cases (non-RDS hosts, missing labels, bad hash, bad Region, s3/dynamodb service endpoints, and a lookalike with a suffix appended). Also asserts the per-hostname cache returns the same instance; 19 rows for the pre-cache suffix gate (commercial and China with trailing dot and :port, upper and mixed case; rejects for non-AWS hosts, .amazonaws.com followed by further labels, and malformed ports) and one asserting 100 non-AWS and 100 suffix-lookalike hostnames do not evict a cached RDS endpoint.
• DatabaseClientDecoratorTest: new onConnection rows asserting the exact tag set per endpoint type with strict interaction counting (0 * _), plus a non-RDS row asserting nothing is added.
• Whole agent-bootstrap module: 686 tests, 0 failures. instrumentation/jdbc module: 423 tests, 0 failures (its fixtures use local hosts, confirming no behaviour change off RDS).
• spotlessApply clean.

How the end-to-end test was run

• Environment: an EKS cluster (managed node group, EC2 nodes) running the Datadog Helm chart with the node Agent's trace receiver; a real DynamoDB table, Kinesis stream, SNS topic, Step Functions state machine, Lambda function, S3 bucket and RDS for PostgreSQL instance in the same account; IRSA credentials on the workload's service account.
• Workload: a small Java 21 application (AWS SDK for Java v2 2.29.52, PostgreSQL JDBC 42.7.4) that once per cycle calls SNS Publish, Kinesis PutRecord, Step Functions StartExecution + DescribeExecution, Lambda Invoke, DynamoDB GetItem (by name and by ARN), S3 GetObject and a JDBC SELECT 1, all inside one traced method so the client spans share a trace.
• Two deployments of the same container image, differing only in the dd-java-agent.jar copied into it: the released 1.66.0 tracer (before) and the jar built from this branch with ./gradlew :dd-java-agent:shadowJar (after). Same node pool, same DD_ENV, distinct DD_VERSION so the two runs are separable in Span Search.
• Verification: the Datadog Span Search API (/api/v2/spans/events/search) was queried for each run's client spans and the per-span attributes compared key by key; the raw responses are retained. Where the change enables a join to the AWS integration metrics, the corresponding /api/v1/query was run against the same org and the returned scope recorded.

Live before/after on a real EKS workload

A Java 21 workload on EKS ran SELECT 1 through the PostgreSQL JDBC driver against a real RDS for PostgreSQL instance, once with stock dd-java-agent 1.66.0 and once with the jar built from this branch (2283a78a22). The postgresql.query span as returned by the Span Search API:

before (1.66.0):
  db.instance   = postgres
  peer.hostname = dd-e2e-20260918-postgres-rds.cgtcuwaucyv9.us-east-1.rds.amazonaws.com

after (this branch):
  db.instance            = postgres
  peer.hostname          = dd-e2e-20260918-postgres-rds.cgtcuwaucyv9.us-east-1.rds.amazonaws.com
  aws.rds.identifier     = dd-e2e-20260918-postgres-rds
  aws.rds.endpoint_type  = instance
  aws.region             = us-east-1
  dbinstanceidentifier   = dd-e2e-20260918-postgres-rds

And the join it enables, from the same org's AWS integration metrics (account masked):

avg:aws.rds.cpuutilization{dbinstanceidentifier:dd-e2e-20260918-postgres-rds} by {dbinstanceidentifier,region,aws_account,engine}
  -> scope: aws_account:123456789012,dbinstanceidentifier:dd-e2e-20260918-postgres-rds,engine:postgres,region:us-east-1

The same workload's AWS SDK spans (SNS, Kinesis, Step Functions, Lambda, DynamoDB) were unchanged between the two runs, as expected.

Design notes for reviewers

• Only aws.region is set, not a bare region tag, to avoid colliding with the host-level region tag when a client in one Region talks to a database in another.
• No configuration flag. The tags are only added when the hostname is provably an RDS endpoint, and the parse is cached, so there is no cost on non-RDS connections. I can gate it behind a dd.trace.db.client.rds.identity.enabled style flag if you prefer that for new tags.
• The hostname does not distinguish an instance endpoint from an Aurora cluster member endpoint (both look like <id>.<hash>.<region>...), so an Aurora instance endpoint is reported as instance with its instance identifier, which is correct for the dbinstanceidentifier join.
• Cache key is the raw hostname string; the 16-entry fixed-size cache follows the existing DatabaseClientDecorator.CACHE pattern. Only hostnames ending in .amazonaws.com / .amazonaws.com.cn reach it (review feedback), and a test asserts that distinct non-AWS and suffix-lookalike hosts do not evict a cached RDS entry.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels — suggested comp: tracing, inst: jdbc, type: feature (external contributor, cannot set labels)
• Avoid using close, fix, or any linking keywords when referencing an issue
• Update the CODEOWNERS file on source file addition, migration, or deletion — the new files sit in dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/, already owned by @DataDog/apm-idm-java via the existing directory entry, so no change needed
• Update public documentation with any new configuration flags or behaviors — no new configuration; new tags only
• Once approved, use merge queue to merge the PR

Suggested labels (cannot set as an external contributor): comp: tracing, inst: jdbc, type: feature.

[mcculls]

+1 from apm-sdk-capabilities-java

I did some rough benchmarking of master vs PR #12599 (commit afcfb6a), 3 forks × 15 iterations (30 samples/hostname)

┌──────────────────────────────────────────────────┬───────────────┬───────────────┬────────────┐
│                     hostname                     │    master     │    PR head    │ added cost │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ localhost                                        │ 0.001 ± 0.001 │ 0.003 ± 0.001 │ +0.002     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ 127.0.0.1                                        │ 0.001 ± 0.001 │ 0.005 ± 0.001 │ +0.004     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ db.internal.corp                                 │ 0.001 ± 0.001 │ 0.006 ± 0.001 │ +0.005     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ postgres.default.svc.cluster.local               │ 0.001 ± 0.001 │ 0.008 ± 0.001 │ +0.007     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ orders-db...rds.amazonaws.com (RDS instance)     │ 0.001 ± 0.001 │ 0.014 ± 0.001 │ +0.013     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ orders-cluster...rds.amazonaws.com (RDS cluster) │ 0.001 ± 0.001 │ 0.014 ± 0.001 │ +0.013     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ orders-db...rds.amazonaws.com.spoofed.example    │ 0.001 ± 0.001 │ 0.006 ± 0.001 │ +0.005     │
├──────────────────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ my-instance...rds.cn-north-1.amazonaws.com.cn    │ 0.001 ± 0.001 │ 0.015 ± 0.001 │ +0.014     │
└──────────────────────────────────────────────────┴───────────────┴───────────────┴────────────┘

Summary: the PR adds a small, bounded per-connection cost (2–7ns for non-RDS hosts, ~13–15ns for genuine RDS hosts due to the extra tagging etc.), negligible next to typical span overhead.

And that last length short-circuit pays off for small hostnames which are common in dev/test environments and local-first setups:

┌──────────────────────────────────────────────────┬──────────────────────┬───────────────────┬────────┐
│                     hostname                     │ without length check │ with length check │ delta  │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ localhost (9 chars)                              │ 0.003                │ 0.001             │ −0.002 │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ 127.0.0.1 (9 chars)                              │ 0.004                │ 0.001             │ −0.003 │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ db.internal.corp (17 chars)                      │ 0.006                │ 0.006             │ 0      │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ postgres.default.svc.cluster.local (35 chars)    │ 0.008                │ 0.007             │ −0.001 │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ orders-db...rds.amazonaws.com (RDS instance)     │ 0.013                │ 0.013             │ 0      │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ orders-cluster...rds.amazonaws.com (RDS cluster) │ 0.013                │ 0.013             │ 0      │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ ...amazonaws.com.spoofed.example (lookalike)     │ 0.006                │ 0.006             │ 0      │
├──────────────────────────────────────────────────┼──────────────────────┼───────────────────┼────────┤
│ China RDS                                        │ 0.015                │ 0.015             │ 0      │
└──────────────────────────────────────────────────┴──────────────────────┴───────────────────┴────────┘