feat(core): Data Collection - #5759
Conversation
|
Add the public Data Collection model, key-value collection behavior, and HTTP body direction types. Preserve unset values internally so the resolver can distinguish legacy bridge mode from explicit configuration. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
📲 Install BuildsAndroid
|
Avoid coupling the Data Collection configuration test to reference identity. The test only needs to verify that setting a nested header behavior marks the configuration explicit. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
Add an always-present DataCollection object to SentryOptions while preserving an unconfigured state for the legacy bridge. Expose public getter and setter APIs and cover explicit-empty and nested override behavior. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
Add a resolver owned by SentryOptions that applies namespace-wide Data Collection defaults and legacy sendDefaultPii fallbacks. Expose boolean, key-value, and directional HTTP body policies without changing production collection paths. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
Control GraphQL documents and variables through the new Data Collection policies across GraphQL and Apollo integrations. Preserve sendDefaultPii and maxRequestBodySize behavior when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Suppress SQL statement descriptions when database query data collection is disabled while retaining database system, name, timing, status, and other structural span metadata. Preserve existing statement collection when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Use the Data Collection incoming request body decision for servlet request caching and event body extraction across all Spring variants. Keep existing body size, content length, and MIME type limits while preserving sendDefaultPii behavior when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Use the Data Collection incoming response body decision when attaching failed GraphQL response content in Apollo 3 and 4. Continue reading responses for error detection and retain status and body size metadata. Co-Authored-By: Claude <noreply@anthropic.com>
Use the Data Collection outgoing request body decision when attaching failed GraphQL request content in Apollo 3 and 4. Retain request body size metadata and continue applying GraphQL document and variable controls when body collection is enabled. Co-Authored-By: Claude <noreply@anthropic.com>
Use the Data Collection outgoing response body decision when attaching GraphQL execution results. Preserve the sendDefaultPii and maxRequestBodySize gate when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Filter automatically collected request headers through the Data Collection policy across Servlet, Spring, OpenTelemetry, OkHttp, Ktor, and Apollo integrations. Preserve each integration’s sendDefaultPii behavior when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Avoid evaluating custom allow or deny terms after a request header has already matched the built-in sensitive policy. Co-Authored-By: Claude <noreply@anthropic.com>
Filter automatically collected response headers through the Data Collection policy in OkHttp, Ktor, and Apollo failed-request events. Preserve sendDefaultPii behavior when Data Collection is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Remove the queue option from the initial Data Collection API because the Java SDK does not collect queue payload data that it could control. Co-Authored-By: Claude <noreply@anthropic.com>
Bring the queue option removal from the foundation branch into the SentryOptions layer.
Use the supported user information option to verify that SentryOptions replaces its Data Collection instance. Co-Authored-By: Claude <noreply@anthropic.com>
Rename the internal override marker to explain that it forces an empty Data Collection object into explicit mode. Align the related tests with the clarified semantics. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
Separate legacy sendDefaultPii fallback coverage from configured Data Collection behavior. Give each resolver test a name that describes one configuration state. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
…n-graphql # Conflicts: # sentry/src/test/java/io/sentry/DataCollectionResolverTest.kt
…-database-query-data
…a-collection-incoming-request-body
…ata-collection-incoming-response-body
…data-collection-outgoing-request-body
…ata-collection-outgoing-response-body
…data-collection-http-request-headers
Verify that Apollo 4 applies configured Data Collection deny-list behavior to captured request headers across both supported execution paths. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com>
…ta-collection-http-response-headers
There was a problem hiding this comment.
Reviewed the full Data Collection stack tip (fix/data-collection-opentelemetry-span-description) against data-collection 0.11.0 and the landed/in-flight peers (JS, Python, Cocoa, Ruby, Go).
What looks solid
- Config surface matches the required core:
userInfo,cookies/httpHeaders/urlQueryParamskey-value modes, directionalhttpBodies,graphQL,databaseQueryData. - Explicit-namespace resolution matches peers: once any
dataCollectionfield is set (or an emptyDataCollectionis assigned), omitted fields take spec defaults, notsendDefaultPii. That matches JSresolveDataCollectionOptionsand Go/Ruby “opt into defaults”. - Built-in sensitive terms match the denylist; partial case-insensitive matching and
[Filtered]substitution look correct inHttpUtils. - Integration wiring for Spring/Servlet/OkHttp/Ktor/Apollo/GraphQL is real and tested; Replay kept independent (spec 0.10.0); Android installation ID restored as non-
userInfoidentity (pragmatic, well motivated). - Span descriptions stripping queries (Ktor/OTel) matches the structuring-data rule.
Please fix before merge
1. Legacy path still ships sensitive query values (and dual-path bypasses the denylist)
Spec: sensitive key-value data MUST be replaced with [Filtered] under automatic collection. Peers always run collection through a resolved policy (JS bridges sendDefaultPii → urlQueryParams: true / deny-with-PII-snippets, then filters; Go/Python/Cocoa same idea).
Java does not:
// UrlUtils.filterQueryParams
return resolver.isDataCollectionConfigured()
? HttpUtils.filterQueryParams(query, resolver.getUrlQueryParams())
: query; // raw, including token=secretThis is covered as intentional in UrlUtilsTest (preserve legacy query values), and HTTP integrations fork the same way (isDataCollectionConfigured → new filter, else sendDefaultPii / old header drop). With default options (dataCollection absent, sendDefaultPii=false), request/span/breadcrumb query strings still attach token=secret whenever a query is collected.
Same dual-path issue on cookies: e.g. OkHttp legacy sendDefaultPii=true returns the raw cookie string with no sensitive denylist pass (Spring at least runs filterOutSecurityCookies*).
Fix shape: always resolve an effective KeyValueCollectionBehavior (spec default deny-list, or a legacy bridge like JS defaultPiiToCollectionOptions) and run all query/cookie/header collection through HttpUtils.filter*. Keep category on/off and PII-term bridges in the resolver; do not bypass scrubbing when the namespace is unset.
2. Public databaseQueryData is a no-op
DataCollection / DataCollectionResolver.isDatabaseQueryData() are public and tested, but tip-of-stack has zero JDBC/SQLite (or other) call sites. #5801’s enforcement was dropped (Preserve query descriptions), so users can set the option with no effect.
Spec note: databaseQueryData gates bound params / payloads / results, not necessarily sanitized db.query.text. If Java cannot separate literals safely, either:
- drop the public option until there is a real consumer, or
- document + enforce a concrete policy in
sentry-jdbc/sentry-android-sqlite(and only suppress what the option is defined to control).
3. Non-Spring config surfaces still only know send-default-pii
Spring Boot binding works via SentryProperties extends SentryOptions + mutable KeyValueCollectionBehavior (#5834). ExternalOptions / sentry.properties and Android ManifestMetadataReader still only map send-default-pii. Peers expose dataCollection on their primary config path. At least property-file (and ideally manifest) keys for the new namespace should land with the stack, or the gap should be explicit in the PR/docs so hybrid and non-Spring users are not stuck on the legacy flag.
Spec / peer deltas (non-blocking, track explicitly)
| Area | Java stack | Spec / peers |
|---|---|---|
genAI, queues, stackFrameVariables, frameContextLines |
omitted | required where platform collects that data; OK to defer if unused |
graphql naming |
getGraphql() |
spec graphQL; Ruby/Python also snake/lower — fine if documented |
Boolean shorthand for key-value (true/false) |
modes + factories only | JS/Ruby accept bool shorthand — nice-to-have |
| Docs / wizard opt-out snippet | not in stack | spec MUST for init snippets |
| Major-version default flip | stay on sendDefaultPii until explicit dataCollection |
same staged approach as JS pre-v11 / Cocoa v10 |
Bottom line
Architecture and explicit-namespace behavior are in good shape and aligned with other SDKs. I would not merge until (1) sensitive scrubbing cannot be skipped on the legacy path, and (2) databaseQueryData is either enforced or removed from the public API. (3) is strongly preferred in the same release train as the feature.
Happy to re-review after those land on the tip branch.
Reframing as non-blocking comment per author request (minor release keeps legacy behavior until dataCollection opt-in).
There was a problem hiding this comment.
Re-posted as a non-blocking comment (dismissed the earlier changes-requested) so this does not block merge.
Reviewed the full Data Collection stack tip (fix/data-collection-opentelemetry-span-description) against data-collection 0.11.0 and the landed/in-flight peers (JS, Python, Cocoa, Ruby, Go).
Minor-release constraint acknowledged: until users opt into dataCollection, preserving current sendDefaultPii behavior is the right call. The notes below are tracking items / polish, not merge blockers for a minor.
What looks solid
- Config surface matches the required core:
userInfo,cookies/httpHeaders/urlQueryParamskey-value modes, directionalhttpBodies,graphQL,databaseQueryData. - Explicit-namespace resolution matches peers: once any
dataCollectionfield is set (or an emptyDataCollectionis assigned), omitted fields take spec defaults, notsendDefaultPii. That matches JSresolveDataCollectionOptionsand Go/Ruby “opt into defaults”. - Built-in sensitive terms match the denylist; partial case-insensitive matching and
[Filtered]substitution look correct inHttpUtils. - Integration wiring for Spring/Servlet/OkHttp/Ktor/Apollo/GraphQL is real and tested; Replay kept independent (spec 0.10.0); Android installation ID restored as non-
userInfoidentity (pragmatic, well motivated). - Span descriptions stripping queries (Ktor/OTel) matches the structuring-data rule.
- Dual-path design (
isDataCollectionConfigured→ new filters, else legacysendDefaultPii) is a deliberate compat bridge for a minor — same staged approach as JS pre-v11.
Non-blocking notes
1. Legacy path intentionally skips the new denylist until opt-in
UrlUtils.filterQueryParams returns raw query when the namespace is unset, and HTTP integrations fork the same way. That matches “keep doing what we did unless opted into data collection.” Spec-full scrubbing (and peer always-resolved policies) apply on the explicit path; full default flip is a major-version concern, not this minor.
Worth documenting clearly for users/hybrid SDKs so the dual path is not mistaken for incomplete wiring. Optional later hardening (major or opt-in-only): always resolve an effective KeyValueCollectionBehavior like JS defaultPiiToCollectionOptions once you are ready to change legacy defaults.
Cookie dual-path is slightly uneven today (OkHttp legacy sendDefaultPii=true can return raw cookies; Spring still runs filterOutSecurityCookies*) — pre-existing-ish, fine to leave for a follow-up unless you touch those call sites anyway.
2. Public databaseQueryData is currently a no-op
DataCollection / DataCollectionResolver.isDatabaseQueryData() are public and tested, but tip-of-stack has zero JDBC/SQLite (or other) call sites after #5801 dropped enforcement. Spec note: the option gates bound params / payloads / results, not necessarily sanitized db.query.text.
Suggestion when convenient: drop the public option until there is a real consumer, or wire a concrete policy in sentry-jdbc / sentry-android-sqlite and document the Java-specific literal limitation.
3. Non-Spring config surfaces still only know send-default-pii
Spring Boot binding works via SentryProperties extends SentryOptions + mutable KeyValueCollectionBehavior (#5834). ExternalOptions / sentry.properties and Android ManifestMetadataReader still only map send-default-pii. Fine for a first minor if programmatic + Spring are the supported opt-in paths — call out the gap in release notes so property-file / manifest users know they need code or Spring binding.
Spec / peer deltas (track explicitly)
| Area | Java stack | Spec / peers |
|---|---|---|
genAI, queues, stackFrameVariables, frameContextLines |
omitted | OK to defer if platform does not collect that data yet |
graphql naming |
getGraphql() |
spec graphQL; other SDKs also localize names |
| Boolean shorthand for key-value | modes + factories only | JS/Ruby accept bool shorthand — nice-to-have |
| Docs / wizard opt-out snippet | not in stack | spec MUST for init snippets when docs land |
| Major-version default flip | stay on sendDefaultPii until explicit dataCollection |
aligned with JS pre-v11 / Cocoa staged rollout |
Bottom line
Architecture and explicit-namespace behavior look good and aligned with other SDKs for a minor opt-in. No merge block from me — treat the items above as follow-ups / docs clarity rather than required changes for this train.
* feat(core): Add Data Collection configuration types Add the public Data Collection model, key-value collection behavior, and HTTP body direction types. Preserve unset values internally so the resolver can distinguish legacy bridge mode from explicit configuration. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com> * test(core): Remove Data Collection identity assertion Avoid coupling the Data Collection configuration test to reference identity. The test only needs to verify that setting a nested header behavior marks the configuration explicit. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com> * ref(core): Remove unused queue collection option Remove the queue option from the initial Data Collection API because the Java SDK does not collect queue payload data that it could control. Co-Authored-By: Claude <noreply@anthropic.com> * ref(core): Rename URL query parameter option Align the public Java API with the canonical Data Collection specification before release. The option has not shipped, so replace the old name without a compatibility alias. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com> * ref(core): Clarify forced Data Collection configuration Rename the internal override marker to explain that it forces an empty Data Collection object into explicit mode. Align the related tests with the clarified semantics. Refs #5666 Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
feat(core): [Data Collection 2] Expose SentryOptions API
feat(core): [Data Collection 3] Add policy resolver
feat(graphql): [Data Collection 4] Apply GraphQL policies
…-query-data feat(database): [Data Collection 5] Apply query policy
…-request-body feat(spring): [Data Collection 6] Apply incoming request body policy
…-response-body feat(apollo): [Data Collection 7] Apply incoming response body policy
…-request-body feat(apollo): [Data Collection 8] Apply outgoing request body policy
…-response-body feat(graphql): [Data Collection 9] Apply outgoing response body policy
…uest-headers feat(http): [Data Collection 10] Apply request header policy
…ponse-headers feat(http): [Data Collection 11] Apply response header policy
…rams feat(http): [Data Collection 12] Apply query parameter policy
PR Stack (Data Collection)
📜 Description
Collection PR for the Data Collection stack. The individual PRs add the configuration model, resolution and compatibility bridge, external configuration, filtering, and integration enforcement.
Squash-merge this PR into
mainonly after every stack PR has been merged into this branch using merge commits.💡 Motivation and Context
Introduce the specification-defined
dataCollectionconfiguration while preserving existingsendDefaultPiibehavior for users who do not opt into the new namespace.Refs #5666
💚 How did you test it?
This collection branch contains only an empty commit. Each stack PR carries its own tests.
📝 Checklist
sendDefaultPIIis enabled.🔮 Next steps
Merge the Data Collection stack into this branch in order, then squash-merge this PR into
main.#skip-changelog