Skip to content

Commit 95065ee

Browse files
leoromanovskyvjfridgedevflow.devflow-routing-intake
authored
feat(openfeature): emit server-side EVP flagevaluation (#11639)
refactor(evp): centralize EVP proxy routing refactor(feature-flagging): share EVP publishing feat(openfeature): add flagevaluation contract refactor(openfeature): name metrics hook explicitly feat(openfeature): add flagevaluation logging hook test(openfeature): cover flagevaluation logging hook fix(openfeature): snapshot flagevaluation context feat(openfeature): register flagevaluation hook feat(feature-flagging): canonicalize flagevaluation context feat(feature-flagging): aggregate flagevaluation rows test(feature-flagging): cover flagevaluation aggregation feat(feature-flagging): encode flagevaluation payloads test(feature-flagging): cover flagevaluation payload encoding feat(telemetry): support tagged core metric counts feat(feature-flagging): run flagevaluation writer lifecycle feat(feature-flagging): post flagevaluation payloads test(feature-flagging): add flagevaluation test support test(feature-flagging): cover flagevaluation writer feat(feature-flagging): wire flagevaluation writer lifecycle test(feature-flagging): cover flagevaluation writer lifecycle perf(feature-flagging): benchmark flagevaluation hot path chore: apply spotless formatting test(feature-flagging): cover flag evaluation jacoco paths fix(feature-flagging): gate flag evaluation enqueue during shutdown test(feature-flagging): cover flag eval event fix(feature-flagging): make aggregator count atomic Merge branch 'master' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Add agentless feature flag configuration source Clean up feature flagging configuration source naming Auto-enable feature flagging for explicit CDN source Add Java agentless feature flag source Address agentless configuration source feedback Refine agentless configuration source contract Merge remote-tracking branch 'origin' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Fix agentless feature flagging CI coverage Add custom endpoint for agentless feature flags Use integer agentless polling durations Fix agentless duration formatting Merge agentless configuration source fix(openfeature): serialize instant context values Define Feature Flagging configuration source contract Add Datadog-managed agentless UFC polling Support custom agentless UFC endpoints Select the Feature Flagging configuration source Merge master into agentless configuration source stack Merge agentless configuration source into aggregate EVP stack # Conflicts: # dd-trace-api/src/main/java/datadog/trace/api/config/FeatureFlaggingConfig.java # products/feature-flagging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java # products/feature-flagging/feature-flagging-agent/src/test/java/com/datadog/featureflag/FeatureFlaggingSystemTest.java Define Feature Flagging configuration source contract Add Datadog-managed agentless UFC polling Support custom agentless UFC endpoints Select the Feature Flagging configuration source Warn on agentless authentication failures Merge remote-tracking branch 'origin/master' into rewrite/java-agentless-narrative # Conflicts: # dd-trace-api/src/main/java/datadog/trace/api/config/FeatureFlaggingConfig.java # products/feature-flagging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java fix(ffe): handle nullable agentless response bodies Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-configuration-source Address agentless poller review feedback Fix feature flag config test imports Address additional agentless source review feedback Support the UFC CDN response contract Harden agentless configuration responses Rely on OkHttp gzip negotiation and cover truncated response cache preservation. Require JSON:API envelopes for custom endpoints. Separate UFC transport parsers Keep Remote Configuration on raw UFC parsing and give agentless JSON:API its own streaming envelope parser. Share UFC adapters without materializing an intermediate response map. Fix feature flagging test formatting Fix feature flagging parser coverage Address final feature flagging review feedback Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-configuration-source Merge branch 'leo.romanovsky/ffl-2693-java-agentless-configuration-source' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Re-anchor the EVP flagevaluation stack onto the updated agentless configuration source branch, dropping the divergent duplicate agentless history so the PR diff is flagevaluation-only. Conflicts resolved by taking 2693's canonical agentless/config work and combining it with the flag-evaluation writer lifecycle (relocated FeatureFlaggingConfig, merged FeatureFlaggingSystem start/stop, unioned tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-configuration-source fix(feature-flags): align configuration source semantics feat(feature-flags): delay agentless polling until provider use feat(communication): support mapped retried HTTP calls fix(feature-flags): align agentless HTTP behavior fix(feature-flags): complete initial poll during activation chore(feature-flags): remove unused agent feature Merge branch 'master' into leo.romanovsky/ffl-2693-java-agentless-configuration-source Fix race in agentless config source listener test scheduledPollContinuesAfterListenerRuntimeException waited on FakeClient.calls, which is incremented when a request starts rather than when it completes. The barrier therefore released as soon as the second poll began, letting the assertion race the poll thread that applies the configuration and notifies the listener. Wait on a CountDownLatch counted down by the listener itself, so the second notification is guaranteed to have happened before the assertions run. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: vickie.fridge <vickie.fridge@datadoghq.com> test(feature-flags): cover agentless lifecycle branches chore(agent): raise jar size budget to 34 MiB fix(feature-flags): document retry policy confinement Revert "chore(agent): raise jar size budget to 34 MiB" This reverts commit 29c1fd4. Merge remote-tracking branch 'origin/master' into agent/pr11892-cross-sdk-contract Merge branch 'leo.romanovsky/ffl-2693-java-agentless-configuration-source' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Bring the parent agentless configuration-source branch, and master through it, into the aggregate flagevaluation EVP stack. Conflict resolutions, all in feature-flagging-agent: - build.gradle.kts: keep the parent's compileOnly dependency on feature-flagging-config so the module stays out of the shaded jar, and restore the testImplementation entry that the test sources need. - FeatureFlaggingSystem: keep the parent's source selection, deferred agentless activation, and STARTED/ACTIVATION_LISTENER lifecycle, then layer the flagevaluation writer on top of initializeSystem so it starts for both the agentless and Remote Configuration sources. stop() now tears down both the activation listener and the flagevaluation writer. Dropped the ConfigurationSource enum, whose only caller the parent replaced with the shared FeatureFlaggingConfig string constants. - FeatureFlaggingSystemTest: union of both suites, minus a duplicate that a git reordering introduced. Two flagevaluation tests now trigger provider activation, since agentless defers initialization until the application provider activates, and startWithOfflineConfigurationSourceSkipsConfigService is dropped because the parent now disables the system entirely for the offline and invalid sources. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Correct FlagEvalLoggingHook hot-path cost documentation The class contract claimed finallyAfter does "ONLY cheap scalar extraction + a non-blocking offer to the writer's bounded queue". It also deep-copies the caller's evaluation context via DDEvaluator.snapshotValues synchronously on the evaluation thread; only flattenValues is deferred to the writer's worker thread. Document the real inline cost, why some copy is required (EvaluationContext is caller-owned and mutable, and the event is consumed asynchronously), and that this snapshot is unbounded even though the writer later prunes the context to 256 fields. Also rename FlagEvaluationHotPathBenchmark.evalThreadCapture to writerEnqueue and scope its javadoc to writer-side queue mechanics. It starts from a pre-built FlagEvalEvent whose supplier returns a ready-made flat map, so it never exercised the hook's inline capture - which is where the inaccurate "cheap capture" claim originated. Addresses review feedback on #11639. Comments only, plus the benchmark method rename; no behaviour change. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Make flagevaluation enqueue lock-free FlagEvaluationWriterImpl.enqueue wrapped its offer in synchronized (lifecycleLock), but the hand-off queue is a JCTools mpscBlockingConsumerArrayQueue - lock-free multi-producer by design. The monitor serialized every flag evaluation across all application threads, negating the reason that queue type was chosen. Measured with the new contention benchmark (16-core, per-producer ns/op): producers with lock lock-free change 1 305 ± 73 261 ± 14 ~1.2x 4 1747 ± 335 505 ± 24 3.5x 16 7551 ± 4561 1810 ± 520 4.2x Cost grew roughly linearly with producer count before, and the error bars shrank by ~9x at 16 producers: the monitor was a variance source as well as a throughput one. The lock did close the enqueue-vs-drain race, so removing it needs replacement accounting. close() set closed=true and called requestShutdown() under the lock, which ordered every offer before the worker's final drain. Without it a producer that already passed the closed check can offer after that drain, stranding an event in a queue nobody polls again. To keep the "loss is observable rather than silent" property: - the worker now makes SHUTDOWN_DRAIN_PASSES drain passes, covering the worker-self-close path where nothing joins the thread; and - close() sweeps the queue after joining the worker and counts the remainder as a `closed` drop. Guarded on the worker being dead, since sweeping beside a live worker would break the single-consumer contract. close() is also restructured so its !isAlive() early return no longer skips shutdown accounting. Benchmark uses @Group/@GroupThreads rather than @threads: writerEnqueue polls the queue, and polling from several threads violates the MPSC single-consumer contract, so producer threads cannot drain their own events. The single consumer batch-drains to keep the queue off its capacity limit, and overflow is reported per iteration so a saturated (invalid) run is visible. enqueueRechecksEnabledStateAfterTakingLifecycleLock existed only to assert that enqueue blocks on the monitor, so it is replaced rather than adapted: closeSweepsAndCountsEventsLeftInTheQueue covers the new shutdown accounting deterministically. Addresses review feedback on #11639. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Restore strict inequality in agent feature discovery The EVP proxy routing refactor incidentally changed discoverIfOutdated from 'elapsed > maxElapsedMs' to '>=' in a routine shared by CI Visibility, LLM Observability, telemetry and remote config. Feature flagging does not depend on it: it reaches discovery only through BackendApiFactory.createBackendApi, which uses the 60s discoverIfOutdated() path. Revert to keep this change out of the shared discovery timing. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Remove duplicate feature flagging shutdown path Agent.shutdown() stopped Feature Flagging twice: once through a no-arg shutdownFeatureFlagging() helper added on this branch, and once through shutdownFeatureFlagging(AGENT_CLASSLOADER), which reached master independently in #11892. Both resolve to FeatureFlaggingSystem.stop(), since shutdown() was only an alias for it, so the second call did nothing but repeat a reflective class load and re-log the stopped message. Drop this branch's helper, its call site, and the shutdown() alias, and point the tests at stop(). Agent.java now matches master, and the surviving path is the one covered by AgentFeatureFlaggingLifecycleTest. The enqueue gate that motivated the original helper is unaffected: it lives in stop(), which the remaining call site still reaches. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Master received the agentless configuration source via the squashed #11892 while this branch already carried the same work through its own merge, so the overlapping changes arrived by two routes. Conflict resolution, all resolving to the branch side after confirming master contributed nothing unique in each file: - AgentThreadFactory / FeatureFlaggingConfig: both sides appended adjacent constants; kept both sets. - FeatureFlaggingSystem and its test: master's copies are the branch's minus the flag-evaluation writer, so the branch side is a superset. The offline configuration-source test exists on both sides in different positions; kept one copy. - FeatureFlaggingGatewayTest: git merged this cleanly but both sides had independently added an identical testProviderActivationListener, giving a duplicate method that failed compilation. Removed the copy. Verified: test sources compile for every module both sides touched, and the feature-flagging, communication, agent-bootstrap, internal-api and dd-trace-api suites pass. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Allow benchmark stdout past forbidden-apis forbiddenApisJmh failed check_base on the queue-overflow warning in FlagEvaluationEnqueueContentionBenchmark: System.out is banned to keep excess logging out of the agent. A JMH benchmark is not agent code and stdout is the harness's own output channel, and the warning matters - it tells the operator the consumer fell behind and the iteration's enqueue timings describe overflow accounting instead of the enqueue path. Annotate the teardown with @SuppressForbidden, matching InjectorBenchmark and HotSpotStackWalkerBenchmark, which print to stdout the same way. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Drop the justification comment on benchmark @SuppressForbidden All thirteen existing @SuppressForbidden annotations in jmh source sets are bare, and the method's javadoc already explains why it prints, so the trailing comment only diverged from the convention without adding anything. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Stop Feature Flagging before telemetry at shutdown Agent.shutdown() ran stopTelemetry() before shutdownFeatureFlagging(), so the drop and degradation counters the flag-evaluation writer queues during its final flush had no telemetry worker left to drain them: TelemetryRunnable emits queued metrics from flushPendingTelemetryData(), which only runs while the telemetry thread is still alive. This branch originally called shutdownFeatureFlagging() from above stopTelemetry(). Deduplicating the two shutdown paths in 9e63356 kept the call site that reached master through #11892, which sits after stopTelemetry(), and moved the ordering as a side effect. Restore it. Costs up to the writer's 5s join before telemetry stops; total agent shutdown stays bounded. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Bound evaluation-context snapshot recursion depth DDEvaluator.snapshotValue() recurses once per nesting level of a caller-owned OpenFeature Value tree, on the application's evaluation thread. A pathologically deep list or structure overflows that stack, and StackOverflowError is neither an Exception nor a LinkageError, so it escapes both FlagEvalLoggingHook.finallyAfter's guard and the unguarded exposure dispatch path - letting default-on telemetry break an otherwise valid flag evaluation. Cap the snapshot at 32 levels and truncate past it to a null Value, matching how the existing cycle guard truncates. flattenValues() is already iterative, so nothing else needed changing. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Remove unused EVP proxy endpoint pinning The flag-evaluation writer posts to whatever EVP proxy endpoint the Agent advertises, exactly like the exposure writer, and disables response compression so the request is valid on both /evp_proxy/v2/ and /evp_proxy/v4/. Nothing in production ever pinned an endpoint: preferredEvpProxyEndpoint was always null, and DDAgentFeaturesDiscovery.supportsEvpProxyEndpoint() had no caller outside the branch it gated. Drop the parameter, the discovery helper, and the endpoint set it read - whose name also misdescribed its contents, since it held every endpoint the Agent advertises, not just the EVP proxy ones. createBackendApi keeps its responseCompression overload, which the flag-evaluation writer does use, and keeps coverage for the no-EVP-proxy and disabled-compression paths. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Clear flag-evaluation aggregator on encode failure A single unserializable value in a customer's evaluation context (for example Double.NaN, which Moshi rejects) would poison the aggregator: the encode path threw, the finally branch skipped `aggregator.clear()` because `aggregatesWereEncoded` stayed false, and every subsequent flush re-encoded and re-threw the same bucket forever. Clear the aggregator in the finally branch regardless of whether encoding succeeded. On success, this preserves the existing behavior. On failure, it discards the poisoned bucket so later flushes recover. Reported by Codex on the current head. Adds a regression test with Double.NaN in the context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Count drop when enqueue gate races shutdown FeatureFlaggingSystem.stop() flips the enqueue-enabled gate before FlagEvaluationWriterImpl.close() runs. An in-flight producer that resolved the writer reference before the gate flip could reach enqueue(), see the gate false while closed=false, and drop the event without incrementing DROP_REASON_CLOSED. Shutdown loss disappeared from telemetry. Rename countClosedDropIfClosed() to countClosedDrop() and remove the inner guard. The early-exit path in enqueue() only fires when isClosedOrEnqueueDisabled() is true, so we can attribute the drop to DROP_REASON_CLOSED unconditionally. Reported by Codex on the current head. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Suppress OkHttp transparent gzip when responseCompression=false OkHttp's BridgeInterceptor adds a transparent Accept-Encoding: gzip whenever the caller does not set the header. EvpProxyApi previously skipped adding the header when responseCompression=false, so the BridgeInterceptor added gzip anyway and the flag had no wire effect. The flagevaluation track was documented as "response compression disabled to match the merged Go behavior" but negotiated gzip on the wire. Set Accept-Encoding: identity explicitly when responseCompression=false to preempt the BridgeInterceptor. The v=true path is unchanged. Reported by Codex on the current head. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Align queue capacity and snapshot depth with cross-SDK RFC The Java hand-off queue (65,536) was 16x wider than the shared RFC target of 4,096 events, and the evaluation-context snapshot depth (32) was 8x deeper than the RFC target of 4. Every other merged SDK uses the RFC numbers: dd-trace-go queue=4,096 depth=4 dd-trace-rb queue=4,096 depth=4 dd-trace-py queue=4,096 depth=4 dd-trace-js queue=4,096 depth=4 libdatadog depth=4 (sidecar coalescer, no app-thread queue) Bring the Java hot path in line so a full queue holds at most 4,096 pre-aggregated events and a caller-owned Value tree is bounded at 4 nesting levels, matching cross-SDK behavior and shrinking the pre-queue capture footprint by ~128x on paper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Cap every context dimension inline via copyPrunedContext and add pre-queue guard The hot path previously called snapshotValues to deep-copy the caller's EvaluationContext (bounded only by depth and cycles) and deferred flatten and prune to the worker. That let a wide, long-string, or long-key context sit un-pruned in the hand-off queue, and let the app thread pay the full snapshot cost even when the queue was already full and the event would be dropped. DDEvaluator.copyPrunedContext replaces snapshotValues + flattenValues + FlagEvaluationAggregator.pruneContext on the EVP hot path. It walks the caller-owned context once and caps every retained-size dimension inline: MAX_CONTEXT_FIELDS = 256 top-level fields kept MAX_KEY_LENGTH = 256 chars per key (closes prior gap) MAX_VALUE_LENGTH = 256 chars per string value MAX_LIST_ELEMENTS = 256 elements walked per list MAX_STRUCTURE_PROPERTIES = 256 properties walked per structure MAX_SNAPSHOT_DEPTH = 4 nesting depth (cross-SDK RFC) Every limit is a named constant so it can be tuned in isolation. Work on the hot path is now proportional to what is retained, never to what the caller supplied. FlagEvaluationWriter gains hasCapacityForEnqueue()/countPreQueueOverflow() so FlagEvalLoggingHook can short-circuit when the queue is saturated, counting the drop as queue_overflow without doing any context-copy work. FlagEvaluationAggregator.pruneContext becomes a passthrough. Aggregator tests assert the new store-what-you-get contract; the pruning contract moves to DDEvaluatorTest.copyPrunedContext coverage. Cross-SDK note: other SDKs still prune on the worker. This change is a Java-specific "do no harm" hardening; the same design is a candidate to fan out. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Emit flagevaluation.context.truncated metric per evaluation with sorted deduplicated reason tag copyPrunedContext now returns a CopyResult carrying the pruned attrs map and an optional truncatedReason string (sorted comma-separated cap names, e.g. "max_key_length,max_value_length"). The hook calls w.countContextTruncated(reason) when any cap fires, and the writer batches counts per unique reason string in a ConcurrentHashMap, draining them at each 10s flush cycle as flagevaluation.context.truncated metrics with "reason:..." tags. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Remove dead lazy-context supplier path and pruneContext passthrough FlagEvalEvent: drop supplier constructor, attrsSupplier field, and contextAttributes() — all callers now pass pre-pruned attrs directly. FlagEvaluationAggregator: read event.attrs directly instead of going through pruneContext(event.contextAttributes()); delete pruneContext() which was a no-op passthrough. FlagEvaluationWriterImpl: remove pruneContext/canonicalContextKey delegation methods, MAX_CONTEXT_FIELDS/MAX_FIELD_LENGTH constants, and DROP_REASON_CONTEXT_ERROR — the supplier-throw path that triggered context_error is gone so the constant and its metric are unreachable. Tests: delete enqueueDoesNotResolveContextBeforeBuffering, contextMaterializationFailureDropsSingleEvent, publicConstructorAndContextHelpersDelegateToSharedImplementations, and aggregatorPruneContextIsPassthrough; update remaining callsites to use event.attrs directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Remove stale javadoc and unused aggregator constant re-exports from writer FlagEvalEvent: fix class javadoc that still described the removed lazy supplier path. FlagEvaluationWriterImpl: remove six unreferenced constant re-exports (EVAL_SCALE_FULL_BUCKET_TARGET, EVAL_SCALE_PER_FLAG_BUCKET_TARGET, EVAL_SCALE_DEGRADED_BUCKET_TARGET, GLOBAL_CAP, PER_FLAG_CAP, DEGRADED_CAP) — all delegated to FlagEvaluationAggregator but had zero callers on the writer side. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Strip Javadoc HTML tags from flagevaluation PR comments Remove <p>, {@code}, {@link}, <ul>/<li>, and <em> tags from all comments written in this PR. Plain prose reads more clearly without IDE rendering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> refactor(flagevaluation): address dd-oleksii review comments - Flatten nested try/finally in FeatureFlaggingSystem.stop() to per-resource try/catch for readability - Use a single evalTimestampMs for both startAt/endAt gating and the metadata timestamp so the logged eval time matches the allocation selection decision - Rename metadata key dd.eval.timestamp_ms -> __dd_eval_timestamp_ms to signal internal-only intent; update FlagEvalLoggingHook reader and tests in lockstep - Move the isEmpty/droppedQueueOverflow guard into shouldFlush() so flushIfNecessary() is a single-responsibility delegate Co-Authored-By: Claude <noreply@anthropic.com> refactor(flagevaluation): narrow FlagEvalLoggingHook catch to LinkageError only The OpenFeature SDK's executeAfterAllHooks already wraps finallyAfter calls in a catch(Exception) and logs them. Our Exception catch was redundant. LinkageError is kept because the SDK catches Exception, not Throwable, leaving classloading failures unguarded. Co-Authored-By: Claude <noreply@anthropic.com> refactor(flagevaluation): swallow close() exceptions in stop() without logging At shutdown there is nowhere to surface the error usefully, and each writer already logs internally on failure. Debug-logging from the catch blocks was misleading noise. True swallow (Exception ignored) matches the intent of the original suggestion. Co-Authored-By: Claude <noreply@anthropic.com> refactor(flagevaluation): extract closeQuietly helper in FeatureFlaggingSystem.stop() Replaces four identical try/catch blocks with a single closeQuietly helper. SpanEnrichmentWriter gains AutoCloseable (it already had a close() method) so all four resources share the same call site. Co-Authored-By: Claude <noreply@anthropic.com> refactor(flagevaluation): rename EVAL_SCALE_* constants in FlagEvaluationAggregator EVAL_SCALE_ prefix was ambiguous — it read as a runtime scale factor rather than design-time sizing assumptions. Split into two named groups: - EXPECTED_* for the design assumptions (flag count, users per flag, etc.) - *_SIZING_BASIS for the derived intermediate values - Inline comments on GLOBAL_CAP and DEGRADED_CAP explain they are the nearest powers of two above the respective sizing bases. Co-Authored-By: Claude <noreply@anthropic.com> refactor(flagevaluation): simplify drainAndFlush to a single pass The multi-pass loop with Thread.yield() between passes was a heuristic for catching producers mid-enqueue during shutdown. A single poll loop already drains everything in the queue at that point. For events that race past the drain, close() sweeps the queue after joining the worker and counts any remainder as an observable drop — making extra passes redundant. Removed SHUTDOWN_DRAIN_PASSES constant and the loop. Co-Authored-By: Claude <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Align FlagEvalEventTest with removed lazy-context API The lazy Supplier overload and contextAttributes() accessor were removed in 29b974d, breaking compileTestJava. Drop the two tests that covered the deleted lazy path and remove the contextAttributes() assertions from the remaining tests. Co-Authored-By: Claude <noreply@anthropic.com> Fix JMH benchmarks broken by removal of lazy-Supplier FlagEvalEvent overload The lazy Supplier constructor was removed in 29b974d; pass attrs directly. Also rename nextLazyEvent -> nextEvent in FlagEvaluationHotPathBenchmark. Co-Authored-By: Claude <noreply@anthropic.com> Restore coverage for SCO constructor and countContextTruncated Commit 29b974d removed the lazy-supplier path and took two tests with it, dropping FlagEvaluationWriterImpl branch/instruction coverage below the 90% Jacoco threshold on Java 8. Add two replacement tests: - scoConstructorCreatesUsableWriter: exercises the public SCO constructor - countContextTruncatedAccumulatesPerReason: exercises countContextTruncated Co-Authored-By: Claude <noreply@anthropic.com> Cover hasCapacityForEnqueue and countPreQueueOverflow on writer impl The pre-queue guard methods were only exercised through mocks in the hook test, leaving FlagEvaluationWriterImpl branch coverage at 0.844 and failing jacocoTestCoverageVerification (threshold 0.9). Add a direct test that observes both branches of hasCapacityForEnqueue and the countPreQueueOverflow counter surfacing as a queue_overflow drop metric. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Parse observeFullEvaluationData and hash targeting_key in flagevaluations events (#12042) * Parse observeFullEvaluationData and hash targeting_key in flagevaluations events Adds the top-level observeFullEvaluationData boolean to the UFC model, plumbs it through to the EVP flagevaluation event serializer, and gates PII handling on it: when the flag is absent/false the targeting key is SHA-256 hashed (sha256_<hex>) and the raw evaluation context is omitted from the wire; when true the raw targeting key and context are emitted. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Extract hashed targeting key prefix into a named constant Replace the inline "sha256_" literal with a documented HASHED_TARGETING_KEY_PREFIX constant describing the cross-SDK wire contract for privacy-preserving hashed targeting keys. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Test observeFullEvaluationData parsing edge cases Parameterize the true/false config-parsing assertions with @valuesource and add a test locking in the fail-closed behaviour for an explicit JSON null: malformed config is rejected so full evaluation data is never observed off the back of it. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData per bucket at aggregation time The flush-time read of FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() was a TOCTOU bug: CURRENT_CONFIG could be overwritten by a later RC update between when an evaluation happened and when the batch flushed, so events could be emitted under the wrong environment's consent (the system test observed a targeting key hashed even though the active UFC said observeFullEvaluationData=true). Capture consent when the evaluation is folded into its EvalBucket instead. On merge the value is folded with AND, so any no-consent evaluation in a bucket's lifetime sinks the whole bucket to hashed/omitted (fail-closed). buildEventList now reads bucket.observeFullEvaluationData rather than the gateway. The gateway accessor is retained; it is read at aggregation time. Adds a writer-level regression guard (a bucket aggregated under consent-off stays hashed even if the gateway later reports consent-on) plus aggregator fold tests, and an end-to-end parse->dispatch->flush test. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData consent at evaluation time Snapshot the PII consent flag on the evaluation thread (in the OpenFeature hook) and carry it on FlagEvalEvent, instead of reading the gateway when the event is aggregated/flushed. This pins the hashed-vs-raw decision to the configuration active at evaluation time, closing a one-directional leak window where a later Remote Config update could retroactively apply a different environment's consent to already-collected evaluations. Aggregation and flush now read event.observeFullEvaluationData and never consult the gateway; the AND-fold across a bucket's evaluations is unchanged (any no-consent evaluation sinks the bucket to hashed/omitted). Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Bind observeFullEvaluationData consent to the evaluator's configuration Address PR #12042 review feedback (Codex P1, leoromanovsky, dd-oleksii): the FlagEvalLoggingHook was reading observeFullEvaluationData from FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() at hook-fire time, which races against a Remote Config swap of CURRENT_CONFIG that happens after DDEvaluator.evaluate() captured its own ServerConfiguration reference. That race can retroactively mark an evaluation performed without consent as consented and leak the raw targeting key / context. DDEvaluator now stamps the boolean directly from the ServerConfiguration it used, onto every ProviderEvaluation via ImmutableMetadata under key "dd.observe_full_evaluation_data". The hook reads consent from that metadata and no longer queries the gateway. Missing metadata (PROVIDER_NOT_READY or a non-DD provider) → false, the privacy-preserving default. The gateway's isObserveFullEvaluationDataEnabled() accessor is removed since its only real caller was the hook and re-adding it would re-open the race. Adds regression tests: hook honours consent metadata (true/false/absent) and ignores a gateway value that disagrees; evaluator stamps the correct boolean on the FLAG_NOT_FOUND path and omits metadata when it holds no config. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Pass the boolean, not the ServerConfiguration, into error/resolveVariant Follow-up to the previous commit: the private error() and resolveVariant() helpers only ever read one field off the ServerConfiguration (observeFullEvaluationData), so pass the boolean directly instead of the whole config. Keeps the internal API narrow and removes the incidental coupling these helpers had to the UFC. While here, PROVIDER_NOT_READY now stamps consent as the privacy-preserving false rather than omitting the metadata. Same on-the-wire outcome the hook would have produced, but the invariant "every DD-produced evaluation carries dd.observe_full_evaluation_data" is now unconditional, which is easier to reason about. The two error() overloads collapse to one (the (String) null casts at call sites disappear along with them). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop the dd. prefix on the evaluation-metadata consent key The key is only ever read by FlagEvalLoggingHook one line later — it never lands on the wire, so it doesn't need the "dd." namespacing that "dd.eval.timestamp_ms" has (that key is re-emitted onto spans). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Trim verbose comments around the observeFullEvaluationData plumbing The race-vs-CURRENT_CONFIG backstory is captured in the previous commits' messages; the code only needs the forward-looking invariants (metadata is source of truth, missing key = false, DD-produced evaluations always stamp). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation context when aggregating consent-off evaluations Address PR #12042 review from leoromanovsky (escalated Codex P2 → P1): on the protected path (observeFullEvaluationData=false) the serializer drops the evaluation context, but the aggregator was still running it through pruneContext + canonicalContextKey and keying every full-tier bucket on it. A high-cardinality field on the evaluation context (request_id, timestamp, correlation id) would fragment buckets that emit byte-identical wire rows, blow out PER_FLAG_CAP (10k) inside one flush window, and force subsequent evaluations into the degraded tier — which drops the targeting key entirely. On the protected path aggregate() now uses ctxKey="" and stores prunedAttrs=null, so different contexts for the same subject collapse into one bucket. The targeting key stays in the aggregation identity, so different subjects still hash to different buckets. The consent-on path is unchanged. Regression tests: protected path collapses differing contexts for one subject; protected path still separates distinct subjects; full path still splits on context. Existing tests that exercise pruneContext / context-differentiation were updated to use consent=on (that's the code path they actually cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation-context capture on the hook hot path when consent is off Companion to the aggregator fix: with observeFullEvaluationData=false the evaluation context is dropped on emit and no longer influences aggregation, so there is no reason to snapshot it on the evaluation thread. The hook now branches on consent up front — the protected path enqueues an event with an empty materialized attrs map (no map copy of the OpenFeature context, no Supplier<Map> allocation, no lambda instance), while the consent-on path is unchanged. Grep confirms the only production consumer of FlagEvalEvent.contextAttributes / FlagEvalEvent.attrs is FlagEvaluationAggregator.aggregate, which already skips them on the protected path. Regression test: mutating the EvaluationContext after finallyAfter returns still yields empty attrs on the enqueued event — proves the hook never snapshotted it. Two existing tests that exercise the snapshot mechanism were switched to pass consent-on metadata (that's the code path they cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Include observeFullEvaluationData in the aggregation bucket key Bucket keys should cover every dimension the emitter will branch on. The serializer branches on observeFullEvaluationData (hashes the targeting key and drops the context when off), so two evaluations that differ only in consent produce different wire rows and must not share a bucket. Before this change they could: same subject, same flag, same empty context would land under the same FullKey regardless of consent, and the AND-fold would silently downgrade a consent-on evaluation to the protected wire shape because a nearby consent-off event merged into its bucket first. No PII leak (fail-closed direction), but arrival-order-dependent semantics and a lost raw-context row. Add observeFullEvaluationData to FullKey / DegradedKey (equals + hashCode). The AND-fold on bucket.observeFullEvaluationData stays as defensive belt- and-suspenders; every event merging into a bucket now carries the matching consent value by construction. Regression test: two events identical except for consent land in two full- tier buckets, one consent-on and one consent-off. Updated the previous "fold to false on mixed consent" test to reflect the new invariant. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add consent metadata to ProviderTest flag-eval-logging hook route test The test asserts that context attributes flow through the logging hook, but the mock metadata omitted the observe-full-evaluation-data flag, so the hook took the privacy-preserving path and dropped context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Redact error messages when observeFullEvaluationData is off Exception messages from the evaluator's outer catch blocks (NumberFormatException, generic Exception) can echo raw evaluation-context values verbatim — for example a GT rule on "id" with a PII-shaped targeting key produced error.message="For input string: \"jane.doe@...\"" on the wire regardless of consent, defeating the PR's own PII guard. Drop the message at DDEvaluator.error() when consent is off, and add a hook-layer fallback that substitutes ErrorCode.name() so operators keep a stable signal (e.g. "TYPE_MISMATCH") even when a third-party provider hands us a raw message. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Exercise every consent-stamp code path in DDEvaluatorTest The only observeFullEvaluationData assertions were on error paths (FLAG_NOT_FOUND, PROVIDER_NOT_READY), leaving the success-path stamp in resolveVariant and the DISABLED/DEFAULT stamps in consentMetadata uncovered — line 448 could be deleted or hardcoded to either value and every existing test would still pass. Add symmetric consent-on/consent-off tests for each of resolveVariant, DISABLED, and DEFAULT so any mutation (delete / hardcode true / hardcode false) flips at least one assertion. Rename the previously misleading …OnSuccess test to reflect what it actually exercises (FLAG_NOT_FOUND error via error()). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop consent from DegradedKey to reclaim effective DEGRADED_CAP Two degraded buckets differing only in observeFullEvaluationData emit byte-identical wire JSON — the degraded serializer (fromBucket with isFullTier=false) drops the targeting key and context regardless of consent — so the consent dimension in DegradedKey halved effective DEGRADED_CAP for zero wire fidelity gain. FullKey correctly keeps consent (the full-tier serializer branches on it for raw-vs-hashed targeting key and context inclusion). Mixed-consent events now merge into one degraded bucket. The AND-fold on EvalBucket.observeFullEvaluationData still runs and collapses to false whenever any consent-off event lands in a mixed bucket; benign because the value has no downstream effect for degraded rows. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tolerate malformed observeFullEvaluationData in UFC parse Before this change ServerConfiguration.observeFullEvaluationData was a primitive boolean — Moshi's reflective adapter rejected the entire UFC whenever the JSON value was null or wrong-typed. Agentless swallows the IOException at DEBUG, so a pod starting after a malformed message had no last-known-good, stranded every flag on PROVIDER_NOT_READY, and served defaults forever. Fail-closed on privacy shouldn't cascade into fail-closed on availability. Box the field to Boolean so null tolerates naturally, register a LenientBooleanAdapter that maps wrong-typed values to null as well, and read via Boolean.TRUE.equals(...) at the DDEvaluator so null falls to the privacy-preserving default. The lenient adapter only intercepts Boolean (not primitive boolean), so mandatory fields like Flag.enabled keep their strict parse; the only other Boolean it touches is Allocation.doLog, which is already read as `!= null && doLog`. Reversed the earlier RejectsExplicitNull test — it had locked in the buggy behaviour — into a family of tolerance tests for null / stringified / numeric. Added a DDEvaluator test that a config with a null consent field evaluates without NPE and stamps the privacy-preserving default. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Set observeFullEvaluationData=true for the NaN-poison flush test The consent-off short-circuit in FlagEvaluationEvent.fromBucket drops the raw context before Moshi encodes it, so a NaN in the attrs never reaches the encoder and the flush succeeds. That defeated the intent of encodeFailureClearsAggregatorSoLaterFlushesRecover, which must observe a real encode failure to prove the aggregator is cleared. Co-Authored-By: Claude <noreply@anthropic.com> * Cover LenientBooleanAdapter read-only and qualifier paths The per-class JaCoCo gate (0.9 minimum, gradle/jacoco.gradle) failed on the new adapter: toJson was never invoked (20/25 instructions) and the factory's !annotations.isEmpty() short-circuit never evaluated true (3/4 branches). Neither path is reachable through the parse-driven tests in JsonApiUfcResponseParserTest. Mirror the tests the sibling FlagMapAdapter and DateAdapter already have. The primitive-boolean assertion documents the guard that keeps this leniency off mandatory fields like Flag.enabled. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> perf(feature-flagging): drop hot-path allocations in aggregator hashing/canonicalization Address review feedback from @AlexeyKuznetsov-DD on the FlagEvaluationAggregator: - Replace String.format("%08x", ...) in appendLengthDelimited with a StringBuilder-based zero-padded hex writer. Format on the per-context-field path allocates a Formatter and boxes the length argument on every call. - Rewrite FullKey.hashCode and DegradedKey.hashCode with HashingUtils.hash / addToHash. Objects.hash allocates an Object[] per call and boxes the boolean dimensions; both keys are looked up on every event dispatch. Co-Authored-By: Claude <noreply@anthropic.com> Roll back partially initialized state when feature flagging start fails The failure handler only cleared STARTED. If initializeSystem threw after initialize() had published CONFIG_SERVICE and EXPOSURE_WRITER, those stayed open and unreachable, and the gateway kept its enqueue flag. Route both start paths through one initializeOrRollBack helper that calls stop() on failure, which closes every published resource, removes any activation listener, and clears the gateway flags. This also removes the duplicated try/catch shared by start() and activateAgentless(). Also publish FLAG_EVAL_WRITER before evalWriter.start(), so a writer whose start fails is still reachable by the rollback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Measure canonicalization and restore the hook hot-path benchmark FlagEvaluationHotPathBenchmark built its events with the convenience constructor, which defaults observeFullEvaluationData to false. Under consent-off the aggregator drops attrs and skips canonicalContextKey, so every field-count profile measured the same scalar-only work: 10 fields and 250 fields both landed near 30 ns. Pass consent true so the benchmark exercises canonicalization again. The profiles now spread from 498 ns at 10 fields to 17.3 us at 250 fields. Add FlagEvalHookHotPathBenchmark in feature-flagging-api to cover the evaluation-thread cost the lib benchmark cannot reach. It splits total inline cost, the bounded context copy alone, and the consent-off floor across flat, nested, and list context shapes. This required enabling the me.champeau.jmh plugin on the module, which had no benchmark source set. Also refresh the lib benchmark javadoc, which still described an attribute-supplier API that no longer exists and claimed the hook cost was unmeasured anywhere. Co-Authored-By: Claude <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffl-2446-evp-flagevaluation-java Clear the interrupt flag before the flagevaluation final flush close() interrupts the worker thread to break it out of queue.poll(100ms). The worker then falls into the finally block in run(), which calls drainAndFlush() to do the final flush that close() exists to guarantee. The interrupt flag is still set at that point. That flush does socket I/O, and OkHttp fails fast on a thread whose interrupt flag is set. The resulting IOException is swallowed by the broad catch in flush(), whose finally then calls aggregator.clear(). So every aggregated evaluation in the final flush window was discarded on every clean shutdown, and the loss was invisible: it surfaced as an error log rather than a drop metric. close()'s post-join sweep does not cover this. The sweep counts events still sitting in the queue. These rows had already been drained out of the queue into the aggregator before flush() threw, so the sweep cannot see them. Save and clear the interrupt flag around drainAndFlush(), then restore it so the thread still exits with the correct interrupt status. The bug needed both halves to appear, and neither is wrong alone: the interrupt came from 5c6cb2b ("Make flagevaluation enqueue lock-free"), which needs it to wake the worker now that the lock no longer orders offers against the drain; the finally-block drain came from 5f88cad. The existing closeDrainsAndFinalFlushesQueuedEvents test passes against the bug because a mocked publisher ignores the interrupt flag - only a real socket reacts to it. The new test therefore asserts on the flag state at the moment post() is invoked. Verified it fails without the production change: "expected: <false> but was: <true>". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: vjfridge <vickie.fridge@datadoghq.com> Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
1 parent 4eabb06 commit 95065ee

50 files changed

Lines changed: 6681 additions & 152 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

communication/src/main/java/datadog/communication/BackendApiFactory.java

Lines changed: 25 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ public BackendApiFactory(Config config, SharedCommunicationObjects sharedCommuni
2424
}
2525

2626
public @Nullable BackendApi createBackendApi(Intake intake) {
27+
return createBackendApi(intake, true);
28+
}
29+
30+
public @Nullable BackendApi createBackendApi(Intake intake, boolean responseCompression) {
2731
HttpRetryPolicy.Factory retryPolicyFactory = new HttpRetryPolicy.Factory(5, 100, 2.0, true);
2832

2933
if (intake.isAgentlessEnabled(config)) {
@@ -46,23 +50,28 @@ public BackendApiFactory(Config config, SharedCommunicationObjects sharedCommuni
4650
DDAgentFeaturesDiscovery featuresDiscovery =
4751
sharedCommunicationObjects.featuresDiscovery(config);
4852
featuresDiscovery.discoverIfOutdated();
49-
if (featuresDiscovery.supportsEvpProxy()) {
50-
String traceId = config.getIdGenerationStrategy().generateTraceId().toString();
51-
String evpProxyEndpoint = featuresDiscovery.getEvpProxyEndpoint();
52-
HttpUrl evpProxyUrl = sharedCommunicationObjects.agentUrl.resolve(evpProxyEndpoint);
53-
String subdomain = intake.getUrlPrefix();
54-
return new EvpProxyApi(
55-
traceId,
56-
evpProxyUrl,
57-
subdomain,
58-
retryPolicyFactory,
59-
sharedCommunicationObjects.agentHttpClient,
60-
true);
53+
if (!featuresDiscovery.supportsEvpProxy()) {
54+
log.warn(
55+
"Cannot create backend API client since agentless mode is disabled, "
56+
+ "and agent does not support EVP proxy");
57+
return null;
6158
}
59+
String evpProxyEndpoint = featuresDiscovery.getEvpProxyEndpoint();
6260

63-
log.warn(
64-
"Cannot create backend API client since agentless mode is disabled, "
65-
+ "and agent does not support EVP proxy");
66-
return null;
61+
String traceId = config.getIdGenerationStrategy().generateTraceId().toString();
62+
log.debug(
63+
"Creating EVP proxy client for {} using endpoint {} with responseCompression={}",
64+
intake,
65+
evpProxyEndpoint,
66+
responseCompression);
67+
HttpUrl evpProxyUrl = sharedCommunicationObjects.agentUrl.resolve(evpProxyEndpoint);
68+
String subdomain = intake.getUrlPrefix();
69+
return new EvpProxyApi(
70+
traceId,
71+
evpProxyUrl,
72+
subdomain,
73+
retryPolicyFactory,
74+
sharedCommunicationObjects.agentHttpClient,
75+
responseCompression);
6776
}
6877
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
package datadog.communication;
2+
3+
/** Shared EVP proxy constants. */
4+
public final class EvpProxy {
5+
6+
public static final String SUBDOMAIN_HEADER = "X-Datadog-EVP-Subdomain";
7+
8+
/**
9+
* Default SDK-side target for uncompressed EVP request bodies. Writers may split batches at or
10+
* below this size to keep Agent proxy requests comfortably bounded.
11+
*/
12+
public static final int PAYLOAD_SIZE_LIMIT_BYTES = 5 * 1024 * 1024;
13+
14+
private EvpProxy() {}
15+
}

communication/src/main/java/datadog/communication/EvpProxyApi.java

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@ public class EvpProxyApi implements BackendApi {
2020
private static final Logger log = LoggerFactory.getLogger(EvpProxyApi.class);
2121

2222
private static final String API_VERSION = "v2";
23-
private static final String X_DATADOG_EVP_SUBDOMAIN_HEADER = "X-Datadog-EVP-Subdomain";
2423
private static final String X_DATADOG_TRACE_ID_HEADER = "x-datadog-trace-id";
2524
private static final String X_DATADOG_PARENT_ID_HEADER = "x-datadog-parent-id";
2625
private static final String ACCEPT_ENCODING_HEADER = "Accept-Encoding";
2726
private static final String CONTENT_ENCODING_HEADER = "Content-Encoding";
2827
private static final String GZIP_ENCODING = "gzip";
28+
private static final String IDENTITY_ENCODING = "identity";
2929

3030
private final String traceId;
3131
private final HttpRetryPolicy.Factory retryPolicyFactory;
@@ -62,7 +62,7 @@ public <T> T post(
6262
Request.Builder requestBuilder =
6363
new Request.Builder()
6464
.url(url)
65-
.addHeader(X_DATADOG_EVP_SUBDOMAIN_HEADER, subdomain)
65+
.addHeader(EvpProxy.SUBDOMAIN_HEADER, subdomain)
6666
.addHeader(X_DATADOG_TRACE_ID_HEADER, traceId)
6767
.addHeader(X_DATADOG_PARENT_ID_HEADER, traceId);
6868

@@ -74,11 +74,21 @@ public <T> T post(
7474
requestBuilder.addHeader(CONTENT_ENCODING_HEADER, GZIP_ENCODING);
7575
}
7676

77+
// OkHttp's BridgeInterceptor adds a transparent Accept-Encoding: gzip when the caller does not
78+
// set one. Set the header explicitly on both paths so responseCompression=false actually
79+
// suppresses gzip negotiation on the wire.
7780
if (responseCompression) {
7881
requestBuilder.addHeader(ACCEPT_ENCODING_HEADER, GZIP_ENCODING);
82+
} else {
83+
requestBuilder.addHeader(ACCEPT_ENCODING_HEADER, IDENTITY_ENCODING);
7984
}
8085

8186
final Request request = requestBuilder.post(requestBody).build();
87+
log.debug(
88+
"Posting EVP request to {} with responseCompression={} requestCompression={}",
89+
url,
90+
responseCompression,
91+
requestCompression);
8292

8393
try (okhttp3.Response response =
8494
OkHttpUtils.sendWithRetries(httpClient, retryPolicyFactory, request)) {
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
package datadog.communication;
2+
3+
import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V4_EVP_PROXY_ENDPOINT;
4+
import static org.junit.jupiter.api.Assertions.assertEquals;
5+
import static org.junit.jupiter.api.Assertions.assertNotNull;
6+
import static org.junit.jupiter.api.Assertions.assertNull;
7+
8+
import datadog.communication.ddagent.DDAgentFeaturesDiscovery;
9+
import datadog.communication.ddagent.SharedCommunicationObjects;
10+
import datadog.metrics.api.Monitoring;
11+
import datadog.trace.api.Config;
12+
import datadog.trace.api.ProtocolVersion;
13+
import datadog.trace.api.intake.Intake;
14+
import java.nio.charset.StandardCharsets;
15+
import okhttp3.HttpUrl;
16+
import okhttp3.MediaType;
17+
import okhttp3.OkHttpClient;
18+
import okhttp3.RequestBody;
19+
import okhttp3.mockwebserver.MockResponse;
20+
import okhttp3.mockwebserver.MockWebServer;
21+
import okhttp3.mockwebserver.RecordedRequest;
22+
import org.junit.jupiter.api.Test;
23+
24+
class BackendApiFactoryTest {
25+
26+
private static final MediaType JSON = MediaType.parse("application/json");
27+
28+
@Test
29+
void noBackendApiWhenAgentDoesNotAdvertiseEvpProxy() {
30+
final FakeFeaturesDiscovery discovery = new FakeFeaturesDiscovery(null);
31+
final BackendApiFactory factory =
32+
new BackendApiFactory(Config.get(), sharedCommunicationObjects(discovery, null));
33+
34+
assertNull(factory.createBackendApi(Intake.EVENT_PLATFORM, false));
35+
}
36+
37+
@Test
38+
void advertisedEvpProxyEndpointSupportsDisabledResponseCompression() throws Exception {
39+
final MockWebServer agent = new MockWebServer();
40+
agent.enqueue(new MockResponse().setResponseCode(200).setBody("{}"));
41+
agent.start();
42+
try {
43+
final FakeFeaturesDiscovery discovery = new FakeFeaturesDiscovery(V4_EVP_PROXY_ENDPOINT);
44+
final BackendApiFactory factory =
45+
new BackendApiFactory(
46+
Config.get(), sharedCommunicationObjects(discovery, agent.url("/")));
47+
final BackendApi api = factory.createBackendApi(Intake.EVENT_PLATFORM, false);
48+
49+
assertNotNull(api);
50+
api.post(
51+
"flagevaluation",
52+
RequestBody.create(JSON, "{}".getBytes(StandardCharsets.UTF_8)),
53+
stream -> null,
54+
null,
55+
false);
56+
57+
final RecordedRequest request = agent.takeRequest();
58+
assertEquals("/evp_proxy/v4/api/v2/flagevaluation", request.getPath());
59+
} finally {
60+
agent.shutdown();
61+
}
62+
}
63+
64+
private static SharedCommunicationObjects sharedCommunicationObjects(
65+
final DDAgentFeaturesDiscovery discovery, final HttpUrl agentUrl) {
66+
final TestSharedCommunicationObjects sco = new TestSharedCommunicationObjects(discovery);
67+
sco.agentUrl = agentUrl != null ? agentUrl : HttpUrl.get("http://localhost:8126/");
68+
sco.agentHttpClient = new OkHttpClient();
69+
return sco;
70+
}
71+
72+
private static final class TestSharedCommunicationObjects extends SharedCommunicationObjects {
73+
private final DDAgentFeaturesDiscovery discovery;
74+
75+
private TestSharedCommunicationObjects(final DDAgentFeaturesDiscovery discovery) {
76+
this.discovery = discovery;
77+
}
78+
79+
@Override
80+
public DDAgentFeaturesDiscovery featuresDiscovery(final Config config) {
81+
return discovery;
82+
}
83+
}
84+
85+
private static final class FakeFeaturesDiscovery extends DDAgentFeaturesDiscovery {
86+
private final String evpProxyEndpoint;
87+
88+
private FakeFeaturesDiscovery(final String evpProxyEndpoint) {
89+
super(
90+
new OkHttpClient(),
91+
Monitoring.DISABLED,
92+
HttpUrl.get("http://localhost:8126/"),
93+
ProtocolVersion.V0_5,
94+
true,
95+
false);
96+
this.evpProxyEndpoint = evpProxyEndpoint;
97+
}
98+
99+
@Override
100+
public void discoverIfOutdated() {}
101+
102+
@Override
103+
public String getEvpProxyEndpoint() {
104+
return evpProxyEndpoint;
105+
}
106+
107+
@Override
108+
public boolean supportsEvpProxy() {
109+
return evpProxyEndpoint != null;
110+
}
111+
}
112+
}

dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -524,15 +524,17 @@ public static void shutdown(final boolean sync) {
524524
if (profilingEnabled) {
525525
shutdownProfilingAgent(sync);
526526
}
527+
// Before telemetry: the feature flagging writers queue drop/degradation metrics during their
528+
// final flush, and only a still-running telemetry worker can drain and transmit them.
529+
if (featureFlaggingEnabled) {
530+
shutdownFeatureFlagging(AGENT_CLASSLOADER);
531+
}
527532
if (telemetryEnabled) {
528533
stopTelemetry();
529534
}
530535
if (flareEnabled) {
531536
stopFlarePoller();
532537
}
533-
if (featureFlaggingEnabled) {
534-
shutdownFeatureFlagging(AGENT_CLASSLOADER);
535-
}
536538

537539
if (agentlessLogSubmissionEnabled) {
538540
shutdownLogsIntake();

dd-trace-core/src/main/java/datadog/trace/common/writer/ddintake/DDEvpProxyApi.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import static datadog.trace.common.writer.DDIntakeWriter.DEFAULT_INTAKE_TIMEOUT;
44
import static datadog.trace.common.writer.DDIntakeWriter.DEFAULT_INTAKE_VERSION;
55

6+
import datadog.communication.EvpProxy;
67
import datadog.communication.http.HttpRetryPolicy;
78
import datadog.communication.http.OkHttpUtils;
89
import datadog.trace.api.civisibility.InstrumentationBridge;
@@ -26,7 +27,6 @@ public class DDEvpProxyApi extends RemoteApi {
2627

2728
private static final Logger log = LoggerFactory.getLogger(DDEvpProxyApi.class);
2829

29-
private static final String DD_EVP_SUBDOMAIN_HEADER = "X-Datadog-EVP-Subdomain";
3030
private static final String CONTENT_ENCODING_HEADER = "Content-Encoding";
3131
private static final String GZIP_CONTENT_TYPE = "gzip";
3232

@@ -131,7 +131,7 @@ public Response sendSerializedTraces(Payload payload) {
131131
Request.Builder builder =
132132
new Request.Builder()
133133
.url(proxiedApiUrl)
134-
.addHeader(DD_EVP_SUBDOMAIN_HEADER, subdomain)
134+
.addHeader(EvpProxy.SUBDOMAIN_HEADER, subdomain)
135135
.tag(OkHttpUtils.CustomListener.class, telemetryListener);
136136

137137
if (isCompressionEnabled()) {

dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import static datadog.communication.http.OkHttpUtils.gzippedMsgpackRequestBodyOf;
44

5+
import datadog.communication.EvpProxy;
56
import datadog.communication.serialization.GrowableBuffer;
67
import datadog.communication.serialization.Writable;
78
import datadog.communication.serialization.msgpack.MsgPackWriter;
@@ -99,7 +100,7 @@ public class LLMObsSpanMapper implements RemoteMapper {
99100
private int spansWritten;
100101

101102
public LLMObsSpanMapper() {
102-
this(5 << 20);
103+
this(EvpProxy.PAYLOAD_SIZE_LIMIT_BYTES);
103104
}
104105

105106
private LLMObsSpanMapper(int size) {

dd-trace-core/src/test/groovy/datadog/trace/common/writer/ddintake/DDEvpProxyApiTest.groovy

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package datadog.trace.common.writer.ddintake
22

33
import com.fasterxml.jackson.core.type.TypeReference
44
import com.fasterxml.jackson.databind.ObjectMapper
5+
import datadog.communication.EvpProxy
56
import datadog.communication.serialization.ByteBufferConsumer
67
import datadog.communication.serialization.FlushingBuffer
78
import datadog.communication.serialization.msgpack.MsgPackWriter
@@ -64,7 +65,7 @@ class DDEvpProxyApiTest extends DDCoreSpecification {
6465
clientResponse.status().present
6566
clientResponse.status().asInt == 200
6667
agentEvpProxy.getLastRequest().path == path
67-
agentEvpProxy.getLastRequest().getHeader(DDEvpProxyApi.DD_EVP_SUBDOMAIN_HEADER) == intakeSubdomain
68+
agentEvpProxy.getLastRequest().getHeader(EvpProxy.SUBDOMAIN_HEADER) == intakeSubdomain
6869

6970
cleanup:
7071
agentEvpProxy.close()
@@ -100,7 +101,7 @@ class DDEvpProxyApiTest extends DDCoreSpecification {
100101
clientResponse.status().present
101102
clientResponse.status().asInt == 200
102103
agentEvpProxy.getLastRequest().path == path
103-
agentEvpProxy.getLastRequest().getHeader(DDEvpProxyApi.DD_EVP_SUBDOMAIN_HEADER) == intakeSubdomain
104+
agentEvpProxy.getLastRequest().getHeader(EvpProxy.SUBDOMAIN_HEADER) == intakeSubdomain
104105

105106
cleanup:
106107
agentEvpProxy.close()

internal-api/src/main/java/datadog/trace/api/telemetry/CoreMetricCollector.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,14 @@ private CoreMetricCollector() {
2929
this.metricsQueue = new ArrayBlockingQueue<>(RAW_QUEUE_SIZE);
3030
}
3131

32+
public void count(String metricName, long value, String tag) {
33+
if (value <= 0) {
34+
return;
35+
}
36+
this.metricsQueue.offer(
37+
new CoreMetric(METRIC_NAMESPACE, true, metricName, "count", value, tag));
38+
}
39+
3240
@Override
3341
public void prepareMetrics() {
3442
// Collect span metrics

internal-api/src/main/java/datadog/trace/util/AgentThreadFactory.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ public enum AgentThread {
6767
LLMOBS_EVALS_PROCESSOR("dd-llmobs-evals-processor"),
6868

6969
FEATURE_FLAG_EXPOSURE_PROCESSOR("dd-ffe-exposure-processor"),
70+
FEATURE_FLAG_EVALUATION_PROCESSOR("dd-ffe-evaluation-processor"),
7071
FEATURE_FLAG_CONFIGURATION_POLLER("dd-feature-flagging-http-poller");
7172

7273
public final String threadName;

0 commit comments

Comments
 (0)