Enable EVP flagevaluation system tests for Java

[leoromanovsky]

Motivation

Java needs the shared server-side EVP flagevaluation system-test contract enabled so APM reviewers can validate that Java evaluations produce the same backend-visible signal as the other SDKs. This PR provides the public system-tests companion for the Java SDK fanout.

Changes

• Enables tests/ffe/test_flag_eval_evp.py for Java spring-boot in manifests/java.yml at v1.66.0-SNAPSHOT.
• Leaves other Java weblogs marked irrelevant.
• Keeps the test implementation inherited from current system-tests main.

Decisions

• Enable only spring-boot because that is the Java weblog covered by the companion SDK validation.
• Keep this PR to the Java manifest surface; SDK behavior remains in DataDog/dd-trace-java#11639.
• Existing OTel metric coverage stays in tests/ffe/test_flag_eval_metrics.py; this PR is only for EVP flagevaluation.
• Version gate updated from v1.64.0-SNAPSHOT → v1.66.0-SNAPSHOT. The original v1.64.0-SNAPSHOT was an estimate from when this PR was opened (June 20); the SDK implementation (DataDog/dd-trace-java#11639) did not merge until Aug 12, after v1.65.1 (Aug 10). The feature is only on dd-trace-java master (1.66.0-SNAPSHOT) and is not in any released tag, so the gate must be v1.66.0-SNAPSHOT (the first version containing the feature). With the old gate, the test would run but fail against v1.64.0/v1.65.0/v1.65.1.

Validation Evidence

System Tests

• Companion SDK PR: Emit server-side EVP flagevaluation dd-trace-java#11639 (MERGED)
• This PR enables the Java manifest surface used by that SDK PR for EVP flagevaluation validation.

Local run (Aug 17)

Ran against a fresh dd-java-agent/dd-trace-api/dd-openfeature build from dd-trace-java master (1.66.0-SNAPSHOT+8180203cd7, which contains the merged feature):

./build.sh java -w spring-boot
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_evp.py

Result: 11 passed, 0 failures, 0 errors, 0 skipped in 96s.

• Test_FFE_EVP_Flagevaluation_Basic
• Test_FFE_EVP_Flagevaluation_Count
• Test_FFE_EVP_Flagevaluation_Context_Bounds
• Test_FFE_EVP_Flagevaluation_Runtime_Default
• Test_FFE_EVP_Flagevaluation_Load_Aggregation
• Test_FFE_EVP_Flagevaluation_Burst_Aggregation
• Test_FFE_EVP_Flagevaluation_High_Cardinality_Aggregation
• Test_FFE_EVP_Flagevaluation_Degradation
• Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed
• Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed
• Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed

[github-actions]

CODEOWNERS have been resolved as:

manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java

[datadog-datadog-prod-us1-2]

 

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (java, dev) / End-to-end #1 / spring-boot 1 — ❌ 11 tests failed · 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitHub Actions

11 failed tests. Timed out waiting for EVP flagevaluation events for multiple flags including 'evp-basic-flag', 'evp-count-flag', and 'evp-degradation-flag'.

Showing tests most relevant to this failure.

❌ tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Basic.test_ffe_evp_flagevaluation_basic[spring-boot] from system_tests_suite  

AssertionError: Timed out waiting for EVP flagevaluation event for flag evp-basic-flag
assert False
 +  where False = <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')>(<function wait_for_evp_flagevaluation_event.<locals>.<lambda> at 0x7f0daec274c0>, timeout=30)
 +    where <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')> = AgentInterfaceValidator('agent').wait_for
 +      where AgentInterfaceValidator('agent') = interfaces.agent

self = <tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Basic object at 0x7f0df8687b90>

    def test_ffe_evp_flagevaluation_basic(self) -> None:
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
...

❌ tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Count.test_ffe_evp_flagevaluation_count[spring-boot] from system_tests_suite  

AssertionError: Timed out waiting for EVP flagevaluation event for flag evp-count-flag
assert False
 +  where False = <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')>(<function wait_for_evp_flagevaluation_event.<locals>.<lambda> at 0x7f0daedaf880>, timeout=30)
 +    where <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')> = AgentInterfaceValidator('agent').wait_for
 +      where AgentInterfaceValidator('agent') = interfaces.agent

self = <tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Count object at 0x7f0df8687770>

    def test_ffe_evp_flagevaluation_count(self) -> None:
        for index, response in enumerate(self.responses):
...

❌ tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Degradation.test_ffe_evp_flagevaluation_degradation[spring-boot] from system_tests_suite  

AssertionError: Timed out waiting for EVP flagevaluation count >= 12000 for flag evp-degradation-flag
assert False
 +  where False = <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')>(<function wait_for_evp_flagevaluation_count.<locals>.<lambda> at 0x7f0daea060c0>, timeout=60)
 +    where <bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator('agent')> = AgentInterfaceValidator('agent').wait_for
 +      where AgentInterfaceValidator('agent') = interfaces.agent

self = <tests.ffe.test_flag_eval_evp.Test_FFE_EVP_Flagevaluation_Degradation object at 0x7f0df8687620>

    def test_ffe_evp_flagevaluation_degradation(self) -> None:
        for index, response in enumerate(self.responses):
...

↳ and 8 more — View all

Testing the test | all-jobs-are-green

View in Datadog · View in GitHub Actions

CI checks or statuses failed during execution.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: leo.romanovsky/ffe-evp-flagevaluation-enable-java-clean

Testing the test | System Tests (java, dev) / End-to-end #1 / spring-boot 1
Commit: a30030588ffa6c20fa52dbe26dc603a94ad0ed05
Error (code / test):
11 failed tests. Timed out waiting for EVP flagevaluation events for multiple flags including 'evp-basic-flag', 'evp-count-flag', and 'evp-degradation-flag'.
CI job: https://github.com/DataDog/system-tests/actions/runs/32178574277/job/95849517716

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dc1a527 | Docs | View more details | Give us feedback!}

[datadog-system-tests-org]

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (java, dev) / End-to-end #1 / spring-boot 1     

Testing the test | all-jobs-are-green     

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 923dc46 | Docs | Give us feedback!}

[vjfridge]

Decision: This PR is blocked until the next dd-trace-java release (expected 9/7/26)

The EVP flagevaluation feature spans two separate dd-trace-java artifacts that are versioned and released together but updated independently in the system-tests dev build.

• dd-java-agent.jar (the consumer-side worker that drains the evaluation queue and flushes to /evp_proxy/v4/api/v2/flagevaluation) is pulled fresh from S3 master on every dev run
• dd-openfeature.jar (the producer-side OpenFeature hook that captures each evaluation and enqueues it) is resolved by the weblog's Maven [1.56.0,) range from the latest released version — currently
  1.65.1, which predates PR #11639 that added FlagEvalLoggingHook.

The Java SDK team acknowledges this cross-jar split is suboptimal and is already pursuing simplification work, so rather than invest in teaching the CI infra to live-update dd-openfeature.jar for this one case, we'll wait foe the next dd-trace-java release (expected ~Sept 7), at which point dd-openfeature 1.66.0 will land on Maven Central with the EVP hook, the weblog will resolve it automatically, and the v1.66.0-SNAPSHOT gate in this manifest will be satisfied for both dev and prod.

The system tests have already been run against a dev dd-java-agent containing the consumer code, giving us confidence they'll pass once the producer side catches up via release.