Skip to content

test(relay): Assert rejection outcomes instead of waiting out ingestion - #131170

Draft
sentry-junior[bot] wants to merge 1 commit into
masterfrom
ci/relay-rejection-outcomes-v2
Draft

sentry-junior[bot] wants to merge 1 commit into
masterfrom
ci/relay-rejection-outcomes-v2

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Supersedes #129071. Builds on the outcome-capture idea from #128956, rebased onto current master.

post_and_try_retrieve_event swallowed the AssertionError from the success path. That made every expected Relay rejection wait out the 16s ingest timeout, and let unrelated failures pass as rejections.

post_and_assert_rejected replaces it. It reads the per-worker outcomes topic from the high watermarks captured before the POST, then waits for the outcome Relay emits for the project's org/project/key. It asserts that outcome's type and reason. Relay aggregates outcomes and drops event IDs, so the match uses org/project/key. That's unambiguous because project IDs are snowflakes and each case posts once.

via @joshuarli.

--

View Session [Sentry]

Negative Relay integration tests caught the AssertionError from the
successful-ingestion helper, so every expected rejection waited the full
16s ingest timeout and any unrelated assertion also counted as a rejection.

Read the per-worker outcomes topic from its current high watermarks and
wait for the error outcome Relay emits for the test's org, project, and key,
then assert its outcome type, reason, and quantity.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 14, 2026
@sentry-junior sentry-junior Bot added the Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling label Nov 14, 2026
@sentry-junior

sentry-junior Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Per-file durations from the full backend run on this PR (run 43539688316, Trigger: Override Selective Testing), compared with the three latest green master runs (43535410847, 43533907107, 43532740801). Each value is the sum of setup, call, and teardown from the backend-test-durations-* artifacts.

file master (3 runs) master median this PR
test_integration.py 144.5 / 145.5 / 135.4 144.5 52.0
test_message_filters.py 99.4 / 101.3 / 100.6 100.6 36.3
test_sdk.py (unchanged) 64.8 / 52.4 / 56.9 56.9 43.9
lang/java/test_plugin.py (unchanged) 201.6 / 171.9 / 189.8 189.8 122.5
lang/javascript/test_plugin.py (unchanged) 185.7 / 178.2 / 171.2 178.2 138.8
lang/javascript/test_example.py (unchanged) 11.6 / 13.7 / 11.0 11.6 10.8

Caveat: the runners on this run were faster than usual. Summed over every file, the whole suite took 14,947s here against 18,481 to 19,525s on master, about 0.78x. That explains the drop in the untouched files. After scaling the master medians by 0.78, the change itself saves about 61s in test_integration.py (113 → 52) and about 42s in test_message_filters.py (79 → 36). That matches nine negative cases going from ~16s to ~2.5s.

Locally, against devservices and the real Relay container, running test_message_filters.py and test_integration.py serially took 209.6s on master and 70.4s on this branch. Both files also pass under -n2 --dist=loadfile. Two deliberate breakages both failed: a wrong expected reason, and a filter turned off for an expected rejection (that one failed with "no error outcome", not a pass).

@linear-code

linear-code Bot commented Nov 14, 2026

Copy link
Copy Markdown

DI-2853

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant