Repository navigation
perf(explore): Add a request deadline to trace item stats - #130922
Open
sentry-junior[bot] wants to merge 5 commits into
Open
sentry-junior[bot] wants to merge 5 commits into
sentry-junior[bot] wants to merge 5 commits into
Conversation
The trace item stats endpoint chains a table RPC, a preflight stats RPC, and parallel stats RPCs. Each RPC has its own 30s Snuba timeout, but nothing bounds the request as a whole. Check a shared time budget before each stage and bound the parallel stats RPCs by the time left, returning the usual Snuba RPC timeout 504 once the budget runs out. Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io>
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3b2df21. Configure here.
15s sits just under the sampled p99 and would turn some requests that currently succeed into 504s. 30s matches SENTRY_SNUBA_TIMEOUT and still caps chained RPCs. Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io>
nsdeschenes
marked this pull request as ready for review
November 12, 2026 17:38
…item-stats-deadline Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io> # Conflicts: # src/sentry/api/endpoints/organization_trace_item_stats.py # tests/snuba/api/endpoints/test_organization_trace_item_stats.py
…item-stats-deadline # Conflicts: # src/sentry/api/endpoints/organization_trace_item_stats.py
…item-stats-deadline # Conflicts: # src/sentry/api/endpoints/organization_trace_item_stats.py
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OrganizationTraceItemStatsEndpoint(the attribute breakdowns tab in Explore) runs a table RPC, then a preflight stats RPC, then up to 4 parallel stats RPCs. Each RPC has its own 30s Snuba timeout (SENTRY_SNUBA_TIMEOUT), but nothing bounds the request as a whole. Slow requests take 13-22s as several sequential 1-7s calls, and the worst case is about 3 × 30s. Browsers often abort these requests while the backend keeps working.Design. The request gets one time budget, created when the request starts:
handle_query_errorsuses for a Snuba RPC timeout, and start no more Snuba work.wait(..., timeout=remaining)). On timeout, cancel futures that haven't started and shut the pool down without waiting, so the response doesn't wait on RPCs that are still in flight.api.trace_item_stats.deadline_exceededmetric, tagged bystage, shows which stage runs out of budget.Budget. New option
explore.trace-items.stats.deadline-seconds(Float, automator-modifiable,0disables it), followingexplore.trace-items.*andperformance.traces.pagination.max-timeout. The default is 30s, the same as the per-RPCSENTRY_SNUBA_TIMEOUT. A lower value such as 15s would sit just under the sampled p99 (16.7s) and turn some requests that succeed today into 504s. At 30s, none of the sampled requests would have hit the budget, and it still caps the worst case of chained RPCs (about 90s today). The UI already has a dedicated 504 state ("Query timed out"), so this PR doesn't touch the frontend. The budget can be tightened through the automator later, usingapi.trace_item_stats.deadline_exceededto see how often requests hit it.Limitation. The RPC layer has no per-call timeout.
_make_rpc_requestuses the pool-wide timeout andrun_table_query/run_stats_querytake no timeout argument, and changingsnuba_rpcwas out of scope here. So an RPC that is already running can't be cut short. The table and preflight RPCs can each still run up to 30s, and a stats RPC that has been abandoned keeps running in the background until its own timeout. Worst case is now about budget + 30s (about 60s with the default) instead of about 90s.Tests. New cases in
tests/snuba/api/endpoints/test_organization_trace_item_stats.pymock the endpoint's clock and don't sleep:via Nick.
--
View Session [Sentry]