Skip to content

perf(explore): Add a request deadline to trace item stats - #130922

Open
sentry-junior[bot] wants to merge 5 commits into
masterfrom
perf/explore-trace-item-stats-deadline
Open

sentry-junior[bot] wants to merge 5 commits into
masterfrom
perf/explore-trace-item-stats-deadline

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

OrganizationTraceItemStatsEndpoint (the attribute breakdowns tab in Explore) runs a table RPC, then a preflight stats RPC, then up to 4 parallel stats RPCs. Each RPC has its own 30s Snuba timeout (SENTRY_SNUBA_TIMEOUT), but nothing bounds the request as a whole. Slow requests take 13-22s as several sequential 1-7s calls, and the worst case is about 3 × 30s. Browsers often abort these requests while the backend keeps working.

Design. The request gets one time budget, created when the request starts:

  • Before the preflight RPC and before the stats RPCs, check how much budget is left. If it has run out, return a 504 with the same message handle_query_errors uses for a Snuba RPC timeout, and start no more Snuba work.
  • Wait for the parallel stats RPCs only as long as the remaining budget (wait(..., timeout=remaining)). On timeout, cancel futures that haven't started and shut the pool down without waiting, so the response doesn't wait on RPCs that are still in flight.
  • A api.trace_item_stats.deadline_exceeded metric, tagged by stage, shows which stage runs out of budget.

Budget. New option explore.trace-items.stats.deadline-seconds (Float, automator-modifiable, 0 disables it), following explore.trace-items.* and performance.traces.pagination.max-timeout. The default is 30s, the same as the per-RPC SENTRY_SNUBA_TIMEOUT. A lower value such as 15s would sit just under the sampled p99 (16.7s) and turn some requests that succeed today into 504s. At 30s, none of the sampled requests would have hit the budget, and it still caps the worst case of chained RPCs (about 90s today). The UI already has a dedicated 504 state ("Query timed out"), so this PR doesn't touch the frontend. The budget can be tightened through the automator later, using api.trace_item_stats.deadline_exceeded to see how often requests hit it.

Limitation. The RPC layer has no per-call timeout. _make_rpc_request uses the pool-wide timeout and run_table_query/run_stats_query take no timeout argument, and changing snuba_rpc was out of scope here. So an RPC that is already running can't be cut short. The table and preflight RPCs can each still run up to 30s, and a stats RPC that has been abandoned keeps running in the background until its own timeout. Worst case is now about budget + 30s (about 60s with the default) instead of about 90s.

Tests. New cases in tests/snuba/api/endpoints/test_organization_trace_item_stats.py mock the endpoint's clock and don't sleep:

  • budget spent before the preflight: 504, and no stats RPCs run
  • budget spent before the stats stage: 504, and only the preflight runs
  • budget runs out while stats RPCs are in flight: 504, returned without waiting for them
  • request within budget: 200 with the normal data

via Nick.

--

View Session [Sentry]

The trace item stats endpoint chains a table RPC, a preflight stats RPC, and parallel stats RPCs. Each RPC has its own 30s Snuba timeout, but nothing bounds the request as a whole. Check a shared time budget before each stage and bound the parallel stats RPCs by the time left, returning the usual Snuba RPC timeout 504 once the budget runs out.

Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 12, 2026
@nsdeschenes nsdeschenes self-assigned this Nov 12, 2026
@nsdeschenes

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 3b2df21. Configure here.

15s sits just under the sampled p99 and would turn some requests that currently succeed into 504s. 30s matches SENTRY_SNUBA_TIMEOUT and still caps chained RPCs.

Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io>
@nsdeschenes
nsdeschenes marked this pull request as ready for review November 12, 2026 17:38
@nsdeschenes
nsdeschenes requested review from a team as code owners November 12, 2026 17:38
…item-stats-deadline

Co-Authored-By: Nicholas Deschenes <nicholas.deschenes@sentry.io>

# Conflicts:
#	src/sentry/api/endpoints/organization_trace_item_stats.py
#	tests/snuba/api/endpoints/test_organization_trace_item_stats.py
…item-stats-deadline

# Conflicts:
#	src/sentry/api/endpoints/organization_trace_item_stats.py
…item-stats-deadline

# Conflicts:
#	src/sentry/api/endpoints/organization_trace_item_stats.py

This branch was successfully deployed

1 active deployment
Preview — a927be6a Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant