Skip to content

feat(testutils): fail tests that overflow Postgres's subtransaction cache - #131123

Open
ldelvoye wants to merge 4 commits into
masterfrom
ldelvoye/feat/savepoint-count-check
Open

ldelvoye wants to merge 4 commits into
masterfrom
ldelvoye/feat/savepoint-count-check

Conversation

@ldelvoye

@ldelvoye ldelvoye commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

part of INFRENG-846

Adds a pytest plugin that counts, for each transaction the code under test opens, the savepoints that hold a Postgres subtransaction id. A test fails once one transaction goes past 64 of them, the point where Postgres overflows its subtransaction cache and every query on the database slows down while that transaction stays open. One function opening 6 or more savepoints in a single transaction is listed in a savepoint loops section at the end of the run, without failing anything.

It lives in sentry.testutils.pytest, so getsentry's tests run it too. Like stale_database_reads, a test that needs to can opt out with self.savepoint_usage.clear().

Code that overflows today goes in a savepoint_baseline list in each repo's pyproject.toml: its overflows are listed in a savepoint baseline section instead of failing, and an entry nothing hits is flagged for removal. Sentry's one entry is the self-hosted relocation import, which runs the whole import in one transaction under monolith databases. getsentry's entry comes from getsentry/getsentry#23235.

It only sees code the tests run, at the sizes the tests use, so a loop over 3 test rows will not reach 64 even if production feeds it thousands. How long a transaction stays open is left to production monitoring.

The current list of baselined and reported code: https://www.notion.so/4178b10e4b5d81fc8baac1545b732fe6

Merge order:

  1. getsentry/getsentry#23235: getsentry baseline entry, plus a filter for the option sentry does not register yet
  2. feat(testutils): fail tests that overflow Postgres's subtransaction cache #131123: the savepoint check; then wait for the bot to bump sentry-version in getsentry
  3. getsentry/getsentry#23236: drop the filter

…ache

Counts, per application transaction, the savepoints that hold a
subtransaction id. A test fails at 64, the point where Postgres overflows
its cache and slows every query on the database. A caller that opens 6
or more in one transaction is listed in a savepoint loops summary
without failing.
Groups a transaction's savepoints by the first application function
that opened them, without fixed frame depths, and stops a write's walk
at the first savepoint that already holds an id.
@linear-code

linear-code Bot commented Nov 13, 2026

Copy link
Copy Markdown

INFRENG-846

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 13, 2026
Adds a savepoint_baseline pytest ini option. An overflowing transaction
that was opened in a listed function, or whose savepoints were mostly
opened by one, no longer fails its test. It is listed under a
"savepoint baseline" section of the terminal summary instead, with each
test that hit the entry, and entries no test hit are flagged for
removal.

Each repo lists its own entries, so getsentry's stay out of sentry.
Sentry's one entry covers the 22 relocation import tests that overflow
under monolith databases.

Opening paths are now relative to the pytest rootdir, so getsentry's
read getsentry/... on any checkout, and the stale_database_reads signal
wrapper no longer claims the savepoints its receivers open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ldelvoye
ldelvoye marked this pull request as ready for review November 14, 2026 00:19
@ldelvoye
ldelvoye requested review from a team as code owners November 14, 2026 00:19
@ldelvoye
ldelvoye requested a review from a team November 14, 2026 00:19

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 20062c3. Configure here.

Postgres caches 64 subtransaction ids per transaction and overflows on
the 65th, so a transaction holding exactly 64 still fits and no longer
fails.

The statement head is now taken after stripping leading whitespace, so
an indented raw statement keeps its verb and its writes are counted.

The rollback case now rolls back a savepoint with released children,
which pins that ROLLBACK TO frees the ids of everything opened after
its target.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant