Repository navigation
fix(ci): CI has been red since PR #25 -- test_orders.py's session-scoped table fixture - #31
Merged
Merged
Conversation
…into the full suite CI has been red on every push to main since PR #25 -- confirmed via `gh run list`, green through PR #24, failing on every merge since. Root cause, reproduced deterministically from a genuinely fresh, isolated Postgres/Redis container matching CI's exact setup (not my local dev DB, which had accumulated its own unrelated cross-session cruft this session and was a red herring for earlier, separate investigations): test_orders.py's `_order_tables` fixture was `scope="session", autouse=True` -- the only session-scoped table fixture in the entire backend test suite; every other API test file (test_portfolios_api.py, test_portfolios_holdings.py, test_valuation_api.py, test_funding.py, ...) uses a function-scoped autouse fixture that creates its tables before each test and drops them after. Session scope meant test_orders.py's 16 tables -- critically `security`, which nearly every other file also manages independently via its own `checkfirst=True` create -- stayed alive for the rest of the pytest *process*, not just for test_orders.py's own tests. Any other file that ran afterward in the same process silently reused this still-alive `security` table instead of getting a fresh one, and when that file's own per-test teardown tried its usual non-CASCADE `DROP TABLE "security"`, it failed with DependentObjectsStillExist because test_orders.py's `tax_lot` (also still alive) still referenced it. That single failed DROP rolls back its whole teardown transaction, so every table in that file's list survives with its data into the next file too -- explaining both observed CI symptoms: `uq_security_symbol` duplicate-key errors and a leftover "Growth" model_portfolio row bleeding into an unrelated assertion. This is why it only ever showed up in CI/a true full-suite run and never in any of this session's many scoped-file or small-combo local test runs -- confirmed by reproducing it with test_orders.py + test_portfolios_api.py in one pytest invocation (fails) versus the exact same two files run as separate invocations against the same DB state (passes) -- the collision requires both files' fixtures to be alive in the same pytest session, not just the same database. Fixed by making `_order_tables` function-scoped like every other file, folding in the now-redundant `_clean_order_tables` TRUNCATE-based second fixture (full create/drop per test already resets everything a TRUNCATE would have). Costs more CREATE/DROP churn across this file's ~50 tests, matching the cost every other file already pays for correct isolation. Verified: full `uv run pytest --cov=app --cov-report=term-missing` (CI's exact command) against a fresh, isolated Postgres/Redis container -- 932 passed, 0 failed, 0 errors, 84.52% coverage (floor 77%).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
mainhas failed CI on every push since PR #25 merged (confirmed viagh run list --branch main— green through PR #24, red on every merge since). Only the "Backend / tests + coverage floor" job
fails; migrations, SAST, secret scanning, and Docker builds all stay green.
Root cause (reproduced deterministically, not guessed)
Spun up a throwaway Postgres 16 + Redis container on separate ports, ran
docker/postgres/init.sqlagainst it, and ran
uv run pytest --cov=app --cov-report=term-missing— CI's exact command — toget a genuinely clean environment matching CI precisely (my own local dev DB had accumulated
unrelated cross-session state this session and was not a reliable baseline).
Reproduced the exact CI failure (2 failed, 930 passed, 31 errors) from a fresh environment,
then bisected it down to two files:
tests/api/test_orders.py+tests/api/test_portfolios_api.pyrun together in one pytest invocation.
test_orders.py's_order_tablesfixture wasscope="session", autouse=True— the onlysession-scoped table fixture in the entire backend test suite. Every other API test file
(
test_portfolios_api.py,test_portfolios_holdings.py,test_valuation_api.py,test_funding.py, ...) uses a function-scoped fixture: create tables before each test, drop themafter. Session scope meant
test_orders.py's 16 tables — criticallysecurity, which nearlyevery other file also manages independently — stayed alive for the rest of the pytest process,
not just for
test_orders.py's own tests.Any file that ran afterward in the same process silently reused this still-alive
securitytableinstead of getting a fresh one. When that file's own per-test teardown ran its usual non-CASCADE
DROP TABLE "security", it failed withDependentObjectsStillExist—test_orders.py'stax_lot(also still alive) still referenced it. That one failedDROProlled back its wholeteardown transaction, so every table in that file's list survived with its data into the next file
too — explaining both CI symptoms:
uq_security_symbolduplicate-key errors, and a leftover"Growth"
model_portfoliorow (fromtest_orders.py's session-scoped seed) bleeding into anunrelated assertion in
test_portfolios_api.py.Why this never showed up locally all session: confirmed it requires both files' fixtures alive
in the same pytest process — running them as two separate
uv run pytestinvocations againstthe identical database state does not reproduce it. Every local check this session ran scoped
files or small combos as separate invocations (by design, to avoid exactly the shared-DB
contention problem this session hit repeatedly with concurrent agents) — which is precisely the
shape that never triggers this bug.
Fix
Made
_order_tablesfunction-scoped, matching every other file's own established convention.Folded in the now-redundant
_clean_order_tablesTRUNCATE-based fixture (a full create/drop pertest already resets everything a
TRUNCATEwould have). Costs moreCREATE/DROPchurn acrossthis file's ~50 tests — the same cost every other file already pays for correct isolation.
Validation evidence
uv run ruff check,uv run mypy --strict app(pre-existing baseline noise only, confirmed viathe identical pattern already present in every other test file this session),
uv run lint-imports— clean.uv run pytest --cov=app --cov-report=term-missing→ 932 passed, 7 skipped, 2 xpassed, 0 failed, 0 errors, 84.52%coverage (floor 77%).
Security / DB / API impact
None. Test-only change — no application code touched.