Skip to content

fix(ci): CI has been red since PR #25 -- test_orders.py's session-scoped table fixture - #31

Merged
Badri-Rajendran merged 1 commit into
mainfrom
fix_ci_order_tables_session_scope
Sep 7, 2026
Merged

Badri-Rajendran merged 1 commit into
mainfrom
fix_ci_order_tables_session_scope

Conversation

@Badri-Rajendran

Copy link
Copy Markdown
Owner

Problem

main has failed CI on every push since PR #25 merged (confirmed via gh run list --branch main
— green through PR #24, red on every merge since). Only the "Backend / tests + coverage floor" job
fails; migrations, SAST, secret scanning, and Docker builds all stay green.

Root cause (reproduced deterministically, not guessed)

Spun up a throwaway Postgres 16 + Redis container on separate ports, ran docker/postgres/init.sql
against it, and ran uv run pytest --cov=app --cov-report=term-missing — CI's exact command — to
get a genuinely clean environment matching CI precisely (my own local dev DB had accumulated
unrelated cross-session state this session and was not a reliable baseline).

Reproduced the exact CI failure (2 failed, 930 passed, 31 errors) from a fresh environment,
then bisected it down to two files: tests/api/test_orders.py + tests/api/test_portfolios_api.py
run together in one pytest invocation.

test_orders.py's _order_tables fixture was scope="session", autouse=True — the only
session-scoped table fixture in the entire backend test suite. Every other API test file
(test_portfolios_api.py, test_portfolios_holdings.py, test_valuation_api.py,
test_funding.py, ...) uses a function-scoped fixture: create tables before each test, drop them
after. Session scope meant test_orders.py's 16 tables — critically security, which nearly
every other file also manages independently — stayed alive for the rest of the pytest process,
not just for test_orders.py's own tests.

Any file that ran afterward in the same process silently reused this still-alive security table
instead of getting a fresh one. When that file's own per-test teardown ran its usual non-CASCADE
DROP TABLE "security", it failed with DependentObjectsStillExist — test_orders.py's
tax_lot (also still alive) still referenced it. That one failed DROP rolled back its whole
teardown transaction, so every table in that file's list survived with its data into the next file
too — explaining both CI symptoms: uq_security_symbol duplicate-key errors, and a leftover
"Growth" model_portfolio row (from test_orders.py's session-scoped seed) bleeding into an
unrelated assertion in test_portfolios_api.py.

Why this never showed up locally all session: confirmed it requires both files' fixtures alive
in the same pytest process — running them as two separate uv run pytest invocations against
the identical database state does not reproduce it. Every local check this session ran scoped
files or small combos as separate invocations (by design, to avoid exactly the shared-DB
contention problem this session hit repeatedly with concurrent agents) — which is precisely the
shape that never triggers this bug.

Fix

Made _order_tables function-scoped, matching every other file's own established convention.
Folded in the now-redundant _clean_order_tables TRUNCATE-based fixture (a full create/drop per
test already resets everything a TRUNCATE would have). Costs more CREATE/DROP churn across
this file's ~50 tests — the same cost every other file already pays for correct isolation.

Validation evidence

  • uv run ruff check, uv run mypy --strict app (pre-existing baseline noise only, confirmed via
    the identical pattern already present in every other test file this session), uv run lint-imports — clean.
  • Full suite, CI's exact command, fresh isolated environment: uv run pytest --cov=app --cov-report=term-missing → 932 passed, 7 skipped, 2 xpassed, 0 failed, 0 errors, 84.52%
    coverage
    (floor 77%).

Security / DB / API impact

None. Test-only change — no application code touched.

…into the full suite

CI has been red on every push to main since PR #25 -- confirmed via
`gh run list`, green through PR #24, failing on every merge since. Root
cause, reproduced deterministically from a genuinely fresh, isolated
Postgres/Redis container matching CI's exact setup (not my local dev DB,
which had accumulated its own unrelated cross-session cruft this
session and was a red herring for earlier, separate investigations):

test_orders.py's `_order_tables` fixture was `scope="session",
autouse=True` -- the only session-scoped table fixture in the entire
backend test suite; every other API test file (test_portfolios_api.py,
test_portfolios_holdings.py, test_valuation_api.py, test_funding.py, ...)
uses a function-scoped autouse fixture that creates its tables before
each test and drops them after. Session scope meant test_orders.py's 16
tables -- critically `security`, which nearly every other file also
manages independently via its own `checkfirst=True` create -- stayed
alive for the rest of the pytest *process*, not just for test_orders.py's
own tests. Any other file that ran afterward in the same process
silently reused this still-alive `security` table instead of getting a
fresh one, and when that file's own per-test teardown tried its usual
non-CASCADE `DROP TABLE "security"`, it failed with
DependentObjectsStillExist because test_orders.py's `tax_lot` (also
still alive) still referenced it. That single failed DROP rolls back its
whole teardown transaction, so every table in that file's list survives
with its data into the next file too -- explaining both observed CI
symptoms: `uq_security_symbol` duplicate-key errors and a leftover
"Growth" model_portfolio row bleeding into an unrelated assertion.

This is why it only ever showed up in CI/a true full-suite run and never
in any of this session's many scoped-file or small-combo local test
runs -- confirmed by reproducing it with test_orders.py +
test_portfolios_api.py in one pytest invocation (fails) versus the exact
same two files run as separate invocations against the same DB state
(passes) -- the collision requires both files' fixtures to be alive in
the same pytest session, not just the same database.

Fixed by making `_order_tables` function-scoped like every other file,
folding in the now-redundant `_clean_order_tables` TRUNCATE-based
second fixture (full create/drop per test already resets everything a
TRUNCATE would have). Costs more CREATE/DROP churn across this file's
~50 tests, matching the cost every other file already pays for correct
isolation.

Verified: full `uv run pytest --cov=app --cov-report=term-missing`
(CI's exact command) against a fresh, isolated Postgres/Redis container
-- 932 passed, 0 failed, 0 errors, 84.52% coverage (floor 77%).
Copilot AI lite review requested due to automatic review settings September 7, 2026 07:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Badri-Rajendran
Badri-Rajendran merged commit bf3bfaf into main Sep 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants