Repository navigation
Speed up the Python test suites: per-worker Postgres DBs, schema reuse, lazy route resolution - #577
Merged
Merged
Conversation
…worker make test-py-pg ran serially (~29 min in CI) because every test drops and recreates public in the one database. Each xdist worker now creates and uses <db>_gw<N> and points SM_TEST_DATABASE_URL at it; the switch happens in pytest_load_initial_conftests so helpers that read the URL at import time (framework/db/tests/_models.py) see the worker's database. Claude-Session: https://claude.ai/code/session_01MeLZyxkDD4sq75m1Siw7om
Deploying simple-module-python with
|
| Latest commit: |
fd09449
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9159060c.simple-module-python.pages.dev |
| Branch Preview URL: | https://claude-keen-pasteur-op0eox.simple-module-python.pages.dev |
Rebuilding public per test cost ~0.25 s; now each process builds it once and later tests delete every row and restart the sequences (~0.05 s). The snapshot is keyed on the built tables and a catalog fingerprint (tables, columns, constraints, types, extensions), so a direct reset_schema() call or a test's own DDL falls back to the full rebuild. alembic_version is created inside the build so the fingerprint covers it. Claude-Session: https://claude.ai/code/session_01MeLZyxkDD4sq75m1Siw7om
…p0eox # Conflicts: # CHANGELOG.md
antosubash
marked this pull request as ready for review
October 22, 2026 12:42
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Each create_app resolved all ~400 routes (dependants, signatures, pydantic adapters; ~0.5 s, ~70% of a test's setup) through three path walks: - FastAPI's empty-path check on include_router without prefix=: module routers now get their prefix on include, so effective paths are the same (route table and OpenAPI diffed identical) and the check is skipped; - the route prefix guard and the SPA overlap check read effective paths via effective_route_contexts(): both now use _route_paths, which composes paths with FastAPI's include_context.combine/path_for and resolves nothing (tests compare it with the resolved paths for every router of a real app). The lifespan still warms all routes at startup outside environment=testing. Locally: SQLite suite 7:29 -> 4:15, Postgres suite 8:15 -> 4:33. Claude-Session: https://claude.ai/code/session_01MeLZyxkDD4sq75m1Siw7om
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
Postgres testsworkflow spent ~29 min inmake test-py-pg(5418 tests, 1745 s). Each cause below was measured before it was changed.publicin one database, so the plugin refused-n.create_appresolved all ~400 routes, about 0.5 s and ~70% of a test's setup, on SQLite and Postgres alike. Once 1 and 2 were fixed, Postgres was at parity with the SQLite job (both ~15 min on CI). This was the remaining cost, and it is the critical path ofpr-checkstoo.Durability is not a bottleneck (~0.5 s of
syncper checkpoint in the CI log;fsync=offchanged nothing locally), so no server config is added.What
1. One database per xdist worker (
simple_module_test)isolate_worker_database()creates<db>_gw<N>and rewritesSM_TEST_DATABASE_URL, so direct env readers and subprocesses follow. It runs inpytest_load_initial_conftests, before conftests capture the URL at import time.make test-py-pgruns-n $(PYTEST_WORKERS) --dist loadgroup. Behaviour change: the role needsCREATEDB.2. Schema reuse (
prepare_schema_once)reset_schema()call or a test's own DDL triggers a full rebuild.alembic_versionis created inside the build so the fingerprint covers it.CREATE DATABASE … TEMPLATE) measured 0.05–0.15 s per test with a 3.3 s spike, so row deletion was chosen.3. Lazy route resolution (
simple_module_hosting)FastAPI ≥0.140 resolves an included router's routes (dependants, signatures, pydantic adapters) on first use, but three path walks forced all of them on every app build:
include_routergets noprefix=.wire_module_routesnow passes the prefix on include instead of on theAPIRouter. The effective route table (192 routes: path, methods, name, schema flag, dependency count) and the full OpenAPI document diff byte-identical before and after._clone_bare_prefix_routenow compares router-relative paths.effective_route_contexts(). Both now use the new_route_paths, which composes paths with FastAPI's owninclude_context.combine/path_forand falls back to FastAPI's per-route build for any other route type.environment="testing"the lifespan still warms every route at startup (guard_included_routers(warm=True)), so production's first requests are unchanged._route_pathsagainst the resolved paths for every router of the real app, and check that it resolves nothing. The private FastAPI members it uses are added totest_route_guard_fastapi_contract.py.Verification (local, 4 cores)
-n 4)make test-py)modules/background_tasksunder-n 4test_tenant_settings.py)On GitHub's runner, steps 1 and 2 alone took the Postgres step from 30.5 min to 13.6–17 min. This push adds step 3, which also speeds up the main
Python testsjob.Also clean:
ci-python-lint,ci-python-typecheck,ci-check-file-size,ci-api-types-fresh,ci-doctor.Known: one full SQLite run had
test_bg_inprocess_sweep.py::test_eager_sweep_recovers_without_schedule[False]time out (a 10 swait_for). It passed 5/5 alone and 3/3 under-n 4on this branch. The committed code also had one unexplained failure in seven loaded runs of the same module, which I could not reproduce. The test exercises the outbox sweep, not routing. Not fixed here.https://claude.ai/code/session_01MeLZyxkDD4sq75m1Siw7om