Speed up the Python test suites: per-worker Postgres DBs, schema reuse, lazy route resolution

.github/workflows/postgres-tests.yml

@@ -1,9 +1,10 @@
 name: Postgres tests
 
 # The full Python suite on Postgres (`make test-py-pg`), including the
-# `@pytest.mark.postgres` tests that SQLite cannot run. Slow and serial (every
-# test resets one database), so it is not part of the PR aggregate: it runs
-# nightly, on demand, and on PRs that touch the database/tenancy layers (#508).
+# `@pytest.mark.postgres` tests that SQLite cannot run. Parallel under xdist,
+# one database per worker (every test resets its schema), but still slower than
+# SQLite, so it is not part of the PR aggregate: it runs nightly, on demand, and
+# on PRs that touch the database/tenancy layers (#508).
 
 on:
   schedule:

CHANGELOG.md

@@ -156,6 +156,26 @@ All notable changes to this project are documented in this file. The format is b
   it, which removes the `users` ↔ `audit_log` import cycle.
 
 ### Changed
+- **Behaviour change:** the `simple_module_test` plugin no longer refuses `-n`
+  on Postgres: each pytest-xdist worker creates and uses its own database
+  (`<db>_gw0`, `<db>_gw1`, ...) and points `SM_TEST_DATABASE_URL` at it, so the
+  role needs `CREATEDB`. `make test-py-pg` now runs with
+  `-n $(PYTEST_WORKERS) --dist loadgroup` like `make test-py`
+  (`PYTEST_WORKERS=0` keeps the old serial run on the named database).
+- `simple_module_test` on Postgres builds the schema once per process and, before
+  each later test, deletes every row and restarts the sequences instead of
+  dropping and re-creating `public` (~0.05 s instead of ~0.25 s per test). A
+  direct `reset_schema()` call or any change to the catalog (a test's own
+  tables, extensions, constraints) falls back to the full rebuild.
+- `create_app` no longer resolves every route's dependencies at build time
+  (~0.5 s for the full module set, ~70% of a test's setup): module routers now
+  get their prefix from `include_router(prefix=...)`, which skips FastAPI's
+  resolving empty-path check (effective paths and the OpenAPI document are
+  unchanged), and the route prefix guard and the SPA overlap check read paths
+  from FastAPI's include contexts without resolving routes. Outside
+  `environment="testing"` the lifespan still resolves all routes at startup,
+  so the first requests after boot stay warm. The SQLite and Postgres suites
+  run ~45% faster.
 - tenants and api_tokens pages call their APIs through the typed client and take
   typed component props (#504). **Behaviour change:** the api_tokens pages no longer
   receive the unused `api_url` prop.

CLAUDE.md

@@ -28,7 +28,7 @@ Everything goes through `make`. Full list: `docs/reference/make-commands.md`.
 | `make install` | `uv sync --all-packages` + `npm install` |
 | `make dev` | Shared Docker stack up + `gen-pages` + API (:8000) and Vite (:5050) |
 | `make test` / `make test-py` / `make test-js` | Pytest (SQLite, e2e/perf excluded; parallel via `pytest-xdist`, `-n auto --dist loadgroup`, `PYTEST_WORKERS=0` for serial) then Vitest |
-| `make test-py-pg` | Pytest on Postgres (`SM_TEST_PG_URL`, default db `sm_test`); serial, the plugin refuses `-n` there; not run in CI |
+| `make test-py-pg` | Pytest on Postgres (`SM_TEST_PG_URL`, default db `sm_test`); parallel like `test-py`, each xdist worker creates its own `sm_test_gw<N>` database (role needs `CREATEDB`); not in `pr-checks` |
 | `make test-e2e` | Playwright smoke tests against a running `make dev` |
 | `make lint` | Ruff format + Ruff + `ty` + Biome + per-workspace `tsc` + file-size cap + hardcoded-string check + untranslated-string check + package metadata/README checks |
 | `make doctor` | Module diagnostics (`SM0xx`); a dev boot runs the same plus `SM012`/`SM025` |

Makefile

@@ -72,11 +72,13 @@ test-py:
 # `@pytest.mark.anyio` would otherwise run on anyio's event loop while the
 # async fixtures ran on pytest-asyncio's, and an asyncpg connection cannot
 # cross loops (aiosqlite's thread hides this on SQLite). asyncio_mode=auto
-# still runs those tests. Serial on purpose: every test drops and recreates
-# `public` in the one database, so parallel workers would wipe each other's.
+# still runs those tests. Parallel like test-py: every test resets the schema
+# (rows emptied, or `public` rebuilt when it changed), so each xdist worker
+# creates and uses its own `<db>_gw<N>` database (the role needs CREATEDB);
+# PYTEST_WORKERS=0 uses the named database itself.
 SM_TEST_PG_URL ?= postgresql+asyncpg://postgres:postgres@localhost:5432/sm_test
 test-py-pg:
-	SM_TEST_DATABASE_URL=$(SM_TEST_PG_URL) uv run pytest -p no:anyio
+	SM_TEST_DATABASE_URL=$(SM_TEST_PG_URL) uv run pytest -p no:anyio -n $(PYTEST_WORKERS) --dist loadgroup
 
 test-js:
 	npm test

docs/database/extensions.md

@@ -67,5 +67,9 @@ pytestmark = pytest.mark.postgres
 
 They are skipped unless `SM_TEST_DATABASE_URL` is a Postgres URL (see
 `make test-py-pg`); the marker is registered by the `simple_module_test`
-plugin. The fixtures drop and re-create `public` per test, and then re-create
-every installed module's declared extensions.
+plugin. The fixtures build the schema once per process and empty its tables
+between tests; whenever they drop and re-create `public` (the first test, or a
+test that changed the schema), they re-create every installed module's declared
+extensions. Under pytest-xdist each worker
+runs on its own database (`<db>_gw0`, ...), which the plugin creates on first
+use, so the role needs `CREATEDB`.

framework/hosting/simple_module_hosting/_lifespan.py

@@ -187,7 +187,8 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
                 app.state.deferred_startup.append(mod)
         # Here rather than in create_app: the host includes its own routers
         # after create_app returns, and those should be guarded too.
-        guard_included_routers(app)
+        settings = getattr(app.state.sm, "settings", None)
+        guard_included_routers(app, warm=getattr(settings, "environment", None) != "testing")
         yield
         for mod in reversed(modules):
             await mod.on_shutdown(app)

framework/hosting/simple_module_hosting/_module_routes.py

@@ -25,9 +25,14 @@ def wire_module_routes(app: FastAPI, module) -> bool:
     The single canonical implementation so ``create_app`` and the test harness
     in ``simple_module_test`` stay in lockstep if ``ModuleBase`` ever gains
     a new router type.
+
+    Each prefix is given to ``include_router``, not to the ``APIRouter``: the
+    effective paths are the same, but FastAPI only checks an un-prefixed
+    include for empty paths, and that check resolves every nested route
+    (dependants, signatures, adapters; ~0.2 s per app build) that would
+    otherwise resolve lazily on first use.
     """
     api_router = APIRouter(
-        prefix=module.meta.route_prefix,
         tags=[module.meta.name],
         generate_unique_id_function=make_unique_id_function(),
     )
@@ -36,31 +41,29 @@ def wire_module_routes(app: FastAPI, module) -> bool:
     # not (``token_guard``).
     pages = [Depends(page_route)]
     view_router = APIRouter(
-        prefix=module.meta.view_prefix,
         tags=[f"{module.meta.name} Views"],
         include_in_schema=False,
         dependencies=pages,
     )
     module.register_routes(api_router, view_router)
     _clone_bare_prefix_route(view_router, module.meta.view_prefix)
     has_views = bool(view_router.routes)
-    app.include_router(api_router)
-    app.include_router(view_router, include_in_schema=False)
+    app.include_router(api_router, prefix=module.meta.route_prefix)
+    app.include_router(view_router, prefix=module.meta.view_prefix, include_in_schema=False)
 
     # Second view router for modules that serve both public and admin pages
     # and therefore cannot express both under a single view_prefix — see
     # ``ModuleMeta.admin_view_prefix``.
     admin_prefix = getattr(module.meta, "admin_view_prefix", "")
     if admin_prefix:
         admin_router = APIRouter(
-            prefix=admin_prefix,
             tags=[f"{module.meta.name} Admin Views"],
             include_in_schema=False,
             dependencies=pages,
         )
         module.register_admin_routes(admin_router)
         _clone_bare_prefix_route(admin_router, admin_prefix)
-        app.include_router(admin_router, include_in_schema=False)
+        app.include_router(admin_router, prefix=admin_prefix, include_in_schema=False)
     return has_views
 
 
@@ -80,9 +83,9 @@ def _clone_bare_prefix_route(router: APIRouter, prefix: str) -> None:
     """
     if not prefix:
         return
-    bare_target = f"{prefix}/"
+    # Paths are relative to the router: the prefix is added on include.
     for route in list(router.routes):
-        if isinstance(route, APIRoute) and route.path == bare_target:
+        if isinstance(route, APIRoute) and route.path == "/":
             router.add_api_route(
                 "",
                 route.endpoint,

framework/hosting/simple_module_hosting/_route_guard.py

@@ -36,6 +36,8 @@
 from starlette.routing import Match
 from starlette.types import Scope
 
+from simple_module_hosting._route_paths import included_route_paths
+
 try:  # private, and absent before FastAPI 0.140
     from fastapi.routing import _IncludedRouter
 except ImportError:  # pragma: no cover - older FastAPI flattens eagerly
@@ -44,14 +46,6 @@
 __all__ = ["guard_included_routers"]
 
 
-def _effective_paths(included: Any) -> list[str]:
-    paths: list[str] = []
-    for ctx in included.effective_route_contexts():
-        route = ctx.starlette_route
-        paths.append((getattr(route, "path", "") if route is not None else ctx.path) or "")
-    return paths
-
-
 def _common_prefix(paths: list[str]) -> str:
     """The literal leading part every path shares — never past a ``{param}``."""
     if not paths:
@@ -73,7 +67,7 @@ def __init__(self, included: Any) -> None:
     def _current_prefix(self) -> str:
         version = self._included.original_router._get_routes_version()
         if version != self._version:
-            self._prefix = _common_prefix(_effective_paths(self._included))
+            self._prefix = _common_prefix(included_route_paths(self._included))
             self._version = version
         return self._prefix
 
@@ -85,14 +79,16 @@ def __call__(self, scope: Scope) -> tuple[Match, Scope]:
         return self._matches(scope)
 
 
-def guard_included_routers(app: FastAPI) -> None:
+def guard_included_routers(app: FastAPI, *, warm: bool = True) -> None:
     """Install a prefix guard on each router included directly into ``app``.
 
-    Computing a guard's prefix resolves its router's effective routes, which is
-    also what FastAPI does lazily on the first request to reach them — building
-    every route's dependant, signature and pydantic adapters. Doing it here
-    moves that one-time cost (hundreds of ms for a full module set) from the
-    first request after boot into startup.
+    A guard's prefix comes from its routes' paths alone, so installing it
+    resolves nothing. ``warm`` then resolves every router's effective routes, as
+    FastAPI would lazily on the first request to reach each one (building every
+    route's dependant, signature and pydantic adapters), moving that one-time
+    cost (hundreds of ms for a full module set) from the first requests after
+    boot into startup. Test apps skip it: each is built for a handful of
+    requests, so resolving all ~400 routes was most of their setup time.
     """
     if _IncludedRouter is None:
         return
@@ -101,3 +97,6 @@ def guard_included_routers(app: FastAPI) -> None:
             guard = _PrefixGuard(route)
             guard._current_prefix()
             route.matches = guard  # ty: ignore[invalid-assignment]
+            if warm:
+                for _ in route.effective_route_contexts():
+                    pass

framework/hosting/simple_module_hosting/_route_paths.py

@@ -0,0 +1,72 @@
+"""Effective route paths of an app, without resolving its routes.
+
+FastAPI >= 0.140 includes routers lazily: an ``_IncludedRouter`` builds each
+route's dependant, signature and pydantic adapters the first time a request
+reaches it, which is ~0.5 s for the full module set. Asking it for its
+``effective_route_contexts()`` just to read their paths forces all of that at
+once, on every app build, for every router.
+
+The path itself is cheap: it is the include context's prefix plus the route's
+own path (``include_context.path_for``), with nested includes folded in by
+``include_context.combine``, the same two steps FastAPI's own resolution uses.
+A route type with any other shape falls back to FastAPI building that one
+route's context. ``tests/test_route_paths.py`` checks the result matches the
+resolved paths for every router of a real app, so a FastAPI upgrade that
+changes how paths compose fails there rather than misrouting.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import FastAPI
+from fastapi.routing import APIRoute, APIWebSocketRoute
+from starlette import routing
+
+# Private; see tests/test_route_guard_fastapi_contract.py before widening the
+# fastapi dependency bound.
+try:
+    from fastapi.routing import _IncludedRouter
+except ImportError:  # pragma: no cover - older FastAPI flattens eagerly
+    _IncludedRouter = None
+
+__all__ = ["app_route_paths", "included_route_paths"]
+
+# Types whose effective path is exactly ``include_context.path_for(route)``.
+_PREFIXED = (APIRoute, routing.Route, APIWebSocketRoute)
+
+
+def included_route_paths(included: Any) -> list[str]:
+    """Every effective path under one ``_IncludedRouter``, nested includes too."""
+    context = included.include_context
+    paths: list[str] = []
+    for route in included.original_router.routes:
+        if _IncludedRouter is not None and isinstance(route, _IncludedRouter):
+            child = _IncludedRouter(
+                original_router=route.original_router,
+                include_context=context.combine(route.include_context),
+            )
+            paths.extend(included_route_paths(child))
+        elif isinstance(route, _PREFIXED):
+            paths.append(context.path_for(route))
+        else:
+            resolved = included._build_effective_context(route)
+            if resolved is not None:
+                paths.append(_resolved_path(resolved))
+    return paths
+
+
+def app_route_paths(app: FastAPI) -> list[str]:
+    """Every path registered on *app*, flattening its included routers."""
+    paths: list[str] = []
+    for route in app.routes:
+        if _IncludedRouter is not None and isinstance(route, _IncludedRouter):
+            paths.extend(included_route_paths(route))
+        else:
+            paths.append(getattr(route, "path", "") or "")
+    return paths
+
+
+def _resolved_path(context: Any) -> str:
+    route = context.starlette_route
+    return (getattr(route, "path", "") if route is not None else context.path) or ""

framework/hosting/simple_module_hosting/_spa_mount.py

@@ -25,6 +25,7 @@
 from starlette.responses import Response
 from starlette.types import Scope
 
+from simple_module_hosting._route_paths import app_route_paths
 from simple_module_hosting.static_files import IMMUTABLE_CACHE_CONTROL
 
 logger = logging.getLogger(__name__)
@@ -100,24 +101,9 @@ def _is_page_request(path: str, scope: Scope) -> bool:
     return "text/html" in Headers(scope=scope).get("accept", "").lower()
 
 
-def _registered_route_paths(app: FastAPI) -> list[str]:
-    """Every path registered on *app*, flattening FastAPI's lazy included routers."""
-    paths: list[str] = []
-    for route in app.routes:
-        contexts = getattr(route, "effective_route_contexts", None)
-        if contexts is None:
-            paths.append(getattr(route, "path", "") or "")
-            continue
-        for ctx in contexts():
-            starlette_route = ctx.starlette_route
-            path = getattr(starlette_route, "path", "") if starlette_route is not None else ctx.path
-            paths.append(path or "")
-    return paths
-
-
 def mount_spa_mounts(app: FastAPI, modules: list) -> None:
     """Mount every module's SPAs, exempt the public ones from auth, publish prefixes."""
-    problems = check_spa_mount_overlaps(modules, _registered_route_paths(app))
+    problems = check_spa_mount_overlaps(modules, app_route_paths(app))
     if problems:
         raise SpaMountError("\n".join(str(p) for p in problems))
 

framework/hosting/tests/test_route_guard_fastapi_contract.py

@@ -92,3 +92,37 @@ def replacement(scope):
     assert included.matches(scope) == (Match.NONE, {}), (
         f"assigned matches(scope) must be callable; update {GUARD}"
     )
+
+
+PATHS = "framework/hosting/simple_module_hosting/_route_paths.py"
+
+
+def test_private_include_context_contract():
+    included_type = routing._IncludedRouter
+    leaf = APIRouter()
+
+    @leaf.get("/item")
+    async def item():
+        return {}
+
+    parent = APIRouter()
+    parent.include_router(leaf, prefix="/leaf")
+    app = FastAPI()
+    app.include_router(parent, prefix="/api")
+    included = next(r for r in app.router.routes if isinstance(r, included_type))
+
+    context = getattr(included, "include_context", None)
+    assert context is not None, f"_IncludedRouter.include_context must exist; update {PATHS}"
+    nested = next(r for r in parent.routes if isinstance(r, included_type))
+    combine = _accepts(
+        getattr(context, "combine", None), "include_context.combine(child)", nested.include_context
+    )
+    child = combine(nested.include_context)
+    route = next(r for r in leaf.routes if getattr(r, "name", "") == "item")
+    path_for = _accepts(getattr(child, "path_for", None), "include_context.path_for(route)", route)
+    assert path_for(route) == "/api/leaf/item", (
+        f"combine/path_for must compose include prefixes; update {PATHS}"
+    )
+    included_type(original_router=leaf, include_context=child)  # constructible by keyword
+    build = getattr(included, "_build_effective_context", None)
+    _accepts(build, "_IncludedRouter._build_effective_context(route)", route)