feat: module data-compatibility checks and smpy migrations prepare (#711)

CHANGELOG.md

@@ -26,6 +26,12 @@ All notable changes to this project are documented in this file. The format is b
   ones) above `direct_upload_threshold_bytes` now use bounded multipart, including
   unknown-length streams, with size/part bounds and abort on failure; the S3
   credentials need `s3:AbortMultipartUpload` (#708).
+- **Behaviour change:** module data compatibility checks run before startup and in
+  Doctor (`SM033`); legacy bearer-token storage refuses boot outside development,
+  while obsolete mail settings warn (#711). `smpy migrations prepare` writes
+  frozen host-owned recipes for mail namespace changes (`d7b3a9e5c1f2`) and token
+  invalidation/column changes (`a626b1c8d023`), without applying migrations. `smpy doctor` is a new alias for `make doctor`
+  on scaffolded hosts.
 - Users auth emails support ordered host template directories through the
   restart-required DB setting `mail_template_dirs`, with per-file bundled
   fallback and optional autoescaped HTML alternatives (#712).

CLAUDE.md

@@ -173,8 +173,10 @@ every Tailwind utility, so keep tokens in `theme.css` and rules in `styles.css`
 `register_design_packs`,
 `register_audit_links`, `register_resource_types`, `register_exception_handlers`, `register_middleware`,
 `register_routes(api_router, view_router)`, `register_admin_routes(admin_router)`
-(only with `ModuleMeta.admin_view_prefix`), then async `on_startup` /
-`on_shutdown` (reverse order). Non-hook overrides: `locale_dirs()`,
+(only with `ModuleMeta.admin_view_prefix`), then async `check_data_compatibility(db)`
+(read-only upgrade checks before settings hydration; an `error` refuses boot
+outside development, `smpy migrations prepare <upgrade-id>` writes the fix into
+the host), then async `on_startup` / `on_shutdown` (reverse order). Non-hook overrides: `locale_dirs()`,
 `db_extensions()`, `template_dirs()`, `static_mounts()`, `spa_mounts()`.
 
 - A module gets one view router per prefix. All-admin modules point
@@ -475,8 +477,9 @@ not run the suite** (only `SM025` is reported there). What fails a production
 boot is strict discovery (a bad entry point, missing `meta`, or a missing
 `depends_on` target), the
 auth-provider check, placeholder secrets (`SM_SECRET_KEY`, users token
-secrets), a localhost Celery broker, and the `SM010` migration check (except
-on a never-migrated first run, which goes to `/setup`). Catch everything else
+secrets), a localhost Celery broker, the `SM010` migration check (except
+on a never-migrated first run, which goes to `/setup`) and an `error` from a
+module's `check_data_compatibility` (`SM033`). Catch everything else
 with `make doctor` before shipping. Codes that
 change how you write code: `SM003`/`SM004` render-key mismatch, `SM009`
 framework→plugin import, `SM011` table missing from migrations, `SM012`

docs/.vitepress/config.ts

@@ -92,6 +92,7 @@ export default defineConfig({
             { text: "Quickstart", link: "/guide/quickstart" },
             { text: "Project structure", link: "/guide/project-structure" },
             { text: "Configuration", link: "/guide/configuration" },
+            { text: "Upgrading a host", link: "/guide/upgrading" },
             { text: "Your first module", link: "/guide/first-module" },
             { text: "OpenAPI & clients", link: "/guide/openapi-clients" },
           ],

docs/database/migrations.md

@@ -8,6 +8,29 @@ All migrations live in `host/migrations/versions/` — **not** in module package
 - **Autogenerate sees everything.** `host/migrations/env.py` calls `build_module_metadata()` to union every installed module's `MetaData`. Autogenerate diffs the DB against that union and writes one migration covering all changes.
 - **Operators run one command.** `make migrate` is the only target. No "did you also run `orders/migrate`?" footgun.
 
+## Module data upgrades in consuming hosts (#711)
+
+Scaffolded hosts own their migration graphs. A schema-only baseline generated
+from current models does not include this repository's historical data changes.
+Before deploying upgraded packages, run `make doctor` or `uv run smpy doctor`.
+Semantic module checks (`SM033`) recognize equivalent/manual ports independently
+of upstream revision ids.
+
+`uv run smpy migrations prepare <upgrade-id> --head <host-head-or-label@head>`
+copies a frozen, self-contained recipe into the host's own `migrations/versions`;
+it never applies it. An explicit current head is required. The bundled ids are
+`mail.settings_namespace_v1` (system settings move, target wins, matching
+`d7b3a9e5c1f2`) and `users.hashed_bearer_tokens_v1` (invalidate credentials before
+altering both token columns, matching `a626b1c8d023`). The first recipe for an
+unlabelled module declares `mail`/`users`; existing labels are not redeclared.
+Review the generated revision before applying it with `upgrade heads`.
+
+Boot checks run before module settings hydration/startup; token incompatibility
+refuses boot outside development, while obsolete mail keys warn. Never-migrated
+first runs defer checks until the wizard applies migrations. No automatic
+credential deletion happens at startup. See the [scaffolded host upgrade
+guide](../guide/upgrading.md) for the operator workflow and token sign-out risk.
+
 ## The real graph
 
 The history is **not** a single linear chain. It has several roots and several heads, and `make migrate` runs `upgrade heads` for that reason. On `main` today:

docs/framework/lifecycle.md

@@ -27,6 +27,7 @@ register_exception_handlers
 register_middleware
 register_routes
 register_admin_routes       (only when meta.admin_view_prefix is set)
+-- check_data_compatibility (async, before module settings hydration)
 -- on_startup (async, after middleware is installed)
 ```
 
@@ -350,6 +351,17 @@ def register_resource_types(self, registry: ResourceTypeRegistry) -> None:
     )
 ```
 
+## `check_data_compatibility(db)`
+
+This async read-only hook returns a sequence of `DataCompatibilityIssue` frozen
+dataclasses (`upgrade_id`, `message`, `severity="warning"|"error"`), empty by default.
+Use stable semantic ids and inspect actual schema/data state so hosts with
+independently ported revisions pass. Checks run before module DB-settings
+hydration and `on_startup`; errors refuse boot outside development. A never-
+migrated first run defers checks until migrations have been applied. Doctor
+reports the same issues as `SM033`. Checks must never clear credentials or apply
+migrations; see [Migrations](../database/migrations.md).
+
 ## `on_startup()` / `on_shutdown()`
 
 Async lifespan hooks that run after all modules are registered.

docs/guide/upgrading.md

@@ -0,0 +1,75 @@
+# Upgrading a scaffolded host
+
+A host generated by `smpy new` or `smpy create-host` owns its Alembic history.
+Its initial revision reflects the models installed at generation time; it does
+not import the framework repository's historical revisions. Updating packages
+and running schema autogenerate cannot reproduce historical data transformations.
+
+## Check and prepare
+
+Back up the database and review the release notes before upgrading packages.
+From the project root, with the host's Python environment active:
+
+```bash
+uv run smpy doctor
+uv run alembic -c host/alembic.ini heads
+uv run smpy migrations prepare mail.settings_namespace_v1 --head <your-host-head>
+uv run smpy migrations prepare users.hashed_bearer_tokens_v1 --head users@head
+```
+
+Replace `<your-host-head>` with an actual head in **your** graph. Use `users@head`
+only if your graph already declares that label; otherwise pass the relevant host
+head id. Preparing the mail revision may advance the same branch, so inspect
+`heads` again before preparing the next revision. For flat hosts use
+`alembic.ini` instead of `host/alembic.ini`. The preparation command detects both
+layouts, or accepts `--config path/to/alembic.ini`.
+
+Preparation writes one self-contained revision into that host's
+`migrations/versions`. It never connects to the database or applies the revision.
+The frozen recipes ship in the CLI package, and generated revisions import only
+Alembic and SQLAlchemy: future module/package upgrades cannot change their code.
+The first prepared recipe for an unlabelled module declares `mail` or `users`;
+existing labels are reused without redeclaration. `--head` is required and must
+resolve to one current host head. No upstream roots are grafted onto your graph.
+
+## Review and apply
+
+Review and keep the generated revision in your application's version control.
+Do not generate a token invalidation revision for a baseline that is already
+compatible: applying it deliberately signs clients out.
+
+| Upgrade id | Effect |
+|---|---|
+| `mail.settings_namespace_v1` | Move system-scope `users.mailer` to `mail.backend` and the six `users.smtp_*` transport keys to `mail.*`. Existing target values win; stale source rows are removed. Tenant/user settings are untouched. Downgrade reverses the mapping with the same target-wins rule. |
+| `users.hashed_bearer_tokens_v1` | Delete access and refresh credentials **before** changing both token columns to `String(64)`. Downgrade deletes credentials again and restores the legacy column types; it cannot recover secrets. |
+
+Stop every old application worker, including background workers, before applying
+the token upgrade. Mixed old/new workers must not mint or use plaintext tokens
+during this upgrade. Clients using bearer tokens or `sm_auth` must sign in again.
+Other session cookies are unaffected. After reviewing and scheduling the upgrade:
+
+```bash
+uv run alembic -c host/alembic.ini upgrade heads
+uv run smpy doctor
+```
+
+Restart the application with the upgraded packages. Role provisioning remains an
+idempotent startup step; it needs no prepared data revision.
+
+## Compatibility checks
+
+Modules expose read-only `check_data_compatibility(db)` hooks with semantic upgrade
+ids. Boot runs them before module settings hydration and module startup. Doctor
+(`make doctor` or `smpy doctor`) reports them as `SM033` and exits nonzero on errors.
+
+Unmoved system mail keys warn. Incompatible token columns or legacy token
+representations refuse boot outside development; development logs warnings.
+Checks inspect state, so equivalent manual ports are accepted without requiring
+upstream revision ids. A current schema-only baseline with empty token tables
+passes and does not trigger credential deletion.
+
+A never-migrated database skips the initial check so the first-run wizard stays
+reachable. Checks replay after wizard/out-of-band migrations before deferred
+settings hydration/startup; an incompatibility keeps the setup gate closed.
+This is detection and explicit preparation for the two bundled upgrades, not an
+automatic migration system for arbitrary third-party module data.

docs/reference/diagnostic-codes.md

@@ -41,6 +41,7 @@ The framework runs a set of static checks over installed modules at app boot. Th
 | `SM030` | WARNING at a dev boot; ERROR in `make doctor` | A module's `depends_on` names a module that is not loaded (not installed, left out of `SM_MODULES_ENABLED`, or dropped because another auth provider was selected — `keycloak` drops `Users`). Outside development the boot raises `MissingDependencyError` listing every pair. | Install and enable the dependency, or leave the dependent module out. A relationship that is only about order belongs in `optional_depends_on`. See [Discovery](/framework/discovery#modulemeta). |
 | `SM031` | WARNING | An `/api` route has no permission, `TokenAllowed` or `Authenticated` gate anywhere in its dependency tree (include-level dependencies count) and no public-route match, so the schema cannot say who may call it. Needs the built app: reported at a **dev boot** (and in the Doctor screen), not by `make doctor`. | Add `Depends(RequiresPermission(...))`, or `Depends(Authenticated())` when any signed-in user may call it, or register it in `register_public_routes`. |
 | `SM032` | ERROR (fails boot in every environment) | A `ModuleBase.spa_mounts()` prefix overlaps a module's `route_prefix`/`view_prefix`/`admin_view_prefix`, a `static_mounts()` prefix, a framework path (`/api`, `/admin`, `/setup`, `/i18n`, `/health`, `/static`, `/modules`) or another SPA mount. The mount would shadow, or be shadowed by, that router. | Pick a prefix nothing else uses. See [Legacy SPA mounts](/framework/spa-mounts). |
+| `SM033` | ERROR or WARNING | A module data compatibility check found a missing semantic upgrade (a WARNING when the database could not be checked; a SQLite file that does not exist yet is skipped). Token schema/legacy credentials are errors; unmoved mail system keys warn. Errors refuse boot outside development. Never-migrated first runs defer checks. | Run the reported `smpy migrations prepare <upgrade-id> --head <host-head-or-label@head>`, review the host revision and apply `upgrade heads`; see [Upgrading a host](/guide/upgrading). |
 
 `SM022`/`SM023` are the two halves of the same invariant: a module's optional [`theme.css` is imported unlayered and `styles.css` into `layer(components)`](/module-authoring#styling), and CSS put in the wrong one silently does nothing (or silently wins everything).
 

framework/cli/simple_module_cli/cli.py

@@ -23,6 +23,7 @@
 from simple_module_cli.add_cmd import add_module, run_add
 from simple_module_cli.case import to_kebab_case
 from simple_module_cli.docker_cmd import docker_app
+from simple_module_cli.migrations_cmd import migrations_app
 from simple_module_cli.module_cmd import module_app
 from simple_module_cli.new import new_project
 from simple_module_cli.openapi_cmd import openapi_app
@@ -49,6 +50,22 @@
 app.add_typer(skills_app, name="skills")
 app.add_typer(module_app, name="module")
 app.add_typer(openapi_app, name="openapi")
+app.add_typer(migrations_app, name="migrations")
+
+
+@app.command("doctor")
+def doctor() -> None:
+    """Check installed modules and their database compatibility."""
+    try:
+        from simple_module_core.__main__ import main as run_doctor
+    except ImportError:
+        # The CLI installs on its own (to scaffold); doctor needs a project's framework.
+        typer.echo(
+            "ERROR: smpy doctor needs simple_module_core; run it inside a project.", err=True
+        )
+        raise typer.Exit(1) from None
+
+    raise typer.Exit(run_doctor())
 
 
 @app.command("create-host")

framework/cli/simple_module_cli/migration_recipes/mail_settings.py

@@ -0,0 +1,48 @@
+"""Frozen system mail-settings move from d7b3a9e5c1f2 (#711).
+
+An existing target value wins; the obsolete source row is removed.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+_TABLE = "settings_setting"
+_SCOPE_SYSTEM = "system"
+_SYSTEM_SCOPE_ID = ""
+
+_KEY_PAIRS = (
+    ("users.mailer", "mail.backend"),
+    *((f"users.smtp_{f}", f"mail.smtp_{f}") for f in ("host", "port", "username", "password")),
+    ("users.smtp_from", "mail.smtp_from"),
+    ("users.smtp_tls", "mail.smtp_tls"),
+)
+
+
+def _rename(old: str, new: str) -> None:
+    scope = {"scope": _SCOPE_SYSTEM, "scope_id": _SYSTEM_SCOPE_ID, "old": old, "new": new}
+    bind = op.get_bind()
+    bind.execute(
+        sa.text(
+            f"DELETE FROM {_TABLE} WHERE scope = :scope AND scope_id = :scope_id AND key = :old"
+            f" AND EXISTS (SELECT 1 FROM {_TABLE} AS t WHERE t.scope = :scope"
+            " AND t.scope_id = :scope_id AND t.key = :new)"
+        ),
+        scope,
+    )
+    bind.execute(
+        sa.text(
+            f"UPDATE {_TABLE} SET key = :new"
+            " WHERE scope = :scope AND scope_id = :scope_id AND key = :old"
+        ),
+        scope,
+    )
+
+
+def upgrade() -> None:
+    for old, new in _KEY_PAIRS:
+        _rename(old, new)
+
+
+def downgrade() -> None:
+    for old, new in _KEY_PAIRS:
+        _rename(new, old)

framework/cli/simple_module_cli/migration_recipes/users_tokens.py

@@ -0,0 +1,34 @@
+"""Frozen token-storage upgrade from a626b1c8d023 (#711).
+
+Upgrade and downgrade invalidate all access/refresh credentials. Stop all old
+application workers before upgrading; clients must sign in again.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+
+def _clear_tokens() -> None:
+    for table in ("users_refresh_token", "users_access_token"):
+        op.execute(sa.table(table).delete())
+
+
+def upgrade() -> None:
+    _clear_tokens()
+    with op.batch_alter_table("users_refresh_token") as batch:
+        batch.alter_column(
+            "token", existing_type=sa.Uuid(), type_=sa.String(64), postgresql_using="token::text"
+        )
+    with op.batch_alter_table("users_access_token") as batch:
+        batch.alter_column("token", existing_type=sa.String(43), type_=sa.String(64))
+
+
+def downgrade() -> None:
+    # Digests cannot be restored to UUIDs or bearer secrets.
+    _clear_tokens()
+    with op.batch_alter_table("users_refresh_token") as batch:
+        batch.alter_column(
+            "token", existing_type=sa.String(64), type_=sa.Uuid(), postgresql_using="token::uuid"
+        )
+    with op.batch_alter_table("users_access_token") as batch:
+        batch.alter_column("token", existing_type=sa.String(64), type_=sa.String(43))

framework/cli/simple_module_cli/migrations_cmd.py

@@ -0,0 +1,66 @@
+"""Prepare frozen framework upgrades in a consuming host's own Alembic graph."""
+
+from __future__ import annotations
+
+import os
+from importlib.resources import files
+from pathlib import Path
+from typing import Annotated
+from uuid import uuid4
+
+import typer
+
+migrations_app = typer.Typer(help="Prepare host-owned framework upgrade revisions.")
+_RECIPES = {
+    "mail.settings_namespace_v1": ("mail", "mail_settings.py"),
+    "users.hashed_bearer_tokens_v1": ("users", "users_tokens.py"),
+}
+
+
+@migrations_app.command("prepare")
+def prepare(
+    upgrade_id: Annotated[str, typer.Argument(help="Stable framework upgrade id.")],
+    head: Annotated[str, typer.Option("--head", help="Explicit host head or label@head.")],
+    config: Annotated[Path | None, typer.Option("--config", help="Host alembic.ini.")] = None,
+) -> None:
+    """Write a self-contained revision; never connect to or migrate the database."""
+    if upgrade_id not in _RECIPES:
+        raise typer.BadParameter(f"Unknown upgrade {upgrade_id!r}; choose {', '.join(_RECIPES)}")
+    try:
+        from alembic.config import Config
+        from alembic.script import ScriptDirectory
+        from alembic.util.exc import CommandError
+    except ImportError as exc:
+        raise typer.BadParameter(
+            "Run this command in the host environment with Alembic installed"
+        ) from exc
+
+    root = Path(os.environ.get("SM_PROJECT_ROOT", Path.cwd()))
+    if config is None:
+        config = root / "alembic.ini"
+        if not config.is_file():
+            config = root / "host/alembic.ini"
+    if not config.is_file():
+        raise typer.BadParameter(f"Host Alembic config not found: {config}")
+    try:
+        script = ScriptDirectory.from_config(Config(str(config)))
+        parent = script.get_revision(head)
+        if parent is None or not parent.is_head:
+            raise typer.BadParameter("--head must resolve to one existing host head")
+        label, recipe = _RECIPES[upgrade_id]
+        existing = {name for rev in script.walk_revisions() for name in rev.branch_labels}
+    except CommandError as exc:
+        raise typer.BadParameter(str(exc)) from exc
+
+    # Frozen source is copied, so future package changes cannot change this revision.
+    source = files("simple_module_cli").joinpath("migration_recipes", recipe).read_text()
+    revision = uuid4().hex[:12]
+    branch_labels = None if label in existing else (label,)
+    identifiers = (
+        f"\nrevision = {revision!r}\ndown_revision = {parent.revision!r}\n"
+        f"branch_labels = {branch_labels!r}\ndepends_on = None\n"
+    )
+    path = Path(script.versions) / f"{revision}_{upgrade_id.replace('.', '_')}.py"
+    with path.open("x") as output:
+        output.write(source + identifiers)
+    typer.echo(f"Prepared {upgrade_id}: {path}. Review it, then run alembic upgrade heads.")