refactor(settings): move the settings mechanism into the framework; SM009 catches dynamic imports (#500)

CHANGELOG.md

@@ -25,6 +25,33 @@ All notable changes to this project are documented in this file. The format is b
 - `captured_mail` pytest fixture in `simple_module_test` records sent mail.
 
 ### Changed
+- The module-settings mechanism moved from the `settings` plugin into the
+  framework (#500): `register_module_settings` / `ModuleSettingsRegistry` to
+  `simple_module_core.module_settings`, hydration (`hydrate_settings`,
+  `build_settings`, `OverrideSource`, field helpers) to
+  `simple_module_core.settings_hydrate`, secret-field masking to
+  `simple_module_core.secret_fields`, and the `Setting` model, its constants,
+  the read-only override queries and `hydrate_settings_sync` to
+  `simple_module_db.stored_settings`. The table (`settings_setting`) is
+  unchanged; no migration. The hosting layer imports these statically instead
+  of reaching the plugin through `importlib` strings, and builds its pre-app
+  and secret-key SQL from the shared column constants.
+- **Behaviour change (#500):** registration and boot-time hydration no longer
+  need the `settings` plugin. The registry is created on `app.state` on first
+  use, `host` settings are always registered, and the lifespan hydrates from the
+  table directly, so module and host settings still come from the DB with the
+  plugin disabled (it keeps the admin UI, hot reload and the key/value API).
+  The pre-app host read and the generated `SM_SECRET_KEY` are likewise stored
+  and read without it; outside development an unreachable database now refuses
+  the boot even when the plugin is not installed.
+- **Out-of-tree hosts without the `settings` plugin (#500):** the framework now
+  always contributes `settings_setting` to the migration metadata, so the next
+  `alembic revision --autogenerate` will emit a `create_table` for it. Keep that
+  operation (the framework reads the table at boot).
+- `make new-module` / `smpy create-module` scaffolds subclass `DbBackedSettings`
+  and register through `register_module_settings` instead of assigning
+  `app.state.<pkg>` directly (#500).
+- SM009 now detects dynamic framework imports of plugin packages, including importlib aliases and fixed f-string package prefixes (#500).
 - CI and local lint now share one definition (#508). `make ci-js-typecheck`
   depends on `gen-pages`, so `make lint` works on a fresh clone. New targets
   `ci-check-metadata` (now also runs `check_readmes.py` in CI), `ci-i18n-fresh`
@@ -153,6 +180,14 @@ All notable changes to this project are documented in this file. The format is b
   `openapi_public` (`SM_OPENAPI_PUBLIC`) is on. The bare `/openapi.json` was
   previously served, and anonymous, in production.
 
+### Deprecated
+- `settings.registration`, `settings.module_registry` and `settings.hydrate`
+  re-export the framework objects above and emit a `DeprecationWarning`;
+  `settings.models.Setting`, `settings._secrets` and the moved names in
+  `settings.store` / `settings.constants` re-export them silently (the same
+  objects, never a second `Setting` mapping). Removal no earlier than 0.1.0
+  (#500).
+
 ### Added
 - `TenantDeleted` event and `TenantService.delete_tenant` (#491). No endpoint
   yet: what deleting a tenant means for other modules' rows is still open.

CLAUDE.md

@@ -341,9 +341,13 @@ development and testing (`session_cookie_secure`).
   is marked restart-required (`test_build_time_settings_marked.py`).
   Module settings are a `DbBackedSettings` subclass (constructor + DB, no env
   source) registered in `register_settings` through
-  `settings.registration.register_module_settings(app, package, SettingsCls,
-  services_factory)`, which mounts the services object on `app.state.<package>`
-  (`SM012`). The exceptions that read env are secrets and process wiring
+  `simple_module_core.module_settings.register_module_settings(app, package,
+  SettingsCls, services_factory)`, which mounts the services object on
+  `app.state.<package>` (`SM012`). Registration, hydration and the
+  `settings_setting` table (`simple_module_db.stored_settings`) belong to the
+  framework and work with the `settings` plugin disabled; the plugin's
+  `settings.registration`/`module_registry`/`hydrate` are deprecated
+  re-exports. The exceptions that read env are secrets and process wiring
   (`SM_BG_TASKS_*`, `SM_KEYCLOAK_*`, `SM_USERS_*_TOKEN_SECRET`).
   `docs/framework/settings.md`, `docs/reference/env-vars.md`.
 - **Framework must not import plugins** (`SM009`): nothing under `framework/`
@@ -397,9 +401,8 @@ development and testing (`session_cookie_secure`).
 `make new-module name=<name>` scaffolds the package, wires it into
 `host/pyproject.toml` and the root `testpaths`/`extra-paths`, and runs
 `uv sync`. Expect `make lint` to fail right after: the scaffold still emits a
-literal `inertia.render("<Class>/Create")`, permission and menu strings
-without constants or `label_key`, and sets `app.state.<pkg>` directly instead
-of calling `register_module_settings`. Fix those first. It also does **not**:
+literal `inertia.render("<Class>/Create")` and permission and menu strings
+without constants or `label_key`. Fix those first. It also does **not**:
 
 1. Run `npm install`, so regenerate and commit `package-lock.json` (CI uses
    `npm ci`). Commit the `uv.lock` change the scaffold's

docs/framework/invalidation.md

@@ -80,7 +80,7 @@ A transport failure never propagates. The caller has already committed; turning
 
 Every framework channel shares that one Redis channel; `Invalidation.channel` routes inside the receiving process.
 
-> **A stored override beats the environment.** `settings.hydrate.hydrate_settings` passes DB values to the settings class as *init args*, and pydantic-settings ranks init args above its env source — so once `background_tasks.broadcast_invalidations` has a row in the settings store (written from the admin UI, or by `smpy settings import-from-env`), the env var above stops having any effect after boot hydration. This inverts the precedence CLAUDE.md states (`env → DB → default`) and is not specific to these two fields: it applies to every env-readable module setting, `broker_url` included. Check the settings screen, not just the container's environment, when a knob appears not to work.
+> **A stored override beats the environment.** `simple_module_core.settings_hydrate.hydrate_settings` passes DB values to the settings class as *init args*, and pydantic-settings ranks init args above its env source — so once `background_tasks.broadcast_invalidations` has a row in the settings store (written from the admin UI, or by `smpy settings import-from-env`), the env var above stops having any effect after boot hydration. This inverts the precedence CLAUDE.md states (`env → DB → default`) and is not specific to these two fields: it applies to every env-readable module setting, `broker_url` included. Check the settings screen, not just the container's environment, when a knob appears not to work.
 
 > **Redis pub/sub ignores the database index.** A publisher on `redis://host/0` and a subscriber on `redis://host//1` with the same channel name *do* reach each other — pub/sub is server-global, not per-database. So the `dev-services` convention of giving each repo its own logical DB (this one owns 4 and 5) isolates keys but **not** invalidations: two installs left on the default channel name will clear each other's caches, and each will see the other's user ids on the wire. Verified by experiment. Rename the channel per app; do not rely on the DB number.
 

docs/framework/settings.md

@@ -37,45 +37,68 @@ Outside the non-prod environments, boot fails if `secret_key == "change-me-in-pr
 
 ## Module settings
 
-Each module that needs configuration declares a **`<Module>Env`** pydantic-settings class with its own `env_prefix` and a **`<Module>State`** dataclass to bundle settings plus any runtime-computed singletons.
+Each module that needs configuration declares a `DbBackedSettings` subclass
+(constructor + DB, no env source) and a services dataclass that bundles the
+settings with any runtime singletons, then registers both from
+`register_settings`:
 
 ```python
-# modules/users/users/settings.py
-from pydantic_settings import BaseSettings
+# modules/orders/orders/settings.py
+from pydantic_settings import SettingsConfigDict
+from simple_module_core.settings_base import DbBackedSettings
 
 
-class UsersEnv(BaseSettings):
-    allow_signup: bool = False
-    base_url: str | None = None
+class OrdersSettings(DbBackedSettings):
+    model_config = SettingsConfigDict(extra="ignore")
 
-    class Config:
-        env_prefix = "SM_USERS_"
+    max_items_per_order: int = 100
 ```
 
 ```python
-# modules/users/users/state.py
-from dataclasses import dataclass
+# modules/orders/orders/module.py
+from simple_module_core.module_settings import register_module_settings
 
 
-@dataclass
-class UsersState:
-    settings: UsersEnv
-    mailer: Mailer  # sends through the mail module's sender
-    principal_serializer: Callable[[User], dict]
+class OrdersModule(ModuleBase):
+    def register_settings(self, app: FastAPI) -> None:
+        from orders.services import OrdersServices
+        from orders.settings import OrdersSettings
+
+        register_module_settings(
+            app, "orders", OrdersSettings, lambda s: OrdersServices(settings=s)
+        )
 ```
 
-Attach in `register_settings`:
+`register_module_settings` records the class in the app's
+`ModuleSettingsRegistry` and mounts `services_factory(OrdersSettings())` on
+`app.state.orders`. The hosting lifespan then replaces
+`app.state.orders.settings` with an instance hydrated from the
+`settings_setting` table (env → DB → default) before any `on_startup` runs.
+
+The whole mechanism lives in the framework (#500), so it works whether or not
+the `settings` plugin is enabled; the plugin adds the admin editor, hot
+reload and the generic key/value API on top.
+
+| What | Import from |
+|---|---|
+| `register_module_settings`, `ModuleSettingsRegistry`, `module_settings_registry(app)` | `simple_module_core.module_settings` |
+| `hydrate_settings`, `build_settings`, `OverrideSource`, `RELATED_FIELDS_CTX`, field helpers | `simple_module_core.settings_hydrate` |
+| Secret-field masking (`conceals_secret`, `mask`, ...) | `simple_module_core.secret_fields` |
+| `Setting` model, scope/table constants, `get_overrides_sync`, `hydrate_settings_sync` | `simple_module_db.stored_settings` |
+
+The old plugin paths (`settings.registration`, `settings.module_registry`,
+`settings.hydrate`) still re-export these and emit a `DeprecationWarning`;
+`settings.models`, `settings._secrets` and the moved names in
+`settings.store` / `settings.constants` re-export them silently. They go no
+earlier than 0.1.0.
+
+A Celery worker or CLI script never runs the lifespan, so it hydrates
+synchronously:
 
 ```python
-# modules/users/users/module.py
-class UsersModule(ModuleBase):
-    def register_settings(self, app: FastAPI) -> None:
-        settings = UsersEnv()
-        app.state.users = UsersState(
-            settings=settings,
-            mailer=build_mailer(settings),
-            principal_serializer=serialize_user,
-        )
+from simple_module_db.stored_settings import hydrate_settings_sync
+
+cfg = hydrate_settings_sync(OrdersSettings, session, "orders")
 ```
 
 ### Why `app.state.<module_lower>`?

docs/guide/configuration.md

@@ -83,15 +83,16 @@ From then on, edit at `/admin/settings/` (requires the `settings.manage` permiss
 
 ## Per-module settings convention
 
-When you write your own module, declare settings as a `pydantic_settings.BaseSettings` subclass and store them on `app.state.<module_lower>` (the lowercase package name) inside `register_settings(app)`. Env-var prefix is `SM_<MODULE>_*`.
+When you write your own module, declare settings as a `DbBackedSettings` subclass (`simple_module_core.settings_base`; constructor + DB, no env source) and register it from `register_settings(app)` with `register_module_settings`, which mounts your services object on `app.state.<module_lower>` (the lowercase package name) and lets the host hydrate it from the database before `on_startup`.
 
 ```python
 # modules/orders/orders/settings.py
-from pydantic_settings import BaseSettings, SettingsConfigDict
+from pydantic_settings import SettingsConfigDict
+from simple_module_core.settings_base import DbBackedSettings
 
 
-class OrdersSettings(BaseSettings):
-    model_config = SettingsConfigDict(env_prefix="SM_ORDERS_", extra="ignore")
+class OrdersSettings(DbBackedSettings):
+    model_config = SettingsConfigDict(extra="ignore")
 
     max_items_per_order: int = 100
 
@@ -107,12 +108,17 @@ class OrdersServices:
 
 
 # modules/orders/orders/module.py
+from simple_module_core.module_settings import register_module_settings
+
+
 class OrdersModule(ModuleBase):
     def register_settings(self, app: FastAPI) -> None:
         from orders.services import OrdersServices
         from orders.settings import OrdersSettings
 
-        app.state.orders = OrdersServices(settings=OrdersSettings())
+        register_module_settings(
+            app, "orders", OrdersSettings, lambda s: OrdersServices(settings=s)
+        )
 ```
 
 If you override `register_settings` but don't write to `app.state.<module_lower>`, diagnostic `SM012` warns in dev. See [Settings & app.state](/framework/settings) for details.

docs/modules/background_tasks.md

@@ -526,4 +526,4 @@ def convert(file_id: int) -> None:
     cfg = settings_for(FileStorageSettings, "file_storage")
 ```
 
-It reads SYSTEM-scope overrides through the worker's sync session and caches per process for 30 seconds (`ttl=0` always re-reads; `clear_settings_cache()` drops the cache). The underlying sync helper is `settings.hydrate.hydrate_settings_sync`.
+It reads SYSTEM-scope overrides through the worker's sync session and caches per process for 30 seconds (`ttl=0` always re-reads; `clear_settings_cache()` drops the cache). The underlying sync helper is `simple_module_db.stored_settings.hydrate_settings_sync`.

docs/modules/settings.md

@@ -20,9 +20,7 @@ Two distinct surfaces:
 
 ### Register module settings
 
-The pattern (pydantic `BaseSettings` subclass + `register_module_settings` in `register_settings`) is documented in [Per-module settings convention](/guide/configuration#per-module-settings-convention). What the settings module adds on top:
-
-- **DB hydration**: `app.state.<package>.settings` is replaced with a fresh instance built from DB overrides + pydantic defaults during the host's hydrate phase, before `on_startup` runs.
+The pattern (a `DbBackedSettings` subclass + `register_module_settings` from `simple_module_core.module_settings` in `register_settings`) is documented in [Settings & app.state](/framework/settings#module-settings). Registration and boot-time DB hydration belong to the framework since #500 and work with this module disabled. What the settings module adds on top:
 - **Env-var migration**: `smpy settings import-from-env` scans every registered class's `env_prefix` and seeds matching `SM_*` env vars as SYSTEM-scoped rows. Idempotent.
 - **Admin editing**: registered fields appear under that package at `/admin/settings/` with type-aware inputs.
 - **Hot reload**: saving via the admin UI calls `apply_changes_and_reload`, which validates the diff against the pydantic class, persists deltas, swaps the live `app.state.<package>.settings`, and publishes [`SettingsReloaded`](#events) so dependents (SMTP clients, Celery configs, …) can rebuild.
@@ -32,7 +30,7 @@ The pattern (pydantic `BaseSettings` subclass + `register_module_settings` in `r
 `DbBackedSettings` subclasses ignore the environment, and only the hosting lifespan hydrates `app.state.<package>.settings`. A Celery worker (or CLI script) never runs that lifespan, so a bare `FileStorageSettings()` there silently returns pydantic defaults. Hydrate synchronously instead:
 
 ```python
-from settings.hydrate import hydrate_settings_sync
+from simple_module_db.stored_settings import hydrate_settings_sync
 
 cfg = hydrate_settings_sync(FileStorageSettings, session, "file_storage")
 ```

docs/reference/diagnostic-codes.md

@@ -17,7 +17,7 @@ The framework runs a set of static checks over installed modules at app boot. Th
 | `SM004` | WARNING | An `inertia.render("<Module>/<name>")` call exists but no matching `.tsx` file is shipped. | Create `pages/<name>.tsx`, or correct the render key. |
 | `SM007` | INFO | Module overrides zero `register_*` hooks and has no `on_startup`/`on_shutdown`. | Delete the module if it's vestigial; otherwise ignore. |
 | `SM008` | ERROR | Two modules declare the same `ModuleMeta.name`. Table-prefix collision. | Rename one. Remember: the name is the table-name prefix (all modules share one DB schema) and the Inertia namespace. |
-| `SM009` | ERROR | A file under `framework/*` imports from any package under `modules/*`. | Invert the dependency: have the module register a callback into the framework (see `principal_serializer` pattern). |
+| `SM009` | ERROR | A file under `framework/*` imports from any package under `modules/*`, including dynamic imports with literal names or fixed f-string package prefixes. | Invert the dependency: have the module register a callback into the framework (see `principal_serializer` pattern). |
 | `SM010` | ERROR | DB `alembic_version` is behind the migration head. | Run `make migrate`. The lifespan migration check raises and aborts boot in every environment before serving. |
 | `SM011` | WARNING | A module's model declares a table that doesn't appear in any Alembic migration. | Run `uv run alembic revision --autogenerate -m "..."`, review, `make migrate`. |
 | `SM012` | WARNING | Module overrides `register_settings` but does not assign to `app.state.<module_lower>`. Dev-only. | Either remove the override or move your state assignment into it. |