feat: durable event delivery via an outbox (#459)

CHANGELOG.md

@@ -18,6 +18,32 @@ All notable changes to this project are documented in this file. The format is b
   previously served, and anonymous, in production.
 
 ### Added
+- Durable event delivery (#459). `bus.subscribe_durable(EventType, handler,
+  name=...)` and `await bus.publish_durable(event, db)` in
+  `simple_module_core.events`; handlers are called as
+  `await handler(event, ctx: DurableContext)`. `background_tasks` installs the
+  transport: an outbox table (`background_tasks_outbox`, **migration**
+  `e878882924e7`) written in the publisher's transaction, a post-commit
+  `background_tasks.dispatch_outbox` task, and a 30 s beat sweep
+  (`background_tasks.sweep_outbox`) for crashes between commit and publish.
+  At-least-once delivery with exponential backoff (10 s doubling, 1 h cap);
+  writes through `ctx.db` commit with the delivery mark and land exactly once
+  (a handler committing `ctx.db` itself is refused with `HandlerCommitError`);
+  other side effects are at-least-once. Handlers run in the scope recorded at
+  publish (`scope`: the publisher's tenant; `all_tenants()` for a publish made
+  inside `all_tenants()`; no tenant — strict — for one made with none bound),
+  with a timeout inside the lease, and
+  may publish durably themselves, in a worker too. Timestamps come from the
+  database clock. Rows go `dead` after `outbox_max_attempts` (8); an unknown
+  handler (rolling deploy) is retried like a failure. New admin page
+  `/admin/background-tasks/outbox` (retry dead, discard) and API under
+  `/api/background_tasks/admin/outbox`. New settings `outbox_max_attempts`,
+  `outbox_sweep_interval_seconds`, `outbox_lease_seconds`,
+  `outbox_handler_timeout_seconds`. Delivered rows are
+  purged after `retention_days`. Without background_tasks, `publish_durable`
+  raises `DurableDeliveryUnavailable`; `publish` / `publish_nowait` are unchanged.
+  New `simple_module_db.no_tenant()` context: no tenant bound and outside any
+  `all_tenants()`.
 - **New `api_tokens` module** (#455): personal access tokens (`smpat_…`) and
   service API keys (`smsk_…`). Only `sha256(token)` is stored and the secret
   is shown once; a token is resolved by one indexed lookup per request (no

docs/database/mixins.md

@@ -95,7 +95,7 @@ INSERTs populate `tenant_id` from the `current_tenant_id` contextvar. Creating a
 
 ### Bypass
 
-A row inserted outside any tenant context (no active `current_tenant_id`) is stamped `None` and, because the column is non-nullable, fails at the DB rather than silently leaking. There is no per-statement `skip_tenant_filter` option or `no_tenant()` helper today; cross-tenant admin operations run outside a tenant-scoped request (e.g. a CLI or background worker where `current_tenant_id` is unset).
+A row inserted outside any tenant context (no active `current_tenant_id`) is stamped `None` and, because the column is non-nullable, fails at the DB rather than silently leaking. There is no per-statement `skip_tenant_filter` option; cross-tenant work uses `all_tenants()` (see [multi-tenancy](../framework/multi-tenancy.md)). `no_tenant()` is the opposite: it forces the unscoped state, where strict mode fails closed.
 
 ## `VersionedMixin`
 

docs/framework/events.md

@@ -1,6 +1,6 @@
 # Events
 
-`EventBus` is an **in-process** pub/sub mechanism. It's not a message broker — no persistence, no retries, no cross-process delivery. For durable async work, use the `background_tasks` module (Celery) or publish a domain event and have a handler enqueue a Celery task. To tell *other worker processes* that something they cached is stale, use the [invalidation bus](/framework/invalidation) instead — same shape, deliberately narrower contract.
+`EventBus` is an **in-process** pub/sub mechanism. `publish` is not a message broker — no persistence, no retries, no cross-process delivery. For events that must be handled even across a crash, use [durable delivery](#durable-delivery) (`subscribe_durable` / `publish_durable`), which the `background_tasks` module backs with an outbox. To tell *other worker processes* that something they cached is stale, use the [invalidation bus](/framework/invalidation) instead — same shape, deliberately narrower contract.
 
 Events are the preferred way for modules to react to **other modules'** actions without creating a direct code dependency.
 
@@ -125,7 +125,7 @@ If you want a handler to see several event types, subscribe it to each one expli
 - **No persistence.** A crash mid-publish loses undelivered events.
 - **Handler failures are isolated.** `publish` gathers with `return_exceptions=True`: each handler exception is logged and the publish itself never raises.
 
-If you need durable delivery across processes, handlers should enqueue a Celery task:
+If you need durable delivery across processes, use [durable delivery](#durable-delivery) below. The older pattern — a handler that enqueues a Celery task — still works, but loses the event if the process dies between the commit and the enqueue:
 
 ```python
 def register_event_handlers(self, bus: EventBus) -> None:
@@ -138,6 +138,59 @@ async def _enqueue_invoice(self, event: OrderPlaced) -> None:
     create_invoice_task.delay(event.order_id)
 ```
 
+## Durable delivery
+
+```python
+from simple_module_core import DurableContext
+
+
+def register_event_handlers(self, bus: EventBus, app=None) -> None:
+    bus.subscribe_durable(OrderPlaced, create_invoice)  # name defaults to the import path
+
+
+async def create_invoice(event: OrderPlaced, ctx: DurableContext) -> None:
+    ctx.db.add(Invoice(order_id=event.order_id))
+
+
+# publisher, inside its transaction:
+await bus.publish_durable(OrderPlaced(order_id=order.id, ...), db)
+```
+
+- `publish_durable(event, db)` hands the event to `bus.durable_transport`, which
+  writes it in `db`'s transaction: a rollback means it was never published. With
+  no transport installed it raises `DurableDeliveryUnavailable` rather than
+  quietly falling back to in-process delivery. It does **not** call `subscribe`
+  handlers; publish separately if you want both.
+- Handlers are called later, by a worker, as `await handler(event, ctx)`.
+  Delivery is **at-least-once** with exponential backoff; database writes made
+  through `ctx.db` commit with the delivery record and so land **exactly once**
+  — provided the handler never commits `ctx.db` itself (the transport refuses
+  that with `HandlerCommitError`; flush instead). Everything else a handler does
+  (HTTP calls, emails, other sessions) may happen more than once: make it
+  idempotent. Savepoints (`ctx.db.begin_nested()`) are allowed. Each attempt
+  has a timeout just inside the transport's lease; handlers must not block the
+  event loop (use `run_in_executor` for blocking work) or swallow
+  `CancelledError`. `ctx.attempt` counts from 1. The handler runs in the scope
+  it was published in: the publisher's tenant; `all_tenants()` for an event
+  published inside `all_tenants()`; or, for one published with no tenant bound
+  at all, no tenant (strict isolation fails closed, as it would have for the
+  publisher).
+- A durable handler may itself `publish_durable(..., ctx.db)`; the new event
+  commits with the handler's delivery.
+- The handler `name=` (default `f"{handler.__module__}.{handler.__qualname__}"`)
+  keys delivery state, so keep it stable across deploys.
+- Events must be `@dataclass` classes serialisable to JSON; `UUID`, `datetime`,
+  `Enum`, `Decimal` and nested dataclasses round-trip. This is checked when you
+  subscribe.
+- The worker builds its own bus by calling every module's
+  `register_event_handlers(bus, None)` — subscribe durably *before* any early
+  `return` taken when `app is None`.
+
+`DurableTransport` is a protocol (`async enqueue(event, handler_names, db)`), so
+another module could provide one; `background_tasks` is the shipped
+implementation — see [its docs](/modules/background_tasks#durable-events-outbox)
+for the outbox, retries, the admin page and the worker side.
+
 ## Testing
 
 Subscribe a spy in a test fixture:

docs/framework/multi-tenancy.md

@@ -85,6 +85,11 @@ for tenant_id in tenant_ids:
 A task or asyncio task started inside `all_tenants()` inherits the bypass
 (ordinary contextvar semantics).
 
+`no_tenant()` is the opposite end: no tenant bound and outside any
+`all_tenants()`, so strict mode fails closed. Use it to replay work for code
+that had no tenant (the durable outbox delivers an event published from a
+public route this way), whatever scope the replaying code itself is in.
+
 ## Work deferred past the request
 
 `db.on_commit(...)` callbacks and FastAPI `BackgroundTasks` run inside the
@@ -167,6 +172,12 @@ runs as that tenant. Request code cannot enqueue as another tenant (an explicit
 bound may name one. Beat tasks have no request: wrap cross-tenant work in
 `all_tenants()`, or loop over tenants with `tenant_context()`.
 
+Durable events (`publish_durable`) record the publisher's scope the same way:
+tenant, platform (published inside `all_tenants()`), or unscoped (neither).
+An unscoped event's handler runs with no tenant bound — never as the platform
+— so a public webhook's payload cannot drive a cross-tenant read. See
+[background_tasks](../modules/background_tasks.md).
+
 The worker never builds the app, so `background_tasks.sync_db` attaches the
 same listeners to its own session class and reads `multi_tenant` from the host
 settings (`scripts/run_worker.py`): task bodies get the same fail-closed rules