antosubash · Oct 22, 2026 · Oct 21, 2026 · Oct 21, 2026 · Oct 21, 2026 · Oct 21, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,6 +17,35 @@ All notable changes to this project are documented in this file. The format is b
 - `host/client_app` no longer re-declares `@simple-module-py/ui`'s runtime dependencies, nor `@base-ui/react`, `@radix-ui/react-slot`, `autoprefixer` or `postcss` (nothing loaded them) (#521).
 
 ### Added
+- `background_tasks`: opt-in retained logs. With `log_sink_enabled` (default off;
+  `log_sink_level` default `INFO`, `log_retention_days` default 7) web and worker
+  log lines are kept in the new `background_tasks_log_record` table, batched off
+  the logging path by the new `simple_module_hosting.log_sinks` (`LogSink`
+  protocol, `QueueSinkHandler`: bounded queue that drops and counts instead of
+  blocking, secrets masked before queueing). The task detail page gains a Logs
+  tab, `/admin/background-tasks/logs` searches all retained lines, and
+  `purge_old_executions` enforces retention. Migration `1f5985daa10b` (#537).
+- `background_tasks`: tasks report progress with `heartbeat(progress=(i, n), message=, phase=)`
+  (buffered under the existing 30 s throttle, `i == n` writes at once, never
+  audited), and `GET /api/background_tasks/tasks/{celery_task_id}/status` lets the user who
+  enqueued a task read its status and progress without `background_tasks.view`
+  (marked `Authenticated()`; tokens are refused)
+  (owner in the active tenant, or `view` holders; anyone else gets 404; never
+  returns args, result or traceback). `TaskExecution` gains `progress_*` and an
+  indexed `owner_user_id` stamped from the new `sm_owner_id` message header;
+  retries keep the original owner. The detail page shows a progress bar.
+  Migration `bb906e1a1089` (first revision labelled `background_tasks`) (#536).
+- Distributed TTL leases in `simple_module_core.leases`: `lease(name, ttl)` (an
+  owner-checked lock with `token`, a strictly increasing `fence`, `refresh()`,
+  `release()` and `async with` waiting) and `semaphore(name, slots, ttl)` (at
+  most N holders across processes). Backed by Redis when `SM_REDIS_URL` is set,
+  else the new framework table `sm_lease` (migration `d263ea727663`, a new root
+  labelled `leases`), else in-process; `create_app` and the Celery worker
+  bootstrap install it. An expired owner can neither refresh nor release (#539).
+- Optional Sentry error reporting: `pip install simple_module_hosting[sentry]` and set `SM_SENTRY_DSN` (plus `SM_SENTRY_ENVIRONMENT`, `SM_SENTRY_TRACES_SAMPLE_RATE`). Web and Celery worker processes tag events with request id, tenant, user id, module and task, and scrub auth headers, cookies, `token`/`code` query values and request bodies. Without a DSN the SDK is never imported (#538).
+- Sentry events no longer leak secrets the SDK's exact-name scrubber misses: auth-, session- and token-like headers (`X-Auth-Token`, `Proxy-Authorization`, `X-Forwarded-For`), nested secrets in `extra`/`contexts`, tokens in log and HTTP breadcrumbs, credentials in exception messages, and Celery task args/kwargs are scrubbed for errors and transactions alike (#538).
+- In-process durable outbox delivery and lifespan-owned recovery sweeps, automatically enabled in eager mode (#565).
+- Sentry scrubbing now covers every event field (tags, `server_name`, top-level stacktraces), camelCase and short key names (`basicAuth`, `pwd`, `otp`), header/query lists of pairs, bytes, quoted and escaped-JSON values, whole `Cookie:` lines, passwords containing `@`, JWTs and bare long tokens; SQL `db.params` are dropped. Behaviour change: a scrubber failure now drops the event instead of letting the SDK decide, and strings over 64 KiB are truncated (the old patterns were quadratic: a 20 KB string stalled `before_send` for over a minute) (#538).
 - Per-route rate limits (#529): modules declare `RateLimitRule(rate, key=...)` in the new
   `register_rate_limits(registry)` hook, and handlers use the
   `RateLimit("10/minute", key="user")` dependency. Keys: `user`, `ip`, `user_or_ip`,
@@ -101,6 +130,9 @@ All notable changes to this project are documented in this file. The format is b
   it, which removes the `users` ↔ `audit_log` import cycle.
 
 ### Changed
+- **Behaviour change:** `simple_module_db` no longer emits the `entity.created` / `entity.updated` /
+  `entity.soft_deleted` log lines for models flagged `__audit_exclude__ = True` (`AuditEntry`,
+  `OutboxMessage`, `RetainedLog`), so retained logs cannot re-log their own writes (#537).
 - **Behaviour change:** `rate_limit_authenticated` (off by default) now keeps one budget per
   signed-in user instead of one per client IP, so users behind one NAT no longer share it (#529).
 - `scripts/bump_version.py` also bumps every `modules/*/package.json`; all
@@ -336,6 +368,7 @@ All notable changes to this project are documented in this file. The format is b
   previously served, and anonymous, in production.
 
 ### Fixed
+- Background task stuck sweeps and retention purges now use DB-backed settings so admin edits apply instead of reading only environment variables (#565).
 - User directory follow-ups (#501): `audit_log` installs its activity feed in
   `on_startup` (cleared on shutdown) instead of `register_settings`, which
   tripped `SM012`; `resolve_user_labels` omits users whose label is empty so

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -305,9 +305,10 @@ development and testing (`session_cookie_secure`).
 - **Migrations** live in `host/migrations/versions/`, never in module packages.
   `host/migrations/env.py` builds metadata from every installed module and runs
   `ensure_db_extensions` first. The history is **not** one linear chain: it has
-  two roots (`77162e7b184b` initial schema, `e36ab09c4a92` tenants) and four
-  heads (`audit_log`, `tenants`, `keycloak`, and the users trunk, which Alembic
-  reports as `api_tokens`). Apply with `upgrade heads` (what `make migrate`
+  three roots (`77162e7b184b` initial schema, `e36ab09c4a92` tenants,
+  `d263ea727663` the framework's `sm_lease`) and five heads (`audit_log`,
+  `tenants`, `keycloak`, `leases`, and the users trunk, which Alembic reports
+  as `api_tokens` and, since #536, `background_tasks`). Apply with `upgrade heads` (what `make migrate`
   does); plain `revision --autogenerate` refuses with "Multiple heads", so name
   the branch:
 

diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts
@@ -129,8 +129,11 @@ export default defineConfig({
             { text: "Deprecations", link: "/framework/deprecations" },
             { text: "Events", link: "/framework/events" },
             { text: "Cache invalidation", link: "/framework/invalidation" },
+            { text: "Leases & semaphores", link: "/framework/leases" },
+            { text: "Sentry", link: "/framework/sentry" },
             { text: "Multi-tenancy", link: "/framework/multi-tenancy" },
             { text: "Internationalization", link: "/framework/i18n" },
+            { text: "Retained logging", link: "/framework/logging" },
           ],
         },
       ],

diff --git a/docs/database/migrations.md b/docs/database/migrations.md
@@ -16,12 +16,14 @@ The history is **not** a single linear chain. It has several roots and several h
 |---|---|
 | `77162e7b184b` initial schema | the users trunk (also holds file_storage, background_tasks, api_tokens revisions) |
 | `e36ab09c4a92` tenants initial schema | tenants (`f1a9c3d7b2e4` head) |
+| `d263ea727663` framework `sm_lease` table | leases (its own head; [leases](../framework/leases.md)) |
 
 | Head | Owning module | Label Alembic reports |
 |---|---|---|
 | `c8e4f1a2b6d9` | users (login lockout) | `api_tokens` |
 | `a7c2e91d4b58` | audit_log | `audit_log` |
 | `f1a9c3d7b2e4` | tenants | `tenants` |
+| `d263ea727663` | framework (`sm_lease`) | `leases` |
 | `f6a3b9d2e8c4` | keycloak | `keycloak` |
 
 `audit_log` forks off the trunk at `41cf2c53660e`, and `keycloak` forks off the users revision `873ca2015033`.
@@ -30,7 +32,7 @@ A module's first migration sets `branch_labels = ("<module_name>",)`. Alembic **
 
 ### Known mislabels
 
-Only four of the revisions set `branch_labels` (`audit_log`, `api_tokens`, `keycloak`, `tenants`). The graph carries these defects; they are documented rather than rewritten, because changing labels on revisions that installed databases already applied is not worth the risk:
+Only five of the revisions set `branch_labels` (`audit_log`, `api_tokens`, `keycloak`, `tenants`, `leases`). The graph carries these defects; they are documented rather than rewritten, because changing labels on revisions that installed databases already applied is not worth the risk:
 
 - `51b2058d8afd_api_tokens_initial` sits in the middle of the trunk, so every revision above it (file_storage, background_tasks including the outbox, and the users revisions through `c8e4f1a2b6d9`) reports the `api_tokens` label. The `api_tokens` head is therefore really a users revision.
 - `keycloak` branches off a users revision, so a Keycloak-only install still builds the users schema.

diff --git a/docs/framework/leases.md b/docs/framework/leases.md
@@ -0,0 +1,66 @@
+# Leases and semaphores
+
+`simple_module_core.leases` gives every process the same two primitives ([#539](https://github.com/antosubash/simple_module_python/issues/539)):
+
+- **`lease(name, ttl)`**: an exclusive lock that expires on its own. Use it to deduplicate work ("only one worker imports dataset 42") or to debounce.
+- **`semaphore(name, slots, ttl)`**: at most `slots` concurrent holders, for example at most 4 downloads per remote host across every web and Celery worker.
+
+Both are *leases*: whoever acquires one gets a random `token`, and only that token can refresh or release it. Once the TTL lapses, the old holder can do neither, even if no one else has taken it yet.
+
+## Usage
+
+```python
+from simple_module_core.leases import LeaseNotAcquiredError, lease, semaphore
+
+# Exclusive. Waits up to 5 s for the current holder, then raises.
+async with lease("import:42", ttl=60, wait=5) as held:
+    for chunk in chunks:
+        await process(chunk)
+        if not await held.refresh():  # extend the TTL from now
+            raise RuntimeError("lost the lease")  # it expired; stop writing
+
+# Try once, never wait.
+job = lease("rebuild-index", ttl=300, blocking=False)
+if await job.acquire():
+    try:
+        await rebuild()
+    finally:
+        await job.release()
+
+# Counted slots, shared across processes.
+async with semaphore("dl:example.org", slots=4, ttl=120, wait=30):
+    await download(url)
+```
+
+| Member | Meaning |
+|---|---|
+| `token` | The owner's identity. Random unless you pass `owner=` (for example a Celery task id, so a retry of the same task can re-acquire its own lease). |
+| `fence` | A number that strictly increases each time the name is acquired. Store it next to whatever the holder writes and refuse a write that carries a smaller fence: that rejects a holder that stalled past its TTL and woke up late. |
+| `acquire(blocking=True, wait=None)` | Returns `True` once held. Non-blocking tries once; blocking polls with jittered back-off until it gets the lease or `wait` seconds pass (`None` waits forever). |
+| `refresh(ttl=None)` | Extends the TTL from now. Returns `False` if the lease expired or moved to another owner. |
+| `release()` | Returns `False` if the lease had already expired or moved on; it never releases someone else's lease. |
+| `async with` | Acquires with the handle's `blocking`/`wait`, raises `LeaseNotAcquiredError` if it cannot, and releases on exit. |
+
+A lease does not refresh itself. Pick a TTL longer than the work, or call `refresh()` from the loop that does it.
+
+## Backends
+
+The backend is process-global because a Celery worker has no `app.state`. `create_app` installs it when the app is built, and the worker bootstrap (`background_tasks.runner.build_runner_celery`) installs the same kind from the same settings:
+
+| Configured | Backend | Notes |
+|---|---|---|
+| `SM_REDIS_URL` | `RedisLeaseBackend` (`simple_module_hosting`) | `SET NX PX` plus Lua compare-and-set for refresh and release; a semaphore is a sorted set scored by expiry against Redis `TIME`; `INCR <key>:fence` for the fence. |
+| a database, no Redis | `DbLeaseBackend` (`simple_module_db`) | Rows in `sm_lease (name, slot, holder, fence, expires_at)`, primary key `(name, slot)`. Every comparison uses the database clock, so clock skew between hosts does not matter. |
+| neither | `InProcessLeaseBackend` | One process only. Also what tests and eager Celery get when nothing else is configured. |
+
+Backend errors propagate. A lease that the backend could not confirm is never reported as held. This is unlike the rate limiter, which falls back to per-process counters.
+
+Override the backend with `configure_leases(backend)`, or pass `backend=` to a single `lease()`/`semaphore()` call. `configure_leases(None)` restores a fresh in-process backend, which is useful in test teardown.
+
+### The `sm_lease` table
+
+The table belongs to the framework, like `settings_setting`, so it exists whether or not any module uses leases. Its migration `d263ea727663` is its own root with the `leases` branch label (see [Migrations](../database/migrations.md)). `make migrate` (`upgrade heads`) applies it.
+
+Rows are never deleted: a released lease is a row whose `expires_at` is in the past. That is what keeps `fence` increasing for the life of the name. A lease uses slot `0`; a semaphore uses slots `0..n-1` and keeps its fence counter in slot `-1`. Use a bounded set of names: `"dl:<host>"` is fine, a fresh UUID per request is not.
+
+On SQLite each attempt takes the database write lock before it reads, so processes queue on `busy_timeout` instead of racing. The table works on SQLite, but production deployments with several hosts should use Postgres or Redis.
diff --git a/docs/framework/logging.md b/docs/framework/logging.md
@@ -0,0 +1,57 @@
+# Retained logging
+
+`simple_module_hosting.log_sinks` lets a module keep log lines somewhere durable
+without putting that work on the logging call. The `background_tasks` module
+ships the database sink; any module can add another.
+
+## The sink protocol
+
+```python
+class LogSink(Protocol):
+    def emit_batch(self, records: Sequence[logging.LogRecord]) -> None: ...
+```
+
+`emit_batch` runs on a background thread, never in a request, and receives
+records that have already been redacted. Raising is allowed: the batch is
+counted in `QueueSinkHandler.failed_batches`, a warning is logged, and the
+records are lost.
+
+## Installing one
+
+```python
+from simple_module_hosting.log_sinks import install_sink, uninstall_sink
+
+handler = install_sink(MySink(), level=logging.INFO)  # attaches to the root logger
+...
+uninstall_sink(handler)  # detaches and flushes
+```
+
+`QueueSinkHandler` is a `logging.handlers.QueueHandler` plus a listener thread:
+
+| Behaviour | Detail |
+|---|---|
+| Batching | up to 200 records, or whatever arrived within 1 second |
+| Bounded | 10,000 queued records; when full a record is dropped and counted in `handler.dropped`. Logging never blocks and cannot deadlock on a slow sink |
+| Level | `level` is a threshold on the handler; the logger it is attached to must also pass the record (a root logger at `WARNING` never offers `INFO` lines) |
+| Self-exclusion | records from `simple_module_hosting.log_sinks` and `sqlalchemy.*` are skipped, so a database sink cannot log about itself |
+| Correlation | `record.correlation_id` is filled from the request context when unset |
+
+Handler filters run on the emitting thread, where the request or task context
+is bound. Add a filter to the handler (as `background_tasks` does for the task
+id and the tenant) to capture context; the listener thread cannot see it.
+
+## Redaction
+
+`redact(text)` and `redact_record(record)` mask `Authorization`,
+`Proxy-Authorization`, `Cookie` and `Set-Cookie` values, bearer tokens, and the
+value of any `key=value` / `"key": "value"` whose key contains `password`,
+`passwd`, `token`, `secret` or `api_key`. Extra attributes whose *name*
+matches are replaced outright. `QueueSinkHandler` redacts a copy of the record
+before queueing it, so the original, shared with every other handler, is never
+changed. Redaction is pattern-based: it cannot recognise a secret that is
+logged without a label.
+
+## Database sink
+
+See [Retained task and app logs](/modules/background_tasks#retained-task-and-app-logs)
+for the table, settings, retention and the admin screens.
diff --git a/docs/framework/sentry.md b/docs/framework/sentry.md
@@ -0,0 +1,74 @@
+# Sentry
+
+Optional error reporting for the web process and Celery workers. It costs
+nothing until you opt in: with `SM_SENTRY_DSN` unset, `init_sentry()` returns
+before `sentry_sdk` is imported.
+
+```bash
+pip install "simple_module_hosting[sentry]"
+export SM_SENTRY_DSN=https://<key>@o0.ingest.sentry.io/<project>
+```
+
+| Variable | Default |
+|---|---|
+| `SM_SENTRY_DSN` | unset (disabled) |
+| `SM_SENTRY_ENVIRONMENT` | `SM_ENVIRONMENT` |
+| `SM_SENTRY_TRACES_SAMPLE_RATE` | `0.0` |
+
+These are env only: the DSN is a secret and the SDK has to start before the DB
+is open. The DSN is never logged.
+
+## Where it starts
+
+- Web: `create_app` calls `init_sentry("web")` before any middleware is built.
+- Worker: background_tasks calls `init_sentry("worker")` on `worker_process_init`,
+  so every prefork child reports.
+
+## Tags
+
+| Tag | Source |
+|---|---|
+| `request_id` | the correlation id |
+| `tenant` | the request's resolved tenant, or the task's tenant |
+| `user_id` | the principal's id (`user.id` is set; never email, username or IP) |
+| `module` | top-level package of the route endpoint, or the prefix of `<module>.<task>` |
+| `task` | the Celery task name (worker events) |
+| `process` | `web` or `worker` |
+
+## Scrubbing
+
+`send_default_pii` is off and local variables are not attached to stack frames
+(they hold the ASGI scope). The SDK's own scrubber matches exact key names
+only, so `simple_module_hosting.sentry_scrub` runs a stricter pass from both
+`before_send` and `before_send_transaction`:
+
+- After targeted passes, *every* event field (tags, `server_name`, threads,
+  a top-level `stacktrace`, fields a future SDK adds) goes through the same
+  recursive pass: deny by default rather than a list of known fields.
+- Any key, at any depth, whose name contains `password`/`passwd`/`pwd`,
+  `passphrase`, `secret`, `token`, `authorization`, `cookie`, `session`,
+  `api_key`, `csrf`, `credential`, `private_key`, `signature`, `dsn`, `jwt`,
+  `bearer`, a card-number word or a client-address word (`forwarded`,
+  `real_ip`, `remote_addr`, …), or has a whole-word `auth`, `pass`, `otp`,
+  `pin`, `sid`, `sig`, `ssn`, `cvv`, `mfa` (snake, kebab and camelCase:
+  `basicAuth`, `DB_PASS`, but not `author`), is replaced with `[Filtered]`.
+  `["Authorization", "…"]` pairs in header/query lists count as keys. Such
+  request headers are removed outright.
+- Request cookies, bodies and `env` (`REMOTE_ADDR`) are dropped; the event
+  user is reduced to `{"id": …}`.
+- Every string (bytes too) loses `key=value` / `key: value` /
+  `"key": "quoted value"` pairs with a sensitive key (escaped JSON included),
+  whole `Cookie:`/`Authorization:` header lines, the OAuth `code` query
+  parameter, `Bearer`/`Basic`/`Digest` credentials, URL userinfo, JWTs and
+  bare 32+ character tokens mixing letters and digits (UUIDs and
+  snake_case identifiers excepted; Sentry's own event/trace/span ids are
+  kept). Strings over 64 KiB are truncated before scanning.
+- Celery task `args`, `kwargs` and return values, SQL `db.params` and frame
+  `vars` are dropped wherever they appear.
+- URL-sourced transaction names and URL paths have id- and
+  token-shaped segments replaced with `*`.
+- If scrubbing raises, the event is dropped (only the exception type is
+  logged); a half-scrubbed event is never sent.
+- Not covered: secrets passed as positional log arguments
+  (`log.info("%s", token)`) with nothing marking them as secret, and
+  base64 tokens with `/` shorter than 32 characters between slashes.