fix: S3 streaming + Range, task heartbeat + after-commit enqueue, token rehash, scaffold fixes, QA follow-ups (#449–#472)

.github/workflows/pr.yml

@@ -217,6 +217,12 @@ jobs:
       # never in the generic module editor" rule is actually exercised — the
       # server-side half of that is a 409 guard, which is worth CI coverage.
       SM_MODULES_ENABLED: '["Auth","Users","Dashboard","Permissions","Settings","BackgroundTasks","FileStorage","FeatureFlags","AuditLog","Branding"]'
+      # Every test signs in from scratch, all from 127.0.0.1: the login page,
+      # the login POST and the landing page are anonymous public-route hits,
+      # ~120 a minute for the whole suite — right at the default
+      # ``rate_limit_public`` of 120/minute, so a fast runner 429s whichever
+      # test logs in last. The limiter itself is covered by unit tests.
+      SM_RATE_LIMIT_PUBLIC: 1000/minute
       E2E_BASE_URL: http://localhost:8000
     steps:
       - uses: actions/checkout@v7

CHANGELOG.md

@@ -12,6 +12,19 @@ All notable changes to this project are documented in this file. The format is b
 ## [Unreleased]
 
 ### Added
+- `from background_tasks import heartbeat` (#451): a long-running task calls
+  `heartbeat()` to refresh `heartbeat_at` (throttled to one write per 30 s; a
+  no-op outside a task). `@shared_task(..., stuck_after=<seconds>)` sets a
+  per-task stuck threshold that the sweep honours ahead of
+  `SM_BG_TASKS_STUCK_AFTER_SECONDS`. No migration.
+- `from background_tasks import enqueue_after_commit` (#452): publishes a task
+  only once the caller's transaction commits (nothing on rollback), and
+  returns the reserved task id so a row can name it up front.
+- HTTP `Range` support on file downloads (#453): single ranges answer `206`
+  (`416` when unsatisfiable, multi-range falls back to `200`), honouring
+  `If-Range`. Storage backends gain `get_range(key, start, end)` (inclusive
+  end); a third-party backend without it keeps serving full `200`s.
+- `smpy host gen-i18n` and a scaffold `make gen-i18n` target (#449).
 - **Host override layer for module copy** (#415). A host ships
   `host/locales/overrides/<lang>.json` (nested or flat dotted keys) and it is
   applied after every module, framework and host catalog, so the server-side
@@ -166,6 +179,40 @@ All notable changes to this project are documented in this file. The format is b
   `tenants` resolver it selects among the user's own memberships.
 
 ### Fixed
+- Timezone-aware timestamp columns read back aware on SQLite (#468). Columns
+  declared `sa_type=DateTime(timezone=True)` (the `AuditMixin`/`SoftDeleteMixin`
+  timestamps, audit log, users, permissions, tenant invitations) came back
+  naive, so the API sent them with no offset and browsers showed UTC as local
+  time. New `simple_module_db.AwareDateTime` replaces that column type; it
+  migrates as `sa.DateTime(timezone=True)`, so no schema change.
+- Background-task Retry sends one request per execution while one is in
+  flight (#467); a burst of confirm clicks no longer queues duplicate retries.
+- The background-tasks list shows its plain empty state when no executions
+  exist at all, instead of claiming some exist outside the search (#469).
+- Deleting files another tab already deleted says so, instead of a success
+  toast reading "0 files deleted" (#470).
+- `smpy`'s workspace-member hint prints `./modules/...` for a module inside
+  the host directory, so the suggested `smpy add` reads it as a path, not a
+  PyPI name (#471).
+- Scaffolded compose apps require a URL-safe `POSTGRES_PASSWORD`: compose puts
+  it unencoded into `SM_DATABASE_URL`, so `make docker-up` now refuses other
+  characters with a clear message (#472).
+- `S3Backend.get` streams objects in 64 KiB reads instead of buffering the
+  whole object in memory (#453). It is now awaited like the filesystem
+  backend, which also fixes S3 thumbnails and the streamed public-file path
+  (they raised `TypeError`), and a missing object answers `404`, not `500`.
+- Admin task retries publish to the broker only after the new
+  `TaskExecution` rows commit (#452); a rolled-back request no longer leaves a
+  queued task with no execution row.
+- `POST /api/users/auth/token` persists the upgraded password hash, as the
+  browser login already did (#457), so bcrypt hashes imported from another
+  system migrate to Argon2 on API/mobile sign-in too.
+- Scaffolded apps regenerate `keys.generated.ts` in development (#449): when
+  `packages/i18n/src` is absent the host writes into the installed
+  `@simple-module-py/i18n` package, merging the keys it already ships.
+- Scaffolded `make kill` stops only this app's ports (`API_PORT`, and
+  `VITE_PORT` read from `SM_VITE_DEV_URL`) instead of `pkill`-ing every
+  uvicorn and vite on the machine, vitest runs included (#461).
 - `render_item` renders sqlmodel>=0.0.48's `UTCDateTime` as
   `sa.DateTime(timezone=True)` and adds `import sqlmodel` for any other
   sqlmodel type, so an initial autogenerate no longer emits a migration that

docs/framework/i18n.md

@@ -283,6 +283,8 @@ The detection logic lives in `scripts/lib/untranslated-strings.mjs` behind its o
 
 `packages/i18n/src/keys.generated.ts` is generated over every **installed** module, not just the ones the running host activated, and `make ci-js-typecheck` runs `tsc -p` for each `modules/*/tsconfig.json` and `packages/*/tsconfig.json` in the workspace. (A module shipping `.tsx` with no `tsconfig.json` fails the target outright, rather than being silently skipped — the same gap `SM017` reports.)
 
+**Scaffolded apps.** There is no `packages/i18n` checkout in a scaffolded app: `@simple-module-py/i18n` is an npm dependency. When `packages/i18n/src` is absent, development boot (and `make gen-i18n` / `smpy host gen-i18n`) writes `keys.generated.ts` and `generated-resources.ts` into the nearest `node_modules/@simple-module-py/i18n/src` found by walking up from the project root (so a hoisted npm-workspace install is found; a flat host also checks `client_app/node_modules`). The shared `ui.*` and framework keys come from catalogs inside `simple_module_hosting`, and any keys already in the published file are kept (the published catalog covers modules this app does not install, so they are not treated as removals). `npm install` restores the published file; `make install` regenerates afterwards. With neither location present (a wheel-installed production app) boot skips silently; `make gen-i18n` fails loudly.
+
 That distinction is load-bearing: with a filtered union, translating an inactive auth provider's pages (say `keycloak`, when this host runs `users`) would break its build on the next regeneration while the app itself ran fine. The runtime registry stays filtered — an inactive module's strings are typed, never served.
 
 `t()` needs literal keys to typecheck; a key assembled at runtime won't resolve.

docs/guide/installation.md

@@ -89,7 +89,7 @@ POSTGRES_PASSWORD=<generated>
 SM_DATABASE_URL=postgresql+asyncpg://postgres:<generated>@localhost:5432/my_app
 ```
 
-The database name is the snake_case app name. `POSTGRES_PASSWORD` is shared with the compose `postgres` service below.
+The database name is the snake_case app name. `POSTGRES_PASSWORD` is shared with the compose `postgres` service below. Compose interpolates it unencoded into the container's `SM_DATABASE_URL`, so it must be URL-safe (letters, digits, `-`, `_`, `.`, `~`); `make docker-up` refuses to start otherwise. The generated value always is.
 
 Every scaffold ships a `docker-compose.yml` that matches your `--db` choice (a migration history is dialect-frozen at autogenerate time, so containers must run the same database the migrations were generated against). With `--db postgres` it brings up `postgres` (db named after your app, user `postgres`, password from `POSTGRES_PASSWORD` in `.env`) on `127.0.0.1:5432` and the `app` service; the default SQLite scaffold runs the `app` container alone with the database on a named volume. Including the `background_tasks` module adds `redis` on `127.0.0.1:6379` plus `worker`/`beat` services that reuse the app image. With `--db postgres`, the container also doubles as a local dev database:
 
@@ -165,7 +165,7 @@ Pass `--loosen` to rewrite every constraint to `>=<latest>` instead.
 Run `uv tool update-shell` (or restart your shell) so the tool's bin dir is on PATH.
 
 **Port already in use.**
-Free `:8000` and `:5050` before the next `make dev`. On Linux/macOS: `lsof -ti:8000,5050 | xargs kill -9`.
+Run `make kill` to free the app's API and Vite ports (`:8000` and the port in `SM_VITE_DEV_URL`, default `:5050`) before the next `make dev`. It only touches those ports, never other apps' servers. To run two apps side by side, give each its own `API_PORT` (`make dev API_PORT=8010`) and `SM_VITE_DEV_URL`.
 
 **Alembic complains about a revision mismatch.**
 The DB is ahead of or behind the migration files. For a dev DB: `rm app.db && make migrate`. For Postgres: `docker compose down -v && docker compose up -d postgres && make migrate`.

docs/modules/background_tasks.md

@@ -32,6 +32,63 @@ send_receipt.delay(order_id=42)
 
 Every dispatch / start / completion / failure / retry / revoke is recorded in the `background_tasks_task_execution` table by Celery signal handlers, so you can see history even after the Celery result backend has expired the result.
 
+## Enqueueing from inside a transaction
+
+`send_receipt.delay(...)` publishes *now*. If the request that called it is still writing the row the task will read, two things go wrong: a fast worker (or eager mode, which runs the task inside the publish) starts before the row is visible, and if the commit then fails the task is queued against a row that never existed. Publish after commit instead:
+
+```python
+from background_tasks import enqueue_after_commit
+
+
+async def place(self, data: OrderCreate) -> OrderOut:
+    order = Order(**data.model_dump())
+    self.db.add(order)
+    await self.db.flush()
+    enqueue_after_commit(self.db, send_receipt, order.id)
+    return OrderOut.model_validate(order)
+```
+
+`enqueue_after_commit(db, task, *args, **kwargs)` sends `task` once `db`'s transaction commits and drops it if the transaction rolls back or the commit fails. It returns the Celery task id the message will carry, reserved up front, so you can store it on your own row in the same transaction.
+
+- `task` is a task (`send_receipt`), a signature for routing options (`send_receipt.s().set(queue="mail", countdown=30)`), or a registered task name (`"orders.send_receipt"`, sent through `celery.current_app`).
+- In a request (`get_db`), the publish runs from the session's `on_commit` hook — after the commit, before the response leaves.
+- Any other session — a task body's sync session, a CLI script's own `async with session.begin()` — hangs it off SQLAlchemy's `after_commit` event instead, and drops it when the outermost transaction ends without committing. Publishing then blocks the committing thread for the length of one kombu publish.
+- The task runs as the tenant bound when you called `enqueue_after_commit`, not whatever is bound at commit time.
+- A publish that fails after the commit is logged, not raised: the transaction is already durable.
+
+The admin retry buttons follow the same rule — the new `pending` row commits first, then the task is sent (GH #452). An orphaned `pending` row that was never sent is visible and retryable; a task with no row is neither.
+
+## Long-running tasks: heartbeats and stuck thresholds
+
+The beat sweep flips a `running` row to `stuck` once its `heartbeat_at` is older than the stuck threshold (`stuck_after_seconds`, default 300), which makes it retry-eligible. The signal handlers only stamp the heartbeat when a task starts and finishes, so a task that runs longer than the threshold must say it is still alive, or it can be retried — and run twice — while the first run is still working.
+
+Call `heartbeat()` from the task body's loop:
+
+```python
+from background_tasks import heartbeat
+from celery import shared_task
+
+
+@shared_task(name="reports.rebuild")
+def rebuild(report_id: int) -> None:
+    for chunk in chunks(report_id):
+        process(chunk)
+        heartbeat()
+```
+
+It refreshes `heartbeat_at` on the current execution's row (found through Celery's `current_task`), writing at most once every 30 seconds per task however often you call it, so calling it on every iteration is fine; `heartbeat(force=True)` skips the throttle. Outside a running task it does nothing and returns `False`, so shared helpers can call it unconditionally. It only touches a row that is still `running` — a heartbeat never revives a row the sweep already declared stuck.
+
+A task whose steps are legitimately longer than the global threshold can declare its own, in seconds:
+
+```python
+@shared_task(name="exports.full_dump", stuck_after=4 * 3600)
+def full_dump() -> None: ...
+```
+
+Celery copies extra decorator options onto the task class; the sweep reads `stuck_after` off the task registry of the worker running it (which autodiscovers every module's tasks), and falls back to `SM_BG_TASKS_STUCK_AFTER_SECONDS` / the default for tasks without one. No column or migration is involved, so the threshold always matches the deployed code. A shorter `stuck_after` works too, for tasks that should never take long. Keep any threshold well above 30 seconds, or the heartbeat throttle alone can let a live task look stale.
+
+Not provided: an automatic heartbeat thread. A task that blocks in one long call (a single huge query, an external process) cannot heartbeat from inside it — give it a `stuck_after` that covers the call instead.
+
 ## Scheduling periodic work (beat)
 
 A module schedules recurring work the same way it registers tasks — by shipping a `tasks.py` — and additionally exporting a module-level `BEAT_SCHEDULE` dict. `build_celery` merges every installed module's `BEAT_SCHEDULE` into the beat schedule at boot (identically in the web and worker processes), so `make beat` runs them alongside the built-ins:
@@ -145,7 +202,7 @@ DB-backed; defaults are in `BackgroundTasksSettings`. Several are marked `requir
 | `task_default_queue` | `"default"` | restart required |
 | `task_always_eager` | `False` | runs tasks in-process (test mode) |
 | `task_eager_propagates` | `True` | re-raises eager-mode errors |
-| `stuck_after_seconds` | `300` | heartbeat-staleness threshold |
+| `stuck_after_seconds` | `300` | heartbeat-staleness threshold; a task's own `stuck_after` overrides it — see [Long-running tasks](#long-running-tasks-heartbeats-and-stuck-thresholds) |
 | `stuck_sweep_interval_seconds` | `60` | beat cadence for the stuck sweep |
 | `purge_interval_seconds` | `86_400` | beat cadence for old-row purge |
 | `retention_days` | `14` | how long to keep terminal rows; range 1-3650 |
@@ -170,7 +227,7 @@ Bootstrap env-var equivalents (`SM_BG_TASKS_*`) only seed pydantic defaults at f
 
 | Task | Schedule | What |
 |---|---|---|
-| `background_tasks.sweep_stuck_tasks` | every `stuck_sweep_interval_seconds` | flips `running` rows with stale heartbeats to `stuck` |
+| `background_tasks.sweep_stuck_tasks` | every `stuck_sweep_interval_seconds` | flips `running` rows whose heartbeat is older than their task's `stuck_after` (else `stuck_after_seconds`) to `stuck` |
 | `background_tasks.purge_old_executions` | every `purge_interval_seconds` | deletes terminal rows older than `retention_days` |
 
 A `background_tasks.demo_echo` task is also registered for smoke tests; not scheduled.
@@ -202,7 +259,7 @@ Synchronous handlers in `signals.py`, all bound through Celery's `signals.<name>
 |---|---|
 | `before_task_publish` | inserts a `pending` row |
 | `task_prerun` | flips to `running`, stamps `started_at` + `heartbeat_at` |
-| `task_postrun` | refreshes `heartbeat_at` |
+| `task_postrun` | refreshes `heartbeat_at` (between prerun and postrun, only `heartbeat()` from the task body does) |
 | `task_success` | flips to `success`, stores `result`, sets `finished_at` |
 | `task_failure` | flips to `failed`, stores `traceback` + `exception_type`, publishes `TaskFailed` |
 | `task_retry` | flips to `retrying`, increments `retries` |

docs/modules/file_storage.md

@@ -22,7 +22,7 @@ Pluggable file storage with two shipped backends — local filesystem and S3-com
 | `GET /api/file-storage/files/{file_id}` | → `StoredFileOut` | `file_storage.download` |
 | `PATCH /api/file-storage/files/{file_id}` | `{"public": bool}` → `StoredFileOut` | `file_storage.upload` |
 | `GET /api/file-storage/files/{file_id}/thumbnail` | `?w=` → `image/webp` | `file_storage.download` |
-| `GET /api/file-storage/files/{file_id}/download` | → 302 (S3) or stream (filesystem) | `file_storage.download` |
+| `GET /api/file-storage/files/{file_id}/download` | → 302 (S3) or stream (filesystem); honours `Range` (see [Range requests](#range-requests)) | `file_storage.download` |
 | `DELETE /api/file-storage/files/{file_id}` | → 204 | `file_storage.delete` |
 | `GET /api/file-storage/public/{file_id}[/{filename}]` | bytes (**anonymous**) | none, `public` files only |
 | `GET /api/file-storage/public/{file_id}/thumbnail` | `?w=` → `image/webp` (**anonymous**) | none, `public` files only |
@@ -85,6 +85,29 @@ Serving rules:
 - Presigning backends (S3) answer with a `302` to the presigned URL, cached
   for at most half the signature's TTL; filesystem backends stream.
 
+### Range requests
+
+Whenever bytes stream through the app (the download endpoint on a
+non-presigning backend, and the public route for filesystem backends or active
+content), a single `Range` is honoured so COG / PMTiles readers can fetch
+just the tiles they need:
+
+- `bytes=a-b`, `bytes=a-` and the suffix form `bytes=-n` answer `206 Partial
+  Content` with `Content-Range: bytes a-b/size` and the slice's
+  `Content-Length`. An end past the object is clamped to `size - 1`.
+- A range starting at or past the end (or `bytes=-0`) is `416` with
+  `Content-Range: bytes */size`.
+- Multiple ranges, another unit, or a malformed header are ignored and the
+  whole file is sent with `200` (RFC 9110 allows this). So is a request whose
+  `If-Range` does not match the current `ETag`.
+- Streamed responses advertise `Accept-Ranges: bytes`. Auth, `ETag`,
+  `Content-Disposition`, `Cache-Control` and CSP headers are the same as for
+  the full response.
+
+Redirected (presigned) downloads are served by the object store, which handles
+`Range` itself. Always redirecting large downloads to a presigned URL, as an
+opt-in setting, is not implemented.
+
 ### View
 
 | Method + path | Inertia component | Permission |
@@ -229,12 +252,19 @@ class GcsBackend(StorageBackend):
     supports_presigned_url = True
 
     async def put(self, key, stream, content_type, size): ...
-    async def get(self, key): ...
+    async def get(self, key): ...  # awaited; returns an AsyncIterator[bytes]
+    async def get_range(self, key, start, end): ...  # end inclusive
     async def delete(self, key): ...
     async def exists(self, key) -> bool: ...
     async def presigned_get_url(self, key, ttl_seconds: int) -> str: ...
 ```
 
+`get` and `get_range` must stream in bounded chunks (never buffer the whole
+object) and raise `StorageNotFoundError` from the awaited call, before the
+first byte. `get_range(key, start, end)` uses HTTP semantics — `end` is
+inclusive — and receives an already-validated `0 <= start <= end < size`. A
+backend without `get_range` still works: the endpoints then ignore `Range`.
+
 Register it in your module's `register_settings` so the file_storage service can pick it up. (The shipped two backends are wired into a private `BACKEND_REGISTRY`; see `backends/__init__.py` for the pattern.)
 
 ## Permissions