fix: S3 streaming + Range, task heartbeat + after-commit enqueue, token rehash, scaffold fixes, QA follow-ups (#449–#472)

[antosubash]

Six bug fixes. Four of them (#451, #452, #453, #457) come from the gis-data-portal port assessment (tracking issue #460).

Changes

• file_storage: S3 get() buffers the whole object in memory; no ranged reads #453: file_storage streaming and Range.
  • S3Backend now streams objects in 64 KiB reads instead of buffering the whole object. The body is closed on completion, on error and on client disconnect.
  • New get_range(key, start, end) on the backend contract (inclusive end), implemented for S3 and filesystem.
  • Download endpoints answer Range with 206. They return 416 bytes */size when the range is unsatisfiable, honour If-Range, and fall back to a full 200 for multi-range or malformed headers.
  • A third-party backend without get_range keeps serving full 200s.
  • Found along the way: S3Backend.get was an async generator, but callers await it, so S3 thumbnails and the streamed public path raised TypeError. It is now awaited like the filesystem backend. A missing object returns 404 (was 500).
  • Out of scope: the optional presigned-redirect setting.
• background_tasks: heartbeat is only stamped at start/end, so any task running >5 min is flagged stuck #451: long tasks flagged stuck.
  • Task bodies can call from background_tasks import heartbeat; heartbeat(). Writes are throttled to one per 30 s. It is a no-op outside a task and never revives a row the sweep already marked stuck.
  • @shared_task(..., stuck_after=<seconds>) sets a per-task threshold. The sweep reads it from the task registry, so no migration is needed.
  • Not done: an automatic heartbeat thread.
• background_tasks: retry publishes to the broker before the TaskExecution row commits #452: retry published before the commit.
  • Single and bulk retries now publish only after the new TaskExecution rows commit. The bulk path publishes in asyncio.to_thread.
  • New public helper enqueue_after_commit(db, task, *args, **kwargs). It returns the reserved task id and works on request sessions (on_commit) and on plain or worker sessions (SQLAlchemy after_commit / rollback events).
• users: /auth/token login discards the upgraded password hash #457: token login dropped the hash upgrade. POST /api/users/auth/token now saves the upgraded password hash, as the browser login already did. The users docs list the hash formats accepted on import: Argon2, and bcrypt $2a$/$2b$/$2y$, which is rehashed on the next sign-in.
• i18n: keys.generated.ts never regenerates in scaffolded apps — app modules' t(keys.<ns>…) can't typecheck #449: i18n keys in scaffolded apps. In dev, when packages/i18n/src is absent, the host writes keys.generated.ts into the installed @simple-module-py/i18n package. It searches up the tree for node_modules (hoisted or the flat client_app/ layout) and keeps the keys the package already ships. Adds smpy host gen-i18n (strict) and a scaffold make gen-i18n, which make install runs.
• cli: scaffolded make kill pkills every uvicorn/vite on the machine (other projects included) #461: make kill killed other apps' servers. The scaffold make kill now only stops this app's ports: API_PORT ?= 8000, and VITE_PORT read from SM_VITE_DEV_URL in .env (default 5050). The flat host Makefile gains the same target.

Verification

• make lint: green.
• make test-py: 4041 passed, 11 skipped, 1 xfailed.
• make test-js: green.
• New tests:
  • range parsing and 206/416 downloads;
  • S3 chunked reads, checked by spying StreamingBody.read;
  • after-commit publishing on rollback and on a failed commit, through the real retry endpoints;
  • heartbeat and per-task thresholds;
  • legacy bcrypt hash upgraded through /auth/token;
  • i18n target resolution;
  • scaffold Makefile has no pkill.
• make -n kill on a real scaffold resolves the ports from API_PORT and .env.
• Not verified: a real broker/worker run for background_tasks: heartbeat is only stamped at start/end, so any task running >5 min is flagged stuck #451/background_tasks: retry publishes to the broker before the TaskExecution row commits #452 (eager and mocked Celery only), a real npm installed scaffold for i18n: keys.generated.ts never regenerates in scaffolded apps — app modules' t(keys.<ns>…) can't typecheck #449 (unit tests only), and the Postgres suite.

Ship pipeline (optimize, review, QA)

• Optimize: 20 lines removed net in 2 behaviour-preserving commits. These remove duplicated code, drop dead guards and leave a single ETag source.
• Review: clean after 3 passes. Pass 1 found that a ranged /download still opened the full object before fetching the range, which on S3 meant an extra full get_object per range read. FileStorageService.download now takes open_body=False, and a test proves a Range request never calls backend.get.
• QA (browser and HTTP): 47 passed. No bugs in this PR's scope.
  • Covered: Range 206/416, If-Range, a malformed or multi-range header falls back to 200, an object missing on disk returns 404 (not 500), byte-identical 0-byte and 20 MB files, public 304, token login, and the retry endpoints.
  • 206 responses are never gzipped.
• Verification on 3c051f96: make lint passed, make test-py 4042 passed, make test-js passed, and the e2e suite 49 passed.
• Pre-existing issues found by QA, to be filed separately: rapid confirm clicks on a single Retry queue duplicates, and created_at has no timezone on SQLite.
• CI (.github/workflows/pr.yml): the e2e job now sets SM_RATE_LIMIT_PUBLIC=1000/minute. The suite signs in 49 times from one IP, about 120 anonymous requests a minute, which is right at the default limit. A fast runner returned a 429 login page to whichever test signed in last; this failed twice in a row on this branch. Nothing in this PR touches those routes.
• Pipeline report and QA report. These are private Claude Artifacts; the author can share them.

QA follow-ups fixed here (#467–#472)

• background_tasks: rapid confirm clicks on single Retry queue duplicate retries (and 500 on SQLite) #467: background-task Retry now sends one request per execution while one is in flight, so a burst of confirm clicks no longer queues duplicate retries. Not changed: under truly concurrent writes, SQLite can still raise database is locked. The cause is that the publish signal does a synchronous DB write on the event loop. Fixing that means moving Celery's publish hook off the loop, which is a design change, and Postgres is unaffected.
• API timestamps lack a timezone on SQLite: fresh uploads show "2h ago" #468: new simple_module_db.AwareDateTime replaces DateTime(timezone=True) in the audit/soft-delete mixins and in the audit_log, users, permissions and tenants models. These columns read back timezone-aware on SQLite, so the API sends an offset. Migrations still render sa.DateTime(timezone=True), so there is no schema change.
• background_tasks: empty state claims executions exist outside the search when there are none #469: the background-tasks list shows its plain empty state when no executions exist (new has_executions prop).
• file_storage: deleting an already-deleted file from a stale tab toasts "0 files deleted" #470: deleting files that another tab already deleted now says so (or "X of Y deleted"), instead of "0 files deleted".
• smpy: workspace hint prints a bare relative path that smpy add parses as a PyPI name #471: the smpy workspace hint prints ./modules/..., which smpy add reads as a path. env_values also ignores indented comments.
• scaffold compose: POSTGRES_PASSWORD is inserted unencoded into SM_DATABASE_URL #472: make docker-up rejects a non-URL-safe POSTGRES_PASSWORD with a clear message, and the scaffolded .env.example and the docs say so.

Second /ship pass over these fixes:

• Optimize: dropped an unreachable guard (2 lines).
• Review: clean after 2 passes. Pass 1 caught that the make docker-up password guard rejected valid .env files with CRLF line endings, quoted values or inline comments, and checked the first definition where compose uses the last. It now reads the value the way compose does, covered by a 10-case test that runs make docker-up against a stubbed docker.
• Browser QA: 9 of 9 passed.
  • background_tasks: rapid confirm clicks on single Retry queue duplicate retries (and 500 on SQLite) #467: 5 rapid confirm clicks send 1 POST, and a deliberate second retry still works.
  • background_tasks: empty state claims executions exist outside the search when there are none #469: the empty-state copy is correct.
  • file_storage: deleting an already-deleted file from a stale tab toasts "0 files deleted" #470: the stale-tab and partial delete messages are correct.
  • API timestamps lack a timezone on SQLite: fresh uploads show "2h ago" #468: a new upload shows "just now", and the API timestamp ends in Z.
• Verification on f16f1699: make lint passed, make test-py 4063 passed, make test-js passed, and the e2e suite 49 passed.

Closes #449
Closes #451
Closes #452
Closes #453
Closes #457
Closes #461
Closes #467
Closes #468
Closes #469
Closes #470
Closes #471
Closes #472

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	✅ Completed 2026-10-13T15:31:45.681441Z	90e69f1	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cloudflare-workers-and-pages]

Deploying simple-module-python with    Cloudflare Pages

Latest commit:	f16f169
Status:	✅  Deploy successful!
Preview URL:	https://b978992f.simple-module-python.pages.dev
Branch Preview URL:	https://fix-issue-batch-449-461.simple-module-python.pages.dev

View logs