Repository navigation
feat: durable event delivery via an outbox (#459) - #485
Merged
Merged
Conversation
…ach pinned by a test - enqueue_after_commit marks a request session written, so a request whose only effect is the enqueue is not rolled back and its publish dropped - a retry whose publish fails after commit flips the pending row to failed instead of stranding it - stuck_after below one second no longer truncates to 0 (every running row instantly stuck) - a 416 response no longer carries Cache-Control Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…r module subset (#447, #448, #450) Adds task_config.py (TASK_ROUTES/QUEUES/BEAT_SCHEDULE), task_routes_override, a Routes table on the Workers page, tuning settings and celery_overrides, and worker_queues/include_modules/exclude_modules filtering for autodiscovery and the eager beat import. make worker passes -Q. The worker process now hydrates DB-backed settings at boot. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
Make celery_overrides env-only with a denylist of serializer/code-loading keys, restrict task_routes_override to queue names and safe route keys, give the override precedence over module routes via a router list, quote make -Q, and correct the env/DB precedence docs. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
task_routes_override now parses the JSON string the settings textbox sends and rejects invalid route mappings with a 422 instead of silently dropping them. Workers empty-state worker command lists every routed queue; the Routes table shows a translated empty state. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
A stored task_routes_override that no longer validates must not take the app or worker down or discard the module's other overrides: boot hydration is now lenient per field (log + ignore); settings writes stay strict (422). Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…ocol (#459) EventBus gains subscribe_durable(EventType, handler, name=...) and publish_durable(event, db). publish_durable hands the event to bus.durable_transport, which must write it in db's transaction; with no transport installed it raises DurableDeliveryUnavailable instead of silently falling back to in-process delivery. Handlers are called later as handler(event, DurableContext(db, attempt)). Events must be @DataClass classes that pydantic can serialise to JSON (UUID, datetime, Enum, Decimal, nested dataclasses), checked at subscribe time. publish/publish_nowait are unchanged. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…in page (#459) OutboxTransport (installed on the app's EventBus in on_startup) writes one background_tasks_outbox row per durable handler in the publisher's session and, after commit, sends background_tasks.dispatch_outbox(ids). A 30 s beat task, background_tasks.sweep_outbox, delivers anything that missed that hand-off and reclaims expired leases. Claiming is one conditional UPDATE (pending -> running, attempts + 1, lease), with FOR UPDATE SKIP LOCKED on Postgres, one row at a time just before its handler runs. The handler's ctx.db commits together with the delivered mark, which is fenced on the claimed attempt, so DB effects land exactly once even if a lease expires and the row is re-run. Failures back off 10 s * 2^(n-1), capped at 1 h, and go dead after outbox_max_attempts (8); unknown handlers or payloads that no longer validate go dead immediately. Handlers run in the publisher's tenant. The worker builds its own EventBus from every module's register_event_handlers(bus, None) and an async NullPool engine on the sync_db URL; Celery task bodies run the dispatcher with asyncio.run (on a private thread when a loop is already running). Eager mode delivers from on_commit on the app's engine. Delivered rows are purged with the execution history. New page /admin/background-tasks/outbox and /api/background_tasks/admin/outbox (list; retry dead; discard pending/dead). Migration e878882924e7. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…459) Outbox rows carry event payloads. A platform admin (global admin role) still sees every tenant; any other principal granted background_tasks.view/manage (a tenant role, via the role editor) now lists, counts, retries and discards only its active tenant's rows, and another tenant's or a platform row's id answers 404. With no tenant bound on a multi-tenant install it sees nothing. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
… outbox section Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…es (#459) `UPDATE ... WHERE id IN (SELECT ... LIMIT n FOR UPDATE SKIP LOCKED)` let Postgres rescan the candidate select per outer row, so a single claim took every due row regardless of its batch, and concurrent dispatchers ended up with nothing. Running the candidates as a MATERIALIZED CTE evaluates them once. The concurrency fixture now empties the shared Postgres schema first, so rows left by an earlier test can't be claimed into its counts. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
- H1: the worker-built bus carries WorkerOutboxTransport, so durable handlers can publish_durable through ctx.db (dispatched after that commit). - M1: handler_guard refuses commits of ctx.db while a handler runs (HandlerCommitError), keeping exactly-once DB effects behind the fence. - M2: unknown handler/event retries with backoff, dead only after max attempts; register_event_handlers failures logged once per worker. - M3: handler timeout = lease minus a safety margin (or the new outbox_handler_timeout_seconds, validated against the lease). - L1-L6: backoff from failure time; DB clock for all outbox timestamps; claim_seq fencing token independent of the attempts budget; eager delivery from non-request sessions; platform events under all_tenants(); pooled worker engine on one long-lived loop per process. - Docs: guarantees, idempotency, timeouts, deploy behaviour. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
Cancel the in-flight outbox delivery when the Celery task body is interrupted (soft time limit, revoke) so it cannot run beside the pool process's next task. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
…1, pass 4) - N1: the handler commit guard allows savepoint releases (begin_nested); a root commit with a savepoint open is still refused. - N2: commits through ctx.db's connection are refused too (engine commit listener scoped to connections the guarded session began on); the guard is lifted before the fenced delivery commit. - N3: one outbox loop and engine per worker thread (thread-local, pid-keyed), pool 1+2 per thread; docs: handlers must not block the event loop. - N4: a handler that swallows the timeout's cancellation is a timed-out attempt. - N5: record the publish scope (tenant/platform/unscoped) in the unreleased e878882924e7 migration; unscoped rows run with no tenant bound (new simple_module_db.no_tenant()), platform rows under all_tenants(). Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
- timeout-vs-lease check is a field validator, so the 422 carries the field loc - settings module form shows model-level (empty loc) errors as a form alert - outbox retry/discard failures use translated messages and always refresh - sweep_outbox beat ticks are no longer recorded unless they fail Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
… (300-line cap) Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying simple-module-python with
|
| Latest commit: |
2e7daff
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://da48e69f.simple-module-python.pages.dev |
| Branch Preview URL: | https://feat-durable-outbox-459.simple-module-python.pages.dev |
antosubash
changed the base branch from
feat/bg-tasks-celery-config-447-450
to
main
October 15, 2026 16:56
# Conflicts: # CHANGELOG.md # CLAUDE.md # docs/modules/background_tasks.md # docs/reference/diagnostic-codes.md # modules/background_tasks/README.md # skills/simple-module-doctor/SKILL.md
…table Autogenerate compared against the whole shared public schema, so tables left by earlier tests in a PG run (qfilter_*, qfrj_*) became DROP ops and failed on their FKs. Filter autogenerate to dbext_node. Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #481 — merge that first; this PR's base will then be retargeted to main.
Summary
Durable event delivery through a transactional outbox. A handler subscribed durably is written to the database in the publisher's own transaction and delivered after commit, so a crash between "commit" and "dispatch" no longer loses the event.
Core API (
simple_module_core):subscribe_durable/publish_durable, theDurableTransportprotocol,DurableContext,DurableDeliveryUnavailable, and JSON event serialization.background_tasksoutboxSKIP LOCKEDthrough a MATERIALIZED CTE on Postgres), withclaim_seqfencing and DB-clock leases.ctx.dbcommits fenced on the claim, with a commit guard (session, savepoint-aware, connection-level).simple_module_db.no_tenant()./admin/background-tasks/outbox(tenant-scoped, retry / discard), plus settings.sweep_outboxruns are no longer recorded.Guarantees and limits
ctx.dbare exactly-once. Side effects outsidectx.dbare at-least-once, so handlers must be idempotent.Review and audit
Adversarial delivery audits found H1, M1-M3, L1-L6 and N1-N5. All are fixed with regression tests. Also fixed: a tenant-isolation hole in the admin page, and a Postgres claim bug found during rebase (claims are now capped to a batch).
Spec deviations
locked_until,claim_seqandscopecolumns added to the outbox table.last_error.QA
Round 1: 5/5 (a P2 settings-form error display fixed, plus 2 P3). Round 2: 4/4.
Not verified in a browser: delivery to a real handler, because no shipped module subscribes durably yet. That path is covered by automated tests on SQLite and Postgres.
Verification (this run)
make lint: passmake test-js: passmake test-py: 4275 passed, 12 skipped, 1 xfailed. 2framework/clivite-build tests failed becausenode_moduleswas broken mid-run (pnpm pollution). After a cleannpm installthey pass (2 passed).modules/background_tasks framework/core framework/db): 1136 passed, 8 skipped, 1 xfailedmake migrations-roundtrip-pg: upgrade / downgrade / upgrade /alembic checkcleanTest plan
/admin/background-tasks/outboxand confirms rows list, retry and discard work and are tenant-scopedCloses #459
https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4