Skip to content

Commit 83ae357

Browse files
antosubashclaude
antosubash
and
claude
authored
Add BackgroundTasks module with Celery + Redis queue and admin UI (#36)
* feat(background_tasks): add Celery + Redis task queue with admin UI Adds a new background_tasks module: Celery app with Redis broker/result backend, a TaskExecution table populated via Celery signals, an admin API + Inertia UI that lists executions with filters, and a retry action gated to failed/stuck rows. Other modules declare tasks by shipping a tasks.py — autodiscovery picks them up without framework changes. Infra: dedicated docker/worker.Dockerfile for worker + beat services, redis service added to docker-compose, .dockerignore, Makefile targets for local and containerized workers. https://claude.ai/code/session_013VMtYvyhYDHEDHGKWAQqPf * refactor(background_tasks): simplify after code review Code-quality and efficiency pass on the Celery module: - reuse framework's ENTRY_POINT_GROUP in celery_app - drop dead constants (CELERY_STATE_*, RUNNING_STATUSES) - drop dead 503 branch in service.retry (celery is always set by on_startup) - merge list+count into one window-function query - purge_old_executions now issues one DELETE instead of SELECT-then-IN - dispose the sync engine on shutdown so lifespan restarts don't leak pools - collapse six signal-handler try/except bodies via a shared _apply helper - extract shared retry flow (fetch + toast) and RetryConfirmDialog so Index.tsx and Detail.tsx stop duplicating the same ~20 lines each https://claude.ai/code/session_013VMtYvyhYDHEDHGKWAQqPf --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 5d4a1cb commit 83ae357

46 files changed

Lines changed: 2972 additions & 3 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Python / uv
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
.venv/
6+
.uv/
7+
.pytest_cache/
8+
.ruff_cache/
9+
.mypy_cache/
10+
.ty_cache/
11+
*.egg-info/
12+
dist/
13+
build/
14+
15+
# Node / Vite
16+
node_modules/
17+
host/client_app/dist/
18+
packages/*/dist/
19+
.npm/
20+
21+
# Editors / OS
22+
.vscode/
23+
.idea/
24+
.DS_Store
25+
26+
# Secrets / local data
27+
.env
28+
*.db
29+
*.sqlite
30+
.memray/
31+
32+
# Tests / docs not needed at runtime
33+
tests/e2e/
34+
docs/

‎.env.example‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,13 @@ SM_USERS_COOKIE_SECURE=false
4646
# Multi-tenancy (default off).
4747
# SM_MULTI_TENANT=false
4848
# SM_TENANT_HEADER=
49+
50+
# BackgroundTasks module — Celery + Redis task queue.
51+
# In docker-compose these point at the `redis` service host (redis://redis:6379/*).
52+
SM_BG_TASKS_BROKER_URL=redis://localhost:6379/0
53+
SM_BG_TASKS_RESULT_BACKEND=redis://localhost:6379/1
54+
# How long a task can stay in `running` without a heartbeat before it's
55+
# flipped to `stuck` by the beat sweep.
56+
SM_BG_TASKS_STUCK_AFTER_SECONDS=300
57+
# Purge terminal rows older than this (days) — keeps history bounded.
58+
SM_BG_TASKS_RETENTION_DAYS=14

‎Makefile‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: install install-py install-js dev dev-api dev-ui build test test-py test-js test-e2e bench memray-run memray-flamegraph loadtest loadtest-memray lint doctor migrate migration downgrade migration-history docker-up docker-down kill new-module gen-pages sync-module-deps ci-python-lint ci-python-typecheck ci-js-lint ci-js-typecheck ci-check-file-size
1+
.PHONY: install install-py install-js dev dev-api dev-ui build test test-py test-js test-e2e bench memray-run memray-flamegraph loadtest loadtest-memray lint doctor migrate migration downgrade migration-history docker-up docker-down kill new-module gen-pages sync-module-deps ci-python-lint ci-python-typecheck ci-js-lint ci-js-typecheck ci-check-file-size worker beat worker-docker
22

33
# Install
44
install:
@@ -143,7 +143,18 @@ kill:
143143

144144
# Docker
145145
docker-up:
146-
docker compose up -d
146+
docker compose up -d postgres redis
147147

148148
docker-down:
149149
docker compose down
150+
151+
# Celery — local dev (fast reload, no container rebuild)
152+
worker: ## Run a Celery worker locally against $(SM_BG_TASKS_BROKER_URL)
153+
uv run celery -A scripts.run_worker:celery worker -l info
154+
155+
beat: ## Run the Celery beat scheduler locally
156+
uv run celery -A scripts.run_worker:celery beat -l info
157+
158+
# Celery — containerized (matches prod image)
159+
worker-docker: ## Build + run the worker + beat services in docker
160+
docker compose up --build worker beat

‎docker-compose.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,73 @@ services:
99
- "5432:5432"
1010
volumes:
1111
- pgdata:/var/lib/postgresql/data
12+
healthcheck:
13+
test: ["CMD-SHELL", "pg_isready -U sm -d simple_module"]
14+
interval: 5s
15+
timeout: 5s
16+
retries: 10
17+
18+
redis:
19+
image: redis:7-alpine
20+
ports:
21+
- "6379:6379"
22+
volumes:
23+
- redisdata:/data
24+
healthcheck:
25+
test: ["CMD", "redis-cli", "ping"]
26+
interval: 5s
27+
timeout: 3s
28+
retries: 10
29+
30+
worker:
31+
build:
32+
context: .
33+
dockerfile: docker/worker.Dockerfile
34+
env_file: .env
35+
environment:
36+
SM_BG_TASKS_BROKER_URL: redis://redis:6379/0
37+
SM_BG_TASKS_RESULT_BACKEND: redis://redis:6379/1
38+
SM_DATABASE_URL: ${SM_DATABASE_URL:-postgresql+asyncpg://sm:sm@postgres:5432/simple_module}
39+
depends_on:
40+
redis:
41+
condition: service_healthy
42+
postgres:
43+
condition: service_healthy
44+
command:
45+
- "uv"
46+
- "run"
47+
- "celery"
48+
- "-A"
49+
- "scripts.run_worker:celery"
50+
- "worker"
51+
- "-l"
52+
- "info"
53+
- "--concurrency=4"
54+
55+
beat:
56+
build:
57+
context: .
58+
dockerfile: docker/worker.Dockerfile
59+
env_file: .env
60+
environment:
61+
SM_BG_TASKS_BROKER_URL: redis://redis:6379/0
62+
SM_BG_TASKS_RESULT_BACKEND: redis://redis:6379/1
63+
SM_DATABASE_URL: ${SM_DATABASE_URL:-postgresql+asyncpg://sm:sm@postgres:5432/simple_module}
64+
depends_on:
65+
redis:
66+
condition: service_healthy
67+
worker:
68+
condition: service_started
69+
command:
70+
- "uv"
71+
- "run"
72+
- "celery"
73+
- "-A"
74+
- "scripts.run_worker:celery"
75+
- "beat"
76+
- "-l"
77+
- "info"
1278

1379
volumes:
1480
pgdata:
81+
redisdata:

‎docker/worker.Dockerfile‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Celery worker image for the BackgroundTasks module.
2+
#
3+
# Serves both the ``worker`` and ``beat`` services in docker-compose — they
4+
# differ only by command. A separate image (rather than reusing a web one)
5+
# keeps the runtime surface narrow: no Vite, no static assets, no hot-reload
6+
# tooling. Everything needed is ``uv sync`` + the project source.
7+
8+
FROM python:3.12-slim AS base
9+
10+
ENV PYTHONDONTWRITEBYTECODE=1 \
11+
PYTHONUNBUFFERED=1 \
12+
UV_LINK_MODE=copy \
13+
UV_COMPILE_BYTECODE=1 \
14+
UV_SYSTEM_PYTHON=1
15+
16+
RUN apt-get update \
17+
&& apt-get install -y --no-install-recommends \
18+
curl \
19+
ca-certificates \
20+
build-essential \
21+
&& rm -rf /var/lib/apt/lists/* \
22+
&& pip install --no-cache-dir uv
23+
24+
WORKDIR /app
25+
26+
# Dependency-only layer so code changes don't bust the wheel cache.
27+
COPY pyproject.toml uv.lock ./
28+
COPY framework/ framework/
29+
COPY modules/ modules/
30+
COPY host/ host/
31+
COPY scripts/ scripts/
32+
33+
# Install the full workspace (all framework + module packages) without dev
34+
# extras. Production workers never need pytest / ruff / ty.
35+
RUN uv sync --all-packages --frozen --no-dev
36+
37+
# Drop privileges. uv writes to /app/.venv so it needs write access until
38+
# sync completes; chown after install to keep the build step rootful.
39+
RUN useradd --system --uid 10001 --home /app --shell /usr/sbin/nologin worker \
40+
&& chown -R worker:worker /app
41+
USER worker
42+
43+
# Container-native healthcheck: ask the worker to ping itself via Celery's
44+
# control protocol. Fails fast if the worker has crashed but the PID lingers.
45+
ENV CELERY_APP=scripts.run_worker:celery
46+
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
47+
CMD uv run celery -A $CELERY_APP inspect ping -d celery@$HOSTNAME || exit 1
48+
49+
# Default command is the worker; docker-compose overrides to `beat` for the
50+
# scheduler service.
51+
CMD ["uv", "run", "celery", "-A", "scripts.run_worker:celery", "worker", "-l", "info"]

‎host/migrations/versions/6df7645cc4d2_add_background_tasks_task_execution.py‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
"""add background_tasks_task_execution
2+
3+
Revision ID: 6df7645cc4d2
4+
Revises: b7e1af4c9d02
5+
Create Date: 2026-04-19 07:44:06.824688
6+
"""
7+
8+
from collections.abc import Sequence
9+
10+
import sqlalchemy as sa
11+
from alembic import op
12+
13+
# revision identifiers, used by Alembic.
14+
revision: str = "6df7645cc4d2"
15+
down_revision: str | None = "b7e1af4c9d02"
16+
branch_labels: str | Sequence[str] | None = None
17+
depends_on: str | Sequence[str] | None = None
18+
19+
20+
def upgrade() -> None:
21+
# ### commands auto generated by Alembic - please adjust! ###
22+
op.create_table(
23+
"background_tasks_task_execution",
24+
sa.Column(
25+
"created_at",
26+
sa.DateTime(timezone=True),
27+
server_default=sa.text("(CURRENT_TIMESTAMP)"),
28+
nullable=False,
29+
),
30+
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
31+
sa.Column("created_by", sa.String(length=255), nullable=True),
32+
sa.Column("updated_by", sa.String(length=255), nullable=True),
33+
sa.Column("id", sa.Uuid(), nullable=False),
34+
sa.Column("celery_task_id", sa.String(length=64), nullable=True),
35+
sa.Column("task_name", sa.String(length=255), nullable=False),
36+
sa.Column("status", sa.String(length=20), nullable=False),
37+
sa.Column("queue", sa.String(length=64), nullable=False),
38+
sa.Column("args", sa.JSON(), nullable=True),
39+
sa.Column("kwargs", sa.JSON(), nullable=True),
40+
sa.Column("result", sa.JSON(), nullable=True),
41+
sa.Column("traceback", sa.String(), nullable=True),
42+
sa.Column("exception_type", sa.String(length=255), nullable=True),
43+
sa.Column("worker", sa.String(length=255), nullable=True),
44+
sa.Column("retries", sa.Integer(), nullable=False),
45+
sa.Column("retried_from_id", sa.Uuid(), nullable=True),
46+
sa.Column("queued_at", sa.DateTime(), nullable=True),
47+
sa.Column("started_at", sa.DateTime(), nullable=True),
48+
sa.Column("finished_at", sa.DateTime(), nullable=True),
49+
sa.Column("heartbeat_at", sa.DateTime(), nullable=True),
50+
sa.ForeignKeyConstraint(
51+
["retried_from_id"],
52+
["background_tasks_task_execution.id"],
53+
name=op.f(
54+
"fk_background_tasks_task_execution_retried_from_id_background_tasks_task_execution"
55+
),
56+
),
57+
sa.PrimaryKeyConstraint("id", name=op.f("pk_background_tasks_task_execution")),
58+
)
59+
op.create_index(
60+
op.f("ix_background_tasks_task_execution_celery_task_id"),
61+
"background_tasks_task_execution",
62+
["celery_task_id"],
63+
unique=False,
64+
)
65+
op.create_index(
66+
op.f("ix_background_tasks_task_execution_retried_from_id"),
67+
"background_tasks_task_execution",
68+
["retried_from_id"],
69+
unique=False,
70+
)
71+
op.create_index(
72+
op.f("ix_background_tasks_task_execution_status"),
73+
"background_tasks_task_execution",
74+
["status"],
75+
unique=False,
76+
)
77+
op.create_index(
78+
"ix_background_tasks_task_execution_status_queued",
79+
"background_tasks_task_execution",
80+
["status", "queued_at"],
81+
unique=False,
82+
)
83+
op.create_index(
84+
op.f("ix_background_tasks_task_execution_task_name"),
85+
"background_tasks_task_execution",
86+
["task_name"],
87+
unique=False,
88+
)
89+
# ### end Alembic commands ###
90+
91+
92+
def downgrade() -> None:
93+
# ### commands auto generated by Alembic - please adjust! ###
94+
op.drop_index(
95+
op.f("ix_background_tasks_task_execution_task_name"),
96+
table_name="background_tasks_task_execution",
97+
)
98+
op.drop_index(
99+
"ix_background_tasks_task_execution_status_queued",
100+
table_name="background_tasks_task_execution",
101+
)
102+
op.drop_index(
103+
op.f("ix_background_tasks_task_execution_status"),
104+
table_name="background_tasks_task_execution",
105+
)
106+
op.drop_index(
107+
op.f("ix_background_tasks_task_execution_retried_from_id"),
108+
table_name="background_tasks_task_execution",
109+
)
110+
op.drop_index(
111+
op.f("ix_background_tasks_task_execution_celery_task_id"),
112+
table_name="background_tasks_task_execution",
113+
)
114+
op.drop_table("background_tasks_task_execution")
115+
# ### end Alembic commands ###

‎host/pyproject.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ dependencies = [
88
"auth",
99
"dashboard",
1010
"products",
11+
"background-tasks",
1112
"file-storage",
1213
"python-multipart>=0.0.6",
1314
]
@@ -17,4 +18,5 @@ simple-module-hosting = { workspace = true }
1718
auth = { workspace = true }
1819
products = { workspace = true }
1920
dashboard = { workspace = true }
21+
background-tasks = { workspace = true }
2022
file-storage = { workspace = true }

‎modules/background_tasks/background_tasks/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""BackgroundTasks module — Celery + Redis task queue with admin UI."""

0 commit comments

Comments
 (0)