Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .deployment.sample.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ server_port = 8080
[controller]
port = 5100

# Connect to an externally provisioned broker. Keep credentials out of version control.
[rabbitmq]
enabled = false
url = "${RABBITMQ_URL:-}"
request_queue = "exaflow.analysis.requests"
dead_letter_queue = "exaflow.analysis.dead"

[privacy]
minimum_row_count = 10
protect_local_data = false
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/prod_env_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,21 @@ jobs:
- name: Print Helm Templates
run: helm template kubernetes/

- name: Deploy test RabbitMQ
run: |
set +x
password=$(openssl rand -hex 24)
echo "::add-mask::$password"
# Pipe the Secret directly to kubectl; never persist credentials in artifacts.
kubectl create secret generic exaflow-rabbitmq \
--from-literal=username=exaflow_tests \
--from-literal=password="$password" \
--from-literal=url="amqp://exaflow_tests:$password@exaflow-rabbitmq:5672/" \
--dry-run=client -o yaml | kubectl apply -f -
echo "RABBITMQ_TEST_URL=amqp://exaflow_tests:$password@127.0.0.1:5679/" >> "$GITHUB_ENV"
kubectl apply -f tests/prod_env_tests/deployment_configs/rabbitmq.yaml
kubectl rollout status statefulset/exaflow-rabbitmq --timeout=180s

- name: Deploy Helm
run: helm install exaflow kubernetes/ --debug

Expand Down Expand Up @@ -257,6 +272,11 @@ jobs:
- name: Run production env tests
run: uv run --no-sync pytest tests/prod_env_tests --verbosity=4

- name: Run isolated RabbitMQ recovery test
env:
RABBITMQ_TEST_RECOVERY: "1"
run: uv run --no-sync pytest -q tests/prod_env_tests/test_rabbitmq_analysis.py::test_redelivery_survives_broker_restart_and_controller_replacement

- name: Pod restarts and probe failures
if: always()
run: |
Expand Down
20 changes: 19 additions & 1 deletion documentation/api-specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -414,12 +414,18 @@ Example:

| Field | Type | Meaning | Validation / UI Notes |
| --- | --- | --- | --- |
| `request_id` | `string \| null` | Optional client request id. | If omitted, the server generates one. |
| `request_id` | `string \| null` | Optional client request id. | Supplied strings must match `^[A-Za-z0-9_-]{1,128}$`. If omitted or null, the server generates one. |
| `inputdata` | `AnalysisInputDataDTO` | Data model, datasets, filter, and source variables loaded before preprocessing. | Required. |
| `preprocessing` | `AnalysisPreprocessingStepDTO[] \| null` | Ordered preprocessing step configuration. | Step names must exist in enabled preprocessing specs, and required preprocessing steps must be included. |
| `algorithm` | `AnalysisAlgorithmDTO` | Selected algorithm name, final algorithm `x`, `y`, and parameter values. | Required. |
| `flags` | `object \| null` | System flags. | Optional. |

**Request ID compatibility:** caller-provided IDs are restricted to 1–128 ASCII
letters, digits, underscores, or hyphens. Empty strings, whitespace, other
punctuation, Unicode, and control characters return HTTP `422` before analysis
orchestration. Accepted IDs are preserved exactly; they are not trimmed or
rewritten. Clients previously using IDs outside this format must update them.

### `AnalysisInputDataDTO`

| Field | Type | Meaning | Validation / UI Notes |
Expand Down Expand Up @@ -570,3 +576,15 @@ ______________________________________________________________________
- `exaflow/controller/http/error_handlers.py`
- Algorithm/preprocessing names:
- `exaflow/algorithms/specifications.py`

## Optional RabbitMQ transport

`POST /analysis` keeps its current body, response, and error behavior. An optional
RabbitMQ consumer accepts the same analysis JSON and publishes a correlated
success/failure envelope; see [RabbitMQ analysis submission](rabbitmq-analysis.md).

`GET /healthcheck/rabbitmq` reports `enabled`, `connected`, `consuming`,
`active_jobs`, and `last_error` (a safe error type or null). Its status is 503 when
enabled but unavailable, otherwise 200. It does not change `/livez` or worker
health checks. The aggregate HTTP/AMQP analysis limit is shared; only analysis
submissions acquire a slot.
25 changes: 25 additions & 0 deletions documentation/context/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,3 +209,28 @@ that caller configured it.
need verification.
- Some Flower and SMPC paths appear optional or partly disabled in CI; verify
before broad changes.

## Optional RabbitMQ analysis transport

FastAPI lifespan can own a bounded async AMQP consumer alongside HTTP. Both call
`analysis_service.run_analysis`; HTTP rejects excess analyses while AMQP waits
for the same capacity and validates after admission. Metadata and health requests
remain independent of analysis slots. The worker landscape refresher keeps its
background thread and analyses keep their dedicated executor.

A queue backlog can make HTTP analysis submissions return 503 until queued
waiters are admitted. Each delivery passively checks its reply destination and
publishes through its own channel on the existing broker connection. A rejected
reply dead-letters only that request. Transient publication retries reuse the
result; completed replies for unsettled deliveries have bounded in-memory
retention for same-controller redelivery.

Controller and worker request logging uses fixed service loggers with per-record
request IDs, rather than permanently registering a logger and handler for every
logical request. Worker request adapters retain their ContextVar isolation.

Logical `request_id` values survive retries; UUID `context_id` values identify
attempts and key all aggregation state. Both are carried to aggregation services
and logged. This changes internal RPC contracts and requires coordinated rollout.
See `documentation/rabbitmq-analysis.md` for delivery, lifecycle, configuration,
and rollback behavior.
12 changes: 12 additions & 0 deletions documentation/context/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,3 +288,15 @@ uv run python .agents/skills/exaflow-algorithm-validate/scripts/validate_algorit
EBRAINS integration path.
- `docker-bake.hcl` defines the multi-image build targets used by the publish
workflows.

## RabbitMQ

Enable the optional external-broker transport through the existing deployment
configuration and privately supplied `RABBITMQ_URL`. Configuration, Helm Secret
wiring, producer examples, and test-broker provisioning are documented in
`documentation/rabbitmq-analysis.md`.

```bash
uv run python examples/rabbitmq_analysis.py analysis-request.json
uv run pytest -q tests/standalone_tests/controller/rabbitmq
```
24 changes: 24 additions & 0 deletions documentation/context/decision-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,3 +213,27 @@ Files affected: `.agents/skills`, `documentation/new-algorithm-setup.md`,
algorithm wrapper/core/test/docs paths.

Date: 2026-05-22

## Separate logical requests from aggregation attempts

RabbitMQ redelivery may repeat a logical request. Preserve `request_id` for
tracing and use a fresh UUID `context_id` for each attempt, including all worker
aggregation operations and cleanup. Propagate both explicitly; aggregation logs
include both. This changes the aggregation RPC contract and requires coordinated
controller, worker, and aggregation-server upgrades. Regenerate protobuf bindings
using `exaflow/protos/README.md`.

## Isolate RabbitMQ replies and retain unsettled results briefly

Each delivery checks that its producer-created reply queue exists and publishes
through an isolated channel on the existing connection. Missing destinations and
broker nacks dead-letter only their request; transient failures retry the
serialized result. Completed replies can survive redelivery within one
controller for five minutes, bounded to 128 entries and 64 MiB. This remains
at-least-once execution without a durable result store. A sustained backlog can
reject HTTP analysis submissions under the intentionally shared capacity limit.

The broker is dedicated to Exaflow; no application queue-prefix allowlist or
request-body collision checks are added. RabbitMQ remains opt-in and externally
provisioned. The independent 30-second publication timeout is interruptible; the
20-second drain cap leaves ten seconds of lifespan shutdown headroom.
10 changes: 10 additions & 0 deletions documentation/context/module-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,3 +225,13 @@ Rules: Preserve dry-run behavior and JSON merge semantics.
Tests: No dedicated test found during inspection; validate manually if changed.

Notes: Can accept piped JSON and override fields with CLI args.

## RabbitMQ analysis submission

`exaflow/controller/services/rabbitmq` owns typed broker settings, bounded async
consumption, reply envelopes, manual acknowledgement, and reconnect/drain behavior.
FastAPI lifespan owns its runtime instance; the HTTP layer exposes independent
consumer health. Both transports call the shared analysis service, with AMQP
waiting for capacity before semantic validation. Production broker provisioning
remains external; test-only kind manifests live with production test deployment
configuration. See `documentation/rabbitmq-analysis.md`.
26 changes: 26 additions & 0 deletions documentation/context/risk-register.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,29 @@ persistence. Publishing workflows build without restoring or saving CI caches;
test caches remain enabled. Controller, worker, and staging aggregation images
retain separate publishing jobs so one failure does not skip other images. Publishing credentials and triggers are unchanged.
Rollback is a revert of these changes, restoring the corresponding exposures.

## RabbitMQ analysis transport

Broker persistence adds storage for analysis results and dead letters. Bound
message size, queue growth, and retention; provide credentials through existing
configuration/Secrets and keep URLs out of logs. AMQP is disabled by default and
normal deployments provide an external broker. HTTP and AMQP share one analysis
capacity limit; other HTTP routes remain independent.

A sustained queue backlog can reject HTTP analyses with 503. Reply failures
must remain isolated from the consuming channel. Finished replies retained for
redelivery are bounded to five minutes, 128 entries, and 64 MiB; eviction or a
controller restart still permits recomputation. Acknowledge only after confirmed
publication, and keep reply bodies and connection URLs out of diagnostics.

Delivery is at least once, with recomputation and duplicate replies allowed.
Preserve logical request IDs and isolate aggregation state using attempt context
IDs, including late calls, cancellation tombstones, and cleanup ownership checks.
The internal protocol change requires coordinated controller/worker/aggregation
rollout. Consumer tasks require explicit draining before existing shutdown cancels
RPCs; cancelling asyncio tasks alone does not stop executor threads.

Real-broker recovery tests may restart only their dedicated kind test broker and
replace the test controller. They require the explicit test recovery environment
flag and must never target shared deployments. See
`documentation/rabbitmq-analysis.md` for configuration, tests, and rollback.
41 changes: 41 additions & 0 deletions documentation/context/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,44 @@ CI also builds the package and smoke-tests the wheel.
| Dependency update | Lockfile update, focused tests, lint/format, and CI-equivalent suite where feasible. |
| Refactor without behavior change | Focused tests for touched modules and broader standalone tests if shared code moved. |
| `exadata-validator` change | `cd exadata-validator && uv run pytest -q`; build/smoke test if packaging or CLI changes. |

## RabbitMQ transport

Focused fake-AMQP tests live in `tests/standalone_tests/controller/rabbitmq`.
Controller concurrency and Uvicorn shutdown tests cover shared admission and queue
cancellation; aggregation gRPC tests cover independent execution contexts.
Algorithm validation stays on HTTP and requires no broker.

RabbitMQ configuration tests cover disabled-feature validation, independent
publication deadlines, and ten seconds of headroom in the controller's lifespan
shutdown budget. Helm rendering checks decimal integer environment values and
fractional timeout values; it requires Helm on `PATH`.

Consumer regressions cover missing destinations before execution, unroutable
replies, broker nacks, transient publish retries, failed settlement, completed
result retention and eviction, and concurrent reply-channel cleanup. Real broker
tests also exercise deleted destinations and a full reply queue while another
analysis succeeds. Shared logging tests check bounded logger/handler counts and
concurrent request context isolation in the controller and worker.

Shared DTO and intake tests enforce `^[A-Za-z0-9_-]{1,128}$` for caller-provided
analysis request IDs. HTTP rejects invalid body IDs with 422 before orchestration;
AMQP validates its correlation property before assigning it to the DTO or
consulting execution/result caches. Cover lengths 1/128/129, control characters,
spaces, punctuation, Unicode, nullable/omitted body IDs, and body/header conflicts.

Run controller/logging regressions without supplying `LOG_LEVEL` or
`FRAMEWORK_LOG_LEVEL`: the config loader defaults missing/empty values to `INFO`.
The full-reply-queue production regression checks the source queue's `x-death`
queue, reason, and count; a dead-letter delivery's `x-delivery-count` is not a
reliable measure of analysis executions.

Production tests provision `tests/prod_env_tests/deployment_configs/rabbitmq.yaml`
in kind. `RABBITMQ_TEST_URL` enables real-broker cases; the CI-only
`RABBITMQ_TEST_RECOVERY=1` flag permits restarting the dedicated test broker and
replacing its test controller. CI sets that flag only in a separate recovery
step after the normal production suite, and recovery cleanup waits for controller
rollout and transport readiness, including on failures. Broker access uses kind
host port 5679 mapped to NodePort 30001 instead of a restart-sensitive port-forward.
Queue and HTTP parity cases share the existing request builder, including required
missing-value preprocessing. See `documentation/rabbitmq-analysis.md`.
Loading
Loading