Skip to content

Feat/receive analysis through message queue - #673

Merged
ThanKarab merged 6 commits into
masterfrom
feat/receive_analysis_through_message_queue
Nov 17, 2026
Merged

ThanKarab merged 6 commits into
masterfrom
feat/receive_analysis_through_message_queue

Conversation

@ThanKarab

@ThanKarab ThanKarab commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Add RabbitMQ analysis submission alongside HTTP

Adds RabbitMQ as a second analysis entry point. HTTP and RabbitMQ can run simultaneously and use the same analysis service, validation, execution strategies, and concurrency limit.

Changes

  • Add an async RabbitMQ consumer managed by FastAPI lifespan, with bounded consumption, reconnect backoff, graceful draining, and independent /healthcheck/rabbitmq reporting.
  • Accept the existing analysis request JSON and publish correlated success/failure replies. Acknowledge requests after confirmed reply publication; support dead-letter handling and message-size/retention limits.
  • Preserve request_id for tracing and reply correlation. Use a fresh context_id per execution attempt to isolate aggregation state, including worker operations and cleanup.
  • Add local configuration, Helm Secret wiring, documentation, and a producer example.
  • Add mocked AMQP standalone tests and a real RabbitMQ deployment for production environment tests. Algorithm validation continues through HTTP.

Delivery is at least once: interrupted requests may execute again, and clients must handle duplicate replies.

Compatibility and rollout

HTTP request/response contracts remain unchanged.

Internal aggregation RPCs and UDF system arguments now require context_id. Controller, workers, and aggregation server must be upgraded together, even when RabbitMQ is disabled.

Disabling the consumer stops queue consumption while preserving HTTP operation. Reverting the internal protocol requires reverting all three services together.

Validation

  • Focused standalone regression: 539 passed, 5 skipped.
  • Helm rendering, TOML/YAML parsing, lockfile checks, and commit hooks passed.
  • Six real-broker production tests added and collected.
  • Environment-backed algorithm validation and real RabbitMQ/kind tests were not run locally; the required deployment tools/services are unavailable.

Open questions

1. Should RabbitMQ be enabled by default?

Currently disabled globally and enabled in production test configuration.

Recommendation: keep it disabled by default, with deployments opting in when a broker is provisioned. Enabling RabbitMQ leaves HTTP enabled.

2. Should an unavailable RabbitMQ instance prevent the controller from running?

Currently, a configured but unreachable broker does not block HTTP. The consumer retries, and /healthcheck/rabbitmq returns 503.

Recommendation: retain this behavior so broker outages do not interrupt HTTP analyses.

A missing or invalid broker URL is treated separately: when RabbitMQ is enabled, configuration validation currently fails controller startup. Keep this fail-fast behavior for configuration errors, while tolerating broker connectivity outages.

3. Should RabbitMQ be part of exaflow k8s deployment or mip?

@ThanKarab
ThanKarab force-pushed the feat/receive_analysis_through_message_queue branch 3 times, most recently from 5178949 to 66ecc47 Compare November 13, 2026 16:19

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 5178949. The structure is good: ack only after a confirmed mandatory reply, shutdown leaves requests unacked, credentials stay in the Secret and out of logs, and the context_id proto fields are appended so the wire format stays compatible. CI is green on this head.

Two things I would fix before this leaves draft, both in the consumer:

  1. A single failed reply publish tears down the shared channel and re-runs analyses. Details inline at consumer.py:249.
  2. A RabbitMQ backlog locks HTTP out of /analysis. Details inline at analysis_execution.py:123.

Smaller points are inline. Two that have no good line to hang on:

  • Shutdown budget has no margin. Worst case is 5 s (iterator close) + 30 s (drain) + 10 s + 5 s (existing shutdown) + 5 s + 5 s (consumer close) = 60 s, which equals SHUTDOWN_TIMEOUT. Either lower the shutdown_drain_timeout cap or raise the budget.
  • Missing tests for the paths above. There is no test for a broker nack (DeliveryError), for a publish failure in one handler while another delivery is in flight on the same channel, or for a redelivery that arrives after the execution finished. set_qos is mocked but the prefetch value is never asserted, and no gRPC test sends an empty context_id. tests/standalone_tests/controller/rabbitmq/ also has no __init__.py, unlike its sibling test packages.

On the open questions: I agree with both recommendations. Keep RabbitMQ disabled by default, and keep a broker outage from blocking HTTP while a missing or invalid URL fails startup.

Nits, take or leave:

  • MAX_REQUEST_BYTES in consumer.py duplicates MAX_REQUEST_BODY_BYTES in request_body_limit.py. Import it so the two transports cannot drift.
  • run_analysis has two branches that differ only in ordering. _validate and the strategy call could share one async with analysis_slot(wait_for_capacity=...).
  • In aggregation_server/server.py, _evict_stale_contexts, _remember_cancelled and CANCELLED_REQUEST_ID_LIMIT still say request_id but now hold context ids. _get_aggregation_context requires a non-empty request_id and then never compares it with the stored one.
  • With a fresh context_id per attempt, ALREADY_CONFIGURED can no longer be produced by the controller. Worth one sentence in the docs that it is now defence in depth.

@ThanKarab
ThanKarab force-pushed the feat/receive_analysis_through_message_queue branch 2 times, most recently from 9540a5c to 8967983 Compare November 16, 2026 11:21
@ThanKarab
ThanKarab marked this pull request as ready for review November 16, 2026 11:39
@ThanKarab
ThanKarab force-pushed the feat/receive_analysis_through_message_queue branch from 8967983 to 6027bb8 Compare November 16, 2026 12:31

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job all feedback is addressed!

@ThanKarab
ThanKarab force-pushed the feat/receive_analysis_through_message_queue branch from 6027bb8 to a520102 Compare November 17, 2026 08:45
@ThanKarab
ThanKarab force-pushed the feat/receive_analysis_through_message_queue branch from a520102 to 712ec62 Compare November 17, 2026 09:24
@ThanKarab
ThanKarab merged commit 716c29d into master Nov 17, 2026
5 checks passed
@ThanKarab
ThanKarab deleted the feat/receive_analysis_through_message_queue branch November 17, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants