Repository navigation
Feat/receive analysis through message queue - #673
Merged
Merged
Conversation
ThanKarab
force-pushed
the
feat/receive_analysis_through_message_queue
branch
3 times, most recently
from
November 13, 2026 16:19
5178949 to
66ecc47
Compare
KFilippopolitis
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed at 5178949. The structure is good: ack only after a confirmed mandatory reply, shutdown leaves requests unacked, credentials stay in the Secret and out of logs, and the context_id proto fields are appended so the wire format stays compatible. CI is green on this head.
Two things I would fix before this leaves draft, both in the consumer:
- A single failed reply publish tears down the shared channel and re-runs analyses. Details inline at
consumer.py:249. - A RabbitMQ backlog locks HTTP out of
/analysis. Details inline atanalysis_execution.py:123.
Smaller points are inline. Two that have no good line to hang on:
- Shutdown budget has no margin. Worst case is 5 s (iterator close) + 30 s (drain) + 10 s + 5 s (existing shutdown) + 5 s + 5 s (consumer close) = 60 s, which equals
SHUTDOWN_TIMEOUT. Either lower theshutdown_drain_timeoutcap or raise the budget. - Missing tests for the paths above. There is no test for a broker nack (
DeliveryError), for a publish failure in one handler while another delivery is in flight on the same channel, or for a redelivery that arrives after the execution finished.set_qosis mocked but the prefetch value is never asserted, and no gRPC test sends an emptycontext_id.tests/standalone_tests/controller/rabbitmq/also has no__init__.py, unlike its sibling test packages.
On the open questions: I agree with both recommendations. Keep RabbitMQ disabled by default, and keep a broker outage from blocking HTTP while a missing or invalid URL fails startup.
Nits, take or leave:
MAX_REQUEST_BYTESinconsumer.pyduplicatesMAX_REQUEST_BODY_BYTESinrequest_body_limit.py. Import it so the two transports cannot drift.run_analysishas two branches that differ only in ordering._validateand the strategy call could share oneasync with analysis_slot(wait_for_capacity=...).- In
aggregation_server/server.py,_evict_stale_contexts,_remember_cancelledandCANCELLED_REQUEST_ID_LIMITstill sayrequest_idbut now hold context ids._get_aggregation_contextrequires a non-emptyrequest_idand then never compares it with the stored one. - With a fresh
context_idper attempt,ALREADY_CONFIGUREDcan no longer be produced by the controller. Worth one sentence in the docs that it is now defence in depth.
ThanKarab
force-pushed
the
feat/receive_analysis_through_message_queue
branch
2 times, most recently
from
November 16, 2026 11:21
9540a5c to
8967983
Compare
ThanKarab
marked this pull request as ready for review
November 16, 2026 11:39
ThanKarab
force-pushed
the
feat/receive_analysis_through_message_queue
branch
from
November 16, 2026 12:31
8967983 to
6027bb8
Compare
KFilippopolitis
approved these changes
Nov 17, 2026
KFilippopolitis
left a comment
Collaborator
There was a problem hiding this comment.
Good job all feedback is addressed!
ThanKarab
force-pushed
the
feat/receive_analysis_through_message_queue
branch
from
November 17, 2026 08:45
6027bb8 to
a520102
Compare
ThanKarab
force-pushed
the
feat/receive_analysis_through_message_queue
branch
from
November 17, 2026 09:24
a520102 to
712ec62
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add RabbitMQ analysis submission alongside HTTP
Adds RabbitMQ as a second analysis entry point. HTTP and RabbitMQ can run simultaneously and use the same analysis service, validation, execution strategies, and concurrency limit.
Changes
/healthcheck/rabbitmqreporting.request_idfor tracing and reply correlation. Use a freshcontext_idper execution attempt to isolate aggregation state, including worker operations and cleanup.Delivery is at least once: interrupted requests may execute again, and clients must handle duplicate replies.
Compatibility and rollout
HTTP request/response contracts remain unchanged.
Internal aggregation RPCs and UDF system arguments now require
context_id. Controller, workers, and aggregation server must be upgraded together, even when RabbitMQ is disabled.Disabling the consumer stops queue consumption while preserving HTTP operation. Reverting the internal protocol requires reverting all three services together.
Validation
Open questions
1. Should RabbitMQ be enabled by default?
Currently disabled globally and enabled in production test configuration.
Recommendation: keep it disabled by default, with deployments opting in when a broker is provisioned. Enabling RabbitMQ leaves HTTP enabled.
2. Should an unavailable RabbitMQ instance prevent the controller from running?
Currently, a configured but unreachable broker does not block HTTP. The consumer retries, and
/healthcheck/rabbitmqreturns503.Recommendation: retain this behavior so broker outages do not interrupt HTTP analyses.
A missing or invalid broker URL is treated separately: when RabbitMQ is enabled, configuration validation currently fails controller startup. Keep this fail-fast behavior for configuration errors, while tolerating broker connectivity outages.
3. Should RabbitMQ be part of exaflow k8s deployment or mip?