Repository navigation
Conversation
…up at startup Add characterization tests for start_executor_process ahead of splitting start_executor_process_inner into phases (apache#2597): the single connect attempt with scheduler_connect_timeout_seconds = 0, the memory pool size reported to ExecutorMemoryMetrics for a configured and an unbounded pool, and the shuffle clean-up: with job_data_clean_up_interval_seconds = 0 neither the periodic cleaner nor the clean-up at shutdown runs, and with a positive interval job data is removed on shutdown. The last two run a pull-based executor against an idle in-process scheduler and shut it down by taking its Flight port. Co-authored-by: Megabot <megabot@megabot.com.br>
…s_inner Part of apache#2597. Moves the vcores resolution verbatim; behaviour is unchanged. Co-authored-by: Megabot <megabot@megabot.com.br>
…_process_inner Part of apache#2597. Moves the periodic shuffle cleaner task verbatim; behaviour is unchanged. Adds a test that it removes expired job dirs and stops on shutdown. Co-authored-by: Megabot <megabot@megabot.com.br>
…ocess_inner Part of apache#2597. Moves the service/signal/stop-request select verbatim; behaviour is unchanged. Adds tests for the stop-request and failed-service outcomes. Co-authored-by: Megabot <megabot@megabot.com.br>
…cutor_process_inner Part of apache#2597. Moves the memory pool resolution and its metrics wiring verbatim; behaviour is unchanged. The metrics handle is passed as Option<&ExecutorMemoryMetrics>. Covered by the memory pool metrics tests. Co-authored-by: Megabot <megabot@megabot.com.br>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #2597 (first of three PRs; it does not close the issue).
Rationale for this change
start_executor_process_innerinballista/executor/src/executor_process.rsis one longfunction that mixes several unrelated startup and shutdown phases. #2597 asks to split it
into smaller steps so that each one can be read, tested and changed on its own. As described
in the plan on the issue, this is the first PR of three: it extracts the phases that do not
overlap with the open PRs touching the same function. The scheduler connection
(
build_endpoint/connect_to_scheduler) andresolve_work_dirare left for the next twoPRs, once their prerequisites are merged.
This is a behaviour-preserving refactor: the moved code is copied literally, in the same
order, with the same log lines, errors and ordering of side effects.
What changes are included in this PR?
Only
ballista/executor/src/executor_process.rschanges. Four phases move out ofstart_executor_process_innerinto their own functions, one commit per step:resolve_vcores: the "0 means use the available parallelism" rule.spawn_shuffle_cleaner: the periodic shuffle clean-up task and its shutdown handle. Thejob_data_clean_up_interval_seconds > 0decisions stay instart_executor_process_inner.run_until_shutdown: waiting for the service handlers, a stop request or a terminationsignal, and notifying the scheduler on failure.
resolve_memory_pool_policy: choosing the memory pool budget, registering the metrics andwrapping the policy with the pool observer. It takes
Option<&ExecutorMemoryMetrics>. (Theplan on the issue called it
build_memory_pool_policy; it is renamed so it does not readlike the existing
memory_pool_policy(bytes, vcores)constructor.)The first commit only adds tests and does not touch production code. They pin the current
behaviour and pass on
mainbefore any extraction:scheduler_connect_timeout_seconds = 0makes a single attempt;job_data_clean_up_interval_seconds = 0the cleaner is not started and job data iskept on shutdown; with a positive interval job data is removed on shutdown. These two run a
whole executor against a fake idle scheduler and stop it by taking its Flight port.
Each extraction commit adds the unit tests of its own function (
resolve_vcores, the cleaner,and both outcomes of
run_until_shutdown).How it was tested, all on Linux, at the last commit:
-D warnings,cargo check,cargo doc, the workspace tests including the standalone and benchmark ones, toml, proto andconfig-docs checks, machete, RAT, prettier) pass.
fails, then restored the file.
one built from
main: one scheduler, one executor (--job-data-clean-up-interval-seconds 1 --job-data-ttl-seconds 1), aSELECT k, count(*), sum(v) ... GROUP BY k ORDER BY kover a100000-row CSV, then
SIGTERMon the executor. The result is the same, the cleaner empties thework dir, and the shutdown sequence in the logs is the same.
load, with no failure.
Not verified: macOS builds and tests, the h2o and TPC-H/TPC-DS jobs, the Kubernetes chaos job,
the Docker build, the Python bindings, CodeQL and the allowlist check.
SIGINT, the unboundedmemory pool and the automatic
--memory-pool-sizewere checked only by unit tests, not on areal cluster. Under very heavy CPU load I saw the existing
scheduler_connect_timeout_is_honouredandscheduler_connect_timeout_bounds_a_hanging_connecttests fail now and then; they are not touched by this PR.
Are there any user-facing changes?
No. There is no change to the public API, configuration, logs, protocol or behaviour.
Co-developed with Megabot, my AI-assisted development pipeline (Claude Sonnet 5.5; reviewed, tested and owned by me).