Skip to content

refactor(executor): extract startup phases from start_executor_process_inner - #2894

Open
juliomax wants to merge 5 commits into
apache:mainfrom
juliomax:refactor/executor-startup-phases
Open

juliomax wants to merge 5 commits into
apache:mainfrom
juliomax:refactor/executor-startup-phases

Conversation

@juliomax

Copy link
Copy Markdown

Which issue does this PR close?

Part of #2597 (first of three PRs; it does not close the issue).

Rationale for this change

start_executor_process_inner in ballista/executor/src/executor_process.rs is one long
function that mixes several unrelated startup and shutdown phases. #2597 asks to split it
into smaller steps so that each one can be read, tested and changed on its own. As described
in the plan on the issue, this is the first PR of three: it extracts the phases that do not
overlap with the open PRs touching the same function. The scheduler connection
(build_endpoint / connect_to_scheduler) and resolve_work_dir are left for the next two
PRs, once their prerequisites are merged.

This is a behaviour-preserving refactor: the moved code is copied literally, in the same
order, with the same log lines, errors and ordering of side effects.

What changes are included in this PR?

Only ballista/executor/src/executor_process.rs changes. Four phases move out of
start_executor_process_inner into their own functions, one commit per step:

  1. resolve_vcores: the "0 means use the available parallelism" rule.
  2. spawn_shuffle_cleaner: the periodic shuffle clean-up task and its shutdown handle. The
    job_data_clean_up_interval_seconds > 0 decisions stay in start_executor_process_inner.
  3. run_until_shutdown: waiting for the service handlers, a stop request or a termination
    signal, and notifying the scheduler on failure.
  4. resolve_memory_pool_policy: choosing the memory pool budget, registering the metrics and
    wrapping the policy with the pool observer. It takes Option<&ExecutorMemoryMetrics>. (The
    plan on the issue called it build_memory_pool_policy; it is renamed so it does not read
    like the existing memory_pool_policy(bytes, vcores) constructor.)

The first commit only adds tests and does not touch production code. They pin the current
behaviour and pass on main before any extraction:

  • the scheduler connect with scheduler_connect_timeout_seconds = 0 makes a single attempt;
  • the configured and the unbounded memory pool are reported to the metrics;
  • with job_data_clean_up_interval_seconds = 0 the cleaner is not started and job data is
    kept on shutdown; with a positive interval job data is removed on shutdown. These two run a
    whole executor against a fake idle scheduler and stop it by taking its Flight port.

Each extraction commit adds the unit tests of its own function (resolve_vcores, the cleaner,
and both outcomes of run_until_shutdown).

How it was tested, all on Linux, at the last commit:

  • The CI commands of this repository (fmt, clippy with -D warnings, cargo check,
    cargo doc, the workspace tests including the standalone and benchmark ones, toml, proto and
    config-docs checks, machete, RAT, prettier) pass.
  • For every new test I removed the effect it protects (mutation) and checked that the test
    fails, then restored the file.
  • Standalone cluster by hand, on localhost, with the binary of the last commit compared with the
    one built from main: one scheduler, one executor (--job-data-clean-up-interval-seconds 1 --job-data-ttl-seconds 1), a SELECT k, count(*), sum(v) ... GROUP BY k ORDER BY k over a
    100000-row CSV, then SIGTERM on the executor. The result is the same, the cleaner empties the
    work dir, and the shutdown sequence in the logs is the same.
  • The two clean-up tests that start a whole executor ran 30 times each, alone and under heavy
    load, with no failure.

Not verified: macOS builds and tests, the h2o and TPC-H/TPC-DS jobs, the Kubernetes chaos job,
the Docker build, the Python bindings, CodeQL and the allowlist check. SIGINT, the unbounded
memory pool and the automatic --memory-pool-size were checked only by unit tests, not on a
real cluster. Under very heavy CPU load I saw the existing
scheduler_connect_timeout_is_honoured and scheduler_connect_timeout_bounds_a_hanging_connect
tests fail now and then; they are not touched by this PR.

Are there any user-facing changes?

No. There is no change to the public API, configuration, logs, protocol or behaviour.

Co-developed with Megabot, my AI-assisted development pipeline (Claude Sonnet 5.5; reviewed, tested and owned by me).

Júlio Max and others added 5 commits November 18, 2026 18:23
…up at startup

Add characterization tests for start_executor_process ahead of splitting
start_executor_process_inner into phases (apache#2597): the single connect attempt
with scheduler_connect_timeout_seconds = 0, the memory pool size reported to
ExecutorMemoryMetrics for a configured and an unbounded pool, and the shuffle
clean-up: with job_data_clean_up_interval_seconds = 0 neither the periodic
cleaner nor the clean-up at shutdown runs, and with a positive interval job
data is removed on shutdown. The last two run a pull-based executor against an
idle in-process scheduler and shut it down by taking its Flight port.

Co-authored-by: Megabot <megabot@megabot.com.br>
…s_inner

Part of apache#2597. Moves the vcores resolution verbatim; behaviour is unchanged.

Co-authored-by: Megabot <megabot@megabot.com.br>
…_process_inner

Part of apache#2597. Moves the periodic shuffle cleaner task verbatim; behaviour is
unchanged. Adds a test that it removes expired job dirs and stops on shutdown.

Co-authored-by: Megabot <megabot@megabot.com.br>
…ocess_inner

Part of apache#2597. Moves the service/signal/stop-request select verbatim;
behaviour is unchanged. Adds tests for the stop-request and failed-service
outcomes.

Co-authored-by: Megabot <megabot@megabot.com.br>
…cutor_process_inner

Part of apache#2597. Moves the memory pool resolution and its metrics wiring
verbatim; behaviour is unchanged. The metrics handle is passed as
Option<&ExecutorMemoryMetrics>. Covered by the memory pool metrics tests.

Co-authored-by: Megabot <megabot@megabot.com.br>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant