Skip to content

fix: keep dedicated workers counted, busy and out of zombie restarts - #12540

Merged
rubenfiszel merged 6 commits into
mainfrom
rf/fix-dedicated-worker-autoscaling
Nov 19, 2026
Merged

rubenfiszel merged 6 commits into
mainfrom
rf/fix-dedicated-worker-autoscaling

Conversation

@rubenfiszel

@rubenfiszel rubenfiszel commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Autoscaling a group of dedicated workers misread it in three ways, all from one cause: the worker's main loop hands each job to a dedicated subprocess through a small channel and, when that channel was full, waited on send().await until a slot freed, a job duration away. That loop is the only thing that writes the worker's ping and its occupancy.

  • Worker count too low. The autoscaler counts workers that pinged in the last 30s. A busy dedicated worker stopped pinging for about a job's duration, so most pods went uncounted (count of 1 < min_workers of 5 with 5 pods up). Below min_workers the autoscaler returns before it reads the queue, so it never scaled out.
  • Occupancy always 0. Time spent in dedicated subprocesses was never added to occupancy, so scale-in always fired.
  • Jobs ran twice. A job's own ping (v2_job_runtime.ping) is set only when it is pulled. A dedicated job that took longer than ZOMBIE_JOB_TIMEOUT (60s) between pull and completion, waiting in the channel or running, was restarted by the zombie detector while still in the subprocess, and ran a second time. Jobs pulled into the channel were also marked running, so they did not count as waiting for the scale-out thresholds.

Changes

  • worker.rs: the main loop no longer pulls while any dedicated channel is full. It sleeps sleep_queue() and loops, so it keeps pinging, and the jobs stay in the queue where the autoscaler counts them. The concurrency semaphore is always Semaphore::new(1), so the subprocesses already run one job at a time and this costs no throughput.
  • common.rs: DedicatedWorkerState, shared by a worker's main loop and its dedicated subprocesses (found by worker name), holds two things:
    • The busy union. Dedicated jobs and the jobs the main loop runs inline both hold a BusyGuard, so overlapping work counts once. For a worker with dedicated subprocesses, this union replaces the main loop's own timers in OccupancyMetrics.
    • The in-flight set. These are jobs handed to a subprocess that it isn't done with yet. The main loop adds a job before the send. The subprocess takes a HandedOver guard on receipt, and dropping it releases the job: at completion, when the subprocess loop exits with jobs still queued (for example a draining generation during an in-place reload), or on an error before the job is queued. A job released unfinished is recovered by the zombie detection, as before.
  • worker.rs: a task of its own pings the in-flight jobs every 5s, keeping only those still running on this worker. It runs apart from the main loop, which can spend longer than the zombie timeout on a job it runs inline (a restricted-token job, a flow-runner wait).
  • EE (dedicated_worker_ee.rs; windmill-ee-private#997, merged, and chore(deps): bump vscode-ws-jsonrpc from 2.0.0 to 2.0.1 in /frontend #998): handle_dedicated_process takes the HandedOver guard when it receives a job, and holds the busy guard from the semaphore permit until the job's result is read.

One behaviour change: a dedicated job that hangs is no longer "restarted" after 60s. That restart was what caused the double runs. It now stays running, like a hung job on a regular worker.

Not covered here: a job handed to a flow runner still blocks the main loop on done_rx.await until the step finishes, so the worker's own ping can still lapse during a long flow-runner step. The in-flight task does not cover those jobs, which have the same zombie exposure. It's a separate path and worth a follow-up.

Test plan

  • Repro before the fix (EE build, one dedi worker, a bun dedicated script sleeping 40s, 8 jobs, DryRun autoscaling): worker ping age climbed to ~35s per job, the autoscaler logged count of 0 < min_workers of 1 with the worker up, and 5 of 8 jobs were zombie-restarted and ran twice ("job already completed by another worker"). The batch took ~9 min.
  • Same repro after the fix: worker ping age ≤ 6s throughout, occupancy_rate_15s = 1.00 while busy and 0 once idle, unstarted jobs stayed running = false and the autoscaler scaled out on them ("5 jobs waiting >= full scaleout threshold"). 0 zombie restarts, 0 double runs, batch took 320s (8 × 40s).
  • 3 jobs of 90s each, every one past the 60s zombie timeout even while executing: all completed once, 0 zombie restarts, 280s total.
  • A plain bun job on the default group still runs.
  • cargo test -p windmill-worker --lib dedicated_worker_counts: an inline job overlapping a dedicated one counts once, and dropping the dedicated guard releases its in-flight job.
  • After the review fixes: 5 dedicated jobs of 40s took 200s, 0 zombie restarts, 0 double runs, job pings ≤ 1s old, occupancy 1.00 while busy and 0.00 once idle.
  • Not exercised: a restricted-token dedicated job run inline while other dedicated jobs are in flight (the reason the in-flight ping has its own task).

🤖 Generated with Claude Code


Summary by cubic

Fixes dedicated worker autoscaling, which misread busy workers as idle and double-ran long jobs.

When a dedicated worker's channel is full, the main loop no longer parks on send().await (which froze the worker's ping and occupancy for a job's duration); it leaves the job queued and keeps looping.

  • In-flight dedicated jobs are now pinged by their own task, so jobs longer than 60s aren't restarted as zombies and run twice.
  • DedicatedOccupancy records time spent in dedicated subprocesses and adds it to the worker's occupancy for scale-in decisions.

One behavior change: a hung dedicated job no longer restarts after 60s; it stays running like a hung regular-worker job.

Not covered: jobs handed to a flow runner still block the main loop and keep their zombie exposure; worth a follow-up.

Written for commit acb2787. Summary will update on new commits.

View guided diff Turn on auto-fix

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 2m 11s —— View job


Good to merge

Summary

I traced the three changes against the main loop (worker.rs), the EE channel and semaphore setup (create_dedicated_worker_map and spawn_dedicated_worker in dedicated_worker_ee.rs) and the zombie query (backend/src/monitor.rs:6300). I found no P0, P1 or P2 issues.

Channel-full gate (worker.rs:3907)

  • It runs after the same-worker channel and killpill checks, so flows already running on this worker keep draining and shutdown still works.
  • The worker ping at the top of the loop (worker.rs:3707) now runs on every pass.
  • In EE, concurrency_semaphore is always Some(Semaphore::new(1)), so every channel has capacity 2 and all subprocesses run one job at a time.
  • So refusing to pull while any one channel is full costs no dedicated throughput, even for workers with several scripts, runner groups or flows. Before this PR the loop was already stuck on send().await in that state.

In-flight pinger (worker.rs:831)

  • The zombie query picks jobs by running = true AND ping < now() - timeout.
  • The pinger's q.running AND q.worker = $2 filter matches that. A job that was restarted, or picked up by another worker, drops out of the set on the next tick.
  • The retain(|id| still_running.contains(id) || !sent.contains(id)) handles the race where a job is handed over during the query.
  • Using Weak stops the task once the main loop drops dedicated_in_flight.
  • Jobs are tracked only after send succeeds. Flow-step dedicated jobs (flow_script_step_runner_key) go through the same send, so they are covered too.

Occupancy (common.rs)

  • busy_secs is cumulative, which fits how the 15s/5m/30m windows are derived from the history.
  • The process-wide DEDICATED_OCCUPANCY map can't carry stale busy time into a new start_time: run_workers runs once per process (main.rs:1329), so run_worker never restarts in place.
  • Capping at 1 handles an inline job overlapping a dedicated one.

Observations (not blocking)

  • Inline jobs wait too. While any dedicated channel is full, the gate also stops the loop from pulling jobs it would run inline, such as restricted-token jobs or non-dedicated flow steps in the same group. The PR description covers this, and these jobs are rare on a dedicated group, but it adds some latency in mixed groups.
  • Hung dedicated jobs. A hung job now stays running, as disclosed. With the shared Semaphore::new(1), one hung job stalls every subprocess on that worker. It keeps pinging and reports occupancy 1, so the autoscaler scales out instead of treating it as gone. That seems like the right failure mode, but it may be worth a sentence in the docs or changelog.
  • EE side not checked. The windmill-ee-private checkout here is older than e88293dd, so I couldn't review where the busy guard is placed in handle_dedicated_process.

AGENTS.md compliance

  • Comments explain constraints (why the loop must not wait on a full channel, why the pinger runs as its own task) rather than narrating the code.
  • No scratch files, and only one test ships.

Test coverage

  • What's covered: dedicated_occupancy_counts_overlapping_jobs_once pins the one piece of new pure logic, that overlapping busy intervals count once. Its timing tolerance is safe because both sides measure the same intervals, and its final assert_eq is exact.
  • What isn't: the gate and the pinger are worker-loop and DB behaviour with no integration test. That's reasonable given the EE and dedicated-subprocess setup they need. The manual repro in the description (40s and 90s jobs, worker ping age ≤ 6s, 0 zombie restarts, scale-out on waiting jobs) is what verifies them.
  • Still to run manually: the one unchecked item. On a dedicated group, run a job of a script with restricted job_token_scopes, so it runs inline, while other dedicated jobs are buffered. Check that the in-flight jobs keep a fresh v2_job_runtime.ping past 60s and none is zombie-restarted.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: 0e1466b
Status: ✅  Deploy successful!
Preview URL: https://76ec26da.windmill.pages.dev
Branch Preview URL: https://rf-fix-dedicated-worker-auto.windmill.pages.dev

View logs

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

cc @rubenfiszel

Should address issues before merging: double-counted occupancy, orphaned reload jobs

  • [P1] double-counted occupancy — backend/windmill-worker/src/common.rs:787

    Adding dedicated busy time to inline execution time counts overlapping work twice. Capping the resulting average cannot remove that overlap: 360 seconds of fully overlapping dedicated and restricted-token inline execution within a 30-minute window reports about 40% occupancy, although the worker was busy for about 20%. This prevents scale-in at the default 25% threshold. Record the union of inline and dedicated busy intervals before updating the cumulative history.

  • [P1] orphaned reload jobs — backend/windmill-worker/src/worker.rs:849

    If a dedicated subprocess exits before returning its result while draining for an in-place reload, its unfinished job is now pinged indefinitely. The EE reload path (dedicated_worker_ee.rs:1578–1602) replaces that generation without stopping the worker, and handle_dedicated_process drops its pending jobs without completing them. Their queue rows remain running on this worker, so this heartbeat keeps refreshing them and prevents the zombie recovery that previously retried them. Track subprocess ownership and recover unfinished jobs when a generation exits, or restart the worker when that happens.

Test coverage

The new Rust unit test covers overlapping dedicated guards. Add a regression for overlap with inline execution and worker/DB integration coverage for heartbeat cleanup after an interrupted reload.

I reviewed the diff and relevant EE lifecycle paths; git diff --check passed. An isolated Rust harness using the extracted occupancy calculation reproduced 40.02% for approximately 20% busy time. I did not run the full Cargo tests or a live worker.

There is no new UI surface. On an EE worker, exercise overlapping dedicated and restricted-token inline jobs, followed by idle time; occupancy should reflect their combined busy interval once.

During a running dedicated job, deploy another version and make the draining subprocess exit before producing its result. The unfinished job should recover or fail explicitly, rather than remain running with fresh heartbeats forever.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

cc @rubenfiszel

Mergeable, but should ideally address nits: head-of-line blocking across dedicated channels when one channel is full (worker.rs:3907)

I read the diff, the surrounding main loop in worker.rs, the occupancy bookkeeping in common.rs, the zombie-detection query in backend/src/monitor.rs, and the EE subprocess loop in dedicated_worker_ee.rs (which the ee-repo-ref.txt bump points at). The core fix is sound: the worker keeps pinging while a channel is full, in-flight dedicated jobs get their own 5s ping (well under the 60s ZOMBIE_JOB_TIMEOUT), and DedicatedOccupancy correctly counts the union of busy intervals via a running counter, so overlapping subprocesses don't inflate occupancy. The min(1.0) caps are the right call given a main-loop job can overlap a dedicated one.

Findings

P2 — worker.rs:3907 — a full channel on one dedicated script gates pulls for every workload on the worker

} else if dedicated_workers.values().any(|tx| tx.capacity() == 0) {
    tokio::time::sleep(Duration::from_millis(sleep_queue())).await;
    continue;
}

The gate is evaluated across all dedicated channels and is placed before the pull, so it applies regardless of which job the queue would return next. On a worker configured with more than one dedicated script — or a dedicated script plus flow-runner steps or restricted-token jobs (which run inline) — a single saturated channel suppresses pulling of jobs that target the other, idle channels.

Before this change, the loop pulled one job and only blocked on send().await when that specific job hit a full channel; jobs for other channels (or inline jobs) that were next in the queue still flowed. Now, under sustained load on one script (the exact scenario this PR targets), its channel stays full nearly continuously, so the other scripts/flow-runners/restricted jobs on the same worker get roughly one pull opportunity per completion of a job on the saturated script — a queue-order-independent starvation that didn't exist before. The PR's own test plan notes the mixed-workload case as "not exercised," but frames it only as the reason the in-flight pinger runs in its own task; the pulling gate's collateral effect on sibling channels is a separate consequence.

This is EE-only and config-dependent (a single-script dedicated worker is unaffected), so it's not a merge blocker, but it's worth either making the gate per-target-channel (only defer when the job that would be pulled has nowhere to go) or documenting the tradeoff.

Considered and dismissed

  • retain(|id| still_running.contains(id) || !sent.contains(id)) in the pinger (worker.rs:849-864): I traced the snapshot logic — sent is the exact set at query start, still_running is the subset the UPDATE actually touched, so jobs handed over mid-query (not in sent) are kept and completed/foreign jobs are dropped. Correct.
  • Pinger's q.running AND q.worker = $2 filter: a dedicated job is only inserted into in_flight after a successful send(), which happens after pull() has already set running = true and worker = worker_name, so the filter matches exactly while the job is in the channel or subprocess. Confirmed against pull() in jobs.rs and the zombie query in monitor.rs.
  • Occupancy double-counting: running_job_started_at is reset to None at the top of every loop iteration, and the dedicated path continues before the inline completion accounting, so dedicated time is only tallied through busy_secs(). No overlap with total_duration_of_running_jobs.
  • DedicatedOccupancy global map grows without eviction: worker names are a small, stable set per process (and native-mode names are distinct per worker), so this is a negligible concern, not a leak worth flagging.
  • capacity() == 0 on a dead subprocess's channel: a dedicated subprocess exit sends the worker's killpill (per the code's own comment), so the gate is moot in that case; the pre-existing "pulled job dropped on failed send" path is unchanged by this PR.
  • Flow-runner jobs still blocking the main loop on done_rx.await: explicitly acknowledged in the PR as a known, pre-existing follow-up, not introduced here.
  • Unit test flakiness (dedicated_occupancy_counts_overlapping_jobs_once): both the measured busy_secs() and the started.elapsed() reference derive from the same wall clock, so scheduling delays cancel out; the 10ms tolerance is ample.

Test coverage

  • Backend (Rust): the PR adds one focused unit test for the overlapping-interval counting, which is the right granularity for DedicatedOccupancy. The worker.rs changes (the full-channel gate and the in-flight pinger) are main-loop/EE paths with no unit or integration coverage in this diff — acceptable given they require a live DB + EE subprocess setup, but worth noting the gate's cross-channel behavior is exactly the kind of logic that would benefit from a small test of the branching condition if it were extracted.
  • CI / config: the ee-repo-ref.txt bump is a pointer update with no testable surface; the behavior it pins lives in the companion private PR.

Manual verification still recommended before merge: the PR's own "not exercised" scenario — a worker with a dedicated script under sustained load while (a) a second dedicated script has queued jobs and (b) a restricted-token dedicated job runs inline — should be run to confirm the sibling script's jobs still make progress and aren't deferred until the first script drains. The already-run repros (ping age ≤ 6s, occupancy_rate_15s reaching 1.0 then 0, 0 zombie restarts over 90s jobs) cover the primary fix well.

This commit updates the EE repository reference after PR #997 was merged in windmill-ee-private.

Previous ee-repo-ref: e88293dda3a13d3a1db7007cbdf9d5efb49214d1

New ee-repo-ref: 32a01d290fc38e2da374d428cc87f2b62e38af0c

Automated by sync-ee-ref workflow.
@windmill-internal-app

Copy link
Copy Markdown
Contributor

🤖 Updated ee-repo-ref.txt to 32a01d290fc38e2da374d428cc87f2b62e38af0c after windmill-ee-private PR #997 was merged.

@windmill-internal-app windmill-internal-app Bot changed the title [ee] fix: keep dedicated workers counted, busy and out of zombie restarts fix: keep dedicated workers counted, busy and out of zombie restarts Nov 19, 2026
@rubenfiszel
rubenfiszel marked this pull request as ready for review November 19, 2026 13:08
@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 1m 56s —— View job


cc @rubenfiszel

Should address issues before merging: a job dropped during an in-place reload is pinged forever (worker.rs:827). Nit: time when an inline job and a dedicated job run together is counted twice (common.rs:785)

  • Read the diff and what changed since my last review (only the ee-repo-ref.txt bump to 32a01d29)
  • Check the Codex P1s against the EE reload path, now checked out at the new ref
  • Post the review

What changed since my last review

My earlier "Good to merge" was written without the EE side: the checkout then was older than the companion PR. With 32a01d29 checked out, I traced the in-place reload in spawn_dedicated_worker and handle_dedicated_process. Codex's "orphaned reload jobs" finding holds up, so I'm withdrawing the earlier verdict.

Findings

P1: a job dropped during an in-place reload is pinged forever (worker.rs:827-830, inline)

  • The pinger's comment says it is safe because "a dedicated subprocess that exits stops the whole worker". That is false during an in-place reload.
  • On reload (dedicated_worker_ee.rs:1573-1584, and :1839-1841 for runner groups), the generation drains after stop_tx. If the subprocess dies before it prints wm_res[ (OOM, segfault, process.exit in user code), handle_dedicated_process breaks on EOF (:200/:271) and drops the pending job without completing it.
  • run still returns Ok, because process_status only logs. The reload arm then starts the next generation without a killpill.
  • The job's row stays running on this worker, and the pinger keeps its ping fresh. The zombie detector, which recovered such jobs before this PR, no longer sees it, so the job looks running until the worker restarts.
  • Suggested fix: when the subprocess exits, fail whatever is left in jobs with an explicit error. That covers every exit path, not only reload.

P2: time when an inline job and a dedicated job run together is counted twice (common.rs:785, inline)

  • Inline time and dedicated busy time are added together. The .min(1.0) caps stop a window from exceeding 100%, but they don't remove the overlap inside a window.
  • Example: an inline job and a dedicated job running during the same half of a 15s window report 100% instead of 50%.
  • This needs a mixed configuration (restricted-token or non-dedicated steps on a dedicated worker), and the error points toward "busy". It can still hold the worker above the scale-in threshold.
  • Suggested fix: take a DedicatedOccupancy guard around inline execution as well, so everything comes from one union.

Considered and dismissed

  • Pi's head-of-line concern on the channel-full gate (worker.rs:3907): the subprocesses share one Semaphore::new(1), so a job pulled for an idle channel would wait for the permit anyway. The gate keeps queue order and loses no throughput. Inline jobs do wait, as I noted last time.
  • Pinger retain race, Weak shutdown, the running AND worker filter: still correct, as in my previous review.

AGENTS.md compliance

  • No violations.
  • The pinger comment states its invariant, which is good practice. The invariant is wrong, though, and the P1 fix should correct it.

Test coverage

  • Backend: dedicated_occupancy_counts_overlapping_jobs_once covers overlap between dedicated guards. It does not cover overlap between an inline job and a dedicated one, which the P2 is about; a unit test for that would fit once the union is shared. The pinger and the reload cleanup need a live EE worker, so a manual check is reasonable there.
  • Manual check still needed: on an EE worker with a dedicated bun script that sleeps about 30s and then calls process.exit(1), start a job, then deploy a new version of the script while it sleeps so a reload drains the subprocess.
    • It passes if the job ends failed with an explicit error.
    • Before the fix, it stays running with a fresh v2_job_runtime.ping indefinitely.
  • Still unexercised: the PR's unchecked item, a restricted-token job run inline alongside in-flight dedicated jobs.

Comment thread backend/windmill-worker/src/worker.rs Outdated
Comment on lines +827 to +830
/// Pings the jobs in `in_flight` every `NUM_SECS_PING`, dropping those no longer running here,
/// until the worker drops `in_flight`. Runs apart from the main loop, which can spend longer
/// than the zombie timeout on a job it runs itself. Safe for as long as the worker lives: a
/// dedicated subprocess that exits stops the whole worker.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: a job dropped during an in-place reload is pinged forever. The comment says this is safe because a dedicated subprocess that exits stops the whole worker. With the in-place reload at the pinned EE ref, that is not always true.

  1. A deploy fires reload_rx, and spawn_dedicated_worker (dedicated_worker_ee.rs:1573-1584) sends stop_tx. The subprocess gets end and drains.
  2. If the subprocess dies before it prints wm_res[ for its current job (OOM, segfault, process.exit in user code), handle_dedicated_process hits EOF and breaks (:200/:271). The job still in jobs is dropped without a JobCompleted.
  3. process_status only logs, so run returns Ok. The reload arm then calls break Some(new), and the next generation starts without a killpill. Runner groups behave the same way (:1839-1841).
  4. The row stays running = true, worker = <this>. The pinger keeps refreshing its ping every 5s, so the zombie detector, which recovered the job before this PR, never sees it. It stays "running" until the worker restarts.

Suggested fix: in handle_dedicated_process, when the process exits with entries left in jobs, send each one as a failed JobCompleted ("dedicated worker process exited before returning a result"). That closes the hole on every exit path, not only reload.

Fix this →

Comment thread backend/windmill-worker/src/common.rs Outdated
.map(|started_at| started_at.elapsed().as_secs_f32())
.unwrap_or(0.0);
let total_occupation = metrics.total_duration_of_running_jobs + current_occupied_duration;
let total_occupation = metrics.total_duration_of_running_jobs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: time when an inline job and a dedicated job run together is counted twice. Inline time (total_duration_of_running_jobs + current_occupied_duration) and dedicated busy time are added together. The .min(1.0) caps only stop a window from going over 100%. They do not remove the overlap inside a window.

Example: an inline restricted-token job and a dedicated job run during the same 7.5s of a 15s window. The worker was busy 50% of the window, but this reports 100%. Over 30m, the same pattern reads as twice the real occupancy and can keep the worker above the scale-in threshold.

This only happens in mixed configurations, and the error points toward "busy". That is why I rate it P2. A fix is to have the inline timer take a DedicatedOccupancy guard too (dedicated.start() while running_job_started_at is set), so everything comes from one union, and then drop the inline term when dedicated is Some.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

cc @rubenfiszel

Should address issues before merging: double-counted occupancy, orphaned reload jobs

  • [P1] double-counted occupancy — backend/windmill-worker/src/common.rs:787

    A restricted-token job runs inline while a dedicated subprocess can remain busy. Adding their elapsed times counts that overlap twice; capping the resulting average cannot remove it. For example, 360 seconds of overlapping execution within a 30-minute window reports about 40% occupancy instead of 20%, preventing scale-in at the default 25% threshold. Record the union of inline and dedicated busy intervals before updating occupancy history.

  • [P1] orphaned reload jobs — backend/windmill-worker/src/worker.rs:849

    If a dedicated subprocess exits before returning a job’s result while draining for an in-place reload, this task keeps pinging the abandoned job indefinitely. At the pinned EE ref, spawn_dedicated_worker’s reload arm starts another generation even when the draining generation exits with unfinished jobs; it does not stop the worker. Those jobs remain running on the same worker, so this query keeps them eligible for heartbeats and prevents zombie recovery. Track generation ownership and recover unfinished jobs when it exits, or restart the worker when a generation exits with unfinished work.

Test coverage

The new Rust unit test covers overlapping dedicated jobs. Worker/DB integration coverage for backpressure and heartbeat cleanup is absent.

I ran git diff --check and a standalone Rust harness extracted from the supplied head. The dedicated-overlap test passed; overlapping inline/dedicated execution reproduced 40.11% reported occupancy versus 20.06% actual busy time. I did not run the Cargo suite or a live worker.

There is no new UI surface. On an EE dedicated worker, exercise overlapping dedicated and restricted-token inline jobs followed by idle time; occupancy should count their overlap once.

During a running dedicated job, trigger an in-place reload and make the draining subprocess exit before returning its result. The unfinished job should recover or fail explicitly instead of remaining running with fresh heartbeats indefinitely.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

cc @rubenfiszel

Should address issues before merging: double-counted occupancy (common.rs:785), orphaned in-flight jobs after an in-place reload (worker.rs:831)

Findings

P1 — backend/windmill-worker/src/common.rs:785 — inline and dedicated busy time are summed, double-counting overlap

let total_occupation = metrics.total_duration_of_running_jobs
    + current_occupied_duration
    + metrics.dedicated.as_ref().map(|d| d.busy_secs()).unwrap_or(0.0);

The inline timers (total_duration_of_running_jobs / current_occupied_duration) and the dedicated subprocess timers (busy_secs()) measure wall-clock time over the same intervals whenever the main loop runs a job inline while a dedicated subprocess runs concurrently. This is exactly the scenario the PR's own test plan lists as unexercised ("a restricted-token dedicated job run inline while other dedicated jobs are in flight"): a restricted-token job (job.job_token_scopes.is_some()) skips the dedicated hand-off and runs in the main loop, where metric_timer feeds total_duration_of_running_jobs (worker.rs:5599), while the dedicated subprocesses for other scripts accumulate busy_secs() in parallel.

The .min(1.0) caps at lines 827/832/837/845 only mask the case where the sum exceeds 100%. They do nothing for partial overlap: 360s of fully-overlapping inline and dedicated execution inside a 1800s window reports (360+360)/1800 = 40% occupancy when the worker was actually busy 20%. This inflates occupancy and suppresses scale-in (default threshold 25%) in mixed workloads — the opposite failure mode from the "occupancy always 0" bug this PR fixes. The windowed rates (15s/5m/30m) and the instantaneous rate are all affected. The prior Pi review's dismissal of this only covered the dedicated-pulled path (which continues before metric_timer); it did not cover an inline job overlapping concurrent subprocesses. Fix by recording the union of inline and dedicated busy intervals, not the sum.

P1 — backend/windmill-worker/src/worker.rs:831 — the in-flight pinger removes zombie recovery for jobs orphaned by an in-place reload

The pinger's own doc comment asserts the safety premise: "Safe for as long as the worker lives: a dedicated subprocess that exits stops the whole worker." That invariant has an exception in the EE reload path (dedicated_worker_ee.rs:1572-1585): next_reload sends stop_tx, awaits the subprocess, then break Some(new) and starts the next generation without restarting the worker.

When the subprocess exits without completing its in-flight job — a crash, process.exit() from user code, or OOM during the drain window — handle_dedicated_process breaks out of its select! loop and drops its jobs deque (dedicated_worker_ee.rs:266-330), so the job's MiniCompletedJob is never sent to job_completed_tx. Its queue row stays running = true with worker = <this worker>. The pinger's UPDATE filter (q.running AND q.worker = $2) keeps matching it, and the retain clause keeps it in dedicated_in_flight, so its ping is refreshed every 5s forever. The zombie detector in backend/src/monitor.rs:6300-6311 (which restarts jobs whose ping is older than ZOMBIE_JOB_TIMEOUT) never fires, so the job hangs permanently with no recovery. Before this PR, the same orphaned job's ping went stale and it was restarted after 60s. This is a silent removal of that safety net; a subprocess generation that exits should either complete/fail its pending jobs or restart the worker, so the pinger's premise holds.

P2 — backend/windmill-worker/src/worker.rs:3907 — a full channel on one dedicated script blocks pulls for every channel

} else if dedicated_workers.values().any(|tx| tx.capacity() == 0) {
    tokio::time::sleep(Duration::from_millis(sleep_queue())).await;
    continue;
}

The gate is evaluated across all dedicated channels before the pull, so it applies regardless of which job the queue would return next. On a worker with more than one dedicated script — or a dedicated script plus flow-runner steps or restricted-token inline jobs — a single saturated channel suppresses pulling of jobs that target the other, idle channels, and of inline jobs that need no channel at all. Before this change, the loop pulled one job and only blocked on send().await when that job hit a full channel; jobs for other channels still flowed. Now, under sustained load on one script (the scenario this PR targets), siblings get roughly one pull opportunity per completion of a job on the saturated script — queue-order-independent starvation that did not exist before. This is EE-only and config-dependent (a single-script dedicated worker is unaffected), so it's not a merge blocker, but the gate should be per-target-channel, or the trade-off documented.

Considered and dismissed

  • Pinger retain(|id| still_running.contains(id) || !sent.contains(id)) race — sent is the exact snapshot at query start and still_running is the subset the UPDATE actually touched, so jobs handed over mid-query (not in sent) are kept and completed/foreign jobs are dropped. Correct.
  • Pinger filter q.running AND q.worker = $2 vs. the zombie query — a job is inserted into dedicated_in_flight only after a successful send(), which follows pull() already setting running = true and worker = worker_name, so the filter matches the zombie detector's selection exactly while the job is in the channel or subprocess. Confirmed against pull() and monitor.rs:6300.
  • DEDICATED_OCCUPANCY global map growth — keyed by worker name, a small stable set per process (distinct names in native mode); not a leak.
  • Blocking std::sync::Mutex in async paths — every critical section (start, Drop, busy_secs) is short with no .await inside, and poisoning is recovered via into_inner().
  • Gate placement vs. shutdown/draining — the gate is an else if after the killpill and killed_but_draining_same_worker_jobs/paused checks, so shutdown and draining still take precedence.
  • Flow-runner zombie exposure ("not covered" in the PR) — disclosed in the body and pre-existing (flow-runner jobs were never in dedicated_in_flight and were never pinged before); not introduced by this PR, so not a finding.

Test coverage

  • Backend: the added dedicated_occupancy_counts_overlapping_jobs_once unit test correctly pins the union logic of DedicatedOccupancy (overlapping guards count once, and the exact final assert_eq is safe because both sides measure the same interval). There is no test for the overlap with inline execution (finding feat: Typescript support for scripts (alpha) #1) and no worker/DB integration test for the pinger's heartbeat-cleanup on an interrupted reload (finding Typo on site homepage #2); both are the gaps worth a regression guard if the fixes land.
  • Frontend / CI / docs: not touched — no tests expected.

Manual verification still needed: on an EE worker with a dedicated script, run a restricted-token job inline while dedicated jobs are in flight, then let it go idle — the reported 15s/5m/30m occupancy should reflect the union of the two busy intervals, not their sum, and scale-in should fire when the union drops below threshold. Separately, start a long dedicated job and trigger an in-place reload, then kill the subprocess mid-drain — the unfinished job should be re-queued or failed (and eventually recovered) rather than remaining running with a perpetually fresh ping.

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

Addressed in 4bde084 (+ EE windmill-labs/windmill-ee-private#998, pinned in ee-repo-ref):

  • Double-counted occupancy (Codex P1, Pi P1, Claude P2): a worker with dedicated subprocesses now measures one busy union. The main loop holds a busy guard on the same DedicatedWorkerState for every job it runs itself, so an inline job overlapping a dedicated one is counted once, and that union replaces the main loop's own timers. The min(1.0) caps are gone, since a union can't exceed elapsed time. The unit test covers an inline guard overlapping a dedicated one.
  • Job dropped during an in-place reload pinged forever (Codex P1, Pi P1, Claude P1): the in-flight set now lives in the same per-worker state. A subprocess takes a HandedOver guard the moment it receives a job, and the guard is carried into the job's busy guard. Dropping it, whether at completion, when the subprocess loop exits with jobs still queued, or on an error before the job is queued, takes the job out of the in-flight set. An abandoned job is then recovered by the zombie detection, as before this PR. The main loop now hands a job over before the send, so a fast subprocess can't take it back before the main loop records it.
  • One full channel gates every pull (Pi P2), kept as is: the dedicated subprocesses share Semaphore::new(1) (create_dedicated_worker_map), so a job for another, idle channel could not have run until the busy one finished either. Before this PR, the loop blocked on the full channel's send().await and could not pull at all. The only work the gate delays is inline work: restricted-token jobs, dependency jobs and flow-runner steps on a dedicated worker. Gating per target channel would mean pulling first, which claims the job and takes it out of the queue counts the autoscaler reads.

Re-verified live on an EE build: 5 dedicated jobs of 40s took 200s, with 0 zombie restarts and 0 double runs. The jobs' own pings stayed ≤ 1s old, occupancy read 1.00 while busy and 0.00 once idle, and the worker's ping stayed ≤ 6s old.

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 1m 27s —— View job


Good to merge

  • Read the diff since my last review (4bde084, 0e1466b)
  • Check the EE side (HandedOver, busy guard) at the pinned ref — not possible here, see below
  • Post the review

What changed since my last review

Both findings from my last round are resolved on the CE side. The P2 that was kept as is (one full channel gates every pull) is justified by the shared Semaphore::new(1), which I agree with, so I'm not raising it again.

Inline and dedicated time counted twice (was P2): fixed

  • common.rs:837: when dedicated is Some, occupancy is now only busy_secs(), the union of all guards. Inline timers no longer add to it.
  • worker.rs:4061: _busy is taken in the Ok(Some(job)) arm and held until the arm exits. That covers inline execution and flow-runner waits. On the dedicated hand-off path it drops at the continue, right after the send. The gate keeps that send from waiting, because the channel can't fill between the capacity() check and the send when the main loop is its only producer.
  • Taking the .min(1.0) caps out restores the behaviour on main for non-dedicated workers, and a union can't exceed elapsed time.

Jobs dropped during a reload pinged forever (was P1): fixed, CE side

  • The in-flight set is now owned by DedicatedWorkerState. A job leaves it when its HandedOver drops, so a job a draining generation abandons stops being pinged. The zombie detector then recovers it after 60s, as it did before this PR.
  • Jobs still buffered in the channel are still pinged during a reload. That's correct: the channel outlives the generation, and the next one receives them.
  • hand_over before send and take_back on a failed send close the race where a fast subprocess finishes a job before the main loop records it. The comment at worker.rs:4125 states that constraint where someone would break it.
  • prune_in_flight keeps the earlier race-safe retain semantics.
  • for_worker now keeps a Weak in the map, and the main loop holds the strong ref for the worker's lifetime. Every lookup by the subprocesses therefore upgrades to the same state, and busy_secs starts at about the same time as start_time.

Not checked

  • The windmill-ee-private checkout in this runner is older than chore(deps): bump chartjs-plugin-zoom from 1.2.1 to 2.0.0 in /frontend #997: it has no DedicatedOccupancy, received or start_busy. So I couldn't confirm the EE side of the fix:
    • that handle_dedicated_process takes received(job_id) on every receive path, including runner groups and flow steps;
    • that the resulting HandedOver actually reaches start_busy(Some(..));
    • that it drops on every early exit.
  • The CE contract is sound if that holds. Your live re-run (5 × 40s, 0 zombie restarts, job pings ≤ 1s old) is good evidence for the normal path, but it doesn't cover the reload-crash path.

AGENTS.md compliance

  • No violations.
  • The pinger doc comment no longer makes the false "a dedicated subprocess that exits stops the whole worker" claim.
  • The new comments state constraints, not history.

Test coverage

  • Backend: dedicated_worker_counts_overlapping_jobs_once now pins both pieces of new logic:
    • an inline guard overlapping a dedicated one counts once;
    • dropping the dedicated guard releases its in-flight job.
  • That's the right minimal guard. The pinger and the gate are worker-loop and DB behaviour that need a live EE worker, so a manual check is reasonable for them.
  • Manual checks still worth running:
    • Reload crash: on an EE worker, run a dedicated bun script that sleeps about 30s and then calls process.exit(1). Deploy a new version while it sleeps. After about 60s the job should be zombie-restarted or failed, not left running with a fresh v2_job_runtime.ping.
    • Restricted-token inline job (still unchecked in the PR): run one while other dedicated jobs are buffered. In-flight jobs should keep fresh pings past 60s, and the 15s occupancy should read the union of the two busy periods.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No new issues found. The shared busy interval fixes double-counted occupancy, and the pinned EE ownership guard lets jobs abandoned during reload reach zombie recovery.

Test coverage

The new Rust unit test covers overlapping inline/dedicated execution and in-flight cleanup when a guard drops. Worker/DB integration coverage for backpressure, heartbeats and interrupted reload is absent.

git diff --check passed. I did not run Cargo tests or a live worker.

On an EE worker, overlap dedicated and restricted-token inline jobs, then leave it idle. Occupancy should count overlapping time once, and dedicated jobs should retain fresh heartbeats beyond the zombie timeout.

During a running job, deploy a new version and make the draining subprocess exit before returning its result. The abandoned job should stop receiving heartbeats and recover or fail; buffered jobs should complete once.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

Good to merge

I re-reviewed the current head (4bde084527 + EE f9654a5) against the two P1s raised in the earlier rounds and traced the fixes through the EE code path. Both are correctly resolved; the P2 (channel-full gate) was kept deliberately with a sound rationale, so I did not re-flag it. I found no new blocking or non-blocking issues.

Considered and dismissed

  • Double-counted occupancy (prior P1). Now resolved by a genuine interval union: Busy keeps a running counter with since/total_secs, so overlapping busy guards count once. The main loop holds start_busy(None) for the full inline job (worker.rs:4060, _busy lives until the Ok(Some(job)) arm ends), and each subprocess holds start_busy(Some(received)) from permit acquisition to wm_res (dedicated_worker_ee.rs:319). The min(1.0) caps were removed, which is correct — a union cannot exceed elapsed time. The shipped test dedicated_worker_counts_overlapping_jobs_once pins exactly this.
  • Orphaned in-flight jobs after an in-place reload (prior P1). Resolved. A subprocess takes a HandedOver via state.received(id) on dequeue and carries it into its BusyGuard; dropping the jobs deque on generation exit drops the guard → take_back removes the job from in_flight, so its ping goes stale and zombie recovery applies as before. Jobs still buffered in the channel are not leaked: the receiver (jobs_rx) outlives generations (dedicated_worker_ee.rs:1537-1553), and the crash/EOF path sends the killpill, restarting the whole worker and dropping the state. Verified the early-exit paths inside the dedicated_job branch (transform_json(...)? and write_stdin(...)?) also drop received, so no in_flight entry is left behind.
  • Head-of-line blocking across channels (prior P2). Kept as-is; the author's rationale (dedicated subprocesses share Semaphore::new(1), so a job for another channel could not run anyway) holds, and only rare inline work is delayed. Respecting the decision, not re-flagging.
  • running_job_started_at never reset to None. Pre-existing; the dedicated path no longer reads it (update_occupancy_metrics uses dedicated.busy_secs() when dedicated is Some), and the non-dedicated path is untouched. Not introduced here.
  • received guard created before acquire_owned().await. Safe: the branch runs to completion (no select! inside), and every early return drops the guard. No leak.
  • Pinger prune_in_flight race. Correct: sent is the exact snapshot at query start and still_running is what the UPDATE ... RETURNING actually matched, so jobs handed over mid-query (!pinged.contains(id)) are retained and completed/foreign jobs are dropped.
  • Global DEDICATED_WORKER_STATES map accumulating dead Weak entries. Bounded by the small, stable set of worker names per process; negligible.
  • Flow-runner jobs not covered by the in-flight pinger. Pre-existing exposure (they still block the main loop on done_rx.await and can lapse past ZOMBIE_JOB_TIMEOUT), explicitly disclosed in the PR body as a follow-up, and not a regression from this change.
  • Dead accounting on dedicated workers (running_job_started_at and total_duration_of_running_jobs still updated but ignored when dedicated is set). No functional impact; inline busy time is captured by the _busy guard instead.

Test coverage

  • Backend (Rust): the one new pure-logic unit, dedicated_worker_counts_overlapping_jobs_once, guards the union-accounting fix (the exact prior P1) with a sound tolerance, and covers an inline guard overlapping a dedicated one. The pinger and the channel-full gate are worker-loop/DB behaviour with no integration test; given they require an EE build plus dedicated subprocesses, that is reasonable, and the PR's manual repro (40s/90s bun jobs, ping age ≤ 6s, occupancy 1.00 busy / 0.00 idle, scale-out on queued jobs, 0 zombie restarts) is what verifies them.
  • Frontend / CI / config: none (only ee-repo-ref.txt, a config pointer, plus Rust). No automated tests expected for the ref bump.

The one manual scenario still worth exercising before merge is the item the PR's own test plan leaves unchecked: on an EE dedicated group, run a restricted-token (job_token_scopes) script inline while other dedicated subprocesses are running jobs, then confirm occupancy reports the combined busy interval once (not summed) — this exercises the _busy union guard added in the final commit, which the unit test covers only in isolation.

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

✅ Review round clean @ 0e1466b

@rubenfiszel
rubenfiszel merged commit 7b23884 into main Nov 19, 2026
14 checks passed
@rubenfiszel
rubenfiszel deleted the rf/fix-dedicated-worker-autoscaling branch November 19, 2026 13:51
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant