Repository navigation
fix: keep dedicated workers counted, busy and out of zombie restarts - #12540
Conversation
|
/review |
|
Claude finished @rubenfiszel's task in 2m 11s —— View job Good to merge SummaryI traced the three changes against the main loop ( Channel-full gate (
In-flight pinger (
Occupancy (
Observations (not blocking)
AGENTS.md compliance
Test coverage
|
Deploying windmill with
|
| Latest commit: |
0e1466b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://76ec26da.windmill.pages.dev |
| Branch Preview URL: | https://rf-fix-dedicated-worker-auto.windmill.pages.dev |
Codex Reviewcc @rubenfiszel Should address issues before merging: double-counted occupancy, orphaned reload jobs
Test coverageThe new Rust unit test covers overlapping dedicated guards. Add a regression for overlap with inline execution and worker/DB integration coverage for heartbeat cleanup after an interrupted reload. I reviewed the diff and relevant EE lifecycle paths; There is no new UI surface. On an EE worker, exercise overlapping dedicated and restricted-token inline jobs, followed by idle time; occupancy should reflect their combined busy interval once. During a running dedicated job, deploy another version and make the draining subprocess exit before producing its result. The unfinished job should recover or fail explicitly, rather than remain running with fresh heartbeats forever. |
Pi Reviewcc @rubenfiszel Mergeable, but should ideally address nits: head-of-line blocking across dedicated channels when one channel is full ( I read the diff, the surrounding main loop in FindingsP2 — } else if dedicated_workers.values().any(|tx| tx.capacity() == 0) {
tokio::time::sleep(Duration::from_millis(sleep_queue())).await;
continue;
}The gate is evaluated across all dedicated channels and is placed before the pull, so it applies regardless of which job the queue would return next. On a worker configured with more than one dedicated script — or a dedicated script plus flow-runner steps or restricted-token jobs (which run inline) — a single saturated channel suppresses pulling of jobs that target the other, idle channels. Before this change, the loop pulled one job and only blocked on This is EE-only and config-dependent (a single-script dedicated worker is unaffected), so it's not a merge blocker, but it's worth either making the gate per-target-channel (only defer when the job that would be pulled has nowhere to go) or documenting the tradeoff. Considered and dismissed
Test coverage
Manual verification still recommended before merge: the PR's own "not exercised" scenario — a worker with a dedicated script under sustained load while (a) a second dedicated script has queued jobs and (b) a restricted-token dedicated job runs inline — should be run to confirm the sibling script's jobs still make progress and aren't deferred until the first script drains. The already-run repros (ping age ≤ 6s, |
This commit updates the EE repository reference after PR #997 was merged in windmill-ee-private. Previous ee-repo-ref: e88293dda3a13d3a1db7007cbdf9d5efb49214d1 New ee-repo-ref: 32a01d290fc38e2da374d428cc87f2b62e38af0c Automated by sync-ee-ref workflow.
|
🤖 Updated |
|
Claude finished @rubenfiszel's task in 1m 56s —— View job cc @rubenfiszel Should address issues before merging: a job dropped during an in-place reload is pinged forever (
What changed since my last reviewMy earlier "Good to merge" was written without the EE side: the checkout then was older than the companion PR. With FindingsP1: a job dropped during an in-place reload is pinged forever (
P2: time when an inline job and a dedicated job run together is counted twice (
Considered and dismissed
AGENTS.md compliance
Test coverage
|
| /// Pings the jobs in `in_flight` every `NUM_SECS_PING`, dropping those no longer running here, | ||
| /// until the worker drops `in_flight`. Runs apart from the main loop, which can spend longer | ||
| /// than the zombie timeout on a job it runs itself. Safe for as long as the worker lives: a | ||
| /// dedicated subprocess that exits stops the whole worker. |
There was a problem hiding this comment.
P1: a job dropped during an in-place reload is pinged forever. The comment says this is safe because a dedicated subprocess that exits stops the whole worker. With the in-place reload at the pinned EE ref, that is not always true.
- A deploy fires
reload_rx, andspawn_dedicated_worker(dedicated_worker_ee.rs:1573-1584) sendsstop_tx. The subprocess getsendand drains. - If the subprocess dies before it prints
wm_res[for its current job (OOM, segfault,process.exitin user code),handle_dedicated_processhits EOF andbreaks (:200/:271). The job still injobsis dropped without aJobCompleted. process_statusonly logs, sorunreturnsOk. The reload arm then callsbreak Some(new), and the next generation starts without a killpill. Runner groups behave the same way (:1839-1841).- The row stays
running = true, worker = <this>. The pinger keeps refreshing its ping every 5s, so the zombie detector, which recovered the job before this PR, never sees it. It stays "running" until the worker restarts.
Suggested fix: in handle_dedicated_process, when the process exits with entries left in jobs, send each one as a failed JobCompleted ("dedicated worker process exited before returning a result"). That closes the hole on every exit path, not only reload.
| .map(|started_at| started_at.elapsed().as_secs_f32()) | ||
| .unwrap_or(0.0); | ||
| let total_occupation = metrics.total_duration_of_running_jobs + current_occupied_duration; | ||
| let total_occupation = metrics.total_duration_of_running_jobs |
There was a problem hiding this comment.
P2: time when an inline job and a dedicated job run together is counted twice. Inline time (total_duration_of_running_jobs + current_occupied_duration) and dedicated busy time are added together. The .min(1.0) caps only stop a window from going over 100%. They do not remove the overlap inside a window.
Example: an inline restricted-token job and a dedicated job run during the same 7.5s of a 15s window. The worker was busy 50% of the window, but this reports 100%. Over 30m, the same pattern reads as twice the real occupancy and can keep the worker above the scale-in threshold.
This only happens in mixed configurations, and the error points toward "busy". That is why I rate it P2. A fix is to have the inline timer take a DedicatedOccupancy guard too (dedicated.start() while running_job_started_at is set), so everything comes from one union, and then drop the inline term when dedicated is Some.
Codex Reviewcc @rubenfiszel Should address issues before merging: double-counted occupancy, orphaned reload jobs
Test coverageThe new Rust unit test covers overlapping dedicated jobs. Worker/DB integration coverage for backpressure and heartbeat cleanup is absent. I ran There is no new UI surface. On an EE dedicated worker, exercise overlapping dedicated and restricted-token inline jobs followed by idle time; occupancy should count their overlap once. During a running dedicated job, trigger an in-place reload and make the draining subprocess exit before returning its result. The unfinished job should recover or fail explicitly instead of remaining running with fresh heartbeats indefinitely. |
Pi Reviewcc @rubenfiszel Should address issues before merging: double-counted occupancy ( FindingsP1 — let total_occupation = metrics.total_duration_of_running_jobs
+ current_occupied_duration
+ metrics.dedicated.as_ref().map(|d| d.busy_secs()).unwrap_or(0.0);The inline timers ( The P1 — The pinger's own doc comment asserts the safety premise: "Safe for as long as the worker lives: a dedicated subprocess that exits stops the whole worker." That invariant has an exception in the EE reload path ( When the subprocess exits without completing its in-flight job — a crash, P2 — } else if dedicated_workers.values().any(|tx| tx.capacity() == 0) {
tokio::time::sleep(Duration::from_millis(sleep_queue())).await;
continue;
}The gate is evaluated across all dedicated channels before the pull, so it applies regardless of which job the queue would return next. On a worker with more than one dedicated script — or a dedicated script plus flow-runner steps or restricted-token inline jobs — a single saturated channel suppresses pulling of jobs that target the other, idle channels, and of inline jobs that need no channel at all. Before this change, the loop pulled one job and only blocked on Considered and dismissed
Test coverage
Manual verification still needed: on an EE worker with a dedicated script, run a restricted-token job inline while dedicated jobs are in flight, then let it go idle — the reported 15s/5m/30m occupancy should reflect the union of the two busy intervals, not their sum, and scale-in should fire when the union drops below threshold. Separately, start a long dedicated job and trigger an in-place reload, then kill the subprocess mid-drain — the unfinished job should be re-queued or failed (and eventually recovered) rather than remaining |
…ing' into rf/fix-dedicated-worker-autoscaling
|
Addressed in 4bde084 (+ EE windmill-labs/windmill-ee-private#998, pinned in ee-repo-ref):
Re-verified live on an EE build: 5 dedicated jobs of 40s took 200s, with 0 zombie restarts and 0 double runs. The jobs' own pings stayed ≤ 1s old, occupancy read 1.00 while busy and 0.00 once idle, and the worker's ping stayed ≤ 6s old. |
|
/review |
|
Claude finished @rubenfiszel's task in 1m 27s —— View job Good to merge
What changed since my last reviewBoth findings from my last round are resolved on the CE side. The P2 that was kept as is (one full channel gates every pull) is justified by the shared Inline and dedicated time counted twice (was P2): fixed
Jobs dropped during a reload pinged forever (was P1): fixed, CE side
Not checked
AGENTS.md compliance
Test coverage
|
Codex ReviewGood to merge No new issues found. The shared busy interval fixes double-counted occupancy, and the pinned EE ownership guard lets jobs abandoned during reload reach zombie recovery. Test coverageThe new Rust unit test covers overlapping inline/dedicated execution and in-flight cleanup when a guard drops. Worker/DB integration coverage for backpressure, heartbeats and interrupted reload is absent.
On an EE worker, overlap dedicated and restricted-token inline jobs, then leave it idle. Occupancy should count overlapping time once, and dedicated jobs should retain fresh heartbeats beyond the zombie timeout. During a running job, deploy a new version and make the draining subprocess exit before returning its result. The abandoned job should stop receiving heartbeats and recover or fail; buffered jobs should complete once. |
Pi ReviewGood to merge I re-reviewed the current head ( Considered and dismissed
Test coverage
The one manual scenario still worth exercising before merge is the item the PR's own test plan leaves unchecked: on an EE dedicated group, run a restricted-token ( |
|
✅ Review round clean @ 0e1466b |
Summary
Autoscaling a group of dedicated workers misread it in three ways, all from one cause: the worker's main loop hands each job to a dedicated subprocess through a small channel and, when that channel was full, waited on
send().awaituntil a slot freed, a job duration away. That loop is the only thing that writes the worker's ping and its occupancy.count of 1 < min_workers of 5with 5 pods up). Belowmin_workersthe autoscaler returns before it reads the queue, so it never scaled out.v2_job_runtime.ping) is set only when it is pulled. A dedicated job that took longer thanZOMBIE_JOB_TIMEOUT(60s) between pull and completion, waiting in the channel or running, was restarted by the zombie detector while still in the subprocess, and ran a second time. Jobs pulled into the channel were also marked running, so they did not count as waiting for the scale-out thresholds.Changes
worker.rs: the main loop no longer pulls while any dedicated channel is full. It sleepssleep_queue()and loops, so it keeps pinging, and the jobs stay in the queue where the autoscaler counts them. The concurrency semaphore is alwaysSemaphore::new(1), so the subprocesses already run one job at a time and this costs no throughput.common.rs:DedicatedWorkerState, shared by a worker's main loop and its dedicated subprocesses (found by worker name), holds two things:BusyGuard, so overlapping work counts once. For a worker with dedicated subprocesses, this union replaces the main loop's own timers inOccupancyMetrics.HandedOverguard on receipt, and dropping it releases the job: at completion, when the subprocess loop exits with jobs still queued (for example a draining generation during an in-place reload), or on an error before the job is queued. A job released unfinished is recovered by the zombie detection, as before.worker.rs: a task of its own pings the in-flight jobs every 5s, keeping only those still running on this worker. It runs apart from the main loop, which can spend longer than the zombie timeout on a job it runs inline (a restricted-token job, a flow-runner wait).dedicated_worker_ee.rs; windmill-ee-private#997, merged, and chore(deps): bump vscode-ws-jsonrpc from 2.0.0 to 2.0.1 in /frontend #998):handle_dedicated_processtakes theHandedOverguard when it receives a job, and holds the busy guard from the semaphore permit until the job's result is read.One behaviour change: a dedicated job that hangs is no longer "restarted" after 60s. That restart was what caused the double runs. It now stays running, like a hung job on a regular worker.
Not covered here: a job handed to a flow runner still blocks the main loop on
done_rx.awaituntil the step finishes, so the worker's own ping can still lapse during a long flow-runner step. The in-flight task does not cover those jobs, which have the same zombie exposure. It's a separate path and worth a follow-up.Test plan
dediworker, a bun dedicated script sleeping 40s, 8 jobs, DryRun autoscaling): worker ping age climbed to ~35s per job, the autoscaler loggedcount of 0 < min_workers of 1with the worker up, and 5 of 8 jobs were zombie-restarted and ran twice ("job already completed by another worker"). The batch took ~9 min.occupancy_rate_15s= 1.00 while busy and 0 once idle, unstarted jobs stayedrunning = falseand the autoscaler scaled out on them ("5 jobs waiting >= full scaleout threshold"). 0 zombie restarts, 0 double runs, batch took 320s (8 × 40s).cargo test -p windmill-worker --lib dedicated_worker_counts: an inline job overlapping a dedicated one counts once, and dropping the dedicated guard releases its in-flight job.🤖 Generated with Claude Code
Summary by cubic
Fixes dedicated worker autoscaling, which misread busy workers as idle and double-ran long jobs.
When a dedicated worker's channel is full, the main loop no longer parks on
send().await(which froze the worker's ping and occupancy for a job's duration); it leaves the job queued and keeps looping.DedicatedOccupancyrecords time spent in dedicated subprocesses and adds it to the worker's occupancy for scale-in decisions.One behavior change: a hung dedicated job no longer restarts after 60s; it stays running like a hung regular-worker job.
Not covered: jobs handed to a flow runner still block the main loop and keep their zombie exposure; worth a follow-up.
Written for commit acb2787. Summary will update on new commits.