Repository navigation
feat: record and download a worker group's autoscaling decisions - #12542
Conversation
|
/review |
1 similar comment
|
/review |
|
Claude finished @rubenfiszel's task in 1m 22s —— View job Good to merge No P0, P1 or P2 issues in the public-repo part of this PR. One limit on what I could review:
What I checked
AGENTS.md compliance
I found no violations. Test coverageBackend: no Rust tests were added. The new handlers are thin CRUD behind an existing auth helper, and the real logic (the decision text on each path of Frontend: this PR adds a component, not a pure-logic utility, so I'm not asking for tests. Manual verification still open (also left unchecked in the PR body): sign in as a non-superadmin user without the devops role, open the Workers page, and expand a worker group's Autoscaling events. "Record decisions" and "Download" should not appear. Calling |
|
/codex |
|
/review |
…on-diagnostics # Conflicts: # backend/ee-repo-ref.txt
|
/review |
|
Claude finished @rubenfiszel's task in 48s —— View job Good to merge The public-repo changes are the same as in the last review: I found nothing new to flag.
What changed since the last reviewThe only new commit is The rest of the PR (migration,
What I could not reviewThe private checkout this job has is at AGENTS.md complianceNo violations:
Test coverageBackend: no Rust tests were added. The CE handlers are thin CRUD behind an existing auth helper, so I don't think a CE integration test is needed. The guard worth having is an EE-crate test that a recorded group gets exactly one decision row per tick, including on the skip and error paths. Frontend: this PR adds a component, not a pure-logic utility, so no tests are expected. Manual verification still open: sign in as a user who is neither superadmin nor devops and open the Workers page. Expanding a worker group's Autoscaling events should show neither "Record decisions" nor "Download". Calling |
Codex ReviewGood to merge No issues found in the pinned public diff or the inspected EE implementation. The private checkout matches Test coverageNo backend unit or integration tests were added for recording, expiry, or skip/error paths. A focused integration regression test would be useful. No frontend component tests are expected.
Still verify the Workers page as a user who is neither superadmin nor devops: recording and download controls should be hidden, and direct requests to both new endpoints should be denied. Also verify expiry by expiring a recording and aging a decision past 48 hours: subsequent ticks should stop recording that group, and cleanup should remove the aged decision while retaining newer decisions. |
Pi ReviewGood to merge I reviewed the public-repo portion of this PR (migration, Considered and dismissed
Test coverageBackend: no Rust tests were added. The CE handlers are thin CRUD behind the existing Frontend: this PR adds Svelte components, not a pure-logic utility with a sibling CI / migration / OpenAPI / docs: no automated tests expected; the Manual verification still neededThe PR body leaves one scenario unchecked, and I concur it is the one worth exercising before merge: sign in as a user who is neither superadmin nor devops, open the Workers page and expand a worker group's Autoscaling events header — neither "Record decisions" nor "Download" should appear, and a direct |
This commit updates the EE repository reference after PR #999 was merged in windmill-ee-private. Previous ee-repo-ref: e2232bda0cea3c1abcdac88140f48f6c7d90d15d New ee-repo-ref: cac1d382a7c22e8a4e9c56832ad12d11de5bba7f Automated by sync-ee-ref workflow.
|
🤖 Updated |
|
Claude finished @rubenfiszel's task in 54s —— View job Good to merge The only commit since my last review is the bot bump of
EE side (
|
Summary
When a worker group does not scale the way someone expects ("workers stayed up for 42 hours", "it never scaled out"), there is nothing to look at after the fact.
autoscaling_eventonly records actions taken. The ticks where the autoscaler decided not to act, and the inputs it decided on (worker count, occupancy, jobs waiting per tag), go only to the server log atinfolevel.This adds an on-demand recording of every autoscaling tick of a worker group, and an export from the Workers page that a customer can send us.
no change: within cooldown, last Scaleout 20s ago, cooldown 30sno change: count 1 (min 1, max 3), occupancy 12.0/8.3/9.1% (scale in below 25%, scale out above 75%), no tag at 3 jobs waitingno change: autoscaling is disabled or the worker group is pausedChanges
autoscaling_recording(group →until,started_by) andautoscaling_decision(one row per recorded tick), indexed on(worker_group, recorded_at).autoscaling_ee.rs(windmill-ee-private companion PR):apply_autoscalingnow returns its decision as text on every path. Each formeradd_autoscaling_event(..).await?; return Ok(())becamereturn add_autoscaling_event(..).await, so the scaling actions are unchanged. The cooldown message now carries the numbers.apply_all_autoscalingwrites a row for each recorded group: the decision, anerror: …when the tick failed, or why the group was skipped (disabled, paused, or invalid config).windmill-api-configs: two endpoints, both requiring the devops role.GET /configs/autoscaling_diagnostics/{worker_group}returns the recording state, the decision count and the decisions, oldest first.include_decisions=falsereturns the state and count only.POST /configs/autoscaling_recording/{worker_group}with{record: bool}starts (or extends) a 24h recording, or stops it. Both are audit-logged.monitor.rs: the sweep deletes decisions older than 48h and expired recordings.AutoscalingDiagnostics.sveltesits in the Autoscaling events header (EE, devops/superadmin only). It shows "Record decisions", or "Recording until … · Stop", plus "Download". The count refreshes with the panel's refresh button, and every 30s while recording.feature_usage):autoscaling_diagnostics/recording_startedanddownloaded, registered inFEATURE_USAGE_KINDS(EE). Plain counters, so no disclosure change.Screenshots
Test plan
dediwith DryRun autoscaling (min 1, max 3), recording started through the API. A row landed per tick, for exampleno change: within cooldown, last Scaleout 20s ago, cooldown 30s,no change: count 1 (min 1, max 3), occupancy -/-/-% …, andFull_Scaleout to 3: count of 0 < min_workers of 1 but queue >= full scaleout threshold of 3. This branch doesn't include fix: keep dedicated workers counted, busy and out of zombie restarts #12540, so the rows also show that bug: a count of 0 while the worker was busy.reportsgroup, which has no autoscaling, writesno change: autoscaling is disabled or the worker group is paused.autoscaling-dedi-<time>.json, holding the autoscaling config, the version, the recording window and all decisions oldest first. Stop → back to "Record decisions", with Download still available.include_decisions=falsereturns the count only.worker_config.autoscaling_recording_startand_stopappear in the audit log, and afeature_usagerow forautoscaling_diagnostics/recording_startedlanded.SQLX_OFFLINE=true cargo check --workspace --features quickjs,enterprise,private,license, plus the CEcargo check -p windmill-api-configs -p windmill-autoscaling. The 8 new.sqlxentries are exactly the new queries.npm run check:fast: no errors in the changed files (the 4 that remain are onmain).$superadmin || $devopsRolethe Workers page uses for config actions.🤖 Generated with Claude Code
Summary by cubic
Records every autoscaling decision of a worker group—including ticks where it decided not to scale, plus the inputs (worker count, occupancy, jobs waiting per tag) it based that on—so scaling behavior can be diagnosed after the fact. Previously those details only went to info-level logs, and
autoscaling_eventrecorded actions taken, not the reasoning.Details
autoscaling_recordingandautoscaling_decisiontables (one row per tick, indexed on(worker_group, recorded_at)) and a 48h retention sweep inmonitor.rs.GET /configs/autoscaling_diagnostics/{worker_group}andPOST /configs/autoscaling_recording/{worker_group}with{record: bool}, both devops-role gated and audit-logged.Written for commit ae7e1be. Summary will update on new commits.