Skip to content

feat: record and download a worker group's autoscaling decisions - #12542

Merged
rubenfiszel merged 4 commits into
mainfrom
rf/autoscaling-decision-diagnostics
Nov 19, 2026
Merged

rubenfiszel merged 4 commits into
mainfrom
rf/autoscaling-decision-diagnostics

Conversation

@rubenfiszel

@rubenfiszel rubenfiszel commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a worker group does not scale the way someone expects ("workers stayed up for 42 hours", "it never scaled out"), there is nothing to look at after the fact. autoscaling_event only records actions taken. The ticks where the autoscaler decided not to act, and the inputs it decided on (worker count, occupancy, jobs waiting per tag), go only to the server log at info level.

This adds an on-demand recording of every autoscaling tick of a worker group, and an export from the Workers page that a customer can send us.

  • Opt-in and self-expiring. "Record decisions" records a group for 24h, then stops by itself, so it can't be forgotten on. Rows are kept 48h, so a recording can still be downloaded a day after it ends. A group writes one row per tick, about 4,300-5,800 rows (a few MB) per recorded day, and nothing when no recording is running.
  • Every tick, with its reason. A row holds the worker count, the 15s/5m/30m occupancy, the jobs waiting on each of the group's tags, and the decision. Decisions that change nothing carry their reason too, for example:
    • no change: within cooldown, last Scaleout 20s ago, cooldown 30s
    • no change: count 1 (min 1, max 3), occupancy 12.0/8.3/9.1% (scale in below 25%, scale out above 75%), no tag at 3 jobs waiting
    • no change: autoscaling is disabled or the worker group is paused
  • Download gives a JSON file with the recorded decisions, the group's autoscaling config, the server version and the recording window.

Changes

  • Migration: autoscaling_recording (group → until, started_by) and autoscaling_decision (one row per recorded tick), indexed on (worker_group, recorded_at).
  • EE autoscaling_ee.rs (windmill-ee-private companion PR):
    • apply_autoscaling now returns its decision as text on every path. Each former add_autoscaling_event(..).await?; return Ok(()) became return add_autoscaling_event(..).await, so the scaling actions are unchanged. The cooldown message now carries the numbers.
    • apply_all_autoscaling writes a row for each recorded group: the decision, an error: … when the tick failed, or why the group was skipped (disabled, paused, or invalid config).
    • Reading which groups record is diagnostics only: if it fails, it is logged and scaling goes on.
  • windmill-api-configs: two endpoints, both requiring the devops role.
    • GET /configs/autoscaling_diagnostics/{worker_group} returns the recording state, the decision count and the decisions, oldest first. include_decisions=false returns the state and count only.
    • POST /configs/autoscaling_recording/{worker_group} with {record: bool} starts (or extends) a 24h recording, or stops it. Both are audit-logged.
  • monitor.rs: the sweep deletes decisions older than 48h and expired recordings.
  • Frontend: AutoscalingDiagnostics.svelte sits in the Autoscaling events header (EE, devops/superadmin only). It shows "Record decisions", or "Recording until … · Stop", plus "Download". The count refreshes with the panel's refresh button, and every 30s while recording.
  • Telemetry (feature_usage): autoscaling_diagnostics / recording_started and downloaded, registered in FEATURE_USAGE_KINDS (EE). Plain counters, so no disclosure change.

Screenshots

autoscaling-recording

Test plan

  • EE build, a dedicated group dedi with DryRun autoscaling (min 1, max 3), recording started through the API. A row landed per tick, for example no change: within cooldown, last Scaleout 20s ago, cooldown 30s, no change: count 1 (min 1, max 3), occupancy -/-/-% …, and Full_Scaleout to 3: count of 0 < min_workers of 1 but queue >= full scaleout threshold of 3. This branch doesn't include fix: keep dedicated workers counted, busy and out of zombie restarts #12540, so the rows also show that bug: a count of 0 while the worker was busy.
  • Recording the reports group, which has no autoscaling, writes no change: autoscaling is disabled or the worker group is paused.
  • UI (Playwright): Record decisions → "Recording until …" with Stop and Download. Download produced autoscaling-dedi-<time>.json, holding the autoscaling config, the version, the recording window and all decisions oldest first. Stop → back to "Record decisions", with Download still available. include_decisions=false returns the count only.
  • worker_config.autoscaling_recording_start and _stop appear in the audit log, and a feature_usage row for autoscaling_diagnostics / recording_started landed.
  • SQLX_OFFLINE=true cargo check --workspace --features quickjs,enterprise,private,license, plus the CE cargo check -p windmill-api-configs -p windmill-autoscaling. The 8 new .sqlx entries are exactly the new queries.
  • npm run check:fast: no errors in the changed files (the 4 that remain are on main).
  • Not exercised: a user without the devops role seeing the controls hidden. The gate is the same $superadmin || $devopsRole the Workers page uses for config actions.

🤖 Generated with Claude Code


Summary by cubic

Records every autoscaling decision of a worker group—including ticks where it decided not to scale, plus the inputs (worker count, occupancy, jobs waiting per tag) it based that on—so scaling behavior can be diagnosed after the fact. Previously those details only went to info-level logs, and autoscaling_event recorded actions taken, not the reasoning.

Details

  • Recording is opt-in per worker group, lasts 24h, and stops by itself; decisions are kept for 48h after the recording window.
  • Devops/admin users can start and stop a recording and download the full set of decisions as JSON from the Autoscaling events header on the Workers page; the controls show to those roles only.
  • Adds autoscaling_recording and autoscaling_decision tables (one row per tick, indexed on (worker_group, recorded_at)) and a 48h retention sweep in monitor.rs.
  • The EE autoscaler returns its decision as text on every path, so no-change ticks record their reason (cooldown, idle, disabled); reading recording state is best-effort and never blocks scaling.
  • Adds GET /configs/autoscaling_diagnostics/{worker_group} and POST /configs/autoscaling_recording/{worker_group} with {record: bool}, both devops-role gated and audit-logged.

Written for commit ae7e1be. Summary will update on new commits.

View guided diff Turn on auto-fix

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

1 similar comment
@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: ae7e1be
Status:⚡️  Build in progress...

View logs

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 1m 22s —— View job


Good to merge

No P0, P1 or P2 issues in the public-repo part of this PR. One limit on what I could review: autoscaling_ee.rs is in the companion windmill-ee-private PR, and the private checkout this job had is at an older ref than ee-repo-ref.txt (f8a140a9). The tick-side writes and the until > now() check on the writer side are therefore unreviewed here.

  • Gather context (diff, AGENTS.md)
  • Review backend (migration, API, monitor, OpenAPI)
  • Review frontend
  • Post review

What I checked

  • Authorization: both new handlers call require_devops_role. That check refuses $WM_TOKEN job tokens and requires the devops email, so the new endpoints are protected the same way as the other /configs autoscaling routes. worker_group only ever reaches the database as a bound parameter.
  • Writes and audit: in set_autoscaling_recording, the upsert or delete and the audit_log row are written in the same transaction. Re-recording replaces both until and started_by, which gives the "extends" behaviour the PR body describes.
  • Cleanup in monitor.rs: the sweep deletes old decisions inside a WITH … DELETE that the outer statement never references. That still works, because Postgres always runs a data-modifying CTE exactly once. Both deletes happen in one round trip, and a failure is logged rather than propagated, like the autoscaling_event sweep above it.
  • Row cap: MAX_AUTOSCALING_DECISIONS = 12_000 is above 48h of 15s ticks (11,520), so a full window is never cut off. Ordering DESC LIMIT and then calling reverse() returns the newest rows, oldest first.
  • OpenAPI: the AutoscalingDiagnostics and AutoscalingDecision schemas match the Rust structs: nullable occupancy fields, int64 count, jobs_waiting as an object, text/plain for the POST response.
  • Frontend:
    • The 30s poll only runs while recordingUntil is set. It stops by itself once the recording expires, because loadState then clears recordingUntil.
    • Bumping refreshKey reloads the state, and untrack keeps loadState from also reacting to its own writes.
    • The controls are gated on $superadmin || $devopsRole inside the existing $enterpriseLicense block.
  • Telemetry docs: 109 kinds and 42 features in the local FEATURE_USAGE_KINDS, plus the two new kinds and one new feature, gives 111 and 43, matching the doc. Adding code_mode to the doc fixes an entry that was missing before this PR.

AGENTS.md compliance

  • Telemetry is proposed and registered for a new user-facing feature.
  • The migration comes as an up/down pair, and summarized_schema.txt is updated.
  • The .sqlx entries were added.
  • Comments explain why a constraint exists rather than narrating the code.

I found no violations.

Test coverage

Backend: no Rust tests were added. The new handlers are thin CRUD behind an existing auth helper, and the real logic (the decision text on each path of apply_autoscaling) is in the EE crate. I don't think a CE integration test is required. The most useful guard would be a test in the EE crate pinning that a recorded group gets exactly one row per tick on the skip and error paths.

Frontend: this PR adds a component, not a pure-logic utility, so I'm not asking for tests.

Manual verification still open (also left unchecked in the PR body): sign in as a non-superadmin user without the devops role, open the Workers page, and expand a worker group's Autoscaling events. "Record decisions" and "Download" should not appear. Calling POST /api/configs/autoscaling_recording/<group> directly as that user should return 401 with the "requires the caller to have the devops role" message.

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/codex

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

…on-diagnostics

# Conflicts:
#	backend/ee-repo-ref.txt
@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 48s —— View job


Good to merge

The public-repo changes are the same as in the last review: I found nothing new to flag.

  • Gather context (diff since the last review)
  • Review the changes
  • Post review

What changed since the last review

The only new commit is f3d6e0e, a merge of origin/main. Within the PR's own files, it changes just one line: backend/ee-repo-ref.txt now pins e2232bda instead of f8a140a9.

The rest of the PR (migration, windmill-api-configs handlers, the monitor.rs sweep, OpenAPI, the two Svelte components, telemetry docs) is byte-identical to 64c1b0a, which the previous review covered. Re-checking it turned up nothing new:

  • Both handlers call require_devops_role.
  • In the POST handler, the change and its audit-log row are written in one transaction.
  • worker_group only reaches the database as a bound parameter.
  • The 12,000-row cap stays above 48h of 15s ticks (11,520).

What I could not review

The private checkout this job has is at f9654a5d, not the pinned e2232bda. It has no references to autoscaling_decision or autoscaling_recording. So the EE side of the feature, autoscaling_ee.rs, is still unreviewed here. That covers the per-tick decision text, the skip and error rows, and reading which groups record without blocking scaling. It should be reviewed on the windmill-ee-private companion PR.

AGENTS.md compliance

No violations:

  • Telemetry is registered for the new user-facing feature.
  • The migration comes as an up/down pair, and summarized_schema.txt is updated.
  • The .sqlx entries were added.
  • Comments explain constraints rather than narrating the code.

Test coverage

Backend: no Rust tests were added. The CE handlers are thin CRUD behind an existing auth helper, so I don't think a CE integration test is needed. The guard worth having is an EE-crate test that a recorded group gets exactly one decision row per tick, including on the skip and error paths.

Frontend: this PR adds a component, not a pure-logic utility, so no tests are expected.

Manual verification still open: sign in as a user who is neither superadmin nor devops and open the Workers page. Expanding a worker group's Autoscaling events should show neither "Record decisions" nor "Download". Calling POST /api/configs/autoscaling_recording/<group> directly as that user should return 401 with the message that the call requires the devops role.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found in the pinned public diff or the inspected EE implementation. The private checkout matches e2232bda; its shallow history lacks the previous pin, so a complete EE before/after comparison was unavailable.

Test coverage

No backend unit or integration tests were added for recording, expiry, or skip/error paths. A focused integration regression test would be useful. No frontend component tests are expected.

git diff --check passed. I did not run builds, application tests, or running-instance checks. The PR reports EE/CE compilation checks, frontend checks, real DryRun recording, and Playwright verification of recording, downloading, and stopping.

Still verify the Workers page as a user who is neither superadmin nor devops: recording and download controls should be hidden, and direct requests to both new endpoints should be denied.

Also verify expiry by expiring a recording and aging a decision past 48 hours: subsequent ticks should stop recording that group, and cleanup should remove the aged decision while retaining newer decisions.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

Good to merge

I reviewed the public-repo portion of this PR (migration, windmill-api-configs handlers, monitor.rs sweep, OpenAPI, telemetry docs, and both Svelte components). No P0/P1/P2 issues in what this checkout contains. One review limitation, not a finding: the autoscaler-side logic (apply_autoscaling returning decision text on every path, apply_all_autoscaling writing the per-tick rows and the skip/error rows, and the best-effort read of which groups record) lives in autoscaling_ee.rs in the windmill-ee-private companion PR pinned at e2232bda, which is not checked out here — there is no autoscaling_ee.rs in this repo. That side should be reviewed on the companion PR.

Considered and dismissed

  • No index on recorded_at for the retention sweep (monitor.rs:2155): the DELETE ... WHERE recorded_at <= now() - interval '48 hours' runs every 10s monitor tick and can only seq-scan the table (the only index leads with worker_group). Dismissed because the table is opt-in and self-expiring — the PR body budgets a few thousand rows per recorded group (a few MB per day) — so a seq scan of that is sub-millisecond, and the existing autoscaling_event sweep above it follows the same no-special-index pattern.
  • decision_count recomputed on every poll: get_autoscaling_diagnostics runs count(*) on each 30s poll and refresh. Dismissed — WHERE worker_group = $1 is served by the new (worker_group, recorded_at) index, and the row counts are small by design.
  • MAX_AUTOSCALING_DECISIONS = 12_000 vs the "oldest first" claim: the DESC LIMIT 12000 then reverse() returns newest-12000 in oldest-first order. Dismissed — 12,000 exceeds 48h of 15s ticks (11,520), so a full retention window is never truncated.
  • "Extends" semantics of POST ... {record: true} (lib.rs:492): the upsert re-arms until = now() + 24h rather than adding to remaining time. Dismissed — re-arming to a fresh 24h is the sensible reading of "starts (or extends)" and matches the previous review.
  • No validation that worker_group exists: a typo'd group starts a recording that writes nothing. Dismissed — the endpoint is devops-gated; nothing harmful results.
  • recording_started_by exposes the starter's email: devops-only endpoint returning an admin's own email to the same role; not a disclosure.
  • Auth: both handlers call require_devops_role, which also rejects $WM_TOKEN job tokens and is capped per GHSA-hfh4-cx4h-3fcr — verified in windmill-api-auth/src/lib.rs:805.
  • Injection/traversal: worker_group reaches the database only as a bound parameter; make_interval(secs => $2) binds AUTOSCALING_RECORDING_SECS as f64 as a constant, not caller input.
  • Audit correctness: the upsert/delete and audit_log share one transaction (&mut *tx), consistent with delete_config above.
  • Telemetry doc arithmetic: 109 local kinds + 2 new = 111 actions; 42 + 1 new feature = 43 features; code_mode was already missing from the doc and is added here.
  • Generated client absent from diff: frontend/src/lib/gen is gitignored and regenerated from openapi.yaml; the new operationIds (getAutoscalingDiagnostics, setAutoscalingRecording) match the component's ConfigService calls.
  • Svelte 5 event/effect wiring: Button accepts onClick as a prop; the first $effect tracks only refreshKey/worker_group and runs loadState inside untrack, so its own $state writes don't re-trigger it; the 30s poll tears down when recordingUntil clears after expiry. All correct.

Test coverage

Backend: no Rust tests were added. The CE handlers are thin CRUD behind the existing require_devops_role helper, so a CE integration test would add little; the logic worth pinning (one decision row per tick, including the skip and error paths) is in the EE crate and belongs on the companion PR as an EE-crate test.

Frontend: this PR adds Svelte components, not a pure-logic utility with a sibling *.test.ts, so no component tests are expected.

CI / migration / OpenAPI / docs: no automated tests expected; the .sqlx offline entries and summarized_schema.txt are updated, and the PR body reports the offline and CE/EE cargo check runs passing.

Manual verification still needed

The PR body leaves one scenario unchecked, and I concur it is the one worth exercising before merge: sign in as a user who is neither superadmin nor devops, open the Workers page and expand a worker group's Autoscaling events header — neither "Record decisions" nor "Download" should appear, and a direct POST /api/configs/autoscaling_recording/<group> as that user should return 401 with the "requires the caller to have the devops role" message. Separately, the tick-side behavior (a row landing per tick with the no-change reasons, and the download JSON shape) is already exercised per the test plan but only on the EE side, which I could not re-verify from this checkout.

This commit updates the EE repository reference after PR #999 was merged in windmill-ee-private.

Previous ee-repo-ref: e2232bda0cea3c1abcdac88140f48f6c7d90d15d

New ee-repo-ref: cac1d382a7c22e8a4e9c56832ad12d11de5bba7f

Automated by sync-ee-ref workflow.
@windmill-internal-app

Copy link
Copy Markdown
Contributor

🤖 Updated ee-repo-ref.txt to cac1d382a7c22e8a4e9c56832ad12d11de5bba7f after windmill-ee-private PR #999 was merged.

@windmill-internal-app windmill-internal-app Bot changed the title [ee] feat: record and download a worker group's autoscaling decisions feat: record and download a worker group's autoscaling decisions Nov 19, 2026
@rubenfiszel
rubenfiszel marked this pull request as ready for review November 19, 2026 17:33
@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 54s —— View job


Good to merge

The only commit since my last review is the bot bump of backend/ee-repo-ref.txt to cac1d382 (windmill-ee-private #999 merged). The rest of the public diff is the same as at 64c1b0a, which the earlier reviews covered. This job's private checkout does contain the companion code, so this pass also reviews the EE side the previous two could not: windmill-autoscaling/src/autoscaling_ee.rs.

  • Gather context (diff since the last review, EE checkout)
  • Review the EE autoscaler changes
  • Post review

EE side (autoscaling_ee.rs)

  • Scaling behaviour is unchanged. Each action path in apply_autoscaling is now return add_autoscaling_event(..).await. add_autoscaling_event still runs the integration first and writes the autoscaling_event row second, so cooldowns read the same table as before. The new decision string is built only after the integration succeeds.
  • The recording is read best-effort. If the SELECT worker_group FROM autoscaling_recording WHERE until > now() query fails, the error is logged and an empty set is used (:170-180). If record_decision fails, the error is logged too (:208-213, record_skipped). A diagnostics failure therefore never changes a scaling decision. The until > now() filter is what makes a recording stop after 24h, and it is applied on the writer side as the PR says.
  • Every tick writes one row per recorded group, covering these cases:
    • a valid config: the decision, or error: … when apply_autoscaling fails;
    • a config that fails to parse: the autoscaling config is invalid;
    • a group the config query leaves out: disabled or the worker group is paused, computed as recorded.difference(&seen);
    • a group with no integration, or with min_workers > max_workers: the matching no change: text.
  • The values recorded are the values used. record_decision receives the same count, occupancy, tags and queue_counts that apply_autoscaling decided on. jobs_waiting is limited to the group's own tags, so the export holds no other group's queue data.
  • Inputs are bound. worker_group and the decision text reach SQL only as bound parameters.

Considered and dismissed

  • An error in one group stops recording for the groups after it. decision? (:215) still returns early, as the old apply_autoscaling(..).await? did. When it does, recorded groups later in the loop, and the "disabled or paused" groups, get no row for that tick. The group that failed does get its error: … row first, and the early return already existed before this PR. So the only effect is a missing diagnostic row for one tick, not a regression. If complete per-tick coverage matters, collecting the first error and returning it after the loop and the skipped-groups block would close this gap.
  • Nothing is recorded on a server that does not hold the lock. That is correct: only the server holding the autoscaling lock decides, so only it should record.

AGENTS.md compliance

No violations. Telemetry is registered for the new user-facing feature. The migration comes as an up/down pair, summarized_schema.txt is updated, and the .sqlx entries are present. The comments state constraints, for example "Diagnostics only: failing to read which groups record must not stop the scaling".

Test coverage

Backend: no Rust tests were added in either repo. The CE handlers are thin CRUD behind require_devops_role, so I don't think they need an integration test. The test worth adding would go in the EE crate: one recorded group per path (decided, error, invalid config, disabled or paused), asserting exactly one autoscaling_decision row per tick. That would pin the "a row every tick" behaviour the export depends on.

Frontend: this PR adds Svelte components, not a pure-logic utility, so no tests are expected.

Manual verification still open: this is the scenario the PR body leaves unchecked. Sign in as a user who is neither superadmin nor devops, open the Workers page and expand a worker group's Autoscaling events. Neither "Record decisions" nor "Download" should appear. A direct POST /api/configs/autoscaling_recording/<group> from that user should return 401 with the message that the call requires the devops role.

@rubenfiszel
rubenfiszel merged commit 5583c68 into main Nov 19, 2026
23 of 26 checks passed
@rubenfiszel
rubenfiszel deleted the rf/autoscaling-decision-diagnostics branch November 19, 2026 17:34
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant