Skip to content

fix: hand flow previews on an agent's tag to a flow worker - #12563

Merged
rubenfiszel merged 3 commits into
mainfrom
fix-agent-flow-preview-requeue
Nov 20, 2026
Merged

rubenfiszel merged 3 commits into
mainfrom
fix-agent-flow-preview-requeue

Conversation

@rubenfiszel

@rubenfiszel rubenfiszel commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

An agent worker cannot drive a flow. When pull finds a flow job through a caller's own query (the agent endpoint's), it moves the job onto the generic flow tag, and the job keeps its own tag in v2_job so its steps still run on the agent. Flow previews were exempt from this when the preview_tags_override instance setting was on (#8649), so with that setting on, testing a flow on an agent's tag handed the flow preview to the agent, which failed it with Could not handle flow job with agent worker.

Regular workers pull with no custom query and never reach this block, so the exemption only ever affected agents (the worker shell pulls on a worker-name tag no preview carries; AI tool claims pull tool jobs, never flow previews).

Changes

  • pull (windmill-queue/src/jobs.rs): requeue flow previews like any other flow job, override or not.
  • handle_queued_job (windmill-worker/src/worker.rs): the agent-side error, kept as a guard, now says why agents never run flow jobs.
  • EE companion (windmill-labs/windmill-ee-private, branch fix-agent-flow-preview-requeue): drops the agent pull_job todo to requeue flow jobs, which pull covers; ee-repo-ref.txt points to it.
  • windmill-queue/tests/custom_query_flow_requeue_test.rs: with the override on, a flow preview on an agent tag pulled through a custom query lands on flow, not running, with v2_job.tag kept. Fails without the fix.

Test plan

  • E2E against a real agent (tag agentx, MODE=agent) on an EE build with agent_worker_server, override on, flow preview tagged agentx:
    • before: the flow preview ran on ag-agentgrp-* and failed with Could not handle flow job with agent worker
    • after: the flow preview ran on wk-default-*, its step on ag-agentgrp-* with tag agentx, both succeeded
  • New test passes, and fails with the exemption restored
  • cargo check -p windmill-queue -p windmill-worker (CE) clean; EE build in the dev pane clean

🤖 Generated with Claude Code


Summary by cubic

Fixes flow previews failing when tested on an agent's tag: they were handed to the agent worker, which cannot drive flows, and failed with Could not handle flow job with agent worker.

Now flow previews are requeued to the generic flow tag like any other flow job, so a regular worker picks them up while the preview's steps still run on the agent's tag. The agent-side error message now explains why agents never run flow jobs, and the EE companion drops its now-redundant agent pull_job requeue.

  • Adds a regression test that fails without the fix.

Written for commit 234a87b. Summary will update on new commits.

View guided diff Turn on auto-fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 1m 0s —— View job


Good to merge

  • Gather diff and context
  • Review changes
  • Post review

Summary

Removing the PREVIEW_TAGS_OVERRIDE exemption in pull's custom-query requeue block (backend/windmill-queue/src/jobs.rs:4994-5033) is correct and narrowly scoped. Here is what I checked:

  • Only agent pulls change. There are three callers that pass query_o: Some(..):
    • the agent endpoint (windmill-api-agent-workers/src/ee.rs:1255), which uses the agent's tags;
    • the worker shell (windmill-worker/src/worker.rs:2786) and the agent-side shell (ee.rs:1239). Both pull on retrieve_common_worker_prefix(worker_name), and no flow preview carries that tag.
    • Regular workers pass None and never reach this block, so the change has no effect on them.
  • Steps still go to the agent. Only v2_job_queue.tag is rewritten, so v2_job.tag keeps agentx and the steps inherit it. The new test asserts this.
  • The preview tag still works. PREVIEW_TAGS_OVERRIDE is still read at jobs.rs:8051 when the tag is computed at push time. A flow preview with the override on and no explicit tag still lands on preview[-ws]. The only difference is that an agent serving preview now hands it to flow instead of failing it, which matches the stated intent.
  • No new requeue loop. If an agent also served flow, the job would be requeued over and over, but the existing pull_loop_count > 10 cap bounds that. Non-preview flows already had this behaviour, so this PR doesn't introduce it.
  • The worker error stays as a guard. Its new text in handle_queued_job explains why agents never run flow jobs, which is more useful for diagnosis.
  • No new public surface. The diff adds no pub fn or other public API.

AGENTS.md compliance

  • The comment at jobs.rs:4995 records a durable constraint (don't bring the exemption back, because agents can't drive flows) in two lines. That fits "Comments record constraints, not narration."
  • ee-repo-ref.txt was bumped together with the EE companion that drops the stale // todo: requeue with default flow tag if flow job at ee.rs:1280, as docs/enterprise.md asks.
  • The test is a single minimal regression guard that fails without the fix, as "Ship only the tests the PR needs" asks.

Test coverage

  • Backend: the new custom_query_flow_requeue_test.rs covers the changed branch: a flow preview pulled through a custom agent-tag query with the override on. It checks that no job comes back, the queue tag is flow, running = false, and v2_job.tag is kept.
    • It sets the global PREVIEW_TAGS_OVERRIDE to true. That is safe because it is the only test in its own test binary.
    • It uses the existing base fixture, and Cargo picks the test up automatically.
  • Manual verification: the PR's E2E run covers the only in-app path: an EE build with agent_worker_server, an agent on tag agentx, the override on, and a flow preview tagged agentx. It confirms the flow ran on a wk-default-* worker while its step ran on ag-agentgrp-*, and both succeeded. I didn't re-run it here, and nothing beyond it is needed before merge.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found in the queue routing, custom-query callers, or step-tag preservation. No new public surface is introduced. The previous EE revision was unavailable locally, so I could inspect the current agent pull implementation but could not compare the private companion diff.

Test coverage

The new SQLx integration test covers preview requeueing, resetting running, and preserving the agent tag. The author reports it passes and fails with the exemption restored, plus successful CE checks, an EE build, and a real-agent E2E confirming the flow runs on a regular worker and its step on the agent.

I did not rerun these checks. The reported E2E covers the affected runtime path; no additional manual scenario is needed.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

cc @rubenfiszel

Mergeable, but should ideally address nits: misleading comment in backend/windmill-queue/src/jobs.rs.

Findings

P2 — backend/windmill-queue/src/jobs.rs:4995 — comment misstates the scenario the fix handles

// An agent worker cannot drive a flow, so this holds for flow previews on
// the preview tag too: the job keeps its tag in v2_job for its steps.
if job.is_flow() || job.is_dependency() {

"Flow previews on the preview tag" doesn't match the case this branch actually rescues. This block only fires when the job is pulled through a caller-supplied query (query_o), and the only such pullers are agents (their own tags) and the worker shell (its common_worker_prefix). A flow preview sitting on the literal preview / preview-<ws> tag is never pulled by an agent's query, so it can't reach this block. What actually breaks — and what the fix and test target — is a flow preview explicitly tagged with the agent's own tag (e.g. agentx). Suggest rewording to something like "…so flow previews under the preview-tags override are requeued too when they land on an agent's tag". The second clause ("keeps its tag in v2_job for its steps") is accurate and worth keeping.

Considered and dismissed

  • Dedicated-worker flow preview regression — I traced JobPayload::RawFlow → JobPayloadUntagged (jobs.rs:7437): dedicated_worker defaults to None, so on_dedicated_worker is always false for flow previews and the dedicated-worker tag path is never taken. No preview carries a dedicated tag that this change could strip.
  • Worker-shell path — the shell (worker.rs:2790) pulls on common_worker_prefix (e.g. wk-<group>); preview tags are preview/preview-<ws>, language, or explicit tags, so a preview only reaches the shell if explicitly tagged wk-<group>. In that case requeueing to flow is the same behavior as every non-preview flow already gets, and a regular worker still drives it. No regression.
  • Suspended-state loss on requeue — the requeue only rewrites tag/running and preserves suspended, identical to the pre-existing non-preview path. Not introduced here.
  • Test leaks global PREVIEW_TAGS_OVERRIDE — it stores true without resetting, but no other test in the suite reads that flag, so there's no cross-test interference today.
  • Error-message change in worker.rs:5317 — pure string change; the guard (conn.as_sql() is None → agent) is unchanged and the new text accurately states the intended routing.

Test coverage

Backend: the new custom_query_flow_requeue_test.rs is a well-formed sqlx::test regression guard that reproduces the exact failure (asserts pulled.job.is_none(), v2_job_queue.tag == "flow", running == false, and v2_job.tag == "agentx" preserved for steps). I traced it against push/pull signatures and the RawFlow → FlowPreview mapping; it correctly fails with the old exemption restored. No additional unit tests are warranted — this is a one-line behavior change fully covered by the integration test.

Manual verification still needed: the agent-side EE companion (pull_job requeue drop, branch fix-agent-flow-preview-requeue) lives in the private repo and can't be exercised from this checkout; the PR body's E2E (real agent agentx, MODE=agent, override on, flow preview tagged agentx running on wk-default-* with its step on ag-agentgrp-*) covers that path, but it should be re-confirmed against the EE commit the updated ee-repo-ref.txt points to before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: 234a87b
Status: ✅  Deploy successful!
Preview URL: https://5e70f9aa.windmill.pages.dev
Branch Preview URL: https://fix-agent-flow-preview-reque.windmill.pages.dev

View logs

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

✅ Review round clean @ fc01deb — nit-only verdicts at 8652706; nits addressed in fc01deb

@rubenfiszel
rubenfiszel marked this pull request as ready for review November 20, 2026 10:50
This commit updates the EE repository reference after PR #1000 was merged in windmill-ee-private.

Previous ee-repo-ref: 5b81b24e5fca5933b23a830fef0c5ae6f36de11a

New ee-repo-ref: 57c9ae5b1b5c594fcdd9b398052a24006a367117

Automated by sync-ee-ref workflow.
@windmill-internal-app

Copy link
Copy Markdown
Contributor

🤖 Updated ee-repo-ref.txt to 57c9ae5b1b5c594fcdd9b398052a24006a367117 after windmill-ee-private PR #1000 was merged.

@windmill-internal-app windmill-internal-app Bot changed the title [ee] fix: hand flow previews on an agent's tag to a flow worker fix: hand flow previews on an agent's tag to a flow worker Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

Good to merge

The one P2 from my prior pass (the misleading pull comment) is fixed in fc01deb; the comment now names the exact scenario the branch rescues. Re-verified the rest of the diff against the current head (234a87b).

Considered and dismissed

  • PREVIEW_TAGS_OVERRIDE leak in the new test (custom_query_flow_requeue_test.rs:17) — the flag is store(true)-ed and never reset. #[sqlx::test] gives each test file its own process, and PREVIEW_TAGS_OVERRIDE is only loaded at jobs.rs:8052 during push; no other test in this binary reads it, so there is no cross-test interference. Dismissed.
  • Requeue loop if an agent also serves flow — the requeue always lands flow jobs on the generic flow/flow-<ws> tag (configured_dependency_job_tag returns None for non-dependency kinds, so flow previews take the flow base tag). An agent that also serves flow would pull and requeue repeatedly, but this is pre-existing behavior for all non-preview flows and is bounded by the existing pull_loop_count > 10 cap at jobs.rs:4952. Not introduced here. Dismissed.
  • Script previews (JobKind::Preview) under the override — is_dependency() matches no preview variant and is_flow() excludes Preview, so only FlowPreview is newly requeued; script previews still reach their worker as before. The change is exactly scoped. Dismissed.
  • Suspended-state / other fields on requeue — the UPDATE v2_job_queue SET tag = $1, running = false at jobs.rs:5020 rewrites only tag and running, identical to the pre-existing non-preview path; suspended and everything else is preserved. Dismissed.
  • Error-message wording (worker.rs:5317) — the guard (conn.as_sql() is None → agent) is unchanged; the new text accurately states that the server routes a flow on an agent's tag to a regular worker serving the flow tag, which is exactly what the requeue does. Dismissed.
  • Comment accuracy (jobs.rs:4995) — "a flow preview on an agent's tag is requeued too, preview tags override or not" correctly describes the removed exemption (job.kind.is_preview() && PREVIEW_TAGS_OVERRIDE) and the block's reach (only agents pull with a caller query on their own tags). No longer conflates the literal preview tag with the agent's tag. Dismissed.
  • Test genuinely guards the regression — I traced the old condition: a RawFlow push maps to JobKind::FlowPreview (is_flow() and is_preview() both true), the override is on, and the explicit agentx tag makes the preview land on agentx (the override only supplies preview when no explicit tag is given, jobs.rs:8051). With the old exemption the agent would have received the job, so assert!(pulled.job.is_none()) fails; with the fix it passes. The queue_tag/running/job_tag assertions match the requeue's UPDATE and the untouched v2_job.tag. Dismissed.

Test coverage

  • Backend: the new custom_query_flow_requeue_test.rs is a minimal sqlx::test regression guard that exercises the changed branch directly and fails with the old exemption restored. It reuses the established migrations = "../migrations" / fixtures("base") pattern, so Cargo discovers it automatically. No additional unit tests are warranted for a one-condition change.
  • Manual verification: the EE companion (agent-side pull_job requeue drop) was previously the only unexercised path; ee-repo-ref.txt now points to 57c9ae5b1b5c594fcdd9b398052a24006a367117, the merged head of windmill-ee-private PR chore(main): release 1.54.0 #1000, so that side is landed. The PR's real-agent E2E (tag agentx, MODE=agent, override on, flow preview tagged agentx running on wk-default-* with its step on ag-agentgrp-*) covers the only in-app path and needs no further re-run beyond the author's report.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found at 234a87b. The corrected comment matches the routing behavior, and the current EE agent endpoint delegates to the shared requeue logic. No new public API is introduced. Earlier private EE revisions are unavailable locally, so I could inspect the current companion implementation but could not compare its history.

Test coverage

The SQLx regression test covers preview requeueing, clearing running, and preserving the agent tag for steps. The author reports it passes and fails with the old exemption restored, plus successful CE checks, an EE build, and a real-agent E2E confirming the flow and its step execute on their intended workers.

I ran git diff --check successfully. I did not rerun Rust tests or E2E; local Postgres is unavailable. The reported E2E covers the affected runtime path; no additional manual scenario is needed.

@rubenfiszel
rubenfiszel merged commit e061d11 into main Nov 20, 2026
12 checks passed
@rubenfiszel
rubenfiszel deleted the fix-agent-flow-preview-requeue branch November 20, 2026 11:10
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant