Skip to content

fix: relax git auto-pull polling only once the webhook has delivered - #12532

Merged
rubenfiszel merged 7 commits into
mainfrom
airgapped-github-webhook-polling-fallback
Nov 20, 2026
Merged

rubenfiszel merged 7 commits into
mainfrom
airgapped-github-webhook-polling-fallback

Conversation

@hugocasa

@hugocasa hugocasa commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Git auto-pull relaxed its safety-net poll from ~1 min to ~10 min as soon as a repository webhook was registered. Registering only proves Windmill can reach the git host; it says nothing about the host reaching Windmill. On an instance that can call GitHub but that GitHub cannot call back (a typical air-gapped setup), the hook was created, every delivery failed, and new commits took up to 10 minutes to deploy, while the card claimed "Syncing instantly via webhook".

Polling now relaxes only once the hook has actually delivered to this instance, and drops back to every minute when a poll finds a head no delivery brought.

Changes

  • AutoPullSettings.webhook_verified_id: the webhook_id a delivery last arrived for. It is server-owned: sanitized from client saves, carried across settings saves, and stripped from the settings export. A newly registered hook has a new id, so it starts unverified without touching the hook create/delete paths.
  • effective_poll_interval_s relaxes to 600s only when webhook_verified_id == webhook_id.
  • GitHub and GitLab receivers (EE) record the id after a delivery authenticates. This happens once per hook, not per delivery, and is best-effort so it can never fail a delivery. GitHub's ping on hook creation verifies a reachable instance within seconds. GitLab sends no ping, so a GitLab repo polls every minute until its first push delivery.
  • The poller clears the id when it has to enqueue a pull for a verified hook, meaning the webhook missed that push. It skips this while the last pull failed: a failed pull rolls its synced sha back, which would otherwise look like a missed delivery and retry the failing pull every minute.
  • set_auto_pull_webhook_verified_id does a targeted jsonb write, guarded against jsonb_set NULLing git_sync.
  • Repository card: says "Syncing instantly via webhook" only for a verified hook. Otherwise it says it checks every minute until the webhook delivers.

Known ceilings:

  • If GitHub's ping lands before the new hook id is persisted, it is ignored, and the repo keeps 60s polling until the first push delivery. That costs extra polling, not missed changes.
  • A webhook that breaks after working costs at most one 10-minute delay before polling tightens again.

Test plan

  • Unit: test_effective_poll_interval covers an unverified hook, a stale verified id from an older hook, and a verified hook.
  • EE backend (quickjs,enterprise,private), local smart-HTTP git remote, with state planted by SQL:
    • A signed ping with a bad signature returns 401 and leaves the id unset.
    • A valid signed ping returns 200 and sets webhook_verified_id=777.
    • A push with no delivery: the next poll enqueues the pull, clears the id, and logs moved without a webhook delivery. The rest of git_sync is intact.
    • Verified hook with a failed last pull: the poll re-enqueues and the id stays. With a successful last pull it is cleared.
  • UI: the card text for unverified and verified hooks (below).

Screenshots

Hook registered, no delivery yet:
ui-unverified

Hook verified:
ui-verified

Follow-up (separate PR): remove AutoPullMode. No UI ever set it, and with polling now a reliable baseline every repo can create a hook when it can hold one and always poll.

🤖 Generated with Claude Code


Summary by cubic

Git auto-pull now relaxes its polling interval only after the webhook has actually delivered to the instance, not when the webhook is registered. This prevents a 10-minute deployment delay on air-gapped setups where the host can reach Windmill but not the reverse.

  • Adds a server-owned webhook_verified_id to AutoPullSettings, sanitized from client saves and stripped from settings exports. Polling stays at every minute until the current webhook_id matches it.
  • The poller clears the verified id when it finds a fresh head that no delivery brought, falling back to 60-second polling. It skips clearing after a failed pull and for a ref's first sync, which no delivery could have brought.
  • The repository card says "Syncing instantly via webhook" only for a verified hook; otherwise it states minute-cadence polling until the webhook delivers.

Written for commit e7802da. Summary will update on new commits.

View guided diff Turn on auto-fix

hugocasa and others added 3 commits November 19, 2026 10:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hugocasa

Copy link
Copy Markdown
Collaborator Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @hugocasa's task in 1m 52s —— View job


cc @hugocasa

Mergeable, but should ideally address nits: the first pull of a newly enabled repo or newly tracked ref clears the ping verification (monitor.rs)

  • Gather diff and context
  • Review backend changes
  • Review frontend changes
  • Post review

Findings

P2: an initial pull is counted as a missed webhook delivery (backend/src/monitor.rs:5362-5368)

The poller clears webhook_verified_id on any Ok(Some(_)). That includes a ref with no entry in last_synced_sha yet, where no webhook could have brought the head.

This happens on the most common path. When auto-pull is newly enabled, sanitize_client_auto_pull leaves last_synced_sha empty, the hook is created after the commit, and GitHub's ping verifies it within seconds. Then the repo's first poll tick, up to ~60s later, enqueues the initial pull. last_pull_status is None, so the guard passes and the verification is cleared, with the misleading log "moved without a webhook delivery". After that the repo polls every minute until its first real push delivery. Switching the tracked branch has the same effect.

This only costs extra polling, never a missed change, but it cancels most of what the ping buys. The fix is to also require auto_pull.last_synced_sha.contains_key(&git_ref) in the guard. Fix this →

Checked and found correct:

  • The server-owned field is handled at every write and export path: it is sanitized from client saves, carried across both settings-save handlers, redacted from the settings export, and printed in Debug.
  • set_auto_pull_webhook_verified_id is safe. The COALESCE guards against jsonb_set NULLing git_sync, the jsonb_array_length > 0 guard stops jsonb_agg returning NULL on an empty array, and the doc comment states the caller contract (checklist (a)).
  • The failed-last-pull exemption correctly avoids demoting the hook because of a rollback.
  • A settings save that runs at the same moment can overwrite a just-written verified id (read-modify-write of the whole git_sync). That is the same lost-update class as last_synced_sha and only costs polling, so I am not flagging it.
  • The frontend card logic matches webhook_verified().

Not verified: the EE receivers. The windmill-ee-private checkout in this runner is not at ee-repo-ref.txt's a202fb5, so I could not read the ping/GitLab call sites of set_auto_pull_webhook_verified_id.

AGENTS.md compliance

No violations found. Comments record constraints rather than narration, the sqlx cache is updated for the new query, and openapi.yaml documents the new field.

Test coverage

  • Backend: test_effective_poll_interval covers the interval rule well (unverified, stale id from an older hook, verified). The demotion branch in the poller has no automated test. It depends on DB state and the head lookup, so the manual SQL-planted checks in the PR body are reasonable. If the guard above is added, the "first sync of a ref" case is worth adding to that manual pass.
  • Frontend: a template-only change, so no tests are expected.

Manual verification still worth doing: enable auto-pull on a GitHub repo whose instance is reachable, and wait past the first poll tick. The card should still read "Syncing instantly via webhook", and the backend log should not show "moved without a webhook delivery". Today I expect it to fall back to the every-minute text.

Comment thread backend/src/monitor.rs
Comment on lines +5362 to +5368
Ok(Some(_))
if auto_pull.webhook_verified()
&& auto_pull
.last_pull_status
.as_ref()
.map_or(true, |s| s.success) =>
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: a ref with no synced sha yet is treated as a missed delivery. Ok(Some(_)) here also covers the first pull of a ref that has never synced. last_synced_sha has no entry for it, so should_pull is true, but no webhook ever had a chance to bring that head.

Common path: auto-pull is enabled on a GitHub repo. sanitize_client_auto_pull leaves last_synced_sha empty, the hook is created after the commit, and the ping sets webhook_verified_id within seconds. The poller's first tick for the repo comes up to ~60s later and enqueues the initial pull. webhook_verified() is true and last_pull_status is None, so the id is cleared and the log says "moved without a webhook delivery". The ping verification is undone, and the repo polls every minute until its first real push delivery. Changing the tracked branch (a new git_ref key) has the same effect.

This only costs extra polling, never a missed change. But it cancels the "ping verifies within seconds" benefit for most new repos. Suggested guard: demote only when a previous synced sha exists for this ref:

Ok(Some(_))
    if auto_pull.webhook_verified()
        && auto_pull.last_synced_sha.contains_key(&git_ref)
        && auto_pull.last_pull_status.as_ref().map_or(true, |s| s.success) =>

(git_ref is moved into the call above, so bind the check to a local before calling reconcile_and_enqueue_pull.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hugocasa

Copy link
Copy Markdown
Collaborator Author

/codex

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: e7802da
Status:⚡️  Build in progress...

View logs

@hugocasa

Copy link
Copy Markdown
Collaborator Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @hugocasa's task in 34s —— View job


Good to merge

  • Gather diff since the previous review
  • Check the fix for the first-pull finding
  • Re-check the rest of the diff
  • Post review

What changed since the last review

One new commit, f8202c9. It adds auto_pull.last_synced_sha.contains_key(&git_ref) to the guard on the demotion arm (backend/src/monitor.rs:5363-5369).

This resolves the earlier P2. Before this change, the initial pull after enabling auto-pull or switching the tracked branch cleared the GitHub ping's verification and logged "moved without a webhook delivery". Now that pull has no last_synced_sha entry for the ref, so it leaves webhook_verified_id alone.

  • git_ref is borrowed (&git_ref) in the call to reconcile_and_enqueue_pull, not moved, so the guard can reference it without a separate local.
  • auto_pull is the snapshot taken before the reconcile runs, so the check sees the state the poll tick started from, which is what the guard needs.
  • If the ref key ever differs between the poller and the stored map (for example a "HEAD" fallback when ls-remote --symref resolution fails), the guard skips the demotion. Skipping is the safe direction: polling stays at whatever interval it already had, and the next head with a matching key is still caught.
  • The updated comment states both exemptions (first sync, failed last pull) as constraints and doesn't narrate.

My other conclusions from the previous review still hold: the field is server-owned on every save and export path, the jsonb_set helper is null-safe, and the card logic matches webhook_verified(). The EE receivers are still unverified for the same reason as before: the local windmill-ee-private checkout doesn't match ee-repo-ref.txt.

AGENTS.md compliance

No violations found.

Test coverage

  • Backend: test_effective_poll_interval still covers the interval rule. The demotion arm, including the new first-sync exemption, has no automated test because it depends on DB state and a live head lookup. The SQL-planted manual checks are the right level for it.
  • Frontend: only the template changed, so no tests are expected.

Manual verification still worth doing: enable auto-pull on a GitHub repo whose instance GitHub can reach, and wait past the first poll tick (about 60s). The card should keep reading "Syncing instantly via webhook", the backend log should not show "moved without a webhook delivery", and webhook_verified_id should still equal webhook_id in git_sync.

…ook-polling-fallback

# Conflicts:
#	backend/ee-repo-ref.txt
@hugocasa

Copy link
Copy Markdown
Collaborator Author

/review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @hugocasa's task in 32s —— View job


Good to merge

  • Gather the diff since the previous review
  • Re-check the PR's own changes
  • Post review

What changed since the last review

Only the merge of origin/main (6356758). In this PR's files, it adds just three things that came from main and are not part of this change:

  • the GitRemoteCaller::AdminOrSystem argument on the head lookup in monitor.rs
  • capabilities: None in load_workspace_authed
  • capabilities: Some(RoleCapabilities::FULL) in two windmill-common test fixtures

None of them touches the auto-pull verification logic. The PR's own diff is the same as at f8202c9. The demotion arm in backend/src/monitor.rs:5364-5371 still requires a verified hook, an existing last_synced_sha entry for the ref, and a successful (or missing) last pull.

My earlier conclusions still hold. The field is server-owned on every save and export path. The jsonb_set helper is null-safe and documents its caller contract. The first-sync and failed-pull exemptions are correct, and the card text matches webhook_verified(). The EE receivers are still unverified, because the runner's windmill-ee-private checkout is not at ee-repo-ref.txt.

AGENTS.md compliance

No violations found.

Test coverage

  • Backend: test_effective_poll_interval covers the interval rule. The demotion arm has no automated test because it depends on DB state and a live head lookup. The manual checks in the PR body, with state planted by SQL, are the right level for it.
  • Frontend: only the template changed, so no tests are expected.

Manual verification still worth doing: enable auto-pull on a GitHub repo whose instance GitHub can reach, then wait past the first poll tick (about 60s). The card should still read "Syncing instantly via webhook" and the backend log should not show "moved without a webhook delivery".

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found. Reviewed the polling logic, settings sanitization/preservation/export paths, and authenticated GitHub/GitLab receivers at the exact pinned EE revision, 095ef503.

Test coverage

The Rust interval test covers unverified hooks, stale verification IDs, verified hooks, and explicit interval overrides. No new integration tests cover receiver verification, the JSONB update, or poller demotion. The frontend change only updates template text; component tests are unnecessary.

Ran git diff --check and confirmed the SQLx cached query matches its source and hash. No automated tests or live UI/webhook scenarios were run during this review.

Before merge, enable auto-pull for a reachable GitHub repository and wait through its first poll. Verification should remain set and the card should continue showing webhook sync.

For GitLab, confirm an authenticated push verifies the hook. Then prevent deliveries and push another commit: the safety-net poll should deploy it, clear verification, and restore minute polling. Invalid-token deliveries should leave verification unchanged.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

Good to merge

I reviewed the full diff (6356758, including the earlier fix commits) against the surrounding code, not just the patch. The design is sound and the one real defect found in the earlier review round (the first-sync pull clearing the ping verification) is fixed in f8202c9. No blockers or nits worth surfacing.

Considered and dismissed

Each candidate from my first pass, with the reason it is not a finding:

  • Poll/webhook race → false demotion or duplicate pull. The poll reads its auto_pull snapshot before the ls-remote round-trip, so a webhook delivering the same head during that window makes the poll enqueue a duplicate and demote the hook (monitor.rs:5356-5390). This is the pre-existing poll-vs-webhook reconcile race, and the new consequence is only extra polling: the receiver's mark_webhook_verified runs on the very next delivery (git_sync_ee.rs:172,350) and re-verifies, so it self-heals and can never miss a change.
  • set_auto_pull_webhook_verified_id NULLing git_sync (workspaces.rs:647-679). Dismissed: COALESCE(to_jsonb($3::bigint), 'null'::jsonb) handles the NULL-id write, and the jsonb_typeof(...) = 'array' / jsonb_array_length(...) > 0 guards stop jsonb_agg returning NULL on an empty array; when git_sync itself is NULL the WHERE clause simply doesn't match.
  • Lost update between this writer and persist_auto_pull_state. Both rewrite the whole {repositories} array with targeted jsonb_sets, so they can clobber each other. This is the same lost-update class the existing last_synced_sha writer already has; the worst outcome is dropping a webhook_verified_id write, i.e. extra polling, never a missed change.
  • Stale webhook_verified_id left behind after a hook is deleted (git_sync_ee.rs:2377 clears webhook_id but not webhook_verified_id). Dismissed: webhook_verified() requires webhook_id.is_some() (workspaces.rs:613-615), and host hook ids are not reused, so a replacement hook starts unverified as intended.
  • Frontend strict === on an i64 field. webhook_verified_id === hookId compares JSON numbers; GitHub/GitLab hook ids are small integers far inside Number's safe range, and the card already compared webhook_id != null the same way.
  • GitLab never verifying (no hook id). Dismissed: create_repo_webhook (git_sync_ee.rs:1032) returns the hook id for both providers, so mark_webhook_verified writes a real id for GitLab too.
  • Server-owned field leaking to clients. Dismissed: sanitize_client_auto_pull zeroes it on both save paths (workspaces.rs:1025, used at 5361 and 5596), both save handlers carry it from the DB row (5460, 5719), and the settings export redacts it (workspace_settings_document.rs:109). The GET endpoint strips only webhook_secret, so the card still sees both ids to compare.
  • False demotion on first sync or after a failed pull. Dismissed: the contains_key(&git_ref) guard (added in f8202c9) exempts first syncs, and the last_pull_status.map_or(true, |s| s.success) guard exempts rollbacks; both behave as the comment documents.

Test coverage

  • Backend (Rust): test_effective_poll_interval (workspaces.rs:4313-4337) covers the interval rule well — unverified hook, stale verified id from an older hook, and verified hook. The demotion arm in monitor.rs and the set_auto_pull_webhook_verified_id SQL have no automated test because they depend on DB state plus a live head lookup; the SQL-planted manual checks in the PR body (bad-signature ping → 401 and unset, valid ping → webhook_verified_id=777, missed-push demotion, failed-last-pull exemption) are the right level for those paths. The .sqlx cache entry for the new query is committed.
  • Frontend: template-only change to GitSyncRepositoryCard.svelte; no component tests expected in this codebase.
  • Docs/config: openapi.yaml documents the new field.

Manual verification still worth doing: enable auto-pull on a GitHub repo whose instance GitHub can actually reach, and wait past the first poll tick (~60s) — the card should keep reading "Syncing instantly via webhook", the backend log should not print "moved without a webhook delivery", and webhook_verified_id should equal webhook_id in git_sync. Separately, push to an air-gapped repo (hook registered but undeliverable) and confirm the card reads "Checking … every minute until the webhook delivers" and the deploy lands within ~1 minute, not 10.

@rubenfiszel
rubenfiszel marked this pull request as ready for review November 20, 2026 15:52
This commit updates the EE repository reference after PR #996 was merged in windmill-ee-private.

Previous ee-repo-ref: 095ef5033bff64ed5b2d08c84bb0037b83449d88

New ee-repo-ref: 6c06d688fe435630e2827bc177f066d2646bb768

Automated by sync-ee-ref workflow.
@windmill-internal-app

Copy link
Copy Markdown
Contributor

🤖 Updated ee-repo-ref.txt to 6c06d688fe435630e2827bc177f066d2646bb768 after windmill-ee-private PR #996 was merged.

@windmill-internal-app windmill-internal-app Bot changed the title [ee] fix: relax git auto-pull polling only once the webhook has delivered fix: relax git auto-pull polling only once the webhook has delivered Nov 20, 2026
@rubenfiszel
rubenfiszel merged commit d538f2a into main Nov 20, 2026
16 of 19 checks passed
@rubenfiszel
rubenfiszel deleted the airgapped-github-webhook-polling-fallback branch November 20, 2026 15:55
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants