E2E test flake: TestWorkerPoolPodTemplateUpdate fails on optimistic-concurrency conflict #994

Symptom

TestWorkerPoolPodTemplateUpdate intermittently fails on main with an optimistic-concurrency conflict:

Operation cannot be fulfilled on workerpools.ate.dev "test-template-update":
the object has been modified; please apply your changes to the latest version and try again

Seen on three independent runs on main. Each was verified: all are failure, all carry the conflict signature in the failed-step log, and in each the failing test is TestWorkerPoolPodTemplateUpdate itself.

Cause

The test does a bare get-then-update on the WorkerPool:

k8sClient.Get(ctx, types.NamespacedName{...}, wp)
wp.Spec.Template.NodeSelector = map[string]string{"workload": "updated"}
k8sClient.Update(ctx, wp)   // no conflict handling

WorkerPoolReconciler writes the same object concurrently — r.Status().Update(ctx, wp) at cmd/atecontroller/internal/controllers/workerpool_controller.go:142 — which bumps resourceVersion. If that lands between the test's read and its write, the write is rejected and the test fails. The test also runs with t.Parallel(), so the reconcile loop is under load from the rest of the package.

The window is narrow enough that it never shows up locally and only surfaces on loaded CI runners; I did not measure its width.

The file already has a helper for exactly this, updateWorkerPoolSpec, which retries on conflict; the comment on its sibling spells the problem out:

the controller reconciles the Deployment concurrently (bumping its resourceVersion on every SSA apply), so a plain get-then-update flakes with "the object has been modified" under load.

This test simply does not use it.

Verification approach

Worth noting for whoever reviews the fix: -count=N proves nothing here. The unfixed test passes at -count=100 -race locally because the window is too narrow to hit. The conflict has to be constructed deliberately — insert a write between the test's read and its write — at which point the old code fails with the exact CI error and the helper-based version retries and lands.