Skip to content

fix(deploy): resolve the proxy probe's named port so the migration can start - #271

Open
yetone wants to merge 1 commit into
mainfrom
fix/migration-job-probe-named-port
Open

fix(deploy): resolve the proxy probe's named port so the migration can start#271
yetone wants to merge 1 commit into
mainfrom
fix/migration-job-probe-named-port

Conversation

@yetone

@yetone yetone commented Sep 10, 2026

Copy link
Copy Markdown
Owner

What breaks

The production deploy has not been able to run its schema migration. Run 34529081984 failed with deploy-release: job_failed after the Job hit DeadlineExceeded at 600s, which reads as "the catch-up migration is too big". It is not — the migration never started at all.

The Deployment's cloud-sql-proxy declares:

ports: [{containerPort: 9090, name: pg-health}]
startupProbe: {httpGet: {path: /readiness, port: pg-health}}

buildMigrationJob copies that probe onto the Job's sidecar, but cleanContainerForJob strips ports. The name then has nothing to resolve against, and kubelet says so:

Warning  Unhealthy  pod/cumora-migrate-...  Startup probe errored and resulted in
unknown state: strconv.Atoi: parsing "pg-health": invalid syntax

A native sidecar whose startup probe never passes never reports started, so the migrate container is never launched. Observed live in cumora-migrate-e11b024-34531956583: proxy ready=false started=false, proxy logs "The proxy has started successfully and is ready for new connections!", migrate container PodInitializing nine minutes in. The Job then burns its whole deadline and its pod is reaped, so there are no logs to reveal that nothing ran.

The fix

Resolve a named httpGet/tcpSocket port against the source container's port list at the point where the probe is copied, while that list is still in hand. When the name cannot be resolved, return null so the caller keeps its existing numeric default — shipping a probe that can never pass is strictly worse than using our own.

The Job's containers still declare no ports; only the probe target changes.

Tests

The fixture's proxy had neither ports nor a startupProbe, so the suite only ever exercised the numeric fallback path — which is exactly why this never showed up before production. Two cases added, both verified to fail on main:

not ok 4 - a proxy probe on a named port is resolved to its number, ...
not ok 5 - an unresolvable named probe port falls back to a probe that can actually pass

…n start

Every deploy since the proxy gained a startup probe has been unable to run
its migration, and the failure looked like a slow migration rather than a
broken one.

The Deployment's cloud-sql-proxy declares `ports: [{containerPort: 9090,
name: pg-health}]` and probes that name. buildMigrationJob copies the probe
onto the Job's sidecar but `cleanContainerForJob` strips `ports`, so the name
has nothing to resolve against. kubelet falls back to parsing it as a number:

  Startup probe errored and resulted in unknown state:
  strconv.Atoi: parsing "pg-health": invalid syntax

A native sidecar whose startup probe never passes never reports `started`,
so the `migrate` container is never launched. The Job sits idle until
activeDeadlineSeconds kills it as DeadlineExceeded, and because the pod is
reaped there are no logs to say nothing ever ran.

Resolve the name to its number while the source container's port list is
still in hand, and fall back to the existing numeric probe when it cannot be
resolved -- shipping a probe that can never pass is strictly worse than
using our own.

The fixture's proxy had neither ports nor a startup probe, so the tests only
ever exercised the numeric fallback. Both new cases fail without this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant