fix(deploy): resolve the proxy probe's named port so the migration can start - #271
Open
yetone wants to merge 1 commit into
Open
fix(deploy): resolve the proxy probe's named port so the migration can start#271yetone wants to merge 1 commit into
yetone wants to merge 1 commit into
Conversation
…n start
Every deploy since the proxy gained a startup probe has been unable to run
its migration, and the failure looked like a slow migration rather than a
broken one.
The Deployment's cloud-sql-proxy declares `ports: [{containerPort: 9090,
name: pg-health}]` and probes that name. buildMigrationJob copies the probe
onto the Job's sidecar but `cleanContainerForJob` strips `ports`, so the name
has nothing to resolve against. kubelet falls back to parsing it as a number:
Startup probe errored and resulted in unknown state:
strconv.Atoi: parsing "pg-health": invalid syntax
A native sidecar whose startup probe never passes never reports `started`,
so the `migrate` container is never launched. The Job sits idle until
activeDeadlineSeconds kills it as DeadlineExceeded, and because the pod is
reaped there are no logs to say nothing ever ran.
Resolve the name to its number while the source container's port list is
still in hand, and fall back to the existing numeric probe when it cannot be
resolved -- shipping a probe that can never pass is strictly worse than
using our own.
The fixture's proxy had neither ports nor a startup probe, so the tests only
ever exercised the numeric fallback. Both new cases fail without this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks
The production deploy has not been able to run its schema migration. Run
34529081984failed withdeploy-release: job_failedafter the Job hitDeadlineExceededat 600s, which reads as "the catch-up migration is too big". It is not — the migration never started at all.The Deployment's
cloud-sql-proxydeclares:buildMigrationJobcopies that probe onto the Job's sidecar, butcleanContainerForJobstripsports. The name then has nothing to resolve against, and kubelet says so:A native sidecar whose startup probe never passes never reports
started, so themigratecontainer is never launched. Observed live incumora-migrate-e11b024-34531956583: proxyready=false started=false, proxy logs"The proxy has started successfully and is ready for new connections!", migrate containerPodInitializingnine minutes in. The Job then burns its whole deadline and its pod is reaped, so there are no logs to reveal that nothing ran.The fix
Resolve a named
httpGet/tcpSocketport against the source container's port list at the point where the probe is copied, while that list is still in hand. When the name cannot be resolved, returnnullso the caller keeps its existing numeric default — shipping a probe that can never pass is strictly worse than using our own.The Job's containers still declare no
ports; only the probe target changes.Tests
The fixture's proxy had neither
portsnor astartupProbe, so the suite only ever exercised the numeric fallback path — which is exactly why this never showed up before production. Two cases added, both verified to fail onmain: