Repository navigation
fix(gcp): exclude uninitialized replicas from GKE and Cloud Run readiness #648
Copy link
Copy link
Closed
Labels
effort/mDaysDaysimpact/manyAffects most usersAffects most userspriority/p2Backlog-worthyBacklog-worthyseverity/mediumModerate harmModerate harmtriagedItem has been triagedItem has been triagedtype/bugDefectDefecturgency/this-quarterWithin the quarterWithin the quarter
Description
Summary
PR #646 (issue #488) introduced a /ready endpoint — 503 until the process completes initialization — and moved every Azure traffic-admission consumer to it. The same cold-replica admission bug class remains on GCP: the GKE readiness probe and the Cloud Run startup probe still gate traffic on /health, which answers HTTP 200 even while degraded.
Current behaviour
terraform/modules/compute/gcp/gke/main.tf:590-593readiness probe usesvar.health_check_path(default/healthatterraform/modules/compute/gcp/gke/variables.tf:221). Kubernetes readiness is exactly the traffic-admission signal; this keeps cold pods in the Service endpoints. Complication: the liveness probe (main.tf:579-582) shares the same variable, so the fix needs a newreadiness_probe_pathvariable defaulting to/ready— liveness must stay on/health.terraform/modules/compute/gcp/cloud-run/main.tf:184-194: the Cloud Run startup probe doubles as the readiness gate (Cloud Run admits an instance once its startup probe passes) and still uses/health. The liveness probe atmain.tf:197-207correctly stays on/health..github/workflows/deploy-gcp.yml:275-298still uses the pre-fix(server/azure): exclude uninitialized replicas from readiness #488 smoke-gate pattern: warm up/api/auth/check-admin, then parse the/healthbody — subject to the same warm-up/hit-different-replica race fixed on Azure.
Steps to verify the gap
Deploy to GKE or Cloud Run with more than one replica; during rollout, cold replicas pass their probes while /ready on the same replicas answers 503.
Expected behaviour
GKE readiness, Cloud Run startup, and the GCP deploy smoke gate all use /ready; liveness probes stay on /health.
Proposed fix
- Add a
readiness_probe_pathvariable (default/ready) to the GKE module and use it atterraform/modules/compute/gcp/gke/main.tf:590-593. - Point the Cloud Run
startup_probeat/ready, but widen its failure budget (failure_threshold = 3xperiod_seconds = 10+ 10 s delay is only ~40 s; background init plus migrations can exceed that, and startup-probe failure restarts the container, risking a restart loop against a slow-but-recovering DB). - Replace the inline smoke gate in
.github/workflows/deploy-gcp.yml:275-298withscripts/wait-for-healthy.sh(mirroringdeploy-azure.ymlafter fix(server/azure): exclude uninitialized replicas from readiness #488).
References
- Parent: fix(server/azure): exclude uninitialized replicas from readiness #488, PR fix(server): exclude uninitialized replicas from readiness #646
Severity
Medium — same admission bug class as #488; the GCP smoke gate's body-parse still catches persistent degraded states within its retry budget.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
effort/mDaysDaysimpact/manyAffects most usersAffects most userspriority/p2Backlog-worthyBacklog-worthyseverity/mediumModerate harmModerate harmtriagedItem has been triagedItem has been triagedtype/bugDefectDefecturgency/this-quarterWithin the quarterWithin the quarter
Activity
IAmJSD commentedon Oct 16, 2026
Claiming: working on this
fix(ci): gate the GCP deploy on /ready via wait-for-healthy.sh
fix(gcp): exclude uninitialized replicas from GKE and Cloud Run readi…