Skip to content

fix(gcp): exclude uninitialized replicas from GKE and Cloud Run readiness #648

Closed

Description

@cristim

Summary

PR #646 (issue #488) introduced a /ready endpoint — 503 until the process completes initialization — and moved every Azure traffic-admission consumer to it. The same cold-replica admission bug class remains on GCP: the GKE readiness probe and the Cloud Run startup probe still gate traffic on /health, which answers HTTP 200 even while degraded.

Current behaviour

  • terraform/modules/compute/gcp/gke/main.tf:590-593 readiness probe uses var.health_check_path (default /health at terraform/modules/compute/gcp/gke/variables.tf:221). Kubernetes readiness is exactly the traffic-admission signal; this keeps cold pods in the Service endpoints. Complication: the liveness probe (main.tf:579-582) shares the same variable, so the fix needs a new readiness_probe_path variable defaulting to /ready — liveness must stay on /health.
  • terraform/modules/compute/gcp/cloud-run/main.tf:184-194: the Cloud Run startup probe doubles as the readiness gate (Cloud Run admits an instance once its startup probe passes) and still uses /health. The liveness probe at main.tf:197-207 correctly stays on /health.
  • .github/workflows/deploy-gcp.yml:275-298 still uses the pre-fix(server/azure): exclude uninitialized replicas from readiness #488 smoke-gate pattern: warm up /api/auth/check-admin, then parse the /health body — subject to the same warm-up/hit-different-replica race fixed on Azure.

Steps to verify the gap

Deploy to GKE or Cloud Run with more than one replica; during rollout, cold replicas pass their probes while /ready on the same replicas answers 503.

Expected behaviour

GKE readiness, Cloud Run startup, and the GCP deploy smoke gate all use /ready; liveness probes stay on /health.

Proposed fix

  • Add a readiness_probe_path variable (default /ready) to the GKE module and use it at terraform/modules/compute/gcp/gke/main.tf:590-593.
  • Point the Cloud Run startup_probe at /ready, but widen its failure budget (failure_threshold = 3 x period_seconds = 10 + 10 s delay is only ~40 s; background init plus migrations can exceed that, and startup-probe failure restarts the container, risking a restart loop against a slow-but-recovering DB).
  • Replace the inline smoke gate in .github/workflows/deploy-gcp.yml:275-298 with scripts/wait-for-healthy.sh (mirroring deploy-azure.yml after fix(server/azure): exclude uninitialized replicas from readiness #488).

References

Severity

Medium — same admission bug class as #488; the GCP smoke gate's body-parse still catches persistent degraded states within its retry budget.

Activity

  1. IAmJSD commented on Oct 16, 2026

    @IAmJSD
    Collaborator

    Claiming: working on this

  2. self-assigned this
    on Oct 16, 2026
  3. added a commit that references this issue on Oct 17, 2026
    b47dbfd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions