Skip to content

fix(aws): exclude uninitialized Fargate tasks from ALB readiness #647

Closed

Description

@cristim

Summary

PR #646 (issue #488) introduced a /ready endpoint — 503 until the process completes initialization — and moved every Azure traffic-admission consumer to it. The same cold-replica admission bug class remains on AWS: the Fargate ALB target group health check still gates traffic on /health, which answers HTTP 200 even while degraded.

Current behaviour

  • terraform/environments/aws/compute.tf:185 sets health_check_path = "/health" (default at terraform/modules/compute/aws/fargate/variables.tf:148), consumed by the ALB health_check block at terraform/modules/compute/aws/fargate/main.tf:468-470. A cold Fargate task answers 200/"degraded" and the ALB routes production traffic to it.
  • .github/workflows/deploy-aws-fargate.yml:289-313 still uses the pre-fix(server/azure): exclude uninitialized replicas from readiness #488 smoke-gate pattern: warm up /api/auth/check-admin to trigger lazy init, then parse the /health body. The warm-up initializes whichever replica the load balancer picked, not the one the next /health request reaches, and its comments ("never triggers the lazy DB init") are contradicted by the background initializer added in fix(server/azure): exclude uninitialized replicas from readiness #488.
  • The ECS container-level healthCheck (terraform/modules/compute/aws/fargate/main.tf:666) is liveness and correctly stays on /health.
  • Lambda deploy/rollback workflows (deploy-aws-lambda.yml:424-442, rollback.yml:295) are correct as-is: Lambda deliberately keeps lazy init and has no /ready route.

Steps to verify the gap

Deploy to AWS with more than one Fargate task; during the rollout, cold tasks pass the ALB health check while /ready on the same tasks answers 503.

Expected behaviour

Every AWS traffic-admission check uses /ready; liveness checks stay on /health; the deploy smoke gate drops the warm-up request and uses scripts/wait-for-healthy.sh like the Azure workflow.

Proposed fix

  • Set health_check_path = "/ready" in terraform/environments/aws/compute.tf:185 and update the variable default/description in terraform/modules/compute/aws/fargate/variables.tf:148.
  • Replace the inline smoke gate in .github/workflows/deploy-aws-fargate.yml:289-313 with scripts/wait-for-healthy.sh (mirroring deploy-azure.yml after fix(server/azure): exclude uninitialized replicas from readiness #488).

References

Severity

Medium — same admission bug class as #488; the AWS smoke gate's body-parse still catches persistent degraded states within its retry budget, so this is less acute than the Azure failure was.

Activity

  1. IAmJSD commented on Oct 16, 2026

    @IAmJSD
    Collaborator

    Claiming: working on this

  2. self-assigned this
    on Oct 16, 2026
  3. added a commit that references this issue on Oct 17, 2026
    663aec9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions