Skip to content

fix(infra): deployment checks and GCP uptime monitor gate on tautological /health 200 #649

Closed

Description

@cristim

Summary

After PR #646 (issue #488), /health is a liveness-only endpoint that answers HTTP 200 by design even when every dependency is down. Two verification layers still treat that 200 as meaningful, so they now pass tautologically on a fully degraded deployment.

Current behaviour

  • terraform/modules/deployment-checks/main.tf:60-81: the post-apply check "health" asserts only HTTP 200 on /health; the warmup provisioner at main.tf:35-56 also accepts any non-000/502/503 status. Since /health is 200-always, this gate passes on a fully degraded deploy. Complication: the module also runs against the Lambda platform (terraform/environments/aws/checks.tf:13-17), where /ready does not exist (the route is registered only on the server mux, not the Lambda API router at internal/api/router.go:355-356).
  • terraform/modules/monitoring/gcp/main.tf:600-612: the GCP uptime check asserts only HTTP 200 on /health, so it reports 100% availability while every replica is 503 on /ready. (The Azure counterpart content-matches "healthy" in the body and still detects degraded.)

Steps to verify the gap

Point the deployment-checks module or the GCP uptime check at an instance whose database is unreachable: both report success while /ready answers 503.

Expected behaviour

Post-apply deployment checks fail when the deployment is degraded; uptime monitoring can distinguish "process alive" from "serving traffic" where that signal is wanted.

Proposed fix

  • Give the deployment-checks module a per-platform health-path variable (/ready for container platforms, /health + body parse for Lambda), or register /ready in the Lambda API router too — noting that Lambda deliberately keeps lazy init, so a Lambda /ready would need to trigger or await initialization rather than just report it.
  • For the GCP uptime check, either add a content match on "healthy" (mirroring the Azure monitor) or leave as a pure availability signal by decision; do NOT point uptime monitors at /ready (deploys would flap alerts).

References

Severity

Medium — deployment verification and monitoring blind spots, not a live outage vector.

Activity

  1. IAmJSD commented on Oct 16, 2026

    @IAmJSD
    Collaborator

    Claiming: working on this

  2. added a commit that references this issue on Oct 17, 2026
    333b13f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions