Skip to content

fix(ci): fail deploy verification on degraded health and Azure log lookup errors - #437

Merged
cristim merged 2 commits into
mainfrom
fix/217-deploy-health-status
Sep 28, 2026
Merged

cristim merged 2 commits into
mainfrom
fix/217-deploy-health-status

Conversation

@cristim

@cristim cristim commented Sep 28, 2026

Copy link
Copy Markdown
Member

Summary

Deployment verification now fails when the app reports degraded, and the Azure log diagnostic queries the real per-environment resource group and fails on lookup errors.

Root cause

  • Test health endpoint only grepped for a "status" key, so {"status":"degraded"} passed.
  • Run smoke tests only checked for HTTP 200, but internal/server/health.go always returns 200 and carries the verdict in the body.
  • Check Container App logs used env.RESOURCE_GROUP (default cudly-rg), not the cudly-<env>-rg group Terraform creates, and || true swallowed ResourceGroupNotFound.
  • deploy-gcp.yml, deploy-aws-fargate.yml and deploy-aws-lambda.yml had the same two health-check weaknesses, so they are fixed here as well (same concern).

Fix

  • All four deploy workflows: both steps parse the body with jq -e '.status == "healthy"'. The top-level status is only healthy when config_store and auth_store are healthy and migrations are healthy or disabled (health.go), so this covers the required component checks.
  • HTTP transports (Azure, GCP, Fargate): each health attempt first sends GET /api/auth/check-admin (public, read-only). On the HTTP server /health never calls ensureDB, so without a warm-up request a fresh deploy would stay degraded (config_store pending, auth_store not initialized, exactly the payload in the issue) and the stricter check would fail every deploy. The warm-up result is ignored (|| true) on purpose; /health stays the gate. Lambda needs no warm-up because HandleLambdaEvent runs ensureDB for every event, /health included.
  • Azure: build-and-deploy exposes the Terraform resource_group_name output as a job output; the log step uses it, errors if either output is empty, and no longer has || true, so a failed az containerapp logs show fails the step. The now-unused RESOURCE_GROUP env var and its README/header docs are removed.

Regression test

There is no workflow test harness in this repo, so I extracted the run: blocks of the two steps from deploy-azure.yml with PyYAML (sleeps set to 0, URL substituted) and ran them under bash -e against a local HTTP server that returns 200 with a given status:

python3 srv.py 18217 healthy &    # returns 200 {"status": "healthy"}
python3 srv.py 18218 degraded &   # returns 200 {"status": "degraded"}
TEST_URL=http://127.0.0.1:<port> bash -e health.sh / smoke.sh
Code Server Step Exit Last line
origin/main degraded health 0 Health check passed!
origin/main degraded smoke 0 All smoke tests passed!
this PR degraded health 1 Health check failed after 5 attempts
this PR degraded smoke 1 Health check 1: failed: {"status": "degraded"}
this PR healthy health 0 Health check passed!
this PR healthy smoke 0 All smoke tests passed!

jq expression against sample payloads (jq -e '.status == "healthy"'):

healthy='{"status":"healthy","checks":{"auth_store":{"status":"healthy"},"config_store":{"status":"healthy"},"migrations":{"status":"healthy"}}}'   -> exit 0
degraded='{"status":"degraded","checks":{"auth_store":{"status":"unhealthy","message":"auth store not initialized"},"config_store":{"status":"pending"},"migrations":{"status":"pending"}}}'   -> exit 1 (old grep: exit 0)
'{}' -> exit 1, 'not json' -> exit 5, '' -> exit 4

The log-step change could not be exercised locally (needs Azure credentials and a deployed app); it has not been run.

Verification

  • actionlint v1.7.12 (the version pinned in .pre-commit-config.yaml), native binary: clean on all deploy-*.yml.
  • pre-commit run on the changed files: zizmor, markdownlint, trivy, whitespace and secrets hooks pass. The actionlint hook was skipped (SKIP=actionlint) because it runs through Docker and the local Docker daemon is hung; the native binary at the same version was run instead.
  • No deploy was triggered.

Closes #217

…okup errors

The deploy health and smoke steps accepted any body containing a status
key, or any HTTP 200, but /health returns 200 with status=degraded by
design. Require .status == "healthy" via jq in all four deploy
workflows, warming the lazy DB init through a public /api/ route on the
HTTP transports first.

The Azure log step queried the hardcoded cudly-rg default instead of the
per-environment resource group and swallowed ResourceGroupNotFound with
|| true. Use the Terraform resource_group_name output and let lookup
errors fail the step.

Closes #217
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 5 billable files and costs up to $1.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 23 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 62 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: LeanerCloud/cloud-commitments-platform/.coderabbit.yaml

Review profile: CHILL

Plan: Essentials

Run ID: e9ad883c-702c-4af4-a646-88045995398e

📥 Commits

Reviewing files that changed from the base of the PR and between fb19eaa and f0ef145.

📒 Files selected for processing (5)
  • .github/workflows/README.md
  • .github/workflows/deploy-aws-fargate.yml
  • .github/workflows/deploy-aws-lambda.yml
  • .github/workflows/deploy-azure.yml
  • .github/workflows/deploy-gcp.yml

Comment @coderabbitai help to get the list of available commands.

@cristim

cristim commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review (Opus 5.5) at b069c70: MERGE. All 4 deploy workflows require jq -e '.status == "healthy"' (degraded, wrong-case, array and concatenated JSON pass on main but fail here; empty, HTML, non-JSON and 503 fail on both); the Azure log step uses the TF resource_group_name output with no || true; retries are bounded; the warm-up GET /api/auth/check-admin is read-only, and any /api path triggers ensureDB, so a route rename can't break it. This head is b069c70 plus a merge of main (#440, the red-main fix); workflow files are unchanged. Follow-ups: #438.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/internal Team-internal only priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): fail Azure deployment verification on degraded health and log lookup errors

1 participant