Add blue/green deployment tooling for cloud-TEE gateway - #62
Merged
Conversation
A new gateway image is a new app_id (compose hash), i.e. a separate CVM, so a zero-downtime release is an atomic flip of the _dstack-app-address pointer, not a load-balancer weight. Add tooling and docs for running two sides and cutting between them. - switch.sh: status / switch <a|b> / rollback / acme <a|b>. Operates only on the delegation zone (the served 0g.ai zone is never touched, so no token for it is needed). Each side runs under its own DELEGATION_ZONE sub-zone so their auto-managed records never collide; the switch layer is operator-owned CNAMEs the script repoints. Pre-switch it reads the target's published app_id and can probe it directly (--probe-url); post-switch it verifies /healthz and auto-rolls-back on failure. Reads app_id straight from Cloudflare (no dig dependency); shellcheck-clean. - blue-green.md: record-chain architecture, one-time setup, per-side deploy params, the ACME issuance switch + Let's Encrypt rate limits, cutover and rollback, standby health-check options, and a zero-downtime migration from the current single instance. Notes the app_id-LB-across-replicas [verify] on the custom-domain path. - README: link the runbook from the Notes section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Load KEY=VALUE config (CF_API_TOKEN and any overrides) from an env file so it need not be exported every run. `switch.env` next to the script is picked up automatically and is git-ignored (it holds the Cloudflare token); --env-file points elsewhere. The real environment still wins over the file, so an inline `CF_API_TOKEN=... ./switch.sh` override keeps working. - Config resolution moved after env-file load + arg parsing. - switch.env.example documents every variable; blue-green.md setup uses it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
…reen-deployment-rgpvyc
Self-review after merging main (whose compose comments spell this out): the
gateway's DOMAIN/DELEGATION_ZONE/ROUTER_URL are ${...} placeholders injected
from the encrypted env at boot, so they are NOT part of the measured compose
text and do NOT change app_id. Only the literal image digest does. Two fixes:
- blue-green.md: drop the wrong "changing DELEGATION_ZONE changes app_id" claim
in the migration section; add a top-of-doc note that a side's identity is its
image digest, so the two sides MUST be different builds — same image => same
app_id => dstack treats them as replicas and the switch cannot select between
them. Reframe migration step 1 accordingly.
- switch.sh: warn in `switch` and `status` when both sides publish the same
app_id (same build); make `--dry-run` never prompt for confirmation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Add a "Releasing a new build (fast path)" TL;DR near the top — the 4-command common case (acme b -> deploy b -> switch b -> delete a) with a focused note on what the certificate step does and why its order matters (side b's own ingress issues the cert on first boot; aim the issuance switch at b BEFORE deploying so it validates cleanly instead of burning failed-validation budget / tripping DNS_SETUP_TIMEOUT). Clarify in the Certificates section that pointing _acme-challenge back at the live side is only needed when staging the standby for days, not in the fast path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
The only record an operator hand-creates in the delegation zone is the static serving alias (`<DOMAIN>.<DELEGATION_ZONE>` CNAME -> GATEWAY_DOMAIN). Add a `setup` subcommand that creates it, so the one-time setup needs no manual DNS in integratenetwork.work at all. - `setup` reads GATEWAY_DOMAIN (env/switch.env); if unset it prints the value the current single-instance container already publishes, to pin as-is. - `status` now shows the serving alias and its target. - switch.env.example documents GATEWAY_DOMAIN; blue-green.md one-time setup uses `./switch.sh setup` and spells out that the switch + per-side records are all created automatically. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
`sh switch.sh` runs under dash on Debian/Ubuntu/WSL, which fails on `set -o pipefail` (and the script's other bashisms). Add a guard that re-execs under bash when not already running there, so `sh switch.sh` works too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Blue/green runs each side under its own delegation sub-zone while the served
CNAMEs stay pinned at the base zone, so dstack-ingress's strict one-hop DNS
pre-check never matches and a side blocks in the default `wait` mode. Expose
DNS_SETUP_MODE=${DNS_SETUP_MODE:-wait} so those sides can set `print` (which
skips only the container's own pre-check; Let's Encrypt and the dstack gateway
follow the full CNAME chain, so issuance and routing still work). Injected, so
the measured compose text is identical for wait or print; single instances keep
the safe `wait` default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Expose ACME_STAGING=${ACME_STAGING:-false} so a side can select Let's Encrypt's
staging CA (much higher rate limits, untrusted certs) from the CVM's encrypted
env instead of editing the compose. Default false reproduces today's behaviour
exactly: entrypoint.sh already defaults an unset ACME_STAGING to the string
"false", which is how production issues real certs now. Injected, so the measured
compose is identical for staging or prod — they share app_id, distinguished by
the served cert's issuer out of band. Update the README's stale "must be the
file, not an env var" guidance to match; needs ACME_STAGING in allowed_envs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
The post-switch `curl /healthz` could be satisfied by a cached routing lookup on the dstack gateway still pointing at the OLD side: it returns 200 (old side healthy), switch.sh declares success and never rolls back, then real traffic flows to a possibly-broken new side once the cache expires. Use the served TLS cert fingerprint as a per-side identity: each side runs its own dstack-ingress with its own cert, so a changed fingerprint proves the gateway now routes to the target side. After the flip, require BOTH /healthz OK AND the served fingerprint to differ from the pre-switch one before declaring success; keep polling (and auto-rollback on timeout) otherwise. Falls back to /healthz-only with a warning when there is no prior side or openssl is absent. Widen the default verify window (VERIFY_RETRIES 10 -> 20) so it outlasts the route cache instead of reading a slow flush as failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Probing a specific side before cutover IS possible: the dstack platform hostname <app_id>-443s.<base> (TLS passthrough) routes by the app-id in the hostname, independent of the custom domain's _dstack-app-address, so it reaches that side directly even before any traffic points at it. Add PLATFORM_BASE (e.g. in1.phala.network): when set, `switch <a|b>` health-checks https://<app_id_target>-443s.<PLATFORM_BASE>/healthz and refuses to switch unless the target side is healthy — the "verify the standby before cutover" gate that the custom-domain path alone could not provide. An explicit --probe-url still overrides it. Also make http_ok use -k (reachability/health, not cert validity — covered elsewhere) so staging and per-side endpoints probe cleanly, and show each side's probe URL in `status`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
The previous rollback recorded the side it left in a per-machine file under /tmp. In a multi-operator setting that file is per-machine and can be stale or absent, so different people could roll back inconsistently. With only two sides, "roll back" is exactly "switch to the other side of whatever is currently live", and the live side is already readable from the shared switch record in DNS — the one authoritative source. Drop the state file entirely and derive the target from DNS, so every operator on any machine computes the same rollback target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Two fixes surfaced by a real switch attempt: - switch.env edited on Windows/WSL had CRLF line endings, so PLATFORM_BASE carried a trailing CR and the constructed probe URL became "https://...phala.network\r/healthz" — a broken URL (and mangled terminal output). load_env_file now strips a trailing CR from each line, so CRLF files work. - The pre-switch target probe was one-shot. Retry it PROBE_RETRIES times (default 5, VERIFY_INTERVAL apart) before refusing to switch, so a just-booted or briefly-unreachable standby isn't rejected on a single transient failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
A CRLF checkout of a shell script breaks it under bash (CR in the shebang and in `set -o pipefail`), and a CR in a value sourced by deploy/phala/switch.sh breaks that value. Force eol=lf for *.sh/*.bash and *.env.example so checkouts are LF regardless of a contributor's core.autocrlf (common on Windows/WSL). Complements the runtime CR-stripping switch.sh already does for the (git-ignored) live env file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
End-to-end tested on staging (router-api-tee-staging.0g.ai). Fold the confirmed facts into the runbook: - DNS_SETUP_MODE=print is required on each side (the served CNAMEs stay at the base zone while a side runs under its sub-zone, so ingress's one-hop pre-check never matches and `wait` would block). Explain why; add it to the deploy table, fast path, and migration; note it must be in allowed_envs. - Standby probe: replace the old "environment-specific/hard" section with the working method — https://<app_id>-443s.<PLATFORM_BASE>/healthz reaches a side directly by app-id (passthrough), independent of the custom domain. switch.sh probes it automatically before a switch when PLATFORM_BASE is set. - Post-switch verification is cache-proof via the served cert fingerprint; the gateway route cache was observed ~30s, so the verify window must exceed it. - Rollback is stateless (live side read from DNS, no local state file). - Limitations updated accordingly (PLATFORM_BASE, openssl, cache window). The whole switch stays in the delegation zone; 0g.ai is untouched after the one-time CNAMEs — no 0g.ai token needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Visualise the cutover in the record-architecture section: the client → gateway → CNAME chain (0g.ai served zone → base switch record → per-side records → CVM), with the one switch-layer CNAME switch.sh flips highlighted, solid = live path, dashed = the alternate the flip selects. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Record that blue/green here assumes both sides run in the same dstack cluster (the serving alias is one static value → one gateway, which only routes to app_ids in its own cluster). Sketch what a future cross-cluster / cluster migration would additionally need — a switched serving alias, per-side GATEWAY_DOMAIN/PLATFORM_BASE, SNI allowlist on both clusters, and a lowered TTL to shrink the cutover's inconsistency window — so the assumption isn't a surprise later. Deferred; not needed until a cluster move actually happens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
served_cert_fp ends in a pipeline that returns non-zero on a transient TLS read failure (pipefail). It is consumed via plain assignments — cert_before=$(...) and cert_now=$(...) — which, unlike `local var=$(...)`, are subject to errexit, so a momentary openssl hiccup aborted the script: - before the flip, skipping the /healthz-only degradation branch just below; - after the flip (worse), aborting mid-verify-loop with traffic already switched and the auto-rollback never reached. Guard both with `|| true` so a failed read yields "" and falls through to the existing handling (degrade / keep polling), as intended. Thanks to the reviewer for catching this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
…llback) Addresses review items 3-7: - switch.env.example: VERIFY_RETRIES default 10 -> 20 (matches the script/header), and add PROBE_RETRIES=5 (the gate-2 knob was undocumented). - Auto-rollback message now distinguishes the two failure modes: "/healthz OK but cert never flipped" (likely gateway route-cache lag — suggests raising VERIFY_RETRIES / lowering TTL) vs "/healthz never healthy". They warrant different operator responses. - Verify loop no longer sleeps after the final attempt (guarded like the probe loop), so the failure path doesn't waste one VERIFY_INTERVAL. - usage(): stop at the first non-comment line instead of at `set -euo pipefail`, so the `if [ -z "$BASH_VERSION" ]; exec bash` re-exec guard no longer leaks into --help output. - rollback: clear PROBE_URL before delegating to switch, so a --probe-url meant for one side can't be used as the gate-2 probe for the rollback target. Thanks to the reviewer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
…_id) Make explicit an assumption the "only the image digest distinguishes the sides" and "staging and prod share app_id" claims silently depend on: allowed_envs is a field of the measured app-compose, so it is part of app_id. Therefore: - allowed_envs must be IDENTICAL on both sides (else they diverge by more than the image digest); - DNS_SETUP_MODE and ACME_STAGING must be listed PERMANENTLY, toggled by value — adding/removing a key edits allowed_envs and changes app_id. Fixes the README's internal inconsistency (it told you to add ACME_STAGING to allowed_envs for staging, then claimed staging and prod share app_id — only true if the list is constant). Updates blue-green.md (app_id note + deploy step), README, and the compose comments to say this once, clearly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
Root cause: side_label's `die` runs inside `$(...)`, so it only exits that subshell — addr_target/acme_target then emitted a name with an empty label (…0g.ai..zone) and exit 0, and side_name echoed unknown input back verbatim. Two live-traffic hazards followed: cmd_switch's auto-rollback with cur_side="?" (move_switches "?" writes the traffic switch to a bogus name), and `acme c` (writes the issuance switch to a bogus name after you confirm — a delayed renewal failure). Three layers, matching the reviewer's suggestions: - side_name now dies on unknown input (not echo it), so `switch c` / `acme c` abort at the argument — before any confirm or DNS write. (cmd_switch/cmd_acme are called directly, so the failed `target=$(side_name …)` trips set -e.) - cmd_switch refuses cur_side="?" up front, like cmd_rollback already did, so auto-rollback never runs against an unrecognized side. - addr_target/acme_target guard side_label with `|| return 1` (its die is swallowed by the subshell), so a bad label yields no output + non-zero rather than a malformed name; the directly-called caller then trips set -e. Verified: `switch c`/`acme c` abort cleanly; move_switches "?" aborts instead of writing a bogus record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
cf failures abort the whole script (cf -> die), so with the traffic switch written first, a 5xx/rate-limit on the subsequent issuance-switch write left traffic pointing at the unverified target with neither the verify loop nor the auto-rollback reached. Write the issuance switch first and the traffic switch last (a single PUT): any failure before it leaves traffic on the current side, and a failure of the traffic write itself doesn't partially apply. Thanks to the reviewer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Zero-downtime blue/green releases for the cloud-TEE gateway (
deploy/phala). Arelease is an atomic DNS pointer flip between two independently-attested CVMs
running different gateway image digests (hence different
app_ids). dstack routesby
app_idand won't load-balance across different images, so a DNS-level flip isthe only viable mechanism. The entire switch lives in the delegation zone
(
integratenetwork.work); the served zone (0g.ai) is touched once at setupand never again, so no
0g.aitoken is required.Validated end-to-end on staging (
router-api-tee-staging.0g.ai): a side bootsunder its sub-zone, issues its cert, is probed directly before cutover, traffic
flips, and the cutover is confirmed cache-proof — with instant rollback.
What's here
deploy/phala/switch.sh— traffic + certificate-issuance switch:setup(one-time serving alias),status,switch a|b,rollback,acme a|bhttps://APP_ID-443s.PLATFORM_BASE/healthz(TLS passthrough; routes by the app-id in the hostname, independent of the
custom domain), retried
PROBE_RETRIEStimes; refuses to switch if the targetis unhealthy
fingerprint to change to the target side's before declaring success (the
dstack gateway caches the
_dstack-app-addresslookup ~30 s, so a bare/healthzcan be answered by the old side for a while); distinguishes"cache still flushing" from "target unhealthy", and auto-rolls-back on failure
record, not a local file, so it is consistent across operators/machines
writing a malformed record, and writes the traffic switch last so a
mid-flight Cloudflare error can't leave traffic flipped-but-unverified
switch.env, git-ignored) that tolerates CRLF; re-execsunder bash if started with
sh;--dry-run,--yes,--probe-url,--no-verifydeploy/phala/docker-compose.yml—DNS_SETUP_MODE=${DNS_SETUP_MODE:-wait}(blue/green sides set
printso ingress skips its strict one-hop DNS pre-checkand can issue under a sub-zone) and
ACME_STAGING=${ACME_STAGING:-false}(injectable; the default reproduces today's real-cert behaviour byte-for-byte)
deploy/phala/blue-green.md— full runbook: record architecture with amermaid diagram of the traffic switch, one-time setup, the required
DNS_SETUP_MODE=print, the issuance switch + Let's Encrypt rate limits, cutover,stateless rollback, standby probing, migration from a single instance, scaling,
and limitations (incl. the same-cluster assumption)
deploy/phala/switch.env.example,deploy/phala/README.md(staging /ACME_STAGING guidance), and
.gitattributes(pin shell scripts to LF)Design notes
0g.aiserved records (static) → the base switchrecord in
integratenetwork.work(the one CNAMEswitch.shflips) → per-sidea./b.records each CVM writes itself. Sides use a per-sideDELEGATION_ZONEso their auto-managed records never collide; dstack-ingress's Cloudflare provider
resolves the longest parent zone, so
a./b.need not be real Cloudflare zonesand one delegation-zone token covers both sides and the switch layer.
DNS_SETUP_MODE=printis required on each side because the served CNAMEsstay pinned at the base zone while a side runs under its sub-zone, so ingress's
one-hop pre-check never matches;
printskips only that pre-check — Let's Encryptand the dstack gateway follow the full CNAME chain, so issuance and routing work.
app_idhashes the app-compose manifest — the compose text andallowed_envs. Only the literal image digest distinguishes the two sides(DOMAIN / DELEGATION_ZONE / ROUTER_URL etc. are
${...}values injected at bootand don't affect
app_id). So the two sides must be different builds, andallowed_envsmust be identical on both — keepDNS_SETUP_MODEandACME_STAGINGpermanently listed and toggle by value; editing the list changesapp_id.routes only to
app_ids in its own cluster. Cross-cluster / cluster migrationneeds extra work (documented under Limitations); deferred until needed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01AVfc5XmR4aKUNLwgFY5oyk