Skip to content

feat(platform): the site's images name a version, and Flagger promotes the pair - #1316

Merged
ExtraToast merged 1 commit into
mainfrom
worktree-platform+lockstep-api-frontend-release
Sep 18, 2026
Merged

ExtraToast merged 1 commit into
mainfrom
worktree-platform+lockstep-api-frontend-release

Conversation

@ExtraToast

@ExtraToast ExtraToast commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Closes #1313. Closes #1294. Part of #1289.

The api and the frontend ran :latest, and Keel polled that tag every two minutes and rolled each Deployment on its own schedule. Two services built from one commit reached production minutes apart, and nothing in git recorded which version was running. These are the first two slices of #1289: name a version, then promote the two services as one thing.

What this achieves

One version is written once, for both images, in apps/stateless/kustomization.yaml. The Deployments carry the image name without a tag, so there is exactly one place a version can be set and the two cannot drift apart.

On top of that, each service is a Flagger blue/green canary. The two confirm-promotion gates read the sibling canary's phase, so whichever finishes analysis first waits, and a failed half parks the other rather than shipping alone.

How

The gate is a symmetric rendezvous rather than an ordering. Each canary's webhook passes when the sibling is neither Progressing nor Failed. Both go Progressing; the first to finish sees the other still progressing and blocks; when the second arrives it sees WaitingPromotion and passes; the first passes on its next 30s check. Promotions land within one interval. If one analysis fails, the other's gate sees Failed and holds indefinitely — confirm gates do not consume the failure threshold, so it waits rather than shipping half a release.

Flagger installs from its own apps-delivery Kustomization, not from apps-core. Everything in the tree depends on apps-core and it carries wait: true, so a Flagger HelmRelease that failed to install there would stop cert-manager, Traefik and Vault reconciling in order to deploy a canary controller. apps-delivery depends on apps-core, and only apps-stateless depends on it.

Flagger owns the apex Services and the replica counts, so the hand-written Services and replicas: 1 are gone from git — leaving replicas in the manifest would make Flux reset it every two minutes against Flagger's scale-to-zero. The Flux health checks name the -primary Deployments, which are what the Services point at.

The frontend gained the probe set the api already had: a startupProbe, and real periodSeconds / timeoutSeconds / failureThreshold on readiness and liveness. Its /healthz deliberately does not probe the api upstream, because an api outage must not also empty the frontend's endpoints. promote in build.yml now moves :latest only for the images Keel still watches.

Not in scope

The digest pin and the pipeline that writes it are #1293, so the tag is bumped by hand until that lands and cutting a release publishes images without deploying them. The migration Job is #1297, canary metrics #1300, Keel's removal #1302.

This does not make the switchover atomic. Promotion copies the canary spec onto the -primary Deployment, which then rolls normally, so the two apex Services finish flipping seconds apart. One release of backward compatibility remains the contract; closing that window needs a version-selected Service, which Flagger's Kubernetes provider does not do.

Worth a reviewer's attention

  • Applying this costs an outage of roughly one api cold start. Flux prunes the hand-written Services in the same apply that creates the Canaries, and Flagger recreates them only once its -primary Deployments are Ready. Flux reconciles main within a minute, so this wants merging with apps-stateless suspended — the runbook now carries that procedure.
  • The canary is not isolated. It shares the production database and, for the api, the same ReadWriteOnce storage volume. Anything later pointed at api-canary must be read-only.
  • Chart value names and Flagger's phase strings come from upstream documentation, not from this cluster. The kubeconfig reachable from this machine points at a different cluster entirely, so none of it has been exercised against ESA production.
  • portDiscovery is left off, so the api's management port 8081 stays pod-local exactly as today and the probes are unchanged.

Verification

  • kubectl kustomize renders apps/stateless, apps/delivery, apps/core and clusters/production clean, with both images resolving to the pinned tag.
  • Every changed manifest and workflow parses as YAML.
  • The gate predicate was exercised directly: WaitingPromotion passes, Progressing and Failed both block.
  • release-please's generic updater behaviour was checked against its upstream source and simulated locally, which is what established that a hand-bumped tag is the honest option here rather than an annotated one.
  • «not verified against a running cluster — no ESA context available to this session»

Diff breakdown added removed, scaled to the largest row.

platform                                          +386    -65   17
  infrastructure     ██████████████████████░░░░   +386    -65   17

ci                                                 +17    -23    1
  build & config     █░                            +17    -23    1

docs                                                +9     -3    1
  docs               █░                             +9     -3    1

──────────────────────────────────────────────────────────────────
total (hand-written)                              +412    -91  19 files

…s the pair

The api and the frontend ran :latest and Keel rolled each on its own
two-minute poll, so two services built from one commit reached production
minutes apart and nothing in git said what was running. The overlay now
names one version for both, in one place, and the keel.sh annotations
come off those two Deployments.

On top of that, each service becomes a Flagger blue/green canary whose
confirm-promotion gate reads the sibling's phase, so a release promotes
both or neither and a failed half parks the other. Flagger installs from
its own apps-delivery Kustomization: everything depends on apps-core, so
a controller only apps-stateless needs must not be able to stall
cert-manager, Traefik and Vault.

The tag is bumped by hand until the release pipeline writes a digest
there, so cutting a release publishes images without deploying them.
Applying this prunes the hand-written Services and Flagger recreates them
only once its -primary Deployments are Ready, which costs about one api
cold start; the runbook carries the suspend-merge-resume procedure.
@ExtraToast ExtraToast added enhancement New feature or request area:ci-cd labels Sep 18, 2026
@ExtraToast ExtraToast self-assigned this Sep 18, 2026
@ExtraToast
ExtraToast merged commit fe4d58f into main Sep 18, 2026
26 checks passed
@ExtraToast
ExtraToast deleted the worktree-platform+lockstep-api-frontend-release branch September 18, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci-cd enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The stateless overlay names a version, and the site's images stop moving on :latest api and frontend promote as a pair, or not at all

1 participant