Merging two PRs to main in quick succession cancels the first site deploy in flight. Because every site's ECS task definition pins the mutable :latest tag, the surviving run does not necessarily deploy the newest commit — and nothing in the pipeline detects it.
Affects all three callers of site-deploy.yml@v1.2.2 identically: site-pondviewlane-com, site-lentago-dev, site-icecreamtofightwith-com. Each holds the same workflow-level group:
concurrency:
group: deploy-main
cancel-in-progress: true
Concurrency groups are per-repository, so the three sites do not contend with each other. The problem is within a single repo.
What was observed
Merging two Dependabot PRs ~10s apart in site-pondviewlane-com on 2026-08-30:
| Run |
Commit |
PR |
Result |
| 33320559226 |
d050020 |
#72 (nginx base digest) |
cancelled |
| 33320565937 |
75856a6 |
#73 (astro 7.2.8 / starlight 0.41.9) |
success |
This instance was benign. The cancelled run died at step 4 (npm ci), long before docker push, and 75856a6 is a superset of both changes — main and the live site agree. It is the mechanism, not this occurrence, that is worth fixing.
Why it is not reliably benign
cancel-in-progress is documented as making "the most recent push always win cleanly" (comment in site-icecreamtofightwith-com/.github/workflows/deploy.yml). Two properties break that guarantee:
1. The deployed artifact is not identified by commit. site-deploy.yml builds and pushes two tags — :latest and the immutable :${{ github.sha }} — then deploys with:
aws ecs update-service --force-new-deployment
against a task definition whose image is <repo_url>:latest (solidago/modules/site/variables.tf:112, container_image_tag defaults to "latest"; no site caller overrides it). ECS resolves the tag at pull time, so what runs is whatever :latest points to when a task starts, not what the deploying run built. The per-commit tag is pushed and then unused. This also means a later task replacement — scale event, crash, AZ rebalance — pulls whatever :latest has become since, which may be a different commit than the deployment that placed the task.
2. Cancellation is not atomic. A step already executing when the cancel signal arrives runs until the runner's grace period expires. A run cancelled inside docker push can therefore finalize its :latest manifest after the superseding run finalized its own. :latest then resolves to the older commit, the newer run's update-service pulls it, and aws ecs wait services-stable passes — it verifies service stability, never image identity. Result: live serves older content than main, with a green check on the newest commit and no failure anywhere.
The window is narrow (it requires the older run to be mid-push when superseded) and the consequence is silent, which is the combination that makes it worth closing rather than watching for.
3. A cancelled-then-failed pair leaves main undeployed. If the run that cancelled its predecessor subsequently fails — Trivy, ECR flake, build error — the predecessor's commit is on main and was never deployed. This one does surface as a red run, so it is the least severe of the three, but there is no drift check asserting live matches main.
Suggested direction
Deploy by immutable reference instead of :latest. solidago#180 §1 reaches the same conclusion from the security side (mutable :latest is what blocks setting ECR image_tag_mutability = IMMUTABLE), and notes the pieces already exist because SLSA attestation verifies by digest. Sketch:
- Register a new task-definition revision per deploy, pinned to the digest resolved in the existing "Get image digest for attestation" step, and
update-service --task-definition <new-arn> instead of --force-new-deployment.
- Drop the
:latest push, or keep it strictly as a human-facing convenience tag nothing deploys from.
solidago/modules/site then stops owning the deployed tag; container_image_tag becomes a bootstrap-only default.
With a digest-pinned deploy, a cancelled run can no longer influence what a surviving run deploys, and cancel-in-progress becomes purely a cost optimization — which is what it was intended to be.
Coordinating change across site-deploy.yml, solidago/modules/site, and the three callers, so it wants a version bump and a staged adoption rather than an in-place edit. Related: solidago#180 §1 (ECR immutability), solidago#124 (task-def replace-diff — relevant if task defs start being registered by CI).
Merging two PRs to
mainin quick succession cancels the first site deploy in flight. Because every site's ECS task definition pins the mutable:latesttag, the surviving run does not necessarily deploy the newest commit — and nothing in the pipeline detects it.Affects all three callers of
site-deploy.yml@v1.2.2identically:site-pondviewlane-com,site-lentago-dev,site-icecreamtofightwith-com. Each holds the same workflow-level group:Concurrency groups are per-repository, so the three sites do not contend with each other. The problem is within a single repo.
What was observed
Merging two Dependabot PRs ~10s apart in
site-pondviewlane-comon 2026-08-30:d05002075856a6This instance was benign. The cancelled run died at step 4 (
npm ci), long beforedocker push, and75856a6is a superset of both changes —mainand the live site agree. It is the mechanism, not this occurrence, that is worth fixing.Why it is not reliably benign
cancel-in-progressis documented as making "the most recent push always win cleanly" (comment insite-icecreamtofightwith-com/.github/workflows/deploy.yml). Two properties break that guarantee:1. The deployed artifact is not identified by commit.
site-deploy.ymlbuilds and pushes two tags —:latestand the immutable:${{ github.sha }}— then deploys with:against a task definition whose image is
<repo_url>:latest(solidago/modules/site/variables.tf:112,container_image_tagdefaults to"latest"; no site caller overrides it). ECS resolves the tag at pull time, so what runs is whatever:latestpoints to when a task starts, not what the deploying run built. The per-commit tag is pushed and then unused. This also means a later task replacement — scale event, crash, AZ rebalance — pulls whatever:latesthas become since, which may be a different commit than the deployment that placed the task.2. Cancellation is not atomic. A step already executing when the cancel signal arrives runs until the runner's grace period expires. A run cancelled inside
docker pushcan therefore finalize its:latestmanifest after the superseding run finalized its own.:latestthen resolves to the older commit, the newer run'supdate-servicepulls it, andaws ecs wait services-stablepasses — it verifies service stability, never image identity. Result: live serves older content thanmain, with a green check on the newest commit and no failure anywhere.The window is narrow (it requires the older run to be mid-push when superseded) and the consequence is silent, which is the combination that makes it worth closing rather than watching for.
3. A cancelled-then-failed pair leaves
mainundeployed. If the run that cancelled its predecessor subsequently fails — Trivy, ECR flake, build error — the predecessor's commit is onmainand was never deployed. This one does surface as a red run, so it is the least severe of the three, but there is no drift check asserting live matchesmain.Suggested direction
Deploy by immutable reference instead of
:latest.solidago#180§1 reaches the same conclusion from the security side (mutable:latestis what blocks setting ECRimage_tag_mutability = IMMUTABLE), and notes the pieces already exist because SLSA attestation verifies by digest. Sketch:update-service --task-definition <new-arn>instead of--force-new-deployment.:latestpush, or keep it strictly as a human-facing convenience tag nothing deploys from.solidago/modules/sitethen stops owning the deployed tag;container_image_tagbecomes a bootstrap-only default.With a digest-pinned deploy, a cancelled run can no longer influence what a surviving run deploys, and
cancel-in-progressbecomes purely a cost optimization — which is what it was intended to be.Coordinating change across
site-deploy.yml,solidago/modules/site, and the three callers, so it wants a version bump and a staged adoption rather than an in-place edit. Related:solidago#180§1 (ECR immutability),solidago#124(task-def replace-diff — relevant if task defs start being registered by CI).