feat(cloud): support EC2 Spot Instances in kirocrew cloud launch (--spot) - #3355
feat(cloud): support EC2 Spot Instances in kirocrew cloud launch (--spot)#3355so0k wants to merge 1 commit into
Conversation
67d3cea to
893f43a
Compare
893f43a to
68992f6
Compare
|
Note for maintainers: the red Fork workflow-change guard is the designed block for fork PRs touching |
Design Review (Fable 5, fork) — 🟡 CONCERNSAdvisory design-level review of Design-Verdict: CONCERNS Sound spot design end-to-end; the one wart is structure flattened into prose at the API boundary and reverse-parsed client-side with sentinel heuristics. Watch
Suggestions
[DESIGN-REVIEWED] c92d069 |
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed BLOCKING -- src/kiro_crew/cloud/ec2.py:1622 -- stopped Spot teardown can delete data before stack deletion BLOCKING -- src/kiro_crew/cloud/iam.py:321 -- launcher credentials can create uncancellable persistent requests |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI've verified the full chain. Let me confirm the The chain is confirmed:
Production inserts exactly one This is a real, grounded defect in an added line, but it lives in a dev screenshot-capture script ( Screenshot evidence fixture misrepresents the fix: FINDING — website/scripts/capture-cloud-spot-remedies.mjs:5194 — [OPUS-REVIEWED] c92d069 |
68992f6 to
4123060
Compare
4123060 to
2df50fb
Compare
|
Follow-up on the Design Review's 🟡 CONCERNS — all three points are addressed as of
|
b53221e to
efd4509
Compare
UX Review (Fable 5, fork) — 🟡 CONCERNSAdvisory UX-level review of UX-Verdict: CONCERNS The honesty contract is right, but the "still billing" remedies live in the most perishable and most muted parts of the UI. Watch
Suggestions
[UX-REVIEWED] c92d069 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSAdvisory premise-level review of All evidence gathered; here is the review. First-Principles-Verdict: CONCERNS Everything here is derived from AWS platform rules and earns its place — except the start-hint's newline-in-the-error-string protocol, which duplicates a body-key mechanism this same diff ships. What this change shipsIntent: let cloud-launcher users pay Spot instead of on-demand rates for their EC2 crew (#3184) — an ADDITION.
WatchThe comment justifying item 7 — "a sibling key would be silently dropped by every existing caller" — is refuted by this same PR: Subtractions
[FIRST-PRINCIPLES-REVIEWED] c92d069 |
efd4509 to
d5ba747
Compare
|
Response to the GPT 5.6 blocking review — three findings are fixed as of Fixed:
Rebutted:
|
d2cff0b to
aa66ffa
Compare
|
Status as of GPT 5.6 blocking finding (ownership): fixed. The sweep's describe now requires BOTH First-principles subtractions: all taken — UX watch items: the Spot start-failure hint now renders as its own neutral note block with one sentence per line ("Do NOT destroy the instance…" is no longer a paragraph tail beside the Delete button), and Full CI is green on this head apart from the workflow-change guard, which awaits the |
aa66ffa to
c92d069
Compare
|
Response to the GPT 5.6 review of 1. "Stopped Spot teardown can delete data before stack deletion." The scenario requires a successful cancel on a stopped Spot stack followed by a failed
2. "Launcher credentials can create uncancellable persistent requests." AWS explicitly documents that untagged spot-request creation cannot be prevented by IAM — the Both trade-offs are documented in |
|
🤖 Kiro Crew [operator: iamwhatever]: The automated drive-to-green pipeline can't clear the remaining blockers here — they need human/maintainer decisions, so I'm routing this out of the auto-triage lane:
No automated changes will be made. When a maintainer clears the workflow-change guard and adjudicates the blocking review, the PR can proceed. |
|
🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
…pot) Adds an opt-in --spot flag to kirocrew cloud launch that provisions the instance as a persistent Spot request with stop-on-interruption, cutting instance-hour cost 60-90% for the intermittent-use case the cloud launcher targets. Deviation from the proposal in kirodotdev#3184: InstanceMarketOptions is not a valid AWS::EC2::Instance property (cfn-lint E3002; confirmed against the CFN registry schema), so the spot options ride on a conditional AWS::EC2::LaunchTemplate referenced by the instance only when Spot=true. The on-demand resource graph is unchanged (verified via a live create-change-set for both parameter values). Because the root volume is DeleteOnTermination and a persistent request relaunches a replacement instance whenever its instance is terminated without cancelling the request first, this also ships the safety rails the naive flag would have lacked: - cloud destroy cancels the tagged Spot request before delete-stack, and REFUSES to delete a Spot stack whose request cannot be confirmed cancelled (CLI exits 1 with runnable remedies and touches nothing; the dashboard answers 409): deleting anyway would terminate the instance and let the un-cancelled request launch an untracked replacement. The stack's own instance is left for CloudFormation to terminate; only orphan replacement instances are terminated directly. Orphaned requests are swept even when the stack is already gone, behind the same confirmation the stack path uses (cancelling a disabled request auto-terminates its stopped instance). - The Spot request never expires (explicit far-future ValidUntil): a request that expired or was cancelled while the box is stopped would auto-terminate it and take ~/.kiro/crew with it. - The request is tagged at create via the launch template, so the least-privilege launcher policy can tag-gate CancelSpotInstanceRequests; the policy also gains the launch-template lifecycle actions and the scoped service-linked-role grant first Spot use requires. - --spot on a resume mirrors the --subnet guard (warn interactively, hard-fail under -y) instead of silently billing on-demand. - Docs/help state the real recovery semantics: only EC2 can restart an interruption-stopped Spot instance; manual cloud stop/start still work. CI's cfn-lint job now actually lints this template (pin bumped to 1.55.0, glob widened) and template tests parse structure instead of grepping strings. Fixes kirodotdev#3184
c92d069 to
d4ef011
Compare
|
Rebased onto main
Gates run locally: A maintainer push makes the maintainer the last pusher, so a second approver is needed under the repo's last-push rule. Reply if anything looks wrong. |
Problem / Motivation
kirocrew cloud launchalways deploys on-demand pricing — there is no way to ask for a Spot Instance. For the intermittent personal-use case the cloud launcher targets, the instance-hours on top of the fixed NAT floor are real money:t4g.xlarge(Balanced) is ~$0.134/hr andm7g.2xlarge(Power) ~$0.326/hr on-demand, while Spot pricing for the same shapes typically runs 60–90% lower. Full write-up in #3184.Why it matters
Spot changes the "how long can I leave this running" calculus without touching the NAT floor, and it compounds with (rather than replaces) the EventBridge scheduled stop/start some users already run. Without it, cost-conscious users either babysit their instance or don't use the cloud launcher at all.
What changed (motivation → approach → change)
Goal: an additive, opt-in
--spotflag threaded like--subnet(CLI → wizard →ec2.deploy()→ CloudFormation), with on-demand behaviour byte-for-byte unchanged by default.Why not the design proposed in the issue: #3184 (which we filed) proposed attaching
InstanceMarketOptionsdirectly to theAWS::EC2::Instance. While implementing we found that property does not exist on that resource type — cfn-lint 1.55 rejects it (E3002), the CFN registry schema has no such property, and a stack create would hard-fail. The current CI cfn-lint job never noticed because it only lints the artifact-deploy templates, notcloud/templates/.Approach actually built:
Spotparameter (default"false") plus anIsSpotcondition gate a newSpotLaunchTemplate(AWS::EC2::LaunchTemplate,Condition: IsSpot) that carries onlyInstanceMarketOptions(MarketType: spot,SpotInstanceType: persistent,InstanceInterruptionBehavior: stop) and aspot-instances-requestTagSpecification. The instance references it viaLaunchTemplate: !If [IsSpot, {...}, !Ref AWS::NoValue].persistent+stopis the only shape that preserves the root volume (it isDeleteOnTermination: true, so the default one-time/terminate would wipe~/.kiro/crewon interruption), and EC2 auto-resumes a persistent-stopped instance when capacity returns.ValidUntilis pinned far-future (2099-01-01): the LaunchTemplate API documents a 7-day default for persistent requests, and an expired/cancelled request auto-terminates its stopped instance — i.e. the exact data loss this design exists to prevent, on a delay. The explicit date removes that failure mode.cloud destroycancels the Spot request beforedelete-stack— and refuses to delete when it can't confirm the request is gone. Terminating a persistent-spot instance without cancelling first flips the request toopenand EC2 launches a replacement outside the stack — a billing zombie invisible tokirocrew's tag discovery. So: aSpot=truestack whose cancel failed (or whose lookup went unanswered) is not deleted — the CLI exits 1 with the runnableawsremedies and touches no local state; the dashboard answers409 spot_sweep_blocked_destroy. The stack's own instance is left for CloudFormation to terminate (so a rejecteddelete-stackcan never strand a half-destroyed stack whose volume we already deleted); only orphan replacement instances are terminated directly. The sweep runs by tag, also fires when the stack is already gone (cleanup after a rolled-back--spotlaunch) — behind the same confirmation the stack path uses, since cancelling adisabledrequest auto-terminates its stopped instance. On-demand destroys pay one describe call and are otherwise unchanged; users on the previous launcher policy (who cannot have created a spot stack) keep a quiet, successful teardown.iam.py):spot-instances-request/*onRunInstances(unconditioned — AWS documentsaws:RequestTagconditions on that resource as not supported; the template always tags the request instead, which is what letsCancelSpotInstanceRequestsstayaws:ResourceTag-gated), the launch-template lifecycle actions, spot/LT describes, andiam:CreateServiceLinkedRolescoped tospot.amazonaws.com(first Spot use in an account needsAWSServiceRoleForEC2Spot; the console auto-creates it, the CLI does not). The policy had to stay under IAM's hard 6,144-char managed-policy cap, so two groups of same-shape statements were merged (rationale documented in the module and pinned by a size-guard test).--spotwhile resuming an existing stack mirrors the--subnetguard — warns interactively, hard-fails under-y— instead of silently keeping on-demand billing.cloud startworks after a manualcloud stop, not after an interruption — wait for auto-resume); an interruption mid-agent-run kills that run ungracefully, same as a host reboot; capacity varies by tier/region.src/kiro_crew/cloud/templates/and the pin is bumped 1.22.3 → 1.55.0 (1.22.3's stale schema false-flags the template's pre-existing, validMetadataOptions).cloud starton aSpot=truestack now explains itself — "likely an interruption-stop; only EC2 can restart it; your data is intact; do not destroy it" — on both the CLI and the dashboard (fetched only on the failure path). The panel renders remedy commands as copyable<code>, and the softernoticesget a neutral tone instead of the warning amber.ec2.grade_spot_sweep()so the CLI and the dashboard destroy route reach identical verdicts. The dashboard attacheswarnings(and, for the no-stack orphan case, softernotices) to its 200 response, auditspartialinstead ofsuccesswhen work is left, sweeps orphaned requests on an already-absent stack exactly like the CLI, andRemoteCrewPanelrenders those lines in a warn-toned notice (screenshots below). Whether a denied lookup may be quietly shrugged off is decided by the stack's ownSpotparameter, never by inference about the destroying principal — and the launch template also tagsinstance/volume, so a replacement instance launched by a re-opened request is discoverable and terminable by the tag-gated policy.Tests
Net +80 tests across
test_cloud_ec2.py/test_cloud_cli.py/test_cloud_wizard.py/test_cloud_iam.py/test_cloud_handlers.pyplus the frontendRemoteCrewPanel.test.tsx(targeted cloud suite: 612 passed; panel suite: 24 passed). Highlights of what they lock in:build_deploy_argv/deployincludeSpot=trueonly when set (dry-run path covered); on-demand argv unchanged.IsSpot, its SpotOptions are persistent+stop+the exactValidUntil, the instance'sLaunchTemplateproperty has theAWS::NoValuefallback, and the request TagSpecification carries both kirocrew tags.find_stackbefore the mutating sweep, orphan sweep reachable from the CLI when no stack exists, terminal request states excluded client-side (thestatefilter valuedisabledis undocumented, so filtering is by exclusion), denied describe vs. denied cancel vs. denied terminate each graded correctly (exit codes + suppression of the billing claim pinned), agent-session chokepoint refusal caught,--helprenders (a literal%in help text crashed argparse — regression-tested).aws:RequestTagcondition on the spot-request ARN anywhere, policy length under the 6,144-char IAM cap.--spotthreads toec2.deploy, prints the tradeoff, defaults to on-demand, resume guard parity with--subnet.Manual verification
Live server-side validation against a real AWS account (ap-southeast-1):
aws cloudformation create-change-set(no execute) for bothSpot=falseandSpot=true→ bothCREATE_COMPLETE;Spot=falseproduces exactly the pre-change 6-resource graph (no launch template),Spot=trueadds onlySpotLaunchTemplate(re-validated after the TagSpecifications were extended). Change sets and review-state stacks deleted afterwards; no resources were created. A full live--spotlaunch/interrupt/destroy cycle was not run — flagging the destroy-sweep behaviour on a real interruption as the one thing a maintainer may want to see exercised end-to-end before merge.Screenshots
The one UI change:
kirocrew cloud destroy's honesty contract, rendered by the dashboard. When the destroy succeeds but the Spot sweep leaves something live (or unprovable), the response'swarnings/noticesrender as a warn-toned status block above the crew row — each line self-contained with the runnableawsremedy. A clean destroy (and every on-demand teardown) renders nothing new.Light:
A failed
cloud starton an interruption-stopped Spot crew explains itself instead of showing a bare AWS error:Dark variants
Screenshots are reproducible via the committed harness:
node website/scripts/capture-cloud-spot-remedies.mjs.Related Issues
Fixes #3184
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)