Skip to content

feat(cloud): support EC2 Spot Instances in kirocrew cloud launch (--spot) - #3355

Open
so0k wants to merge 1 commit into
kirodotdev:mainfrom
so0k:feat/cloud-spot-launch
Open

feat(cloud): support EC2 Spot Instances in kirocrew cloud launch (--spot)#3355
so0k wants to merge 1 commit into
kirodotdev:mainfrom
so0k:feat/cloud-spot-launch

Conversation

@so0k

@so0k so0k commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Problem / Motivation

kirocrew cloud launch always deploys on-demand pricing — there is no way to ask for a Spot Instance. For the intermittent personal-use case the cloud launcher targets, the instance-hours on top of the fixed NAT floor are real money: t4g.xlarge (Balanced) is ~$0.134/hr and m7g.2xlarge (Power) ~$0.326/hr on-demand, while Spot pricing for the same shapes typically runs 60–90% lower. Full write-up in #3184.

Why it matters

Spot changes the "how long can I leave this running" calculus without touching the NAT floor, and it compounds with (rather than replaces) the EventBridge scheduled stop/start some users already run. Without it, cost-conscious users either babysit their instance or don't use the cloud launcher at all.

What changed (motivation → approach → change)

Goal: an additive, opt-in --spot flag threaded like --subnet (CLI → wizard → ec2.deploy() → CloudFormation), with on-demand behaviour byte-for-byte unchanged by default.

Why not the design proposed in the issue: #3184 (which we filed) proposed attaching InstanceMarketOptions directly to the AWS::EC2::Instance. While implementing we found that property does not exist on that resource type — cfn-lint 1.55 rejects it (E3002), the CFN registry schema has no such property, and a stack create would hard-fail. The current CI cfn-lint job never noticed because it only lints the artifact-deploy templates, not cloud/templates/.

Approach actually built:

  • Template: a Spot parameter (default "false") plus an IsSpot condition gate a new SpotLaunchTemplate (AWS::EC2::LaunchTemplate, Condition: IsSpot) that carries only InstanceMarketOptions (MarketType: spot, SpotInstanceType: persistent, InstanceInterruptionBehavior: stop) and a spot-instances-request TagSpecification. The instance references it via LaunchTemplate: !If [IsSpot, {...}, !Ref AWS::NoValue]. persistent + stop is the only shape that preserves the root volume (it is DeleteOnTermination: true, so the default one-time/terminate would wipe ~/.kiro/crew on interruption), and EC2 auto-resumes a persistent-stopped instance when capacity returns.
  • ValidUntil is pinned far-future (2099-01-01): the LaunchTemplate API documents a 7-day default for persistent requests, and an expired/cancelled request auto-terminates its stopped instance — i.e. the exact data loss this design exists to prevent, on a delay. The explicit date removes that failure mode.
  • cloud destroy cancels the Spot request before delete-stack — and refuses to delete when it can't confirm the request is gone. Terminating a persistent-spot instance without cancelling first flips the request to open and EC2 launches a replacement outside the stack — a billing zombie invisible to kirocrew's tag discovery. So: a Spot=true stack whose cancel failed (or whose lookup went unanswered) is not deleted — the CLI exits 1 with the runnable aws remedies and touches no local state; the dashboard answers 409 spot_sweep_blocked_destroy. The stack's own instance is left for CloudFormation to terminate (so a rejected delete-stack can never strand a half-destroyed stack whose volume we already deleted); only orphan replacement instances are terminated directly. The sweep runs by tag, also fires when the stack is already gone (cleanup after a rolled-back --spot launch) — behind the same confirmation the stack path uses, since cancelling a disabled request auto-terminates its stopped instance. On-demand destroys pay one describe call and are otherwise unchanged; users on the previous launcher policy (who cannot have created a spot stack) keep a quiet, successful teardown.
  • Least-privilege launcher policy (iam.py): spot-instances-request/* on RunInstances (unconditioned — AWS documents aws:RequestTag conditions on that resource as not supported; the template always tags the request instead, which is what lets CancelSpotInstanceRequests stay aws:ResourceTag-gated), the launch-template lifecycle actions, spot/LT describes, and iam:CreateServiceLinkedRole scoped to spot.amazonaws.com (first Spot use in an account needs AWSServiceRoleForEC2Spot; the console auto-creates it, the CLI does not). The policy had to stay under IAM's hard 6,144-char managed-policy cap, so two groups of same-shape statements were merged (rationale documented in the module and pinned by a size-guard test).
  • Resume guard: --spot while resuming an existing stack mirrors the --subnet guard — warns interactively, hard-fails under -y — instead of silently keeping on-demand billing.
  • Honest semantics in help + docs: only EC2 can restart an interruption-stopped Spot instance (cloud start works after a manual cloud stop, not after an interruption — wait for auto-resume); an interruption mid-agent-run kills that run ungracefully, same as a host reboot; capacity varies by tier/region.
  • CI: the cfn-lint job's glob now covers src/kiro_crew/cloud/templates/ and the pin is bumped 1.22.3 → 1.55.0 (1.22.3's stale schema false-flags the template's pre-existing, valid MetadataOptions).
  • Interruption UX (from the AI UX review): a failed cloud start on a Spot=true stack now explains itself — "likely an interruption-stop; only EC2 can restart it; your data is intact; do not destroy it" — on both the CLI and the dashboard (fetched only on the failure path). The panel renders remedy commands as copyable <code>, and the softer notices get a neutral tone instead of the warning amber.
  • Dashboard parity (added after the AI design review flagged the gap): the sweep grading lives in ec2.grade_spot_sweep() so the CLI and the dashboard destroy route reach identical verdicts. The dashboard attaches warnings (and, for the no-stack orphan case, softer notices) to its 200 response, audits partial instead of success when work is left, sweeps orphaned requests on an already-absent stack exactly like the CLI, and RemoteCrewPanel renders those lines in a warn-toned notice (screenshots below). Whether a denied lookup may be quietly shrugged off is decided by the stack's own Spot parameter, never by inference about the destroying principal — and the launch template also tags instance/volume, so a replacement instance launched by a re-opened request is discoverable and terminable by the tag-gated policy.

Tests

Net +80 tests across test_cloud_ec2.py / test_cloud_cli.py / test_cloud_wizard.py / test_cloud_iam.py / test_cloud_handlers.py plus the frontend RemoteCrewPanel.test.tsx (targeted cloud suite: 612 passed; panel suite: 24 passed). Highlights of what they lock in:

  • build_deploy_argv/deploy include Spot=true only when set (dry-run path covered); on-demand argv unchanged.
  • Template structure is parsed, not grepped: a CFN-short-tag-tolerant loader asserts the LT exists only under IsSpot, its SpotOptions are persistent+stop+the exact ValidUntil, the instance's LaunchTemplate property has the AWS::NoValue fallback, and the request TagSpecification carries both kirocrew tags.
  • Destroy sweep: cancel-then-terminate ordering, find_stack before the mutating sweep, orphan sweep reachable from the CLI when no stack exists, terminal request states excluded client-side (the state filter value disabled is undocumented, so filtering is by exclusion), denied describe vs. denied cancel vs. denied terminate each graded correctly (exit codes + suppression of the billing claim pinned), agent-session chokepoint refusal caught, --help renders (a literal % in help text crashed argparse — regression-tested).
  • IAM: policy statements re-pinned, no aws:RequestTag condition on the spot-request ARN anywhere, policy length under the 6,144-char IAM cap.
  • Wizard: --spot threads to ec2.deploy, prints the tradeoff, defaults to on-demand, resume guard parity with --subnet.
  • Dashboard: destroy responses surface sweep warnings/notices with the runnable remedies; clean sweeps leave the response byte-identical; the panel renders them (and a clean destroy renders nothing new).

Manual verification

Live server-side validation against a real AWS account (ap-southeast-1): aws cloudformation create-change-set (no execute) for both Spot=false and Spot=true → both CREATE_COMPLETE; Spot=false produces exactly the pre-change 6-resource graph (no launch template), Spot=true adds only SpotLaunchTemplate (re-validated after the TagSpecifications were extended). Change sets and review-state stacks deleted afterwards; no resources were created. A full live --spot launch/interrupt/destroy cycle was not run — flagging the destroy-sweep behaviour on a real interruption as the one thing a maintainer may want to see exercised end-to-end before merge.

Screenshots

The one UI change: kirocrew cloud destroy's honesty contract, rendered by the dashboard. When the destroy succeeds but the Spot sweep leaves something live (or unprovable), the response's warnings/notices render as a warn-toned status block above the crew row — each line self-contained with the runnable aws remedy. A clean destroy (and every on-demand teardown) renders nothing new.

Light:

Destroy sweep warnings, light theme

A failed cloud start on an interruption-stopped Spot crew explains itself instead of showing a bare AWS error:

Start failure hint on an interrupted Spot crew, light theme

Dark variants

Destroy sweep warnings, dark theme

Start failure hint, dark theme

Screenshots are reproducible via the committed harness: node website/scripts/capture-cloud-spot-remedies.mjs.

Related Issues

Fixes #3184

Checklist

  • Single commit with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@so0k
so0k requested a review from a team as a code owner August 13, 2026 18:09
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Aug 13, 2026
Comment thread docs/guides/remote-crew-on-ec2.md
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 67d3cea to 893f43a Compare August 14, 2026 00:21
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 893f43a to 68992f6 Compare August 14, 2026 00:36
@so0k

so0k commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Note for maintainers: the red Fork workflow-change guard is the designed block for fork PRs touching .github/** — this PR deliberately edits ci.yml in one place: the CloudFormation Lint step's pin is bumped cfn-lint 1.22.3 → 1.55.0 (1.22.3's stale AWS::EC2::Instance schema false-flags the template's long-valid MetadataOptions) and its glob is widened to also cover src/kiro_crew/cloud/templates/ — the cloud launcher template was previously not linted at all, which is exactly how the invalid-property design originally proposed in #3184 would have sailed through CI. Both template directories pass cfn-lint 1.55.0 locally, and the Lint CloudFormation templates check in this very run is green. Everything else in the run is green (2026-08-14). Once the workflow diff is reviewed, the allow-fork-workflow-change label re-evaluates the guard.

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Advisory design-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — updated in place on each push; does not block merge.

Design-Verdict: CONCERNS

Sound spot design end-to-end; the one wart is structure flattened into prose at the API boundary and reverse-parsed client-side with sentinel heuristics.

Watch

  • The backend holds every remedy structured (grade_spot_sweep's {summary, details}, the SPOT_START_FAILURE_HINT tuple), then flattens to strings the panel re-derives via lastIndexOf('aws ec2 '), the failed: regex, and a sentence-split on /\.\s+(?=[A-Z])/ — three exported parsers whose contract is "the command is last" and "the hint is after the first newline". Cause → mechanism → consequence: a future remedy that isn't aws ec2 … (e.g. an aws cloudformation command), or a translated hint (the panel ships in 12 languages; the capital-letter split is Latin-only), silently degrades back into the wrapped-paragraph rendering this machinery exists to prevent — with nothing failing loudly. The wording comments say "reword freely," but the format IS load-bearing.

Suggestions

  • Return problems as structured JSON ({summary, aws_error, command}) on the 200/409 bodies and let the CLI and panel each flatten for display — same shared grader, same no-drift property, zero client-side sentinel parsing; keep the newline-in-error trick only for the start-hint path if touching friendlyErrText is out of scope.

[DESIGN-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline; updated in place on each push.

BLOCKING -- src/kiro_crew/cloud/ec2.py:1622 -- stopped Spot teardown can delete data before stack deletion
sweep = cancel_spot_requests(tag, profile, region, exclude_instance_id=...)
Stopped Spot stack -> cancellation auto-terminates the instance and its DeleteOnTermination volume -> a subsequent delete-stack failure leaves the stack but destroys ~/.kiro/crew.
Anchor: residual/crash-data-loss-corruption
Fix: Revert the Spot launch/destroy hunks until stopped-stack teardown preserves the volume when stack deletion fails.

BLOCKING -- src/kiro_crew/cloud/iam.py:321 -- launcher credentials can create uncancellable persistent requests
`"[REDACTED-ARN]
Compromised launcher credentials -> tagged instance plus untagged persistent Spot request -> this unconditional grant authorizes creation, while the resource-tagged cancel grant rejects cleanup -> replacements continue after termination.
Anchor: residual/security
Fix: Revert the unconditional Spot-request grant and Spot feature until every authorized request can be cancelled fail-closed.

[BLOCK-MERGE] c92d069
[GPT-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline; updated in place on each push.

Review details

I've verified the full chain. Let me confirm the errMsgsplitSpotStartHint path once more and check the production join format is a single newline (already seen at patch line 2373: message = "\n".join([" ".join(message.split()), " ".join(hint)])).

The chain is confirmed:

  • errMsg returns e.message unchanged (RemoteCrewPanel.tsx:695).
  • For the fixture's 502 {error: START_ERROR}, friendlyErrText unwraps error → the raw START_ERROR string (client.ts:830-831).
  • START_ERROR (patch 5194-5205) is space-joined with no \n.
  • splitSpotStartHint (patch 5410-5412): indexOf('\n') = -1 → returns {error: whole message, hint: []}.
  • onError (patch 5467-5471): setActionErr(whole string), setActionNote([]).

Production inserts exactly one \n (patch 2373); the fixture omits it, so the screenshot renders the hint inside the red ErrorNotice banner rather than the neutral note block. The unit test's flattened (patch 5785) correctly uses ${awsError}\n, confirming the intended format. The fixture's own comment (patch 5192-5193) states it reproduces the gateway's error + appended hint, so the omission contradicts its stated contract.

This is a real, grounded defect in an added line, but it lives in a dev screenshot-capture script (website/scripts/) — not a production path. No security/crash/data-loss/corruption/removed-guard. It classifies as advisory FINDING.

Screenshot evidence fixture misrepresents the fix: START_ERROR omits the \n the gateway inserts, so start-interrupted-*.png shows the interruption hint jammed in the red error banner — the exact pre-fix UX the panel change eliminates.

FINDING — website/scripts/capture-cloud-spot-remedies.mjs:5194 — START_ERROR is space-joined with no newline, so splitSpotStartHint returns hint: [] and the "Do NOT destroy the instance…" hint renders inside the red ErrorNotice instead of the neutral note block, making the committed evidence screenshot demonstrate the pre-fix behavior → Fix: join the AWS error and hint sentences with '\n' exactly as handlers_cloud._mutate_instance does (`${awsError}\n` + hint.join(' ')), matching the unit test's flattened fixture.

[OPUS-REVIEWED] c92d069

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 68992f6 to 4123060 Compare August 14, 2026 02:00
@so0k
so0k requested a review from a team August 14, 2026 02:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 4123060 to 2df50fb Compare August 14, 2026 02:12
@so0k

so0k commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up on the Design Review's 🟡 CONCERNS — all three points are addressed as of 2df50fbb2:

  1. Dashboard destroy ignored spot_sweep → the grading now lives at the seam, as suggested: ec2.grade_spot_sweep() is the single grader both surfaces render. The dashboard destroy route attaches warnings to its 200 (same wording and runnable aws remedies the CLI prints before exiting 1), audits partial instead of success when the sweep left work behind, and also performs the CLI's no-stack orphan sweep on an already_absent result. RemoteCrewPanel renders the lines in a warn-toned status block — screenshots now in the PR body.

  2. SWEEP_ERROR_ACCESS_DENIED assumed destroyer == launcher → the quiet path is no longer an inference about the principal: the decision comes from the destroyed stack's own Spot template parameter (read from the describe-stacks payload find_stack already fetched — zero extra calls). Denied lookup on a Spot=true stack is a sweep failure (warn + manual describe remedy + no billing reassurance + rc 1); on a Spot=false stack the quiet path survives, now justified by the stack; with no stack at all the CLI and dashboard both emit the honest "nothing proves it either way" line.

  3. Replacement instances inherit no tags → the launch template's TagSpecifications now covers instance and volume alongside spot-instances-request (verified against AWS docs that LT-level and request-level tag specs merge, request wins on duplicates — so the primary launch is a harmless duplicate). A replacement launched from a re-opened request is thereby discoverable by the sweep's tag-filtered describe and terminable by the tag-gated ec2:TerminateInstances. The template re-validated server-side via a fresh no-execute change set after this change.

@so0k
so0k force-pushed the feat/cloud-spot-launch branch 2 times, most recently from b53221e to efd4509 Compare August 16, 2026 06:17
@github-actions

github-actions Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

UX Review (Fable 5, fork) — 🟡 CONCERNS

Advisory UX-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — updated in place on each push; does not block merge.

UX-Verdict: CONCERNS

The honesty contract is right, but the "still billing" remedies live in the most perishable and most muted parts of the UI.

Watch

  • The consequence clause gets the muted treatment. splitSweepError mutes everything from the AWS-error mark to the end of the prose, which includes the panel-authored "Cancel them yourself or EC2 keeps launching replacements:" tail (the PR's own test pins awsError swallowing it). Every warned destroy with an AWS error — the common case — greys out the one sentence saying why to run the command. Fix: end the muted span at the AWS sentence, or move the instruction into the summary.
  • Warnings are one-shot component state. actionWarn dies on dismiss, tab switch, or reload; after that the dashboard has no trace that a Spot request is still billing, and destroy can't be re-run to resurface it. Rare × real money × unrecoverable. Smallest fix: persist sweep warnings with the launch/job record or a notification.
  • "Do NOT destroy the instance…" ships in the neutral info tone (actionNote, accent/Info) while a Delete affordance sits in the same panel — the diff's highest-stakes, data-loss-preventing line gets the softest treatment users are trained to skim. Promote that one line (not the whole hint) to warn tone.

Suggestions

  • Hint lines render backticks literally in the note block ("check `kirocrew cloud status`.") — strip them or render as <code> in SweepRemedy's verbatim path.

[UX-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Advisory premise-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push; does not block merge.

All evidence gathered; here is the review.

First-Principles-Verdict: CONCERNS

Everything here is derived from AWS platform rules and earns its place — except the start-hint's newline-in-the-error-string protocol, which duplicates a body-key mechanism this same diff ships.

What this change ships

Intent: let cloud-launcher users pay Spot instead of on-demand rates for their EC2 crew (#3184) — an ADDITION.

  1. cloud launch --spot provisions the box on Spot pricing — justified
  2. Persistent + stop + never-expire Spot shape via conditional launch template — justified (E3002, DeleteOnTermination, 7-day-expiry rules)
  3. destroy cancels the request first; refuses delete when unconfirmed (exit 1 / 409) — justified
  4. destroy on a stackless tag probes leftover requests, asks before cancelling — justified
  5. Failed start on a Spot crew explains the interruption (CLI + dashboard) — justified
  6. Destroy 200 carries warnings/notices; panel renders copyable remedy commands — justified
  7. Start hint flattened into the 502 error string, re-split client-side — duplicate of ApiError.body key (this diff's own 409 path)
  8. IAM: spot/LT grants + Spot service-linked role; 3 statements merged under the 6,144-char cap — justified
  9. cfn-lint glob now covers cloud/templates/; pin 1.22.3 → 1.55.0 — justified, cause-level (the unlinted dir is how feat(cloud): support EC2 Spot Instances in kirocrew cloud launch for cost savings #3184's invalid design would have shipped; all template dirs now covered — 0 unfixed siblings)
  10. Destroy audits partial when sweep work is left — justified

Watch

The comment justifying item 7 — "a sibling key would be silently dropped by every existing caller" — is refuted by this same PR: sweepRemediesFromError (RemoteCrewPanel.tsx) already parses a sibling warnings key out of ApiError.body for the 409, and the only caller that renders the hint is the panel this PR edits. Callers that read only error would drop the hint — which is exactly today's behavior, so nobody is worse off.

Subtractions

  • Replace the in-band hint protocol with a hint: string[] sibling key on the 502 JSON body, read via ApiError.body like the 409's warnings: deletes splitSpotStartHint, the sentence-boundary regex (/\.\s+(?=[A-Z])/), and the whitespace-collapse step in handlers_cloud.py whose sole job is keeping the delimiter unique.

[FIRST-PRINCIPLES-REVIEWED] c92d069

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Aug 16, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from efd4509 to d5ba747 Compare August 16, 2026 07:45
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Aug 16, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Aug 16, 2026
@so0k

so0k commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

Response to the GPT 5.6 blocking review — three findings are fixed as of d5ba74701, two are rebutted with evidence:

Fixed:

  • "Deletion proceeds after Spot cancellation fails"ec2.destroy now refuses to issue delete-stack on a Spot=true stack whose sweep left live risk (failed cancel, or an unanswered lookup — both hide exactly the request that would zombie). CLI exits 1 with the runnable remedies and touches no local state; the dashboard answers 409 spot_sweep_blocked_destroy and starts no teardown. A failed terminate after a successful cancel is deliberately not live risk (the request is dead; nothing can relaunch). Pinned by TestDestroyRefusesToDeleteAfterABadSweep incl. a live-risk truth table.
  • "Pre-delete sweep terminates the stack-owned instance" → the stack's own InstanceId (read from the describe payload find_stack already had) is now excluded from the sweep's terminate on the stack-exists path — CloudFormation does that terminating. Only orphan replacement instances are terminated directly. The no-stack path still terminates everything its requests point at.
  • "Orphan cleanup bypasses destroy confirmation" → the no-stack branch now probes read-only (ec2.probe_spot_requests), prints what it found including the stopped-instance consequence, and requires the same confirmation the stack path uses (-y skips; decline touches nothing; nothing-found keeps the old silent path).

Rebutted:

  • "CancelSpotInstanceRequests bypasses the agent command guard" — this conflates the IAM launcher policy (human AWS credentials, aws:ResourceTag-gated) with the agent-session chokepoint. ec2 cancel-spot-instance-requests (and the describe, and terminate-instances) are not in aws._AGENT_READ_ALLOWLIST, so any agent-session invocation is refused at run_aws before AWS is reached — the sweep even catches CloudActionDenied to keep its no-raise contract, pinned by test_agent_session_refusal_is_caught_not_raised and the handler-level agent-guard test. Injected content driving the agent cannot reach this verb; removing the grant would only break the human teardown path.
  • "Unknown Spot ownership fails open (not stack_is_spot treats None as harmless)" — deliberate and documented: None means no stack exists, where the common case by far is an old-policy on-demand user destroying a stale tag; escalating that to a failure would exit 1 on every such teardown (and in scripts, loop-fail them all). The case is not silent: both surfaces emit the honest "nothing proves it either way — check the EC2 console if this tag ever ran --spot" line. And after the fix above, the no-stack path now probes and confirms before acting, which further narrows the window this finding worried about. If maintainers prefer fail-closed here it is a one-expression change (stack_is_spot is False), but we think the current grading is the right default.

@so0k
so0k force-pushed the feat/cloud-spot-launch branch 2 times, most recently from d2cff0b to aa66ffa Compare August 16, 2026 08:39
@so0k

so0k commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

Status as of aa66ffae1 — responding to the latest review round and declaring this iteration settled from our side:

GPT 5.6 blocking finding (ownership): fixed. The sweep's describe now requires BOTH kirocrew:managed=true AND kirocrew:instance=<tag> (AWS ANDs the filters), matching the ownership rule find_stack and the tag-gated IAM cancel already enforce — a foreign Spot request that merely shares the instance tag can no longer be cancelled (or its instance terminated) by cloud destroy. The printed manual remedy carries both filters too, so the "check it yourself" command answers exactly the question the sweep would have.

First-principles subtractions: all taken — DESTROY_ABORT_SPOT_SWEEP/abort_reason deleted (aborted is the branch every caller takes), exclude_instance_ids singularized, and the 409 body trimmed to exactly {error, code, warnings}.

UX watch items: the Spot start-failure hint now renders as its own neutral note block with one sentence per line ("Do NOT destroy the instance…" is no longer a paragraph tail beside the Delete button), and SweepRemedy mutes the raw AWS error segment so the actionable sentences carry the weight — screenshots in the body regenerated via the committed harness. Deferred as follow-up work rather than grown into this PR: persisting the post-destroy billing warning beyond component state — that needs server-side unresolved-leftover state (the audit trail already records the partial outcome), and we'd rather ship it as its own reviewed change than widen this diff further. Same for the Design review's standing notes (IAM policy at ~98% of the 6,144-char cap, the prose-parsed remedy contract): both are documented in-module with guard tests and flagged for maintainer judgement.

Full CI is green on this head apart from the workflow-change guard, which awaits the allow-fork-workflow-change label after maintainer review of the ci.yml diff (cfn-lint pin bump + widened glob, described in the body).

@so0k
so0k force-pushed the feat/cloud-spot-launch branch from aa66ffa to c92d069 Compare August 16, 2026 08:50
@so0k

so0k commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

Response to the GPT 5.6 review of c92d0697f — both findings are analyzed and rebutted; no code change is warranted, and "revert the feature" is not proportionate to either:

1. "Stopped Spot teardown can delete data before stack deletion." The scenario requires a successful cancel on a stopped Spot stack followed by a failed delete-stack. Three points:

  • The instance termination in that scenario is AWS's own semantics — cancelling a disabled request auto-terminates its stopped instance (documented, and quoted in the sweep's docstring). Our terminate call already excludes the stack-owned instance.
  • The user invoked destroy: volume deletion is the requested outcome. The residual — "data gone, stack shell remains until destroy is re-run" — is exactly the residual the existing on-demand path has (CloudFormation terminates the instance before deleting the SG/role; any later step can fail and leave a half-deleted stack whose volume is already gone). This PR does not introduce that shape.
  • There is provably no safer ordering under AWS's rules: delete-before-cancel re-opens the request and launches an untracked billing replacement (the hazard this PR's abort logic exists to stop); cancel-before-delete lets the Spot service reap a stopped instance. Between "user's requested deletion happens slightly early in a rare partial-failure" and "un-tracked instance bills forever," the PR picks the former — and refuses to proceed at all when the cancel did NOT succeed.

2. "Launcher credentials can create uncancellable persistent requests." AWS explicitly documents that untagged spot-request creation cannot be prevented by IAM — the aws:RequestTag condition on the spot-instances-request resource for RunInstances is the construct AWS's own example policy labels NOT SUPPORTED (the resource is simply not evaluated when no tag spec is passed). So the create side is un-gateable by design; the only available lever is the cancel grant's scope. Unconditioning ec2:CancelSpotInstanceRequests would let the same compromised credential cancel unrelated workloads' Spot requests — trading a bounded self-inflicted cost (an account admin can always cancel the rogue request; the console shows it) for a real cross-workload destructive grant. The tag-gated cancel is the deliberate, in-module-documented choice; flipping it is a one-line maintainer decision if they weigh the trade differently.

Both trade-offs are documented in iam.py/ec2.py comments and the module spec precisely so maintainers can arbitrate them with full context. Happy to implement either alternative if maintainers prefer — but neither finding identifies a defect in what this PR ships.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Aug 16, 2026
@iamwhatever iamwhatever added the needs-pr-triage PR scanner: awaiting automated triage label Aug 21, 2026
@iamwhatever

Copy link
Copy Markdown
Collaborator

🤖 Kiro Crew [operator: iamwhatever]: The automated drive-to-green pipeline can't clear the remaining blockers here — they need human/maintainer decisions, so I'm routing this out of the auto-triage lane:

  • Fork workflow-change guard (red, by design): this PR deliberately edits .github/ci.yml. That guard is a hard, maintainer-only control for fork PRs touching .github/**; a rebase/auto-push won't clear it, so a maintainer has to decide whether to accept the workflow edit.
  • GPT 5.6 blocking review is disputed: you've fixed several findings and rebutted the remaining two as "no code change warranted / revert not proportionate." Adjudicating a blocking-review-vs-author disagreement is a maintainer call — the pipeline won't side against your design or revert the feature.
  • Merge conflict: the branch is behind main; a rebase is needed, but on its own it won't get this green given the guard above.

No automated changes will be made. When a maintainer clears the workflow-change guard and adjudicates the blocking review, the PR can proceed.

@iamwhatever iamwhatever added needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Aug 21, 2026
@iamwhatever

Copy link
Copy Markdown
Collaborator

🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • Data-loss risk in stopped-Spot teardown (src/kiro_crew/cloud/ec2.py:1622, GPT blocking): cancelling the Spot request auto-terminates the instance and its DeleteOnTermination volume before stack deletion, so a later delete-stack failure can destroy ~/.kiro/crew. The safe teardown ordering / volume-preservation approach is a design decision in a data-destroying path — not a mechanical fix.
  • IAM grant scoping (src/kiro_crew/cloud/iam.py:321, GPT blocking): compromised launcher credentials could create an untagged, uncancellable persistent Spot request (the cancel grant is resource-tagged but the create grant is unconditional), so replacements keep launching after termination. Making cancellation fail-closed is a policy decision.
  • Fork CI-workflow change: this PR edits .github/workflows/ci.yml (bumps cfn-lint 1.22.3→1.55.0 and widens the CFN lint glob). The Fork workflow-change guard blocks fork PRs that modify workflows by design — a maintainer must decide whether to accept it; the automated pipeline will not touch .github/**.
  • Merge conflict: the branch conflicts with main and needs a rebase, but resolving it depends on the two design decisions above.

When you've addressed these, the pipeline will re-assess on its next cycle.

@bolichen97
bolichen97 enabled auto-merge (squash) August 24, 2026 07:01
@bolichen97

Copy link
Copy Markdown
Collaborator

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #2188 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #2188: CONTINUE_DEVELOPMENT. Independent features in the same module; both can land, with only ordinary same-file rebase churn in the docs and cloud tests. Files: src/kiro_crew/cloud/templates/kirocrew-ec2.yaml. The two independent directions used different labels; the matrix conservatively retains OVERLAPPING for coordination.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

…pot)

Adds an opt-in --spot flag to kirocrew cloud launch that provisions the
instance as a persistent Spot request with stop-on-interruption, cutting
instance-hour cost 60-90% for the intermittent-use case the cloud
launcher targets.

Deviation from the proposal in kirodotdev#3184: InstanceMarketOptions is not a
valid AWS::EC2::Instance property (cfn-lint E3002; confirmed against
the CFN registry schema), so the spot options ride on a conditional
AWS::EC2::LaunchTemplate referenced by the instance only when Spot=true.
The on-demand resource graph is unchanged (verified via a live
create-change-set for both parameter values).

Because the root volume is DeleteOnTermination and a persistent request
relaunches a replacement instance whenever its instance is terminated
without cancelling the request first, this also ships the safety rails
the naive flag would have lacked:

- cloud destroy cancels the tagged Spot request before delete-stack,
  and REFUSES to delete a Spot stack whose request cannot be confirmed
  cancelled (CLI exits 1 with runnable remedies and touches nothing;
  the dashboard answers 409): deleting anyway would terminate the
  instance and let the un-cancelled request launch an untracked
  replacement. The stack's own instance is left for CloudFormation to
  terminate; only orphan replacement instances are terminated directly.
  Orphaned requests are swept even when the stack is already gone,
  behind the same confirmation the stack path uses (cancelling a
  disabled request auto-terminates its stopped instance).
- The Spot request never expires (explicit far-future ValidUntil): a
  request that expired or was cancelled while the box is stopped would
  auto-terminate it and take ~/.kiro/crew with it.
- The request is tagged at create via the launch template, so the
  least-privilege launcher policy can tag-gate CancelSpotInstanceRequests;
  the policy also gains the launch-template lifecycle actions and the
  scoped service-linked-role grant first Spot use requires.
- --spot on a resume mirrors the --subnet guard (warn interactively,
  hard-fail under -y) instead of silently billing on-demand.
- Docs/help state the real recovery semantics: only EC2 can restart an
  interruption-stopped Spot instance; manual cloud stop/start still work.

CI's cfn-lint job now actually lints this template (pin bumped to
1.55.0, glob widened) and template tests parse structure instead of
grepping strings.

Fixes kirodotdev#3184
@bolichen97
bolichen97 force-pushed the feat/cloud-spot-launch branch from c92d069 to d4ef011 Compare September 8, 2026 22:28
@bolichen97

Copy link
Copy Markdown
Collaborator

Rebased onto main 6c24f116e by a maintainer as part of the 2026-09-08 open-PR audit (was 2909 commits behind, mergeable_state: dirty). Please review the resolutions:

  • src/kiro_crew/cloud/ec2.py — 5 hunks: dropped the dashboard_port plumbing this branch re-added (main removed it in chore(cloud): pin the stock-port invariant and drop the dead dashboard_port plumbing #5452) and kept only the spot parameter/override/pass-through.
  • src/kiro_crew/cloud/iam.py — kept main's new Route53DnsPreflight statement (fix(cloud): detect Q VPC DNS shadowing of kiro-cli download #7888) and this branch's rewritten request-tag-gate comment.
  • CHANGELOG.md — main released [0.6.0], so the Spot entry moved verbatim into a new [Unreleased] section.
  • docs/system-specs/modules/cloud.md — main's reworded aws.py row plus this branch's ec2.py row.
  • RemoteCrewPanel.tsx / RemoteCrewPanel.test.tsx — kept main's askAgent error notice and its comments, re-added the X lucide import the merge dropped (4 tests failed without it), merged the import lists.
  • black formatted 4 lines in test_cloud_cli.py / test_cloud_iam.py (main's new formatting gate does not baseline them).

Gates run locally: black/isort/flake8 on changed files, pytest on the 6 touched test files (386 passed), tsc --noEmit, and vitest on the 2 touched panel test files (93 passed). No behaviour changed.

A maintainer push makes the maintainer the last pusher, so a second approver is needed under the repo's last-push rule. Reply if anything looks wrong.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) needs-author-decision PR blocked on author input readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cloud): support EC2 Spot Instances in kirocrew cloud launch for cost savings

3 participants