Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/guides/cloud-instance-ssm-vs-ssh.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ older SSH-over-`ProxyCommand` workaround.
> inbound port, and no `~/.ssh/config` edits**. Reachability is an IAM decision
> (grant/revoke `ssm:StartSession`, CloudTrail-audited), not a networking decision
> gated by a key someone holds.
>
> New here? [remote-crew-on-ec2.md](remote-crew-on-ec2.md#deploy-your-first-kiro-crew-on-ec2-step-by-step)
> is the step-by-step first-launch walkthrough; this page is the transport deep-dive.

## The two transports

Expand Down
95 changes: 95 additions & 0 deletions docs/guides/remote-crew-on-ec2.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,63 @@ plus the EC2-specific gotchas people actually hit (from `kirocrew doctor`).
Both end at the same loopback gateway and both are managed identically once
registered in **Settings → Remote Instances**.

## Deploy your first Kiro Crew on EC2 (step by step)

This is the shortest path from a fresh laptop to a working cloud Kiro Crew, using
the one-command launcher (`kirocrew cloud`). Plan on ~10 minutes plus one browser
sign-in.

**Before you start (laptop prerequisites):**

1. **AWS CLI installed and configured.** `aws sts get-caller-identity` must print
your account — if it errors, run `aws configure` (or `aws sso login`) first.
2. **Session Manager plugin** — `kirocrew cloud doctor` offers to install it, or
see [cloud-instance-ssm-vs-ssh.md](cloud-instance-ssm-vs-ssh.md).
3. **npm on your laptop (recommended)** — with `npm` installed (Node.js 18+),
the launcher builds the dashboard frontend automatically from the packaged
source (`npm ci && npm run build` in an isolated temporary tree — your
checkout's own `website/dist` and `static/dist` are never read or touched)
and ships the result, so the box skips the slow, failure-prone on-box npm
build. `kirocrew cloud doctor` reports whether the frontend will ship.
Without npm — or if the local build fails — the box builds the frontend
itself.
4. **A git checkout of this repo** with Kiro Crew installed from it — the launcher
packages your local source and uploads it to S3 (private-repo safe), it does
not clone GitHub on the box.

Then run the checks and launch:

```bash
kirocrew cloud doctor # verifies AWS CLI, credentials, plugin, frontend build
kirocrew cloud launch # interactive: picks the recommended tier, region, etc.
```

- Accept the recommended **balanced** tier. The **light** tier is sized under
Kiro Crew's working set — fine for a quick look, but expect pressure under
real workloads. (`kirocrew cloud launch` prints each tier's current instance
type and memory; the source of truth is `src/kiro_crew/cloud/sizes.py`.)
- Launch blocks until the box is **actually serving the dashboard** (the stack's
health check verifies the SPA loads, not just that something answers on the
port). Typical wait: 10–20 minutes on a cold box.
- When it finishes it prints a dashboard URL and opens it. **Sign in to Kiro in
the browser window** that appears (device-code flow) — chats don't work until
this is done. If you launched with `--yes` (non-interactive), run
`kirocrew cloud login` afterwards to do this step.

**Coming back later:**

```bash
kirocrew cloud connect # reopens the dashboard tunnel (alias: tunnel)
kirocrew cloud status # instance state
kirocrew cloud stop # stop the box (EBS kept; cheap)
kirocrew cloud start # start it again
kirocrew cloud destroy # tear everything down (stack + uploaded source)
```

If `connect` ever prints a "stale dashboard" warning, the box is missing its
frontend assets — see
[Dashboard HTML not found](#dashboard-html-not-found-after-launch) below.

## Way 1 — SSH tunnel

1. On your laptop, forward the gateway's port over SSH (use the **real** gateway
Expand Down Expand Up @@ -59,6 +116,44 @@ port, no SSH key.

These map to warnings in `kirocrew doctor`.

### "Dashboard HTML not found" after launch

The gateway is up but has no frontend assets (`src/kiro_crew/static/dist` is
missing) — `kirocrew cloud connect` also prints a "stale dashboard" warning in
this state. Fixes, cheapest first:

1. On the box: `sudo systemctl restart kirocrew` — picks up assets staged after
the gateway started.
2. Still missing? The on-box npm build failed at install time. On a cloud box
the install is fail-closed (`KIROCREW_REQUIRE_FRONTEND=1` plus the stack's
own dist check), so re-running `kirocrew cloud launch` (with Node/npm on
your laptop, so the frontend ships pre-built) is the reliable fix.
3. To inspect by hand: `kirocrew cloud connect` keeps working for SSH/SSM access
(`aws ssm start-session --target i-…`); the install log is
`/var/log/kirocrew-install.log` on the box.

### Launch fails during package install / pip output scrolls by

What looks like "python testing" is pip's dependency resolver on a cold box. The
template already retries the install once for transient mirror flakes; if the
stack still rolls back, just re-run `kirocrew cloud launch` — a second run reuses
the warm dnf/pip caches and almost always gets through. Persistent failures point
at the instance tier: use **balanced**, not light.

### Stack rolls back with a WaitCondition timeout

The box didn't report healthy within 25 minutes. Read the actual failure reason:

```bash
aws cloudformation describe-stack-events \
--stack-name kirocrew-<tag> \
--query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].ResourceStatusReason'
```

The reason carries the last lines of the on-box install log, which names the
failing step (package install, npm build, gateway start). Fix that step, then
re-run `kirocrew cloud launch`.

### MCP tools all fail: "Sandbox backend unavailable … `allow_unsandboxed_exec` is not set"

On Linux, agent subprocesses run inside a **user-namespace sandbox**. Many hardened
Expand Down
22 changes: 18 additions & 4 deletions docs/system-specs/modules/cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ claim that a hostile in-process agent is fully contained.
| `ssm.py` | SSM `send-command` run-and-poll (base64-wrapped remote scripts) + `start-session` port-forward; `open_port_forward()` directly spawns the streaming `aws ssm start-session` child because `run_aws` captures output, and calls `aws.assert_human_action()` before doing so; `port_is_free` / `wait_for_local_port`. |
| `login.py` | `kiro-cli` device-code / social sign-in on the box over SSM, plus `logout` — the account switch. `login` short-circuits on an existing session, so `logout` is what makes a different Kiro account reachable without a hand-run SSM command. It kills any still-polling background `kiro-cli login` **and** any live `kiro-cli acp` runtime **before** signing out (otherwise the login re-authenticates the old account, and an ACP runtime keeps serving the old account's in-memory credential until its next 401), removes the login log/PID/FIFO (they hold the previous device-code URL + code, which must never be re-shown as a fresh prompt), and confirms the result with `is_logged_in` rather than the exit code — `kiro-cli logout` exits non-zero when there was no session to drop, which is still the requested state. That confirmation fails CLOSED: it requires a positive signed-out sentinel (`__NOAUTH__`), so an SSM timeout or transport error — where the session may still be active — reports failure rather than a false "signed out". The same fail-closed applies to the cleanup command itself: if that SSM invocation doesn't return `Success`, the kills it was meant to do can't be trusted and logout reports failure without probing. The CLI warns the operator that in-flight chats/cron sessions are stopped (their runtimes are killed). |
| `connect.py` | SSM port-forward + token mint + open browser; Instances-registry integration; `redact_token`. `is_launched_instance()` prevents the generic instance PATCH endpoint from rewriting a correlated launch’s connection method, SSM target, AWS profile, or region, so Stop/Start/Delete retain the stack address and a running billable instance is not stranded. |
| `source.py` | Detect and package an editable local checkout (`git archive`, tarfile fallback) and upload it to a per-account S3 bucket; packaged installs instead use the template's public-repo clone fallback. The secret-excluding filter is shared by both packaging paths. Also **`ensure_instance_boundary`** — creates the shared, immutable `kirocrew-ec2-boundary` managed policy once (create-if-not-exists, never re-versioned) and returns its ARN; `delete_instance_boundary` for admin cleanup. |
| `source.py` | Detect and package an editable local checkout (`git archive`, tarfile fallback) and upload it to a per-account S3 bucket; packaged installs instead use the template's public-repo clone fallback. The secret-excluding filter is shared by both packaging paths. `_inject_dist` extracts only `website/` from that already-filtered archive into an isolated temporary root, runs lockfile-exact `npm ci --ignore-scripts` plus the stock build there, and appends the resulting frontend. It never reads the gitignored checkout `static/dist`, so a bundle left by another branch cannot override the source being shipped; edition composition variables are removed so an external composition root cannot enter the build. The built dist must be a real tree contained by the isolated root. Members are then admitted through four gates: the secret filter (`allow_dist_under` exempts only the literal destination `dist` component for that exact prefix — every other excluded dir still refuses the member; `.env`/`.pem`/credential-name checks still apply), a **type allowlist** (`_DIST_ALLOWED_SUFFIXES` — only build-artifact extensions ship), the hardened nolink read gate (`hooks.safe_read_file_bytes_nolink`: O_NOFOLLOW + same-fd fstat rejects symlinks/hardlinks/non-regular files race-free, fd-real-path containment), and a bounded text-content scan using the shared credential redactor as a detector. The content scan covers UTF-8 HTML, JavaScript, CSS, source maps, JSON/manifests, and SVG; it blocks distinctive/plaintext and encoded credentials plus exact 40-character bare AWS secret keys, while ignoring the redactor's bare-secret warning for longer base64-shaped runs because real minified bundles and data URIs trigger that heuristic. Shipping is **atomic over one hardened-read byte snapshot**: the scanner and index-reference check inspect the same bytes written to the archive, and any build, containment, read/content, or referenced-chunk rejection aborts injection entirely — the original archive ships unchanged and the box uses its required npm-build fallback, so a partial dist never rides. Diagnostics repr-escape untrusted filenames and index references before logging or terminal display. Also **`ensure_instance_boundary`** — creates the shared, immutable `kirocrew-ec2-boundary` managed policy once (create-if-not-exists, never re-versioned) and returns its ARN; `delete_instance_boundary` for admin cleanup. |
| `config.py` | Persisted profile / region / tag (**never credentials**); `load()` tolerates a hand-edited/corrupt `cloud.json` — bad JSON *or* a non-object shape falls back to defaults rather than crashing every cloud command. |
| `sizes.py` | arm64/Graviton size tiers (16 GB default `t4g.xlarge`). |
| `ui.py` / `wizard.py` | Terminal UI + the interactive launch flow. `_deploy_with_progress` runs the blocking deploy on a daemon thread and captures the `aws cloudformation deploy` child via a `proc_sink`, so a Ctrl+C on the main (poll) thread terminates it instead of orphaning it (~1800s). An unknown `--size`/`size_key` on the public `launch()` entrypoint yields a clean rc=1 + message, not an uncaught `KeyError`. Resuming a saved stack (`launch` after `stop`) first calls `_ensure_running_and_ssm_ready` — starts a `stopped` instance and waits for SSM `Online` before sign-in/tunnel (which are SSM-only and would otherwise fail); a `terminated` instance fails clean pointing at `--new`. `last_tag` is persisted (`cfg.save()`) **only after** a deploy confirms healthy — a failed first launch leaves no saved pointer, so the next `launch` retries clean instead of resuming a rolled-back/instance-less stack; `_saved_launch_is_usable` additionally ignores a stale saved tag (from an older build) whose stack is in a `_FAILED_STATES` status or has no instance. |
Expand All @@ -84,8 +84,12 @@ claim that a hostile in-process agent is fully contained.
CloudFormation stack, one `aws cloudformation deploy` (change-set based), atomic
rollback, one-command `delete-stack` teardown. AMI resolves from the public
`resolve:ssm` Amazon-Linux-2023 alias per arch (no hardcoded AMI ids). A
`WaitCondition` + `cfn-signal` blocks the deploy until the gateway is healthy; a
failed bootstrap folds the on-box setup-log tail into the signal reason so the cause survives the rollback. Bootstrap failure reasons are normalized to printable ASCII before CloudFormation receives them; otherwise CloudFormation replaces the setup error with a charset error and masks it during rollback (`test_cloud_ec2.py::test_failure_reason_is_filtered_to_printable_ascii`).
`WaitCondition` + `cfn-signal` blocks the deploy until the gateway is healthy —
where healthy means the root URL answers **and** its body lacks the gateway's
"Dashboard HTML not found" marker (the dashboard-less fallback page is still
HTTP 200, so a bare status check would bless a frontend-less box). A failed
bootstrap folds the on-box setup-log tail into the signal reason so the cause
survives the rollback. Bootstrap failure reasons are normalized to printable ASCII before CloudFormation receives them; otherwise CloudFormation replaces the setup error with a charset error and masks it during rollback (`test_cloud_ec2.py::test_failure_reason_is_filtered_to_printable_ascii`).

Bootstrap is reboot-resilient. UserData performs no package or build work directly:
it first copies its already-rendered script to
Expand Down Expand Up @@ -113,7 +117,17 @@ role (`s3:GetObject` scoped to the single object). Wheel and desktop installs
have no checkout to package, so `ec2.deploy` omits `SourceBucket` by default and
the template clones the public repository/ref instead. An explicit
`ship_source=True` remains fail-closed rather than packaging an unrelated
`site-packages` ancestor.
`site-packages` ancestor. Packaging builds the stock frontend from the exact
filtered source archive in a temporary root, with dependency lifecycle scripts
disabled, and injects only the admitted result. It never reads or mutates the
checkout's live `website/dist` or gitignored `static/dist`; this binds shipped
same-origin JavaScript to the source tarball even after a branch switch. If that
isolated build fails, the original archive ships unchanged and the box builds
with npm. `install.sh` skips the on-box npm build only when the source-bound
bundle is present. A cloud box that still ends up without a dist is failed by
the template's existing gates (`KIROCREW_REQUIRE_FRONTEND=1` makes an on-box
build failure fatal, and the bootstrap's dist check fails the stack before the
gateway starts); local installs stay non-fatal.

`discover_network` is **egress-kind-aware**, not just "has a default route":
`_subnet_egress_kinds` classifies each subnet's effective route table (explicit
Expand Down
35 changes: 34 additions & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,40 @@ cd "$KIROCREW_APP_DIR"
# ── Frontend (npm + vite) ──
# Vite emits to website/dist; we stage it into src/kiro_crew/static/dist
# where setup.py copies it into the package at install time.
if has node && [ -d "$KIROCREW_APP_DIR/website" ]; then
# A launch that ships the prebuilt bundle (kirocrew cloud launch injects it
# into the source tarball) skips this step entirely — the on-box npm build is
# the fallback path. The skip is gated on KIROCREW_REQUIRE_FRONTEND=1 (exported
# by the CFn template) so it only fires on cloud boxes: a LOCAL checkout also
# has a staged static/dist (a symlink on source installs), and skipping there
# would freeze the dashboard on whatever was built last. index.html alone does
# not prove a usable bundle: a torn staging or interrupted earlier build can
# leave an index whose hashed chunks are missing, and an install RETRY that
# trusted it would skip the rebuild and serve a shell whose every chunk 404s.
# So the skip additionally requires every /assets/ chunk the index references
# to exist (the same completeness signal frontend._incomplete_bundle_reason
# uses); an incomplete tree falls through to the npm rebuild below. A cloud box
# that ends up with no dist is failed by the existing gates:
# KIROCREW_REQUIRE_FRONTEND=1 makes a build failure fatal below, and the
# template's own DIST_INDEX check fails the stack before the gateway starts.
# Local installs stay non-fatal.
_dist_bundle_complete() {
# Complete = index.html exists and every /assets/*.js|.css it references
# is present on disk. Gateway-served routes (/manifest.js) are not under
# /assets/ and are deliberately not matched.
local _dist_dir _refs _ref
_dist_dir="$1"
[ -f "$_dist_dir/index.html" ] || return 1
_refs="$(grep -oE '(src|href)="/assets/[^"?#]+\.(js|css)"' \
"$_dist_dir/index.html" 2>/dev/null | sed -E 's/^[^"]*"//; s/"$//')"
for _ref in $_refs; do
[ -f "$_dist_dir$_ref" ] || return 1
done
return 0
}
if [ "${KIROCREW_REQUIRE_FRONTEND:-0}" = "1" ] \
&& _dist_bundle_complete "$KIROCREW_APP_DIR/src/kiro_crew/static/dist"; then
ok "Frontend shipped pre-built — skipping npm build"
elif has node && [ -d "$KIROCREW_APP_DIR/website" ]; then
info "Building frontend (website/)…"
_fe_log="$(mktemp)"
(
Expand Down
23 changes: 22 additions & 1 deletion src/kiro_crew/cli_cloud.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""CLI ``kirocrew cloud`` command group — thin dispatchers into :mod:`cloud`.

Every verb here is a small wrapper that calls into the testable ``cloud/``
Expand All @@ -18,7 +18,7 @@
from kiro_crew.cloud import connect as connect_mod
from kiro_crew.cloud import ec2, iam
from kiro_crew.cloud import login as login_mod
from kiro_crew.cloud import sizes, ssm, ui, wizard
from kiro_crew.cloud import sizes, source, ssm, ui, wizard
from kiro_crew.cloud.aws import AWSError, CloudActionDenied
from kiro_crew.cloud.config import DEFAULT_REGION, CloudConfig
from kiro_crew.deploy.engine import resolve_aws_bin
Expand Down Expand Up @@ -118,6 +118,14 @@
ui.fail(str(exc))
return 1
if conn.ready and conn.url:
# Warn early if the box serves the "Dashboard HTML not found" page (no
# static/dist) instead of letting the user open a broken dashboard.
# Deferred import: cli_server pulls the gateway's heavy module chain
# (vector_memory → numpy), which must not load at CLI import time
# (test_cli_lazy_imports, issue #3504).
from kiro_crew.cli_server import _probe_dashboard_health

_probe_dashboard_health(conn.local_port)
if not conn.token:
# Tunnel is up but the token mint failed — the URL will hit the
# dashboard's login wall. Say so instead of implying it's ready.
Expand Down Expand Up @@ -402,6 +410,19 @@
else:
ui.fail("AWS not reachable")
ui.detail(reach.get("note", ""))
# A launch builds the stock frontend from the exact source archive in an
# isolated temporary tree, so a residual gitignored static/dist can never
# influence what ships. Doctor checks only the local prerequisites; the
# packaging path remains fail-closed and falls back to the box build.
try:
reason = source.dist_ineligible_reason(source.repo_root())
except Exception as exc:
reason = f"could not inspect the checkout ({exc})"
if not reason:
ui.ok("frontend source ready (launch builds an isolated prebuilt bundle)")
else:
ui.warn(f"frontend cannot be prebuilt for launch: {reason}")
ui.detail("The box will use its required npm-build fallback.")
return 0


Expand Down
3 changes: 2 additions & 1 deletion src/kiro_crew/cli_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,8 @@ def _probe_dashboard_health(port: int) -> None:
print(
"⚠️ Warning: gateway is serving a stale dashboard "
"(assets missing — likely an update pruned the "
"running install). Restart the gateway to fix.",
"running install). Restart the gateway to fix "
"(on a cloud box: sudo systemctl restart kirocrew).",
file=sys.stderr,
)
except Exception:
Expand Down
Loading
Loading