Repository navigation
Sync upstream worker registration and snapshot updates - #95
Merged
Merged
Conversation
…navailable (agent-substrate#2744) When a micro-VM restore in `restoreMemoryFidelity` fails to resume the guest, reach `kata-agent`, or reseed the guest CRNG before serving any inbound traffic, the worker's deferred cleanup already stops cloud-hypervisor and virtiofsd and unhosts the actor. However, `restoreMemoryFidelity` returned a plain `fmt.Errorf` (which the worker reports as `codes.Internal`, or `DeadlineExceeded` when the cause was a timeout) and `Atelet.Restore` wrapped all worker `RestoreWorkload` errors with `fmt.Errorf` (`codes.Internal`). So `ateapi`'s `handleAteletError` treated transient post-resume guest failures as fatal and moved the actor to `ACTOR_STATE_CRASHED` instead of keeping `ACTOR_STATE_RESUMING` for `ActorResumer` to retry. This PR: - updates the `apierror.Unavailable` doc comment in `internal/apierror/apierror.go` to cover an attempt that was rolled back so retrying it is safe, - adds `restoreGuestReadyError` in `cmd/ateworker-microvm/restore.go`, so pre-serving guest readiness failures (`client.Resume`, `dialAgentRetry`, `reseedGuestCRNG`) report `apierror.Unavailable` when the caller's context is still active. When it wraps `ctx.Err()`, it formats the cause with `%v`, so a child `context.DeadlineExceeded` does not shadow a caller `context.Canceled`. "Pre-serving" means before inbound traffic (not even the wakeup probe has connected yet). After `Resume`, the guest and its workload do run until the agent is dialed and the CRNG reseeded: atunnel's egress proxy closes their outbound TCP connections until `tunnel.Activate`, and the retry re-extracts the rootfs upper and durable-dir volumes from the snapshot, whereas any background writes made to a CSI volume during that window stay while guest memory and its page cache resume from the snapshot, - moves the `restoreMemoryFidelity` failure cleanup `defer` right after `PrepareEgress`, so it runs last (after the untar drain and the cloud-hypervisor and virtiofsd kills). It now sweeps the sandbox state, removes the rootfs upper dir, and detaches the bundle rootfs overlays (`cleanupFailedRestoreRootfs`, mirroring `teardownActor`), so nothing stays mounted across a retry. Two side effects: - failures from `buildActorContainers` through `stageMergedRootfs`, which used to leave the bundle overlays mounted, now detach them too; - like `teardownActor`, the sweep removes `/run/vc/vm/<id>`, so a failed restore no longer leaves the guest's console, serial and virtiofsd logs on the node, - adds `workerRestoreError` in `cmd/atelet/main.go` (resolving the `TODO` in `Atelet.Restore`). It propagates a worker `codes.Unavailable`, which a transport error also arrives as, as `apierror.Unavailable`, so `ateapi` preserves `ACTOR_STATE_RESUMING` and `WorkerAssignment` for retry. Other worker errors still crash the actor, including a worker-side `Canceled` or `DeadlineExceeded`, which can come from any step, - bounds retries of a restore that keeps failing, so it does not keep the actor `RESUMING` forever: - a guest found stopped while dialing the agent (`errGuestStopped`, e.g. a guest kernel panic after resume or a corrupt memory image) is reported as an error rather than `Unavailable` (`coldBootActorRetrying` likewise caps retries of that case); - `atelet` reports an actor's third consecutive retryable worker failure to restore the same snapshot as `Internal`; - a successful restore, a restore error that crashes the actor, or a successful `Terminate` ends the streak, while an error that keeps the actor `RESUMING` (a context error, or another `Unavailable`) leaves it as is, and - adds unit tests in `cmd/ateworker-microvm/restore_test.go`, `cmd/ateworker-microvm/restore_linux_test.go` (the overlay case is root-gated), and `cmd/atelet/lifecycle_test.go`. #### How I tested it - `go test -race ./internal/apierror/... ./cmd/ateworker-microvm/... ./cmd/atelet/...` - `go build` and `go vet` at each commit, and `go vet ./...` on the head - The root-gated `TestCleanupFailedRestoreRootfs` overlay case, run in a user and mount namespace (`unshare -r -m`) - `./hack/verify/gofmt.sh` and `./hack/verify/boilerplate.sh` - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
## What this PR does and why we need it ? When detaching an actor (either via background timer detach or explicit `DetachActor` RPC), `atelet` previously only pruned entries inside `/var/lib/ate/snapshots/<uid>/local-checkpoint/`, leaving the actor's snapshot directory (`/var/lib/ate/snapshots/<uid>`) and actor directory (`/var/lib/ate/actors/<uid>`) on the node indefinitely. Because `sandbox-assets/` holds hard links into the sandbox asset cache (`st_nlink > 1`), leaving the actor directory also permanently pinned sandbox binaries in the cache after detach. ### Changes - Move `ResetDir`, `ResetActorDirs`, `ResetActorDirsKeepingVolumes`, `RemoveActorFilesExceptState`, and `RemoveActorDirs` into `cmd/atelet/internal/actordirs` so `Atelet` and `snapshottracker.Tracker` share directory cleanup logic (preserving populated external volume mount directories, removing `ateletpath.ActorSnapshotPath`, and handling read-only bundle upper directories via `imagecache.RemoveAllWritable`). - Update `Tracker.finishDetach` to call `actordirs.RemoveActorFilesExceptState` (removing `/var/lib/ate/snapshots/<uid>` and all entries under `/var/lib/ate/actors/<uid>` except `snapshot-state.json`) while `opDetach` is active, and mark a detach report due (`markReportDueLocked`) for both background and foreground `Detach` calls. - Keep `snapshot-state.json` (with `ReportDueSince`) on disk until the detach report finishes so a retried `DetachActor` RPC still sees `LatestDurable` and a pending detach report survives an `atelet` restart. - Remove `snapshot-state.json` and the empty `/var/lib/ate/actors/<uid>` directory (via `removeDetachedActorState` in `finishReportDueLocked`) once `!Started && LatestLocal == nil && Uploading == nil` and the control plane acknowledges the detach report or refuses it with `NotFound`. On give-up or any other refusal (`Unimplemented`, `FailedPrecondition`, `InvalidArgument`), `snapshot-state.json` stays on disk with `ReportDueSince` cleared so `DetachActor` keeps returning `LatestDurable`. - Require `--snapshot-report-workers >= 1` in `validateSnapshotWorkerFlags` so background detach reports are always sent. - In `cleanUpAfterCheckpoint`, call `actordirs.RemoveActorFilesExceptState` instead of `actordirs.ResetActorDirs` when no local snapshot is kept on the node, so empty subdirectories are not recreated. - Document in `atelet.proto` (`ActorSnapshotState.latest_durable` and `DetachActorResponse.state`) that `DetachActor` queues a background detach report and forgets the actor once acknowledged. > **Merge order**: Land after agent-substrate#2797 so `Terminate` tolerates a missing `sandbox-assets.json` after `cleanUpAfterCheckpoint` calls `RemoveActorFilesExceptState`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#2797) ## What is this PR and Why we need it ? A deleted worker pod is deregistered by `workersync`, which calls `DeleteWorker` to mark its assigned actors `CRASHED` and clear their `WorkerAssignment`. Those actors are never terminated on the node, so `/var/lib/ate/actors/<uid>` and `/var/lib/ate/snapshots/<uid>` stay on disk until the actor is later deleted or reverted, if it ever is. (Note: this covers worker-pod removal via `DeleteWorker`; tearing down a crashed sandbox on a still-live worker is tracked separately in agent-substrate#2355.) ### Changes - **`atelet` workerless / gone-worker `Terminate`**: - Skips `TerminateWorkload` when `worker_pod_uid` is empty, the worker socket is gone, `sandbox-assets.json` is absent (`Run` and `Restore` write it before `RunWorkload`/`RestoreWorkload`, and an earlier `Terminate` removes it with the actor directory), or `TerminateWorkload` returns `Unavailable`/`NotFound`. Any other error from a live worker fails `Terminate` before touching disk. - Volume unmount failures in `Terminate` return `Internal` without `%w`-wrapping downstream gRPC errors, so a transient CSI failure does not surface to `ateapi` as `Unavailable` (and reaches `DeleteActor`/`RevertActor` callers as `Internal` instead of `Unavailable`). - **`ateapi` (`DeleteWorker`)**: - `DeleteWorker` takes each actor's lease (`ABORTED` if held, requeued by `workersync`), skips actors that already reached `SUSPENDED`, and calls best-effort `atelet.Terminate` with a 30s timeout to reclaim the actor's node state and local snapshots. Once a call times out (`DeadlineExceeded`) or returns `Unavailable`, remaining actors on that worker skip the node and leave cleanup to revert/delete. --- ### Testing - **Unit & functional tests**: `go test ./cmd/atelet/... ./cmd/ateapi/internal/controlapi/...` - **Live-cluster verification** (3 × `c3-standard-4` `gvisor` nodes, 3 × `c3-highcpu-4` `microvm` nodes): | Suite | `gvisor` (3 × `c3-standard-4`) | `microvm` (3 × `c3-highcpu-4`) | |---|---|---| | **`durdir_data_baseline` 1m smoke** (8 MiB `glutton-durdir-data`) | `20` reqs, `0` fails (`0.00%`) | `15` reqs, `0` fails (`0.00%`) | | **`DeleteWorker` scale-down reclaim** (3 actors, 256 MiB `durable-dir`/actor) | `/var/lib/ate/actors`: `0.00 GB -> 1.41 GB (3 dirs) -> 0.00 GB (0 dirs)`; `-0.25 GB`/node reclaimed (`+0.00 GB` vs baseline) | `/var/lib/ate/actors`: `0.00 GB -> 1.64 GB (3 dirs) -> 0.00 GB (0 dirs)`; `-0.25 GB`/node reclaimed (`+0.00 GB` vs baseline) | | **`DeleteActor` on paused actor with no worker** (`worker_pod_uid=""`) | `local-checkpoint` pruned & `/var/lib/ate/actors/<uid>` removed (`1.75 ms`) | `local-checkpoint` pruned & `/var/lib/ate/actors/<uid>` removed (`9.45 ms`) | Fixes agent-substrate#1688 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…bstrate#3050) Not every file of a node-local snapshot is synced to disk, so a local copy written before an unclean node reboot may be left empty or truncated even when its manifest is intact. Stamp the node's boot ID into the manifest of each local snapshot copy (omitting it from uploaded manifests). At atelet startup, delete all pre-reboot local snapshots on disk, detaching and reporting any that are already durable. Restore and upload never read a local copy from an earlier boot, in case one is left behind. A local snapshot not yet uploaded is lost on reboot. An actor that needs durability across planned node maintenance should suspend with an immediate upload. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…ir tokens no longer run short near the end of each hour (agent-substrate#3093) Fixes agent-substrate#2722 The credential exchange downscoped ate-api-server's own token, so GCS audit logs named ate-api-server for every node's snapshot transfer, and the metadata server's cached token left downscoped tokens short for the last twenty minutes of each hour. The exchange now impersonates a dedicated Google service account, `ate-snapshot-transfer` by default, and fetches a new token once less than 40 minutes are left. setup-gcp creates the account and its grants, and ate-setup passes it to the exchange. #### Breaking change GKE installs must run `go run ./tools/setup-gcp create iam` and set `SNAPSHOT_TRANSFER_GSA_NAME` before deploying. ate-setup refuses to deploy ate-api-server on GKE without it. With Cloud SQL, also rerun `create cloudsql`, so its account may create the transfer account's tokens. Tested on a GKE cluster: the full e2e suite passed on gVisor workers, a micro-VM actor suspended and resumed, and GCS audit logs named the transfer account for every node snapshot transfer. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…rate#1954) (agent-substrate#3026) Fixes agent-substrate#1954 , depends on agent-substrate#3050 The `f.Sync()` in `tarutil` was a high cost of microVM suspend under load, and per the discussion on the issue local-only snapshots aren't guaranteed to survive a node restart, and local snapshots from an earlier boot aren't trusted (agent-substrate#3050 ). Since all actors share one filesystem, the fsync also waits behind other actors' writes in the ext4 journal. On a bare-metal microVM node (sweperf, one worker), the worker spent ~95% of its blocked time in this fsync (`jbd2_log_wait_commit`) at saturation. Removing it cut SuspendActor p50 from 10.6 s → 0.81 s at 40 users (full table in agent-substrate#1954). It doesn't change where the node saturates. - [ X] Tests pass - [ ] Docs updated
…e#3087) At 1k nodes and 200k actors, atenet-router starts failing ingress with `503 upstream connect error or disconnect/reset before headers. reset reason: remote connection failure`. The requests that fail are the ones that need a new tunnel. `actor_connect` refuses the tunnel once a circuit breaker set by `tunnelCircuitBreakerLimit` (65536) is full. - Split the limit into tunnelCircuitBreakerLimit and actorConnectCircuitBreakerLimit, so each cluster can be tuned on its own. - Set both to 1048576. For actor_connect that covers 11,000 workers on up to about 95 Envoy threads. Leaving the breakers unset would mean Envoy's default of 1024. - Tuning actor_connect's limits is tracked in agent-substrate#3100. - [ ] Tests pass - [ ] Docs updated
…it rotates (agent-substrate#3081) Part of agent-substrate#3032 atenet-egress checks actor certificates in Envoy and again in the router ext_proc sidecar. Both read `/run/actor-id-ca-certs/ca.crt` once at startup, so after a CA rotation they reject certificates from the new CA until the pod restarts. - Envoy loads the CA from a filesystem SDS resource with `watched_directory`, like the serving cert. - The router passes a `credbundle.PoolLoader` to `egress.New`, which checks the file on each CONNECT. AI use: used Claude Code to write the change and tests. I reviewed the diff. - [x] Tests pass - [ ] Docs updated
…bstrate#3049) Why is this needed? When ctx was done, schedule closed ordered and Read returned io.EOF, even with bytes of the object still unread. A plain zstd stream cut at a frame boundary then decoded as a clean, truncated success. schedule now records the context's cause when it stops early, and Read returns it instead of io.EOF. Fixes agent-substrate#3047 AI use: Claude assisted. - [ ] Tests pass - [ ] Docs updated
… in /hack/tools/ko (agent-substrate#2980) Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.6 to 1.18.7. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/klauspost/compress/releases">github.com/klauspost/compress's releases</a>.</em></p> <blockquote> <h2>v1.18.7</h2> <ul> <li>Fix s2.NewDict OOB read.</li> </ul> <p>Security release without other changes.</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/klauspost/compress/compare/v1.18.6...v1.18.7">https://github.com/klauspost/compress/compare/v1.18.6...v1.18.7</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/klauspost/compress/commit/8668e357e776d5152ed62f33c17f21b8690664fa"><code>8668e35</code></a> Merge commit from fork</li> <li>See full diff in <a href="https://github.com/klauspost/compress/compare/v1.18.6...v1.18.7">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/agent-substrate/substrate/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…t-substrate#2988) Restart detection relied on the worker container's restartCount, which can restart from zero or stay the same. It was also recorded at bind time, so a resume whose Restore reached the restarted process was later crashed as if its sandbox had been lost. Each worker process now acquires an epoch from ateapi and activates it with RegisterWorker, once, along with its pod IPs and capacity. Sandbox runtimes move to SetWorkerSandboxRuntimes. Run and Restore return the worker's epoch, and finalizeRunning records it, or returns Unavailable. A worker refuses workloads with Unavailable until it registers, and atelet passes a worker's Unavailable on, so the resume is retried rather than the actor crashed. The reconciler crashes actors recorded below the active epoch, and leaves RESUMING actors and claims never committed to an actor alone. A worker reports ready only once registered. Worker.ips moves to status.ips and follows a recreated pod sandbox. Not handled yet: a worker that crashes before registering leaves its lost actors RUNNING. Fixes agent-substrate#2692 Fixes agent-substrate#1837 AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…t to async (agent-substrate#3097) Rename the `--lifecycle-mode` choices from `suspend` and `pause` to `sync` and `async`, and switch the default in `lifecycle_mode.py` and the boomer `agentsession`, `ateenv`, `glutton`, `durdir`, and `sweperf` user classes to `async` (`DURABILITY_MODE_ASYNCHRONOUS`) to match `Control.SuspendActor`'s default (agent-substrate#2813). - [x] Tests pass - [x] Docs updated
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
marked this pull request as ready for review
October 22, 2026 17:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork up to upstream
e88ffcdb5(12 commits), preserving both histories and the fork's agentgateway configuration.Includes worker restart registration, actor cleanup after worker deletion, snapshot invalidation after node reboots, GCS transfer credentials, and microVM restore error handling. Resolve the workflow conflicts while retaining the fork's deadlines, and wire
objectStorePlugin.transferServiceAccountthrough Helm.Upgrade API clients, control-plane components, and workers together: worker registration RPCs and protobuf fields changed. GCS Helm installs must set
objectStorePlugin.transferServiceAccountand provision the transfer account IAM grants. S3/RustFS installs do not need this setting.Validation: all 36 Helm tests, chart linting, manifest rendering, and CRD parity passed. The full Go race suite and repository verification hit the local 20 GB temporary filesystem limit (
no space left on device). Local E2E was not run; CI validation is pending. Keeping this draft until validation is complete. Independent agent review was attempted, but each session failed with a service disconnect.Use a merge commit to retain upstream ancestry.
AI use: Codex prepared the merge, resolved conflicts, updated Helm and docs, and reviewed and tested the integration.