Skip to content

fix(simulate): give each run its own diff servers and reclaim dead runs #500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jcogilvie merged 1 commit into from
Nov 5, 2026
Merged

fix(simulate): give each run its own diff servers and reclaim dead runs #500

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cmd/crossplane/render/op/help/render.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,10 @@ reference.
## Cleaning up after a crashed run

If a run is killed before it cleans up (for example by `kill -9`), the next
`composition render` or `operation render` run cleans up what it left behind,
following each container's `runtime-docker-cleanup` policy. This doesn't apply
to runs started with `--session-id`, to runs on Windows, or to a different
`composition render` or `operation render` run, or `resource simulate` run
that starts a diff server container, cleans up what it left behind, following
each container's `runtime-docker-cleanup` policy. This doesn't apply to runs
started with `--session-id`, to runs on Windows, or to a different
`DOCKER_HOST`.

If a container named by `runtime-docker-name` already exists:
Expand Down
7 changes: 4 additions & 3 deletions cmd/crossplane/render/xr/help/render.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,10 @@ reference.
## Cleaning up after a crashed run

If a run is killed before it cleans up (for example by `kill -9`), the next
`composition render` or `operation render` run cleans up what it left behind,
following each container's `runtime-docker-cleanup` policy. This doesn't apply
to runs started with `--session-id`, to runs on Windows, or to a different
`composition render` or `operation render` run, or `resource simulate` run
that starts a diff server container, cleans up what it left behind, following
each container's `runtime-docker-cleanup` policy. This doesn't apply to runs
started with `--session-id`, to runs on Windows, or to a different
`DOCKER_HOST`.

If a container named by `runtime-docker-name` already exists:
Expand Down
25 changes: 20 additions & 5 deletions cmd/crossplane/simulate/help/simulate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,28 @@ Nothing simulate does writes to the cluster or the cloud: it reads the
cluster with your kubeconfig and RBAC, and the diff servers run locally with
no credentials and no cluster access.

Each run starts its own diff servers, named
`crossplane-diff-server-DIGEST-SUFFIX`, and removes them when it exits.

Diff server startup takes seconds per provider image. In an
edit-simulate-edit loop, annotate a provider manifest with
`simulate.crossplane.io/runtime-docker-cleanup: Orphan` (like render's
per-function cleanup annotation) to leave its container running so the next
run reuses it. The containers have names like `crossplane-diff-server-DIGEST`
and carry `simulate.crossplane.io/managed-by` and session labels; any later
run without the annotation cleans them up, or remove them with docker rm.
`simulate.crossplane.io/runtime-docker-cleanup: Orphan` to leave its
container, named `crossplane-diff-server-DIGEST`, running for every later run
with the annotation to share. No run removes it, including a later run without
the annotation, which starts its own server instead. Remove kept containers
with:

```shell
docker rm -f $(docker ps -aq --filter label=simulate.crossplane.io/cleanup=Orphan)
```

If a run is killed before it cleans up (for example by `kill -9`), the next
`crossplane resource simulate` run that starts a diff server container, or
`crossplane composition render` or `crossplane operation render` run, removes
what it left, except kept containers. A run whose providers are all
Development starts no containers, so it doesn't. This doesn't apply to runs
started with `--session-id`, to runs on Windows, or to a different
`DOCKER_HOST`.

The command exits with one of these statuses, so a CI pipeline can gate on
the result:
Expand Down
26 changes: 16 additions & 10 deletions cmd/crossplane/simulate/simulate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import (

"github.com/crossplane/crossplane/apis/v2/pkg"

"github.com/crossplane/cli/v2/internal/containerruntime"
"github.com/crossplane/cli/v2/internal/diffserver"
"github.com/crossplane/cli/v2/internal/docker"
"github.com/crossplane/cli/v2/internal/kube"
Expand All @@ -57,14 +58,14 @@ var helpDetail string
type Cmd struct {
Filename string `help:"YAML files or directory with managed resources, or '-' for stdin (comma-separated)." predictor:"yaml_file_or_directory" required:"" short:"f"`

Providers string `help:"YAML file or directory with Provider manifests to run as diff servers, overriding discovery from the cluster. Supports per-provider runtime annotations." name:"providers" optional:"" predictor:"yaml_file_or_directory" type:"path"`
ImageConfigs string `help:"YAML file with ImageConfig manifests that rewrite provider image paths for local pulls. Defaults to the project's imageConfigs when a project file is present." name:"image-configs" optional:"" predictor:"yaml_file" type:"existingfile"`
ProjectFile string `help:"Path to a project definition file to read imageConfigs from. Defaults to ${project_file} when present." name:"project-file" optional:""`
SessionID string `help:"Session ID to apply as the simulate.crossplane.io/session label on diff server containers. Defaults to a random per-process ID." placeholder:"ID"`
Context string `default:"" help:"Kubernetes context." name:"context" predictor:"context" short:"c"`
CacheDir string `default:"~/.crossplane/cache" help:"Cache directory for downloaded provider packages." name:"cache-dir" predictor:"directory"`
Timeout time.Duration `default:"2m" help:"Per-resource plan timeout."`
Output string `default:"plan" enum:"plan,json,yaml" help:"Output format. One of: plan, json, yaml." name:"output" short:"o"`
Providers string `help:"YAML file or directory with Provider manifests to run as diff servers, overriding discovery from the cluster. Supports per-provider runtime annotations." name:"providers" optional:"" predictor:"yaml_file_or_directory" type:"path"`
ImageConfigs string `help:"YAML file with ImageConfig manifests that rewrite provider image paths for local pulls. Defaults to the project's imageConfigs when a project file is present." name:"image-configs" optional:"" predictor:"yaml_file" type:"existingfile"`
ProjectFile string `help:"Path to a project definition file to read imageConfigs from. Defaults to ${project_file} when present." name:"project-file" optional:""`
SessionID string `help:"Session ID to apply as the simulate.crossplane.io/session label on diff server containers. Defaults to a random per-run ID. A supplied ID isn't leased, so its containers aren't reclaimed after a crash." placeholder:"ID"`
Context string `default:"" help:"Kubernetes context." name:"context" predictor:"context" short:"c"`
CacheDir string `default:"~/.crossplane/cache" help:"Cache directory for downloaded provider packages." name:"cache-dir" predictor:"directory"`
Timeout time.Duration `default:"2m" help:"Per-resource plan timeout."`
Output string `default:"plan" enum:"plan,json,yaml" help:"Output format. One of: plan, json, yaml." name:"output" short:"o"`

Impersonation kube.ImpersonationFlags `embed:""`
}
Expand Down Expand Up @@ -154,11 +155,16 @@ func (c *Cmd) Run(ctx context.Context, k *kong.Context, logger logging.Logger, s
return errors.Wrap(err, "cannot create package client")
}

mgr := diffserver.NewManager(pkgClient, logger, diffserver.WithSessionID(c.SessionID))
mgr := diffserver.NewManager(pkgClient, logger,
diffserver.WithSessionID(c.SessionID),
diffserver.WithLeaseDir(containerruntime.DefaultLeaseDir()),
)
// Stop before EnsureServers returns, so an interrupted startup still
// stops the servers it started. Stop is safe when none started.
defer isimulate.StopServers(ctx, k.Stderr, mgr)
if err := isimulate.EnsureServers(ctx, sp, k.Stderr, mgr, inputs); err != nil {
return err
}
defer isimulate.StopServers(ctx, k.Stderr, mgr)

plans := make([]*plan.ResourcePlan, 0, len(resources))
planner := isimulate.NewPlanner(kubeClient, rmapper, defaultNamespace, mgr, c.Timeout)
Expand Down
26 changes: 15 additions & 11 deletions design/one-pager-cli-simulate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,17 +354,21 @@ read.

Startup is the one cost that hurts across runs: a diff server takes a few
seconds to come up, and an edit-simulate-edit loop pays that on every run
if containers are torn down at exit. So containers get deterministic
names derived from the provider image digest, and a provider manifest
annotated `simulate.crossplane.io/runtime-docker-cleanup: Orphan`, the same
cleanup policy render's function containers take through their annotation,
leaves its container running when the command exits: the next run finds a
live server with a matching name and reuses it instead of starting a new
one. Because the name carries the digest, a changed image never matches a
stale server, and leftovers are easy to find and remove since they all
follow one naming scheme and carry render-style managed-by and session
labels. The default stays clean, start and tear down per run; Orphan is
for the loops where those seconds add up.
if containers are torn down at exit. So a provider manifest annotated
`simulate.crossplane.io/runtime-docker-cleanup: Orphan`, the same cleanup
policy render's function containers take through their annotation, leaves
its container running when the command exits, under a deterministic
`crossplane-diff-server-<digest12>` name derived from the provider image
digest: the next run finds a live server with a matching name and reuses it
instead of starting a new one. Because the name carries the digest, a
changed image never matches a stale server. Only Orphan servers are shared
this way; every other server gets a per-run suffix on that name, so
concurrent runs never stop each other's servers, and a run that dies before
cleaning up has its servers reclaimed by the next run through the lease
mechanism render uses. All of them carry render-style managed-by and session
labels, so leftovers are easy to find and remove. The default stays clean,
start and tear down per run; Orphan is for the loops where those seconds
add up.

### From Response to Printed Diff

Expand Down
Loading
Loading