Skip to content

Recreate a local builder whose Docker context was removed - #1999

Open
viktorianer wants to merge 2 commits into
basecamp:mainfrom
viktorianer:recreate-builder-on-removed-context
Open

viktorianer wants to merge 2 commits into
basecamp:mainfrom
viktorianer:recreate-builder-on-removed-context

Conversation

@viktorianer

@viktorianer viktorianer commented Nov 15, 2026 •

Copy link
Copy Markdown

A local builder whose Docker context was removed is never recreated. docker buildx inspect still exits 0 for it, reporting the broken node only in its output, so Kamal keeps the builder and every build fails:

ERROR: failed to build: no valid drivers found: unable to parse docker host `gone`

This makes inspect_builder fail when the builder's node reports unable to parse docker host, with a message the existing rescue in Kamal::Cli::Build#push already treats as a missing builder. Every other inspect failure keeps its own error, since the message is printed only after that match. Kamal removes the builder and creates it again. docker buildx rm exits 1 for such a builder even though it removes it, and the rescue already tolerates that for context not found.

The check matches that one error, not any node error. A stopped engine also reports a node error (failed to connect to the docker API), and recreating that builder would throw away its cache while the build fails anyway.

Reproduction

host=$(docker context inspect --format '{{.Endpoints.docker.Host}}')
docker context create gone --docker "host=$host"
docker buildx create --name kamal-local-docker-container --driver docker-container gone
docker context rm gone
docker buildx inspect kamal-local-docker-container; echo "exit=$?"
Nodes:
Name:     kamal-local-docker-container0
Endpoint: gone
Error:    unable to parse docker host `gone`
exit=0

Verification

The new inspect_builder runs docker buildx inspect once under sh -c, keeps its output and exits with its status when it fails. Executed through SSHKit's local backend against Docker 29.5.2:

Builder state Exit Output
Node on a removed context 1 builder context not found
No such builder 1 ERROR: no builder "…" found, as before
Unreadable builder state 1 buildx's own error, as before, so the builder is not recreated
Healthy 0
Engine not running 0 node reports failed to connect to the docker API, so the builder is kept

bin/kamal build push --output docker from this branch, against a project whose builder sits on a removed context, warns Missing compatible builder, so creating a new one first, recreates the builder and builds. On main the same run fails with no valid drivers found. A second push keeps the recreated builder.

The new tests run the generated command through SSHKit's local backend against a fake docker, covering the same states and one inspect per check.

Remote and hybrid builders keep their own inspect_builder. A remote builder whose context was removed already fails its context check. The local node of a hybrid builder can go stale the same way and is not covered here.

🤖 Generated with Claude Code

docker buildx inspect exits 0 for a builder whose node points at a
removed Docker context, so the push kept the builder and every build
failed with "no valid drivers found: unable to parse docker host".
inspect_builder now fails when the node reports that error, with a
message the existing rescue treats as a missing builder, so the builder
is removed and created again.

Only that error counts: a stopped engine also reports a node error, and
recreating its builder would drop the build cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@viktorianer
viktorianer marked this pull request as ready for review November 15, 2026 21:11
Copilot AI lite review requested due to automatic review settings November 15, 2026 21:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Executable coverage is still missing for stopped-engine and unrelated inspect-error paths, leaving the shell safety condition vulnerable to recreating builders and discarding caches.

2 open issues
What changed in this PR

This pull request updates local Buildx builder inspection to detect builders backed by removed Docker contexts and recreate them before builds fail.

Changes:

  • Detects unable to parse docker host during local builder inspection.
  • Preserves existing handling for other inspection failures.
  • Updates command and CLI tests for the new inspection and recreation flow.
File Description
test/​commands/​builder_test.rb Verifies the generated local builder inspection command.
test/​cli/​cli_test_case.rb Updates shared SSHKit stubs for the revised builder inspection command.
test/​cli/​build_test.rb Covers builder inspection expectations and recreation after a Docker context is removed.
lib/​kamal/​commands/​builder/​base.rb Adds detection for local builders whose Docker context no longer exists.

Review effort: Lite

Comment thread lib/kamal/commands/builder/base.rb Resolved
Comment thread test/commands/builder_test.rb Resolved
Comment thread lib/kamal/commands/builder/base.rb Resolved
Comment thread test/commands/builder_test.rb Resolved
The check ran docker buildx inspect twice, so a context on another host
paid for two round trips before every build. It now keeps the output of
one inspect, exits with its status when it fails, and looks for the
removed-context error in that output.

The new tests run the generated command through SSHKit's local backend
against a fake docker, so they cover the shell control flow and SSHKit's
command mapping, not only the command string: a removed context fails
with the marker, a healthy builder and a stopped engine pass, and a
missing builder or an unrelated failure keeps its own error. Each
inspects once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants