Kubernetes Dockerless build drops .devsy-internal during StreamMount and fails before build starts #1108

Activity

1. coderabbitai commented on Aug 19, 2026

   

   coderabbitaibot

   on Aug 19, 2026 – with coderabbitai

   Last edited by coderabbitai

   More actions

   🔗 Related PRs

   #398 - fix: use actual container mount path after workspace rename [merged]
   #486 - fix(workspace): delete container on rename to fix stale bind mounts [merged]
   #519 - fix(docker): stale bind-mount inode to Docker Desktop on up/delete/up [merged]
   #603 - feat(ci): add devsy ci command for devcontainer CI [merged]
   #615 - fix(delivery): cleanup build files for mounted volume [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   ✅ Successfully generated implementation plan.

   💡 Use @coderabbitai <feedback> to replan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
2. joshyorko commented on Aug 19, 2026

   

   joshyorko

   on Aug 19, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Additional independent reproducer: joshyorko/room-of-requirement.

   This is a useful control because the same repo is known to build successfully on Kubernetes with Skevetter's DevPod provider path, while Devsy 1.15.0 fails at the same .devsy-internal handoff before Dockerless starts.

   Devsy run details:

   • workspace: https://github.com/joshyorko/room-of-requirement
   • provider: Kubernetes
   • Kubernetes context: ror
   • namespace: devsy
   • Dockerless image: ghcr.io/devsy-org/dockerless:0.2.0
   • architecture: amd64

   Devsy successfully resolves the Dockerfile and the docker-in-docker devcontainer feature, calculates the prebuild hash, creates the PVC/pod, injects the agent, and begins streaming the workspace.

   The pod is configured with:

   DOCKERLESS_CONTEXT=/workspaces/room-of-requirement
   DOCKERLESS_DOCKERFILE=/workspaces/room-of-requirement/.devsy-internal/Dockerfile-with-features
   DOCKERLESS_TARGET=dev_containers_target_stage
   

   Then immediately after the workspace copy begins, setup fails with:

   Copy .../room-of-requirement into DevContainer /workspaces/room-of-requirement
   dockerless build: rename dir: open /workspaces/.dockerless/.devsy-internal: no such file or directory
   

   This reproduces the same failure with a second Dockerfile-backed devcontainer, including a generated Dockerfile-with-features path rather than the Dockerfile-without-features path in the original actions reproducer.

   That makes the failure appear independent of the specific Dockerfile contents and independent of whether Devcontainer Features are involved; the common failure point is the generated .devsy-internal directory not surviving the streamed Kubernetes workspace handoff.
3. coderabbitai commented on Aug 19, 2026

   

   coderabbitaibot

   on Aug 19, 2026 – with coderabbitai

   More actions

   Coding Plan

   Summary

   • Remove the config.BuildArtifactExcludes() exclusion from StreamMount so the streamed workspace mount carries the generated .devsy-internal Dockerless build directory into the Kubernetes pod.
   • Keep .devsyignore handling and all other BuildArtifactExcludes() callers unchanged, because .devsy-internal is generated only in the Dockerless flow that needs it.
   • Leave host-side and pod-side cleanup unchanged, so generated artifacts are still removed after setup.
   • Add a StreamMount regression test that asserts the .devsy-internal build directory is present in the streamed tar.

   Design Choices

   Design Choice 1: How should the generated `.devsy-internal` build directory reach the pod?

   Options Considered:

   1. Stop applying config.BuildArtifactExcludes() in StreamMount, so the streamed mount carries .devsy-internal into the pod. This matches the working DevPod control behavior, where the mount stream does not exclude the equivalent internal folder.
   2. Keep the exclude in StreamMount, but add a separate step that seeds/persists .devsy-internal into the /workspaces/.dockerless volume before excluding it from the normal payload. This requires a new transport path, because /workspaces/.dockerless is a Type: "volume" mount that is never part of config.GetMounts() and is never streamed.

   Chosen Option: 1

   Rationale: This option matches the proven DevPod control flow. This option is the minimal change. .devsy-internal is generated only for the Dockerless (Kubernetes/non-Docker) flow, which is exactly the flow that needs the folder in the pod.

   💡 User Tips

   Regenerate the plan with different choices with @coderabbitai <feedback>.

   Implementation Steps

   Phase 1: Deliver the Dockerless build directory to the pod through the mount stream

   This phase makes the streamed workspace mount carry the generated .devsy-internal build directory to the Kubernetes pod. The goal is that prepareBuildDirectory finds the build directory at the workspace context path before DockerlessBuild runs.

   Task 1: Stop excluding build artifacts in StreamMount

   Remove the build-artifact exclusion from the mount stream so the generated Dockerless build directory reaches the pod.

   • Update StreamMount in pkg/agent/tunnelserver/tunnelserver.go. Change its exclude list so it no longer appends config.BuildArtifactExcludes().
   • Keep workspaceIgnoreExcludes() in place. The .devsyignore behavior must not change.
   • Rationale: .devsy-internal is generated only in the Dockerless (non-Docker driver) build path. The Docker driver never creates this folder. So removing the exclude affects only the Dockerless flow, where the pod requires the folder.
   • Add a short code comment that explains why StreamMount must not exclude .devsy-internal. State that the Dockerless build directory travels to the pod through this mount stream.
   • Scope note: Do not change StreamWorkspace or StreamSnapshotVolumes in this ticket. The reported failure uses the StreamMount path for the primary workspace mount. Leave the other RPCs and pkg/agent/delivery/workspace_seed.go and pkg/devcontainer/config/prebuild.go unchanged.

   Task 2: Confirm host cleanup and pod fallback remain intact

   Verify that existing cleanup behavior still runs and that no additional change is required for the fallback path.

   • Confirm cleanupBuildInformation (deferred in pkg/devcontainer/run.go Up()) still removes the host .devsy-internal folder after setup. Do not change this behavior.
   • Confirm the pod-side cleanupBuildDirectory in pkg/agent/dockerless.go still moves .devsy-internal into /workspaces/.dockerless after the build. Do not change this behavior.
   • Confirm config.BuildArtifactExcludes() and config.RemoveBuildArtifacts() in pkg/devcontainer/config/artifacts.go remain unchanged, because other callers (prebuild hash, volume seeding) still need them.

   🤖 Prompt for AI agents

   Implement the fix that lets the generated Dockerless build directory reach the
   Kubernetes pod through the streamed workspace mount. The goal is that
   `prepareBuildDirectory` finds the build directory before `DockerlessBuild` runs.
   
   - In `pkg/agent/tunnelserver/tunnelserver.go`, modify `StreamMount`. Remove
   `config.BuildArtifactExcludes()` from the excludes list, so `.devsy-internal` is
   no longer stripped from the streamed tar.
   - Keep `workspaceIgnoreExcludes()` unchanged in `StreamMount`. Do not change
   `.devsyignore` behavior.
   - Add a short code comment in `StreamMount` that explains why the exclude was
   removed: the Dockerless build directory must travel to the pod through this
   mount stream.
   - Do not modify `StreamWorkspace` or `StreamSnapshotVolumes`. Do not modify
   `pkg/agent/delivery/workspace_seed.go` or `pkg/devcontainer/config/prebuild.go`.
   - Verify, without code changes, that `cleanupBuildInformation` (deferred in
   `pkg/devcontainer/run.go` `Up()`) still removes the host `.devsy-internal`
   folder after setup.
   - Verify, without code changes, that `cleanupBuildDirectory` in
   `pkg/agent/dockerless.go` still moves `.devsy-internal` into
   `/workspaces/.dockerless` after the build.
   - Verify, without code changes, that `config.BuildArtifactExcludes()` and
   `config.RemoveBuildArtifacts()` in `pkg/devcontainer/config/artifacts.go` remain
   unchanged, since other callers (prebuild hash, volume seeding) depend on them.
   

   Phase 2: Add regression coverage

   This phase adds a test that fails if a future change re-excludes the Dockerless build directory from the mount stream.

   Add a StreamMount regression test

   Add a test that proves StreamMount includes .devsy-internal in the streamed tar.

   • Add the test to pkg/agent/tunnelserver/tunnelserver_test.go. Follow the existing TestStreamSnapshotVolumes_TarsMountTargets pattern.
   • Define a fakeStreamMountServer that embeds tunnel.Tunnel_StreamMountServer and collects sent chunks, with Send and Context methods.
   • In the test, create a temp dir with t.TempDir(). Add a normal file. Add a config.DevsyContextFeatureFolder subdirectory (.devsy-internal) that contains a file such as Dockerfile-without-features.
   • Build a tunnelServer struct literal with mounts set to a single *config.Mount whose Source is the temp dir.
   • Call srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()}, fake). Reassemble the tar from the collected chunks. Walk the tar with archive/tar.
   • Assert that the .devsy-internal/Dockerfile-without-features entry is present in the tar. This is the core regression assertion.
   • Assert that the normal file is also present, to confirm the stream is otherwise complete.
   • Use github.com/stretchr/testify/require, consistent with the existing test file.

   🤖 Prompt for AI agents

   Add a regression test that fails if a future change re-excludes the Dockerless
   build directory from the mount stream.
   
   - Add the test to `pkg/agent/tunnelserver/tunnelserver_test.go`, following the
   existing `TestStreamSnapshotVolumes_TarsMountTargets` pattern.
   - Define `fakeStreamMountServer`, embedding `tunnel.Tunnel_StreamMountServer`,
   and collect sent chunks. Implement `Send` and `Context` methods on it.
   - In the test, create a temp directory with `t.TempDir()`. Add a normal file to
   it. Add a `config.DevsyContextFeatureFolder` subdirectory (`.devsy-internal`)
   that contains a file such as `Dockerfile-without-features`.
   - Build a `tunnelServer` struct literal. Set `mounts` to a single
   `*config.Mount` whose `Source` is the temp directory.
   - Call `srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()},
   fake)`. Reassemble the tar from the collected chunks. Walk the tar with
   `archive/tar`.
   - Assert that the `.devsy-internal/Dockerfile-without-features` entry is present
   in the tar. This assertion is the core regression check.
   - Assert that the normal file is also present in the tar, to confirm the stream
   is otherwise complete.
   - Use `github.com/stretchr/testify/require`, consistent with the existing test
   file.
   - Use `config.DevsyContextFeatureFolder` for the folder name in the test instead
   of a string literal, after confirming it equals `.devsy-internal`.
   - Do not alter `BuildArtifactExcludes()` or its other callers.
   - Run `go test ./pkg/agent/tunnelserver/...` and `go test
   ./pkg/devcontainer/config/...` to confirm no regressions.
   

   Research

   The host CLI process resolves the devcontainer config and builds the image. For non-Docker drivers (Kubernetes), dockerlessFallback in pkg/devcontainer/build.go writes .devsy-internal/Dockerfile-without-features under the build context path. This path sits inside the local workspace folder.

   The pod agent runs cmd/internal/agentcontainer/setup.go. This process first calls syncMounts, which streams each mount from the host through the StreamMount RPC in pkg/agent/tunnelserver/tunnelserver.go. The process then runs agent.DockerlessBuild (pkg/agent/dockerless.go).

   prepareBuildDirectory expects .devsy-internal at <DOCKERLESS_CONTEXT>/.devsy-internal or at the fallback /workspaces/.dockerless/.devsy-internal.

   StreamMount currently appends config.BuildArtifactExcludes() (which returns .devsy-internal) to its exclude list. As a result, the only host copy of the build directory is stripped from the tar before it reaches the pod.

   Host-side cleanup runs later, at the end of Up(), through a deferred RemoveBuildArtifacts call.

   ---

   🚀 Next Steps

   🤖 All AI agent prompts combined

   Task: 1
   
   Implement the fix that lets the generated Dockerless build directory reach the
   Kubernetes pod through the streamed workspace mount. The goal is that
   `prepareBuildDirectory` finds the build directory before `DockerlessBuild` runs.
   
   - In `pkg/agent/tunnelserver/tunnelserver.go`, modify `StreamMount`. Remove
   `config.BuildArtifactExcludes()` from the excludes list, so `.devsy-internal` is
   no longer stripped from the streamed tar.
   - Keep `workspaceIgnoreExcludes()` unchanged in `StreamMount`. Do not change
   `.devsyignore` behavior.
   - Add a short code comment in `StreamMount` that explains why the exclude was
   removed: the Dockerless build directory must travel to the pod through this
   mount stream.
   - Do not modify `StreamWorkspace` or `StreamSnapshotVolumes`. Do not modify
   `pkg/agent/delivery/workspace_seed.go` or `pkg/devcontainer/config/prebuild.go`.
   - Verify, without code changes, that `cleanupBuildInformation` (deferred in
   `pkg/devcontainer/run.go` `Up()`) still removes the host `.devsy-internal`
   folder after setup.
   - Verify, without code changes, that `cleanupBuildDirectory` in
   `pkg/agent/dockerless.go` still moves `.devsy-internal` into
   `/workspaces/.dockerless` after the build.
   - Verify, without code changes, that `config.BuildArtifactExcludes()` and
   `config.RemoveBuildArtifacts()` in `pkg/devcontainer/config/artifacts.go` remain
   unchanged, since other callers (prebuild hash, volume seeding) depend on them.
   ===============================================================================
   
   Task: 2
   
   Add a regression test that fails if a future change re-excludes the Dockerless
   build directory from the mount stream.
   
   - Add the test to `pkg/agent/tunnelserver/tunnelserver_test.go`, following the
   existing `TestStreamSnapshotVolumes_TarsMountTargets` pattern.
   - Define `fakeStreamMountServer`, embedding `tunnel.Tunnel_StreamMountServer`,
   and collect sent chunks. Implement `Send` and `Context` methods on it.
   - In the test, create a temp directory with `t.TempDir()`. Add a normal file to
   it. Add a `config.DevsyContextFeatureFolder` subdirectory (`.devsy-internal`)
   that contains a file such as `Dockerfile-without-features`.
   - Build a `tunnelServer` struct literal. Set `mounts` to a single
   `*config.Mount` whose `Source` is the temp directory.
   - Call `srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()},
   fake)`. Reassemble the tar from the collected chunks. Walk the tar with
   `archive/tar`.
   - Assert that the `.devsy-internal/Dockerfile-without-features` entry is present
   in the tar. This assertion is the core regression check.
   - Assert that the normal file is also present in the tar, to confirm the stream
   is otherwise complete.
   - Use `github.com/stretchr/testify/require`, consistent with the existing test
   file.
   - Use `config.DevsyContextFeatureFolder` for the folder name in the test instead
   of a string literal, after confirming it equals `.devsy-internal`.
   - Do not alter `BuildArtifactExcludes()` or its other callers.
   - Run `go test ./pkg/agent/tunnelserver/...` and `go test
   ./pkg/devcontainer/config/...` to confirm no regressions.
   

   💡 Iterate on the plan with: `@coderabbitai` <feedback>

   Example Feedback
   - `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
   - `@coderabbitai` For design choice 1 go ahead with option 3 and replan.
   

   ---

   💬 Have feedback or questions? Drop into our discord!
4. skevetter

   closed this as completedin #1109on Aug 20, 2026