diff --git a/SECURITY.md b/SECURITY.md index 83e97df..b9d68c1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -29,16 +29,25 @@ as it can: grants by default are dropped - a pids limit and a memory limit (`ram_mib` + 256 MiB) - only the directories the Boxfile declares, `/data` and any `mounts:` -- with `isolation: strict`, a subordinate host UID that owns nothing of yours +- with `isolation: strict`, a subordinate host UID of its own, which owns + nothing of yours and nothing of any other sandbox's Under standard isolation the VMM runs as your user. Escaping both the guest kernel and libkrun then lands in your account. Use `isolation: strict` for agents. +Under strict, each sandbox gets its own slot of 65536 IDs from your +subordinate range (`/etc/subuid`, `/etc/subgid`), recorded with the sandbox +and freed when it is destroyed; a fork shares its parent's, because they share +files. Its `/data` belongs to that UID, with your group and mode 0750: you can +read it, no other sandbox's UID can. When the range has no free slot, box +refuses to start the sandbox and prints the `usermod` line that adds one, +rather than let two sandboxes share a UID. `box doctor` shows how many strict +sandboxes your range holds. A destroyed sandbox's kept `/data` is handed back +to you, so a later sandbox given the same slot does not own it. + ## What it is not -- **Not a boundary between strict sandboxes.** They share one subordinate UID. - Strict separates sandboxes from you, not from each other. - **Not a sandbox for image builds.** `box build` runs `podman build`, so a Boxfile's `run:` and `blueprint` steps execute in an ordinary rootless container, on the host kernel. Build only Boxfiles you would run yourself. @@ -93,8 +102,9 @@ directory. A guest can write anything into `/data`: symlinks to `/etc/passwd`, `..` entries in an archive, device nodes, setuid bits. box's own tools never follow a guest's symlink on the host and host-side `/data` operations on a strict sandbox go -through `podman unshare`, where the subordinate UID is reachable and yours is -not. Your own tools are your responsibility: do not `cp -L` or run scripts out +through `podman unshare`, where the sandbox's subordinate UID is reachable. +Files written into a strict sandbox's `/data` from outside (`box data import`, +`restore`) are given to that sandbox afterwards, so its guest owns all of it. Your own tools are your responsibility: do not `cp -L` or run scripts out of `/data` without looking. `restore` reads archives itself, in Go, rather than with the host's `tar`, diff --git a/docs/docs.html b/docs/docs.html index d1bf69c..246d031 100644 --- a/docs/docs.html +++ b/docs/docs.html @@ -77,7 +77,7 @@

Boxfile

One YAML file per sandbox at ~/.box/sandboxes/<name>/Boxfile. Unknown keys are rejected, so a typo fails loudly.

- +
FieldValuesDefaultMeaning
baseimage referencealpine:latestThe image every VM starts from.
backendpodman, krun, firecrackerpodmanWhat boots the VM. See Architecture.
isolationstandard, strictstandardstrict runs the VMM as a subordinate UID.
cpus1 to 162vCPUs.
ram_mib128 and up2048Guest RAM in MiB.
networkbridge, nonebridgeInternet access, or none at all.
readonlyboolfalseRead-only root; /tmp and /data stay writable.
timeout_secondsint0Per-command limit; exits 124. 0 is unlimited.
warm0 to 80VMs kept booted for run (podman, krun).
warmupshell linesRun once in each warm VM before it serves a run.
packagesnamesInstalled with the image's package manager.
runshell linesExtra build steps.
envmapEnvironment for every command.
mountshost, guest, modeHost directories shared in (podman, krun); ro by default.
blueprintusers, write_files, runcmdProvisioning applied at build time.
pkgmgrapk, apt, dnfinferredOverride the package manager.
passtboolfalseA real NIC with a default route (podman, krun).
seccomppathA seccomp profile for the VMM.
baseimage referencealpine:latestThe image every VM starts from.
backendpodman, krun, firecrackerpodmanWhat boots the VM. See Architecture.
isolationstandard, strictstandardstrict runs the VMM as a subordinate UID of the sandbox's own.
cpus1 to 162vCPUs.
ram_mib128 and up2048Guest RAM in MiB.
networkbridge, nonebridgeInternet access, or none at all.
readonlyboolfalseRead-only root; /tmp and /data stay writable.
timeout_secondsint0Per-command limit; exits 124. 0 is unlimited.
warm0 to 80VMs kept booted for run (podman, krun).
warmupshell linesRun once in each warm VM before it serves a run.
packagesnamesInstalled with the image's package manager.
runshell linesExtra build steps.
envmapEnvironment for every command.
mountshost, guest, modeHost directories shared in (podman, krun); ro by default.
blueprintusers, write_files, runcmdProvisioning applied at build time.
pkgmgrapk, apt, dnfinferredOverride the package manager.
passtboolfalseA real NIC with a default route (podman, krun).
seccomppathA seccomp profile for the VMM.

CLI

diff --git a/docs/index.html b/docs/index.html index 03b8d21..b21e964 100644 --- a/docs/index.html +++ b/docs/index.html @@ -229,7 +229,7 @@

Not sure it fits? Ask your AI.

-
you5.1 KB · plain text
+
you5.2 KB · plain text
I'm evaluating box (https://github.com/chxperiments/box) for something I'm building. Help me decide whether it fits, and if it does, how to set it up correctly. Use only the facts below about box; if something isn't covered here, say so instead of guessing.
 
 ## What box is
@@ -259,7 +259,7 @@ 

Not sure it fits? Ask your AI.

## Security model Two layers: the guest runs on its own kernel under KVM, and the VMM process on the host is itself confined (no_new_privs, seccomp, its own network namespace, pids and memory limits, a reduced capability set; Firecracker's VMM runs with zero capabilities). With `isolation: strict` the VMM runs as a subordinate UID instead of yours, so escaping both the VM and the VMM does not land in your account. Use strict for agents. An escape test suite ships in the repo. -Not covered: all strict sandboxes share one subordinate UID, so strict separates sandboxes from you, not from each other. The kernel check catches a fallback to a plain container but cannot catch a guest that lies about its kernel. It is not designed for hostile multi-tenant hosting. +Each strict sandbox gets its own subordinate UID, so one sandbox cannot reach another's files; the number of strict sandboxes is limited by the user's /etc/subuid range (box doctor shows it, and box prints the fix when it runs out). Not covered: the kernel check catches a fallback to a plain container but cannot catch a guest that lies about its kernel. It is not designed for hostile multi-tenant hosting. ## Requirements and limits Linux with KVM (/dev/kvm), podman and libkrun (crun built with +LIBKRUN). macOS works through a podman machine, but up/exec, the warm pool, forks and the krun and firecracker backends are Linux-only. It runs on your machine or your server: it is not a hosted service and has no GPU support. `box shell` has a TTY; `exec` does not yet. @@ -273,8 +273,8 @@

Not sure it fits? Ask your AI.

- Ask Claude - Ask ChatGPT + Ask Claude + Ask ChatGPT
@@ -287,9 +287,9 @@

Not sure it fits? Ask your AI.

Questions, answered.

Still unsure? Ask your AI with the prompt above.

-
01How is this different from running code in a container?

A container shares your host's kernel, so a kernel bug or a careless mount reaches your machine. box runs each sandbox as a microVM under KVM with its own kernel, and refuses to run one whose kernel matches the host's, which would mean it quietly became a plain container.

02Do I need root?

Only once, to install podman and libkrun and create the krun symlink. After that everything runs as you, with rootless podman. Your user needs write access to /dev/kvm (usually the kvm group), and isolation: strict needs a range in /etc/subuid. box doctor checks all of it and prints the fix.

03Does it work on macOS or Windows?

Linux with KVM is the main target. On macOS it runs through a podman machine, but up/exec, the warm pool, forks and the krun and firecracker backends are Linux-only. On Windows, use WSL2 with /dev/kvm available: the published benchmarks were measured that way.

04How fast is it?

A cold run takes about 1.3 s on podman, 0.75 s on krun and 0.3 s on firecracker. With warm: set, a fresh-VM run takes 20 to 45 ms, and exec into a running sandbox about 15 ms. Method and numbers.

05What is kept between runs?

Only /data, a directory on your host at ~/.box/data/<name>/. Every run is a new VM, so installed packages, files elsewhere and processes are gone. up keeps one VM alive when you want state to last across commands.

06Can a sandbox reach the internet or my machine?

With network: bridge it reaches the internet; with network: none it has no network at all. Either way it has its own network namespace and cannot reach services on your host's loopback. It sees only the host directories its Boxfile declares, read-only unless you say rw.

07Can my AI agent break anything?

Not outside its sandbox. Inside, it can do anything, even rm -rf /, and the next run starts clean. Only /data and the host folders you declare are kept, and forks let you review its changes before they reach your data. Use isolation: strict: the VMM around the agent is confined and runs as a UID that is not yours. box is built for your own agents, not for hosting strangers' code, and strict sandboxes share one UID between them. The threat model covers what it does not protect.

08Which backend should I use?

Start with podman, the default: it supports everything. krun boots faster but supports isolation: standard only, for now. firecracker is fastest to start but is x86_64-only and has no network, host mounts or forks yet.

09How do I let an agent use it?

Add the MCP server: claude mcp add box -- box mcp. The agent can run, exec, read and write files, and fork, but cannot apply a fork to real data unless you start the server with --allow-apply. Creating and building sandboxes stays with you.

10Is it a hosted service? What does it cost?

Neither. box runs on your laptop or your own server, MIT-licensed and free. Nothing is sent anywhere. It also has no GPU support, so GPU workloads need something else.

+
01How is this different from running code in a container?

A container shares your host's kernel, so a kernel bug or a careless mount reaches your machine. box runs each sandbox as a microVM under KVM with its own kernel, and refuses to run one whose kernel matches the host's, which would mean it quietly became a plain container.

02Do I need root?

Only once, to install podman and libkrun and create the krun symlink. After that everything runs as you, with rootless podman. Your user needs write access to /dev/kvm (usually the kvm group), and isolation: strict needs a range in /etc/subuid. box doctor checks all of it and prints the fix.

03Does it work on macOS or Windows?

Linux with KVM is the main target. On macOS it runs through a podman machine, but up/exec, the warm pool, forks and the krun and firecracker backends are Linux-only. On Windows, use WSL2 with /dev/kvm available: the published benchmarks were measured that way.

04How fast is it?

A cold run takes about 1.3 s on podman, 0.75 s on krun and 0.3 s on firecracker. With warm: set, a fresh-VM run takes 20 to 45 ms, and exec into a running sandbox about 15 ms. Method and numbers.

05What is kept between runs?

Only /data, a directory on your host at ~/.box/data/<name>/. Every run is a new VM, so installed packages, files elsewhere and processes are gone. up keeps one VM alive when you want state to last across commands.

06Can a sandbox reach the internet or my machine?

With network: bridge it reaches the internet; with network: none it has no network at all. Either way it has its own network namespace and cannot reach services on your host's loopback. It sees only the host directories its Boxfile declares, read-only unless you say rw.

07Can my AI agent break anything?

Not outside its sandbox. Inside, it can do anything, even rm -rf /, and the next run starts clean. Only /data and the host folders you declare are kept, and forks let you review its changes before they reach your data. Use isolation: strict: the VMM around the agent is confined and runs as a UID that is not yours. Each strict sandbox gets a UID of its own, so one cannot reach another's files. box is built for your own agents, not for hosting strangers' code. The threat model covers what it does not protect.

08Which backend should I use?

Start with podman, the default: it supports everything. krun boots faster but supports isolation: standard only, for now. firecracker is fastest to start but is x86_64-only and has no network, host mounts or forks yet.

09How do I let an agent use it?

Add the MCP server: claude mcp add box -- box mcp. The agent can run, exec, read and write files, and fork, but cannot apply a fork to real data unless you start the server with --allow-apply. Creating and building sandboxes stays with you.

10Is it a hosted service? What does it cost?

Neither. box runs on your laptop or your own server, MIT-licensed and free. Nothing is sent anywhere. It also has no GPU support, so GPU workloads need something else.

- +
diff --git a/docs/security.html b/docs/security.html index 2a80502..552be63 100644 --- a/docs/security.html +++ b/docs/security.html @@ -55,7 +55,7 @@

Two layers

-

Use isolation: strict for agents. Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to your first subordinate UID instead.

+

Use isolation: strict for agents. Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to a subordinate UID of the sandbox's own instead, which owns nothing of yours and nothing of any other sandbox's.

@@ -89,7 +89,6 @@

What it is not

diff --git a/internal/cli/commands.go b/internal/cli/commands.go index ac47ced..dde28ae 100644 --- a/internal/cli/commands.go +++ b/internal/cli/commands.go @@ -358,6 +358,13 @@ func restoreCmd() *cobra.Command { if err := sandbox.Restore(name, archive); err != nil { return err } + // The archive was unpacked from outside; give its files to the + // sandbox, so a strict sandbox's guest owns all of /data again. + if s, err := loadSpec(name); err == nil { + if err := runtime.ReownData(name, s); err != nil { + return err + } + } runtime.SpawnTender(name) fmt.Printf("restored %s from %s\n", name, filepath.Base(archive)) return nil @@ -524,6 +531,12 @@ func destroyCmd() *cobra.Command { if err := sandbox.RemoveFork(name); err != nil { return err } + } else if !withData { + // The sandbox's slot is freed with it; the kept /data must + // not stay owned by a UID the next strict sandbox may get. + if err := runtime.ReleaseData(name); err != nil { + return err + } } if err := sandbox.Remove(name, withData); err != nil { return err @@ -567,7 +580,13 @@ func nukeCmd() *cobra.Command { runtime.Down(n) runtime.RemovePool(n) runtime.RemoveImage(n) + fork := sandbox.IsFork(n) sandbox.RemoveFork(n) + if noData && !fork { + if err := runtime.ReleaseData(n); err != nil { + return fmt.Errorf("%s: %w", n, err) + } + } if err := sandbox.Remove(n, !noData); err != nil { return fmt.Errorf("%s: %w", n, err) } diff --git a/internal/runtime/backend_firecracker.go b/internal/runtime/backend_firecracker.go index 52b6ab2..64048e2 100644 --- a/internal/runtime/backend_firecracker.go +++ b/internal/runtime/backend_firecracker.go @@ -72,7 +72,7 @@ type fcRun struct { Disk string `json:"disk"` CPUs int `json:"cpus"` RAMMiB int `json:"ram_mib"` - Strict bool `json:"strict"` + Shift int `json:"shift"` // 0 standard; strict: the sandbox's slot in podman's namespace ReadOnly bool `json:"readonly"` Argv []string `json:"argv,omitempty"` Stdin bool `json:"stdin,omitempty"` @@ -96,6 +96,10 @@ func (b fcBackend) Launch(name string, s boxfile.Spec, l Launch) (*exec.Cmd, err if err != nil { return nil, err } + shift, err := idShift(name, s) + if err != nil { + return nil, err + } disk, err := fcDataDisk(name) if err != nil { return nil, err @@ -106,7 +110,7 @@ func (b fcBackend) Launch(name string, s boxfile.Spec, l Launch) (*exec.Cmd, err } r := fcRun{ Mode: "run", Sandbox: name, VM: vm, Image: img, Disk: disk, - CPUs: s.CPUs, RAMMiB: s.RAMMiB, Strict: s.Isolation == "strict", + CPUs: s.CPUs, RAMMiB: s.RAMMiB, Shift: shift, ReadOnly: s.ReadOnlyRootfs, Argv: l.Argv, Stdin: l.Interactive, Labels: l.Labels, Spec: s, } @@ -324,14 +328,18 @@ func fcStart(dir string, r fcRun) error { if err != nil { return err } - if r.Strict { - // The VMM runs as the subordinate UID (1 in this namespace). It owns - // the /data disk; the VM directory stays yours, shared with it through - // the group, so it can create its sockets there and nobody else can. - if err := os.Lchown(r.Disk, 1, 1); err != nil { + if r.Shift > 0 { + // The VMM runs as the sandbox's own subordinate UID (its shift in + // this namespace). It owns the /data disk; the VM directory stays + // yours, shared with it through the group, so it can create its + // sockets there and nobody else can -- no other sandbox either. + if err := os.Lchown(r.Disk, r.Shift, r.Shift); err != nil { return err } - if err := os.Lchown(dir, 0, 1); err != nil { + if err := os.Chmod(r.Disk, 0o600); err != nil { + return err + } + if err := os.Lchown(dir, 0, r.Shift); err != nil { return err } if err := os.Chmod(dir, 0o770); err != nil { @@ -460,13 +468,14 @@ func fcSpec(r fcRun, dir, jail, binDir, binName, kernelDir string) ([]byte, erro Seccomp: seccomp, }, } - if r.Strict { - // The VMM runs as UID 1 of podman's namespace -- your first - // subordinate UID on the host -- rather than in a nested user - // namespace: crun sets the mounts up as that namespace's root, where - // every path is reachable whatever your home directory's mode, and - // only then drops to the VMM's identity, with no capabilities. - spec.Process.User = ociUser{UID: 1, GID: 1} + if r.Shift > 0 { + // The VMM runs as the sandbox's own UID in podman's namespace -- a + // subordinate UID on the host that no other sandbox has -- rather + // than in a nested user namespace: crun sets the mounts up as that + // namespace's root, where every path is reachable whatever your home + // directory's mode, and only then drops to the VMM's identity, with + // no capabilities. + spec.Process.User = ociUser{UID: uint32(r.Shift), GID: uint32(r.Shift)} } return json.MarshalIndent(spec, "", " ") } @@ -593,12 +602,14 @@ func fcRestore(api fcClient, dir string, r fcRun, token string) error { return fmt.Errorf("preparing the restored VM: exit %d: %s", res.Code, strings.TrimSpace(errb.String())) } // The VMM created the sockets as itself; under strict that is the - // subordinate UID. Hand them to you, owner and group only, so the CLI - // can connect and other users cannot. - for _, s := range []string{"v.sock", "api.sock"} { - p := filepath.Join(dir, s) - os.Lchown(p, 0, 1) - os.Chmod(p, 0o660) + // sandbox's subordinate UID. Hand them to you, shared with that UID + // through the group, so the CLI can connect and nobody else can. + if r.Shift > 0 { + for _, s := range []string{"v.sock", "api.sock"} { + p := filepath.Join(dir, s) + os.Lchown(p, 0, r.Shift) + os.Chmod(p, 0o660) + } } // Kept (owner-only) so teardown can reach the agent to flush /data, // whoever tears the VM down. diff --git a/internal/runtime/data.go b/internal/runtime/data.go index 9c03ac7..6d16551 100644 --- a/internal/runtime/data.go +++ b/internal/runtime/data.go @@ -49,7 +49,12 @@ func ImportData(name string, s boxfile.Spec, r io.Reader) error { return err } if mine, err := sandbox.OwnedByMe(data); err == nil && !mine { - return hostHelper(r, io.Discard, "__tarin", data) + // A strict sandbox's /data: unpack inside podman's namespace, then + // give what was unpacked to the sandbox, so its guest owns all of it. + if err := hostHelper(r, io.Discard, "__tarin", data); err != nil { + return err + } + return ReownData(name, s) } _, err = sandbox.SafeUntar(r, data) return err diff --git a/internal/runtime/doctor.go b/internal/runtime/doctor.go index 94e8ce5..cbdd26a 100644 --- a/internal/runtime/doctor.go +++ b/internal/runtime/doctor.go @@ -80,8 +80,20 @@ func Doctor() []Check { add(Check{Name: "subordinate UIDs", Warn: true, Detail: "none for " + currentUser() + " in /etc/subuid: isolation: strict cannot run", Fix: "sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 " + currentUser()}) + } else if n := slotCapacity(); n == 0 { + add(Check{Name: "subordinate UIDs", Warn: true, + Detail: ids + fmt.Sprintf(": fewer than %d IDs, so isolation: strict cannot run", slotSize), + Fix: "sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 " + currentUser()}) } else { - add(Check{Name: "subordinate UIDs", OK: true, Detail: ids}) + detail := ids + fmt.Sprintf(": room for %d strict sandbox(es), each with its own UIDs", n) + check := Check{Name: "subordinate UIDs", OK: true, Detail: detail} + if n < 4 { + start := freeSubordinateStart() + check = Check{Name: "subordinate UIDs", Warn: true, Detail: detail, + Fix: fmt.Sprintf("sudo usermod --add-subuids %d-%d --add-subgids %d-%d %s && podman system migrate", + start, start+64*slotSize-1, start, start+64*slotSize-1, currentUser())} + } + add(check) } if exe, err := os.Executable(); err == nil { diff --git a/internal/runtime/fc_image.go b/internal/runtime/fc_image.go index 000ec96..8442632 100644 --- a/internal/runtime/fc_image.go +++ b/internal/runtime/fc_image.go @@ -34,9 +34,9 @@ func fcImageDir(name string, s boxfile.Spec) (string, error) { if err != nil { return "", err } - shift := 0 - if s.Isolation == "strict" { - shift = 1 + shift, err := idShift(name, s) + if err != nil { + return "", err } // The agent is baked into the root disk and the snapshot, so a new // box means a new image; the key includes its hash. @@ -202,6 +202,10 @@ func fcDataDisk(name string) (string, error) { // fcSnapshot boots the image once, waits for its agent, pauses it and // snapshots it into dir. func fcSnapshot(name string, s boxfile.Spec, dir string) error { + shift, err := idShift(name, s) + if err != nil { + return err + } token, err := newToken() if err != nil { return err @@ -221,7 +225,7 @@ func fcSnapshot(name string, s boxfile.Spec, dir string) error { vm := "box-snap-" + name cmd, err := fcLaunchCmd(fcRun{ Mode: "snapshot", Sandbox: name, VM: vm, Image: dir, Disk: placeholder, - CPUs: s.CPUs, RAMMiB: s.RAMMiB, Strict: s.Isolation == "strict", + CPUs: s.CPUs, RAMMiB: s.RAMMiB, Shift: shift, ReadOnly: s.ReadOnlyRootfs, BootToken: token, Spec: s, }) if err != nil { diff --git a/internal/runtime/runtime.go b/internal/runtime/runtime.go index 6c194d3..ae945d3 100644 --- a/internal/runtime/runtime.go +++ b/internal/runtime/runtime.go @@ -13,6 +13,7 @@ import ( "path/filepath" "strconv" "strings" + "syscall" "time" "box/internal/agent" @@ -45,7 +46,11 @@ func vmArgs(name string, s boxfile.Spec, interactive, useKrun bool) ([]string, e if useKrun { args = append(args, "--runtime", "krun") } - args = append(args, hardening(s)...) + shift, err := idShift(name, s) + if err != nil { + return nil, err + } + args = append(args, hardening(s, shift)...) args = append(args, "--network="+s.Network, "--annotation", "krun.cpus="+strconv.Itoa(s.CPUs), @@ -161,33 +166,28 @@ const vmmPidsLimit = 512 // hardening confines the VMM process on the host. It is the boundary between // a guest that has broken out of libkrun and the rest of the machine. -func hardening(s boxfile.Spec) []string { +func hardening(s boxfile.Spec, shift int) []string { args := []string{ "--security-opt", "no-new-privileges", "--cap-drop=all", "--cap-add=" + vmmCaps, "--pids-limit=" + strconv.Itoa(vmmPidsLimit), "--memory=" + strconv.Itoa(s.RAMMiB+vmmOverheadMiB) + "m", } - if s.Isolation == "strict" { + if shift > 0 { // Rootless podman maps the sandbox's root to your own UID, so a guest // that escaped libkrun would land in your account. Strict maps it to - // the first UID of your subordinate range instead and leaves your UID - // out of the mapping entirely: an escape lands in an account that owns - // nothing of yours. The range is the same for every strict sandbox, so - // it separates sandboxes from you, not from each other. - args = append(args, "--uidmap", strictMap, "--gidmap", strictMap) + // the sandbox's own slot of your subordinate range instead and leaves + // your UID out of the mapping entirely: an escape lands in an account + // that owns nothing of yours, and nothing of any other sandbox's. + args = append(args, "--uidmap", idMap(shift), "--gidmap", idMap(shift)) } return args } -// strictMap maps sandbox IDs 0-65535 onto IDs 1-65536 of the rootless user -// namespace, i.e. onto your subordinate UIDs and never onto you (ID 0 there). -const strictMap = "0:1:65536" - // prepareData gives /data to whoever the sandbox's root is on the host: you -// under standard isolation, the first subordinate UID under strict. Ownership -// only changes when a sandbox switches between the two, so this is a stat on -// every boot and a chown once. +// under standard isolation, the sandbox's own subordinate UID under strict. +// Ownership only changes when a sandbox switches mode or slot, so this is a +// stat on every boot and a chown once. func prepareData(name string, s boxfile.Spec) error { if parent, err := sandbox.Parent(name); err == nil { // A fork's /data is the parent's, overlaid. The lower layer is @@ -208,39 +208,82 @@ func prepareData(name string, s boxfile.Spec) error { if err != nil { return err } - return ownData(upper, s) + shift, err := idShift(name, s) // the parent's slot + if err != nil { + return err + } + return ownData(upper, shift, false) } data, err := sandbox.DataDir(name) if err != nil { return err } - return ownData(data, s) + shift, err := idShift(name, s) + if err != nil { + return err + } + return ownData(data, shift, false) } -// ownData gives a data directory to whoever the sandbox's root is on the host. -func ownData(data string, s boxfile.Spec) error { +// ownData gives a data directory to whoever the sandbox's root is on the +// host. Under strict that is the sandbox's slot, and the top directory is +// 0750 with your group: you can still read it, other sandboxes' UIDs cannot. +// force re-owns everything below too, after files were written into it from +// outside (an import, a restore); otherwise only a wrong owner on the top +// directory triggers it. +func ownData(data string, shift int, force bool) error { if err := os.MkdirAll(data, 0o755); err != nil { return err } - mine, err := sandbox.OwnedByMe(data) + fi, err := os.Stat(data) if err != nil { return err } - strict := s.Isolation == "strict" - if mine != strict { - return nil // already owned by the sandbox's root + st, ok := fi.Sys().(*syscall.Stat_t) + if !ok { + return nil // not a Unix filesystem: nothing to map } - // Inside podman unshare, ID 0 is you and ID 1 the first subordinate UID. - owner := "0:0" - if strict { - owner = "1:1" + wantUID, ok := hostID(shift, os.Getuid(), subordinateRanges("/etc/subuid")) + if !ok { + return fmt.Errorf("/data's owner (namespace ID %d) is outside your subordinate range; see box doctor", shift) } - if out, err := exec.Command("podman", "unshare", "chown", "-R", owner, data).CombinedOutput(); err != nil { - return fmt.Errorf("re-owning /data for isolation %s: %s", s.Isolation, strings.TrimSpace(string(out))) + if !force && int(st.Uid) == wantUID && (shift == 0 || (int(st.Gid) == os.Getgid() && fi.Mode().Perm() == 0o750)) { + return nil + } + if int(st.Uid) != wantUID || force { + owner := fmt.Sprintf("%d:%d", shift, shift) + if out, err := exec.Command("podman", "unshare", "chown", "-R", owner, data).CombinedOutput(); err != nil { + return fmt.Errorf("re-owning /data: %s", strings.TrimSpace(string(out))) + } + } + if shift > 0 { + // The top directory keeps your group (ID 0 in the namespace) and + // shuts everyone else out, including every other sandbox's UID. + if out, err := exec.Command("podman", "unshare", "sh", "-c", + `chown "$1:0" "$2" && chmod 0750 "$2"`, "sh", strconv.Itoa(shift), data).CombinedOutput(); err != nil { + return fmt.Errorf("closing /data to other sandboxes: %s", strings.TrimSpace(string(out))) + } } return nil } +// ReownData re-owns a sandbox's whole /data after files were written into it +// from outside, so the guest owns all of it. +func ReownData(name string, s boxfile.Spec) error { + if s.Backend == "firecracker" || sandbox.IsFork(name) { + return nil // a disk the guest writes itself; forks import via parents + } + data, err := sandbox.DataDir(name) + if err != nil { + return err + } + shift, err := idShift(name, s) + if err != nil { + return err + } + return ownData(data, shift, true) +} + // Build renders the Containerfile from the spec, writes it, and builds the image. func Build(name string, s boxfile.Spec) error { cfPath, err := sandbox.ContainerfilePath(name) diff --git a/internal/runtime/slots.go b/internal/runtime/slots.go new file mode 100644 index 0000000..13e4017 --- /dev/null +++ b/internal/runtime/slots.go @@ -0,0 +1,261 @@ +package runtime + +import ( + "bufio" + "fmt" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "syscall" + + "box/internal/boxfile" + "box/internal/sandbox" +) + +// Every strict sandbox runs as its own block of subordinate IDs. Inside +// podman's user namespace, ID 0 is you and IDs 1.. are your subordinate +// range, in /etc/subuid order. A sandbox's IDs 0-65535 need 65536 of them, so +// the range is cut into slots of that size, and slot k maps the sandbox onto +// namespace IDs 1+k*65536 onward. Two strict sandboxes therefore never share +// a UID: a guest that escapes its VM and its VMM lands in an account that +// owns nothing of yours and nothing of any other sandbox's. +// +// A sandbox's slot is recorded in its directory, so it survives rename and +// is freed by destroy. A fork shares its parent's slot, because the two share +// files and files have one owner. + +const slotSize = 65536 + +// idShift is how far a sandbox's IDs move in podman's user namespace: 0 +// under standard isolation (the sandbox's root is you), 1+slot*65536 under +// strict. +func idShift(name string, s boxfile.Spec) (int, error) { + if s.Isolation != "strict" { + return 0, nil + } + owner := name + if parent, err := sandbox.Parent(name); err == nil { + owner = parent + } + slot, err := strictSlot(owner) + if err != nil { + return 0, err + } + return 1 + slot*slotSize, nil +} + +// idMap is the --uidmap/--gidmap value for a shift: the sandbox's 65536 IDs +// onto namespace IDs shift.. -- never onto you, who are ID 0. +func idMap(shift int) string { return fmt.Sprintf("0:%d:%d", shift, slotSize) } + +func slotPath(name string) (string, error) { + dir, err := sandbox.Dir(name) + if err != nil { + return "", err + } + return filepath.Join(dir, "slot"), nil +} + +func readSlot(name string) (int, bool) { + p, err := slotPath(name) + if err != nil { + return 0, false + } + b, err := os.ReadFile(p) + if err != nil { + return 0, false + } + n, err := strconv.Atoi(strings.TrimSpace(string(b))) + if err != nil || n < 0 { + return 0, false + } + return n, true +} + +// strictSlot returns the sandbox's slot, allocating the lowest free one the +// first time. Allocation holds a lock, so concurrent box processes (a warm +// pool refilling, two runs) never hand out the same slot twice. +func strictSlot(name string) (int, error) { + if n, ok := readSlot(name); ok { + return n, nil + } + home, err := sandbox.Home() + if err != nil { + return 0, err + } + if err := os.MkdirAll(home, 0o700); err != nil { + return 0, err + } + lock, err := os.OpenFile(filepath.Join(home, "slots.lock"), os.O_CREATE|os.O_RDWR, 0o600) + if err != nil { + return 0, err + } + defer lock.Close() + if err := syscall.Flock(int(lock.Fd()), syscall.LOCK_EX); err != nil { + return 0, err + } + defer syscall.Flock(int(lock.Fd()), syscall.LOCK_UN) + if n, ok := readSlot(name); ok { // allocated while we waited + return n, nil + } + capacity := slotCapacityFn() + // Every sandbox directory counts, Boxfile or not: a slot recorded + // anywhere is taken, so no two sandboxes can ever be handed the same one. + entries, err := os.ReadDir(filepath.Join(home, "sandboxes")) + if err != nil && !os.IsNotExist(err) { + return 0, err + } + used := map[int]bool{} + for _, e := range entries { + if !e.IsDir() || sandbox.ValidName(e.Name()) != nil { + continue + } + if n, ok := readSlot(e.Name()); ok { + used[n] = true + } + } + for n := 0; n < capacity; n++ { + if used[n] { + continue + } + p, err := slotPath(name) + if err != nil { + return 0, err + } + tmp := p + ".tmp" + if err := os.WriteFile(tmp, []byte(strconv.Itoa(n)+"\n"), 0o600); err != nil { + return 0, err + } + if err := os.Rename(tmp, p); err != nil { + return 0, err + } + return n, nil + } + return 0, slotsExhausted(capacity, len(used)) +} + +func slotsExhausted(capacity, used int) error { + if capacity == 0 { + return fmt.Errorf("isolation: strict needs at least %d subordinate IDs for %s and has none in /etc/subuid and /etc/subgid; see box doctor", + slotSize, currentUser()) + } + want := 64 * slotSize + start := freeSubordinateStart() + return fmt.Errorf("every strict sandbox needs its own %d subordinate IDs, and all %d slot(s) in your range are taken by %d strict sandbox(es).\n"+ + "Add a range no other user has, then let podman pick it up:\n"+ + " sudo usermod --add-subuids %d-%d --add-subgids %d-%d %s\n"+ + " podman system migrate # stops running containers\n"+ + "That makes room for 64 more strict sandboxes.", + slotSize, capacity, used, start, start+want-1, start, start+want-1, currentUser()) +} + +// subRange is one line of /etc/subuid or /etc/subgid for this user. +type subRange struct{ start, count int } + +// subordinateRanges returns this user's ranges from path, in file order -- +// the order podman lays them out in its namespace. +func subordinateRanges(path string) []subRange { + f, err := os.Open(path) + if err != nil { + return nil + } + defer f.Close() + user, uid := currentUser(), strconv.Itoa(os.Getuid()) + var out []subRange + sc := bufio.NewScanner(f) + for sc.Scan() { + p := strings.Split(strings.TrimSpace(sc.Text()), ":") + if len(p) != 3 || (p[0] != user && p[0] != uid) { + continue + } + start, err1 := strconv.Atoi(p[1]) + count, err2 := strconv.Atoi(p[2]) + if err1 == nil && err2 == nil && count > 0 { + out = append(out, subRange{start, count}) + } + } + return out +} + +func rangeTotal(rs []subRange) int { + n := 0 + for _, r := range rs { + n += r.count + } + return n +} + +// slotCapacityFn is slotCapacity, replaceable in tests. +var slotCapacityFn = slotCapacity + +// slotCapacity is how many strict sandboxes the subordinate ranges hold: +// whichever of the UID and GID ranges is smaller decides. +func slotCapacity() int { + n := min(rangeTotal(subordinateRanges("/etc/subuid")), rangeTotal(subordinateRanges("/etc/subgid"))) + return n / slotSize +} + +// hostID maps an ID in podman's namespace to the host: 0 is you, 1.. walk +// your subordinate ranges in order. +func hostID(nsID int, self int, rs []subRange) (int, bool) { + if nsID == 0 { + return self, true + } + i := nsID - 1 + for _, r := range rs { + if i < r.count { + return r.start + i, true + } + i -= r.count + } + return 0, false +} + +// freeSubordinateStart suggests where a new range can begin: past the end of +// every range any user has, rounded up to a slot. +func freeSubordinateStart() int { + end := 100000 + for _, path := range []string{"/etc/subuid", "/etc/subgid"} { + b, err := os.ReadFile(path) + if err != nil { + continue + } + for _, line := range strings.Split(string(b), "\n") { + p := strings.Split(strings.TrimSpace(line), ":") + if len(p) != 3 { + continue + } + s, err1 := strconv.Atoi(p[1]) + c, err2 := strconv.Atoi(p[2]) + if err1 == nil && err2 == nil && s+c > end { + end = s + c + } + } + } + return (end + slotSize - 1) / slotSize * slotSize +} + +// releaseData hands a strict sandbox's kept /data back to you when the +// sandbox is destroyed without --data. Its slot is freed with it, and the +// next strict sandbox given that slot must not own the files. +func releaseData(name string) error { + data, err := sandbox.DataDir(name) + if err != nil { + return err + } + if _, err := os.Stat(data); err != nil { + return nil + } + if mine, err := sandbox.OwnedByMe(data); err != nil || mine { + return err + } + if out, err := exec.Command("podman", "unshare", "chown", "-R", "0:0", data).CombinedOutput(); err != nil { + return fmt.Errorf("handing %s's /data back to you: %s", name, strings.TrimSpace(string(out))) + } + return nil +} + +// ReleaseData is releaseData for the CLI's destroy and nuke. +func ReleaseData(name string) error { return releaseData(name) } diff --git a/internal/runtime/slots_test.go b/internal/runtime/slots_test.go new file mode 100644 index 0000000..65a0ff6 --- /dev/null +++ b/internal/runtime/slots_test.go @@ -0,0 +1,151 @@ +package runtime + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "box/internal/boxfile" + "box/internal/sandbox" +) + +func TestIDMap(t *testing.T) { + if got := idMap(1); got != "0:1:65536" { + t.Errorf("slot 0: %s", got) + } + if got := idMap(1 + 3*slotSize); got != "0:196609:65536" { + t.Errorf("slot 3: %s", got) + } +} + +func TestHostIDWalksRangesInOrder(t *testing.T) { + rs := []subRange{{start: 100000, count: 65536}, {start: 500000, count: 131072}} + for _, c := range []struct{ ns, host int }{ + {0, 1000}, // you + {1, 100000}, // first subordinate ID + {65536, 165535}, // end of the first range + {65537, 500000}, // start of the second + {1 + 2*slotSize, 565536}, // slot 2 begins inside the second range + } { + got, ok := hostID(c.ns, 1000, rs) + if !ok || got != c.host { + t.Errorf("hostID(%d) = %d, %v; want %d", c.ns, got, ok, c.host) + } + } + if _, ok := hostID(1+3*slotSize, 1000, rs); ok { + t.Error("an ID past the ranges mapped") + } +} + +func TestSubordinateRanges(t *testing.T) { + t.Setenv("USER", "alice") + p := filepath.Join(t.TempDir(), "subuid") + os.WriteFile(p, []byte("bob:100000:65536\nalice:165536:65536\n# x\nalice:900000:131072\n"), 0o644) + rs := subordinateRanges(p) + if len(rs) != 2 || rs[0] != (subRange{165536, 65536}) || rs[1] != (subRange{900000, 131072}) { + t.Fatalf("ranges: %+v", rs) + } + if rangeTotal(rs)/slotSize != 3 { + t.Errorf("capacity %d, want 3", rangeTotal(rs)/slotSize) + } +} + +func withSlots(t *testing.T, capacity int) { + t.Helper() + t.Setenv("BOX_HOME", t.TempDir()) + old := slotCapacityFn + slotCapacityFn = func() int { return capacity } + t.Cleanup(func() { slotCapacityFn = old }) +} + +func newSandbox(t *testing.T, name string) { + t.Helper() + dir, err := sandbox.Create(name) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "Boxfile"), []byte("base: alpine\n"), 0o644); err != nil { + t.Fatal(err) + } +} + +func TestStrictSandboxesGetTheirOwnSlots(t *testing.T) { + withSlots(t, 2) + strict := boxfile.Spec{Isolation: "strict"} + for _, n := range []string{"a", "b", "c"} { + newSandbox(t, n) + } + sa, err := idShift("a", strict) + if err != nil { + t.Fatal(err) + } + sb, err := idShift("b", strict) + if err != nil { + t.Fatal(err) + } + if sa == sb { + t.Fatalf("a and b share shift %d", sa) + } + if again, _ := idShift("a", strict); again != sa { + t.Errorf("a's slot moved: %d then %d", sa, again) + } + // No third slot: refuse, with the fix, rather than share a UID. + if _, err := idShift("c", strict); err == nil || !strings.Contains(err.Error(), "usermod --add-subuids") { + t.Fatalf("third strict sandbox: %v", err) + } + // Destroying a frees its slot for c. + if err := sandbox.Remove("a", false); err != nil { + t.Fatal(err) + } + sc, err := idShift("c", strict) + if err != nil || sc != sa { + t.Errorf("c got %d, %v; want a's freed %d", sc, err, sa) + } +} + +func TestSlotsCountedWithoutABoxfile(t *testing.T) { + withSlots(t, 2) + strict := boxfile.Spec{Isolation: "strict"} + newSandbox(t, "a") + sa, err := idShift("a", strict) + if err != nil { + t.Fatal(err) + } + // a's Boxfile goes missing: its slot must still count as taken. + dir, _ := sandbox.Dir("a") + os.Remove(filepath.Join(dir, "Boxfile")) + newSandbox(t, "b") + sb, err := idShift("b", strict) + if err != nil || sb == sa { + t.Errorf("b got %d, %v; a holds %d", sb, err, sa) + } +} + +func TestStandardSandboxesTakeNoSlot(t *testing.T) { + withSlots(t, 0) + newSandbox(t, "plain") + if shift, err := idShift("plain", boxfile.Spec{Isolation: "standard"}); err != nil || shift != 0 { + t.Errorf("standard: %d, %v", shift, err) + } + if _, err := idShift("plain", boxfile.Spec{Isolation: "strict"}); err == nil || !strings.Contains(err.Error(), "none in /etc/subuid") { + t.Errorf("strict without a range: %v", err) + } +} + +func TestSlotSurvivesRename(t *testing.T) { + withSlots(t, 4) + newSandbox(t, "old") + strict := boxfile.Spec{Isolation: "strict"} + before, err := idShift("old", strict) + if err != nil { + t.Fatal(err) + } + if err := sandbox.Rename("old", "new"); err != nil { + t.Fatal(err) + } + after, err := idShift("new", strict) + if err != nil || after != before { + t.Errorf("after rename: %d, %v; want %d", after, err, before) + } +} diff --git a/security/escape-test.sh b/security/escape-test.sh index 84612c8..a954a3d 100755 --- a/security/escape-test.sh +++ b/security/escape-test.sh @@ -183,6 +183,44 @@ else fail "could not inspect the VMM process" fi +if [ "$ISOLATION" = strict ] && [ -n "$vmm" ] && [ -r "/proc/$vmm/status" ]; then + echo + echo "other sandboxes" + # Each strict sandbox runs as its own slot of subordinate IDs; what it + # keeps must be closed to every other sandbox's UID. + if [ "$BACKEND" = firecracker ]; then + target="$BOX_HOME/disks/$NAME.ext4"; want_mode=600 + else + target="$BOX_HOME/data/$NAME"; want_mode=750 + fi + read -r t_uid t_gid t_mode < <(stat -c '%u %g %a' "$target" 2>/dev/null) + [ "$t_uid" = "$uid" ] && pass "the VMM's UID ($uid) owns this sandbox's /data, and only this sandbox runs as it" \ + || fail "/data is owned by $t_uid, but the VMM runs as $uid" + if [ "$t_mode" = "$want_mode" ] && { [ "$BACKEND" = firecracker ] || [ "$t_gid" = "$(id -g)" ]; }; then + pass "/data is closed to other sandboxes (mode $t_mode)" + else + fail "/data is open to other sandboxes (mode $t_mode, group $t_gid)" + fi + # Then try it: a UID from another slot reads the sandbox's files. + slot=$(cat "$BOX_HOME/sandboxes/$NAME/slot" 2>/dev/null) + total=$(awk -F: -v u="$(id -un)" -v n="$(id -u)" '($1==u||$1==n){s+=$3} END{print s+0}' /etc/subuid) + other="" + if [ -n "$slot" ] && [ $(( (slot + 2) * 65536 )) -le "$total" ]; then other=$(( 1 + (slot + 1) * 65536 )) + elif [ -n "$slot" ] && [ "$slot" -gt 0 ]; then other=$(( 1 + (slot - 1) * 65536 )); fi + if [ -z "$other" ]; then + warn "skipped reading as another sandbox's UID: /etc/subuid holds one strict sandbox ($total IDs)" + elif ! command -v setpriv >/dev/null; then + warn "skipped reading as another sandbox's UID: setpriv is not installed" + else + if [ "$BACKEND" = firecracker ]; then probe="$target"; else probe="$target/w"; fi + if podman unshare setpriv --reuid="$other" --regid="$other" --clear-groups cat "$probe" >/dev/null 2>&1; then + fail "another sandbox's UID read this sandbox's /data" + else + pass "another sandbox's UID cannot read this sandbox's /data" + fi + fi +fi + echo echo "resource limits" if [ "$BACKEND" != podman ] && [ -n "$vmm" ]; then diff --git a/tools/site/build.py b/tools/site/build.py index 6b3f640..d15eb15 100644 --- a/tools/site/build.py +++ b/tools/site/build.py @@ -29,7 +29,7 @@ ("Can a sandbox reach the internet or my machine?", "With network: bridge it reaches the internet; with network: none it has no network at all. Either way it has its own network namespace and cannot reach services on your host's loopback. It sees only the host directories its Boxfile declares, read-only unless you say rw."), ("Can my AI agent break anything?", - "Not outside its sandbox. Inside, it can do anything, even rm -rf /, and the next run starts clean. Only /data and the host folders you declare are kept, and forks let you review its changes before they reach your data. Use isolation: strict: the VMM around the agent is confined and runs as a UID that is not yours. box is built for your own agents, not for hosting strangers' code, and strict sandboxes share one UID between them. The threat model covers what it does not protect."), + "Not outside its sandbox. Inside, it can do anything, even rm -rf /, and the next run starts clean. Only /data and the host folders you declare are kept, and forks let you review its changes before they reach your data. Use isolation: strict: the VMM around the agent is confined and runs as a UID that is not yours. Each strict sandbox gets a UID of its own, so one cannot reach another's files. box is built for your own agents, not for hosting strangers' code. The threat model covers what it does not protect."), ("Which backend should I use?", "Start with podman, the default: it supports everything. krun boots faster but supports isolation: standard only, for now. firecracker is fastest to start but is x86_64-only and has no network, host mounts or forks yet."), ("How do I let an agent use it?", @@ -550,7 +550,7 @@ def conf_table():
{art.security()}
-

Use isolation: strict for agents. Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to your first subordinate UID instead.

+

Use isolation: strict for agents. Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to a subordinate UID of the sandbox's own instead, which owns nothing of yours and nothing of any other sandbox's.

@@ -581,7 +581,6 @@ def conf_table(): @@ -693,7 +692,7 @@ def api_code(name, py, ts, go, rs): BOXFILE_FIELDS = [ ("base", "image reference", "alpine:latest", "The image every VM starts from."), ("backend", "podman, krun, firecracker", "podman", "What boots the VM. See Architecture."), - ("isolation", "standard, strict", "standard", "strict runs the VMM as a subordinate UID."), + ("isolation", "standard, strict", "standard", "strict runs the VMM as a subordinate UID of the sandbox's own."), ("cpus", "1 to 16", "2", "vCPUs."), ("ram_mib", "128 and up", "2048", "Guest RAM in MiB."), ("network", "bridge, none", "bridge", "Internet access, or none at all."), diff --git a/tools/site/fit_prompt.txt b/tools/site/fit_prompt.txt index 8014e6d..cc92f56 100644 --- a/tools/site/fit_prompt.txt +++ b/tools/site/fit_prompt.txt @@ -27,7 +27,7 @@ base (image ref), backend, cpus (max 16), ram_mib, network (bridge = internet, n ## Security model Two layers: the guest runs on its own kernel under KVM, and the VMM process on the host is itself confined (no_new_privs, seccomp, its own network namespace, pids and memory limits, a reduced capability set; Firecracker's VMM runs with zero capabilities). With `isolation: strict` the VMM runs as a subordinate UID instead of yours, so escaping both the VM and the VMM does not land in your account. Use strict for agents. An escape test suite ships in the repo. -Not covered: all strict sandboxes share one subordinate UID, so strict separates sandboxes from you, not from each other. The kernel check catches a fallback to a plain container but cannot catch a guest that lies about its kernel. It is not designed for hostile multi-tenant hosting. +Each strict sandbox gets its own subordinate UID, so one sandbox cannot reach another's files; the number of strict sandboxes is limited by the user's /etc/subuid range (box doctor shows it, and box prints the fix when it runs out). Not covered: the kernel check catches a fallback to a plain container but cannot catch a guest that lies about its kernel. It is not designed for hostile multi-tenant hosting. ## Requirements and limits Linux with KVM (/dev/kvm), podman and libkrun (crun built with +LIBKRUN). macOS works through a podman machine, but up/exec, the warm pool, forks and the krun and firecracker backends are Linux-only. It runs on your machine or your server: it is not a hosted service and has no GPU support. `box shell` has a TTY; `exec` does not yet.