strict: a UID of its own for every sandbox (#27)

SECURITY.md

@@ -29,16 +29,25 @@ as it can:
   grants by default are dropped
 - a pids limit and a memory limit (`ram_mib` + 256 MiB)
 - only the directories the Boxfile declares, `/data` and any `mounts:`
-- with `isolation: strict`, a subordinate host UID that owns nothing of yours
+- with `isolation: strict`, a subordinate host UID of its own, which owns
+  nothing of yours and nothing of any other sandbox's
 
 Under standard isolation the VMM runs as your user. Escaping both the guest
 kernel and libkrun then lands in your account. Use `isolation: strict` for
 agents.
 
+Under strict, each sandbox gets its own slot of 65536 IDs from your
+subordinate range (`/etc/subuid`, `/etc/subgid`), recorded with the sandbox
+and freed when it is destroyed; a fork shares its parent's, because they share
+files. Its `/data` belongs to that UID, with your group and mode 0750: you can
+read it, no other sandbox's UID can. When the range has no free slot, box
+refuses to start the sandbox and prints the `usermod` line that adds one,
+rather than let two sandboxes share a UID. `box doctor` shows how many strict
+sandboxes your range holds. A destroyed sandbox's kept `/data` is handed back
+to you, so a later sandbox given the same slot does not own it.
+
 ## What it is not
 
-- **Not a boundary between strict sandboxes.** They share one subordinate UID.
-  Strict separates sandboxes from you, not from each other.
 - **Not a sandbox for image builds.** `box build` runs `podman build`, so a
   Boxfile's `run:` and `blueprint` steps execute in an ordinary rootless
   container, on the host kernel. Build only Boxfiles you would run yourself.
@@ -93,8 +102,9 @@ directory.
 A guest can write anything into `/data`: symlinks to `/etc/passwd`, `..`
 entries in an archive, device nodes, setuid bits. box's own tools never
 follow a guest's symlink on the host and host-side `/data` operations on a strict sandbox go
-through `podman unshare`, where the subordinate UID is reachable and yours is
-not. Your own tools are your responsibility: do not `cp -L` or run scripts out
+through `podman unshare`, where the sandbox's subordinate UID is reachable.
+Files written into a strict sandbox's `/data` from outside (`box data import`,
+`restore`) are given to that sandbox afterwards, so its guest owns all of it. Your own tools are your responsibility: do not `cp -L` or run scripts out
 of `/data` without looking.
 
 `restore` reads archives itself, in Go, rather than with the host's `tar`,

docs/docs.html

@@ -77,7 +77,7 @@ <h2 id="boxfile">Boxfile</h2>
     <p>One YAML file per sandbox at <code>~/.box/sandboxes/&lt;name&gt;/Boxfile</code>. Unknown keys are rejected, so a typo fails loudly.</p>
     <div class="table-wrap build"><table>
       <thead><tr><th>Field</th><th>Values</th><th>Default</th><th>Meaning</th></tr></thead>
-      <tbody><tr><td class="mono">base</td><td>image reference</td><td class="mono">alpine:latest</td><td>The image every VM starts from.</td></tr><tr><td class="mono">backend</td><td>podman, krun, firecracker</td><td class="mono">podman</td><td>What boots the VM. See Architecture.</td></tr><tr><td class="mono">isolation</td><td>standard, strict</td><td class="mono">standard</td><td>strict runs the VMM as a subordinate UID.</td></tr><tr><td class="mono">cpus</td><td>1 to 16</td><td class="mono">2</td><td>vCPUs.</td></tr><tr><td class="mono">ram_mib</td><td>128 and up</td><td class="mono">2048</td><td>Guest RAM in MiB.</td></tr><tr><td class="mono">network</td><td>bridge, none</td><td class="mono">bridge</td><td>Internet access, or none at all.</td></tr><tr><td class="mono">readonly</td><td>bool</td><td class="mono">false</td><td>Read-only root; /tmp and /data stay writable.</td></tr><tr><td class="mono">timeout_seconds</td><td>int</td><td class="mono">0</td><td>Per-command limit; exits 124. 0 is unlimited.</td></tr><tr><td class="mono">warm</td><td>0 to 8</td><td class="mono">0</td><td>VMs kept booted for run (podman, krun).</td></tr><tr><td class="mono">warmup</td><td>shell lines</td><td class="mono"></td><td>Run once in each warm VM before it serves a run.</td></tr><tr><td class="mono">packages</td><td>names</td><td class="mono"></td><td>Installed with the image's package manager.</td></tr><tr><td class="mono">run</td><td>shell lines</td><td class="mono"></td><td>Extra build steps.</td></tr><tr><td class="mono">env</td><td>map</td><td class="mono"></td><td>Environment for every command.</td></tr><tr><td class="mono">mounts</td><td>host, guest, mode</td><td class="mono"></td><td>Host directories shared in (podman, krun); ro by default.</td></tr><tr><td class="mono">blueprint</td><td>users, write_files, runcmd</td><td class="mono"></td><td>Provisioning applied at build time.</td></tr><tr><td class="mono">pkgmgr</td><td>apk, apt, dnf</td><td class="mono">inferred</td><td>Override the package manager.</td></tr><tr><td class="mono">passt</td><td>bool</td><td class="mono">false</td><td>A real NIC with a default route (podman, krun).</td></tr><tr><td class="mono">seccomp</td><td>path</td><td class="mono"></td><td>A seccomp profile for the VMM.</td></tr></tbody>
+      <tbody><tr><td class="mono">base</td><td>image reference</td><td class="mono">alpine:latest</td><td>The image every VM starts from.</td></tr><tr><td class="mono">backend</td><td>podman, krun, firecracker</td><td class="mono">podman</td><td>What boots the VM. See Architecture.</td></tr><tr><td class="mono">isolation</td><td>standard, strict</td><td class="mono">standard</td><td>strict runs the VMM as a subordinate UID of the sandbox's own.</td></tr><tr><td class="mono">cpus</td><td>1 to 16</td><td class="mono">2</td><td>vCPUs.</td></tr><tr><td class="mono">ram_mib</td><td>128 and up</td><td class="mono">2048</td><td>Guest RAM in MiB.</td></tr><tr><td class="mono">network</td><td>bridge, none</td><td class="mono">bridge</td><td>Internet access, or none at all.</td></tr><tr><td class="mono">readonly</td><td>bool</td><td class="mono">false</td><td>Read-only root; /tmp and /data stay writable.</td></tr><tr><td class="mono">timeout_seconds</td><td>int</td><td class="mono">0</td><td>Per-command limit; exits 124. 0 is unlimited.</td></tr><tr><td class="mono">warm</td><td>0 to 8</td><td class="mono">0</td><td>VMs kept booted for run (podman, krun).</td></tr><tr><td class="mono">warmup</td><td>shell lines</td><td class="mono"></td><td>Run once in each warm VM before it serves a run.</td></tr><tr><td class="mono">packages</td><td>names</td><td class="mono"></td><td>Installed with the image's package manager.</td></tr><tr><td class="mono">run</td><td>shell lines</td><td class="mono"></td><td>Extra build steps.</td></tr><tr><td class="mono">env</td><td>map</td><td class="mono"></td><td>Environment for every command.</td></tr><tr><td class="mono">mounts</td><td>host, guest, mode</td><td class="mono"></td><td>Host directories shared in (podman, krun); ro by default.</td></tr><tr><td class="mono">blueprint</td><td>users, write_files, runcmd</td><td class="mono"></td><td>Provisioning applied at build time.</td></tr><tr><td class="mono">pkgmgr</td><td>apk, apt, dnf</td><td class="mono">inferred</td><td>Override the package manager.</td></tr><tr><td class="mono">passt</td><td>bool</td><td class="mono">false</td><td>A real NIC with a default route (podman, krun).</td></tr><tr><td class="mono">seccomp</td><td>path</td><td class="mono"></td><td>A seccomp profile for the VMM.</td></tr></tbody>
     </table></div>
 
     <h2 id="cli">CLI</h2>

docs/security.html

@@ -55,7 +55,7 @@ <h2 style="margin-bottom:1rem">Two layers</h2>
   <g class="tick" style="--i:0"><path class="s" d="M250,248 l6,7 l12,-14"/></g><g class="tick" style="--i:1"><path class="s" d="M302,248 l6,7 l12,-14"/></g><g class="tick" style="--i:2"><path class="s" d="M354,248 l6,7 l12,-14"/></g><g class="tick" style="--i:3"><path class="s" d="M406,248 l6,7 l12,-14"/></g><g class="tick" style="--i:4"><path class="s" d="M458,248 l6,7 l12,-14"/></g><g class="tick" style="--i:5"><path class="s" d="M510,248 l6,7 l12,-14"/></g>
 </svg>
       <div class="note prose">
-      <p><b>Use <code>isolation: strict</code> for agents.</b> Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to your first subordinate UID instead.</p>
+      <p><b>Use <code>isolation: strict</code> for agents.</b> Under standard isolation the VMM runs as your user, so escaping both the guest kernel and the VMM lands in your account. Strict maps it to a subordinate UID of the sandbox's own instead, which owns nothing of yours and nothing of any other sandbox's.</p>
       </div>
     </div>
   </div>
@@ -89,7 +89,6 @@ <h2 style="margin-bottom:1rem">What it is not</h2>
       <ul>
         <li><b>Not egress control, yet.</b> A sandbox with a network can send what it can read. Per-sandbox allowlists are planned.</li>
         <li><b>Not a secrets boundary.</b> Anything in <code>/data</code>, a mount or <code>env:</code> is the workload's. Credential brokering is planned.</li>
-        <li><b>Not a boundary between strict sandboxes.</b> They share one subordinate UID.</li>
         <li><b>Not a check on a lying guest.</b> The kernel comparison catches a runtime that fell back to a container, not a guest that fakes <code>uname</code>.</li>
         <li><b>Not multi-tenant.</b> box is one user's tool on their own machine.</li>
       </ul>

internal/cli/commands.go

@@ -358,6 +358,13 @@ func restoreCmd() *cobra.Command {
 			if err := sandbox.Restore(name, archive); err != nil {
 				return err
 			}
+			// The archive was unpacked from outside; give its files to the
+			// sandbox, so a strict sandbox's guest owns all of /data again.
+			if s, err := loadSpec(name); err == nil {
+				if err := runtime.ReownData(name, s); err != nil {
+					return err
+				}
+			}
 			runtime.SpawnTender(name)
 			fmt.Printf("restored %s from %s\n", name, filepath.Base(archive))
 			return nil
@@ -524,6 +531,12 @@ func destroyCmd() *cobra.Command {
 				if err := sandbox.RemoveFork(name); err != nil {
 					return err
 				}
+			} else if !withData {
+				// The sandbox's slot is freed with it; the kept /data must
+				// not stay owned by a UID the next strict sandbox may get.
+				if err := runtime.ReleaseData(name); err != nil {
+					return err
+				}
 			}
 			if err := sandbox.Remove(name, withData); err != nil {
 				return err
@@ -567,7 +580,13 @@ func nukeCmd() *cobra.Command {
 				runtime.Down(n)
 				runtime.RemovePool(n)
 				runtime.RemoveImage(n)
+				fork := sandbox.IsFork(n)
 				sandbox.RemoveFork(n)
+				if noData && !fork {
+					if err := runtime.ReleaseData(n); err != nil {
+						return fmt.Errorf("%s: %w", n, err)
+					}
+				}
 				if err := sandbox.Remove(n, !noData); err != nil {
 					return fmt.Errorf("%s: %w", n, err)
 				}

internal/runtime/backend_firecracker.go

@@ -72,7 +72,7 @@ type fcRun struct {
 	Disk      string            `json:"disk"`
 	CPUs      int               `json:"cpus"`
 	RAMMiB    int               `json:"ram_mib"`
-	Strict    bool              `json:"strict"`
+	Shift     int               `json:"shift"` // 0 standard; strict: the sandbox's slot in podman's namespace
 	ReadOnly  bool              `json:"readonly"`
 	Argv      []string          `json:"argv,omitempty"`
 	Stdin     bool              `json:"stdin,omitempty"`
@@ -96,6 +96,10 @@ func (b fcBackend) Launch(name string, s boxfile.Spec, l Launch) (*exec.Cmd, err
 	if err != nil {
 		return nil, err
 	}
+	shift, err := idShift(name, s)
+	if err != nil {
+		return nil, err
+	}
 	disk, err := fcDataDisk(name)
 	if err != nil {
 		return nil, err
@@ -106,7 +110,7 @@ func (b fcBackend) Launch(name string, s boxfile.Spec, l Launch) (*exec.Cmd, err
 	}
 	r := fcRun{
 		Mode: "run", Sandbox: name, VM: vm, Image: img, Disk: disk,
-		CPUs: s.CPUs, RAMMiB: s.RAMMiB, Strict: s.Isolation == "strict",
+		CPUs: s.CPUs, RAMMiB: s.RAMMiB, Shift: shift,
 		ReadOnly: s.ReadOnlyRootfs, Argv: l.Argv, Stdin: l.Interactive,
 		Labels: l.Labels, Spec: s,
 	}
@@ -324,14 +328,18 @@ func fcStart(dir string, r fcRun) error {
 	if err != nil {
 		return err
 	}
-	if r.Strict {
-		// The VMM runs as the subordinate UID (1 in this namespace). It owns
-		// the /data disk; the VM directory stays yours, shared with it through
-		// the group, so it can create its sockets there and nobody else can.
-		if err := os.Lchown(r.Disk, 1, 1); err != nil {
+	if r.Shift > 0 {
+		// The VMM runs as the sandbox's own subordinate UID (its shift in
+		// this namespace). It owns the /data disk; the VM directory stays
+		// yours, shared with it through the group, so it can create its
+		// sockets there and nobody else can -- no other sandbox either.
+		if err := os.Lchown(r.Disk, r.Shift, r.Shift); err != nil {
 			return err
 		}
-		if err := os.Lchown(dir, 0, 1); err != nil {
+		if err := os.Chmod(r.Disk, 0o600); err != nil {
+			return err
+		}
+		if err := os.Lchown(dir, 0, r.Shift); err != nil {
 			return err
 		}
 		if err := os.Chmod(dir, 0o770); err != nil {
@@ -460,13 +468,14 @@ func fcSpec(r fcRun, dir, jail, binDir, binName, kernelDir string) ([]byte, erro
 			Seccomp:       seccomp,
 		},
 	}
-	if r.Strict {
-		// The VMM runs as UID 1 of podman's namespace -- your first
-		// subordinate UID on the host -- rather than in a nested user
-		// namespace: crun sets the mounts up as that namespace's root, where
-		// every path is reachable whatever your home directory's mode, and
-		// only then drops to the VMM's identity, with no capabilities.
-		spec.Process.User = ociUser{UID: 1, GID: 1}
+	if r.Shift > 0 {
+		// The VMM runs as the sandbox's own UID in podman's namespace -- a
+		// subordinate UID on the host that no other sandbox has -- rather
+		// than in a nested user namespace: crun sets the mounts up as that
+		// namespace's root, where every path is reachable whatever your home
+		// directory's mode, and only then drops to the VMM's identity, with
+		// no capabilities.
+		spec.Process.User = ociUser{UID: uint32(r.Shift), GID: uint32(r.Shift)}
 	}
 	return json.MarshalIndent(spec, "", " ")
 }
@@ -593,12 +602,14 @@ func fcRestore(api fcClient, dir string, r fcRun, token string) error {
 		return fmt.Errorf("preparing the restored VM: exit %d: %s", res.Code, strings.TrimSpace(errb.String()))
 	}
 	// The VMM created the sockets as itself; under strict that is the
-	// subordinate UID. Hand them to you, owner and group only, so the CLI
-	// can connect and other users cannot.
-	for _, s := range []string{"v.sock", "api.sock"} {
-		p := filepath.Join(dir, s)
-		os.Lchown(p, 0, 1)
-		os.Chmod(p, 0o660)
+	// sandbox's subordinate UID. Hand them to you, shared with that UID
+	// through the group, so the CLI can connect and nobody else can.
+	if r.Shift > 0 {
+		for _, s := range []string{"v.sock", "api.sock"} {
+			p := filepath.Join(dir, s)
+			os.Lchown(p, 0, r.Shift)
+			os.Chmod(p, 0o660)
+		}
 	}
 	// Kept (owner-only) so teardown can reach the agent to flush /data,
 	// whoever tears the VM down.

internal/runtime/data.go

@@ -49,7 +49,12 @@ func ImportData(name string, s boxfile.Spec, r io.Reader) error {
 		return err
 	}
 	if mine, err := sandbox.OwnedByMe(data); err == nil && !mine {
-		return hostHelper(r, io.Discard, "__tarin", data)
+		// A strict sandbox's /data: unpack inside podman's namespace, then
+		// give what was unpacked to the sandbox, so its guest owns all of it.
+		if err := hostHelper(r, io.Discard, "__tarin", data); err != nil {
+			return err
+		}
+		return ReownData(name, s)
 	}
 	_, err = sandbox.SafeUntar(r, data)
 	return err

internal/runtime/doctor.go

@@ -80,8 +80,20 @@ func Doctor() []Check {
 		add(Check{Name: "subordinate UIDs", Warn: true,
 			Detail: "none for " + currentUser() + " in /etc/subuid: isolation: strict cannot run",
 			Fix:    "sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 " + currentUser()})
+	} else if n := slotCapacity(); n == 0 {
+		add(Check{Name: "subordinate UIDs", Warn: true,
+			Detail: ids + fmt.Sprintf(": fewer than %d IDs, so isolation: strict cannot run", slotSize),
+			Fix:    "sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 " + currentUser()})
 	} else {
-		add(Check{Name: "subordinate UIDs", OK: true, Detail: ids})
+		detail := ids + fmt.Sprintf(": room for %d strict sandbox(es), each with its own UIDs", n)
+		check := Check{Name: "subordinate UIDs", OK: true, Detail: detail}
+		if n < 4 {
+			start := freeSubordinateStart()
+			check = Check{Name: "subordinate UIDs", Warn: true, Detail: detail,
+				Fix: fmt.Sprintf("sudo usermod --add-subuids %d-%d --add-subgids %d-%d %s && podman system migrate",
+					start, start+64*slotSize-1, start, start+64*slotSize-1, currentUser())}
+		}
+		add(check)
 	}
 
 	if exe, err := os.Executable(); err == nil {

internal/runtime/fc_image.go

@@ -34,9 +34,9 @@ func fcImageDir(name string, s boxfile.Spec) (string, error) {
 	if err != nil {
 		return "", err
 	}
-	shift := 0
-	if s.Isolation == "strict" {
-		shift = 1
+	shift, err := idShift(name, s)
+	if err != nil {
+		return "", err
 	}
 	// The agent is baked into the root disk and the snapshot, so a new
 	// box means a new image; the key includes its hash.
@@ -202,6 +202,10 @@ func fcDataDisk(name string) (string, error) {
 // fcSnapshot boots the image once, waits for its agent, pauses it and
 // snapshots it into dir.
 func fcSnapshot(name string, s boxfile.Spec, dir string) error {
+	shift, err := idShift(name, s)
+	if err != nil {
+		return err
+	}
 	token, err := newToken()
 	if err != nil {
 		return err
@@ -221,7 +225,7 @@ func fcSnapshot(name string, s boxfile.Spec, dir string) error {
 	vm := "box-snap-" + name
 	cmd, err := fcLaunchCmd(fcRun{
 		Mode: "snapshot", Sandbox: name, VM: vm, Image: dir, Disk: placeholder,
-		CPUs: s.CPUs, RAMMiB: s.RAMMiB, Strict: s.Isolation == "strict",
+		CPUs: s.CPUs, RAMMiB: s.RAMMiB, Shift: shift,
 		ReadOnly: s.ReadOnlyRootfs, BootToken: token, Spec: s,
 	})
 	if err != nil {