diff --git a/README.md b/README.md index ca0baae..93e9b21 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ backend: auto # firecracker when it fits, else podman; or name one isolation: strict # the VMM runs as a UID that is not yours network: bridge # or none, for no network at all warm: 2 # keep VMs booted so runs start in ms +data_size: 10G # cap /data, so an agent cannot fill your disk packages: [python3, git] ``` diff --git a/SECURITY.md b/SECURITY.md index c6ff077..fdebee8 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,6 +28,7 @@ as it can: - 6 capabilities, the ones virtiofs and low ports need; the other 5 podman grants by default are dropped - a pids limit and a memory limit (`ram_mib` + 256 MiB) +- `/data` capped by `data_size` (see below), so an agent cannot fill your disk - only the directories the Boxfile declares, `/data` and any `mounts:` - with `isolation: strict`, a subordinate host UID of its own, which owns nothing of yours and nothing of any other sandbox's @@ -46,6 +47,20 @@ rather than let two sandboxes share a UID. `box doctor` shows how many strict sandboxes your range holds. A destroyed sandbox's kept `/data` is handed back to you, so a later sandbox given the same slot does not own it. +## /data's size + +`data_size` in the Boxfile caps `/data` (new Boxfiles say 10G). On +firecracker `/data` is a disk of exactly that size, so the guest's own kernel +refuses a write that does not fit. On podman and krun `/data` is a host +directory shared over virtiofs, and a rootless process cannot put a quota on +it, so box measures it: it refuses to start a sandbox that is over, and +checks every 2 seconds while one runs -- a cold run, an `exec`, a warm run, or +a process left running in an `up` sandbox -- stopping the VM when it goes +over. A fast writer can overshoot by what it writes in those 2 seconds; the +limit is what the sandbox must get back under, not a wall. A Boxfile without +`data_size` has no limit on podman and krun (8G on firecracker), and +`box inspect` warns about it. The escape suite fills `/data` and checks both. + ## Checking it yourself `box inspect --security-report` (or `=json`) states what a sandbox's diff --git a/docs/docs.html b/docs/docs.html index e896d5d..885f9a3 100644 --- a/docs/docs.html +++ b/docs/docs.html @@ -77,7 +77,7 @@

Boxfile

One YAML file per sandbox at ~/.box/sandboxes/<name>/Boxfile. Unknown keys are rejected, so a typo fails loudly.

- +
FieldValuesDefaultMeaning
baseimage referencealpine:latestThe image every VM starts from.
backendauto, podman, krun, firecrackerpodman (new Boxfiles: auto)What boots the VM. auto picks firecracker when the sandbox is offline, has no host mounts, no warm pool and no forks, and the host can run it; podman otherwise. The pick is made once and shown by box inspect. See Architecture.
isolationstandard, strictstandardstrict runs the VMM as a subordinate UID of the sandbox's own.
cpus1 to 162vCPUs.
ram_mib128 and up2048Guest RAM in MiB.
networkbridge, nonebridgeInternet access, or none at all.
readonlyboolfalseRead-only root; /tmp and /data stay writable.
timeout_secondsint0Per-command limit; exits 124. 0 is unlimited.
warm0 to 80VMs kept booted for run (podman, krun).
warmupshell linesRun once in each warm VM before it serves a run.
packagesnamesInstalled with the image's package manager.
runshell linesExtra build steps.
envmapEnvironment for every command.
mountshost, guest, modeHost directories shared in (podman, krun); ro by default.
blueprintusers, write_files, runcmdProvisioning applied at build time.
pkgmgrapk, apt, dnfinferredOverride the package manager.
passtboolfalseA real NIC with a default route (podman, krun).
seccomppathA seccomp profile for the VMM.
baseimage referencealpine:latestThe image every VM starts from.
backendauto, podman, krun, firecrackerpodman (new Boxfiles: auto)What boots the VM. auto picks firecracker when the sandbox is offline, has no host mounts, no warm pool and no forks, and the host can run it; podman otherwise. The pick is made once and shown by box inspect. See Architecture.
isolationstandard, strictstandardstrict runs the VMM as a subordinate UID of the sandbox's own.
cpus1 to 162vCPUs.
ram_mib128 and up2048Guest RAM in MiB.
networkbridge, nonebridgeInternet access, or none at all.
readonlyboolfalseRead-only root; /tmp and /data stay writable.
timeout_secondsint0Per-command limit; exits 124. 0 is unlimited.
warm0 to 80VMs kept booted for run (podman, krun).
data_sizee.g. 512M, 10Gnone (8G on firecracker); new Boxfiles: 10GCaps /data. Exact on firecracker, where /data is a disk of this size. On podman and krun box measures it: it will not start a sandbox that is over, and stops a running one within 2 s of going over.
warmupshell linesRun once in each warm VM before it serves a run.
packagesnamesInstalled with the image's package manager.
runshell linesExtra build steps.
envmapEnvironment for every command.
mountshost, guest, modeHost directories shared in (podman, krun); ro by default.
blueprintusers, write_files, runcmdProvisioning applied at build time.
pkgmgrapk, apt, dnfinferredOverride the package manager.
passtboolfalseA real NIC with a default route (podman, krun).
seccomppathA seccomp profile for the VMM.

CLI

diff --git a/docs/index.html b/docs/index.html index c4d8fbf..879fe6b 100644 --- a/docs/index.html +++ b/docs/index.html @@ -229,7 +229,7 @@

Not sure it fits? Ask your AI.

-
you5.5 KB · plain text
+
you5.7 KB · plain text
I'm evaluating box (https://github.com/chxperiments/box) for something I'm building. Help me decide whether it fits, and if it does, how to set it up correctly. Use only the facts below about box; if something isn't covered here, say so instead of guessing.
 
 ## What box is
@@ -251,7 +251,7 @@ 
Not sure it fits? Ask your AI.

 Measured: a warm `run` is ~20–45 ms; `exec` in a running sandbox is ~15 ms for a no-op.
 
 ## Boxfile reference (unknown keys are rejected)
-base (image ref), backend, cpus (max 16), ram_mib, network (bridge = internet, none = offline), readonly (read-only root; /tmp and /data stay writable), isolation (standard | strict), timeout_seconds (per run; a timed-out run exits 124), warm (0–8, needs network: bridge), warmup (commands run in each pooled VM first), packages (apk, apt or dnf, chosen from the base), run (build steps), env, mounts (host dir → guest path, ro by default; strict allows ro only), blueprint (users, write_files, runcmd, applied at build time).
+base (image ref), backend, cpus (max 16), ram_mib, network (bridge = internet, none = offline), data_size (caps /data: exact on firecracker, where /data is a disk; on podman/krun box measures it every 2 s and stops a VM that goes over), readonly (read-only root; /tmp and /data stay writable), isolation (standard | strict), timeout_seconds (per run; a timed-out run exits 124), warm (0–8, needs network: bridge), warmup (commands run in each pooled VM first), packages (apk, apt or dnf, chosen from the base), run (build steps), env, mounts (host dir → guest path, ro by default; strict allows ro only), blueprint (users, write_files, runcmd, applied at build time).
 
 ## Driving it
 - CLI: new [--from example], build, run, shell, up, exec, down, fork, diff, apply, discard, snapshot, restore, reset, data import/export, doctor (checks the host and prints fixes), verify, logs, destroy.
@@ -274,8 +274,8 @@ 
Not sure it fits? Ask your AI.
- Ask Claude - Ask ChatGPT + Ask Claude + Ask ChatGPT
diff --git a/internal/boxfile/boxfile.go b/internal/boxfile/boxfile.go index 8bfa735..50b60b6 100644 --- a/internal/boxfile/boxfile.go +++ b/internal/boxfile/boxfile.go @@ -46,6 +46,7 @@ type Spec struct { ReadOnlyRootfs bool `yaml:"readonly"` TimeoutSeconds int `yaml:"timeout_seconds"` // 0 = unlimited Warm int `yaml:"warm"` // VMs kept booted for run; 0 = boot per run + DataSize string `yaml:"data_size"` // /data's limit, e.g. 10G; empty = none (8G disk on firecracker) Isolation string `yaml:"isolation"` // "standard" or "strict": VMM under its own host UID Backend string `yaml:"backend"` // what boots the VM: podman (default), krun, firecracker Seccomp string `yaml:"seccomp"` // profile path, filters the VMM @@ -132,6 +133,41 @@ func FirecrackerFits(s Spec) error { return nil } +// MinDataSize is the smallest data_size: below it ext4 has little room left. +const MinDataSize = 64 << 20 + +// DataBytes is data_size in bytes, or 0 when the Boxfile sets none. +// Sizes are 1024-based: 512M, 10G, 1T; a bare number is bytes. +func (s Spec) DataBytes() (int64, error) { + v := strings.TrimSpace(s.DataSize) + if v == "" { + return 0, nil + } + mult := int64(1) + switch strings.ToUpper(v[len(v)-1:]) { + case "K": + mult = 1 << 10 + case "M": + mult = 1 << 20 + case "G": + mult = 1 << 30 + case "T": + mult = 1 << 40 + } + num := v + if mult > 1 { + num = v[:len(v)-1] + } + n, err := strconv.ParseInt(strings.TrimSpace(num), 10, 64) + if err != nil || n <= 0 || n > (1<<62)/mult { + return 0, fmt.Errorf("data_size must be a size like 512M, 10G or 1T, got %q", s.DataSize) + } + if n*mult < MinDataSize { + return 0, fmt.Errorf("data_size must be at least 64M, got %q", s.DataSize) + } + return n * mult, nil +} + // BackendNames lists the accepted backends for error messages. func BackendNames() string { return strings.Join(Backends, ", ") } @@ -150,6 +186,7 @@ isolation: standard # strict: VMM runs as a UID that is not yours (use for age backend: auto # auto: firecracker when it fits (offline, no mounts, no warm pool), else podman | podman | krun | firecracker timeout_seconds: 0 # per-run wall clock limit, 0 = unlimited warm: 0 # VMs kept booted so run starts in ms (each holds its RAM) +data_size: 10G # /data's limit: exact on firecracker, checked every 2 s on podman # Tools baked into the image. Reset every run; keep work in /data. packages: @@ -245,6 +282,9 @@ func (s Spec) validate() error { return err } } + if _, err := s.DataBytes(); err != nil { + return err + } if s.Isolation != "standard" && s.Isolation != "strict" { return fmt.Errorf("isolation must be standard or strict, got %q", s.Isolation) } diff --git a/internal/boxfile/datasize_test.go b/internal/boxfile/datasize_test.go new file mode 100644 index 0000000..c3681fd --- /dev/null +++ b/internal/boxfile/datasize_test.go @@ -0,0 +1,20 @@ +package boxfile + +import "testing" + +func TestDataBytes(t *testing.T) { + for in, want := range map[string]int64{ + "": 0, "64M": 64 << 20, "512M": 512 << 20, "10G": 10 << 30, "10g": 10 << 30, "1T": 1 << 40, + " 2G ": 2 << 30, "67108864": 64 << 20, + } { + got, err := Spec{DataSize: in}.DataBytes() + if err != nil || got != want { + t.Errorf("%q: %d, %v; want %d", in, got, err, want) + } + } + for _, bad := range []string{"10X", "G", "-1G", "0", "1M", "63M", "ten", "99999999999T"} { + if _, err := (Spec{DataSize: bad}).DataBytes(); err == nil { + t.Errorf("%q accepted", bad) + } + } +} diff --git a/internal/cli/inspect.go b/internal/cli/inspect.go index fde9ca0..c72bbdf 100644 --- a/internal/cli/inspect.go +++ b/internal/cli/inspect.go @@ -97,6 +97,12 @@ func printReport(r runtime.Report) { data += ", owner " + d.Owner + ", mode " + d.Mode } row("/data", data+"), kept between runs") + limit := d.Limit + if d.Used != "" { + limit += ", " + d.Used + " used" + } + row("/data limit", limit) + row("enforced", d.Enforcement) if len(r.Filesystem.Mounts) == 0 { row("host mounts", "none") } diff --git a/internal/cli/root.go b/internal/cli/root.go index 82bf80e..14c9ca5 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -65,7 +65,7 @@ func newRoot() *cobra.Command { dataCmd(), envCmd(), logsCmd(), verifyCmd(), inspectCmd(), doctorCmd(), renameCmd(), destroyCmd(), nukeCmd(), - agentCmd(), prepareCmd(), tendCmd(), overlayCmd(), krunCmd(), exportCmd(), netnsCmd(), fcCmd(), tarOutCmd(), tarInCmd(), + agentCmd(), prepareCmd(), tendCmd(), watchDataCmd(), overlayCmd(), krunCmd(), exportCmd(), netnsCmd(), fcCmd(), tarOutCmd(), tarInCmd(), ) return root } diff --git a/internal/cli/up.go b/internal/cli/up.go index 9e984f7..7529aa5 100644 --- a/internal/cli/up.go +++ b/internal/cli/up.go @@ -312,6 +312,17 @@ func netnsCmd() *cobra.Command { // tendCmd tops up a sandbox's warm pool. It is started detached by run and // build, never by hand, and reports failures to the sandbox's log. +func watchDataCmd() *cobra.Command { + return &cobra.Command{ + Use: "__watch-data ", Hidden: true, + Args: cobra.ExactArgs(1), + ValidArgsFunction: completeNothing, + RunE: func(_ *cobra.Command, args []string) error { + return runtime.WatchData(args[0]) + }, + } +} + func tendCmd() *cobra.Command { return &cobra.Command{ Use: "__tend ", Hidden: true, diff --git a/internal/runtime/backend_firecracker.go b/internal/runtime/backend_firecracker.go index 9212e3a..a9e1a94 100644 --- a/internal/runtime/backend_firecracker.go +++ b/internal/runtime/backend_firecracker.go @@ -100,7 +100,7 @@ func (b fcBackend) Launch(name string, s boxfile.Spec, l Launch) (*exec.Cmd, err if err != nil { return nil, err } - disk, err := fcDataDisk(name) + disk, err := fcDataDisk(name, fcDataBytes(s)) if err != nil { return nil, err } diff --git a/internal/runtime/fc_assets.go b/internal/runtime/fc_assets.go index 450ac9b..550f02c 100644 --- a/internal/runtime/fc_assets.go +++ b/internal/runtime/fc_assets.go @@ -27,8 +27,8 @@ const ( fcKernelName = "vmlinux-6.1.155" fcKernelURL = "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.15/x86_64/" + fcKernelName fcKernelSHA = "e20e46d0c36c55c0d1014eb20576171b3f3d922260d9f792017aeff53af3d4f2" - fcToolsImage = "localhost/box/fc-tools:latest" - fcDataDiskGiB = 8 // sparse: only what /data actually holds is stored + fcToolsImage = "localhost/box/fc-tools:2" // 2: adds resize2fs; a new tag so an old build is not reused + fcDataDiskGiB = 8 // sparse: only what /data actually holds is stored ) // fcDir holds Firecracker's assets and every image's disk and snapshot. @@ -154,7 +154,7 @@ func fcTools() error { } cmd := exec.Command("podman", "build", "-q", "-t", fcToolsImage, "-f", "-", ".") cmd.Dir = os.TempDir() - cmd.Stdin = strings.NewReader("FROM docker.io/library/alpine:latest\nRUN apk add --no-cache e2fsprogs tar\n") + cmd.Stdin = strings.NewReader("FROM docker.io/library/alpine:latest\nRUN apk add --no-cache e2fsprogs e2fsprogs-extra tar\n") if out, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("building %s: %s", fcToolsImage, strings.TrimSpace(string(out))) } diff --git a/internal/runtime/fc_image.go b/internal/runtime/fc_image.go index 9632bf2..c289c46 100644 --- a/internal/runtime/fc_image.go +++ b/internal/runtime/fc_image.go @@ -8,6 +8,7 @@ import ( "os" "os/exec" "path/filepath" + "strconv" "strings" "box/internal/boxfile" @@ -45,7 +46,7 @@ func fcImageDir(name string, s boxfile.Spec) (string, error) { return "", err } return filepath.Join(dir, "images", name, - fmt.Sprintf("%s-a%s-s%d-r%d-c%d-%s", id[:12], agentHash, shift, s.RAMMiB, s.CPUs, roFlag(s))), nil + fmt.Sprintf("%s-a%s-s%d-r%d-c%d-d%d-%s", id[:12], agentHash, shift, s.RAMMiB, s.CPUs, fcDataBytes(s)>>20, roFlag(s))), nil } func roFlag(s boxfile.Spec) string { @@ -100,9 +101,33 @@ func fcImage(name string, s boxfile.Spec) (string, error) { if err := fcSnapshot(name, s, dir); err != nil { return "", err } + removeStaleImages(dir) return dir, nil } +// removeStaleImages deletes a sandbox's other images. Each is a rootfs and a +// full memory snapshot (its RAM in size), keyed on everything that shapes it +// -- the image, box's own build, isolation, RAM, CPUs, /data's size -- so +// every change, a box upgrade included, makes a new one; without this the +// old ones stayed forever. A VM still restored from one keeps its files +// open, and Linux keeps them until it exits. +func removeStaleImages(current string) { + parent := filepath.Dir(current) + entries, err := os.ReadDir(parent) + if err != nil { + return + } + var stale []string + for _, e := range entries { + if e.IsDir() && e.Name() != filepath.Base(current) { + stale = append(stale, filepath.Join(parent, e.Name())) + } + } + if len(stale) > 0 { + inPodmanNS(append([]string{"rm", "-rf", "--"}, stale...)...).Run() + } +} + // fcRootfs writes rootfs.ext4: the image's files exported by podman and // written into an ext4 filesystem by the helper image, with the agent binary // and the image's environment added under /.box. @@ -203,13 +228,40 @@ func FirecrackerDataExists(name string) bool { return err == nil } -func fcDataDisk(name string) (string, error) { +// fcDataBytes is the size of a sandbox's /data disk: data_size, or 8 GiB. +// The disk is sparse, so only what /data holds takes space. +func fcDataBytes(s boxfile.Spec) int64 { + if n, err := s.DataBytes(); err == nil && n > 0 { + return n + } + return int64(fcDataDiskGiB) << 30 +} + +// fcDataDisk returns the sandbox's /data disk, created at size the first +// time. The disk is data_size's limit: the guest's kernel refuses a write +// that does not fit. A disk smaller than size is grown, offline, keeping +// its files; one larger is refused, since shrinking a filesystem in place +// is how data is lost. +func fcDataDisk(name string, size int64) (string, error) { disk, err := fcDataDiskPath(name) if err != nil { return "", err } - if _, err := os.Stat(disk); err == nil { - return disk, nil + if fi, err := os.Stat(disk); err == nil { + switch { + case fi.Size() == size: + return disk, nil + case fi.Size() > size: + return "", fmt.Errorf("%s's /data disk is %s, larger than its data_size %s, and a disk cannot shrink in place: "+ + "box data export %s ./%s-data && box reset %s -y, then box data import %s ./%s-data", + name, humanBytes(fi.Size()), humanBytes(size), name, name, name, name, name) + } + // Never resize a disk a running VM has mounted. + if holder, err := os.ReadFile(diskLockPath(name)); err == nil && fcAlive(string(holder)) { + return "", fmt.Errorf("%s's /data disk is in use by %s; bring it down to grow /data to its new data_size", + name, strings.TrimSpace(string(holder))) + } + return disk, growDisk(disk, size) } if err := os.MkdirAll(filepath.Dir(disk), 0o700); err != nil { return "", err @@ -220,7 +272,7 @@ func fcDataDisk(name string) (string, error) { tmp := name + ".ext4.partial" cmd := exec.Command("podman", "run", "--rm", "--pull=never", "--network=none", "-v", filepath.Dir(disk)+":/out", fcToolsImage, - "sh", "-c", fmt.Sprintf("truncate -s %dG /out/%s && mkfs.ext4 -q -F -L data /out/%s", fcDataDiskGiB, tmp, tmp)) + "sh", "-c", fmt.Sprintf("truncate -s %d /out/%s && mkfs.ext4 -q -F -L data /out/%s", size, tmp, tmp)) if out, err := cmd.CombinedOutput(); err != nil { return "", fmt.Errorf("creating %s's /data disk: %s", name, strings.TrimSpace(string(out))) } @@ -230,6 +282,36 @@ func fcDataDisk(name string) (string, error) { return disk, nil } +// growDisk enlarges an unmounted ext4 disk and its filesystem. A strict +// sandbox's disk belongs to its UID; it is given back to you for the resize +// and handed to the sandbox again at its next start. +func growDisk(disk string, size int64) error { + if err := fcTools(); err != nil { + return err + } + if out, err := exec.Command("podman", "unshare", "chown", "0:0", disk).CombinedOutput(); err != nil { + return fmt.Errorf("growing %s: %s", disk, strings.TrimSpace(string(out))) + } + // Check for the tools first, and put the size back if the resize fails: + // a grow either happens whole or leaves the disk as it was. + cmd := exec.Command("podman", "run", "--rm", "--pull=never", "--network=none", + "-v", filepath.Dir(disk)+":/out", fcToolsImage, "sh", "-c", ` +d=/out/$1 +command -v resize2fs >/dev/null || { echo "resize2fs is missing from the tools image"; exit 1; } +orig=$(stat -c %s "$d") +truncate -s "$2" "$d" || exit 1 +e2fsck -fy "$d" >/dev/null 2>&1 +if [ $? -gt 1 ] || ! resize2fs "$d" >/dev/null 2>&1; then + truncate -s "$orig" "$d" + echo "the filesystem could not be grown; the disk is unchanged" + exit 1 +fi`, "sh", filepath.Base(disk), strconv.FormatInt(size, 10)) + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("growing %s to %s: %s", disk, humanBytes(size), strings.TrimSpace(string(out))) + } + return nil +} + // fcSnapshot boots the image once, waits for its agent, pauses it and // snapshots it into dir. func fcSnapshot(name string, s boxfile.Spec, dir string) error { @@ -249,7 +331,7 @@ func fcSnapshot(name string, s boxfile.Spec, dir string) error { // a restored copy is prepared. placeholder := filepath.Join(dir, "placeholder.ext4") if f, err := os.Create(placeholder); err == nil { - f.Truncate(int64(fcDataDiskGiB) << 30) + f.Truncate(fcDataBytes(s)) // the snapshot's disk must be the size every restore attaches f.Close() } defer os.Remove(placeholder) diff --git a/internal/runtime/persistent.go b/internal/runtime/persistent.go index fdc0ca2..0710bc4 100644 --- a/internal/runtime/persistent.go +++ b/internal/runtime/persistent.go @@ -269,7 +269,10 @@ func Exec(name string, s boxfile.Spec, argv []string, streams Streams) error { if err != nil { return err } - err = execOn(name, st, s, "exec", argv, streams) + if err := overQuota(name, s); err != nil { + return err + } + err = execOn(name, st, s, "exec", argv, streams, func() { Down(name) }) if errors.Is(err, errLost) { return fmt.Errorf("%s is not responding (%v); restart it: box down %s && box up %s", name, err, name, name) @@ -282,7 +285,7 @@ var errLost = errors.New("agent unreachable") // execOn runs argv through the agent of a running VM, with output streamed // and logged the way Run does, and every command gated on the guest kernel. -func execOn(name string, st upState, s boxfile.Spec, verb string, argv []string, streams Streams) error { +func execOn(name string, st upState, s boxfile.Spec, verb string, argv []string, streams Streams, stop func()) error { host, err := HostKernel() if err != nil { return err @@ -296,11 +299,20 @@ func execOn(name string, st upState, s boxfile.Spec, verb string, argv []string, out, errw := tee(streams, log) started := time.Now() + done := make(chan struct{}) + exceeded := watchQuota(name, s, done, stop) res, err := agent.Exec(st.Addr, agent.Request{ Token: st.Token, Argv: argv, TimeoutSeconds: s.TimeoutSeconds, }, kernelGate(host), streams.Stdin, out, errw) + close(done) + if qerr := exceeded(); qerr != nil { + if log != nil { + fmt.Fprintf(log, "=== stopped: %v\n", qerr) + } + return fmt.Errorf("%w; box stopped the VM", qerr) + } if log != nil { code := res.Code if err != nil { diff --git a/internal/runtime/pool.go b/internal/runtime/pool.go index b2755e7..f3e7f00 100644 --- a/internal/runtime/pool.go +++ b/internal/runtime/pool.go @@ -130,7 +130,15 @@ func RunWarm(name string, s boxfile.Spec, argv []string, streams Streams) (bool, return false, nil } streams.Stdin = nil // run never forwards stdin, warm or cold - err := execOn(name, e.upState, s, "run", argv, streams) + if qerr := overQuota(name, s); qerr != nil { + os.Rename(claimed, stalePath(claimed)) + return true, qerr + } + err := execOn(name, e.upState, s, "run", argv, streams, func() { + if b, err := backendFor(s); err == nil { + b.Remove(e.Container) + } + }) // One run per VM, whatever happened in it: that is what makes it fresh. os.Rename(claimed, stalePath(claimed)) if errors.Is(err, errLost) { diff --git a/internal/runtime/quota.go b/internal/runtime/quota.go new file mode 100644 index 0000000..a03f05a --- /dev/null +++ b/internal/runtime/quota.go @@ -0,0 +1,225 @@ +package runtime + +import ( + "errors" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "syscall" + "time" + + "box/internal/boxfile" + "box/internal/sandbox" +) + +// data_size limits /data. On firecracker /data is a disk of exactly that +// size, so the guest's own kernel stops a write that would not fit. On +// podman and krun /data is a host directory shared over virtiofs, and a +// rootless process has no way to put a quota on it; box measures it instead +// -- before a VM starts, and every quotaInterval while one runs -- and stops +// the VM when it goes over. A fast writer can overshoot by what it writes in +// one interval; the limit is what it settles back under, not a wall. + +var quotaInterval = 2 * time.Second // a var so tests can shorten it + +// ErrDataFull is returned when a run is stopped for going over data_size. +var ErrDataFull = errors.New("/data went over its data_size") + +// quotaApplies reports whether box enforces data_size by measuring: a +// directory-backed /data with a limit set. +func quotaApplies(s boxfile.Spec) (int64, bool) { + if s.Backend == "firecracker" { + return 0, false // the disk is the limit + } + limit, err := s.DataBytes() + return limit, err == nil && limit > 0 +} + +// dataUsage is the space a sandbox's /data takes on the host. A fork is its +// parent's data plus its own layer: counted in full, which errs high. +func dataUsage(name string) (int64, error) { + dirs := []string{} + if parent, err := sandbox.Parent(name); err == nil { + d, err := sandbox.DataDir(parent) + if err != nil { + return 0, err + } + u, err := sandbox.UpperDir(name) + if err != nil { + return 0, err + } + dirs = append(dirs, d, u) + } else { + d, err := sandbox.DataDir(name) + if err != nil { + return 0, err + } + dirs = append(dirs, d) + } + var total int64 + for _, d := range dirs { + n, err := dirUsage(d) + if err != nil { + return 0, err + } + total += n + } + return total, nil +} + +// dirUsage sums the blocks a tree holds, hard links once. A strict +// sandbox's tree belongs to its own UID and may hold directories you cannot +// enter, so it is measured from podman's namespace, where every file is. +func dirUsage(dir string) (int64, error) { + if _, err := os.Stat(dir); errors.Is(err, fs.ErrNotExist) { + return 0, nil + } + if mine, err := sandbox.OwnedByMe(dir); err == nil && !mine { + out, err := exec.Command("podman", "unshare", "du", "-s", "-B1", "--", dir).Output() + if err != nil { + return 0, fmt.Errorf("measuring %s: %v", dir, err) + } + f := strings.Fields(string(out)) + if len(f) == 0 { + return 0, fmt.Errorf("measuring %s: no output from du", dir) + } + return strconv.ParseInt(f[0], 10, 64) + } + seen := map[[2]uint64]bool{} + var total int64 + err := filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + fi, err := d.Info() + if err != nil { + return nil // removed while walking + } + st, ok := fi.Sys().(*syscall.Stat_t) + if !ok { + total += fi.Size() + return nil + } + if st.Nlink > 1 && !fi.IsDir() { + k := [2]uint64{uint64(st.Dev), uint64(st.Ino)} + if seen[k] { + return nil + } + seen[k] = true + } + total += st.Blocks * 512 + return nil + }) + return total, err +} + +// overQuota says how a sandbox stands against data_size, or nil when it is +// within it or has none. +func overQuota(name string, s boxfile.Spec) error { + limit, ok := quotaApplies(s) + if !ok { + return nil + } + used, err := dataUsage(name) + if err != nil { + return err + } + if used > limit { + return fmt.Errorf("%w: it holds %s, its data_size is %s. Free space (box data export, then box reset) or raise data_size", + ErrDataFull, humanBytes(used), s.DataSize) + } + return nil +} + +// watchQuota measures /data every quotaInterval until done is closed, and +// calls stop once if it goes over. It returns a function that reports +// whether it did. +func watchQuota(name string, s boxfile.Spec, done <-chan struct{}, stop func()) func() error { + if _, ok := quotaApplies(s); !ok { + return func() error { return nil } + } + result := make(chan error, 1) + go func() { + t := time.NewTicker(quotaInterval) + defer t.Stop() + for { + select { + case <-done: + result <- nil + return + case <-t.C: + if err := overQuota(name, s); errors.Is(err, ErrDataFull) { + stop() + result <- err + return + } + } + } + }() + var got error + var read bool + return func() error { + if !read { + select { + case got = <-result: + case <-time.After(quotaInterval + time.Second): + } + read = true + } + return got + } +} + +func humanBytes(n int64) string { + switch { + case n >= 1<<30: + return fmt.Sprintf("%.1fG", float64(n)/(1<<30)) + case n >= 1<<20: + return fmt.Sprintf("%.1fM", float64(n)/(1<<20)) + case n >= 1<<10: + return fmt.Sprintf("%.1fK", float64(n)/(1<<10)) + } + return fmt.Sprintf("%dB", n) +} + +// spawnDataWatch starts a detached `box __watch-data ` for an up +// sandbox: between execs, a process left running in it can still write. +func spawnDataWatch(name string, s boxfile.Spec) { + if _, ok := quotaApplies(s); !ok { + return + } + exe, err := os.Executable() + if err != nil { + return + } + cmd := exec.Command(exe, "__watch-data", name) + cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true} + if cmd.Start() == nil { + cmd.Process.Release() + } +} + +// WatchData is `box __watch-data`: while the sandbox is up, measure /data +// every quotaInterval and bring the sandbox down if it goes over. +func WatchData(name string) error { + for sandbox.IsUp(name) { + time.Sleep(quotaInterval) + s, err := parseSpec(name) + if err != nil { + return err + } + if err := overQuota(name, s); errors.Is(err, ErrDataFull) { + Down(name) + if log := openLog(name); log != nil { + fmt.Fprintf(log, "\n=== %s data: brought down: %v\n", time.Now().UTC().Format(time.RFC3339), err) + log.Close() + } + return nil + } + } + return nil +} diff --git a/internal/runtime/quota_test.go b/internal/runtime/quota_test.go new file mode 100644 index 0000000..897f4e7 --- /dev/null +++ b/internal/runtime/quota_test.go @@ -0,0 +1,98 @@ +package runtime + +import ( + "errors" + "os" + "path/filepath" + "testing" + "time" + + "box/internal/boxfile" + "box/internal/sandbox" +) + +func writeBytes(t *testing.T, p string, n int) { + t.Helper() + if err := os.WriteFile(p, make([]byte, n), 0o644); err != nil { + t.Fatal(err) + } +} + +func TestDirUsageCountsBlocksAndHardLinksOnce(t *testing.T) { + d := t.TempDir() + writeBytes(t, filepath.Join(d, "a"), 1<<20) + if err := os.Link(filepath.Join(d, "a"), filepath.Join(d, "b")); err != nil { + t.Fatal(err) + } + // A sparse file holds almost nothing, whatever its size says. + f, _ := os.Create(filepath.Join(d, "sparse")) + f.Truncate(1 << 30) + f.Close() + got, err := dirUsage(d) + if err != nil { + t.Fatal(err) + } + if got < 1<<20 || got > 1<<20+64<<10 { + t.Errorf("usage %d, want about 1 MiB", got) + } +} + +func quotaSandbox(t *testing.T, size string) boxfile.Spec { + t.Helper() + t.Setenv("BOX_HOME", t.TempDir()) + newSandbox(t, "q") + return boxfile.Spec{Backend: "podman", DataSize: size} +} + +func TestOverQuotaRefusesAFullSandbox(t *testing.T) { + s := quotaSandbox(t, "64M") + data, _ := sandbox.DataDir("q") + writeBytes(t, filepath.Join(data, "small"), 1<<20) + if err := overQuota("q", s); err != nil { + t.Fatalf("1 MiB of 64: %v", err) + } + writeBytes(t, filepath.Join(data, "big"), 65<<20) + if err := overQuota("q", s); !errors.Is(err, ErrDataFull) { + t.Fatalf("66 MiB of 64: %v", err) + } + // No data_size, no limit; firecracker's limit is its disk, not this. + if err := overQuota("q", boxfile.Spec{Backend: "podman"}); err != nil { + t.Errorf("no data_size: %v", err) + } + if err := overQuota("q", boxfile.Spec{Backend: "firecracker", DataSize: "64M"}); err != nil { + t.Errorf("firecracker: %v", err) + } +} + +func TestWatchQuotaStopsARunThatGoesOver(t *testing.T) { + old := quotaInterval + quotaInterval = 50 * time.Millisecond + defer func() { quotaInterval = old }() + s := quotaSandbox(t, "64M") + data, _ := sandbox.DataDir("q") + + stopped := make(chan struct{}, 1) + done := make(chan struct{}) + exceeded := watchQuota("q", s, done, func() { stopped <- struct{}{} }) + writeBytes(t, filepath.Join(data, "big"), 65<<20) // the run writes too much + select { + case <-stopped: + case <-time.After(5 * time.Second): + t.Fatal("the watcher never stopped the run") + } + close(done) + if err := exceeded(); !errors.Is(err, ErrDataFull) { + t.Errorf("result: %v", err) + } + + // Under the limit, it never stops anything. + os.Remove(filepath.Join(data, "big")) + done2 := make(chan struct{}) + calls := 0 + ex2 := watchQuota("q", s, done2, func() { calls++ }) + time.Sleep(200 * time.Millisecond) + close(done2) + if err := ex2(); err != nil || calls != 0 { + t.Errorf("under the limit: %v, %d stops", err, calls) + } +} diff --git a/internal/runtime/report.go b/internal/runtime/report.go index 6b51f50..d508136 100644 --- a/internal/runtime/report.go +++ b/internal/runtime/report.go @@ -68,10 +68,14 @@ type FSReport struct { // DataReport is the one thing that persists between runs. type DataReport struct { - Host string `json:"host"` - Kind string `json:"kind"` // directory | disk - Owner string `json:"owner,omitempty"` - Mode string `json:"mode,omitempty"` + Host string `json:"host"` + Kind string `json:"kind"` // directory | disk + Owner string `json:"owner,omitempty"` + Mode string `json:"mode,omitempty"` + Limit string `json:"limit"` // data_size, or none + Used string `json:"used,omitempty"` + Over bool `json:"over_limit"` + Enforcement string `json:"enforcement"` } // MountReport is one host directory the Boxfile shares in. @@ -336,13 +340,23 @@ func parseVolume(v string) (MountReport, bool) { func dataReport(name string, s boxfile.Spec) DataReport { if s.Backend == "firecracker" { d, _ := fcDataDiskPath(name) // a report never creates the disk - dr := DataReport{Host: d, Kind: "disk"} + dr := DataReport{Host: d, Kind: "disk", Limit: humanBytes(fcDataBytes(s)), + Enforcement: "exact: the disk is the limit, and the guest's kernel refuses a write that does not fit"} statInto(&dr, d) return dr } src, _ := sandbox.DataSource(name) - dr := DataReport{Host: src, Kind: "directory"} + dr := DataReport{Host: src, Kind: "directory", Limit: "none", + Enforcement: "none: /data can grow until the host disk is full"} statInto(&dr, src) + if used, err := dataUsage(name); err == nil { + dr.Used = humanBytes(used) + if limit, ok := quotaApplies(s); ok { + dr.Limit = s.DataSize + dr.Over = used > limit + dr.Enforcement = "measured: box refuses to start the sandbox while over, and checks every 2 s while it runs, stopping the VM when it goes over; a fast writer can overshoot by what it writes in 2 s" + } + } return dr } @@ -516,6 +530,13 @@ func findings(r Report, s boxfile.Spec) []Finding { if r.Network.Egress == "all" { add("warn", "full internet egress: anything the guest can read, including /data and mounts, can leave. Per-sandbox allowlists are planned (#7).") } + if r.Filesystem.Data.Over { + add("fail", fmt.Sprintf("/data holds %s, over its data_size %s: box will not start the sandbox until it is back under", + r.Filesystem.Data.Used, r.Filesystem.Data.Limit)) + } + if r.Filesystem.Data.Limit == "none" { + add("warn", "no data_size: /data can grow until the host disk is full. Set data_size in the Boxfile.") + } for _, m := range r.Filesystem.Mounts { if m.Mode == "rw" { add("warn", fmt.Sprintf("the guest can write %s on the host (mounted at %s)", m.Host, m.Guest)) diff --git a/internal/runtime/runtime.go b/internal/runtime/runtime.go index ae945d3..48270c8 100644 --- a/internal/runtime/runtime.go +++ b/internal/runtime/runtime.go @@ -103,6 +103,9 @@ var ErrNoSandbox = errors.New("no sandbox") // a warm pool and one is waiting, otherwise one booted now. notice, if not // nil, hears about slow paths such as a re-verification. func RunFresh(name string, s boxfile.Spec, argv []string, streams Streams, notice func(string)) error { + if err := overQuota(name, s); err != nil { + return err + } // A pooled VM was checked when it booted and its kernel is checked again // before the command starts, so the warm path skips the toolchain checks // below and costs a connection, not a boot. @@ -122,7 +125,14 @@ func UpChecked(name string, s boxfile.Spec, notice func(string)) (time.Duration, if err := checked(name, s, notice); err != nil { return 0, err } - return Up(name, s) + if err := overQuota(name, s); err != nil { + return 0, err + } + d, err := Up(name, s) + if err == nil { + spawnDataWatch(name, s) + } + return d, err } // checked confirms the toolchain is present and the kernel boundary still @@ -382,7 +392,22 @@ func Run(name string, s boxfile.Spec, argv []string, streams Streams) error { }) defer stop() } + done := make(chan struct{}) + exceeded := watchQuota(name, s, done, func() { + if cmd.Process != nil { + cmd.Process.Kill() + } + b.Remove(runName) + }) err = streamTee(cmd, streams, log) + close(done) + if qerr := exceeded(); qerr != nil { + b.Remove(runName) + if log != nil { + fmt.Fprintf(log, "=== stopped: %v\n", qerr) + } + return fmt.Errorf("%w; box stopped the VM", qerr) + } if log != nil { code := 0 if err != nil { diff --git a/security/escape-test.sh b/security/escape-test.sh index 690a46b..ff3826b 100755 --- a/security/escape-test.sh +++ b/security/escape-test.sh @@ -66,6 +66,7 @@ network: none readonly: true isolation: $ISOLATION backend: firecracker +data_size: 64M EOF else cat > "$BOX_HOME/sandboxes/$NAME/Boxfile" </dev/null 2>&1 +"$BB" up "$NAME" >/dev/null 2>&1 +wrote=$(guest 'dd if=/dev/zero of=/data/fill bs=1M count=300 2>&1; sync') +if [ "$BACKEND" = firecracker ]; then + # /data is a disk of data_size: the guest's own kernel refuses the rest. + size=$(stat -c %s "$BOX_HOME/disks/$NAME.ext4" 2>/dev/null || echo 0) + if echo "$wrote" | grep -qi "no space left" && [ "$size" -le $((64 << 20)) ]; then + pass "/data cannot outgrow data_size: the guest ran out of space at the 64 MiB disk" + else + fail "/data outgrew data_size (disk $((size >> 20)) MiB; dd said: $(echo "$wrote" | tail -1))" + fi +else + # Directory-backed /data is measured: going over stops the sandbox, and it + # will not start again until it is back under. + out=$("$BB" run "$NAME" -- true 2>&1) + if echo "$wrote$out" | grep -q "over its data_size" && echo "$out" | grep -q "over its data_size"; then + pass "a sandbox that filled /data past data_size was stopped and will not start again" + else + fail "a sandbox past its data_size still runs (dd: $(echo "$wrote" | tail -1); box run: $(echo "$out" | head -1))" + fi +fi + echo if [ "$FAILS" -eq 0 ]; then echo "no escapes." diff --git a/tools/site/build.py b/tools/site/build.py index 2141ec0..03b54ce 100644 --- a/tools/site/build.py +++ b/tools/site/build.py @@ -699,6 +699,7 @@ def api_code(name, py, ts, go, rs): ("readonly", "bool", "false", "Read-only root; /tmp and /data stay writable."), ("timeout_seconds", "int", "0", "Per-command limit; exits 124. 0 is unlimited."), ("warm", "0 to 8", "0", "VMs kept booted for run (podman, krun)."), + ("data_size", "e.g. 512M, 10G", "none (8G on firecracker); new Boxfiles: 10G", "Caps /data. Exact on firecracker, where /data is a disk of this size. On podman and krun box measures it: it will not start a sandbox that is over, and stops a running one within 2 s of going over."), ("warmup", "shell lines", "", "Run once in each warm VM before it serves a run."), ("packages", "names", "", "Installed with the image's package manager."), ("run", "shell lines", "", "Extra build steps."), diff --git a/tools/site/fit_prompt.txt b/tools/site/fit_prompt.txt index 11e4258..d83f104 100644 --- a/tools/site/fit_prompt.txt +++ b/tools/site/fit_prompt.txt @@ -19,7 +19,7 @@ An open-source CLI (Go, MIT) that runs code in disposable microVMs on your own m Measured: a warm `run` is ~20–45 ms; `exec` in a running sandbox is ~15 ms for a no-op. ## Boxfile reference (unknown keys are rejected) -base (image ref), backend, cpus (max 16), ram_mib, network (bridge = internet, none = offline), readonly (read-only root; /tmp and /data stay writable), isolation (standard | strict), timeout_seconds (per run; a timed-out run exits 124), warm (0–8, needs network: bridge), warmup (commands run in each pooled VM first), packages (apk, apt or dnf, chosen from the base), run (build steps), env, mounts (host dir → guest path, ro by default; strict allows ro only), blueprint (users, write_files, runcmd, applied at build time). +base (image ref), backend, cpus (max 16), ram_mib, network (bridge = internet, none = offline), data_size (caps /data: exact on firecracker, where /data is a disk; on podman/krun box measures it every 2 s and stops a VM that goes over), readonly (read-only root; /tmp and /data stay writable), isolation (standard | strict), timeout_seconds (per run; a timed-out run exits 124), warm (0–8, needs network: bridge), warmup (commands run in each pooled VM first), packages (apk, apt or dnf, chosen from the base), run (build steps), env, mounts (host dir → guest path, ro by default; strict allows ro only), blueprint (users, write_files, runcmd, applied at build time). ## Driving it - CLI: new [--from example], build, run, shell, up, exec, down, fork, diff, apply, discard, snapshot, restore, reset, data import/export, doctor (checks the host and prints fixes), verify, logs, destroy.