diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index 860e43fd6..ae2834d2a 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -34,7 +34,7 @@ its own in turn. | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | | [MicroVM scenarios](#microvm-scenarios) | 54 | 50 | 3 | 1 | | [CI](#ci) | 69 | 42 | 3 | 24 | -| [Docs](#docs) | 36 | 25 | 3 | 8 | +| [Docs](#docs) | 36 | 18 | 2 | 16 | ## The adapter's scenarios @@ -480,22 +480,22 @@ this matrix. The README's own code is no doctest. | `nvx-rs/src/gpt/mod.rs - gpt (line 43)` | kept | | | `nvx-rs/src/guest/activation.rs - guest::activation (line 158)` | kept | SL6.11c rewrote it on `guest::snapshot::capture`, when its capture preparation and admission fence left (#1087) | | `nvx-rs/src/guest/activation/state.rs - guest::activation::state::Activation (line 56)` | kept | | -| `nvx-rs/src/guest/cgroup.rs - guest::cgroup (line 36)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | -| `nvx-rs/src/guest/entry.rs - guest::entry (line 90)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | -| `nvx-rs/src/guest/exec.rs - guest::exec (line 79)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | -| `nvx-rs/src/guest/exec.rs - guest::exec::Execution::adopt (line 619)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | +| `nvx-rs/src/guest/cgroup.rs - guest::cgroup (line 36)` | retired | `guest::cgroup` leaves (SL6.11b) | +| `nvx-rs/src/guest/entry.rs - guest::entry (line 90)` | retired | `guest::entry` leaves (SL6.11b) | +| `nvx-rs/src/guest/exec.rs - guest::exec (line 79)` | retired | `guest::exec` leaves (SL6.11b) | +| `nvx-rs/src/guest/exec.rs - guest::exec::Execution::adopt (line 619)` | retired | `guest::exec` leaves (SL6.11b) | | `nvx-rs/src/guest/gpt.rs - guest::gpt (line 18)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/mapper.rs - guest::mappings::mapper (line 107)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/table.rs - guest::mappings::table::MappingTable (line 47)` | kept | | -| `nvx-rs/src/guest/mod.rs - guest (line 39)` | kept | The guest-agent guidelines quote it | +| `nvx-rs/src/guest/mod.rs - guest (line 34)` | kept | The guest-agent guidelines quote it | | `nvx-rs/src/guest/nvxs.rs - guest::nvxs (line 22)` | kept | | -| `nvx-rs/src/guest/process.rs - guest::process (line 12)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | -| `nvx-rs/src/guest/process.rs - guest::process (line 39)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | +| `nvx-rs/src/guest/process.rs - guest::process (line 12)` | retired | `guest::process` leaves (SL6.11b) | +| `nvx-rs/src/guest/process.rs - guest::process (line 39)` | retired | `guest::process` leaves (SL6.11b) | | `nvx-rs/src/guest/reseed.rs - guest::reseed (line 17)` | kept | | -| `nvx-rs/src/guest/root.rs - guest::root (line 130)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | +| `nvx-rs/src/guest/root.rs - guest::root (line 126)` | kept | SL6.11b removed its bootstrap, which left with the cgroups. The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/snapshot.rs - guest::snapshot (line 25)` | kept | The guest-agent guidelines quote it | -| `nvx-rs/src/guest/spawn.rs - guest::spawn (line 41)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | -| `nvx-rs/src/guest/spawn.rs - guest::spawn::LaunchStage::from_code (line 201)` | rewritten | `nvx-rs/src/guest/spawn/tests.rs::launch_stages_keep_their_numbers`. The exec engine's launch failure and outcome, which the example read, leave with the engine (SL6.11a) | +| `nvx-rs/src/guest/spawn.rs - guest::spawn (line 41)` | retired | `guest::spawn` leaves (SL6.11b) | +| `nvx-rs/src/guest/spawn.rs - guest::spawn::LaunchStage::from_code (line 201)` | retired | `LaunchStage::from_code` and its test, `launch_stages_keep_their_numbers`, leave with `guest::spawn` (SL6.11b). The stages of nvx-init's executions keep their numbers in the crate's own fork boundary, whose launch report `nvx-rs/src/guest/init/tests/exec.rs::native_workload_cwd_accepts_procfs_links_but_closes_inherited_descriptors` reads from a real launch | | `nvx-rs/src/guest/terminal.rs - guest::terminal (line 10)` | kept | | | `nvx-rs/src/guest/time/mod.rs - guest::time (line 9)` | kept | SL6.11c removed its capture and restore finish, which became crate-private: `guest::snapshot::capture` runs them | | `nvx-rs/src/host/sandbox/prepared.rs - host::sandbox::prepared::PreparedLaunch (line 92)` | retired | `PreparedLaunch` leaves with the checked-file handoff (SL6.3, #1122) | @@ -505,9 +505,9 @@ this matrix. The README's own code is no doctest. | `nvx-rs/src/host/template/plan.rs - host::template::plan::TemplatePlan (line 137)` | retired | `TemplatePlan` leaves with the template store (SL6.2a, #1121) | | `nvx-rs/src/host/template/store.rs - host::template::store::Store (line 473)` | retired | The template store leaves (SL6.2a, #1121) | | `nvx-rs/src/host/transport.rs - host::transport (line 22)` | kept | | -| `nvx-rs/src/lib.rs - (line 58)` | rewritten | Its `Instance` and sandbox store leave (SL6.1, #1121): `nvx-rs/src/lib.rs - (line 57)` compiles a cold boot and a stop through `host::vm` | +| `nvx-rs/src/lib.rs - (line 58)` | rewritten | Its `Instance` and sandbox store leave (SL6.1, #1121): `nvx-rs/src/lib.rs - (line 53)` compiles a cold boot and a stop through `host::vm` | | `nvx-rs/src/protocol/nvxs/mod.rs - protocol::nvxs (line 68)` | kept | | | `nvx-rs/src/host/snapshot/capture.rs - host::snapshot::capture::finish_capture (line 48)` | kept | | | `nvx-rs/src/host/store/mod.rs - host::store (line 104)` | retired | The sandbox store leaves (SL6.1, #1121) | -| `nvx-rs/README.md`, its Example: an instance through `LaunchSpec` and the sandbox store | rewritten | Nothing compiles it. `nvx-rs/examples/host-workflows.rs` compiles the documented host workflows, and the library's guide (SL6.10a) replaces the example. SL6.1 (#1121) made it the cold boot that `nvx-rs/src/lib.rs - (line 57)` compiles | +| `nvx-rs/README.md`, its Example: an instance through `LaunchSpec` and the sandbox store | rewritten | Nothing compiles it. `nvx-rs/examples/host-workflows.rs` compiles the documented host workflows, and the library's guide (SL6.10a) replaces the example. SL6.1 (#1121) made it the cold boot that `nvx-rs/src/lib.rs - (line 53)` compiles | | `nvx-rs/examples/external-launch.rs` | kept | Compiled with the crate's targets, and run by hand on a host. Since SL6.3 (#1122), which deletes `host::supervisor`, a minimal launcher of its own on `std::process` starts its VMM | diff --git a/doc/build.md b/doc/build.md index d35a9ed93..9601998b7 100644 --- a/doc/build.md +++ b/doc/build.md @@ -380,8 +380,8 @@ host, which remains supported. The inventory was built from the configuration that CI's kernel build resolved before it existed, which set 1,131 symbols, 1,055 of them `y` and 503 with a prompt: it keeps what an NVX component, contract, or workload -needs, and leaves the rest off (#344). The generated configuration sets 769 -symbols: 175 inventory settings and 594 derived ones, 53 of which have a +needs, and leaves the rest off (#344). The generated configuration sets 767 +symbols: 174 inventory settings and 593 derived ones, 52 of which have a prompt. The checked-in file had also drifted from what `olddefconfig` resolved: it omitted 130 symbols that the build set, such as the wireless and Ethernet vendor menus, and listed four that the build never set, among them @@ -397,11 +397,12 @@ builds, so it needs neither the serial core nor `CONFIG_SERIAL_EARLYCON`. | Kexec, crash dumps, `/proc/vmcore`, `/proc/kcore`, KASLR (`RANDOMIZE_BASE`, `RANDOMIZE_MEMORY`) and the relocatable kernel | OpenVMM loads `vmlinux` at its link address and never sets `boot_params`' KASLR flag, so randomization never ran | | The crypto API, keyrings, X.509 and PKCS#7 parsers, the security module framework and integrity | No guest component uses them; the kernel's own users, such as module signing, are off | | Filesystems and formats: ISO 9660, autofs, quotas, hugetlbfs, NLS code pages, MBR partitions, EROFS's file backing and LZMA and DEFLATE decompression, FUSE's io_uring and passthrough, `binfmt_misc`, and every initramfs compression but gzip | NVX's layers are EROFS (LZ4 or Zstandard) on virtio-blk, raw or GPT, its scratch ext4, its shares virtio-fs, and its initramfses gzip | -| Cgroup controllers and namespaces that the sandbox never enables: CPU, cpuset, cpuacct, devices v1, freezer v1, hugetlb, RDMA, misc, perf, and the I/O controller's policies, and the IPC and time namespaces | The sandbox enables memory and process-count limits and creates mount, PID, and UTS namespaces; workloads have no capabilities to create others | +| Cgroup controllers and namespaces beyond those that the inventory keeps for root programs in a guest: CPU, cpuset, cpuacct, devices v1, freezer v1, hugetlb, RDMA, misc, perf, and the I/O controller's policies, and the IPC and time namespaces | No NVX component creates a cgroup, or a namespace other than a mount namespace; the inventory keeps the memory, process-count, and block cgroups and the PID, UTS, and network namespaces that root programs in a guest, such as an `nvx-rs run` shell, may use, and workloads have no capabilities to create others | | Swap, the block layer's debugfs files, scheduler statistics, SLUB debugging, memory-init debugging, BIOS-corruption checks, process accounting, relayfs, `PM` and its debugging, the CPU idle framework, the I/O schedulers, `/dev/cpu/*/cpuid` | No guest runs swap or reads them; with no cpuidle driver the kernel idles with HLT either way, and virtio-blk uses no scheduler | +| The magic SysRq key: `/proc/sysrq-trigger`, `kernel.sysrq`, and the SysRq sequences of the consoles | No NVX component uses it, and the hvc console took `^O` on the kernel console, where an `nvx-rs run` shell reads, as a SysRq prefix: `^O c` crashed the guest | | Obsolete interfaces: 16-bit UIDs, `sgetmask`, 16-bit segments, `TIOCSTI` for unprivileged users, line-discipline autoloading, writes to mounted block devices | No supported workload uses them | | CPU vendors other than Intel and AMD | CPU profiles serve Intel and AMD hosts | -| x86 platform options without an effect on the microVM: extended platforms, IRQ rerouting quirks, bus-lock detection, the AMD uncore PMU, ISA DMA and its memory zone, CPU isolation, `devtmpfs` automounting, the firmware build options, the vhost menu, accelerated RFS, ethtool's netlink API, serial sysrq, runtime tests, the decompressor's verbose boot, and the version suffix from Git | Their hardware, boot path, or consumer is absent | +| x86 platform options without an effect on the microVM: extended platforms, IRQ rerouting quirks, bus-lock detection, the AMD uncore PMU, ISA DMA and its memory zone, CPU isolation, `devtmpfs` automounting, the firmware build options, the vhost menu, accelerated RFS, ethtool's netlink API, runtime tests, the decompressor's verbose boot, and the version suffix from Git | Their hardware, boot path, or consumer is absent | The image, the span of guest memory that the kernel's loadable segments cover, `vmlinux`, and its main sections, of the kernels that CI's Docker toolchain @@ -448,7 +449,7 @@ rationales say why NVX needs them. | --- | --- | --- | | Direct boot | `direct-boot`: ACPI, PVH, and PCI off; MP-table parsing, local APIC, IOAPIC, and virtio-mmio devices from the command line | Boot through the [ACPI-free MP-table loader](design/5.1-boot-protocol.md#511-linux-direct-loader) | | Virtio console | `virtio-console`: virtio console and HVC driver over virtio-mmio | Drive the [virtio consoles](design/3.3-consoles.md) | -| Sandbox | `block-storage`, `sandbox-filesystems`, `sandbox-isolation`, `guest-internal-networking`, `unix-sockets`: virtio-blk with GPT partitions; ext4; EROFS with zstd compression; overlayfs that does not always follow redirects; tmpfs extended attributes; memory, PID, and writeback cgroups; BPF and cgroup BPF; seccomp filters; mount, PID, and UTS namespaces without user namespaces; Unix-domain sockets; network namespaces and veth | Layer EROFS over ext4 scratch and confine the workload in the [sandbox bootstrap](design/4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) | +| Sandbox | `block-storage`, `sandbox-filesystems`, `sandbox-isolation`, `guest-internal-networking`, `unix-sockets`: virtio-blk with GPT partitions; ext4; EROFS with zstd compression; overlayfs that does not always follow redirects; tmpfs extended attributes; memory, PID, and writeback cgroups; BPF and cgroup BPF; seccomp filters; mount, PID, and UTS namespaces without user namespaces; Unix-domain sockets; network namespaces and veth | Layer EROFS over an ext4 scratch or tmpfs upper in nvx-init's [layered root](design/4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), let workloads filter their system calls with seccomp, and keep the namespaces, cgroups, and cgroup BPF that root programs in a guest may use | | Network | `network`: IPv4 with TCP socket diagnostics; IPv6 without the SIT tunnel driver; virtio-net | Configure the dual-stack NIC of the [portable network profile](design/3.5-network.md); SIT would add an `sit0` device that decapsulates IPv6 from IPv4 packets | | Shared interrupt status | `shared-interrupt-status`: `CONFIG_VIRTIO_MMIO_SHARED_STATUS` from the kernel patches | Read [shared virtio interrupt status](design/2.4-interrupt-and-io-architecture.md#244-shared-interrupt-status) | | Time ABI | `time-abi`: hypervisor-guest and paravirtualization support on; Hyper-V, KVM guest, and cpufreq off; tickless idle at 100 Hz with high-resolution timers; the RCU stall timeouts; a 256 KiB log buffer | Meet the [time ABI](design/3.2-clock-and-time.md), as described above | diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index c2b42c089..7cb5b9763 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -53,13 +53,8 @@ forks a child, which waits for a grandchild that forces an RCU grace period, and collects only the child, so a grandchild that outlives the RCU stall timeout passes to PID 1, or to the nearest subreaper. No call starts the announcer, which is a program, `/sbin/nvx-ingress`: an init runs it itself, -or has `nvx-ingress start` run it from a socket that it binds for it. Until -the nvx-rs slimming -([microsoft/nvx#1076](https://github.com/microsoft/nvx/issues/1076)) removes -them, the crate also keeps the older process -modules, `guest::{exec, spawn, entry, cgroup, process}`, which NVX's older -guest agents use. nvx-init uses none of these modules, and its time daemon -runs for the VM's life. +or has `nvx-ingress start` run it from a socket that it binds for it. +nvx-init's time daemon runs for the VM's life. ## Build the space diff --git a/kernel/config-microvm b/kernel/config-microvm index 1bc42816a..4b87115c8 100644 --- a/kernel/config-microvm +++ b/kernel/config-microvm @@ -1874,9 +1874,7 @@ CONFIG_OBJTOOL=y # # Generic Kernel Debugging Instruments # -CONFIG_MAGIC_SYSRQ=y -CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1 -# CONFIG_MAGIC_SYSRQ_SERIAL is not set +# CONFIG_MAGIC_SYSRQ is not set CONFIG_DEBUG_FS=y CONFIG_DEBUG_FS_ALLOW_ALL=y # CONFIG_DEBUG_FS_DISALLOW_MOUNT is not set diff --git a/kernel/config-microvm.derived b/kernel/config-microvm.derived index 609fdf296..0e625c4d1 100644 --- a/kernel/config-microvm.derived +++ b/kernel/config-microvm.derived @@ -429,7 +429,6 @@ CONFIG_LOCK_SPIN_ON_OWNER=y # no prompt CONFIG_LOG_CPU_MAX_BUF_SHIFT=12 # default CONFIG_LTO_NONE=y # choice default CONFIG_LZ4_DECOMPRESS=y # no prompt -CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1 # default CONFIG_MAX_SKB_FRAGS=17 # default CONFIG_MEMFD_CREATE=y # selected by TMPFS CONFIG_MEMORY_ISOLATION=y # no prompt diff --git a/nvx-rs/README.md b/nvx-rs/README.md index f87896aef..e63e423a1 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -52,13 +52,11 @@ nvx-rs/ │ ├── testing/ feature `testing`: the fake OpenVMM │ ├── guest/ feature `guest`: the programs that run in Linux guests, the guest's NVXS │ │ connection, the libraries through which a guest init or agent opens the -│ │ terminal on which it serves its session, starts a program, drives its -│ │ execution, collects its children's statuses, settles its executions' -│ │ cgroups, mounts host mappings beneath the descriptors of the roots that it -│ │ names, assembles and tears down a layered root, reads a GPT disk's partition -│ │ table through its block device, and prepares a template's guest for a -│ │ capture before its root's assembly, and the typed calls through which it -│ │ runs the time ABI's steps and the reseed +│ │ terminal on which it serves its session, mounts host mappings beneath the +│ │ descriptors of the roots that it names, assembles and tears down a layered +│ │ root, reads a GPT disk's partition table through its block device, and +│ │ prepares a template's guest for a capture before its root's assembly, and +│ │ the typed calls through which it runs the time ABI's steps and the reseed │ └── bin/ nvx-rs, nvx-fake-openvmm, and the guest binaries ├── tests/ fake-OpenVMM, CLI, NVXS session, process and spawn, checked-handoff, │ host-module layering, and guest-binary tests, the CLI's scenario tests and @@ -128,16 +126,11 @@ changes only when the module's purpose does. | [`nvx::guest::mappings`](#nvxguestmappings) | `Mapping` and `MappingTable`: host mappings, and the account of a host's mapping table, for any guest init or agent | | [`nvx::guest::nvxs`](#nvxguestnvxs) | `GuestConnection`: the guest's role of the NVXS session over a byte stream, such as the control console | | [`nvx::guest::terminal`](#nvxguestterminal) | `Terminal`: the terminal on which a guest agent serves its session, such as the VM's control console | -| [`nvx::guest::spawn`](#nvxguestspawn) | `Command`: starts a program in a new process, for any guest init or agent | -| [`nvx::guest::entry`](#nvxguestentry) | `Entry`: the namespaces, view, root, and identity in which a program runs, for any guest init or agent | -| [`nvx::guest::process`](#nvxguestprocess) | `Child`: the status of one child that the caller names, and of no other | -| [`nvx::guest::cgroup`](#nvxguestcgroup) | Cgroup v2 handles through which a guest init or agent places, accounts, kills, and settles its executions | -| [`nvx::guest::exec`](#nvxguestexec) | `Execution`: drives one program as an execution, for any guest init or agent | | [`nvx::guest::time`](#nvxguesttime) | x86_64: the boot steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent | | [`nvx::guest::snapshot`](#nvxguestsnapshot) | x86_64: bounded Fresh capture with inline restore repair and staged failure dispositions, without execution control | | [`nvx::guest::activation`](#nvxguestactivation) | x86_64: fence-free activation of each clone that its capture repaired | | [`nvx::guest::reseed`](#nvxguestreseed) | A restore's fresh entropy, and the reseed of the kernel's random number generator with it | -| [`nvx::guest::root`](#nvxguestroot) | The operations of a layered root: its bootstrap, assembly, shares, export, mappings, and teardown | +| [`nvx::guest::root`](#nvxguestroot) | The operations of a layered root: its assembly, shares, export, mappings, and teardown | | [`nvx::guest::view`](#nvxguestview) | The VM-wide hostname, machine ID, and single-server resolver, whose binds the layered root owns | | [`nvx::guest::gpt`](#nvxguestgpt) | `read_disk`: the partition table of a GPT disk, read and checked through its block device | @@ -1170,101 +1163,6 @@ session's bytes as they are, and drains it, so that it sends what the agent wrote before the agent ends the VM. NVX's managed agent and other guest agents serve their sessions on it. -### `nvx::guest::spawn` - -`Command`: starts a program in a new process for any guest init or agent, with -its arguments, environment, a `PATH` in which it looks for a program named -without a `/`, root and working directories, standard streams, inherited -descriptors, cgroup, process group or new session, a `Barrier` at which the -program waits until the caller releases it, a reset that gives every signal its -default disposition and blocks none, and an `entry::Prepared` execution to -enter, without NVX's control protocol, managed agent, or workload identity. - -Its `LaunchReport` tells whether the program started, or which `LaunchStage` -failed and why. Each stage keeps its number, `LaunchStage::code`, which -`LaunchStage::from_code` turns back into the stage, and a new stage takes a -number of its own. - -### `nvx::guest::entry` - -`Entry`: the execution that a program enters, for any guest init or agent, from -typed inputs: a root, which is a mount, moved over `/` in new mount, PID, and -UTS namespaces, where the program is PID 1 and leads a new session; a hostname; -a fresh procfs, a read-only sysfs, and a `/dev` of its own with only the -allowlisted `DeviceNode`s, a devpts, and a `/dev/shm`, every other mount -`nodev`; the `MachineId`, the image's own or one generated by -`MachineId::generate`, the time status, and the resolver file that -`Entry::resolver` names, if any, which `Entry::bind_runtime` binds read-only -into the root once, through `mappings::bind`'s walk and attach, for every later -execution, and returns as `RuntimeBinds`, which `RuntimeBinds::remove` unmounts -in reverse order before the root's teardown, and of which a failed bind removes -those made before it, with an `Error::Rollback` that names any that stays; and -the `Identity`: user and group IDs, supplementary groups, and `Capabilities` -from an allowlist, which only user ID 0 may hold, without `CAP_NET_BIND_SERVICE` -and `CAP_NET_RAW` while the execution's listeners are `Publication::Published`, -taken with `no_new_privs`. - -Its program starts with every signal's default disposition and none blocked. For -user ID 0, the risky procfs and sysfs entries are read-only or masked. - -`Entry::prepare` checks it, reads the existing `Account` of the IDs from the -root's account databases unless `AccountMode::Numeric` takes them as they are, -and `spawn::Command::enter` runs it, with each step a `LaunchStage`. It reads no -kernel command line or boot token and creates no account. - -### `nvx::guest::process` - -`Child`: the status of one child that the caller names, by its pidfd or its -process ID. It waits for that child alone, so another child's status stays for -the caller; a caller that collects every status itself, such as a PID 1 that -adopts orphans, hands each one to its `Child` with `observe`. Statuses are -`std::process::ExitStatus`, real-time signals included. - -### `nvx::guest::cgroup` - -`Layout`: the agent's cgroup and the workload cgroup, which the caller names, -and which may not hold the agent's. - -`ExecutionCgroup`: the cgroup of one execution, the workload cgroup itself or a -child of it, which it prepares with the caller's resource values, places -processes in, checks for occupancy through `pids.current`, kills with -`cgroup.kill`, and settles, over the execution's cgroup or the whole workload -subtree, as the caller's `Scope` chooses. It holds its cgroups' directories -open, and the process's removals and preparations of cgroups take turns, so that -it never reaches another cgroup that takes one of their paths. Settling calls -the caller's status collector between its checks, and collects no status itself. - -### `nvx::guest::exec` - -`Execution`: drives one program that the caller spawned with the program's ends -of its `Pipes`, or that the caller started itself and adopts with its own ends -of the program's pipes and a `Child` that names it, without blocking, from the -caller's own loop or reactor, and reports each step as an `Event`: the launch -report, which an adopted program has at once, the output of each stream, kill -failures, and the `Outcome`. - -It reads a stream only as far as the caller's `grant` allows, so a stalled sink -stalls its own execution alone, writes the program's standard input without -`SIGPIPE`, and ends the program on a cancellation, a timeout, or a -disconnection, with `SIGKILL` through the program's pidfd to the program and, if -the caller asks, its process group, while no collector has taken its status, -and, through the opened directory of a cgroup that the caller names, to that -cgroup. - -`signal` sends the program a signal that the caller chooses, such as `SIGTERM` -for a graceful stop, as container runtimes stop their workloads, to the same -processes but the cgroup's, each of them once unless the program leaves its -group meanwhile, while no collector has taken its status. It ends neither the -program nor the execution: it records no cause, so a program that exits after it -ends with its own status, and the caller ends one that outlives the caller's own -deadline with `terminate` or `cancel`. It refuses `SIGKILL`, which only the -execution's causes send. - -It closes the program's input once the program has ended. It waits for its -program alone, by a pidfd, or takes the status from the caller's collector with -`observe`; with `Options::caller_reaps`, that collector is the program's only -one. - ### `nvx::guest::time` x86_64: `TimeAbi`, the boot steps of NVX time ABI v1 as typed calls in the @@ -1287,7 +1185,7 @@ snapshot capture below prepares strictly. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and -from which `nvx::guest::spawn` starts none. No call waits for a child that it +from which NVX's guest programs start none. No call waits for a child that it did not start. `nvx-time` runs its steps through them. @@ -1449,11 +1347,6 @@ The operations of a layered root, for a guest init or agent, whether or not it is PID 1. The module is hidden: NVX's internal reference code of the guest's space, unstable, and crate-private once nothing outside NVX uses it. -`bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with -every process of their parent moved into the agent's, the agent's `memory.low`, -and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its -`memory.max` and `pids.max`, and user namespaces kept unavailable. - `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic @@ -2656,17 +2549,12 @@ The `guest` feature adds the programs that run in NVX's Linux guests: the `nvx::guest` module and one binary per program. The Alpine, Ubuntu, and Azure Linux images ship all ten at their established `/sbin/nvx-*` paths. The module's libraries are not programs: `terminal` opens the terminal on which an -agent serves its session, `spawn` starts a program in a new process, `entry` -has the program enter an execution's namespaces, view, root, and identity, -`process` collects the status of a child that the caller names, `cgroup` -places, kills, and settles the processes of an execution, `exec` drives one -program as an execution, and `mappings` describes host mappings and keeps the +agent serves its session, and `mappings` describes host mappings and keeps the account of a host's mapping table. NVX's own guest space is internal reference code, hidden, unstable, and crate-private once nothing outside NVX uses it: the mapper of `mappings` mounts host mappings beneath the descriptors of the -roots that its caller names, and `root` bootstraps the guest for a layered root -through `cgroup`, assembles the root, mounts live shares and mappings in it -through the mapper, and tears it down. +roots that its caller names, and `root` assembles a layered root, mounts live +shares and mappings in it through the mapper, and tears it down. `nvx-managed-agent` opens its control terminal and runs direct workloads through nvx-init's plain-process core: each child resets its priority, drops its credentials and capabilities, and enters its working directory before exec. diff --git a/nvx-rs/public-api/linux.txt b/nvx-rs/public-api/linux.txt index 3b57f757f..501e4423a 100644 --- a/nvx-rs/public-api/linux.txt +++ b/nvx-rs/public-api/linux.txt @@ -512,24 +512,6 @@ field nvx::guest::activation::RootTemplate::runtime (hidden) field nvx::guest::activation::RootTemplate::scratch (hidden) field nvx::guest::activation::RootTemplate::timeout (hidden) fn nvx::guest::activation::apply (hidden) -mod nvx::guest::cgroup -struct nvx::guest::cgroup::ExecutionCgroup -method nvx::guest::cgroup::ExecutionCgroup::is_occupied -method nvx::guest::cgroup::ExecutionCgroup::kill -method nvx::guest::cgroup::ExecutionCgroup::open -method nvx::guest::cgroup::ExecutionCgroup::path -method nvx::guest::cgroup::ExecutionCgroup::place -method nvx::guest::cgroup::ExecutionCgroup::placement -method nvx::guest::cgroup::ExecutionCgroup::prepare -method nvx::guest::cgroup::ExecutionCgroup::remove -method nvx::guest::cgroup::ExecutionCgroup::settle -struct nvx::guest::cgroup::Layout -method nvx::guest::cgroup::Layout::agent -method nvx::guest::cgroup::Layout::new -method nvx::guest::cgroup::Layout::workload -enum nvx::guest::cgroup::Scope -variant nvx::guest::cgroup::Scope::Execution -variant nvx::guest::cgroup::Scope::Workload mod nvx::guest::clone struct nvx::guest::clone::Activation method nvx::guest::clone::Activation::complete @@ -724,199 +706,6 @@ mod nvx::guest::console_pending (hidden) fn nvx::guest::console_pending::main (hidden) mod nvx::guest::device_io (hidden) fn nvx::guest::device_io::main (hidden) -mod nvx::guest::entry -struct nvx::guest::entry::Account -method nvx::guest::entry::Account::home -method nvx::guest::entry::Account::name -enum nvx::guest::entry::AccountError -variant nvx::guest::entry::AccountError::Duplicate -variant nvx::guest::entry::AccountError::Group -variant nvx::guest::entry::AccountError::Home -variant nvx::guest::entry::AccountError::Malformed -variant nvx::guest::entry::AccountError::Missing -variant nvx::guest::entry::AccountError::NotFile -field nvx::guest::entry::AccountError::NotFile::0 -variant nvx::guest::entry::AccountError::PrimaryGroup -field nvx::guest::entry::AccountError::PrimaryGroup::0 -variant nvx::guest::entry::AccountError::Read -field nvx::guest::entry::AccountError::Read::0 -field nvx::guest::entry::AccountError::Read::1 -variant nvx::guest::entry::AccountError::TooLarge -field nvx::guest::entry::AccountError::TooLarge::0 -enum nvx::guest::entry::AccountMode -variant nvx::guest::entry::AccountMode::Existing -variant nvx::guest::entry::AccountMode::Numeric -struct nvx::guest::entry::Capabilities -const nvx::guest::entry::Capabilities::ALLOWED -const nvx::guest::entry::Capabilities::EMPTY -const nvx::guest::entry::Capabilities::PUBLISHED_REFUSED -method nvx::guest::entry::Capabilities::bits -method nvx::guest::entry::Capabilities::contains -method nvx::guest::entry::Capabilities::is_empty -method nvx::guest::entry::Capabilities::iter -method nvx::guest::entry::Capabilities::with -method nvx::guest::entry::Capabilities::without -struct nvx::guest::entry::Capability -const nvx::guest::entry::Capability::AUDIT_WRITE -const nvx::guest::entry::Capability::CHOWN -const nvx::guest::entry::Capability::DAC_OVERRIDE -const nvx::guest::entry::Capability::FOWNER -const nvx::guest::entry::Capability::FSETID -const nvx::guest::entry::Capability::KILL -const nvx::guest::entry::Capability::NET_BIND_SERVICE -const nvx::guest::entry::Capability::NET_RAW -const nvx::guest::entry::Capability::SETFCAP -const nvx::guest::entry::Capability::SETGID -const nvx::guest::entry::Capability::SETPCAP -const nvx::guest::entry::Capability::SETUID -const nvx::guest::entry::Capability::SYS_CHROOT -method nvx::guest::entry::Capability::is_allowed -method nvx::guest::entry::Capability::new -method nvx::guest::entry::Capability::number -const nvx::guest::entry::DEFAULT_HOSTNAME -enum nvx::guest::entry::Database -variant nvx::guest::entry::Database::Group -variant nvx::guest::entry::Database::Passwd -enum nvx::guest::entry::DeviceNode -const nvx::guest::entry::DeviceNode::ALL -variant nvx::guest::entry::DeviceNode::Full -variant nvx::guest::entry::DeviceNode::Null -variant nvx::guest::entry::DeviceNode::Random -variant nvx::guest::entry::DeviceNode::Tty -variant nvx::guest::entry::DeviceNode::Urandom -variant nvx::guest::entry::DeviceNode::Zero -struct nvx::guest::entry::Entry -method nvx::guest::entry::Entry::bind_runtime -method nvx::guest::entry::Entry::check -method nvx::guest::entry::Entry::devices -method nvx::guest::entry::Entry::hostname -method nvx::guest::entry::Entry::identity -method nvx::guest::entry::Entry::machine_id -method nvx::guest::entry::Entry::new -method nvx::guest::entry::Entry::prepare -method nvx::guest::entry::Entry::publication -method nvx::guest::entry::Entry::resolver -method nvx::guest::entry::Entry::time_status -enum nvx::guest::entry::Error -variant nvx::guest::entry::Error::Account -field nvx::guest::entry::Error::Account::0 -variant nvx::guest::entry::Error::Capability -field nvx::guest::entry::Error::Capability::0 -variant nvx::guest::entry::Error::Groups -variant nvx::guest::entry::Error::Hostname -variant nvx::guest::entry::Error::InvalidId -variant nvx::guest::entry::Error::MachineId -field nvx::guest::entry::Error::MachineId::0 -variant nvx::guest::entry::Error::NonRootCapabilities -variant nvx::guest::entry::Error::Published -field nvx::guest::entry::Error::Published::0 -variant nvx::guest::entry::Error::Resolver -field nvx::guest::entry::Error::Resolver::0 -variant nvx::guest::entry::Error::Rollback -field nvx::guest::entry::Error::Rollback::error -field nvx::guest::entry::Error::Rollback::teardown -variant nvx::guest::entry::Error::Root -field nvx::guest::entry::Error::Root::0 -variant nvx::guest::entry::Error::RootPath -variant nvx::guest::entry::Error::TimeStatus -field nvx::guest::entry::Error::TimeStatus::0 -struct nvx::guest::entry::Identity -method nvx::guest::entry::Identity::account -method nvx::guest::entry::Identity::account_mode -method nvx::guest::entry::Identity::capabilities -method nvx::guest::entry::Identity::capability_set -method nvx::guest::entry::Identity::gid -method nvx::guest::entry::Identity::groups -method nvx::guest::entry::Identity::new -method nvx::guest::entry::Identity::supplementary_groups -method nvx::guest::entry::Identity::uid -const nvx::guest::entry::MAX_ACCOUNT_DATABASE_BYTES -const nvx::guest::entry::MAX_GROUPS -const nvx::guest::entry::MAX_HOSTNAME_BYTES -enum nvx::guest::entry::MachineId -variant nvx::guest::entry::MachineId::Generated -field nvx::guest::entry::MachineId::Generated::0 -variant nvx::guest::entry::MachineId::Image -method nvx::guest::entry::MachineId::generate -struct nvx::guest::entry::Prepared -method nvx::guest::entry::Prepared::account -enum nvx::guest::entry::Publication -variant nvx::guest::entry::Publication::Published -variant nvx::guest::entry::Publication::Unpublished -struct nvx::guest::entry::RuntimeBinds -method nvx::guest::entry::RuntimeBinds::remove -mod nvx::guest::exec -enum nvx::guest::exec::Cause -variant nvx::guest::exec::Cause::Cancelled -variant nvx::guest::exec::Cause::Disconnected -variant nvx::guest::exec::Cause::Terminated -variant nvx::guest::exec::Cause::TimedOut -enum nvx::guest::exec::Event -variant nvx::guest::exec::Event::Closed -field nvx::guest::exec::Event::Closed::0 -variant nvx::guest::exec::Event::Discarded -field nvx::guest::exec::Event::Discarded::0 -field nvx::guest::exec::Event::Discarded::1 -variant nvx::guest::exec::Event::Ended -field nvx::guest::exec::Event::Ended::0 -variant nvx::guest::exec::Event::KillFailed -field nvx::guest::exec::Event::KillFailed::0 -field nvx::guest::exec::Event::KillFailed::1 -variant nvx::guest::exec::Event::Launch -field nvx::guest::exec::Event::Launch::0 -variant nvx::guest::exec::Event::Output -field nvx::guest::exec::Event::Output::0 -field nvx::guest::exec::Event::Output::1 -variant nvx::guest::exec::Event::ReadFailed -field nvx::guest::exec::Event::ReadFailed::0 -field nvx::guest::exec::Event::ReadFailed::1 -struct nvx::guest::exec::Execution -method nvx::guest::exec::Execution::adopt -method nvx::guest::exec::Execution::cancel -method nvx::guest::exec::Execution::close_input -method nvx::guest::exec::Execution::deadline -method nvx::guest::exec::Execution::descriptors -method nvx::guest::exec::Execution::discard -method nvx::guest::exec::Execution::disconnect -method nvx::guest::exec::Execution::grant -method nvx::guest::exec::Execution::id -method nvx::guest::exec::Execution::input -method nvx::guest::exec::Execution::new -method nvx::guest::exec::Execution::next_event -method nvx::guest::exec::Execution::observe -method nvx::guest::exec::Execution::signal -method nvx::guest::exec::Execution::status -method nvx::guest::exec::Execution::terminate -method nvx::guest::exec::Execution::write_input -enum nvx::guest::exec::KillTarget -variant nvx::guest::exec::KillTarget::Cgroup -variant nvx::guest::exec::KillTarget::ProcessGroup -variant nvx::guest::exec::KillTarget::Program -struct nvx::guest::exec::Options -method nvx::guest::exec::Options::caller_reaps -method nvx::guest::exec::Options::cgroup -method nvx::guest::exec::Options::new -method nvx::guest::exec::Options::process_group -method nvx::guest::exec::Options::timeout -struct nvx::guest::exec::Outcome -method nvx::guest::exec::Outcome::cause -method nvx::guest::exec::Outcome::discarded -method nvx::guest::exec::Outcome::status -struct nvx::guest::exec::Pipes -method nvx::guest::exec::Pipes::from_ends -method nvx::guest::exec::Pipes::new -struct nvx::guest::exec::ProgramEnds -field nvx::guest::exec::ProgramEnds::stderr -field nvx::guest::exec::ProgramEnds::stdin -field nvx::guest::exec::ProgramEnds::stdout -enum nvx::guest::exec::Source -variant nvx::guest::exec::Source::Exit -variant nvx::guest::exec::Source::Launch -variant nvx::guest::exec::Source::Output -field nvx::guest::exec::Source::Output::0 -enum nvx::guest::exec::Stream -variant nvx::guest::exec::Stream::Stderr -variant nvx::guest::exec::Stream::Stdout mod nvx::guest::exit (hidden) fn nvx::guest::exit::main (hidden) mod nvx::guest::gpt (hidden) @@ -1114,17 +903,6 @@ variant nvx::guest::nvxs::GuestConnectionError::Io field nvx::guest::nvxs::GuestConnectionError::Io::0 variant nvx::guest::nvxs::GuestConnectionError::Protocol field nvx::guest::nvxs::GuestConnectionError::Protocol::0 -mod nvx::guest::process -struct nvx::guest::process::Child -method nvx::guest::process::Child::from_pidfd -method nvx::guest::process::Child::id -method nvx::guest::process::Child::new -method nvx::guest::process::Child::observe -method nvx::guest::process::Child::pidfd -method nvx::guest::process::Child::status -method nvx::guest::process::Child::try_wait -method nvx::guest::process::Child::wait -method nvx::guest::process::Child::with_pidfd mod nvx::guest::reseed enum nvx::guest::reseed::ReseedError variant nvx::guest::reseed::ReseedError::OpenRandom @@ -1156,18 +934,6 @@ struct nvx::guest::root::AssemblyError (hidden) method nvx::guest::root::AssemblyError::cleanup (hidden) method nvx::guest::root::AssemblyError::error (hidden) method nvx::guest::root::AssemblyError::into_parts (hidden) -struct nvx::guest::root::Bootstrap (hidden) -method nvx::guest::root::Bootstrap::layout (hidden) -method nvx::guest::root::Bootstrap::moved (hidden) -method nvx::guest::root::Bootstrap::workload (hidden) -enum nvx::guest::root::BootstrapStep (hidden) -variant nvx::guest::root::BootstrapStep::AgentCgroup (hidden) -variant nvx::guest::root::BootstrapStep::AgentMemoryLow (hidden) -variant nvx::guest::root::BootstrapStep::Hierarchy (hidden) -variant nvx::guest::root::BootstrapStep::Layout (hidden) -variant nvx::guest::root::BootstrapStep::Placement (hidden) -variant nvx::guest::root::BootstrapStep::UserNamespaces (hidden) -variant nvx::guest::root::BootstrapStep::WorkloadCgroup (hidden) struct nvx::guest::root::Device (hidden) field nvx::guest::root::Device::mmio (hidden) field nvx::guest::root::Device::sectors (hidden) @@ -1184,9 +950,6 @@ variant nvx::guest::root::Error::Access (hidden) field nvx::guest::root::Error::Access::device (hidden) field nvx::guest::root::Error::Access::read_only (hidden) field nvx::guest::root::Error::Access::role (hidden) -variant nvx::guest::root::Error::Bootstrap (hidden) -field nvx::guest::root::Error::Bootstrap::error (hidden) -field nvx::guest::root::Error::Bootstrap::step (hidden) variant nvx::guest::root::Error::Device (hidden) field nvx::guest::root::Error::Device::device (hidden) field nvx::guest::root::Error::Device::error (hidden) @@ -1299,10 +1062,6 @@ enum nvx::guest::root::OverlayMode (hidden) variant nvx::guest::root::OverlayMode::TrustedLowers (hidden) variant nvx::guest::root::OverlayMode::UntrustedLowers (hidden) method nvx::guest::root::OverlayMode::options (hidden) -struct nvx::guest::root::Resources (hidden) -field nvx::guest::root::Resources::agent_memory_low (hidden) -field nvx::guest::root::Resources::workload_memory_max (hidden) -field nvx::guest::root::Resources::workload_pids_max (hidden) enum nvx::guest::root::Role (hidden) variant nvx::guest::root::Role::Distro (hidden) variant nvx::guest::root::Role::Scratch (hidden) @@ -1373,7 +1132,6 @@ enum nvx::guest::root::UpperKind (hidden) variant nvx::guest::root::UpperKind::Scratch (hidden) variant nvx::guest::root::UpperKind::Tmpfs (hidden) fn nvx::guest::root::assemble (hidden) -fn nvx::guest::root::bootstrap (hidden) mod nvx::guest::snapshot enum nvx::guest::snapshot::CaptureCause variant nvx::guest::snapshot::CaptureCause::Io @@ -1480,63 +1238,6 @@ method nvx::guest::snapshot::Repaired::cpus method nvx::guest::snapshot::Repaired::generation_id method nvx::guest::snapshot::Repaired::memory fn nvx::guest::snapshot::capture -mod nvx::guest::spawn -struct nvx::guest::spawn::Barrier -method nvx::guest::spawn::Barrier::release -struct nvx::guest::spawn::Cgroup -method nvx::guest::spawn::Cgroup::open -struct nvx::guest::spawn::Command -method nvx::guest::spawn::Command::barrier -method nvx::guest::spawn::Command::cgroup -method nvx::guest::spawn::Command::check_working_directory_as -method nvx::guest::spawn::Command::enter -method nvx::guest::spawn::Command::environment -method nvx::guest::spawn::Command::export_physical_working_directory -method nvx::guest::spawn::Command::inherit -method nvx::guest::spawn::Command::new -method nvx::guest::spawn::Command::new_process_group -method nvx::guest::spawn::Command::new_session -method nvx::guest::spawn::Command::reset_signals -method nvx::guest::spawn::Command::root -method nvx::guest::spawn::Command::search_path -method nvx::guest::spawn::Command::spawn -method nvx::guest::spawn::Command::stderr -method nvx::guest::spawn::Command::stdin -method nvx::guest::spawn::Command::stdout -method nvx::guest::spawn::Command::working_directory -struct nvx::guest::spawn::LaunchFailure -method nvx::guest::spawn::LaunchFailure::raw_os_error -method nvx::guest::spawn::LaunchFailure::stage -struct nvx::guest::spawn::LaunchReport -method nvx::guest::spawn::LaunchReport::wait -enum nvx::guest::spawn::LaunchStage -variant nvx::guest::spawn::LaunchStage::Barrier -variant nvx::guest::spawn::LaunchStage::Bounding -variant nvx::guest::spawn::LaunchStage::Capabilities -variant nvx::guest::spawn::LaunchStage::Cgroup -variant nvx::guest::spawn::LaunchStage::Credentials -variant nvx::guest::spawn::LaunchStage::Descriptors -variant nvx::guest::spawn::LaunchStage::Devices -variant nvx::guest::spawn::LaunchStage::Exec -variant nvx::guest::spawn::LaunchStage::Hostname -variant nvx::guest::spawn::LaunchStage::Identity -variant nvx::guest::spawn::LaunchStage::MoveRoot -variant nvx::guest::spawn::LaunchStage::NoNewPrivileges -variant nvx::guest::spawn::LaunchStage::PhysicalWorkingDirectory -variant nvx::guest::spawn::LaunchStage::Proc -variant nvx::guest::spawn::LaunchStage::ProcessGroup -variant nvx::guest::spawn::LaunchStage::Propagation -variant nvx::guest::spawn::LaunchStage::Root -variant nvx::guest::spawn::LaunchStage::Session -variant nvx::guest::spawn::LaunchStage::Signals -variant nvx::guest::spawn::LaunchStage::Sys -variant nvx::guest::spawn::LaunchStage::WorkingDirectory -method nvx::guest::spawn::LaunchStage::code -method nvx::guest::spawn::LaunchStage::from_code -struct nvx::guest::spawn::Spawned -method nvx::guest::spawn::Spawned::id -method nvx::guest::spawn::Spawned::into_report -method nvx::guest::spawn::Spawned::take_barrier mod nvx::guest::terminal struct nvx::guest::terminal::Terminal method nvx::guest::terminal::Terminal::configure_raw diff --git a/nvx-rs/src/guest/account.rs b/nvx-rs/src/guest/account.rs index 966e463f8..555b5d765 100644 --- a/nvx-rs/src/guest/account.rs +++ b/nvx-rs/src/guest/account.rs @@ -7,7 +7,7 @@ use super::parse::parse_decimal; use super::sys::{self, Errno, FileDescriptor, Mode, OpenFlags}; /// The largest account database, `/etc/passwd` or `/etc/group`, that existing-account mode reads. -pub const MAX_ACCOUNT_DATABASE_BYTES: u64 = 1 << 20; +pub(crate) const MAX_ACCOUNT_DATABASE_BYTES: u64 = 1 << 20; /// The path of the passwd database below the root. const PASSWD: &std::ffi::CStr = c"etc/passwd"; /// The path of the group database below the root. @@ -15,7 +15,7 @@ const GROUP: &std::ffi::CStr = c"etc/group"; /// The account of an identity, as its passwd entry names it. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct Account { +pub(crate) struct Account { /// The user name. name: Vec, /// The home directory. @@ -24,19 +24,19 @@ pub struct Account { impl Account { /// Returns the user name, as bytes, since passwd entries need not be UTF-8. - pub fn name(&self) -> &[u8] { + pub(crate) fn name(&self) -> &[u8] { &self.name } /// Returns the home directory, an absolute path. - pub fn home(&self) -> &[u8] { + pub(crate) fn home(&self) -> &[u8] { &self.home } } /// An account database of existing-account mode. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -pub enum Database { +pub(crate) enum Database { /// `/etc/passwd`. Passwd, /// `/etc/group`. @@ -56,7 +56,7 @@ impl fmt::Display for Database { /// Why existing-account mode refused an identity. #[derive(Debug)] #[non_exhaustive] -pub enum AccountError { +pub(crate) enum AccountError { /// The database cannot be opened or read below the root, which a symbolic link on its path /// fails too. Read(Database, io::Error), diff --git a/nvx-rs/src/guest/activation.rs b/nvx-rs/src/guest/activation.rs index ddc872d47..16c7c1e11 100644 --- a/nvx-rs/src/guest/activation.rs +++ b/nvx-rs/src/guest/activation.rs @@ -134,8 +134,8 @@ //! directory exists and is not empty, since a clone's root starts as its image alone, while a cold //! sandbox's scratch keeps its upper across boots; the check reads at most one entry of each, and //! no file. The record holds no execution identity: the adapter gives each execution the identity -//! that the execution names, which [`Entry::check`](super::entry::Entry::check) checks, as for a sandbox -//! without a workload identity at a cold boot. If the record has an export, it mounts the host +//! that the execution names, as for a sandbox without a workload identity at a cold boot. If the +//! record has an export, it mounts the host //! export through the template's //! filesystem device, with `Root::mount_export`, read-write, at //! `hostfs/root` in the runtime directory, outside the root, where only the guest's root user diff --git a/nvx-rs/src/guest/cgroup.rs b/nvx-rs/src/guest/cgroup.rs deleted file mode 100644 index bb8667d77..000000000 --- a/nvx-rs/src/guest/cgroup.rs +++ /dev/null @@ -1,565 +0,0 @@ -//! Cgroup v2 handles through which a guest init or agent places, accounts, kills, and settles the -//! processes of its executions. -//! -//! A [`Layout`] names the two cgroups that a caller uses: the agent's, which holds the caller -//! itself, and the workload cgroup, in which or below which its executions run. The library fixes -//! neither, and refuses a workload cgroup that holds the agent's, so that ending a workload's -//! processes never ends the agent. NVX's legacy sandbox, for example, keeps its agents in the -//! `nvx` cgroup and its workloads in the sibling `container` cgroup. -//! -//! An [`ExecutionCgroup`] is the handle of one execution's cgroup: the workload cgroup itself, -//! for a caller that runs one execution at a time, or a child cgroup of it, for one that runs -//! several, whose workload cgroup then holds no process of its own. Through it, the caller: -//! -//! - [prepares](ExecutionCgroup::prepare) the cgroup, with the resource values that it chooses; -//! - [places](ExecutionCgroup::placement) the execution's process in it, which its descendants -//! then inherit, however they detach from its process group or session; -//! - checks whether any process [occupies](ExecutionCgroup::is_occupied) it; -//! - [kills](ExecutionCgroup::kill) every process in it with `cgroup.kill`; -//! - and [settles](ExecutionCgroup::settle) it: waits until no process is left. -//! -//! Each of the last three covers a [`Scope`] that the caller chooses: the execution's cgroup -//! alone, or the workload cgroup's whole subtree. -//! -//! Occupancy is the cgroup's `pids.current`, which counts every process that the cgroup charges, -//! a zombie that no process has collected included. The library collects no status: settling -//! calls the caller's own status collector between its checks, as [`process`](super::process) -//! describes. -//! -//! A handle holds the directories of its cgroups open, from when it was -//! [opened](ExecutionCgroup::open), and each operation opens the interface files that it uses in -//! them anew. So a handle stays valid whatever controllers its cgroup gains, and reaches its own -//! cgroups alone, even once the caller removes one and prepares another at its path. A removal, -//! which only a path names, takes its turn with the process's other removals and preparations, -//! so it never reaches such a cgroup either. -//! -//! ```no_run -//! use std::time::Duration; -//! -//! use nvx::guest::cgroup::{ExecutionCgroup, Layout, Scope}; -//! use nvx::guest::process::Child; -//! use nvx::guest::spawn::Command; -//! -//! # fn main() -> std::io::Result<()> { -//! let layout = Layout::new("/sys/fs/cgroup/agent", "/sys/fs/cgroup/workloads")?; -//! let cgroup = ExecutionCgroup::prepare(&layout, Some("exec-1"), &[("pids.max", "64")])?; -//! let spawned = Command::new(c"/bin/true".to_owned(), vec![c"true".to_owned()]) -//! .cgroup(cgroup.placement()?) -//! .spawn()?; -//! let mut child = Child::new(spawned.id())?; -//! child.wait()?; -//! // End whatever the execution left behind, without touching the other executions. -//! cgroup.kill(Scope::Execution)?; -//! cgroup.settle(Scope::Execution, Duration::from_secs(2), || Ok(()))?; -//! cgroup.remove()?; -//! # Ok(()) -//! # } -//! ``` - -use std::fs; -use std::io; -use std::path::{Component, Path, PathBuf}; -use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; -use std::time::{Duration, Instant}; - -use super::spawn::Cgroup; -use super::sys::{self, Errno, FileDescriptor, Mode, OpenFlags, ProcessId}; - -/// A cgroup's membership file. -const PROCS: &str = "cgroup.procs"; -/// A cgroup's file that kills every process in it and below it. -const KILL: &str = "cgroup.kill"; -/// A cgroup's file that enables controllers for its children. -const SUBTREE_CONTROL: &str = "cgroup.subtree_control"; -/// The `pids` controller's count of the processes in a cgroup and below it. -const PIDS_CURRENT: &str = "pids.current"; -/// The longest canonical `pids.current` that a check reads, with its newline. -const COUNT_CAPACITY: usize = 32; -/// The interval between two checks of a settling cgroup. -const SETTLE_INTERVAL: Duration = Duration::from_millis(10); - -/// The turns of the process's removals of execution cgroups, and of its preparations of -/// execution cgroups of their own, which take them one at a time: see -/// [`ExecutionCgroup::remove`]. -static TURNS: Mutex<()> = Mutex::new(()); - -/// The cgroups that a guest init or agent uses: its own, and the workload cgroup in which or -/// below which its executions run. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Layout { - /// The cgroup that holds the agent itself. - agent: PathBuf, - /// The cgroup in which or below which the executions run. - workload: PathBuf, -} - -impl Layout { - /// Names the agent's cgroup, `agent`, and the workload cgroup, `workload`, as the absolute - /// paths of their directories in a mounted cgroup v2 hierarchy. - /// - /// The paths are compared as written, so the caller names both without symbolic links or - /// other aliases: the check keeps a mistaken layout from ending the agent, not a deliberate - /// one. A relative path, one with a `..` component, a workload cgroup without a parent, and - /// one that is the agent's or holds it are `EINVAL`. The workload cgroup is not the - /// hierarchy's root either, which has none of the interface files that the handles use. - pub fn new(agent: impl Into, workload: impl Into) -> io::Result { - let agent = agent.into(); - let workload = workload.into(); - if !well_formed(&agent) - || !well_formed(&workload) - || workload.parent().is_none() - || agent.starts_with(&workload) - { - return Err(Errno::EINVAL.into()); - } - Ok(Self { agent, workload }) - } - - /// Returns the directory of the agent's cgroup. - pub fn agent(&self) -> &Path { - &self.agent - } - - /// Returns the directory of the workload cgroup. - pub fn workload(&self) -> &Path { - &self.workload - } -} - -/// The processes that an operation of an [`ExecutionCgroup`] covers. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -pub enum Scope { - /// The execution's cgroup, and the cgroups below it. For an execution with a cgroup of its - /// own, the workload's other executions keep running; one that runs in the workload cgroup - /// itself covers the whole workload subtree. - Execution, - /// The workload cgroup, and the whole subtree below it, every execution's cgroup included. - Workload, -} - -/// The handle of one execution's cgroup: the workload cgroup itself, or a child cgroup of it. -/// -/// It holds the directory of the execution's cgroup open, and the workload cgroup's for an -/// execution with a cgroup of its own, as [`open`](Self::open) found them, and each operation -/// opens the interface files that it uses in them. So it reaches those cgroups alone, even once -/// another cgroup takes one of their paths: a cgroup that was removed has no interface file any -/// more, and an operation on it fails with `ENOENT`. Clones share the directories, and dropping -/// the last one closes them, but neither kills nor removes the cgroup: the caller does that with -/// [`kill`](Self::kill), [`settle`](Self::settle), which takes its status collector, and -/// [`remove`](Self::remove), and sees their failures. -#[derive(Clone, Debug)] -pub struct ExecutionCgroup { - /// The path of the execution's cgroup. - path: PathBuf, - /// The path of the workload cgroup. - workload: PathBuf, - /// Whether the execution has a cgroup of its own below the workload cgroup. - own: bool, - /// The directories of the cgroups, which the clones share. - directories: Arc, -} - -/// The open directories of an [`ExecutionCgroup`]'s cgroups. -#[derive(Debug)] -struct Directories { - /// The execution's cgroup. - execution: FileDescriptor, - /// The workload cgroup, for an execution with a cgroup of its own; otherwise `execution` - /// names it. - workload: Option, -} - -impl ExecutionCgroup { - /// Opens the cgroup of an execution in `layout`, without changing the hierarchy: the - /// workload cgroup itself, or, with `name`, its child cgroup `name`. It holds the directory - /// of that cgroup open, and the workload cgroup's for a child cgroup, so both must exist. - /// - /// A name that is not one path component, such as an empty one, `.`, `..`, or one with a - /// `/` or a NUL, is `EINVAL`, and a cgroup that does not exist `ENOENT`. - pub fn open(layout: &Layout, name: Option<&str>) -> io::Result { - let path = execution_path(layout, name)?; - let workload = name.map(|_| open_directory(&layout.workload)).transpose()?; - let execution = open_directory(&path)?; - Ok(Self { - path, - workload: layout.workload.clone(), - own: name.is_some(), - directories: Arc::new(Directories { - execution, - workload, - }), - }) - } - - /// Prepares the cgroup of an execution in `layout`, which [`open`](Self::open) names, and - /// returns its handle. - /// - /// The workload cgroup is created unless it exists, and an execution's own cgroup is created - /// below it, which fails with `EEXIST` if it exists, so that no two executions share one. - /// Each of them gets the `pids` controller, which counts its processes, and the controller of - /// each resource: the controllers whose interface files it lacks are enabled in its parent's - /// `cgroup.subtree_control`, in one write. The workload cgroup's parent must have them, as - /// the hierarchy's root does. Then each resource's value is written to its file in the - /// execution's cgroup, in order: `("pids.max", "64")`, for example, limits the execution to - /// 64 processes. An empty value is written as a newline, as `echo` writes it, so that the - /// kernel applies or refuses it too. A workload cgroup that gives its executions' cgroups a - /// domain controller, such as `memory`, holds no process of its own, as cgroup v2 requires. - /// - /// A resource file that is not a controller's interface file, `CONTROLLER.NAME`, is `EINVAL`, - /// before anything changes. A failure once the execution's own cgroup exists, such as a value - /// that the kernel refuses, removes that cgroup again, so that its name can be prepared anew; - /// the workload cgroup stays, with the controllers that it enabled, as the executions share - /// it. The preparation of a cgroup of the execution's own takes its turn with the process's - /// removals, as [`remove`](Self::remove) describes, so that the handle holds the cgroup that - /// it created, and a failure removes that cgroup alone. - pub fn prepare( - layout: &Layout, - name: Option<&str>, - resources: &[(&str, &str)], - ) -> io::Result { - for (file, _) in resources { - controller(file)?; - } - let path = execution_path(layout, name)?; - let above_workload = layout - .workload - .parent() - .expect("a layout's workload cgroup has a parent"); - let files = || std::iter::once(PIDS_CURRENT).chain(resources.iter().map(|(file, _)| *file)); - create(&layout.workload, false)?; - provide(&layout.workload, above_workload, files())?; - // In its turn, no removal of the process removes the new cgroup, so its path names it. - let _turn = name.map(|_| turn()); - if name.is_some() { - create(&path, true)?; - #[cfg(test)] - tests::pause(); - } - let prepared = Self::open(layout, name).and_then(|execution| { - execution.configure(files(), resources)?; - Ok(execution) - }); - if prepared.is_err() && name.is_some() { - // The new cgroup holds no process yet. - let _ = fs::remove_dir(&path); - } - prepared - } - - /// Gives the execution's own cgroup, if it has one, the interface files `files` through the - /// workload cgroup, then writes each resource's value to its file, in order. - fn configure<'a>( - &self, - files: impl IntoIterator, - resources: &[(&str, &str)], - ) -> io::Result<()> { - if self.own { - provide(&self.path, &self.workload, files)?; - } - for (file, value) in resources { - // Writing nothing would reach no controller. - let value = if value.is_empty() { "\n" } else { value }; - write_in(&self.directories.execution, file, value.as_bytes())?; - } - Ok(()) - } - - /// Returns the directory of the execution's cgroup. - pub fn path(&self) -> &Path { - &self.path - } - - /// Returns the placement that makes a [spawned](super::spawn::Command::cgroup) child join - /// the execution's cgroup before its program runs, so that the program and its descendants - /// start in it. - pub fn placement(&self) -> io::Result { - Cgroup::open_in(&self.directories.execution) - } - - /// Moves the process `id`, with all of its threads, into the execution's cgroup. An ID of 0, - /// or one above `i32::MAX`, is `EINVAL`. - pub fn place(&self, id: u32) -> io::Result<()> { - let process = ProcessId::from_positive(i64::from(id)) - .ok_or_else(|| io::Error::from(Errno::EINVAL))?; - write_in(&self.directories.execution, PROCS, &process.line()) - } - - /// Returns whether `scope` holds a process, a zombie that no process has collected included. - /// - /// It reads `pids.current`, which must be one canonical decimal and a newline: another form - /// is `EPROTO`, and one of more than 31 bytes `EOVERFLOW`. - pub fn is_occupied(&self, scope: Scope) -> io::Result { - let descriptor = open_file(self.directory(scope), PIDS_CURRENT, OpenFlags::O_RDONLY)?; - let mut buffer = [0u8; COUNT_CAPACITY]; - let mut length = 0; - let result = loop { - match descriptor.read(&mut buffer[length..]) { - Ok(count) if count > 0 => { - length += count; - if length < buffer.len() { - continue; - } - break Err(Errno::EOVERFLOW); - } - Ok(_) => break parse_count(&buffer[..length]), - Err(Errno::EINTR) => {} - Err(error) => break Err(error), - } - }; - let occupied = result?; - descriptor.close()?; - Ok(occupied) - } - - /// Kills every process in `scope` with `SIGKILL`, through `cgroup.kill`, which also kills a - /// process that one of them forks meanwhile. It does not wait for them to end: see - /// [`settle`](Self::settle). - pub fn kill(&self, scope: Scope) -> io::Result<()> { - write_in(self.directory(scope), KILL, b"1") - } - - /// Waits at most `timeout` until `scope` holds no process, and fails with `ETIMEDOUT` if one - /// is left then. It ends no process itself: a caller [kills](Self::kill) the scope first, or - /// waits for its processes to end on their own. - /// - /// Before each check, it calls `reap`, the caller's status collector, which collects the - /// statuses of the caller's children that have ended, and whose failure ends the wait: a - /// zombie stays in `pids.current` until its parent collects it. A caller whose executions' - /// orphaned descendants another process collects, such as its PID 1, still collects the - /// status of each child of its own. A caller that drives its own poll loop checks - /// [`is_occupied`](Self::is_occupied) there instead. - pub fn settle( - &self, - scope: Scope, - timeout: Duration, - mut reap: impl FnMut() -> io::Result<()>, - ) -> io::Result<()> { - let deadline = Instant::now().checked_add(timeout); - loop { - reap()?; - if !self.is_occupied(scope)? { - return Ok(()); - } - let remaining = match deadline { - Some(deadline) => deadline.saturating_duration_since(Instant::now()), - None => SETTLE_INTERVAL, - }; - if remaining.is_zero() { - return Err(Errno::ETIMEDOUT.into()); - } - std::thread::sleep(remaining.min(SETTLE_INTERVAL)); - } - } - - /// Removes the execution's own cgroup unless it is gone already, and fails with `EBUSY` - /// while a process, a zombie included, or a cgroup is left in it, so that a removal never - /// hides a status that the caller has yet to collect. An execution that runs in the workload - /// cgroup itself leaves it in place. - /// - /// The removal names the cgroup by its path, which no other cgroup takes while the cgroup - /// exists, as cgroup v2 never renames one. So the process's removals take turns, with its - /// preparations of cgroups of the executions' own: in its turn, a removal finds its cgroup - /// still there before it removes the cgroup at its path, and a cgroup that the caller - /// prepared at the path of a removed one stays, even when clones of one handle remove the - /// cgroup at once. - /// - /// The caller places no process in the cgroup while it removes it: neither with - /// [`place`](Self::place), nor with a child that it spawned with the cgroup's - /// [`placement`](Self::placement) and whose launch has yet to report. The kernel removes a - /// cgroup that only zombies occupy, so a process that joined the cgroup after the removal's - /// check, and ended before the removal, would be a zombie that the removal hides. - pub fn remove(&self) -> io::Result<()> { - if !self.own { - return Ok(()); - } - let _turn = turn(); - match self.is_occupied(Scope::Execution) { - Ok(false) => {} - Ok(true) => return Err(Errno::EBUSY.into()), - Err(error) if error.kind() == io::ErrorKind::NotFound && self.is_removed() => { - return Ok(()); - } - Err(error) => return Err(error), - } - #[cfg(test)] - tests::pause(); - match fs::remove_dir(&self.path) { - Err(error) if error.kind() == io::ErrorKind::NotFound => Ok(()), - result => result, - } - } - - /// Returns whether the execution's cgroup was removed, which leaves its directory without - /// even the membership file that every cgroup has. - fn is_removed(&self) -> bool { - open_file(&self.directories.execution, PROCS, OpenFlags::O_RDONLY) - .is_err_and(|error| error.kind() == io::ErrorKind::NotFound) - } - - /// Returns the directory of the cgroup that `scope` covers. - fn directory(&self, scope: Scope) -> &FileDescriptor { - match (scope, &self.directories.workload) { - (Scope::Workload, Some(workload)) => workload, - _ => &self.directories.execution, - } - } -} - -/// Waits for the process's turn to remove an execution's own cgroup, or to prepare one, which -/// lasts until the guard drops. -fn turn() -> MutexGuard<'static, ()> { - TURNS.lock().unwrap_or_else(PoisonError::into_inner) -} - -/// Returns the path of an execution's cgroup in `layout`: the workload cgroup itself, or its -/// child cgroup `name`, which is one path component, or `EINVAL`. -fn execution_path(layout: &Layout, name: Option<&str>) -> io::Result { - let Some(name) = name else { - return Ok(layout.workload.clone()); - }; - let mut components = Path::new(name).components(); - if name.contains('\0') - || !matches!(components.next(), Some(Component::Normal(_))) - || components.next().is_some() - || name.contains('/') - { - return Err(Errno::EINVAL.into()); - } - Ok(layout.workload.join(name)) -} - -/// Opens the directory of the cgroup at `path`, which names that cgroup from then on, whatever -/// takes its path. -fn open_directory(path: &Path) -> io::Result { - Ok(sys::open( - &sys::path_c_string(path)?, - OpenFlags::O_RDONLY | OpenFlags::O_DIRECTORY | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?) -} - -/// Opens the interface file `file` of the cgroup whose directory `directory` holds, with `flags`. -fn open_file( - directory: &FileDescriptor, - file: &str, - flags: OpenFlags, -) -> io::Result { - let name = sys::c_string(file.as_bytes())?; - Ok(sys::open_at( - directory, - &name, - flags | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?) -} - -/// Writes a whole control value to the interface file `file` of the cgroup whose directory -/// `directory` holds, and reports a failure to close it, with which the kernel may refuse the -/// value. -fn write_in(directory: &FileDescriptor, file: &str, bytes: &[u8]) -> io::Result<()> { - let descriptor = open_file(directory, file, OpenFlags::O_WRONLY)?; - descriptor.write_all(bytes)?; - Ok(descriptor.close()?) -} - -/// Returns whether `path` is absolute and has no `..` component, so that its components name -/// the cgroups that it reaches. -fn well_formed(path: &Path) -> bool { - path.is_absolute() - && !path - .components() - .any(|component| component == Component::ParentDir) -} - -/// Returns the controller of the interface file `file`, `CONTROLLER.NAME`, or `EINVAL` for a -/// file that is not a controller's, such as one of the core `cgroup.*` files. -fn controller(file: &str) -> io::Result<&str> { - let name = |part: &str| { - !part.is_empty() - && part - .bytes() - .all(|byte| byte.is_ascii_alphanumeric() || byte == b'_') - }; - match file.split_once('.') { - Some((controller, rest)) - if controller != "cgroup" - && name(controller) - && controller.bytes().all(|byte| !byte.is_ascii_uppercase()) - && rest.split('.').all(name) => - { - Ok(controller) - } - _ => Err(Errno::EINVAL.into()), - } -} - -/// Creates the cgroup `path`, which fails with `EEXIST` if it exists and `exclusive` is set. -/// Otherwise an existing cgroup is accepted, and a file in its place is `ENOTDIR`. -pub(crate) fn create(path: &Path, exclusive: bool) -> io::Result<()> { - match sys::create_directory(&sys::path_c_string(path)?, Mode::from_bits_retain(0o755)) { - Ok(()) => Ok(()), - Err(Errno::EEXIST) if !exclusive && fs::metadata(path)?.is_dir() => Ok(()), - Err(Errno::EEXIST) if !exclusive => Err(Errno::ENOTDIR.into()), - Err(error) => Err(error.into()), - } -} - -/// Makes sure that the cgroup `cgroup` has each of the interface files `files`: the controllers -/// of those that it lacks are enabled together in `parent`'s `cgroup.subtree_control`, which -/// gives them to every child of `parent`. -pub(crate) fn provide<'a>( - cgroup: &Path, - parent: &Path, - files: impl IntoIterator, -) -> io::Result<()> { - let mut missing = Vec::new(); - for file in files { - match fs::metadata(cgroup.join(file)) { - Ok(_) => {} - Err(error) if error.kind() == io::ErrorKind::NotFound => { - let controller = controller(file)?; - if !missing.contains(&controller) { - missing.push(controller); - } - } - Err(error) => return Err(error), - } - } - if missing.is_empty() { - return Ok(()); - } - let enable = missing - .iter() - .map(|controller| format!("+{controller}")) - .collect::>() - .join(" "); - write(&parent.join(SUBTREE_CONTROL), enable.as_bytes()) -} - -/// Parses the canonical decimal form of `pids.current`, and returns whether it is nonzero. -fn parse_count(bytes: &[u8]) -> Result { - if bytes.len() < 2 - || bytes.last() != Some(&b'\n') - || (bytes.first() == Some(&b'0') && bytes.len() != 2) - || !bytes[..bytes.len() - 1].iter().all(u8::is_ascii_digit) - { - return Err(Errno::EPROTO); - } - Ok(bytes.first() != Some(&b'0')) -} - -/// Writes a whole control value to the existing file `path`, and reports a failure to close it, -/// with which the kernel may refuse the value. -fn write(path: &Path, bytes: &[u8]) -> io::Result<()> { - let path = sys::path_c_string(path)?; - let descriptor = sys::open( - &path, - OpenFlags::O_WRONLY | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?; - descriptor.write_all(bytes)?; - Ok(descriptor.close()?) -} - -#[cfg(test)] -mod tests; diff --git a/nvx-rs/src/guest/cgroup/tests.rs b/nvx-rs/src/guest/cgroup/tests.rs deleted file mode 100644 index c5b2c67a3..000000000 --- a/nvx-rs/src/guest/cgroup/tests.rs +++ /dev/null @@ -1,759 +0,0 @@ -use std::cell::Cell; -use std::ffi::CString; -use std::io::Read; -use std::os::fd::AsFd; -use std::os::unix::process::ExitStatusExt; - -use nix::poll::{PollFd, PollFlags, PollTimeout}; - -use super::*; -use crate::guest::process::Child; -use crate::guest::spawn::Command; - -thread_local! { - /// What the next turn on this thread runs once it found its cgroup, before it acts on the - /// cgroup's path: the pause in which a test races it. - static PAUSE: std::cell::Cell>> = - const { std::cell::Cell::new(None) }; -} - -/// Runs the pause that a test set for this thread's next turn, if any. -pub(super) fn pause() { - if let Some(pause) = PAUSE.take() { - pause(); - } -} - -/// Returns a command that runs `/bin/sh -c script`. -fn shell(script: &str) -> Command { - Command::new( - c"/bin/sh".to_owned(), - vec![ - c"sh".to_owned(), - c"-c".to_owned(), - CString::new(script).unwrap(), - ], - ) -} - -/// Returns the error number of `result`'s error. -fn error(result: io::Result) -> Option { - result.unwrap_err().raw_os_error() -} - -/// Returns a fake hierarchy of regular files in a new temporary directory, which the caller -/// keeps: the root, which holds the agent, with an empty `cgroup.subtree_control`, and the -/// workload cgroup `workload`, without processes, as the layout of an agent that runs in the -/// root. -fn hierarchy() -> (tempfile::TempDir, Layout) { - let temporary = tempfile::tempdir().unwrap(); - let root = temporary.path().join("cgroup"); - let workload = root.join("workload"); - fs::create_dir_all(&workload).unwrap(); - fs::write(root.join(SUBTREE_CONTROL), b"").unwrap(); - for (file, contents) in [ - (PIDS_CURRENT, b"0\n".as_slice()), - (KILL, b""), - (PROCS, b""), - (SUBTREE_CONTROL, b""), - ] { - fs::write(workload.join(file), contents).unwrap(); - } - let layout = Layout::new(&root, &workload).unwrap(); - (temporary, layout) -} - -/// Creates the fake cgroup `name` of an execution below `layout`'s workload cgroup, with -/// the processes that `pids.current` counts, and returns its handle. -fn fake_execution(layout: &Layout, name: &str, pids: &[u8]) -> ExecutionCgroup { - let path = layout.workload().join(name); - fs::create_dir(&path).unwrap(); - for (file, contents) in [(PIDS_CURRENT, pids), (KILL, b""), (PROCS, b"")] { - fs::write(path.join(file), contents).unwrap(); - } - ExecutionCgroup::open(layout, Some(name)).unwrap() -} - -#[test] -fn layouts_keep_the_agent_out_of_the_workload_subtree() { - for (agent, workload) in [ - ("/sys/fs/cgroup", "/sys/fs/cgroup/nvx-exec"), - ("/sys/fs/cgroup/agent", "/sys/fs/cgroup/container"), - ("/sys/fs/cgroup/agent", "/sys/fs/cgroup/agent-workloads"), - ( - "/sys/fs/cgroup/workloads/agent", - "/sys/fs/cgroup/workloads/executions", - ), - ] { - let layout = Layout::new(agent, workload).unwrap(); - assert_eq!( - (layout.agent(), layout.workload()), - (Path::new(agent), Path::new(workload)) - ); - } - for (agent, workload) in [ - ("sys/fs/cgroup", "/sys/fs/cgroup/nvx-exec"), - ("/sys/fs/cgroup", "sys/fs/cgroup/nvx-exec"), - ("/sys/fs/cgroup/agent", "/"), - ("/sys/fs/cgroup/workload", "/sys/fs/cgroup/workload"), - ("/sys/fs/cgroup/workload/", "/sys/fs/cgroup/workload"), - ("/sys/fs/cgroup/workload/agent", "/sys/fs/cgroup/workload"), - ("/sys/fs/cgroup", "/sys/fs/cgroup/nvx-exec/.."), - ( - "/sys/fs/cgroup/workload/../agent", - "/sys/fs/cgroup/workload", - ), - ] { - assert_eq!( - error(Layout::new(agent, workload)), - Some(Errno::EINVAL as i32), - "{agent} {workload}" - ); - } -} - -#[test] -fn executions_run_in_the_workload_cgroup_or_in_one_child_of_it() { - let (_temporary, layout) = hierarchy(); - let workload = ExecutionCgroup::open(&layout, None).unwrap(); - assert_eq!(workload.path(), layout.workload()); - for name in ["exec-1", "exec.1", "..."] { - fs::create_dir(layout.workload().join(name)).unwrap(); - let execution = ExecutionCgroup::open(&layout, Some(name)).unwrap(); - assert_eq!(execution.path(), layout.workload().join(name)); - } - for name in ["", ".", "..", "a/b", "/a", "a/", "./a", "a\0b"] { - assert_eq!( - error(ExecutionCgroup::open(&layout, Some(name))), - Some(Errno::EINVAL as i32), - "{name:?}" - ); - assert_eq!( - error(ExecutionCgroup::prepare(&layout, Some(name), &[])), - Some(Errno::EINVAL as i32), - "{name:?}" - ); - } - // A handle holds its cgroups' directories, which must exist. - assert_eq!( - error(ExecutionCgroup::open(&layout, Some("missing"))), - Some(Errno::ENOENT as i32) - ); - fs::write(layout.workload().join("file"), b"").unwrap(); - assert_eq!( - error(ExecutionCgroup::open(&layout, Some("file"))), - Some(Errno::ENOTDIR as i32) - ); - let missing = Layout::new(layout.agent(), layout.agent().join("missing")).unwrap(); - assert_eq!( - error(ExecutionCgroup::open(&missing, None)), - Some(Errno::ENOENT as i32) - ); -} - -#[test] -fn preparation_creates_the_cgroup_and_enables_its_controllers_together() { - let (temporary, _) = hierarchy(); - let root = temporary.path().join("cgroup"); - let layout = Layout::new(&root, root.join("fresh")).unwrap(); - let control = root.join(SUBTREE_CONTROL); - // A new cgroup lacks the files of both controllers, which one write enables. - let failure = ExecutionCgroup::prepare(&layout, None, &[("memory.max", "max")]).unwrap_err(); - assert!(layout.workload().is_dir()); - assert_eq!(fs::read(&control).unwrap(), b"+pids +memory"); - // A fake hierarchy gains no file when a controller is enabled. - assert_eq!(failure.raw_os_error(), Some(Errno::ENOENT as i32)); - - // With every file in place, nothing is enabled, and the values are written in order. - fs::write(&control, b"").unwrap(); - for file in [PIDS_CURRENT, "memory.max", "memory.high"] { - fs::write(layout.workload().join(file), b"").unwrap(); - } - let execution = ExecutionCgroup::prepare( - &layout, - None, - &[("memory.max", "1024"), ("memory.high", "512")], - ) - .unwrap(); - assert_eq!(execution.path(), layout.workload()); - assert_eq!(fs::read(&control).unwrap(), b""); - assert_eq!( - fs::read(layout.workload().join("memory.max")).unwrap(), - b"1024" - ); - assert_eq!( - fs::read(layout.workload().join("memory.high")).unwrap(), - b"512" - ); - - // An existing workload cgroup is accepted, a file in its place is not. - fs::remove_dir_all(layout.workload()).unwrap(); - fs::write(layout.workload(), b"").unwrap(); - assert_eq!( - error(ExecutionCgroup::prepare(&layout, None, &[])), - Some(Errno::ENOTDIR as i32) - ); -} - -#[test] -fn invalid_resources_change_nothing() { - let (temporary, _) = hierarchy(); - let root = temporary.path().join("cgroup"); - let layout = Layout::new(&root, root.join("fresh")).unwrap(); - for file in [ - "cgroup.procs", - "cgroup.subtree_control", - "pids", - ".max", - "pids.", - "pids..max", - "Pids.max", - "pids/max", - "../pids.max", - "pids.max/x", - "pids.max\n", - ] { - assert_eq!( - error(ExecutionCgroup::prepare(&layout, None, &[(file, "1")])), - Some(Errno::EINVAL as i32), - "{file:?}" - ); - } - assert!(!layout.workload().exists()); - assert_eq!(fs::read(root.join(SUBTREE_CONTROL)).unwrap(), b""); - for file in [ - "hugetlb.2MB.max", - "memory.swap.max", - "cpuset.cpus.partition", - "io.max", - ] { - assert_eq!(controller(file).unwrap(), file.split('.').next().unwrap()); - } -} - -#[test] -fn an_execution_s_own_cgroup_is_new_and_takes_its_controllers_from_the_workload() { - let (_temporary, layout) = hierarchy(); - let root = layout.agent().to_owned(); - let execution = ExecutionCgroup::prepare(&layout, Some("exec-1"), &[]).unwrap(); - assert!(execution.path().is_dir()); - // The workload cgroup counts its processes already, so only it enables the controller. - assert_eq!(fs::read(root.join(SUBTREE_CONTROL)).unwrap(), b""); - assert_eq!( - fs::read(layout.workload().join(SUBTREE_CONTROL)).unwrap(), - b"+pids" - ); - // A second execution of the same name would share the first one's cgroup. - let duplicate = ExecutionCgroup::prepare(&layout, Some("exec-1"), &[]).unwrap_err(); - assert_eq!(duplicate.kind(), io::ErrorKind::AlreadyExists); - // A workload cgroup that lacks the controller gets it from its parent first. - fs::remove_file(layout.workload().join(PIDS_CURRENT)).unwrap(); - ExecutionCgroup::prepare(&layout, Some("exec-2"), &[]).unwrap(); - assert_eq!(fs::read(root.join(SUBTREE_CONTROL)).unwrap(), b"+pids"); -} - -#[test] -fn a_failed_preparation_leaves_no_cgroup_of_the_execution_s_own() { - let (_temporary, layout) = hierarchy(); - // No controller creates `pids.max` in a fake cgroup, so its write fails once the - // execution's cgroup exists. - let failed = ExecutionCgroup::prepare(&layout, Some("exec-1"), &[("pids.max", "16")]); - assert_eq!(failed.unwrap_err().kind(), io::ErrorKind::NotFound); - assert!(!layout.workload().join("exec-1").exists()); - // The name can be prepared anew. - let execution = ExecutionCgroup::prepare(&layout, Some("exec-1"), &[]).unwrap(); - assert!(execution.path().is_dir()); - // An execution in the workload cgroup itself leaves the shared cgroup in place. - let failed = ExecutionCgroup::prepare(&layout, None, &[("pids.max", "16")]); - assert_eq!(failed.unwrap_err().kind(), io::ErrorKind::NotFound); - assert!(layout.workload().join(PIDS_CURRENT).is_file()); -} - -#[test] -fn an_empty_resource_value_reaches_its_file_as_a_newline() { - let (_temporary, layout) = hierarchy(); - let limit = layout.workload().join("pids.max"); - fs::write(&limit, b"").unwrap(); - ExecutionCgroup::prepare(&layout, None, &[("pids.max", "")]).unwrap(); - // The kernel gets a value to apply or refuse, as from `echo`, not a write of nothing. - assert_eq!(fs::read(&limit).unwrap(), b"\n"); -} - -#[test] -fn occupancy_is_the_scope_s_canonical_process_count() { - let (_temporary, layout) = hierarchy(); - let execution = fake_execution(&layout, "exec-1", b"1\n"); - assert!(execution.is_occupied(Scope::Execution).unwrap()); - assert!(!execution.is_occupied(Scope::Workload).unwrap()); - let count = layout.workload().join(PIDS_CURRENT); - for (contents, expected) in [ - (b"0\n".as_slice(), Ok(false)), - (b"10\n", Ok(true)), - (b"4194304\n", Ok(true)), - (b"", Err(Errno::EPROTO)), - (b"0", Err(Errno::EPROTO)), - (b"00\n", Err(Errno::EPROTO)), - (b"-1\n", Err(Errno::EPROTO)), - (b" 1\n", Err(Errno::EPROTO)), - (b"max\n", Err(Errno::EPROTO)), - (&[b'1'; 40], Err(Errno::EOVERFLOW)), - ] { - fs::write(&count, contents).unwrap(); - assert_eq!( - execution - .is_occupied(Scope::Workload) - .map_err(|error| Errno::from_raw(error.raw_os_error().unwrap())), - expected, - "{contents:?}" - ); - } - fs::remove_file(&count).unwrap(); - assert_eq!( - error(execution.is_occupied(Scope::Workload)), - Some(Errno::ENOENT as i32) - ); -} - -#[test] -fn a_kill_reaches_the_scope_s_cgroup_alone() { - let (_temporary, layout) = hierarchy(); - let execution = fake_execution(&layout, "exec-1", b"0\n"); - execution.kill(Scope::Execution).unwrap(); - assert_eq!(fs::read(execution.path().join(KILL)).unwrap(), b"1"); - assert_eq!(fs::read(layout.workload().join(KILL)).unwrap(), b""); - execution.kill(Scope::Workload).unwrap(); - assert_eq!(fs::read(layout.workload().join(KILL)).unwrap(), b"1"); - fs::remove_file(execution.path().join(KILL)).unwrap(); - assert_eq!( - error(execution.kill(Scope::Execution)), - Some(Errno::ENOENT as i32) - ); -} - -#[test] -fn settlement_collects_statuses_between_its_checks_until_the_deadline() { - let (_temporary, layout) = hierarchy(); - let execution = fake_execution(&layout, "exec-1", b"2\n"); - let count = execution.path().join(PIDS_CURRENT); - // Each collection lets one process go, as collecting a zombie does. - let collections = Cell::new(0); - execution - .settle(Scope::Execution, Duration::from_secs(10), || { - collections.set(collections.get() + 1); - let left = 3 - collections.get().min(3); - fs::write(&count, format!("{left}\n")) - }) - .unwrap(); - assert_eq!(collections.get(), 3); - - fs::write(&count, b"1\n").unwrap(); - let failed = execution.settle(Scope::Execution, Duration::from_secs(10), || { - Err(Errno::ECHILD.into()) - }); - assert_eq!(error(failed), Some(Errno::ECHILD as i32)); - collections.set(0); - let timed_out = execution.settle(Scope::Execution, Duration::from_millis(50), || { - collections.set(collections.get() + 1); - Ok(()) - }); - assert_eq!(error(timed_out), Some(Errno::ETIMEDOUT as i32)); - assert!(collections.get() > 1); - fs::write(&count, b"bad\n").unwrap(); - assert_eq!( - error(execution.settle(Scope::Execution, Duration::ZERO, || Ok(()))), - Some(Errno::EPROTO as i32) - ); - // The workload cgroup is empty, whatever its execution's count. - execution - .settle(Scope::Workload, Duration::ZERO, || Ok(())) - .unwrap(); -} - -#[test] -fn processes_are_placed_in_the_execution_s_cgroup() { - let (_temporary, layout) = hierarchy(); - let execution = fake_execution(&layout, "exec-1", b"0\n"); - let procs = execution.path().join(PROCS); - let spawned = shell("exit 0") - .cgroup(execution.placement().unwrap()) - .spawn() - .unwrap(); - let mut child = Child::new(spawned.id()).unwrap(); - assert!(child.wait().unwrap().success()); - assert_eq!( - fs::read(&procs).unwrap(), - format!("{}\n", child.id()).into_bytes() - ); - execution.place(4_194_304).unwrap(); - assert_eq!(fs::read(&procs).unwrap(), b"4194304\n"); - for id in [0, u32::MAX] { - assert_eq!(error(execution.place(id)), Some(Errno::EINVAL as i32)); - } -} - -#[test] -fn a_handle_reaches_its_own_cgroups_alone_once_another_takes_a_path() { - let (_temporary, layout) = hierarchy(); - let old = fake_execution(&layout, "exec-1", b"0\n"); - let clone = old.clone(); - // The caller removes the cgroup, as `remove` does in a real hierarchy, and prepares - // another of the same name. - fs::remove_dir_all(old.path()).unwrap(); - let new = fake_execution(&layout, "exec-1", b"1\n"); - for handle in [&old, &clone] { - assert_eq!( - error(handle.kill(Scope::Execution)), - Some(Errno::ENOENT as i32) - ); - assert_eq!( - error(handle.is_occupied(Scope::Execution)), - Some(Errno::ENOENT as i32) - ); - assert_eq!(error(handle.place(1)), Some(Errno::ENOENT as i32)); - assert_eq!(error(handle.placement()), Some(Errno::ENOENT as i32)); - handle.remove().unwrap(); - } - assert!(new.path().is_dir()); - assert_eq!(fs::read(new.path().join(KILL)).unwrap(), b""); - assert_eq!(fs::read(new.path().join(PROCS)).unwrap(), b""); - assert!(new.is_occupied(Scope::Execution).unwrap()); - // The workload cgroup is still the one that the old handle opened. - old.kill(Scope::Workload).unwrap(); - assert_eq!(fs::read(layout.workload().join(KILL)).unwrap(), b"1"); - new.kill(Scope::Execution).unwrap(); - assert_eq!(fs::read(new.path().join(KILL)).unwrap(), b"1"); -} - -#[test] -fn only_an_empty_cgroup_of_the_execution_s_own_is_removed() { - let (_temporary, layout) = hierarchy(); - ExecutionCgroup::open(&layout, None) - .unwrap() - .remove() - .unwrap(); - assert!(layout.workload().is_dir()); - let execution = fake_execution(&layout, "exec-1", b"1\n"); - assert_eq!(error(execution.remove()), Some(Errno::EBUSY as i32)); - fs::remove_file(execution.path().join(PIDS_CURRENT)).unwrap(); - assert_eq!(error(execution.remove()), Some(Errno::ENOENT as i32)); - fs::remove_dir_all(execution.path()).unwrap(); - execution.remove().unwrap(); -} - -#[test] -fn cgroup_operations_leave_an_unrelated_child_s_status() { - let mut unrelated = std::process::Command::new("/bin/true").spawn().unwrap(); - let process = ProcessId::from_child(&unrelated); - while !sys::tests::has_exited(process).unwrap() { - std::thread::yield_now(); - } - let (_temporary, layout) = hierarchy(); - let workload = ExecutionCgroup::prepare(&layout, None, &[]).unwrap(); - let execution = fake_execution(&layout, "exec-1", b"0\n"); - for cgroup in [&workload, &execution] { - let spawned = shell("exit 0") - .cgroup(cgroup.placement().unwrap()) - .spawn() - .unwrap(); - let mut child = Child::new(spawned.id()).unwrap(); - for scope in [Scope::Execution, Scope::Workload] { - assert!(!cgroup.is_occupied(scope).unwrap()); - cgroup.kill(scope).unwrap(); - cgroup - .settle(scope, Duration::from_secs(10), || { - child.try_wait().map(drop) - }) - .unwrap(); - } - assert!(child.wait().unwrap().success()); - } - // Nothing collected a status that it did not name. - assert!(unrelated.try_wait().unwrap().unwrap().success()); -} - -/// A workload cgroup in the real hierarchy, which the test removes, with everything in it, -/// when it ends. -struct RealWorkload(Layout); - -impl RealWorkload { - /// Creates a new workload cgroup below the root of the hierarchy at `/sys/fs/cgroup`, - /// with the test's own cgroup as the agent's. It returns `None` where the test cannot - /// change the hierarchy: unless the test runs as root in a cgroup v2 hierarchy with the - /// `pids` controller, in which it can create a cgroup, as a read-only mount, or one - /// that a user namespace's root does not own, does not let it. - fn new(name: &str) -> Option { - let root = Path::new("/sys/fs/cgroup"); - let controllers = fs::read_to_string(root.join("cgroup.controllers")).ok()?; - if !nix::unistd::geteuid().is_root() - || !controllers - .split_whitespace() - .any(|controller| controller == "pids") - { - return None; - } - let own = fs::read_to_string("/proc/self/cgroup").ok()?; - let own = own.lines().find_map(|line| line.strip_prefix("0::"))?; - let agent = root.join(own.trim_start_matches('/')); - let workload = root.join(format!("nvx-test-{}-{name}", std::process::id())); - // The test's cgroup lies outside the mount's view in another cgroup namespace, - // where it reads as a path with `..` components, which no layout accepts. - let layout = Layout::new(agent, workload).ok()?; - match fs::create_dir(layout.workload()) { - Ok(()) => Some(Self(layout)), - Err(error) - if matches!( - error.kind(), - io::ErrorKind::ReadOnlyFilesystem | io::ErrorKind::PermissionDenied - ) => - { - None - } - Err(error) => panic!( - "cannot create the test's cgroup {}: {error}", - layout.workload().display() - ), - } - } -} - -impl Drop for RealWorkload { - /// Kills whatever is left in the workload cgroup, then removes its cgroups. - fn drop(&mut self) { - let workload = self.0.workload(); - let _ = write(&workload.join(KILL), b"1"); - for _ in 0..1000 { - if fs::read(workload.join(PIDS_CURRENT)).is_ok_and(|count| count == b"0\n") { - break; - } - std::thread::sleep(SETTLE_INTERVAL); - } - if let Ok(entries) = fs::read_dir(workload) { - for entry in entries.flatten() { - if entry.file_type().is_ok_and(|kind| kind.is_dir()) { - let _ = fs::remove_dir(entry.path()); - } - } - } - let _ = fs::remove_dir(workload); - } -} - -/// Returns whether `reader` polls readable, with data or end of file, within `timeout`. -fn readable(reader: &std::io::PipeReader, timeout: Duration) -> bool { - let mut descriptors = [PollFd::new(reader.as_fd(), PollFlags::POLLIN)]; - nix::poll::poll(&mut descriptors, PollTimeout::try_from(timeout).unwrap()).unwrap() == 1 -} - -#[test] -fn descendants_that_hold_pipes_are_found_and_settled() { - // The test changes the real hierarchy, which needs root; elsewhere it does nothing. - let Some(workload) = RealWorkload::new("settle") else { - return; - }; - let layout = &workload.0; - let first = ExecutionCgroup::prepare(layout, Some("first"), &[("pids.max", "16")]).unwrap(); - let second = ExecutionCgroup::prepare(layout, Some("second"), &[]).unwrap(); - assert_eq!( - fs::read_to_string(first.path().join("pids.max")).unwrap(), - "16\n" - ); - assert_eq!( - ExecutionCgroup::prepare(layout, Some("first"), &[]) - .unwrap_err() - .kind(), - io::ErrorKind::AlreadyExists - ); - // A value that the kernel refuses, an empty one included, leaves no cgroup behind, so the - // name can be prepared anew. - for value in ["-1", ""] { - assert_eq!( - error(ExecutionCgroup::prepare( - layout, - Some("third"), - &[("pids.max", value)] - )), - Some(Errno::EINVAL as i32), - "{value:?}" - ); - assert!(!layout.workload().join("third").exists()); - } - ExecutionCgroup::prepare(layout, Some("third"), &[("pids.max", "8")]) - .unwrap() - .remove() - .unwrap(); - - // The first execution's program ends at once, but leaves a descendant in a session of its - // own, outside the program's process group, which holds the program's output open. - let (mut output, writer) = std::io::pipe().unwrap(); - let spawned = shell("setsid sleep 60 & echo started") - .stdout(writer) - .cgroup(first.placement().unwrap()) - .spawn() - .unwrap(); - let mut child = Child::with_pidfd(spawned.id()).unwrap(); - // A child joins its cgroup before its program runs, which its launch report waits for. - assert!(spawned.into_report().wait().unwrap().is_none()); - assert!(child.wait().unwrap().success()); - let mut started = [0u8; 8]; - output.read_exact(&mut started).unwrap(); - assert_eq!(&started, b"started\n"); - assert!(!readable(&output, Duration::from_millis(200))); - assert!(first.is_occupied(Scope::Execution).unwrap()); - - // The second execution keeps running. - let spawned = shell("exec sleep 60") - .cgroup(second.placement().unwrap()) - .spawn() - .unwrap(); - let mut sibling = Child::new(spawned.id()).unwrap(); - assert!(spawned.into_report().wait().unwrap().is_none()); - - // The first execution's cgroup finds the descendant: once it is settled, the output ends. - first.kill(Scope::Execution).unwrap(); - first - .settle(Scope::Execution, Duration::from_secs(10), || Ok(())) - .unwrap(); - assert!(readable(&output, Duration::ZERO)); - let mut rest = Vec::new(); - output.read_to_end(&mut rest).unwrap(); - assert!(rest.is_empty()); - first.remove().unwrap(); - assert!(!first.path().exists()); - // A cgroup prepared at the removed one's path is out of the old handle's reach. - let again = ExecutionCgroup::prepare(layout, Some("first"), &[]).unwrap(); - assert_eq!( - error(first.kill(Scope::Execution)), - Some(Errno::ENOENT as i32) - ); - assert_eq!( - error(first.is_occupied(Scope::Execution)), - Some(Errno::ENOENT as i32) - ); - first.remove().unwrap(); - assert!(again.path().is_dir()); - again.remove().unwrap(); - assert_eq!(sibling.try_wait().unwrap(), None); - assert!(second.is_occupied(Scope::Execution).unwrap()); - - // Settling the whole workload subtree ends the sibling too. - second.kill(Scope::Workload).unwrap(); - second - .settle(Scope::Workload, Duration::from_secs(10), || { - sibling.try_wait().map(drop) - }) - .unwrap(); - assert_eq!(sibling.status().unwrap().signal(), Some(nix::libc::SIGKILL)); - second.remove().unwrap(); -} - -/// Runs a racer on another thread once this thread's next turn pauses, and lets the turn go -/// on once the racer has returned, or half a second after its setup, as it does when the turn -/// holds the racer off. The racer runs `setup` first, which the turn waits for, so that it -/// takes place within the pause however late the thread starts, then `racer`, with what -/// `setup` returned. Joining the thread returns what `racer` returned, or `None` if no turn -/// paused. -fn race( - setup: impl FnOnce() -> S + Send + 'static, - racer: impl FnOnce(S) -> T + Send + 'static, -) -> std::thread::JoinHandle> { - let (go, start) = std::sync::mpsc::channel(); - let (ready, set_up) = std::sync::mpsc::channel(); - let (finished, done) = std::sync::mpsc::channel(); - let thread = std::thread::spawn(move || { - start.recv().ok()?; - let state = setup(); - let _ = ready.send(()); - let result = racer(state); - let _ = finished.send(()); - Some(result) - }); - PAUSE.set(Some(Box::new(move || { - go.send(()).unwrap(); - // A setup that fails drops its sender, which ends the wait at once. - let _ = set_up.recv_timeout(Duration::from_secs(10)); - let _ = done.recv_timeout(Duration::from_millis(500)); - }))); - thread -} - -/// Returns what the racer of `thread` returned, once no turn of this thread is left to pause. -fn finish(thread: std::thread::JoinHandle>) -> T { - drop(PAUSE.take()); - thread.join().unwrap().expect("the turn paused") -} - -#[test] -fn a_removal_leaves_a_cgroup_prepared_at_its_path_by_a_concurrent_one() { - // The test changes the real hierarchy, which needs root; elsewhere it does nothing. - let Some(workload) = RealWorkload::new("removals") else { - return; - }; - let layout = workload.0.clone(); - let first = ExecutionCgroup::prepare(&layout, Some("exec"), &[]).unwrap(); - let delayed = first.clone(); - // Once the delayed clone found the cgroup empty, the racer removes it through the other - // clone and prepares another of the same name, unless the delayed removal holds it off. - let racer = race( - move || { - // The delayed removal pauses between its check and its removal. - assert!(!first.is_occupied(Scope::Execution).unwrap()); - first - }, - { - let layout = layout.clone(); - move |first: ExecutionCgroup| { - first.remove().unwrap(); - ExecutionCgroup::prepare(&layout, Some("exec"), &[]).unwrap() - } - }, - ); - delayed.remove().unwrap(); - let replacement = finish(racer); - assert!(replacement.path().is_dir()); - assert!(!replacement.is_occupied(Scope::Execution).unwrap()); - replacement.remove().unwrap(); - assert!(!replacement.path().exists()); -} - -#[test] -fn a_failed_preparation_removes_its_own_cgroup_alone() { - // The test changes the real hierarchy, which needs root; elsewhere it does nothing. - let Some(workload) = RealWorkload::new("preparations") else { - return; - }; - let layout = workload.0.clone(); - // A first preparation gives the workload cgroup's children the `pids` controller. - ExecutionCgroup::prepare(&layout, Some("first"), &[]) - .unwrap() - .remove() - .unwrap(); - // Once the preparation created its cgroup, the racer opens that cgroup, within the pause, - // removes it, and prepares another of the same name, unless the preparation holds it off. - // The preparation then fails, as the kernel refuses its value, and removes its cgroup - // again. - let racer = race( - { - let layout = layout.clone(); - move || ExecutionCgroup::open(&layout, Some("exec")).unwrap() - }, - { - let layout = layout.clone(); - move |opened: ExecutionCgroup| { - opened.remove().unwrap(); - ExecutionCgroup::prepare(&layout, Some("exec"), &[]).unwrap() - } - }, - ); - assert_eq!( - error(ExecutionCgroup::prepare( - &layout, - Some("exec"), - &[("pids.max", "-1")] - )), - Some(Errno::EINVAL as i32) - ); - let replacement = finish(racer); - assert!(replacement.path().is_dir()); - assert!(!replacement.is_occupied(Scope::Execution).unwrap()); - replacement.remove().unwrap(); -} diff --git a/nvx-rs/src/guest/entry.rs b/nvx-rs/src/guest/entry.rs deleted file mode 100644 index 012d8faa2..000000000 --- a/nvx-rs/src/guest/entry.rs +++ /dev/null @@ -1,904 +0,0 @@ -//! Entering an execution, for any guest init or agent: the namespaces, the view, the root, and -//! the identity in which a program runs, from the adapter's typed inputs. -//! -//! An [`Entry`] describes how an adapter's executions enter its root: -//! -//! - the root: the path of a mount, such as a layered root's; -//! - the [`Identity`]: the user and group IDs, the supplementary groups, the [`Capabilities`], -//! from an allowlist of 13, and the [`AccountMode`]; -//! - the hostname, `nvx-sandbox` unless the adapter names another; -//! - the [`MachineId`] mode, the [`Publication`] state, the source of the time status, if any, -//! the resolver file, if any, and the [`DeviceNode`]s of `/dev`. -//! -//! [`Entry::bind_runtime`] binds the runtime's files into the root once, for every execution -//! that enters it later, and returns the [`RuntimeBinds`], which a caller removes before it tears -//! the root down in the same mount namespace. [`Entry::prepare`] checks the inputs for one -//! execution, and reads and checks the account in existing-account mode; [`Command::enter`] then -//! has the child enter that execution, in launch stages of [`spawn`](super::spawn), in the order -//! of [`LaunchStage`]. Nothing here reads the kernel command line or a boot token, and nothing -//! creates an account. -//! -//! # The runtime's binds -//! -//! [`Entry::bind_runtime`] binds the file of a [generated](MachineId::Generated) machine ID over -//! the root's `/etc/machine-id`, the time status's directory at the root's `/run/nvx/time`, and -//! a [resolver](Entry::resolver) file, which names the guest's DNS servers, over the root's -//! `/etc/resolv.conf`, through the walk and attach of the mapper's `bind`: each is read-only, -//! `nosuid`, and `nodev` before anything sees it, and the walk creates what the root lacks only -//! on the root's own mount, through no symbolic link, and refuses a target of another kind than -//! its source. A symbolic link at `/etc/machine-id` or `/etc/resolv.conf` itself, which no bind -//! can cover, is replaced with an empty file first. In [image](MachineId::Image) mode, the -//! image's own `/etc/machine-id` is left alone, and without a resolver file, so is the image's -//! own `/etc/resolv.conf`. The time status's directory must hold only what any execution may -//! use, since one of user ID 0 can open every file in it that user ID 0 owns, and hold its lock: -//! the guest's time tools keep their locks and private records outside it. -//! -//! The binds are made in the caller's mount namespace, of which each execution's starts as a -//! copy, so every execution that enters the root later sees them, and sees their sources as they -//! change, such as a machine ID that the guest rewrites in place after a restore. They last as -//! long as that namespace, unless [`RuntimeBinds::remove`] removes them, in reverse order. A -//! root's teardown detaches no busy mount, so a caller that tears the root down in the namespace -//! where it bound them removes them first. A caller whose root is torn down after it ends, such -//! as an agent that an init outlives, may instead bind them in a mount namespace of its own, -//! which ends with it, and drop them. A bind that fails has the call remove the binds that it -//! made before it, and an [`Error::Rollback`] names any of them that stays mounted. A bind that a -//! rename moved while the walk ran, and that the attach failed to detach, fails the call, and -//! stays mounted wherever it went, as the mapper's `bind` does. -//! -//! # The execution -//! -//! The child starts with `clone3` in new mount, PID, and UTS namespaces: it is PID 1 of its own -//! PID namespace, with no process between it and its caller, so that its end ends every process -//! of the namespace. It then: -//! -//! 1. leads a new session, without a controlling terminal; -//! 2. waits at its [barrier](Command::barrier), if its command has one, until its caller has -//! placed it, then places its descriptors, and closes every other descriptor of its caller's -//! before it waits or places them, so that no descriptor of the old root, of the caller's -//! control session, or of another namespace or process reaches the program; -//! 3. joins its [cgroup](Command::cgroup), if its command names one; -//! 4. makes every mount of its mount namespace private, recursively, and sets its hostname; -//! 5. moves the root over `/`, after it makes every mount of the root's tree `nosuid` and -//! `nodev`. The root hides the former root, which no path reaches from it, `..` from the top -//! included, and the child never enters the root by `chroot` alone; -//! 6. mounts a procfs of its own PID namespace at `/proc`, `nosuid`, `nodev`, and `noexec`; -//! 7. mounts a read-only sysfs at `/sys`, `nosuid`, `nodev`, and `noexec`; -//! 8. mounts a tmpfs at `/dev`, `nosuid`, which holds only the allowlisted device nodes, the -//! links `fd`, `stdin`, `stdout`, `stderr`, and `ptmx`, a devpts instance of its own at -//! `/dev/pts`, `nosuid` and `noexec`, and a tmpfs at `/dev/shm`, `nosuid` and `nodev`. Every -//! other mount of the execution is `nodev`; -//! 9. takes its [identity](Identity), and sets `no_new_privs`; -//! 10. closes every descriptor that its stages used or opened but its program's, and enters its -//! working directory, `/` unless its command names another, with that identity, through no -//! magic link, so that neither a `/proc` link to a descriptor, a working directory, or a root, -//! nor a symbolic link to one, places it outside the root; -//! 11. gives every signal its default disposition, and blocks none, so that its program inherits -//! no signal that its caller ignores or blocks, and runs its program, which a command that -//! [searches a `PATH`](Command::search_path) looks for in the root. -//! -//! For user ID 0, `/proc/sys`, `/proc/sysrq-trigger`, `/proc/bus`, `/proc/fs`, and `/proc/irq` -//! are read-only, and `/proc/kcore`, `/proc/keys`, `/proc/timer_list`, `/proc/timer_stats`, -//! `/proc/latency_stats`, `/proc/sched_debug`, `/proc/acpi`, `/proc/asound`, `/proc/scsi`, and -//! `/sys/firmware` are masked by an empty, read-only file or directory; an entry that the kernel -//! lacks needs neither. -//! -//! The view creates what the root lacks of `/proc`, `/sys`, and `/dev` on the root's own mount, -//! through no symbolic link and inside no other mount, and attaches its mounts there; a symbolic -//! link or another mount on the way fails the stage. The guest's kernel has no user namespaces, -//! so the entry creates none. -//! -//! ```no_run -//! use std::os::unix::ffi::OsStrExt; -//! -//! use nvx::guest::entry::{Entry, Identity, MachineId}; -//! use nvx::guest::spawn::Command; -//! use nvx::protocol::cmdline; -//! -//! # fn main() -> Result<(), Box> { -//! let machine_id = MachineId::generate("/run/nvx/workload-machine-id")?; -//! let mut entry = Entry::new("/run/nvx/rootfs", Identity::new(65534, 65534)) -//! .hostname("sandbox") -//! .machine_id(machine_id) -//! .time_status("/run/nvx/time"); -//! // The guest's init wrote its resolver file from the DNS server that the command line names. -//! if cmdline::dns_server(&std::fs::read("/proc/cmdline")?)?.is_some() { -//! entry = entry.resolver("/etc/resolv.conf"); -//! } -//! // Once, before the first execution. -//! let binds = entry.bind_runtime()?; -//! let prepared = entry.prepare()?; -//! let home = prepared.account().map(|account| account.home().to_vec()); -//! let mut spawned = Command::new(c"/bin/true".to_owned(), vec![c"true".to_owned()]) -//! .barrier() -//! .enter(prepared) -//! .spawn()?; -//! let barrier = spawned.take_barrier().expect("the command has a barrier"); -//! // Place the child, by `spawned.id()`, in its cgroup; then: -//! barrier.release()?; -//! // Once the executions have ended, before the root's teardown in this mount namespace: -//! binds.remove()?; -//! # Ok(()) -//! # } -//! ``` - -use std::fmt; -use std::io; -use std::os::unix::ffi::OsStrExt; -use std::path::{Path, PathBuf}; - -use super::account::existing_account; -pub use super::account::{Account, AccountError, Database, MAX_ACCOUNT_DATABASE_BYTES}; -use super::mappings::{ - self, Kernel, Kind, Mapped, MappingError, SystemKernel, TargetRoot, TeardownError, -}; -use super::root::{TIME_STATUS_TARGET, bind_file_retained, bind_runtime_file}; -#[cfg(doc)] -use super::spawn::{Command, LaunchStage}; -use super::sys::{self, EntryChild, Mode, OpenFlags}; -use super::view::{MACHINE_ID_TARGET, RESOLVER_TARGET}; -#[cfg(doc)] -use crate::protocol::cmdline::write_resolver; - -pub use super::view::{DEFAULT_HOSTNAME, MAX_HOSTNAME_BYTES, MachineId}; - -/// The most supplementary groups that an identity holds: Linux's `NGROUPS_MAX`. -pub const MAX_GROUPS: usize = 65536; -/// The ID that `setresuid` and `setresgid` read as "unchanged", which no identity may hold. -const UNCHANGED_ID: u32 = u32::MAX; -/// The names of the Linux capabilities, by number. -const CAPABILITY_NAMES: [&str; 41] = [ - "CAP_CHOWN", - "CAP_DAC_OVERRIDE", - "CAP_DAC_READ_SEARCH", - "CAP_FOWNER", - "CAP_FSETID", - "CAP_KILL", - "CAP_SETGID", - "CAP_SETUID", - "CAP_SETPCAP", - "CAP_LINUX_IMMUTABLE", - "CAP_NET_BIND_SERVICE", - "CAP_NET_BROADCAST", - "CAP_NET_ADMIN", - "CAP_NET_RAW", - "CAP_IPC_LOCK", - "CAP_IPC_OWNER", - "CAP_SYS_MODULE", - "CAP_SYS_RAWIO", - "CAP_SYS_CHROOT", - "CAP_SYS_PTRACE", - "CAP_SYS_PACCT", - "CAP_SYS_ADMIN", - "CAP_SYS_BOOT", - "CAP_SYS_NICE", - "CAP_SYS_RESOURCE", - "CAP_SYS_TIME", - "CAP_SYS_TTY_CONFIG", - "CAP_MKNOD", - "CAP_LEASE", - "CAP_AUDIT_WRITE", - "CAP_AUDIT_CONTROL", - "CAP_SETFCAP", - "CAP_MAC_OVERRIDE", - "CAP_MAC_ADMIN", - "CAP_SYSLOG", - "CAP_WAKE_ALARM", - "CAP_BLOCK_SUSPEND", - "CAP_AUDIT_READ", - "CAP_PERFMON", - "CAP_BPF", - "CAP_CHECKPOINT_RESTORE", -]; - -/// A Linux capability, by its number. -/// -/// Any of the 64 that a capability set holds can be named, but an entry allows only those of -/// [`Capabilities::ALLOWED`], which have constants here. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] -pub struct Capability(u8); - -impl Capability { - /// `CAP_CHOWN`: changes the owners and groups of files. - pub const CHOWN: Self = Self(0); - /// `CAP_DAC_OVERRIDE`: bypasses file read, write, and execute permission checks. - pub const DAC_OVERRIDE: Self = Self(1); - /// `CAP_FOWNER`: bypasses the checks that require the caller to own a file. - pub const FOWNER: Self = Self(3); - /// `CAP_FSETID`: keeps the set-user-ID and set-group-ID bits of a file that changes. - pub const FSETID: Self = Self(4); - /// `CAP_KILL`: signals processes of other users. - pub const KILL: Self = Self(5); - /// `CAP_SETGID`: changes group IDs and supplementary groups. - pub const SETGID: Self = Self(6); - /// `CAP_SETUID`: changes user IDs. - pub const SETUID: Self = Self(7); - /// `CAP_SETPCAP`: drops capabilities from the bounding set, and changes securebits. - pub const SETPCAP: Self = Self(8); - /// `CAP_NET_BIND_SERVICE`: binds ports below the unprivileged port start. - pub const NET_BIND_SERVICE: Self = Self(10); - /// `CAP_NET_RAW`: uses raw and packet sockets. - pub const NET_RAW: Self = Self(13); - /// `CAP_SYS_CHROOT`: changes the root directory. - pub const SYS_CHROOT: Self = Self(18); - /// `CAP_AUDIT_WRITE`: writes records to the kernel's audit log. - pub const AUDIT_WRITE: Self = Self(29); - /// `CAP_SETFCAP`: sets file capabilities. - pub const SETFCAP: Self = Self(31); - - /// Returns the capability numbered `number`, or `None` above 63. - pub const fn new(number: u8) -> Option { - if number < 64 { - Some(Self(number)) - } else { - None - } - } - - /// Returns the capability's number. - pub const fn number(self) -> u8 { - self.0 - } - - /// Returns whether an entry allows the capability: whether - /// [`Capabilities::ALLOWED`] holds it. - pub const fn is_allowed(self) -> bool { - Capabilities::ALLOWED.contains(self) - } -} - -impl fmt::Display for Capability { - /// Writes the capability's name, such as `CAP_CHOWN`, or `capability N` for a number that - /// Linux does not name. - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - match CAPABILITY_NAMES.get(usize::from(self.0)) { - Some(name) => formatter.write_str(name), - None => write!(formatter, "capability {}", self.0), - } - } -} - -/// A set of capabilities. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -pub struct Capabilities(u64); - -impl Capabilities { - /// No capability. - pub const EMPTY: Self = Self(0); - /// The 13 capabilities that an entry allows: `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `FSETID`, - /// `KILL`, `SETGID`, `SETUID`, `SETPCAP`, `SETFCAP`, `NET_BIND_SERVICE`, `NET_RAW`, - /// `SYS_CHROOT`, and `AUDIT_WRITE`. - pub const ALLOWED: Self = Self::EMPTY - .with(Capability::CHOWN) - .with(Capability::DAC_OVERRIDE) - .with(Capability::FOWNER) - .with(Capability::FSETID) - .with(Capability::KILL) - .with(Capability::SETGID) - .with(Capability::SETUID) - .with(Capability::SETPCAP) - .with(Capability::SETFCAP) - .with(Capability::NET_BIND_SERVICE) - .with(Capability::NET_RAW) - .with(Capability::SYS_CHROOT) - .with(Capability::AUDIT_WRITE); - /// The capabilities that an entry refuses while listeners are - /// [published](Publication::Published): `NET_BIND_SERVICE`, which binds the ports from which - /// the guest accepts announcements, and `NET_RAW`, which forges packets from them. - pub const PUBLISHED_REFUSED: Self = Self::EMPTY - .with(Capability::NET_BIND_SERVICE) - .with(Capability::NET_RAW); - - /// Returns this set and `capability`. - pub const fn with(self, capability: Capability) -> Self { - Self(self.0 | 1 << capability.0) - } - - /// Returns this set without `capability`. - pub const fn without(self, capability: Capability) -> Self { - Self(self.0 & !(1 << capability.0)) - } - - /// Returns whether the set holds `capability`. - pub const fn contains(self, capability: Capability) -> bool { - self.0 & 1 << capability.0 != 0 - } - - /// Returns whether the set is empty. - pub const fn is_empty(self) -> bool { - self.0 == 0 - } - - /// Returns the capabilities of the set, in the order of their numbers. - pub fn iter(self) -> impl Iterator { - (0..64) - .map(Capability) - .filter(move |capability| self.contains(*capability)) - } - - /// Returns the set's bits: bit `n` for capability `n`. - pub const fn bits(self) -> u64 { - self.0 - } -} - -impl FromIterator for Capabilities { - fn from_iter>(capabilities: I) -> Self { - capabilities - .into_iter() - .fold(Self::EMPTY, |set, capability| set.with(capability)) - } -} - -/// How an entry checks its identity's account. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum AccountMode { - /// The root's `/etc/passwd` holds exactly one entry for the user ID, whose primary group is - /// the identity's group ID, which `/etc/group` holds, and whose home is absolute. Each - /// database is read below the root, through no symbolic link, and refused unless it is a - /// regular file of at most [`MAX_ACCOUNT_DATABASE_BYTES`]. - #[default] - Existing, - /// The IDs are used as given, without an account. - Numeric, -} - -/// The identity of an execution: its IDs, its capabilities, and how its account is checked. -/// -/// An identity with user ID 0 keeps exactly its [`Capabilities`], from the allowlist, in its -/// bounding, permitted, and effective sets, so that its program keeps them too, and nothing -/// else. Any other identity gets none: its bounding, permitted, and effective sets are empty. -/// Every identity has empty inheritable and ambient sets and `no_new_privs`, so no program that -/// it runs gains a privilege, set-user-ID ones included. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Identity { - /// The user ID. - uid: u32, - /// The group ID. - gid: u32, - /// The supplementary groups. - groups: Vec, - /// The capabilities, for user ID 0. - capabilities: Capabilities, - /// How the account is checked. - account: AccountMode, -} - -impl Identity { - /// Returns the identity of user ID `uid` and group ID `gid`, without supplementary groups or - /// capabilities, whose account an entry checks in [existing](AccountMode::Existing) mode. - pub fn new(uid: u32, gid: u32) -> Self { - Self { - uid, - gid, - groups: Vec::new(), - capabilities: Capabilities::EMPTY, - account: AccountMode::Existing, - } - } - - /// Gives the identity the supplementary groups `groups`. - pub fn groups(mut self, groups: impl Into>) -> Self { - self.groups = groups.into(); - self - } - - /// Gives the identity the capabilities `capabilities`, which only user ID 0 may hold. - pub fn capabilities(mut self, capabilities: Capabilities) -> Self { - self.capabilities = capabilities; - self - } - - /// Checks the identity's account in `mode`. - pub fn account(mut self, mode: AccountMode) -> Self { - self.account = mode; - self - } - - /// Returns the user ID. - pub fn uid(&self) -> u32 { - self.uid - } - - /// Returns the group ID. - pub fn gid(&self) -> u32 { - self.gid - } - - /// Returns the supplementary groups. - pub fn supplementary_groups(&self) -> &[u32] { - &self.groups - } - - /// Returns the capabilities. - pub fn capability_set(&self) -> Capabilities { - self.capabilities - } - - /// Returns how the account is checked. - pub fn account_mode(&self) -> AccountMode { - self.account - } -} - -/// Whether an execution's listeners are published, through private-network ingress or the -/// host's loopback, which decides whether the entry allows the capabilities of -/// [`Capabilities::PUBLISHED_REFUSED`]. -/// -/// The runtime owns `net.ipv4.ip_unprivileged_port_start`, which no execution can lower: its -/// `/proc/sys` is read-only for user ID 0, and no other user may write it. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -pub enum Publication { - /// Listeners are published, so the entry refuses `NET_BIND_SERVICE` and `NET_RAW`. - #[default] - Published, - /// No listener is published. - Unpublished, -} - -/// A device node that an execution's `/dev` may hold, which the character device of the same -/// name backs. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum DeviceNode { - /// `/dev/null`. - Null, - /// `/dev/zero`. - Zero, - /// `/dev/full`. - Full, - /// `/dev/random`. - Random, - /// `/dev/urandom`. - Urandom, - /// `/dev/tty`, the controlling terminal, of which the execution has none. - Tty, -} - -impl DeviceNode { - /// Every device node, which an execution's `/dev` holds by default. - pub const ALL: [Self; 6] = [ - Self::Null, - Self::Zero, - Self::Full, - Self::Random, - Self::Urandom, - Self::Tty, - ]; - - /// Returns the node's bit in the child's device set, in the order of the entry's device - /// table. - fn bit(self) -> u8 { - 1 << match self { - Self::Null => 0, - Self::Zero => 1, - Self::Full => 2, - Self::Random => 3, - Self::Urandom => 4, - Self::Tty => 5, - } - } -} - -/// How an adapter's executions enter its root: see the [module](self) documentation. -/// -/// An entry holds only the adapter's inputs, so that the adapter builds it once, binds its -/// runtime's files once, and [prepares](Self::prepare) it for each execution. -#[derive(Clone, Debug)] -pub struct Entry { - /// The root: the path of a mount. - root: PathBuf, - /// The identity. - identity: Identity, - /// The hostname. - hostname: Vec, - /// The machine ID mode. - machine_id: MachineId, - /// Whether listeners are published. - publication: Publication, - /// The directory of the time status, if the executions see one. - time_status: Option, - /// The resolver file, if the executions see one. - resolver: Option, - /// The device nodes of `/dev`, as [`DeviceNode::bit`]s. - devices: u8, -} - -impl Entry { - /// Describes executions that enter the mount at `root`, an absolute path, as `identity`, - /// with the hostname `nvx-sandbox`, the image's machine ID, published listeners, no time - /// status, the image's resolver file, and every [`DeviceNode`]. - pub fn new(root: impl Into, identity: Identity) -> Self { - Self { - root: root.into(), - identity, - hostname: DEFAULT_HOSTNAME.as_bytes().to_vec(), - machine_id: MachineId::Image, - publication: Publication::Published, - time_status: None, - resolver: None, - devices: DeviceNode::ALL - .iter() - .fold(0, |devices, node| devices | node.bit()), - } - } - - /// Gives the executions the hostname `hostname`: 1 to 64 bytes, without a NUL. - pub fn hostname(mut self, hostname: impl Into>) -> Self { - self.hostname = hostname.into(); - self - } - - /// Gives the executions the machine ID of `mode`: a generated one once - /// [`bind_runtime`](Self::bind_runtime) has bound it. - pub fn machine_id(mut self, mode: MachineId) -> Self { - self.machine_id = mode; - self - } - - /// Tells whether the executions' listeners are published. - pub fn publication(mut self, publication: Publication) -> Self { - self.publication = publication; - self - } - - /// Has the executions see the directory `source`, which holds the guest's time status, - /// read-only at `/run/nvx/time`, once [`bind_runtime`](Self::bind_runtime) has bound it. - /// - /// The directory must hold only what any execution may use: one of user ID 0 can open, and - /// lock, every file in it that user ID 0 owns. The guest's time tools publish only their - /// state and their daemon's process ID in `/run/nvx/time`, and keep their locks and private - /// records outside it. - pub fn time_status(mut self, source: impl Into) -> Self { - self.time_status = Some(source.into()); - self - } - - /// Has the executions see the regular file `source`, a resolver file in the format of - /// `/etc/resolv.conf` that names the guest's DNS servers, read-only at `/etc/resolv.conf`, - /// once [`bind_runtime`](Self::bind_runtime) has bound it. Without one, they see the image's - /// own `/etc/resolv.conf`. - /// - /// The file may be the one that the guest's init wrote from the `virtnet_dns=` token, or one - /// that [`write_resolver`] wrote. Every execution must be able to read it. The bind shows the - /// file as it is, so what its owner rewrites in place reaches every execution. - pub fn resolver(mut self, source: impl Into) -> Self { - self.resolver = Some(source.into()); - self - } - - /// Gives the executions' `/dev` exactly the device nodes `devices`. - pub fn devices(mut self, devices: impl IntoIterator) -> Self { - self.devices = devices - .into_iter() - .fold(0, |devices, node| devices | node.bit()); - self - } - - /// Returns the identity. - pub fn identity(&self) -> &Identity { - &self.identity - } - - /// Checks the entry's inputs, without reading anything: the root's path, the IDs, the - /// groups, the capabilities, and the hostname. - /// - /// The root's path must be absolute and hold no NUL, and no ID may be `u32::MAX`, which the - /// kernel reads as "unchanged". An identity holds at most [`MAX_GROUPS`] supplementary - /// groups. A capability outside [`Capabilities::ALLOWED`] is refused, as is any capability - /// of an identity whose user ID is not 0, and, while listeners are - /// [published](Publication::Published), `NET_BIND_SERVICE` and `NET_RAW`. - pub fn check(&self) -> Result<(), Error> { - let root = self.root.as_os_str().as_bytes(); - if root.first() != Some(&b'/') || root.contains(&0) { - return Err(Error::RootPath); - } - let identity = &self.identity; - if identity.uid == UNCHANGED_ID - || identity.gid == UNCHANGED_ID - || identity.groups.contains(&UNCHANGED_ID) - { - return Err(Error::InvalidId); - } - if identity.groups.len() > MAX_GROUPS { - return Err(Error::Groups); - } - if let Some(capability) = identity - .capabilities - .iter() - .find(|capability| !capability.is_allowed()) - { - return Err(Error::Capability(capability)); - } - if identity.uid != 0 && !identity.capabilities.is_empty() { - return Err(Error::NonRootCapabilities); - } - if self.publication == Publication::Published - && let Some(capability) = identity - .capabilities - .iter() - .find(|capability| Capabilities::PUBLISHED_REFUSED.contains(*capability)) - { - return Err(Error::Published(capability)); - } - if self.hostname.is_empty() - || self.hostname.len() > MAX_HOSTNAME_BYTES - || self.hostname.contains(&0) - { - return Err(Error::Hostname); - } - Ok(()) - } - - /// Prepares one execution: [checks](Self::check) the inputs, opens the root, and reads and - /// checks the account in [existing](AccountMode::Existing) mode, for [`Command::enter`]. - /// - /// The execution sees the runtime's binds that [`bind_runtime`](Self::bind_runtime) made in - /// the caller's mount namespace before it starts. - pub fn prepare(&self) -> Result { - self.check()?; - let root = sys::path_c_string(&self.root).map_err(|_| Error::RootPath)?; - let directory = sys::open( - &root, - OpenFlags::O_PATH | OpenFlags::O_DIRECTORY | OpenFlags::O_CLOEXEC, - Mode::empty(), - ) - .map_err(|error| Error::Root(error.into()))?; - let account = match self.identity.account { - AccountMode::Existing => Some( - existing_account(&directory, self.identity.uid, self.identity.gid) - .map_err(Error::Account)?, - ), - AccountMode::Numeric => None, - }; - Ok(Prepared { - child: EntryChild { - root, - hostname: self.hostname.clone().into_boxed_slice(), - devices: self.devices, - uid: self.identity.uid, - gid: self.identity.gid, - groups: self.identity.groups.clone().into_boxed_slice(), - capabilities: self.identity.capabilities.bits(), - #[cfg(test)] - simulated: None, - }, - account, - }) - } - - /// Binds the runtime's files into the root, in the caller's mount namespace, for every - /// execution that enters the root later, as the [module](self) describes, and returns the - /// binds that it made: the file of a [generated](MachineId::Generated) machine ID over - /// `/etc/machine-id`, then the [time status](Self::time_status) at `/run/nvx/time`, then the - /// [resolver](Self::resolver) file over `/etc/resolv.conf`. An entry with none of them binds - /// nothing. - /// - /// The binds last as long as the caller's mount namespace, unless [`RuntimeBinds::remove`] - /// removes them, as a caller that tears the root down in that namespace does first. If a bind - /// fails, the call removes the binds that it made before it, in reverse order, and returns - /// the failure, or, if any of them stays mounted, an [`Error::Rollback`] that names them too. - /// - /// Each call binds again, over the earlier binds, so an adapter calls it once, before its - /// first execution. It needs the privilege to mount, and Linux 6.8 or later. - pub fn bind_runtime(&self) -> Result { - self.bind_runtime_with(&mut SystemKernel) - } - - /// Implements [`bind_runtime`](Self::bind_runtime) with `kernel`. - pub(crate) fn bind_runtime_with(&self, kernel: &mut dyn Kernel) -> Result { - self.check()?; - if self.machine_id == MachineId::Image - && self.time_status.is_none() - && self.resolver.is_none() - { - return Ok(RuntimeBinds { - root: None, - binds: Vec::new(), - }); - } - let root = TargetRoot::open_with(kernel, &self.root).map_err(Error::Root)?; - let mut binds = Vec::new(); - if let Err(error) = self.bind_each(kernel, &root, &mut binds) { - // The caller cannot remove the binds before the failure, which a teardown in this - // namespace would trip over, so it learns of any that stays. - return Err(match mappings::unbind_with(kernel, &root, &binds) { - Ok(()) => error, - Err(teardown) => Error::Rollback { - error: Box::new(error), - teardown, - }, - }); - } - Ok(RuntimeBinds { - root: Some(root), - binds, - }) - } - - /// Makes the binds of [`bind_runtime`](Self::bind_runtime) beneath `root`, the root's mount, - /// in order, with `kernel`, and adds each to `binds`, until one fails. - fn bind_each( - &self, - kernel: &mut dyn Kernel, - root: &TargetRoot, - binds: &mut Vec, - ) -> Result<(), Error> { - if let MachineId::Generated(source) = &self.machine_id { - let bound = bind_file(kernel, root, &self.root, source, MACHINE_ID_TARGET); - binds.push(bound.map_err(Error::MachineId)?); - } - if let Some(source) = &self.time_status { - let bound = - bind_runtime_file(kernel, root, source, TIME_STATUS_TARGET, Kind::Directory); - binds.push(bound.map_err(Error::TimeStatus)?); - } - if let Some(source) = &self.resolver { - let bound = bind_file(kernel, root, &self.root, source, RESOLVER_TARGET); - binds.push(bound.map_err(Error::Resolver)?); - } - Ok(()) - } -} - -/// The runtime's binds that [`Entry::bind_runtime`] made in the caller's mount namespace, in the -/// order in which it made them. -/// -/// A caller that tears the root down in that namespace [removes](Self::remove) them first: a -/// teardown detaches no busy mount, and the root's mount is busy while they are on it. A caller -/// whose namespace ends before the teardown, such as an agent that an init outlives, may drop -/// them instead, which leaves them mounted. Until they are removed or dropped, they hold a -/// descriptor of the root, which keeps its mount busy too. -#[derive(Debug)] -pub struct RuntimeBinds { - /// The root beneath which the binds were made, unless there are none. - root: Option, - /// The binds, in the order in which they were made. - binds: Vec, -} - -impl RuntimeBinds { - /// Removes the binds, in reverse order: unmounts each that is still the topmost mount at its - /// target, by its name in its parent directory, without detaching a busy one, which needs - /// procfs at `/proc`. It attempts each, whatever failed before, and reports every one that it - /// failed to unmount, which stays mounted. - pub fn remove(self) -> Result<(), TeardownError> { - self.remove_with(&mut SystemKernel) - } - - /// Implements [`remove`](Self::remove) with `kernel`. - pub(crate) fn remove_with(self, kernel: &mut dyn Kernel) -> Result<(), TeardownError> { - match &self.root { - Some(root) => mappings::unbind_with(kernel, root, &self.binds), - None => Ok(()), - } - } -} - -/// Binds the regular file at `source` read-only over `target`, a file of the root's `/etc`, -/// beneath `root`, the mount at `path`, with `kernel`, and returns the bind. A symbolic link at -/// `target` itself, which no bind can cover, is first replaced with an empty file, which the bind -/// creates. -fn bind_file( - kernel: &mut dyn Kernel, - root: &TargetRoot, - path: &Path, - source: &Path, - target: &str, -) -> Result { - bind_file_retained(kernel, root, path, source, target).map_err(|failure| failure.error) -} - -/// One execution that an [`Entry`] prepared, which [`Command::enter`] takes. -#[derive(Debug)] -pub struct Prepared { - /// What the child needs. - child: EntryChild, - /// The account that existing-account mode found. - account: Option, -} - -impl Prepared { - /// Returns the identity's account, in [existing](AccountMode::Existing) mode. - pub fn account(&self) -> Option<&Account> { - self.account.as_ref() - } - - /// Returns what the child needs. - pub(crate) fn into_child(self) -> EntryChild { - self.child - } -} - -/// Why an entry refused to prepare an execution. -#[derive(Debug)] -#[non_exhaustive] -pub enum Error { - /// The root's path is not absolute, or holds a NUL. - RootPath, - /// The root cannot be opened as a directory, or, for [`Entry::bind_runtime`], is not the root - /// of a mount. - Root(io::Error), - /// The user ID, the group ID, or a supplementary group is `u32::MAX`, which the kernel reads - /// as "unchanged". - InvalidId, - /// The identity has more than [`MAX_GROUPS`] supplementary groups. - Groups, - /// The capability is outside [`Capabilities::ALLOWED`]. - Capability(Capability), - /// The identity's user ID is not 0, and it has capabilities. - NonRootCapabilities, - /// The capability is refused while listeners are published. - Published(Capability), - /// The hostname is empty, longer than [`MAX_HOSTNAME_BYTES`], or holds a NUL. - Hostname, - /// Existing-account mode refused the identity. - Account(AccountError), - /// [`Entry::bind_runtime`] cannot bind the machine ID's file over the root's - /// `/etc/machine-id`: it is missing or not a regular file, the target cannot hold it, or the - /// caller cannot mount. - MachineId(MappingError), - /// [`Entry::bind_runtime`] cannot bind the time status's directory at the root's - /// `/run/nvx/time`. - TimeStatus(MappingError), - /// [`Entry::bind_runtime`] cannot bind the resolver file over the root's `/etc/resolv.conf`: - /// it is missing or not a regular file, the target cannot hold it, or the caller cannot - /// mount. - Resolver(MappingError), - /// [`Entry::bind_runtime`] failed to make a bind, and to remove some of the binds that it made - /// before it, which stay mounted, so that a teardown of the root in the caller's mount - /// namespace fails. - Rollback { - /// Why the bind failed: an [`Error::MachineId`], [`Error::TimeStatus`], or - /// [`Error::Resolver`]. - error: Box, - /// The binds made before it that stay mounted. - teardown: TeardownError, - }, -} - -impl fmt::Display for Error { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::RootPath => formatter.write_str("the root's path is not absolute"), - Self::Root(error) => write!(formatter, "cannot open the root: {error}"), - Self::InvalidId => formatter.write_str("an ID of the identity is invalid"), - Self::Groups => write!( - formatter, - "the identity has more than {MAX_GROUPS} supplementary groups" - ), - Self::Capability(capability) => write!(formatter, "{capability} is not allowed"), - Self::NonRootCapabilities => { - formatter.write_str("an identity other than user ID 0 has capabilities") - } - Self::Published(capability) => write!( - formatter, - "{capability} is not allowed while listeners are published" - ), - Self::Hostname => formatter.write_str("the hostname is invalid"), - Self::Account(error) => write!(formatter, "the account is unavailable: {error}"), - Self::MachineId(error) => write!(formatter, "cannot bind the machine ID: {error}"), - Self::TimeStatus(error) => write!(formatter, "cannot bind the time status: {error}"), - Self::Resolver(error) => write!(formatter, "cannot bind the resolver file: {error}"), - Self::Rollback { error, teardown } => write!( - formatter, - "{error}; the binds made before it stay mounted: {teardown}" - ), - } - } -} - -impl std::error::Error for Error { - fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { - match self { - Self::Root(error) => Some(error), - Self::MachineId(error) | Self::TimeStatus(error) | Self::Resolver(error) => Some(error), - Self::Account(error) => Some(error), - Self::Rollback { error, .. } => Some(error.as_ref()), - _ => None, - } - } -} - -#[cfg(test)] -mod tests; diff --git a/nvx-rs/src/guest/entry/tests.rs b/nvx-rs/src/guest/entry/tests.rs deleted file mode 100644 index 9821173c9..000000000 --- a/nvx-rs/src/guest/entry/tests.rs +++ /dev/null @@ -1,1459 +0,0 @@ -//! Tests of the entry's checks, accounts, machine IDs, runtime binds, and launch stages. -//! -//! The launch tests run each entry without privileges: the child starts without new namespaces -//! and skips the system calls of every entry stage, but fails the one that a test names, so that -//! each stage reports its own failure. The runtime binds' tests run on the root operations' fake -//! kernel, which simulates mounts by their unique IDs. The tests whose names start with -//! `privileged_` run the entry, and the runtime's binds, for real, and only where the test -//! process may mount, as root does. - -use std::ffi::CString; -use std::io::Read; -use std::os::unix::ffi::OsStrExt; -use std::os::unix::fs::PermissionsExt; - -use super::*; -use crate::guest::account::tests::{GROUP_DATABASE, PASSWD_DATABASE}; -use crate::guest::mappings::{Access, Step}; -use crate::guest::root::tests::{Call, FakeKernel, Operation, attach_calls}; -use crate::guest::spawn::{Command, LaunchFailure, LaunchStage}; -use crate::guest::sys::tests::Simulation; -use crate::guest::sys::{ChildStatus, Errno}; -use crate::protocol::cmdline::write_resolver; - -/// Returns an entry of a root at `/run/nvx/rootfs` as `identity`, whose account is numeric, so -/// that checking it reads nothing. -fn entry(identity: Identity) -> Entry { - Entry::new("/run/nvx/rootfs", identity.account(AccountMode::Numeric)) -} - -/// Returns a temporary root whose `etc` holds `passwd` and `group`, when given. -fn root_with(passwd: Option<&[u8]>, group: Option<&[u8]>) -> tempfile::TempDir { - let root = tempfile::tempdir().unwrap(); - let etc = root.path().join("etc"); - std::fs::create_dir(&etc).unwrap(); - if let Some(passwd) = passwd { - std::fs::write(etc.join("passwd"), passwd).unwrap(); - } - if let Some(group) = group { - std::fs::write(etc.join("group"), group).unwrap(); - } - root -} - -/// Returns the account error with which an entry of `root` refuses `identity`. -fn account_error(root: &Path, identity: Identity) -> AccountError { - match Entry::new(root, identity).prepare() { - Err(Error::Account(error)) => error, - other => panic!("expected an account error, not {other:?}"), - } -} - -#[test] -fn the_allowlist_holds_the_thirteen_capabilities() { - let numbers: Vec = Capabilities::ALLOWED - .iter() - .map(Capability::number) - .collect(); - assert_eq!(numbers, [0, 1, 3, 4, 5, 6, 7, 8, 10, 13, 18, 29, 31]); - let names: Vec = Capabilities::ALLOWED - .iter() - .map(|capability| capability.to_string()) - .collect(); - assert_eq!( - names, - [ - "CAP_CHOWN", - "CAP_DAC_OVERRIDE", - "CAP_FOWNER", - "CAP_FSETID", - "CAP_KILL", - "CAP_SETGID", - "CAP_SETUID", - "CAP_SETPCAP", - "CAP_NET_BIND_SERVICE", - "CAP_NET_RAW", - "CAP_SYS_CHROOT", - "CAP_AUDIT_WRITE", - "CAP_SETFCAP", - ] - ); - assert_eq!(Capability::new(21).unwrap().to_string(), "CAP_SYS_ADMIN"); - assert_eq!( - Capability::new(40).unwrap().to_string(), - "CAP_CHECKPOINT_RESTORE" - ); - assert_eq!(Capability::new(41).unwrap().to_string(), "capability 41"); - assert_eq!(Capability::new(63).map(Capability::number), Some(63)); - assert_eq!(Capability::new(64), None); - assert!(Capability::SYS_CHROOT.is_allowed()); - assert!(!Capability::new(27).unwrap().is_allowed(), "CAP_MKNOD"); -} - -#[test] -fn capability_sets_add_remove_and_list_their_members() { - let set = Capabilities::EMPTY - .with(Capability::SETFCAP) - .with(Capability::CHOWN) - .with(Capability::CHOWN); - assert_eq!(set.bits(), 1 | 1 << 31); - assert!(set.contains(Capability::CHOWN) && set.contains(Capability::SETFCAP)); - assert!(!set.contains(Capability::KILL)); - assert_eq!( - set.iter().collect::>(), - [Capability::CHOWN, Capability::SETFCAP] - ); - assert_eq!(set.without(Capability::CHOWN).bits(), 1 << 31); - assert_eq!(set.without(Capability::KILL), set); - assert!( - set.without(Capability::CHOWN) - .without(Capability::SETFCAP) - .is_empty() - ); - let highest = Capability::new(63).unwrap(); - assert_eq!(Capabilities::EMPTY.with(highest).bits(), 1 << 63); - assert_eq!( - [Capability::KILL, highest] - .into_iter() - .collect::(), - Capabilities::EMPTY.with(Capability::KILL).with(highest) - ); - assert_eq!(Capabilities::default(), Capabilities::EMPTY); - assert_eq!( - Capabilities::PUBLISHED_REFUSED.iter().collect::>(), - [Capability::NET_BIND_SERVICE, Capability::NET_RAW] - ); -} - -#[test] -fn every_capability_outside_the_allowlist_is_refused() { - for number in 0..64 { - let capability = Capability::new(number).unwrap(); - let identity = Identity::new(0, 0).capabilities(Capabilities::EMPTY.with(capability)); - let result = entry(identity) - .publication(Publication::Unpublished) - .check(); - if Capabilities::ALLOWED.contains(capability) { - assert!(result.is_ok(), "{capability}: {result:?}"); - } else { - assert!( - matches!(result, Err(Error::Capability(refused)) if refused == capability), - "{capability}: {result:?}" - ); - } - } - // The first refused capability is reported, whatever else the set holds. - let identity = Identity::new(0, 0).capabilities( - Capabilities::ALLOWED - .with(Capability::new(21).unwrap()) - .with(Capability::new(27).unwrap()), - ); - let error = entry(identity).check().unwrap_err(); - assert!(matches!(error, Error::Capability(capability) if capability.number() == 21)); - assert_eq!(error.to_string(), "CAP_SYS_ADMIN is not allowed"); -} - -#[test] -fn only_user_id_zero_may_hold_capabilities() { - for uid in [1, 1000, 65534] { - let identity = - Identity::new(uid, 0).capabilities(Capabilities::EMPTY.with(Capability::KILL)); - assert!(matches!( - entry(identity).check(), - Err(Error::NonRootCapabilities) - )); - assert!(entry(Identity::new(uid, 0)).check().is_ok()); - } - // Root with no capability is a valid identity, and so is root with all thirteen while no - // listener is published. - assert!(entry(Identity::new(0, 0)).check().is_ok()); - let identity = Identity::new(0, 0).capabilities(Capabilities::ALLOWED); - assert!( - entry(identity) - .publication(Publication::Unpublished) - .check() - .is_ok() - ); - // A refused capability outranks the non-root rule. - let identity = - Identity::new(1000, 0).capabilities(Capabilities::EMPTY.with(Capability::new(21).unwrap())); - assert!(matches!(entry(identity).check(), Err(Error::Capability(_)))); -} - -#[test] -fn published_listeners_refuse_binding_and_raw_sockets() { - for capability in [Capability::NET_BIND_SERVICE, Capability::NET_RAW] { - let identity = Identity::new(0, 0) - .capabilities(Capabilities::EMPTY.with(capability).with(Capability::CHOWN)); - let published = entry(identity.clone()); - assert_eq!(published.publication, Publication::Published); - let error = published.check().unwrap_err(); - assert!( - matches!(error, Error::Published(refused) if refused == capability), - "{error:?}" - ); - assert_eq!( - error.to_string(), - format!("{capability} is not allowed while listeners are published") - ); - assert!( - entry(identity) - .publication(Publication::Unpublished) - .check() - .is_ok() - ); - } - let identity = Identity::new(0, 0).capabilities( - Capabilities::ALLOWED - .without(Capability::NET_BIND_SERVICE) - .without(Capability::NET_RAW), - ); - assert!(entry(identity).check().is_ok()); -} - -#[test] -fn invalid_roots_ids_groups_and_hostnames_are_refused() { - let check = |entry: Entry| entry.check(); - assert!(matches!( - check(Entry::new("relative", Identity::new(1, 1))), - Err(Error::RootPath) - )); - assert!(matches!( - check(Entry::new( - std::ffi::OsStr::from_bytes(b"/root\0"), - Identity::new(1, 1) - )), - Err(Error::RootPath) - )); - for identity in [ - Identity::new(u32::MAX, 1), - Identity::new(1, u32::MAX), - Identity::new(1, 1).groups([2, u32::MAX]), - ] { - assert!(matches!(check(entry(identity)), Err(Error::InvalidId))); - } - let groups: Vec = (0..u32::try_from(MAX_GROUPS).unwrap()).collect(); - assert!(check(entry(Identity::new(1, 1).groups(groups.clone()))).is_ok()); - let mut too_many = groups; - too_many.push(1 << 20); - assert!(matches!( - check(entry(Identity::new(1, 1).groups(too_many))), - Err(Error::Groups) - )); - for hostname in [b"".to_vec(), vec![b'h'; 65], b"host\0name".to_vec()] { - assert!(matches!( - check(entry(Identity::new(1, 1)).hostname(hostname)), - Err(Error::Hostname) - )); - } - assert!(check(entry(Identity::new(1, 1)).hostname(vec![b'h'; 64])).is_ok()); - assert_eq!( - entry(Identity::new(1, 1)).hostname, - DEFAULT_HOSTNAME.as_bytes() - ); -} - -#[test] -fn existing_accounts_are_read_below_the_root() { - let root = root_with(Some(PASSWD_DATABASE), Some(GROUP_DATABASE)); - let prepared = Entry::new(root.path(), Identity::new(1000, 1001)) - .prepare() - .unwrap(); - let account = prepared.account().unwrap(); - assert_eq!( - (account.name(), account.home()), - (b"workload".as_slice(), b"/home/workload".as_slice()) - ); - let child = prepared.into_child(); - assert_eq!(child.root.as_bytes(), root.path().as_os_str().as_bytes()); - assert_eq!((child.uid, child.gid, child.capabilities), (1000, 1001, 0)); - assert_eq!(child.devices, 0b11_1111); - assert_eq!(&*child.hostname, DEFAULT_HOSTNAME.as_bytes()); - - assert!(matches!( - account_error(root.path(), Identity::new(1000, 1000)), - AccountError::PrimaryGroup(1001) - )); - assert!(matches!( - account_error(root.path(), Identity::new(2000, 2000)), - AccountError::Missing - )); - let missing_group = root_with(Some(PASSWD_DATABASE), Some(b"root:x:0:\n")); - assert!(matches!( - account_error(missing_group.path(), Identity::new(1000, 1001)), - AccountError::Group - )); - // Numeric mode reads no database. - let empty = root_with(None, None); - let prepared = Entry::new( - empty.path(), - Identity::new(1000, 1001).account(AccountMode::Numeric), - ) - .prepare() - .unwrap(); - assert!(prepared.account().is_none()); - assert!(matches!( - account_error(empty.path(), Identity::new(1000, 1001)), - AccountError::Read(Database::Passwd, error) - if error.raw_os_error() == Some(Errno::ENOENT as i32) - )); - let no_group = root_with(Some(PASSWD_DATABASE), None); - assert!(matches!( - account_error(no_group.path(), Identity::new(1000, 1001)), - AccountError::Read(Database::Group, _) - )); -} - -#[test] -fn account_databases_are_bounded_regular_files_reached_through_no_link() { - let identity = || Identity::new(1000, 1001); - // A symbolic link, as the database or on its path, fails the read, even one that leads to a - // valid database inside the root. - let root = root_with(None, Some(GROUP_DATABASE)); - std::fs::write(root.path().join("passwd"), PASSWD_DATABASE).unwrap(); - std::os::unix::fs::symlink("../passwd", root.path().join("etc/passwd")).unwrap(); - assert!(matches!( - account_error(root.path(), identity()), - AccountError::Read(Database::Passwd, error) - if error.raw_os_error() == Some(Errno::ELOOP as i32) - )); - let root = tempfile::tempdir().unwrap(); - let real = root_with(Some(PASSWD_DATABASE), Some(GROUP_DATABASE)); - std::os::unix::fs::symlink(real.path().join("etc"), root.path().join("etc")).unwrap(); - assert!(matches!( - account_error(root.path(), identity()), - AccountError::Read(Database::Passwd, error) - if error.raw_os_error() == Some(Errno::ELOOP as i32) - )); - // A directory or a FIFO is no database, and the FIFO does not block the read. - let root = root_with(None, Some(GROUP_DATABASE)); - std::fs::create_dir(root.path().join("etc/passwd")).unwrap(); - assert!(matches!( - account_error(root.path(), identity()), - AccountError::NotFile(Database::Passwd) - )); - let root = root_with(Some(PASSWD_DATABASE), None); - let fifo = sys::path_c_string(&root.path().join("etc/group")).unwrap(); - sys::tests::create_fifo(&fifo, Mode::from_bits_retain(0o644)).unwrap(); - assert!(matches!( - account_error(root.path(), identity()), - AccountError::NotFile(Database::Group) - )); - // A database of exactly the bound is read, and one byte more is refused. - let mut database = PASSWD_DATABASE.to_vec(); - database.resize(usize::try_from(MAX_ACCOUNT_DATABASE_BYTES).unwrap(), b'\n'); - let root = root_with(Some(&database), Some(GROUP_DATABASE)); - assert!(Entry::new(root.path(), identity()).prepare().is_ok()); - database.push(b'\n'); - std::fs::write(root.path().join("etc/passwd"), &database).unwrap(); - assert!(matches!( - account_error(root.path(), identity()), - AccountError::TooLarge(Database::Passwd) - )); -} - -#[test] -fn an_entry_needs_a_root_directory() { - let file = tempfile::NamedTempFile::new().unwrap(); - let result = Entry::new( - "/nonexistent/root", - Identity::new(1, 1).account(AccountMode::Numeric), - ) - .prepare(); - assert!(matches!(result, Err(Error::Root(_))), "{result:?}"); - let result = Entry::new( - file.path(), - Identity::new(1, 1).account(AccountMode::Numeric), - ) - .prepare(); - assert!( - matches!(&result, Err(Error::Root(error)) - if error.raw_os_error() == Some(Errno::ENOTDIR as i32)), - "{result:?}" - ); -} - -#[test] -fn runtime_binds_need_a_mount_root_unless_there_are_none() { - let root = root_with(None, None); - let identity = || Identity::new(1, 1).account(AccountMode::Numeric); - // The image's machine ID and resolver file, and no time status, bind nothing, so any root - // does, and there is nothing to remove. - let binds = Entry::new(root.path(), identity()).bind_runtime().unwrap(); - assert!(binds.root.is_none() && binds.binds.is_empty()); - binds.remove().unwrap(); - let machine_id = MachineId::generate(root.path().join("machine-id")).unwrap(); - let result = Entry::new(root.path(), identity()) - .machine_id(machine_id) - .bind_runtime(); - assert!( - matches!(&result, Err(Error::Root(error)) if error.kind() == io::ErrorKind::InvalidInput), - "{result:?}" - ); - let result = Entry::new(root.path(), identity()) - .time_status(root.path().join("etc")) - .bind_runtime(); - assert!(matches!(result, Err(Error::Root(_))), "{result:?}"); - let result = Entry::new(root.path(), identity()) - .resolver(root.path().join("machine-id")) - .bind_runtime(); - assert!(matches!(result, Err(Error::Root(_))), "{result:?}"); - // The inputs are checked first. - let result = Entry::new(root.path(), identity()) - .hostname("") - .time_status(root.path().join("etc")) - .bind_runtime(); - assert!(matches!(result, Err(Error::Hostname)), "{result:?}"); -} - -/// A root, `root`, on a mount of the fake kernel's, with an empty `etc`, and the sources of the -/// runtime's files, in `sources`: a generated machine ID, `machine-id`, the time status's -/// directory, `time`, and a resolver file that names `10.0.0.1`, `resolv.conf`, below a -/// temporary directory. -struct RuntimeFixture { - /// The temporary directory, which the fixture holds only so that dropping it removes it. - _directory: tempfile::TempDir, - /// The root. - root: PathBuf, - /// The directory of the sources. - sources: PathBuf, - /// The fake kernel. - kernel: FakeKernel, -} - -impl RuntimeFixture { - fn new() -> Self { - let directory = tempfile::tempdir().unwrap(); - let base = directory.path().canonicalize().unwrap(); - let root = base.join("root"); - let sources = base.join("sources"); - std::fs::create_dir_all(root.join("etc")).unwrap(); - std::fs::create_dir_all(sources.join("time")).unwrap(); - MachineId::generate(sources.join("machine-id")).unwrap(); - let resolver = std::fs::File::create(sources.join("resolv.conf")).unwrap(); - write_resolver(resolver, "10.0.0.1".parse().unwrap()).unwrap(); - let kernel = FakeKernel::default(); - kernel.add_mount(&root); - Self { - _directory: directory, - root, - sources, - kernel, - } - } - - /// Returns the entry of the root, of user ID 0 in numeric-account mode, with every runtime - /// file of the sources. - fn entry(&self) -> Entry { - Entry::new( - &self.root, - Identity::new(0, 0).account(AccountMode::Numeric), - ) - .machine_id(MachineId::Generated(self.sources.join("machine-id"))) - .time_status(self.sources.join("time")) - .resolver(self.sources.join("resolv.conf")) - } - - /// Returns the mount points of the runtime's targets in the root, in the order of the binds. - fn points(&self) -> [PathBuf; 3] { - ["etc/machine-id", "run/nvx/time", "etc/resolv.conf"].map(|target| self.root.join(target)) - } - - /// Returns the mount points as the fake kernel records them, in reverse order. - fn removals(&self, points: &[PathBuf]) -> Vec { - points - .iter() - .rev() - .map(|point| point.display().to_string()) - .collect() - } -} - -#[test] -fn runtime_binds_are_made_in_order_and_removed_in_reverse() { - let fixture = RuntimeFixture::new(); - // The image's resolver file is a link, as a resolver daemon's image has, which no bind - // covers; it gives way to an empty file, and its target is left alone. - let outside = fixture.sources.join("stub-resolv.conf"); - std::fs::write(&outside, b"nameserver 127.0.0.53\n").unwrap(); - std::os::unix::fs::symlink(&outside, fixture.root.join("etc/resolv.conf")).unwrap(); - let binds = fixture - .entry() - .bind_runtime_with(&mut fixture.kernel.clone()) - .unwrap(); - let points = fixture.points(); - let expected: Vec = ["machine-id", "time", "resolv.conf"] - .iter() - .zip(&points) - .flat_map(|(source, point)| { - attach_calls(&fixture.sources.join(source), point, Access::ReadOnly) - }) - .collect(); - assert_eq!(fixture.kernel.calls(), expected); - for (bind, point) in binds.binds.iter().zip(&points) { - assert_eq!(fixture.root.join(bind.target()), *point); - assert!(fixture.kernel.mount_at(point).is_some(), "{point:?}"); - } - let replaced = std::fs::symlink_metadata(fixture.root.join("etc/resolv.conf")).unwrap(); - assert!(replaced.is_file() && replaced.len() == 0); - assert_eq!(replaced.permissions().mode() & 0o7777, 0o644); - assert_eq!(std::fs::read(&outside).unwrap(), b"nameserver 127.0.0.53\n"); - // Their removal takes them off, last first. - binds.remove_with(&mut fixture.kernel.clone()).unwrap(); - assert_eq!(fixture.kernel.unmounts(), fixture.removals(&points)); - assert!( - points - .iter() - .all(|point| fixture.kernel.mount_at(point).is_none()) - ); -} - -#[test] -fn a_failed_runtime_bind_removes_the_binds_before_it() { - // A missing resolver file, after the machine ID and the time status. - let fixture = RuntimeFixture::new(); - let points = fixture.points(); - let result = fixture - .entry() - .resolver(fixture.sources.join("missing")) - .bind_runtime_with(&mut fixture.kernel.clone()); - assert!( - matches!( - &result, - Err(Error::Resolver(MappingError::System { step: Step::Source, error })) - if *error == Errno::ENOENT as i32 - ), - "{result:?}" - ); - assert_eq!(fixture.kernel.unmounts(), fixture.removals(&points[..2])); - assert!( - points - .iter() - .all(|point| fixture.kernel.mount_at(point).is_none()) - ); - // A target of another kind for the time status, after the machine ID. - let fixture = RuntimeFixture::new(); - let points = fixture.points(); - std::fs::create_dir_all(fixture.root.join("run/nvx")).unwrap(); - std::fs::write(fixture.root.join("run/nvx/time"), b"").unwrap(); - let result = fixture - .entry() - .bind_runtime_with(&mut fixture.kernel.clone()); - assert!( - matches!(result, Err(Error::TimeStatus(MappingError::TargetKind))), - "{result:?}" - ); - assert_eq!(fixture.kernel.unmounts(), fixture.removals(&points[..1])); - assert_eq!(fixture.kernel.mount_at(&points[0]), None); - // A failure of the first bind leaves nothing to remove. - let fixture = RuntimeFixture::new(); - std::fs::create_dir(fixture.root.join("etc/machine-id")).unwrap(); - let result = fixture - .entry() - .bind_runtime_with(&mut fixture.kernel.clone()); - assert!( - matches!(result, Err(Error::MachineId(MappingError::TargetKind))), - "{result:?}" - ); - assert!(fixture.kernel.calls().is_empty()); -} - -#[test] -fn a_failed_runtime_bind_names_the_binds_before_it_that_stay() { - // A missing resolver file, after the machine ID and the time status, whose bind is busy. - let fixture = RuntimeFixture::new(); - let points = fixture.points(); - fixture - .kernel - .fail(Operation::Unmount, &points[1], Errno::EBUSY); - let error = fixture - .entry() - .resolver(fixture.sources.join("missing")) - .bind_runtime_with(&mut fixture.kernel.clone()) - .unwrap_err(); - let Error::Rollback { - error: failure, - teardown, - } = &error - else { - panic!("{error:?}"); - }; - assert!( - matches!( - failure.as_ref(), - Error::Resolver(MappingError::System { step: Step::Source, error: code }) - if *code == Errno::ENOENT as i32 - ), - "{failure:?}" - ); - let stayed: Vec<&Path> = teardown - .failures() - .iter() - .map(|stayed| stayed.target()) - .collect(); - assert_eq!(stayed, [Path::new("/run/nvx/time")]); - assert_eq!( - error.to_string(), - format!("{failure}; the binds made before it stay mounted: {teardown}") - ); - assert!(std::error::Error::source(&error).is_some()); - // The rollback attempts each bind, so the machine ID's is removed all the same. - assert_eq!(fixture.kernel.unmounts(), fixture.removals(&points[..2])); - assert!(fixture.kernel.mount_at(&points[1]).is_some()); - assert!(fixture.kernel.mount_at(&points[0]).is_none()); -} - -#[test] -fn a_runtime_bind_that_stays_is_reported() { - let fixture = RuntimeFixture::new(); - let binds = fixture - .entry() - .bind_runtime_with(&mut fixture.kernel.clone()) - .unwrap(); - let points = fixture.points(); - fixture - .kernel - .fail(Operation::Unmount, &points[2], Errno::EBUSY); - let error = binds.remove_with(&mut fixture.kernel.clone()).unwrap_err(); - assert_eq!( - error.to_string(), - format!( - "failed to unmount the mapping at /etc/resolv.conf: failed to unmount the mapping: {}", - io::Error::from_raw_os_error(Errno::EBUSY as i32) - ) - ); - // The others are removed all the same. - assert_eq!(fixture.kernel.unmounts(), fixture.removals(&points)); - assert!(fixture.kernel.mount_at(&points[2]).is_some()); - assert!(fixture.kernel.mount_at(&points[0]).is_none()); -} - -#[test] -fn device_sets_select_their_nodes() { - let entry = entry(Identity::new(1, 1)); - assert_eq!(entry.devices, 0b11_1111); - assert_eq!(entry.clone().devices([]).devices, 0); - assert_eq!( - entry - .devices([DeviceNode::Null, DeviceNode::Tty, DeviceNode::Null]) - .devices, - 0b10_0001 - ); -} - -/// Returns an execution, prepared without privileges, whose child skips every entry stage's -/// system calls and fails `stage` with `error`. -fn simulated(stage: LaunchStage, error: Errno) -> Prepared { - Prepared { - child: EntryChild { - root: c"/".to_owned(), - hostname: b"simulated".to_vec().into_boxed_slice(), - devices: 0, - uid: 0, - gid: 0, - groups: Box::new([]), - capabilities: 0, - simulated: Some(Simulation::Fail(stage, error)), - }, - account: None, - } -} - -/// Returns a command that runs the shell script `script`. -fn shell(script: &str) -> Command { - Command::new( - c"/bin/sh".to_owned(), - vec![ - c"sh".to_owned(), - c"-c".to_owned(), - CString::new(script).unwrap(), - ], - ) -} - -/// Spawns `command`, which waits at a barrier, releases it, and returns the child's launch -/// failure, status, and standard output. -fn run_released(command: Command) -> (Option, Option, Vec) { - let (mut output, writer) = std::io::pipe().unwrap(); - let mut spawned = command.barrier().stdout(writer).spawn().unwrap(); - let process = spawned.process(); - spawned.take_barrier().unwrap().release().unwrap(); - let failure = spawned.into_report().wait().unwrap(); - let mut bytes = Vec::new(); - output.read_to_end(&mut bytes).unwrap(); - (failure, sys::wait(process), bytes) -} - -#[test] -fn each_entry_stage_reports_its_own_failure_and_runs_nothing() { - let directory = tempfile::tempdir().unwrap(); - let marker = directory.path().join("ran"); - let script = format!("touch '{}'", marker.display()); - let stages = [ - (LaunchStage::Propagation, Errno::EINVAL), - (LaunchStage::Hostname, Errno::EPERM), - (LaunchStage::MoveRoot, Errno::EBUSY), - (LaunchStage::Proc, Errno::ENODEV), - (LaunchStage::Sys, Errno::EACCES), - (LaunchStage::Devices, Errno::ENOSPC), - (LaunchStage::Bounding, Errno::EPERM), - (LaunchStage::Identity, Errno::EINVAL), - (LaunchStage::Capabilities, Errno::EPERM), - (LaunchStage::NoNewPrivileges, Errno::EINVAL), - ]; - for (stage, error) in stages { - let (failure, status, _) = run_released(shell(&script).enter(simulated(stage, error))); - let failure = failure.unwrap_or_else(|| panic!("{stage} did not fail")); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (stage, error as i32) - ); - assert_eq!(status, Some(ChildStatus::Exited(125)), "{stage}"); - assert!(!marker.exists(), "{stage} ran the program"); - assert_eq!( - failure.to_string(), - format!( - "the {stage} launch stage failed: {}", - io::Error::from_raw_os_error(error as i32) - ) - ); - } -} - -#[test] -fn a_denied_umask_fails_the_proc_stage_and_runs_nothing() { - // A seccomp filter lasts as long as the thread that installs it, and passes to the processes - // that the thread starts, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_ENTRY_DENIED_UMASK_TEST_CHILD", - "guest::entry::tests::a_denied_umask_fails_the_proc_stage_and_runs_nothing", - ) { - return; - } - let directory = tempfile::tempdir().unwrap(); - let marker = directory.path().join("ran"); - // The simulated stage is no entry stage, so every entry stage succeeds, and the child's own - // call, which clears its umask before it mounts the view, is the one that fails. - sys::tests::deny_system_call(nix::libc::SYS_umask); - let (failure, status, _) = run_released( - shell(&format!("touch '{}'", marker.display())) - .enter(simulated(LaunchStage::Credentials, Errno::EPERM)), - ); - let failure = failure.expect("the denied umask fails the launch"); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Proc, Errno::EPERM as i32) - ); - assert_eq!(status, Some(ChildStatus::Exited(125))); - assert!(!marker.exists()); -} - -#[test] -fn an_entered_program_leads_a_session_without_a_terminal_or_the_callers_descriptors() { - // A descriptor that the caller opened without close-on-exec, as another thread of a caller - // might, and one that the command passes on. - let leaked = nix::fcntl::open(c"/dev/null", OpenFlags::O_RDONLY, Mode::empty()).unwrap(); - let passed = tempfile::tempfile().unwrap(); - let leaked_number = std::os::fd::AsRawFd::as_raw_fd(&leaked); - let passed_number = std::os::fd::AsRawFd::as_raw_fd(&passed); - // The simulated stage is no entry stage, so every entry stage succeeds. - let script = format!( - "read -r pid comm state ppid pgrp session tty rest = ids.split(' ').collect(); - // `sh -c` reads its own status file: the entry's child leads its session and process group, - // without a terminal. - assert_eq!(ids[1], ids[0], "{output}"); - assert_eq!(ids[2], ids[0], "{output}"); - assert_eq!(ids[3], "0", "{output}"); - assert_eq!(descriptors, "passed"); -} - -/// Runs `cat /proc/self/status` as the execution that `prepared` enters, while the caller ignores -/// `SIGHUP` and `SIGPIPE`, and its thread blocks `SIGCHLD`, as one that collects its children -/// through a signal descriptor does, and checks that the program ignores and blocks no signal. -fn check_signals(prepared: Prepared) { - sys::ignore_signals(&[sys::Signal::SIGHUP, sys::Signal::SIGPIPE]); - let blocked = nix::sys::signal::SigSet::from(sys::Signal::SIGCHLD); - blocked.thread_block().unwrap(); - let (failure, status, output) = run_released( - Command::new( - c"/bin/cat".to_owned(), - vec![c"cat".to_owned(), c"/proc/self/status".to_owned()], - ) - .enter(prepared), - ); - blocked.thread_unblock().unwrap(); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - let output = String::from_utf8(output).unwrap(); - for set in ["SigIgn", "SigBlk"] { - let value = output - .lines() - .find_map(|line| line.strip_prefix(set)?.strip_prefix(':')) - .unwrap_or_else(|| panic!("{set} is missing from {output}")); - assert_eq!(value.trim(), "0000000000000000", "{set}"); - } -} - -#[test] -fn an_entered_program_ignores_and_blocks_no_signal_of_its_caller_s() { - // Ignoring a signal changes the whole process, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_ENTRY_SIGNALS_TEST_CHILD", - "guest::entry::tests::an_entered_program_ignores_and_blocks_no_signal_of_its_caller_s", - ) { - return; - } - // The simulated stage is no entry stage, so every entry stage succeeds. - check_signals(simulated(LaunchStage::Credentials, Errno::EPERM)); -} - -#[test] -fn an_entry_needs_a_placement_and_neither_a_root_nor_a_check() { - let invalid = |command: Command| { - let error = command.spawn().unwrap_err(); - assert_eq!(error.raw_os_error(), Some(Errno::EINVAL as i32)); - }; - let prepared = || simulated(LaunchStage::Credentials, Errno::EPERM); - invalid(shell("exit 0").enter(prepared())); - let root = std::fs::File::open("/").unwrap(); - invalid(shell("exit 0").barrier().root(root).enter(prepared())); - invalid( - shell("exit 0") - .barrier() - .working_directory(c"/".to_owned()) - .check_working_directory_as(1, 1) - .enter(prepared()), - ); - // A cgroup places the child as well as a barrier. - let directory = tempfile::tempdir().unwrap(); - std::fs::write(directory.path().join("cgroup.procs"), b"").unwrap(); - let cgroup = crate::guest::spawn::Cgroup::open(directory.path()).unwrap(); - let spawned = shell("exit 0") - .cgroup(cgroup) - .enter(prepared()) - .spawn() - .unwrap(); - let process = spawned.process(); - assert_eq!(spawned.into_report().wait().unwrap(), None); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0))); - let written = std::fs::read_to_string(directory.path().join("cgroup.procs")).unwrap(); - assert_eq!(written, format!("{}\n", process.raw())); -} - -/// A probe that prints what an entered program sees of its identity and view, one `name=value` -/// line each. -const PROBE: &str = r#" -field() { - while IFS= read -r line; do - case "$line" in "$1:"*) set -- ${line#*:}; printf '%s' "$*"; return ;; esac - done /dev/null; then echo "null=writable"; fi -if mknod /tmp/node c 1 3 2>/dev/null; then echo "mknod=allowed"; else echo "mknod=refused"; fi -if (echo 1023 >/proc/sys/net/ipv4/ip_unprivileged_port_start) 2>/dev/null; then - echo "port-start=written" -else - echo "port-start=refused" -fi -echo "kcore=$(wc -c 2>/dev/null /dev/null | tr '\n' ' ')" -echo "leaked=$(test -e /proc/self/fd/$LEAKED && echo open || echo closed)" -echo "root=$(ls -A / | tr '\n' ' ')" -if [ "$(id -u)" = 0 ] && command -v python3 >/dev/null; then - python3 -c ' -import os -os.makedirs("/tmp/jail", exist_ok=True) -try: - os.chroot("/tmp/jail") -except PermissionError: - print("escape=refused") - raise SystemExit -for _ in range(16): - os.chdir("..") -os.chroot(".") -print("escape=" + " ".join(sorted(os.listdir("/"))) + " ") -' -fi -"#; - -/// Returns the values of the probe's lines named `name`, in their order. -fn values<'a>(output: &'a str, name: &str) -> Vec<&'a str> { - output - .lines() - .filter_map(|line| line.strip_prefix(name)?.strip_prefix('=')) - .collect() -} - -/// Returns the probe's one value named `name`. -fn value<'a>(output: &'a str, name: &str) -> &'a str { - match values(output, name)[..] { - [value] => value, - ref found => panic!("{name}: {found:?} in\n{output}"), - } -} - -/// Returns the options of the one mount at `point` that the probe lists. -fn mount_options<'a>(output: &'a str, point: &str) -> Vec<&'a str> { - let mounts: Vec<&str> = values(output, "mount") - .into_iter() - .filter_map(|mount| mount.strip_prefix(point)?.strip_prefix(' ')) - .collect(); - match mounts[..] { - [options] => options.split(',').collect(), - ref found => panic!("mounts at {point}: {found:?} in\n{output}"), - } -} - -/// Runs the probe as an execution of `entry`, with a descriptor of the caller's root leaked to -/// it, and returns its output. -fn probe(entry: &Entry) -> String { - let leaked = nix::fcntl::open( - c"/", - OpenFlags::O_RDONLY | OpenFlags::O_DIRECTORY, - Mode::empty(), - ) - .unwrap(); - let number = std::os::fd::AsRawFd::as_raw_fd(&leaked); - let (mut output, writer) = std::io::pipe().unwrap(); - let mut spawned = shell(PROBE) - .environment(vec![ - c"PATH=/usr/sbin:/usr/bin:/sbin:/bin".to_owned(), - c"LC_ALL=C".to_owned(), - CString::new(format!("LEAKED={number}")).unwrap(), - ]) - .stdout(writer) - .barrier() - .enter(entry.prepare().unwrap()) - .spawn() - .unwrap(); - let process = spawned.process(); - spawned.take_barrier().unwrap().release().unwrap(); - let failure = spawned.into_report().wait().unwrap(); - let mut bytes = Vec::new(); - output.read_to_end(&mut bytes).unwrap(); - drop(leaked); - let output = String::from_utf8(bytes).unwrap(); - assert_eq!(failure, None, "{output}"); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0)), "{output}"); - output -} - -/// Mounts a root for the privileged entries in `base`, a tmpfs that holds the account databases -/// and reaches the caller's programs through a read-only bind of `/usr`, and returns its path. -fn mount_test_root(base: &Path) -> PathBuf { - let root = base.join("root"); - std::fs::create_dir(&root).unwrap(); - let path = |path: &Path| sys::path_c_string(path).unwrap(); - sys::mount( - Some(c"tmpfs"), - &path(&root), - Some(c"tmpfs"), - sys::MountFlags::empty(), - Some(c"mode=0755"), - ) - .unwrap(); - for directory in ["usr", "etc", "tmp"] { - std::fs::create_dir(root.join(directory)).unwrap(); - } - std::fs::set_permissions(root.join("tmp"), std::fs::Permissions::from_mode(0o1777)).unwrap(); - for (link, target) in [ - ("bin", "usr/bin"), - ("sbin", "usr/sbin"), - ("lib", "usr/lib"), - ("lib64", "usr/lib64"), - ] { - std::os::unix::fs::symlink(target, root.join(link)).unwrap(); - } - sys::mount( - Some(c"/usr"), - &path(&root.join("usr")), - None, - sys::MountFlags::MS_BIND | sys::MountFlags::MS_REC, - None, - ) - .unwrap(); - sys::mount( - None, - &path(&root.join("usr")), - None, - sys::MountFlags::MS_BIND | sys::MountFlags::MS_REMOUNT | sys::MountFlags::MS_RDONLY, - None, - ) - .unwrap(); - std::fs::write(root.join("etc/passwd"), PASSWD_DATABASE).unwrap(); - std::fs::write(root.join("etc/group"), GROUP_DATABASE).unwrap(); - root -} - -/// Checks what a workload of `entry`, which `output` probed, sees of the view that every -/// identity shares. -fn check_view(output: &str, machine_id: &str) { - assert_eq!(value(output, "pid"), "1", "{output}"); - assert_eq!(value(output, "first-process"), "1", "{output}"); - assert_eq!(value(output, "NoNewPrivs"), "1", "{output}"); - assert_eq!(value(output, "CapInh"), "0000000000000000", "{output}"); - assert_eq!(value(output, "CapAmb"), "0000000000000000", "{output}"); - assert_eq!(value(output, "hostname"), "entry-test", "{output}"); - assert_eq!(value(output, "machine-id"), machine_id, "{output}"); - assert_eq!(value(output, "resolver"), "nameserver 10.0.0.1", "{output}"); - assert_eq!(value(output, "time"), "time-state", "{output}"); - assert_eq!( - value(output, "dev"), - "fd full null ptmx pts random shm stderr stdin stdout tty urandom zero" - ); - assert_eq!(value(output, "pts"), "ptmx"); - assert_eq!(value(output, "null"), "writable"); - assert_eq!(value(output, "mknod"), "refused"); - assert_eq!(value(output, "port-start"), "refused"); - assert_eq!(value(output, "leaked"), "closed"); - assert_eq!( - value(output, "root"), - "bin dev etc lib lib64 proc run sbin sys tmp usr " - ); - for (point, required) in [ - ("/", &["nosuid", "nodev"][..]), - ("/usr", &["ro", "nosuid", "nodev"]), - ("/proc", &["nosuid", "nodev", "noexec"]), - ("/sys", &["ro", "nosuid", "nodev", "noexec"]), - ("/dev", &["nosuid"]), - ("/dev/pts", &["nosuid", "noexec"]), - ("/dev/shm", &["nosuid", "nodev"]), - ("/etc/machine-id", &["ro", "nosuid", "nodev"]), - ("/etc/resolv.conf", &["ro", "nosuid", "nodev"]), - ("/run/nvx/time", &["ro", "nosuid", "nodev"]), - ] { - let options = mount_options(output, point); - for option in required { - assert!( - options.contains(option), - "{point} lacks {option}: {options:?}" - ); - } - } - // Only `/dev` and its devpts may hold devices. - for mount in values(output, "mount") { - let (point, options) = mount.split_once(' ').unwrap(); - if !matches!(point, "/dev" | "/dev/pts") { - assert!( - options.split(',').any(|option| option == "nodev"), - "{mount}" - ); - } - } -} - -/// Returns the mount points below `path`, not `path` itself, in the caller's mount namespace. -fn mounts_below(path: &Path) -> Vec { - let table = std::fs::read_to_string("/proc/self/mountinfo").unwrap(); - table - .lines() - .filter_map(|line| line.split(' ').nth(4)) - .map(PathBuf::from) - .filter(|point| point.starts_with(path) && point != path) - .collect() -} - -/// Checks that [`Entry::bind_runtime`] refuses what it cannot bind into `root`, the privileged -/// entries' mounted root, from the sources in `base`, and leaves the root as it was. -fn check_runtime_binds(base: &Path, root: &Path) { - let mounts = mounts_below(root); - let entry = || Entry::new(root, Identity::new(0, 0).account(AccountMode::Numeric)); - let generated = |path: PathBuf| { - entry() - .machine_id(MachineId::Generated(path)) - .bind_runtime() - }; - let result = generated(base.join("missing")); - assert!( - matches!( - &result, - Err(Error::MachineId(MappingError::System { step: Step::Source, error })) - if *error == Errno::ENOENT as i32 - ), - "{result:?}" - ); - let result = generated(base.join("time")); - assert!( - matches!(result, Err(Error::MachineId(MappingError::SourceKind))), - "{result:?}" - ); - let result = entry().time_status(base.join("machine-id")).bind_runtime(); - assert!( - matches!(result, Err(Error::TimeStatus(MappingError::SourceKind))), - "{result:?}" - ); - let result = entry().resolver(base.join("time")).bind_runtime(); - assert!( - matches!(result, Err(Error::Resolver(MappingError::SourceKind))), - "{result:?}" - ); - // A target that the image holds as another kind, or below a link, is refused. - std::fs::create_dir(root.join("etc/machine-id")).unwrap(); - let result = generated(base.join("machine-id")); - assert!( - matches!(result, Err(Error::MachineId(MappingError::TargetKind))), - "{result:?}" - ); - std::fs::remove_dir(root.join("etc/machine-id")).unwrap(); - std::os::unix::fs::symlink(base, root.join("run")).unwrap(); - let result = entry().time_status(base.join("time")).bind_runtime(); - assert!( - matches!(&result, Err(Error::TimeStatus(MappingError::Link { path })) - if path == Path::new("/run")), - "{result:?}" - ); - std::fs::remove_file(root.join("run")).unwrap(); - // A failure after the first binds removes them. - std::fs::create_dir(root.join("etc/resolv.conf")).unwrap(); - let result = entry() - .machine_id(MachineId::Generated(base.join("machine-id"))) - .time_status(base.join("time")) - .resolver(base.join("resolv.conf")) - .bind_runtime(); - assert!( - matches!(result, Err(Error::Resolver(MappingError::TargetKind))), - "{result:?}" - ); - std::fs::remove_dir(root.join("etc/resolv.conf")).unwrap(); - assert_eq!(mounts_below(root), mounts); - // What the walk created stays, but for the machine ID's file, which the caller's test puts a - // link in place of. - std::fs::remove_file(root.join("etc/machine-id")).unwrap(); -} - -/// Checks that an execution of `entry`, whose root is the privileged entries' mounted `root`, -/// enters its working directory, by default `/`, through no magic link: neither a `/proc` link to -/// a descriptor, such as one of the caller's root that leaks to the child or one that the view's -/// stages opened, a working directory, or a root, nor a symbolic link to one, which could name a -/// directory outside the root. -fn check_working_directories(entry: &Entry, root: &Path) { - let run = |directory: Option<&str>| { - let command = shell("pwd -P"); - let command = match directory { - Some(directory) => command.working_directory(CString::new(directory).unwrap()), - None => command, - }; - run_released(command.enter(entry.prepare().unwrap())) - }; - for (directory, expected) in [(None, "/\n"), (Some("/tmp"), "/tmp\n")] { - let (failure, status, output) = run(directory); - assert_eq!( - (failure, status), - (None, Some(ChildStatus::Exited(0))), - "{directory:?}" - ); - assert_eq!(String::from_utf8(output).unwrap(), expected); - } - let leaked = nix::fcntl::open( - c"/", - OpenFlags::O_RDONLY | OpenFlags::O_DIRECTORY, - Mode::empty(), - ) - .unwrap(); - let number = std::os::fd::AsRawFd::as_raw_fd(&leaked); - std::os::unix::fs::symlink("/proc/self/cwd", root.join("tmp/cwd-link")).unwrap(); - let mut refused: Vec<(String, &[Errno])> = vec![ - (format!("/proc/self/fd/{number}"), &[Errno::ENOENT][..]), - ("/proc/self/cwd".to_owned(), &[Errno::ELOOP][..]), - ("/proc/self/root".to_owned(), &[Errno::ELOOP][..]), - ("/proc/1/cwd".to_owned(), &[Errno::ELOOP][..]), - ("/tmp/cwd-link".to_owned(), &[Errno::ELOOP][..]), - ]; - // The child closes every other descriptor first, so a link names a standard stream or the - // launch report's writer, or nothing. - refused.extend((0..64).map(|number| { - ( - format!("/proc/self/fd/{number}"), - &[Errno::ELOOP, Errno::ENOENT][..], - ) - })); - for (directory, errors) in refused { - let (failure, status, output) = run(Some(&directory)); - let failure = failure.unwrap_or_else(|| panic!("{directory} was entered: {output:?}")); - assert_eq!( - failure.stage(), - LaunchStage::WorkingDirectory, - "{directory}" - ); - assert!( - errors - .iter() - .any(|error| failure.raw_os_error() == *error as i32), - "{directory}: {failure}" - ); - assert_eq!(status, Some(ChildStatus::Exited(125)), "{directory}"); - assert!(output.is_empty(), "{directory}"); - } - std::fs::remove_file(root.join("tmp/cwd-link")).unwrap(); - drop(leaked); -} - -/// Checks that an execution of `entry`, whose root is the privileged entries' mounted `root`, -/// looks for its program in the directories of its command's search path in that root, and for -/// an empty one in the working directory that it entered. -fn check_search(entry: &Entry, root: &Path) { - std::fs::create_dir(root.join("tmp/only-inside")).unwrap(); - std::os::unix::fs::symlink("/bin/sh", root.join("tmp/only-inside/inside-shell")).unwrap(); - for (path, directory) in [ - (c"/missing:/tmp/only-inside", None), - (c"/missing:", Some(c"/tmp/only-inside")), - ] { - let command = Command::new( - c"inside-shell".to_owned(), - vec![ - c"inside-shell".to_owned(), - c"-c".to_owned(), - c"printf found".to_owned(), - ], - ) - .search_path(path.to_owned()); - let command = match directory { - Some(directory) => command.working_directory(directory.to_owned()), - None => command, - }; - let (failure, status, output) = run_released(command.enter(entry.prepare().unwrap())); - assert_eq!( - (failure, status), - (None, Some(ChildStatus::Exited(0))), - "{path:?}" - ); - assert_eq!(output, b"found", "{path:?}"); - } - std::fs::remove_dir_all(root.join("tmp/only-inside")).unwrap(); -} - -#[test] -fn privileged_entries_see_their_own_view_and_identity() { - if crate::guest::sys::tests::isolate_process_test( - "NVX_ENTRY_PRIVILEGED_TEST_CHILD", - "guest::entry::tests::privileged_entries_see_their_own_view_and_identity", - ) { - return; - } - if !sys::tests::holds_capability(sys::tests::CAP_SYS_ADMIN).unwrap() { - return; - } - // The test mounts its root in a mount namespace of its own, which only a process with one - // thread may enter, so a child of the harness's process makes the calls. - match sys::tests::fork_isolated_test().unwrap() { - sys::ForkResult::Child => { - let outcome = std::panic::catch_unwind(|| { - sys::enter_private_mount_namespace().unwrap(); - let base = tempfile::tempdir().unwrap(); - let root = mount_test_root(base.path()); - let machine_id = MachineId::generate(base.path().join("machine-id")).unwrap(); - let id = std::fs::read_to_string(base.path().join("machine-id")).unwrap(); - std::fs::create_dir(base.path().join("time")).unwrap(); - std::fs::write(base.path().join("time/state"), "time-state\n").unwrap(); - let resolver = std::fs::File::create(base.path().join("resolv.conf")).unwrap(); - write_resolver(resolver, "10.0.0.1".parse().unwrap()).unwrap(); - let entry = |identity: Identity| { - Entry::new(&root, identity) - .hostname("entry-test") - .machine_id(machine_id.clone()) - .time_status(base.path().join("time")) - .resolver(base.path().join("resolv.conf")) - }; - check_runtime_binds(base.path(), &root); - // The image's links at the machine ID and at the resolver file, which no bind - // covers, give way to empty files, and their targets are left alone. - std::fs::write(base.path().join("outside"), "outside\n").unwrap(); - std::os::unix::fs::symlink( - base.path().join("outside"), - root.join("etc/machine-id"), - ) - .unwrap(); - std::os::unix::fs::symlink( - base.path().join("outside"), - root.join("etc/resolv.conf"), - ) - .unwrap(); - entry(Identity::new(65534, 65534)).bind_runtime().unwrap(); - assert_eq!( - std::fs::read_to_string(root.join("etc/machine-id")).unwrap(), - id - ); - assert_eq!( - std::fs::read_to_string(root.join("etc/resolv.conf")).unwrap(), - "nameserver 10.0.0.1\n" - ); - assert_eq!( - std::fs::read_to_string(base.path().join("outside")).unwrap(), - "outside\n" - ); - - // The non-root profile, with its existing account. - let output = probe(&entry(Identity::new(65534, 65534))); - check_view(&output, id.trim_end()); - assert_eq!(value(&output, "ids"), "65534 65534 65534"); - for set in ["CapPrm", "CapEff", "CapBnd"] { - assert_eq!(value(&output, set), "0000000000000000", "{set}"); - } - assert_eq!(value(&output, "kcore"), "unreadable"); - - // A machine ID that the guest rewrites in place, as after a restore, is what the - // later executions see. - let id = "0123456789abcdef0123456789abcdef\n"; - std::fs::write(base.path().join("machine-id"), id).unwrap(); - - // Root, with three allowlisted capabilities and its masked and read-only proc. - let capabilities = Capabilities::EMPTY - .with(Capability::CHOWN) - .with(Capability::KILL) - .with(Capability::SYS_CHROOT); - let privileged = entry( - Identity::new(0, 0) - .capabilities(capabilities) - .account(AccountMode::Numeric), - ) - .publication(Publication::Unpublished); - let output = probe(&privileged); - check_view(&output, id.trim_end()); - assert_eq!(value(&output, "ids"), "0 0 0"); - let mask = format!("{:016x}", capabilities.bits()); - for set in ["CapPrm", "CapEff", "CapBnd"] { - assert_eq!(value(&output, set), mask, "{set}"); - } - assert_eq!(value(&output, "kcore"), "0"); - assert_eq!(value(&output, "firmware"), ""); - for point in [ - "/proc/sys", - "/proc/sysrq-trigger", - "/proc/kcore", - "/sys/firmware", - ] { - assert!(mount_options(&output, point).contains(&"ro"), "{point}"); - } - // A nested `chroot` and `..` from it land at the top of the root, never above. - for escape in values(&output, "escape") { - assert_eq!(escape, value(&output, "root")); - } - check_working_directories(&privileged, &root); - check_search(&privileged, &root); - check_signals(privileged.prepare().unwrap()); - - // Root with no capability. - let output = probe(&entry(Identity::new(0, 0).account(AccountMode::Numeric))); - for set in ["CapPrm", "CapEff", "CapBnd"] { - assert_eq!(value(&output, set), "0000000000000000", "{set}"); - } - assert_eq!(value(&output, "port-start"), "refused"); - assert_eq!(value(&output, "mknod"), "refused"); - for escape in values(&output, "escape") { - assert_eq!(escape, "refused"); - } - }); - sys::exit_immediately(u8::from(outcome.is_err())); - } - sys::ForkResult::Parent(child) => { - assert_eq!(sys::wait(child), Some(ChildStatus::Exited(0))); - } - } -} - -#[test] -fn privileged_runtime_binds_are_removed_before_a_teardown() { - if crate::guest::sys::tests::isolate_process_test( - "NVX_ENTRY_TEARDOWN_TEST_CHILD", - "guest::entry::tests::privileged_runtime_binds_are_removed_before_a_teardown", - ) { - return; - } - if !sys::tests::holds_capability(sys::tests::CAP_SYS_ADMIN).unwrap() { - return; - } - // As for the entries' own test, a child of the harness's process makes the calls. - match sys::tests::fork_isolated_test().unwrap() { - sys::ForkResult::Child => { - let outcome = std::panic::catch_unwind(|| { - use nix::sys::statvfs::{FsFlags, statvfs}; - - sys::enter_private_mount_namespace().unwrap(); - let directory = tempfile::tempdir().unwrap(); - let base = directory.path().canonicalize().unwrap(); - // A root that is a mount of its own, as a layered root's overlay is, whose - // resolver file is a link, as a resolver daemon's image has. - let root = base.join("root"); - std::fs::create_dir(&root).unwrap(); - let point = sys::path_c_string(&root).unwrap(); - sys::mount( - Some(c"tmpfs"), - &point, - Some(c"tmpfs"), - sys::MountFlags::empty(), - Some(c"mode=0755"), - ) - .unwrap(); - std::fs::create_dir(root.join("etc")).unwrap(); - std::os::unix::fs::symlink( - "../run/resolver/resolv.conf", - root.join("etc/resolv.conf"), - ) - .unwrap(); - let machine_id = MachineId::generate(base.join("machine-id")).unwrap(); - std::fs::create_dir(base.join("time")).unwrap(); - let resolver = std::fs::File::create(base.join("resolv.conf")).unwrap(); - write_resolver(resolver, "fd00::a00:1".parse().unwrap()).unwrap(); - let binds = Entry::new(&root, Identity::new(0, 0).account(AccountMode::Numeric)) - .machine_id(machine_id) - .time_status(base.join("time")) - .resolver(base.join("resolv.conf")) - .bind_runtime() - .unwrap(); - // Each bind is read-only, `nosuid`, and `nodev`. - let targets = ["etc/machine-id", "run/nvx/time", "etc/resolv.conf"] - .map(|target| root.join(target)); - assert_eq!(mounts_below(&root), targets); - for target in &targets { - let flags = statvfs(target).unwrap().flags(); - assert!( - flags.contains(FsFlags::ST_RDONLY | FsFlags::ST_NOSUID | FsFlags::ST_NODEV), - "{}: {flags:?}", - target.display() - ); - } - assert_eq!( - std::fs::read_to_string(root.join("etc/resolv.conf")).unwrap(), - "nameserver fd00::a00:1\n" - ); - // A teardown that finds them fails, since it detaches no busy mount. - assert_eq!(sys::unmount(&point), Err(Errno::EBUSY)); - binds.remove().unwrap(); - assert!(mounts_below(&root).is_empty()); - // The image's link gave way to an empty file, which stays. - let replaced = std::fs::symlink_metadata(root.join("etc/resolv.conf")).unwrap(); - assert!(replaced.is_file() && replaced.len() == 0); - // Once they are removed, the teardown succeeds. - sys::unmount(&point).unwrap(); - }); - sys::exit_immediately(u8::from(outcome.is_err())); - } - sys::ForkResult::Parent(child) => { - assert_eq!(sys::wait(child), Some(ChildStatus::Exited(0))); - } - } -} diff --git a/nvx-rs/src/guest/exec.rs b/nvx-rs/src/guest/exec.rs deleted file mode 100644 index 5700634d1..000000000 --- a/nvx-rs/src/guest/exec.rs +++ /dev/null @@ -1,1201 +0,0 @@ -//! Driving a program as one execution, for any guest init or agent. -//! -//! An [`Execution`] drives a program that the caller [spawned](super::spawn) with the program's -//! ends of its [`Pipes`], or that the caller started itself and [adopts](Execution::adopt): it -//! tells the caller how the program's launch went, reads what the program writes to its standard -//! output and error, writes what the caller gives it to the program's standard input, and ends -//! once the program has ended and its output is all taken. It ends the program when the caller -//! cancels it, when its timeout passes, or when the caller loses the peer for which it runs it, -//! and it reports each step as an [`Event`]. -//! -//! It needs neither NVX's control protocol nor an async runtime, and it keeps no policy: the -//! caller decides which programs run and how many at once, where their output goes, how much of -//! it to take, and what each outcome means. -//! -//! # Driving -//! -//! The caller drives each execution without blocking, from its own loop or reactor. After it -//! acts on an execution, and whenever one of the execution's -//! [descriptors](Execution::descriptors) polls readable or its [deadline](Execution::deadline) -//! passes, it calls [`next_event`](Execution::next_event) until that returns `None`, which it -//! does only once every descriptor would block. Then it waits until one of them polls readable, -//! the deadline passes, or the caller has something to do, such as input to -//! [write](Execution::write_input) once the [input](Execution::input) polls writable. A reactor -//! that tracks readiness by edges may clear every descriptor's readiness when `next_event` -//! returns `None`. Executions share a loop without waiting for each other: the one whose output -//! the caller cannot take stalls alone. -//! -//! # Flow control -//! -//! The execution reads an output stream only as far as the caller allows with -//! [`grant`](Execution::grant), its flow-control seam: without an allowance, it leaves the stream -//! unread, so the program blocks once its pipe fills, and nothing else waits. Each output event -//! uses up as much of the stream's allowance, which the caller grants again as its sink takes -//! the bytes; a caller whose peer grants credit grants what the peer credits. The execution reads -//! into a buffer that the caller lends and holds no input, so its memory stays bounded. A stream -//! that the caller [discards](Execution::discard) is read into the same buffer and dropped -//! instead, so the buffer bounds every read. -//! -//! # Adoption -//! -//! A caller that starts a program by means of its own, such as a `clone3` into namespaces and a -//! cgroup that [`spawn`](super::spawn) does not make, [adopts](Execution::adopt) it: it hands -//! the execution the [`Child`] that names the program, and its own ends of the program's pipes, -//! which [`Pipes::from_ends`] takes. The caller saw how the program's launch went, so the -//! execution waits on no launch report: it reports the launch at once, with no failure, and then -//! drives the program as one that the caller spawned. -//! -//! # Status and ending -//! -//! By default, the execution collects the program's status itself, by a pidfd, and never another -//! child's, as [`process::Child`](super::process::Child) does. A caller that collects the status -//! of every child itself, such as a PID 1, hands the program's status to the execution with -//! [`observe`](Execution::observe): before it next calls into the execution, from the thread -//! that drives it, or whenever its collector takes the status, once [`Options::caller_reaps`] -//! has made that collector the program's only one. -//! -//! The execution ends the program with `SIGKILL`: the program alone, or the process group that -//! it leads too, as long as no collector has taken the program's status, through the program's -//! pidfd, which names them whatever later takes their IDs; and the processes of a cgroup that the -//! caller names, which also reaches descendants that left the group. Once the program has ended, -//! as its status tells, or as a collector of the caller's that took the status first does, the -//! execution closes the program's input, so that a descendant that still reads it sees its end, -//! and kills that cgroup again if the program's output is still open, as only another process -//! can hold it then. Ending the program ends neither stream: an execution ends once the caller -//! has taken or discarded what remains in them, and normal completion loses no byte. -//! -//! A caller that stops the program gracefully, as a container runtime does, first -//! [signals](Execution::signal) it, such as with `SIGTERM`, which reaches the program alone, or -//! its process group too, as the execution's kill does, but not the processes of the cgroup, and -//! each process once, unless the program leaves its group meanwhile. The signal ends neither the -//! program nor the execution: the execution records no cause, keeps the program's input open, and -//! leaves its timeout in place. The caller then drives the execution until a deadline of its own, -//! such as the end of a grace period: a program that handles the signal and exits ends with its -//! own status, and an outcome without a [cause](Outcome::cause), and loses no output. A program -//! whose status is still unknown by then, as one that ignores the signal, the caller ends with -//! [`terminate`](Execution::terminate) or [`cancel`](Execution::cancel), whose cause the outcome -//! reports. -//! -//! ```no_run -//! use nvx::guest::exec::{Event, Execution, Options, Pipes, Stream}; -//! use nvx::guest::spawn::Command; -//! -//! # fn main() -> std::io::Result<()> { -//! let (pipes, ends) = Pipes::new(false)?; -//! let spawned = Command::new(c"/bin/echo".to_owned(), vec![c"echo".to_owned(), c"hi".to_owned()]) -//! .stdout(ends.stdout) -//! .stderr(ends.stderr) -//! .new_process_group() -//! .spawn()?; -//! let mut execution = Execution::new(spawned, pipes, Options::new().process_group())?; -//! execution.grant(Stream::Stdout, 64 * 1024); -//! execution.grant(Stream::Stderr, 64 * 1024); -//! let mut buffer = [0u8; 4096]; -//! loop { -//! match execution.next_event(&mut buffer)? { -//! Some(Event::Output(stream, bytes)) => { -//! // The caller's sink took the bytes, so it allows as many again. -//! let length = bytes.len(); -//! execution.grant(stream, length); -//! } -//! Some(Event::Ended(outcome)) => { -//! assert!(outcome.status().success()); -//! break; -//! } -//! Some(_) => {} -//! // Poll `execution.descriptors()` until one is readable or `execution.deadline()`. -//! None => {} -//! } -//! } -//! # Ok(()) -//! # } -//! ``` - -use std::io; -use std::os::fd::{BorrowedFd, OwnedFd}; -use std::process::ExitStatus; -use std::time::{Duration, Instant}; - -use super::cgroup::{ExecutionCgroup, Scope}; -use super::process::Child; -use super::spawn::{LaunchFailure, LaunchReport, Spawned}; -use super::sys::{self, Errno, FileDescriptor, ProcessId}; - -/// One of a program's output streams. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -pub enum Stream { - /// The program's standard output. - Stdout, - /// The program's standard error. - Stderr, -} - -impl Stream { - /// Both streams, in the order in which an execution first reads them. - const ALL: [Self; 2] = [Self::Stdout, Self::Stderr]; - - /// Returns the stream's position in [`ALL`](Self::ALL). - fn index(self) -> usize { - match self { - Self::Stdout => 0, - Self::Stderr => 1, - } - } - - /// Returns the other stream. - fn other(self) -> Self { - match self { - Self::Stdout => Self::Stderr, - Self::Stderr => Self::Stdout, - } - } -} - -/// The pipes between an execution and its program. -/// -/// [`new`](Self::new) creates them, and hands the program's ends to the caller, whose -/// [`Command`](super::spawn::Command) gives them to the program as its standard streams. The -/// execution's ends stay here, nonblocking, until [`Execution::new`] takes them. For a program -/// that the caller started itself, [`from_ends`](Self::from_ends) takes the caller's own ends of -/// the program's pipes instead, for [`Execution::adopt`]. -#[derive(Debug)] -pub struct Pipes { - /// The writer of the program's standard input, if the execution writes it. - input: Option, - /// The readers of the program's standard output and error. - outputs: [FileDescriptor; 2], -} - -/// The program's ends of its execution's [`Pipes`], which its command takes as its standard -/// streams. They block, as a program expects of its standard streams, and the spawned child -/// keeps them only at its standard streams' numbers. -#[derive(Debug)] -pub struct ProgramEnds { - /// The reader of the program's standard input, if its execution writes it: the command's - /// [`stdin`](super::spawn::Command::stdin). Without it, the caller gives the program a - /// standard input of its choice, such as the null device. - pub stdin: Option, - /// The writer of the program's standard output: the command's - /// [`stdout`](super::spawn::Command::stdout). - pub stdout: OwnedFd, - /// The writer of the program's standard error: the command's - /// [`stderr`](super::spawn::Command::stderr). - pub stderr: OwnedFd, -} - -impl Pipes { - /// Creates a pipe for the program's standard output, one for its standard error, and, with - /// `input`, one for its standard input, and returns the execution's ends and the program's. - /// - /// Every end closes on `exec`, except the ones that a spawned child places at its standard - /// streams, and the execution's ends are nonblocking: a pipe end of its own has an open file - /// description that no other descriptor shares, so the program's ends keep blocking. - pub fn new(input: bool) -> io::Result<(Self, ProgramEnds)> { - let (stdout_reader, stdout_writer) = sys::create_pipe()?; - let (stderr_reader, stderr_writer) = sys::create_pipe()?; - stdout_reader.set_nonblocking()?; - stderr_reader.set_nonblocking()?; - let (input, stdin) = if input { - let (reader, writer) = sys::create_pipe()?; - writer.set_nonblocking()?; - (Some(writer), Some(reader.into())) - } else { - (None, None) - }; - Ok(( - Self { - input, - outputs: [stdout_reader, stderr_reader], - }, - ProgramEnds { - stdin, - stdout: stdout_writer.into(), - stderr: stderr_writer.into(), - }, - )) - } - - /// Takes the caller's own ends of the pipes of a program that it started itself, for - /// [`Execution::adopt`]: `input`, the writer of the program's standard input if the execution - /// writes it, and `stdout` and `stderr`, the readers of the program's standard output and - /// error. - /// - /// Each must be an end of a pipe whose open file description the program does not share, as - /// one end of a new pipe is, since `from_ends` makes each nonblocking, and the program's own - /// ends must keep blocking. It makes each close-on-exec too. Each must also be an end of a - /// pipe that carries bytes rather than packets, which `O_DIRECT` makes its writer write, as no - /// check can tell from the reader: a read that takes part of a packet drops the rest. An end - /// of another kind of file, or one open for the other direction, is `EINVAL`, and an error - /// closes every end. - pub fn from_ends(input: Option, stdout: OwnedFd, stderr: OwnedFd) -> io::Result { - let take = |descriptor: OwnedFd, writes: bool| -> io::Result { - let descriptor = FileDescriptor::from_owned(descriptor); - descriptor.take_pipe_end(writes)?; - Ok(descriptor) - }; - let input = input.map(|writer| take(writer, true)).transpose()?; - let stdout = take(stdout, false)?; - let stderr = take(stderr, false)?; - Ok(Self { - input, - outputs: [stdout, stderr], - }) - } -} - -/// How an execution ends its program, and when, and who collects its status. -#[derive(Clone, Debug, Default)] -pub struct Options { - /// Whether the program leads a process group of its own. - process_group: bool, - /// The cgroup whose processes ending the program kills too, and the scope that it kills. - cgroup: Option<(ExecutionCgroup, Scope)>, - /// How long the program may run. - timeout: Option, - /// Whether the caller's own collector takes the program's status, which the execution then - /// never waits for. - caller_reaps: bool, -} - -impl Options { - /// Returns options that end the program alone, give it no timeout, and leave its status for - /// the execution to collect. - pub fn new() -> Self { - Self::default() - } - - /// Ends the program's whole process group, whose leader the program is, as - /// [`Command::new_process_group`](super::spawn::Command::new_process_group) makes it, not - /// the program alone, and sends the group the program's [signals](Execution::signal) too. The - /// execution signals the group through the program's pidfd, which names it whatever later - /// takes its ID. Before Linux 6.9, which cannot signal a group through a pidfd, it signals the - /// group by its ID instead, right after the program's pidfd showed the program unreaped. - pub fn process_group(mut self) -> Self { - self.process_group = true; - self - } - - /// Kills every process in `scope` of `cgroup` too, with `cgroup.kill`, when the execution - /// ends the program, and once the program has ended while its output is still open. The - /// handle holds the cgroup's directory, so that those kills reach this cgroup alone, even - /// once another cgroup takes its path. [`Scope::Workload`] reaches every execution of the - /// workload, which a caller that runs several at once rarely wants. The program's - /// [signals](Execution::signal) do not reach the cgroup's processes, as `cgroup.kill` sends - /// only `SIGKILL`. - pub fn cgroup(mut self, cgroup: &ExecutionCgroup, scope: Scope) -> Self { - self.cgroup = Some((cgroup.clone(), scope)); - self - } - - /// Ends the program once `timeout` has passed since [`Execution::new`] or - /// [`Execution::adopt`], unless a collector has taken the program's status by then, or another - /// cause ended the program first. - pub fn timeout(mut self, timeout: Duration) -> Self { - self.timeout = Some(timeout); - self - } - - /// Leaves the program's status to the caller's own collector, such as the `waitpid(-1)` - /// loop of a PID 1 on a thread of its own, which hands it to the execution with - /// [`Execution::observe`]. The execution then never waits for the program, nor lists its - /// pidfd among its descriptors, so that the program has one status collector: the caller - /// learns that the program has ended from its own, as from `SIGCHLD`. The collector leaves - /// the program's status alone until [`Execution::new`] has returned, which opens the - /// program's pidfd by its ID, as [`Execution::adopt`] does for a child without one. - pub fn caller_reaps(mut self) -> Self { - self.caller_reaps = true; - self - } -} - -/// What an execution waits on, as one of its [descriptors](Execution::descriptors). -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum Source { - /// The program's launch report, which arrives once its program runs or its launch fails. An - /// [adopted](Execution::adopt) program has none. - Launch, - /// An output stream, which has bytes, or which every writer closed. - Output(Stream), - /// The program's pidfd, which polls readable once the program has ended, while the - /// execution collects the program's status itself. - Exit, -} - -/// Why an execution ended its program. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum Cause { - /// The caller [cancelled](Execution::cancel) the execution. - Cancelled, - /// The caller [terminated](Execution::terminate) the execution for a reason of its own. - Terminated, - /// The program's [timeout](Options::timeout) passed. - TimedOut, - /// The caller [disconnected](Execution::disconnect) the execution. - Disconnected, -} - -/// The processes that a kill reaches. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum KillTarget { - /// The program alone. - Program, - /// The program's process group. - ProcessGroup, - /// The processes of the execution's [cgroup](Options::cgroup). - Cgroup, -} - -impl KillTarget { - /// Every target, in the order in which the execution reports their failures. - const ALL: [Self; 3] = [Self::Program, Self::ProcessGroup, Self::Cgroup]; - - /// Returns the target's position in [`ALL`](Self::ALL). - fn index(self) -> usize { - match self { - Self::Program => 0, - Self::ProcessGroup => 1, - Self::Cgroup => 2, - } - } -} - -/// How an execution ended. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct Outcome { - /// The program's status. - status: ExitStatus, - /// Why the execution ended the program, if it did. - cause: Option, - /// The bytes of each stream that the execution discarded. - discarded: [u64; 2], -} - -impl Outcome { - /// Returns the program's status: an exit, or the signal that ended it. - pub fn status(&self) -> ExitStatus { - self.status - } - - /// Returns why the execution ended the program, or `None` if the program ended on its own. - /// Even with a cause, the program may have exited by itself, before the execution's kill - /// reached it or after a [failed kill](Event::KillFailed), which its status tells. - pub fn cause(&self) -> Option { - self.cause - } - - /// Returns how many bytes of `stream` the execution [discarded](Execution::discard): none, - /// unless the caller discarded or disconnected the stream. - pub fn discarded(&self, stream: Stream) -> u64 { - self.discarded[stream.index()] - } -} - -/// One step of an execution, which [`Execution::next_event`] reports. -#[derive(Debug)] -#[non_exhaustive] -pub enum Event<'a> { - /// The program's launch report, once, as [`LaunchReport::wait`] returns it: no failure, which - /// means that the program runs, unless its child ended before it could, as when the - /// execution ended it first; the launch stage that failed, after which the child exits with - /// status 125 without running the program; or an error, if the report could not be read. An - /// [adopted](Execution::adopt) program, whose launch its caller saw, reports no failure, at - /// once. - Launch(io::Result>), - /// Bytes that the program, or another process that shares its stream, wrote to the stream, - /// in order. They use up as much of the stream's allowance. - Output(Stream, &'a [u8]), - /// Bytes of the stream that the execution read and dropped. - Discarded(Stream, usize), - /// Every writer of the stream closed it, after its last byte. - Closed(Stream), - /// Reading the stream failed, which ends it as if every writer closed it. - ReadFailed(Stream, io::Error), - /// A kill that ending the program needed failed. A target that is gone, such as a process - /// group whose processes have all ended, is no failure. The cause for which the execution - /// ended the program counts all the same, and the execution goes on. - KillFailed(KillTarget, io::Error), - /// The execution ended: the program's status is known, its launch was reported, and its - /// output streams are closed. It is the last event, and the execution has closed the - /// program's input. - Ended(Outcome), -} - -/// One of a program's output streams, as its execution reads it. -#[derive(Debug)] -struct Output { - /// The stream's reader, until the stream closes. - reader: Option, - /// How many more bytes the caller lets the execution read, or `usize::MAX` for any number. - allowance: usize, - /// Whether the execution reads and drops the stream's bytes. - discarding: bool, - /// How many bytes the execution has dropped. - discarded: u64, -} - -/// What one read of an output stream did. -enum Read { - /// The stream had nothing to read now, or the execution may not read it. - Idle, - /// The execution read this many bytes into the caller's buffer. - Data(usize), - /// The execution read this many bytes into the caller's buffer, and dropped them. - Discarded(usize), - /// Every writer closed the stream. - Closed, - /// The read failed, which closed the stream. - Failed(Errno), -} - -impl Output { - /// Starts reading the stream from `reader`, with no allowance. - fn new(reader: FileDescriptor) -> Self { - Self { - reader: Some(reader), - allowance: 0, - discarding: false, - discarded: 0, - } - } - - /// Returns whether the stream is open. - fn is_open(&self) -> bool { - self.reader.is_some() - } - - /// Returns whether the execution reads the stream now: it is open, and the execution may - /// read or drop its bytes. - fn is_wanted(&self) -> bool { - self.is_open() && (self.discarding || self.allowance > 0) - } - - /// Reads the stream once into `buffer`: as far as the allowance lets it, or, while the - /// execution discards the stream, as far as the buffer goes, dropping what it read. - fn read(&mut self, buffer: &mut [u8]) -> Read { - if !self.is_open() { - return Read::Idle; - } - let length = if self.discarding { - buffer.len() - } else { - self.allowance.min(buffer.len()) - }; - if length == 0 { - return Read::Idle; - } - match self.read_into(&mut buffer[..length]) { - Read::Data(length) if self.discarding => { - self.discarded += length as u64; - Read::Discarded(length) - } - Read::Data(length) => { - if self.allowance != usize::MAX { - self.allowance -= length; - } - Read::Data(length) - } - other => other, - } - } - - /// Reads the open stream once into `buffer`, which is not empty, retrying interruptions. - fn read_into(&mut self, buffer: &mut [u8]) -> Read { - let Some(reader) = &self.reader else { - return Read::Idle; - }; - loop { - match reader.read(buffer) { - Ok(0) => { - self.reader = None; - return Read::Closed; - } - Ok(length) => return Read::Data(length), - Err(Errno::EINTR) => {} - Err(Errno::EAGAIN) => return Read::Idle, - Err(error) => { - self.reader = None; - return Read::Failed(error); - } - } - } - } -} - -/// What an execution reports of its program's launch. -#[derive(Debug)] -enum Launch { - /// The launch report of a program that the caller spawned, until it arrives. - Report(LaunchReport), - /// No failure, for an adopted program, whose launch its caller saw. - Adopted, -} - -/// One program that runs as an execution: see the [module](self) documentation. -/// -/// Dropping an execution that has not ended ends its program, as -/// [`disconnect`](Self::disconnect) does, without waiting; its status is still the caller's to -/// collect, with its own collector or a [`Child`]. -#[derive(Debug)] -pub struct Execution { - /// The program, which the execution names by a pidfd, and whose status it keeps. - child: Child, - /// The program's launch, until the execution reports it. - launch: Option, - /// The program's standard output and error. - outputs: [Output; 2], - /// The writer of the program's standard input, until it closes. - input: Option, - /// The stream that the execution reads first in its next step. - turn: Stream, - /// Whether the program leads its own process group, which ending the program kills whole, and - /// which the program's signals reach. - process_group: bool, - /// The cgroup whose processes ending the program kills too, and the scope that it kills. - cgroup: Option<(ExecutionCgroup, Scope)>, - /// When the program's timeout passes. - deadline: Option, - /// Whether the caller's own collector takes the program's status. - caller_reaps: bool, - /// Whether a collector of the caller's took the program's status before handing it over. - reaped: bool, - /// Why the execution ended the program, once it did. - cause: Option, - /// Whether the execution has learned that the program has ended. - exited: bool, - /// The latest kill failure of each target, which the next events report. - failures: [Option; 3], - /// Whether the execution has ended. - ended: bool, -} - -impl Execution { - /// Starts driving the program that `spawned` names, which the caller spawned with the - /// program's ends of `pipes`, as `options` say. Nothing collects the program's status before - /// `new` returns: `new` opens a pidfd for the program by its ID, which names the program alone - /// only until then, and the execution names the program by that pidfd from then on. - /// - /// If opening the pidfd fails, `new` ends what the options name before it returns the error: - /// the program, and then the process group that it led if the options say so, by their IDs, - /// unless `ESRCH` tells that the program's status is gone already, as its ID may name another - /// process then; and the processes of the options' cgroup. A program that joined another - /// process group ends too. It closes the pipes and the launch report before that kill, so - /// that a failure for want of descriptors leaves it one. The program's status is still the - /// caller's to collect, by `spawned`'s ID. - pub fn new(spawned: Spawned, pipes: Pipes, options: Options) -> io::Result { - let process = spawned.process(); - let report = spawned.into_report(); - let child = match Child::with_pidfd(process.raw().unsigned_abs()) { - Ok(child) => child, - Err(error) => { - let unreaped = error.raw_os_error() != Some(Errno::ESRCH as i32); - abandon(process, unreaped, (report, pipes), &options); - return Err(error); - } - }; - Ok(Self::start(child, Launch::Report(report), pipes, options)) - } - - /// Starts driving the program that `child` names, which the caller started itself, through - /// `pipes`, which hold the caller's own ends of the program's pipes, as `options` say. The - /// caller saw how the program's launch went, so the execution waits on no launch report: - /// [`next_event`](Self::next_event) reports the launch at once, with no failure, and the - /// caller calls it right after `adopt` returns, as no descriptor polls readable for it. - /// - /// The execution names the program by the child's pidfd. A child without one, which - /// [`Child::new`] names by its ID, gets one that `adopt` opens by that ID, as - /// [`new`](Self::new) does, so nothing may collect the program's status before `adopt` - /// returns then. A child whose status a collector of the caller's took already, and handed - /// over with [`Child::observe`], needs none: the program has ended, so the execution closes - /// its input, kills the processes of the options' cgroup if its output is still open, and - /// reads what remains of that output. - /// - /// If opening the pidfd fails, `adopt` ends what the options name before it returns the - /// error, as [`new`](Self::new) does, and closes the pipes before that kill. The program's - /// status is still the caller's to collect. - /// - /// ```no_run - /// use std::os::unix::process::CommandExt; - /// use std::process::{Command, Stdio}; - /// - /// use nvx::guest::exec::{Event, Execution, Options, Pipes}; - /// use nvx::guest::process::Child; - /// - /// # fn main() -> std::io::Result<()> { - /// let (stdout, stdout_writer) = std::io::pipe()?; - /// let (stderr, stderr_writer) = std::io::pipe()?; - /// let program = Command::new("/bin/echo") - /// .arg("hi") - /// .stdin(Stdio::null()) - /// .stdout(stdout_writer) - /// .stderr(stderr_writer) - /// .process_group(0) - /// .spawn()?; - /// // Nothing collects the program's status before the child holds its pidfd. - /// let child = Child::with_pidfd(program.id())?; - /// let pipes = Pipes::from_ends(None, stdout.into(), stderr.into())?; - /// let mut execution = Execution::adopt(child, pipes, Options::new().process_group())?; - /// let mut buffer = [0u8; 4096]; - /// assert!(matches!(execution.next_event(&mut buffer)?, Some(Event::Launch(Ok(None))))); - /// // Drive it on as one that the caller spawned. - /// # Ok(()) - /// # } - /// ``` - pub fn adopt(mut child: Child, pipes: Pipes, options: Options) -> io::Result { - if let Err(error) = child.open_pidfd() { - let unreaped = error.raw_os_error() != Some(Errno::ESRCH as i32); - abandon(child.process(), unreaped, pipes, &options); - return Err(error); - } - let mut execution = Self::start(child, Launch::Adopted, pipes, options); - if execution.child.status().is_some() { - execution.exited(); - } - Ok(execution) - } - - /// Starts driving the program that `child` names, whose launch is `launch`, through `pipes`, - /// as `options` say. - fn start(child: Child, launch: Launch, pipes: Pipes, options: Options) -> Self { - let [stdout, stderr] = pipes.outputs; - Self { - child, - launch: Some(launch), - outputs: [Output::new(stdout), Output::new(stderr)], - input: pipes.input, - turn: Stream::Stdout, - process_group: options.process_group, - cgroup: options.cgroup, - deadline: options - .timeout - .and_then(|timeout| Instant::now().checked_add(timeout)), - caller_reaps: options.caller_reaps, - reaped: false, - cause: None, - exited: false, - failures: [None, None, None], - ended: false, - } - } - - /// Returns the program's process ID. - pub fn id(&self) -> u32 { - self.child.id() - } - - /// Returns the program's status, once the execution or the caller's collector collected it. - pub fn status(&self) -> Option { - self.child.status() - } - - /// Lets the execution read `bytes` more of `stream`: its flow-control seam. It reads a - /// stream only while the stream has an allowance, and each output event uses up as much of - /// it. An allowance that reaches `usize::MAX` never runs out. - pub fn grant(&mut self, stream: Stream, bytes: usize) { - let output = &mut self.outputs[stream.index()]; - output.allowance = output.allowance.saturating_add(bytes); - } - - /// Reads and drops the rest of `stream`, whatever its allowance, from the next event on. It - /// reads into the buffer that the caller lends [`next_event`](Self::next_event), as output - /// does, reports each read as [`Event::Discarded`], and counts the bytes in its [`Outcome`]. - pub fn discard(&mut self, stream: Stream) { - self.outputs[stream.index()].discarding = true; - } - - /// Writes as much of `bytes` to the program's standard input as its pipe takes now, with - /// one write, and returns how many it took. A full pipe is `WouldBlock`: the caller writes - /// again once [`input`](Self::input) polls writable. - /// - /// Once the input is closed, by [`close_input`](Self::close_input), by the end of the - /// program or the execution, or because the program closed its end, a write is `EPIPE`. The - /// write never raises `SIGPIPE`, whatever the process's disposition of it. - pub fn write_input(&mut self, bytes: &[u8]) -> io::Result { - let Some(input) = &self.input else { - return Err(Errno::EPIPE.into()); - }; - loop { - match sys::write_without_sigpipe(input, bytes) { - Ok(length) => return Ok(length), - Err(Errno::EINTR) => {} - Err(Errno::EPIPE) => { - self.input = None; - return Err(Errno::EPIPE.into()); - } - Err(error) => return Err(error.into()), - } - } - } - - /// Closes the program's standard input, which the program reads to its end after every byte - /// that a write took. - pub fn close_input(&mut self) { - self.input = None; - } - - /// Lends the writer of the program's standard input while it is open, which a caller with - /// input to write polls for writing. It polls with an error once the program has closed its - /// end. - pub fn input(&self) -> Option> { - self.input.as_ref().map(FileDescriptor::borrowed) - } - - /// Ends the program for a cancellation, and closes its input. Returns whether the - /// cancellation is why the execution ended the program, as it is unless a collector had - /// taken the program's status, or the execution had ended the program, already. - pub fn cancel(&mut self) -> bool { - self.end(Cause::Cancelled) - } - - /// Ends the program for a reason of the caller's own, such as a limit on its output, and - /// closes its input. Returns whether that is why the execution ended the program, as for - /// [`cancel`](Self::cancel). - pub fn terminate(&mut self) -> bool { - self.end(Cause::Terminated) - } - - /// Ends the program because the caller lost the peer for which it ran it: the execution - /// discards both output streams from now on, closes the input, and ends the program while it - /// is unreaped, and the processes of its cgroup. - pub fn disconnect(&mut self) { - for output in &mut self.outputs { - output.discarding = true; - } - self.end(Cause::Disconnected); - } - - /// Sends the program the signal numbered `signal`, such as `SIGTERM`, and returns whether the - /// program was unreaped. The signal reaches what the execution's kill reaches but the - /// processes of its [cgroup](Options::cgroup): the program alone, or its - /// [process group](Options::process_group) too, through the program's pidfd, or, before Linux - /// 6.9, the group by its ID, right after the pidfd showed the program unreaped. - /// - /// Each process receives the signal once, so that no handler runs twice for one call: a - /// program that leads its group receives the group's signal alone, once a wait through its - /// pidfd, which needs no permission to signal it, has shown it unreaped, and one that left its - /// group receives a signal of its own. A program that leaves its group while it is signalled - /// receives a signal of its own too, so that it never misses one, though the group's signal - /// may have reached it first. - /// - /// The signal ends neither the program nor the execution: the execution records no cause, - /// keeps the program's input open, and leaves its timeout in place, so a program that ends - /// after the signal ends with its own status, and its outcome reports no - /// [cause](Outcome::cause), as for a program that ended by itself. `SIGKILL` is `EINVAL`, and - /// sends nothing, as the execution ends its program with a cause, through - /// [`cancel`](Self::cancel) and [`terminate`](Self::terminate), so that its outcome tells why. - /// - /// Once a collector has taken the program's status, it sends nothing, and returns `false`, so - /// that it never reaches a process that took the program's ID, or its group's, since. Finding - /// the status taken changes nothing in the execution, which learns of the program's end as - /// ever: from its own collection of the status, or from the caller's - /// [`observe`](Self::observe). A signal that the kernel refuses to the program or to its - /// process group, such as one that the caller may not send, or a number that names no signal, - /// is an error, which no event reports. The group's signal fails only if the kernel refuses it - /// to every process of the group, as `kill` does. A lookup of the program's group that fails - /// other than because the program is gone is an error too: a first one sends nothing, and one - /// after the group's signal sends the program no signal of its own. - pub fn signal(&mut self, signal: i32) -> io::Result { - if signal == sys::Signal::SIGKILL as i32 { - return Err(Errno::EINVAL.into()); - } - if !self.uncollected() { - return Ok(false); - } - let Some(pidfd) = self.child.pidfd() else { - return Ok(false); - }; - // The program's group is looked up by the program's ID before the pidfd shows the program - // unreaped, which proves that the ID still named the program then. - let process = self.child.process(); - if self.process_group && leads_its_group(process)? { - if !sys::is_unreaped(pidfd)? { - return Ok(false); - } - self.signal_group(pidfd, signal)?; - // The program may have left its group since the lookup, before the group's signal. - if !leads_its_group(process)? { - match signal_through_pidfd(pidfd, signal, false) { - Ok(()) | Err(Errno::ESRCH) => {} - Err(error) => return Err(error.into()), - } - } - return Ok(true); - } - let program = match signal_through_pidfd(pidfd, signal, false) { - Ok(()) => Ok(()), - Err(Errno::ESRCH) => return Ok(false), - Err(error) => Err(error), - }; - let group = if self.process_group { - self.signal_group(pidfd, signal) - } else { - Ok(()) - }; - program.and(group).map(|()| true).map_err(io::Error::from) - } - - /// Hands the execution the status that the caller's own collector collected for the process - /// `id`, and returns whether it was the program's final status, as - /// [`Child::observe`](super::process::Child::observe) does. Once it has the program's - /// status, the execution closes the program's input, and kills the processes of its cgroup - /// if the program's output is still open. - pub fn observe(&mut self, id: u32, status: ExitStatus) -> bool { - let observed = self.child.observe(id, status); - if observed { - self.exited(); - } - observed - } - - /// Returns the descriptors that move the execution on, each with what it waits on, which the - /// caller polls for reading: any event on one, such as a hangup, means that the next event - /// may follow. They change as the execution proceeds, and none is left once it has ended. - /// - /// An output stream is among them only while the execution may read it, so a caller grants - /// an allowance before it waits on the stream. - pub fn descriptors(&self) -> impl Iterator)> { - let active = !self.ended; - let launch = match &self.launch { - Some(Launch::Report(report)) if active => Some((Source::Launch, report.descriptor())), - _ => None, - }; - let outputs = Stream::ALL.map(|stream| { - let output = &self.outputs[stream.index()]; - output - .reader - .as_ref() - .filter(|_| active && output.is_wanted()) - .map(|reader| (Source::Output(stream), reader.borrowed())) - }); - let exit = self - .child - .pidfd() - .filter(|_| active && self.collects()) - .map(|pidfd| (Source::Exit, pidfd)); - launch - .into_iter() - .chain(outputs.into_iter().flatten()) - .chain(exit) - } - - /// Returns when the program's timeout passes, while it can still end the program: until a - /// collector takes the program's status, or another cause ends the program. - pub fn deadline(&self) -> Option { - self.deadline - .filter(|_| !self.ended && self.cause.is_none() && self.uncollected()) - } - - /// Takes the next step of the execution without blocking, and returns its event, or `None` - /// if no step can be taken until a descriptor polls readable, the deadline passes, or the - /// caller acts on the execution. Every read of an output stream goes into `buffer`, which - /// bounds its event, whether the execution returns the bytes or discards them; an empty - /// buffer reads nothing. - /// - /// Within a step, the execution reports a failed kill first, then the launch report, then - /// collects the program's status, unless the caller reaps the program, ends it if its - /// timeout has passed, and reads the output streams in turn, one read at a time, so that - /// neither starves the other. It ends once nothing is left. After that, it returns `None`. - /// - /// It fails only if collecting the program's status fails. A collector of the caller's that - /// took the status first makes it fail with `ECHILD`, once: the execution then leaves the - /// status to that collector, as [`Options::caller_reaps`] does, and waits until the caller - /// hands it over with [`observe`](Self::observe). The program has ended, though, so the - /// execution closes its input, and kills the processes of its cgroup if its output is still - /// open, as once it has the status. - pub fn next_event<'b>(&mut self, buffer: &'b mut [u8]) -> io::Result>> { - if self.ended { - return Ok(None); - } - if let Some(event) = self.kill_failure() { - return Ok(Some(event)); - } - if let Some(report) = self.launch() { - return Ok(Some(Event::Launch(report))); - } - if self.collect()? - && let Some(event) = self.kill_failure() - { - return Ok(Some(event)); - } - if self - .deadline() - .is_some_and(|deadline| Instant::now() >= deadline) - { - self.end(Cause::TimedOut); - if let Some(event) = self.kill_failure() { - return Ok(Some(event)); - } - } - for _ in Stream::ALL { - let stream = self.turn; - self.turn = stream.other(); - match self.outputs[stream.index()].read(&mut buffer[..]) { - Read::Idle => {} - Read::Data(length) => return Ok(Some(Event::Output(stream, &buffer[..length]))), - Read::Discarded(length) => return Ok(Some(Event::Discarded(stream, length))), - Read::Closed => return Ok(Some(Event::Closed(stream))), - Read::Failed(error) => return Ok(Some(Event::ReadFailed(stream, error.into()))), - } - } - let Some(status) = self.child.status() else { - return Ok(None); - }; - if self.launch.is_some() || self.outputs.iter().any(Output::is_open) { - return Ok(None); - } - self.ended = true; - self.input = None; - Ok(Some(Event::Ended(Outcome { - status, - cause: self.cause, - discarded: self.outputs.each_ref().map(|output| output.discarded), - }))) - } - - /// Takes the oldest kill failure that the execution has yet to report, as its event. - pub(crate) fn kill_failure(&mut self) -> Option> { - KillTarget::ALL.into_iter().find_map(|target| { - self.failures[target.index()] - .take() - .map(|error| Event::KillFailed(target, error)) - }) - } - - /// Takes the program's launch report without blocking once it has arrived, as - /// [`next_event`](Self::next_event) does within a step, and returns it, or `None` while it is - /// pending or once it was taken. An adopted program's launch, with no failure, is never - /// pending. - pub(crate) fn launch(&mut self) -> Option>> { - let ready = match self.launch.as_ref()? { - Launch::Report(report) => arrived(report), - Launch::Adopted => true, - }; - if !ready { - return None; - } - match self.launch.take().expect("the launch is pending") { - Launch::Report(report) => Some(report.wait()), - Launch::Adopted => Some(Ok(None)), - } - } - - /// Collects the program's status without blocking, if the execution collects it itself and - /// the program has ended, as [`next_event`](Self::next_event) does within a step, and returns - /// whether it collected it now. It fails as `next_event` does if collecting the status fails. - pub(crate) fn collect(&mut self) -> io::Result { - if !self.collects() { - return Ok(false); - } - match self.child.try_wait() { - Ok(Some(_)) => { - self.exited(); - Ok(true) - } - Ok(None) => Ok(false), - Err(error) => { - if error.raw_os_error() == Some(Errno::ECHILD as i32) { - self.taken(); - } - Err(error) - } - } - } - - /// Ends the program for `cause`, unless the execution has ended, and returns whether `cause` - /// is why it ended the program: the first cause while the program is unreaped. - fn end(&mut self, cause: Cause) -> bool { - if self.ended { - return false; - } - self.input = None; - let first = self.kill() && self.cause.is_none(); - if first { - self.cause = Some(cause); - } - first - } - - /// Kills what the execution may: the program, and the process group that it leads if the - /// options say so, while no collector has taken the program's status, and the cgroup's - /// processes. It records each failure for the next events, and returns whether the program - /// was unreaped. - fn kill(&mut self) -> bool { - let unreaped = self.signal_program(); - if let Some((cgroup, scope)) = &self.cgroup - && let Err(error) = cgroup.kill(*scope) - { - self.failures[KillTarget::Cgroup.index()] = Some(error); - } - unreaped - } - - /// Sends `SIGKILL` to the program through its pidfd, unless a collector is known to have - /// taken the program's status, and returns whether the program was unreaped. - /// - /// The pidfd names the program alone, whatever later takes its ID, so the signal fails with - /// `ESRCH` once a collector of the caller's has taken the status, which the execution notes. - /// Then, if the options say so, it [signals the process group](Self::signal_group) that the - /// program leads. A signal that the kernel refuses otherwise, such as one that it does not - /// permit, is a failure to kill its target, which the next events report. Only `ESRCH` tells - /// that the program is gone, so it still counts as unreaped then: the cause for which the - /// execution ended it stands, and a timeout fires once. - fn signal_program(&mut self) -> bool { - if !self.uncollected() { - return false; - } - let Some(pidfd) = self.child.pidfd() else { - return false; - }; - let kill = sys::Signal::SIGKILL as i32; - match signal_through_pidfd(pidfd, kill, false) { - Ok(()) => {} - Err(Errno::ESRCH) => { - self.reaped = true; - return false; - } - Err(error) => self.failures[KillTarget::Program.index()] = Some(error.into()), - } - if self.process_group - && let Err(error) = self.signal_group(pidfd, kill) - { - self.failures[KillTarget::ProcessGroup.index()] = Some(error.into()); - } - true - } - - /// Sends `signal` to the process group that the program leads, right after the program's - /// `pidfd` showed the program unreaped: through the pidfd, which names that group even if a - /// collector takes the program's status meanwhile, or, from a kernel that cannot signal a - /// group through a pidfd, before Linux 6.9, by the group's ID, which the program, unreaped a - /// moment before, kept from any other group. A group whose processes have all ended is no - /// failure. - fn signal_group(&self, pidfd: BorrowedFd<'_>, signal: i32) -> Result<(), Errno> { - let signalled = match signal_through_pidfd(pidfd, signal, true) { - Err(Errno::EINVAL) => sys::signal_process_group(self.child.process(), signal), - signalled => signalled, - }; - match signalled { - Err(Errno::ESRCH) => Ok(()), - signalled => signalled, - } - } - - /// Returns whether the program's status is unknown to the execution, and no collector of - /// the caller's is known to have taken it. - fn uncollected(&self) -> bool { - self.child.status().is_none() && !self.reaped - } - - /// Returns whether the execution collects the program's status itself, by its pidfd: until - /// it knows the status, unless the caller reaps the program, or a collector of the caller's - /// took the status first. - fn collects(&self) -> bool { - !self.caller_reaps && self.uncollected() - } - - /// Notes that a collector of the caller's took the program's status first, as a collection - /// that fails with `ECHILD` tells, so that the program has ended. - fn taken(&mut self) { - self.reaped = true; - self.exited(); - } - - /// Notes that the program has ended, the first time that the execution learns it: from the - /// program's status, or because a collector of the caller's took the status first. It closes - /// the program's input, which only another process can read now, and kills the processes of - /// the cgroup if the program's output is still open, as only another process can hold it now. - fn exited(&mut self) { - if self.exited { - return; - } - self.exited = true; - self.input = None; - if self.outputs.iter().any(Output::is_open) - && let Some((cgroup, scope)) = &self.cgroup - && let Err(error) = cgroup.kill(*scope) - { - self.failures[KillTarget::Cgroup.index()] = Some(error); - } - } -} - -impl Drop for Execution { - /// Ends the program of an execution that has not ended, as [`Execution::disconnect`] does. - fn drop(&mut self) { - if !self.ended { - self.kill(); - } - } -} - -/// Ends what the program of an execution that could not start may leave running, while it is -/// `unreaped`: the program, by its ID, and then, if the options say so, the process group that it -/// led when it started, as the execution's own kill does through the pidfd, so that a program -/// that joined another group since ends too; and the processes of the options' cgroup. It closes -/// the execution's `descriptors` before the cgroup's kill, which opens `cgroup.kill`, so that a -/// failure for want of descriptors leaves that kill one. -fn abandon(process: ProcessId, unreaped: bool, descriptors: D, options: &Options) { - if unreaped { - let _ = sys::kill_process(process); - if options.process_group { - let _ = sys::kill_process_group(process); - } - } - drop(descriptors); - if let Some((cgroup, scope)) = &options.cgroup { - let _ = cgroup.kill(*scope); - } -} - -/// Returns whether `report` has arrived: its pipe polls readable or hung up. -fn arrived(report: &LaunchReport) -> bool { - loop { - match sys::readable_or_hung_up(&[report.descriptor()], 0) { - Ok(ready) => return ready[0], - Err(Errno::EINTR) => {} - Err(_) => return false, - } - } -} - -/// Returns whether `process` leads its process group, by a lookup by its ID, unless a test has -/// given the lookup's next answer, as for a program that leaves its group right after it. A -/// process that the lookup finds no more, as `ESRCH` tells, leads none; any other failure of the -/// lookup is an error. -fn leads_its_group(process: ProcessId) -> Result { - #[cfg(test)] - if let Some(leads) = tests::LEADS.take() { - return Ok(leads); - } - match sys::process_group_of(process) { - Ok(group) => Ok(group == process), - Err(Errno::ESRCH) => Ok(false), - Err(error) => Err(error), - } -} - -/// Sends the signal numbered `signal` through `pidfd`, as [`sys::signal_through_pidfd`] does, -/// unless a test has the kernel refuse this thread's signals through pidfds, or only those to -/// process groups, as a kernel before Linux 6.9 does. A test sees what this thread sent. -fn signal_through_pidfd( - pidfd: BorrowedFd<'_>, - signal: i32, - process_group: bool, -) -> Result<(), Errno> { - #[cfg(test)] - tests::SENT.with_borrow_mut(|sent| sent.push((signal, process_group))); - #[cfg(test)] - if let Some(error) = tests::REFUSED.get() { - return Err(error); - } - #[cfg(test)] - if process_group && tests::BEFORE_LINUX_6_9.get() { - return Err(Errno::EINVAL); - } - sys::signal_through_pidfd(pidfd, signal, process_group) -} - -#[cfg(test)] -mod tests; diff --git a/nvx-rs/src/guest/exec/tests.rs b/nvx-rs/src/guest/exec/tests.rs deleted file mode 100644 index 7d8b8f360..000000000 --- a/nvx-rs/src/guest/exec/tests.rs +++ /dev/null @@ -1,1829 +0,0 @@ -//! Checks of executions of real programs. - -use std::ffi::CString; -use std::os::fd::{AsFd, AsRawFd}; -use std::os::unix::ffi::OsStrExt; -use std::os::unix::process::{CommandExt, ExitStatusExt}; -use std::path::Path; -use std::time::{Duration, Instant}; - -use nix::fcntl::{FcntlArg, FdFlag, OFlag, fcntl}; -use nix::poll::{PollFd, PollFlags}; - -use super::*; -use crate::guest::cgroup::Layout; -use crate::guest::spawn::tests::shell; -use crate::guest::spawn::{Command, LaunchStage}; - -thread_local! { - /// The error with which a test has the kernel refuse this thread's signals through pidfds. - pub(super) static REFUSED: std::cell::Cell> = const { std::cell::Cell::new(None) }; - /// Whether a test has the kernel refuse this thread's signals to process groups through - /// pidfds, as a kernel before Linux 6.9 does. - pub(super) static BEFORE_LINUX_6_9: std::cell::Cell = const { std::cell::Cell::new(false) }; - /// The signals that this thread sent through pidfds, in order, each with whether it went to - /// a process group. - pub(super) static SENT: std::cell::RefCell> = const { - std::cell::RefCell::new(Vec::new()) - }; - /// The answer that a test has the next lookup of whether a program leads its process group - /// give on this thread. - pub(super) static LEADS: std::cell::Cell> = const { std::cell::Cell::new(None) }; -} - -/// How long a test waits for an execution before it fails. -const PATIENCE: Duration = Duration::from_secs(20); - -/// The most bytes that one event carries in these tests: the size of the buffer that they lend -/// the execution, and the allowance that they grant each stream at a time. -const MAX_EVENT_BYTES: usize = 4096; - -/// More bytes than a program's output pipe can hold, even after the program grows the pipe, as -/// uutils' `head -c` (Ubuntu 26.04's `head`) does to fit what it copies. Without privileges, a -/// program can grow a pipe to `/proc/sys/fs/pipe-max-size` at most, 1 MiB by default, and uutils -/// grows it to 1 MiB at most. -const MORE_THAN_A_PIPE_HOLDS: usize = 4 * 1024 * 1024; - -/// Spawns `command` with new pipes, its standard input piped if `input`, in a process group of -/// its own, and starts its execution with `options`. -fn start(command: Command, input: bool, options: Options) -> Execution { - let (pipes, ends) = Pipes::new(input).unwrap(); - let mut command = command - .stdout(ends.stdout) - .stderr(ends.stderr) - .new_process_group(); - if let Some(stdin) = ends.stdin { - command = command.stdin(stdin); - } - Execution::new(command.spawn().unwrap(), pipes, options.process_group()).unwrap() -} - -/// Makes `SIGPIPE` end the process, as a caller's default disposition does. -fn default_sigpipe() { - sys::tests::default_sigpipe(); -} - -/// What a test saw of one execution. -#[derive(Debug, Default)] -struct Record { - /// The launch report, once it arrived. - launch: Option>, - /// The bytes of each stream, in order. - output: [Vec; 2], - /// The streams of the output events, in order. - order: Vec, - /// The streams that closed, in order. - closed: Vec, - /// The bytes that the execution discarded, from its events. - discarded: [u64; 2], - /// The kill failures, in order. - kill_failures: Vec<(KillTarget, Errno)>, - /// The outcome, once the execution ended. - outcome: Option, -} - -impl Record { - /// Takes the execution's events until it has no step to take, and returns whether it took - /// one. With `regrant`, it grants each stream's output back, as a sink that takes everything - /// would. - fn pump(&mut self, execution: &mut Execution, regrant: bool) -> bool { - let mut buffer = [0u8; MAX_EVENT_BYTES]; - let mut progressed = false; - while let Some(event) = execution.next_event(&mut buffer).unwrap() { - progressed = true; - match event { - Event::Launch(report) => { - assert!(self.launch.is_none(), "one launch report"); - self.launch = Some(report.unwrap()); - } - Event::Output(stream, bytes) => { - assert!(!bytes.is_empty()); - self.output[stream.index()].extend_from_slice(bytes); - self.order.push(stream); - let length = bytes.len(); - if regrant { - execution.grant(stream, length); - } - } - Event::Discarded(stream, length) => { - // Discarding reads into the same buffer as output. - assert!((1..=MAX_EVENT_BYTES).contains(&length), "{length}"); - self.discarded[stream.index()] += length as u64; - } - Event::Closed(stream) => { - assert!(!self.closed.contains(&stream), "{stream:?} closes once"); - self.closed.push(stream); - } - Event::ReadFailed(stream, error) => panic!("reading {stream:?} failed: {error}"), - Event::KillFailed(target, error) => { - self.kill_failures.push((target, sys::errno_of(&error))); - } - Event::Ended(outcome) => { - assert!(self.launch.is_some(), "the launch report comes first"); - assert_eq!(self.closed.len(), 2, "both streams close first"); - self.outcome = Some(outcome); - } - } - } - progressed - } - - /// Pumps `execution` and waits on it, as a caller's loop does, until `done` holds. - fn drive_until( - &mut self, - execution: &mut Execution, - regrant: bool, - mut done: impl FnMut(&Self, &Execution) -> bool, - ) { - let started = Instant::now(); - loop { - self.pump(execution, regrant); - if done(self, execution) { - return; - } - assert!( - started.elapsed() < PATIENCE, - "the execution moved on in time" - ); - wait_any(&[execution]); - } - } -} - -/// Waits on the descriptors of `executions` until one polls ready or the earliest deadline -/// passes, as a caller's loop does, and never wakes on its own sooner than 10 seconds. -fn wait_any(executions: &[&Execution]) { - let descriptors = executions - .iter() - .flat_map(|execution| execution.descriptors().map(|(_, descriptor)| descriptor)) - .collect::>(); - let deadline = executions - .iter() - .filter_map(|execution| execution.deadline()) - .min(); - let timeout = deadline.map_or(10_000, |deadline| { - deadline - .saturating_duration_since(Instant::now()) - .as_millis() - .clamp(1, 10_000) as u16 - }); - match sys::readable_or_hung_up(&descriptors, timeout) { - Ok(_) | Err(Errno::EINTR) => {} - Err(error) => panic!("poll failed: {error}"), - } -} - -/// Drives `execution` until it ends, granting each stream's output back as it arrives, and -/// returns what it saw. -fn run(execution: &mut Execution) -> Record { - let mut record = Record::default(); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - record.drive_until(execution, true, |record, _| record.outcome.is_some()); - assert!(execution.descriptors().next().is_none()); - assert!(execution.next_event(&mut [0u8; 16]).unwrap().is_none()); - record -} - -/// Returns the signal that ended `status`, if one did. -fn signal_of(status: ExitStatus) -> Option { - status.signal() -} - -#[test] -fn a_program_s_launch_output_and_status_are_reported() { - let mut execution = start( - shell("printf out; printf err >&2; exit 3"), - false, - Options::new(), - ); - let record = run(&mut execution); - assert_eq!(record.launch, Some(None)); - assert_eq!(record.output, [b"out".to_vec(), b"err".to_vec()]); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.status().code(), Some(3)); - assert_eq!(outcome.cause(), None); - assert_eq!(outcome.discarded(Stream::Stdout), 0); - assert_eq!(outcome.discarded(Stream::Stderr), 0); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn a_failed_launch_stage_is_reported_and_the_execution_still_ends() { - let mut execution = start( - shell("exit 0").working_directory(c"/nonexistent/directory".to_owned()), - false, - Options::new(), - ); - let record = run(&mut execution); - let failure = record.launch.unwrap().expect("the launch failed"); - assert_eq!(failure.stage(), LaunchStage::WorkingDirectory); - assert_eq!(failure.raw_os_error(), Errno::ENOENT as i32); - assert_eq!(record.outcome.unwrap().status().code(), Some(125)); -} - -#[test] -fn an_execution_without_output_ends_through_its_descriptors_alone() { - // Only the launch report's hangup and the pidfd wake the loop, which would otherwise wait - // ten seconds in each poll. - let started = Instant::now(); - let mut execution = start( - shell("exec >/dev/null 2>&1; sleep 0.2"), - false, - Options::new(), - ); - let sources = execution - .descriptors() - .map(|(source, _)| source) - .collect::>(); - assert_eq!(sources, [Source::Launch, Source::Exit]); - let record = run(&mut execution); - assert!(record.outcome.unwrap().status().success()); - assert!( - started.elapsed() < Duration::from_secs(5), - "{:?}", - started.elapsed() - ); -} - -#[test] -fn concurrent_executions_progress_independently() { - let mut executions = (0..16) - .map(|index| { - start( - shell(&format!("cat; printf {index} >&2")), - true, - Options::new(), - ) - }) - .collect::>(); - let mut records = (0..16).map(|_| Record::default()).collect::>(); - for (index, execution) in executions.iter_mut().enumerate() { - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let input = format!("input {index}\n"); - assert_eq!( - execution.write_input(input.as_bytes()).unwrap(), - input.len() - ); - execution.close_input(); - assert!(execution.input().is_none()); - } - let started = Instant::now(); - while records.iter().any(|record| record.outcome.is_none()) { - for (execution, record) in executions.iter_mut().zip(&mut records) { - record.pump(execution, true); - } - assert!(started.elapsed() < PATIENCE); - let waiting = executions - .iter() - .zip(&records) - .filter(|(_, record)| record.outcome.is_none()) - .map(|(execution, _)| execution) - .collect::>(); - if !waiting.is_empty() { - wait_any(&waiting); - } - } - for (index, record) in records.iter().enumerate() { - assert_eq!(record.output[0], format!("input {index}\n").as_bytes()); - assert_eq!(record.output[1], index.to_string().as_bytes()); - assert!(record.outcome.unwrap().status().success()); - } -} - -#[test] -fn a_stalled_sink_stalls_only_its_own_execution_and_loses_nothing() { - const LENGTH: usize = MORE_THAN_A_PIPE_HOLDS; - let mut stalled = start( - shell(&format!("head -c {LENGTH} /dev/zero")), - false, - Options::new(), - ); - let mut other = start(shell("printf done"), false, Options::new()); - stalled.grant(Stream::Stderr, MAX_EVENT_BYTES); - other.grant(Stream::Stdout, MAX_EVENT_BYTES); - other.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut stalled_record = Record::default(); - let mut other_record = Record::default(); - let started = Instant::now(); - while other_record.outcome.is_none() { - stalled_record.pump(&mut stalled, false); - other_record.pump(&mut other, true); - assert!(started.elapsed() < PATIENCE); - if other_record.outcome.is_none() { - wait_any(&[&stalled, &other]); - } - } - assert_eq!(other_record.output[0], b"done"); - // The stalled program fills its pipe and blocks, and its execution waits for the caller. - std::thread::sleep(Duration::from_millis(200)); - while stalled_record.launch.is_none() { - assert!(stalled_record.pump(&mut stalled, false)); - } - assert!(!stalled_record.pump(&mut stalled, false)); - assert!(stalled.status().is_none()); - assert!(stalled_record.output[0].is_empty()); - assert!( - stalled - .descriptors() - .all(|(source, _)| source != Source::Output(Stream::Stdout)) - ); - // Once its sink takes output again, every byte arrives. - stalled.grant(Stream::Stdout, MAX_EVENT_BYTES); - stalled_record.drive_until(&mut stalled, true, |record, _| record.outcome.is_some()); - assert_eq!(stalled_record.output[0].len(), LENGTH); - assert!(stalled_record.output[0].iter().all(|&byte| byte == 0)); - let outcome = stalled_record.outcome.unwrap(); - assert!(outcome.status().success()); - assert_eq!(outcome.discarded(Stream::Stdout), 0); -} - -#[test] -fn cancellation_ends_a_stalled_program_and_the_caller_discards_the_rest() { - // The program fills its pipe with the first page, which any pipe holds, before it says so. - let mut execution = start( - shell(&format!( - "head -c 4096 /dev/zero; printf ready >&2; \ - exec head -c {MORE_THAN_A_PIPE_HOLDS} /dev/zero" - )), - true, - Options::new(), - ); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, false, |record, _| { - record.output[1] == b"ready" - }); - assert!(execution.input().is_some()); - assert!(execution.cancel()); - assert!(!execution.cancel()); - assert!(!execution.terminate()); - assert!(execution.input().is_none()); - assert_eq!( - sys::errno_of(&execution.write_input(b"late").unwrap_err()), - Errno::EPIPE - ); - // The program ends at once, but its pipe still holds what it wrote, which the caller has - // yet to take. - record.drive_until(&mut execution, false, |_, execution| { - execution.status().is_some() - }); - assert_eq!(signal_of(execution.status().unwrap()), Some(9)); - assert!(!record.pump(&mut execution, false)); - assert!(record.outcome.is_none()); - execution.discard(Stream::Stdout); - record.drive_until(&mut execution, false, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert!(outcome.discarded(Stream::Stdout) > 0); - assert_eq!( - outcome.discarded(Stream::Stdout), - record.discarded[Stream::Stdout.index()] - ); - assert!(record.output[0].is_empty()); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn stdin_eof_follows_every_byte_that_a_write_took() { - let input = (0..300_000u32) - .map(|index| (index % 251) as u8) - .collect::>(); - let mut execution = start( - Command::new(c"/bin/cat".to_owned(), vec![c"cat".to_owned()]), - true, - Options::new(), - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - let mut written = 0; - let started = Instant::now(); - while record.outcome.is_none() { - while written < input.len() { - match execution.write_input(&input[written..]) { - Ok(length) => written += length, - Err(error) if error.kind() == io::ErrorKind::WouldBlock => break, - Err(error) => panic!("writing the input failed: {error}"), - } - } - if written == input.len() && execution.input().is_some() { - execution.close_input(); - } - record.pump(&mut execution, true); - assert!(started.elapsed() < PATIENCE); - if record.outcome.is_some() { - break; - } - // Wait for output, or for `cat` to read the full input pipe. - let mut descriptors = execution - .descriptors() - .map(|(_, descriptor)| PollFd::new(descriptor, PollFlags::POLLIN)) - .collect::>(); - if let Some(input) = execution.input() { - descriptors.push(PollFd::new(input, PollFlags::POLLOUT)); - } - let _ = nix::poll::poll(&mut descriptors, 1_000u16); - } - assert_eq!(record.output[0], input); - assert!(record.outcome.unwrap().status().success()); -} - -#[test] -fn a_program_that_closes_its_input_ends_the_writes_without_sigpipe() { - // A `SIGPIPE` would end this process under its default disposition, so the test runs in a - // process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_SIGPIPE_TEST_CHILD", - "guest::exec::tests::a_program_that_closes_its_input_ends_the_writes_without_sigpipe", - ) { - return; - } - default_sigpipe(); - let mut execution = start(shell("exec 0<&-; sleep 0.3"), true, Options::new()); - let started = Instant::now(); - // The writes meet the closed pipe, at the latest once the pipe is full. - let error = loop { - match execution.write_input(&[1u8; 4096]) { - Ok(_) => {} - Err(error) if error.kind() == io::ErrorKind::WouldBlock => { - std::thread::sleep(Duration::from_millis(5)); - } - Err(error) => break sys::errno_of(&error), - } - assert!(started.elapsed() < PATIENCE); - }; - assert_eq!(error, Errno::EPIPE); - assert!(execution.input().is_none()); - assert_eq!( - sys::errno_of(&execution.write_input(b"late").unwrap_err()), - Errno::EPIPE - ); - assert!(run(&mut execution).outcome.unwrap().status().success()); -} - -#[test] -fn concurrent_writers_meet_closed_inputs_without_sigpipe() { - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_SIGPIPE_THREADS_TEST_CHILD", - "guest::exec::tests::concurrent_writers_meet_closed_inputs_without_sigpipe", - ) { - return; - } - default_sigpipe(); - let mut executions = (0..4) - .map(|_| start(shell("exec 0<&-; sleep 0.3"), true, Options::new())) - .collect::>(); - std::thread::scope(|scope| { - for execution in &mut executions { - scope.spawn(move || { - let started = Instant::now(); - loop { - match execution.write_input(&[1u8; 4096]) { - Ok(_) => {} - Err(error) if error.kind() == io::ErrorKind::WouldBlock => { - std::thread::sleep(Duration::from_millis(1)); - } - Err(error) => { - assert_eq!(sys::errno_of(&error), Errno::EPIPE); - break; - } - } - assert!(started.elapsed() < PATIENCE); - } - }); - } - }); - for execution in &mut executions { - assert!(run(execution).outcome.unwrap().status().success()); - } -} - -#[test] -fn a_timeout_ends_the_program() { - let mut execution = start( - shell("sleep 30"), - false, - Options::new().timeout(Duration::from_millis(100)), - ); - assert!(execution.deadline().is_some()); - let started = Instant::now(); - let record = run(&mut execution); - assert!(started.elapsed() < Duration::from_secs(10)); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::TimedOut)); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert!(execution.deadline().is_none()); -} - -#[test] -fn a_timeout_whose_signals_the_kernel_refuses_fires_once() { - // The kernel refuses the execution's signals, as it does those that the caller may not - // send, so the program outlives its timeout. - REFUSED.set(Some(Errno::EPERM)); - let mut execution = start( - shell("sleep 0.3; echo done"), - false, - Options::new().timeout(Duration::from_millis(50)), - ); - let record = run(&mut execution); - REFUSED.set(None); - // The timeout ends the program once, and the execution goes on with its output. - assert_eq!( - record.kill_failures, - [ - (KillTarget::Program, Errno::EPERM), - (KillTarget::ProcessGroup, Errno::EPERM) - ] - ); - assert_eq!(record.output[0], b"done\n"); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success()); - assert_eq!(outcome.cause(), Some(Cause::TimedOut)); -} - -#[test] -fn a_disconnection_discards_the_output_and_ends_the_execution() { - let mut execution = start(shell("yes"), true, Options::new()); - execution.grant(Stream::Stdout, 10_000); - let mut record = Record::default(); - record.drive_until(&mut execution, false, |record, _| { - record.output[0].len() == 10_000 - }); - execution.disconnect(); - assert!(execution.input().is_none()); - record.drive_until(&mut execution, false, |record, _| record.outcome.is_some()); - assert_eq!(record.output[0].len(), 10_000); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Disconnected)); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert_eq!( - outcome.discarded(Stream::Stdout), - record.discarded[Stream::Stdout.index()] - ); -} - -#[test] -fn causes_count_only_while_the_program_runs() { - let mut execution = start(shell("printf x"), false, Options::new()); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, false, |_, execution| { - execution.status().is_some() - }); - // The program has ended, but its output waits for the caller. - assert!(record.outcome.is_none()); - assert!(!execution.cancel()); - assert!(!execution.terminate()); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!(record.output[0], b"x"); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), None); - assert!(outcome.status().success()); - assert!(!execution.cancel()); -} - -#[test] -fn an_observed_status_ends_the_execution() { - let mut execution = start(shell("exit 4"), true, Options::new()); - let id = execution.id(); - // The caller's own collector takes the status, as a PID 1 does, and hands it over. - let mut collector = Child::new(id).unwrap(); - let status = collector.wait().unwrap(); - assert!(!execution.observe(id + 1, status)); - assert!(execution.input().is_some()); - assert!(execution.observe(id, status)); - assert!(execution.input().is_none()); - assert!(!execution.observe(id, status)); - let record = run(&mut execution); - assert_eq!(record.outcome.unwrap().status().code(), Some(4)); -} - -#[test] -fn a_caller_that_reaps_is_the_program_s_only_collector() { - let mut execution = start(shell("exit 6"), false, Options::new().caller_reaps()); - assert!( - execution - .descriptors() - .all(|(source, _)| source != Source::Exit) - ); - let id = execution.id(); - let process = ProcessId::from_positive(i64::from(id)).unwrap(); - let started = Instant::now(); - while !sys::tests::has_exited(process).unwrap() { - assert!(started.elapsed() < PATIENCE, "the program ended in time"); - std::thread::sleep(Duration::from_millis(1)); - } - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.launch.is_some() && record.closed.len() == 2 - }); - // The execution took every other step, but left the status, which its collector finds. - assert!(execution.status().is_none()); - assert!(execution.descriptors().next().is_none()); - let status = Child::new(id).unwrap().wait().unwrap(); - assert!(execution.observe(id, status)); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.status().code(), Some(6)); - assert_eq!(outcome.cause(), None); -} - -/// Returns a command whose program starts a descendant in its process group, which outlives the -/// program, writes the descendant's ID to `file`, and exits with `code`. -fn leave_a_descendant(file: &Path, code: u8) -> Command { - shell(&format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'; exit {code}", - file.display() - )) -} - -/// The descendant that [`leave_a_descendant`] started, with a pidfd that polls readable once it -/// ends. -struct Descendant(ProcessId, FileDescriptor); - -impl Descendant { - /// Opens the descendant whose ID `file` holds. - fn open(file: &Path) -> Self { - let id = std::fs::read_to_string(file).unwrap(); - let process = ProcessId::from_positive(id.trim().parse().unwrap()).unwrap(); - Self(process, sys::open_pidfd(process).unwrap()) - } - - /// Asserts that the descendant still runs, as it does unless a signal to its process group - /// reached it, which would end it at once, and ends it. - fn assert_runs(self) { - assert_eq!( - sys::readable_or_hung_up(&[self.1.borrowed()], 200).unwrap(), - [false] - ); - sys::kill_process(self.0).unwrap(); - } - - /// Asserts that the descendant ends, as a signal to its process group ends it. - fn assert_ends(self) { - let patience = u16::try_from(PATIENCE.as_millis()).unwrap(); - assert_eq!( - sys::readable_or_hung_up(&[self.1.borrowed()], patience).unwrap(), - [true] - ); - } -} - -#[test] -fn ending_the_program_ends_the_process_group_that_it_leads() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - // Without a cgroup, only a signal to the program's process group reaches the descendant. - let mut execution = start( - shell(&format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'; echo started; exec sleep 30", - file.display() - )), - false, - Options::new(), - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"started\n" - }); - let descendant = Descendant::open(&file); - assert!(execution.cancel()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - descendant.assert_ends(); -} - -#[test] -fn a_status_that_another_collector_took_first_is_left_to_it() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - // Before the execution looks for the status itself, the caller may cancel it. - for cancel_first in [false, true] { - let mut execution = start( - leave_a_descendant(&file, 5), - false, - Options::new().timeout(Duration::from_secs(3600)), - ); - let id = execution.id(); - // A collector of the caller's, such as a reaper on another thread, takes the status - // before the execution looks for it. The program's ID may name another process from - // then on, and so may that of its process group once the descendant leaves it. - let status = Child::new(id).unwrap().wait().unwrap(); - let descendant = Descendant::open(&file); - if cancel_first { - assert!(!execution.cancel(), "nothing to cancel"); - } - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - let mut buffer = [0u8; 64]; - let mut failures = 0; - let started = Instant::now(); - // The descendant holds the output until it redirects its own. - while record.closed.len() < 2 { - match execution.next_event(&mut buffer) { - Ok(Some(Event::Launch(report))) => record.launch = Some(report.unwrap()), - Ok(Some(Event::Closed(stream))) => record.closed.push(stream), - Ok(Some(event)) => panic!("{event:?}"), - Ok(None) => { - assert!(started.elapsed() < PATIENCE, "the output closed in time"); - wait_any(&[&execution]); - } - Err(error) => { - assert_eq!(error.raw_os_error(), Some(Errno::ECHILD as i32)); - failures += 1; - } - } - } - assert!(execution.next_event(&mut buffer).unwrap().is_none()); - // Collecting fails once, unless the cancellation found the status gone first. - assert_eq!(failures, usize::from(!cancel_first), "{cancel_first}"); - assert!(execution.status().is_none()); - assert!(execution.deadline().is_none()); - assert!( - execution - .descriptors() - .all(|(source, _)| source != Source::Exit) - ); - assert!(!execution.cancel()); - execution.disconnect(); - descendant.assert_runs(); - assert!(execution.observe(id, status)); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.status().code(), Some(5)); - assert_eq!(outcome.cause(), None); - } -} - -#[test] -fn a_start_after_another_collector_took_the_status_ends_nothing_by_id() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - let (pipes, ends) = Pipes::new(false).unwrap(); - let spawned = leave_a_descendant(&file, 0) - .stdout(ends.stdout) - .stderr(ends.stderr) - .new_process_group() - .spawn() - .unwrap(); - let status = Child::new(spawned.id()).unwrap().wait().unwrap(); - assert!(status.success()); - let descendant = Descendant::open(&file); - let error = Execution::new(spawned, pipes, Options::new().process_group()).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::ESRCH); - descendant.assert_runs(); -} - -#[test] -fn the_program_s_end_closes_its_input_for_a_descendant_that_reads_it() { - // Without a cgroup, nothing ends the descendant, which holds the output open until its - // input ends. A shell gives a background job the null device as its input, unless the job - // redirects it. - let mut execution = start( - shell("exec 3<&0; cat <&3 & printf started"), - true, - Options::new(), - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |_, execution| { - execution.status().is_some() - }); - assert!(execution.input().is_none()); - assert_eq!( - sys::errno_of(&execution.write_input(b"late").unwrap_err()), - Errno::EPIPE - ); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!(record.output[0], b"started"); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success()); - assert_eq!(outcome.cause(), None); -} - -#[test] -fn dropping_an_unfinished_execution_ends_its_program() { - let execution = start(shell("sleep 30"), false, Options::new()); - let id = execution.id(); - drop(execution); - let mut child = Child::new(id).unwrap(); - assert_eq!(signal_of(child.wait().unwrap()), Some(9)); -} - -#[test] -fn an_unrelated_child_s_status_is_left_alone() { - let mut unrelated = std::process::Command::new("/bin/sh") - .args(["-c", "exit 7"]) - .spawn() - .unwrap(); - while !sys::tests::has_exited(ProcessId::from_child(&unrelated)).unwrap() { - std::thread::yield_now(); - } - let mut execution = start(shell("printf ran"), false, Options::new()); - assert_eq!(run(&mut execution).output[0], b"ran"); - // Nothing collected the unrelated child's status, which its own wait still finds. - assert_eq!(unrelated.wait().unwrap().code(), Some(7)); -} - -#[test] -fn both_streams_take_turns_while_each_has_output() { - // Each pipe takes the program's 8 KiB whole, and the execution reads neither stream without - // an allowance, so both hold all of their output once the program has ended. - let mut execution = start( - shell("head -c 8192 /dev/zero; head -c 8192 /dev/zero >&2"), - false, - Options::new(), - ); - let mut record = Record::default(); - // The launch report may arrive after the status: a process that another test's thread forked - // while the program was spawning holds the report's writer until it replaces itself with its - // program or ends. - record.drive_until(&mut execution, false, |record, execution| { - record.launch.is_some() && execution.status().is_some() - }); - assert!(record.order.is_empty()); - execution.grant(Stream::Stdout, usize::MAX); - execution.grant(Stream::Stderr, usize::MAX); - // Reads smaller than what each pipe holds leave both streams ready for several turns. - let mut buffer = [0u8; 1024]; - while let Some(event) = execution.next_event(&mut buffer).unwrap() { - match event { - Event::Output(stream, bytes) => { - record.output[stream.index()].extend_from_slice(bytes); - record.order.push(stream); - } - Event::Closed(stream) => record.closed.push(stream), - Event::Ended(outcome) => record.outcome = Some(outcome), - event => panic!("{event:?}"), - } - } - assert_eq!(record.output.each_ref().map(Vec::len), [8192, 8192]); - assert_eq!(record.order.len(), 16); - assert!( - record.order.windows(2).all(|pair| pair[0] != pair[1]), - "{:?}", - record.order - ); - assert!(record.outcome.unwrap().status().success()); -} - -/// Creates a fake cgroup in `root`, a workload cgroup whose `cgroup.kill` records the kills, and -/// returns its handle. -fn fake_cgroup(root: &Path) -> ExecutionCgroup { - let layout = Layout::new(root.join("agent"), root.join("workload")).unwrap(); - std::fs::create_dir_all(layout.workload()).unwrap(); - std::fs::write(layout.workload().join("cgroup.kill"), b"").unwrap(); - ExecutionCgroup::open(&layout, None).unwrap() -} - -#[test] -fn the_cgroup_is_killed_once_another_collector_took_the_status_with_the_output_held() { - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let file = temporary.path().join("descendant"); - // The descendant holds the program's output until it ends. - let mut execution = start( - shell(&format!("sleep 30 & echo $! >'{}'", file.display())), - true, - Options::new().cgroup(&cgroup, Scope::Execution), - ); - let id = execution.id(); - // A collector of the caller's, such as the kernel's while `SIGCHLD` is ignored, takes the - // status before the execution looks for it. - let status = Child::new(id).unwrap().wait().unwrap(); - let descendant = Descendant::open(&file); - let mut record = Record::default(); - let mut buffer = [0u8; 64]; - let started = Instant::now(); - let error = loop { - match execution.next_event(&mut buffer) { - Ok(Some(Event::Launch(report))) => record.launch = Some(report.unwrap()), - Ok(Some(event)) => panic!("{event:?}"), - Ok(None) => { - assert!( - started.elapsed() < PATIENCE, - "the collection failed in time" - ); - wait_any(&[&execution]); - } - Err(error) => break error, - } - }; - assert_eq!(error.raw_os_error(), Some(Errno::ECHILD as i32)); - // The program has ended, so only another process can hold its output, or read its input. - assert_eq!( - std::fs::read(cgroup.path().join("cgroup.kill")).unwrap(), - b"1" - ); - assert!(execution.input().is_none()); - // The fake cgroup ends nothing, so the test ends the descendant. - descendant.assert_runs(); - assert!(execution.observe(id, status)); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success()); - assert_eq!(outcome.cause(), None); -} - -#[test] -fn the_cgroup_is_killed_once_the_program_ends_with_its_output_held() { - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let kill = cgroup.path().join("cgroup.kill"); - // The descendant holds the output open, and the fake cgroup kills nothing. - let mut execution = start( - shell("sleep 30 & printf $!"), - false, - Options::new().cgroup(&cgroup, Scope::Execution), - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, execution| { - execution.status().is_some() && !record.output[0].is_empty() - }); - assert_eq!(std::fs::read(&kill).unwrap(), b"1"); - assert!(record.outcome.is_none()); - let descendant = std::str::from_utf8(&record.output[0]) - .unwrap() - .parse::() - .unwrap(); - nix::sys::signal::kill( - nix::unistd::Pid::from_raw(descendant), - nix::sys::signal::Signal::SIGKILL, - ) - .unwrap(); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert!(record.outcome.unwrap().status().success()); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn a_failed_kill_is_reported() { - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - std::fs::remove_file(cgroup.path().join("cgroup.kill")).unwrap(); - let mut execution = start( - shell("sleep 30"), - false, - Options::new().cgroup(&cgroup, Scope::Execution), - ); - assert!(execution.cancel()); - let record = run(&mut execution); - assert_eq!(record.kill_failures[0], (KillTarget::Cgroup, Errno::ENOENT)); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - assert_eq!(signal_of(outcome.status()), Some(9)); -} - -#[test] -fn a_kill_never_reaches_a_cgroup_that_took_the_path_later() { - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let kill = cgroup.path().join("cgroup.kill"); - let mut execution = start( - shell("sleep 30"), - false, - Options::new().cgroup(&cgroup, Scope::Execution), - ); - std::fs::remove_file(&kill).unwrap(); - std::fs::remove_dir(cgroup.path()).unwrap(); - std::fs::create_dir(cgroup.path()).unwrap(); - std::fs::write(&kill, b"").unwrap(); - assert!(execution.cancel()); - let record = run(&mut execution); - assert_eq!(record.kill_failures[0], (KillTarget::Cgroup, Errno::ENOENT)); - assert_eq!(std::fs::read(&kill).unwrap(), b""); -} - -#[test] -fn a_failed_start_ends_the_program_and_its_cgroup_even_without_descriptors() { - // Other tests could open nothing under the lowered descriptor limit, so this test runs in a - // process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_DESCRIPTOR_LIMIT_TEST_CHILD", - "guest::exec::tests::a_failed_start_ends_the_program_and_its_cgroup_even_without_descriptors", - ) { - return; - } - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let (pipes, ends) = Pipes::new(false).unwrap(); - let spawned = shell("sleep 30") - .stdout(ends.stdout) - .stderr(ends.stderr) - .new_process_group() - .spawn() - .unwrap(); - let id = spawned.id(); - // Without a spare descriptor, the pidfd does not open. `new` then ends the program, closes - // the descriptors that it took, and kills the cgroup, which opens `cgroup.kill` with one of - // them. The process opens its lowest free descriptor next. - let lowest = std::fs::File::open("/dev/null").unwrap().as_raw_fd(); - let previous = sys::tests::set_descriptor_limit(lowest as _).unwrap(); - let started = Execution::new( - spawned, - pipes, - Options::new() - .process_group() - .cgroup(&cgroup, Scope::Execution), - ); - sys::tests::set_descriptor_limit(previous).unwrap(); - assert_eq!(sys::errno_of(&started.unwrap_err()), Errno::EMFILE); - assert_eq!( - std::fs::read(cgroup.path().join("cgroup.kill")).unwrap(), - b"1" - ); - assert_eq!(signal_of(Child::new(id).unwrap().wait().unwrap()), Some(9)); -} - -#[test] -fn a_failed_start_ends_a_program_that_joined_another_process_group() { - const TEST: &str = - "guest::exec::tests::a_failed_start_ends_a_program_that_joined_another_process_group"; - // The test runs its own executable again as the program, with this variable naming a process - // group and a file. The program joins that group, which leaves the group that it led, - // creates the file to say so, and waits to be ended. - const PROGRAM: &str = "NVX_EXEC_GROUP_JOINER_PROGRAM"; - if let Some(value) = std::env::var_os(PROGRAM) { - let value = value.into_string().unwrap(); - let (group, joined) = value.split_once(':').unwrap(); - nix::unistd::setpgid( - nix::unistd::Pid::from_raw(0), - nix::unistd::Pid::from_raw(group.parse().unwrap()), - ) - .unwrap(); - std::fs::write(joined, b"").unwrap(); - std::thread::sleep(PATIENCE); - return; - } - // As the test above, it lowers the descriptor limit, so it runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test("NVX_EXEC_GROUP_JOINER_TEST_CHILD", TEST) { - return; - } - let temporary = tempfile::tempdir().unwrap(); - let joined = temporary.path().join("joined"); - let executable = std::env::current_exe().unwrap(); - let executable = CString::new(executable.as_os_str().as_bytes()).unwrap(); - let (pipes, ends) = Pipes::new(false).unwrap(); - let spawned = Command::new( - executable.clone(), - vec![ - executable, - c"--exact".to_owned(), - CString::new(TEST).unwrap(), - c"--nocapture".to_owned(), - ], - ) - .environment(vec![ - CString::new(format!( - "{PROGRAM}={}:{}", - nix::unistd::getpgrp(), - joined.display() - )) - .unwrap(), - ]) - .stdout(ends.stdout) - .stderr(ends.stderr) - .new_process_group() - .spawn() - .unwrap(); - let id = spawned.id(); - let started = Instant::now(); - while !joined.exists() { - assert!( - started.elapsed() < PATIENCE, - "the program joined the test's group in time" - ); - std::thread::sleep(Duration::from_millis(10)); - } - // No process is left in the group that the program led, so a signal to the group alone - // would reach nothing. - let program = nix::unistd::Pid::from_raw(i32::try_from(id).unwrap()); - assert_eq!(nix::sys::signal::killpg(program, None), Err(Errno::ESRCH)); - // Without a spare descriptor, the pidfd does not open, and no cgroup holds the program. - let lowest = std::fs::File::open("/dev/null").unwrap().as_raw_fd(); - let previous = sys::tests::set_descriptor_limit(lowest as _).unwrap(); - let started = Execution::new(spawned, pipes, Options::new().process_group()); - sys::tests::set_descriptor_limit(previous).unwrap(); - assert_eq!(sys::errno_of(&started.unwrap_err()), Errno::EMFILE); - assert_eq!(signal_of(Child::new(id).unwrap().wait().unwrap()), Some(9)); -} - -#[test] -fn the_program_s_ends_block_and_the_execution_s_do_not() { - let nonblocking = |descriptor: BorrowedFd<'_>| { - OFlag::from_bits_retain(fcntl(descriptor, FcntlArg::F_GETFL).unwrap()) - .contains(OFlag::O_NONBLOCK) - }; - let (pipes, ends) = Pipes::new(true).unwrap(); - for end in [ - ends.stdin.as_ref().unwrap().as_fd(), - ends.stdout.as_fd(), - ends.stderr.as_fd(), - ] { - assert!(!nonblocking(end)); - } - assert!(nonblocking(pipes.input.as_ref().unwrap().borrowed())); - for output in &pipes.outputs { - assert!(nonblocking(output.borrowed())); - } - let (pipes, ends) = Pipes::new(false).unwrap(); - assert!(pipes.input.is_none() && ends.stdin.is_none()); -} - -/// Starts `/bin/sh -c script` by a launcher other than [`spawn`](crate::guest::spawn), the -/// standard library's, as a caller with a launcher of its own does: in a process group of its -/// own, with new pipes, its standard input piped if `input` and the null device otherwise. It -/// returns the program's ID and the test's ends of its pipes. -fn start_own(script: &str, input: bool) -> (u32, Pipes) { - let (stdout, stdout_writer) = std::io::pipe().unwrap(); - let (stderr, stderr_writer) = std::io::pipe().unwrap(); - let mut command = std::process::Command::new("/bin/sh"); - command - .args(["-c", script]) - .stdout(stdout_writer) - .stderr(stderr_writer) - .process_group(0); - let writer = if input { - let (reader, writer) = std::io::pipe().unwrap(); - command.stdin(reader); - Some(OwnedFd::from(writer)) - } else { - command.stdin(std::process::Stdio::null()); - None - }; - // The execution collects the program's status, rather than the standard library's handle. - #[allow(clippy::zombie_processes)] - let program = command.spawn().unwrap(); - // The command holds the program's ends until it drops. - drop(command); - let pipes = Pipes::from_ends(writer, stdout.into(), stderr.into()).unwrap(); - (program.id(), pipes) -} - -/// Adopts the program `id` that [`start_own`] started, by a pidfd that the test opened for it, as -/// a caller's launcher holds one, with `options` that end its process group too. -fn adopt(id: u32, pipes: Pipes, options: Options) -> Execution { - let pidfd = sys::open_pidfd(ProcessId::from_positive(i64::from(id)).unwrap()).unwrap(); - let child = Child::from_pidfd(id, pidfd.into()).unwrap(); - Execution::adopt(child, pipes, options.process_group()).unwrap() -} - -#[test] -fn an_adopted_program_s_launch_comes_at_once_then_its_output_and_status() { - let (id, pipes) = start_own("printf out; printf err >&2; exit 3", false); - let mut execution = adopt(id, pipes, Options::new()); - assert_eq!(execution.id(), id); - // No descriptor waits for the launch, which the first step reports with no failure, and the - // execution collects the status through the program's pidfd. - assert!( - execution - .descriptors() - .all(|(source, _)| source != Source::Launch) - ); - assert!( - execution - .descriptors() - .any(|(source, _)| source == Source::Exit) - ); - let mut buffer = [0u8; MAX_EVENT_BYTES]; - assert!(matches!( - execution.next_event(&mut buffer).unwrap(), - Some(Event::Launch(Ok(None))) - )); - let mut record = Record { - launch: Some(None), - ..Record::default() - }; - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!(record.output, [b"out".to_vec(), b"err".to_vec()]); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.status().code(), Some(3)); - assert_eq!(outcome.cause(), None); - assert!(execution.descriptors().next().is_none()); -} - -#[test] -fn an_adopted_program_reads_its_input_to_its_end() { - let (id, pipes) = start_own("cat", true); - let mut execution = adopt(id, pipes, Options::new()); - assert_eq!(execution.write_input(b"in").unwrap(), 2); - execution.close_input(); - let record = run(&mut execution); - assert_eq!(record.launch, Some(None)); - assert_eq!(record.output[0], b"in"); - assert!(record.outcome.unwrap().status().success()); -} - -#[test] -fn an_adopted_program_that_a_nonblocking_pidfd_names_runs_as_any_other() { - let (id, pipes) = start_own("printf done; exit 4", false); - let process = ProcessId::from_positive(i64::from(id)).unwrap(); - let pidfd = OwnedFd::from(sys::open_pidfd(process).unwrap()); - // As `PIDFD_NONBLOCK` opens one, through which a blocking wait fails while the program runs. - let flags = OFlag::from_bits_retain(fcntl(&pidfd, FcntlArg::F_GETFL).unwrap()); - fcntl(&pidfd, FcntlArg::F_SETFL(flags | OFlag::O_NONBLOCK)).unwrap(); - let child = Child::from_pidfd(id, pidfd).unwrap(); - let mut execution = Execution::adopt(child, pipes, Options::new().process_group()).unwrap(); - let record = run(&mut execution); - assert_eq!(record.output[0], b"done"); - assert_eq!(record.outcome.unwrap().status().code(), Some(4)); -} - -#[test] -fn an_adopted_program_that_its_id_names_is_cancelled_with_its_process_group() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - // Without a cgroup, only a signal to the program's process group reaches the descendant. - let (id, pipes) = start_own( - &format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'; echo started; exec sleep 30", - file.display() - ), - false, - ); - // Nothing has collected the program's status, so `adopt` opens its pidfd by its ID. - let mut execution = Execution::adopt( - Child::new(id).unwrap(), - pipes, - Options::new().process_group(), - ) - .unwrap(); - assert!( - execution - .descriptors() - .any(|(source, _)| source == Source::Exit) - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"started\n" - }); - let descendant = Descendant::open(&file); - assert!(execution.cancel()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - descendant.assert_ends(); -} - -#[test] -fn an_adopted_program_whose_status_was_observed_has_ended_and_its_output_is_read() { - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let (id, pipes) = start_own("printf done", true); - // A collector of the caller's took the status and handed it over before the adoption, so the - // program's ID may name another process, and `adopt` opens nothing by it. - let status = Child::new(id).unwrap().wait().unwrap(); - let mut child = Child::new(id).unwrap(); - assert!(child.observe(id, status)); - let mut execution = Execution::adopt( - child, - pipes, - Options::new() - .process_group() - .cgroup(&cgroup, Scope::Execution), - ) - .unwrap(); - // The program has ended: its input is closed, nothing waits for its status, and, as its - // output is still open, the cgroup's processes are killed. - assert_eq!(execution.status(), Some(status)); - assert!(execution.input().is_none()); - assert!( - execution - .descriptors() - .all(|(source, _)| source != Source::Exit) - ); - assert_eq!( - std::fs::read(cgroup.path().join("cgroup.kill")).unwrap(), - b"1" - ); - assert!(!execution.cancel(), "nothing to cancel"); - let record = run(&mut execution); - assert_eq!(record.launch, Some(None)); - assert_eq!(record.output[0], b"done"); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success()); - assert_eq!(outcome.cause(), None); -} - -#[test] -fn adopting_a_program_whose_status_another_collector_took_ends_nothing_by_id() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - let (id, pipes) = start_own( - &format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'", - file.display() - ), - false, - ); - let status = Child::new(id).unwrap().wait().unwrap(); - assert!(status.success()); - let descendant = Descendant::open(&file); - // Nothing handed the status over, so `adopt` looks for the program by its ID, which names no - // process now, while the descendant keeps the program's process group. - let error = Execution::adopt( - Child::new(id).unwrap(), - pipes, - Options::new().process_group(), - ) - .unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::ESRCH); - descendant.assert_runs(); -} - -#[test] -fn a_failed_adoption_ends_the_program_and_its_cgroup_without_descriptors() { - // As for a failed start, the lowered descriptor limit runs this test in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_ADOPT_DESCRIPTOR_LIMIT_TEST_CHILD", - "guest::exec::tests::a_failed_adoption_ends_the_program_and_its_cgroup_without_descriptors", - ) { - return; - } - let temporary = tempfile::tempdir().unwrap(); - let cgroup = fake_cgroup(temporary.path()); - let (id, pipes) = start_own("sleep 30", false); - // Without a spare descriptor, the pidfd does not open by the program's ID. `adopt` then ends - // the program, closes the descriptors that it took, and kills the cgroup. - let lowest = std::fs::File::open("/dev/null").unwrap().as_raw_fd(); - let previous = sys::tests::set_descriptor_limit(lowest as _).unwrap(); - let adopted = Execution::adopt( - Child::new(id).unwrap(), - pipes, - Options::new() - .process_group() - .cgroup(&cgroup, Scope::Execution), - ); - sys::tests::set_descriptor_limit(previous).unwrap(); - assert_eq!(sys::errno_of(&adopted.unwrap_err()), Errno::EMFILE); - assert_eq!( - std::fs::read(cgroup.path().join("cgroup.kill")).unwrap(), - b"1" - ); - assert_eq!(signal_of(Child::new(id).unwrap().wait().unwrap()), Some(9)); -} - -#[test] -fn adopted_ends_are_pipe_ends_that_go_the_way_of_their_streams() { - // For a moment the caller's ends are inheritable, and a child that another test's thread - // started meanwhile could keep them, so this test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_ADOPTED_ENDS_TEST_CHILD", - "guest::exec::tests::adopted_ends_are_pipe_ends_that_go_the_way_of_their_streams", - ) { - return; - } - let nonblocking = |descriptor: BorrowedFd<'_>| { - OFlag::from_bits_retain(fcntl(descriptor, FcntlArg::F_GETFL).unwrap()) - .contains(OFlag::O_NONBLOCK) - }; - let cloexec = |descriptor: BorrowedFd<'_>| { - FdFlag::from_bits_retain(fcntl(descriptor, FcntlArg::F_GETFD).unwrap()) - .contains(FdFlag::FD_CLOEXEC) - }; - // The caller's ends, which its launcher left inheritable. - let pipe = || nix::unistd::pipe2(OFlag::O_CLOEXEC).unwrap(); - let inheritable = |end: OwnedFd| { - fcntl(&end, FcntlArg::F_SETFD(FdFlag::empty())).unwrap(); - end - }; - let ((stdin, input), (stdout, stdout_writer), (stderr, stderr_writer)) = - (pipe(), pipe(), pipe()); - let pipes = Pipes::from_ends( - Some(inheritable(input)), - inheritable(stdout), - inheritable(stderr), - ) - .unwrap(); - for end in [ - pipes.input.as_ref().unwrap(), - &pipes.outputs[0], - &pipes.outputs[1], - ] { - assert!(nonblocking(end.borrowed()) && cloexec(end.borrowed())); - } - // The program's ends of the same pipes keep blocking. - for end in [stdin.as_fd(), stdout_writer.as_fd(), stderr_writer.as_fd()] { - assert!(!nonblocking(end)); - } - // An end of another kind of file, or one that goes the other way, is refused. - let null = |writes: bool| { - let file = std::fs::OpenOptions::new() - .read(!writes) - .write(writes) - .open("/dev/null") - .unwrap(); - OwnedFd::from(file) - }; - for (input, stdout, stderr) in [ - (Some(pipe().0), pipe().0, pipe().0), - (None, pipe().1, pipe().0), - (None, pipe().0, null(false)), - (Some(null(true)), pipe().0, pipe().0), - ] { - let error = Pipes::from_ends(input, stdout, stderr).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::EINVAL); - } -} - -/// The signal with which these tests stop a program gracefully. -const SIGTERM: i32 = nix::libc::SIGTERM; - -/// Spawns `command` with new pipes and no input, as the leader of a process group of its own if -/// `leads`, and starts its execution with `options` as they are. -fn start_leading(command: Command, leads: bool, options: Options) -> Execution { - let (pipes, ends) = Pipes::new(false).unwrap(); - let mut command = command.stdout(ends.stdout).stderr(ends.stderr); - if leads { - command = command.new_process_group(); - } - Execution::new(command.spawn().unwrap(), pipes, options).unwrap() -} - -/// Waits until the program `id` has ended, and collects its status, as the caller's own -/// collector does. -fn collect_once_ended(id: u32) -> ExitStatus { - let process = ProcessId::from_positive(i64::from(id)).unwrap(); - let started = Instant::now(); - while !sys::tests::has_exited(process).unwrap() { - assert!(started.elapsed() < PATIENCE, "the program ended in time"); - std::thread::sleep(Duration::from_millis(1)); - } - Child::new(id).unwrap().wait().unwrap() -} - -#[test] -fn a_program_that_handles_a_signal_ends_with_its_own_status_and_all_of_its_output() { - // The program handles `SIGTERM`, starts a descendant in its process group, which the signal - // ends too, says that it is ready, and waits for the descendant, without dash's report of the - // signal that ended it. It exits 3 unless its handler runs. - let script = "trap 'printf bye; exit 0' TERM; sleep 30 /dev/null 2>&1 & \ - printf ready; wait $! 2>/dev/null; exit 3"; - for (adopted, caller_reaps) in [(false, false), (false, true), (true, false), (true, true)] { - let mut options = Options::new().timeout(Duration::from_secs(3600)); - if caller_reaps { - options = options.caller_reaps(); - } - let mut execution = if adopted { - let (id, pipes) = start_own(script, true); - adopt(id, pipes, options) - } else { - start(shell(script), true, options) - }; - let id = execution.id(); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"ready" - }); - let deadline = execution.deadline(); - assert!(deadline.is_some()); - assert!( - execution.signal(SIGTERM).unwrap(), - "{adopted} {caller_reaps}" - ); - // The signal ended neither the execution nor its timeout, and left the input open. - assert_eq!(execution.deadline(), deadline); - assert!(execution.input().is_some()); - if caller_reaps { - let status = collect_once_ended(id); - assert!(execution.observe(id, status)); - } - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!( - record.output, - [b"readybye".to_vec(), Vec::new()], - "{adopted} {caller_reaps}" - ); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.status().code(), Some(0), "{adopted} {caller_reaps}"); - assert_eq!(outcome.cause(), None); - assert!(record.kill_failures.is_empty()); - // The program has ended, so the execution signals it no more. - assert!(!execution.signal(SIGTERM).unwrap()); - } -} - -#[test] -fn a_program_that_ignores_a_signal_runs_until_the_caller_terminates_it() { - let mut execution = start( - shell("trap '' TERM; printf ready; exec sleep 30"), - true, - Options::new().timeout(Duration::from_secs(3600)), - ); - let process = ProcessId::from_positive(i64::from(execution.id())).unwrap(); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"ready" - }); - let deadline = execution.deadline(); - for _ in 0..2 { - assert!(execution.signal(SIGTERM).unwrap()); - } - std::thread::sleep(Duration::from_millis(200)); - record.pump(&mut execution, true); - assert!(!sys::tests::has_exited(process).unwrap()); - assert!(execution.status().is_none()); - assert_eq!(execution.deadline(), deadline); - assert!(execution.input().is_some()); - // The caller's own deadline passed, so it ends the program, and the outcome says why. - assert!(execution.terminate()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!(record.output[0], b"ready"); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Terminated)); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn a_signal_reaches_the_process_group_that_the_program_leads_if_the_options_say_so() { - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - // The descendant starts before the program ignores `SIGTERM`, so the signal ends it alone. - let script = format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'; trap '' TERM; echo started; \ - exec sleep 30", - file.display() - ); - // Whether the options name the group, and whether the execution signals it through the - // program's pidfd or, as before Linux 6.9, by its ID. - for (process_group, before_linux_6_9) in [(false, false), (true, false), (true, true)] { - BEFORE_LINUX_6_9.set(before_linux_6_9); - // The program leads a process group of its own either way. - let options = if process_group { - Options::new().process_group() - } else { - Options::new() - }; - let mut execution = start_leading(shell(&script), true, options); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"started\n" - }); - let descendant = Descendant::open(&file); - assert!(execution.signal(SIGTERM).unwrap()); - if process_group { - descendant.assert_ends(); - } else { - descendant.assert_runs(); - } - // The program ignores the signal, so it runs until the caller ends it. - assert!(execution.cancel()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - BEFORE_LINUX_6_9.set(false); - let outcome = record.outcome.unwrap(); - assert_eq!( - outcome.cause(), - Some(Cause::Cancelled), - "{process_group} {before_linux_6_9}" - ); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert!(record.kill_failures.is_empty()); - } -} - -#[test] -fn a_signal_reaches_each_process_once() { - // The program handles `SIGTERM` once and exits: a second signal would run its handler again. - // Its handler also ends the descendant, which the group's signal may not reach, and it waits - // for the descendant without dash's report of the signal that ended it. - let script = "trap 'kill $! 2>/dev/null; printf bye; exit 0' TERM; \ - sleep 30 /dev/null 2>&1 & printf ready; \ - wait $! 2>/dev/null; exit 3"; - // Whether the program leads its process group, whether the options name the group, what the - // first lookup of the program's group finds, if a test hook gives it, and what the execution - // sends through the program's pidfd. A program that leads its group receives the group's - // signal alone, with no signal 0 beforehand, which would need permission to signal it. One - // that the lookup found leading its group, but that left it before the group's signal, as the - // hook has it, receives a signal of its own after the group's. - let cases = [ - (true, true, None, vec![(SIGTERM, true)]), - (false, true, None, vec![(SIGTERM, false), (SIGTERM, true)]), - ( - false, - true, - Some(true), - vec![(SIGTERM, true), (SIGTERM, false)], - ), - (true, false, None, vec![(SIGTERM, false)]), - ]; - for (leads, process_group, looked_up, sent) in cases { - let case = format!("{leads} {process_group} {looked_up:?}"); - let options = if process_group { - Options::new().process_group() - } else { - Options::new() - }; - let mut execution = start_leading(shell(script), leads, options); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"ready" - }); - SENT.take(); - LEADS.set(looked_up); - assert!(execution.signal(SIGTERM).unwrap(), "{case}"); - assert_eq!(SENT.take(), sent, "{case}"); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - assert_eq!(record.output, [b"readybye".to_vec(), Vec::new()], "{case}"); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success(), "{case}"); - assert_eq!(outcome.cause(), None); - } -} - -#[test] -fn a_signal_reaches_nothing_once_a_collector_took_the_program_s_status() { - // Whether the program was adopted, whether the caller reaps it, whether a signal to its group - // would go through its pidfd or, as before Linux 6.9, by the group's ID, and whether the - // lookup of the program's group finds a leader, as a test hook has it, as when a group leader - // took the program's ID since. - for case in 0..16 { - signal_once_a_collector_took_the_status( - case & 1 != 0, - case & 2 != 0, - case & 4 != 0, - case & 8 != 0, - ); - } -} - -/// Checks that a signal reaches nothing once a collector of the caller's took the status of a -/// program, which may have been `adopted`, and whose status the caller may reap, that leaves a -/// descendant in its process group, when a signal to the group would go as before Linux 6.9 or -/// not, and when the lookup of the program's group finds a `leader` or not. -fn signal_once_a_collector_took_the_status( - adopted: bool, - caller_reaps: bool, - before_linux_6_9: bool, - leader: bool, -) { - let case = format!("{adopted} {caller_reaps} {before_linux_6_9} {leader}"); - let temporary = tempfile::tempdir().unwrap(); - let file = temporary.path().join("descendant"); - let script = format!( - "sleep 30 /dev/null 2>&1 & echo $! >'{}'", - file.display() - ); - BEFORE_LINUX_6_9.set(before_linux_6_9); - let mut options = Options::new(); - if caller_reaps { - options = options.caller_reaps(); - } - let mut execution = if adopted { - let (id, pipes) = start_own(&script, true); - adopt(id, pipes, options) - } else { - start(shell(&script), true, options) - }; - let id = execution.id(); - // A collector of the caller's takes the status before the execution looks for it, or before - // the caller hands it over. The program's ID may name another process from then on, and so - // may that of its process group once the descendant leaves it. - let status = collect_once_ended(id); - let descendant = Descendant::open(&file); - SENT.take(); - for _ in 0..2 { - LEADS.set(leader.then_some(true)); - assert!(!execution.signal(SIGTERM).unwrap(), "{case}"); - } - // Nothing reached the program's process group, which the descendant still holds, and the - // signals changed nothing in the execution. A program that the lookup found leading its group - // got nothing at all, as the wait through its pidfd found its status taken. - let sent = SENT.take(); - assert!(sent.iter().all(|&(_, group)| !group), "{case}"); - assert!(!leader || sent.is_empty(), "{case}"); - descendant.assert_runs(); - assert!(execution.input().is_some()); - // The execution learns of the program's end as ever: one that collects the status itself - // fails to, once, and closes the program's input. - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - let mut buffer = [0u8; 64]; - let mut failures = 0; - let started = Instant::now(); - while record.closed.len() < 2 { - match execution.next_event(&mut buffer) { - Ok(Some(Event::Launch(report))) => record.launch = Some(report.unwrap()), - Ok(Some(Event::Closed(stream))) => record.closed.push(stream), - Ok(Some(event)) => panic!("{event:?}"), - Ok(None) => { - assert!(started.elapsed() < PATIENCE, "the output closed in time"); - wait_any(&[&execution]); - } - Err(error) => { - assert_eq!(error.raw_os_error(), Some(Errno::ECHILD as i32)); - failures += 1; - } - } - } - assert_eq!(failures, usize::from(!caller_reaps), "{case}"); - assert_eq!(execution.input().is_none(), !caller_reaps, "{case}"); - assert!(execution.observe(id, status)); - assert!(execution.input().is_none()); - assert!(!execution.signal(SIGTERM).unwrap()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - BEFORE_LINUX_6_9.set(false); - let outcome = record.outcome.unwrap(); - assert!(outcome.status().success(), "{case}"); - assert_eq!(outcome.cause(), None); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn a_failed_group_lookup_is_an_error_that_sends_no_second_signal() { - // A seccomp filter lasts as long as the thread that installs it, and passes to the processes - // that the thread starts, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_EXEC_DENIED_GETPGID_TEST_CHILD", - "guest::exec::tests::a_failed_group_lookup_is_an_error_that_sends_no_second_signal", - ) { - return; - } - // The program leads its group, and ignores `SIGTERM`, so it runs until the test ends it. - let mut execution = start( - shell("trap '' TERM; printf ready; exec sleep 30"), - false, - Options::new(), - ); - execution.grant(Stream::Stdout, MAX_EVENT_BYTES); - execution.grant(Stream::Stderr, MAX_EVENT_BYTES); - let mut record = Record::default(); - record.drive_until(&mut execution, true, |record, _| { - record.output[0] == b"ready" - }); - sys::tests::deny_system_call(nix::libc::SYS_getpgid); - // A first lookup that fails sends nothing. - SENT.take(); - let error = execution.signal(SIGTERM).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::EPERM); - assert!(SENT.take().is_empty()); - // One that fails after the group's signal, where the hook answers the first, sends the program - // no signal of its own. - LEADS.set(Some(true)); - let error = execution.signal(SIGTERM).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::EPERM); - assert_eq!(SENT.take(), [(SIGTERM, true)]); - // The kill needs no lookup. - assert!(execution.cancel()); - record.drive_until(&mut execution, true, |record, _| record.outcome.is_some()); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - assert_eq!(signal_of(outcome.status()), Some(9)); - assert!(record.kill_failures.is_empty()); -} - -#[test] -fn a_kill_and_a_signal_that_the_kernel_refuses_are_errors_that_end_nothing() { - let mut execution = start(shell("exec sleep 30"), false, Options::new()); - let process = ProcessId::from_positive(i64::from(execution.id())).unwrap(); - // A kill stays with the execution's causes, and the kernel refuses a number that names no - // signal. - for signal in [nix::libc::SIGKILL, 65] { - let error = execution.signal(signal).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::EINVAL, "{signal}"); - } - // The kernel refuses the execution's signals, as it does those that the caller may not send. - REFUSED.set(Some(Errno::EPERM)); - let refused = execution.signal(SIGTERM); - REFUSED.set(None); - assert_eq!(sys::errno_of(&refused.unwrap_err()), Errno::EPERM); - // The program runs on, the execution has no cause yet, and no event reports the refusals. - std::thread::sleep(Duration::from_millis(100)); - assert!(!sys::tests::has_exited(process).unwrap()); - assert!(execution.cancel()); - let record = run(&mut execution); - assert!(record.kill_failures.is_empty()); - let outcome = record.outcome.unwrap(); - assert_eq!(outcome.cause(), Some(Cause::Cancelled)); - assert_eq!(signal_of(outcome.status()), Some(9)); -} diff --git a/nvx-rs/src/guest/init/tests/exec.rs b/nvx-rs/src/guest/init/tests/exec.rs index 1d9d5a9e9..e63ee197d 100644 --- a/nvx-rs/src/guest/init/tests/exec.rs +++ b/nvx-rs/src/guest/init/tests/exec.rs @@ -741,13 +741,13 @@ fn a_stolen_or_auto_reaped_status_is_reported_once_and_never_reclassified_by_an_ let mut values = request("printf retained; exit 7"); values.timeout = Some(Duration::from_secs(30)); let mut execution = Exec::start_simulated(values, None).unwrap(); - let status = if automatic.is_some() { - 7 << 8 - } else { - sys::wait_named(sys::WaitTarget::Process(execution.leader), true) - .unwrap() - .unwrap() - }; + if automatic.is_none() { + assert_eq!( + sys::wait(execution.leader), + Some(sys::ChildStatus::Exited(7)) + ); + } + let status = 7 << 8; let deadline = Instant::now() + Duration::from_secs(5); loop { match execution.next_event(&mut []) { @@ -1248,7 +1248,11 @@ fn privileged_capture_sweep_kills_setsid_escapees_and_spares_both_helpers() { .unwrap(); let escaped = pid(escaped); let deadline = Instant::now() + Duration::from_secs(5); - while sys::process_group_of(escaped).unwrap() != escaped { + while nix::unistd::getpgid(Some(nix::unistd::Pid::from_raw(escaped.raw()))) + .unwrap() + .as_raw() + != escaped.raw() + { assert!(Instant::now() < deadline); std::thread::sleep(Duration::from_millis(1)); } @@ -1266,7 +1270,11 @@ fn privileged_capture_sweep_kills_setsid_escapees_and_spares_both_helpers() { .parse() .unwrap()); let deadline = Instant::now() + Duration::from_secs(5); - while sys::process_group_of(held_id).unwrap() != held_id { + while nix::unistd::getpgid(Some(nix::unistd::Pid::from_raw(held_id.raw()))) + .unwrap() + .as_raw() + != held_id.raw() + { assert!(Instant::now() < deadline); std::thread::sleep(Duration::from_millis(1)); } diff --git a/nvx-rs/src/guest/managed_agent/dispatch/tests/runtime.rs b/nvx-rs/src/guest/managed_agent/dispatch/tests/runtime.rs index 85d85d27c..20a4589f8 100644 --- a/nvx-rs/src/guest/managed_agent/dispatch/tests/runtime.rs +++ b/nvx-rs/src/guest/managed_agent/dispatch/tests/runtime.rs @@ -144,7 +144,7 @@ fn termination(helper_ends: bool) { sys::kill_through_pidfd(helper.descriptor.borrowed(), false).unwrap(); assert!(sys::readable_or_hung_up(&[helper.descriptor.borrowed()], 5000).unwrap()[0]); } - sys::signal_through_pidfd(parent.borrowed(), Signal::SIGTERM as i32, false).unwrap(); + sys::tests::signal_through_pidfd(parent.borrowed(), Signal::SIGTERM as i32, false).unwrap(); let mut traffic = Vec::new(); host.read_to_end(&mut traffic).unwrap(); assert!( @@ -247,7 +247,8 @@ fn prelaunch_termination(helper_ends: bool) { assert!(sys::readable_or_hung_up(&[helper.descriptor.borrowed()], 5000).unwrap()[0]); } else { let process = sys::open_pidfd(sys::current_process()).unwrap(); - sys::signal_through_pidfd(process.borrowed(), Signal::SIGTERM as i32, false).unwrap(); + sys::tests::signal_through_pidfd(process.borrowed(), Signal::SIGTERM as i32, false) + .unwrap(); } let mut system = TestSystem::new(Paths::default()); let config = @@ -584,7 +585,7 @@ fn backpressured_termination(helper_ends: bool) { sys::kill_through_pidfd(helper.descriptor.borrowed(), false).unwrap(); assert!(sys::readable_or_hung_up(&[helper.descriptor.borrowed()], 5000).unwrap()[0]); } - sys::signal_through_pidfd(parent.borrowed(), Signal::SIGTERM as i32, false).unwrap(); + sys::tests::signal_through_pidfd(parent.borrowed(), Signal::SIGTERM as i32, false).unwrap(); let mut traffic = Vec::new(); host.read_to_end(&mut traffic).unwrap(); only_stdout_prefix(&traffic); @@ -651,7 +652,8 @@ fn privileged_layered_runtime_stops_before_attach_on_pending_sigterm() { Session::with_runtime(FileDescriptor::from_owned(guest.into()), &mut runtime) .unwrap(); let process = sys::open_pidfd(sys::current_process()).unwrap(); - sys::signal_through_pidfd(process.borrowed(), Signal::SIGTERM as i32, false).unwrap(); + sys::tests::signal_through_pidfd(process.borrowed(), Signal::SIGTERM as i32, false) + .unwrap(); let mut system = TestSystem::new(Paths::default()); let config = AgentConfig::layered(b"nvx_identity=1000:1000", c"workload".into(), c"/".into()) diff --git a/nvx-rs/src/guest/mappings.rs b/nvx-rs/src/guest/mappings.rs index 732b57628..d7ac38c05 100644 --- a/nvx-rs/src/guest/mappings.rs +++ b/nvx-rs/src/guest/mappings.rs @@ -45,7 +45,7 @@ pub use self::mapper::{ ApplyError, Mapper, MappingError, Reserved, SourceRoot, Step, TargetRoot, TeardownError, UnmountFailure, bind, }; -pub(crate) use self::mapper::{Mapped, bind_with, unbind_with}; +pub(crate) use self::mapper::{Mapped, unbind_with}; pub use self::table::{MappingTable, Refusal, TableError}; /// The longest source or target of a [`Mapping`], in bytes: Linux's `PATH_MAX` without its diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 2fef49f0c..b3cf61b8a 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -1,13 +1,8 @@ //! The programs that run in NVX's Linux guests, which ship as `/sbin/nvx-*`, and the libraries //! on which any guest init or agent builds: [`nvxs`], the guest's I/O driver of the NVXS session; -//! [`terminal`], the terminal on which it serves that session; [`spawn`], through which it starts -//! a program; [`entry`], through which a program that it starts enters an execution's -//! namespaces, view, root, and identity; [`process`], through which it collects the status of -//! each child that it names, and of no other; [`cgroup`], through which it places, accounts, -//! kills, and settles the processes of its executions; [`mappings`], the host mappings that it -//! mounts, and its account of a host's mapping table; [`exec`], which drives each program as one -//! execution, with its launch, its bounded output and input, its ending, and its outcome; and, -//! on x86-64, [`snapshot`], which composes a strictly checked Fresh capture by one caller deadline, +//! [`terminal`], the terminal on which it serves that session; [`mappings`], the host mappings +//! that it mounts, and its account of a host's mapping table; and, on x86-64, [`snapshot`], which +//! composes a strictly checked Fresh capture by one caller deadline, //! without execution control, with nothing mounted from the devices that a template's clones //! replace, and which repairs each clone that a restore of the snapshot starts before it returns: //! it activates the restore's memory target, reseeds the kernel's random number generator, and @@ -23,8 +18,8 @@ //! The guest's space, the root that an init assembles and what it mounts in it, is each init's //! own. NVX's own space is internal reference code, which NVX's guest programs use: it is hidden //! from this documentation and unstable, and it becomes crate-private once nothing outside NVX -//! uses it. It is `root`, which bootstraps the guest for a layered root, assembles the root, -//! mounts live shares and mappings in it, and tears it down; `view`, which sets the VM's hostname, +//! uses it. It is `root`, which assembles a layered root, mounts live shares and mappings in it, +//! and tears it down; `view`, which sets the VM's hostname, //! machine ID, and resolver file, whose binds the root owns; `gpt`, which reads the partition //! table of a GPT disk that the kernel attached as a block device; the mapper of `mappings`, which //! mounts host mappings beneath the descriptors of the roots that it names; and the activation's @@ -127,7 +122,6 @@ pub use generation::GenerationId; #[cfg(target_arch = "x86_64")] pub mod activation; -pub mod cgroup; /// Compatibility name for the guest's activation. #[cfg(target_arch = "x86_64")] pub use activation as clone; @@ -136,8 +130,6 @@ pub mod console_pending; #[cfg(target_arch = "x86_64")] #[doc(hidden)] pub mod device_io; -pub mod entry; -pub mod exec; #[cfg(target_arch = "x86_64")] #[doc(hidden)] pub mod exit; @@ -156,13 +148,11 @@ pub mod mappings; #[doc(hidden)] pub mod mmio_write; pub mod nvxs; -pub mod process; pub mod reseed; #[doc(hidden)] pub mod root; #[cfg(target_arch = "x86_64")] pub mod snapshot; -pub mod spawn; pub mod terminal; #[cfg(target_arch = "x86_64")] pub mod time; diff --git a/nvx-rs/src/guest/process.rs b/nvx-rs/src/guest/process.rs deleted file mode 100644 index 26327ebfc..000000000 --- a/nvx-rs/src/guest/process.rs +++ /dev/null @@ -1,227 +0,0 @@ -//! Collecting the statuses of the processes that a guest init or agent starts. -//! -//! A process has one status collector: the caller. Nothing in the guest libraries waits for an -//! arbitrary child, which could take the status of a child that another part of the caller -//! started. A [`Child`] waits only for the child that it names, with `waitid` on the child's -//! pidfd or on its process ID, and leaves the status of every other child to the caller. -//! -//! A caller that must collect the status of every child, such as a PID 1 that adopts orphaned -//! processes, keeps its own `waitpid(-1)` loop instead, and hands each status that it collects to -//! the [`Child`] that it belongs to with [`Child::observe`]: -//! -//! ```no_run -//! use std::process::ExitStatus; -//! -//! use nvx::guest::process::Child; -//! use nvx::guest::spawn::Command; -//! -//! /// The caller's own `waitpid(-1, WNOHANG)` loop, which returns the ID and the status of one -//! /// child that has ended. -//! fn reap_any() -> Option<(u32, ExitStatus)> { -//! # unimplemented!() -//! } -//! -//! # fn main() -> std::io::Result<()> { -//! let spawned = Command::new(c"/bin/true".to_owned(), vec![c"true".to_owned()]).spawn()?; -//! let mut child = Child::new(spawned.id())?; -//! while child.status().is_none() { -//! if let Some((id, status)) = reap_any() { -//! child.observe(id, status); -//! } -//! } -//! # Ok(()) -//! # } -//! ``` -//! -//! Any other caller lets each [`Child`] collect its own status, for example once its -//! [pidfd](Child::pidfd) polls readable: -//! -//! ```no_run -//! use nvx::guest::process::Child; -//! use nvx::guest::spawn::Command; -//! -//! # fn main() -> std::io::Result<()> { -//! let spawned = Command::new(c"/bin/true".to_owned(), vec![c"true".to_owned()]).spawn()?; -//! let mut child = Child::with_pidfd(spawned.id())?; -//! // Poll `child.pidfd()` with the caller's other descriptors until it is readable, then: -//! let status = child.wait()?; -//! assert!(status.success()); -//! # Ok(()) -//! # } -//! ``` - -use std::io; -use std::os::fd::{AsFd, BorrowedFd, OwnedFd}; -use std::os::unix::process::ExitStatusExt; -use std::process::ExitStatus; - -use super::sys::{self, Errno, ProcessId, WaitTarget}; - -/// A child process that the caller started and whose status it collects. -/// -/// Unlike [`std::process::Child`], it neither starts the process nor holds its standard streams: -/// it names a child of the caller, for example one that [`spawn`](super::spawn) started, and -/// keeps its status once it has one. Dropping it neither waits for the child nor stops it. -/// -/// Its status is the one with which it ended: an exit, or a terminating signal. A stop does not -/// end it, not even one that a wait reports to the caller as the child's tracer. A wait names the -/// child whatever signal the child reports its end with: one that `clone` started with an exit -/// signal other than `SIGCHLD`, or none, reports with that signal until it runs a program. -#[derive(Debug)] -pub struct Child { - /// The child's process ID. - process: ProcessId, - /// A pidfd that refers to the child, through which waits name it. - pidfd: Option, - /// The status with which the child ended, once a wait or an observation collected it. - status: Option, -} - -impl Child { - /// Names the caller's child `id`, whose status nothing has collected yet. Its waits use - /// `waitid` on the process ID, which names the child for as long as its status stays - /// uncollected: a caller with a collector of its own hands the child every status that - /// the collector takes for `id`, with [`observe`](Self::observe), before the child waits, as - /// the process ID may name another process afterwards. A pidfd keeps naming the child - /// whatever collects its status; see [`with_pidfd`](Self::with_pidfd). - /// - /// An ID of 0, or one above `i32::MAX`, which no process can have, is `EINVAL`. - pub fn new(id: u32) -> io::Result { - Ok(Self { - process: process_id(id)?, - pidfd: None, - status: None, - }) - } - - /// Names the caller's child `id`, whose status nothing has collected yet, by a new pidfd, - /// which [`pidfd`](Self::pidfd) lends for polling: it polls readable once the child ends. - /// Its waits use `waitid` on the pidfd, which keeps naming the child even after another - /// collector takes its status. The pidfd is opened at once, while the process ID still names - /// the child. - /// - /// An ID of 0, or one above `i32::MAX`, is `EINVAL`, and one that names no process `ESRCH`. - pub fn with_pidfd(id: u32) -> io::Result { - let process = process_id(id)?; - let pidfd = sys::open_pidfd(process)?; - Ok(Self { - process, - pidfd: Some(pidfd.into()), - status: None, - }) - } - - /// Names the caller's child `id` by `pidfd`, a pidfd for it that the caller holds, such as the - /// one that `clone3` returns with `CLONE_PIDFD`, which [`pidfd`](Self::pidfd) lends for - /// polling. Its waits use `waitid` on the pidfd, as with [`with_pidfd`](Self::with_pidfd). - /// Nothing is opened by the child's ID, so a collector of the caller's may have taken the - /// child's status already: a wait is `ECHILD` then, and that collector hands the status over - /// with [`observe`](Self::observe). A pidfd that its opener made nonblocking, as - /// `PIDFD_NONBLOCK` does, makes [`wait`](Self::wait) fail with `WouldBlock` while the child - /// runs, as `waitid` through it does; [`try_wait`](Self::try_wait) and the pidfd's polls work - /// as with any other, and so does an execution that adopts the child, which never blocks on - /// its status. - /// - /// The pidfd must refer to the process `id`, which is not checked: statuses are matched to the - /// child by that ID, and an execution signals the process group that the child leads by that - /// ID before Linux 6.9. - /// - /// An ID of 0, or one above `i32::MAX`, is `EINVAL`. - pub fn from_pidfd(id: u32, pidfd: OwnedFd) -> io::Result { - Ok(Self { - process: process_id(id)?, - pidfd: Some(pidfd), - status: None, - }) - } - - /// Opens a pidfd for the child by its ID, unless the child has one already or has ended: the - /// ID names the child alone only while its status stays uncollected, which the caller - /// guarantees until this returns. - pub(crate) fn open_pidfd(&mut self) -> io::Result<()> { - if self.pidfd.is_none() && self.status.is_none() { - self.pidfd = Some(sys::open_pidfd(self.process)?.into()); - } - Ok(()) - } - - /// Returns the child's process ID. - pub fn id(&self) -> u32 { - self.process.raw().unsigned_abs() - } - - /// Returns the child's process ID for the crate's own callers. - pub(crate) fn process(&self) -> ProcessId { - self.process - } - - /// Lends the child's pidfd, if [`with_pidfd`](Self::with_pidfd) opened one or - /// [`from_pidfd`](Self::from_pidfd) took one, which polls readable once the child ends. - pub fn pidfd(&self) -> Option> { - self.pidfd.as_ref().map(AsFd::as_fd) - } - - /// Returns the status with which the child ended, once a wait or - /// [`observe`](Self::observe) collected it. - pub fn status(&self) -> Option { - self.status - } - - /// Hands the child the status that the caller's own collector, such as its `waitpid(-1)` - /// loop, collected for the process `id`, and returns whether it was this child's final - /// status, which the child then keeps. - /// - /// Only a status with which a process ended counts: an exit, or a terminating signal. A stop - /// or a continuation, which a collector that waits with `WUNTRACED` or `WCONTINUED` reports - /// while the process lives, is not recorded, and this returns `false`. Only the first final - /// status counts too: once the child has one, a later status for its ID belongs to another - /// process that the ID was reused for, and this returns `false`. - pub fn observe(&mut self, id: u32, status: ExitStatus) -> bool { - let ended = status.code().is_some() || status.signal().is_some(); - if !ended || self.status.is_some() || id != self.id() { - return false; - } - self.status = Some(status); - true - } - - /// Collects the child's status if it has ended, without blocking, and returns it, or `None` - /// while the child runs. Once the child has a status, this returns it again. - /// - /// It collects no other child's status. A child that a pidfd names, and whose status another - /// collector took without handing it over with [`observe`](Self::observe), is `ECHILD`. - pub fn try_wait(&mut self) -> io::Result> { - self.collect(false) - } - - /// Waits until the child ends, collects its status, and returns it. Once the child has a - /// status, this returns it again at once. - /// - /// It collects no other child's status. A child that a pidfd names, and whose status another - /// collector took without handing it over with [`observe`](Self::observe), is `ECHILD`. - pub fn wait(&mut self) -> io::Result { - self.collect(true)? - .ok_or_else(|| io::Error::from(Errno::ECHILD)) - } - - /// Returns the child's status, which a wait for this child alone collects if the child has - /// none yet, blocking until it ends when `block` is set. - fn collect(&mut self, block: bool) -> io::Result> { - if self.status.is_none() { - let target = match &self.pidfd { - Some(pidfd) => WaitTarget::Pidfd(pidfd.as_fd()), - None => WaitTarget::Process(self.process), - }; - self.status = sys::wait_named(target, block)?.map(ExitStatus::from_raw); - } - Ok(self.status) - } -} - -/// Returns the process ID `id`, or `EINVAL` for one that no process can have. -fn process_id(id: u32) -> io::Result { - ProcessId::from_positive(i64::from(id)).ok_or_else(|| io::Error::from(Errno::EINVAL)) -} - -#[cfg(test)] -mod tests; diff --git a/nvx-rs/src/guest/process/tests.rs b/nvx-rs/src/guest/process/tests.rs deleted file mode 100644 index b89d2def8..000000000 --- a/nvx-rs/src/guest/process/tests.rs +++ /dev/null @@ -1,243 +0,0 @@ -use std::time::Duration; - -use nix::poll::{PollFd, PollFlags, PollTimeout}; - -use super::*; -use crate::guest::spawn::tests::shell; - -/// Spawns `/bin/sh -c script` and returns its process ID. -fn spawn_shell(script: &str) -> u32 { - shell(script).spawn().unwrap().id() -} - -/// Returns whether `pidfd` polls readable within `timeout`. -fn readable(pidfd: BorrowedFd<'_>, timeout: Duration) -> bool { - let mut descriptors = [PollFd::new(pidfd, PollFlags::POLLIN)]; - let timeout = PollTimeout::try_from(timeout).unwrap(); - nix::poll::poll(&mut descriptors, timeout).unwrap() == 1 -} - -/// Spawns a child that ends at once, and waits until it has ended without collecting its -/// status. -fn unrelated_zombie() -> std::process::Child { - let child = std::process::Command::new("/bin/true").spawn().unwrap(); - let process = ProcessId::from_child(&child); - while !sys::tests::has_exited(process).unwrap() { - std::thread::yield_now(); - } - child -} - -#[test] -fn a_named_child_s_status_is_collected_and_no_other() { - let mut unrelated = unrelated_zombie(); - for (script, code, signal) in [ - ("exit 3", Some(3), None), - ("kill -TERM $$", None, Some(nix::libc::SIGTERM)), - // A real-time signal, which `nix`'s wait decoding cannot represent. - ("kill -40 $$", None, Some(40)), - ] { - for with_pidfd in [false, true] { - let id = spawn_shell(script); - let mut child = if with_pidfd { - Child::with_pidfd(id).unwrap() - } else { - Child::new(id).unwrap() - }; - assert_eq!(child.id(), id); - assert_eq!(child.pidfd().is_some(), with_pidfd); - let status = child.wait().unwrap(); - assert_eq!((status.code(), status.signal()), (code, signal), "{script}"); - // The status stays, and no further wait takes place. - assert_eq!(child.try_wait().unwrap(), Some(status)); - assert_eq!(child.wait().unwrap(), status); - assert_eq!(child.status(), Some(status)); - } - } - // Every wait named its child, so the unrelated child's status is still there. - assert!(unrelated.try_wait().unwrap().unwrap().success()); -} - -#[test] -fn a_pidfd_polls_readable_once_its_child_ends() { - let (reader, mut writer) = std::io::pipe().unwrap(); - let spawned = shell("read line").stdin(reader).spawn().unwrap(); - let mut child = Child::with_pidfd(spawned.id()).unwrap(); - let pidfd = child.pidfd().unwrap().try_clone_to_owned().unwrap(); - assert!(!readable(pidfd.as_fd(), Duration::ZERO)); - assert_eq!(child.try_wait().unwrap(), None); - std::io::Write::write_all(&mut writer, b"go\n").unwrap(); - assert!(readable(pidfd.as_fd(), Duration::from_secs(10))); - assert!(child.try_wait().unwrap().unwrap().success()); -} - -#[test] -fn a_child_named_by_the_caller_s_pidfd_is_waited_for_through_it() { - let id = spawn_shell("exit 4"); - let pidfd = sys::open_pidfd(process_id(id).unwrap()).unwrap(); - let mut child = Child::from_pidfd(id, pidfd.into()).unwrap(); - assert_eq!(child.id(), id); - assert!(child.pidfd().is_some()); - assert_eq!(child.wait().unwrap().code(), Some(4)); - // A collector of the caller's that took the status first leaves the pidfd's wait none, - // until it hands the status over. - let id = spawn_shell("exit 5"); - let pidfd = sys::open_pidfd(process_id(id).unwrap()).unwrap(); - let collected = Child::new(id).unwrap().wait().unwrap(); - let mut child = Child::from_pidfd(id, pidfd.into()).unwrap(); - assert_eq!(sys::errno_of(&child.try_wait().unwrap_err()), Errno::ECHILD); - assert!(child.observe(id, collected)); - assert_eq!(child.wait().unwrap().code(), Some(5)); - // No process can have these IDs. - for id in [0, 1 << 31] { - let pidfd = OwnedFd::from(std::fs::File::open("/dev/null").unwrap()); - let error = Child::from_pidfd(id, pidfd).unwrap_err(); - assert_eq!(sys::errno_of(&error), Errno::EINVAL); - } -} - -#[test] -fn a_child_that_reports_its_end_with_no_signal_is_waited_for() { - let id = sys::tests::spawn_child_without_exit_signal(7) - .unwrap() - .raw() - .unsigned_abs(); - let mut child = Child::with_pidfd(id).unwrap(); - assert!(readable(child.pidfd().unwrap(), Duration::from_secs(10))); - assert_eq!(child.try_wait().unwrap().unwrap().code(), Some(7)); - let id = sys::tests::spawn_child_without_exit_signal(8) - .unwrap() - .raw() - .unsigned_abs(); - assert_eq!(Child::new(id).unwrap().wait().unwrap().code(), Some(8)); -} - -#[test] -fn an_observed_status_belongs_to_its_child_alone() { - // The test collects the child's status itself, as a PID 1's own loop would. - let mut process = std::process::Command::new("/bin/sh") - .args(["-c", "exit 7"]) - .spawn() - .unwrap(); - let id = process.id(); - let collected = process.wait().unwrap(); - let mut child = Child::new(id).unwrap(); - assert!(!child.observe(id + 1, ExitStatus::from_raw(0))); - assert_eq!(child.status(), None); - assert!(child.observe(id, collected)); - // A later status for the ID belongs to another process. - assert!(!child.observe(id, ExitStatus::from_raw(0))); - assert_eq!(child.status().unwrap().code(), Some(7)); - // The observed status answers the waits, which would otherwise find no child. - assert_eq!(child.try_wait().unwrap(), Some(collected)); - assert_eq!(child.wait().unwrap(), collected); -} - -#[test] -fn stops_and_continuations_do_not_end_an_observed_child() { - // The statuses that `waitpid` reports with `WUNTRACED` and `WCONTINUED` for a child that - // lives on. - let stopped = ExitStatus::from_raw((nix::libc::SIGSTOP << 8) | 0x7f); - let continued = ExitStatus::from_raw(0xffff); - assert_eq!(stopped.stopped_signal(), Some(nix::libc::SIGSTOP)); - assert!(continued.continued()); - let mut child = Child::new(4242).unwrap(); - for status in [stopped, continued] { - assert!(!child.observe(4242, status), "{status:?}"); - assert_eq!(child.status(), None); - } - // The final status still counts. - let killed = ExitStatus::from_raw(nix::libc::SIGKILL); - assert!(child.observe(4242, killed)); - assert_eq!(child.status().unwrap().signal(), Some(nix::libc::SIGKILL)); - assert!(!child.observe(4242, stopped)); - assert_eq!(child.status(), Some(killed)); -} - -/// Returns whether the process `id` reaches a tracing stop within ten seconds, or `false` once -/// it has ended instead. -fn stops_for_its_tracer(id: u32) -> bool { - let deadline = std::time::Instant::now() + Duration::from_secs(10); - loop { - let stat = std::fs::read_to_string(format!("/proc/{id}/stat")).unwrap(); - // The state follows the command name, in parentheses, which may hold any character. - match stat - .rsplit_once(") ") - .and_then(|(_, rest)| rest.chars().next()) - { - Some('t') => return true, - Some('Z') => return false, - _ => {} - } - assert!( - std::time::Instant::now() < deadline, - "{id} neither stopped nor ended" - ); - std::thread::sleep(Duration::from_millis(5)); - } -} - -#[test] -fn a_traced_child_s_stop_does_not_end_it() { - for with_pidfd in [false, true] { - let id = sys::tests::spawn_traced_child(7).raw().unsigned_abs(); - let mut child = if with_pidfd { - Child::with_pidfd(id) - } else { - Child::new(id) - } - .unwrap(); - if !stops_for_its_tracer(id) { - // The system refuses tracing, which leaves nothing to check. - assert_eq!(child.wait().unwrap().code(), Some(99)); - return; - } - // A wait reports the stop to the child's tracer whatever its options, but the child - // still runs. - assert_eq!(child.try_wait().unwrap(), None); - assert_eq!(child.status(), None); - sys::tests::continue_traced(child.process()); - assert_eq!(child.wait().unwrap().code(), Some(7)); - } - // A blocking wait goes on past the stop until the child ends, here by `SIGKILL`. - let id = sys::tests::spawn_traced_child(7).raw().unsigned_abs(); - let mut child = Child::new(id).unwrap(); - assert!(stops_for_its_tracer(id)); - let killer = std::thread::spawn(move || { - std::thread::sleep(Duration::from_millis(200)); - let process = nix::unistd::Pid::from_raw(i32::try_from(id).unwrap()); - nix::sys::signal::kill(process, nix::sys::signal::Signal::SIGKILL).unwrap(); - }); - assert_eq!(child.wait().unwrap().signal(), Some(nix::libc::SIGKILL)); - killer.join().unwrap(); -} - -#[test] -fn a_child_whose_status_another_collector_took_is_echild() { - let mut process = std::process::Command::new("/bin/true").spawn().unwrap(); - let mut child = Child::with_pidfd(process.id()).unwrap(); - // Another collector takes the status, and hands it over to no one. - process.wait().unwrap(); - assert_eq!( - child.try_wait().unwrap_err().raw_os_error(), - Some(Errno::ECHILD as i32) - ); - assert_eq!( - child.wait().unwrap_err().raw_os_error(), - Some(Errno::ECHILD as i32) - ); - assert_eq!(child.status(), None); -} - -#[test] -fn ids_that_no_process_can_have_are_refused() { - for id in [0, i32::MAX.unsigned_abs() + 1, u32::MAX] { - for result in [Child::new(id), Child::with_pidfd(id)] { - assert_eq!( - result.unwrap_err().raw_os_error(), - Some(Errno::EINVAL as i32), - "{id}" - ); - } - } -} diff --git a/nvx-rs/src/guest/root.rs b/nvx-rs/src/guest/root.rs index 5786aa577..b6017272e 100644 --- a/nvx-rs/src/guest/root.rs +++ b/nvx-rs/src/guest/root.rs @@ -1,14 +1,10 @@ -//! Operations that bootstrap a guest for a layered root, assemble the root, mount live shares and -//! mappings in it, and the host export from which mappings come, and tear it down, for a guest -//! init or agent, whether or not it is PID 1. +//! Operations that assemble a layered root, mount live shares and mappings in it, and the host +//! export from which mappings come, and tear it down, for a guest init or agent, whether or not +//! it is PID 1. //! //! This module is NVX's internal reference code of the guest's space, hidden from the //! documentation and unstable: it becomes crate-private once nothing outside NVX uses it. //! -//! - [`bootstrap`](fn@bootstrap) prepares the guest once: it creates the agent's and the -//! workload's sibling cgroups of a [`Layout`](super::cgroup::Layout), moves every process of -//! their parent into the agent's, applies the agent's and the workload's resource values, and -//! keeps user namespaces unavailable. //! - [`assemble`] finds the block devices of an [`Assembly`] by their MMIO windows, checks and //! flushes them, and mounts the root: an overlay, mounted `nosuid,nodev` with the option set of //! the caller's [`OverlayMode`], which it checks the kernel applied, of the read-only layers of @@ -39,7 +35,7 @@ //! failures. //! //! A failure is an [`Error`] that names what failed. The operations need the guest's root user, -//! in the guest's initial mount and cgroup namespaces. They never collect a child's status, +//! in the guest's initial mount namespace. They never collect a child's status, //! change the caller's credentials, start an async runtime, or power the VM off, and only //! `Root::switch` changes the caller's root directory, so a caller that is not PID 1 keeps its own //! reaping, transport, and shutdown. @@ -132,19 +128,12 @@ //! use std::time::Duration; //! //! use nvx::gpt::Expected; -//! use nvx::guest::cgroup::Layout; //! use nvx::guest::mappings::{Access, Mapping}; -//! use nvx::guest::root::{self, Assembly, Device, Image, OverlayMode, Resources, Upper}; +//! use nvx::guest::root::{self, Assembly, Device, Image, OverlayMode, Upper}; //! use nvx::protocol::export::HostExport; //! use nvx::protocol::root::DISTRO_MMIO_BASE; //! //! # fn main() -> Result<(), Box> { -//! let layout = Layout::new("/sys/fs/cgroup/agent", "/sys/fs/cgroup/container")?; -//! let resources = Resources { -//! agent_memory_low: Some(16 << 20), -//! ..Resources::default() -//! }; -//! let bootstrap = root::bootstrap(&layout, &resources)?; //! let mut root = root::assemble(&Assembly { //! runtime: "/run/nvx".into(), //! distro: Device { mmio: DISTRO_MMIO_BASE, sectors: None }, @@ -166,14 +155,13 @@ //! root.map(&source, &[Mapping::new("0/src", "/src", Access::ReadOnly)?])?; //! // Dropping the export's root here lets the teardown unmount the export. //! } -//! // Run the workload in `bootstrap.workload()`, in the root at `root.path()`; then: +//! // Run the workload in the root at `root.path()`; then: //! root.teardown()?; //! # Ok(()) //! # } //! ``` mod block; -mod bootstrap; mod kernel; mod share; #[cfg(test)] @@ -202,7 +190,6 @@ use crate::protocol::root::RAM_UPPER_CAPS; use crate::uuid::Uuid; pub(crate) use self::block::window_directories; -pub use self::bootstrap::{Bootstrap, BootstrapStep, Resources, bootstrap}; pub(crate) use self::kernel::Paths; pub use self::share::{ShareChild, ShareKind, ShareRefusal}; @@ -267,7 +254,7 @@ const RUNTIME_FILE_SYSTEMS: [(RuntimeMounts, &CStr, &FileSystem); 6] = [ (RuntimeMounts::Run, c"run", &sys::RUN), ]; /// The mode of a directory that [`Root::mount_runtime`] creates for a mount, but for what the -/// umask withholds, as an execution's entry creates it. +/// umask withholds. const RUNTIME_DIRECTORY_MODE: u32 = 0o755; /// The most bytes of options that `mount(2)` passes a file system whole: one 4 KiB page, the @@ -881,7 +868,7 @@ impl Root { /// Mounts the runtime's file systems in the root, as an init does once for its VM before it /// switches into the root. It first makes every mount of the root's tree `nosuid` and `nodev`, - /// then mounts, in order, an execution's file systems, with an execution's options: + /// then mounts, in order, these file systems, with these options: /// /// - procfs at `/proc`, `nosuid`, `nodev`, and `noexec`; /// - sysfs at `/sys`, read-only, `nosuid`, `nodev`, and `noexec`; @@ -894,9 +881,7 @@ impl Root { /// /// With a `time_status`, the directory that holds the guest's time status, it then binds that /// directory read-only, `nosuid`, and `nodev` at `/run/nvx/time`, creating `/run/nvx` and - /// `/run/nvx/time` on the `/run` tmpfs with mode 0755, as an execution's - /// [`Entry::bind_runtime`](super::entry::Entry::bind_runtime) binds it. Without one, it binds - /// nothing there. + /// `/run/nvx/time` on the `/run` tmpfs with mode 0755. Without one, it binds nothing there. /// /// It opens each mount point beneath the root, through no symbolic link, and creates one that /// is missing with mode 0755, but for what the umask withholds. A mount point where something @@ -1032,11 +1017,10 @@ impl Root { } /// Switches into the root, as an init does once for its VM, after the mounts and the - /// descriptors that it needs: it moves the root's mount onto `/`, as an execution's child - /// moves its root in its own mount namespace, then makes the root the calling process's root - /// directory, with `chroot`, and its working directory, with `chdir`. The move is in the - /// caller's mount namespace, and the root and working directories are those that the caller's - /// threads share, but for a thread that unshared them. + /// descriptors that it needs: it moves the root's mount onto `/`, then makes the root the + /// calling process's root directory, with `chroot`, and its working directory, with `chdir`. + /// The move is in the caller's mount namespace, and the root and working directories are + /// those that the caller's threads share, but for a thread that unshared them. /// /// A root switches once, for good: a second switch is refused as [`SwitchError::Switched`]. /// Once its mount is on `/`, the root has switched, even if entering it then fails: from then @@ -1575,19 +1559,8 @@ pub(crate) fn bind_file_retained( /// Binds the file or directory at `path`, which must be of `kind`, read-only at `target` beneath /// `root`, with `kernel`, through the walk and attach of [`mappings::bind`] from a source root at -/// `path`'s parent directory, and returns the bind. -pub(crate) fn bind_runtime_file( - kernel: &mut dyn mappings::Kernel, - root: &TargetRoot, - path: &Path, - target: &str, - kind: Kind, -) -> Result { - let (source, mapping) = runtime_mapping(path, target, kind)?; - mappings::bind_with(kernel, root, &source, &mapping) -} - -/// Binds as [`bind_runtime_file`] does, retaining a failed bind that its caller must detach. +/// `path`'s parent directory, and returns the bind, retaining a failed bind that its caller must +/// detach. pub(crate) fn bind_runtime_file_retained( kernel: &mut dyn mappings::Kernel, root: &TargetRoot, @@ -2045,13 +2018,6 @@ impl RuntimeMounts { #[derive(Debug)] #[non_exhaustive] pub enum Error { - /// A step of [`bootstrap`](fn@bootstrap) failed. - Bootstrap { - /// The step that failed. - step: BootstrapStep, - /// The step's error. - error: io::Error, - }, /// The runtime directory's path is not absolute and canonical, holds a reserved character, /// or leads through a symbolic link. Runtime { @@ -2253,7 +2219,6 @@ pub enum Error { impl fmt::Display for Error { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match self { - Self::Bootstrap { step, error } => write!(formatter, "failed to {step}: {error}"), Self::Runtime { path } => write!( formatter, "the runtime directory is not a canonical absolute path without `,`, `:`, `\\`, \ diff --git a/nvx-rs/src/guest/root/bootstrap.rs b/nvx-rs/src/guest/root/bootstrap.rs deleted file mode 100644 index 9d3821834..000000000 --- a/nvx-rs/src/guest/root/bootstrap.rs +++ /dev/null @@ -1,306 +0,0 @@ -//! The guest's bootstrap: the agent's and the workload's sibling cgroups, and user namespaces. - -use std::fmt; -use std::fs; -use std::io; -use std::path::Path; - -use super::Error; -use super::kernel::{Kernel, Paths, SystemKernel}; -use crate::guest::cgroup::{self, ExecutionCgroup, Layout}; -use crate::guest::parse::parse_decimal; -use crate::guest::sys::{self, Errno, MountFlags}; - -/// The flag of a kernel thread among a process's flags in `/proc/PID/stat`: Linux's -/// `PF_KTHREAD`. -const KERNEL_THREAD: u64 = 0x0020_0000; -/// The most passes in which the placement moves the parent cgroup's processes. A process that a -/// moved one starts meanwhile starts in the parent too, and the next pass moves it. -const PLACEMENT_PASSES: usize = 16; -/// The sysctl, below procfs, of the user namespaces that each user may create. -const USER_NAMESPACES: &str = "sys/user/max_user_namespaces"; -/// The calling process's user namespace, below procfs, which only a kernel with user namespaces -/// has. -const OWN_USER_NAMESPACE: &str = "self/ns/user"; - -/// The resource values that [`bootstrap`] applies, each only when the caller gives it. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -pub struct Resources { - /// The agent's `memory.low`, in bytes: memory of the agent's cgroup that the kernel reclaims - /// only when no unprotected memory is left. - pub agent_memory_low: Option, - /// The workload cgroup's `memory.max`, in bytes. - pub workload_memory_max: Option, - /// The workload cgroup's `pids.max`: the most processes in it and below it. - pub workload_pids_max: Option, -} - -/// A step of [`bootstrap`], which an [`Error::Bootstrap`] names. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum BootstrapStep { - /// Checking that the layout's cgroups are siblings. - Layout, - /// Setting `user.max_user_namespaces` to 0. - UserNamespaces, - /// Mounting the cgroup v2 hierarchy at the cgroups' parent. - Hierarchy, - /// Creating the agent's cgroup. - AgentCgroup, - /// Finding the parent cgroup's processes, and moving them into the agent's cgroup. - Placement, - /// Enabling the `memory` controller for the agent's cgroup unless it has it, and writing the - /// agent's `memory.low`. - AgentMemoryLow, - /// Preparing the workload cgroup with [`ExecutionCgroup::prepare`]: creating it, enabling - /// the controllers that it lacks, and writing its `memory.max` and `pids.max`. - WorkloadCgroup, -} - -impl fmt::Display for BootstrapStep { - /// Writes the step as the verb phrase of a failure: `failed to {step}`. - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter.write_str(match self { - Self::Layout => "check that the agent's and the workload cgroups are siblings", - Self::UserNamespaces => "disable user namespaces", - Self::Hierarchy => "mount the cgroup hierarchy", - Self::AgentCgroup => "create the agent's cgroup", - Self::Placement => "place the agent's processes in its cgroup", - Self::AgentMemoryLow => "set the agent's memory reservation", - Self::WorkloadCgroup => "prepare the workload cgroup", - }) - } -} - -/// The cgroups that [`bootstrap`] prepared, and the processes that it moved into the agent's. -#[derive(Clone, Debug)] -pub struct Bootstrap { - /// The agent's cgroup and the workload cgroup. - layout: Layout, - /// The handle of the workload cgroup itself. - workload: ExecutionCgroup, - /// The processes that the placement moved, in the order in which it moved them. - moved: Vec, -} - -impl Bootstrap { - /// Returns the agent's cgroup and the workload cgroup. - pub fn layout(&self) -> &Layout { - &self.layout - } - - /// Returns the handle of the workload cgroup itself, through which a caller that runs one - /// execution at a time places, kills, and settles it, and below which another prepares the - /// cgroup of each execution. - pub fn workload(&self) -> &ExecutionCgroup { - &self.workload - } - - /// Returns the IDs of the processes that the bootstrap moved into the agent's cgroup, in the - /// order in which it moved them. - pub fn moved(&self) -> &[u32] { - &self.moved - } -} - -/// Bootstraps the guest for a layered root: user namespaces, and the agent's and the workload's -/// cgroups of `layout`, which must be siblings. In order, it: -/// -/// 1. sets `user.max_user_namespaces` to 0, so that no process creates a user namespace, even on -/// a kernel that supports them. A kernel without sysctls lacks it, which the bootstrap accepts -/// only when the kernel has no user namespaces either, and otherwise fails with `ENOENT`; -/// 2. mounts the cgroup v2 hierarchy, `nosuid,nodev,noexec`, at the cgroups' parent, unless the -/// parent is a cgroup already, which its `cgroup.controllers` shows; -/// 3. creates the agent's cgroup unless it exists; -/// 4. moves every process of the parent cgroup into the agent's cgroup, except kernel threads, -/// which stay: the caller, and every daemon that its prelude started. It repeats until no -/// such process is left, for at most 16 passes, so that one that a moved process starts -/// meanwhile moves too, and fails with `EBUSY` if one is still left; -/// 5. enables the `memory` controller in the parent's `cgroup.subtree_control` unless the -/// agent's cgroup has it, and writes the agent's `memory.low`; -/// 6. and prepares the workload cgroup with [`ExecutionCgroup::prepare`], as the managed agent -/// prepares its own: it creates the cgroup unless it exists, enables the `pids` controller, -/// and that of each value, in the parent's `cgroup.subtree_control`, in one write, unless the -/// cgroup has them, and writes the cgroup's `memory.max` and `pids.max`. -/// -/// Each value is written only when `resources` holds one, and step 5 runs only for the agent's -/// `memory.low`. A failure names its step, and leaves the earlier steps' changes in place. The -/// bootstrap collects no status, and starts and stops no process. -pub fn bootstrap(layout: &Layout, resources: &Resources) -> Result { - bootstrap_with(&mut SystemKernel, &Paths::default(), layout, resources) -} - -/// Implements [`bootstrap`] with `kernel`, and with procfs at `paths.proc`. -pub(super) fn bootstrap_with( - kernel: &mut dyn Kernel, - paths: &Paths, - layout: &Layout, - resources: &Resources, -) -> Result { - let failed = |step| move |error| Error::Bootstrap { step, error }; - let parent = match (layout.agent().parent(), layout.workload().parent()) { - (Some(agent), Some(workload)) if agent == workload => agent, - _ => return Err(failed(BootstrapStep::Layout)(Errno::EINVAL.into())), - }; - disable_user_namespaces(kernel, paths).map_err(failed(BootstrapStep::UserNamespaces))?; - mount_hierarchy(kernel, parent).map_err(failed(BootstrapStep::Hierarchy))?; - cgroup::create(layout.agent(), false).map_err(failed(BootstrapStep::AgentCgroup))?; - let moved = - place(kernel, paths, parent, layout.agent()).map_err(failed(BootstrapStep::Placement))?; - if let Some(low) = resources.agent_memory_low { - cgroup::provide(layout.agent(), parent, ["memory.low"]) - .and_then(|()| { - write( - kernel, - &layout.agent().join("memory.low"), - low.to_string().as_bytes(), - ) - }) - .map_err(failed(BootstrapStep::AgentMemoryLow))?; - } - let values = [ - ("memory.max", resources.workload_memory_max), - ("pids.max", resources.workload_pids_max), - ] - .into_iter() - .filter_map(|(file, value)| Some((file, value?.to_string()))) - .collect::>(); - let values: Vec<(&str, &str)> = values - .iter() - .map(|(file, value)| (*file, value.as_str())) - .collect(); - let workload = ExecutionCgroup::prepare(layout, None, &values) - .map_err(failed(BootstrapStep::WorkloadCgroup))?; - Ok(Bootstrap { - layout: layout.clone(), - workload, - moved, - }) -} - -/// Writes `value` to the control file `path` through `kernel`. -fn write(kernel: &mut dyn Kernel, path: &Path, value: &[u8]) -> io::Result<()> { - let path = sys::path_c_string(path)?; - Ok(kernel.write(&path, value)?) -} - -/// Sets `user.max_user_namespaces` to 0, through `kernel`, with procfs at `paths.proc`. Without -/// the sysctl, the kernel needs no limit if it has no user namespaces, and otherwise cannot be -/// limited, which is the write's `ENOENT`. -fn disable_user_namespaces(kernel: &mut dyn Kernel, paths: &Paths) -> io::Result<()> { - let Err(error) = write(kernel, &paths.proc.join(USER_NAMESPACES), b"0") else { - return Ok(()); - }; - if error.kind() != io::ErrorKind::NotFound { - return Err(error); - } - match fs::symlink_metadata(paths.proc.join(OWN_USER_NAMESPACE)) { - Err(missing) if missing.kind() == io::ErrorKind::NotFound => Ok(()), - Ok(_) => Err(error), - Err(other) => Err(other), - } -} - -/// Mounts the cgroup v2 hierarchy at `parent`, which must exist, unless `parent` is a cgroup. -fn mount_hierarchy(kernel: &mut dyn Kernel, parent: &Path) -> io::Result<()> { - match fs::symlink_metadata(parent.join("cgroup.controllers")) { - Ok(_) => return Ok(()), - Err(error) if error.kind() == io::ErrorKind::NotFound => {} - Err(error) => return Err(error), - } - if !fs::metadata(parent)?.is_dir() { - return Err(Errno::ENOTDIR.into()); - } - let target = sys::path_c_string(parent)?; - Ok(kernel.mount( - Some(c"cgroup2"), - &target, - Some(c"cgroup2"), - MountFlags::MS_NOSUID | MountFlags::MS_NODEV | MountFlags::MS_NOEXEC, - None, - )?) -} - -/// Moves every process of the cgroup `parent` into the cgroup `agent`, except kernel threads, in -/// passes until none is left, and returns the IDs that it moved. -fn place( - kernel: &mut dyn Kernel, - paths: &Paths, - parent: &Path, - agent: &Path, -) -> io::Result> { - let members = parent.join("cgroup.procs"); - let destination = sys::path_c_string(&agent.join("cgroup.procs"))?; - let mut moved = Vec::new(); - for _ in 0..PLACEMENT_PASSES { - let left = user_processes(paths, &members)?; - if left.is_empty() { - return Ok(moved); - } - for process in left { - match kernel.write(&destination, process.to_string().as_bytes()) { - Ok(()) => moved.push(process), - Err(Errno::ESRCH) => {} - Err(error) => return Err(error.into()), - } - } - } - // The last pass may have moved the last of them. - if user_processes(paths, &members)?.is_empty() { - Ok(moved) - } else { - Err(Errno::EBUSY.into()) - } -} - -/// Returns the processes that the membership file `members` lists, except kernel threads and -/// processes that have ended. -fn user_processes(paths: &Paths, members: &Path) -> io::Result> { - let mut left = Vec::new(); - for process in processes(&fs::read(members)?)? { - match fs::read(paths.proc.join(process.to_string()).join("stat")) { - Ok(stat) if is_kernel_thread(&stat).ok_or(Errno::EPROTO)? => {} - Ok(_) => left.push(process), - // The process has ended. - Err(error) if error.kind() == io::ErrorKind::NotFound => {} - Err(error) if error.raw_os_error() == Some(Errno::ESRCH as i32) => {} - Err(error) => return Err(error), - } - } - Ok(left) -} - -/// Parses a cgroup's `cgroup.procs`: one decimal process ID per line. -fn processes(members: &[u8]) -> io::Result> { - let Some(members) = members.strip_suffix(b"\n") else { - return if members.is_empty() { - Ok(Vec::new()) - } else { - Err(Errno::EPROTO.into()) - }; - }; - members - .split(|&byte| byte == b'\n') - .map(|line| { - parse_decimal(line) - .and_then(|process| u32::try_from(process).ok()) - .filter(|&process| process != 0) - .ok_or_else(|| Errno::EPROTO.into()) - }) - .collect() -} - -/// Returns whether the `/proc/PID/stat` record `stat` is a kernel thread's, or `None` for a -/// malformed record. Its flags are the seventh field after the command, which is parenthesized -/// and may hold any character, a `)` and spaces included. -fn is_kernel_thread(stat: &[u8]) -> Option { - let end = stat.iter().rposition(|&byte| byte == b')')?; - let flags = stat[end + 1..] - .split(|&byte| byte == b' ') - .filter(|field| !field.is_empty()) - .nth(6)?; - Some(parse_decimal(flags)? & KERNEL_THREAD != 0) -} - -#[cfg(test)] -mod tests; diff --git a/nvx-rs/src/guest/root/bootstrap/tests.rs b/nvx-rs/src/guest/root/bootstrap/tests.rs deleted file mode 100644 index 3aea21ba3..000000000 --- a/nvx-rs/src/guest/root/bootstrap/tests.rs +++ /dev/null @@ -1,47 +0,0 @@ -use super::*; - -#[test] -fn kernel_threads_are_known_by_their_flags() { - let user = b"512 (sh) S 1 512 512 34816 512 4194560 147 0 0 0 0 0 0 0 20 0 1 0 72\n"; - let kernel = b"2 (kthreadd) S 0 0 0 0 -1 2129984 0 0 0 0 0 0 0 0 20 0 1 0 1\n"; - assert_eq!(is_kernel_thread(user), Some(false)); - assert_eq!(is_kernel_thread(kernel), Some(true)); - // A command with a parenthesis and spaces of its own. - let tricky = b"77 (a) b (c)) S 1 77 77 0 -1 4194560 0 0 0 0 0 0 0 0 20 0 1 0 9\n"; - assert_eq!(is_kernel_thread(tricky), Some(false)); - for malformed in [ - &b""[..], - b"77 (sh S 1 77 77 0 -1 4194560", - b"77 (sh) S 1 77 77 0 -1", - b"77 (sh) S 1 77 77 0 -1 0x10 0", - ] { - assert_eq!( - is_kernel_thread(malformed), - None, - "{}", - malformed.escape_ascii() - ); - } -} - -#[test] -fn membership_lists_are_decimal_lines() { - assert_eq!(processes(b"").unwrap(), Vec::::new()); - assert_eq!(processes(b"1\n42\n4194304\n").unwrap(), [1, 42, 4_194_304]); - for malformed in [ - &b"1"[..], - b"\n", - b"1\n\n", - b"0\n", - b"x\n", - b"-1\n", - b"4294967296\n", - ] { - assert_eq!( - processes(malformed).unwrap_err().raw_os_error(), - Some(Errno::EPROTO as i32), - "{}", - malformed.escape_ascii() - ); - } -} diff --git a/nvx-rs/src/guest/root/kernel.rs b/nvx-rs/src/guest/root/kernel.rs index 5c58418c1..d3313e0da 100644 --- a/nvx-rs/src/guest/root/kernel.rs +++ b/nvx-rs/src/guest/root/kernel.rs @@ -5,10 +5,9 @@ //! What a test cannot do in a temporary directory, such as a mount, an unmount, a flush of a //! block device's buffers, a sync of a mounted file system, a reread of a disk's partition table, //! or a read of a GPT disk's partition table at the geometry that its block device reports, and -//! the writes to the kernel's control files, the check for a mount point, and the options of a -//! mount, whose effects and answers a test simulates, go through a [`Kernel`], which also performs -//! the mapper's operations for the root's live shares and mappings, the runtime's mounts, and the -//! switch into the root. +//! the check for a mount point and the options of a mount, whose effects and answers a test +//! simulates, go through a [`Kernel`], which also performs the mapper's operations for the root's +//! live shares and mappings, the runtime's mounts, and the switch into the root. use std::ffi::CStr; use std::fs::{self, File}; @@ -29,16 +28,16 @@ const MOUNT_TABLE: &str = "/proc/thread-self/mountinfo"; /// The kernel's operations that the root operations perform, other than reads: mounts and /// unmounts, flushes of a block device's buffers, syncs of a mounted file system, rereads of a -/// disk's partition table, writes of a control file, and the sleeps between two polls for a block -/// device; the check for a mount point and the options of a mount, which depend on them; since -/// only a block device reports its geometry, the read of a GPT disk's partition table; the -/// mapper's operations, through which the root mounts its live shares and mappings; the new -/// mounts, the restriction of the root's tree, and the device nodes of the runtime's file systems; -/// and the change of root directory into a switched root. +/// disk's partition table, and the sleeps between two polls for a block device; the check for a +/// mount point and the options of a mount, which depend on them; since only a block device +/// reports its geometry, the read of a GPT disk's partition table; the mapper's operations, +/// through which the root mounts its live shares and mappings; the new mounts, the restriction of +/// the root's tree, and the device nodes of the runtime's file systems; and the change of root +/// directory into a switched root. /// /// [`SystemKernel`] performs them. The tests' fake records the operations, simulates the effect -/// of a cgroup's membership file and its own mounts, reads a GPT disk's table from a regular -/// file, and fails the operations that a test chooses. +/// of its own mounts, reads a GPT disk's table from a regular file, and fails the operations that +/// a test chooses. pub(super) trait Kernel: MappingKernel + Send + Sync { /// Mounts `source` at `target`, as `mount(2)` does, with the file system's own options /// `data`. @@ -75,10 +74,6 @@ pub(super) trait Kernel: MappingKernel + Send + Sync { expected: &Expected, ) -> Result; - /// Writes `value` whole to the existing control file `path`, such as a cgroup's or a - /// sysctl's, and reports a failure to close it. - fn write(&mut self, path: &CStr, value: &[u8]) -> Result<(), Errno>; - /// Returns whether the directory `path` is a mount point: whether it is on another mount /// than its parent directory, as it is where a file system is mounted, or where a directory /// of the parent's own file system is bound. @@ -164,16 +159,6 @@ impl Kernel for SystemKernel { gpt::read_disk(file, expected) } - fn write(&mut self, path: &CStr, value: &[u8]) -> Result<(), Errno> { - let descriptor = sys::open( - path, - OpenFlags::O_WRONLY | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?; - descriptor.write_all(value)?; - descriptor.close() - } - fn is_mount_point(&mut self, path: &Path) -> io::Result { let parent = path.parent().ok_or(Errno::EINVAL)?; let mount = sys::mount_id(&sys::path_c_string(path)?)?; diff --git a/nvx-rs/src/guest/root/kernel/tests.rs b/nvx-rs/src/guest/root/kernel/tests.rs index 71e16f3d1..7760979c3 100644 --- a/nvx-rs/src/guest/root/kernel/tests.rs +++ b/nvx-rs/src/guest/root/kernel/tests.rs @@ -39,8 +39,6 @@ pub(crate) enum Operation { SyncFileSystem, /// A reread of a disk's partition table, which the disk's block device names. Reread, - /// A write of a control file, which its path names. - Write, /// An open beneath a directory, which the opened path names. Open, /// A creation of a directory or of a regular file, which its path names. @@ -95,8 +93,6 @@ pub(crate) enum Call { Reread(String), /// A read of the partition table of the GPT disk on a block device, which its node names. ReadDisk(String), - /// A write of a value to a control file. - Write(String, String), /// A copy, or an open, of the mount of a file by `open_tree`. OpenTree { /// The file's path. @@ -167,9 +163,6 @@ struct State { calls: Vec, /// The operations that fail, with their error, each at the path that names it. failures: Vec<(Operation, PathBuf, Errno)>, - /// The processes that writes of their ID to a membership file leave where they are, each - /// with the number of such writes left. - unmovable: Vec<(u32, usize)>, /// The mounts that no unmount has undone, in mount order: each mount point, with the /// mount's unique ID. mounted: Vec<(PathBuf, u64)>, @@ -225,10 +218,6 @@ impl State { /// A kernel that records each mount operation, performs no mount, unmount, flush, sync, or /// reread of a partition table, and fails the operations that a test chooses. /// -/// A write of a control file writes the regular file at its path, which it creates if -/// needed, except that a write of a process ID to a cgroup's `cgroup.procs` moves the ID -/// there from the `cgroup.procs` of the cgroup's parent, as the kernel moves the process. -/// /// It simulates its mounts by their mount points and unique IDs: a mount, a move of a tree /// onto a file, and a test's [`add_mount`](Self::add_mount) put a mount on a path, and an /// unmount takes it off. Each shows its whole file system, but one that a test adds with @@ -270,12 +259,6 @@ impl FakeKernel { }); } - /// Makes the next `writes` writes of the ID `process` to a membership file leave the - /// process where it is, as for a process that reappears as fast as it is moved. - pub(crate) fn pin(&self, process: u32, writes: usize) { - self.with(|state| state.unmovable.push((process, writes))); - } - /// Makes the kernel show `options` as the options of the mounts at `path` from now on. pub(in crate::guest::root) fn show(&self, path: impl Into, options: &str) { self.with(|state| state.shown.push((path.into(), options.to_owned()))); @@ -377,17 +360,6 @@ impl FakeKernel { .collect() } - /// Returns the paths and values of the writes so far, in order. - pub(crate) fn writes(&self) -> Vec<(String, String)> { - self.calls() - .into_iter() - .filter_map(|call| match call { - Call::Write(path, value) => Some((path, value)), - _ => None, - }) - .collect() - } - /// Returns the number of sleeps so far. pub(crate) fn sleeps(&self) -> usize { self.with(|state| state.sleeps) @@ -519,27 +491,6 @@ fn path_of(descriptor: &FileDescriptor) -> Result { .map_err(|error| sys::errno_of(&error)) } -/// Moves the ID `process` into the membership file `path`, from that of the parent of -/// `path`'s cgroup. -fn move_process(path: &Path, process: u32) -> Result<(), Errno> { - let source = path - .parent() - .and_then(Path::parent) - .map(|parent| parent.join("cgroup.procs")) - .ok_or(Errno::EINVAL)?; - let listed = std::fs::read_to_string(&source).map_err(|_| Errno::ENOENT)?; - let line = process.to_string(); - let remaining: String = listed - .lines() - .filter(|listed| *listed != line) - .map(|listed| format!("{listed}\n")) - .collect(); - std::fs::write(&source, remaining).map_err(|_| Errno::EIO)?; - let mut members = std::fs::read_to_string(path).unwrap_or_default(); - members.push_str(&format!("{process}\n")); - std::fs::write(path, members).map_err(|_| Errno::EIO) -} - impl Kernel for FakeKernel { fn mount( &mut self, @@ -612,33 +563,6 @@ impl Kernel for FakeKernel { Disk::read(file, LOGICAL_BLOCK_SIZE, size, expected).map_err(DeviceError::Table) } - fn write(&mut self, file: &CStr, value: &[u8]) -> Result<(), Errno> { - let call = Call::Write(text(file), String::from_utf8_lossy(value).into_owned()); - let file = path(file); - self.record(call, Operation::Write, &file)?; - let process = std::str::from_utf8(value) - .ok() - .and_then(|value| value.trim_end().parse::().ok()); - match process { - Some(process) if file.ends_with("cgroup.procs") => { - let pinned = self.with(|state| { - state - .unmovable - .iter_mut() - .find(|(pinned, writes)| *pinned == process && *writes > 0) - .map(|(_, writes)| *writes -= 1) - .is_some() - }); - if pinned { - Ok(()) - } else { - move_process(&file, process) - } - } - _ => std::fs::write(&file, value).map_err(|_| Errno::EIO), - } - } - fn is_mount_point(&mut self, path: &Path) -> std::io::Result { // The kernel's answer needs the directory, too. std::fs::metadata(path)?; diff --git a/nvx-rs/src/guest/root/tests.rs b/nvx-rs/src/guest/root/tests.rs index b9f30a342..ed7f87021 100644 --- a/nvx-rs/src/guest/root/tests.rs +++ b/nvx-rs/src/guest/root/tests.rs @@ -1,6 +1,5 @@ -//! Tests of the root operations with the fake kernel, and with the kernel's interfaces, a cgroup -//! hierarchy, and the runtime directory in a temporary directory, and of the kernel's own check -//! for a mount point. +//! Tests of the root operations with the fake kernel, and with the kernel's interfaces and the +//! runtime directory in a temporary directory, and of the kernel's own check for a mount point. use std::fs; use std::io; @@ -13,7 +12,6 @@ pub(crate) use super::kernel::tests::{Call, FakeKernel, Operation, attach_calls} use super::kernel::{Paths, super_options}; use super::*; use crate::gpt::ExpectedPartition; -use crate::guest::cgroup::Layout; use crate::guest::mappings::{Access, Mapping, SourceRoot}; use crate::guest::sys::MountAttributes; @@ -4040,295 +4038,3 @@ fn a_dropped_root_is_torn_down_once() { .unwrap(); assert_eq!(guest.kernel.unmounts().len(), 3); } - -/// A process of the bootstrap's parent cgroup. -enum Process { - /// A process, which the bootstrap moves. - User(u32), - /// A kernel thread, which stays. - Kernel(u32), - /// A process that ended after the cgroup listed it, which has no `/proc` entry. - Ended(u32), -} - -/// Prepares a cgroup hierarchy at `cgroup` in `guest`, whose root, already a cgroup if -/// `mounted`, holds `processes`, and returns the layout of its `agent` and `container` cgroups. -fn hierarchy(guest: &Guest, mounted: bool, processes: &[Process]) -> Layout { - let cgroup = guest.path("cgroup"); - write( - &guest.paths.proc.join("sys/user/max_user_namespaces"), - "15019\n", - ); - if mounted { - write(&cgroup.join("cgroup.controllers"), "cpu io memory pids\n"); - } - write(&cgroup.join("cgroup.subtree_control"), ""); - let mut members = String::new(); - for process in processes { - let (id, flags) = match *process { - Process::User(id) => (id, Some(0x0040_0100)), - Process::Kernel(id) => (id, Some(0x0020_8040)), - Process::Ended(id) => (id, None), - }; - members.push_str(&format!("{id}\n")); - if let Some(flags) = flags { - let stat = format!("{id} (p {id}) S 0 {id} {id} 0 -1 {flags} 0 0 0 0 0 0 0 0 20 0 1\n"); - write(&guest.paths.proc.join(id.to_string()).join("stat"), stat); - } - } - write(&cgroup.join("cgroup.procs"), members); - Layout::new(cgroup.join("agent"), cgroup.join("container")).unwrap() -} - -/// Bootstraps `layout` with `resources` and a clone of `guest`'s kernel. -fn bootstrap_in(guest: &Guest, layout: &Layout, resources: &Resources) -> Result { - bootstrap::bootstrap_with(&mut guest.kernel.clone(), &guest.paths, layout, resources) -} - -/// Creates the interface files `files` of the workload cgroup, `cgroup/container`, in `guest`, -/// as the kernel shows them once its parent enables their controllers, which a hierarchy of -/// regular files does not. -fn interface_files(guest: &Guest, files: &[&str]) { - for file in files { - write(&guest.path("cgroup/container").join(file), ""); - } -} - -#[test] -fn bootstrap_moves_every_process_but_kernel_threads_and_applies_the_resources() { - let guest = Guest::bare(); - let processes = [ - Process::User(1), - Process::Kernel(2), - Process::User(45), - Process::Ended(99), - ]; - let layout = hierarchy(&guest, true, &processes); - interface_files(&guest, &["pids.current", "memory.max", "pids.max"]); - let resources = Resources { - agent_memory_low: Some(16 << 20), - workload_memory_max: Some(512 << 20), - workload_pids_max: Some(65), - }; - let bootstrap = bootstrap_in(&guest, &layout, &resources).unwrap(); - assert_eq!(bootstrap.moved(), [1, 45]); - assert_eq!(bootstrap.layout(), &layout); - assert_eq!(bootstrap.workload().path(), layout.workload()); - let writes: Vec<(String, String)> = [ - ("proc/sys/user/max_user_namespaces", "0"), - ("cgroup/agent/cgroup.procs", "1"), - ("cgroup/agent/cgroup.procs", "45"), - ("cgroup/agent/memory.low", "16777216"), - ] - .into_iter() - .map(|(path, value)| (guest.text(path), value.to_owned())) - .collect(); - assert_eq!(guest.kernel.writes(), writes); - assert!(guest.kernel.mounts().is_empty()); - // The agent's `memory.low` enabled the `memory` controller. The workload cgroup has every - // file already, so its preparation enabled nothing, and wrote its values. - assert_eq!( - fs::read_to_string(guest.path("cgroup/cgroup.subtree_control")).unwrap(), - "+memory" - ); - for (file, value) in [("memory.max", "536870912"), ("pids.max", "65")] { - assert_eq!( - fs::read_to_string(guest.path("cgroup/container").join(file)).unwrap(), - value, - "{file}" - ); - } - assert_eq!( - fs::read_to_string(guest.path("cgroup/cgroup.procs")).unwrap(), - "2\n99\n" - ); - assert_eq!( - fs::read_to_string(guest.path("cgroup/agent/cgroup.procs")).unwrap(), - "1\n45\n" - ); -} - -#[test] -fn bootstrap_mounts_the_hierarchy_unless_the_parent_is_a_cgroup() { - let guest = Guest::bare(); - let layout = hierarchy(&guest, false, &[Process::User(1)]); - bootstrap_in(&guest, &layout, &Resources::default()).unwrap(); - assert_eq!( - guest.kernel.calls()[1], - mount_call( - Some("cgroup2"), - &guest.text("cgroup"), - Some("cgroup2"), - MountFlags::MS_NOSUID | MountFlags::MS_NODEV | MountFlags::MS_NOEXEC, - None, - ) - ); - // Without a value, the bootstrap writes only the placement, and the preparation of the new - // workload cgroup enables only the `pids` controller. - assert_eq!(guest.kernel.writes().len(), 2); - assert_eq!( - fs::read_to_string(guest.path("cgroup/cgroup.subtree_control")).unwrap(), - "+pids" - ); - assert!(guest.path("cgroup/container").is_dir()); -} - -#[test] -fn bootstrap_names_the_failed_step_and_keeps_the_earlier_changes() { - // The workload cgroup's `memory.max` cannot be written: a directory takes its place. - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1)]); - interface_files(&guest, &["pids.current", "pids.max"]); - fs::create_dir(guest.path("cgroup/container/memory.max")).unwrap(); - let resources = Resources { - agent_memory_low: Some(1 << 20), - workload_memory_max: Some(1), - workload_pids_max: Some(8), - }; - let error = bootstrap_in(&guest, &layout, &resources).unwrap_err(); - assert!( - matches!( - &error, - Error::Bootstrap { step: BootstrapStep::WorkloadCgroup, error } - if error.raw_os_error() == Some(Errno::EISDIR as i32) - ), - "{error:?}" - ); - assert_eq!( - error.to_string(), - format!( - "failed to prepare the workload cgroup: {}", - os_error(Errno::EISDIR) - ) - ); - assert_eq!( - fs::read_to_string(guest.path("cgroup/agent/memory.low")).unwrap(), - "1048576" - ); - - // The agent's value is refused before the workload cgroup is prepared. - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1)]); - guest.kernel.fail( - Operation::Write, - guest.path("cgroup/agent/memory.low"), - Errno::EINVAL, - ); - let error = bootstrap_in(&guest, &layout, &resources).unwrap_err(); - assert!( - matches!( - &error, - Error::Bootstrap { step: BootstrapStep::AgentMemoryLow, error } - if error.raw_os_error() == Some(Errno::EINVAL as i32) - ), - "{error:?}" - ); - assert_eq!( - error.to_string(), - format!( - "failed to set the agent's memory reservation: {}", - os_error(Errno::EINVAL) - ) - ); - assert_eq!( - fs::read_to_string(guest.path("cgroup/agent/cgroup.procs")).unwrap(), - "1\n" - ); - assert!(!guest.path("cgroup/container").exists()); - - // Cgroups that are not siblings are refused before anything changes. - let guest = Guest::bare(); - hierarchy(&guest, true, &[Process::User(1)]); - let cgroup = guest.path("cgroup"); - let layout = Layout::new(cgroup.join("agent"), cgroup.join("workloads/container")).unwrap(); - let error = bootstrap_in(&guest, &layout, &Resources::default()).unwrap_err(); - assert!( - matches!( - &error, - Error::Bootstrap { step: BootstrapStep::Layout, error } - if error.raw_os_error() == Some(Errno::EINVAL as i32) - ), - "{error:?}" - ); - assert!(guest.kernel.calls().is_empty()); -} - -#[test] -fn bootstrap_accepts_a_missing_sysctl_only_without_user_namespaces() { - // A kernel without sysctls lacks the limit, which one without user namespaces needs no more. - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1)]); - guest.kernel.fail( - Operation::Write, - guest.path("proc/sys/user/max_user_namespaces"), - Errno::ENOENT, - ); - bootstrap_in(&guest, &layout, &Resources::default()).unwrap(); - assert_eq!( - fs::read_to_string(guest.path("cgroup/agent/cgroup.procs")).unwrap(), - "1\n" - ); - - // One with user namespaces cannot be limited, so nothing else changes. - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1)]); - guest.kernel.fail( - Operation::Write, - guest.path("proc/sys/user/max_user_namespaces"), - Errno::ENOENT, - ); - write(&guest.paths.proc.join("self/ns/user"), ""); - let error = bootstrap_in(&guest, &layout, &Resources::default()).unwrap_err(); - assert!( - matches!( - &error, - Error::Bootstrap { step: BootstrapStep::UserNamespaces, error } - if error.raw_os_error() == Some(Errno::ENOENT as i32) - ), - "{error:?}" - ); - assert_eq!(guest.kernel.calls().len(), 1); - assert!(!guest.path("cgroup/agent").exists()); -} - -#[test] -fn bootstrap_fails_while_a_process_keeps_returning_to_the_parent() { - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1), Process::User(7)]); - guest.kernel.pin(7, usize::MAX); - let error = bootstrap_in(&guest, &layout, &Resources::default()).unwrap_err(); - assert!( - matches!( - &error, - Error::Bootstrap { step: BootstrapStep::Placement, error } - if error.raw_os_error() == Some(Errno::EBUSY as i32) - ), - "{error:?}" - ); - let attempts = guest - .kernel - .writes() - .into_iter() - .filter(|(path, value)| path.ends_with("agent/cgroup.procs") && value == "7") - .count(); - assert_eq!(attempts, 16); -} - -#[test] -fn bootstrap_succeeds_when_the_last_pass_empties_the_parent() { - let guest = Guest::bare(); - let layout = hierarchy(&guest, true, &[Process::User(1), Process::User(7)]); - // The sixteenth pass is the first that moves the process. - guest.kernel.pin(7, 15); - let bootstrap = bootstrap_in(&guest, &layout, &Resources::default()).unwrap(); - assert_eq!(bootstrap.moved()[0], 1); - assert_eq!(bootstrap.moved()[1..], [7; 16]); - assert_eq!( - fs::read_to_string(guest.path("cgroup/cgroup.procs")).unwrap(), - "" - ); - assert_eq!( - fs::read_to_string(guest.path("cgroup/agent/cgroup.procs")).unwrap(), - "1\n7\n" - ); -} diff --git a/nvx-rs/src/guest/spawn.rs b/nvx-rs/src/guest/spawn.rs deleted file mode 100644 index e55bd0cbf..000000000 --- a/nvx-rs/src/guest/spawn.rs +++ /dev/null @@ -1,622 +0,0 @@ -//! Starting a program in a new process, for any guest init or agent. -//! -//! A [`Command`] describes a program and how its process starts: its arguments and environment, a -//! `PATH` in which it looks for a program that it names without a `/`, if any, an optional root -//! directory and a working directory in it, the descriptors of its standard streams and of any -//! others that it inherits, the cgroup that it joins, whether it leads a new process group or -//! session, whether it waits at a [`Barrier`] until its caller releases it, whether it resets its -//! signals, and the execution that it [enters](Command::enter), if any. [`Command::spawn`] -//! allocates and opens everything that the child needs before it starts the child, with `fork`, -//! or for an entry with `clone3`; the child then makes only system calls over that storage, in -//! the order of [`LaunchStage`], and replaces itself with the program. A caller may therefore -//! spawn from any thread. -//! -//! A command owns the descriptors that it takes, and hands each one over once: the child places -//! the standard streams and keeps the inherited descriptors open at their numbers, and the -//! caller's process closes all of them as soon as it has started the child, so that the reader -//! of a workload's pipe sees end of file once the program and its descendants close their ends. -//! Before the child starts, a command moves any descriptor that the child uses after it places -//! its standard streams above their numbers and makes it close-on-exec, so the program receives -//! the command's descriptors only as its standard streams and inherited descriptors. Descriptors -//! that the caller opened without close-on-exec stay open in the program, as across any `exec`, -//! except in the program of a child that waits at a [`Barrier`] or enters an execution: such a -//! child closes every descriptor that it does not use before it waits or enters, so that it holds -//! no copy of another child's pipes while it waits, and the program of an entry none of its -//! caller's. -//! -//! The signals that the caller ignores, and its thread's signal mask, also pass to the program, -//! as across any `exec`, unless the command [resets its signals](Command::reset_signals), as an -//! entry's does: a caller that ignores `SIGPIPE`, as Rust programs do, or that blocks `SIGCHLD` -//! to collect its children through a signal descriptor, would otherwise start every program so. -//! -//! No step needs NVX's control protocol, its managed agent and helper, or its workload identity: -//! the caller chooses the program, and the identity, if any, that an entry gives it, or as which -//! the child checks its working directory. -//! -//! The caller owns the child. [`Spawned::id`] names it, and the caller collects its status, for -//! example through a [`process::Child`](super::process::Child): this module never waits for a -//! process. [`LaunchReport::wait`] tells whether the program started, or which stage failed and -//! why. -//! -//! ```no_run -//! use std::fs::File; -//! -//! use nvx::guest::spawn::Command; -//! -//! # fn main() -> std::io::Result<()> { -//! let spawned = Command::new(c"/bin/echo".to_owned(), vec![c"echo".to_owned(), c"hi".to_owned()]) -//! .working_directory(c"/tmp".to_owned()) -//! .stdin(File::open("/dev/null")?) -//! .new_process_group() -//! .spawn()?; -//! let process = spawned.id(); -//! if let Some(failure) = spawned.into_report().wait()? { -//! eprintln!("process {process} did not start: {failure}"); -//! } -//! // The caller reaps `process`. -//! # Ok(()) -//! # } -//! ``` - -use std::ffi::CString; -use std::fmt; -use std::io; -use std::os::fd::{BorrowedFd, OwnedFd}; -use std::path::Path; - -use super::sys::{ - self, ChildSpec, Errno, FileDescriptor, Mode, OpenFlags, PreparedChild, ProcessId, - REPORT_LENGTH, -}; - -/// A step that a child takes after it starts and before its program runs. -/// -/// The variants are in the order of the steps, and a child skips the steps that its [`Command`] -/// does not ask for. A child whose step fails reports it and exits with status 125 without -/// running its program. The steps from [`Propagation`](Self::Propagation) to -/// [`Devices`](Self::Devices), and from [`Bounding`](Self::Bounding) to -/// [`NoNewPrivileges`](Self::NoNewPrivileges), are an execution's [entry](super::entry), which -/// [`Command::enter`] asks for; the child of an entry also takes the [`Signals`](Self::Signals) -/// step, which [`Command::reset_signals`] asks for otherwise. -/// -/// Each stage has a number, its [`code`](Self::code), with which the child's launch report names -/// the stage that failed. -/// [`from_code`](Self::from_code) turns a number back into its stage. The numbers are part of the -/// API: a stage keeps its number, and a new stage takes a number of its own, wherever it comes in -/// the order. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum LaunchStage { - /// The child makes itself the leader of a new process group. - ProcessGroup, - /// The child makes itself the leader of a new session, and of its process group, without a - /// controlling terminal. - Session, - /// The child closes every descriptor that it does not use, then waits until its caller - /// releases its [`Barrier`]. - Barrier, - /// The child places its standard streams and keeps its inherited descriptors open. The child - /// of an entry first closes every descriptor that it does not use. - Descriptors, - /// The child joins its cgroup. - Cgroup, - /// The child makes every mount of its new mount namespace private. - Propagation, - /// The child sets the hostname of its new UTS namespace. - Hostname, - /// The child moves its root over `/`, and makes it its root and working directory. - MoveRoot, - /// The child clears its umask, then mounts its procfs, and restricts it for user ID 0. - Proc, - /// The child mounts its read-only sysfs, and restricts it for user ID 0. - Sys, - /// The child mounts its `/dev`, with its device nodes, devpts, and shm, then restores its - /// umask. - Devices, - /// The child makes the root directory its root. - Root, - /// The child drops every capability but its own from its bounding set. - Bounding, - /// The child takes its supplementary groups, group ID, and user ID. - Identity, - /// The child applies its capabilities, and empties its inheritable and ambient sets. - Capabilities, - /// The child sets `no_new_privs`. - NoNewPrivileges, - /// The child takes the identity of its working-directory check, or gives it back after. - Credentials, - /// The child enters its working directory. The child of an entry first closes every - /// descriptor but its program's and its launch report's, and resolves the directory through - /// no magic link. - WorkingDirectory, - /// The child records the physical path of its working directory in `PWD`. - PhysicalWorkingDirectory, - /// The child gives every signal its default disposition, then unblocks every signal. - Signals, - /// The child replaces itself with its program: the first of those that its - /// [search](Command::search_path) finds that it can run, if its command searches for it. - Exec, -} - -impl LaunchStage { - /// Every stage, in order. - const ALL: [Self; 21] = [ - Self::ProcessGroup, - Self::Session, - Self::Barrier, - Self::Descriptors, - Self::Cgroup, - Self::Propagation, - Self::Hostname, - Self::MoveRoot, - Self::Proc, - Self::Sys, - Self::Devices, - Self::Root, - Self::Bounding, - Self::Identity, - Self::Capabilities, - Self::NoNewPrivileges, - Self::Credentials, - Self::WorkingDirectory, - Self::PhysicalWorkingDirectory, - Self::Signals, - Self::Exec, - ]; - - /// Returns the stage's number: from 1, for [`ProcessGroup`](Self::ProcessGroup), to 21, for - /// [`Exec`](Self::Exec), in the order of the stages. A stage that NVX adds takes 22, then 23, - /// and so on, wherever it comes in the order, and no stage is numbered 0. - pub const fn code(self) -> u32 { - match self { - Self::ProcessGroup => 1, - Self::Session => 2, - Self::Barrier => 3, - Self::Descriptors => 4, - Self::Cgroup => 5, - Self::Propagation => 6, - Self::Hostname => 7, - Self::MoveRoot => 8, - Self::Proc => 9, - Self::Sys => 10, - Self::Devices => 11, - Self::Root => 12, - Self::Bounding => 13, - Self::Identity => 14, - Self::Capabilities => 15, - Self::NoNewPrivileges => 16, - Self::Credentials => 17, - Self::WorkingDirectory => 18, - Self::PhysicalWorkingDirectory => 19, - Self::Signals => 20, - Self::Exec => 21, - } - } - - /// Returns the stage that `code` numbers, or `None` if no stage has that number, such as 0. - pub fn from_code(code: u32) -> Option { - Self::ALL.into_iter().find(|stage| stage.code() == code) - } -} - -impl fmt::Display for LaunchStage { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter.write_str(match self { - Self::ProcessGroup => "process group", - Self::Session => "session", - Self::Barrier => "barrier", - Self::Descriptors => "descriptors", - Self::Cgroup => "cgroup", - Self::Propagation => "mount propagation", - Self::Hostname => "hostname", - Self::MoveRoot => "root move", - Self::Proc => "procfs", - Self::Sys => "sysfs", - Self::Devices => "devices", - Self::Root => "root directory", - Self::Bounding => "bounding set", - Self::Identity => "identity", - Self::Capabilities => "capabilities", - Self::NoNewPrivileges => "no new privileges", - Self::Credentials => "credentials", - Self::WorkingDirectory => "working directory", - Self::PhysicalWorkingDirectory => "physical working directory", - Self::Signals => "signals", - Self::Exec => "exec", - }) - } -} - -/// The launch stage at which a child stopped, and the `errno` of the failure. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct LaunchFailure { - /// The stage that failed. - stage: LaunchStage, - /// The `errno` with which it failed. - error: i32, -} - -impl LaunchFailure { - /// Returns the stage that failed. - pub fn stage(&self) -> LaunchStage { - self.stage - } - - /// Returns the `errno` with which the stage failed. - pub fn raw_os_error(&self) -> i32 { - self.error - } -} - -impl fmt::Display for LaunchFailure { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - write!( - formatter, - "the {} launch stage failed: {}", - self.stage, - io::Error::from_raw_os_error(self.error) - ) - } -} - -impl std::error::Error for LaunchFailure {} - -/// A cgroup v2 that a child joins before its program runs. -#[derive(Debug)] -pub struct Cgroup { - /// The cgroup's membership file, open for writing. - procs: FileDescriptor, -} - -impl Cgroup { - /// Opens the membership file, `cgroup.procs`, of the cgroup v2 directory `directory`. The - /// child writes its process ID to it. - pub fn open(directory: &Path) -> io::Result { - let path = sys::path_c_string(&directory.join("cgroup.procs"))?; - let procs = sys::open( - &path, - OpenFlags::O_WRONLY | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?; - Ok(Self { procs }) - } - - /// Opens the membership file of the cgroup v2 whose directory `directory` holds open, so - /// that a child joins that cgroup even once another cgroup takes its path. - pub(crate) fn open_in(directory: &FileDescriptor) -> io::Result { - let procs = sys::open_at( - directory, - c"cgroup.procs", - OpenFlags::O_WRONLY | OpenFlags::O_CLOEXEC, - Mode::empty(), - )?; - Ok(Self { procs }) - } -} - -/// A program, and how a new process starts it. -/// -/// By default the child keeps its parent's environment, root and working directories, standard -/// streams, process group, cgroup, ignored signals, and signal mask, inherits no other descriptor -/// of the command's, and runs the program at the path that the command names. Each method -/// changes one of them. -#[derive(Debug)] -pub struct Command { - /// The child's inputs. - spec: ChildSpec, -} - -impl Command { - /// Describes a process that runs the executable at `program`, which is not searched for in a - /// `PATH` unless the command [searches one](Self::search_path), with `arguments`, argument - /// zero first. A command without arguments fails to spawn. - pub fn new(program: CString, arguments: Vec) -> Self { - Self { - spec: ChildSpec::new(program, arguments), - } - } - - /// Gives the program exactly the `NAME=value` entries of `environment`, instead of the - /// parent's environment. - pub fn environment(mut self, environment: Vec) -> Self { - self.spec.set_environment(environment); - self - } - - /// Looks for a program that the command names without a `/` in the directories of `path`, a - /// `PATH` value, as `execvp` does, once the child has entered its root and working directory: - /// the child runs the first `DIRECTORY/NAME`, in the order of the directories, that it can - /// run, where an empty directory, such as the one between two adjacent colons, stands for the - /// working directory. A program whose name is empty or holds a `/` runs as the command names - /// it. The child searches `path`, not the `PATH` of the program's environment: a caller whose - /// program's own `PATH` should decide passes that one. - /// - /// The search takes the same memory however long the name and `path` are, as `execvp`'s does: - /// one buffer of `PATH_MAX` bytes, 4096, which the command prepares before the child starts, - /// and in which the child builds each path in turn, so that the child allocates nothing. A - /// path that does not fit in the buffer with its terminating NUL is one that the kernel - /// refuses with `ENAMETOOLONG`, and the child skips it as such without trying it. A name - /// longer than `NAME_MAX`, 255 bytes, fails the [stage](LaunchStage::Exec) with - /// `ENAMETOOLONG` before the child tries a path, as it does in `execvp`. - /// - /// The child skips a path that does not exist, which `ENOENT`, `ENOTDIR`, or `ENAMETOOLONG` - /// reports, and one that it may not run, `EACCES`, which it remembers. Any other failure, - /// such as `ENOEXEC` for a file that the kernel cannot run, ends the search: unlike some - /// `execvp`s, the child runs no shell for such a file. A search that runs nothing fails the - /// [stage](LaunchStage::Exec) with `EACCES` if the child skipped a path that it may not run, - /// and otherwise with the last path's error. - pub fn search_path(mut self, path: CString) -> Self { - self.spec.set_search_path(path); - self - } - - /// Makes the directory that `root` opens the child's root directory, and its working - /// directory unless [`working_directory`](Self::working_directory) names another one. Changing - /// the root needs the privilege to `chroot`. - pub fn root(mut self, root: impl Into) -> Self { - self.spec.set_root(FileDescriptor::from_owned(root.into())); - self - } - - /// Makes the absolute path `directory`, which the child resolves in its root, its working - /// directory. A relative path fails to spawn. - pub fn working_directory(mut self, directory: CString) -> Self { - self.spec.set_directory(directory); - self - } - - /// Makes the child enter its working directory with the effective user ID `uid`, the - /// effective group ID `gid`, and no supplementary groups, then take back its own, so that the - /// directory's permissions apply as they will to a program that runs as `uid` and `gid`. - /// - /// The check needs a [working directory](Self::working_directory), without which the command - /// fails to spawn, and the privilege to change the child's credentials. - pub fn check_working_directory_as(mut self, uid: u32, gid: u32) -> Self { - self.spec.set_credentials(uid, gid); - self - } - - /// Sets `PWD`, in the place of the environment's first `PWD` entry or after the last entry, - /// to the physical path of the child's working directory once the child has entered it. - /// - /// The path may hold 4095 bytes. The child records it without allocating, in an entry of - /// the [environment](Self::environment), without which the command fails to spawn. A longer - /// path fails the [stage](LaunchStage::PhysicalWorkingDirectory) with `ENAMETOOLONG`, and a - /// working directory outside the child's root, which has no path in it, with `ENOENT`. - pub fn export_physical_working_directory(mut self) -> Self { - self.spec.export_physical_pwd(); - self - } - - /// Makes `descriptor` the program's standard input. - pub fn stdin(mut self, descriptor: impl Into) -> Self { - self.spec - .set_stream(0, FileDescriptor::from_owned(descriptor.into())); - self - } - - /// Makes `descriptor` the program's standard output. - pub fn stdout(mut self, descriptor: impl Into) -> Self { - self.spec - .set_stream(1, FileDescriptor::from_owned(descriptor.into())); - self - } - - /// Makes `descriptor` the program's standard error. - pub fn stderr(mut self, descriptor: impl Into) -> Self { - self.spec - .set_stream(2, FileDescriptor::from_owned(descriptor.into())); - self - } - - /// Keeps `descriptor` open in the program at its current number, which the caller passes to - /// the program, for example as an argument. A descriptor numbered 0, 1, or 2, which are the - /// standard streams', fails to spawn. - pub fn inherit(mut self, descriptor: impl Into) -> Self { - self.spec - .inherit(FileDescriptor::from_owned(descriptor.into())); - self - } - - /// Makes the child join `cgroup` once it has placed its descriptors, before the rest of its - /// stages and its program, so that the program and its descendants start in it. - pub fn cgroup(mut self, cgroup: Cgroup) -> Self { - self.spec.set_cgroup(cgroup.procs); - self - } - - /// Makes the child the leader of a new process group, whose ID is its process ID. - pub fn new_process_group(mut self) -> Self { - self.spec.new_process_group(); - self - } - - /// Makes the child the leader of a new session, without a controlling terminal, and of its - /// process group, whose IDs are its process ID. The session replaces a - /// [new process group](Self::new_process_group). - pub fn new_session(mut self) -> Self { - self.spec.new_session(); - self - } - - /// Makes the child wait, before it places its descriptors, until its caller releases the - /// [`Barrier`] that [`Spawned::take_barrier`] returns, for example once it has placed the - /// child in its cgroup. A child whose barrier is dropped unreleased, with the [`Spawned`] that - /// holds it too, fails the [stage](LaunchStage::Barrier) with `ECANCELED` and never runs its - /// program. - /// - /// Before it waits, the child closes every descriptor that it does not use, those that the - /// caller opened without close-on-exec too, so that it holds no copy of the descriptors of a - /// child that the caller starts meanwhile, which would keep that child's pipes open. - pub fn barrier(mut self) -> Self { - self.spec.set_barrier(); - self - } - - /// Gives every signal its default disposition, and then blocks none, just before the program - /// runs, so that the program ignores and blocks no signal because its caller does, as `exec` - /// would keep both: a write to a pipe without a reader then ends the program by `SIGPIPE`, - /// which Rust programs ignore, and a `SIGCHLD` reaches it once a child of its own ends. The - /// child of an [entry](Self::enter) always does. - pub fn reset_signals(mut self) -> Self { - self.spec.reset_signals(); - self - } - - /// Makes the child enter the execution that `entry` prepares before its program runs, as the - /// [entry](super::entry) module describes: its namespaces, view, root, and identity. The - /// child then leads a [new session](Self::new_session), closes every descriptor but those that - /// its program receives, and enters its working directory, by default `/`, with that - /// identity, through no magic link, such as `/proc/self/cwd` or a `/proc/self/fd` entry, which - /// could name a directory outside the root. Its program starts with every signal's default - /// disposition and none blocked, as [`reset_signals`](Self::reset_signals) has it. - /// - /// The command must also place the child: it needs a [cgroup](Self::cgroup) or a - /// [barrier](Self::barrier), at which its caller places it. It may name neither a - /// [root](Self::root) nor a [check](Self::check_working_directory_as), which the entry's root - /// and identity replace. A command that breaks either rule fails to spawn. - pub fn enter(mut self, entry: super::entry::Prepared) -> Self { - self.spec.set_entry(entry.into_child()); - self - } - - /// Starts the child. - /// - /// The call returns once it has started the child, before the child's launch stages finish; - /// the child's [`LaunchReport`] tells how they end. It fails without a child if the command - /// is invalid, or if a resource that the child needs cannot be allocated or opened. On - /// x86-64, it also fails, with `PermissionDenied`, while the calling thread may still hold - /// the permission for OpenVMM's I/O ports that a time call could not revoke, as - /// `TimeError::PortRelease` reports, because the child would inherit it. - pub fn spawn(self) -> io::Result { - let started = PreparedChild::new(self.spec)?.spawn()?; - Ok(Spawned { - process: started.process, - report: LaunchReport { - reader: started.report, - }, - barrier: started.barrier.map(|writer| Barrier { writer }), - }) - } -} - -/// A child that [`Command::spawn`] started. -/// -/// Dropping it neither waits for the child nor stops it, but it drops the child's -/// [`Barrier`], if the caller has not taken it, which cancels the child's launch. -#[derive(Debug)] -pub struct Spawned { - /// The child's process ID. - process: ProcessId, - /// The report of the child's launch. - report: LaunchReport, - /// The barrier at which the child waits, until the caller takes it. - barrier: Option, -} - -impl Spawned { - /// Returns the child's process ID, which is also its process group's ID when its command - /// asked for a [new process group](Command::new_process_group) or a - /// [new session](Command::new_session). - pub fn id(&self) -> u32 { - self.process.raw().unsigned_abs() - } - - /// Returns the child's process ID for the crate's own callers. - pub(crate) fn process(&self) -> ProcessId { - self.process - } - - /// Takes the [`Barrier`] at which the child waits, if its command asked for - /// [one](Command::barrier) and it has not been taken. - pub fn take_barrier(&mut self) -> Option { - self.barrier.take() - } - - /// Returns the report of the child's launch. - pub fn into_report(self) -> LaunchReport { - self.report - } -} - -/// The barrier at which a child waits before it places its descriptors, until its caller -/// [releases](Self::release) it. -/// -/// Dropping it unreleased cancels the child's launch: the child reads end of file, fails the -/// [stage](LaunchStage::Barrier) with `ECANCELED`, and never runs its program. So a caller that -/// cannot place the child, or that cancels it or times out first, never releases it by mistake. -/// The drop cancels the launch at once, even while another child that started meanwhile, and has -/// not run its program yet, holds a copy of the barrier's descriptor, which closing it alone -/// would leave open. -#[derive(Debug)] -pub struct Barrier { - /// The caller's end of the barrier's connected socket pair, which the drop shuts down. - writer: FileDescriptor, -} - -impl Barrier { - /// Lets the child continue its launch. - /// - /// A child that has ended already fails the call with `BrokenPipe`, without a `SIGPIPE` for - /// the caller; its launch report tells why it ended. - pub fn release(self) -> io::Result<()> { - match sys::write_without_sigpipe(&self.writer, &[1]) { - Ok(1) => Ok(()), - Ok(_) => Err(io::ErrorKind::WriteZero.into()), - Err(error) => Err(error.into()), - } - } -} - -impl Drop for Barrier { - fn drop(&mut self) { - // The child of a released barrier reads the release before the end of file. A failure - // leaves only the close, which still ends the wait once no copy remains. - let _ = self.writer.shut_down_sending(); - } -} - -/// The outcome of a child's launch stages. -#[derive(Debug)] -pub struct LaunchReport { - /// The read end of the child's report pipe, whose write end only the child holds. - reader: FileDescriptor, -} - -impl LaunchReport { - /// Lends the report's pipe, which polls readable, or hung up, once the report has arrived: - /// the child wrote it, or replaced itself with its program or exited without one. - pub(crate) fn descriptor(&self) -> BorrowedFd<'_> { - self.reader.borrowed() - } - - /// Waits until the child's program runs, which returns `None`, or a launch stage fails, - /// which returns the failure. - /// - /// The wait ends when the child replaces itself with its program or exits. A report that is - /// cut short or names no stage is `EPROTO`. - pub fn wait(self) -> io::Result> { - let mut bytes = [0u8; REPORT_LENGTH]; - let mut length = 0; - while length < bytes.len() { - match self.reader.read(&mut bytes[length..]) { - Ok(0) => break, - Ok(count) => length += count, - Err(Errno::EINTR) => {} - Err(error) => return Err(error.into()), - } - } - if length == 0 { - return Ok(None); - } - let stage = u32::from_ne_bytes(bytes[..4].try_into().expect("four bytes")); - let error = i32::from_ne_bytes(bytes[4..].try_into().expect("four bytes")); - match LaunchStage::from_code(stage) { - Some(stage) if length == REPORT_LENGTH => Ok(Some(LaunchFailure { stage, error })), - _ => Err(Errno::EPROTO.into()), - } - } -} - -#[cfg(test)] -pub(crate) mod tests; diff --git a/nvx-rs/src/guest/spawn/tests.rs b/nvx-rs/src/guest/spawn/tests.rs deleted file mode 100644 index a5cd650d6..000000000 --- a/nvx-rs/src/guest/spawn/tests.rs +++ /dev/null @@ -1,765 +0,0 @@ -use std::ffi::CStr; -use std::io::{Read, Write}; -use std::os::unix::ffi::OsStrExt; -use std::os::unix::fs::PermissionsExt; - -use super::*; -use crate::guest::sys::ChildStatus; - -/// Returns a command that runs `/bin/sh -c script`, with `$0` set to `sh`, for the tests of the -/// libraries that start programs. -pub(crate) fn shell(script: &str) -> Command { - Command::new( - c"/bin/sh".to_owned(), - vec![ - c"sh".to_owned(), - c"-c".to_owned(), - CString::new(script).unwrap(), - ], - ) -} - -impl Command { - /// Prepares the child without starting it, for focused launch tests. - pub(crate) fn prepare(self) -> PreparedChild { - PreparedChild::new(self.spec).expect("the command prepares") - } - - /// Replaces the system calls of the child's credential stage, which then succeed, or fail - /// with `failure`, without privileges. - pub(crate) fn simulate_credentials(mut self, failure: Option) -> Self { - self.spec.simulate_credentials(failure); - self - } -} - -/// Returns `path` as a C string. -fn c_path(path: &Path) -> CString { - CString::new(path.as_os_str().as_bytes()).unwrap() -} - -/// Returns the bytes that `reader` yields until end of file. -fn read_all(mut reader: std::io::PipeReader) -> Vec { - let mut bytes = Vec::new(); - reader.read_to_end(&mut bytes).unwrap(); - bytes -} - -/// Spawns `command` with piped standard output and error, and returns its launch failure, -/// status, output, and error output. -fn run(command: Command) -> (Option, Option, Vec, Vec) { - let (stdout, stdout_writer) = std::io::pipe().unwrap(); - let (stderr, stderr_writer) = std::io::pipe().unwrap(); - let spawned = command - .stdout(stdout_writer) - .stderr(stderr_writer) - .spawn() - .unwrap(); - let process = spawned.process(); - let failure = spawned.into_report().wait().unwrap(); - // The command took the writers, so the readers end when the child and its descendants - // close theirs. - let output = read_all(stdout); - let error = read_all(stderr); - (failure, sys::wait(process), output, error) -} - -#[test] -fn a_program_runs_with_its_arguments_environment_and_streams() { - let (stdin, mut stdin_writer) = std::io::pipe().unwrap(); - stdin_writer.write_all(b"input\n").unwrap(); - drop(stdin_writer); - let command = Command::new( - c"/bin/sh".to_owned(), - vec![ - c"zero".to_owned(), - c"-c".to_owned(), - c"read -r line; printf '%s|%s|%s|%s' \"$0\" \"$1\" \"$VALUE\" \"$line\"; printf err >&2" - .to_owned(), - c"argument zero".to_owned(), - c"one".to_owned(), - ], - ) - .environment(vec![c"VALUE=exact".to_owned()]) - .stdin(stdin); - let (failure, status, output, error) = run(command); - assert_eq!(failure, None); - assert_eq!(status, Some(ChildStatus::Exited(0))); - assert_eq!(output, b"argument zero|one|exact|input"); - assert_eq!(error, b"err"); -} - -#[test] -fn a_program_without_an_environment_inherits_its_parent_s() { - let path = std::env::var_os("PATH").unwrap_or_default(); - let (failure, status, output, _) = run(shell("printf %s \"$PATH\"")); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(output, path.as_bytes()); -} - -#[test] -fn inherited_descriptors_stay_open_at_their_numbers_and_others_close() { - let mut file = tempfile::tempfile().unwrap(); - file.write_all(b"handed over").unwrap(); - std::io::Seek::rewind(&mut file).unwrap(); - let inherited = std::os::fd::AsRawFd::as_raw_fd(&file); - let unrelated = tempfile::tempfile().unwrap(); - let other = std::os::fd::AsRawFd::as_raw_fd(&unrelated); - // `cat` reads the inherited descriptor through its own descriptor table, because `sh` - // accepts only single-digit redirections. - let (failure, status, output, _) = run(shell(&format!( - "cat /proc/self/fd/{inherited}; test ! -e /proc/self/fd/{other} && printf ' closed'" - )) - .inherit(file)); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(output, b"handed over closed"); - drop(unrelated); -} - -#[test] -fn the_working_directory_and_physical_pwd_follow_the_command() { - let directory = tempfile::tempdir().unwrap(); - let physical = directory.path().join("physical"); - std::fs::create_dir(&physical).unwrap(); - std::os::unix::fs::symlink(&physical, directory.path().join("link")).unwrap(); - let requested = directory.path().join("link/."); - let (failure, status, output, _) = run(shell("printf '%s|' \"$PWD\"; pwd -P") - .environment(vec![c"PWD=/stale".to_owned(), c"KEPT=1".to_owned()]) - .working_directory(c_path(&requested)) - .export_physical_working_directory()); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - let physical = std::fs::canonicalize(&physical).unwrap(); - let expected = [ - physical.as_os_str().as_bytes(), - b"|", - physical.as_os_str().as_bytes(), - b"\n", - ] - .concat(); - assert_eq!(output, expected); -} - -#[test] -fn a_program_named_without_a_slash_is_searched_for_in_the_command_s_path() { - // No program here was ever open for writing, which a child that another thread forks - // meanwhile could still hold, and fail the `exec` with `ETXTBSY`: the one that runs is a - // link to the shell, and the empty one is made by `mknod`. - let directory = tempfile::tempdir().unwrap(); - let path = |name: &str| directory.path().join(name); - for name in ["found", "denied", "empty", "other"] { - std::fs::create_dir(path(name)).unwrap(); - } - // `found/prog` runs; no one may run `denied/prog`; the kernel cannot run the empty - // `empty/prog`; and `file` is no directory. - std::os::unix::fs::symlink("/bin/sh", path("found/prog")).unwrap(); - std::fs::write(path("denied/prog"), b"").unwrap(); - nix::sys::stat::mknod( - &path("empty/prog"), - nix::sys::stat::SFlag::S_IFREG, - Mode::from_bits_truncate(0o755), - 0, - ) - .unwrap(); - std::fs::set_permissions(path("empty/prog"), std::fs::Permissions::from_mode(0o755)).unwrap(); - std::fs::write(path("file"), b"").unwrap(); - let named = |program: &CStr| { - Command::new( - program.to_owned(), - vec![ - c"prog".to_owned(), - c"-c".to_owned(), - c"printf ran".to_owned(), - ], - ) - }; - // An empty name stands for the empty directory, the working directory. - let searched = |directories: &[&str]| { - let directories: Vec = directories - .iter() - .map(|&name| match name { - "" => String::new(), - name => path(name).display().to_string(), - }) - .collect(); - named(c"prog").search_path(CString::new(directories.join(":")).unwrap()) - }; - for command in [ - searched(&["missing", "file", "denied", "found", "empty"]), - searched(&["missing", ""]).working_directory(c_path(&path("found"))), - // A name with a slash is the program's path, from the working directory. - named(c"found/prog") - .search_path(c_path(&path("other"))) - .working_directory(c_path(directory.path())), - ] { - let (failure, status, output, _) = run(command); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(output, b"ran"); - } - // A search that runs nothing fails with `EACCES` once it has met a program that it may - // not run, and otherwise with its last path's error. A program that the kernel cannot run - // ends it. - for (command, error) in [ - (searched(&["missing", "denied", "file"]), Errno::EACCES), - (searched(&["missing", "file"]), Errno::ENOTDIR), - (searched(&["file", "missing"]), Errno::ENOENT), - (searched(&["empty", "found"]), Errno::ENOEXEC), - ( - named(c"found/prog") - .search_path(c_path(directory.path())) - .working_directory(c_path(&path("other"))), - Errno::ENOENT, - ), - ] { - let (failure, status, output, _) = run(command); - let failure = failure.expect("the search fails"); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Exec, error as i32) - ); - assert_eq!(status, Some(ChildStatus::Exited(125))); - assert!(output.is_empty()); - } -} - -/// Returns a command that looks for the program `name` in `directories`, and runs it as -/// `sh -c 'printf ran'`. -fn searched_for(name: &str, directories: &[&str]) -> Command { - Command::new( - CString::new(name).unwrap(), - vec![c"sh".to_owned(), c"-c".to_owned(), c"printf ran".to_owned()], - ) - .search_path(CString::new(directories.join(":")).unwrap()) -} - -/// Runs `command`, whose search fails, and returns the search's error once the child has -/// exited with the launch failure status without output. -fn search_error(command: Command) -> i32 { - let (failure, status, output, _) = run(command); - assert_eq!(status, Some(ChildStatus::Exited(125))); - assert!(output.is_empty()); - let failure = failure.expect("the search fails"); - assert_eq!(failure.stage(), LaunchStage::Exec); - failure.raw_os_error() -} - -#[test] -fn a_search_skips_a_path_longer_than_path_max() { - let directory = tempfile::tempdir().unwrap(); - std::os::unix::fs::symlink("/bin/sh", directory.path().join("prog")).unwrap(); - let found = directory.path().display().to_string(); - let missing = directory.path().join("missing").display().to_string(); - // The path from this directory to any name is longer than `PATH_MAX`. - let long = format!("/{}", "d/".repeat(2048)); - let (failure, status, output, _) = run(searched_for("prog", &[&long, &found])); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(output, b"ran"); - // The search skips that directory as if the kernel had refused its path with - // `ENAMETOOLONG`, which a later path's error replaces. - assert_eq!( - search_error(searched_for("prog", &[&missing, &long])), - Errno::ENAMETOOLONG as i32 - ); - assert_eq!( - search_error(searched_for("prog", &[&long, &missing])), - Errno::ENOENT as i32 - ); -} - -#[test] -fn a_search_refuses_a_name_longer_than_name_max() { - let directory = tempfile::tempdir().unwrap(); - let found = directory.path().display().to_string(); - let missing = directory.path().join("missing").display().to_string(); - // The longest name that a search looks for, 255 bytes, is found. - let longest = "n".repeat(255); - std::os::unix::fs::symlink("/bin/sh", directory.path().join(&longest)).unwrap(); - let (failure, status, output, _) = run(searched_for(&longest, &[&missing, &found])); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(output, b"ran"); - // A longer name fails before any path, even where the last path's error would be - // `ENOENT`. - assert_eq!( - search_error(searched_for(&"n".repeat(256), &[&found, &missing])), - Errno::ENAMETOOLONG as i32 - ); -} - -/// Returns the peak resident memory of the calling process, in KiB: its `VmHWM`. -fn peak_resident_kib() -> u64 { - std::fs::read_to_string("/proc/self/status") - .unwrap() - .lines() - .find_map(|line| line.strip_prefix("VmHWM:")) - .and_then(|value| value.trim().strip_suffix("kB")) - .and_then(|value| value.trim().parse().ok()) - .expect("the status holds the peak resident memory") -} - -#[test] -fn a_search_s_memory_is_bounded_whatever_its_name_and_path() { - if crate::guest::sys::tests::isolate_process_test( - "NVX_SPAWN_BOUNDED_SEARCH_TEST_CHILD", - "guest::spawn::tests::a_search_s_memory_is_bounded_whatever_its_name_and_path", - ) { - return; - } - // The peak resident memory of the isolated process tells how much a launch prepared, - // once a first launch has brought in the code that every launch runs, which it counts - // too. - let directory = tempfile::tempdir().unwrap(); - let missing = directory.path().join("missing").display().to_string(); - assert_eq!( - search_error(searched_for("prog", &[&missing])), - Errno::ENOENT as i32 - ); - // A 4,096-byte name in each of the 32,764 empty directories of 32,763 colons would make - // 32,764 paths of 4,097 bytes: 128 MiB. - let name = CString::new("n".repeat(4096)).unwrap(); - let command = Command::new(name.clone(), vec![name]) - .search_path(CString::new(":".repeat(32_763)).unwrap()); - let before = peak_resident_kib(); - let error = search_error(command); - let grown = peak_resident_kib() - before; - assert_eq!(error, Errno::ENAMETOOLONG as i32); - assert!( - grown < 16 * 1024, - "the launch grew the peak resident memory by {grown} KiB" - ); -} - -#[test] -fn launch_stage_failures_are_reported_and_run_nothing() { - let directory = tempfile::tempdir().unwrap(); - let file = directory.path().join("file"); - std::fs::write(&file, b"").unwrap(); - let marker = directory.path().join("ran"); - let touch = |command: Command| run(command).0; - let script = format!("touch '{}'", marker.display()); - for (path, stage, error) in [ - ( - directory.path().join("missing"), - LaunchStage::WorkingDirectory, - Errno::ENOENT, - ), - (file.clone(), LaunchStage::WorkingDirectory, Errno::ENOTDIR), - ] { - let failure = touch(shell(&script).working_directory(c_path(&path))).unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (stage, error as i32), - "{path:?}" - ); - } - let failure = touch( - shell(&script) - .working_directory(c_path(directory.path())) - .check_working_directory_as(1000, 1000) - .simulate_credentials(Some(Errno::EPERM)), - ) - .unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Credentials, Errno::EPERM as i32) - ); - let failure = touch(Command::new( - c_path(&directory.path().join("missing")), - vec![c"missing".to_owned()], - )) - .unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Exec, Errno::ENOENT as i32) - ); - // A cgroup whose membership file is full refuses the child's process ID. - let cgroup = directory.path().join("cgroup"); - std::fs::create_dir(&cgroup).unwrap(); - std::os::unix::fs::symlink("/dev/full", cgroup.join("cgroup.procs")).unwrap(); - let failure = touch(shell(&script).cgroup(Cgroup::open(&cgroup).unwrap())).unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Cgroup, Errno::ENOSPC as i32) - ); - assert!(!marker.exists()); - assert_eq!( - failure.to_string(), - format!( - "the cgroup launch stage failed: {}", - io::Error::from_raw_os_error(Errno::ENOSPC as i32) - ) - ); -} - -#[test] -fn a_child_joins_its_cgroup_and_enters_its_checked_directory() { - let directory = tempfile::tempdir().unwrap(); - let cgroup = directory.path().join("cgroup"); - std::fs::create_dir(&cgroup).unwrap(); - std::fs::write(cgroup.join("cgroup.procs"), b"").unwrap(); - let (stdout, stdout_writer) = std::io::pipe().unwrap(); - let spawned = shell("pwd -P") - .working_directory(c_path(directory.path())) - .check_working_directory_as(1000, 1000) - .simulate_credentials(None) - .cgroup(Cgroup::open(&cgroup).unwrap()) - .stdout(stdout_writer) - .spawn() - .unwrap(); - let (id, process) = (spawned.id(), spawned.process()); - let failure = spawned.into_report().wait().unwrap(); - let output = read_all(stdout); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0))); - assert_eq!(failure, None); - let physical = std::fs::canonicalize(directory.path()).unwrap(); - assert_eq!(output, [physical.as_os_str().as_bytes(), b"\n"].concat()); - assert_eq!( - std::fs::read(cgroup.join("cgroup.procs")).unwrap(), - format!("{id}\n").into_bytes() - ); - assert!(Cgroup::open(&directory.path().join("missing")).is_err()); -} - -#[test] -fn invalid_commands_fail_to_spawn() { - for command in [ - Command::new(c"/bin/true".to_owned(), Vec::new()), - shell("exit 0").working_directory(c"relative".to_owned()), - shell("exit 0").check_working_directory_as(1000, 1000), - shell("exit 0").export_physical_working_directory(), - ] { - let error = command.spawn().unwrap_err(); - assert_eq!(error.raw_os_error(), Some(Errno::EINVAL as i32)); - } -} - -#[test] -fn short_and_unknown_reports_are_protocol_errors() { - for (bytes, expected) in [ - (Vec::new(), Ok(None)), - ( - [18u32.to_ne_bytes(), 2i32.to_ne_bytes()].concat(), - Ok(Some(LaunchFailure { - stage: LaunchStage::WorkingDirectory, - error: 2, - })), - ), - (vec![1, 0, 0], Err(Errno::EPROTO as i32)), - ( - [22u32.to_ne_bytes(), 2i32.to_ne_bytes()].concat(), - Err(Errno::EPROTO as i32), - ), - ( - [0u32.to_ne_bytes(), 2i32.to_ne_bytes()].concat(), - Err(Errno::EPROTO as i32), - ), - ] { - let (reader, writer) = sys::create_pipe().unwrap(); - writer.write_all(&bytes).unwrap(); - drop(writer); - let report = LaunchReport { reader }.wait(); - assert_eq!( - report.map_err(|error| error.raw_os_error().unwrap()), - expected, - "{bytes:?}" - ); - } - for stage in LaunchStage::ALL { - assert_eq!(LaunchStage::from_code(stage.code()), Some(stage)); - } -} - -#[test] -fn launch_stages_keep_their_numbers() { - // Callers tell a failed stage by its number, so no stage changes its number: a new stage - // takes 22, then 23, and so on, wherever it comes in the order. - let numbers = [ - (LaunchStage::ProcessGroup, 1), - (LaunchStage::Session, 2), - (LaunchStage::Barrier, 3), - (LaunchStage::Descriptors, 4), - (LaunchStage::Cgroup, 5), - (LaunchStage::Propagation, 6), - (LaunchStage::Hostname, 7), - (LaunchStage::MoveRoot, 8), - (LaunchStage::Proc, 9), - (LaunchStage::Sys, 10), - (LaunchStage::Devices, 11), - (LaunchStage::Root, 12), - (LaunchStage::Bounding, 13), - (LaunchStage::Identity, 14), - (LaunchStage::Capabilities, 15), - (LaunchStage::NoNewPrivileges, 16), - (LaunchStage::Credentials, 17), - (LaunchStage::WorkingDirectory, 18), - (LaunchStage::PhysicalWorkingDirectory, 19), - (LaunchStage::Signals, 20), - (LaunchStage::Exec, 21), - ]; - assert_eq!(numbers.map(|(stage, _)| stage), LaunchStage::ALL); - for (stage, number) in numbers { - assert_eq!(stage.code(), number, "{stage:?}"); - assert_eq!(LaunchStage::from_code(number), Some(stage), "{number}"); - } - for number in [0, 22, u32::MAX] { - assert_eq!(LaunchStage::from_code(number), None, "{number}"); - } -} - -#[test] -fn launch_stages_are_in_their_order() { - // The entry's view comes after the cgroup, which a child joins before it drops its - // privileges, and its identity before the working directory, which it enters as that - // identity. The signals take their defaults last, just before the program. - let position = |stage: LaunchStage| { - LaunchStage::ALL - .iter() - .position(|other| *other == stage) - .unwrap() - }; - assert!(position(LaunchStage::Cgroup) < position(LaunchStage::Identity)); - assert!(position(LaunchStage::MoveRoot) < position(LaunchStage::Proc)); - assert!(position(LaunchStage::NoNewPrivileges) < position(LaunchStage::WorkingDirectory)); - assert!(position(LaunchStage::Barrier) < position(LaunchStage::Descriptors)); - assert!(position(LaunchStage::PhysicalWorkingDirectory) < position(LaunchStage::Signals)); - assert!(position(LaunchStage::Signals) < position(LaunchStage::Exec)); -} - -/// Returns whether the launch report of `spawned` has arrived, without waiting. -fn report_arrived(spawned: &Spawned) -> bool { - sys::readable_or_hung_up(&[spawned.report.descriptor()], 0).unwrap()[0] -} - -#[test] -fn a_child_waits_at_its_barrier_until_released() { - let directory = tempfile::tempdir().unwrap(); - let marker = directory.path().join("ran"); - let (stdout, stdout_writer) = std::io::pipe().unwrap(); - let mut spawned = shell(&format!("touch '{}'; printf released", marker.display())) - .barrier() - .stdout(stdout_writer) - .spawn() - .unwrap(); - let process = spawned.process(); - let barrier = spawned.take_barrier().unwrap(); - assert!(spawned.take_barrier().is_none()); - std::thread::sleep(std::time::Duration::from_millis(50)); - assert!(!report_arrived(&spawned)); - assert!(!marker.exists()); - barrier.release().unwrap(); - assert_eq!(spawned.into_report().wait().unwrap(), None); - assert_eq!(read_all(stdout), b"released"); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0))); - assert!(marker.exists()); -} - -#[test] -fn a_dropped_barrier_cancels_the_launch() { - let directory = tempfile::tempdir().unwrap(); - let marker = directory.path().join("ran"); - let script = format!("touch '{}'", marker.display()); - // Dropping the barrier, or the spawned child that still holds it, cancels the launch. - let mut spawned = shell(&script).barrier().spawn().unwrap(); - drop(spawned.take_barrier()); - let cancelled = spawned.process(); - let first = spawned.into_report().wait().unwrap(); - let spawned = shell(&script).barrier().spawn().unwrap(); - let held = spawned.process(); - let second = spawned.into_report().wait().unwrap(); - for (failure, process) in [(first, cancelled), (second, held)] { - let failure = failure.unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Barrier, Errno::ECANCELED as i32) - ); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(125))); - } - assert!(!marker.exists()); -} - -#[test] -fn a_dropped_barrier_cancels_its_launch_while_another_child_waits_at_its_own() { - let directory = tempfile::tempdir().unwrap(); - let marker = |name: &str| directory.path().join(name); - let command = |name: &str| shell(&format!("touch '{}'", marker(name).display())).barrier(); - let mut first = command("first").spawn().unwrap(); - let first_barrier = first.take_barrier().unwrap(); - // The second child starts while the first waits, with a copy of the first's barrier, - // which it closes before it waits at its own; the drop cancels the first's launch - // whichever copies remain. - let mut second = command("second").spawn().unwrap(); - let second_barrier = second.take_barrier().unwrap(); - let (first_process, second_process) = (first.process(), second.process()); - drop(first_barrier); - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); - while !report_arrived(&first) { - assert!( - std::time::Instant::now() < deadline, - "the first child still waits at its dropped barrier" - ); - std::thread::sleep(std::time::Duration::from_millis(10)); - } - let failure = first.into_report().wait().unwrap().unwrap(); - assert_eq!( - (failure.stage(), failure.raw_os_error()), - (LaunchStage::Barrier, Errno::ECANCELED as i32) - ); - assert_eq!(sys::wait(first_process), Some(ChildStatus::Exited(125))); - assert!(!report_arrived(&second)); - second_barrier.release().unwrap(); - assert_eq!(second.into_report().wait().unwrap(), None); - assert_eq!(sys::wait(second_process), Some(ChildStatus::Exited(0))); - assert!(!marker("first").exists()); - assert!(marker("second").exists()); -} - -#[test] -fn another_childs_pipes_hang_up_while_a_child_waits_at_its_barrier() { - // The first child's launch report and output pipe exist before the waiting child - // starts, which therefore starts with copies of their writers, and closes them before it - // waits: the first child's pipes hang up once it ends, while the other still waits. - let (stdout, stdout_writer) = std::io::pipe().unwrap(); - let first = shell("printf done").stdout(stdout_writer).prepare(); - let mut waiting = shell("exit 0").barrier().spawn().unwrap(); - let barrier = waiting.take_barrier().unwrap(); - let first = first.spawn().unwrap(); - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); - while !(sys::hung_up(first.report.borrowed()).unwrap() - && sys::hung_up(std::os::fd::AsFd::as_fd(&stdout)).unwrap()) - { - assert!( - std::time::Instant::now() < deadline, - "the waiting child holds the first child's pipes" - ); - std::thread::sleep(std::time::Duration::from_millis(10)); - } - assert!(!report_arrived(&waiting)); - let report = LaunchReport { - reader: first.report, - }; - assert_eq!(report.wait().unwrap(), None); - assert_eq!(read_all(stdout), b"done"); - assert_eq!(sys::wait(first.process), Some(ChildStatus::Exited(0))); - let process = waiting.process(); - barrier.release().unwrap(); - assert_eq!(waiting.into_report().wait().unwrap(), None); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0))); -} - -#[test] -fn releasing_the_barrier_of_an_ended_child_fails() { - // Another test's child holds a copy of every descriptor of this process from its fork to - // its `exec`, which would read the release, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_SPAWN_ENDED_BARRIER_TEST_CHILD", - "guest::spawn::tests::releasing_the_barrier_of_an_ended_child_fails", - ) { - return; - } - let mut spawned = shell("exit 0").barrier().spawn().unwrap(); - let process = spawned.process(); - let barrier = spawned.take_barrier().unwrap(); - sys::kill_process(process).unwrap(); - assert_eq!( - sys::wait(process), - Some(ChildStatus::Signaled(sys::Signal::SIGKILL as i32)) - ); - let error = barrier.release().unwrap_err(); - assert_eq!(error.kind(), io::ErrorKind::BrokenPipe); -} - -#[test] -fn a_new_session_has_no_controlling_terminal() { - for command in [ - shell(STAT_SCRIPT).new_session(), - // A session replaces a new process group, which the parent then leaves alone. - shell(STAT_SCRIPT).new_process_group().new_session(), - ] { - let (failure, status, output, _) = run(command); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - let output = String::from_utf8(output).unwrap(); - let ids: Vec<&str> = output.split(' ').collect(); - assert_eq!(ids[1], ids[0], "{output}"); - assert_eq!(ids[2], ids[0], "{output}"); - assert_eq!(ids[3], "0", "{output}"); - } -} - -/// Returns a command that prints its program's status file. -fn print_status() -> Command { - Command::new( - c"/bin/cat".to_owned(), - vec![c"cat".to_owned(), c"/proc/self/status".to_owned()], - ) -} - -/// Returns the signals that the process whose status file is `status` ignores, and those -/// that it blocks, from its `SigIgn` and `SigBlk` lines, in which bit `n - 1` stands for -/// signal `n`. -fn ignored_and_blocked(status: &[u8]) -> (u64, u64) { - let status = std::str::from_utf8(status).unwrap(); - let set = |name: &str| { - let digits = status - .lines() - .find_map(|line| line.strip_prefix(name)?.strip_prefix(':')) - .unwrap_or_else(|| panic!("{name} is missing from {status}")); - u64::from_str_radix(digits.trim(), 16).unwrap() - }; - (set("SigIgn"), set("SigBlk")) -} - -#[test] -fn a_program_keeps_its_caller_s_ignored_and_blocked_signals_unless_its_command_resets_them() { - // Ignoring a signal changes the whole process, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_SPAWN_SIGNALS_TEST_CHILD", - "guest::spawn::tests::a_program_keeps_its_caller_s_ignored_and_blocked_signals_unless_its_command_resets_them", - ) { - return; - } - // The caller ignores `SIGHUP` and `SIGPIPE`, and its thread blocks `SIGCHLD`, as one - // that collects its children through a signal descriptor does. - let bit = |signal: sys::Signal| 1u64 << (signal as i32 - 1); - sys::ignore_signals(&[sys::Signal::SIGHUP, sys::Signal::SIGPIPE]); - nix::sys::signal::SigSet::from(sys::Signal::SIGCHLD) - .thread_block() - .unwrap(); - let caller = ignored_and_blocked(&std::fs::read("/proc/thread-self/status").unwrap()); - let ignored = bit(sys::Signal::SIGHUP) | bit(sys::Signal::SIGPIPE); - assert_eq!(caller.0 & ignored, ignored); - assert_ne!(caller.1 & bit(sys::Signal::SIGCHLD), 0); - - let (failure, status, output, _) = run(print_status()); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(ignored_and_blocked(&output), caller); - let (failure, status, output, _) = run(print_status().reset_signals()); - assert_eq!((failure, status), (None, Some(ChildStatus::Exited(0)))); - assert_eq!(ignored_and_blocked(&output), (0, 0)); -} - -#[test] -fn a_child_whose_report_has_no_reader_still_exits_with_the_launch_failure_status() { - // Another test's child holds a copy of every descriptor of this process from its fork to - // its `exec`, the report's reader too, so the test runs in a process of its own. - if crate::guest::sys::tests::isolate_process_test( - "NVX_SPAWN_UNREAD_REPORT_TEST_CHILD", - "guest::spawn::tests::a_child_whose_report_has_no_reader_still_exits_with_the_launch_failure_status", - ) { - return; - } - // The child writes its report with `SIGPIPE` blocked, since the default disposition that - // the reset gives it would end the child at a write that no reader takes. - let mut spawned = Command::new(c"/nonexistent".to_owned(), vec![c"nonexistent".to_owned()]) - .reset_signals() - .barrier() - .spawn() - .unwrap(); - let process = spawned.process(); - let barrier = spawned.take_barrier().unwrap(); - drop(spawned.into_report()); - barrier.release().unwrap(); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(125))); -} - -/// A script that prints its process ID, process group ID, session ID, and controlling -/// terminal from its status file. -const STAT_SCRIPT: &str = "read -r pid comm state ppid pgrp session tty rest , - /// The device nodes of the child's `/dev`, as bits of [`DEVICE_NODES`]. - pub(crate) devices: u8, - /// The user ID that the child takes. - pub(crate) uid: u32, - /// The group ID that the child takes. - pub(crate) gid: u32, - /// The supplementary groups that the child takes. - pub(crate) groups: Box<[libc::gid_t]>, - /// The capabilities that the child keeps, bit `n` for capability `n`. - pub(crate) capabilities: u64, - /// Test-only simulation of the entry stages, without privileges. - #[cfg(test)] - pub(crate) simulated: Option, -} - -impl EntryChild { - /// Returns the `clone3` flags of the child, which tests that simulate the stages, without - /// privileges, leave empty. - pub(super) fn clone_flags(&self) -> u64 { - #[cfg(test)] - if self.simulated.is_some() { - return 0; - } - NAMESPACES - } - - /// Makes the child's mounts private, sets its hostname, moves its root over `/`, and mounts - /// its view: procfs, sysfs, and `/dev` with its devpts and shm, in the order of their stages. - /// It creates its directories and nodes with exact modes, whatever the process's umask, which - /// it clears before it mounts the view and restores before it returns: a failure to clear it - /// is the [`Proc`](LaunchStage::Proc) stage's, and one to restore it the - /// [`Devices`](LaunchStage::Devices) stage's. - /// - /// Each failure names its stage. Only the post-clone child of an entry may call this. - pub(super) fn enter_view(&self) -> Result<(), (LaunchStage, Errno)> { - self.stage(LaunchStage::Propagation, make_mounts_private)?; - self.stage(LaunchStage::Hostname, || set_hostname(&self.hostname))?; - self.stage(LaunchStage::MoveRoot, || move_root(&self.root))?; - let umask = set_umask(0).map_err(|error| (LaunchStage::Proc, error))?; - let view = self.mount_view(); - let restored = set_umask(umask).map_err(|error| (LaunchStage::Devices, error)); - view.and(restored.map(drop)) - } - - /// Mounts the view's file systems, as [`enter_view`](Self::enter_view) describes. - fn mount_view(&self) -> Result<(), (LaunchStage, Errno)> { - let root_user = self.uid == 0; - let mut mask = None; - self.stage(LaunchStage::Proc, || { - let proc = mount_proc()?; - if root_user { - let source = mask_source()?; - mask = Some(source); - restrict_proc(proc, source)?; - } - Ok(()) - })?; - self.stage(LaunchStage::Sys, || { - let sys = mount_sys()?; - if let Some(source) = mask { - for name in SYS_MASKED { - mask_entry(sys, name, source)?; - } - } - Ok(()) - })?; - self.stage(LaunchStage::Devices, || mount_devices(self.devices)) - } - - /// Takes the child's identity: drops every capability but its own from the bounding set, - /// takes its supplementary groups, group ID, and user ID, applies its capabilities to the - /// permitted and effective sets, with empty inheritable and ambient sets, and sets - /// `no_new_privs`, in the order of their stages. - /// - /// Each failure names its stage. Only the post-clone child of an entry may call this. - pub(super) fn take_identity(&self) -> Result<(), (LaunchStage, Errno)> { - self.stage(LaunchStage::Bounding, || drop_bounding(self.capabilities))?; - self.stage(LaunchStage::Identity, || { - set_identity(self.uid, self.gid, &self.groups) - })?; - self.stage(LaunchStage::Capabilities, || { - set_capabilities(self.capabilities) - })?; - self.stage(LaunchStage::NoNewPrivileges, || { - prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0) - }) - } - - /// Runs the system calls of `stage`, and names it in their failure. A test that simulates - /// the stages skips them, and fails the stage that it names. - fn stage( - &self, - stage: LaunchStage, - calls: impl FnOnce() -> Result<(), Errno>, - ) -> Result<(), (LaunchStage, Errno)> { - #[cfg(test)] - if let Some(simulation) = self.simulated { - return match simulation { - tests::Simulation::Fail(failing, error) if failing == stage => Err((stage, error)), - _ => Ok(()), - }; - } - calls().map_err(|error| (stage, error)) - } -} - -/// Takes a plain workload's IDs with empty groups and capabilities, without an entry or mounts. +/// Takes a plain workload's IDs with empty groups and capabilities. pub(super) fn take_workload_identity(uid: u32, gid: u32) -> Result<(), (LaunchStage, Errno)> { drop_bounding(0).map_err(|error| (LaunchStage::Bounding, error))?; set_identity(uid, gid, &[]).map_err(|error| (LaunchStage::Identity, error))?; @@ -380,62 +174,6 @@ pub(super) fn take_workload_identity(uid: u32, gid: u32) -> Result<(), (LaunchSt prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0).map_err(|error| (LaunchStage::NoNewPrivileges, error)) } -/// Starts a child in the namespaces that `flags` names, as `fork` starts one otherwise, and -/// returns its process ID in the parent, or `None` in the child. The child's end sends its -/// parent `SIGCHLD`, and the child keeps a copy of its parent's memory and descriptors. -/// -/// # Safety -/// -/// The child may only make system calls over storage that the parent prepared, as the -/// [module](self) describes, and must end in `execve` or `_exit`. -pub(super) unsafe fn clone_into_namespaces(flags: u64) -> Result, Errno> { - let mut arguments = CloneArgs { - flags, - exit_signal: libc::SIGCHLD as u64, - ..CloneArgs::default() - }; - // SAFETY: the arguments are a valid first-version `clone_args` without a stack, so the child - // runs on its copy of the parent's, as after `fork`; the caller keeps the child's promise. - let result = unsafe { - libc::syscall( - libc::SYS_clone3, - &mut arguments as *mut CloneArgs, - std::mem::size_of::(), - ) - }; - match result { - 0 => Ok(None), - child if child > 0 => Ok(Some(Pid::from_raw(child as libc::pid_t))), - _ => Err(Errno::last()), - } -} - -/// Makes the directory at `path` the child's working directory, resolving it in the child's root -/// through no magic link: neither a `/proc` link to a descriptor, a working directory, or a root, -/// nor a symbolic link to one, may then place the child outside its root, or in a directory that -/// a mount of the view covers, such as the writable `/proc/sys` below its read-only bind. -pub(super) fn enter_directory(path: &CStr) -> Result<(), Errno> { - let how = OpenHow { - flags: (libc::O_PATH | libc::O_DIRECTORY | libc::O_CLOEXEC) as u64, - mode: 0, - resolve: libc::RESOLVE_NO_MAGICLINKS, - }; - // SAFETY: the path is a NUL-terminated string, and `how` a valid `open_how` of its size. - let directory = descriptor(unsafe { - libc::syscall( - libc::SYS_openat2, - libc::AT_FDCWD, - path.as_ptr(), - &how, - std::mem::size_of::(), - ) - })?; - // SAFETY: `fchdir` takes a descriptor, which the call above opened. - let entered = unit(unsafe { libc::fchdir(directory) }.into()); - close(directory); - entered -} - /// Returns whether `file` is a regular file, and its size. pub(crate) fn file_size(file: &FileDescriptor) -> Result<(bool, u64), Errno> { let status = nix::sys::stat::fstat(file.borrowed())?; @@ -457,157 +195,12 @@ fn capability_data(capabilities: u64) -> [CapabilityData; 2] { [word(capabilities), word(capabilities >> 32)] } -/// Makes every mount of the child's new mount namespace private, recursively, so that no mount -/// propagates to or from another namespace. -fn make_mounts_private() -> Result<(), Errno> { - nix::mount::mount( - None::<&CStr>, - c"/", - None::<&CStr>, - MsFlags::MS_REC | MsFlags::MS_PRIVATE, - None::<&CStr>, - ) -} - /// Sets the hostname of the current UTS namespace. pub(crate) fn set_hostname(hostname: &[u8]) -> Result<(), Errno> { // SAFETY: the kernel reads `hostname.len()` bytes from the slice. unit(unsafe { libc::syscall(libc::SYS_sethostname, hostname.as_ptr(), hostname.len()) }) } -/// Moves the root mount at `root` over `/` and makes it the child's root and working directory. -/// -/// It first makes every mount of the root's tree `nosuid` and `nodev` in the child's namespace, -/// whatever its options elsewhere. The root then hides the namespace's former root, which no -/// path reaches from it: a `..` from the root's top stays there, as the root's mount point is the -/// top of the former root. -fn move_root(root: &CStr) -> Result<(), Errno> { - nix::unistd::chdir(root)?; - mount_setattr(libc::AT_FDCWD, c".", AT_RECURSIVE, TREE_ATTRIBUTES)?; - nix::mount::mount( - Some(c"."), - c"/", - None::<&CStr>, - MsFlags::MS_MOVE, - None::<&CStr>, - )?; - nix::unistd::chroot(c".")?; - nix::unistd::chdir(c"/") -} - -/// Sets the calling process's umask to `mask`, and returns the umask that it replaces. -/// -/// The child makes the system call itself, because `nix`'s `umask` panics when the call fails, -/// as a seccomp policy can make it. -fn set_umask(mask: libc::mode_t) -> Result { - // SAFETY: `umask` takes an integer and dereferences nothing. - let previous = unsafe { libc::syscall(libc::SYS_umask, mask) }; - if previous < 0 { - Err(Errno::last()) - } else { - // A umask fits a `mode_t`. - Ok(previous as libc::mode_t) - } -} - -/// Mounts a new procfs instance of the child's PID namespace at `/proc`, `nosuid`, `nodev`, and -/// `noexec`, and returns its mount's descriptor. -fn mount_proc() -> Result { - let target = directory_target(libc::AT_FDCWD, c"proc")?; - let proc = new_mount(&PROC)?; - move_mount(proc, target)?; - Ok(proc) -} - -/// Makes the procfs entries of [`PROC_READ_ONLY`] read-only, and masks those of [`PROC_MASKED`] -/// with `source`, in the procfs whose mount `proc` holds. An entry that this kernel lacks needs -/// neither. -fn restrict_proc(proc: RawFd, source: RawFd) -> Result<(), Errno> { - for name in PROC_READ_ONLY { - let Some(target) = existing_entry(proc, name)? else { - continue; - }; - let tree = open_tree( - target, - c"", - OPEN_TREE_CLONE | OPEN_TREE_CLOEXEC | libc::AT_EMPTY_PATH as c_uint, - )?; - mount_setattr(tree, c"", libc::AT_EMPTY_PATH as c_uint, READ_ONLY_BIND)?; - move_mount(tree, target)?; - } - for name in PROC_MASKED { - mask_entry(proc, name, source)?; - } - Ok(()) -} - -/// Mounts a new sysfs at `/sys`, read-only, `nosuid`, `nodev`, and `noexec`, and returns its -/// mount's descriptor. -fn mount_sys() -> Result { - let target = directory_target(libc::AT_FDCWD, c"sys")?; - let sys = new_mount(&SYS)?; - move_mount(sys, target)?; - Ok(sys) -} - -/// Creates the source of the masks of an execution as user ID 0: a tmpfs, `nosuid`, `nodev`, -/// and `noexec`, that holds one empty file and one empty directory, and that is read-only once -/// it holds them. It returns the tmpfs's mount descriptor, of which each mask binds one entry. -fn mask_source() -> Result { - let source = new_mount(&MASK_SOURCE)?; - let file = descriptor( - // SAFETY: the path is a NUL-terminated string, and the mode an integer. - unsafe { - libc::openat( - source, - MASK_FILE.as_ptr(), - libc::O_WRONLY | libc::O_CREAT | libc::O_EXCL | libc::O_CLOEXEC, - 0o444 as c_uint, - ) - } - .into(), - )?; - close(file); - make_directory(source, MASK_DIRECTORY, 0o555)?; - mount_setattr( - source, - c"", - libc::AT_EMPTY_PATH as c_uint, - MOUNT_ATTR_RDONLY, - )?; - Ok(source) -} - -/// Masks the entry `name` of the directory `directory` with the empty directory, or the empty -/// file, of the mask source `source`. An entry that does not exist needs no mask. -fn mask_entry(directory: RawFd, name: &CStr, source: RawFd) -> Result<(), Errno> { - let Some(target) = existing_entry(directory, name)? else { - return Ok(()); - }; - let mask = if file_kind(target)? == libc::S_IFDIR { - MASK_DIRECTORY - } else { - MASK_FILE - }; - let tree = open_tree(source, mask, OPEN_TREE_CLONE | OPEN_TREE_CLOEXEC)?; - move_mount(tree, target) -} - -/// Mounts the child's `/dev`: a tmpfs, `nosuid` and with mode 0755, which holds the device -/// nodes that `devices` selects, with mode 0666, the symbolic links of [`DEVICE_LINKS`], a -/// devpts instance of its own at `pts`, `nosuid` and `noexec`, and a tmpfs at `shm`, `nosuid`, -/// `nodev`, and with mode 1777. -fn mount_devices(devices: u8) -> Result<(), Errno> { - let target = directory_target(libc::AT_FDCWD, c"dev")?; - let dev = new_mount(&DEV)?; - move_mount(dev, target)?; - populate_devices(dev, devices)?; - let pts = new_mount(&DEV_PTS)?; - move_mount(pts, directory_target(dev, c"pts")?)?; - let shm = new_mount(&DEV_SHM)?; - move_mount(shm, directory_target(dev, c"shm")?) -} - /// Creates, in the directory `dev`, the device nodes of [`DEVICE_NODES`] that `devices` selects, /// with mode 0666 but for what the umask withholds, and the symbolic links of [`DEVICE_LINKS`]. fn populate_devices(dev: RawFd, devices: u8) -> Result<(), Errno> { @@ -636,7 +229,7 @@ fn populate_devices(dev: RawFd, devices: u8) -> Result<(), Errno> { } /// Mounts a new instance of `file_system`, detached, and returns its mount's descriptor, which a -/// layered root owns, unlike an entry's child. +/// layered root owns. pub(crate) fn mount_detached(file_system: &FileSystem) -> Result { let mount = new_mount(file_system)?; // SAFETY: `fsmount` returned a new descriptor that nothing else owns. @@ -644,15 +237,14 @@ pub(crate) fn mount_detached(file_system: &FileSystem) -> Result Result<(), Errno> { let flags = AT_RECURSIVE | libc::AT_EMPTY_PATH as c_uint; mount_setattr(directory.raw(), c"", flags, TREE_ATTRIBUTES) } /// Creates in the directory `dev` every device node of [`DEVICE_NODES`], with mode 0666 whatever -/// the umask, which an entry's child clears and a layered root's caller keeps, and the symbolic -/// links of [`DEVICE_LINKS`]. +/// the umask, which a layered root's caller keeps, and the symbolic links of [`DEVICE_LINKS`]. pub(crate) fn create_devices(dev: &FileDescriptor) -> Result<(), Errno> { populate_devices(dev.raw(), u8::MAX)?; let mode = nix::sys::stat::Mode::from_bits_truncate(DEVICE_MODE); @@ -723,45 +315,6 @@ fn prctl(option: c_int, first: c_ulong, second: c_ulong) -> Result<(), Errno> { }) } -/// Returns a descriptor of the directory `name` in `directory`, below it and on its mount, -/// through no symbolic link, and creates the directory, with mode 0755, if it is missing. -fn directory_target(directory: RawFd, name: &CStr) -> Result { - let flags = libc::O_PATH | libc::O_DIRECTORY | libc::O_NOFOLLOW | libc::O_CLOEXEC; - match open_target(directory, name, flags) { - Err(Errno::ENOENT) => { - match make_directory(directory, name, DIRECTORY_MODE) { - Ok(()) | Err(Errno::EEXIST) => {} - Err(error) => return Err(error), - } - open_target(directory, name, flags) - } - result => result, - } -} - -/// Returns a descriptor of the entry `name` of `directory`, without following it, or `None` if -/// there is none. -fn existing_entry(directory: RawFd, name: &CStr) -> Result, Errno> { - match open_target( - directory, - name, - libc::O_PATH | libc::O_NOFOLLOW | libc::O_CLOEXEC, - ) { - Ok(target) => Ok(Some(target)), - Err(Errno::ENOENT) => Ok(None), - Err(error) => Err(error), - } -} - -/// Returns the file type bits of `descriptor`'s file. -fn file_kind(descriptor: RawFd) -> Result { - let mut status = MaybeUninit::::uninit(); - // SAFETY: the kernel fills the status, which is initialized once the call succeeds. - unit(unsafe { libc::fstat(descriptor, status.as_mut_ptr()) }.into())?; - // SAFETY: the successful call initialized the status. - Ok(unsafe { status.assume_init() }.st_mode & libc::S_IFMT) -} - /// Creates the file system `file_system` with its string parameters, mounts it detached with its /// mount attributes, and returns the mount's descriptor. fn new_mount(file_system: &FileSystem) -> Result { @@ -816,31 +369,6 @@ fn configure_and_mount( }) } -/// Opens `path` below `directory`, on its mount, and through no symbolic link, with `flags`. -fn open_target(directory: RawFd, path: &CStr, flags: c_int) -> Result { - let how = OpenHow { - flags: flags as u64, - mode: 0, - resolve: BENEATH, - }; - // SAFETY: the path is a NUL-terminated string, and `how` a valid `open_how` of its size. - descriptor(unsafe { - libc::syscall( - libc::SYS_openat2, - directory, - path.as_ptr(), - &how, - std::mem::size_of::(), - ) - }) -} - -/// Opens the mount tree at `path` in `directory` with `open_tree`'s `flags`. -fn open_tree(directory: RawFd, path: &CStr, flags: c_uint) -> Result { - // SAFETY: the path is a NUL-terminated string. - descriptor(unsafe { libc::syscall(libc::SYS_open_tree, directory, path.as_ptr(), flags) }) -} - /// Sets the mount attributes `attributes` on the mount at `path` in `directory`, with /// `mount_setattr`'s `flags`. fn mount_setattr( @@ -869,32 +397,10 @@ fn mount_setattr( }) } -/// Attaches the mount that `source` holds at the file or directory that `target` holds. -fn move_mount(source: RawFd, target: RawFd) -> Result<(), Errno> { - // SAFETY: both paths are empty NUL-terminated strings, which the flags take as the - // descriptors themselves. - unit(unsafe { - libc::syscall( - libc::SYS_move_mount, - source, - c"".as_ptr(), - target, - c"".as_ptr(), - MOVE_MOUNT_F_EMPTY_PATH | MOVE_MOUNT_T_EMPTY_PATH, - ) - }) -} - -/// Creates the directory `name` in `directory` with `mode`. -fn make_directory(directory: RawFd, name: &CStr, mode: u32) -> Result<(), Errno> { - // SAFETY: the name is a NUL-terminated string, and the mode an integer. - unit(unsafe { libc::mkdirat(directory, name.as_ptr(), mode) }.into()) -} - -/// Closes a descriptor that the child opened, ignoring a failure, as the descriptor is gone +/// Closes a descriptor that this module opened, ignoring a failure, as the descriptor is gone /// either way. fn close(descriptor: RawFd) { - // SAFETY: the child opened `descriptor`, and no owner of it exists. + // SAFETY: this module opened `descriptor`, and no owner of it exists. unsafe { libc::close(descriptor) }; } diff --git a/nvx-rs/src/guest/sys/entry/tests.rs b/nvx-rs/src/guest/sys/entry/tests.rs index 4df423a93..3efd6257f 100644 --- a/nvx-rs/src/guest/sys/entry/tests.rs +++ b/nvx-rs/src/guest/sys/entry/tests.rs @@ -19,14 +19,6 @@ pub(crate) fn seed_inheritable_and_ambient() -> Result<(), Errno> { ) } -/// How a test without privileges simulates the stages of an entry: the child starts without -/// new namespaces and skips the system calls of every entry stage but the one that fails. -#[derive(Clone, Copy, Debug)] -pub(crate) enum Simulation { - /// The stage fails with the error. - Fail(LaunchStage, Errno), -} - #[test] fn capability_sets_split_into_two_data_words() { assert_eq!( @@ -55,8 +47,6 @@ fn capability_sets_split_into_two_data_words() { #[test] fn the_kernel_structures_have_their_uapi_sizes() { - assert_eq!(std::mem::size_of::(), 64); - assert_eq!(std::mem::size_of::(), 24); assert_eq!(std::mem::size_of::(), 32); assert_eq!(std::mem::size_of::(), 8); assert_eq!(std::mem::size_of::(), 12); diff --git a/nvx-rs/src/guest/sys/fd.rs b/nvx-rs/src/guest/sys/fd.rs index b2f7ba2fc..a097938b9 100644 --- a/nvx-rs/src/guest/sys/fd.rs +++ b/nvx-rs/src/guest/sys/fd.rs @@ -7,9 +7,8 @@ use std::thread; use std::time::{Duration, Instant}; pub(crate) use nix::fcntl::OFlag as OpenFlags; -use nix::fcntl::{FcntlArg, FdFlag, Flock, FlockArg, fcntl}; +use nix::fcntl::{FcntlArg, Flock, FlockArg, fcntl}; use nix::poll::{PollFd, PollFlags}; -use nix::sys::socket::{AddressFamily, Shutdown, SockFlag, SockType, shutdown, socketpair}; pub(crate) use nix::sys::stat::Mode; use nix::sys::stat::SFlag; use nix::unistd::Whence; @@ -139,12 +138,6 @@ impl FileDescriptor { nix::sys::stat::fchmod(&self.0, mode) } - /// Shuts down the sending direction of this connected socket, so that its peer reads end of - /// file, which closing it would not tell while another process holds a copy of it. - pub(crate) fn shut_down_sending(&self) -> Result<(), Errno> { - shutdown(self.0.as_raw_fd(), Shutdown::Write) - } - /// Waits at most `timeout_ms` milliseconds for data to read, with one `poll` call. /// /// Returns whether `poll` reported `POLLIN`. An interrupted call is an error, not a retry. @@ -169,27 +162,6 @@ impl FileDescriptor { fcntl(&self.0, FcntlArg::F_SETFL(flags | OpenFlags::O_NONBLOCK)).map(|_| ()) } - /// Checks that this is an end of a pipe, which reads unless `writes`, and makes it - /// nonblocking and close-on-exec, as the ends of the pipes that an execution creates are. An - /// end of another kind of file, or one open for the other direction, is `EINVAL`. - pub(crate) fn take_pipe_end(&self, writes: bool) -> Result<(), Errno> { - let status = nix::sys::stat::fstat(&self.0)?; - if SFlag::from_bits_truncate(status.st_mode) & SFlag::S_IFMT != SFlag::S_IFIFO { - return Err(Errno::EINVAL); - } - let flags = fcntl(&self.0, FcntlArg::F_GETFL).map(OpenFlags::from_bits_retain)?; - let access = if writes { - OpenFlags::O_WRONLY - } else { - OpenFlags::O_RDONLY - }; - if flags & OpenFlags::O_ACCMODE != access { - return Err(Errno::EINVAL); - } - fcntl(&self.0, FcntlArg::F_SETFL(flags | OpenFlags::O_NONBLOCK))?; - fcntl(&self.0, FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC)).map(|_| ()) - } - /// Moves this descriptor to the end of its file. On the kernel log device, it skips the /// records already written, so that reads return only later ones. pub(crate) fn seek_end(&self) -> Result<(), Errno> { @@ -243,7 +215,8 @@ impl io::Write for FileDescriptor { } } -/// Gives up the descriptor to its new owner, which `crate::guest::spawn` hands to a child. +/// Gives up the descriptor to its new owner, such as a socket that takes over an inherited +/// descriptor. impl From for OwnedFd { fn from(descriptor: FileDescriptor) -> Self { descriptor.0 @@ -255,34 +228,12 @@ pub(crate) fn open(path: &CStr, flags: OpenFlags, mode: Mode) -> Result Result { - nix::fcntl::openat(&directory.0, name, flags, mode).map(FileDescriptor) -} - /// Creates a close-on-exec pipe and returns its owned read and write ends. pub(crate) fn create_pipe() -> Result<(FileDescriptor, FileDescriptor), Errno> { nix::unistd::pipe2(OpenFlags::O_CLOEXEC) .map(|(reader, writer)| (FileDescriptor(reader), FileDescriptor(writer))) } -/// Creates a connected pair of close-on-exec Unix stream sockets and returns its owned ends. -pub(crate) fn create_socket_pair() -> Result<(FileDescriptor, FileDescriptor), Errno> { - socketpair( - AddressFamily::Unix, - SockType::Stream, - None, - SockFlag::SOCK_CLOEXEC, - ) - .map(|(first, second)| (FileDescriptor(first), FileDescriptor(second))) -} - /// Makes the null device the standard input, output, and error, and closes every other /// descriptor, best effort, in a forked child that may outlive its parent's interest in it. The /// child then holds none of the parent's descriptors, such as the writer of a pipe, whose diff --git a/nvx-rs/src/guest/sys/fd/tests.rs b/nvx-rs/src/guest/sys/fd/tests.rs index 0a4cd0c53..d1583f9ca 100644 --- a/nvx-rs/src/guest/sys/fd/tests.rs +++ b/nvx-rs/src/guest/sys/fd/tests.rs @@ -80,25 +80,6 @@ fn an_expired_write_deadline_leaves_the_pipe_unchanged() { assert_eq!(&bytes, b"on time"); } -/// Sets the soft limit of the process's open descriptors to `limit`, and returns the limit that -/// it replaced, for a test that runs out of descriptors in a process of its own. -/// -/// The crate does not enable `nix`'s resource limits, so this calls `getrlimit` and `setrlimit` -/// through libc. -pub(crate) fn set_descriptor_limit(limit: nix::libc::rlim_t) -> Result { - let mut limits = nix::libc::rlimit { - rlim_cur: 0, - rlim_max: 0, - }; - // SAFETY: `limits` is a writable `rlimit`, which the call only fills. - Errno::result(unsafe { nix::libc::getrlimit(nix::libc::RLIMIT_NOFILE, &mut limits) })?; - let previous = limits.rlim_cur; - limits.rlim_cur = limit; - // SAFETY: `limits` is a valid `rlimit`, which the call only reads. - Errno::result(unsafe { nix::libc::setrlimit(nix::libc::RLIMIT_NOFILE, &limits) })?; - Ok(previous) -} - /// Creates `path` as a FIFO with `mode`, with which tests stand for a console, and keep their /// system calls in this module. pub(crate) fn create_fifo(path: &CStr, mode: Mode) -> Result<(), Errno> { diff --git a/nvx-rs/src/guest/sys/mod.rs b/nvx-rs/src/guest/sys/mod.rs index 669326e24..56d49f625 100644 --- a/nvx-rs/src/guest/sys/mod.rs +++ b/nvx-rs/src/guest/sys/mod.rs @@ -26,12 +26,11 @@ //! which reach libc through `nix`'s re-export: a spawned child, which may neither allocate nor //! drop an owner, fills its preallocated `PWD` entry with the `getcwd` system call, closes the //! descriptors that it does not use with the `close_range` system call, and executes and exits -//! through libc; an execution's entry starts its child with `clone3`, and the child changes its -//! credentials, capabilities, hostname, and umask, and mounts its view and enters its working -//! directory with the new mount API and `openat2`, through libc's `syscall`, as the entry's -//! module describes, because libc does not know of that child; the agent's wait-status decoding, -//! the waits for one named child, and an execution's signal to its program's process group by the -//! group's ID, before Linux 6.9, keep the real-time signals that `nix`'s `Signal` cannot represent; +//! through libc; a workload's child changes its credentials and capabilities, the VM's hostname +//! is set, and a layered root's runtime file systems are mounted with the new mount API, through +//! libc's `syscall`, as the entry module describes; the agent's wait-status decoding, the waits +//! for one named child, and a signal to a process group by the group's ID keep the real-time +//! signals that `nix`'s `Signal` cannot represent; //! a pidfd is opened with the `pidfd_open` system call, and signals are sent through it with //! `pidfd_send_signal`, which neither `nix` nor musl wraps; a write to an execution's input //! takes back the `SIGPIPE` that a closed reader raises with `sigtimedwait`, once `sigpending` @@ -43,9 +42,9 @@ //! and `getpid` system calls, which the latency probe measures without the vDSO's or libc's //! shortcuts. The guest mappings need `statx`, for the kinds and unique mount IDs of open files, //! and the mount API's `open_tree`, `mount_setattr`, and `move_mount`, which neither `nix` nor -//! musl wraps. The tests trace a child with libc's `ptrace`, and lower the process's descriptor -//! limit with libc's `setrlimit`, as the crate enables neither for `nix`, and deny a system call -//! to a thread with libc's `prctl`, as `nix` does not wrap seccomp. Each such call says why. +//! musl wraps. The tests lower a resource limit with libc's `setrlimit`, as the crate does not +//! enable it for `nix`, and deny a system call to a thread with libc's `prctl`, as `nix` does not +//! wrap seccomp. Each such call says why. //! //! The x86 instructions of the time tools are `core::arch` intrinsics, except the port //! instructions, which only `port` executes, as inline assembly. The constants of the kernel's @@ -90,14 +89,14 @@ pub(crate) use clock::{Clock, clock_time, monotonic_ms, sleep_once, sleep_until_ pub(crate) use clock::{monotonic_ns, syscall_monotonic_nanoseconds}; pub(crate) use discipline::{ClockAdjustment, ClockDiscipline, FrequencyHold, adjust_clock}; pub(crate) use entry::{ - DEV, DEV_PTS, DEV_SHM, EntryChild, FileSystem, PROC, RUN, SYS, create_devices, file_size, - mount_detached, restrict_tree, set_hostname, + DEV, DEV_PTS, DEV_SHM, FileSystem, PROC, RUN, SYS, create_devices, file_size, mount_detached, + restrict_tree, set_hostname, }; #[cfg(target_arch = "x86_64")] pub(crate) use fd::close_inherited_descriptors; pub(crate) use fd::{ CreatedFile, FileDescriptor, FileLock, Mode, OpenFlags, create_directory, create_pipe, - create_socket_pair, lock_file, open, open_at, page_size, try_lock_directory, unlink, + lock_file, open, page_size, try_lock_directory, unlink, }; pub(crate) use filesystem::{ MountFlags, PATH_MAX, ScratchDirectory, TmpfsMount, check_exists, chmod, chown, @@ -117,11 +116,10 @@ pub(crate) use port::{PORT_DEVICE, Port}; pub(crate) use port::{PortAccess, PortError, check_released, power_off, probe_port}; pub(crate) use process::set_command_name; pub(crate) use process::{ - ChildStatus, DirectStdout, ForkResult, ProcessId, Stdout, WaitTarget, current_process, detach, - exec, exit_immediately, fork, ignore_interrupt_and_hangup, is_unreaped, kill_process, - kill_process_group, kill_through_pidfd, open_pidfd, process_exists, process_group_of, - process_name, reap_any_nonblocking, restore_sigpipe, signal_process_group, - signal_through_pidfd, take_inherited_descriptor, try_wait, wait, wait_named, + ChildStatus, DirectStdout, ForkResult, ProcessId, Stdout, current_process, detach, exec, + exit_immediately, fork, ignore_interrupt_and_hangup, is_unreaped, kill_process, + kill_process_group, kill_through_pidfd, open_pidfd, process_exists, process_name, + reap_any_nonblocking, restore_sigpipe, take_inherited_descriptor, try_wait, wait, write_port_diagnostic, write_standard_error, }; #[cfg(target_arch = "x86_64")] diff --git a/nvx-rs/src/guest/sys/port/tests.rs b/nvx-rs/src/guest/sys/port/tests.rs index df2d06923..519ed1d90 100644 --- a/nvx-rs/src/guest/sys/port/tests.rs +++ b/nvx-rs/src/guest/sys/port/tests.rs @@ -155,27 +155,35 @@ mod transactions { #[test] fn a_thread_that_may_hold_the_permission_spawns_no_command() { - use crate::guest::spawn::Command; - use crate::guest::sys::{self, ChildStatus}; + use std::io::Read; - let command = || Command::new(c"/bin/true".to_owned(), vec![c"true".to_owned()]); - let refusal = |command: Command| command.spawn().unwrap_err().raw_os_error(); + use crate::guest::sys::{self, ChildSpec, ChildStatus, PreparedChild}; + + let spawn = || { + PreparedChild::new(ChildSpec::new( + c"/bin/true".to_owned(), + vec![c"true".to_owned()], + )) + .unwrap() + .spawn() + }; let backend = FakePermission::default(); let access = PortAccess::with_backend(&backend).unwrap(); - assert_eq!(refusal(command()), Some(Errno::EPERM as i32)); + assert_eq!(spawn().err(), Some(Errno::EPERM)); backend.fail_revocations(Some(Errno::ENOMEM)); assert_eq!(access.release(), Err(PortError::Revoke(Errno::ENOMEM))); - assert_eq!(refusal(command()), Some(Errno::EPERM as i32)); + assert_eq!(spawn().err(), Some(Errno::EPERM)); // A later transaction revokes the permission, and the thread starts a command again. backend.fail_revocations(None); PortAccess::with_backend(&backend) .unwrap() .release() .unwrap(); - let spawned = command().spawn().unwrap(); - let process = spawned.process(); - assert_eq!(spawned.into_report().wait().unwrap(), None); - assert_eq!(sys::wait(process), Some(ChildStatus::Exited(0))); + let mut started = spawn().unwrap(); + let mut report = Vec::new(); + started.report.read_to_end(&mut report).unwrap(); + assert!(report.is_empty()); + assert_eq!(sys::wait(started.process), Some(ChildStatus::Exited(0))); } #[test] diff --git a/nvx-rs/src/guest/sys/process.rs b/nvx-rs/src/guest/sys/process.rs index 47d1d6310..89fd21259 100644 --- a/nvx-rs/src/guest/sys/process.rs +++ b/nvx-rs/src/guest/sys/process.rs @@ -304,8 +304,7 @@ fn child_status(process: nix::libc::pid_t, status: nix::libc::c_int) -> ReapedCh /// /// Only a program that is its process's one status collector calls this: the managed agent, which /// as the direct agent is PID 1 and also collects its workloads' orphaned descendants, and the -/// time daemon. The guest libraries wait only for the child that the caller names, with -/// [`wait_named`]. +/// time daemon. /// /// libc is used instead of `nix` because `nix` converts a terminating signal to its finite /// `Signal` enum and rejects Linux real-time signal numbers, after the child is reaped. @@ -382,88 +381,13 @@ pub(crate) fn kill_process(process: ProcessId) -> Result<(), Errno> { kill(process.0, Signal::SIGKILL) } -/// Returns the ID of the process group to which `process` belongs. The lookup goes by -/// `process`'s ID, which names `process` only while its status stays uncollected. -pub(crate) fn process_group_of(process: ProcessId) -> Result { - nix::unistd::getpgid(Some(process.0)).map(ProcessId) -} - -/// One child that a wait names, which is all that it may collect. -#[derive(Clone, Copy, Debug)] -pub(crate) enum WaitTarget<'a> { - /// The child with this process ID. - Process(ProcessId), - /// The child to which this pidfd refers. - Pidfd(BorrowedFd<'a>), -} - -/// Collects the status of the child that `target` names once it has ended, and returns its raw -/// wait status, in the form that `waitpid` reports. Without `block`, it returns `None` while the -/// child runs. -/// -/// Only an exit or a terminating signal ends the child. `waitid` also reports each stop of a -/// child that this process traces, whatever its options, which leaves the child running: a -/// blocking wait goes on past such a stop, and one without `block` returns `None`. -/// -/// Only the named child's status is collected, so every other child's stays for its collector. -/// The wait names the child whatever signal it reports its end with, as `__WALL` makes it, so a -/// child that `clone` started with an exit signal other than `SIGCHLD`, or none, and that ends -/// before it runs a program, which would reset that signal, is collected too. Interruptions are -/// retried. A target that is not a child of this process, or whose status was collected already, -/// is `ECHILD`, and a blocking wait through a pidfd that its opener made nonblocking, while the -/// child runs, is `EAGAIN`. -/// -/// libc is used because `nix`'s `waitid` decodes the state into a `WaitStatus`, which fails for a -/// real-time signal after the child is reaped. The raw status is rebuilt from the child event's -/// code and status, which keeps the signal number and the core dump. -pub(crate) fn wait_named(target: WaitTarget<'_>, block: bool) -> Result, Errno> { - let (kind, id) = match target { - WaitTarget::Process(process) => (nix::libc::P_PID, process.0.as_raw() as nix::libc::id_t), - WaitTarget::Pidfd(descriptor) => ( - nix::libc::P_PIDFD, - descriptor.as_raw_fd() as nix::libc::id_t, - ), - }; - let options = if block { - nix::libc::WEXITED | nix::libc::__WALL - } else { - nix::libc::WEXITED | nix::libc::WNOHANG | nix::libc::__WALL - }; - loop { - // SAFETY: zero is a valid initial representation for `siginfo_t`, and leaves `si_pid` zero - // unless a child event fills it. - let mut information = unsafe { std::mem::zeroed::() }; - // SAFETY: `information` is writable, and the call collects at most the named child. - let result = unsafe { nix::libc::waitid(kind, id, &mut information, options) }; - if result < 0 { - match Errno::last() { - Errno::EINTR => continue, - error => return Err(error), - } - } - // SAFETY: a successful `waitid` either initialized the Linux child-event fields, or, for a - // child that still runs, left the zeroed `si_pid`. - let (process, status) = unsafe { (information.si_pid(), information.si_status()) }; - if process == 0 { - return Ok(None); - } - return Ok(Some(match information.si_code { - nix::libc::CLD_EXITED => (status & 0xff) << 8, - nix::libc::CLD_KILLED => status & 0x7f, - nix::libc::CLD_DUMPED => (status & 0x7f) | 0x80, - // A traced child's stop, which does not end it. - _ if block => continue, - _ => return Ok(None), - })); - } -} - /// Returns whether the child to which `pidfd` refers is unreaped: whether it runs, or has ended /// with a status that no collector has taken yet, which the check leaves in place, as `WNOWAIT` /// makes it. Unlike a signal 0, the check needs no permission to signal the child. A child whose /// status a collector took, as the wait's `ECHILD` tells, is not. /// -/// libc is used for the same reason as in [`wait_named`], whose options the check shares. +/// libc is used because `nix`'s `waitid` decodes the state into a `WaitStatus`, which fails for a +/// real-time signal. pub(crate) fn is_unreaped(pidfd: BorrowedFd<'_>) -> Result { probe_unreaped(nix::libc::P_PIDFD, pidfd.as_raw_fd() as nix::libc::id_t) } diff --git a/nvx-rs/src/guest/sys/process/tests.rs b/nvx-rs/src/guest/sys/process/tests.rs index ca1512d01..9739ac972 100644 --- a/nvx-rs/src/guest/sys/process/tests.rs +++ b/nvx-rs/src/guest/sys/process/tests.rs @@ -74,67 +74,6 @@ pub(crate) fn fork_isolated_test() -> Result { } } -/// Starts a child that ends at once with `status`, before it could run a program, and that -/// reports its end with no signal, as a child that `clone` starts with no exit signal does, for -/// the tests of the waits that name such a child. -pub(crate) fn spawn_child_without_exit_signal(status: u8) -> Result { - let none: nix::libc::c_long = 0; - // SAFETY: without `CLONE_VM`, and with no stack of its own, the child runs in a copy of this - // process, as after `fork`, where it calls only `_exit`, which takes no lock that another of - // this process's threads could hold. No flag asks for an exit signal. - let child = unsafe { nix::libc::syscall(nix::libc::SYS_clone, none, none, none, none, none) }; - match child { - 0 => exit_immediately(status), - child if child < 0 => Err(Errno::last()), - child => { - let child = nix::libc::pid_t::try_from(child).map_err(|_| Errno::EOVERFLOW)?; - Ok(ProcessId::from_pid(Pid::from_raw(child))) - } - } -} - -/// Forks a child that makes the calling thread its tracer, stops with `SIGSTOP`, and exits with -/// `status` once its tracer continues it, or with 99 if it cannot be traced. -/// -/// libc's `ptrace` is used because the crate does not enable `nix`'s. -pub(crate) fn spawn_traced_child(status: u8) -> ProcessId { - // SAFETY: the child makes only the async-signal-safe `ptrace`, `raise`, and `_exit` calls, - // as a child of a multithreaded process may. - match unsafe { nix::unistd::fork() }.expect("fork") { - nix::unistd::ForkResult::Child => { - // SAFETY: `PTRACE_TRACEME` reads none of the other arguments. - let traced = unsafe { - nix::libc::ptrace( - nix::libc::PTRACE_TRACEME, - 0, - std::ptr::null_mut::(), - std::ptr::null_mut::(), - ) - }; - if traced != 0 { - exit_immediately(99); - } - let _ = nix::sys::signal::raise(Signal::SIGSTOP); - exit_immediately(status) - } - nix::unistd::ForkResult::Parent { child } => ProcessId::from_pid(child), - } -} - -/// Continues `process`, which the calling thread traces and which is stopped, without a signal. -pub(crate) fn continue_traced(process: ProcessId) { - // SAFETY: `PTRACE_CONT` reads the process ID, and a null signal, which delivers none. - let result = unsafe { - nix::libc::ptrace( - nix::libc::PTRACE_CONT, - process.0.as_raw(), - std::ptr::null_mut::(), - std::ptr::null_mut::(), - ) - }; - assert_eq!(result, 0, "PTRACE_CONT: {}", Errno::last()); -} - /// Makes the system call `number` fail with `EPERM` in the calling thread, and in every process /// that it starts from now on, through a seccomp filter that allows every other call, for a test /// that runs in a process of its own, since no filter can be removed. It first sets the thread's @@ -406,7 +345,6 @@ fn a_real_time_signal_goes_through_a_pidfd_and_to_a_process_group() { .spawn() .unwrap(); let process = ProcessId::from_child(&leader); - assert_eq!(process_group_of(process), Ok(process)); assert_eq!(signal_process_group(process, 65), Err(Errno::EINVAL)); assert_eq!(signal_process_group(process, 41), Ok(())); assert_eq!(leader.wait().unwrap().signal(), Some(41)); diff --git a/nvx-rs/src/guest/sys/signal/tests.rs b/nvx-rs/src/guest/sys/signal/tests.rs index b8c963f3c..497fcc06f 100644 --- a/nvx-rs/src/guest/sys/signal/tests.rs +++ b/nvx-rs/src/guest/sys/signal/tests.rs @@ -139,12 +139,12 @@ fn a_write_takes_back_its_own_sigpipe_beside_one_pending_for_the_process() { crate::guest::sys::exit_immediately(check().err().unwrap_or(0)) } nix::unistd::ForkResult::Parent { child } => { - let id = child.as_raw().unsigned_abs(); - let status = crate::guest::process::Child::new(id) - .unwrap() - .wait() - .unwrap(); - assert_eq!(status.code(), Some(0), "{status:?}"); + let status = crate::guest::sys::wait(ProcessId::from_pid(child)); + assert_eq!( + status, + Some(crate::guest::sys::ChildStatus::Exited(0)), + "{status:?}" + ); } } } diff --git a/nvx-rs/src/guest/sys/spawn.rs b/nvx-rs/src/guest/sys/spawn.rs index 4c0461287..e64687287 100644 --- a/nvx-rs/src/guest/sys/spawn.rs +++ b/nvx-rs/src/guest/sys/spawn.rs @@ -1,8 +1,7 @@ -//! The fork/exec boundary of [`crate::guest::spawn`]. +//! The fork/exec boundary of nvx-init's plain-process executions. //! //! [`PreparedChild::new`] checks a [`ChildSpec`] and allocates or opens everything that the child -//! uses after `fork`, and [`PreparedChild::spawn`] forks it, or, for an execution's -//! [entry](super::EntryChild), starts it with `clone3` in new namespaces. The child branch +//! uses after `fork`, and [`PreparedChild::spawn`] forks it. The child branch //! performs no allocation, locking, formatting, or Rust cleanup: it makes direct system calls //! over that storage, in the order of [`LaunchStage`], and ends in `execve`, `execv`, or `_exit`. //! It makes them through `nix` where its wrappers neither allocate nor consume an owner, and @@ -10,20 +9,19 @@ //! may walk the file system, closes the descriptors that the child does not use with //! `close_range`, resets the signals with `rt_sigaction` and `rt_sigprocmask`, since libc's //! `sigaction` refuses the signals that libc reserves for itself and musl's takes a lock for -//! `SIGABRT`, and makes the entry's calls that the [entry](super::entry) module lists; and its -//! `execve`, `execv`, and `_exit` end the child. A stage that fails writes its number and `errno` -//! to the launch report, a close-on-exec pipe, with `SIGPIPE` blocked, and ends the child with -//! status 125, so that the report's reader reaches end of file without a report once the program -//! runs, and a report that no reader takes still ends the child with that status. +//! `SIGABRT`, and makes the workload identity's calls that the [entry](super::entry) module +//! lists; and its `execve`, `execv`, and `_exit` end the child. A stage that fails writes its +//! number and `errno` to the launch report, a close-on-exec pipe, with `SIGPIPE` blocked, and +//! ends the child with status 125, so that the report's reader reaches end of file without a +//! report once the program runs, and a report that no reader takes still ends the child with +//! that status. //! //! Every descriptor that the child uses after it places its standard streams has a number above -//! theirs, so that placing them replaces none of it: [`PreparedChild::new`] moves a stream source, -//! the root, the cgroup's membership file, the barrier's ends, or the report's writer that has a -//! lower number. It also makes them close-on-exec, so that the program receives -//! them only as its standard streams. A child that waits at a barrier, or enters an execution, -//! first closes every other descriptor, so that it holds no copy of another child's while it -//! waits; an entry's child also closes, before it enters its working directory, every descriptor -//! that its stages used or opened but those that its program inherits. +//! theirs, so that placing them replaces none of it: [`PreparedChild::new`] moves a stream source +//! or the report's writer that has a lower number. It also makes them close-on-exec, so that the +//! program receives them only as its standard streams. A workload's child first closes every +//! other descriptor, so that it holds no copy of another child's, and closes, before it enters +//! its working directory, every descriptor but its standard streams and its launch report's. use std::ffi::{CStr, CString}; use std::os::fd::RawFd; @@ -31,10 +29,9 @@ use std::os::unix::process::CommandExt; use nix::fcntl::{FcntlArg, FdFlag}; use nix::libc::{c_char, c_int, c_uint, c_ulong}; -use nix::unistd::{ForkResult, Gid, Pid, Uid}; +use nix::unistd::{ForkResult, Pid}; -use super::{EntryChild, Errno, FileDescriptor, PATH_MAX, ProcessId}; -use crate::guest::spawn::LaunchStage; +use super::{Errno, FileDescriptor, PATH_MAX, ProcessId}; /// Prefix of the mutable physical-working-directory environment entry. const PWD_PREFIX: &[u8] = b"PWD="; @@ -55,7 +52,64 @@ pub(crate) const REPORT_LENGTH: usize = 8; /// signal sets `rt_sigaction` and `rt_sigprocmask` take as one 64-bit word. const SIGNALS: c_int = 64; -/// Everything that a caller asks of one child, as [`crate::guest::spawn::Command`] collects it. +/// A step that a child takes after it starts and before its program runs. +/// +/// The variants are in the order of the steps, and a child skips the steps that its +/// [`ChildSpec`] does not ask for. A child whose step fails reports it and exits with status 125 +/// without running its program. +/// +/// Each stage has a number, its [`code`](Self::code), with which the child's launch report names +/// the stage that failed, and which the report's readers decode: a stage keeps its number, and a +/// new stage takes a number of its own, wherever it comes in the order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum LaunchStage { + /// The child makes itself the leader of a new process group. + ProcessGroup, + /// The child places its standard streams. A workload's child first closes every descriptor + /// that it does not use. + Descriptors, + /// The child drops every capability but its own from its bounding set. + Bounding, + /// The child takes its supplementary groups, group ID, and user ID. + Identity, + /// The child applies its capabilities, and empties its inheritable and ambient sets. + Capabilities, + /// The child sets `no_new_privs`. + NoNewPrivileges, + /// The child enters its working directory. A workload's child first closes every descriptor + /// but its standard streams and its launch report's. + WorkingDirectory, + /// The child records the physical path of its working directory in `PWD`. + PhysicalWorkingDirectory, + /// The child gives every signal its default disposition, then unblocks every signal. + Signals, + /// The child replaces itself with its program: the first of those that its search finds that + /// it can run, if its spec has a search path. + Exec, +} + +impl LaunchStage { + /// Returns the stage's number: from 1, for [`ProcessGroup`](Self::ProcessGroup), to 21, for + /// [`Exec`](Self::Exec), in the order of the stages, with gaps where stages left. A stage + /// that NVX adds takes 22, then 23, and so on, wherever it comes in the order, and no stage + /// is numbered 0. + pub(crate) const fn code(self) -> u32 { + match self { + Self::ProcessGroup => 1, + Self::Descriptors => 4, + Self::Bounding => 13, + Self::Identity => 14, + Self::Capabilities => 15, + Self::NoNewPrivileges => 16, + Self::WorkingDirectory => 18, + Self::PhysicalWorkingDirectory => 19, + Self::Signals => 20, + Self::Exec => 21, + } + } +} + +/// Everything that a caller asks of one child. #[derive(Debug)] pub(crate) struct ChildSpec { /// Path of the program. @@ -69,29 +123,15 @@ pub(crate) struct ChildSpec { environment: Option>, /// Whether `PWD` receives the physical working directory. physical_pwd: bool, - /// Directory that becomes the child's root. - root: Option, /// Absolute working directory, resolved in the root, or `None` to keep the current one. directory: Option, - /// Effective user and group IDs with which the child enters `directory`. - credentials: Option<(u32, u32)>, /// Sources of standard input, output, and error; `None` keeps the caller's. streams: [Option; 3], - /// Descriptors that stay open across `exec`, at their numbers. - inherited: Vec, - /// Membership file of the cgroup that the child joins. - cgroup: Option, /// Whether the child leads a new process group. process_group: bool, - /// Whether the child leads a new session. - session: bool, - /// Whether the child waits at a barrier until its caller releases it. - barrier: bool, /// Whether the child gives every signal its default disposition, and blocks none, before its /// program runs. reset_signals: bool, - /// The execution that the child enters, if any. - entry: Option, /// The identity of an init workload, without an entry or per-execution containment. workload: Option<(u32, u32)>, /// Test-only outcome of the credential system calls, which the child then skips: @@ -110,17 +150,10 @@ impl ChildSpec { search_path: None, environment: None, physical_pwd: false, - root: None, directory: None, - credentials: None, streams: [None, None, None], - inherited: Vec::new(), - cgroup: None, process_group: false, - session: false, - barrier: false, reset_signals: false, - entry: None, workload: None, #[cfg(test)] simulated_credentials: None, @@ -143,62 +176,16 @@ impl ChildSpec { self.physical_pwd = true; } - /// Makes the directory that `root` opens the child's root. - pub(crate) fn set_root(&mut self, root: FileDescriptor) { - self.root = Some(root); - } - /// Makes the absolute path `directory`, resolved in the root, the child's working directory. pub(crate) fn set_directory(&mut self, directory: CString) { self.directory = Some(directory); } - /// Makes the child enter its working directory with the effective `uid` and `gid`. - pub(crate) fn set_credentials(&mut self, uid: u32, gid: u32) { - self.credentials = Some((uid, gid)); - } - /// Makes `descriptor` the child's standard stream numbered `stream`: 0, 1, or 2. pub(crate) fn set_stream(&mut self, stream: usize, descriptor: FileDescriptor) { self.streams[stream] = Some(descriptor); } - /// Keeps `descriptor` open in the child at its number. - pub(crate) fn inherit(&mut self, descriptor: FileDescriptor) { - self.inherited.push(descriptor); - } - - /// Makes the child join the cgroup whose membership file `cgroup` opens. - pub(crate) fn set_cgroup(&mut self, cgroup: FileDescriptor) { - self.cgroup = Some(cgroup); - } - - /// Makes the child the leader of a new process group. - pub(crate) fn new_process_group(&mut self) { - self.process_group = true; - } - - /// Makes the child the leader of a new session, without a controlling terminal. - pub(crate) fn new_session(&mut self) { - self.session = true; - } - - /// Makes the child wait at a barrier until its caller releases it. - pub(crate) fn set_barrier(&mut self) { - self.barrier = true; - } - - /// Makes the child give every signal its default disposition, and block none, before its - /// program runs. - pub(crate) fn reset_signals(&mut self) { - self.reset_signals = true; - } - - /// Makes the child enter the execution that `entry` prepares. - pub(crate) fn set_entry(&mut self, entry: EntryChild) { - self.entry = Some(entry); - } - /// Gives an init workload its identity, its own process group, and reset signals. pub(crate) fn set_workload(&mut self, uid: u32, gid: u32) { self.workload = Some((uid, gid)); @@ -277,17 +264,6 @@ impl Search { } } -/// Credentials of the working-directory check, and the supplementary groups that the child gives -/// back after it. -struct Credentials { - /// Effective user ID of the check. - uid: Uid, - /// Effective group ID of the check. - gid: Gid, - /// The parent's supplementary groups, which the child shares until the check. - groups: Vec, -} - /// A checked child whose every input is allocated or opened, ready to fork. pub(crate) struct PreparedChild { /// Path of the program. @@ -300,44 +276,25 @@ pub(crate) struct PreparedChild { environment: Option>, /// Physical `PWD` entry that the child fills. physical_pwd: Option, - /// Directory that becomes the child's root. - root: Option, /// Absolute working directory, resolved in the root. directory: Option, - /// Credentials of the working-directory check. - credentials: Option, /// Sources of standard input, output, and error. streams: [Option; 3], - /// Descriptors that stay open across `exec`, at their numbers. - inherited: Vec, - /// Membership file of the cgroup that the child joins. - cgroup: Option, /// Whether the child leads a new process group. process_group: bool, - /// Whether the child leads a new session, which an entry implies. - session: bool, - /// The reader and the writer of the barrier at which the child waits, if any: the ends of a - /// connected socket pair, of which the child keeps the reader, and its caller the writer, - /// whose shutdown the child reads as end of file even while other children hold copies of - /// it. - barrier: Option<(FileDescriptor, FileDescriptor)>, - /// Whether the child resets its signals before its program runs, which an entry implies. + /// Whether the child resets its signals before its program runs. reset_signals: bool, - /// The execution that the child enters, if any. - entry: Option, /// The identity of an init workload, whose priority is reset before it takes it. workload: Option<(u32, u32)>, /// Reader of the launch report, which the caller receives. report_reader: FileDescriptor, /// Writer of the launch report, which only the child keeps. report_writer: FileDescriptor, - /// The numbers of every descriptor that the child uses before its program runs, or that its - /// program inherits, in ascending order: the stream sources, the root, the cgroup's - /// membership file, the barrier's reader, the report's writer, and the inherited - /// descriptors. A child that waits at a barrier, or enters an execution, closes every other. + /// The numbers of every descriptor that the child uses before its program runs, in ascending + /// order: the stream sources and the report's writer. A workload's child closes every other. used: Box<[RawFd]>, - /// The numbers of the descriptors that an entry's program inherits, and of the report's - /// writer, in ascending order: all that its child keeps once its stages have run. + /// The number of the report's writer: all that a workload's child keeps, but its standard + /// streams, once its stages have run. kept: Box<[RawFd]>, /// Test-only outcome of the credential system calls. #[cfg(test)] @@ -350,36 +307,22 @@ pub(crate) struct Started { pub(crate) process: ProcessId, /// The reader of the child's launch report. pub(crate) report: FileDescriptor, - /// The writer of the barrier at which the child waits, if it waits at one. - pub(crate) barrier: Option, } impl PreparedChild { /// Checks `spec` and prepares every input of its child. /// - /// The working directory must be absolute, credentials need a working directory, the physical - /// `PWD` an exact environment, and an inherited descriptor a number above the standard - /// streams'; a command with no argument zero is invalid too. An entry needs a cgroup or a - /// barrier, at which its caller places it, and takes neither a root directory nor - /// credentials, as its own root and identity replace them. Each fails with `EINVAL`. A program - /// that the spec names without a `/`, in a spec with a search path, gets here the one buffer - /// in which its child builds, in turn, each path at which it looks for it. + /// The working directory must be absolute, and the physical `PWD` needs an exact + /// environment; a command with no argument zero is invalid too. Each fails with `EINVAL`. A + /// program that the spec names without a `/`, in a spec with a search path, gets here the one + /// buffer in which its child builds, in turn, each path at which it looks for it. pub(crate) fn new(spec: ChildSpec) -> Result { if spec.arguments.is_empty() || spec .directory .as_ref() .is_some_and(|directory| directory.as_bytes().first() != Some(&b'/')) - || (spec.credentials.is_some() && spec.directory.is_none()) || (spec.physical_pwd && spec.environment.is_none()) - || spec - .inherited - .iter() - .any(|descriptor| descriptor.raw() < FIRST_NONSTANDARD) - || (spec.entry.is_some() - && ((spec.cgroup.is_none() && !spec.barrier) - || spec.root.is_some() - || spec.credentials.is_some())) { return Err(Errno::EINVAL); } @@ -389,44 +332,15 @@ impl PreparedChild { stdout.map(above_standard_streams).transpose()?, stderr.map(above_standard_streams).transpose()?, ]; - let root = spec.root.map(above_standard_streams).transpose()?; - let cgroup = spec.cgroup.map(above_standard_streams).transpose()?; - let barrier = if spec.barrier { - let (reader, writer) = super::create_socket_pair()?; - Some(( - above_standard_streams(reader)?, - above_standard_streams(writer)?, - )) - } else { - None - }; - let entry = spec.entry; let (report_reader, report_writer) = super::create_pipe()?; let report_writer = above_standard_streams(report_writer)?; - let credentials = match spec.credentials { - Some((uid, gid)) => Some(Credentials { - uid: Uid::from_raw(uid), - gid: Gid::from_raw(gid), - groups: nix::unistd::getgroups()?, - }), - None => None, - }; let mut environment = spec.environment; let physical_pwd = match (&mut environment, spec.physical_pwd) { (Some(environment), true) => Some(PhysicalPwd::new(environment)), _ => None, }; - let used = ascending_numbers( - streams - .iter() - .flatten() - .chain(&root) - .chain(&cgroup) - .chain(barrier.as_ref().map(|(reader, _)| reader)) - .chain([&report_writer]) - .chain(&spec.inherited), - ); - let kept = ascending_numbers(spec.inherited.iter().chain([&report_writer])); + let used = ascending_numbers(streams.iter().flatten().chain([&report_writer])); + let kept = ascending_numbers([&report_writer]); let search = spec .search_path .and_then(|path| Search::new(&spec.program, path)); @@ -436,17 +350,10 @@ impl PreparedChild { search, environment, physical_pwd, - root, directory: spec.directory, - credentials, streams, - inherited: spec.inherited, - cgroup, process_group: spec.process_group, - session: spec.session || entry.is_some(), - barrier, - reset_signals: spec.reset_signals || entry.is_some(), - entry, + reset_signals: spec.reset_signals, workload: spec.workload, report_reader, report_writer, @@ -458,9 +365,7 @@ impl PreparedChild { } /// Forks the child, which runs its launch stages and then its program, and returns its - /// process ID, the reader of its launch report, and the writer of its barrier, if any. The - /// child of an entry starts with `clone3`, in new mount, PID, and UTS namespaces, of which it - /// is PID 1. + /// process ID and the reader of its launch report. /// /// On x86-64, it fails with `EPERM`, before it forks, while the calling thread may still /// hold the permission for OpenVMM's I/O ports that a time call could not revoke, which the @@ -510,14 +415,9 @@ impl PreparedChild { // SAFETY: every child input and pointer vector was allocated above, and the child runs // only the non-returning routine below. - let child = match self.entry.as_ref().map(EntryChild::clone_flags) { - // SAFETY: as above; the entry's child makes only the system calls that its module - // permits. - Some(flags) => unsafe { super::entry::clone_into_namespaces(flags) }?, - None => match unsafe { nix::unistd::fork() }? { - ForkResult::Child => None, - ForkResult::Parent { child } => Some(child), - }, + let child = match unsafe { nix::unistd::fork() }? { + ForkResult::Child => None, + ForkResult::Parent { child } => Some(child), }; match child { None => { @@ -533,17 +433,15 @@ impl PreparedChild { } } Some(child) => { - if self.process_group && !self.session { + if self.process_group { // The child makes the same call; whichever runs first creates the group, so // that the caller may signal it at once. The child's `exec` or exit makes a - // failure here harmless. A child that starts a session leads the group that - // it creates, which the caller's call would forbid. + // failure here harmless. let _ = nix::unistd::setpgid(child, child); } Ok(Started { process: ProcessId::from_pid(child), report: self.report_reader, - barrier: self.barrier.take().map(|(_, writer)| writer), }) } } @@ -566,102 +464,36 @@ impl PreparedChild { ) -> ! { let report = &self.report_writer; if self.process_group - && !self.session && let Err(error) = nix::unistd::setpgid(Pid::from_raw(0), Pid::from_raw(0)) { // SAFETY: the caller establishes that this is the post-fork child. unsafe { fail(report, LaunchStage::ProcessGroup, error) } } - if self.session - && let Err(error) = nix::unistd::setsid() - { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Session, error) } - } - if let Some((reader, _)) = &self.barrier { - // Before it waits, the child closes every descriptor that it does not use, its copy - // of the barrier's writer among them. It then holds no copy of another child's pipes - // or launch report, which would stay open while it waits, and reads end of file once - // its caller shuts the writer down, or once no other copy of it remains. - if let Err(error) = - close_other_descriptors(&self.used).and_then(|()| wait_for_release(reader)) - { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Barrier, error) } - } - } if let Err(error) = self.place_descriptors() { // SAFETY: as above. unsafe { fail(report, LaunchStage::Descriptors, error) } } - if let Some(cgroup) = &self.cgroup - && let Err(error) = write_decimal(cgroup, nix::unistd::getpid().as_raw()) - { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Cgroup, error) } - } - if let Some(entry) = &self.entry - && let Err((stage, error)) = entry.enter_view() - { - // SAFETY: as above. - unsafe { fail(report, stage, error) } - } - if let Some(root) = &self.root - && let Err(error) = enter_root(root) - { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Root, error) } - } - if let Some(entry) = &self.entry - && let Err((stage, error)) = entry.take_identity() - { - // SAFETY: as above. - unsafe { fail(report, stage, error) } - } if let Some((uid, gid)) = self.workload && let Err((stage, error)) = self.take_workload(uid, gid) { // SAFETY: as above. unsafe { fail(report, stage, error) } } - if self.entry.is_some() || self.workload.is_some() { - // Close the stages' descriptors before resolving either working directory. - // Entries refuse magic links that could escape their root or bypass a view mount; - // plain workloads keep chdir's resolution under their final credentials. + if self.workload.is_some() { + // Close the stages' descriptors before resolving the working directory, with chdir's + // resolution under the workload's final credentials. let directory = self.directory.as_deref().unwrap_or(c"/"); if let Err(error) = - close_other_descriptors(&self.kept).and_then(|()| match self.entry.as_ref() { - Some(_) => super::entry::enter_directory(directory), - None => nix::unistd::chdir(directory), - }) + close_other_descriptors(&self.kept).and_then(|()| nix::unistd::chdir(directory)) { // SAFETY: as above. unsafe { fail(report, LaunchStage::WorkingDirectory, error) } } - } else if let Some(directory) = &self.directory { - match &self.credentials { - Some(credentials) => { - let saved = (nix::unistd::geteuid(), nix::unistd::getegid()); - if let Err(error) = self.take_credentials(credentials) { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Credentials, error) } - } - if let Err(error) = nix::unistd::chdir(directory.as_c_str()) { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::WorkingDirectory, error) } - } - if let Err(error) = self.restore_credentials(credentials, saved) { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::Credentials, error) } - } - } - None => { - if let Err(error) = nix::unistd::chdir(directory.as_c_str()) { - // SAFETY: as above. - unsafe { fail(report, LaunchStage::WorkingDirectory, error) } - } - } - } + } else if let Some(directory) = &self.directory + && let Err(error) = nix::unistd::chdir(directory.as_c_str()) + { + // SAFETY: as above. + unsafe { fail(report, LaunchStage::WorkingDirectory, error) } } if let Some(entry) = physical_pwd { // `nix`'s `getcwd` allocates its result, and glibc's walks the file system, which @@ -713,14 +545,14 @@ impl PreparedChild { super::entry::take_workload_identity(uid, gid) } - /// Places the standard streams and clears close-on-exec on the inherited descriptors. + /// Places the standard streams. /// /// Every source has a number above the standard streams', so `dup2` replaces each stream - /// without closing a source, and clears close-on-exec on its copy. The child of an entry - /// first closes every descriptor that it does not use, those that its caller opened without + /// without closing a source, and clears close-on-exec on its copy. A workload's child first + /// closes every descriptor that it does not use, those that its caller opened without /// close-on-exec too, so that its program holds none of the caller's. fn place_descriptors(&self) -> Result<(), Errno> { - if self.entry.is_some() || self.workload.is_some() { + if self.workload.is_some() { close_other_descriptors(&self.used)?; } let [stdin, stdout, stderr] = &self.streams; @@ -733,39 +565,8 @@ impl PreparedChild { if let Some(source) = stderr { retry(|| nix::unistd::dup2_stderr(source.borrowed()))?; } - for descriptor in &self.inherited { - nix::fcntl::fcntl(descriptor.borrowed(), FcntlArg::F_SETFD(FdFlag::empty()))?; - } Ok(()) } - - /// Takes the credentials of the working-directory check: no supplementary groups, then the - /// effective group, then the effective user. - fn take_credentials(&self, credentials: &Credentials) -> Result<(), Errno> { - #[cfg(test)] - if let Some(simulated) = self.simulated_credentials { - return simulated.map_or(Ok(()), Err); - } - nix::unistd::setgroups(&[])?; - nix::unistd::setegid(credentials.gid)?; - nix::unistd::seteuid(credentials.uid) - } - - /// Gives back the effective user, then the effective group and the supplementary groups, - /// that the child had before the check. - fn restore_credentials( - &self, - credentials: &Credentials, - (uid, gid): (Uid, Gid), - ) -> Result<(), Errno> { - #[cfg(test)] - if self.simulated_credentials.is_some() { - return Ok(()); - } - nix::unistd::seteuid(uid)?; - nix::unistd::setegid(gid)?; - nix::unistd::setgroups(&credentials.groups) - } } /// Resets an init workload's out-of-memory adjustment and nice value before it loses privilege. @@ -847,21 +648,6 @@ pub(crate) fn enter_root(root: &FileDescriptor) -> Result<(), Errno> { nix::unistd::chroot(c".") } -/// Waits until the caller releases the barrier whose reader is `reader`, which a byte tells. -/// End of file without a release, once the caller has shut the writer down, or once no copy of -/// the writer remains, is `ECANCELED`. -fn wait_for_release(reader: &FileDescriptor) -> Result<(), Errno> { - let mut byte = [0u8; 1]; - loop { - match reader.read(&mut byte) { - Ok(0) => return Err(Errno::ECANCELED), - Ok(_) => return Ok(()), - Err(Errno::EINTR) => {} - Err(error) => return Err(error), - } - } -} - /// Repeats `call` while it is interrupted. fn retry(mut call: impl FnMut() -> Result<(), Errno>) -> Result<(), Errno> { loop { diff --git a/nvx-rs/src/guest/sys/spawn/tests.rs b/nvx-rs/src/guest/sys/spawn/tests.rs index cc815fc9c..71ab84ede 100644 --- a/nvx-rs/src/guest/sys/spawn/tests.rs +++ b/nvx-rs/src/guest/sys/spawn/tests.rs @@ -20,10 +20,7 @@ pub(crate) const CAP_SETGID: u32 = 6; #[cfg(feature = "cli")] pub(crate) const CAP_SETUID: u32 = 7; -/// Capability to change the root directory. -pub(crate) const CAP_SYS_CHROOT: u32 = 18; - -/// Capability to mount, and to enter new namespaces, which an execution's entry needs. +/// Capability to mount, and to enter new namespaces. pub(crate) const CAP_SYS_ADMIN: u32 = 21; /// Capability to lower nice values. @@ -160,10 +157,6 @@ fn invalid_specs_are_refused_before_forking() { spec.directory = Some(c"relative".to_owned()); assert_eq!(PreparedChild::new(spec).err(), Some(Errno::EINVAL)); - let mut spec = shell("exit 0"); - spec.credentials = Some((1000, 1000)); - assert_eq!(PreparedChild::new(spec).err(), Some(Errno::EINVAL)); - let mut spec = shell("exit 0"); spec.physical_pwd = true; assert_eq!(PreparedChild::new(spec).err(), Some(Errno::EINVAL)); @@ -189,19 +182,10 @@ fn stream_sources_below_the_standard_streams_are_raised_and_kept_from_the_progra let mut spec = shell("printf placed; test ! -e /proc/self/fd/0"); spec.streams[1] = Some(FileDescriptor::from_owned(file.into())); let outcome = run(spec); - - // An inherited descriptor keeps its number, which must not be a standard stream's. - let file = std::fs::File::open(output.path()).unwrap(); - assert_eq!(file.as_raw_fd(), 0); - let mut spec = shell("exit 0"); - spec.inherited - .push(FileDescriptor::from_owned(OwnedFd::from(file))); - let refused = PreparedChild::new(spec).err(); drop(streams); assert_eq!(outcome, (None, Some(ChildStatus::Exited(0)))); assert_eq!(std::fs::read(output.path()).unwrap(), b"placed"); - assert_eq!(refused, Some(Errno::EINVAL)); } #[test] @@ -253,46 +237,6 @@ fn a_new_process_group_is_led_by_the_child() { } } -#[test] -fn a_child_enters_its_root_or_reports_why_not() { - let root = tempfile::tempdir().unwrap(); - std::fs::create_dir(root.path().join("inside")).unwrap(); - let mut spec = ChildSpec::new(c"/bin/sh".to_owned(), vec![c"sh".to_owned()]); - spec.root = Some( - sys::open( - &sys::path_c_string(root.path()).unwrap(), - sys::OpenFlags::O_RDONLY | sys::OpenFlags::O_DIRECTORY | sys::OpenFlags::O_CLOEXEC, - sys::Mode::empty(), - ) - .unwrap(), - ); - spec.directory = Some(c"/inside".to_owned()); - let (report, status) = run(spec); - assert_eq!(status, Some(ChildStatus::Exited(LAUNCH_FAILURE_STATUS))); - // Without the privilege to change its root, the child stops there. With it, the child - // finds `/inside` in the new root, which has no `/bin/sh`. - let (stage, error) = if holds_capability(CAP_SYS_CHROOT).unwrap() { - (LaunchStage::Exec, Errno::ENOENT) - } else { - (LaunchStage::Root, Errno::EPERM) - }; - assert_eq!(report, Some((stage.code(), error as i32))); -} - -#[test] -fn the_check_s_credentials_keep_the_parent_s_supplementary_groups() { - let mut spec = shell("exit 0"); - spec.directory = Some(c"/".to_owned()); - spec.credentials = Some((1000, 1001)); - let prepared = PreparedChild::new(spec).unwrap(); - let credentials = prepared.credentials.as_ref().unwrap(); - assert_eq!(credentials.groups, nix::unistd::getgroups().unwrap()); - assert_eq!( - (credentials.uid, credentials.gid), - (Uid::from_raw(1000), Gid::from_raw(1001)) - ); -} - /// Returns a spec that records the physical path of the working directory that the child /// inherits. fn inherited_physical_pwd() -> ChildSpec { @@ -376,11 +320,9 @@ fn a_working_directory_outside_the_root_has_no_physical_path() { #[test] fn the_launch_descriptors_are_close_on_exec() { let mut spec = shell("exit 0"); - spec.cgroup = Some(descriptor()); spec.streams[2] = Some(descriptor()); let prepared = PreparedChild::new(spec).unwrap(); for descriptor in [ - prepared.cgroup.as_ref().unwrap(), prepared.streams[2].as_ref().unwrap(), &prepared.report_reader, &prepared.report_writer, @@ -509,29 +451,3 @@ fn the_kernel_refuses_every_path_that_a_search_s_buffer_cannot_hold() { ); } } - -#[test] -fn an_entry_leads_a_session_and_keeps_its_barrier_from_the_program() { - if crate::guest::sys::tests::isolate_process_test( - "NVX_SPAWN_ENTRY_BARRIER_TEST_CHILD", - "guest::sys::spawn::tests::an_entry_leads_a_session_and_keeps_its_barrier_from_the_program", - ) { - return; - } - // With the standard streams closed, the barrier's ends take their numbers. - let streams = ClosedStreams::new(&[0, 1, 2]); - let mut spec = shell("exit 0"); - spec.barrier = true; - spec.entry = Some(EntryChild::default()); - let prepared = PreparedChild::new(spec); - drop(streams); - let prepared = prepared.unwrap(); - assert!(prepared.session && !prepared.process_group); - assert!(prepared.entry.is_some()); - let (reader, writer) = prepared.barrier.as_ref().unwrap(); - for descriptor in [reader, writer] { - assert!(descriptor.raw() >= FIRST_NONSTANDARD); - let flags = nix::fcntl::fcntl(descriptor.borrowed(), FcntlArg::F_GETFD).unwrap(); - assert!(FdFlag::from_bits_retain(flags).contains(FdFlag::FD_CLOEXEC)); - } -} diff --git a/nvx-rs/src/guest/sys/tests.rs b/nvx-rs/src/guest/sys/tests.rs index 4569f4bcb..a45c1ff6f 100644 --- a/nvx-rs/src/guest/sys/tests.rs +++ b/nvx-rs/src/guest/sys/tests.rs @@ -4,24 +4,26 @@ use std::io::{self, Write}; use super::*; -pub(crate) use super::entry::tests::Simulation; pub(crate) use super::entry::tests::seed_inheritable_and_ambient; -pub(crate) use super::fd::tests::{create_fifo, set_descriptor_limit}; +pub(crate) use super::fd::tests::create_fifo; #[cfg(target_arch = "x86_64")] pub(crate) use super::kernel::tests::nice_of; #[cfg(target_arch = "x86_64")] pub(crate) use super::port::tests::PortBackend; -pub(crate) use super::process::tests::{ - continue_traced, deny_system_call, fork_isolated_test, has_exited, isolate_process_test, - spawn_child_without_exit_signal, spawn_traced_child, -}; +#[cfg(feature = "cli")] +pub(crate) use super::process::signal_through_pidfd; +#[cfg(feature = "cli")] +pub(crate) use super::process::tests::has_exited; #[cfg(target_arch = "x86_64")] pub(crate) use super::process::tests::{deny_group_kill, exit_main_thread, forbid_system_calls}; +pub(crate) use super::process::tests::{ + deny_system_call, fork_isolated_test, isolate_process_test, +}; pub(crate) use super::rooted::tests::{ exchange_paths, restrict_new_entries, thread_umask, unshare_standard_descriptors, with_thread_umask, }; -pub(crate) use super::signal::tests::{default_sigpipe, has_child, reap_children_automatically}; +pub(crate) use super::signal::tests::{has_child, reap_children_automatically}; pub(crate) use super::sock_diag::tests::{encode_inet_diag_done, encode_inet_diag_socket}; #[cfg(feature = "cli")] pub(crate) use super::spawn::tests::{ diff --git a/nvx-rs/src/guest/time/api.rs b/nvx-rs/src/guest/time/api.rs index 801bc3111..5c4597466 100644 --- a/nvx-rs/src/guest/time/api.rs +++ b/nvx-rs/src/guest/time/api.rs @@ -44,8 +44,8 @@ const REPORT_ONLY_TOKEN: Token<'static> = Token { /// write access to `/run/nvx`. It takes the permission for the ports only for each of its port /// transactions, on the calling thread, and returns only once the thread holds none of it, or /// fails with [`TimeError::PortRelease`], after which the thread must start neither a thread -/// nor a process, which would inherit the permission; [`Command`](crate::guest::spawn::Command) -/// refuses to start one from it. No call waits for a child that it did not start. +/// nor a process, which would inherit the permission, as NVX's guest programs start none from +/// it. No call waits for a child that it did not start. /// /// A violation of the time ABI is reported by a violation event on the console, in the kernel /// log, and on OpenVMM's debug console. In enforcing mode, the event stops the VM, so the call @@ -1385,7 +1385,7 @@ pub enum TimeError { PortAccess(io::Error), /// The kernel did not revoke the calling thread's permission for OpenVMM's ports. The /// thread may still hold some of it, and must start neither a thread nor a process, which - /// [`Command`](crate::guest::spawn::Command) refuses to start from it. + /// would inherit it. PortRelease(io::Error), /// The lock that serializes the transactions of port B's window failed. PortLock(io::Error), diff --git a/nvx-rs/src/guest/view.rs b/nvx-rs/src/guest/view.rs index ad4503f1d..6c23e8702 100644 --- a/nvx-rs/src/guest/view.rs +++ b/nvx-rs/src/guest/view.rs @@ -10,8 +10,6 @@ use std::net::IpAddr; use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; use std::path::{Path, PathBuf}; -#[cfg(doc)] -use super::entry::Entry; use super::root::Root; use super::sys::{self, Mode, OpenFlags}; use crate::activation::{self, MachineIdMode, View}; @@ -42,9 +40,8 @@ pub enum MachineId { /// The image's own `/etc/machine-id`, left untouched. #[default] Image, - /// The ID in this regular file, bound read-only over `/etc/machine-id`, as - /// [`Entry::bind_runtime`] also does. Rewriting the file in place, such as after a restore, - /// reaches every execution that sees the bind. + /// The ID in this regular file, bound read-only over `/etc/machine-id`. Rewriting the file in + /// place, such as after a restore, reaches every execution that sees the bind. Generated(PathBuf), } diff --git a/nvx-rs/src/lib.rs b/nvx-rs/src/lib.rs index dfc83f4e3..f7d333324 100644 --- a/nvx-rs/src/lib.rs +++ b/nvx-rs/src/lib.rs @@ -33,13 +33,9 @@ //! manifests and compatibility keys of templates. //! - `guest` (feature `guest`, Linux): the programs that run in NVX guests, such as //! `nvx-console-pending`, the guest's NVXS connection, the libraries through which any guest -//! init or agent opens the terminal on which it serves its session (`guest::terminal`), starts -//! a program in a new process (`guest::spawn`), has the program enter an execution's -//! namespaces, view, root, and identity (`guest::entry`), collects the status of each child -//! that it names (`guest::process`), places, kills, and settles the processes of its -//! executions in cgroups (`guest::cgroup`), describes host mappings and keeps the account of a -//! host's mapping table (`guest::mappings`), and drives each program as one execution, with its -//! launch, bounded output, input, ending, and outcome (`guest::exec`), the typed calls through +//! init or agent opens the terminal on which it serves its session (`guest::terminal`), and +//! describes host mappings and keeps the account of a host's mapping table +//! (`guest::mappings`), the typed calls through //! which it runs the time ABI's steps (`guest::time`) and reseeds after a restore //! (`guest::reseed`) in its own process, //! and the capture of a template's guest, which repairs its clones (`guest::snapshot`), and diff --git a/scripts/nvx_tools/kernel_features.py b/scripts/nvx_tools/kernel_features.py index dc055fb72..c55c8ba53 100644 --- a/scripts/nvx_tools/kernel_features.py +++ b/scripts/nvx_tools/kernel_features.py @@ -100,19 +100,19 @@ class KernelFeature: rationale=( "A microVM has 1, 2, 4, or 8 virtual processors, whose CPU " "profiles serve Intel and AMD hosts alone, so the other vendors' " - "support stays off. Clone repair and the snapshot helper bring " - "processors online after a restore. The local APIC driver keeps " - "its x2APIC support, and PAT, on MTRRs, gives mappings their " - "memory types, as on every x86 kernel." + "support stays off. nvx-init's restore repair and the snapshot " + "helper bring processors online after a restore. The local APIC " + "driver keeps its x2APIC support, and PAT, on MTRRs, gives " + "mappings their memory types, as on every x86 kernel." ), consumers=( "every multiprocessor microVM", - "nvx::guest::clone::repair", + "nvx::guest::snapshot::capture", "guest/common/nvx-snapshot", ), validation=( "the --processors runs of the microVM scenarios", - "the runtime adapter's clone tests", + "the restore-processors and platform-restore-processors scenarios", ), settings=( "CONFIG_SMP=y", @@ -255,7 +255,7 @@ class KernelFeature: ), consumers=( "nvx-managed-agent", - "nvx-init-agent", + "nvx-init", "nvx-rs run", "nvx-rs sandbox", "the test-microvm and benchmark harnesses", @@ -340,17 +340,14 @@ class KernelFeature: ), KernelFeature( name="memory-hotplug", - summary="Memory blocks that clone repair adds through the probe interface", + summary="Memory blocks that restore repair adds through the probe interface", rationale=( - "After a restore, clone repair and the snapshot helper add the " - "memory blocks of the restored RAM through " + "After a restore, nvx-init's restore repair and the snapshot " + "helper add the memory blocks of the restored RAM through " "/sys/devices/system/memory/probe and online them themselves." ), - consumers=("nvx::guest::clone::repair", "guest/common/nvx-snapshot"), - validation=( - "the runtime adapter's clone tests", - "the snapshot scenarios with a RAM target", - ), + consumers=("nvx::guest::snapshot::capture", "guest/common/nvx-snapshot"), + validation=("the restore-memory scenario",), settings=( "CONFIG_MEMORY_HOTPLUG=y", "CONFIG_ARCH_MEMORY_PROBE=y", @@ -384,11 +381,14 @@ class KernelFeature: "(nvx::guest::gpt); no NVX image has an MBR partition table." ), consumers=( - "nvx-init-agent", + "nvx-init", "nvx::guest::root", - "nvx::guest::clone", + "nvx::guest::activation", + ), + validation=( + "the sandbox scenarios", + "the sandbox-gpt-layer and snapshot-tiers scenarios", ), - validation=("the sandbox scenarios", "the runtime adapter's GPT tests"), settings=( "CONFIG_BLOCK=y", "CONFIG_BLK_DEV=y", @@ -401,7 +401,7 @@ class KernelFeature: name="sandbox-filesystems", summary="EROFS layers under overlayfs, on an ext4 or tmpfs upper", rationale=( - "The layered sandbox root mounts EROFS layers, compressed with " + "nvx-init's layered root mounts EROFS layers, compressed with " "LZ4 or Zstandard, under overlayfs, whose upper is the ext4 " "scratch or a tmpfs (doc/design/4.1-guest-filesystem-and-image-model.md). " "Every layer keeps extended attributes, POSIX ACLs, and security " @@ -409,7 +409,7 @@ class KernelFeature: "never follows a redirect that it did not create, and the scratch " "template's metadata checksums use the optimized CRCs." ), - consumers=("nvx-init-agent", "nvx::guest::root", "generic sandbox workloads"), + consumers=("nvx-init", "nvx::guest::root", "generic sandbox workloads"), validation=("the sandbox scenarios", "the RAM-upper scenarios"), settings=( "CONFIG_EXT4_FS=y", @@ -431,27 +431,28 @@ class KernelFeature: ), KernelFeature( name="sandbox-isolation", - summary="The namespaces, cgroups, seccomp, and BPF of the sandbox", + summary="The namespaces, cgroups, seccomp, and BPF of programs in a guest", rationale=( - "The workload gets private mount, PID, and UTS namespaces, and " - "sibling memory and process-count cgroups whose freezing captures " - "it; cgroup writeback charges its dirty scratch pages to its " - "memory cgroup (doc/design/4.3-isolation-and-resource-model.md). " - "Generic sandbox workloads filter system calls with seccomp and " - "control devices with cgroup BPF programs, and unprivileged BPF " - "stays off. Without capabilities or user namespaces, workloads " - "create no namespaces of their own, so the other namespace and " - "cgroup controller types stay off." + "Since SL7.8b, SL7.8c, and SL6.11b (#1123) deleted the old layered " + "sandbox root, no NVX component creates a cgroup, or a namespace " + "other than a mount namespace: nvx-init and the managed agent run " + "their workloads in the namespaces that they run in, without " + "cgroups (doc/design/4.3-isolation-and-resource-model.md). Only " + "root programs in a guest, such as an nvx-rs run shell, can use " + "the PID and UTS namespaces, the memory, process-count, and block " + "cgroups with their writeback, and the cgroup BPF programs that " + "this feature keeps, and NAMESPACES also stays for NET_NS " + "(guest-internal-networking). Workloads, which hold no " + "capabilities, can filter their system calls with seccomp, and " + "unprivileged BPF stays off. Without capabilities or user " + "namespaces, workloads create no namespaces of their own, so the " + "other namespace and cgroup controller types stay off." ), consumers=( - "nvx-init-agent", - "nvx-container-launch", - "nvx::guest::cgroup", - "nvx::guest::entry", - "guest/common/nvx-snapshot", + "root programs in a guest, such as an nvx-rs run shell", "generic sandbox workloads", ), - validation=("the sandbox and managed scenarios", "the runtime adapter"), + validation=("build-kernel's sandbox group",), settings=( "CONFIG_NAMESPACES=y", "CONFIG_UTS_NS=y", @@ -526,7 +527,7 @@ class KernelFeature: "Live host shares and a managed sandbox's host export are " "virtio-fs mounts (doc/design/3.6-host-filesystem.md)." ), - consumers=("nvx-hostmount", "nvx-init-agent"), + consumers=("nvx-hostmount", "nvx-init", "nvx::guest::root"), validation=("the virtio-fs scenarios",), settings=("CONFIG_FUSE_FS=y", "CONFIG_VIRTIO_FS=y"), ), @@ -639,7 +640,7 @@ class KernelFeature: "and sgetmask calls and 16-bit segments, stay off." ), consumers=("generic sandbox workloads", "the guest agents"), - validation=("every microVM scenario", "the runtime adapter"), + validation=("every microVM scenario",), settings=( "CONFIG_MULTIUSER=y", "CONFIG_SYSVIPC=y", @@ -676,21 +677,15 @@ class KernelFeature: ), KernelFeature( name="diagnostics", - summary="Readable oopses and the sysrq trigger of the entry contract", - rationale=( - "Oopses and BUGs carry symbol and error names, which CI logs " - "need. The layered sandbox entry makes /proc/sysrq-trigger " - "read-only for user ID 0, which the runtime adapter's " - "vm_layered test checks." - ), - consumers=("CI logs", "nvx::guest::sys::entry"), - validation=("the runtime adapter's vm_layered test",), + summary="Readable oopses and BUGs", + rationale=("Oopses and BUGs carry symbol and error names, which CI logs need."), + consumers=("CI logs",), + validation=("build-kernel's kernel configuration checks",), settings=( "CONFIG_BUG=y", "CONFIG_DEBUG_BUGVERBOSE=y", "CONFIG_KALLSYMS=y", "CONFIG_SYMBOLIC_ERRNAME=y", - "CONFIG_MAGIC_SYSRQ=y", ), ), ) diff --git a/scripts/test-layout-allowlist.json b/scripts/test-layout-allowlist.json index ddb768d07..6ba886ddc 100644 --- a/scripts/test-layout-allowlist.json +++ b/scripts/test-layout-allowlist.json @@ -2,14 +2,10 @@ "schema": "nvx-test-layout-allowlist", "schema_version": 2, "seams": { - "nvx-rs/src/guest/cgroup.rs": 2, - "nvx-rs/src/guest/entry.rs": 1, - "nvx-rs/src/guest/exec.rs": 4, "nvx-rs/src/guest/init.rs": 7, "nvx-rs/src/guest/init/boot.rs": 1, "nvx-rs/src/guest/reseed.rs": 1, - "nvx-rs/src/guest/sys/entry.rs": 3, - "nvx-rs/src/guest/sys/spawn.rs": 7, + "nvx-rs/src/guest/sys/spawn.rs": 5, "nvx-rs/src/guest/time/system.rs": 3, "nvx-rs/src/host/network/address.rs": 2, "nvx-rs/src/host/platform/linux/mod.rs": 1, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 664881ade..581add13a 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -2395,7 +2395,7 @@ def test_readme_gives_each_module_a_short_row_and_a_section(self): for index, line in enumerate(lines) if line.startswith("### ") } - self.assertGreater(len(rows), 40) + self.assertGreater(len(rows), 30) for row in rows: link = row.split(" | ")[0].removeprefix("| ") with self.subTest(module=link):