Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions nvx-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1587,6 +1587,14 @@ mount whose attachment fails and which cannot be detached keeps its mount
descriptor, so rollback and teardown can retry cleanup without trusting its
original path.

`Root::switch`: the switch into the root, which an init makes once for its VM:
it moves the root's mount onto `/`, then makes the root the caller's root
directory with `chroot`, and its working directory with `chdir`. Once the mount
is on `/`, the root has switched for good: a second switch and the teardown are
refused, and dropping the root unmounts nothing, while descriptors opened before
the switch, such as a `TargetRoot` and a `SourceRoot`, still map. A move that
fails leaves the root where it was.

`Root::teardown`: the shares and mappings in reverse order, a runtime mount whose
attachment failed and which could not be detached, then the mounts of an
aggregate, of the export, and of the runtime, the later first, the overlay,
Expand All @@ -1596,9 +1604,10 @@ ext4 scratch's file system is synced before its unmount, since a mount namespace
that copied the root's mounts keeps them, unsynced, after this one unmounts
them, and a scratch whose sync fails stays mounted.

Each failure is a typed `Error`. The operations collect no child's status,
change neither the caller's root nor its credentials, start no async runtime,
and never power the VM off.
Each failure is a typed `Error`, or a `SwitchError` for the switch. The
operations collect no child's status, change the caller's root only by the
switch and never its credentials, start no async runtime, and never power the VM
off.

### `nvx::guest::gpt`

Expand Down Expand Up @@ -2868,11 +2877,13 @@ NVX_BUNDLE=skip cargo test --locked --all-features --lib -- \
--ignored --exact guest::root::tests::a_thread_with_private_tables_finds_its_mount_points_and_options
```

Ignored tests of `Root::mount_runtime` mount the runtime's file systems on a
temporary tmpfs root, in a thread's private mount namespace, and check their
options, device nodes, and time status, a call's undo, the teardown, and the
order of the runtime and the mappings beneath `/run`. They need root and Linux
6.8 or later; run them as root with:
Ignored tests of `Root::mount_runtime` and `Root::switch` mount the runtime's
file systems on a temporary tmpfs root, in a thread's private mount namespace,
and check their options, device nodes, and time status, a call's undo, the
teardown, and the order of the runtime and the mappings beneath `/run`. They
also switch into the root from a thread with root and working directories of
its own, and check the mounts that stay and a mapping through descriptors opened
before the switch. They need root and Linux 6.8 or later; run them as root with:

```bash
NVX_BUNDLE=skip cargo test --locked --all-features --lib -- \
Expand Down
127 changes: 116 additions & 11 deletions nvx-rs/src/guest/root.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@
//! as an init does once for its VM, before the root's first live share or mapping: it makes
//! every mount of the root's tree `nosuid` and `nodev`, then mounts `/proc`, `/sys`, `/dev` with
//! `/dev/pts` and `/dev/shm`, and `/run`, and binds the guest's time status at `/run/nvx/time`.
//! - `Root::switch`, which is crate-private, switches into the root, as an init does once for its
//! VM: it moves the root's mount onto `/`, then makes the root the caller's root directory and
//! working directory. The root's mounts stay from then on: its teardown refuses, and dropping it
//! unmounts nothing.
//! - [`Root::teardown`] unmounts, in order, the live shares and mappings in reverse order, the
//! mounts of an aggregate live share, of the host export, and of the runtime, the later first,
//! the overlay, the layers from the top down, and the upper, whose file system it syncs first
Expand All @@ -36,8 +40,9 @@
//!
//! A failure is an [`Error`] that names what failed. The operations need the guest's root user,
//! in the guest's initial mount and cgroup namespaces. They never collect a child's status,
//! change the caller's root directory or credentials, start an async runtime, or power the VM
//! off, so a caller that is not PID 1 keeps its own reaping, transport, and shutdown.
//! change the caller's credentials, start an async runtime, or power the VM off, and only
//! `Root::switch` changes the caller's root directory, so a caller that is not PID 1 keeps its own
//! reaping, transport, and shutdown.
//!
//! # Runtime directory
//!
Expand Down Expand Up @@ -626,6 +631,7 @@ fn assemble_with(
mapper: None,
mapped: Vec::new(),
latched: false,
switched: false,
reservation: None,
};
match root.mount_layers(paths, assembly, deadline, contents) {
Expand Down Expand Up @@ -679,8 +685,9 @@ impl Lower {
/// those of its live shares, host export, and mappings, and reports what the assembly mounted.
///
/// [`teardown`](Self::teardown) unmounts them and reports each failure. Dropping the root
/// unmounts them the same way, and ignores failures. Until then, the root holds the lock of its
/// runtime directory, where no other root assembles.
/// unmounts them the same way, and ignores failures, unless `Root::switch` has switched into the
/// root: it then unmounts nothing. Until then, the root holds the lock of its runtime directory,
/// where no other root assembles.
pub struct Root {
/// The kernel that mounts and unmounts.
kernel: Box<dyn Kernel>,
Expand Down Expand Up @@ -716,13 +723,17 @@ pub struct Root {
/// Whether a share, the export, or a mapping failed, after which the root refuses every later
/// one.
latched: bool,
/// Whether the root's mount has moved onto `/`, after which the root refuses another switch
/// and its teardown, and unmounts nothing.
switched: bool,
/// The exclusive lock of the runtime directory, from the start of the assembly; dropping the
/// root releases it after the unmounts.
reservation: Option<sys::FileLock>,
}

impl Root {
/// Returns the root: the overlay's mount point, `rootfs` in the runtime directory.
/// Returns the root's original overlay mount point, `rootfs` in the runtime directory.
/// This path is unchanged by `Root::switch`; paths then resolve inside the switched root.
pub fn path(&self) -> &Path {
&self.path
}
Expand Down Expand Up @@ -781,6 +792,9 @@ impl Root {
/// goes, which may be after the VM powers off. So a teardown that succeeds leaves every write
/// to the root on the scratch device. A scratch whose sync fails is not unmounted, and the
/// failure is reported as its own.
///
/// Once `Root::switch` has switched into the root, the teardown refuses: it unmounts nothing,
/// and its error's `TeardownError::is_switched` is true.
pub fn teardown(mut self) -> Result<(), TeardownError> {
self.unmount()
}
Expand Down Expand Up @@ -938,6 +952,44 @@ impl Root {
Ok(())
}

/// Switches into the root, as an init does once for its VM, after the mounts and the
/// descriptors that it needs: it moves the root's mount onto `/`, as an execution's child
/// moves its root in its own mount namespace, then makes the root the calling process's root
/// directory, with `chroot`, and its working directory, with `chdir`. The move is in the
/// caller's mount namespace, and the root and working directories are those that the caller's
/// threads share, but for a thread that unshared them.
///
/// A root switches once, for good: a second switch is refused as [`SwitchError::Switched`].
/// Once its mount is on `/`, the root has switched, even if entering it then fails: from then
/// on its [`teardown`](Self::teardown) refuses and dropping it unmounts nothing, so its mounts
/// stay until the VM powers off. A move that fails leaves the root where it was, unswitched.
///
/// Paths resolve in the root from then on, but descriptors opened before the switch stay
/// usable: a [`Mapper`] still maps from a [`SourceRoot`](super::mappings::SourceRoot) beneath
/// a [`TargetRoot`] opened before it, as [`map`](Self::map) does if the root had mounted a
/// live share or a mapping before it.
#[cfg_attr(not(test), expect(dead_code, reason = "nvx-init calls it (#1115)"))]
pub(crate) fn switch(&mut self) -> Result<(), SwitchError> {
if self.switched {
return Err(SwitchError::Switched);
}
let moved = |error: Errno| SwitchError::Move {
error: error.into(),
};
// The root is entered through its descriptor, so a move that fails changes nothing.
let directory = share::open_mount_point(&self.path).map_err(moved)?;
let root = sys::path_c_string(&self.path).map_err(moved)?;
self.kernel
.mount(Some(root.as_c_str()), c"/", None, MountFlags::MS_MOVE, None)
.map_err(moved)?;
self.switched = true;
self.kernel
.enter_root(&directory)
.map_err(|error| SwitchError::Enter {
error: error.into(),
})
}

/// Implements [`assemble`]'s steps after the first, waiting for each device until `deadline`
/// at the latest, if any, and accepting the upper's `contents`.
fn mount_layers(
Expand Down Expand Up @@ -1278,8 +1330,14 @@ impl Root {
}

/// Unmounts what the root has mounted, as [`teardown`](Self::teardown) describes, and
/// forgets it, failed unmounts included.
/// forgets it, failed unmounts included, unless the root has switched.
fn unmount(&mut self) -> Result<(), TeardownError> {
if self.switched {
// An empty list of failures is a switched root's refusal.
return Err(TeardownError {
failures: Vec::new(),
});
}
let mut failures = Vec::new();
// The mapper's descriptor of the root keeps the overlay busy, so it goes first.
if let Some(mut mapper) = self.mapper.take() {
Expand Down Expand Up @@ -1350,7 +1408,8 @@ impl Root {
}

impl Drop for Root {
/// Unmounts what the root has mounted, as [`Root::teardown`] does, and ignores failures.
/// Unmounts what the root has mounted, as [`Root::teardown`] does, and ignores failures. A
/// root that has switched unmounts nothing.
fn drop(&mut self) {
let _ = self.unmount();
}
Expand All @@ -1375,6 +1434,7 @@ impl fmt::Debug for Root {
.field("runtime_stray", &self.runtime_stray)
.field("mapped", &self.mapped)
.field("latched", &self.latched)
.field("switched", &self.switched)
.finish_non_exhaustive()
}
}
Expand Down Expand Up @@ -2145,24 +2205,69 @@ impl fmt::Display for AssemblyError {

impl std::error::Error for AssemblyError {}

/// The syncs and unmounts of a teardown that failed, in the order in which they were attempted.
/// Why [`Root::switch`] failed. Its text names the step that failed, and the system's error, if
/// any.
#[derive(Debug)]
pub(crate) enum SwitchError {
/// The root has switched already, and a root switches once.
Switched,
/// Opening the root or moving its mount onto `/` failed: the root stays where it was, and
/// has not switched.
Move {
/// The system's error.
error: io::Error,
},
/// Making the root the caller's root directory and working directory failed once its mount
/// was on `/`: the root has switched.
Enter {
/// The system's error.
error: io::Error,
},
}

impl fmt::Display for SwitchError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Switched => formatter.write_str("the root has switched already"),
Self::Move { error } => write!(formatter, "failed to move the root onto /: {error}"),
Self::Enter { error } => write!(
formatter,
"failed to enter the root, whose mount is on / already: {error}"
),
}
}
}

impl std::error::Error for SwitchError {}

/// The syncs and unmounts of a teardown that failed, in the order in which they were attempted,
/// or the refusal of a root that has switched.
#[derive(Debug)]
pub struct TeardownError {
/// The failures, which is never empty.
/// The failures, which is empty only for the refusal of a root that has switched.
failures: Vec<Unmount>,
}

impl TeardownError {
/// Returns the failures, each of a mount that stays in place, in the order in which they were
/// attempted.
/// attempted: none if the root had switched.
pub fn failures(&self) -> &[Unmount] {
&self.failures
}

/// Returns whether the teardown was refused because the root had
/// [switched](Root::switch), so that it unmounted nothing.
pub(crate) fn is_switched(&self) -> bool {
self.failures.is_empty()
}
}

impl fmt::Display for TeardownError {
/// Writes each failure, separated by `; `.
/// Writes each failure, separated by `; `, or the refusal.
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
if self.is_switched() {
return formatter.write_str("the root has switched, so its mounts stay");
}
for (index, failure) in self.failures.iter().enumerate() {
if index > 0 {
formatter.write_str("; ")?;
Expand Down
17 changes: 14 additions & 3 deletions nvx-rs/src/guest/root/kernel.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
//! or a read of a GPT disk's partition table at the geometry that its block device reports, and
//! the writes to the kernel's control files, the check for a mount point, and the options of a
//! mount, whose effects and answers a test simulates, go through a [`Kernel`], which also performs
//! the mapper's operations for the root's live shares and mappings, and the runtime's mounts.
//! the mapper's operations for the root's live shares and mappings, the runtime's mounts, and the
//! switch into the root.

use std::ffi::CStr;
use std::fs::{self, File};
Expand All @@ -31,8 +32,9 @@ const MOUNT_TABLE: &str = "/proc/thread-self/mountinfo";
/// disk's partition table, writes of a control file, and the sleeps between two polls for a block
/// device; the check for a mount point and the options of a mount, which depend on them; since
/// only a block device reports its geometry, the read of a GPT disk's partition table; the
/// mapper's operations, through which the root mounts its live shares and mappings; and the new
/// mounts, the restriction of the root's tree, and the device nodes of the runtime's file systems.
/// mapper's operations, through which the root mounts its live shares and mappings; the new
/// mounts, the restriction of the root's tree, and the device nodes of the runtime's file systems;
/// and the change of root directory into a switched root.
///
/// [`SystemKernel`] performs them. The tests' fake records the operations, simulates the effect
/// of a cgroup's membership file and its own mounts, reads a GPT disk's table from a regular
Expand Down Expand Up @@ -101,6 +103,10 @@ pub(super) trait Kernel: MappingKernel + Send + Sync {
/// Creates the device nodes and the symbolic links of an execution's `/dev` in the directory
/// `dev`, the nodes with mode 0666 whatever the umask.
fn create_devices(&mut self, dev: &FileDescriptor) -> Result<(), Errno>;

/// Makes the directory `root` the calling process's root directory, then its working
/// directory the new root's top: `chroot`, then `chdir`.
fn enter_root(&mut self, root: &FileDescriptor) -> Result<(), Errno>;
}

impl Kernel for SystemKernel {
Expand Down Expand Up @@ -196,6 +202,11 @@ impl Kernel for SystemKernel {
fn create_devices(&mut self, dev: &FileDescriptor) -> Result<(), Errno> {
sys::create_devices(dev)
}

fn enter_root(&mut self, root: &FileDescriptor) -> Result<(), Errno> {
sys::enter_root(root)?;
nix::unistd::chdir(c"/")
}
}

/// Returns the options of the file system of the mount whose ID is `mount`, the last field of its
Expand Down
12 changes: 12 additions & 0 deletions nvx-rs/src/guest/root/kernel/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ pub(crate) enum Operation {
Options,
/// A new mount of a file system, which its type names.
NewMount,
/// An entry into a directory as the root directory and the working directory, which the
/// directory names.
EnterRoot,
}

/// One operation that the fake kernel recorded, with its paths and options as text.
Expand Down Expand Up @@ -133,6 +136,9 @@ pub(crate) enum Call {
RestrictTree(String),
/// A creation of the device nodes and the links of a `/dev`, which its directory names.
CreateDevices(String),
/// An entry into a directory as the root directory and the working directory, which the
/// directory names.
EnterRoot(String),
}

/// The type of the action that runs after each sleep.
Expand Down Expand Up @@ -733,6 +739,12 @@ impl Kernel for FakeKernel {
let call = Call::CreateDevices(path.display().to_string());
self.record(call, Operation::Create, &path)
}

fn enter_root(&mut self, root: &FileDescriptor) -> Result<(), Errno> {
let path = path_of(root)?;
let call = Call::EnterRoot(path.display().to_string());
self.record(call, Operation::EnterRoot, &path)
}
}

impl MappingKernel for FakeKernel {
Expand Down
1 change: 1 addition & 0 deletions nvx-rs/src/guest/root/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ use crate::guest::sys::MountAttributes;

mod privileged;
mod runtime;
mod switch;

/// The base address of the distro device's MMIO window.
const DISTRO: u64 = 0xd000_0000;
Expand Down
Loading
Loading