Skip to content

SL2.4a1: the root's kernel tokens carry what VmConfig sets, and the guest caps a RAM upper #1278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into from
Nov 7, 2026
Merged

SL2.4a1: the root's kernel tokens carry what VmConfig sets, and the guest caps a RAM upper #1278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
75b863e
nvx-rs: carry the root's settings in its boot tokens, and cap a RAM u…
Nov 4, 2026
c016e2c
guest: validate a bound before refusing its GPT image
Nov 7, 2026
ba9b85f
Match missing-image diagnostics to root token order
Nov 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions doc/design/4.1-guest-filesystem-and-image-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@ stored and cached image bytes, because the guest kernel decompresses blocks as
it reads them. No FUSE daemon or in-guest fetch service is involved in the root
filesystem.

The launcher describes the root to the guest in two boot tokens, whose grammar
one contract module defines, with the host's renderer and a parser for guests:
The launcher describes the root to the guest in two boot tokens, and in up to
four more for its settings, whose grammar one contract module defines, with the
host's renderer and a parser for guests:

- `nvx_layer=distro,0xd0003000,FORMAT[:IDENTITY]` names the image on the
`distro` block device: its format, `erofs` for an EROFS filesystem or `gpt`
Expand All @@ -56,13 +57,23 @@ one contract module defines, with the host's renderer and a parser for guests:
- `nvx_scratch=0xd0006000,ext4` keeps the overlay's upper on the whole-device
ext4 `scratch` disk, and `nvx_overlay_upper=ramfs` keeps it in guest
memory, on a tmpfs, with no `scratch` block device.
- `nvx_overlay_mode=trusted-lowers` or `untrusted-lowers` names the overlay's
option set, `nvx_scratch_journal=allowed` or `forbidden` whether the
scratch's ext4 filesystem may have a journal, `nvx_first_partition_bound`
the most bytes of a GPT disk's first partition, and `nvx_ram_upper_percent`
the tmpfs upper's cap, from 1 to 90 percent of the guest's memory. A number
is decimal, without a sign or leading zeros. The guest takes its default for
a setting without a token: trusted lowers, a journal allowed, and the
kernel's default size of a tmpfs.

Guests refuse a token without a value, a repeated token, both uppers, an image
without an upper or an upper without an image, a RAM upper beside a `scratch`
block device, and every other form, including the earlier `nvx_layer` form
without a format, rather than probe the devices. The guest agents described
here assemble only an EROFS image, over either upper: they refuse a GPT image
with a message that says so.
without an upper or an upper or a setting without an image, a setting of
another upper or of another format, a RAM upper beside a `scratch` block
device, and every other form, including the earlier `nvx_layer` form without a
format, rather than probe the devices. The guest agents described here
assemble only an EROFS image, over either upper, for trusted lowers, and don't
check a scratch for a journal: they refuse a GPT image, untrusted lowers, and
a forbidden journal with a message that says so.

The guest agents assemble the root in this order:

Expand All @@ -77,8 +88,9 @@ The guest agents assemble the root in this order:
the token names one.
4. Mount the upper's file system, and create the overlay's upper and work
directories on it: resolve and flush scratch and mount it as ext4 with
`rw,nosuid,nodev`, or mount a tmpfs with `mode=0755,nosuid,nodev` at the
kernel's default size, half of the guest's RAM.
`rw,nosuid,nodev`, or mount a tmpfs with `mode=0755,nosuid,nodev`, and
`size=N%` when `nvx_ram_upper_percent=N` caps it, or else at the kernel's
default size, half of the guest's RAM.
5. Mount the overlay with the `distro` layer as its lower directory,
metadata-only copy-up, and extended inode numbers.
6. Verify the workload identity in the assembled root, mount each live share
Expand Down
100 changes: 81 additions & 19 deletions guest/common/nvx-init-agent
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,11 +178,11 @@ is_canonical_uuid() {
return 1
}

# Reads the value of the nvx_layer token for read_root_tokens:
# distro,0xd0003000,FORMAT[:IDENTITY], where this init accepts only the erofs
# format, with or without the filesystem UUID that the image must have.
# Reads the nvx_layer format and EROFS identity for read_root_tokens.
# This init assembles only EROFS; it refuses GPT after root-token validation.
read_layer_token() {
layer_image=${1#distro,0xd0003000,}
layer_format=${layer_image%%:*}
case "$1" in
distro,0xd0003000,*,*) ;;
distro,0xd0003000,erofs)
Expand All @@ -196,10 +196,9 @@ read_layer_token() {
return 0
;;
distro,0xd0003000,gpt | distro,0xd0003000,gpt:*)
fatal "nvx_layer=$1 names a GPT image, which this init cannot assemble: it assembles only an EROFS image"
return 0
;;
distro,0xd0003000,*)
layer_format=${layer_image%%:*}
# The form without a format names the UUID in its place.
if [ -z "$layer_format" ] || is_canonical_uuid "$layer_format"; then
fatal "the nvx_layer kernel parameter names no image format, erofs or gpt: '$1'"
Expand All @@ -212,24 +211,37 @@ read_layer_token() {

# Reads the boot tokens of the layered root, with the rules of the host's
# renderer, nvx::protocol::root: nvx_layer names the image of the distro block
# device, and nvx_scratch=0xd0006000,ext4 or nvx_overlay_upper=ramfs the upper,
# each once. This init assembles only an EROFS image, so it also refuses a GPT
# image. Sets layer_uuid to the filesystem UUID that the image must have, or to
# nothing if the token names none, and upper to scratch or ram. The parser also
# refuses a RAM upper beside a scratch block device, which this init checks
# against the kernel's devices instead, with refuse_scratch_device. The tests
# replay nvx-rs/tests/fixtures/root-tokens.json through it.
# device, nvx_scratch=0xd0006000,ext4 or nvx_overlay_upper=ramfs the upper, and
# nvx_overlay_mode, nvx_scratch_journal, nvx_first_partition_bound, and
# nvx_ram_upper_percent the root's settings, each once. This init assembles
# only an EROFS image, mounts the overlay only for trusted lowers, and reads
# nothing of a scratch's superblock, so after those rules it refuses a GPT
# image, untrusted lowers, and a forbidden journal too. A bound is checked as
# an unsigned decimal even though it needs a GPT image, which this init cannot
# assemble. Sets layer_uuid to the filesystem UUID that the image must have, or to
# nothing if the token names none, upper to scratch or ram, and ram_percent to
# the RAM upper's cap, or to nothing for the kernel's default size. The parser
# also refuses a RAM upper beside a scratch block device, which this init
# checks against the kernel's devices instead, with refuse_scratch_device. The
# tests replay nvx-rs/tests/fixtures/root-tokens.json through it.
read_root_tokens() {
layer_seen=false
scratch_seen=false
ram_seen=false
layer_uuid=
first_root_parameter=
# The names of the settings that the command line names, and the values of
# three of them.
settings=
overlay_mode=
scratch_journal=
ram_percent=
# Kernel command-line tokens are deliberately split on spaces.
# shellcheck disable=SC2086
set -f
for token in $cmdline; do
case "$token" in
nvx_layer | nvx_scratch | nvx_overlay_upper)
nvx_layer | nvx_scratch | nvx_overlay_upper | nvx_overlay_mode | nvx_scratch_journal | nvx_first_partition_bound | nvx_ram_upper_percent)
fatal "the $token kernel parameter has no value"
;;
nvx_layer=*)
Expand All @@ -239,24 +251,61 @@ read_root_tokens() {
read_layer_token "${token#*=}"
;;
nvx_scratch=*)
[ -n "$first_root_parameter" ] || first_root_parameter=nvx_scratch
[ "$scratch_seen" = false ] ||
fatal "the nvx_scratch kernel parameter is repeated"
scratch_seen=true
[ "${token#*=}" = 0xd0006000,ext4 ] ||
fatal "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not '${token#*=}'"
;;
nvx_overlay_upper=*)
[ -n "$first_root_parameter" ] || first_root_parameter=nvx_overlay_upper
[ "$ram_seen" = false ] ||
fatal "the nvx_overlay_upper kernel parameter is repeated"
ram_seen=true
[ "${token#*=}" = ramfs ] ||
fatal "the nvx_overlay_upper kernel parameter must be ramfs, not '${token#*=}'"
;;
nvx_overlay_mode=* | nvx_scratch_journal=* | nvx_first_partition_bound=* | nvx_ram_upper_percent=*)
setting=${token%%=*}
[ -n "$first_root_parameter" ] || first_root_parameter=$setting
case " $settings " in *" $setting "*) fatal "the $setting kernel parameter is repeated" ;; esac
settings="$settings $setting"
case "$token" in
nvx_overlay_mode=trusted-lowers | nvx_overlay_mode=untrusted-lowers) overlay_mode=${token#*=} ;;
nvx_overlay_mode=*) fatal "the $setting kernel parameter must be trusted-lowers or untrusted-lowers, not '${token#*=}'" ;;
nvx_scratch_journal=allowed | nvx_scratch_journal=forbidden) scratch_journal=${token#*=} ;;
nvx_scratch_journal=*) fatal "the $setting kernel parameter must be allowed or forbidden, not '${token#*=}'" ;;
nvx_ram_upper_percent=[1-9] | nvx_ram_upper_percent=[1-8][0-9] | nvx_ram_upper_percent=90) ram_percent=${token#*=} ;;
nvx_ram_upper_percent=*) fatal "the $setting kernel parameter must be a decimal number from 1 to 90, without a sign or leading zeros, not '${token#*=}'" ;;
nvx_first_partition_bound= | nvx_first_partition_bound=0* | nvx_first_partition_bound=*[!0-9]*) fatal "the $setting kernel parameter must be a decimal number from 1 to 18446744073709551615, without a sign or leading zeros, not '${token#*=}'" ;;
Comment thread
esaurez marked this conversation as resolved.
Resolved
nvx_first_partition_bound=*)
bound_digits=${token#*=}
[ "${#bound_digits}" -le 20 ] ||
fatal "the $setting kernel parameter must be a decimal number from 1 to 18446744073709551615, without a sign or leading zeros, not '${token#*=}'"
Comment thread
Copilot marked this conversation as resolved.
Resolved
if [ "${#bound_digits}" -eq 20 ]; then
bound_limit=18446744073709551615
# Compare digits, never the full unsigned value.
while [ -n "$bound_digits" ]; do
bound_digit=${bound_digits%"${bound_digits#?}"}
limit_digit=${bound_limit%"${bound_limit#?}"}
[ "$bound_digit" -le "$limit_digit" ] ||
fatal "the $setting kernel parameter must be a decimal number from 1 to 18446744073709551615, without a sign or leading zeros, not '${token#*=}'"
[ "$bound_digit" -eq "$limit_digit" ] || break
bound_digits=${bound_digits#?}
bound_limit=${bound_limit#?}
done
fi
;;
esac
;;
esac
done
set +f
[ "$scratch_seen" = false ] || [ "$ram_seen" = false ] ||
fatal "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper"
[ "$layer_seen" = true ] || [ -z "$first_root_parameter" ] ||
fatal "the $first_root_parameter kernel parameter needs an image, which nvx_layer names"
[ "$layer_seen" = true ] ||
fatal "the command line names no distro layer: this init needs nvx_layer=distro,0xd0003000,erofs[:UUID]"
if [ "$scratch_seen" = true ]; then
Expand All @@ -266,6 +315,19 @@ read_root_tokens() {
else
fatal "the nvx_layer kernel parameter needs an upper, nvx_scratch or nvx_overlay_upper"
fi
[ -z "$scratch_journal" ] || [ "$upper" = scratch ] ||
fatal "the nvx_scratch_journal kernel parameter needs a scratch upper, nvx_scratch"
[ -z "$ram_percent" ] || [ "$upper" = ram ] ||
fatal "the nvx_ram_upper_percent kernel parameter needs a RAM upper, nvx_overlay_upper"
if [ "$layer_format" != gpt ]; then
case " $settings " in *" nvx_first_partition_bound "*) fatal "the nvx_first_partition_bound kernel parameter needs a GPT image" ;; esac
fi
[ "$layer_format" != gpt ] ||
fatal "nvx_layer=distro,0xd0003000,$layer_image names a GPT image, which this init cannot assemble: it assembles only an EROFS image"
[ "$overlay_mode" != untrusted-lowers ] ||
fatal "nvx_overlay_mode=untrusted-lowers names untrusted lowers, which this init cannot assemble: it mounts the overlay only for trusted lowers, with metacopy=on,xino=on"
[ "$scratch_journal" != forbidden ] ||
fatal "nvx_scratch_journal=forbidden forbids the scratch a journal, which this init cannot check: it reads nothing of the scratch's superblock"
}

# Fails if the kernel has a virtio-mmio device at the scratch block device's
Expand All @@ -288,11 +350,11 @@ refuse_scratch_device() {
# Mounts the upper that read_root_tokens read, and creates the overlay's upper
# and work directories on it: the ext4 scratch disk at 0xd0006000, or a tmpfs,
# with the mode of an ext4 scratch's root rather than tmpfs's world-writable
# default, and with the kernel's default size, half of the guest's RAM, as
# nvx::guest::root mounts them. Sets upper_point to the upper's mount point,
# and upper_name to what messages call it. A tmpfs upper's pages are charged to
# the memory cgroup of the process that writes them, so the workload's writes
# count against its nvx_memory_max.
# default, and with the size of its cap, size=N%, or else the kernel's default
# size, half of the guest's RAM, as nvx::guest::root mounts them. Sets
# upper_point to the upper's mount point, and upper_name to what messages call
# it. A tmpfs upper's pages are charged to the memory cgroup of the process
# that writes them, so the workload's writes count against its nvx_memory_max.
mount_upper() {
case "$upper" in
scratch)
Expand All @@ -310,7 +372,7 @@ mount_upper() {
upper_point=$tmpfs
upper_name="the tmpfs upper"
mkdir -p "$upper_point"
mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs "$upper_point" ||
mount -t tmpfs -o "mode=0755,nosuid,nodev${ram_percent:+,size=$ram_percent%}" tmpfs "$upper_point" ||
fatal "failed to mount the tmpfs upper"
;;
esac
Expand Down
31 changes: 21 additions & 10 deletions nvx-rs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ changes only when the module's purpose does.
| [`nvx::protocol::export`](#nvxprotocolexport) | The boot tokens of the host export, the virtio-fs device from which a guest maps host paths, and the guest's parser of them |
| [`nvx::protocol::nvxs`](#nvxprotocolnvxs) | The NVXS session, independent of any application protocol: its records and the state machines of its host and guest roles |
| [`nvx::protocol::restore_ready`](#nvxprotocolrestore_ready) | The exact parser of OpenVMM's restore-ready event, including its terminating newline |
| [`nvx::protocol::root`](#nvxprotocolroot) | The boot tokens that tell the guest which image its layered root assembles and where the root keeps its upper |
| [`nvx::protocol::root`](#nvxprotocolroot) | The boot tokens that tell the guest which image its layered root assembles, where the root keeps its upper, and how the guest assembles them |
| [`nvx::protocol::share`](#nvxprotocolshare) | The guest targets that a sandbox's live shares may not take, which the host and the guest both refuse |
| [`nvx::protocol::status`](#nvxprotocolstatus) | The exit statuses with which a clone's guest ends its VM once it fails fast, and what each means |
| [`nvx::activation`](#nvxactivation) | The activation record of a layered root, which a clone reads in place of its captured configuration, and what Ready reports |
Expand Down Expand Up @@ -247,16 +247,25 @@ trailing bytes fail with `ProtocolError`.
The boot tokens of a layered root, which the host renders and guests parse by
the same rules: a `RootLayout` is a `RootImage` on the `distro` block device, an
EROFS filesystem or a GPT disk with the `nvx::uuid::Uuid` that the guest expects
of it, if any, over a `RootUpper`, the ext4 scratch disk or RAM.
of it, if any, over a `RootUpper`, the ext4 scratch disk or RAM, with the
settings of the root's assembly that it names: the overlay's `OverlayMode` and
a scratch's `Journal` policy, both of `nvx::activation`, a GPT disk's
first-partition bound, and a RAM upper's cap, from 1 to 90 percent of the VM's
memory. The guest takes its own default for a setting that the layout leaves
out.

`RootLayout::tokens` renders `nvx_layer=distro,0xd0003000,FORMAT[:IDENTITY]` and
`nvx_scratch=0xd0006000,ext4` or `nvx_overlay_upper=ramfs`, and
`nvx_scratch=0xd0006000,ext4` or `nvx_overlay_upper=ramfs`, then
`nvx_overlay_mode=trusted-lowers` or `untrusted-lowers`,
`nvx_scratch_journal=allowed` or `forbidden`, `nvx_first_partition_bound=BYTES`,
and `nvx_ram_upper_percent=PERCENT` for the settings that it names, and
`RootLayout::parse` reads them from a kernel command line, and refuses
malformed, repeated, and conflicting tokens and the earlier form without a
format, with no fallback that probes the devices. It reads the virtio-mmio
devices of the command line with `nvx::protocol::cmdline`, to refuse a RAM upper
beside one at the scratch base. `tests/fixtures/root-tokens.json` holds its
cases, which the tests of the guest's shell init replay too.
malformed, repeated, and conflicting tokens, a setting of an upper or a format
that the layout lacks, and the earlier form without a format, with no fallback
that probes the devices. It reads the virtio-mmio devices of the command line
with `nvx::protocol::cmdline`, to refuse a RAM upper beside one at the scratch
base. `tests/fixtures/root-tokens.json` holds its cases, which the tests of the
guest's shell init replay too.

### `nvx::protocol::share`

Expand Down Expand Up @@ -1708,8 +1717,10 @@ option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers
that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`,
with no fallback, whose upper and work directories are on the `Upper`: an ext4
scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that
the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the
kernel's default size. It refuses overlay options longer than
the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` with
`size=N%` for the caller's cap of N percent of the guest's memory, which it
refuses outside 1 to 90 before anything changes, or else at the kernel's
default size. It refuses overlay options longer than
`MOUNT_OPTIONS_LIMIT`, the most that a mount passes, which, below `/run/nvx`,
fit up to 195 layers of a GPT disk, and an overlay that the kernel's mount table
shows read-only or without the mode's options in effect, since the kernel mounts
Expand Down
Loading
Loading