Skip to content

host: locate private files and directories without reading them on musl (O_PATH) - #1522

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
feature/nvx-refactorfrom
agents/slim-musl-opath
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
feature/nvx-refactorfrom
agents/slim-musl-opath

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1076 (slice musl-opath: the O_PATH follow-up that SL5.1c, #1500, recorded).

What and why

SL5.1c (#1500) found that the musl build of nvx-rs, which CI and the releases ship, could restore no named snapshot. It fixed that in c19dc62 ("host: open the restore-ready socket with O_PATH on musl too"). The cause: std's OpenOptionsExt::custom_flags masks out the bits of O_ACCMODE, and musl's O_ACCMODE is 0o10000003, which includes O_PATH; glibc's is 0o3. So on musl, custom_flags(O_PATH | …) silently drops O_PATH, and the open becomes an ordinary read open. #1500 named three more such sites as a follow-up. SL6.1 (#1520) has since deleted one of them, Directory::search. This PR fixes the other two, and a test helper with the same bug.

The production sites. restore_private_file_mode and open_plain_dir, in nvx-rs/src/host/platform/linux/private.rs, open their path only to locate it. On musl, they opened it for reading instead. That needs the permission to read it, and waits for a writer of a FIFO:

  • create_store_dir, with which the CLI creates its store, failed with EACCES under a umask that withholds the owner's permission to read the new directory, though its docs say that it sets the mode again for that case. check_store_dir and check_snapshot_dir failed the same way on a directory that its owner may not read.
  • A lock file of the CLI's store that its owner may neither read nor write never got its mode 0600 back: open_lock failed with EACCES.
  • A FIFO with read permission in place of a lock file blocked restore_private_file_mode until a writer came.

The fix. Both sites open with nix::fcntl::open, which passes the flags on as they are, as c19dc62 did, through a private helper, open_to_locate(path, flags), which opens with O_PATH | O_CLOEXEC | flags. They keep their error handling. Nothing changes on glibc: std adds O_CLOEXEC to every open, and O_RDONLY is 0. std retries an open on EINTR and nix doesn't, but NVX's one signal handler, SIGINT's, has SA_RESTART.

The test helper. open_metadata_only in nvx-rs/src/host/seal/tests.rs, with which tests open a file for its metadata alone, had the same bug. So on musl, the tests that check that no read goes through such a file, a_seal_needs_no_access_to_the_content and the_staged_check_reads_no_content, failed. It now opens with nix::fcntl::open too.

No other site. No other custom_flags call in nvx-rs/src, host or guest, on any platform, passes O_PATH or another bit of musl's O_ACCMODE (O_WRONLY, O_RDWR), including those of the helpers that SL6.0d (#1518) moved into host::platform. The others pass O_NONBLOCK, O_NOFOLLOW or O_DIRECTORY, or Windows' FILE_FLAG_* values.

Tests. Nothing caught this because CI runs the unit tests only on glibc; a CI run of them on musl stays the follow-up that #1500 recorded. The existing test lock_files_get_back_the_permissions_that_the_umask_cleared fails on musl without this PR. It is the regression test of restore_private_file_mode without the permission to read the file, so this PR adds no duplicate of it. It adds tests, in host::platform::linux::tests, only for what no test covered:

  • a_store_directory_gets_mode_700_even_where_the_umask_withholds_reading_it: open_plain_dir, through create_store_dir under a umask of 0477.
  • a_fifo_in_place_of_a_private_file_is_left_as_it_is_without_waiting_for_a_writer: restore_private_file_mode of a FIFO, behind cli as that function is. The call runs in a thread of its own, so that a regression fails after 10 s instead of hanging.

Commits.

  1. host: locate private files and directories with O_PATH on musl too: the fix and the two tests.
  2. host: open a file for its metadata alone with O_PATH on musl too: the test helper.

Size. 13 core lines in 1 counted file, private.rs, against an estimate of about 30 lines in 2 or 3 files. The public API doesn't change, since the helper is private. Neither does the case matrix: no test that it lists or names changes, and it doesn't inventory unit tests.

Validation

At the head 3b698a76cb3, on the base 00c55b965f0.

On Windows:

  • cargo fmt --all --check: passed.
  • cargo clippy --locked --all-targets --all-features -- -D warnings and cargo clippy --locked -p nvx --lib --no-default-features -- -D warnings: passed. The changed code is Linux-only, so this shows only that Windows still builds.
  • python scripts/check_test_layout.py: passed.
  • python scripts/check_test_changes.py --base origin/feature/nvx-refactor: one change to existing test code (see Test changes) and three additions (the two tests and a helper, permission_bits).
  • SL5.2's case-matrix check, check_matrix.py, against 99d2022085f (the last revision with the adapter's inventory): passed.

On Linux: nvx-kvm-test (KVM, virtual machine, kernel 6.6) and nvx-mshv-test (MSHV, virtual machine, kernel 6.18), Rust 1.95.0, the glibc target and x86_64-unknown-linux-musl, as an unprivileged user, since root would pass the permission checks.

The two new tests and the three existing ones that fail on musl at the base, with cargo test --locked --lib --all-features [--target x86_64-unknown-linux-musl] -- --exact <the five>:

Host Target Base's production code, this PR's tests This PR
nvx-kvm-test glibc 5 passed 5 passed
nvx-kvm-test musl 5 failed, the FIFO test after its 10 s timeout 5 passed
nvx-mshv-test glibc 5 passed 5 passed
nvx-mshv-test musl 5 failed, likewise 5 passed

Also at the head, on nvx-kvm-test:

  • cargo test --locked --lib --all-features, glibc and musl: the same 28 guest:: tests fail on both targets, which fail on that host's kernel 6.6 without this PR too. Before the rebase, on 07c508c74a5, the whole suite's failures were the same tests at the base and the head on glibc on both hosts. On musl, they were the same as glibc's at the head, and the base failed only these sites' tests more. nvx-mshv-test passed all of its tests on both targets.
  • cargo test --locked --lib, with the default features and so without cli: builds without warnings in the changed files, and the store-directory test passes.
  • every clippy command of check-nvx-rs's Lint step, with -D warnings: no diagnostics. Likewise for musl, cargo clippy with --all-targets --all-features, --lib, --lib --bins --features cli and --lib --no-default-features.

CI: nvx-rs / ubuntu-latest and NVX CLI / Linux fail on this head exactly as on the base's own push run, 43212226280 at 00c55b965f0. Both ran on the ubuntu-26.04 image, where ubuntu-latest now points: guest::exec::tests::a_stalled_sink_stalls_only_its_own_execution_and_loses_nothing fails its assertion at src/guest/exec/tests.rs:318, and test_linux_runners_terminate_openvmm_left_by_earlier_jobs fails because that image's multi-call coreutils refuses to run under another name (coreutils: unknown program 'nvx-leak-…'). The previous base run, 43208645731 at 671a192d0a5, passed both on ubuntu-24.04.

Left to CI: the Windows unit tests, as no Windows code changes; the guest binaries' checks, as no guest code changes; and the microVM scenarios. On glibc, nothing changes. On musl, the opens now match glibc's, and the scenarios, which run the musl build, reach these paths only with permissions that both opens allow.

Test changes

check_test_changes.py lists one change to existing test code, hence tests-modified. The second commit changes the helper open_metadata_only in host::seal::tests to open with nix::fcntl::open in place of custom_flags(O_PATH), so that it opens a file for its metadata alone on musl too. No test's assertions change.

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 1 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified helper fn open_metadata_only in host::seal::tests nvx-rs/src/host/seal/tests.rs:14 nvx-rs/src/host/seal/tests.rs:14

Relocated unchanged: 0; added: 3. Compared the merge base 35af82af1562 with a31ef77ef7f4.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the Linux metadata-only open update.

No open issues 🎉

What changed in this PR

This PR completes the musl O_PATH follow-up from #1500 within the #1076 workstream. It ensures Linux metadata-only opens preserve O_PATH, avoiding unnecessary read-permission checks and blocking FIFO opens without changing the public API.

Changes:

  • Introduces a shared nix::fcntl::open helper for three production path-lookup sites.
  • Fixes the metadata-only test helper to preserve O_PATH on musl.
  • Adds regression tests for restrictive directory permissions and nonblocking FIFO handling.
File Description
nvx-rs/​src/​host/​seal/​tests.rs Uses nix to open metadata-only test files with O_PATH.
nvx-rs/​src/​host/​platform/​linux/​tests.rs Adds restrictive-umask and FIFO regression tests.
nvx-rs/​src/​host/​platform/​linux/​private.rs Adds the shared path-lookup helper and updates file-mode and directory operations.
nvx-rs/​src/​host/​platform/​linux/​identity.rs Routes directory search through the shared helper.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the metadata-only open changes.

No open issues 🎉

Review effort: Balanced

restore_private_file_mode and open_plain_dir open their path only to
locate it, with O_PATH, through OpenOptionsExt::custom_flags, which
masks out the bits of O_ACCMODE. On musl, O_ACCMODE is 0o10000003 and
includes O_PATH, so the musl build of nvx-rs, which CI and the releases
ship, opened the file or directory for reading instead, which needs the
permission to read it and waits for a writer of a FIFO. A store
directory created under a umask that withholds its owner's permission
to read it failed with EACCES instead of getting its mode 0700, a lock
file that its owner may neither read nor write never got its mode back,
and a FIFO in place of a lock file blocked the restore of its mode
until a writer came.

Open them with nix::fcntl::open, which passes the flags on as they are,
through a helper, open_to_locate, as c19dc62 did for the
restore-ready socket (#1500). Nothing changes on glibc, whose O_ACCMODE
is 0o3.

On nvx-kvm-test (Linux 6.6) and nvx-mshv-test (Linux 6.18), a musl
build without this fails the existing test
lock_files_get_back_the_permissions_that_the_umask_cleared, of
restore_private_file_mode, and two new tests in host::platform::linux,
of open_plain_dir under such a umask and of a FIFO, which waits for a
writer until the test's timeout. With this, the three pass on musl;
they pass on glibc either way. CI runs the unit tests only on glibc.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
open_metadata_only, the helper with which tests open a file for its
metadata alone, opened it with O_PATH through
OpenOptionsExt::custom_flags, which masks out the bits of O_ACCMODE. On
musl, O_ACCMODE includes O_PATH, so a musl build opened the file for
reading, and the tests that check that no read goes through such a
file, a_seal_needs_no_access_to_the_content and
the_staged_check_reads_no_content, failed. Open it with
nix::fcntl::open, as the host's own opens with O_PATH do.

On nvx-kvm-test and nvx-mshv-test, the two tests fail on musl without
this and pass with it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) merged commit 9bb42b2 into feature/nvx-refactor Nov 12, 2026
45 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/slim-musl-opath branch November 12, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants