usb: a console helper for headless Linux hosts (script, helper disk, console test)

[dormancygrace]

USB serial gave the managed computer a CDC ACM port, but the login on it was left to the administrator: a udev rule and a getty by hand, and docs/usb-composition.md said the Linux side was never tried. This adds a helper that sets the login up on the host, two ways to get it there (the NanoKVM's web server, and a read-only helper disk next to the virtual drive), a console test from the USB settings page, and the same helper wrapped for the Proxmox VE Helper-Scripts.

The script: nanokvm-serial-console.sh

POSIX sh (dash, busybox ash, bash), runs on the managed Linux host, about 560 lines, nothing beyond coreutils.

• plan (default) and status only read. They report the distribution (os-release), the init system (systemd, sysvinit or busybox init with /etc/inittab, runit, OpenRC), whether this is a container, an immutable or read-only system (OSTree, NixOS), the cdc_acm driver for the running kernel (loaded, built in, or a module file), the getty, and the NanoKVM on the USB bus (by 3346:1009, with its serial number and ttyACM name). Then they print what apply would change.
• apply asks for confirmation (--yes skips it; a non-terminal without --yes changes nothing) and:
  • on systemd hosts writes /etc/udev/rules.d/90-nanokvm-serial-console.rules, two rules bound to the vendor, product and serial number: on the USB device ENV{ID_MM_DEVICE_IGNORE}="1" (ModemManager stays away, old and new ModemManager), and on the tty ACTION=="add", SUBSYSTEM=="tty", KERNEL=="ttyACM[0-9]*", the same mark plus ID_MM_PORT_IGNORE, TAG+="systemd", ENV{SYSTEMD_WANTS}+="serial-getty@%k.service", SYMLINK+="nanokvm-console". The login starts for whichever ttyACMn the NanoKVM becomes, only while it is present. Then udevadm control --reload and, when the port is present, udevadm trigger --subsystem-match=tty --sysname-match=<port> --action=add (flags that old udevadm has too), and it checks serial-getty@<port> is active; when the trigger alone did not start it (systemd 247, Debian 11, after an undo), it starts the unit and checks again. restorecon on the rule when SELinux enforces.
  • on busybox init (Alpine) appends ttyACMn::respawn:getty … to /etc/inittab, on sysvinit nkac:2345:respawn:… (backup kept), and sends init SIGHUP.
  • Everything goes into a manifest with backups under /var/backups/nanokvm-serial-console/<timestamp>/. A second apply changes nothing and leaves no record. undo reverts the last change, reloads udev and stops the getty.
• Refuses (exit 2, or 3 for an unhandled system) and prints the manual steps: in a container (/.dockerenv, container= in PID 1's environment, cgroup), on OSTree/NixOS/read-only root, without cdc_acm for the running kernel (with the known reasons: Ubuntu cloud images without linux-modules-extra, Debian's cloud kernel), without serial-getty@.service or agetty/getty, and on runit or OpenRC as PID 1.
• When a login for this NanoKVM is already set up another way (a udev rule with serial-getty@ for its serial number, or a running serial-getty@<port> that the rule did not start), apply says so and adds no second rule.
• --serial SERIAL binds the rule when the NanoKVM is not plugged in yet (the USB page's command includes it); a connected NanoKVM wins over the option.
• The bootloader's console= is out of scope: early boot cannot go through CDC ACM anyway.

Getting it onto a headless host

• Over the network: GET /nanokvm-serial-console.sh on the NanoKVM, public (the host has no credentials, and the script is firmware source), with X-Checksum-Sha256. The USB page shows curl -fsS -k https://<nanokvm>/nanokvm-serial-console.sh -o nanokvm-serial-console.sh && sh nanokvm-serial-console.sh --serial <serial> (-k only when the page itself is HTTPS), the same for the NanoKVM's USB network address when the USB network is up (a host with no other network), and a wget form for busybox hosts.
• Without a network, the helper disk: with USB serial and the virtual disk enabled, S03usbdev adds a second LUN to mass_storage.disk0, read-only, removable, inquiry_string "NanoKVM Setup Disk", backed by /kvmapp/system/share/serial-console/nanokvm-serial-console.img: a 1440 KiB FAT12 image (volume label NANOKVM) with the script as setup.sh and a readme.txt. mount -o ro /dev/disk/by-label/NANOKVM /mnt && sh /mnt/setup.sh. The LUN is removed again when serial is off, so the drives match the composition. A LUN can only be added or removed while the function is not linked into the configuration (the kernel answers "Resource busy"), so the script settles it before the ln -s.
  • The image is built by scripts/build-serial-console-disk.py, in Python, byte-for-byte reproducible (fixed timestamps and volume id), no mtools needed on the build host; platform/build.sh tools builds it and packages.list ships both files under /kvmapp/system/share/serial-console/. Its boot sector carries the stub mkfs.fat writes ("This is not a bootable disk", wait for a key, next device), so a BIOS with a USB drive first in its boot order does not run zeros.
  • Why a disk and not a CD-ROM: auto-mounting is a desktop (udisks) matter and applies to both; a headless host mounts either by hand, /dev/disk/by-label/NANOKVM names both. A CD-ROM LUN would need an ISO 9660 image (FAT cannot be read through a 2048-byte-sector device); that can follow if /dev/sr0 naming is wanted.
  • Separate from the virtual drive: it does not replace a mounted image and costs no endpoints.
  • /etc/init.d/S03usbdev on a device is a symlink to the base package's copy (packages.list:46, /usr/libexec/nanokvm/legacy/S03usbdev), so the helper disk needs nanokvm-base rebuilt, not only the app package.
• Proxmox VE Helper-Scripts: firmware/services/community-scripts/nanokvm-serial-console.sh is a separate bash script for Proxmox VE and Proxmox Backup Server hosts only (Debian, systemd), built on that project's Core as their host tools are: it sources core.func, tools.func, error_handler.func and api.func from COMMUNITY_SCRIPTS_CORE_URL (so it needs that reachable, or a local copy), and uses load_functions, catch_errors, header_info (APP_TYPE=tools), root_check, on_pve_host, the prompt_* helpers for the menu, and plan|status|apply|undo [--yes] with NANOKVM_SERIAL for use without prompts. PVE 8.x/9.x and PBS on bookworm/trixie; pve_check is not called because it requires /etc/pve to be mounted. It writes the same rule file with the same content and records to the same backup directory as the POSIX helper, so a host set up by one is "already set up" for the other and undo works across them. Submitted as New Script: NanoKVM USB serial console (tools/pve) community-scripts/DevScripts#2414.

Server and web

• GET /api/vm/serial-console (admin): the gadget's USB serial number, the script's path, size and SHA-256, and whether the helper disk is shipped and attached (LUN 1 backed by it). When a package upgrade replaces the image while the LUN holds the old one, the kernel shows the path as deleted: reported as attached and stale, and the page says the computer sees the new disk once the USB devices reconnect (the next composition change or boot).
• POST /api/vm/serial-console/test (admin): sends one carriage return on /dev/ttyGS0, raw 8N1, and reads the answer for up to five seconds, as the browser terminal's first Enter does. Reports login, password, shell (the prompt ends the text), text, silent, busy (picocom's lock file, or any other process holding the port, found through /proc/*/fd), unavailable (no USB serial), error. ANSI sequences are stripped. Some gettys (seen on Arch and Ubuntu 26.04) reset the terminal and ask for the cursor position before the prompt, and wait for the answer: the probe replies as the browser terminal does, so the prompt can follow.
• USB settings page, while USB serial is enabled: the fetch commands (copyable), the SHA-256, the helper disk state (attached, pending after reconnect, or "enable the virtual disk as well"), the disk command, Test the console, and a download link. en and ru.
• docs/usb-composition.md describes the helper, and now says the terminal's 9600 baud is nominal (it said 115200 while the server always opens ttyGS at 9600). README.md points to the helper.

Checked

• firmware/services/test_serial_console.py (in CI): fake roots for Debian 12 and 8, Devuan (sysvinit), Rocky with SELinux, Alpine (busybox init with inittab), runit, unknown init, no os-release; every refusal (three container signals, OSTree, NixOS, read-only root, no cdc_acm with the Ubuntu and Debian cloud hints, no unit, no agetty); the foreign rule and the foreign getty; plan and status change no file and run no command, for every case, under dash, busybox sh and bash; apply writes the expected rules or inittab line and runs exactly udevadm control --reload, the targeted trigger and the is-active check, and starts the unit when the trigger did not; a second apply changes nothing; undo restores files and stops the getty; --serial and the connected device; a rule for another serial is replaced with a backup; --help and a bad argument. shellcheck clean.
• scripts/test-serial-console-disk.py (in CI): layout, both FATs, the root directory entries with lower-case names, reproducibility, and mtools (mdir, mtype) reading the files back.
• Go: classifySerialResponse (prompts, ANSI, the captured terminal setup); the probe against a pty with a fake getty (login prompt, a getty that waits for the cursor position reply, ANSI shell prompt, silent host within the timeout, missing device, a lock file, another process holding the port); the state and the handlers. go vet, go test -tags teststub ./....
• Web: tsc --noEmit, eslint, prettier, test:i18n. scripts/test-platform-build.py, test-platform-config.py.
• The Proxmox script: shellcheck at default severity with that project's disables; on the Proxmox host (NanoKVM passed through to Debian 13, 12 and 11 VMs, a stub pveversion, and once a stub proxmox-backup-manager for the PBS branch; Core from its main branch): status and plan without the device and with NANOKVM_SERIAL, apply without --yes and without a terminal changes nothing, apply --yes → nkvm-test-debianNN login: read on the NanoKVM, apply again (nothing to do), undo (inactive, port silent), apply after undo, undo again, undo with nothing left, --help, a bad argument. PBS is exercised through the stub only. Read-only status and plan on the real PVE 9.2 host, with a local Core copy, report the hand-made setup and change nothing.
• On the stand (Windows host): the script URL answers 200 with the checksum header; GET /api/vm/serial-console reports the serial d19506b31359c372; applying the console preset creates lun.1 with file = the image, ro=1, removable=1, inquiry "NanoKVM Setup Disk"; turning serial off removes the LUN and attached goes false, turning it on brings it back; the console test reports silent, right for a Windows host without a login on the COM port, and busy while another process holds /dev/ttyGS0.

Linux hosts end to end

A second NanoKVM (serial-only composition, NanoKVM OS 1.0.0-beta.11) on a Proxmox VE 9.2 host (pve-manager 9.2.21, kernel 7.0.14-19-pve, Debian 13.7), passed through (qm set --usb0 host=3346:1009, hot-plugged into running q35 VMs) to one VM per distribution, one at a time. For each VM the driver ran plan without the device and with it, apply, read the getty state, sent Enter from the NanoKVM side and captured what came back, ran apply again (nothing changed), undo (getty gone, port silent), apply again, undo again, then detached the device and checked the host's own getty was back on /dev/ttyACM0.

Host	Kernel	Init	apply → login prompt on the NanoKVM	undo → silent	apply again, undo again
Debian 13	6.12.111+deb13	systemd	yes	yes	yes
Debian 12	6.1.0-53	systemd	yes	yes	yes
Debian 11 (systemd 247, what PVE 7 runs)	5.10.0-46	systemd	yes	yes	yes: after undo the trigger alone did not restart the unit there, the script's explicit systemctl start did
Ubuntu 26.04	7.0.0-38-generic	systemd	terminal setup, prompt after the capture window (see below)	yes	yes
Ubuntu 24.04	6.8.0-146-generic	systemd	yes	yes	yes
Ubuntu 22.04	5.15.0-198-generic	systemd	yes	yes	yes
Fedora 42	6.14.0-63	systemd	yes	yes	yes
Rocky 9.8	5.14.0-687	systemd	yes	yes	yes
openSUSE Leap 15.6	6.4.0-150600.23.103	systemd	yes	yes	yes
Arch (2026-10)	7.2.9-arch1	systemd	terminal setup, prompt after the capture window (see below)	yes	yes
Alpine 3.21	6.12.112-0-lts	busybox init, inittab	yes (ttyACM0::respawn:/sbin/getty -L 115200 ttyACM0 vt220)	yes (inittab restored, getty gone)	yes (line appended again, getty running)

• On Arch and Ubuntu 26.04 the getty first sends a terminal reset and a cursor position query (CSI 6 n) and waits for the reply before the prompt; the passive capture ended before that. The console test now answers the query (commit c859e2b); the browser terminal always did.
• Refusals, as designed (exit 2, nothing written, the manual steps printed): an LXC container (Debian 13 template) on the same host; and every cloud image whose kernel has no cdc_acm: Debian genericcloud (cloud kernel), Ubuntu cloud without linux-modules-extra (on 26.04 the running kernel's package had already left the archive: the hint now says linux-generic and reboot), Alpine cloud linux-virt, openSUSE Minimal-VM kernel-default-base. The table rows are after the ordinary kernel was installed (Debian generic image; linux-modules-extra/linux-generic; linux-lts; kernel-default), which is what an installed host runs.
• Read-only on the Proxmox host itself, where the console was set up by hand before this script existed: status and plan report "already set up by /etc/udev/rules.d/70-nanokvm-serial-console.rules" and would change nothing. No apply was run there.

Not covered

• The web terminal on ttyGS0 fails on current builds with picocom: executable file not found: scripts/build-picocom.sh exists but nothing builds or packages it, although the docs say it is bundled. Found while checking the busy state; terminal: bridge serial ports in the server, without picocom #78 replaces picocom with a native session.
• The optional resize bridge (docs/USB-SERIAL.md) is not installed by the script.
• platform/expected.sha256 lists the two tools/serial-console/ outputs (the script byte for byte, the reproducible FAT image); the web assets still need the platform build, as for any web change.
• The cursor position reply in the console test is verified against the captured sequence in the pty test; a live Arch or Ubuntu 26.04 host with this server build is still to run.
• Devuan and Linux Mint have no cloud images; Devuan's sysvinit path is covered by the fake-root test only, Mint by Ubuntu.

🤖 Generated with Claude Code

[dormancygrace]

Review of 41ab2ae8aafec682e81b4400821c618521a3dd9f against current main 1a4723e5.

Two fixes are needed before merging:

1. [P2] Acquire exclusive ownership for the whole console probe. In server/service/vm/serial_console.go:167–193, the busy check is only an observation, and deviceInUse deliberately skips this process. Two requests to /vm/serial-console/test therefore both open/read the same tty and save/restore its termios. Reproduced with the existing fakeHost(t, ""): start a 600 ms probe, wait for received() (the first CR), then run a second 200 ms probe. The second returns silent, not busy. They can consume each other's responses and restore stale tty settings. Use an atomic per-device lease covering open, read/write and termios restoration; share it with terminal sessions when terminal: bridge serial ports in the server, without picocom #78 is integrated. Test overlapping probes and both probe/terminal start orders.

2. [P2] Register the new build outputs in the checksum manifest. platform/build.sh:579–586 now creates tools/serial-console/nanokvm-serial-console.sh and tools/serial-console/nanokvm-serial-console.img, but neither is in platform/expected.sha256. verify() explicitly rejects every unlisted output (lines 982–989), so even after accounting for intended changed binaries, the documented build verification cannot succeed. Generate both artifacts reproducibly and add their verified hashes; do not weaken the verifier.

Integration note: current main also introduces a content conflict in .github/workflows/ci.yml. Keep both the new serial-helper tests and main's platform-library tests when resolving it. Confirmed with git merge-tree --write-tree 1a4723e5 HEAD.

Validation: all 18 host-helper tests and all 5 FAT-image tests pass. The existing targeted Go serial tests pass; the added overlapping-probe regression fails as described. Tests ran on local PTYs/fake roots; the physical NanoKVM was not touched.

[dormancygrace]

Both fixed, and main merged in (dc2d44e, the ci.yml conflict resolved with both test lists: mmf-light-link from main, serial-console-disk and test_serial_console.py from here).

1. Lease for the whole probe — 3d61c10. New serial_lease.go holds the per-device registry (claimSerialPort / serialPortOpen); probeSerialConsole claims it before anything else and releases it after the device is closed and the termios restored, so an overlapping probe reports busy instead of reading the first one's answer. terminal: bridge serial ports in the server, without picocom #78's terminal session will use the same registry, which gives the exclusion in both start orders; that part lands there. Regression: TestProbeSerialConsoleHoldsTheDeviceForItsWholeRun (600 ms probe on a silent host, a 200 ms probe started after the first CR is busy, the lease is gone when the first ends). go test -race on the probe tests is clean.

2. Manifest — 9b0c91f. tools/serial-console/nanokvm-serial-console.sh (the source file byte for byte) and nanokvm-serial-console.img (from scripts/build-serial-console-disk.py, two builds give the same bytes) are in platform/expected.sha256 with their hashes. The web assets still need the platform build, as for any web change; the verifier is untouched.

Tests after the merge: the 18 helper tests, the 5 FAT-image tests, test-platform-build.py, test-platform-config.py, go vet, go test -tags teststub ./service/vm/, shellcheck. PTYs and fake roots only; the devices were not touched.

[dormancygrace]

Re-review complete on 81414b234c22fe8b7e5417adf1ac38622b7311f9. The previous findings are fixed. The shared probe lease test passes under -race; all 18 helper and 5 FAT-image tests pass, and both generated output hashes match expected.sha256. I merged current main and resolved the USB-menu conflict by retaining SerialConsole plus #77's conditional MouseJiggler. TypeScript and all 7 USB preset tests pass. No device changes. Waiting for CI on the integrated head before merging, as authorized.