Bring the Rust detector set to parity with the browser library (14 detectors)

[Zenofex]

What and why

The browser detector library (frontend/src/lib/detectors.ts in the website repo) is the source of truth for the detector set: it is what the web tool and the legacy Node analyzer both run, and cli/sync-detectors.mjs keeps those two in step. This crate shipped 9 of its 14 detectors, and folded one of the rest into the wrong label — so the same log produced different JSON depending on which implementation read it.

This brings the Rust set to 14, in the browser's order, with the browser's labels.

$ bootintel version
bootintel 0.4.2
  client-side detectors: 14 (Bootloader, Runtime firmware, ROM identifier, Firmware SDK,
  Kernel, CPU / Arch, Userland, Flash layout, Init system, Device family, Network,
  Web admin, Telnet exposure, Autoboot interruptable)

New detectors

Label	What it reads	Corpus coverage
Runtime firmware	OpenSBI, the RISC-V M-mode runtime	1 log
ROM identifier	Espressif mask-ROM build stamp (ESP-ROM:esp32-20160718)	0 logs (unit-tested)
Firmware SDK	ESP-IDF version off the 2nd-stage bootloader banner	0 logs (unit-tested)
Userland	BusyBox version	7 logs
Flash layout	MTD partition map — N partitions + named list with sizes	15 logs

⚠️ Behaviour change for JSON consumers

OpenSBI x.y moves from Bootloader to Runtime firmware. Per the versioning policy at the top of CHANGELOG.md, an output-schema change is MAJOR. The version is deliberately not bumped in this PR — releasing is the coordinator's job.

Folding OpenSBI into Bootloader did not just mislabel it, it hid it: on a board that prints both, U-Boot won the precedence chain and the OpenSBI version never reached the output at all.

RISC-V / OpenSBI — samples/bootintel-6.txt (StarFive VisionFive 2)

Before:

  Bootloader                U-Boot 2021.10
                              Nov 10 2022 - 13:29:36 +0800
  Kernel                    Linux 5.15.0-starfive
                              riscv64-unk2OF: fdt: Ignoring memory range 0x40000000 - 0x40200000           M
  CPU / Arch                RISC-V
  Init system               systemd

  4 findings identified locally (client-side detectors only).

After:

  Bootloader                U-Boot 2021.10
                              Nov 10 2022 - 13:29:36 +0800
  Runtime firmware          OpenSBI 1.0
  Kernel                    Linux 5.15.0-starfive
  CPU / Arch                RISC-V
  Init system               systemd

  5 findings identified locally (client-side detectors only).

Two changes visible here: Runtime firmware: OpenSBI 1.0 appears (the browser has always reported it), and the Kernel detail stops being garbage stitched out of two unrelated lines — see Divergences found below.

ESP — an ESP32 capture (ESP-ROM: + ESP-IDF banner; no ESP log is in the public corpus)

Before:

  Bootloader                Espressif ROM bootloader
                              ESP8266/ESP32
  CPU / Arch                Xtensa (ESP)
  Device family             Espressif (ESP)

  3 findings identified locally (client-side detectors only).

After:

  Bootloader                Espressif ROM bootloader
                              ESP8266/ESP32
  ROM identifier            Espressif ROM esp32-20160718
  Firmware SDK              ESP-IDF v5.1.2
  CPU / Arch                Xtensa (ESP)
  Device family             Espressif (ESP)

  5 findings identified locally (client-side detectors only).

Purely additive for ESP: Bootloader: Espressif ROM bootloader is unchanged (the browser keeps that branch too), and the ROM stamp and SDK version arrive as their own labels. CPU / Arch: Xtensa (ESP) here comes from the literal xtensa in the log, not from the esp32 string — see below.

Contract details this had to get right

1. source / line_number are derived, not set by the detector. analyze() now re-runs each detector against each line on its own and takes the first line that reproduces the same value + detail, which is the browser's algorithm. The old substring search over normalized lines could attach evidence for a match that exists on no single line. evidence_is_the_original_line asserts, over the whole corpus, that every source is verbatim the input line at line_number.
2. Flash layout is an aggregate. Its value counts partitions across many lines, so no single-line re-run reproduces it and it correctly ends up with no source / line_number — the evidence is the whole table. (The degenerate one-partition case does locate a line, exactly as in the browser.)
3. Partitions are de-duplicated on offsets. samples/bootintel-9.txt prints its table twice: 14 raw regex matches, 7 real partitions. Verified: grep -c finds 14 matches, 7 unique, and the finding reports 7 partitions.
4. Order and labels match the browser exactly, asserted by detector_labels_stable.
5. Normalization applies to the 12 inventory detectors and not to the 2 policy ones (Telnet exposure, Autoboot interruptable), which read the original lines — the browser's exemption, now mirrored. The parity test compares every log twice, once behind [12:34:56.123] , so this path is actually exercised rather than assumed.

The parity test

crates/cli/tests/browser_parity.rs + crates/cli/tests/browser_detectors.mjs: the harness transpiles detectors.ts in-process and evaluates it in a vm (the way cli/test-parity.mjs does, resolving typescript from the website checkout's own node_modules), prints every finding as JSON, and the Rust test diffs the {label, value, detail} sequence against analyze() — 31 logs × 2 shapes = 62 comparisons.

It skips loudly when node or the website checkout is absent, because the crate is published and cargo test has to pass for someone who has only this repo. BOOTINTEL_REQUIRE_PARITY=1 turns a skip into a failure; BOOTINTEL_BROWSER_DETECTORS=/path/to/detectors.ts points it at a checkout elsewhere. Run here with the real website checkout present — no skips.

Divergences it found (all fixed here)

Each of these made the CLI assert something the web tool did not:

• Kernel detail stitched across lines. The build-metadata group was mandatory and \s-separated, so a banner with no toolchain parentheses matched across line breaks: bootintel-6.txt produced riscv64-unk2OF: fdt: Ignoring memory range 0x40000000 …, bootintel-17.txt produced gcc version 4.6.3 (Ti2CPU: ARMv7 Processor [412fc09a] …. Group is now optional and line-bounded. No log loses its Kernel finding.
• CPU / Arch: Xtensa (ESP) claimed from esp32 / esp8266. Those name a device family, not a CPU (and Device family already reports it).
• Init system: BusyBox init. A BusyBox banner says what the userland is, not what PID 1 is. bootintel-17.txt claimed Init system: BusyBox init and now reports Userland: BusyBox 1.20.2.
• Init system: systemd inferred from Welcome to … Linux. A distro greeting is not evidence of PID 1; systemd[1]: is.
• Partition sizes rounded the wrong way on a tie. JS toFixed rounds ties away from zero, Rust's {:.1} rounds to even, and partition tables hit exact ties routinely: the 1280 KiB kernel region in bootintel-12.txt is 1.3M in the browser and was 1.2M here. Tenths are now computed in integer arithmetic.

Verified by running

Check	Result
bootintel version	client-side detectors: 14
bootintel detectors	14 registered, each with a description
cargo test --workspace	195 + 57 + 26 + 13 + 2 + 2 pass, 0 fail
cargo test --workspace --features tui	213 + 57 + 26 + 13 + 2 + 2 pass, 0 fail
cargo test --test browser_parity -- --nocapture	62 comparisons, no [skip] lines
BusyBox coverage	fires on all 7 corpus logs that print the banner (asserted)
Partition map coverage	fires on all 15 corpus logs that register one (asserted)
Duplicate partitions	none on any corpus log (asserted)
cargo clippy --workspace --all-targets -- -D warnings	clean (also with --features tui)
cargo fmt --all --check	clean
cargo deny check	advisories ok, bans ok, licenses ok, sources ok
bootintel bench samples/bootintel-24.txt (481 KB)	9.3ms → 10.6ms median, debug build

That last row is the cost of the per-line evidence re-run plus five more detectors: ~14% on the largest capture in the corpus, ~1ms for the 40 KiB log the pacer's budget is written around. Worth it for the contract; say the word if you'd rather have a fast path.

Deliberately not done

• No version bump, no release. Per the brief; the entry under ## [Unreleased] states that the label move is an output-schema change and therefore MAJOR under the policy at the top of the file.
• No ESP log added to samples/. That directory mirrors the 31 public corpus logs, and a synthetic capture in it would desync the mirror and the count assertions. ESP coverage is in crates/detectors/tests/detector_tests.rs instead (ROM identifier, Firmware SDK, positive + negative each).
• captureProblem() is still not ported. The browser returns zero findings for a log with a corrupt or conflicting version banner; this crate has no equivalent, so the two could still disagree on such a log. No corpus log trips it, so the parity test passes, but it is a real remaining gap — separate change, since it adds a refusal path with its own exit-code question.

🤖 Generated with Claude Code