Bring the Rust detector set to parity with the browser library (14 detectors)

CHANGELOG.md

@@ -8,6 +8,119 @@ All notable changes to bootintel-cli are documented here. Format follows [Keep a
 
 ## [Unreleased]
 
+Detector-set parity with the browser detector library at
+bootintel.com/tools/fingerprint, which is the source of truth for the
+set: it is what the web tool and the legacy Node analyzer both run. This
+crate shipped 9 of its 14 detectors and folded two others into the wrong
+label, so the same log produced different JSON depending on which
+implementation read it.
+
+**This is an output-schema change, which the policy at the top of this
+file calls MAJOR.** One finding moves to a different label, one moves out
+of a label it never belonged in, and five new labels appear (all three
+below), so anything parsing `.findings[]` by label needs a look before
+upgrading. The version is deliberately not bumped here —
+releasing is handled separately, and the 0.4.0 entry records how this
+project numbers a breaking change while pre-1.0.
+
+### Added
+- Five detectors, bringing the set to 14 and `bootintel version` to
+  `client-side detectors: 14`:
+  - **Runtime firmware** — OpenSBI, the RISC-V M-mode runtime.
+  - **ROM identifier** — the Espressif mask-ROM build stamp
+    (`ESP-ROM:esp32-20160718`), which is what a ROM-level exploit is
+    written against.
+  - **Firmware SDK** — ESP-IDF version off the 2nd-stage bootloader
+    banner. On an ESP target this is the CVE-relevant identifier; the ROM
+    stamp rarely moves, the SDK does.
+  - **Userland** — BusyBox version. It is printed by 7 of the 31 public
+    corpus logs and this crate reported it zero times, so the offline
+    story was missing the most common userland component in the category.
+  - **Flash layout** — the MTD partition map (15 of the 31 corpus logs
+    register one), as `N partitions` plus the named list with sizes. This
+    is what a flash-clip read needs in order to know where to read.
+- `cargo test -p bootintel --test browser_parity` — runs the browser
+  library over all 31 corpus logs (transpiled in-process and evaluated in
+  a `vm`, the same way the website repo's own parity test does) and
+  asserts the `{label, value, detail}` sequence matches this crate's, log
+  for log. Each log is compared twice, as captured and with
+  `[12:34:56.123] ` in front of every line, because line-prefix
+  normalization is the one place the two could agree on a bare log and
+  still disagree on a real capture. It skips loudly when `node` or the website checkout is absent,
+  since the crate is published and `cargo test` has to pass for someone
+  who only has this repo; `BOOTINTEL_REQUIRE_PARITY=1` turns a skip into
+  a failure and `BOOTINTEL_BROWSER_DETECTORS` points at `detectors.ts`.
+- Corpus coverage guards: BusyBox must fire on every corpus log that
+  prints its banner (7), the partition map on every log that registers
+  one (15), and no log may report the same partition twice.
+
+### Changed
+- **`OpenSBI x.y` moves from `Bootloader` to `Runtime firmware`.** It is
+  the M-mode runtime that hands off to U-Boot, not a bootloader, and the
+  browser has always reported it under its own label. Folding it into
+  `Bootloader` also *hid* it: on a RISC-V board that prints both (sample
+  `bootintel-6.txt`), U-Boot won the precedence chain and the OpenSBI
+  version disappeared from CLI output entirely. It is now reported
+  alongside the bootloader.
+- **An ESP log gains `ROM identifier` and `Firmware SDK` findings.** The
+  `Bootloader: Espressif ROM bootloader` finding is unchanged — the ROM
+  build stamp and the SDK version are additional labels, not a rename.
+- Consumers parsing JSON by label: the two bullets above are the whole
+  behaviour change. `Bootloader` for U-Boot / coreboot / GRUB / ESP,
+  `Kernel`, `CPU / Arch`, `Init system`, `Device family`, `Network`,
+  `Web admin`, `Telnet exposure` and `Autoboot interruptable` keep their
+  labels, values and relative order; the five new labels are interleaved
+  in the browser's registration order (`Bootloader`, `Runtime firmware`,
+  `ROM identifier`, `Firmware SDK`, `Kernel`, `CPU / Arch`, `Userland`,
+  `Flash layout`, `Init system`, …) because order is part of what makes
+  the two implementations comparable.
+
+### Fixed
+- Four divergences from the browser found by the new parity test, each of
+  which made the CLI assert something the web tool did not:
+  - **Garbled `Kernel` detail.** The build-metadata group was mandatory
+    and `\s`-separated, so on a kernel banner that prints no toolchain
+    parentheses the match ran across line breaks and swept up unrelated
+    text: `bootintel-6.txt` reported a detail of
+    `riscv64-unk2OF: fdt: Ignoring memory range 0x40000000 …` and
+    `bootintel-17.txt` one of
+    `gcc version 4.6.3 (Ti2CPU: ARMv7 Processor [412fc09a] revision 10 …`,
+    each stitched out of two different lines. The group is now optional
+    and line-bounded, so a banner with no toolchain parentheses yields a
+    version and no detail. No log loses its `Kernel` finding.
+  - **`CPU / Arch: Xtensa (ESP)` claimed from `esp32` / `esp8266`.**
+    Those name a device family, not a CPU, and are reported as such by
+    `Device family`. Only the literal arch name counts now.
+  - **`Init system: BusyBox init`.** A BusyBox banner says what the
+    userland is, not what PID 1 is. The claim is replaced by the new
+    `Userland` finding — `bootintel-17.txt` reported
+    `Init system: BusyBox init` and now reports
+    `Userland: BusyBox 1.20.2`.
+  - **`Init system: systemd` inferred from a `Welcome to … Linux`
+    greeting.** A distro greeting is not evidence of PID 1;
+    `systemd[1]:` is.
+- `source` and `line_number` are now derived exactly the way the browser
+  derives them — the detector is re-run against each line on its own and
+  the first line reproducing the same `value` + `detail` is the evidence.
+  The old substring search over the normalized lines could attach a
+  `source` for a match that exists on no single line. An aggregate
+  finding (`Flash layout`, whose value counts partitions across the whole
+  table) correctly carries neither field.
+- The two policy detectors (`Telnet exposure`, `Autoboot interruptable`)
+  read the original lines rather than the normalized ones, as the browser
+  does. Stripping timestamps and ANSI is a presentation choice for the
+  inventory detectors; a policy rule should see what the capture
+  contained.
+- Partition offsets are de-duplicated, so a capture that prints the table
+  twice (a reset loop, or two flash devices registering) does not inflate
+  the count — `bootintel-9.txt` reports its real 7 partitions rather
+  than 14.
+- Partition sizes round the way JavaScript's `toFixed` does (ties away
+  from zero) rather than the way Rust's formatter does (ties to even).
+  Exact ties are common in a partition table: the 1280 KiB `kernel`
+  region in `bootintel-12.txt` is `1.3M` in the browser and was `1.2M`
+  here.
+
 ## [0.4.2] — 2026-09-26 — published as `bootintel`
 
 ### Changed

CONTRIBUTING.md

@@ -119,13 +119,14 @@ Component names loosely mirror the tree — `cli`, `detectors`, `tui`, `api`, `d
 
 ## Adding a detector
 
-New detectors live in `crates/detectors/src/lib.rs`.
+New detectors live in `crates/detectors/src/lib.rs`, but the browser detector library at bootintel.com/tools/fingerprint is the source of truth for the set — it is what the web tool and the legacy Node analyzer run. Add the detector there first, then port it here.
 
 1. Add the regex as a `static LazyLock<Regex>`.
-2. Add a `fn detect_thing(&str) -> Option<Finding>` that runs the regex and returns a `Finding`.
-3. Wire it into `analyze()` alongside the existing detectors.
-4. Add unit tests: at least one positive case and one negative-case log excerpt that shouldn't match.
-5. If the detector should sync with the browser detector library at bootintel.com/tools/fingerprint, coordinate with a maintainer — the browser + CLI detector sets should stay label-aligned.
+2. Add a `fn run_thing(&str) -> Option<Finding>` that runs the regex and returns a `Finding`. Do not set `source` from anything but the matched text, and never set `line_number` — `analyze()` derives both by re-running the detector per line, so a hand-set `source` breaks the original-evidence contract.
+3. Wire it into `ALL_DETECTORS` at the same position its browser counterpart occupies. Order and labels are part of the output contract.
+4. Add a description to `cmd::detectors::DESCRIPTIONS` (a missing one fails a test) and update `detector_labels_stable`.
+5. Add unit tests: at least one positive case and one negative-case log excerpt that shouldn't match.
+6. Run `cargo test -p bootintel --test browser_parity` against a checkout of the website repo (`BOOTINTEL_BROWSER_DETECTORS=/path/to/frontend/src/lib/detectors.ts`, `BOOTINTEL_REQUIRE_PARITY=1` so it cannot silently skip). It runs the browser library over all 31 corpus logs and diffs the findings against this crate's.
 
 ## Adding a subcommand
 

Cargo.toml

@@ -10,7 +10,7 @@
 #     `lz-str` (for /decode-compatible share URLs).
 #
 # Ships `bootintel scan / share / ports / version / term / analyze`
-# with the 9-detector streaming subset + json/text/sarif/junit output.
+# with the 14-detector set + json/text/sarif/junit output.
 # Client-side identification is offline + free; full CVE matching runs
 # server-side via bootintel.com's API (opt-in via --api).
 

README.md

@@ -307,7 +307,7 @@ Distribution scope deliberately excludes: no auto-update (attack surface), no te
 Cargo.toml                       # workspace manifest
 crates/
 ├── detectors/                   # pure client-side detector library
-│   ├── src/lib.rs               # 9 detectors
+│   ├── src/lib.rs               # 14 detectors
 │   └── tests/detector_tests.rs  # unit tests
 └── cli/                         # the `bootintel` binary
     ├── src/
@@ -392,6 +392,8 @@ Two fields were added and no existing key changed name or meaning:
 | `analysis_status` | envelope | `matched` or `unrecognized` — pairs with exit codes 0 / 3 |
 | `line_number` | each finding | 1-based line of `source` in the analyzed log; omitted when unknown |
 
+Both are **derived** from the log, not set by the detector: the detector is re-run against each line on its own and the first line that reproduces the same `value` + `detail` is the evidence. An aggregate finding whose value spans many lines — `Flash layout` counts partitions across the whole MTD table — therefore carries neither field, because no single line is its source.
+
 `source` now carries the **original** log line, prefix and all, rather
 than only the substring the detector regex matched. Prefixed captures
 (`[12:34:56.789] `, ISO-8601 timestamps, ANSI colour) are normalized
@@ -404,7 +406,7 @@ SARIF v2.1.0 and JUnit XML outputs conform to their respective specs and validat
 
 ## Detector sync discipline
 
-Nine detectors live at `crates/detectors/src/lib.rs` and mirror the browser detector library at bootintel.com/tools/fingerprint. When adding a detector, keep the label set aligned between the two so JSON output round-trips cleanly.
+Fourteen detectors live at `crates/detectors/src/lib.rs` and mirror the browser detector library at bootintel.com/tools/fingerprint, which is the source of truth for the set — labels *and* registration order, so JSON output is comparable between the two. `cargo test -p bootintel --test browser_parity` proves it: it runs the browser library over all 31 corpus logs and asserts the `{label, value, detail}` sequence matches this crate's, log for log. When adding a detector, add it to the browser library first.
 
 ## Contributing
 

crates/cli/src/analyze/pacer.rs

@@ -6,7 +6,7 @@
 //! CPU on high-baud lines; running on a plain timer would delay
 //! findings on quiet lines that emit one banner and then wait.
 //!
-//! The full re-run of all 9 detectors against a 40 KiB log takes
+//! The full re-run of all 14 detectors against a 40 KiB log takes
 //! <5ms on modern hardware, so the pacer's job is really about
 //! amortizing display flicker + not blocking the serial reader,
 //! not about compute cost.

crates/cli/src/cmd/detectors.rs

@@ -23,11 +23,28 @@ pub struct Args {
 /// detector without a description here fails the build.
 const DESCRIPTIONS: &[(&str, &str)] = &[
     ("Bootloader", "U-Boot / Barebox / CFE + build timestamp"),
+    (
+        "Runtime firmware",
+        "OpenSBI — the RISC-V M-mode runtime below the bootloader",
+    ),
+    ("ROM identifier", "Mask-ROM build stamp (ESP-ROM)"),
+    (
+        "Firmware SDK",
+        "Vendor SDK the image was built against (ESP-IDF)",
+    ),
     ("Kernel", "Linux version + build metadata"),
     (
         "CPU / Arch",
         "ARMv7 / ARM64 / MIPS / RISC-V / x86_64 family",
     ),
+    (
+        "Userland",
+        "BusyBox version — what the userland is built on",
+    ),
+    (
+        "Flash layout",
+        "MTD partition map — offsets, names, sizes for a flash read",
+    ),
     (
         "Init system",
         "systemd / procd / OpenRC / SysV — what PID 1 is",

crates/cli/tests/browser_detectors.mjs

@@ -0,0 +1,67 @@
+// Browser-parity harness: run the browser detector library (the source of
+// truth) over a directory of boot logs and print its findings as JSON.
+//
+// Usage: node browser_detectors.mjs <path/to/detectors.ts> <logs-dir>
+//
+// Output: { "<file>": { "raw": [ { label, value, detail }, … ], "prefixed": [ … ] }, … }
+//
+// The `prefixed` variant is every line behind `[12:34:56.123] `. It is not
+// padding: line-prefix normalization is the one place the two
+// implementations could agree on a bare log and still disagree on a real
+// capture, and the browser exempts two policy detectors from it.
+//
+// `detectors.ts` is TypeScript living in the website repo, so it is
+// transpiled in-process with the `typescript` package resolved from that
+// repo's own node_modules (found by walking up from the file) and evaluated
+// in a fresh vm context — the same trick cli/test-parity.mjs uses, so both
+// parity harnesses agree on what "the browser implementation" means.
+import fs from 'node:fs'
+import path from 'node:path'
+import vm from 'node:vm'
+import { createRequire } from 'node:module'
+
+const [detectorsPath, logsDir] = process.argv.slice(2)
+if (!detectorsPath || !logsDir) {
+  console.error('usage: browser_detectors.mjs <detectors.ts> <logs-dir>')
+  process.exit(2)
+}
+
+function findTypescript(from) {
+  let dir = path.resolve(path.dirname(from))
+  for (;;) {
+    const candidate = path.join(dir, 'node_modules', 'typescript', 'package.json')
+    if (fs.existsSync(candidate)) return createRequire(candidate)('typescript')
+    const parent = path.dirname(dir)
+    if (parent === dir) break
+    dir = parent
+  }
+  // Last resort: whatever `typescript` resolves to for this script.
+  return createRequire(import.meta.url)('typescript')
+}
+
+const ts = findTypescript(detectorsPath)
+const source = fs.readFileSync(detectorsPath, 'utf8')
+const context = { exports: {} }
+vm.runInNewContext(
+  ts.transpileModule(source, {
+    compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022 },
+  }).outputText,
+  context,
+)
+
+const rows = (log) =>
+  context.exports.analyze(log).map((f) => ({
+    label: f.label,
+    value: f.value,
+    detail: f.detail === undefined ? null : f.detail,
+  }))
+
+const out = {}
+for (const file of fs.readdirSync(logsDir).filter((f) => f.endsWith('.txt')).sort()) {
+  const log = fs.readFileSync(path.join(logsDir, file), 'utf8')
+  out[file] = {
+    raw: rows(log),
+    prefixed: rows(log.split(/\r\n|\n|\r/).map((line) => `[12:34:56.123] ${line}`).join('\n')),
+  }
+}
+process.stdout.write(JSON.stringify(out, null, 2) + '\n')