Evaluate Project516's rebuilt-fuel model and make its export reproducible

README.md

@@ -721,7 +721,7 @@ This setup stands on other people's work. We link to or patch their code rather
 **Models:**
 
 - **FUEL YOLO11n by [Team 2826 Wave Robotics](https://www.chiefdelphi.com/t/introducing-wave-robotics-yolov11-model-for-rebuilt/512701).** It's the model PhotonVision ships for 2026; Wave gave PhotonVision permission to include it.
-- **FUEL YOLO26n by [Project516](https://huggingface.co/project516/rebuilt-fuel-model)** (AGPL-3.0), downloaded for comparison.
+- **FUEL YOLO26n by [Project516](https://huggingface.co/project516/rebuilt-fuel-model)** (AGPL-3.0). `tools/fuelmodel` exports it and compares it with Team 2826's.
 
 We don't redistribute either; see [docs/GAME-PIECE-MODELS.md](docs/GAME-PIECE-MODELS.md).
 

docs/GAME-PIECE-MODELS.md

@@ -16,8 +16,8 @@ which to use. Prepared 2026-09-24 by a Claude subagent; setup and usage are in t
 
 ## Exports and engines
 
-- **Exported on the laptop** with Ultralytics 8.4.161: `format=onnx imgsz=640 batch=1 dynamic=False opset=17 simplify=True`, plus an explicit `nms` setting.
-  - The venv is `~/build/yolo/venv`, and the export script is `~/build/yolo/export_fuel.py`.
+- **Exported** with Ultralytics 8.4.161: `format=onnx imgsz=640 batch=1 dynamic=False opset=17 simplify=True`, plus an explicit `nms` setting.
+  - `tools/fuelmodel/export.py` makes `project516_yolo26n_fuel_v1_raw.onnx` on any Linux machine, CPU only (see [its README](../tools/fuelmodel/README.md)). It pins the Hugging Face revision, checks the checkpoint's SHA-256, and fails unless the output is `[1, 5, 8400]`.
   - The ONNX files are in `staging/fuel-models/` (not in git) and `~/models` on the Jetson.
 - **Engines are built on the Jetson** with `trtexec --fp16` (TensorRT 10.3, ~7–8 min each, no warnings). `scripts/jetson/12-install-yolo-model.sh` does this and installs the result.
 
@@ -36,9 +36,39 @@ which to use. Prepared 2026-09-24 by a Claude subagent; setup and usage are in t
 In PhotonVision the full pipeline (colour decode, letterbox, inference, NMS) ran at 76 fps uncapped;
 we cap it at 30 fps.
 
+## Accuracy on the rebuilt-fuel photos
+
+`tools/fuelmodel/eval.py` scores both models the same way: the photo letterboxed to 640 px, NMS at
+IoU 0.45 (PhotonVision's default), then COCO-style AP. Team 2826's model here is PhotonVision's
+int8 TFLite (`fuelV1-yolo11n.tflite`), not the FP16 engine we run, so its numbers may be a little
+low. Run on GitHub Actions on 2026-09-30:
+
+| Model | Photos | Balls | Image | AP50 | AP50-95 | Recall @0.5 | Recall @0.9 |
+|---|---|---|---|---|---|---|---|
+| Project516 v1 | 11 held out | 13 | colour | 1.000 | 0.932 | 1.000 | 0.769 |
+| Team 2826 | 11 held out | 13 | colour | 1.000 | 0.772 | 1.000 | 0.615 |
+| Project516 v1 | 11 held out | 13 | gray | 1.000 | 0.844 | 0.923 | 0.000 |
+| Team 2826 | 11 held out | 13 | gray | 1.000 | 0.731 | 1.000 | 0.615 |
+| Team 2826 | all 53 | 111 | colour | 0.817 | 0.537 | 0.784 | 0.586 |
+| Team 2826 | all 53 | 111 | gray | 0.743 | 0.509 | 0.604 | 0.495 |
+
+- **Held out** is the 11 photos v1 didn't train on (v1's own 80/20 split, rebuilt). v1's scores on all
+  53 include its training photos, so they're left out.
+- **Gray** is the photo converted to one channel and back, which is what a mono camera gives the model.
+- **Recall @0.9** is at PhotonVision's default confidence threshold (0.9).
+- 13 balls is a small sample; one missed ball moves recall by 0.077.
+
+What this says:
+- **v1 works with our TensorRT backend.** The raw export has the same input and output as Team 2826's.
+- **In colour, v1 is at least as good,** with tighter boxes (AP50-95 0.932 against 0.772).
+- **On a mono camera, v1 needs a lower threshold.** In gray it still finds the balls, but with low
+  confidence: none pass 0.9, and 12 of 13 pass 0.5. Team 2826's model keeps its confidence in gray.
+- **Neither test is our camera.** These are phone photos, mostly close up.
+
 ## Which to use
 
 - **Now:** `wave2826_yolo11n_fuel_raw`. It's installed on the Jetson as "Fuel (Wave 2826 YOLO11n)", and it's the model PhotonVision ships.
-- **Next:** A/B it against `project516_yolo26n_fuel_v1_raw` (same input and output, ~17% fewer FLOPs, similar speed) on footage from our own colour camera. Rewind recordings are good for this.
+- **Mono cameras:** keep Team 2826's model. v1 at the default 0.9 threshold found no balls in gray.
+- **Colour camera:** try `project516_yolo26n_fuel_v1_raw` (same input and output, ~17% fewer FLOPs, similar speed) at a 0.5 threshold, and A/B it on Rewind recordings from our own camera before switching.
 - **By October:** fine-tune on our own camera's frames, with motion blur, bumpers and yellow distractors, and keep the 640 input. Through an ultrawide lens, a ball 5 m away is only ~9 px wide at 640.
 - **Rebuild the engines** from the ONNX after any JetPack or TensorRT change. Engines only run on the TensorRT version and GPU they were built for.

tools/fuelmodel/README.md

@@ -0,0 +1,22 @@
+# fuelmodel: FUEL model export and comparison
+
+Exports Project516's [rebuilt-fuel](https://github.com/Project516/rebuilt-fuel) YOLO26n (v1) to the
+ONNX that `scripts/jetson/12-install-yolo-model.sh` takes, and scores it against Team 2826's
+YOLO11n. Runs on the CPU of any Linux machine; not on the Jetson or a Pi (it installs PyTorch).
+
+```bash
+python3 -m venv ~/build/fuelmodel-venv
+~/build/fuelmodel-venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch==2.14.0 torchvision==0.29.0
+~/build/fuelmodel-venv/bin/pip install -r tools/fuelmodel/requirements.txt
+~/build/fuelmodel-venv/bin/python tools/fuelmodel/export.py
+~/build/fuelmodel-venv/bin/python tools/fuelmodel/eval.py staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx --wave
+```
+
+- `export.py` writes `staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx` (about a minute).
+  Copy it to `~/models` on the Jetson and install it:
+  `12-install-yolo-model.sh ~/models/project516_yolo26n_fuel_v1_raw.onnx "Fuel (Project516 YOLO26n)" Fuel`.
+- `eval.py` downloads the 53 rebuilt-fuel photos and prints a Markdown table (a few minutes).
+  What the numbers mean, and the last results: [docs/GAME-PIECE-MODELS.md](../../docs/GAME-PIECE-MODELS.md#accuracy-on-the-rebuilt-fuel-photos).
+
+Every download is pinned: the model and dataset to a Hugging Face revision, Team 2826's TFLite to a
+PhotonVision commit and SHA-256.

tools/fuelmodel/eval.py

@@ -0,0 +1,249 @@
+"""Compare FUEL models on Project516's rebuilt-fuel photos, in colour and in gray.
+
+    tools/fuelmodel/eval.py staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx [--wave]
+
+Downloads the rebuilt-fuel dataset (53 labelled phone photos, pinned revision) and scores each
+model the same way: the photo letterboxed to 640x640 as PhotonVision does, NMS at IoU 0.45
+(PhotonVision's default), then COCO-style AP. --wave adds Team 2826's YOLO11n as PhotonVision ships
+it (int8 TFLite, pinned commit), so both models see the same pixels.
+
+Two splits:
+- val: the photos Project516's v1 did not train on. It rebuilds v1's split (train_test_split,
+  80/20, random_state=42, over the train.txt entries that have a photo, as v1's training script
+  did), so it's the only fair split for v1.
+- all: all 53 photos. Team 2826 never trained on any of them.
+
+Gray is each photo converted to one channel and back to three, which is what PhotonVision feeds
+the model from a mono camera like the Thriftiest Cam.
+
+Prints a Markdown table: AP at IoU 0.5 and 0.5-0.95, then precision and recall at confidence 0.5
+and recall at 0.9 (PhotonVision's default threshold). Needs the packages in requirements.txt.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import urllib.request
+from pathlib import Path
+
+import cv2
+import numpy as np
+
+DATASET_REPO = "project516/rebuilt-fuel-dataset"
+DATASET_REVISION = "4040182cc1e330c40d217d6ac7581bc6b2daabda"
+# train.txt's order decides v1's train/val split.
+TRAIN_TXT_URL = ("https://raw.githubusercontent.com/Project516/rebuilt-fuel/"
+                 "acad6f1b3feb68cf05ee8e0bfdaf1637905eab68/dataset1/annotated/train.txt")
+TRAIN_TXT_SHA256 = "eaf6637dd36d69add7f7694b64ae6cf9efd71874858a91515e75303d8a5fada8"
+WAVE_URL = ("https://raw.githubusercontent.com/PhotonVision/photonvision/"
+            "1f419c9de3a6a0787571648d18211537a83fdca9/"
+            "photon-server/src/main/resources/models/fuelV1-yolo11n.tflite")
+WAVE_SHA256 = "bd74b36f230df1e560ce92d0e06b7504dda169462882774fe6154b598a7b9bcb"
+SIZE = 640
+NMS_IOU = 0.45
+MIN_CONF = 0.001
+
+
+def sha256(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def download(url: str, dest: Path, digest: str | None = None) -> Path:
+    if not dest.exists():
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        urllib.request.urlretrieve(url, dest)
+    if digest and sha256(dest) != digest:
+        raise SystemExit(f"{dest} SHA-256 is {sha256(dest)}, expected {digest}")
+    return dest
+
+
+def v1_val_names(train_txt: Path, src: Path) -> set[str]:
+    from sklearn.model_selection import train_test_split
+
+    names = [Path(line.strip()).name for line in train_txt.read_text().splitlines() if line.strip()]
+    names = [n for n in names if (src / n).exists()]
+    _, val = train_test_split(names, train_size=0.8, random_state=42)
+    return set(val)
+
+
+def letterbox(bgr: np.ndarray) -> tuple[np.ndarray, float, int, int]:
+    """640x640 RGB with grey (114) bars, plus the scale and padding used."""
+    h, w = bgr.shape[:2]
+    r = min(SIZE / w, SIZE / h)
+    nw, nh = round(w * r), round(h * r)
+    px, py = (SIZE - nw) // 2, (SIZE - nh) // 2
+    out = np.full((SIZE, SIZE, 3), 114, np.uint8)
+    out[py:py + nh, px:px + nw] = cv2.resize(bgr, (nw, nh), interpolation=cv2.INTER_LINEAR)
+    return cv2.cvtColor(out, cv2.COLOR_BGR2RGB), r, px, py
+
+
+def labels_xyxy(label: Path, w: int, h: int, r: float, px: int, py: int) -> np.ndarray:
+    rows = [list(map(float, line.split()[1:])) for line in label.read_text().splitlines() if line.strip()]
+    if not rows:
+        return np.zeros((0, 4))
+    cx, cy, bw, bh = np.array(rows).T
+    x1 = (cx - bw / 2) * w * r + px
+    y1 = (cy - bh / 2) * h * r + py
+    return np.stack([x1, y1, x1 + bw * w * r, y1 + bh * h * r], 1)
+
+
+class OnnxRaw:
+    """An Ultralytics raw export: float RGB [1, 3, 640, 640] in, [1, 5, N] cx cy w h score out."""
+
+    def __init__(self, path: Path):
+        import onnxruntime as ort
+
+        self.s = ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])
+        self.input = self.s.get_inputs()[0].name
+
+    def __call__(self, rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+        x = rgb.transpose(2, 0, 1)[None].astype(np.float32) / 255
+        out = self.s.run(None, {self.input: x})[0][0]
+        cx, cy, w, h, score = out[:5]
+        return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1), score
+
+
+class PhotonTflite:
+    """PhotonVision's int8 TFLite: uint8 RGB in; boxes (x1 y1 x2 y2), scores, class_idx out."""
+
+    def __init__(self, path: Path):
+        from ai_edge_litert.interpreter import Interpreter
+
+        self.i = Interpreter(str(path))
+        self.i.allocate_tensors()
+        self.input = self.i.get_input_details()[0]["index"]
+        self.out = {d["name"]: d for d in self.i.get_output_details()}
+
+    def _get(self, name: str) -> np.ndarray:
+        d = self.out[name]
+        scale, zero = d["quantization"]
+        return (self.i.get_tensor(d["index"]).astype(np.float32) - zero) * scale
+
+    def __call__(self, rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+        self.i.set_tensor(self.input, rgb[None])
+        self.i.invoke()
+        return self._get("boxes")[0], self._get("scores")[0]
+
+
+def iou(a: np.ndarray, b: np.ndarray) -> np.ndarray:
+    """IoU of every box in a against every box in b, both x1 y1 x2 y2."""
+    lt = np.maximum(a[:, None, :2], b[None, :, :2])
+    rb = np.minimum(a[:, None, 2:], b[None, :, 2:])
+    inter = np.clip(rb - lt, 0, None).prod(2)
+    area = lambda x: (x[:, 2] - x[:, 0]) * (x[:, 3] - x[:, 1])
+    return inter / (area(a)[:, None] + area(b)[None] - inter + 1e-9)
+
+
+def nms(boxes: np.ndarray, scores: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+    keep_mask = scores >= MIN_CONF
+    boxes, scores = boxes[keep_mask], scores[keep_mask]
+    order = np.argsort(-scores)
+    keep = []
+    while order.size:
+        i = order[0]
+        keep.append(i)
+        order = order[1:][iou(boxes[i:i + 1], boxes[order[1:]])[0] <= NMS_IOU]
+    return boxes[keep], scores[keep]
+
+
+def match(pred: np.ndarray, truth: np.ndarray, thresholds: np.ndarray) -> np.ndarray:
+    """Per prediction (best first), whether it is a true positive at each IoU threshold."""
+    tp = np.zeros((len(pred), len(thresholds)), bool)
+    if not len(pred) or not len(truth):
+        return tp
+    overlaps = iou(pred, truth)
+    for t, thr in enumerate(thresholds):
+        used = np.zeros(len(truth), bool)
+        for p in range(len(pred)):
+            cand = np.where((overlaps[p] >= thr) & ~used)[0]
+            if cand.size:
+                used[cand[np.argmax(overlaps[p, cand])]] = True
+                tp[p, t] = True
+    return tp
+
+
+def average_precision(tp: np.ndarray, conf: np.ndarray, n_truth: int) -> np.ndarray:
+    """COCO 101-point AP per IoU threshold."""
+    if not len(tp):
+        return np.zeros(tp.shape[1])
+    order = np.argsort(-conf)
+    tp = tp[order]
+    ctp = np.cumsum(tp, 0)
+    recall = ctp / max(n_truth, 1)
+    precision = ctp / np.arange(1, len(tp) + 1)[:, None]
+    aps = []
+    grid = np.linspace(0, 1, 101)
+    for t in range(tp.shape[1]):
+        envelope = np.flip(np.maximum.accumulate(np.flip(precision[:, t])))
+        idx = np.searchsorted(recall[:, t], grid, side="left")
+        found = idx < len(envelope)
+        aps.append(np.where(found, envelope[np.minimum(idx, len(envelope) - 1)], 0.0).mean())
+    return np.array(aps)
+
+
+def score(model, photos: list[tuple[np.ndarray, Path]]) -> dict[str, float]:
+    thresholds = np.linspace(0.5, 0.95, 10)
+    tps, confs, n_truth = [], [], 0
+    for bgr, label in photos:
+        rgb, r, px, py = letterbox(bgr)
+        truth = labels_xyxy(label, bgr.shape[1], bgr.shape[0], r, px, py)
+        boxes, scores = nms(*model(rgb))
+        tps.append(match(boxes, truth, thresholds))
+        confs.append(scores)
+        n_truth += len(truth)
+    tp, conf = np.concatenate(tps), np.concatenate(confs)
+    ap = average_precision(tp, conf, n_truth)
+
+    def at(c: float) -> tuple[float, float]:
+        hit = conf >= c
+        found = int(tp[hit, 0].sum())
+        return found / max(int(hit.sum()), 1), found / max(n_truth, 1)
+
+    p50, r50 = at(0.5)
+    _, r90 = at(0.9)
+    return {"balls": n_truth, "ap50": ap[0], "ap": ap.mean(), "p50": p50, "r50": r50, "r90": r90}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("models", nargs="*", type=Path, help="Ultralytics raw ONNX exports")
+    parser.add_argument("--wave", action="store_true", help="also score Team 2826's model")
+    parser.add_argument("--work", type=Path, default=Path.home() / "build" / "fuel-eval")
+    args = parser.parse_args()
+
+    from huggingface_hub import snapshot_download
+
+    src = Path(snapshot_download(DATASET_REPO, repo_type="dataset", revision=DATASET_REVISION))
+    images = sorted(src.glob("*.png"), key=lambda p: int(p.stem))
+    if len(images) != 53:
+        raise SystemExit(f"expected 53 photos in {src}, found {len(images)}")
+    val = v1_val_names(download(TRAIN_TXT_URL, args.work / "train.txt", TRAIN_TXT_SHA256), src)
+
+    models = [(m.stem, OnnxRaw(m)) for m in args.models]
+    if args.wave:
+        tflite = download(WAVE_URL, args.work / "fuelV1-yolo11n.tflite", WAVE_SHA256)
+        models.append(("wave2826_yolo11n_fuel_int8", PhotonTflite(tflite)))
+    if not models:
+        raise SystemExit("nothing to score: pass a model or --wave")
+
+    colour = [(cv2.imread(str(p)), p.with_suffix(".txt")) for p in images]
+    gray = [(cv2.cvtColor(cv2.cvtColor(b, cv2.COLOR_BGR2GRAY), cv2.COLOR_GRAY2BGR), l)
+            for b, l in colour]
+    in_val = [p.name in val for p in images]
+    sets = {}
+    for mode, photos in (("colour", colour), ("gray", gray)):
+        sets[("val", mode)] = [x for x, v in zip(photos, in_val) if v]
+        sets[("all", mode)] = photos
+
+    print("| Model | Split | Photos | Balls | Image | AP50 | AP50-95 | P @0.5 | R @0.5 | R @0.9 |")
+    print("|---|---|---|---|---|---|---|---|---|---|")
+    for name, model in models:
+        for (split, mode), photos in sets.items():
+            s = score(model, photos)
+            print(f"| {name} | {split} | {len(photos)} | {s['balls']} | {mode} | {s['ap50']:.3f}"
+                  f" | {s['ap']:.3f} | {s['p50']:.3f} | {s['r50']:.3f} | {s['r90']:.3f} |", flush=True)
+
+
+if __name__ == "__main__":
+    main()