diff --git a/README.md b/README.md index 9cd7dcd..89259e0 100644 --- a/README.md +++ b/README.md @@ -721,7 +721,7 @@ This setup stands on other people's work. We link to or patch their code rather **Models:** - **FUEL YOLO11n by [Team 2826 Wave Robotics](https://www.chiefdelphi.com/t/introducing-wave-robotics-yolov11-model-for-rebuilt/512701).** It's the model PhotonVision ships for 2026; Wave gave PhotonVision permission to include it. -- **FUEL YOLO26n by [Project516](https://huggingface.co/project516/rebuilt-fuel-model)** (AGPL-3.0), downloaded for comparison. +- **FUEL YOLO26n by [Project516](https://huggingface.co/project516/rebuilt-fuel-model)** (AGPL-3.0). `tools/fuelmodel` exports it and compares it with Team 2826's. We don't redistribute either; see [docs/GAME-PIECE-MODELS.md](docs/GAME-PIECE-MODELS.md). diff --git a/docs/GAME-PIECE-MODELS.md b/docs/GAME-PIECE-MODELS.md index 285f106..4a20a98 100644 --- a/docs/GAME-PIECE-MODELS.md +++ b/docs/GAME-PIECE-MODELS.md @@ -16,8 +16,8 @@ which to use. Prepared 2026-09-24 by a Claude subagent; setup and usage are in t ## Exports and engines -- **Exported on the laptop** with Ultralytics 8.4.161: `format=onnx imgsz=640 batch=1 dynamic=False opset=17 simplify=True`, plus an explicit `nms` setting. - - The venv is `~/build/yolo/venv`, and the export script is `~/build/yolo/export_fuel.py`. +- **Exported** with Ultralytics 8.4.161: `format=onnx imgsz=640 batch=1 dynamic=False opset=17 simplify=True`, plus an explicit `nms` setting. + - `tools/fuelmodel/export.py` makes `project516_yolo26n_fuel_v1_raw.onnx` on any Linux machine, CPU only (see [its README](../tools/fuelmodel/README.md)). It pins the Hugging Face revision, checks the checkpoint's SHA-256, and fails unless the output is `[1, 5, 8400]`. - The ONNX files are in `staging/fuel-models/` (not in git) and `~/models` on the Jetson. - **Engines are built on the Jetson** with `trtexec --fp16` (TensorRT 10.3, ~7–8 min each, no warnings). `scripts/jetson/12-install-yolo-model.sh` does this and installs the result. @@ -36,9 +36,39 @@ which to use. Prepared 2026-09-24 by a Claude subagent; setup and usage are in t In PhotonVision the full pipeline (colour decode, letterbox, inference, NMS) ran at 76 fps uncapped; we cap it at 30 fps. +## Accuracy on the rebuilt-fuel photos + +`tools/fuelmodel/eval.py` scores both models the same way: the photo letterboxed to 640 px, NMS at +IoU 0.45 (PhotonVision's default), then COCO-style AP. Team 2826's model here is PhotonVision's +int8 TFLite (`fuelV1-yolo11n.tflite`), not the FP16 engine we run, so its numbers may be a little +low. Run on GitHub Actions on 2026-09-30: + +| Model | Photos | Balls | Image | AP50 | AP50-95 | Recall @0.5 | Recall @0.9 | +|---|---|---|---|---|---|---|---| +| Project516 v1 | 11 held out | 13 | colour | 1.000 | 0.932 | 1.000 | 0.769 | +| Team 2826 | 11 held out | 13 | colour | 1.000 | 0.772 | 1.000 | 0.615 | +| Project516 v1 | 11 held out | 13 | gray | 1.000 | 0.844 | 0.923 | 0.000 | +| Team 2826 | 11 held out | 13 | gray | 1.000 | 0.731 | 1.000 | 0.615 | +| Team 2826 | all 53 | 111 | colour | 0.817 | 0.537 | 0.784 | 0.586 | +| Team 2826 | all 53 | 111 | gray | 0.743 | 0.509 | 0.604 | 0.495 | + +- **Held out** is the 11 photos v1 didn't train on (v1's own 80/20 split, rebuilt). v1's scores on all + 53 include its training photos, so they're left out. +- **Gray** is the photo converted to one channel and back, which is what a mono camera gives the model. +- **Recall @0.9** is at PhotonVision's default confidence threshold (0.9). +- 13 balls is a small sample; one missed ball moves recall by 0.077. + +What this says: +- **v1 works with our TensorRT backend.** The raw export has the same input and output as Team 2826's. +- **In colour, v1 is at least as good,** with tighter boxes (AP50-95 0.932 against 0.772). +- **On a mono camera, v1 needs a lower threshold.** In gray it still finds the balls, but with low + confidence: none pass 0.9, and 12 of 13 pass 0.5. Team 2826's model keeps its confidence in gray. +- **Neither test is our camera.** These are phone photos, mostly close up. + ## Which to use - **Now:** `wave2826_yolo11n_fuel_raw`. It's installed on the Jetson as "Fuel (Wave 2826 YOLO11n)", and it's the model PhotonVision ships. -- **Next:** A/B it against `project516_yolo26n_fuel_v1_raw` (same input and output, ~17% fewer FLOPs, similar speed) on footage from our own colour camera. Rewind recordings are good for this. +- **Mono cameras:** keep Team 2826's model. v1 at the default 0.9 threshold found no balls in gray. +- **Colour camera:** try `project516_yolo26n_fuel_v1_raw` (same input and output, ~17% fewer FLOPs, similar speed) at a 0.5 threshold, and A/B it on Rewind recordings from our own camera before switching. - **By October:** fine-tune on our own camera's frames, with motion blur, bumpers and yellow distractors, and keep the 640 input. Through an ultrawide lens, a ball 5 m away is only ~9 px wide at 640. - **Rebuild the engines** from the ONNX after any JetPack or TensorRT change. Engines only run on the TensorRT version and GPU they were built for. diff --git a/tools/fuelmodel/README.md b/tools/fuelmodel/README.md new file mode 100644 index 0000000..5cbaa89 --- /dev/null +++ b/tools/fuelmodel/README.md @@ -0,0 +1,22 @@ +# fuelmodel: FUEL model export and comparison + +Exports Project516's [rebuilt-fuel](https://github.com/Project516/rebuilt-fuel) YOLO26n (v1) to the +ONNX that `scripts/jetson/12-install-yolo-model.sh` takes, and scores it against Team 2826's +YOLO11n. Runs on the CPU of any Linux machine; not on the Jetson or a Pi (it installs PyTorch). + +```bash +python3 -m venv ~/build/fuelmodel-venv +~/build/fuelmodel-venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch==2.14.0 torchvision==0.29.0 +~/build/fuelmodel-venv/bin/pip install -r tools/fuelmodel/requirements.txt +~/build/fuelmodel-venv/bin/python tools/fuelmodel/export.py +~/build/fuelmodel-venv/bin/python tools/fuelmodel/eval.py staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx --wave +``` + +- `export.py` writes `staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx` (about a minute). + Copy it to `~/models` on the Jetson and install it: + `12-install-yolo-model.sh ~/models/project516_yolo26n_fuel_v1_raw.onnx "Fuel (Project516 YOLO26n)" Fuel`. +- `eval.py` downloads the 53 rebuilt-fuel photos and prints a Markdown table (a few minutes). + What the numbers mean, and the last results: [docs/GAME-PIECE-MODELS.md](../../docs/GAME-PIECE-MODELS.md#accuracy-on-the-rebuilt-fuel-photos). + +Every download is pinned: the model and dataset to a Hugging Face revision, Team 2826's TFLite to a +PhotonVision commit and SHA-256. diff --git a/tools/fuelmodel/eval.py b/tools/fuelmodel/eval.py new file mode 100644 index 0000000..f5052db --- /dev/null +++ b/tools/fuelmodel/eval.py @@ -0,0 +1,249 @@ +"""Compare FUEL models on Project516's rebuilt-fuel photos, in colour and in gray. + + tools/fuelmodel/eval.py staging/fuel-models/project516_yolo26n_fuel_v1_raw.onnx [--wave] + +Downloads the rebuilt-fuel dataset (53 labelled phone photos, pinned revision) and scores each +model the same way: the photo letterboxed to 640x640 as PhotonVision does, NMS at IoU 0.45 +(PhotonVision's default), then COCO-style AP. --wave adds Team 2826's YOLO11n as PhotonVision ships +it (int8 TFLite, pinned commit), so both models see the same pixels. + +Two splits: +- val: the photos Project516's v1 did not train on. It rebuilds v1's split (train_test_split, + 80/20, random_state=42, over the train.txt entries that have a photo, as v1's training script + did), so it's the only fair split for v1. +- all: all 53 photos. Team 2826 never trained on any of them. + +Gray is each photo converted to one channel and back to three, which is what PhotonVision feeds +the model from a mono camera like the Thriftiest Cam. + +Prints a Markdown table: AP at IoU 0.5 and 0.5-0.95, then precision and recall at confidence 0.5 +and recall at 0.9 (PhotonVision's default threshold). Needs the packages in requirements.txt. +""" + +from __future__ import annotations + +import argparse +import hashlib +import urllib.request +from pathlib import Path + +import cv2 +import numpy as np + +DATASET_REPO = "project516/rebuilt-fuel-dataset" +DATASET_REVISION = "4040182cc1e330c40d217d6ac7581bc6b2daabda" +# train.txt's order decides v1's train/val split. +TRAIN_TXT_URL = ("https://raw.githubusercontent.com/Project516/rebuilt-fuel/" + "acad6f1b3feb68cf05ee8e0bfdaf1637905eab68/dataset1/annotated/train.txt") +TRAIN_TXT_SHA256 = "eaf6637dd36d69add7f7694b64ae6cf9efd71874858a91515e75303d8a5fada8" +WAVE_URL = ("https://raw.githubusercontent.com/PhotonVision/photonvision/" + "1f419c9de3a6a0787571648d18211537a83fdca9/" + "photon-server/src/main/resources/models/fuelV1-yolo11n.tflite") +WAVE_SHA256 = "bd74b36f230df1e560ce92d0e06b7504dda169462882774fe6154b598a7b9bcb" +SIZE = 640 +NMS_IOU = 0.45 +MIN_CONF = 0.001 + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def download(url: str, dest: Path, digest: str | None = None) -> Path: + if not dest.exists(): + dest.parent.mkdir(parents=True, exist_ok=True) + urllib.request.urlretrieve(url, dest) + if digest and sha256(dest) != digest: + raise SystemExit(f"{dest} SHA-256 is {sha256(dest)}, expected {digest}") + return dest + + +def v1_val_names(train_txt: Path, src: Path) -> set[str]: + from sklearn.model_selection import train_test_split + + names = [Path(line.strip()).name for line in train_txt.read_text().splitlines() if line.strip()] + names = [n for n in names if (src / n).exists()] + _, val = train_test_split(names, train_size=0.8, random_state=42) + return set(val) + + +def letterbox(bgr: np.ndarray) -> tuple[np.ndarray, float, int, int]: + """640x640 RGB with grey (114) bars, plus the scale and padding used.""" + h, w = bgr.shape[:2] + r = min(SIZE / w, SIZE / h) + nw, nh = round(w * r), round(h * r) + px, py = (SIZE - nw) // 2, (SIZE - nh) // 2 + out = np.full((SIZE, SIZE, 3), 114, np.uint8) + out[py:py + nh, px:px + nw] = cv2.resize(bgr, (nw, nh), interpolation=cv2.INTER_LINEAR) + return cv2.cvtColor(out, cv2.COLOR_BGR2RGB), r, px, py + + +def labels_xyxy(label: Path, w: int, h: int, r: float, px: int, py: int) -> np.ndarray: + rows = [list(map(float, line.split()[1:])) for line in label.read_text().splitlines() if line.strip()] + if not rows: + return np.zeros((0, 4)) + cx, cy, bw, bh = np.array(rows).T + x1 = (cx - bw / 2) * w * r + px + y1 = (cy - bh / 2) * h * r + py + return np.stack([x1, y1, x1 + bw * w * r, y1 + bh * h * r], 1) + + +class OnnxRaw: + """An Ultralytics raw export: float RGB [1, 3, 640, 640] in, [1, 5, N] cx cy w h score out.""" + + def __init__(self, path: Path): + import onnxruntime as ort + + self.s = ort.InferenceSession(str(path), providers=["CPUExecutionProvider"]) + self.input = self.s.get_inputs()[0].name + + def __call__(self, rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + x = rgb.transpose(2, 0, 1)[None].astype(np.float32) / 255 + out = self.s.run(None, {self.input: x})[0][0] + cx, cy, w, h, score = out[:5] + return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1), score + + +class PhotonTflite: + """PhotonVision's int8 TFLite: uint8 RGB in; boxes (x1 y1 x2 y2), scores, class_idx out.""" + + def __init__(self, path: Path): + from ai_edge_litert.interpreter import Interpreter + + self.i = Interpreter(str(path)) + self.i.allocate_tensors() + self.input = self.i.get_input_details()[0]["index"] + self.out = {d["name"]: d for d in self.i.get_output_details()} + + def _get(self, name: str) -> np.ndarray: + d = self.out[name] + scale, zero = d["quantization"] + return (self.i.get_tensor(d["index"]).astype(np.float32) - zero) * scale + + def __call__(self, rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + self.i.set_tensor(self.input, rgb[None]) + self.i.invoke() + return self._get("boxes")[0], self._get("scores")[0] + + +def iou(a: np.ndarray, b: np.ndarray) -> np.ndarray: + """IoU of every box in a against every box in b, both x1 y1 x2 y2.""" + lt = np.maximum(a[:, None, :2], b[None, :, :2]) + rb = np.minimum(a[:, None, 2:], b[None, :, 2:]) + inter = np.clip(rb - lt, 0, None).prod(2) + area = lambda x: (x[:, 2] - x[:, 0]) * (x[:, 3] - x[:, 1]) + return inter / (area(a)[:, None] + area(b)[None] - inter + 1e-9) + + +def nms(boxes: np.ndarray, scores: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + keep_mask = scores >= MIN_CONF + boxes, scores = boxes[keep_mask], scores[keep_mask] + order = np.argsort(-scores) + keep = [] + while order.size: + i = order[0] + keep.append(i) + order = order[1:][iou(boxes[i:i + 1], boxes[order[1:]])[0] <= NMS_IOU] + return boxes[keep], scores[keep] + + +def match(pred: np.ndarray, truth: np.ndarray, thresholds: np.ndarray) -> np.ndarray: + """Per prediction (best first), whether it is a true positive at each IoU threshold.""" + tp = np.zeros((len(pred), len(thresholds)), bool) + if not len(pred) or not len(truth): + return tp + overlaps = iou(pred, truth) + for t, thr in enumerate(thresholds): + used = np.zeros(len(truth), bool) + for p in range(len(pred)): + cand = np.where((overlaps[p] >= thr) & ~used)[0] + if cand.size: + used[cand[np.argmax(overlaps[p, cand])]] = True + tp[p, t] = True + return tp + + +def average_precision(tp: np.ndarray, conf: np.ndarray, n_truth: int) -> np.ndarray: + """COCO 101-point AP per IoU threshold.""" + if not len(tp): + return np.zeros(tp.shape[1]) + order = np.argsort(-conf) + tp = tp[order] + ctp = np.cumsum(tp, 0) + recall = ctp / max(n_truth, 1) + precision = ctp / np.arange(1, len(tp) + 1)[:, None] + aps = [] + grid = np.linspace(0, 1, 101) + for t in range(tp.shape[1]): + envelope = np.flip(np.maximum.accumulate(np.flip(precision[:, t]))) + idx = np.searchsorted(recall[:, t], grid, side="left") + found = idx < len(envelope) + aps.append(np.where(found, envelope[np.minimum(idx, len(envelope) - 1)], 0.0).mean()) + return np.array(aps) + + +def score(model, photos: list[tuple[np.ndarray, Path]]) -> dict[str, float]: + thresholds = np.linspace(0.5, 0.95, 10) + tps, confs, n_truth = [], [], 0 + for bgr, label in photos: + rgb, r, px, py = letterbox(bgr) + truth = labels_xyxy(label, bgr.shape[1], bgr.shape[0], r, px, py) + boxes, scores = nms(*model(rgb)) + tps.append(match(boxes, truth, thresholds)) + confs.append(scores) + n_truth += len(truth) + tp, conf = np.concatenate(tps), np.concatenate(confs) + ap = average_precision(tp, conf, n_truth) + + def at(c: float) -> tuple[float, float]: + hit = conf >= c + found = int(tp[hit, 0].sum()) + return found / max(int(hit.sum()), 1), found / max(n_truth, 1) + + p50, r50 = at(0.5) + _, r90 = at(0.9) + return {"balls": n_truth, "ap50": ap[0], "ap": ap.mean(), "p50": p50, "r50": r50, "r90": r90} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("models", nargs="*", type=Path, help="Ultralytics raw ONNX exports") + parser.add_argument("--wave", action="store_true", help="also score Team 2826's model") + parser.add_argument("--work", type=Path, default=Path.home() / "build" / "fuel-eval") + args = parser.parse_args() + + from huggingface_hub import snapshot_download + + src = Path(snapshot_download(DATASET_REPO, repo_type="dataset", revision=DATASET_REVISION)) + images = sorted(src.glob("*.png"), key=lambda p: int(p.stem)) + if len(images) != 53: + raise SystemExit(f"expected 53 photos in {src}, found {len(images)}") + val = v1_val_names(download(TRAIN_TXT_URL, args.work / "train.txt", TRAIN_TXT_SHA256), src) + + models = [(m.stem, OnnxRaw(m)) for m in args.models] + if args.wave: + tflite = download(WAVE_URL, args.work / "fuelV1-yolo11n.tflite", WAVE_SHA256) + models.append(("wave2826_yolo11n_fuel_int8", PhotonTflite(tflite))) + if not models: + raise SystemExit("nothing to score: pass a model or --wave") + + colour = [(cv2.imread(str(p)), p.with_suffix(".txt")) for p in images] + gray = [(cv2.cvtColor(cv2.cvtColor(b, cv2.COLOR_BGR2GRAY), cv2.COLOR_GRAY2BGR), l) + for b, l in colour] + in_val = [p.name in val for p in images] + sets = {} + for mode, photos in (("colour", colour), ("gray", gray)): + sets[("val", mode)] = [x for x, v in zip(photos, in_val) if v] + sets[("all", mode)] = photos + + print("| Model | Split | Photos | Balls | Image | AP50 | AP50-95 | P @0.5 | R @0.5 | R @0.9 |") + print("|---|---|---|---|---|---|---|---|---|---|") + for name, model in models: + for (split, mode), photos in sets.items(): + s = score(model, photos) + print(f"| {name} | {split} | {len(photos)} | {s['balls']} | {mode} | {s['ap50']:.3f}" + f" | {s['ap']:.3f} | {s['p50']:.3f} | {s['r50']:.3f} | {s['r90']:.3f} |", flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/fuelmodel/export.py b/tools/fuelmodel/export.py new file mode 100644 index 0000000..3cc5868 --- /dev/null +++ b/tools/fuelmodel/export.py @@ -0,0 +1,79 @@ +"""Export Project516's rebuilt-fuel YOLO26n model to the ONNX that 12-install-yolo-model.sh takes. + + tools/fuelmodel/export.py [--out staging/fuel-models] + +Downloads rebuilt-fuel-model-v1.pt from Hugging Face at a pinned revision, checks its SHA-256, and +exports the raw head: output [1, 5, 8400] (cx, cy, w, h, score), no NMS inside the model. Our +TensorRT runner does the NMS, so its thresholds stay tunable. The script fails if the ONNX has any +other output shape. + +Needs the packages in requirements.txt. Runs on the CPU in about a minute. +""" + +from __future__ import annotations + +import argparse +import hashlib +import shutil +from pathlib import Path + +MODEL_REPO = "project516/rebuilt-fuel-model" +MODEL_REVISION = "7661f55574604588bee7dfbf889c01828558886b" +MODEL_FILE = "rebuilt-fuel-model-v1.pt" +MODEL_SHA256 = "94954a468c210c703929eff18a19c380a03865a1194f1584027ae2847263f0c6" +ONNX_NAME = "project516_yolo26n_fuel_v1_raw.onnx" +RAW_SHAPE = [1, 5, 8400] + + +def sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def download_checkpoint() -> Path: + from huggingface_hub import hf_hub_download + + pt = Path(hf_hub_download(MODEL_REPO, MODEL_FILE, revision=MODEL_REVISION)) + got = sha256(pt) + if got != MODEL_SHA256: + raise SystemExit(f"{MODEL_FILE} SHA-256 is {got}, expected {MODEL_SHA256}") + return pt + + +def export(pt: Path, out_dir: Path) -> Path: + import onnx + from ultralytics import YOLO + + work = out_dir / "work" + work.mkdir(parents=True, exist_ok=True) + local_pt = work / MODEL_FILE + shutil.copyfile(pt, local_pt) + # end2end=False keeps YOLO26's one-to-many head, the same layout as a YOLO11 raw export. + exported = YOLO(str(local_pt)).export( + format="onnx", imgsz=640, batch=1, dynamic=False, opset=17, simplify=True, + end2end=False, nms=False) + onnx_path = out_dir / ONNX_NAME + shutil.move(exported, onnx_path) + shutil.rmtree(work) + + model = onnx.load(str(onnx_path)) + shapes = [[d.dim_value for d in o.type.tensor_type.shape.dim] for o in model.graph.output] + if shapes != [RAW_SHAPE]: + raise SystemExit(f"{onnx_path} outputs {shapes}, expected [{RAW_SHAPE}]") + return onnx_path + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--out", type=Path, default=Path("staging/fuel-models")) + args = parser.parse_args() + args.out.mkdir(parents=True, exist_ok=True) + onnx_path = export(download_checkpoint(), args.out) + print(f"{onnx_path} sha256 {sha256(onnx_path)} output {RAW_SHAPE}") + + +if __name__ == "__main__": + main() diff --git a/tools/fuelmodel/requirements.txt b/tools/fuelmodel/requirements.txt new file mode 100644 index 0000000..9e57f98 --- /dev/null +++ b/tools/fuelmodel/requirements.txt @@ -0,0 +1,9 @@ +ultralytics==8.4.161 +onnx==1.23.0 +onnxslim==0.1.96 +onnxruntime==1.30.0 +huggingface_hub==2.0.0 +scikit-learn==1.9.1 +ai-edge-litert==2.2.0 +opencv-python-headless==5.0.0.93 +numpy==2.5.2