Direct .insv dual-fisheye → pinhole rig ingestion (skip the equirectangular intermediate)

[kfarr]

Why

360° scenes processed through the equirectangular (ER) path show weak/missing ground, and both suspected causes are real and compound each other:

1. The ER projection stretches the poles. By the time Insta360 Studio has stitched the dual fisheyes into a 2:1 image, the ground directly under the camera is smeared across the entire bottom pixel row.
2. The ER virtual rig never looks down. pano_sfm.py renders 12 views at pitches (−35°, 0°, +35°) with 90° FOV — nothing below −80° pitch is covered, so the nadir gets zero direct observations.

This PR adds a path that crops virtual pinhole views straight out of the two raw fisheye sensor streams in an .insv file (the approach used by LichtFeld's fisheye mode):

• Native sensor pixels everywhere — an equidistant fisheye has roughly uniform angular resolution, and the nadir sits ~90° off each lens axis, well inside a ~200° lens.
• Default grid: 3×3 lens-local views at ±60° with 75° crops → 9 per lens, 18 per frame pair. The down-pitched views put the nadir ~30° from their axes, so the ground is covered by up to 6 views per frame pair instead of 0.
• Insta360 Studio is no longer needed: upload the .insv directly.

What's in here

• fisheye_projection.py — equidistant fisheye model (+ optional Kannala-Brandt k1–k4), MEI (unified omnidirectional) model for the factory calibration, lens-local view grids, rig rotations with per-lens mounting corrections, cv2.remap grid construction. Pure numpy/scipy, fully unit-tested.
• insv_calibration.py — Insta360 per-unit factory calibration parsing: a minimal protobuf wire walker (no protobuf dependency) reads the trailer metadata record's offset_v3 (20 fields/lens, layout per telemetry-parser) and the .insv.pb sidecar's extended calibration (X5; 27 fields/lens incl. k4 + thin-prism, layout validated by insv-stitch against in-camera stitching), then rescales from the per-model reference resolution to the demuxed stream (window_crop_info-aware, centered aspect-fit fallback). Best-effort: any failure falls back to the idealized model.
• insv_extract.py — ffmpeg demux of the three known .insv layouts (dual-stream single file; two-file _00_/_10_ pairs; side-by-side single stream with a dark-corner guard against mis-feeding ER video). Best-effort trailer parsing (ExifTool/Sub-Etha layout).
• fisheye_sfm.py — renders the views with two mask sets (SfM masks = lens-validity ∩ closest-view partition, to avoid duplicate features in overlaps; training masks = validity only, so gsplat doesn't train on black corners), builds the 18-camera two-lens rig, runs the shared SfM. Lens intrinsics precedence: explicit --insv_calibration JSON > factory calibration > idealized model.
• pano_sfm.py refactor — feature extraction → rig config → matching → mapping extracted into run_rig_sfm_pipeline(), shared by both pipelines. No behavior change to the ER path.
• vid2scene.py — --insv_fisheye (auto-enabled for .insv inputs), --insv_lens_fov, --insv_calibration, --insv_no_factory_calibration. Image budget matches the ER path (~89 frame pairs × 18 ≈ 1,600 images at the 800 default).
• Tests — 51 unit tests (vid2scene_core/tests/): projection round-trips with distortion, MEI projection math, nadir-coverage of the view grid, remap validity, trailer record walking, protobuf wire decoding, both factory-calibration layouts, X4/X5 reference-to-stream scaling, companion-file detection.
• docs/insv_fisheye.md — usage, calibration resolution order, factory-calibration sources, JSON schema, limitations.

Validated

• All 51 unit tests pass.
• End-to-end smoke test on a synthetic dual-stream file with pycolmap 3.13.0 (the version the worker pins): demux → render → feature extraction → rig config → sequential matching → mapping all execute; 18 cameras / 1 rig / N frames land correctly in the COLMAP database.
• Mask conventions verified empirically against pycolmap 3.13: masks are found under both name.png and name.png.png, and an image without a mask is skipped with MASK_ERROR once mask_path is set — the processor therefore writes a mask for every rendered image.
• The refactored ER path was smoke-tested end-to-end (renders → SfM steps run unchanged).
• Factory-calibration parsing is exercised against synthetic data matching the community-documented layouts (telemetry-parser's prost definitions; insv-stitch's X5 findings, which were validated against in-camera stitching output).

Needs validation on real footage (why this is a draft)

• Factory calibration on real recordings. Parsing follows community-documented layouts but hasn't run against a real X4/X5 file yet. Specifically to confirm: the reference-resolution scaling on models other than X4/X5, and the sign convention of the sub-degree mounting corrections. --insv_no_factory_calibration gives an immediate A/B fallback to the idealized model (principal point at center, inscribed circle, 200° FOV).
• Rear-lens mounting uses the factory yaw/pitch/roll corrections when available, otherwise exactly 180° yaw / 0° roll (overridable in the calibration JSON).
• Lens baseline (~2–3 cm) is ignored, same as the ER path; the factory per-lens translation is parsed but not applied (observed as zeros in community dumps; a metric value would also pin reconstruction scale, which needs deliberate handling).
• No IMU use yet (horizon leveling, rolling-shutter correction), no SAM3 ego-masking on the fisheye path, no GPS extraction from the trailer.
• Server upload flow only exposes equirectangular; .insv uploads through the web UI need a form field + pass-through (auto-detection already works at the pipeline level). Same for the cog/Modal wrapper (separate repo).

Suggested A/B test

Same Bernal .insv clip three ways: (a) ER path as-is, (b) ER path with training_max_num_gaussians=3M, (c) this path. If (c) fills in the ground where (a)/(b) don't, the input geometry was the bottleneck, as suspected. With factory calibration now in, (c) can additionally be run with --insv_no_factory_calibration to isolate how much the calibration itself contributes.

🤖 Generated with Claude Code

[kfarr]

Downstream wiring to make this testable through the 3DStreet app (generator#splat → "360 → Splat (.insv)") is now in draft PRs: vid2scene-cog#1 (insv_fisheye pass-through + image pin cog-phase2 = this PR's head) and 3dstreet#1673 (model entry + splat-tab UI). Deploy order and the real-footage test plan are in both PR descriptions.