Merge main into new API

.github/workflows/ci.yml

@@ -222,7 +222,7 @@ jobs:
           open4d/*.py open4d/streaming/*.py open4d/codec open4d/core open4d/io
           open4d/torch_ops open4d/visualization
           integrations/__init__.py integrations/open3d
-          examples/visualization scripts
+          examples/visualization examples/streaming_demo.py scripts
       - name: Check shell syntax
         run: bash -n scripts/*.sh
       - name: Check local Markdown links

README.md

@@ -17,13 +17,18 @@ the extras you use:
 
 ```bash
 python -m pip install -e '.[player]'    # interactive mesh viewer and GIF export
-python -m pip install -e '.[open3d]'    # RGB-D reconstruction
+python -m pip install -e '.[open3d]'    # RGB-D reconstruction (Open3D 0.19.x)
 python -m pip install -e '.[gaussians]' # read Gaussian PLY files
 ```
 
 Research methods have additional setup below. Their source, native programs
 and model weights are not bundled in the Python wheel.
 
+RGB-D reconstruction requires Open3D 0.19.x. The legacy TSDF integrator in
+Open3D 0.20 rescales already-metric float depth and can return empty meshes;
+the extra selects the supported version and reconstruction rejects an
+incompatible manually installed runtime before processing frames.
+
 ## Try a sequence
 
 No dataset is needed for this example:
@@ -151,10 +156,10 @@ with receive() as frames:
 Then send from another:
 
 ```python
-from open4d import stream
+from open4d import send
 from open4d.demo import mesh_sequence
 
-stream(mesh_sequence(frames=30))
+send(mesh_sequence(frames=30))
 ```
 
 This sends decoded mesh arrays over TCP, at their recorded frame timing. It is

THIRD_PARTY.md

@@ -51,7 +51,7 @@ distribution and still needs separate review before any redistribution.
 | `open4d/reconstruction/queen` | Imported research tree including MiDaS, SIBR, and rasterizers | Top-level NVIDIA non-commercial license plus multiple subtree licenses | `BLOCK`; record revisions, patches, all notices, model/data rights, and redistribution limits |
 | `open4d/reconstruction/gs_tools` | Consolidated Gaussian tooling tree containing SIBR viewers, GLM, and three rasterizer imports; one immutable upstream/patch manifest is absent | Component-local SIBR, GLM, and rasterizer license files exist with differing terms | `BLOCK`; inventory exact upstream revisions and patches, preserve every notice, and determine compatible source/binary distribution terms |
 | `open4d/reconstruction/vega` | Copied from `4DVideoStreaming` `baselines/Vega`, working tree above commit `6c2569de85ebe4592a8294c8cb0268efd3659212`, so not a clean revision: it carries three uncommitted modifications (`vega/encoder.py`, `vega/gov.py`, `orbitvega/prepare.py`) and two never-committed files (`orbitvega/scene_export.py`, `orbitvega/scene_render.py`). Local re-implementation of Vega (Kim et al., ACM MobiCom '25, `10.1145/3680207.3765267`); no upstream code release exists. Open4D modifications: import roots rerooted off `baselines.Vega`, `pytest.ini` added | No component license file; `citation.txt` records the paper only, and `vega_engine_pyproject.toml` names it a research prototype | `BLOCK`; identify the implementation's copyright holder and distribution terms, and pin an immutable source revision |
-| `open4d/reconstruction/nevo` | Copied from `4DVideoStreaming` `baselines/NeVo` at its introducing commit `6c2569de85ebe4592a8294c8cb0268efd3659212` plus one uncommitted modification (`orbitnevo/train.py`). NeVo (Wu et al., ACM MobiCom '25, `10.1145/3680207.3723473`) has no released code, so the simulator is local; it vendors `https://github.com/aoliao12138/ReRF` (CVPR 2023) under `rerf/`, **cloned at depth 1 so no upstream revision is recorded**, byte-identical apart from four non-code deletions listed in `rerf/PATCHES.md`. Open4D modifications: import roots rerooted off `baselines.NeVo`, `MODULE_ROOT` repointed, `pytest.ini` and `orbitnevo/objects.py` added | `rerf/LICENSE` is GPL-3.0 carrying a research-purposes-only rider, and its code base derives from DVGO; no top-level component license | `BLOCK`; recover and record the exact ReRF revision, resolve the GPL-3.0 plus research-only boundary and the DVGO lineage, and identify terms for the local simulator |
+| `open4d/reconstruction/rerf` | `rerf_stream/` is project-authored. `upstream/` is a clone of `https://github.com/aoliao12138/ReRF` (Wang et al., CVPR 2023) at `510e6073827ff15f58efda17a8729f2c47e13401` (2023-07-24), byte-identical to that tree apart from four non-code deletions enumerated in the module README (`.git/`, `ac_dc/` CMake intermediates, a README image); upstream carries no Open4D patches, all local adaptation living in `rerf_stream/env.py`. Supersedes the removed `open4d/reconstruction/nevo`, which vendored the same ReRF code at an unrecorded depth-1 revision and remains in history at `3d33655`. `upstream/ac_dc/` ships five prebuilt `.so` modules and an `ncvv_code` binary with no sources, only CMake residue, over a pybind11 checkout whose source is likewise absent | `upstream/LICENSE` is GPL-3.0 prefixed with a `THIS CODE CAN ONLY BE USED FOR RESEARCH PURPOSES` rider that contradicts the GPL grant; the code base derives from DVGO, whose kernels are `upstream/lib/cuda/`. No component license covers `rerf_stream/` beyond root MIT | `BLOCK`; resolve the GPL-3.0 against research-only conflict and the DVGO lineage, obtain reproducible sources and a dependency bill for the `ac_dc` binaries or exclude them from every artifact, and record terms for the project-authored `rerf_stream/` layer |
 | `integrations/unity` | Project glue, copied Eigen, prebuilt plugins, encoded archive | No integration-wide manifest; Eigen has multiple license files; binaries/data unresolved | `BLOCK`; inventory producers, licenses, build revisions, symbols, and fixture rights |
 | `integrations/unity/TVMCUnity/Unity Files/Plugins` | Prebuilt Android `.so` and macOS `.dylib` | Reproducible build and dependency bill absent | `BLOCK`; exclude until reproduced from reviewed source with notices |
 | `integrations/unity/TVMCUnity/EncodedExample/DanceSequence.zip` | Historical encoded dataset archive | Source dataset license/consent absent | `BLOCK`; identify redistribution permission or replace with a licensed synthetic fixture |

docs/api.md

@@ -0,0 +1,137 @@
+# Python API
+
+The public API loads, saves, unloads, and visualizes whole finite triangle-mesh
+sequences independently of their storage format:
+
+```python
+import open4d
+
+with open4d.load("capture.usdc") as sequence:
+    open4d.save(sequence, "capture.o4d")
+    open4d.visualize(sequence)
+
+# A path can go straight to the lazy viewer; it is closed when the window exits.
+open4d.visualize("capture.o4d")
+```
+
+`.usd`, `.usda`, `.usdc`, and `.usdz` are OpenUSD interchange containers.
+`.o4d` and the registered codec suffixes are codec artifacts. Both carry whole
+sequences and use the same `Sequence` interface. `open4d.unload(sequence)` is an
+explicit, idempotent alternative to the context manager.
+
+Frame folders and individual meshes remain supported as import paths; frames
+are decoded on access:
+
+```python
+from open4d.io import open_sequence
+
+with open_sequence("path/to/frames", fps=30.0) as sequence:
+    print(len(sequence), sequence.duration, sequence.fps)
+    mesh = sequence[0].geometry          # TriangleMesh: positions, triangles
+```
+
+## Representations
+
+What a decoded frame *is* is deliberately separate from the codec that produced
+it: a triangle mesh is a mesh whether it arrived as OBJ, as a Draco payload, or
+out of a V-DMC bitstream. `open4d.Representation` is the axis to gate on, and
+`MESH`, `POINTS`, and `GAUSSIANS` have concrete types — `TriangleMesh`,
+`PointCloud`, and `GaussianCloud`. Already-rendered pixels are named in the
+taxonomy but have no concrete type yet; they land with the camera model they
+need in order to be comparable at a known pose.
+
+The containers and codecs described below are the mesh path, and the one most
+completely covered. Check a specific codec before assuming it round-trips
+points or Gaussians.
+
+## Writing sequences
+
+`write_sequence(sequence, "frames/", format="ply")` writes a versioned
+`open4d.sequence.json` beside the frame files, so reopening the directory keeps
+source frame indices, timestamps, frame/sequence metadata, and topology
+declarations. Empty sequences are rejected before the destination is changed.
+
+Single mesh-file exports require `allow_lossy=True` because that storage cannot
+preserve sequence timing, metadata, or topology declarations. Trimesh-backed
+OFF/GLB/glTF color export also requires that opt-in because OFF drops vertex
+color and GLB/glTF quantize canonical float colors to eight bits.
+
+OpenUSD is the public interchange container. `--pack-usd out.usdc` packs any
+source into one compressed `.usdc` file carrying the frame rate, the key-frame
+index, and per-frame streams alongside the geometry — see the
+[visualization guide](../examples/visualization/README.md#the-openusd-container).
+
+## Streaming
+
+`open4d.stream` exports a file as a bundle and serves it to a browser:
+
+```python
+import open4d
+
+open4d.stream("capture.usdc")
+open4d.stream("capture.usdc", rungs=["draco", "draco@11"], out_dir="bundle/")
+```
+
+For a loaded sequence, pass browser options such as `name="capture"` and
+`out_dir="bundle/"`. `open4d.send(sequence, host, port)` explicitly selects
+decoded-mesh TCP transport and pairs with `open4d.receive`. Existing
+`open4d.stream(sequence, host, port)` calls, and frame iterables without browser
+options, retain that TCP behavior without requiring `open4d-streamer`.
+
+`rungs` is the quality ladder. The first is the rendition a client plays by
+default and the rest are what it can switch to mid-playback, so a one-entry
+list is a fixed-quality stream and says so. A spec is a frame format —
+`ply` for interchange, `draco` for delivery — optionally with a position
+quantisation, as in `draco@11`. Sizes are measured off disk rather than
+predicted; quality is left unscored until something scores it.
+
+[`examples/streaming_demo.py`](../examples/streaming_demo.py) runs the whole
+of it on the ten basketball frames the TVMC codec vendors: three rungs
+built and scored, served over HTTP with the counters read back, then thirty
+seconds simulated over a link that collapses mid-run.
+
+The implementation is the separate `open4d-streamer` package, imported on the
+call rather than at load: it depends on `open4d`, so `open4d` must not depend
+on it. Without it installed the call raises `open4d.StreamerDependencyError`
+saying how to install it. For several clips in one bundle, a constrained link,
+or the delivered-quality metrics, use that package directly —
+`streamer.Bundle`, `streamer.Link`, `streamer.serve`.
+
+Separately, [`open4d/webclients`](../open4d/webclients) is the vendored
+browser-client research tree that compares five delivery systems against each
+other. It is not this API and shares no code with it.
+
+## Codecs
+
+Five lossless, in-process reference codecs are included: `raw`, `deflate`,
+`bzip2`, `lzma`, and byte-level `rle` (`npz` remains the default DEFLATE alias).
+They share a safe NumPy-array container so they compare storage strategies, not
+research geometry models.
+
+Source checkouts register in-process adapters for `klt`, `n4mc`, `qndf`, and
+`qndf-int8`; the lightweight wheel omits them until their provenance review is
+complete. Open4D's separate `temporal-delta` and `temporal-pca` experiments are
+not the repository's TVMC or TSMC pipelines. The V-DMC adapters do not execute
+shell scripts, but they do invoke configured native encoder and decoder
+processes once per sequence. Callers can also register another
+`open4d.codec.Codec`.
+
+For an all-registered-codec attempt using `4d_files/Rafa_Approves_hd_4k`, open
+[`examples/open4d_sequence_codec.ipynb`](../examples/open4d_sequence_codec.ipynb).
+Set `OPEN4D_NOTEBOOK_REQUIRE_ALL=1` in a fully provisioned environment to make
+any codec failure stop the notebook instead of appearing only in its result
+table.
+
+### Device selection
+
+The N4MC and QNDF adapters accept `device="auto"` (CUDA, then Apple Metal/MPS,
+then CPU), or an explicit `"cuda"`, `"mps"`, or `"cpu"`. QNDF-int8 can train on
+CUDA or Metal, but its quantized decoder remains CPU-only. Override the notebook
+selection with `OPEN4D_NOTEBOOK_DEVICE=mps` when needed.
+
+### Test coverage
+
+Normal CI runs dependency-complete CPU encode/fresh-decode contracts for KLT,
+N4MC, QNDF, and QNDF-int8. The larger two-format Rafa quality/export matrix is
+an additional CUDA acceptance test gated by `OPEN4D_TEST_RESEARCH_CODECS=1` and
+`OPEN4D_RAFA_DATASET`; it is not presented as part of ordinary CI coverage.

docs/components.md

@@ -0,0 +1,37 @@
+# Components
+
+Each component has its own README and may add native tools, GPU extensions, or
+hardware requirements to the shared Python baseline. See
+[Requirements](requirements.md) for those additions.
+
+## Mesh codecs
+
+| Codec | |
+|---|---|
+| [**N4MC**](../open4d/codecs/n4mc/README.md) | Neural TSDF-based mesh compression, including a newer modular codec under its `data`, `models`, `losses`, `training`, and `evaluation` packages |
+| [**QNDF**](../open4d/codecs/qndf/README.md) | Quantized Neural Displacement Fields: static mesh compression using an SSP coarse mesh and an implicit displacement decoder. [`qndf_int8`](../open4d/codecs/qndf_int8/README.md) is its quantized variant |
+| [**TVMC**](../open4d/codecs/tvmc/README.md) | A Python, .NET, and Draco pipeline for tracked time-varying mesh compression, with setup and resumable pipeline scripts |
+| [**TSMC**](../open4d/codecs/tsmc/README.md) | Scene-mesh compression with optional SAM-based static/dynamic separation, ARAP volume tracking, deformation, displacement compression, and evaluation |
+| [**KLT**](../open4d/codecs/klt/README.md) | Karhunen–Loève Transform baseline that compresses TSDF voxel blocks with a learned linear basis and quantized coefficients, reconstructing meshes via marching cubes |
+| [**Draco**](../open4d/codecs/draco/README.md) | Google Draco mesh-compression baseline. Wraps the vendored `draco_encoder`/`draco_decoder` binaries into a per-frame encode/decode/eval pipeline for benchmarking against the neural codecs |
+| [**MPEG V-DMC test model**](../open4d/codecs/vdmc/README.md) | The pinned MPEG reference implementation for video-based dynamic mesh coding — reference encoder, decoder, metric tools, and unit tests. Separate from Open4D's TVMC research pipeline |
+| [**Faster V-DMC**](../open4d/codecs/faster_vdmc/README.md) | A pinned performance-oriented fork of the same test model, with exact-output and higher-throughput modes recorded in the [benchmark report](benchmarks/faster-vdmc.md) |
+
+## Reconstruction and streaming
+
+| Module | |
+|---|---|
+| [**RGB-D**](../open4d/streaming/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline |
+| [**QUEEN**](../open4d/reconstruction/queen/README.md) | Quantized efficient encoding of dynamic Gaussians for streaming free-viewpoint video (NeurIPS 2024) |
+| [**3DGStream**](../open4d/reconstruction/3dgstream/README.md) | On-the-fly training of 3D Gaussians for streaming photo-realistic free-viewpoint video (CVPR 2024) |
+| [**Vega**](../open4d/reconstruction/vega/README.md) | An ORBIT adaptation of Vega (MobiCom 2025): mobile volumetric video streaming with 3D Gaussian splatting |
+| [**ReRF**](../open4d/reconstruction/rerf/README.md) | Neural residual radiance fields (CVPR 2023) as a streamable compression method |
+| [**gs-tools**](../open4d/reconstruction/gs_tools/README.md) | The one environment, rasterizers, and viewer the Gaussian methods share, plus `gs-tools view` for putting several methods under one camera |
+| [**streamer**](../open4d/streamer/README.md) | Streaming and playback for 4D reconstructions, whatever their representation. Producers import it to describe and serve their output; it imports none of them |
+
+## Integrations
+
+| | |
+|---|---|
+| [**Open3D**](../integrations/open3d/README.md) | Converts decoded Open4D geometry into standard Open3D `TriangleMesh` or `PointCloud` objects. An adapter, not a loader |
+| [**Unity**](../integrations/unity/README.md) | A Unity playback system for TVMC-encoded sequences: a C++ decoder backend and a C# front end, for XR targets |