Move rgbd under reconstruction, transport to core, study under streamer

.github/workflows/ci.yml

@@ -70,7 +70,7 @@ jobs:
       - name: Run every Open3D-backed test
         run: >-
           python -m pytest -m open3d
-          integrations/open3d/tests open4d/torch_ops/tests open4d/streaming/tests
+          integrations/open3d/tests open4d/torch_ops/tests open4d/reconstruction/rgbd/tests
           open4d/codec/tests/test_notebook.py
 
   research-codecs:
@@ -219,7 +219,8 @@ jobs:
       - name: Compile supported Python sources
         run: >-
           python -m compileall -q
-          open4d/*.py open4d/streaming/*.py open4d/codec open4d/core open4d/io
+          open4d/*.py open4d/reconstruction/*.py open4d/reconstruction/rgbd/*.py
+          open4d/transport open4d/codec open4d/core open4d/io
           open4d/torch_ops open4d/visualization
           integrations/__init__.py integrations/open3d
           examples/visualization examples/streaming_demo.py scripts

CONTRIBUTING.md

@@ -18,15 +18,16 @@ For research and reconstruction tests on Python 3.12:
 ```bash
 python -m pip install -e '.[dev,klt,n4mc,qndf,open3d,gaussians]'
 python -m pytest open4d/codec/tests/test_research_cpu.py -m 'not gpu'
-python -m pytest open4d/streaming/tests integrations/open3d/tests
+python -m pytest open4d/reconstruction/rgbd/tests integrations/open3d/tests
 ```
 
 ## Code
 
 - Keep the base install NumPy-only and import optional dependencies when used.
 - Put public mesh codec adapters in `open4d/codec`, Gaussian APIs in
-  `open4d/gaussians.py`, and transport/reconstruction in `open4d/streaming`.
-- Preserve the research implementations under `open4d/codecs` and
+  `open4d/gaussians.py`, RGB-D reconstruction in `open4d/reconstruction/rgbd`,
+  and TCP frame transport in `open4d/transport`.
+- Preserve the research implementations under `open4d/codecs` and the rest of
   `open4d/reconstruction`. Connect them through small adapters.
 - A decoded artifact must work without the original input or hidden encoder
   files. Keep native backends separate from the installed Python package.
@@ -38,7 +39,7 @@ python -m pytest open4d/streaming/tests integrations/open3d/tests
 
 ```bash
 python -m pytest
-python -m compileall -q open4d/*.py open4d/streaming/*.py open4d/codec open4d/core open4d/io open4d/torch_ops open4d/visualization integrations/__init__.py integrations/open3d examples/visualization scripts
+python -m compileall -q open4d/*.py open4d/reconstruction/*.py open4d/reconstruction/rgbd/*.py open4d/transport open4d/codec open4d/core open4d/io open4d/torch_ops open4d/visualization integrations/__init__.py integrations/open3d examples/visualization scripts
 python scripts/check_markdown_links.py
 python scripts/check_provenance.py
 python scripts/check_release_gate.py --expect-blocked

MANIFEST.in

@@ -25,9 +25,12 @@ global-exclude __pycache__
 global-exclude *.py[cod]
 global-exclude .DS_Store
 
-recursive-exclude open4d/streaming/tests *
-prune open4d/streaming/app
-prune open4d/streaming/src
-prune open4d/streaming/include
-prune open4d/streaming/python
-prune open4d/streaming/tools
+include open4d/reconstruction/__init__.py
+recursive-include open4d/reconstruction/rgbd *.py
+recursive-exclude open4d/reconstruction/rgbd/tests *
+prune open4d/reconstruction/rgbd/app
+prune open4d/reconstruction/rgbd/src
+prune open4d/reconstruction/rgbd/include
+prune open4d/reconstruction/rgbd/python
+prune open4d/reconstruction/rgbd/tools
+recursive-exclude open4d/transport/tests *

README.md

@@ -169,18 +169,18 @@ network or SSH tunnel; this protocol has no authentication or encryption.
 Use `realtime=False` to transfer a recorded sequence as fast as possible.
 
 The camera capture and native reconstruction programs are in
-[open4d/streaming](open4d/streaming/README.md), formerly `reconstruction/rgbd`.
+[open4d/reconstruction/rgbd](open4d/reconstruction/rgbd/README.md).
 
 ## Compare methods in a user study
 
-[open4d/webclients](open4d/webclients/README.md) is a browser app that runs one
+[open4d/streamer/study](open4d/streamer/study/README.md) is a browser app that runs one
 participant through several streaming methods — Ours, ViVo, NAVA and Vega —
 under the same network trace, from the same start view, along the same camera
 path, and asks them to rate each clip. The trace can be uploaded in the browser;
 the results export as CSV.
 
 ```bash
-cd open4d/webclients/system/WebClient && npm install && node build.js
+cd open4d/streamer/study/system/WebClient && npm install && node build.js
 cd ../.. && PYTHONPATH=/path/to/4DVideoStreaming PYTHON_BIN=<env-python> scripts/run_web_demo.sh
 # then open http://<host>:3000/web/
 ```

THIRD_PARTY.md

@@ -21,7 +21,7 @@ all required notices to be assembled, and a recorded maintainer approval.
 
 | Package | Basis | Candidate decision |
 | --- | --- | --- |
-| `open4d`, `open4d.codec`, `open4d.core`, `open4d.io`, `open4d.visualization`, `open4d.streaming` | Project-authored, root MIT; NumPy base | Candidate after final review; research implementations remain excluded |
+| `open4d`, `open4d.codec`, `open4d.core`, `open4d.io`, `open4d.visualization`, `open4d.reconstruction.rgbd`, `open4d.transport` | Project-authored, root MIT; NumPy base | Candidate after final review; research implementations remain excluded |
 | `open4d.torch_ops` | Project-authored source; Torch optional and not redistributed | Candidate after final review |
 | `integrations`, `integrations.open3d` | Project-authored adapter source; Open3D optional and not redistributed | Candidate after final review |
 
@@ -46,7 +46,7 @@ distribution and still needs separate review before any redistribution.
 | `open4d/codecs/tsmc` | Research pipeline, copied trees, Draco `47238930f698250f474163e1a29d77858aa5c158`, SAM3 `5dd401d1c5c1d5c3eedff06d41b77af824517619`, tracked datasets/paper. Local change: entropy coding supports constant displacement matrices and rejects values outside the int32/ANS range | No top-level component license; copied subtrees have separate terms | `BLOCK`; inventory source, patches, paper/media, data rights, SAM3 terms, and notices |
 | `open4d/codecs/vdmc` | MPEG reference submodule `ecffe4212e5e956761c4fa14a17c453ae916b0b1` | Not audited in the uninitialized tree | `BLOCK`; initialize, record license/notices, and review intended source/binary distribution |
 | `open4d/codecs/faster_vdmc` | Fork `cicm4/mpeg-vdmc-tm` at `93cdd5e1367b0f9f81c251ef89255bcb2f0d2d3f`, derived from `MPEGGroup/mpeg-vdmc-tm` at `ecffe4212e5e956761c4fa14a17c453ae916b0b1` with eight encoder/decoder performance commits | Fork `COPYING` contains the ISO/IEC BSD 3-Clause terms and expressly excludes patent and other third-party rights; build-fetched dependencies need separate review | `BLOCK`; preserve the fork history and modification record, inventory dependency licenses/notices and patent implications, and review source/binary distribution before release |
-| `open4d/streaming` | Project reconstruction and protocol experiments | Root MIT is current repository evidence; camera SDK/native dependencies external | `EXCLUDED` for native/capture code; project-authored top-level Python API only is in the candidate package; camera/native dependency review remains open |
+| `open4d/reconstruction/rgbd` | Project reconstruction and protocol experiments | Root MIT is current repository evidence; camera SDK/native dependencies external | `EXCLUDED` for native/capture code; project-authored top-level Python API only is in the candidate package; camera/native dependency review remains open |
 | `open4d/reconstruction/3dgstream` | Imported research tree; consolidated upstream revision/patch list absent | Top-level MIT plus separate SIBR/rasterizer/Gaussian terms | `BLOCK`; record all upstream revisions, patches, subtree licenses, data/model rights, and allowed artifact boundary |
 | `open4d/reconstruction/queen` | Imported research tree including MiDaS, SIBR, and rasterizers | Top-level NVIDIA non-commercial license plus multiple subtree licenses | `BLOCK`; record revisions, patches, all notices, model/data rights, and redistribution limits |
 | `open4d/reconstruction/gs_tools` | Consolidated Gaussian tooling tree containing SIBR viewers, GLM, and three rasterizer imports; one immutable upstream/patch manifest is absent | Component-local SIBR, GLM, and rasterizer license files exist with differing terms | `BLOCK`; inventory exact upstream revisions and patches, preserve every notice, and determine compatible source/binary distribution terms |

docs/api.md

@@ -97,7 +97,7 @@ saying how to install it. For several clips in one bundle, a constrained link,
 or the delivered-quality metrics, use that package directly —
 `streamer.Bundle`, `streamer.Link`, `streamer.serve`.
 
-Separately, [`open4d/webclients`](../open4d/webclients) is the vendored
+Separately, [`open4d/streamer/study`](../open4d/streamer/study) is the vendored
 browser-client research tree that compares five delivery systems against each
 other. It is not this API and shares no code with it.
 

docs/components.md

@@ -21,14 +21,14 @@ hardware requirements to the shared Python baseline. See
 
 | Module | |
 |---|---|
-| [**RGB-D**](../open4d/streaming/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline |
+| [**RGB-D**](../open4d/reconstruction/rgbd/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline |
 | [**QUEEN**](../open4d/reconstruction/queen/README.md) | Quantized efficient encoding of dynamic Gaussians for streaming free-viewpoint video (NeurIPS 2024) |
 | [**3DGStream**](../open4d/reconstruction/3dgstream/README.md) | On-the-fly training of 3D Gaussians for streaming photo-realistic free-viewpoint video (CVPR 2024) |
 | [**Vega**](../open4d/reconstruction/vega/README.md) | An ORBIT adaptation of Vega (MobiCom 2025): mobile volumetric video streaming with 3D Gaussian splatting |
 | [**ReRF**](../open4d/reconstruction/rerf/README.md) | Neural residual radiance fields (CVPR 2023) as a streamable compression method |
 | [**gs-tools**](../open4d/reconstruction/gs_tools/README.md) | The one environment, rasterizers, and viewer the Gaussian methods share, plus `gs-tools view` for putting several methods under one camera |
 | [**streamer**](../open4d/streamer/README.md) | Streaming and playback for 4D reconstructions, whatever their representation. Producers import it to describe and serve their output; it imports none of them |
-| [**webclients**](../open4d/webclients/README.md) | A browser user study: one participant, several streaming methods, the same network trace and camera path for each, then a questionnaire and CSV export. Runs Ours, ViVo, NAVA and Vega; needs the 4DVideoStreaming research repo at run time |
+| [**streamer/study**](../open4d/streamer/study/README.md) | A browser user study: one participant, several streaming methods, the same network trace and camera path for each, then a questionnaire and CSV export. Runs Ours, ViVo, NAVA and Vega; needs the 4DVideoStreaming research repo at run time |
 
 ## Integrations
 

docs/requirements.md

@@ -136,5 +136,5 @@ with `Hardware MFT failed to start`. 5 synchronized pairs/s held over Wi-Fi and
 VPN; 15 did not.
 
 Calibration layout and the step-by-step session walkthrough are in
-[`open4d/streaming/README.md`](../open4d/streaming/README.md),
+[`open4d/reconstruction/rgbd/README.md`](../open4d/reconstruction/rgbd/README.md),
 which covers how to run the pipeline and leaves requirements to this page.

open4d/__init__.py

@@ -26,7 +26,7 @@
 from ._api import load, save, unload, reconstruct, stream
 from ._streamer import StreamerDependencyError
 from .codec import available_codecs, decode_sequence as decode, encode_sequence as encode
-from .streaming import receive, send
+from .transport import receive, send
 from .gaussians import GaussianSplats, GaussianRun, NeuralGaussianFrame, load_gaussians
 from .metrics import compare_meshes, compare_sequences
 from .visualization import visualize

open4d/_api.py

@@ -153,7 +153,7 @@ def reconstruct(source, output=None, *, method="rgbd", **options):
     if method == "rgbd":
         if output is not None:
             raise TypeError("RGB-D reconstruction returns a Sequence; omit output")
-        from .streaming import reconstruct as reconstruct_rgbd
+        from .reconstruction.rgbd import reconstruct as reconstruct_rgbd
 
         return reconstruct_rgbd(source, **options)
     if method in ("queen", "3dgstream"):
@@ -176,7 +176,7 @@ def stream(source, *address, **options):
     browser_options = {"out_dir", "name", "title", "rungs", "fps", "open_browser", "block"}
     if address or (not isinstance(source, (str, os.PathLike))
                    and not browser_options.intersection(options)):
-        from .streaming import send
+        from .transport import send
 
         return send(source, *address, **options)
     from ._streamer import stream as stream_to_browser

open4d/reconstruction/__init__.py

@@ -0,0 +1 @@
+"""Reconstruction methods. Only rgbd is part of the package; the rest is vendored research code."""

open4d/reconstruction/rgbd/README.md

@@ -0,0 +1,132 @@
+# RGB-D reconstruction
+
+Two synchronized RGB-D cameras send frames to an Ubuntu GPU machine. That
+machine fuses them into a live point cloud and CUDA meshes, and you watch the
+result in a browser. This module was formerly called `MeshReduce`.
+
+```text
+2 cameras -> Windows capture host -> SSH tunnel -> Ubuntu receiver (fuse + CUDA TSDF) -> Open3D WebRTC -> browser
+```
+
+The point cloud updates on every camera pair; the mesh is rebuilt
+periodically. From Python, `open4d.reconstruct(source, method="rgbd")` runs
+the same reconstruction on saved captures.
+
+## Setup
+
+Hardware and the tested configuration are in the [repository README](../../../README.md).
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+python -m pip install numpy opencv-python zstandard
+# plus a CUDA-enabled Open3D build (the pip wheel may not include CUDA)
+python -c "import open3d as o3d; print(o3d.__version__, o3d.core.cuda.is_available())"   # want True
+```
+
+**Calibration.** Camera 1 is the reference frame. Export each camera's
+factory calibration, calibrate camera 2 against camera 1, and lay out the files
+like this (`EY` = camera 1 and `J3` = camera 2, legacy names). This module only
+reads calibration; it doesn't create it. Recalibrate if you move either camera.
+
+```text
+<calibration-dir>/
+├── source/work/calibration_stepwise/factory/{ey,j3}_factory_calibration.json
+└── final_validated_fusion/j3_depth_to_ey_depth_refined.txt
+```
+
+## Run a live session
+
+Start these in order.
+
+**1. Ubuntu receiver**, run from this directory:
+
+```bash
+export FOURD_CALIBRATION_DIR=/abs/path/to/calibration
+export PYTHON=/path/to/python-with-open3d
+export FOURD_CUDA_DEVICE=0
+export FOURD_CAMERA1_SERIAL=... FOURD_CAMERA2_SERIAL=...
+DISPLAY_MODE=pointcloud MESH_WINDOW=1 ./tools/run_browser_viewer.sh
+# -> listening on 127.0.0.1:17000, viewer on 127.0.0.1:8888
+```
+
+**2. Windows data tunnel.** Close Orbbec Viewer first.
+
+```powershell
+ssh -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -N `
+  -L 127.0.0.1:17000:127.0.0.1:17000 user@gpu-host
+Test-NetConnection 127.0.0.1 -Port 17000     # TcpTestSucceeded : True
+```
+
+**3. Windows camera sender.** Use any sender that speaks OBP1 (packet format
+in `python/protocol.py`). The tested sender is a separate companion script, so
+copy it to the capture host and set your camera serials in it.
+
+```powershell
+& C:\path\to\python.exe C:\path\to\windows_sender.py `
+  --sdk-bin "C:\path\to\OrbbecSDK-K4A-Wrapper\bin" `
+  --host 127.0.0.1 --port 17000 --fps 5 --report sender_report.json
+```
+
+Keep the default 5 FPS: at higher rates the receiver can't prepare point clouds
+fast enough and replaces frames. The Ubuntu log should show `Sender connected`
+and a rising `processed_pairs`.
+
+**4. Browser tunnel**, from the machine you're watching on:
+
+```bash
+ssh -N -o ExitOnForwardFailure=yes -L 127.0.0.1:18888:127.0.0.1:8888 user@gpu-host
+```
+
+Then open <http://127.0.0.1:18888/>. If the page loads but nothing moves, the
+sender isn't connected. To stop, press Ctrl+C in the sender first, then close
+the tunnels and the receiver. If everything runs on one machine, skip both
+tunnels and open <http://127.0.0.1:8888/>.
+
+## Options
+
+| Variable | Values |
+|---|---|
+| `DISPLAY_MODE` | `pointcloud` (motion), `mesh` (still scenes), `auto` (points until the first mesh), `both` (debugging) |
+| `MESH_WINDOW` | how many recent camera pairs go into each mesh: `1` for live, `7` for a still scene |
+| `MESH_FUSION_MODE` | `independent-merge` (default; one TSDF per camera, then merge) or `shared-tsdf` (one volume; usually cleaner) |
+| `MESH_MERGE_MODE` | `concatenate` (keep every triangle) or `weld` (merge vertices within `MESH_WELD_RADIUS`, metres) |
+
+Merging does no cropping, decimation or hole filling, so overlapping views can
+show doubled surfaces. The MeshReduce paper's overlap removal and stitching
+step isn't implemented yet.
+
+Output goes to `output/two-camera-fusion/`:
+`latest_live_full_scene_{pointcloud,mesh}.ply`, `live_fusion_report.json` and
+`live_browser.log`. A healthy mesh log shows `"backend": "CUDA:0"`.
+
+## Without the live cameras
+
+```bash
+# replay saved pairs through the live receiver (start step 1 first)
+"$PYTHON" tools/replay_obp1_sender.py --captures-root /abs/path/to/saved-pairs --fps 2
+
+# bounded run from the remote sender, or one-shot reconstruction of a saved sequence
+MAX_PAIRS=30 ./tools/run_remote_two_camera_fusion.sh
+FOURD_CAPTURE_ROOT=/path/to/capture-data ./tools/reconstruct_saved_two_camera.py
+```
+
+## Native C++ pipeline
+
+For K4A-compatible cameras plugged directly into this machine, or recorded
+`.mkv` files. It writes PLY, textured OBJ, Draco and stage metrics.
+
+```bash
+sudo apt install -y build-essential cmake ninja-build git \
+  libopencv-dev libeigen3-dev libjsoncpp-dev libdraco-dev draco
+cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
+cmake --build build --target rgbd_streamer dual_camera_fusion -j"$(nproc)"
+./build/app/rgbd_streamer <config>.json        # one camera or .mkv
+./build/app/dual_camera_fusion config.dual.json
+ctest --test-dir build --output-on-failure
+```
+
+## Files
+
+- `python/protocol.py`: OBP1 packet definitions
+- `tools/receive_mesh_frame.py`, `tools/receive_live_stream.py`: reference receivers for MRD1/2 and MRD3
+- [`docs/artifacts.md`](../../../docs/artifacts.md): generated-data policy (outputs are git-ignored)

open4d/reconstruction/rgbd/__init__.py

@@ -0,0 +1,5 @@
+"""Reconstruct meshes from calibrated RGB-D captures."""
+
+from ._reconstruction import reconstruct
+
+__all__ = ["reconstruct"]

open4d/streaming/_reconstruction.py → open4d/reconstruction/rgbd/_reconstruction.py

@@ -7,7 +7,7 @@
 
 import numpy as np
 
-from ..core import Frame, Sequence, TopologyMode, TriangleMesh
+from ...core import Frame, Sequence, TopologyMode, TriangleMesh
 
 
 def _positive(value, name):