diff --git a/README.md b/README.md index 316f463d..0431e039 100644 --- a/README.md +++ b/README.md @@ -17,13 +17,18 @@ the extras you use: ```bash python -m pip install -e '.[player]' # interactive mesh viewer and GIF export -python -m pip install -e '.[open3d]' # RGB-D reconstruction +python -m pip install -e '.[open3d]' # RGB-D reconstruction (Open3D 0.19.x) python -m pip install -e '.[gaussians]' # read Gaussian PLY files ``` Research methods have additional setup below. Their source, native programs and model weights are not bundled in the Python wheel. +RGB-D reconstruction requires Open3D 0.19.x. The legacy TSDF integrator in +Open3D 0.20 rescales already-metric float depth and can return empty meshes; +the extra selects the supported version and reconstruction rejects an +incompatible manually installed runtime before processing frames. + ## Try a sequence No dataset is needed for this example: @@ -151,10 +156,10 @@ with receive() as frames: Then send from another: ```python -from open4d import stream +from open4d import send from open4d.demo import mesh_sequence -stream(mesh_sequence(frames=30)) +send(mesh_sequence(frames=30)) ``` This sends decoded mesh arrays over TCP, at their recorded frame timing. It is diff --git a/docs/api.md b/docs/api.md index c18e0fa5..e2e7ace2 100644 --- a/docs/api.md +++ b/docs/api.md @@ -63,7 +63,7 @@ index, and per-frame streams alongside the geometry — see the ## Streaming -`open4d.stream` exports a sequence as a bundle and serves it to a browser: +`open4d.stream` exports a file as a bundle and serves it to a browser: ```python import open4d @@ -72,6 +72,12 @@ open4d.stream("capture.usdc") open4d.stream("capture.usdc", rungs=["draco", "draco@11"], out_dir="bundle/") ``` +For a loaded sequence, pass browser options such as `name="capture"` and +`out_dir="bundle/"`. `open4d.send(sequence, host, port)` explicitly selects +decoded-mesh TCP transport and pairs with `open4d.receive`. Existing +`open4d.stream(sequence, host, port)` calls, and frame iterables without browser +options, retain that TCP behavior without requiring `open4d-streamer`. + `rungs` is the quality ladder. The first is the rendition a client plays by default and the rest are what it can switch to mid-playback, so a one-entry list is a fixed-quality stream and says so. A spec is a frame format — diff --git a/docs/components.md b/docs/components.md index 93947080..973316e1 100644 --- a/docs/components.md +++ b/docs/components.md @@ -21,7 +21,7 @@ hardware requirements to the shared Python baseline. See | Module | | |---|---| -| [**RGB-D**](../open4d/reconstruction/rgbd/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline | +| [**RGB-D**](../open4d/streaming/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline | | [**QUEEN**](../open4d/reconstruction/queen/README.md) | Quantized efficient encoding of dynamic Gaussians for streaming free-viewpoint video (NeurIPS 2024) | | [**3DGStream**](../open4d/reconstruction/3dgstream/README.md) | On-the-fly training of 3D Gaussians for streaming photo-realistic free-viewpoint video (CVPR 2024) | | [**Vega**](../open4d/reconstruction/vega/README.md) | An ORBIT adaptation of Vega (MobiCom 2025): mobile volumetric video streaming with 3D Gaussian splatting | diff --git a/docs/requirements.md b/docs/requirements.md index d40a92a9..a34b71ea 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -121,7 +121,7 @@ build. Those are optional and separate, with install commands in | `codecs/klt` | `kaolin` and an NVIDIA GPU; 24 GB is the same ceiling at resolution 128–256 | | `codecs/draco` | A CMake build of the vendored Draco submodule. Open3D, pymeshlab, and OpenCV are for evaluation only | | `codecs/vdmc`, `codecs/faster_vdmc` | The MPEG reference and optimized test models' own build requirements | -| `reconstruction/rgbd` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper | +| `streaming` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper | | `reconstruction/gs_tools`, `queen`, `3dgstream`, `vega` | The separate `open4d-gs` conda environment and five CUDA extensions built with `--no-build-isolation`, per [`gs_tools`](../open4d/reconstruction/gs_tools/README.md). Build on ext4; on an ntfs3 mount ninja deadlocks in `ntfs_file_write_iter` | | `reconstruction/rerf` | Python 3.8, because `ac_dc/ncvv_ac_dc.cpython-38-*.so` ships without sources and cannot be rebuilt for a newer interpreter. Plus torch with CUDA, mmcv, bitarray, Pillow, NumPy — a separate environment from every other module here | | `integrations/unity` | Unity, plus a C++ toolchain to rebuild the backend for anything other than the prebuilt macOS and Android/Quest 3 plugins | @@ -136,5 +136,5 @@ with `Hardware MFT failed to start`. 5 synchronized pairs/s held over Wi-Fi and VPN; 15 did not. Calibration layout and the step-by-step session walkthrough are in -[`open4d/reconstruction/rgbd/README.md`](../open4d/reconstruction/rgbd/README.md), +[`open4d/streaming/README.md`](../open4d/streaming/README.md), which covers how to run the pipeline and leaves requirements to this page. diff --git a/open4d/__init__.py b/open4d/__init__.py index d427e05d..895cf6b7 100644 --- a/open4d/__init__.py +++ b/open4d/__init__.py @@ -23,8 +23,12 @@ TriangleMesh, dtypes, ) -from ._api import load, save, unload -from ._streamer import StreamerDependencyError, stream +from ._api import load, save, unload, reconstruct, stream +from ._streamer import StreamerDependencyError +from .codec import available_codecs, decode_sequence as decode, encode_sequence as encode +from .streaming import receive, send +from .gaussians import GaussianSplats, GaussianRun, NeuralGaussianFrame, load_gaussians +from .metrics import compare_meshes, compare_sequences from .visualization import visualize __all__ = [ @@ -64,7 +68,7 @@ "receive", "stream", "save", - "stream", + "send", "unload", "visualize", ] diff --git a/open4d/_api.py b/open4d/_api.py index 21106ad4..27d40704 100644 --- a/open4d/_api.py +++ b/open4d/_api.py @@ -163,3 +163,22 @@ def reconstruct(source, output=None, *, method="rgbd", **options): return reconstruct_gaussians(source, output, method=method, **options) raise ValueError("method must be 'rgbd', 'queen' or '3dgstream'") + + +def stream(source, *address, **options): + """Send mesh frames over TCP or export a sequence for browser playback. + + A path, or browser options such as out_dir/name/rungs, selects the optional + browser streamer. A frame iterable without browser options retains the + original TCP behavior, including positional or keyword host/port arguments. + Use send() to select TCP explicitly. + """ + browser_options = {"out_dir", "name", "title", "rungs", "fps", "open_browser", "block"} + if address or (not isinstance(source, (str, os.PathLike)) + and not browser_options.intersection(options)): + from .streaming import send + + return send(source, *address, **options) + from ._streamer import stream as stream_to_browser + + return stream_to_browser(source, **options) diff --git a/open4d/core/tests/test_gaussians.py b/open4d/core/tests/test_gaussians.py index b9d23cd6..ae1fd985 100644 --- a/open4d/core/tests/test_gaussians.py +++ b/open4d/core/tests/test_gaussians.py @@ -171,6 +171,7 @@ def test_reconstruction_refuses_existing_output_before_launch(tmp_path, cli_runt def test_native_render_commands_do_not_call_destructive_extractor(monkeypatch, tmp_path): + monkeypatch.setitem(sys.modules, "streamer", None) root = Path(__file__).resolve().parents[2] / "reconstruction" / "gs_tools" monkeypatch.syspath_prepend(str(root)) try: diff --git a/open4d/io/tests/test_public_stream_dispatch.py b/open4d/io/tests/test_public_stream_dispatch.py new file mode 100644 index 00000000..180f84a5 --- /dev/null +++ b/open4d/io/tests/test_public_stream_dispatch.py @@ -0,0 +1,38 @@ +"""Public TCP compatibility and optional browser streaming coexist.""" +from concurrent.futures import ThreadPoolExecutor +import sys + +import numpy as np +import pytest + +import open4d +from open4d.demo import mesh_sequence + + +@pytest.mark.parametrize("entry", ["send", "stream"]) +@pytest.mark.parametrize("address_style", ["positional", "keyword"]) +def test_public_tcp_round_trip_without_browser_dependency(monkeypatch, entry, address_style): + monkeypatch.setitem(sys.modules, "streamer", None) + with mesh_sequence(side=3, frames=2) as source: + with open4d.receive(port=0, timeout=3) as receiver, ThreadPoolExecutor(1) as pool: + if address_style == "positional": + result = pool.submit(getattr(open4d, entry), source, *receiver.address, + realtime=False, timeout=3) + else: + result = pool.submit(getattr(open4d, entry), source, + host=receiver.address[0], port=receiver.address[1], timeout=3) + restored = list(receiver) + assert result.result(timeout=3) == 2 + for expected, actual in zip(source, restored): + np.testing.assert_array_equal(actual.geometry.positions, expected.geometry.positions) + np.testing.assert_array_equal(actual.geometry.triangles, expected.geometry.triangles) + assert actual.timestamp == expected.timestamp + + +@pytest.mark.parametrize("use_path", [True, False]) +def test_browser_requests_report_missing_optional_streamer(monkeypatch, tmp_path, use_path): + monkeypatch.setitem(sys.modules, "streamer", None) + with mesh_sequence(side=3, frames=1) as frames: + source = tmp_path/'capture.ply' if use_path else frames + with pytest.raises(open4d.StreamerDependencyError, match="pip install -e open4d/streamer"): + open4d.stream(source, name="capture", out_dir=tmp_path/'bundle', open_browser=False, block=False) diff --git a/open4d/reconstruction/gs_tools/gs_tools/methods/__init__.py b/open4d/reconstruction/gs_tools/gs_tools/methods/__init__.py index 48d4e60a..8ccc9544 100644 --- a/open4d/reconstruction/gs_tools/gs_tools/methods/__init__.py +++ b/open4d/reconstruction/gs_tools/gs_tools/methods/__init__.py @@ -11,6 +11,16 @@ and the other cannot. """ -from . import base, gstream, queen, rerf, vega +from importlib import import_module __all__ = ["base", "gstream", "queen", "rerf", "vega"] + + +def __getattr__(name): + # Building a QUEEN/3DGStream command must not import the optional browser + # bundle exporters, which depend on the separately installed streamer. + if name not in __all__: + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + module = import_module(f".{name}", __name__) + globals()[name] = module + return module diff --git a/open4d/streaming/_reconstruction.py b/open4d/streaming/_reconstruction.py index 4bc56c3e..473af6dd 100644 --- a/open4d/streaming/_reconstruction.py +++ b/open4d/streaming/_reconstruction.py @@ -128,7 +128,7 @@ def reconstruct(depth, color=None, *, intrinsics, camera_poses=None, fps=30.0, or not np.allclose(np.linalg.det(rotation), 1, atol=1e-5)): raise ValueError("camera_poses must be rigid camera-to-world transforms") try: - importlib.import_module("open3d") + o3d = importlib.import_module("open3d") except ModuleNotFoundError as error: if error.name != "open3d": raise @@ -136,5 +136,12 @@ def reconstruct(depth, color=None, *, intrinsics, camera_poses=None, fps=30.0, "RGB-D reconstruction needs Open3D. Install open4d[open3d].", name="open3d", ) from error + if o3d.__version__.split(".")[:2] != ["0", "19"]: + raise RuntimeError( + f"RGB-D reconstruction requires Open3D 0.19.x; found {o3d.__version__}. " + "Open3D 0.20's legacy TSDF integration rescales floating-point depth " + "and can silently return empty meshes. " + "Install a supported runtime with: python -m pip install 'open3d>=0.19,<0.20'" + ) return Sequence(_RGBDProvider(depth, color, intrinsics, camera_poses, fps, depth_scale, depth_max, voxel_size, truncation)) diff --git a/open4d/streaming/tests/test_python.py b/open4d/streaming/tests/test_python.py index a454e8b3..2002a57f 100644 --- a/open4d/streaming/tests/test_python.py +++ b/open4d/streaming/tests/test_python.py @@ -2,6 +2,8 @@ import importlib import json import socket +import sys +from types import SimpleNamespace import numpy as np import pytest @@ -121,6 +123,13 @@ def test_reconstruct_validates_before_loading_optional_dependencies(kwargs, matc reconstruct(np.ones((1, 32, 32)), **settings) +@pytest.mark.parametrize("version", ["0.18.0", "0.20.0"]) +def test_reconstruct_rejects_unsupported_open3d_before_returning_a_sequence(monkeypatch, version): + monkeypatch.setitem(sys.modules, "open3d", SimpleNamespace(__version__=version)) + with pytest.raises(RuntimeError, match=r"Open3D 0\.19\.x.*open3d>=0\.19,<0\.20"): + reconstruct(np.ones((1, 32, 32)), intrinsics=(20, 20, 16, 16)) + + @pytest.mark.open3d def test_reconstruct_depth_planes_preserves_motion_units_and_camera_pose(): pytest.importorskip("open3d") diff --git a/pyproject.toml b/pyproject.toml index a4746b1f..2360eddd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,15 +23,17 @@ open4d = "open4d._cli:main" tools = ["trimesh"] metrics = ["scipy"] player = ["PyQt6", "pyqtgraph", "PyOpenGL", "pillow>=9", "scipy"] -open3d = ["open3d>=0.19", "pillow>=9"] +# 0.20 applies depth_scale to float depth, but its legacy TSDF integrator +# still passes 1000 for images already in metres and silently extracts no mesh. +open3d = ["open3d>=0.19,<0.20", "pillow>=9"] usd = ["usd-core>=24.3"] torch = ["torch"] klt = ["torch", "trimesh", "zstd", "tqdm", "point-cloud-utils"] n4mc = ["torch", "trimesh", "scikit-image", "point-cloud-utils"] -qndf = ["torch", "open3d>=0.19", "trimesh", "tqdm"] +qndf = ["torch", "open3d>=0.19,<0.20", "trimesh", "tqdm"] gaussians = ["plyfile"] dev = ["build>=1.2", "pytest>=8", "scipy", "tomli>=2; python_version < '3.11'"] -all = ["plyfile", "trimesh", "PyQt6", "pyqtgraph", "PyOpenGL", "open3d>=0.19", "pillow>=9", "usd-core>=24.3", "scipy", "torch", "zstd", "tqdm", "point-cloud-utils", "scikit-image"] +all = ["plyfile", "trimesh", "PyQt6", "pyqtgraph", "PyOpenGL", "open3d>=0.19,<0.20", "pillow>=9", "usd-core>=24.3", "scipy", "torch", "zstd", "tqdm", "point-cloud-utils", "scikit-image"] [tool.setuptools] include-package-data = false