Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,18 @@ the extras you use:

```bash
python -m pip install -e '.[player]' # interactive mesh viewer and GIF export
python -m pip install -e '.[open3d]' # RGB-D reconstruction
python -m pip install -e '.[open3d]' # RGB-D reconstruction (Open3D 0.19.x)
python -m pip install -e '.[gaussians]' # read Gaussian PLY files
```

Research methods have additional setup below. Their source, native programs
and model weights are not bundled in the Python wheel.

RGB-D reconstruction requires Open3D 0.19.x. The legacy TSDF integrator in
Open3D 0.20 rescales already-metric float depth and can return empty meshes;
the extra selects the supported version and reconstruction rejects an
incompatible manually installed runtime before processing frames.

## Try a sequence

No dataset is needed for this example:
Expand Down Expand Up @@ -151,10 +156,10 @@ with receive() as frames:
Then send from another:

```python
from open4d import stream
from open4d import send
from open4d.demo import mesh_sequence

stream(mesh_sequence(frames=30))
send(mesh_sequence(frames=30))
```

This sends decoded mesh arrays over TCP, at their recorded frame timing. It is
Expand Down
8 changes: 7 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ index, and per-frame streams alongside the geometry — see the

## Streaming

`open4d.stream` exports a sequence as a bundle and serves it to a browser:
`open4d.stream` exports a file as a bundle and serves it to a browser:

```python
import open4d
Expand All @@ -72,6 +72,12 @@ open4d.stream("capture.usdc")
open4d.stream("capture.usdc", rungs=["draco", "draco@11"], out_dir="bundle/")
```

For a loaded sequence, pass browser options such as `name="capture"` and
`out_dir="bundle/"`. `open4d.send(sequence, host, port)` explicitly selects
decoded-mesh TCP transport and pairs with `open4d.receive`. Existing
`open4d.stream(sequence, host, port)` calls, and frame iterables without browser
options, retain that TCP behavior without requiring `open4d-streamer`.

`rungs` is the quality ladder. The first is the rendition a client plays by
default and the rest are what it can switch to mid-playback, so a one-entry
list is a fixed-quality stream and says so. A spec is a frame format —
Expand Down
2 changes: 1 addition & 1 deletion docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ hardware requirements to the shared Python baseline. See

| Module | |
|---|---|
| [**RGB-D**](../open4d/reconstruction/rgbd/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline |
| [**RGB-D**](../open4d/streaming/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline |
| [**QUEEN**](../open4d/reconstruction/queen/README.md) | Quantized efficient encoding of dynamic Gaussians for streaming free-viewpoint video (NeurIPS 2024) |
| [**3DGStream**](../open4d/reconstruction/3dgstream/README.md) | On-the-fly training of 3D Gaussians for streaming photo-realistic free-viewpoint video (CVPR 2024) |
| [**Vega**](../open4d/reconstruction/vega/README.md) | An ORBIT adaptation of Vega (MobiCom 2025): mobile volumetric video streaming with 3D Gaussian splatting |
Expand Down
4 changes: 2 additions & 2 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ build. Those are optional and separate, with install commands in
| `codecs/klt` | `kaolin` and an NVIDIA GPU; 24 GB is the same ceiling at resolution 128–256 |
| `codecs/draco` | A CMake build of the vendored Draco submodule. Open3D, pymeshlab, and OpenCV are for evaluation only |
| `codecs/vdmc`, `codecs/faster_vdmc` | The MPEG reference and optimized test models' own build requirements |
| `reconstruction/rgbd` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper |
| `streaming` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper |
| `reconstruction/gs_tools`, `queen`, `3dgstream`, `vega` | The separate `open4d-gs` conda environment and five CUDA extensions built with `--no-build-isolation`, per [`gs_tools`](../open4d/reconstruction/gs_tools/README.md). Build on ext4; on an ntfs3 mount ninja deadlocks in `ntfs_file_write_iter` |
| `reconstruction/rerf` | Python 3.8, because `ac_dc/ncvv_ac_dc.cpython-38-*.so` ships without sources and cannot be rebuilt for a newer interpreter. Plus torch with CUDA, mmcv, bitarray, Pillow, NumPy — a separate environment from every other module here |
| `integrations/unity` | Unity, plus a C++ toolchain to rebuild the backend for anything other than the prebuilt macOS and Android/Quest 3 plugins |
Expand All @@ -136,5 +136,5 @@ with `Hardware MFT failed to start`. 5 synchronized pairs/s held over Wi-Fi and
VPN; 15 did not.

Calibration layout and the step-by-step session walkthrough are in
[`open4d/reconstruction/rgbd/README.md`](../open4d/reconstruction/rgbd/README.md),
[`open4d/streaming/README.md`](../open4d/streaming/README.md),
which covers how to run the pipeline and leaves requirements to this page.
10 changes: 7 additions & 3 deletions open4d/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,12 @@
TriangleMesh,
dtypes,
)
from ._api import load, save, unload
from ._streamer import StreamerDependencyError, stream
from ._api import load, save, unload, reconstruct, stream
from ._streamer import StreamerDependencyError
from .codec import available_codecs, decode_sequence as decode, encode_sequence as encode
from .streaming import receive, send
from .gaussians import GaussianSplats, GaussianRun, NeuralGaussianFrame, load_gaussians
from .metrics import compare_meshes, compare_sequences
from .visualization import visualize

__all__ = [
Expand Down Expand Up @@ -64,7 +68,7 @@
"receive",
"stream",
"save",
"stream",
"send",
"unload",
"visualize",
]
19 changes: 19 additions & 0 deletions open4d/_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,22 @@ def reconstruct(source, output=None, *, method="rgbd", **options):

return reconstruct_gaussians(source, output, method=method, **options)
raise ValueError("method must be 'rgbd', 'queen' or '3dgstream'")


def stream(source, *address, **options):
"""Send mesh frames over TCP or export a sequence for browser playback.

A path, or browser options such as out_dir/name/rungs, selects the optional
browser streamer. A frame iterable without browser options retains the
original TCP behavior, including positional or keyword host/port arguments.
Use send() to select TCP explicitly.
"""
browser_options = {"out_dir", "name", "title", "rungs", "fps", "open_browser", "block"}
if address or (not isinstance(source, (str, os.PathLike))
and not browser_options.intersection(options)):
from .streaming import send

return send(source, *address, **options)
from ._streamer import stream as stream_to_browser

return stream_to_browser(source, **options)
1 change: 1 addition & 0 deletions open4d/core/tests/test_gaussians.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ def test_reconstruction_refuses_existing_output_before_launch(tmp_path, cli_runt


def test_native_render_commands_do_not_call_destructive_extractor(monkeypatch, tmp_path):
monkeypatch.setitem(sys.modules, "streamer", None)
root = Path(__file__).resolve().parents[2] / "reconstruction" / "gs_tools"
monkeypatch.syspath_prepend(str(root))
try:
Expand Down
38 changes: 38 additions & 0 deletions open4d/io/tests/test_public_stream_dispatch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
"""Public TCP compatibility and optional browser streaming coexist."""
from concurrent.futures import ThreadPoolExecutor
import sys

import numpy as np
import pytest

import open4d
from open4d.demo import mesh_sequence


@pytest.mark.parametrize("entry", ["send", "stream"])
@pytest.mark.parametrize("address_style", ["positional", "keyword"])
def test_public_tcp_round_trip_without_browser_dependency(monkeypatch, entry, address_style):
monkeypatch.setitem(sys.modules, "streamer", None)
with mesh_sequence(side=3, frames=2) as source:
with open4d.receive(port=0, timeout=3) as receiver, ThreadPoolExecutor(1) as pool:
if address_style == "positional":
result = pool.submit(getattr(open4d, entry), source, *receiver.address,
realtime=False, timeout=3)
else:
result = pool.submit(getattr(open4d, entry), source,
host=receiver.address[0], port=receiver.address[1], timeout=3)
restored = list(receiver)
assert result.result(timeout=3) == 2
for expected, actual in zip(source, restored):
np.testing.assert_array_equal(actual.geometry.positions, expected.geometry.positions)
np.testing.assert_array_equal(actual.geometry.triangles, expected.geometry.triangles)
assert actual.timestamp == expected.timestamp


@pytest.mark.parametrize("use_path", [True, False])
def test_browser_requests_report_missing_optional_streamer(monkeypatch, tmp_path, use_path):
monkeypatch.setitem(sys.modules, "streamer", None)
with mesh_sequence(side=3, frames=1) as frames:
source = tmp_path/'capture.ply' if use_path else frames
with pytest.raises(open4d.StreamerDependencyError, match="pip install -e open4d/streamer"):
open4d.stream(source, name="capture", out_dir=tmp_path/'bundle', open_browser=False, block=False)
12 changes: 11 additions & 1 deletion open4d/reconstruction/gs_tools/gs_tools/methods/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,16 @@
and the other cannot.
"""

from . import base, gstream, queen, rerf, vega
from importlib import import_module

__all__ = ["base", "gstream", "queen", "rerf", "vega"]


def __getattr__(name):
# Building a QUEEN/3DGStream command must not import the optional browser
# bundle exporters, which depend on the separately installed streamer.
if name not in __all__:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
module = import_module(f".{name}", __name__)
globals()[name] = module
return module
9 changes: 8 additions & 1 deletion open4d/streaming/_reconstruction.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,20 @@ def reconstruct(depth, color=None, *, intrinsics, camera_poses=None, fps=30.0,
or not np.allclose(np.linalg.det(rotation), 1, atol=1e-5)):
raise ValueError("camera_poses must be rigid camera-to-world transforms")
try:
importlib.import_module("open3d")
o3d = importlib.import_module("open3d")
except ModuleNotFoundError as error:
if error.name != "open3d":
raise
raise ModuleNotFoundError(
"RGB-D reconstruction needs Open3D. Install open4d[open3d].",
name="open3d",
) from error
if o3d.__version__.split(".")[:2] != ["0", "19"]:
raise RuntimeError(
f"RGB-D reconstruction requires Open3D 0.19.x; found {o3d.__version__}. "
"Open3D 0.20's legacy TSDF integration rescales floating-point depth "
"and can silently return empty meshes. "
"Install a supported runtime with: python -m pip install 'open3d>=0.19,<0.20'"
)
return Sequence(_RGBDProvider(depth, color, intrinsics, camera_poses, fps,
depth_scale, depth_max, voxel_size, truncation))
9 changes: 9 additions & 0 deletions open4d/streaming/tests/test_python.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
import importlib
import json
import socket
import sys
from types import SimpleNamespace

import numpy as np
import pytest
Expand Down Expand Up @@ -121,6 +123,13 @@ def test_reconstruct_validates_before_loading_optional_dependencies(kwargs, matc
reconstruct(np.ones((1, 32, 32)), **settings)


@pytest.mark.parametrize("version", ["0.18.0", "0.20.0"])
def test_reconstruct_rejects_unsupported_open3d_before_returning_a_sequence(monkeypatch, version):
monkeypatch.setitem(sys.modules, "open3d", SimpleNamespace(__version__=version))
with pytest.raises(RuntimeError, match=r"Open3D 0\.19\.x.*open3d>=0\.19,<0\.20"):
reconstruct(np.ones((1, 32, 32)), intrinsics=(20, 20, 16, 16))


@pytest.mark.open3d
def test_reconstruct_depth_planes_preserves_motion_units_and_camera_pose():
pytest.importorskip("open3d")
Expand Down
8 changes: 5 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,17 @@ open4d = "open4d._cli:main"
tools = ["trimesh"]
metrics = ["scipy"]
player = ["PyQt6", "pyqtgraph", "PyOpenGL", "pillow>=9", "scipy"]
open3d = ["open3d>=0.19", "pillow>=9"]
# 0.20 applies depth_scale to float depth, but its legacy TSDF integrator
# still passes 1000 for images already in metres and silently extracts no mesh.
open3d = ["open3d>=0.19,<0.20", "pillow>=9"]
usd = ["usd-core>=24.3"]
torch = ["torch"]
klt = ["torch", "trimesh", "zstd", "tqdm", "point-cloud-utils"]
n4mc = ["torch", "trimesh", "scikit-image", "point-cloud-utils"]
qndf = ["torch", "open3d>=0.19", "trimesh", "tqdm"]
qndf = ["torch", "open3d>=0.19,<0.20", "trimesh", "tqdm"]
gaussians = ["plyfile"]
dev = ["build>=1.2", "pytest>=8", "scipy", "tomli>=2; python_version < '3.11'"]
all = ["plyfile", "trimesh", "PyQt6", "pyqtgraph", "PyOpenGL", "open3d>=0.19", "pillow>=9", "usd-core>=24.3", "scipy", "torch", "zstd", "tqdm", "point-cloud-utils", "scikit-image"]
all = ["plyfile", "trimesh", "PyQt6", "pyqtgraph", "PyOpenGL", "open3d>=0.19,<0.20", "pillow>=9", "usd-core>=24.3", "scipy", "torch", "zstd", "tqdm", "point-cloud-utils", "scikit-image"]

[tool.setuptools]
include-package-data = false
Expand Down
Loading