Make VMESH the standalone format for all codecs

.agents.txt

@@ -0,0 +1,11 @@
+Repository instructions for all agents
+
+Never create random or unsolicited Markdown (.md) files in this repository.
+Create a new Markdown file only when the user explicitly requests that file.
+Update the relevant existing documentation when documenting code changes.
+Keep task plans, audit reports, progress notes, summaries, and scratch notes in
+the conversation unless the user explicitly requests a saved artifact.
+Do not evade this rule by creating unsolicited reports with another extension.
+Keep temporary files outside the repository and remove your temporary artifacts.
+Preserve existing useful documentation, licenses, and vendored documentation.
+Follow these instructions throughout the repository, including subdirectories.

.codex/config.toml

@@ -0,0 +1 @@
+project_doc_fallback_filenames = [".agents.txt"]

.github/workflows/ci.yml

@@ -84,14 +84,46 @@ jobs:
           python-version: "3.12"
           cache: pip
       - name: Install research codec dependencies
-        run: python -m pip install -e '.[dev,klt,n4mc,qndf]'
-      - name: Run real in-process encode and fresh-decode contracts on CPU
+        run: python -m pip install -e '.[dev,klt,n4mc,qndf,usd,gaussians]' constriction 'DracoPy>=2,<3'
+      - name: Run default suite with research dependencies
+        run: python -m pytest
+      - name: Run native N4MC model and loss contracts
         run: >-
           python -m pytest
-          open4d/codec/tests/test_research_cpu.py
-          open4d/codec/tests/test_qndf_preprocess.py
-          open4d/codec/tests/test_qndf_entrypoints.py
           open4d/codecs/n4mc/tests/test_codec.py
+          open4d/codecs/n4mc/tests/test_losses.py
+
+  native-numerics:
+    name: Native tracker and editor / .NET 10
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    timeout-minutes: 15
+    env:
+      DOTNET_NOLOGO: "1"
+      DOTNET_CLI_TELEMETRY_OPTOUT: "1"
+      OPEN4D_TEST_DOTNET: dotnet
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "10.0.x"
+      - name: Install native regression test dependencies
+        run: python -m pip install -e '.[dev]'
+      - name: Build and execute native numerical contracts
+        run: |
+          dotnet --info
+          python -m pytest open4d/codec/tests/test_tracker_numerics.py open4d/codec/tests/test_editor_numerics.py --junitxml="${RUNNER_TEMP}/native-numerics.xml"
+          python - <<'PY'
+          import os
+          from pathlib import Path
+          from xml.etree import ElementTree
+          report = ElementTree.parse(Path(os.environ["RUNNER_TEMP"]) / "native-numerics.xml")
+          cases = report.findall(".//testcase")
+          assert cases and not any(case.find("skipped") is not None for case in cases), "native numerical contracts must execute"
+          PY
 
   io-tools:
     name: I/O / real optional formats and benchmark smoke
@@ -154,6 +186,37 @@ jobs:
           if-no-files-found: ignore
           retention-days: 7
 
+  browser:
+    name: Browser / streamer and web clients
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "24"
+          cache: npm
+          cache-dependency-path: |
+            open4d/webclients/system/Server/package-lock.json
+            open4d/webclients/system/WebClient/package-lock.json
+      - name: Install browser test dependencies and standalone streamer
+        run: python -m pip install -e '.[dev]' -e open4d/streamer 'DracoPy>=2,<3' scikit-image
+      - name: Check installed streamer and run its headless tests
+        run: |
+          python -I -c "import streamer; print(streamer.__file__)"
+          python -m pytest open4d/streamer/streamer_tests
+      - name: Install and build browser clients
+        run: |
+          npm ci --prefix open4d/webclients/system/Server
+          npm ci --prefix open4d/webclients/system/WebClient
+          npm run build --prefix open4d/webclients/system/WebClient
+      - name: Run web client and server JavaScript tests
+        run: node --test open4d/webclients/tests/*.js open4d/webclients/system/Server/test_*.js
+
   package:
     name: Install / ${{ matrix.os }} / Python ${{ matrix.python-version }}
     runs-on: ${{ matrix.os }}

README.md

@@ -57,7 +57,7 @@ texture coordinates and custom attributes.
 Choose a research codec explicitly. For example, after setting up V-DMC:
 
 ```python
-encoded = open4d.encode(sequence, "wave.v4d", codec="vdmc")
+encoded = open4d.encode(sequence, "wave.vmesh", codec="vdmc")
 decoded = open4d.decode(encoded)
 open4d.visualize(decoded)
 decoded.close()
@@ -67,25 +67,49 @@ decoded.close()
 artifact. Close a decoded mesh sequence when finished, or use `with`:
 
 ```python
-with open4d.decode("wave.v4d") as decoded:
+with open4d.decode("wave.vmesh") as decoded:
     print(len(decoded), "frames")
 ```
 
 The [notebook](examples/open4d_sequence_codec.ipynb) walks through these calls,
 reconstruction and streaming in separate short cells.
+Short `.vmesh` examples:
+
+- [Preserve compressed state through USDC](examples/vmesh/01_container_and_usdc.ipynb)
+- [The eight mesh codecs](examples/vmesh/02_mesh_codecs.ipynb)
+- [Vega, QUEEN and 3DGStream](examples/vmesh/03_gaussian_codecs.ipynb)
+- [ReRF](examples/vmesh/04_rerf.ipynb)
 
 | Codec | Input | Output | Backend setup |
 | --- | --- | --- | --- |
-| `vdmc` | Mesh sequence | `.v4d` | Build the V-DMC submodule and configure its encoder and decoder |
-| `faster_vdmc` | Mesh sequence | `.v4d` | Build the faster V-DMC submodule and configure its encoder and decoder |
-| `tvmc` | Mesh sequence | `.tvmc` directory | [TVMC setup](open4d/codecs/tvmc/README.md) |
-| `tsmc` | Mesh sequence | `.tsmc` directory | [TSMC setup](open4d/codecs/tsmc/README.md) |
-| `klt` | Mesh sequence converted to TSDF volumes | `.k4d` | Research source and `.[klt]` |
-| `n4mc` | Mesh sequence converted to TSDF volumes | `.n4d` | Research source and `.[n4mc]` |
-| `qndf`, `qndf-int8` | Mesh frames | `.q4d`, `.qi4d` | Research source and `.[qndf]` |
-| `vega` | Gaussian splat frames | `.vega` directory | [Vega CUDA environment](open4d/reconstruction/vega/README.md) |
-
-QNDF-int8 now writes version 2 artifacts. Version 1 artifacts must be encoded again.
+| `vdmc` | Mesh sequence | `.vmesh` | Build the V-DMC submodule and configure its encoder and decoder |
+| `faster_vdmc` | Mesh sequence | `.vmesh` | Build the faster V-DMC submodule and configure its encoder and decoder |
+| `tvmc` | Mesh sequence | `.vmesh` | [TVMC setup](open4d/codecs/tvmc/README.md) |
+| `tsmc` | Mesh sequence | `.vmesh` | [TSMC setup](open4d/codecs/tsmc/README.md) |
+| `klt` | Mesh sequence converted to TSDF volumes | `.vmesh` | Research source and `.[klt]` |
+| `n4mc` | Mesh sequence converted to TSDF volumes | `.vmesh` | Research source and `.[n4mc]` |
+| `qndf`, `qndf-int8` | Mesh frames | `.vmesh` | Research source and `.[qndf]` |
+| `vega` | Gaussian splat frames or native run | `.vmesh` | [Vega CUDA environment](open4d/reconstruction/vega/README.md) |
+| `queen`, `3dgstream`, `rerf` | Native temporal research output | `.vmesh`, with native USDC interchange | Method-specific CUDA runtime for evaluation |
+
+All twelve public codecs write standalone `.vmesh` files. The embedded profile
+identifies the codec; there is no separate public compressed output type for each
+method. Existing native outputs can be packed without recompression. Standard
+PLY, OBJ and USD import/export remain available.
+
+Mesh codecs decode to mesh sequences for ordinary USDC geometry export.
+To preserve a compressed artifact exactly, construct `open4d.NativeSequence`
+and save it to USDC using the custom `VMESH` prim, as shown in the
+[container notebook](examples/vmesh/01_container_and_usdc.ipynb). This works with
+every VMESH profile. Gaussian and field methods load as `NativeSequence` and
+evaluate their native models explicitly. N4MC shares a model across independent
+frame latents; QNDF has independent frame models.
+[VMESH](docs/api.md#vmesh-format) is a standalone custom format: its own versioned
+header and metadata directly carry native codec payloads. It has no private
+Open4D/V3C wrapper and does not claim MPEG V-DMC interoperability. A V-DMC
+profile carries the actual encoder bitstream as a native payload. Older artifacts
+require explicit migration with `open4d.migrate_legacy`; normal load/save paths
+accept VMESH rather than silently interpreting an older compressed type.
 
 KLT, N4MC and QNDF run in Python. TVMC, TSMC, V-DMC and Gaussian methods use
 separate research runtimes. N4MC and QNDF currently process frames independently;
@@ -163,7 +187,7 @@ send(mesh_sequence(frames=30))
 ```
 
 This sends decoded mesh arrays over TCP, at their recorded frame timing. It is
-not a compression method. Both calls default to this computer on port 7000.
+not a compression method. Both calls default to this computer on port 47004.
 Pass `host=` and `port=` for another address. Remote transport needs a trusted
 network or SSH tunnel; this protocol has no authentication or encryption.
 Use `realtime=False` to transfer a recorded sequence as fast as possible.
@@ -196,8 +220,9 @@ Read splat frames, then encode them with the local Vega adaptation:
 from open4d import load_gaussians, encode, decode
 
 frames = [load_gaussians("frame_0000.ply"), load_gaussians("frame_0001.ply")]
-encoded = encode(frames, "capture.vega", codec="vega")
-decoded = decode(encoded)
+encoded = encode(frames, "capture.vmesh", codec="vega")
+native = decode(encoded)   # owned native state; inspecting it needs no CUDA
+decoded = native.decode()  # runs the Vega runtime, returning NeuralGaussianFrame values
 ```
 
 A `GaussianSplats` frame contains `positions` `(N, 3)`, positive `scales`
@@ -230,11 +255,10 @@ requires its native viewer. The Qt viewer and TCP stream currently take meshes.
 
 - `open4d demo`, `open4d inspect` and `open4d view` provide command-line access.
 - `open4d.io.write_sequence` exports mesh folders; `open4d.save` writes OpenUSD
-  or explicitly selected codec artifacts. There is no default `.o4d` encoder.
+  or `.vmesh` with an explicitly selected codec.
 - `open4d.compare_sequences` measures mesh error with the `.[metrics]` extra.
 - [Viewer examples](examples/visualization/README.md) include GIF export and comparisons.
 - [Contributor setup and tests](CONTRIBUTING.md) cover optional dependencies and packaging.
 
-The general `.o4d` format is separate work. Existing codec-specific formats
-remain in use. Publication is still blocked by the unresolved component rights
+Publication is still blocked by the unresolved component rights
 in [THIRD_PARTY.md](THIRD_PARTY.md); preparing the package does not resolve them.

SECURITY.md

@@ -28,4 +28,39 @@ private data, and identifying capture content.
   references.
 - Revoke exposed credentials before removing them from Git history.
 
+## Known limitations
+
+VMESH framing and hashes validate container integrity; they do not authenticate
+the publisher or make native payloads safe to execute. Inspection, packing,
+and compressed USD round trips do not invoke native models. Decoding can:
+
+- [QUEEN's compressed loader](open4d/reconstruction/queen/scene/gaussian_model.py)
+  uses `pickle.load`, which can execute code from a malicious residual.
+- [ReRF's player](open4d/reconstruction/rerf/rerf_stream/bitstream.py) loads
+  checkpoints with `torch.load` without explicitly selecting `weights_only`;
+  older Torch runtimes can execute serialized code. Other research entry points
+  also load executable checkpoints. A worker subprocess is not a security sandbox.
+
+The [Unity TVMC backend](integrations/unity/TVMCUnity/CPP_Backend/src) has
+unchecked OBJ face indices, a decoder access to row 100 without a minimum-row
+check, and a `FetchFrame` interface without an output-capacity argument.
+Its matrix loader also permits very large allocations before checking that all
+declared bytes exist. These are source-level memory safety and resource risks;
+runtime exploitation and sanitizer validation have not been performed here.
+
+The Node control API, decoded-frame TCP transport, and research WebSocket
+bridges have no authentication. Origin and Host checks do not authenticate
+native clients. Some bridges bind to all interfaces, and static file routes
+follow symlinks; their network exposure and asset roots must be controlled.
+
+Vendored QUEEN and 3DGStream trainer/renderer argument loaders evaluate
+`cfg_args`. Some 3DGStream conversion scripts and the ReRF LLFF loader construct
+shell commands from paths or options. Use trusted inputs for those entry points;
+the VMESH import path uses JSON instead of these evaluated configurations.
+
+Legacy ZIP/NPZ paths do not consistently bound aggregate expanded bytes.
+Repacking a legacy N4MC archive can exhaust disk before VMESH validation, and
+neural architecture or volume dimensions can trigger large allocations even
+after safe deserialization.
+
 The absence of a warning is not a claim that a component is production-ready.

THIRD_PARTY.md

@@ -82,6 +82,11 @@ boundaries; they do not resolve any release block above:
   frame on longer clips, use the last available frame on short clips, and
   remove unused alignment work that also accessed an unavailable sixth frame.
 
+Additional local patches (2026-09-23): N4MC restores SSIM loss; ReRF validates
+inputs before native loading and makes test data configurable; webclients use
+a local tile record and default to loopback. Licenses and release boundaries
+are unchanged.
+
 ## Required record for new material
 
 Any copied source, submodule, binary, model, dataset, paper, image, archive, or