From 962dc0486e13236a85cc491218b716b49e89b54c Mon Sep 17 00:00:00 2001 From: ryanmkim Date: Thu, 10 Sep 2026 10:22:14 -0400 Subject: [PATCH 01/11] updated README --- README.md | 453 ++++++++----------------------------------- docs/api.md | 97 +++++++++ docs/components.md | 37 ++++ docs/requirements.md | 144 ++++++++++++++ 4 files changed, 358 insertions(+), 373 deletions(-) create mode 100644 docs/api.md create mode 100644 docs/components.md create mode 100644 docs/requirements.md diff --git a/README.md b/README.md index 58467140..31b8a38a 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,23 @@ -# Open4D - -![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue?logo=python&logoColor=white) ![Ubuntu](https://img.shields.io/badge/Ubuntu-24.04-E95420?logo=ubuntu&logoColor=white) - -## Tools for 3D data that changes over time - -Open4D brings code for loading, viewing, compressing, and comparing mesh and -point-cloud sequences into one open-source research project. In this project, -**4D** means 3D geometry that changes over time. - -[Try the sequence viewer](#try-the-sequence-viewer) | -[Browse the codecs](#codecs-reconstruction-and-integrations) +# Open4D: tools for 4D spatial data + +

+ Quick start | + Install | + Viewer & comparison | + Python API | + Components | + Contribute | + Issues +

+ +![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue?logo=python&logoColor=white) ![Ubuntu](https://img.shields.io/badge/Ubuntu-24.04-E95420?logo=ubuntu&logoColor=white) ![License](https://img.shields.io/badge/license-MIT-green) + +Open4D aims to provide reusable, high-performance libraries and tools for +modern spatial representations, including point clouds, triangle meshes, +Gaussian splats, and future spatial data formats. Our goal is to create a +common open-source infrastructure that accelerates research and development +across applications in XR, robotics, physical AI, autonomous systems, digital +twins, graphics, vision, and spatial computing.

A reference mesh beside decoded results from N4MC, QNDF, TVMC, and TSMC, coloured by distance from the reference @@ -17,414 +25,113 @@ point-cloud sequences into one open-source research project. In this project,

A reference sequence beside results from four research codecs. Colour shows distance from the reference.

-### What works today +## Core features -- A small Python model for triangle meshes, frames, and finite sequences. -- One-file OpenUSD and Open4D codec sequences, plus `.obj`/`.ply` import paths. +- A small Python model for frames and finite sequences, over geometry that + may be a triangle mesh, a point cloud, or a Gaussian cloud. +- One-file OpenUSD and Open4D codec containers, plus `.obj`/`.ply` import paths. - A viewer for inspecting, playing, scrubbing, and exporting mesh sequences. + It runs on macOS, Linux, and Windows and does not need a GPU. - A comparison tool that measures a decoded sequence against its reference and displays both under one camera. -- Research codecs for mesh compression, plus RGB-D reconstruction and Open3D - and Unity integrations. These larger components still have their own setup - and dependencies. +- Research codecs for mesh compression, Gaussian-splatting reconstruction and + streaming, and Open3D and Unity integrations. These larger components still + have their own setup and dependencies. -### Try the sequence viewer +## Quick start -The lightweight viewer runs on macOS, Linux, and Windows and does not need a -GPU. Its normal input is one 4D sequence file: +The viewer's normal input is one 4D sequence file: ```bash git clone https://github.com/open4dfoundation/Open4D.git cd Open4D python -m venv .venv -source .venv/bin/activate +source .venv/bin/activate # Windows: .venv\Scripts\activate python -m pip install -e '.[player,usd]' + +# does it load? python examples/visualization/visualize_sequence.py capture.usdc --info +# play it python examples/visualization/visualize_sequence.py capture.usdc ``` -On Windows, activate the environment with `.venv\Scripts\activate` instead. -See the [visualization guide](examples/visualization/README.md) for supported -inputs, controls, OpenUSD packing, and sequence comparison. - -

- The Open4D sequence viewer playing a ten-frame mesh sequence -

- -> **Project status:** Open4D is early research software. The core data model, -> viewer, comparison tool, and individual research components work today, but -> the shared API and complete cross-codec workflows are still being built. - -> **Release safety:** redistribution is currently blocked while the -> third-party provenance and license audit is incomplete. See -> [`THIRD_PARTY.md`](THIRD_PARTY.md). +Try it on the 10 basketball frames the TVMC codec vendors, if you have no +sequence of your own to hand: -## Repository layout - -```text -Open4D/ -├── open4d/ -│ ├── core/ shared temporal geometry and sequence abstractions -│ ├── io/ public mesh-file and manifested-directory I/O -│ ├── codec/ shared sequence codec API and adapters -│ ├── visualization/ public viewer and GIF renderer -│ ├── torch_ops/ optional Torch geometry helpers -│ ├── codecs/ -│ │ ├── draco/ -│ │ ├── faster_vdmc/ -│ │ ├── klt/ -│ │ ├── n4mc/ -│ │ ├── qndf/ -│ │ ├── qndf_int8/ -│ │ ├── tsmc/ -│ │ ├── tvmc/ -│ │ └── vdmc/ -│ └── reconstruction/ -│ ├── rgbd/ -│ ├── queen/ -│ ├── 3dgstream/ -│ └── gs_tools/ -├── integrations/ -│ ├── open3d/ -│ └── unity/ -├── examples/ -│ └── visualization/ runnable sequence loading, visualization, and -│ reference-versus-decoded comparison -├── apps/ placeholder for end-to-end pipelines; a README only -├── scripts/ repository-level setup utilities -└── docs/ architecture and repository policies +```bash +python examples/visualization/visualize_sequence.py \ + open4d/codecs/tvmc/arap-volume-tracking/data/basketball_player \ + --up y --fps 10 --azimuth 180 ```

- How the Open4D repository's data, codec, evaluation, and playback components fit together + The Open4D sequence viewer playing a ten-frame mesh sequence

-### Codecs, reconstruction, and integrations - -- **N4MC** — neural TSDF-based mesh compression, including a newer modular - codec under its `data`, `models`, `losses`, `training`, and `evaluation` - packages. -- **Quantized Neural Displacement Fields (QNDF)** — static mesh compression - using an SSP coarse mesh and an implicit displacement decoder. -- **TVMC** — a Python, .NET, and Draco pipeline for tracked time-varying mesh - compression. It includes setup and resumable pipeline scripts. -- **TSMC** — scene-mesh compression with optional SAM-based static/dynamic - separation, ARAP volume tracking, deformation, displacement compression, and - evaluation. -- **Unity integration** — a C++ decoder backend and C# Unity front end for - playback on XR targets. -- **Draco** — Google Draco mesh-compression baseline. Wraps the vendored - `draco_encoder`/`draco_decoder` binaries into a per-frame encode/decode/eval - pipeline for benchmarking against the neural codecs. -- **KLT** — Karhunen–Loève Transform baseline that compresses TSDF voxel blocks - with a learned linear basis and quantized coefficients, reconstructing meshes - via marching cubes. -- **4D reconstruction** — synchronized multi-camera RGB-D ingestion, calibrated - point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. - It includes both the original native reconstruction code and the Python - two-camera streaming pipeline. -- **MPEG V-DMC test model** — the pinned MPEG reference implementation for - video-based dynamic mesh coding. The `open4d/codecs/vdmc` submodule provides - the standard's reference encoder, decoder, metric tools, and unit tests; it is - separate from Open4D's TVMC research pipeline. -- **Faster V-DMC** — a pinned performance-oriented fork of the same test model, - with exact-output and higher-throughput modes recorded in the - [benchmark report](docs/benchmarks/faster-vdmc.md). - -Each component has its own README and may add native tools, GPU extensions, or -hardware requirements to the shared Python baseline. See -[Requirements](#requirements) for those additions. - -## Requirements - -One baseline covers the repository itself — the shared data model and -`examples/visualization`: - -| | | -|---|---| -| Python | 3.10–3.13 | -| Operating system | macOS, Linux, or Windows | -| CPU | Any x86-64 or arm64; no particular core count | -| GPU | Not required. The viewers open a real OpenGL window, so a graphical session is needed even for `--save` | -| Memory | Roughly 1 MB of RAM per frame of playback. | -| Disk | About 1.5 GB for a clone with submodules initialized| - -`pip install -e .` needs only NumPy, and reads `.obj` and `.ply` with no further -dependencies. Extras add optional readers and viewers — see -[Installation](#installation). The comparison program additionally needs SciPy, -which the `[player]` extra installs, for its nearest-neighbour search — the same -`cKDTree` query TVMC's own evaluation uses. - -### One Python dependency set for the codecs - -The supported baseline for codec Python stages is described by -[`environment.yml`](environment.yml) at the repository root: - -```bash -conda env create -f environment.yml -conda activate open4d -pip install -e . -``` - -The Python set is Python 3.12, NumPy 1.26.4, Open3D 0.19, and PyTorch 2.7.0. -Native projects use one external .NET 10 SDK. This replaces three Python -versions, two Open3D versions, two PyTorch versions, and three .NET targets. The -Python pins themselves live in -[`requirements-codecs.txt`](requirements-codecs.txt), which `environment.yml` -installs; it lists direct dependencies only, so inside an existing Python 3.12 -environment `pip install -r requirements-codecs.txt` is equivalent. - -Codec-local setup scripts may create a convenience virtual environment, but -they must use these same Python and package pins rather than defining a second -dependency baseline. Native tools and GPU extensions remain separate. - -One trap worth naming, because its error message points the wrong way. The .NET -projects target `net10.0`, and a distribution's own `dotnet` under -`/usr/lib/dotnet` will shadow a newer SDK in `~/.dotnet` on `PATH`. The build -then fails with `NETSDK1045: The current .NET SDK does not support targeting -.NET 10.0`, which reads as a missing SDK when the SDK is usually installed and -merely second in line. Check with `dotnet --list-sdks` before installing -anything. Downgrading the projects to `net9.0` is not the fix: .NET 9 left -support in May 2026, and moving off end-of-life targets is why they are on -`net10.0`. - -Some codecs additionally need compiled extensions that pip cannot resolve from a -version number alone, because each is built against one exact PyTorch and CUDA -build. Those are optional and separate, with install commands in -[`requirements-gpu.txt`](requirements-gpu.txt): - -| Extra | Needed by | -|---|---| -| `cupy-cuda12x` | `n4mc`, `tsmc` | -| `torch-scatter` | `n4mc` | -| `nvdiffrast` | `n4mc` | -| `kaolin` | `n4mc`, `klt` | - -What each module needs beyond that shared environment: - -| Module | Adds | -|---|---| -| `codecs/tvmc` | .NET 10 SDK, CMake; Homebrew macOS or Ubuntu | -| `codecs/tsmc` | .NET 10 SDK, SAM3, `cupy`; Ubuntu 24.04, tested against Meta Quest 3. `convert_to_std_obj.py` runs inside Blender, which supplies `bpy` | -| `codecs/n4mc` | All four GPU extras and an NVIDIA GPU — 24 GB holds only about two training frames at resolution 256 | -| `codecs/qndf`, `codecs/qndf_int8` | An NVIDIA GPU for training. Evaluation (`mesh_errors.py`) runs on CPU. Building the `ssp_remesh` preprocessor needs CMake and Eigen (`libeigen3-dev`/`brew install eigen`), plus the pinned libigl submodule | -| `codecs/klt` | `kaolin` and an NVIDIA GPU; 24 GB is the same ceiling at resolution 128–256 | -| `codecs/draco` | A CMake build of the vendored Draco submodule. Open3D, pymeshlab, and OpenCV are for evaluation only | -| `codecs/vdmc`, `codecs/faster_vdmc` | The MPEG reference and optimized test models' own build requirements | -| `reconstruction/rgbd` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper | -| `integrations/unity` | Unity, plus a C++ toolchain to rebuild the backend for anything other than the prebuilt macOS and Android/Quest 3 plugins | - -Open3D ships no 3.13 wheels, capping .[open3d] and the codecs at 3.12. - -### RGB-D capture on Windows - -The RGB-D capture host is Windows and only encodes and forwards frames, so it needs no NVIDIA GPU: just the camera vendor SDK (tested: Orbbec K4A Wrapper 1.10.5, SDK 1.10.28, two Femto Bolts), both cameras on separate USB 3 ports with a sync hub, and an OpenSSH client. Close Orbbec Viewer first or the sender fails with Hardware MFT failed to start. 5 synchronized pairs/s held over Wi-Fi and VPN; 15 did not. - -Calibration layout and the step-by-step session walkthrough are in -[`open4d/reconstruction/rgbd/README.md`](open4d/reconstruction/rgbd/README.md), -which covers how to run the pipeline and leaves requirements to this page. - -## Installation - -Clone with submodules to obtain the pinned Draco, libigl, SAM3, and MPEG V-DMC -source: - -```bash -git clone --recurse-submodules https://github.com/open4dfoundation/Open4D.git -cd Open4D -``` - -For the lightweight core package: - -```bash -python -m venv .venv -source .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e . -``` - -Optional local tooling is available through extras: - -```bash -python -m pip install -e ".[player]" # the example viewer (PyQt6 + pyqtgraph) -python -m pip install -e ".[usd]" # OpenUSD containers -python -m pip install -e ".[tools]" # trimesh, for extra mesh formats -python -m pip install -e ".[open3d]" # Open3D adapter; Python 3.12 or older -python -m pip install -e ".[qndf]" # QNDF/QNDF-INT8 in-process adapters -python -m pip install -e ".[temporal]" # experimental temporal-delta/PCA codecs -python -m pip install -e ".[all]" -``` - -These extras do not install the heavyweight codec environments. Use the setup -instructions inside the selected codec before running it. Research codec -implementations remain source-checkout-only and are excluded from the -lightweight wheel until their provenance review is complete. - -If an existing clone is missing Draco, initialize and build all three copies — -the Draco baseline codec's own, plus TSMC's and TVMC's — with: - -```bash -./scripts/setup_draco.sh -``` - -## Sequence viewer details - -The public Python API loads, saves, unloads, and visualizes whole finite -triangle-mesh sequences independently of their storage format: +In Python: ```python import open4d with open4d.load("capture.usdc") as sequence: - open4d.save(sequence, "capture.o4d") - open4d.visualize(sequence) - -# A path can go straight to the lazy viewer; it is closed when the window exits. -open4d.visualize("capture.o4d") -``` - -`.usd`, `.usda`, `.usdc`, and `.usdz` are OpenUSD interchange containers. -`.o4d` and the registered codec suffixes are codec artifacts. Both carry whole -sequences and use the same `Sequence` interface. `open4d.unload(sequence)` is an -explicit, idempotent alternative to the context manager. - -`write_sequence(sequence, "frames/", format="ply")` writes a versioned -`open4d.sequence.json` beside the frame files, so reopening the directory keeps -source frame indices, timestamps, frame/sequence metadata, and topology -declarations. Empty sequences are rejected before the destination is changed. -Single mesh-file exports require `allow_lossy=True` because that storage cannot -preserve sequence timing, metadata, or topology declarations. -Trimesh-backed OFF/GLB/glTF color export also requires that opt-in because OFF -drops vertex color and GLB/glTF quantize canonical float colors to eight bits. - -Five lossless, in-process reference codecs are included: `raw`, `deflate`, -`bzip2`, `lzma`, and byte-level `rle` (`npz` remains the default DEFLATE alias). -They share a safe NumPy-array container so they compare storage strategies, not -research geometry models. Source checkouts register in-process adapters for -`klt`, `n4mc`, `qndf`, and `qndf-int8`; the lightweight wheel omits them until -their provenance review is complete. Open4D's separate `temporal-delta` and -`temporal-pca` experiments are not the repository's TVMC or TSMC pipelines. -The V-DMC adapters do not execute shell scripts, but they do invoke configured -native encoder and decoder processes once per sequence. Callers can also -register another `open4d.codec.Codec`. For an all-registered-codec attempt using -`4d_files/Rafa_Approves_hd_4k`, open -[`examples/open4d_sequence_codec.ipynb`](examples/open4d_sequence_codec.ipynb). -Set `OPEN4D_NOTEBOOK_REQUIRE_ALL=1` in a fully provisioned environment to make -any codec failure stop the notebook instead of appearing only in its result table. -The N4MC and QNDF adapters accept `device="auto"` (CUDA, then Apple Metal/MPS, -then CPU), or an explicit `"cuda"`, `"mps"`, or `"cpu"`. QNDF-int8 can train -on CUDA or Metal, but its quantized decoder remains CPU-only. Override the -notebook selection with `OPEN4D_NOTEBOOK_DEVICE=mps` when needed. - -This API slice standardizes files around `Sequence[Frame[TriangleMesh]]`; it is -not yet representation-independent. First-class point-cloud, volume, Gaussian, -and live-stream values require separate contracts. - -Normal CI runs dependency-complete CPU encode/fresh-decode contracts for KLT, -N4MC, QNDF, and QNDF-int8. The larger two-format Rafa quality/export matrix is -an additional CUDA acceptance test gated by `OPEN4D_TEST_RESEARCH_CODECS=1` and -`OPEN4D_RAFA_DATASET`; it is not presented as part of ordinary CI coverage. - -`examples/visualization/visualize_sequence.py` is the command-line client: - -```bash -python examples/visualization/visualize_sequence.py my_capture/ --info -python examples/visualization/visualize_sequence.py my_capture/ -``` - -Playback uses `open4d.visualization`'s PyQt6 window: drag to orbit, scroll to zoom, drag the slider -to scrub, space to pause, left/right to step a frame. `--save out.gif` writes an -animated GIF through the same renderer. - -A viewer source may be a single OpenUSD or codec sequence file, one mesh file, -or a folder holding one mesh file per frame. OBJ and PLY need no extra dependency; -the `[tools]` extra adds OFF, STL, GLB, and glTF, while `[usd]` adds OpenUSD -sequence files. `--info` reports frame count, timing, and topology without decoding -geometry, which is the quickest way to check a dataset loads. - -Frame folders and individual meshes remain supported as import paths; frames -are decoded on access: - -```python -from open4d.io import open_sequence - -with open_sequence("path/to/frames", fps=30.0) as sequence: print(len(sequence), sequence.duration, sequence.fps) - mesh = sequence[0].geometry # TriangleMesh: positions, triangles + open4d.visualize(sequence) ``` -OpenUSD is the public interchange container. `--pack-usd out.usdc` packs any -source into one compressed `.usdc` file carrying the frame rate, the key-frame -index, and per-frame streams alongside the geometry: +> **Project status:** Open4D is early research software. The core data model, +> viewer, comparison tool, and individual research components work today, but +> the shared API and complete cross-codec workflows are still being built. -```bash -python -m pip install -e '.[usd]' -python examples/visualization/visualize_sequence.py my_capture/ --pack-usd out.usdc --info -``` +> **Release safety:** redistribution is currently blocked while the +> third-party provenance and license audit is incomplete. See +> [`THIRD_PARTY.md`](THIRD_PARTY.md). -The TVMC codec vendors 10 frames of a basketball player, useful for checking the -program runs before pointing it at your own data. See -[`examples/visualization/README.md`](examples/visualization/README.md) for that -command, the full format list, and the container layout. +## Documentation -## Comparing a codec against its reference +| | | +|---|---| +| [Requirements and installation](docs/requirements.md) | The baseline, the one codec dependency set, GPU extras, and what each module adds | +| [Viewer and comparison guide](examples/visualization/README.md) | Supported inputs, flags, controls, reading the error numbers, and OpenUSD packing | +| [Python API](docs/api.md) | Loading, saving, the codec registry, and device selection | +| [Components](docs/components.md) | Every codec, reconstruction module, and integration, with links to their own READMEs | +| [Artifacts policy](docs/artifacts.md) | What not to commit, and what a published result must record | -`examples/visualization/compare_sequences.py` measures one sequence against -another and shows both at once — the reference as geometry, the decoded mesh -coloured by its distance from it, in synchronized panes under one camera: +## Repository layout -```bash -python examples/visualization/compare_sequences.py reference/ decoded/ --info -python examples/visualization/compare_sequences.py reference/ decoded/ +```text +open4d/ +├── core/ shared temporal geometry and sequence abstractions +├── io/ public mesh-file and manifested-directory I/O +├── codec/ shared sequence codec API and adapters +├── visualization/ public viewer and GIF renderer +├── torch_ops/ optional Torch geometry helpers +├── codecs/ draco, faster_vdmc, klt, n4mc, qndf, qndf_int8, tsmc, tvmc, vdmc +└── reconstruction/ rgbd, queen, 3dgstream, vega, rerf, gs_tools, streamer +integrations/ open3d, unity +examples/ runnable sequence loading, visualization, and comparison +scripts/ repository-level setup utilities +docs/ requirements, API, components, and repository policies ``` -Error is a nearest-neighbour distance, since a decoded mesh has its own vertex -count and connectivity: point-to-point by default, `--metric plane` for the MPEG -point-to-plane definition. Both are one-sided, so RMS, Hausdorff and PSNR are -reported in each direction and the symmetric figure is the worse of the two. -`--info` prints the per-frame table without opening a window, and `--csv` writes -it for a paper or a regression run. - -This is codec-independent: both sides are read through the same loader, so -anything the viewer opens can be compared. It does not replace a codec's own -evaluation — the figures are per-vertex rather than area-weighted over faces, so -they compare codecs against a shared reference rather than substituting for a -metric tool that integrates over the surface. - -## Reproducibility and artifacts - -Do not commit local datasets, virtual environments, benchmark jobs, training -runs, checkpoints, logs, or decoded outputs. The expected local directories, -publication-manifest requirements, and policy for existing historical fixtures -are documented in [`docs/artifacts.md`](docs/artifacts.md). - -Per-codec evaluation still lives inside each codec rather than in a -repository-wide suite; `examples/visualization/compare_sequences.py` is the one -shared piece, covering geometric error between any two sequences the loader can -read. Results should identify the exact component revision, configuration, -dataset/frame range, encoded byte count, runtime environment, and metric -implementation. +

+ How the Open4D repository's data, codec, evaluation, and playback components fit together +

## Contributing Contributions are welcome, especially around shared data abstractions, common metrics, codec adapters, documentation, and performance. Keep codec dependencies isolated and document any new binary fixture or external artifact -alongside the code that consumes it. - -Please contact the Open4D maintainers before adding a large dataset, checkpoint, -or third-party source tree. +alongside the code that consumes it. Please contact the maintainers before +adding a large dataset, checkpoint, or third-party source tree. See +[`CONTRIBUTING.md`](CONTRIBUTING.md). -## License +## License and citation Open4D is distributed under the [MIT License](LICENSE) and is intended to be -useful in academic, educational, and commercial projects. You may use, adapt, -and redistribute the Open4D code subject to the attribution and license-notice -requirements in the license. Bundled third-party components and submodules -remain subject to their respective license terms. +useful in academic, educational, and commercial projects. Bundled third-party +components and submodules remain subject to their respective license terms. If Open4D contributes to published research, please acknowledge the project using the repository's [citation metadata](CITATION.cff), and cite the original papers for any individual codecs, datasets, or algorithms used in your work. -We also welcome feedback through the project's issue tracker: sharing real-world -use cases, limitations, and improvement ideas helps guide future development. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 00000000..2948cc80 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,97 @@ +# Python API + +The public API loads, saves, unloads, and visualizes whole finite triangle-mesh +sequences independently of their storage format: + +```python +import open4d + +with open4d.load("capture.usdc") as sequence: + open4d.save(sequence, "capture.o4d") + open4d.visualize(sequence) + +# A path can go straight to the lazy viewer; it is closed when the window exits. +open4d.visualize("capture.o4d") +``` + +`.usd`, `.usda`, `.usdc`, and `.usdz` are OpenUSD interchange containers. +`.o4d` and the registered codec suffixes are codec artifacts. Both carry whole +sequences and use the same `Sequence` interface. `open4d.unload(sequence)` is an +explicit, idempotent alternative to the context manager. + +Frame folders and individual meshes remain supported as import paths; frames +are decoded on access: + +```python +from open4d.io import open_sequence + +with open_sequence("path/to/frames", fps=30.0) as sequence: + print(len(sequence), sequence.duration, sequence.fps) + mesh = sequence[0].geometry # TriangleMesh: positions, triangles +``` + +## Representations + +What a decoded frame *is* is deliberately separate from the codec that produced +it: a triangle mesh is a mesh whether it arrived as OBJ, as a Draco payload, or +out of a V-DMC bitstream. `open4d.Representation` is the axis to gate on, and +`MESH`, `POINTS`, and `GAUSSIANS` have concrete types — `TriangleMesh`, +`PointCloud`, and `GaussianCloud`. Already-rendered pixels are named in the +taxonomy but have no concrete type yet; they land with the camera model they +need in order to be comparable at a known pose. + +The containers and codecs described below are the mesh path, and the one most +completely covered. Check a specific codec before assuming it round-trips +points or Gaussians. + +## Writing sequences + +`write_sequence(sequence, "frames/", format="ply")` writes a versioned +`open4d.sequence.json` beside the frame files, so reopening the directory keeps +source frame indices, timestamps, frame/sequence metadata, and topology +declarations. Empty sequences are rejected before the destination is changed. + +Single mesh-file exports require `allow_lossy=True` because that storage cannot +preserve sequence timing, metadata, or topology declarations. Trimesh-backed +OFF/GLB/glTF color export also requires that opt-in because OFF drops vertex +color and GLB/glTF quantize canonical float colors to eight bits. + +OpenUSD is the public interchange container. `--pack-usd out.usdc` packs any +source into one compressed `.usdc` file carrying the frame rate, the key-frame +index, and per-frame streams alongside the geometry — see the +[visualization guide](../examples/visualization/README.md#the-openusd-container). + +## Codecs + +Five lossless, in-process reference codecs are included: `raw`, `deflate`, +`bzip2`, `lzma`, and byte-level `rle` (`npz` remains the default DEFLATE alias). +They share a safe NumPy-array container so they compare storage strategies, not +research geometry models. + +Source checkouts register in-process adapters for `klt`, `n4mc`, `qndf`, and +`qndf-int8`; the lightweight wheel omits them until their provenance review is +complete. Open4D's separate `temporal-delta` and `temporal-pca` experiments are +not the repository's TVMC or TSMC pipelines. The V-DMC adapters do not execute +shell scripts, but they do invoke configured native encoder and decoder +processes once per sequence. Callers can also register another +`open4d.codec.Codec`. + +For an all-registered-codec attempt using `4d_files/Rafa_Approves_hd_4k`, open +[`examples/open4d_sequence_codec.ipynb`](../examples/open4d_sequence_codec.ipynb). +Set `OPEN4D_NOTEBOOK_REQUIRE_ALL=1` in a fully provisioned environment to make +any codec failure stop the notebook instead of appearing only in its result +table. + +### Device selection + +The N4MC and QNDF adapters accept `device="auto"` (CUDA, then Apple Metal/MPS, +then CPU), or an explicit `"cuda"`, `"mps"`, or `"cpu"`. QNDF-int8 can train on +CUDA or Metal, but its quantized decoder remains CPU-only. Override the notebook +selection with `OPEN4D_NOTEBOOK_DEVICE=mps` when needed. + +### Test coverage + +Normal CI runs dependency-complete CPU encode/fresh-decode contracts for KLT, +N4MC, QNDF, and QNDF-int8. The larger two-format Rafa quality/export matrix is +an additional CUDA acceptance test gated by `OPEN4D_TEST_RESEARCH_CODECS=1` and +`OPEN4D_RAFA_DATASET`; it is not presented as part of ordinary CI coverage. diff --git a/docs/components.md b/docs/components.md new file mode 100644 index 00000000..4380dd89 --- /dev/null +++ b/docs/components.md @@ -0,0 +1,37 @@ +# Components + +Each component has its own README and may add native tools, GPU extensions, or +hardware requirements to the shared Python baseline. See +[Requirements](requirements.md) for those additions. + +## Mesh codecs + +| Codec | | +|---|---| +| [**N4MC**](../open4d/codecs/n4mc/README.md) | Neural TSDF-based mesh compression, including a newer modular codec under its `data`, `models`, `losses`, `training`, and `evaluation` packages | +| [**QNDF**](../open4d/codecs/qndf/README.md) | Quantized Neural Displacement Fields: static mesh compression using an SSP coarse mesh and an implicit displacement decoder. [`qndf_int8`](../open4d/codecs/qndf_int8/README.md) is its quantized variant | +| [**TVMC**](../open4d/codecs/tvmc/README.md) | A Python, .NET, and Draco pipeline for tracked time-varying mesh compression, with setup and resumable pipeline scripts | +| [**TSMC**](../open4d/codecs/tsmc/README.md) | Scene-mesh compression with optional SAM-based static/dynamic separation, ARAP volume tracking, deformation, displacement compression, and evaluation | +| [**KLT**](../open4d/codecs/klt/README.md) | Karhunen–Loève Transform baseline that compresses TSDF voxel blocks with a learned linear basis and quantized coefficients, reconstructing meshes via marching cubes | +| [**Draco**](../open4d/codecs/draco/README.md) | Google Draco mesh-compression baseline. Wraps the vendored `draco_encoder`/`draco_decoder` binaries into a per-frame encode/decode/eval pipeline for benchmarking against the neural codecs | +| [**MPEG V-DMC test model**](../open4d/codecs/vdmc/README.md) | The pinned MPEG reference implementation for video-based dynamic mesh coding — reference encoder, decoder, metric tools, and unit tests. Separate from Open4D's TVMC research pipeline | +| [**Faster V-DMC**](../open4d/codecs/faster_vdmc/README.md) | A pinned performance-oriented fork of the same test model, with exact-output and higher-throughput modes recorded in the [benchmark report](benchmarks/faster-vdmc.md) | + +## Reconstruction and streaming + +| Module | | +|---|---| +| [**RGB-D**](../open4d/reconstruction/rgbd/README.md) | Synchronized multi-camera RGB-D ingestion, calibrated point-cloud fusion, CUDA TSDF mesh reconstruction, and live browser playback. Includes both the original native reconstruction code and the Python two-camera streaming pipeline | +| [**QUEEN**](../open4d/reconstruction/queen/README.md) | Quantized efficient encoding of dynamic Gaussians for streaming free-viewpoint video (NeurIPS 2024) | +| [**3DGStream**](../open4d/reconstruction/3dgstream/README.md) | On-the-fly training of 3D Gaussians for streaming photo-realistic free-viewpoint video (CVPR 2024) | +| [**Vega**](../open4d/reconstruction/vega/README.md) | An ORBIT adaptation of Vega (MobiCom 2025): mobile volumetric video streaming with 3D Gaussian splatting | +| [**ReRF**](../open4d/reconstruction/rerf/README.md) | Neural residual radiance fields (CVPR 2023) as a streamable compression method | +| [**gs-tools**](../open4d/reconstruction/gs_tools/README.md) | The one environment, rasterizers, and viewer the Gaussian methods share, plus `gs-tools view` for putting several methods under one camera | +| [**streamer**](../open4d/reconstruction/streamer/README.md) | Streaming and playback for 4D reconstructions, whatever their representation. Producers import it to describe and serve their output; it imports none of them | + +## Integrations + +| | | +|---|---| +| [**Open3D**](../integrations/open3d/README.md) | Converts decoded Open4D geometry into standard Open3D `TriangleMesh` or `PointCloud` objects. An adapter, not a loader | +| [**Unity**](../integrations/unity/README.md) | A Unity playback system for TVMC-encoded sequences: a C++ decoder backend and a C# front end, for XR targets | diff --git a/docs/requirements.md b/docs/requirements.md new file mode 100644 index 00000000..dd96c6a4 --- /dev/null +++ b/docs/requirements.md @@ -0,0 +1,144 @@ +# Requirements and installation + +One baseline covers the repository itself — the shared data model and +`examples/visualization`. Individual codecs, reconstruction modules, and +integrations add to it; those additions are listed further down. + +## Baseline + +| | | +|---|---| +| Python | 3.10–3.13 | +| Operating system | macOS, Linux, or Windows | +| CPU | Any x86-64 or arm64; no particular core count | +| GPU | Not required. The viewers open a real OpenGL window, so a graphical session is needed even for `--save` | +| Memory | Roughly 1 MB of RAM per frame of playback | +| Disk | About 1.5 GB for a clone with submodules initialized | + +`pip install -e .` needs only NumPy, and reads `.obj` and `.ply` with no further +dependencies. Extras add optional readers and viewers. The comparison program +additionally needs SciPy, which the `[player]` extra installs, for its +nearest-neighbour search — the same `cKDTree` query TVMC's own evaluation uses. + +Open3D ships no 3.13 wheels, capping `.[open3d]` and the codecs at 3.12. + +## Installation + +Clone with submodules to obtain the pinned Draco, libigl, SAM3, and MPEG V-DMC +source: + +```bash +git clone --recurse-submodules https://github.com/open4dfoundation/Open4D.git +cd Open4D +``` + +For the lightweight core package: + +```bash +python -m venv .venv +source .venv/bin/activate # Windows: .venv\Scripts\activate +python -m pip install --upgrade pip +python -m pip install -e . +``` + +Optional local tooling is available through extras: + +```bash +python -m pip install -e ".[player]" # the example viewer (PyQt6 + pyqtgraph) +python -m pip install -e ".[usd]" # OpenUSD containers +python -m pip install -e ".[tools]" # trimesh, for extra mesh formats +python -m pip install -e ".[open3d]" # Open3D adapter; Python 3.12 or older +python -m pip install -e ".[qndf]" # QNDF/QNDF-INT8 in-process adapters +python -m pip install -e ".[temporal]" # experimental temporal-delta/PCA codecs +python -m pip install -e ".[all]" +``` + +These extras do not install the heavyweight codec environments. Use the setup +instructions inside the selected codec before running it. Research codec +implementations remain source-checkout-only and are excluded from the +lightweight wheel until their provenance review is complete. + +If an existing clone is missing Draco, initialize and build all three copies — +the Draco baseline codec's own, plus TSMC's and TVMC's — with: + +```bash +./scripts/setup_draco.sh +``` + +## One Python dependency set for the codecs + +The supported baseline for codec Python stages is described by +[`environment.yml`](../environment.yml) at the repository root: + +```bash +conda env create -f environment.yml +conda activate open4d +pip install -e . +``` + +The Python set is Python 3.12, NumPy 1.26.4, Open3D 0.19, and PyTorch 2.7.0. +Native projects use one external .NET 10 SDK. This replaces three Python +versions, two Open3D versions, two PyTorch versions, and three .NET targets. The +Python pins themselves live in +[`requirements-codecs.txt`](../requirements-codecs.txt), which `environment.yml` +installs; it lists direct dependencies only, so inside an existing Python 3.12 +environment `pip install -r requirements-codecs.txt` is equivalent. + +Codec-local setup scripts may create a convenience virtual environment, but +they must use these same Python and package pins rather than defining a second +dependency baseline. Native tools and GPU extensions remain separate. + +### The .NET SDK trap + +One trap worth naming, because its error message points the wrong way. The .NET +projects target `net10.0`, and a distribution's own `dotnet` under +`/usr/lib/dotnet` will shadow a newer SDK in `~/.dotnet` on `PATH`. The build +then fails with `NETSDK1045: The current .NET SDK does not support targeting +.NET 10.0`, which reads as a missing SDK when the SDK is usually installed and +merely second in line. Check with `dotnet --list-sdks` before installing +anything. Downgrading the projects to `net9.0` is not the fix: .NET 9 left +support in May 2026, and moving off end-of-life targets is why they are on +`net10.0`. + +## Compiled GPU extensions + +Some codecs additionally need compiled extensions that pip cannot resolve from a +version number alone, because each is built against one exact PyTorch and CUDA +build. Those are optional and separate, with install commands in +[`requirements-gpu.txt`](../requirements-gpu.txt): + +| Extra | Needed by | +|---|---| +| `cupy-cuda12x` | `n4mc`, `tsmc` | +| `torch-scatter` | `n4mc` | +| `nvdiffrast` | `n4mc` | +| `kaolin` | `n4mc`, `klt` | + +## What each module adds + +| Module | Adds | +|---|---| +| `codecs/tvmc` | .NET 10 SDK, CMake; Homebrew macOS or Ubuntu | +| `codecs/tsmc` | .NET 10 SDK, SAM3, `cupy`; Ubuntu 24.04, tested against Meta Quest 3. `convert_to_std_obj.py` runs inside Blender, which supplies `bpy` | +| `codecs/n4mc` | All four GPU extras and an NVIDIA GPU — 24 GB holds only about two training frames at resolution 256 | +| `codecs/qndf`, `codecs/qndf_int8` | An NVIDIA GPU for training. Evaluation (`mesh_errors.py`) runs on CPU. Building the `ssp_remesh` preprocessor needs CMake and Eigen (`libeigen3-dev`/`brew install eigen`), plus the pinned libigl submodule | +| `codecs/klt` | `kaolin` and an NVIDIA GPU; 24 GB is the same ceiling at resolution 128–256 | +| `codecs/draco` | A CMake build of the vendored Draco submodule. Open3D, pymeshlab, and OpenCV are for evaluation only | +| `codecs/vdmc`, `codecs/faster_vdmc` | The MPEG reference and optimized test models' own build requirements | +| `reconstruction/rgbd` | Two hardware-synchronized RGB-D cameras, a Windows capture host, and an Ubuntu host with Python 3.10+, an NVIDIA GPU, and CUDA-enabled Open3D. Its legacy C++ pipeline additionally wants CUDA 12.x, Open3D 0.18, OpenCV, Eigen, jsoncpp, Draco, CMake, Ninja, and either the Azure Kinect SDK or the Orbbec K4A wrapper | +| `reconstruction/gs_tools`, `queen`, `3dgstream`, `vega` | The separate `open4d-gs` conda environment and five CUDA extensions built with `--no-build-isolation`, per [`gs_tools`](../open4d/reconstruction/gs_tools/README.md). Build on ext4; on an ntfs3 mount ninja deadlocks in `ntfs_file_write_iter` | +| `reconstruction/rerf` | Python 3.8, because `ac_dc/ncvv_ac_dc.cpython-38-*.so` ships without sources and cannot be rebuilt for a newer interpreter. Plus torch with CUDA, mmcv, bitarray, Pillow, NumPy — a separate environment from every other module here | +| `integrations/unity` | Unity, plus a C++ toolchain to rebuild the backend for anything other than the prebuilt macOS and Android/Quest 3 plugins | + +## RGB-D capture on Windows + +The RGB-D capture host is Windows and only encodes and forwards frames, so it +needs no NVIDIA GPU: just the camera vendor SDK (tested: Orbbec K4A Wrapper +1.10.5, SDK 1.10.28, two Femto Bolts), both cameras on separate USB 3 ports with +a sync hub, and an OpenSSH client. Close Orbbec Viewer first or the sender fails +with `Hardware MFT failed to start`. 5 synchronized pairs/s held over Wi-Fi and +VPN; 15 did not. + +Calibration layout and the step-by-step session walkthrough are in +[`open4d/reconstruction/rgbd/README.md`](../open4d/reconstruction/rgbd/README.md), +which covers how to run the pipeline and leaves requirements to this page. From 987e92345ce0dac84fa793b659056e4c8b12b64e Mon Sep 17 00:00:00 2001 From: "Ryan M Kim." <74478729+ryanmkim@users.noreply.github.com> Date: Thu, 10 Sep 2026 10:30:43 -0400 Subject: [PATCH 02/11] Update requirements.md --- docs/requirements.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/requirements.md b/docs/requirements.md index dd96c6a4..d40a92a9 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,9 +1,5 @@ # Requirements and installation -One baseline covers the repository itself — the shared data model and -`examples/visualization`. Individual codecs, reconstruction modules, and -integrations add to it; those additions are listed further down. - ## Baseline | | | From 4b238031c12c9f8f57bcc6fc57a0c580ef5f8528 Mon Sep 17 00:00:00 2001 From: ryanmkim Date: Thu, 17 Sep 2026 22:48:54 -0400 Subject: [PATCH 03/11] Add streaming: browser clients for volumetric adaptive streaming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A sibling of reconstruction/, holding the client side rather than the methods. system/ClientCore is platform-free streaming logic — segment loop, MCKP ABR, bandwidth estimator — shared by a Node and a browser client behind one ClientPlatform contract. system/WebClient has five pages: our adaptive mesh system, ViVo and NAVA point clouds over a WebSocket-to-TCP bridge, Vega splats and NeVo panels. Copied from the 4DVideoStreaming research repo, so the ladder solver, the baseline servers and the corpora still live there; see the README. --- open4d/streaming/.gitignore | 4 + open4d/streaming/README.md | 72 + open4d/streaming/scripts/run_web_demo.sh | 101 + .../scripts/serve_pointcloud_baseline.sh | 75 + open4d/streaming/scripts/shape_web_demo.sh | 59 + .../system/Client/viewpoints/view_00.json | 41 + open4d/streaming/system/ClientCore/README.md | 119 + open4d/streaming/system/ClientCore/abr.js | 1038 +++++++ .../system/ClientCore/bandwidth-estimator.js | 18 + .../streaming/system/ClientCore/bandwidth.js | 187 ++ .../system/ClientCore/download-plan.js | 189 ++ .../system/ClientCore/link-throughput.js | 100 + open4d/streaming/system/ClientCore/metrics.js | 239 ++ .../streaming/system/ClientCore/payloads.js | 207 ++ .../streaming/system/ClientCore/platform.js | 280 ++ open4d/streaming/system/ClientCore/stalls.js | 103 + .../system/ClientCore/stream-config.js | 61 + .../system/ClientCore/streaming-client.js | 1086 +++++++ .../ClientCore/testing/fake-platform.js | 362 +++ .../streaming/system/Server/manifest.mpd.json | 321 +++ .../streaming/system/Server/package-lock.json | 782 +++++ open4d/streaming/system/Server/package.json | 6 + .../streaming/system/Server/public/index.html | 0 .../streaming/system/Server/quest-launch.json | 31 + .../system/Server/quest-trace-player.js | 370 +++ open4d/streaming/system/Server/server.js | 2539 +++++++++++++++++ .../system/Server/test_offline_trajectory.js | 60 + .../system/Server/test_quest_trace_player.js | 31 + open4d/streaming/system/WebClient/README.md | 342 +++ .../system/WebClient/bridge/v4ds-bridge.js | 280 ++ open4d/streaming/system/WebClient/build.js | 76 + .../system/WebClient/package-lock.json | 522 ++++ .../streaming/system/WebClient/package.json | 18 + .../system/WebClient/public/baseline.html | 88 + .../system/WebClient/public/compare.html | 62 + .../system/WebClient/public/index.html | 85 + .../system/WebClient/public/nevo.html | 82 + .../system/WebClient/public/vega.html | 84 + .../system/WebClient/src/baseline-client.js | 397 +++ .../system/WebClient/src/baseline-main.js | 147 + .../system/WebClient/src/browser-platform.js | 494 ++++ .../system/WebClient/src/camera-pose.js | 169 ++ .../streaming/system/WebClient/src/chooser.js | 227 ++ .../system/WebClient/src/decode-cache.js | 166 ++ .../system/WebClient/src/draco-worker.js | 191 ++ .../system/WebClient/src/link-rate.js | 55 + open4d/streaming/system/WebClient/src/main.js | 224 ++ .../system/WebClient/src/nevo-client.js | 274 ++ .../system/WebClient/src/nevo-main.js | 95 + .../system/WebClient/src/nevo-manifest.js | 158 + .../WebClient/src/point-reconstruction.js | 259 ++ .../system/WebClient/src/point-renderer.js | 246 ++ .../system/WebClient/src/splat-renderer.js | 422 +++ .../system/WebClient/src/texture-decoder.js | 208 ++ .../system/WebClient/src/v4ds-protocol.js | 455 +++ .../system/WebClient/src/vega-client.js | 524 ++++ .../system/WebClient/src/vega-main.js | 158 + .../system/WebClient/src/vgs-format.js | 211 ++ .../system/WebClient/src/webgl-renderer.js | 579 ++++ .../WebClient/vendor/draco/draco_decoder.js | 34 + .../WebClient/vendor/draco/draco_decoder.wasm | Bin 0 -> 285948 bytes .../tests/fixtures_nevo_manifest.json | 83 + .../tests/fixtures_recon_golden.gen.py | 112 + .../tests/fixtures_recon_golden.json | 2014 +++++++++++++ .../tests/fixtures_v4ds_golden.gen.py | 59 + .../streaming/tests/fixtures_v4ds_golden.json | 9 + .../tests/fixtures_vgs_golden.gen.py | 87 + .../streaming/tests/fixtures_vgs_golden.json | 1650 +++++++++++ .../tests/test_bandwidth_estimator.js | 15 + .../streaming/tests/test_browser_platform.js | 545 ++++ open4d/streaming/tests/test_camera_pose.js | 290 ++ open4d/streaming/tests/test_client_core.js | 537 ++++ .../tests/test_client_core_parity.js | 359 +++ .../tests/test_client_core_purity.js | 207 ++ .../streaming/tests/test_client_platform.js | 375 +++ open4d/streaming/tests/test_decode_cache.js | 181 ++ open4d/streaming/tests/test_link_rate.js | 79 + open4d/streaming/tests/test_nevo_manifest.js | 216 ++ .../tests/test_point_reconstruction.js | 286 ++ .../streaming/tests/test_streaming_client.js | 689 +++++ open4d/streaming/tests/test_tile_ladder.py | 150 + open4d/streaming/tests/test_v4ds_bridge.js | 401 +++ open4d/streaming/tests/test_v4ds_protocol.js | 382 +++ open4d/streaming/tests/test_vgs_format.js | 332 +++ .../streaming/tests/test_web_client_bundle.js | 354 +++ open4d/streaming/tile_ladder.py | 297 ++ 86 files changed, 25522 insertions(+) create mode 100644 open4d/streaming/.gitignore create mode 100644 open4d/streaming/README.md create mode 100755 open4d/streaming/scripts/run_web_demo.sh create mode 100755 open4d/streaming/scripts/serve_pointcloud_baseline.sh create mode 100755 open4d/streaming/scripts/shape_web_demo.sh create mode 100644 open4d/streaming/system/Client/viewpoints/view_00.json create mode 100644 open4d/streaming/system/ClientCore/README.md create mode 100644 open4d/streaming/system/ClientCore/abr.js create mode 100644 open4d/streaming/system/ClientCore/bandwidth-estimator.js create mode 100644 open4d/streaming/system/ClientCore/bandwidth.js create mode 100644 open4d/streaming/system/ClientCore/download-plan.js create mode 100644 open4d/streaming/system/ClientCore/link-throughput.js create mode 100644 open4d/streaming/system/ClientCore/metrics.js create mode 100644 open4d/streaming/system/ClientCore/payloads.js create mode 100644 open4d/streaming/system/ClientCore/platform.js create mode 100644 open4d/streaming/system/ClientCore/stalls.js create mode 100644 open4d/streaming/system/ClientCore/stream-config.js create mode 100644 open4d/streaming/system/ClientCore/streaming-client.js create mode 100644 open4d/streaming/system/ClientCore/testing/fake-platform.js create mode 100644 open4d/streaming/system/Server/manifest.mpd.json create mode 100644 open4d/streaming/system/Server/package-lock.json create mode 100644 open4d/streaming/system/Server/package.json create mode 100644 open4d/streaming/system/Server/public/index.html create mode 100644 open4d/streaming/system/Server/quest-launch.json create mode 100644 open4d/streaming/system/Server/quest-trace-player.js create mode 100644 open4d/streaming/system/Server/server.js create mode 100644 open4d/streaming/system/Server/test_offline_trajectory.js create mode 100644 open4d/streaming/system/Server/test_quest_trace_player.js create mode 100644 open4d/streaming/system/WebClient/README.md create mode 100644 open4d/streaming/system/WebClient/bridge/v4ds-bridge.js create mode 100644 open4d/streaming/system/WebClient/build.js create mode 100644 open4d/streaming/system/WebClient/package-lock.json create mode 100644 open4d/streaming/system/WebClient/package.json create mode 100644 open4d/streaming/system/WebClient/public/baseline.html create mode 100644 open4d/streaming/system/WebClient/public/compare.html create mode 100644 open4d/streaming/system/WebClient/public/index.html create mode 100644 open4d/streaming/system/WebClient/public/nevo.html create mode 100644 open4d/streaming/system/WebClient/public/vega.html create mode 100644 open4d/streaming/system/WebClient/src/baseline-client.js create mode 100644 open4d/streaming/system/WebClient/src/baseline-main.js create mode 100644 open4d/streaming/system/WebClient/src/browser-platform.js create mode 100644 open4d/streaming/system/WebClient/src/camera-pose.js create mode 100644 open4d/streaming/system/WebClient/src/chooser.js create mode 100644 open4d/streaming/system/WebClient/src/decode-cache.js create mode 100644 open4d/streaming/system/WebClient/src/draco-worker.js create mode 100644 open4d/streaming/system/WebClient/src/link-rate.js create mode 100644 open4d/streaming/system/WebClient/src/main.js create mode 100644 open4d/streaming/system/WebClient/src/nevo-client.js create mode 100644 open4d/streaming/system/WebClient/src/nevo-main.js create mode 100644 open4d/streaming/system/WebClient/src/nevo-manifest.js create mode 100644 open4d/streaming/system/WebClient/src/point-reconstruction.js create mode 100644 open4d/streaming/system/WebClient/src/point-renderer.js create mode 100644 open4d/streaming/system/WebClient/src/splat-renderer.js create mode 100644 open4d/streaming/system/WebClient/src/texture-decoder.js create mode 100644 open4d/streaming/system/WebClient/src/v4ds-protocol.js create mode 100644 open4d/streaming/system/WebClient/src/vega-client.js create mode 100644 open4d/streaming/system/WebClient/src/vega-main.js create mode 100644 open4d/streaming/system/WebClient/src/vgs-format.js create mode 100644 open4d/streaming/system/WebClient/src/webgl-renderer.js create mode 100644 open4d/streaming/system/WebClient/vendor/draco/draco_decoder.js create mode 100644 open4d/streaming/system/WebClient/vendor/draco/draco_decoder.wasm create mode 100644 open4d/streaming/tests/fixtures_nevo_manifest.json create mode 100644 open4d/streaming/tests/fixtures_recon_golden.gen.py create mode 100644 open4d/streaming/tests/fixtures_recon_golden.json create mode 100644 open4d/streaming/tests/fixtures_v4ds_golden.gen.py create mode 100644 open4d/streaming/tests/fixtures_v4ds_golden.json create mode 100644 open4d/streaming/tests/fixtures_vgs_golden.gen.py create mode 100644 open4d/streaming/tests/fixtures_vgs_golden.json create mode 100644 open4d/streaming/tests/test_bandwidth_estimator.js create mode 100644 open4d/streaming/tests/test_browser_platform.js create mode 100644 open4d/streaming/tests/test_camera_pose.js create mode 100644 open4d/streaming/tests/test_client_core.js create mode 100644 open4d/streaming/tests/test_client_core_parity.js create mode 100644 open4d/streaming/tests/test_client_core_purity.js create mode 100644 open4d/streaming/tests/test_client_platform.js create mode 100644 open4d/streaming/tests/test_decode_cache.js create mode 100644 open4d/streaming/tests/test_link_rate.js create mode 100644 open4d/streaming/tests/test_nevo_manifest.js create mode 100644 open4d/streaming/tests/test_point_reconstruction.js create mode 100644 open4d/streaming/tests/test_streaming_client.js create mode 100644 open4d/streaming/tests/test_tile_ladder.py create mode 100644 open4d/streaming/tests/test_v4ds_bridge.js create mode 100644 open4d/streaming/tests/test_v4ds_protocol.js create mode 100644 open4d/streaming/tests/test_vgs_format.js create mode 100644 open4d/streaming/tests/test_web_client_bundle.js create mode 100644 open4d/streaming/tile_ladder.py diff --git a/open4d/streaming/.gitignore b/open4d/streaming/.gitignore new file mode 100644 index 00000000..9cc5fe3a --- /dev/null +++ b/open4d/streaming/.gitignore @@ -0,0 +1,4 @@ +# Generated, both regenerated by `cd system/WebClient && npm install && node build.js`. +# Scoped here rather than in the repo root so this directory stays self-contained. +node_modules/ +dist/ diff --git a/open4d/streaming/README.md b/open4d/streaming/README.md new file mode 100644 index 00000000..179e6c29 --- /dev/null +++ b/open4d/streaming/README.md @@ -0,0 +1,72 @@ +# streaming + +Browser clients for volumetric adaptive streaming, and the platform-free +streaming logic they share with the desktop client. Copied from the +`4DVideoStreaming` research repo; a sibling of [`../reconstruction`](../reconstruction), +which vendors the *reconstruction* methods (`vega`, `rerf`, `queen`, …) that +some of these clients play. + +## What is here + +| Path | Role | +|---|---| +| `system/ClientCore/` | Platform-free streaming logic: segment loop, MCKP ABR, bandwidth estimator, metrics, behind a `ClientPlatform` contract | +| `system/WebClient/` | Five browser pages — our adaptive mesh system, ViVo/NAVA point clouds, Vega splats, NeVo — plus a WebSocket↔TCP bridge | +| `system/Server/` | Express server: publishes the per-segment ladder, serves media, logs viewpoints and selections | +| `tile_ladder.py` | Catalogue-backed ladder letting ViVo and NAVA serve prepared tiles without the RGB-D source | +| `tests/` | 294 tests, including decoders cross-checked against the authoritative Python implementations | +| `scripts/` | Demo launcher, baseline supervisor, trace-based bandwidth shaping | + +The `system/` layout is preserved deliberately: every relative `require()` +(`tests/` → `../system/ClientCore/…`, `WebClient/src` → `../../ClientCore/…`) +keeps working, so nothing needed import rewriting. + +Read [`system/WebClient/README.md`](system/WebClient/README.md) for how the +pages work and the pitfalls that cost real debugging time, and +[`system/ClientCore/README.md`](system/ClientCore/README.md) for the platform +contract. + +## What this does NOT include + +This is the **web demo scope**. Three things it depends on live in the research +repo and were deliberately not copied, so the demo is **not runnable from here +as-is**: + +1. **`vstream/`** — the Python package that solves the bitrate ladder. The + server spawns `python -m vstream.ladder.ladder_service`, so without it no + manifest is published and the mesh client has nothing to select from. +2. **`baselines/`** — the ViVo/NAVA/DeltaStream servers. `tile_ladder.py` is + here for reference but imports `baselines.ViVo.orbitvivo.ladder`, and the + point-cloud pages need one of those servers running behind the bridge. +3. **The corpora** — encoded media, the prepared ViVo tiles, the Vega export. + All gitignored; they are hundreds of GB. + +`scripts/run_web_demo.sh` and `scripts/shape_web_demo.sh` came across for +reference but expect the full repo (the latter also needs +`system/Client/traces/*.csv`). Point them at a checkout of the research repo, +or treat this directory as the client-side source of truth that gets vendored +back. + +## Tests + +```bash +cd system/WebClient && npm install && node build.js # bundles; some tests load dist/ +cd ../.. && node --test tests/*.js +``` + +292 of 294 pass standalone. The two that do not are cross-repo **by design** — +they exist to prove the JavaScript decoders match the Python ones byte for +byte, which is exactly the check you lose if you let them drift: + +```bash +PYTHONPATH=/path/to/4DVideoStreaming \ +VS4D_TEST_PYTHON=/path/to/env/bin/python \ + node --test tests/*.js +``` + +That fixes the V4DS protocol round-trip. The last one, +`test_vgs_format.js`'s "golden fixture and its source asset are present", +wants the real 1.1 MB `results/vega-web/dancer/frame_0000.vgs` export — run +from a repo checkout that has it, or re-export with +`orbitvega.export_quest`. The rest of that file's assertions run against the +committed golden JSON and pass without the asset. diff --git a/open4d/streaming/scripts/run_web_demo.sh b/open4d/streaming/scripts/run_web_demo.sh new file mode 100755 index 00000000..3cd59772 --- /dev/null +++ b/open4d/streaming/scripts/run_web_demo.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Bring up the whole browser demo: five systems, one command. +# +# PYTHON_BIN= scripts/run_web_demo.sh +# PYTHON_BIN=... scripts/run_web_demo.sh --objects dancer,thomas +# +# Starts the Node server on the H.264 corpus (so browsers can decode textures) +# plus a supervised ViVo and NAVA, then prints the URLs. Ctrl-C stops all of +# them. +# +# Not started here, because it needs root and shapes the whole host: +# sudo scripts/shape_web_demo.sh cascade-20 +set -uo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PY="${PYTHON_BIN:-python}" +PORT="${PORT:-3000}" +OBJECTS="dancer,thomas" +while [[ $# -gt 0 ]]; do + case "$1" in + --objects) OBJECTS="$2"; shift 2 ;; + *) echo "unknown option $1" >&2; exit 2 ;; + esac +done + +# H.264 rather than HEVC: HEVC is Firefox-no and Chrome-only-with-hardware, +# and H.264 measures at 1.003x the bitrate at matched quality on this corpus, +# so serving it costs essentially nothing and every browser can decode it. +CORPUS="${VS4D_COMPRESSED_ROOT:-/media/frozzzen/LocalDisk/ORBIT_datasets_compressed_h264}" +# A files root PER CORPUS. Representation ids carry no codec, so a cache filled +# by one corpus would serve its encodes under another's manifest; the server +# refuses to start if this is violated. +FILES="${VS4D_FILES_ROOT:-$REPO/files-h264}" +VEGA="${VS4D_VEGA_WEB_ROOT:-$REPO/results/vega-web-all}" +TILES="${VS4D_VIVO_TILES_ROOT:-/media/frozzzen/DataDrive/ORBIT_vivo_tiles}" +# The server's own default is 10 segments -- 20 seconds, which is a trial +# length, not a demo length: the page reaches "run complete" before you have +# finished looking at it. 300 segments is ten minutes, and the shaping trace +# repeats every three, so a viewer sees the ladder move several times. +SEGMENTS="${VS4D_TOTAL_SEGMENTS:-300}" + +for path in "$CORPUS/megamanifest.json" "$CORPUS/models/quality_model.joblib"; do + [[ -f "$path" ]] || { echo "missing $path" >&2; exit 1; } +done + +PIDS=() +cleanup() { + trap - INT TERM EXIT + echo + echo "stopping…" + for pid in "${PIDS[@]:-}"; do [[ -n "$pid" ]] && kill "$pid" 2>/dev/null; done + wait 2>/dev/null + echo "stopped" +} +trap cleanup INT TERM EXIT + +echo "corpus $CORPUS" +echo "files $FILES" +echo "segments $SEGMENTS ($((SEGMENTS * 2))s per run)" +echo + +( cd "$REPO/system/Server" && VS4D_COMPRESSED_ROOT="$CORPUS" \ + VS4D_FILES_ROOT="$FILES" VS4D_VEGA_WEB_ROOT="$VEGA" \ + VS4D_VIVO_TILES_ROOT="$TILES" VS4D_TOTAL_SEGMENTS="$SEGMENTS" \ + PYTHON_BIN="$PY" PORT="$PORT" \ + node server.js ) & +PIDS+=($!) +sleep 8 + +if [[ -f "$TILES/catalog.json" ]]; then + for baseline in vivo nava; do + PYTHON_BIN="$PY" "$REPO/scripts/serve_pointcloud_baseline.sh" \ + "$baseline" "$OBJECTS" & + PIDS+=($!) + done + sleep 10 +else + echo "no tile corpus at $TILES — ViVo and NAVA will show as unavailable" +fi + +HOST="$(hostname -I | awk '{print $1}')" +cat <&2 + echo "MetaStream/DeltaStream/LiVo need the RGB-D corpus and cannot be" >&2 + echo "served from the prepared tiles." >&2 + exit 2 ;; +esac + +if [[ ! -f "$TILES/catalog.json" ]]; then + echo "no tile catalogue at $TILES/catalog.json" >&2 + exit 1 +fi +mkdir -p "$OUT" +cd "$REPO" + +cleanup() { + trap - INT TERM EXIT + [[ -n "${BRIDGE_PID:-}" ]] && kill "$BRIDGE_PID" 2>/dev/null + [[ -n "${SERVER_PID:-}" ]] && kill "$SERVER_PID" 2>/dev/null + echo "stopped" +} +trap cleanup INT TERM EXIT + +node system/WebClient/bridge/v4ds-bridge.js \ + --baseline-port "$PORT" --listen-port "$BRIDGE_PORT" & +BRIDGE_PID=$! +echo "bridge pid $BRIDGE_PID on ws://0.0.0.0:$BRIDGE_PORT" + +# shellcheck disable=SC2086 +while kill -0 "$BRIDGE_PID" 2>/dev/null; do + "$PY" -m "$MODULE" \ + --prepared-dir "$TILES" --output-dir "$OUT" --tile-catalog-ladder \ + --objects ${OBJECTS//,/ } --port "$PORT" & + SERVER_PID=$! + wait "$SERVER_PID" + SERVER_PID="" + # A crash loop would otherwise spin as fast as the process can fail. + sleep 1 + echo "--- $BASELINE session ended; ready for the next page load ---" +done diff --git a/open4d/streaming/scripts/shape_web_demo.sh b/open4d/streaming/scripts/shape_web_demo.sh new file mode 100755 index 00000000..a1d52e95 --- /dev/null +++ b/open4d/streaming/scripts/shape_web_demo.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Replay a bandwidth trace against the browser demo, so adaptation is visible. +# +# sudo scripts/shape_web_demo.sh # cascade-20, eth0 +# sudo scripts/shape_web_demo.sh poor-wifi +# sudo scripts/shape_web_demo.sh cascade-20 --scale 0.5 +# +# WHAT THIS AFFECTS. It installs a root token bucket on the server's egress +# NIC, so it shapes EVERY outbound flow on this machine, not just the demo. +# That is deliberate and inherited from the Quest methodology: a destination +# u32 filter silently misses traffic on a multiqueue NIC, and this NIC is +# multiqueue (`qdisc mq 0: root`). The cost is that other users and other +# services on this host are shaped too for as long as it runs. Do not leave it +# running, and do not run it on a shared box without telling whoever else is +# on it. +# +# Ctrl-C restores the original qdisc. So does the script exiting for any other +# reason; if it is killed with -9, remove the rule by hand with +# sudo tc qdisc del dev root +set -uo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TRACE_NAME="${1:-cascade-20}" +shift || true +NIC="${VS4D_SHAPED_INTERFACE:-$(ip route show default | grep -oP 'dev \K\S+' | head -1)}" +TRACE="$REPO/system/Client/traces/${TRACE_NAME}.csv" + +if [[ ! -f "$TRACE" ]]; then + echo "no such trace: $TRACE" >&2 + echo "available:" >&2 + ls "$REPO/system/Client/traces/" | sed 's/\.csv$//' | sed 's/^/ /' >&2 + exit 2 +fi +if [[ -z "$NIC" ]]; then + echo "could not determine the egress NIC; set VS4D_SHAPED_INTERFACE" >&2 + exit 2 +fi +if [[ "$(id -u)" -ne 0 ]]; then + echo "tc needs root: re-run as" >&2 + echo " sudo $0 $TRACE_NAME $*" >&2 + exit 1 +fi + +echo "trace $TRACE" +echo "interface $NIC (shaping ALL egress on this host)" +python3 - "$TRACE" <<'PY' +import csv, sys +rows = list(csv.reader(open(sys.argv[1])))[1:] +vals = [float(b) for _, b in rows if b] +print(f"profile {len(vals)} points, {min(vals):.1f}-{max(vals):.1f} Mbps, " + f"{rows[-1][0]}s long") +PY +echo +echo "Watch the 'link:' line on /web/ and /web/baseline.html follow this." +echo "Ctrl-C restores the original qdisc." +echo + +exec node "$REPO/system/Server/quest-trace-player.js" \ + --interface "$NIC" --hold "$@" "$TRACE" diff --git a/open4d/streaming/system/Client/viewpoints/view_00.json b/open4d/streaming/system/Client/viewpoints/view_00.json new file mode 100644 index 00000000..927dfc24 --- /dev/null +++ b/open4d/streaming/system/Client/viewpoints/view_00.json @@ -0,0 +1,41 @@ +{ + "class_name" : "PinholeCameraParameters", + "extrinsic" : + [ + 0.99595296240202125, + -0.0010699242563095551, + -0.08986963861462792, + 0.0, + 0.0010881203398058592, + -0.9997123107282031, + 0.023960629637030998, + 0.0, + -0.089869420142583326, + -0.023961449049726267, + -0.99566527321317844, + 0.0, + 1683.562466691084, + 407.74266067856468, + 6515.1305376613482, + 1.0 + ], + "intrinsic" : + { + "height" : 1920, + "intrinsic_matrix" : + [ + 1662.7687752661222, + 0.0, + 0.0, + 0.0, + 1662.7687752661222, + 0.0, + 959.5, + 959.5, + 1.0 + ], + "width" : 1920 + }, + "version_major" : 1, + "version_minor" : 0 +} \ No newline at end of file diff --git a/open4d/streaming/system/ClientCore/README.md b/open4d/streaming/system/ClientCore/README.md new file mode 100644 index 00000000..e5b91fb8 --- /dev/null +++ b/open4d/streaming/system/ClientCore/README.md @@ -0,0 +1,119 @@ +# ClientCore + +Platform-free streaming logic, shared by the Node desktop client +(`system/Client`) and the browser client (`system/WebClient`). + +The rule: **nothing here may import `fs`, `http`, `https`, `path`, `os` or +`process`, or touch `console`, `Date.now`, `setInterval` or `fetch` directly.** +Everything platform-shaped goes through the `ClientPlatform` contract in +[platform.js](platform.js), and `tests/test_client_core_purity.js` enforces it. +If both clients run the same core, a difference between their results is a real +difference in the system under test rather than a difference between two +hand-written clients. + +## Contents + +| File | Role | +|---|---| +| [streaming-client.js](streaming-client.js) | `StreamingClient` — segment loop, playback clock, download orchestration, server reporting, shutdown | +| [platform.js](platform.js) | The contract: typedefs, `PLATFORM_CONTRACT` as data, `validatePlatform` | +| [testing/fake-platform.js](testing/fake-platform.js) | In-memory platform — virtual clock, routed transport, Map-backed storage. Test substrate and reference implementation | +| [abr.js](abr.js) | The MCKP selector and per-object buffers | +| [link-throughput.js](link-throughput.js) | `LinkThroughputMeter` — byte/busy-time accounting across overlapping downloads | +| [bandwidth.js](bandwidth.js) | `BandwidthEstimator` + `computeBudget` — harmonic estimate and health-scaled spend budget | +| [bandwidth-estimator.js](bandwidth-estimator.js) | The causal harmonic-mean throughput estimate itself | +| [download-plan.js](download-plan.js) | What a segment needs, and how to judge the outcome | +| [metrics.js](metrics.js) | The run's metrics record and its builders | +| [payloads.js](payloads.js) | Server request bodies (wire formats read by `system/Server`) | +| [stalls.js](stalls.js) | Per-segment stall collection and stall-event folding | +| [stream-config.js](stream-config.js) | `resolveStreamConfig` — validated timing from `/api/config` | + +## The seven capabilities + +| Capability | Methods | +|---|---| +| `transport` | `getJson`, `postJson`, `assetUrl`, `fetchAsset` | +| `storage` | `handle`, `createScratch`, `release`, `writeText`, `writeResult`, `openAppendStream` | +| `viewpoints` | `list` | +| `clock` | `now`, `every`, `cancel`, `delay` | +| `logger` | `emit` | +| `renderer` | `start`, `stop`, `stageSegment`, `setPlaybackState`, `setCallbacks`, `latestCamera` (nullable) | +| `lifecycle` | `exit`, `onShutdownRequest` | + +Nothing here is speculative: the set was derived by auditing every platform +touchpoint in the pre-refactor `client.js` (10 fetch sites, 9 distinct `fs` +calls, 5 `process` calls, 4 timer sites, the renderer child process, console +logging), and each method backs at least one real call site. The two +implementations are [`Client/node-platform.js`](../Client/node-platform.js) and +[`WebClient/src/browser-platform.js`](../WebClient/src/browser-platform.js). + +Call `validatePlatform(platform, { requireRenderer })` once at startup. It +reports *every* missing method in one error, because the alternative is +`undefined is not a function` twelve segments into a run — which leaves a +truncated metrics file that looks like a legitimate result. + +## Asset handles + +The core never learns where a downloaded byte lives. `storage.handle(...)` +mints an **opaque** handle, `transport.fetchAsset` fills it, +`renderer.stageSegment` consumes it. In Node a handle is a filesystem path; in +the browser it is an OPFS path or a key into a buffer map. Treat handles as +values to pass along, never to parse. This is why +[download-plan.js](download-plan.js) takes `destinationFor` injected: it slots +assets by frame index without knowing what a destination is. + +## Constraints a new platform must respect + +1. **`fetchAsset` must not be cacheable** (`cache: 'no-store'`). The segment + loop has to keep generating real network load or the shaped-bandwidth + experiment stops meaning anything — the same reason + `system/Client/decode_cache.py` caches decodes but never downloads. +2. **`lifecycle.exit` must not navigate away.** The final `POST /api/results` + happens during shutdown. Stop timers, resolve the run promise, leave the + page alive. +3. **`openAppendStream.write` must not block.** Render telemetry arrives at + 30 Hz; in Node a synchronous write per frame filled the renderer's stdout + pipe and blocked the loop calling `poll_events()`, so client disk I/O was + directly stalling the Open3D window. Buffer and flush. (OPFS + `createSyncAccessHandle` is worker-only.) +4. **Keep the `renderer: null` path working.** Simulated mode credits segments + straight from the download result; interactive mode waits for + `object_ready`. They measure different things and both must survive. + +## Testing against the fake platform + +`createFakePlatform()` returns a contract-valid platform with a **virtual +clock**, so a 40-segment run costs a millisecond of test time instead of +80 seconds. `tests/test_streaming_client.js` drives complete runs this way. + +```js +const { createFakePlatform } = require('../system/ClientCore/testing/fake-platform'); + +const platform = createFakePlatform(); +platform.transport.onGet('/api/config', () => ({ + segmentDuration: 2, framesPerSegment: 60, segmentIntervalMs: 2000, + totalSegments: 20, updateIntervalSegments: 1 +})); +platform.transport.onGet('/api/manifest', () => menu); +platform.transport.failAsset('dancer_fr0003'); // one bad geometry frame + +await platform.clock.advance(20 * 2000); // 20 segments, instantly + +assert.ok(platform.storage.result); // the run wrote its metrics +assert.strictEqual(platform.clock.pending, 0); // no timer leaked +``` + +Assert on observable state — `storage.files`, `transport.requests`, +`renderer.staged`, `logger.at('WARN')` — rather than on call expectations. + +```bash +node --test tests/test_bandwidth_estimator.js tests/test_client_core.js \ + tests/test_client_core_parity.js tests/test_client_platform.js \ + tests/test_client_core_purity.js tests/test_streaming_client.js +``` + +`node --test ` does not work in this environment; name the files. +`test_client_core_parity.js` runs the extracted download planner against the +pre-refactor implementation across 12 manifest shapes — keep it until the +end-to-end parity harness (same trace through both clients, compared with +`vstream/evaluation/QoE_full.py`) is running. diff --git a/open4d/streaming/system/ClientCore/abr.js b/open4d/streaming/system/ClientCore/abr.js new file mode 100644 index 00000000..8ec59dc9 --- /dev/null +++ b/open4d/streaming/system/ClientCore/abr.js @@ -0,0 +1,1038 @@ +// MCKP DP-based ABR with per-object buffers (was Client/client-abr-mckp.js) +// V5.1: Fixed budget usage - don't skip objects when bandwidth is abundant + +// MISSING = unintentional starvation (real stall, playback halted for the +// object). FROZEN = deliberate policy decision under bandwidth deficit: the +// object keeps showing its last frame while the rest of the scene plays. +// Frozen time is tracked separately from stall time and penalized less, but +// the penalty grows with frozen duration so long-frozen objects get +// reconsidered when bandwidth allows. +'use strict'; + +/** + * Both platform seams are mandatory throughout this module — see the note in + * each constructor. Failing at construction is deliberate: the alternative is a + * test that appears to pass while reading wall time. + */ +function requireSeam(value, name) { + if (typeof value !== 'function') { + throw new TypeError( + `MCKP ABR requires a ${name} function (pass platform.clock.now / a logger)`); + } + return value; +} + +const STALL_COSTS = { 'OK': 0.0, 'MISSING': 1.0, 'FROZEN': 0.3 }; + +class ObjectBuffer { + constructor(objectName, maxBufferSec = 20, minBufferSec = 3, deps = {}) { + // Platform seams, both REQUIRED. A Date.now default would let a test + // driving a virtual clock silently measure wall time (the startup grace + // period below is wall-time based), and a console default would keep a + // platform global in ClientCore forever. Callers pass platform.clock.now + // and a logger; see ClientCore/platform.js. + this._now = requireSeam(deps.now, 'now'); + this._log = requireSeam(deps.log, 'log'); + this.objectName = objectName; + this.maxBuffer = maxBufferSec; + this.minBuffer = minBufferSec; + this.level = 0; + this.state = 'OK'; + this.stallDuration = 0; + this.totalStallTime = 0; + this.stallEvents = []; + this.initialized = false; + this.segmentsDownloaded = 0; + this.segmentsConsumed = 0; + this.stallStartTimestamp = null; + this.everHadContent = false; + this.segmentStallTime = 0; + this.segmentStallCount = 0; + this.segmentsSkipped = 0; + this.lastSkipReason = null; + this.independentPlaybackTime = 0; + // Deliberate freeze (bandwidth-deficit policy) vs unintentional stall + this.frozen = false; + this.frozenDuration = 0; // current continuous frozen time + this.totalFrozenTime = 0; + this.segmentFrozenTime = 0; + this.frozenSegments = 0; + } + + freeze(reason = 'bandwidth-deficit') { + if (!this.frozen) this.frozenDuration = 0; + this.frozen = true; + this.lastSkipReason = reason; + this.frozenSegments++; + } + + unfreeze() { + this.frozen = false; + this.frozenDuration = 0; + if (this.state === 'FROZEN') { + this.state = this.level > 0 ? 'OK' : 'MISSING'; + } + } + + addSegment(segmentDuration = 1.0) { + const prevLevel = this.level; + const wasStalling = this.state === 'MISSING'; + + this.level = Math.min(this.maxBuffer, this.level + segmentDuration); + this.initialized = true; + this.everHadContent = true; + this.segmentsDownloaded++; + this.unfreeze(); // fresh content ends a deliberate freeze + + let stallEndEvent = null; + + if (wasStalling && this.level > 0) { + stallEndEvent = this._endStall(this._now()); + } + + return { + prevLevel, + newLevel: this.level, + added: segmentDuration, + stallEndEvent + }; + } + + skipSegment(reason = 'low-priority') { + this.segmentsSkipped++; + this.lastSkipReason = reason; + return { skipped: true, reason, level: this.level }; + } + + consume(elapsedWallTime, timestamp = 0) { + this.segmentsConsumed++; + + const prevLevel = this.level; + const prevState = this.state; + + if (this.level <= 0) { + if (this.frozen) { + this._recordFrozenTime(elapsedWallTime); + return null; // deliberate freeze: last frame shown, no stall event + } + const stallTime = elapsedWallTime; + this._recordStallTime(stallTime); + + if (prevState === 'OK') { + return this._startStallEvent(timestamp, prevLevel, stallTime, 'buffer-empty'); + } + + return this._reportOngoingStall(timestamp, stallTime); + } + + const playableTime = Math.min(this.level, elapsedWallTime); + const stallTime = elapsedWallTime - playableTime; + + this.level = Math.max(0, this.level - playableTime); + this.independentPlaybackTime += playableTime; + + if (stallTime <= 0) { + if (prevState === 'MISSING') { + return this._endStall(timestamp); + } + // playing buffered content counts as OK even under a freeze + // decision; FROZEN only applies once the buffer runs dry + this.state = 'OK'; + return null; + } + + if (this.frozen) { + this._recordFrozenTime(stallTime); + return null; + } + + this._recordStallTime(stallTime); + + if (prevState === 'OK') { + return this._startStallEvent(timestamp, prevLevel, stallTime, 'buffer-depleted'); + } + + return this._reportOngoingStall(timestamp, stallTime); + } + + _recordStallTime(stallTime) { + this.state = 'MISSING'; + this.stallDuration += stallTime; + this.totalStallTime += stallTime; + this.segmentStallTime += stallTime; + } + + _recordFrozenTime(frozenTime) { + this.state = 'FROZEN'; + this.frozenDuration += frozenTime; + this.totalFrozenTime += frozenTime; + this.segmentFrozenTime += frozenTime; + } + + _startStallEvent(timestamp, bufferBefore, stallTime, reason) { + this.stallStartTimestamp = timestamp; + this.segmentStallCount++; + + this.stallEvents.push({ + startTime: timestamp, + endTime: null, + duration: 0, + objectName: this.objectName, + reason: this.everHadContent ? reason : 'no-data-received' + }); + + return { + objectName: this.objectName, + timestamp, + type: 'start', + bufferBefore, + stallTime, + reason: this.everHadContent ? reason : 'no-data-received' + }; + } + + _reportOngoingStall(timestamp, stallTime) { + const prevSeconds = Math.floor(this.stallDuration - stallTime); + const currSeconds = Math.floor(this.stallDuration); + + if (currSeconds > prevSeconds) { + return { + objectName: this.objectName, + timestamp, + type: 'ongoing', + stallDuration: this.stallDuration, + stallTime, + bufferLevel: this.level + }; + } + return null; + } + + _endStall(timestamp) { + const stallInfo = { + objectName: this.objectName, + timestamp, + type: 'end', + stallDuration: this.stallDuration, + bufferAfter: this.level + }; + + if (this.stallEvents.length > 0) { + const lastEvent = this.stallEvents[this.stallEvents.length - 1]; + if (lastEvent.endTime === null) { + lastEvent.endTime = timestamp; + lastEvent.duration = this.stallDuration; + } + } + + this.stallDuration = 0; + this.stallStartTimestamp = null; + this.state = 'OK'; + + return stallInfo; + } + + getSegmentStallStats() { + return { + stallTime: this.segmentStallTime, + stallCount: this.segmentStallCount, + isCurrentlyStalling: this.state === 'MISSING', + currentOngoingStallDuration: this.stallDuration, + bufferLevel: this.level, + totalStallTime: this.totalStallTime, + frozenTime: this.segmentFrozenTime, + isFrozen: this.state === 'FROZEN' || this.frozen + }; + } + + resetSegmentStalls() { + this.segmentStallTime = 0; + this.segmentStallCount = 0; + this.segmentFrozenTime = 0; + } + + getStatus() { + return { + objectName: this.objectName, + level: this.level, + state: this.state, + stallDuration: this.stallDuration, + totalStallTime: this.totalStallTime, + stallEventCount: this.stallEvents.length, + isHealthy: this.level >= this.minBuffer, + isCritical: this.level < 1.0, + initialized: this.initialized, + everHadContent: this.everHadContent, + segmentsDownloaded: this.segmentsDownloaded, + segmentsConsumed: this.segmentsConsumed, + segmentsSkipped: this.segmentsSkipped, + segmentStallTime: this.segmentStallTime, + segmentStallCount: this.segmentStallCount, + independentPlaybackTime: this.independentPlaybackTime, + frozen: this.frozen, + frozenDuration: this.frozenDuration, + totalFrozenTime: this.totalFrozenTime, + frozenSegments: this.frozenSegments + }; + } + + reset() { + this.level = 0; + this.state = 'OK'; + this.stallDuration = 0; + this.totalStallTime = 0; + this.stallEvents = []; + this.initialized = false; + this.everHadContent = false; + this.segmentsDownloaded = 0; + this.segmentsConsumed = 0; + this.segmentsSkipped = 0; + this.stallStartTimestamp = null; + this.segmentStallTime = 0; + this.segmentStallCount = 0; + this.lastSkipReason = null; + this.independentPlaybackTime = 0; + } +} + +class PerObjectBufferManager { + constructor(config = {}, deps = {}) { + // Platform seams, both REQUIRED. A Date.now default would let a test + // driving a virtual clock silently measure wall time (the startup grace + // period below is wall-time based), and a console default would keep a + // platform global in ClientCore forever. Callers pass platform.clock.now + // and a logger; see ClientCore/platform.js. + this._now = requireSeam(deps.now, 'now'); + this._log = requireSeam(deps.log, 'log'); + this._deps = deps; + this.maxBuffer = config.maxBuffer ?? 15; + this.minBuffer = config.minBuffer ?? 2; + this.buffers = new Map(); + this.playbackStarted = false; + this.startupBufferTarget = config.startupBuffer ?? 2; + this.segmentCount = 0; + this.totalPlaybackTime = 0; + this.firstSegmentRequestTime = null; + this.startupGracePeriod = config.startupGracePeriod ?? 3; + } + + getBuffer(objectName) { + if (!this.buffers.has(objectName)) { + this.buffers.set(objectName, new ObjectBuffer( + objectName, + this.maxBuffer, + this.minBuffer, + this._deps + )); + } + return this.buffers.get(objectName); + } + + initFromManifest(manifest) { + for (const objName of Object.keys(manifest.objects)) { + this.getBuffer(objName); + } + if (this.firstSegmentRequestTime === null) { + this.firstSegmentRequestTime = this._now(); + } + } + + addSegmentForObject(objectName, segmentDuration = 1.0) { + const buffer = this.getBuffer(objectName); + const result = buffer.addSegment(segmentDuration); + return { [objectName]: result, stallEndEvent: result.stallEndEvent }; + } + + addDownloadedSegments(downloadedObjects, segmentDuration = 1.0) { + const results = {}; + const stallEndEvents = []; + + for (const [objName, info] of Object.entries(downloadedObjects)) { + const buffer = this.getBuffer(objName); + if (info.downloaded) { + const result = buffer.addSegment(segmentDuration); + results[objName] = result; + if (result.stallEndEvent) { + stallEndEvents.push(result.stallEndEvent); + } + } else if (info.skipped) { + results[objName] = buffer.skipSegment(info.reason || 'low-priority'); + } else { + results[objName] = { prevLevel: buffer.level, newLevel: buffer.level, added: 0 }; + } + } + this.segmentCount++; + return { results, stallEndEvents }; + } + + shouldStartPlayback() { + if (this.playbackStarted) return true; + + if (this.firstSegmentRequestTime !== null) { + const elapsed = (this._now() - this.firstSegmentRequestTime) / 1000; + if (elapsed >= this.startupGracePeriod) { + this._log(`[BUFFER] Starting playback after ${elapsed.toFixed(1)}s grace period`); + this.playbackStarted = true; + return true; + } + } + + const minLevel = this.getMinBufferLevel(); + if (minLevel >= this.startupBufferTarget) { + this._log(`[BUFFER] Starting playback with ${minLevel.toFixed(2)}s buffer`); + this.playbackStarted = true; + return true; + } + return false; + } + + consumeAll(elapsedWallTime = 1.0, timestamp = 0) { + if (!this.shouldStartPlayback()) return []; + + this.totalPlaybackTime += elapsedWallTime; + const stallEvents = []; + + for (const buffer of this.buffers.values()) { + const event = buffer.consume(elapsedWallTime, timestamp); + if (event) stallEvents.push(event); + } + + return stallEvents; + } + + getSegmentStallStatus() { + let totalSegmentStallDuration = 0; + let totalSegmentStallCount = 0; + let stallingCount = 0; + const perObject = {}; + + for (const [name, buffer] of this.buffers.entries()) { + const stats = buffer.getSegmentStallStats(); + perObject[name] = stats; + totalSegmentStallDuration += stats.stallTime; + totalSegmentStallCount += stats.stallCount; + if (stats.isCurrentlyStalling) stallingCount++; + } + + return { + totalSegmentStallDuration, + totalSegmentStallCount, + stallingCount, + perObject + }; + } + + resetAllSegmentStalls() { + for (const buffer of this.buffers.values()) { + buffer.resetSegmentStalls(); + } + } + + getAndResetSegmentStalls() { + const status = this.getSegmentStallStatus(); + this.resetAllSegmentStalls(); + return status; + } + + getStallStates() { + const states = {}; + for (const [name, buffer] of this.buffers.entries()) { + states[name] = buffer.state; + } + return states; + } + + getStallDurations() { + const durations = {}; + for (const [name, buffer] of this.buffers.entries()) { + durations[name] = buffer.stallDuration; + } + return durations; + } + + getTotalStallTimes() { + const times = {}; + for (const [name, buffer] of this.buffers.entries()) { + times[name] = buffer.totalStallTime; + } + return times; + } + + getFrozenDurations() { + const durations = {}; + for (const [name, buffer] of this.buffers.entries()) { + durations[name] = buffer.frozenDuration; + } + return durations; + } + + getTotalFrozenMetrics() { + let sumFrozenTime = 0; + let frozenSegments = 0; + const perObject = {}; + for (const buffer of this.buffers.values()) { + perObject[buffer.objectName] = { + totalFrozenTime: buffer.totalFrozenTime, + frozenSegments: buffer.frozenSegments + }; + sumFrozenTime += buffer.totalFrozenTime; + frozenSegments += buffer.frozenSegments; + } + return { sumFrozenTime, frozenSegments, perObject }; + } + + getBufferLevels() { + const levels = {}; + for (const [name, buffer] of this.buffers.entries()) { + levels[name] = buffer.level; + } + return levels; + } + + getMinBufferLevel({ excludeFrozen = false } = {}) { + let min = Infinity; + for (const buffer of this.buffers.values()) { + if (excludeFrozen && buffer.frozen) continue; + min = Math.min(min, buffer.level); + } + return min === Infinity ? 0 : min; + } + + getBufferLevel(objectName) { + const buffer = this.buffers.get(objectName); + return buffer ? buffer.level : 0; + } + + getSummary() { + const statuses = Array.from(this.buffers.values()).map(b => b.getStatus()); + const missingCount = statuses.filter(s => s.state === 'MISSING').length; + const frozenCount = statuses.filter(s => s.state === 'FROZEN' || s.frozen).length; + const criticalCount = statuses.filter(s => s.isCritical).length; + const avgLevel = statuses.length > 0 + ? statuses.reduce((sum, s) => sum + s.level, 0) / statuses.length + : 0; + + return { + totalObjects: statuses.length, + missingCount, + frozenCount, + criticalCount, + avgBufferLevel: avgLevel, + minBufferLevel: statuses.length > 0 ? Math.min(...statuses.map(s => s.level)) : 0, + maxBufferLevel: statuses.length > 0 ? Math.max(...statuses.map(s => s.level)) : 0, + allHealthy: missingCount === 0 && criticalCount === 0, + playbackStarted: this.playbackStarted, + segmentCount: this.segmentCount, + totalPlaybackTime: this.totalPlaybackTime, + perObjectStallTimes: Object.fromEntries( + statuses.map(s => [s.objectName, s.totalStallTime]) + ), + objects: statuses + }; + } + + getTotalStallMetrics() { + let maxStallTime = 0; + let sumStallTime = 0; + let totalStallEvents = 0; + const perObject = {}; + + for (const buffer of this.buffers.values()) { + const objStallTime = buffer.totalStallTime; + const completedEvents = buffer.stallEvents.filter(e => e.endTime !== null).length; + const ongoingEvent = buffer.state === 'MISSING' ? 1 : 0; + + perObject[buffer.objectName] = { + totalStallTime: objStallTime, + stallEventCount: completedEvents + ongoingEvent + }; + + maxStallTime = Math.max(maxStallTime, objStallTime); + sumStallTime += objStallTime; + totalStallEvents += completedEvents + ongoingEvent; + } + + return { + totalStallTime: maxStallTime, + sumStallTime, + avgStallTime: this.buffers.size > 0 ? sumStallTime / this.buffers.size : 0, + totalStallEvents, + perObject + }; + } + + reset() { + for (const buffer of this.buffers.values()) { + buffer.reset(); + } + this.playbackStarted = false; + this.segmentCount = 0; + this.totalPlaybackTime = 0; + this.firstSegmentRequestTime = null; + } +} + +class MCKPAdaptiveBitrate { + constructor(config = {}, deps = {}) { + // Platform seams, both REQUIRED. A Date.now default would let a test + // driving a virtual clock silently measure wall time (the startup grace + // period below is wall-time based), and a console default would keep a + // platform global in ClientCore forever. Callers pass platform.clock.now + // and a logger; see ClientCore/platform.js. + this._now = requireSeam(deps.now, 'now'); + this._log = requireSeam(deps.log, 'log'); + this.delta = config.delta ?? 0.1; + this.switchPenalty = config.switchPenalty ?? 0.25; + this.stallLambda = config.stallLambda ?? 20.0; + this.stallGamma = config.stallGamma ?? 2.0; + this.stallBeta = config.stallBeta ?? 1.3; + + this.enablePriorityDropping = config.enablePriorityDropping ?? true; + this.minObjectsToDownload = config.minObjectsToDownload ?? 1; + this.bufferThresholdForDropping = config.bufferThresholdForDropping ?? 3.0; + this.highBufferThreshold = config.highBufferThreshold ?? 5.0; + this.abundantBandwidthMultiplier = config.abundantBandwidthMultiplier ?? 1.5; // NEW + + this.bufferManager = new PerObjectBufferManager({ + maxBuffer: config.maxBuffer ?? 15, + minBuffer: config.minBuffer ?? 2, + startupBuffer: config.startupBuffer ?? 2, + startupGracePeriod: config.startupGracePeriod ?? 3 + }, deps); + + this.prevRepByObj = new Map(); + this.prevQualityByObj = new Map(); + } + + getWeightsFromManifest(manifest) { + const weights = {}; + for (const [objName, objData] of Object.entries(manifest.objects)) { + weights[objName] = objData.weight || 1.0; + } + return weights; + } + + calculateStallPenalty(weights) { + if (this.stallLambda === 0) return { total: 0, perObject: {} }; + + const stallStates = this.bufferManager.getStallStates(); + const stallDurations = this.bufferManager.getStallDurations(); + const frozenDurations = this.bufferManager.getFrozenDurations(); + const totalStallTimes = this.bufferManager.getTotalStallTimes(); + + let totalPenalty = 0; + const perObject = {}; + + for (const [objName, weight] of Object.entries(weights)) { + const state = stallStates[objName] || 'OK'; + // FROZEN uses its own (continuous) duration and a lower cost: + // deliberate freezes are cheaper than real stalls, but the + // penalty still grows so long-frozen objects get unfrozen when + // bandwidth allows. + const currentDuration = state === 'FROZEN' + ? (frozenDurations[objName] || 0) + : (stallDurations[objName] || 0); + const totalDuration = totalStallTimes[objName] || 0; + const cost = STALL_COSTS[state] || 0; + + let penalty = 0; + if (cost > 0 && currentDuration > 0) { + penalty = this.stallLambda * + Math.pow(weight * cost, this.stallGamma) * + Math.pow(currentDuration, this.stallBeta); + } + + perObject[objName] = { + state, + currentDuration, + totalDuration, + cost, + penalty + }; + totalPenalty += penalty; + } + return { total: totalPenalty, perObject }; + } + + calculateSwitchPenalty(objName, newQuality, weight) { + if (this.switchPenalty === 0) return 0; + const prevQuality = this.prevQualityByObj.get(objName) || 0; + const drop = Math.max(0, prevQuality - newQuality); + return weight * this.switchPenalty * drop; + } + + calculateEffectivePriority(objName, viewpointPriorities, weights) { + const vp = viewpointPriorities[objName] || { inFOV: true, priority: 3, distance: 5 }; + const weight = weights[objName] || 1.0; + const bufferLevel = this.bufferManager.getBufferLevel(objName); + + let priority = 10 - (vp.priority || 3); + + if (vp.inFOV) priority += 5; + priority += weight * 2; + + if (bufferLevel < 1.0) priority += 10; + else if (bufferLevel < 2.0) priority += 5; + else if (bufferLevel < 3.0) priority += 2; + + if (bufferLevel > 8.0) priority -= 3; + else if (bufferLevel > 5.0) priority -= 1; + + priority -= Math.min(5, (vp.distance || 5) / 2); + + return { + objectName: objName, + effectivePriority: priority, + inFOV: vp.inFOV, + viewpointPriority: vp.priority, + distance: vp.distance, + weight, + bufferLevel + }; + } + + // FIXED: Don't skip objects when bandwidth is abundant + selectObjectsToDownload(manifest, budget, viewpointPriorities) { + const weights = this.getWeightsFromManifest(manifest); + const objectNames = Object.keys(manifest.objects); + + // Calculate minimum total bitrate needed (lowest rep for each object) + const minReps = {}; + let minTotalBitrate = 0; + + for (const [objName, objData] of Object.entries(manifest.objects)) { + const minRep = objData.representations.reduce((a, b) => + a.predicted.bitrate_mbps < b.predicted.bitrate_mbps ? a : b + ); + minReps[objName] = minRep; + minTotalBitrate += minRep.predicted.bitrate_mbps; + } + + // Check if we have abundant bandwidth (>1.5x minimum needed) + const abundantBandwidth = budget > (minTotalBitrate * this.abundantBandwidthMultiplier); + // Sustained deficit: even the cheapest full-scene ladder exceeds the + // budget. Some objects must be frozen (keep showing their last frame) + // so the rest fit. + const deficit = budget < minTotalBitrate; + + const priorities = objectNames.map(objName => + this.calculateEffectivePriority(objName, viewpointPriorities, weights) + ); + + priorities.sort((a, b) => b.effectivePriority - a.effectivePriority); + + const selectedObjects = []; + const skippedObjects = []; + let usedBudget = 0; + + if (deficit) { + // Freeze order = ascending manifest weight (the server-computed + // viewpoint importance): keep the highest-weight objects live, + // freeze from the least important upward until the rest fit. + const byWeight = [...objectNames].sort((a, b) => { + const dw = (weights[b] || 0) - (weights[a] || 0); + if (dw !== 0) return dw; + const pa = priorities.find(p => p.objectName === a)?.effectivePriority || 0; + const pb = priorities.find(p => p.objectName === b)?.effectivePriority || 0; + return pb - pa; + }); + + for (const objName of byWeight) { + const minRep = minReps[objName]; + const minBitrate = minRep.predicted.bitrate_mbps; + const mustInclude = selectedObjects.length < this.minObjectsToDownload; + const fitsInBudget = (usedBudget + minBitrate) <= budget; + + if (mustInclude || fitsInBudget) { + selectedObjects.push(objName); + usedBudget += minBitrate; + this.bufferManager.getBuffer(objName).unfreeze(); + } else { + skippedObjects.push({ + objectName: objName, + reason: 'frozen-bandwidth-deficit', + frozen: true, + weight: weights[objName] || 0, + bufferLevel: this.bufferManager.getBufferLevel(objName), + repId: minRep.id, + bitrate: minBitrate, + quality: minRep.predicted.quality + }); + this.bufferManager.getBuffer(objName).freeze('bandwidth-deficit'); + } + } + } else { + for (const p of priorities) { + const objName = p.objectName; + const minRep = minReps[objName]; + const minBitrate = minRep.predicted.bitrate_mbps; + + const mustInclude = selectedObjects.length < this.minObjectsToDownload; + const bufferLow = p.bufferLevel < this.highBufferThreshold; + const fitsInBudget = (usedBudget + minBitrate) <= budget; + + if (mustInclude || (fitsInBudget && (bufferLow || abundantBandwidth))) { + selectedObjects.push(objName); + usedBudget += minBitrate; + this.bufferManager.getBuffer(objName).unfreeze(); + } else { + // High buffer with tight (but sufficient) bandwidth, or a + // straggler that no longer fits: skip this segment. Not a + // freeze - the object still has buffer to play. + skippedObjects.push({ + objectName: objName, + reason: fitsInBudget ? 'high-buffer' : 'budget-exceeded', + frozen: false, + weight: weights[objName] || 0, + priority: p.effectivePriority, + bufferLevel: p.bufferLevel, + repId: minRep.id, + bitrate: minBitrate, + quality: minRep.predicted.quality + }); + } + } + } + + return { + selectedObjects, + skippedObjects, + frozenObjects: skippedObjects.filter(s => s.frozen).map(s => s.objectName), + priorities, + usedBudget, + totalBudget: budget, + abundantBandwidth, + deficit, + minTotalBitrate + }; + } + + solveMCKP_DP(keptRepsByObj, weights, budgetMbps) { + const objNames = Object.keys(keptRepsByObj); + const n = objNames.length; + + if (n === 0) return { totalBitrate: 0, totalValue: 0, choices: {} }; + + // One quantization for budget and item costs (was 100 vs a shadowed + // 10 inside the loop, inflating the DP budget 10x and misreporting + // the selected bitrate). + const SCALE = Math.round(1 / this.delta); // e.g. 10 for delta=0.1 + const maxBudget = Math.floor(budgetMbps * SCALE); + + const minPossibleBitrate = objNames.reduce((sum, objName) => { + const reps = keptRepsByObj[objName]; + const minRep = reps.reduce((a, b) => + a.predicted.bitrate_mbps < b.predicted.bitrate_mbps ? a : b + ); + return sum + minRep.predicted.bitrate_mbps; + }, 0); + + if (budgetMbps < minPossibleBitrate) { + const choices = {}; + let usedBudget = 0; + let totalValue = 0; + + for (const objName of objNames) { + const reps = keptRepsByObj[objName]; + const lowest = reps.reduce((a, b) => + a.predicted.bitrate_mbps < b.predicted.bitrate_mbps ? a : b + ); + choices[objName] = lowest; + usedBudget += lowest.predicted.bitrate_mbps; + totalValue += (weights[objName] || 1.0) * lowest.predicted.quality; + } + return { totalBitrate: usedBudget, totalValue, choices }; + } + + let dp = new Map(); + dp.set(maxBudget, { value: 0, choices: {} }); + + for (const objName of objNames) { + const reps = keptRepsByObj[objName]; + const w = weights[objName] || 1.0; + const newDp = new Map(); + + for (const [remainBudget, state] of dp.entries()) { + for (const rep of reps) { + const bitrate = Math.floor(rep.predicted.bitrate_mbps * SCALE + 1e-9); + + if (bitrate <= remainBudget) { + const quality = rep.predicted.quality; + const switchPen = this.calculateSwitchPenalty(objName, quality, w); + const repValue = w * quality - switchPen; + const newBudget = remainBudget - bitrate; + const newValue = state.value + repValue; + + const existing = newDp.get(newBudget); + if (!existing || newValue > existing.value) { + newDp.set(newBudget, { + value: newValue, + choices: { ...state.choices, [objName]: rep } + }); + } + } + } + } + + dp = newDp; + + } + + let bestValue = -Infinity; + let bestChoices = null; + let usedBudget = 0; + + for (const [remainBudget, state] of dp.entries()) { + if (Object.keys(state.choices).length === n && state.value > bestValue) { + bestValue = state.value; + bestChoices = state.choices; + usedBudget = (maxBudget - remainBudget) / SCALE; + } + } + + if (!bestChoices) { + bestChoices = {}; + usedBudget = 0; + bestValue = 0; + + for (const objName of objNames) { + const reps = keptRepsByObj[objName]; + const lowest = reps.reduce((a, b) => + a.predicted.bitrate_mbps < b.predicted.bitrate_mbps ? a : b + ); + bestChoices[objName] = lowest; + usedBudget += lowest.predicted.bitrate_mbps; + bestValue += (weights[objName] || 1.0) * lowest.predicted.quality; + } + } + + return { totalBitrate: usedBudget, totalValue: bestValue, choices: bestChoices }; + } + + selectBestCombination(manifest, bitrateBudget, options = {}) { + const { viewpointPriorities = {} } = options; + + this.bufferManager.initFromManifest(manifest); + + const weights = this.getWeightsFromManifest(manifest); + + let safetyMargin = 1; + const safeBudget = bitrateBudget * safetyMargin; + + let objectSelection = null; + let keptRepsByObj = {}; + + if (this.enablePriorityDropping) { + objectSelection = this.selectObjectsToDownload(manifest, safeBudget, viewpointPriorities); + + for (const objName of objectSelection.selectedObjects) { + keptRepsByObj[objName] = manifest.objects[objName].representations; + } + + const frozen = objectSelection.skippedObjects.filter(s => s.frozen); + const softSkips = objectSelection.skippedObjects.filter(s => !s.frozen); + + if (frozen.length > 0) { + const detail = frozen.map(s => + `${s.objectName}(w:${(s.weight || 0).toFixed(3)},buf:${s.bufferLevel.toFixed(1)})` + ).join(', '); + this._log(` [FREEZE] Bandwidth deficit (budget ${safeBudget.toFixed(1)} < min ladder ${objectSelection.minTotalBitrate.toFixed(1)} Mbps): freezing ${frozen.length} lowest-weight objects: ${detail}`); + } + if (softSkips.length > 0) { + const skipReasons = softSkips.map(s => + `${s.objectName}(buf:${s.bufferLevel.toFixed(1)},${s.reason})` + ).join(', '); + this._log(` [PRIORITY-DROP] Skipping ${softSkips.length} objects: ${skipReasons}`); + } + } else { + for (const [objName, objData] of Object.entries(manifest.objects)) { + keptRepsByObj[objName] = objData.representations; + } + } + + const stallPenalties = this.calculateStallPenalty(weights); + + const { totalBitrate, totalValue, choices } = this.solveMCKP_DP( + keptRepsByObj, weights, safeBudget + ); + + for (const [objName, rep] of Object.entries(choices)) { + this.prevRepByObj.set(objName, rep.id); + this.prevQualityByObj.set(objName, rep.predicted.quality); + } + + const totalQuality = Object.values(choices).reduce( + (sum, rep) => sum + rep.predicted.quality, 0 + ); + + return { + combo: choices, + totalBitrate, + totalQuality, + skippedObjects: objectSelection?.skippedObjects || [], + frozenObjects: objectSelection?.frozenObjects || [], + deficit: objectSelection?.deficit || false, + objectSelection, + metadata: { + algorithm: 'mckp-dp-adaptive-budget-v5.2-freeze', + stallPenalties, + adjustedValue: totalValue - stallPenalties.total, + weights, + safetyMargin, + safeBudget, + bufferSummary: this.bufferManager.getSummary(), + priorityDropping: this.enablePriorityDropping, + abundantBandwidth: objectSelection?.abundantBandwidth || false, + minTotalBitrate: objectSelection?.minTotalBitrate || 0 + } + }; + } + + onSegmentDownloaded(downloadedObjects, segmentDuration = 1.0) { + return this.bufferManager.addDownloadedSegments(downloadedObjects, segmentDuration); + } + + onObjectDownloaded(objectName, segmentDuration = 1.0) { + return this.bufferManager.addSegmentForObject(objectName, segmentDuration); + } + + onPlaybackTick(playbackTime = 1.0, timestamp = 0) { + return this.bufferManager.consumeAll(playbackTime, timestamp); + } + + getBufferManager() { + return this.bufferManager; + } + + reset() { + this.prevRepByObj.clear(); + this.prevQualityByObj.clear(); + this.bufferManager.reset(); + } + + getState() { + return { + prevReps: Object.fromEntries(this.prevRepByObj), + bufferSummary: this.bufferManager.getSummary(), + config: { + delta: this.delta, + switchPenalty: this.switchPenalty, + stallLambda: this.stallLambda, + stallGamma: this.stallGamma, + stallBeta: this.stallBeta, + enablePriorityDropping: this.enablePriorityDropping, + minObjectsToDownload: this.minObjectsToDownload, + bufferThresholdForDropping: this.bufferThresholdForDropping, + highBufferThreshold: this.highBufferThreshold, + abundantBandwidthMultiplier: this.abundantBandwidthMultiplier + } + }; + } +} + +module.exports = { + MCKPAdaptiveBitrate, + PerObjectBufferManager, + ObjectBuffer, + STALL_COSTS +}; \ No newline at end of file diff --git a/open4d/streaming/system/ClientCore/bandwidth-estimator.js b/open4d/streaming/system/ClientCore/bandwidth-estimator.js new file mode 100644 index 00000000..dfbc4de4 --- /dev/null +++ b/open4d/streaming/system/ClientCore/bandwidth-estimator.js @@ -0,0 +1,18 @@ +'use strict'; + +/** Return a causal harmonic-mean throughput estimate from recent samples. */ +function harmonicBandwidthEstimate(samples, windowSize = 2) { + if (!Number.isInteger(windowSize) || windowSize <= 0) { + throw new RangeError('windowSize must be a positive integer'); + } + if (!Array.isArray(samples) || samples.length === 0) { + throw new RangeError('at least one bandwidth sample is required'); + } + const recent = samples.slice(-windowSize); + if (recent.some(value => !Number.isFinite(value) || value <= 0)) { + throw new RangeError('bandwidth samples must be finite and positive'); + } + return recent.length / recent.reduce((sum, value) => sum + 1 / value, 0); +} + +module.exports = { harmonicBandwidthEstimate }; diff --git a/open4d/streaming/system/ClientCore/bandwidth.js b/open4d/streaming/system/ClientCore/bandwidth.js new file mode 100644 index 00000000..5ce093ec --- /dev/null +++ b/open4d/streaming/system/ClientCore/bandwidth.js @@ -0,0 +1,187 @@ +'use strict'; + +/** + * Bandwidth estimation and the client's spend budget. + * + * Extracted verbatim from system/Client/client.js. Platform-free: the caller + * injects a clock, a logger, and a `signals()` view of delivery health, so this + * module works unchanged in Node and in a browser. + */ + +const { harmonicBandwidthEstimate } = require('./bandwidth-estimator'); + +const BANDWIDTH_SAMPLE_WINDOW = 5; +const BANDWIDTH_ESTIMATOR_WINDOW = 2; + +/** + * The client commits only a fraction of its bandwidth ESTIMATE each segment. + * + * LIVE system: segments are produced one per segment interval, so a client can + * never buffer ahead more than ~one segment. VOD-style absolute thresholds + * ("buffer <= 2s is critical") are therefore always true here and were used to + * halve the budget permanently. Instead, scale on delivery health: are we + * actually failing to keep up? + * - an object starving unintentionally (MISSING; deliberate freezes excluded), + * - a download backlog at the in-flight cap, + * - or the last segment download arriving late + * all mean the link is behind -> leave recovery headroom. Otherwise use a + * standard safety margin (the estimate is already min-biased). + * + * Mirrors vstream/config.py CLIENT_BUDGET_MULTIPLIER{,_STRUGGLING}: the server + * needs the same number to decide whether the ladder's published floor is + * affordable, so the two must be changed together. + * + * @returns {{budget: number, multiplier: number, reason: string, struggles: string[]}} + */ +function computeBudget(estimate, signals, multiplier, multiplierStruggling) { + const struggles = []; + if (signals.missingCount > 0) struggles.push('missing-objects'); + if (signals.inFlightCount >= signals.maxInFlight) struggles.push('download-backlog'); + if (signals.lastDownloadLate) struggles.push('late-download'); + + const chosen = struggles.length > 0 ? multiplierStruggling : multiplier; + return { + budget: estimate * chosen, + multiplier: chosen, + reason: struggles.length > 0 ? struggles.join('+') : 'normal', + struggles + }; +} + +class BandwidthEstimator { + /** + * @param {object} deps + * @param {() => number} deps.elapsedMs ms since run start, for history stamps + * @param {() => Iterable<[number, {startTime: number, bytesExpected: number}]>} deps.inFlightEntries + * @param {() => {missingCount: number, inFlightCount: number, maxInFlight: number, lastDownloadLate: boolean}} deps.signals + * @param {object} [deps.logger] + * @param {() => number} deps.now platform clock (`platform.clock.now`). + * REQUIRED for the same reason as LinkThroughputMeter's. + * @param {number} [deps.initialEstimate] + * @param {number} [deps.multiplier] + * @param {number} [deps.multiplierStruggling] + */ + constructor({ + elapsedMs, + inFlightEntries, + signals, + logger = null, + now, + initialEstimate = 5, + multiplier = 1, + multiplierStruggling = 1 + }) { + if (typeof now !== 'function') { + throw new TypeError('now must be a function (pass platform.clock.now)'); + } + this._elapsedMs = elapsedMs; + this._inFlightEntries = inFlightEntries; + this._signals = signals; + this._logger = logger; + this._now = now; + this._multiplier = multiplier; + this._multiplierStruggling = multiplierStruggling; + + this.estimate = initialEstimate; + this.samples = []; + /** + * Owned here but shared by reference with the run's metrics record, so + * the serialized output is the same array this module appends to. + */ + this.history = []; + } + + /** Fold one aggregate throughput sample into the estimate. */ + update(downloadSizeBytes, downloadTimeMs) { + if (!downloadSizeBytes || downloadSizeBytes <= 0 + || !downloadTimeMs || downloadTimeMs <= 0) { + return; + } + + const measuredMbps = (downloadSizeBytes * 8) / downloadTimeMs / 1000; + + if (!isFinite(measuredMbps) || measuredMbps <= 0) { + return; + } + + this.samples.push(measuredMbps); + if (this.samples.length > BANDWIDTH_SAMPLE_WINDOW) { + this.samples.shift(); + } + + // Two-sample harmonic mean: causal and responsive, while still weighting + // a low sample more heavily than an arithmetic average. Offline replay of + // all six archived GTA-VI trials reduced next-sample MAPE from 30.7% (the + // old minimum-of-two rule) to 24.6%. The ABR's separate budget multiplier + // keeps headroom; estimator bias and budget safety must not be applied + // twice. + this.estimate = harmonicBandwidthEstimate(this.samples, BANDWIDTH_ESTIMATOR_WINDOW); + + this.history.push({ + timestamp: this._elapsedMs(), + measured: measuredMbps, + estimated: this.estimate, + samples: [...this.samples] + }); + + this._logger?.debug?.('BANDWIDTH', 'Updated estimate', { + measured: measuredMbps.toFixed(2), + harmonic: this.estimate.toFixed(2), + estimated: this.estimate.toFixed(2) + }); + } + + /** + * Current estimate, degraded by any in-flight download that is visibly + * slower than the standing estimate. Only consulted before the first real + * sample lands, otherwise the samples speak for themselves. + */ + current() { + if (this.samples.length === 0) { + const now = this._now(); + for (const [segId, info] of this._inFlightEntries()) { + const elapsedSec = (now - info.startTime) / 1000; + if (elapsedSec > 5 && info.bytesExpected > 0) { + const estimatedMbps = (info.bytesExpected * 8) / (elapsedSec * 1000) / 1000; + if (estimatedMbps < this.estimate) { + this._logger?.warn?.('BANDWIDTH', + 'In-flight download slow, reducing estimate', { + segId, + elapsedSec: elapsedSec.toFixed(1), + oldEstimate: this.estimate.toFixed(2), + newEstimate: Math.max(1, estimatedMbps).toFixed(2) + }); + this.estimate = Math.max(1, estimatedMbps); + } + } + } + } + + return this.estimate; + } + + /** Mbps this segment is allowed to spend. */ + budget() { + const currentBW = this.current(); + const signals = this._signals(); + const result = computeBudget( + currentBW, signals, this._multiplier, this._multiplierStruggling); + + this._logger?.debug?.('BUDGET', 'Calculated', { + estimatedBW: currentBW.toFixed(2), + multiplier: result.multiplier.toFixed(2), + budget: result.budget.toFixed(2), + reason: result.reason, + inFlight: signals.inFlightCount + }); + + return result.budget; + } +} + +module.exports = { + BandwidthEstimator, + computeBudget, + BANDWIDTH_SAMPLE_WINDOW, + BANDWIDTH_ESTIMATOR_WINDOW +}; diff --git a/open4d/streaming/system/ClientCore/download-plan.js b/open4d/streaming/system/ClientCore/download-plan.js new file mode 100644 index 00000000..ec2f1048 --- /dev/null +++ b/open4d/streaming/system/ClientCore/download-plan.js @@ -0,0 +1,189 @@ +'use strict'; + +/** + * Segment download planning and result summarization. + * + * Extracted verbatim from system/Client/client.js `downloadSegmentFiles`. This + * module decides WHAT to fetch and HOW to judge the outcome; it never fetches + * anything, so the same rules apply in Node and in a browser. + * + * The transport half (a sliding window of `DOWNLOAD_CONCURRENCY` workers + * draining `tasks`) stays with the platform adapter, because that is where + * keep-alive agents, destinations and byte crediting live. + */ + +/** + * Build the flat task list for one segment across ALL objects. + * + * Collecting every file up front and draining the list with a sliding window + * means a new request starts the moment one finishes, so the link never idles + * at batch or object boundaries. + * + * @param {object} args + * @param {object} args.selection ABR selection (`combo` maps object -> rep) + * @param {object} args.manifest the published menu.json + * @param {number} args.framesPerSegment geometry frames in this segment + * @param {boolean} args.interactive interactive mode writes files to disk + * @param {(objName: string, repId: string) => (string|null)} args.objectDirFor + * per-object staging directory, or null when nothing is written + * @param {(objName: string, dir: string|null, file: object) => (string|null)} args.destinationFor + * where a planned file should land, or null to keep it in memory + * @param {(assetPath: string) => (string|null)} args.pathToUrl manifest path -> URL + * @returns {{tasks: object[], perObj: Map}} + */ +function planSegmentDownload({ + selection, + manifest, + framesPerSegment, + interactive, + objectDirFor, + destinationFor, + pathToUrl +}) { + const tasks = []; + const perObj = new Map(); // objName -> accumulator (insertion = combo order) + + for (const [objName, rep] of Object.entries(selection.combo)) { + const objData = manifest.objects[objName]; + const startNumber = objData.start_number || 1; + const baseDir = rep.paths.base_dir; + const files = []; + const objectDir = interactive ? objectDirFor(objName, rep.id) : null; + + const textureAssets = ( + Array.isArray(rep.paths.texture_urls) && rep.paths.texture_urls.length > 0 + ? rep.paths.texture_urls + : Array.isArray(rep.paths.texture_mp4s) && rep.paths.texture_mp4s.length > 0 + ? rep.paths.texture_mp4s + : [rep.paths.texture_url || rep.paths.texture_mp4].filter(Boolean) + ); + textureAssets.forEach((textureAsset, textureIndex) => { + const url = pathToUrl(textureAsset); + if (url) { + const file = { url, kind: 'texture', textureIndex, frameNum: null }; + files.push({ + ...file, + destination: interactive + ? destinationFor(objName, objectDir, file) : null + }); + } + }); + + const geometryPattern = rep.paths.geometry_url_pattern || rep.paths.geometry_drc_pattern; + if (geometryPattern) { + for (let f = 0; f < framesPerSegment; f++) { + const frameNum = startNumber + f; + const drcFile = geometryPattern.replace( + '%04d', String(frameNum).padStart(4, '0')); + const asset = drcFile.startsWith('/files/') || /^https?:\/\//.test(drcFile) + ? drcFile + : `${baseDir}/${drcFile}`; + const url = pathToUrl(asset); + if (url) { + const file = { url, kind: 'geometry', frameNum, frameIndex: f }; + files.push({ + ...file, + destination: interactive + ? destinationFor(objName, objectDir, file) : null + }); + } + } + } + + perObj.set(objName, { + repId: rep.id, + size: 0, + success: 0, + total: files.length, + geometryTotal: files.filter(f => f.kind === 'geometry').length, + geometrySuccess: 0, + objectDir, + textureFile: null, + textureFiles: new Array(textureAssets.length).fill(null), + // Index-aligned to the segment's frames. A frame whose download + // fails keeps its slot as null so the renderer can hold the previous + // frame; collecting only the successes used to shift every later + // frame by one and silently desynchronise the clip. + geometryFiles: new Array(framesPerSegment).fill(null) + }); + for (const file of files) tasks.push({ objName, ...file }); + } + + return { tasks, perObj }; +} + +/** + * Credit one completed task against its object's accumulator. + * + * @param {object} acc accumulator from planSegmentDownload + * @param {object} task the planned task + * @param {{success: boolean, size: number}} result transport outcome + * @param {boolean} interactive + */ +function creditTaskResult(acc, task, result, interactive) { + acc.size += result.size; + if (!result.success) return; + + acc.success++; + if (interactive && task.kind === 'texture') { + acc.textureFiles[task.textureIndex] = task.destination; + // Legacy renderer/message consumers still understand one textureFile. + // Multi-group-aware consumers use the list. + if (acc.textureFiles.length === 1) acc.textureFile = task.destination; + } + if (task.kind === 'geometry') { + acc.geometrySuccess++; + if (interactive) acc.geometryFiles[task.frameIndex] = task.destination; + } +} + +/** + * Turn the accumulators into per-object results plus the segment total. + * + * The old `|| acc.size > 0` success rule let a 1-of-61 download report success, + * which then failed hard in the renderer's frame-count check. Geometry is judged + * on its own: the texture is optional (an untextured mesh still plays) but a clip + * needs most of its frames. + * + * @param {Map} perObj + * @returns {{totalSize: number, objects: object[]}} + */ +function summarizeSegmentDownload(perObj) { + let totalSize = 0; + const results = []; + for (const [objName, acc] of perObj.entries()) { + totalSize += acc.size; + const successRatio = acc.total > 0 ? acc.success / acc.total : 0; + const geometryRatio = acc.geometryTotal > 0 ? acc.geometrySuccess / acc.geometryTotal : 0; + results.push({ + objectName: objName, + repId: acc.repId, + filesDownloaded: acc.success, + totalFiles: acc.total, + geometryDownloaded: acc.geometrySuccess, + geometryTotal: acc.geometryTotal, + size: acc.size, + success: successRatio >= 0.9 && geometryRatio >= 0.9, + successRatio, + geometryRatio, + objectDir: acc.objectDir, + textureFiles: acc.textureFiles, + textureFile: acc.textureFile, + geometryFiles: acc.geometryFiles + }); + } + return { totalSize, objects: results }; +} + +/** Expected bytes for a selection, used to seed in-flight download tracking. */ +function expectedSelectionBytes(selection) { + return Object.values(selection.combo).reduce( + (sum, rep) => sum + (rep.predicted.bitrate_mbps * 1000000 / 8), 0); +} + +module.exports = { + planSegmentDownload, + creditTaskResult, + summarizeSegmentDownload, + expectedSelectionBytes +}; diff --git a/open4d/streaming/system/ClientCore/link-throughput.js b/open4d/streaming/system/ClientCore/link-throughput.js new file mode 100644 index 00000000..54e24d10 --- /dev/null +++ b/open4d/streaming/system/ClientCore/link-throughput.js @@ -0,0 +1,100 @@ +'use strict'; + +/** + * Aggregate link throughput measurement. + * + * Extracted verbatim from system/Client/client.js. Per-download measurement is + * biased low whenever segment downloads overlap (each sees a share of the + * link); instead accumulate bytes and busy time across the link's busy periods + * and emit a sample once enough busy time has accumulated (short bursts from + * fast links accumulate across downloads rather than being discarded). + * + * Progressive byte accounting: bytes are credited as each FILE lands, not in + * one lump when the whole segment download finishes. Busy time accumulates + * continuously, so crediting bytes only at segment completion made the + * numerator and denominator cover different windows. When a long download + * completed while the next was still in flight, the rolling-window branch + * divided ONE segment's bytes by a window during which the link had also been + * serving the concurrent download, whose bytes had not landed yet. Measured a + * ~30% under-read (e.g. 58.18 MiB / 3638 ms = 134 Mbps on a ~190 Mbps link), + * which then tripped the drop guard in BandwidthEstimator.update and replaced a + * 245 Mbps estimate with 134 Mbps on a link that had not slowed at all. In one + * 84-sample run the guard fired on 13% of samples at a median 0.70x the + * harmonic mean — a systematic bias, triggered whenever a download overran the + * segment interval (29 late downloads in that run). + * + * `segmentIntervalMs` is a FUNCTION, not a number: the stream config arrives + * from the server after this meter is constructed, so reading a captured value + * would freeze the rolling-window threshold at its pre-config state. + */ + +const LINK_MIN_SAMPLE_MS = 100; // enough busy time for a sample... +const LINK_MIN_SAMPLE_BYTES = 2e6; // ...or enough bytes (fast links finish + // a whole download in tens of ms) + +class LinkThroughputMeter { + /** + * @param {object} deps + * @param {() => number} deps.segmentIntervalMs live segment interval, ms + * @param {(bytes: number, busyMs: number) => void} deps.onSample emit a sample + * @param {() => number} deps.now platform clock (`platform.clock.now`). + * REQUIRED, deliberately: with a Date.now default, a test driving a + * virtual clock that forgot to pass this would silently measure wall + * time and appear to pass. + */ + constructor({ segmentIntervalMs, onSample, now }) { + if (typeof segmentIntervalMs !== 'function') { + throw new TypeError('segmentIntervalMs must be a function'); + } + if (typeof onSample !== 'function') { + throw new TypeError('onSample must be a function'); + } + if (typeof now !== 'function') { + throw new TypeError('now must be a function (pass platform.clock.now)'); + } + this._segmentIntervalMs = segmentIntervalMs; + this._onSample = onSample; + this._now = now; + + this.activeCount = 0; + this.busyStartMs = 0; + this.bytesAccum = 0; + this.busyMsAccum = 0; + } + + started() { + if (this.activeCount === 0) { + this.busyStartMs = this._now(); + } + this.activeCount++; + } + + bytesDelivered(bytes) { + if (bytes > 0) this.bytesAccum += bytes; + } + + finished() { + this.activeCount = Math.max(0, this.activeCount - 1); + const now = this._now(); + + if (this.activeCount === 0) { + this.busyMsAccum += now - this.busyStartMs; + } else if (now - this.busyStartMs >= this._segmentIntervalMs()) { + // sustained load: roll the window so we still sample periodically + this.busyMsAccum += now - this.busyStartMs; + this.busyStartMs = now; + } else { + return; // other downloads still running inside the current window + } + + const enough = this.busyMsAccum >= LINK_MIN_SAMPLE_MS || + (this.bytesAccum >= LINK_MIN_SAMPLE_BYTES && this.busyMsAccum >= 10); + if (enough && this.bytesAccum > 0) { + this._onSample(this.bytesAccum, this.busyMsAccum); + this.bytesAccum = 0; + this.busyMsAccum = 0; + } + } +} + +module.exports = { LinkThroughputMeter, LINK_MIN_SAMPLE_MS, LINK_MIN_SAMPLE_BYTES }; diff --git a/open4d/streaming/system/ClientCore/metrics.js b/open4d/streaming/system/ClientCore/metrics.js new file mode 100644 index 00000000..41e4af5a --- /dev/null +++ b/open4d/streaming/system/ClientCore/metrics.js @@ -0,0 +1,239 @@ +'use strict'; + +/** + * The run's metrics record and its builders. + * + * Extracted from system/Client/client.js (`metrics`, `recordSegmentMetrics`, + * `recordBitrateSelection`, and the summary fill-in at the end of + * `finishStream`). Pure: every builder is a function of its arguments, so the + * serialized shape can be tested without running a stream. + * + * The shape is a published interface, not an internal detail — + * system/Client/calculate-qoe.js, plot-qoe.py and the server's /api/results + * consumer all read these fields by name. Renaming one silently empties a + * column in the QoE report rather than raising anything. + */ + +/** + * @param {object} args + * @param {string} args.clientMode 'interactive' | 'simulated' + * @param {number} args.startTime run epoch, from platform.clock.now() + * @param {Array} args.bandwidthHistory the estimator's OWN array, shared by + * reference so its appends land here without re-assignment + */ +function createMetricsRecord({ clientMode, startTime, bandwidthHistory }) { + return { + startTime, + clientMode, + broadcastId: null, + segments: [], + stalls: [], + objectStalls: {}, + summary: { + totalSegments: 0, + rebuffers: 0, + totalStallDuration: 0, + perObjectStalls: {}, + fallbackCount: 0, + lateDownloads: 0, + missedSegments: 0 + }, + bitrateRequestCounts: {}, + bitrateRequestCountsPerObject: {}, + bufferHistory: [], + bandwidthHistory, + decodeEvents: [], + renderSummary: { + framesPresented: 0, + framesDropped: 0, + objectDecodeFailures: 0, + earlyWindowClose: false + } + }; +} + +/** + * Count one requested representation, both cumulatively and for the current + * reporting interval. `intervalCounts` is mutated and periodically drained by + * the /api/bitrate-counts-interval reporter. + */ +function recordBitrateSelection(metrics, intervalCounts, objectName, repId) { + if (!metrics.bitrateRequestCountsPerObject[objectName]) { + metrics.bitrateRequestCountsPerObject[objectName] = {}; + } + metrics.bitrateRequestCountsPerObject[objectName][repId] = + (metrics.bitrateRequestCountsPerObject[objectName][repId] || 0) + 1; + metrics.bitrateRequestCounts[repId] = (metrics.bitrateRequestCounts[repId] || 0) + 1; + + if (!intervalCounts[objectName]) { + intervalCounts[objectName] = {}; + } + intervalCounts[objectName][repId] = (intervalCounts[objectName][repId] || 0) + 1; +} + +/** + * Per-object decisions for QoE. + * + * calculate-qoe.js reads `seg.objectQualities`: downloaded objects carry the + * selected rep quality; frozen objects carry the LAST SHOWN quality with + * bufferState FROZEN once their buffer runs dry — a frozen object is still + * showing a frame, so scoring it as zero quality would double-count the freeze + * penalty. + */ +function buildObjectQualities({ selection, bufferBefore, prevQualityByObj }) { + const prioByObj = Object.fromEntries( + (selection.objectSelection?.priorities || []).map(p => [p.objectName, p])); + const objectQualities = []; + + for (const [objName, rep] of Object.entries(selection.combo)) { + const bufObj = bufferBefore.objects.find(o => o.objectName === objName) || {}; + const p = prioByObj[objName] || {}; + objectQualities.push({ + objectName: objName, + decision: 'download', + repId: rep.id, + quality: rep.predicted.quality, + bitrate: rep.predicted.bitrate_mbps, + weight: selection.metadata?.weights?.[objName] ?? null, + priority: p.viewpointPriority ?? 3, + inFOV: p.inFOV !== false, + bufferState: bufObj.state ?? 'OK', + bufferLevel: bufObj.level ?? 0 + }); + } + for (const s of (selection.skippedObjects || [])) { + const bufObj = bufferBefore.objects.find(o => o.objectName === s.objectName) || {}; + const p = prioByObj[s.objectName] || {}; + objectQualities.push({ + objectName: s.objectName, + decision: s.frozen ? 'freeze' : 'skip', + reason: s.reason, + repId: null, + quality: prevQualityByObj.get(s.objectName) ?? s.quality ?? 0, + bitrate: 0, + weight: s.weight ?? null, + priority: p.viewpointPriority ?? 3, + inFOV: p.inFOV !== false, + bufferState: bufObj.state ?? 'OK', + bufferLevel: bufObj.level ?? 0 + }); + } + return objectQualities; +} + +/** One row of metrics.segments. */ +function buildSegmentRecord({ + segmentId, selection, bufferBefore, budget, timestamp, segmentStalls, + summary, playbackTime, estimatedBandwidth, inFlightCount, prevQualityByObj +}) { + return { + segmentId, + timestamp, + playbackTime, + selectedBitrate: selection.totalBitrate, + budget, + estimatedBandwidth, + minBufferLevel: bufferBefore.minBufferLevel, + avgBufferLevel: bufferBefore.avgBufferLevel, + missingCount: bufferBefore.missingCount, + frozenCount: (selection.frozenObjects || []).length, + frozenObjects: selection.frozenObjects || [], + deficit: selection.deficit || false, + inFlightDownloads: inFlightCount, + objectQualities: buildObjectQualities({ + selection, bufferBefore, prevQualityByObj + }), + + // PER-SEGMENT stall data (not cumulative) + segmentStallDurationSec: segmentStalls.totalSegmentStallDuration, + segmentStallCount: segmentStalls.totalSegmentStallCount, + stallingObjectCount: segmentStalls.stallingCount, + perObjectSegmentStalls: segmentStalls.perObject, + + // CUMULATIVE totals (for reference) + cumulativeTotalStallSec: summary.objects.reduce( + (sum, o) => sum + o.totalStallTime, 0), + cumulativeTotalFrozenSec: summary.objects.reduce( + (sum, o) => sum + (o.totalFrozenTime || 0), 0) + }; +} + +/** One row of metrics.bufferHistory. */ +function buildBufferHistoryRecord({ + timestamp, playbackTime, bufferBefore, segmentStalls +}) { + return { + timestamp, + playbackTime, + avgLevel: bufferBefore.avgBufferLevel, + minLevel: bufferBefore.minBufferLevel, + missingCount: bufferBefore.missingCount, + stallingCount: segmentStalls.stallingCount, + segmentStallDurationSec: segmentStalls.totalSegmentStallDuration, + // calculate-qoe.js's analyzeBufferHistory expects this; without it the + // whole QoE report threw and no *_qoe.txt was ever produced. + objects: bufferBefore.objects.map(o => ({ + objectName: o.objectName, + level: o.level, + state: o.state + })) + }; +} + +/** + * Fill in metrics.summary at the end of a run. Mutates and returns `metrics`. + * + * @param {object} args + * @param {object} args.metrics + * @param {object} args.bufferManager the ABR's buffer manager + * @param {number} args.totalSegments segments actually ticked + * @param {number} args.totalPlaybackTime seconds + * @param {number} args.totalWallTime seconds + * @param {number} args.estimatedBandwidth final estimate + * @param {number[]} args.bandwidthSamples final sample window + * @param {string|null} args.renderTraceFile + * @param {string|null} args.decodeEventFile + */ +function finalizeMetrics({ + metrics, bufferManager, totalSegments, totalPlaybackTime, totalWallTime, + estimatedBandwidth, bandwidthSamples, renderTraceFile, decodeEventFile +}) { + const finalStallMetrics = bufferManager.getTotalStallMetrics(); + + metrics.summary.totalSegments = totalSegments; + metrics.summary.totalPlaybackTime = totalPlaybackTime; + metrics.summary.totalWallTime = totalWallTime; + metrics.summary.totalStallDuration = finalStallMetrics.totalStallTime * 1000; + metrics.summary.rebuffers = finalStallMetrics.totalStallEvents; + metrics.summary.finalEstimatedBandwidth = estimatedBandwidth; + metrics.summary.bandwidthSamples = [...bandwidthSamples]; + metrics.summary.renderTraceFile = renderTraceFile; + metrics.summary.decodeEventFile = decodeEventFile; + + const frozenMetrics = bufferManager.getTotalFrozenMetrics(); + metrics.summary.totalFrozenTime = frozenMetrics.sumFrozenTime; + metrics.summary.frozenSegments = frozenMetrics.frozenSegments; + metrics.summary.perObjectFrozen = frozenMetrics.perObject; + + for (const [objName, buffer] of bufferManager.buffers.entries()) { + const status = buffer.getStatus(); + metrics.summary.perObjectStalls[objName] = { + count: buffer.stallEvents.length, + totalDuration: status.totalStallTime, + totalFrozenTime: status.totalFrozenTime, + frozenSegments: status.frozenSegments, + finalState: status.state + }; + } + + return { metrics, finalStallMetrics, frozenMetrics }; +} + +module.exports = { + createMetricsRecord, + recordBitrateSelection, + buildObjectQualities, + buildSegmentRecord, + buildBufferHistoryRecord, + finalizeMetrics +}; diff --git a/open4d/streaming/system/ClientCore/payloads.js b/open4d/streaming/system/ClientCore/payloads.js new file mode 100644 index 00000000..39696624 --- /dev/null +++ b/open4d/streaming/system/ClientCore/payloads.js @@ -0,0 +1,207 @@ +'use strict'; + +/** + * Server request bodies. + * + * Extracted from system/Client/client.js (`sendSegmentToServer`, + * `sendDownloadCompleteToServer`, `sendBitrateCountsToServer`, + * `enqueueRenderTraceUpload`). Pure builders: the transport is the platform's + * job, the SHAPE is the core's. + * + * These are wire formats read by system/Server/server.js. Field names are + * load-bearing — `estimatedBandwidth`, `segmentStallDurationSec` and the + * per-object `decision` strings are all parsed by name on the server and by + * vstream/evaluation/QoE*.py afterwards. + */ + +/** Per-object block of the segment report: downloaded objects. */ +function downloadedObjectEntries({ selection, bufferBefore, segmentStalls }) { + return Object.fromEntries( + Object.entries(selection.combo).map(([objName, rep]) => { + const bufferObj = bufferBefore.objects.find( + o => o.objectName === objName) || {}; + const objStall = segmentStalls.perObject[objName] || { + segmentStallDuration: 0, + segmentStallCount: 0 + }; + return [objName, { + decision: 'download', + repId: rep.id, + weight: selection.metadata?.weights?.[objName] ?? null, + bitrate: rep.predicted.bitrate_mbps, + quality: rep.predicted.quality, + bufferLevel: bufferObj.level ?? 0, + bufferState: bufferObj.state ?? 'UNKNOWN', + // Per-segment stall data only + segmentStallDurationSec: objStall.segmentStallDuration, + segmentStallCount: objStall.segmentStallCount, + // Keep cumulative for reference + totalStallDurationSec: bufferObj.totalStallTime ?? 0 + }]; + }) + ); +} + +/** + * Per-object block: skipped and frozen objects. + * + * `decision` distinguishes a deliberate freeze under bandwidth deficit from a + * high-buffer skip. Collapsing them would make a policy decision look like a + * delivery failure in the QoE report. + */ +function skippedObjectEntries({ selection, bufferBefore, segmentStalls }) { + return Object.fromEntries( + (selection.skippedObjects || []).map(s => { + const bufferObj = bufferBefore.objects.find( + o => o.objectName === s.objectName) || {}; + const objStall = segmentStalls.perObject[s.objectName] || {}; + return [s.objectName, { + decision: s.frozen ? 'freeze' : 'skip', + reason: s.reason, + weight: s.weight, + repId: null, + bitrate: 0, + bufferLevel: bufferObj.level ?? 0, + bufferState: bufferObj.state ?? 'UNKNOWN', + segmentStallDurationSec: objStall.segmentStallDuration ?? 0, + segmentFrozenDurationSec: objStall.segmentFrozenDuration ?? 0, + totalStallDurationSec: bufferObj.totalStallTime ?? 0, + totalFrozenTimeSec: bufferObj.totalFrozenTime ?? 0 + }]; + }) + ); +} + +/** + * POST /api/segment/:id + * + * Sent BEFORE downloads start, so ladder generation for this segment's + * viewpoint begins immediately rather than after the transfer. + * + * @param {object} args + * @param {boolean} args.isEmpty no manifest yet — report the tick and nothing else + * @param {object|null} args.data selection/buffer/budget/stalls, null when empty + * @param {object|null} args.viewpoint included only on ladder-update segments + */ +function buildSegmentPayload({ + broadcastId, segmentId, timestamp, playbackTime, estimatedBandwidth, + isEmpty, data, viewpoint, bufferLevels +}) { + const payload = { + broadcastId, + segmentId, + timestamp, + playbackTime, + estimatedBandwidth, + isEmpty + }; + if (isEmpty) return payload; + + payload.viewpoint = viewpoint; + payload.selection = { + segmentId, + bufferLevels, + minBufferLevel: data.bufferBefore.minBufferLevel, + bitrateBudget: data.budget, + totalBitrate: data.selection.totalBitrate, + totalQuality: data.selection.totalQuality, + algorithm: 'mckp-realtime-fixed-bw', + estimatedBandwidth, + playbackTime, + segmentStallDurationSec: data.segmentStalls.totalSegmentStallDuration, + segmentStallEvents: data.segmentStalls.totalSegmentStallCount, + deficit: data.selection.deficit || false, + frozenCount: (data.selection.frozenObjects || []).length, + frozenObjects: data.selection.frozenObjects || [], + objects: { + ...downloadedObjectEntries({ + selection: data.selection, + bufferBefore: data.bufferBefore, + segmentStalls: data.segmentStalls + }), + ...skippedObjectEntries({ + selection: data.selection, + bufferBefore: data.bufferBefore, + segmentStalls: data.segmentStalls + }) + } + }; + return payload; +} + +/** POST /api/segment/:id/download-complete */ +function buildDownloadCompletePayload({ + broadcastId, segmentId, timestamp, estimatedBandwidth, downloadTimeMs, + downloadSizeBytes, measuredBandwidthMbps, bufferAfter, isLate, usedFallback +}) { + return { + broadcastId, + segmentId, + downloadTimeMs, + downloadSizeBytes, + measuredBandwidthMbps, + bufferAfter, + isLate, + usedFallback, + estimatedBandwidth, + timestamp + }; +} + +/** POST /api/bitrate-counts-interval */ +function buildBitrateCountsPayload({ + segmentId, timestamp, intervalMs, countsPerObject +}) { + return { + segmentId, + timestamp, + intervalMs, + countsPerObject: { ...countsPerObject } + }; +} + +/** POST /api/render-frames */ +function buildRenderFramesPayload({ broadcastId, frames }) { + return { broadcastId, frames }; +} + +/** POST /api/viewpoint */ +function buildViewpointPayload({ broadcastId, viewpoint, segId }) { + return { broadcastId, viewpoint, segId }; +} + +/** + * POST /api/broadcast/start + * + * `sceneObjects` restricts the run to a subset of the server's object catalog. + * An absent or empty list resets the server to its full catalog (it stores + * whatever arrives and passes a non-empty list to the ladder service), so a + * run never silently inherits the previous run's scene. + * + * This matters more than it looks: the ladder must publish at least one + * representation per object in the scene, so the scene size sets an + * irreducible bitrate floor. Nine ORBIT objects floor at ~116 Mbps, which no + * ordinary link can carry, and every object the MCKP cannot afford is frozen. + * Choosing the scene is therefore the difference between demonstrating + * adaptation and demonstrating a permanent deficit. + */ +function buildBroadcastStartPayload({ clientMode, label, sceneObjects }) { + const scene = Array.isArray(sceneObjects) + ? sceneObjects.map(name => String(name).trim()).filter(Boolean) : []; + return { + algorithm: `mckp-abr-realtime-${clientMode}`, + label: label || undefined, + sceneObjects: scene.length ? scene : undefined + }; +} + +module.exports = { + buildSegmentPayload, + buildDownloadCompletePayload, + buildBitrateCountsPayload, + buildRenderFramesPayload, + buildViewpointPayload, + buildBroadcastStartPayload, + downloadedObjectEntries, + skippedObjectEntries +}; diff --git a/open4d/streaming/system/ClientCore/platform.js b/open4d/streaming/system/ClientCore/platform.js new file mode 100644 index 00000000..e10eec0a --- /dev/null +++ b/open4d/streaming/system/ClientCore/platform.js @@ -0,0 +1,280 @@ +'use strict'; + +/** + * The platform contract the client core runs on. + * + * `system/ClientCore` holds the streaming logic — segment timing, the MCKP ABR, + * bandwidth estimation, download planning, stall accounting, metrics. It must + * never import `fs`, `http`, `path`, `os`, `process` or touch `console`, + * `Date.now` or `setInterval` directly. Everything platform-shaped goes through + * one object implementing the seven capabilities below, so the Node desktop + * client and the browser client run identical logic. + * + * Derived by auditing every platform touchpoint in system/Client/client.js: + * 10 fetch sites, 9 distinct fs calls, 5 process calls, 4 timer sites, the + * renderer child process, and console logging. Nothing here is speculative — + * each method backs at least one existing call site. + * + * ------------------------------------------------------------------------ + * Asset handles + * ------------------------------------------------------------------------ + * The core never learns where a downloaded byte lives. `storage.handle(...)` + * mints an OPAQUE handle, `transport.fetchAsset` fills it, and + * `renderer.stageSegment` consumes it. The core only moves handles around. + * + * Node handle = a filesystem path string + * Browser handle = an OPFS path, or a key into an in-memory buffer map + * + * This is why ClientCore/download-plan.js takes `destinationFor` injected: the + * planner decides WHICH assets a segment needs and slots them by frame index, + * without knowing what a destination is. Treat handles as values to pass along, + * never to parse. + * + * ------------------------------------------------------------------------ + * What is NOT in the contract + * ------------------------------------------------------------------------ + * Deliberately absent, because they are implementation details of one adapter: + * - HTTP keep-alive agents and socket limits (Node transport internals) + * - the `.part--` write-then-rename dance (Node storage internals) + * - process.env / process.argv: these become the plain config object the core + * is constructed with, not a capability it calls out to + */ + +/** + * @typedef {string|object} AssetHandle + * Opaque locator for a media asset. Minted by `storage.handle`, never parsed + * by the core. + * + * @typedef {object} HttpResult + * @property {boolean} ok transport succeeded AND status was 2xx + * @property {number} status HTTP status, or 0 when the request never landed + * @property {any} body parsed JSON, or null when there was no JSON body + * + * @typedef {object} AssetResult + * @property {boolean} success the asset arrived intact + * @property {number} size bytes received (0 on failure) + * @property {number} timeMs wall time for this asset + * @property {string} [error] failure reason, for logs only + * + * @typedef {object} AppendStream + * @property {(line: string) => void} write MUST NOT block; telemetry runs at 30 Hz + * @property {() => Promise} close resolves once flushed + * + * @typedef {object} TimerHandle opaque, only ever passed back to clock.cancel + */ + +/** + * @typedef {object} TransportAdapter Everything that crosses the network. + * @property {(path: string) => Promise} getJson + * GET a server API path (e.g. '/api/manifest'). The adapter owns the base URL. + * @property {(path: string, body: object) => Promise} postJson + * POST JSON. Resolves for any HTTP status; rejects only if the request never + * completed. Callers decide what a non-ok status means — /api/config treats it + * as fatal, /api/manifest silently skips, telemetry ignores it. + * @property {(assetPath: string) => (string|null)} assetUrl + * Manifest path -> absolute URL. Needs the base URL, hence platform-side. + * Returns null for an unusable path, which the planner treats as "no asset". + * @property {(url: string, handle: AssetHandle|null) => Promise} fetchAsset + * Fetch one media file into `handle`. A null handle means "count the bytes, + * keep nothing" (simulated mode). MUST resolve, never reject: a failed asset + * is normal and is judged by download-plan's 90% rules. + */ + +/** + * @typedef {object} StorageAdapter Durable artifacts and per-run staging. + * @property {(...parts: string[]) => AssetHandle} handle + * Compose a handle under the session scratch. Replaces path.join in the core. + * @property {() => Promise} createScratch + * Per-run staging root (Node: mkdtemp; browser: an OPFS directory). + * @property {(handle: AssetHandle) => Promise} release + * Recursively drop a handle. Called per object-segment once decoded, and once + * for the scratch root at shutdown. MUST tolerate an already-gone handle. + * @property {(name: string, text: string) => Promise} writeText + * Small named debug artifact (the fetched menu.json). + * @property {(text: string) => Promise} writeResult + * The run's final metrics JSON, to wherever this platform keeps results. + * @property {(name: string) => Promise} openAppendStream + * Line-oriented telemetry (render frames, decode events). Streamed rather than + * written at exit so an aborted run keeps its diagnostics. + */ + +/** + * @typedef {object} ViewpointAdapter Where camera poses come from. + * @property {() => Promise>} list + * Ordered poses. Interactive mode uses only [0] as the startup pose and then + * follows the live camera; simulated mode cycles the whole list. + * MUST reject if no pose is available — a run with no initial pose would + * solve the first ladder against a default camera and silently invalidate it. + */ + +/** + * @typedef {object} ClockAdapter All time and scheduling. + * @property {() => number} now epoch ms + * @property {(ms: number, fn: Function) => TimerHandle} every repeating timer + * @property {(handle: TimerHandle) => void} cancel idempotent + * @property {(ms: number) => Promise} delay one-shot, for the bounded + * manifest-fetch race in processSegmentTick + */ + +/** + * @typedef {object} LoggerAdapter Line sink. + * @property {(level: string, line: string, data: object|null) => void} emit + * The core formats the `[wall:+Xs][play:Ys][seg:N][LEVEL][COMPONENT]` prefix, + * because it owns the run clock and segment counter. The adapter only decides + * where the line goes (stdout, DevTools, an on-page pane). + */ + +/** + * @typedef {object} RendererAdapter Decode and display. May be null. + * @property {(opts: object) => Promise} start + * @property {() => Promise} stop + * @property {(segmentId: number, objects: object) => void} stageSegment + * Hand over one segment's per-object asset handles for decode. + * @property {(segmentId: number, objects: object) => void} setPlaybackState + * Per-tick buffer state, so the renderer knows what may be shown. + * @property {(callbacks: object) => void} setCallbacks + * Subscribe to renderer events: `onFrame`, `onObjectReady`, `onClosed`, + * `onLog`. The core calls this once during startup — it cannot pass them at + * construction because the platform is built before the core exists. + * @property {object|null} latestCamera + * Live pose, read every segment tick in interactive mode. + * + * Null in simulated mode: the core credits segments straight from the download + * result instead of waiting for an object_ready event. Both paths must stay, + * because they measure different things. + */ + +/** + * @typedef {object} LifecycleAdapter Process/page control. + * @property {(code: number) => void} exit + * Node: process.exit. Browser: resolve the run promise and stop timers; it + * must NOT navigate away, or the final POST /api/results would be cancelled. + * @property {(fn: () => void) => void} onShutdownRequest + * Node: SIGINT/SIGTERM. Browser: a Stop control or beforeunload. Fires the + * partial-run finalizer so an interrupted run still uploads its metrics. + */ + +/** + * @typedef {object} ClientPlatform + * @property {TransportAdapter} transport + * @property {StorageAdapter} storage + * @property {ViewpointAdapter} viewpoints + * @property {ClockAdapter} clock + * @property {LoggerAdapter} logger + * @property {RendererAdapter|null} renderer + * @property {LifecycleAdapter} lifecycle + */ + +/** + * Freeze the contract all the way down, method arrays included. + * + * A shallow Object.freeze leaves `methods` mutable, so one stray + * `PLATFORM_CONTRACT.transport.methods.push(...)` would silently add a + * requirement that every adapter then fails — and because the contract is a + * module singleton, the corruption outlives the code that caused it. + */ +function deepFreezeContract(contract) { + for (const spec of Object.values(contract)) { + Object.freeze(spec.methods); + Object.freeze(spec.properties); + Object.freeze(spec); + } + return Object.freeze(contract); +} + +/** + * The contract, as data. `validatePlatform` and ClientCore/README.md are both + * driven by this, so an adapter cannot drift from its documentation. + * + * `renderer` is listed but optional — see RendererAdapter. + */ +const PLATFORM_CONTRACT = deepFreezeContract({ + transport: { + methods: ['getJson', 'postJson', 'assetUrl', 'fetchAsset'], + properties: [] + }, + storage: { + methods: ['handle', 'createScratch', 'release', 'writeText', 'writeResult', + 'openAppendStream'], + properties: [] + }, + viewpoints: { methods: ['list'], properties: [] }, + clock: { + methods: ['now', 'every', 'cancel', 'delay'], + properties: [] + }, + logger: { methods: ['emit'], properties: [] }, + renderer: { + methods: ['start', 'stop', 'stageSegment', 'setPlaybackState', 'setCallbacks'], + properties: ['latestCamera'], + optional: true + }, + lifecycle: { methods: ['exit', 'onShutdownRequest'], properties: [] } +}); + +/** + * Fail loudly and completely on an incomplete platform. + * + * Called once at startup rather than letting a missing method surface as + * `undefined is not a function` twelve segments into a run — in a research + * client that produces a half-finished result file that looks legitimate. + * Every problem is reported at once so a new adapter can be finished in one + * pass instead of one error per run. + * + * @param {ClientPlatform} platform + * @param {{requireRenderer?: boolean}} [options] + * requireRenderer: true in interactive mode, where a null renderer would mean + * nothing is ever decoded and every object would be reported missing. + * @returns {ClientPlatform} the same object, for chaining + * @throws {TypeError} listing every missing capability, method and property + */ +function validatePlatform(platform, { requireRenderer = false } = {}) { + if (!platform || typeof platform !== 'object') { + throw new TypeError('ClientPlatform must be an object'); + } + + const problems = []; + + for (const [name, spec] of Object.entries(PLATFORM_CONTRACT)) { + const capability = platform[name]; + const isRenderer = name === 'renderer'; + + if (capability == null) { + // renderer is the one optional capability, and only in simulated + // mode: interactive mode without a renderer would decode nothing + // and report every object missing. + if (spec.optional && !(isRenderer && requireRenderer)) continue; + problems.push(isRenderer + ? 'renderer is required in interactive mode' + : `missing capability: ${name}`); + continue; + } + if (typeof capability !== 'object') { + problems.push(`${name} must be an object, got ${typeof capability}`); + continue; + } + for (const method of spec.methods) { + if (typeof capability[method] !== 'function') { + problems.push(`${name}.${method} must be a function`); + } + } + for (const property of spec.properties) { + if (!(property in capability)) { + problems.push(`${name}.${property} must be present`); + } + } + } + + if (problems.length) { + throw new TypeError( + `Incomplete ClientPlatform:\n - ${problems.join('\n - ')}`); + } + return platform; +} + +/** Capability names, in contract order. */ +function platformCapabilities() { + return Object.keys(PLATFORM_CONTRACT); +} + +module.exports = { PLATFORM_CONTRACT, validatePlatform, platformCapabilities }; diff --git a/open4d/streaming/system/ClientCore/stalls.js b/open4d/streaming/system/ClientCore/stalls.js new file mode 100644 index 00000000..dd4f2807 --- /dev/null +++ b/open4d/streaming/system/ClientCore/stalls.js @@ -0,0 +1,103 @@ +'use strict'; + +/** + * Stall accounting. + * + * Extracted from system/Client/client.js (`getSegmentStallStatus`, + * `processStallEvents`). + * + * Two distinct notions of "not playing" live here and must not be merged: + * + * MISSING unintentional starvation — a real stall, playback halted. + * FROZEN a deliberate policy decision under bandwidth deficit: the object + * keeps showing its last frame while the rest of the scene plays. + * + * Frozen time is tracked separately and penalised less (see STALL_COSTS in + * ClientCore/abr.js), so collapsing the two would misreport the system's + * behaviour as worse than it is. + */ + +/** + * Per-segment stall data. RESETS the manager's counters, so call exactly once + * per segment tick and before anything else that reads them. + * + * The manager returns `{ totalSegmentStallDuration, totalSegmentStallCount, + * stallingCount, perObject: { obj: {stallTime, stallCount, ...} } }`. + * Iterating the top-level object instead of `perObject` used to sum undefined + * into NaN. + */ +function collectSegmentStalls(bufferManager) { + const s = bufferManager.getAndResetSegmentStalls(); + + const perObject = {}; + for (const [objName, stats] of Object.entries(s.perObject || {})) { + perObject[objName] = { + segmentStallDuration: stats.stallTime, + segmentStallCount: stats.stallCount, + isCurrentlyStalling: stats.isCurrentlyStalling, + ongoingStallDuration: stats.currentOngoingStallDuration, + segmentFrozenDuration: stats.frozenTime || 0, + isFrozen: stats.isFrozen || false + }; + } + + return { + totalSegmentStallDuration: s.totalSegmentStallDuration || 0, + totalSegmentStallCount: s.totalSegmentStallCount || 0, + stallingCount: s.stallingCount || 0, + perObject + }; +} + +/** + * Fold playback-tick stall events into the metrics record. Mutates `metrics`. + * + * @param {object} metrics + * @param {Array} events from bufferManager.onPlaybackTick + * @param {number} timestamp ms since run start + * @param {{info: Function, warn: Function, debug: Function}} log + */ +function applyStallEvents(metrics, events, timestamp, log) { + for (const event of events) { + if (!event) continue; + + const objName = event.objectName; + + if (event.type === 'start') { + log.warn('STALL', `Stall start: ${objName}`, { + buffer: event.bufferBefore?.toFixed(3), + reason: event.reason || 'unknown' + }); + + if (!metrics.objectStalls[objName]) metrics.objectStalls[objName] = []; + metrics.objectStalls[objName].push({ + startTime: timestamp, endTime: null, duration: 0 + }); + + } else if (event.type === 'end') { + const duration = event.stallDuration || 0; + log.info('STALL', `Stall end: ${objName}`, { + durationSec: duration.toFixed(3) + }); + + const objStalls = metrics.objectStalls[objName]; + if (objStalls && objStalls.length > 0) { + const last = objStalls[objStalls.length - 1]; + if (last.endTime === null) { + last.endTime = timestamp; + last.duration = duration; + } + } + + metrics.summary.rebuffers++; + metrics.summary.totalStallDuration += duration * 1000; + + } else if (event.type === 'ongoing') { + log.debug('STALL', `Stall ongoing: ${objName}`, { + durationSec: event.stallDuration?.toFixed(1) + }); + } + } +} + +module.exports = { collectSegmentStalls, applyStallEvents }; diff --git a/open4d/streaming/system/ClientCore/stream-config.js b/open4d/streaming/system/ClientCore/stream-config.js new file mode 100644 index 00000000..40b5e37d --- /dev/null +++ b/open4d/streaming/system/ClientCore/stream-config.js @@ -0,0 +1,61 @@ +'use strict'; + +/** + * Stream timing configuration, as published by the server's /api/config. + * + * Extracted from system/Client/client.js `applyStreamConfig`, which mutated + * five module-level globals. Here it is a pure function returning a frozen + * config, so the browser client and the Node client cannot drift on validation + * and a test can exercise the error paths without a server. + */ + +/** + * @param {object} config raw /api/config body + * @param {object} [previous] values to fall back on for absent fields + * @returns {Readonly<{segmentDuration: number, framesPerSegment: number, + * segmentIntervalMs: number, totalSegments: number, + * updateIntervalSegments: number}>} + * @throws {Error} listing every field that is still missing or invalid + */ +function resolveStreamConfig(config, previous = {}) { + const resolved = { + segmentDuration: previous.segmentDuration ?? null, + framesPerSegment: previous.framesPerSegment ?? null, + segmentIntervalMs: previous.segmentIntervalMs ?? null, + totalSegments: previous.totalSegments ?? null, + updateIntervalSegments: previous.updateIntervalSegments ?? null + }; + + const positive = value => Number.isFinite(value) && value > 0; + + if (config) { + if (positive(config.segmentDuration)) resolved.segmentDuration = config.segmentDuration; + if (positive(config.framesPerSegment)) resolved.framesPerSegment = config.framesPerSegment; + if (positive(config.segmentIntervalMs)) { + resolved.segmentIntervalMs = config.segmentIntervalMs; + } else { + // Derived rather than defaulted: a segment interval that disagreed + // with the segment duration would silently desynchronise the segment + // clock from playback. + resolved.segmentIntervalMs = Math.round(resolved.segmentDuration * 1000); + } + if (positive(config.totalSegments)) resolved.totalSegments = config.totalSegments; + if (positive(config.updateIntervalSegments)) { + resolved.updateIntervalSegments = config.updateIntervalSegments; + } + } + + const missing = []; + if (!positive(resolved.segmentDuration)) missing.push('segmentDuration'); + if (!positive(resolved.framesPerSegment)) missing.push('framesPerSegment'); + if (!positive(resolved.segmentIntervalMs)) missing.push('segmentIntervalMs'); + if (!positive(resolved.totalSegments)) missing.push('totalSegments'); + if (!positive(resolved.updateIntervalSegments)) missing.push('updateIntervalSegments'); + if (missing.length) { + throw new Error(`Invalid stream config from server: missing ${missing.join(', ')}`); + } + + return Object.freeze(resolved); +} + +module.exports = { resolveStreamConfig }; diff --git a/open4d/streaming/system/ClientCore/streaming-client.js b/open4d/streaming/system/ClientCore/streaming-client.js new file mode 100644 index 00000000..f0de544e --- /dev/null +++ b/open4d/streaming/system/ClientCore/streaming-client.js @@ -0,0 +1,1086 @@ +'use strict'; + +/** + * The streaming client: segment loop, playback clock, download orchestration, + * server reporting, shutdown. + * + * Moved from system/Client/client.js. Everything platform-shaped goes through + * `platform` (see platform.js), so this same class drives the Node desktop + * client and the browser client. Where the original used module-level `let`, + * this holds instance state — the behaviour is otherwise intended to be + * identical, and tests/test_client_core_parity.js pins the parts most likely + * to drift. + * + * Two modes, both load-bearing: + * interactive a renderer decodes and displays; objects become playable only + * on `object_ready`, and the camera comes from the live window. + * simulated no renderer; segments are credited straight from the download + * result and canned viewpoints are cycled. + * They measure different things. Do not collapse them. + */ + +const { MCKPAdaptiveBitrate } = require('./abr'); +const { BandwidthEstimator } = require('./bandwidth'); +const { LinkThroughputMeter } = require('./link-throughput'); +const { resolveStreamConfig } = require('./stream-config'); +const { validatePlatform } = require('./platform'); +const { + planSegmentDownload, creditTaskResult, summarizeSegmentDownload, + expectedSelectionBytes +} = require('./download-plan'); +const { + createMetricsRecord, recordBitrateSelection, buildSegmentRecord, + buildBufferHistoryRecord, finalizeMetrics +} = require('./metrics'); +const { + buildSegmentPayload, buildDownloadCompletePayload, buildBitrateCountsPayload, + buildRenderFramesPayload, buildViewpointPayload, buildBroadcastStartPayload +} = require('./payloads'); +const { collectSegmentStalls, applyStallEvents } = require('./stalls'); + +/** + * Tunables that were environment variables in the Node client. The adapter + * reads its environment and passes them in; the core never looks at one. + */ +const DEFAULT_CONFIG = Object.freeze({ + clientMode: 'interactive', + // One report per segment: the server-side ladder re-solves every segment + // and uses the freshest report with segmentId <= t — at 10s it flew blind + // for the first 5 segments and reused one stale snapshot for the rest of a + // short run. + bitrateReportIntervalMs: 2000, + playbackTickIntervalMs: 100, + // Bounded so a slow server cannot skew the tick clock. + manifestFetchTimeoutMs: 500, + // Dead-man fallback only; the manifest is refreshed at every segment tick. + manifestPollIntervalMs: 10000, + // Backpressure: never stack more than this many segment downloads. Without + // a cap, a link slower than the selected ladder accumulates concurrent + // downloads that share bandwidth, so every download gets slower each tick + // (queueing death spiral) and the bandwidth estimate collapses. + maxInflightSegments: 2, + // Keep a continuous sliding window of requests in flight across ALL objects + // of the segment, so the link never idles at object boundaries. + downloadConcurrency: 10, + // Fraction of the bandwidth ESTIMATE committed per segment. SHARED WITH THE + // SERVER: vstream/config.py reads the same two values, because the ladder + // service must know what the client can afford to decide whether the menu's + // published floor is within budget. + budgetMultiplier: 1, + budgetMultiplierStruggling: 1, + // How many segments each canned viewpoint is held (simulated mode). + // 0 follows updateIntervalSegments. + viewpointHoldSegments: 0, + renderTraceBatchSize: 30, + manifestDebugName: 'menu.json', + renderTraceName: 'render_frames.jsonl', + decodeEventName: 'decode_events.jsonl', + runLabel: null, + abr: { + delta: 0.1, + switchPenalty: 0.25, + stallLambda: 20.0, + stallGamma: 2.0, + stallBeta: 1.3, + maxBuffer: 15, + minBuffer: 2, + startupBuffer: 2 + } +}); + +class StreamingClient { + /** + * @param {object} args + * @param {import('./platform').ClientPlatform} args.platform + * @param {object} [args.config] overrides for DEFAULT_CONFIG + */ + constructor({ platform, config = {} }) { + this.config = { ...DEFAULT_CONFIG, ...config, abr: { ...DEFAULT_CONFIG.abr, ...(config.abr || {}) } }; + + const mode = String(this.config.clientMode).toLowerCase(); + if (!['interactive', 'simulated'].includes(mode)) { + throw new Error( + `clientMode must be "interactive" or "simulated", got "${mode}"`); + } + this.config.clientMode = mode; + this.interactive = mode === 'interactive'; + + this.platform = validatePlatform(platform, { + requireRenderer: this.interactive + }); + + const { clock } = this.platform; + this.startTime = clock.now(); + + // --- stream timing, from the server --- + this.streamConfig = null; + + // --- run state --- + this.broadcastId = null; + this.viewpoints = []; + this.currentViewpoint = null; + this.currentSegmentId = 0; + this.currentManifest = null; + this.previousManifest = null; + this.finishing = false; + this._finishPromise = null; + this.scratchRoot = null; + this.lastDownloadLate = false; + + this.inFlightDownloads = new Map(); + this.pendingDownloads = new Map(); + this.intervalBitrateCounts = {}; + + this.playbackState = { totalPlaybackTime: 0, lastTickTime: null }; + + // --- timers --- + this.segmentTimer = null; + this.playbackTimer = null; + this.manifestTimer = null; + this.bitrateReportTimer = null; + + // --- renderer plumbing --- + this.latestRendererCamera = null; + this.rendererReadyBySegment = new Map(); + this.renderTraceBatch = []; + this.renderTraceStream = null; + this.decodeEventStream = null; + this.renderUploadPromise = Promise.resolve(); + + // The ABR gets the platform clock and logger too: its startup grace + // period is measured in wall time, and its debug lines belong in the + // run log rather than on stdout. + this.abr = new MCKPAdaptiveBitrate(this.config.abr, { + now: () => clock.now(), + log: message => this._logInfo('ABR', message) + }); + + this.bandwidth = new BandwidthEstimator({ + elapsedMs: () => this._elapsed(), + inFlightEntries: () => this.inFlightDownloads.entries(), + signals: () => ({ + missingCount: this.abr.getBufferManager().getSummary().missingCount, + inFlightCount: this.inFlightDownloads.size, + maxInFlight: this.config.maxInflightSegments, + lastDownloadLate: this.lastDownloadLate + }), + logger: { + debug: (c, m, d) => this._log('DEBUG', c, m, d), + warn: (c, m, d) => this._log('WARN', c, m, d) + }, + now: () => clock.now(), + initialEstimate: 5, + multiplier: this.config.budgetMultiplier, + multiplierStruggling: this.config.budgetMultiplierStruggling + }); + + this.linkMeter = new LinkThroughputMeter({ + // Read through a function: the interval arrives from the server + // after this meter is constructed. + segmentIntervalMs: () => this.streamConfig?.segmentIntervalMs ?? 0, + onSample: (bytes, busyMs) => this.bandwidth.update(bytes, busyMs), + now: () => clock.now() + }); + + this.metrics = createMetricsRecord({ + clientMode: mode, + startTime: this.startTime, + bandwidthHistory: this.bandwidth.history + }); + } + + // ---------------------------------------------------------------- logging + + _elapsed() { return this.platform.clock.now() - this.startTime; } + + /** + * The core formats the prefix because it owns the run clock, the playback + * clock and the segment counter; the adapter only decides where the line + * goes. + */ + _log(level, component, message, data = null) { + const elapsed = (this._elapsed() / 1000).toFixed(2); + const playback = this.playbackState.totalPlaybackTime.toFixed(2); + const prefix = `[wall:+${elapsed}s][play:${playback}s]` + + `[seg:${this.currentSegmentId}][${level}][${component}]`; + this.platform.logger.emit(level, `${prefix} ${message}`, data); + } + + _logInfo(c, m, d = null) { this._log('INFO', c, m, d); } + _logWarn(c, m, d = null) { this._log('WARN', c, m, d); } + _logError(c, m, d = null) { this._log('ERROR', c, m, d); } + _logDebug(c, m, d = null) { this._log('DEBUG', c, m, d); } + + get _log4() { + return { + info: (c, m, d) => this._logInfo(c, m, d), + warn: (c, m, d) => this._logWarn(c, m, d), + error: (c, m, d) => this._logError(c, m, d), + debug: (c, m, d) => this._logDebug(c, m, d) + }; + } + + // ------------------------------------------------------- renderer events + + _enqueueRenderTraceUpload(force = false) { + if (!this.broadcastId || this.renderTraceBatch.length === 0) { + return this.renderUploadPromise; + } + if (!force && this.renderTraceBatch.length < this.config.renderTraceBatchSize) { + return this.renderUploadPromise; + } + const frames = this.renderTraceBatch.splice(0, this.renderTraceBatch.length); + this.renderUploadPromise = this.renderUploadPromise.then(async () => { + try { + const res = await this.platform.transport.postJson('/api/render-frames', + buildRenderFramesPayload({ broadcastId: this.broadcastId, frames })); + if (!res.ok) throw new Error(`HTTP ${res.status}`); + } catch (err) { + this._logError('RENDER', + `Failed to upload ${frames.length} frame records: ${err.message}`); + } + }); + return this.renderUploadPromise; + } + + _onRendererFrame(event) { + this.latestRendererCamera = event.camera || this.latestRendererCamera; + const record = { ...event, clientElapsedMs: this._elapsed() }; + this.metrics.renderSummary.framesPresented++; + this.metrics.renderSummary.framesDropped += + Number(event.droppedFramesBefore || 0); + this.renderTraceStream?.write(JSON.stringify(record) + '\n'); + this.renderTraceBatch.push(record); + this._enqueueRenderTraceUpload(false); + } + + async _onRendererObjectReady(event) { + const key = `${event.segmentId}:${event.objectName}`; + const pending = this.rendererReadyBySegment.get(key); + const record = { + timestamp: this._elapsed(), + segmentId: event.segmentId, + objectName: event.objectName, + repId: event.repId, + status: event.type === 'object_ready' ? 'ready' : 'error', + decodeMs: event.decodeMs ?? null, + cacheHit: event.cacheHit === true, + diskCacheHit: event.diskCacheHit === true, + decodeShared: event.decodeShared === true, + superseded: event.superseded === true, + error: event.message ?? null + }; + this.metrics.decodeEvents.push(record); + // Also streamed out: decodeEvents used to be serialized only at the end + // of a run, so an aborted run lost every decode timing and error — the + // one telemetry needed to diagnose a starved renderer. + this.decodeEventStream?.write(JSON.stringify(record) + '\n'); + + if (event.type === 'object_ready' && pending && !pending.credited) { + pending.credited = true; + this.abr.onObjectDownloaded( + event.objectName, this.streamConfig.segmentDuration); + this._logInfo('DECODE', `${event.objectName} ready`, { + segmentId: event.segmentId, + repId: event.repId, + decodeMs: Number(event.decodeMs || 0).toFixed(1), + cache: event.cacheHit ? 'memory' + : (event.diskCacheHit ? 'disk' : 'decoded') + }); + } else if (event.type === 'object_error' && event.superseded) { + // Not a failure: a newer segment picked a different representation + // before this decode started. No work was lost and the new rep is + // already queued. + this._logDebug('DECODE', `${event.objectName} superseded`, { + segmentId: event.segmentId, repId: event.repId + }); + } else if (event.type === 'object_error') { + this.metrics.renderSummary.objectDecodeFailures++; + this._logError('DECODE', `${event.objectName} failed: ${event.message}`, { + segmentId: event.segmentId, repId: event.repId + }); + } + + if (pending?.objectDir) { + try { + await this.platform.storage.release(pending.objectDir); + } catch (err) { + this._logWarn('CACHE', + `Could not release ${pending.objectDir}: ${err.message}`); + } + } + this.rendererReadyBySegment.delete(key); + } + + _onRendererClosed(event) { + if (this.finishing) return; + this.metrics.renderSummary.earlyWindowClose = + this.currentSegmentId < (this.streamConfig?.totalSegments || Infinity); + this._logInfo('RENDER', 'Window closed', event); + this._stopSegmentTimer(); + this.finish(); + } + + // --------------------------------------------------------------- timers + + _startPlaybackTimer() { + const { clock } = this.platform; + this.playbackState.lastTickTime = clock.now(); + this.playbackState.totalPlaybackTime = 0; + + this._logInfo('PLAYBACK', 'Starting playback timer', { + intervalMs: this.config.playbackTickIntervalMs + }); + + this.playbackTimer = clock.every(this.config.playbackTickIntervalMs, () => { + const now = clock.now(); + const elapsedSec = (now - this.playbackState.lastTickTime) / 1000; + this.playbackState.lastTickTime = now; + this.playbackState.totalPlaybackTime += elapsedSec; + + const timestamp = now - this.startTime; + const stallEvents = this.abr.onPlaybackTick(elapsedSec, timestamp); + + if (stallEvents && stallEvents.length > 0) { + applyStallEvents(this.metrics, stallEvents, timestamp, this._log4); + } + if (this.platform.renderer) { + const objects = Object.fromEntries( + this.abr.getBufferManager().getSummary().objects.map(obj => [ + obj.objectName, + { state: obj.state, bufferLevel: obj.level } + ]) + ); + this.platform.renderer.setPlaybackState( + Math.max(0, this.currentSegmentId - 1), objects); + } + }); + } + + _stopPlaybackTimer() { + if (this.playbackTimer) { + this.platform.clock.cancel(this.playbackTimer); + this.playbackTimer = null; + } + } + + _startSegmentTimer() { + this._logInfo('SEGMENT', 'Starting segment timer', { + intervalMs: this.streamConfig.segmentIntervalMs + }); + this._processSegmentTick(); + this.segmentTimer = this.platform.clock.every( + this.streamConfig.segmentIntervalMs, () => this._processSegmentTick()); + } + + _stopSegmentTimer() { + if (this.segmentTimer) { + this.platform.clock.cancel(this.segmentTimer); + this.segmentTimer = null; + } + } + + // --------------------------------------------------------- segment tick + + async _processSegmentTick() { + if (this.currentSegmentId >= this.streamConfig.totalSegments) { + this._logInfo('SEGMENT', 'All segments complete, stopping...'); + this._stopSegmentTimer(); + this.finish(); + return; + } + + // Refresh the manifest every tick — BEFORE the no-manifest guard, so a + // client that started against an empty server picks up the bootstrap + // ladder on the next tick instead of waiting for the slow fallback poll. + await Promise.race([ + this._fetchManifest(), + this.platform.clock.delay(this.config.manifestFetchTimeoutMs) + ]); + + if (!this.currentManifest) { + this._logWarn('SEGMENT', 'No manifest yet, sending empty segment'); + this._sendSegmentToServer(this.currentSegmentId, null, true); + this.currentSegmentId++; + return; + } + + const segmentId = this.currentSegmentId; + const timestamp = this._elapsed(); + + // Per-segment stalls BEFORE processing: this RESETS the counters. + const segmentStalls = collectSegmentStalls(this.abr.getBufferManager()); + + if (this.interactive) { + if (this.latestRendererCamera) { + this.currentViewpoint = { + filename: 'interactive-camera', + data: this.latestRendererCamera + }; + } + } else { + const hold = this.config.viewpointHoldSegments + || this.streamConfig.updateIntervalSegments; + if (segmentId % hold === 0) { + this.currentViewpoint = this.viewpoints[ + Math.floor(segmentId / hold) % this.viewpoints.length]; + this._logInfo('VIEWPOINT', 'Changed', { + filename: this.currentViewpoint.filename + }); + } + } + + const viewpointPriorities = extractViewpointPriorities( + this.currentViewpoint.data); + const bufferBefore = this.abr.getBufferManager().getSummary(); + const budget = this.bandwidth.budget(); + + const selection = this.abr.selectBestCombination( + this.currentManifest, budget, { viewpointPriorities }); + + this._logInfo('SEGMENT', `>>> Tick ${segmentId}`, { + bufferMin: bufferBefore.minBufferLevel.toFixed(2), + estBW: this.bandwidth.estimate.toFixed(2), + budget: budget.toFixed(2), + bitrate: selection.totalBitrate.toFixed(2), + inFlight: this.inFlightDownloads.size, + segmentStallSec: segmentStalls.totalSegmentStallDuration.toFixed(3), + ...(selection.deficit + ? { deficit: true, frozen: selection.frozenObjects } : {}), + stallStates: Object.fromEntries( + bufferBefore.objects.map(o => [o.objectName, o.state])) + }); + + // Report BEFORE kicking off downloads so ladder generation for this + // segment's viewpoint starts immediately. + this._sendSegmentToServer(segmentId, { + selection, bufferBefore, budget, timestamp, viewpointPriorities, + segmentStalls + }, false); + + if (this.inFlightDownloads.size >= this.config.maxInflightSegments) { + // Link can't keep up with the segment clock: skip this segment's + // download instead of queueing another concurrent transfer. + // Playback stalls honestly on the starved buffers. + this._logWarn('DOWNLOAD', `Skipping segment ${segmentId} download`, { + inFlight: this.inFlightDownloads.size, + maxInFlight: this.config.maxInflightSegments + }); + this.metrics.summary.skippedDownloads = + (this.metrics.summary.skippedDownloads || 0) + 1; + // This path used to bypass the per-object bookkeeping inside + // _startSegmentDownload, so the whole scene lost a segment's credit + // with nothing but a counter to show for it. + for (const objName of Object.keys(selection.combo)) { + this.abr.getBufferManager().getBuffer(objName) + .skipSegment('download-skipped-backlog'); + } + } else { + this._startSegmentDownload( + segmentId, selection, this.currentManifest, viewpointPriorities); + } + + this._recordSegmentMetrics( + segmentId, selection, bufferBefore, budget, timestamp, segmentStalls); + + this.currentSegmentId++; + } + + // ------------------------------------------------------------ downloads + + _startSegmentDownload(segmentId, selection, manifest, viewpointPriorities) { + const { clock } = this.platform; + const downloadStart = clock.now(); + + this.inFlightDownloads.set(segmentId, { + startTime: downloadStart, + bytesExpected: expectedSelectionBytes(selection) + }); + + const downloadPromise = (async () => { + this.linkMeter.started(); + let linkAccounted = false; + let effectiveSelection = selection; + try { + let downloadResult = await this._downloadSegmentFiles( + effectiveSelection, segmentId, manifest); + let usedFallback = false; + + if (downloadResult.totalSize === 0 && this.previousManifest) { + this._logWarn('DOWNLOAD', + `Segment ${segmentId} failed, trying fallback`); + const prevSelection = this.abr.selectBestCombination( + this.previousManifest, this.bandwidth.budget(), + { viewpointPriorities }); + downloadResult = await this._downloadSegmentFiles( + prevSelection, segmentId, this.previousManifest); + if (downloadResult.totalSize > 0) { + effectiveSelection = prevSelection; + usedFallback = true; + this.metrics.summary.fallbackCount++; + } + } + + const downloadTimeMs = clock.now() - downloadStart; + + // bytes were already credited per-file by linkMeter.bytesDelivered + this.linkMeter.finished(); + linkAccounted = true; + + const isLate = downloadTimeMs > this.streamConfig.segmentIntervalMs; + this.lastDownloadLate = isLate; + if (isLate) { + this.metrics.summary.lateDownloads++; + this._logWarn('DOWNLOAD', `Segment ${segmentId} download LATE`, { + downloadTimeMs, + expectedMs: this.streamConfig.segmentIntervalMs + }); + } + + await this._applyDownloadResult( + segmentId, effectiveSelection, downloadResult); + + const bufferAfter = this.abr.getBufferManager().getSummary(); + const measuredBW = downloadTimeMs > 0 + ? (downloadResult.totalSize * 8) / downloadTimeMs / 1000 + : 0; + + // Backfill the transferred size onto the segment record. It is + // not known at tick time (the download has not run yet), and its + // absence made every data-volume figure in the QoE report read 0. + const segmentRecord = this.metrics.segments.find( + s => s.segmentId === segmentId); + if (segmentRecord) { + segmentRecord.size = downloadResult.totalSize; + segmentRecord.downloadTimeMs = downloadTimeMs; + segmentRecord.measuredBandwidth = measuredBW; + } + + this._logInfo('DOWNLOAD', `Segment ${segmentId} complete`, { + sizeMB: (downloadResult.totalSize / 1024 / 1024).toFixed(2), + timeMs: downloadTimeMs, + measuredBW: measuredBW.toFixed(2), + estBW: this.bandwidth.estimate.toFixed(2), + bufferAfter: bufferAfter.minBufferLevel.toFixed(2), + late: isLate, + fallback: usedFallback + }); + + this._sendDownloadCompleteToServer(segmentId, { + downloadTimeMs, + downloadSizeBytes: downloadResult.totalSize, + measuredBandwidthMbps: measuredBW, + bufferAfter, + isLate, + usedFallback + }); + + return { success: true, downloadTimeMs, downloadResult }; + + } catch (err) { + // partial bytes are already credited; just release the busy counter + if (!linkAccounted) this.linkMeter.finished(); + this._logError('DOWNLOAD', + `Segment ${segmentId} failed: ${err.message}`); + return { success: false, error: err.message }; + } finally { + this.inFlightDownloads.delete(segmentId); + this.pendingDownloads.delete(segmentId); + } + })(); + + this.pendingDownloads.set(segmentId, downloadPromise); + } + + /** Per-object bookkeeping and renderer staging once a segment has landed. */ + async _applyDownloadResult(segmentId, selection, downloadResult) { + const downloadedObjects = {}; + const renderObjects = {}; + + for (const [objName, rep] of Object.entries(selection.combo)) { + const objResult = downloadResult.objects.find( + o => o.objectName === objName); + // Interactive mode used to demand all ~61 files, so one 404 or + // timeout discarded a whole object-segment; the renderer now holds + // the previous frame for any gap instead. + const success = Boolean(objResult && objResult.success && objResult.size > 0); + if (success && objResult.geometryDownloaded < objResult.geometryTotal) { + this._logWarn('DOWNLOAD', `${objName} partial geometry`, { + segmentId, + got: objResult.geometryDownloaded, + want: objResult.geometryTotal + }); + } + downloadedObjects[objName] = { downloaded: success, repId: rep.id }; + recordBitrateSelection( + this.metrics, this.intervalBitrateCounts, objName, rep.id); + + if (this.interactive && success) { + const decodeDir = this.platform.storage.handle( + objResult.objectDir, 'decoded'); + this.rendererReadyBySegment.set(`${segmentId}:${objName}`, { + repId: rep.id, + objectDir: objResult.objectDir, + decodeDir, + credited: false + }); + renderObjects[objName] = { + state: 'download', + repId: rep.id, + geometryFiles: objResult.geometryFiles, + textureFiles: objResult.textureFiles, + textureFile: objResult.textureFile, + decodeDir + }; + } else if (this.interactive && objResult?.objectDir) { + try { + await this.platform.storage.release(objResult.objectDir); + } catch (err) { + this._logWarn('CACHE', + `Could not release failed download: ${err.message}`); + } + } + } + + // Skipped/frozen objects: record the skip so per-object segment + // accounting stays complete (the freeze itself was applied at selection + // time). + for (const s of (selection.skippedObjects || [])) { + if (!downloadedObjects[s.objectName]) { + downloadedObjects[s.objectName] = { + downloaded: false, skipped: true, reason: s.reason + }; + } + } + + if (this.interactive) { + if (this.platform.renderer && Object.keys(renderObjects).length > 0) { + this.platform.renderer.stageSegment(segmentId, renderObjects); + } + // Selected objects become playable only on object_ready. + for (const [objName, info] of Object.entries(downloadedObjects)) { + if (!info.downloaded) { + // Carry the real reason: a deliberate freeze or a + // high-buffer skip is not a download failure, and labelling + // all three the same hid which was which. + this.abr.getBufferManager().getBuffer(objName) + .skipSegment(info.reason || 'download-failed'); + } + } + } else { + this.abr.onSegmentDownloaded( + downloadedObjects, this.streamConfig.segmentDuration); + } + } + + async _downloadSegmentFiles(selection, segmentIdx, manifest) { + const { clock, storage, transport } = this.platform; + const downloadStart = clock.now(); + + const { tasks, perObj } = planSegmentDownload({ + selection, + manifest, + framesPerSegment: this.streamConfig.framesPerSegment, + interactive: this.interactive, + objectDirFor: (objName, repId) => storage.handle( + this.scratchRoot, + `segment_${String(segmentIdx).padStart(4, '0')}`, + objName, + repId.replace(/[^a-zA-Z0-9._-]+/g, '-') + ), + destinationFor: (objName, objectDir, file) => (file.kind === 'texture' + ? storage.handle(objectDir, + `texture_${String(file.textureIndex).padStart(4, '0')}.mp4`) + : storage.handle(objectDir, + `geometry_${String(file.frameIndex).padStart(4, '0')}.drc`)), + pathToUrl: assetPath => transport.assetUrl(assetPath) + }); + + // Drain the list with a sliding window: a new request starts the moment + // one finishes, so the link never idles at batch or object boundaries. + let next = 0; + const worker = async () => { + while (next < tasks.length) { + const task = tasks[next++]; + const r = await transport.fetchAsset( + task.url, this.interactive ? task.destination : null); + // Credit the link estimator as this file lands, so bytes and + // busy time cover the same window even when segment downloads + // overlap. + this.linkMeter.bytesDelivered(r.size); + creditTaskResult(perObj.get(task.objName), task, r, this.interactive); + } + }; + await Promise.all(Array.from( + { length: Math.min(this.config.downloadConcurrency, tasks.length) }, + worker)); + + const { totalSize, objects } = summarizeSegmentDownload(perObj); + return { totalSize, totalTimeMs: clock.now() - downloadStart, objects }; + } + + // -------------------------------------------------------------- reports + + async _sendSegmentToServer(segmentId, data, isEmpty) { + try { + const updateInterval = this.streamConfig.updateIntervalSegments; + const payload = buildSegmentPayload({ + broadcastId: this.broadcastId, + segmentId, + timestamp: this._elapsed(), + playbackTime: this.playbackState.totalPlaybackTime, + estimatedBandwidth: this.bandwidth.estimate, + isEmpty, + data, + viewpoint: (segmentId % updateInterval === 0) + ? this.currentViewpoint.data : null, + bufferLevels: this.abr.getBufferManager().getBufferLevels() + }); + + const res = await this.platform.transport.postJson( + `/api/segment/${segmentId}`, payload); + + // Manifest staleness telemetry: how many segments behind the newest + // server-side ladder this client is. + const latest = res?.body?.latestManifestSegId; + if (typeof latest === 'number') { + const manifestSeg = this.currentManifest?.segment?.t ?? -1; + this._logDebug('MANIFEST', 'Staleness', { + segId: segmentId, + latestManifestSegId: latest, + usingManifestSeg: manifestSeg, + behind: latest - manifestSeg + }); + } + + this._logDebug('API', `Segment ${segmentId} sent to server`, { isEmpty }); + + } catch (err) { + this._logError('API', + `Failed to send segment ${segmentId}: ${err.message}`); + } + } + + async _sendDownloadCompleteToServer(segmentId, data) { + try { + await this.platform.transport.postJson( + `/api/segment/${segmentId}/download-complete`, + buildDownloadCompletePayload({ + broadcastId: this.broadcastId, + segmentId, + timestamp: this._elapsed(), + estimatedBandwidth: this.bandwidth.estimate, + ...data + })); + } catch (err) { + this._logDebug('API', + `Failed to send download complete for ${segmentId}`); + } + } + + async _sendBitrateCountsToServer() { + if (Object.keys(this.intervalBitrateCounts).length === 0) return; + try { + await this.platform.transport.postJson('/api/bitrate-counts-interval', + buildBitrateCountsPayload({ + segmentId: this.currentSegmentId, + timestamp: this._elapsed(), + intervalMs: this.config.bitrateReportIntervalMs, + countsPerObject: this.intervalBitrateCounts + })); + this.intervalBitrateCounts = {}; + } catch (err) { + this._logError('BITRATE', `Failed to send: ${err.message}`); + } + } + + _recordSegmentMetrics( + segmentId, selection, bufferBefore, budget, timestamp, segmentStalls) { + const bufferManager = this.abr.getBufferManager(); + this.metrics.segments.push(buildSegmentRecord({ + segmentId, selection, bufferBefore, budget, timestamp, segmentStalls, + summary: bufferManager.getSummary(), + playbackTime: this.playbackState.totalPlaybackTime, + estimatedBandwidth: this.bandwidth.estimate, + inFlightCount: this.inFlightDownloads.size, + prevQualityByObj: this.abr.prevQualityByObj + })); + this.metrics.bufferHistory.push(buildBufferHistoryRecord({ + timestamp, + playbackTime: this.playbackState.totalPlaybackTime, + bufferBefore, + segmentStalls + })); + } + + // ------------------------------------------------------------- manifest + + async _fetchStreamConfig() { + const res = await this.platform.transport.getJson('/api/config'); + if (!res.ok) { + throw new Error(`Config fetch failed: HTTP ${res.status}`); + } + this.streamConfig = resolveStreamConfig(res.body, this.streamConfig || {}); + this._logInfo('CONFIG', 'Loaded from server', { + totalSegments: this.streamConfig.totalSegments, + segmentDurationS: this.streamConfig.segmentDuration, + framesPerSegment: this.streamConfig.framesPerSegment, + segmentIntervalMs: this.streamConfig.segmentIntervalMs, + updateIntervalSegments: this.streamConfig.updateIntervalSegments + }); + } + + async _fetchManifest() { + try { + const res = await this.platform.transport.getJson('/api/manifest'); + if (!res.ok) return; + const newManifest = res.body; + const newSeg = newManifest?.segment?.t; + const curSeg = this.currentManifest?.segment?.t; + // Same ladder version: keep previousManifest meaningfully older (it + // backs the download fallback path) and skip the write. + if (this.currentManifest && newSeg === curSeg) return; + if (this.currentManifest) this.previousManifest = this.currentManifest; + this.currentManifest = newManifest; + await this.platform.storage.writeText( + this.config.manifestDebugName, + JSON.stringify(this.currentManifest, null, 2)); + this._logInfo('MANIFEST', 'Updated', { + manifestSeg: newSeg, + objects: Object.keys(this.currentManifest.objects || {}).length, + nFrames: this.streamConfig.framesPerSegment, + segmentDurationS: this.streamConfig.segmentDuration + }); + } catch (err) { + this._logError('MANIFEST', `Fetch failed: ${err.message}`); + } + } + + // ------------------------------------------------------------ lifecycle + + /** Start streaming. Resolves once all timers are armed. */ + async run() { + const { platform } = this; + this._logInfo('STREAM', + `=== VOLUMETRIC STREAMING (${this.config.clientMode.toUpperCase()}) ===`); + + platform.lifecycle.onShutdownRequest(() => { + this._logInfo('STREAM', 'Shutdown requested, finalizing partial run'); + this.finish().catch(err => { + this._logError('STREAM', `Shutdown failed: ${err.message}`); + platform.lifecycle.exit(1); + }); + }); + + try { + await this._fetchStreamConfig(); + + this.viewpoints = await platform.viewpoints.list(); + this.currentViewpoint = this.viewpoints[0]; + if (this.interactive) { + this.viewpoints = [this.currentViewpoint]; + this._logInfo('VIEWPOINT', 'Using startup pose', { + filename: this.currentViewpoint.filename, + behavior: 'live camera; canned viewpoints will not be cycled' + }); + } else { + this._logInfo('VIEWPOINT', + `Loaded ${this.viewpoints.length} simulation viewpoints`); + } + + this.scratchRoot = await platform.storage.createScratch(); + + if (this.interactive) { + this.renderTraceStream = await platform.storage.openAppendStream( + this.config.renderTraceName); + this.decodeEventStream = await platform.storage.openAppendStream( + this.config.decodeEventName); + + platform.renderer.setCallbacks({ + onFrame: event => this._onRendererFrame(event), + onObjectReady: event => this._onRendererObjectReady(event), + onClosed: event => this._onRendererClosed(event), + onLog: (level, message, data) => this._log( + String(level || 'INFO').toUpperCase(), 'RENDER', message, data) + }); + await platform.renderer.start({ + fps: Math.round(this.streamConfig.framesPerSegment + / this.streamConfig.segmentDuration), + frameCount: this.streamConfig.framesPerSegment, + initialCamera: this.currentViewpoint.data + }); + this.latestRendererCamera = + platform.renderer.latestCamera || this.currentViewpoint.data; + this.currentViewpoint = { + filename: 'interactive-camera', + data: this.latestRendererCamera + }; + this._logInfo('RENDER', 'Interactive window ready', { + scratchRoot: this.scratchRoot + }); + } + + const started = await platform.transport.postJson('/api/broadcast/start', + buildBroadcastStartPayload({ + clientMode: this.config.clientMode, + label: this.config.runLabel, + sceneObjects: this.config.sceneObjects + })); + this.broadcastId = started?.body?.broadcastId ?? null; + this.metrics.broadcastId = this.broadcastId; + this._logInfo('STREAM', 'Broadcast started', { + broadcastId: this.broadcastId + }); + + await platform.transport.postJson('/api/viewpoint', + buildViewpointPayload({ + broadcastId: this.broadcastId, + viewpoint: this.currentViewpoint.data, + segId: 0 + })); + + await this._fetchManifest(); + + this.manifestTimer = platform.clock.every( + this.config.manifestPollIntervalMs, () => this._fetchManifest()); + this.bitrateReportTimer = platform.clock.every( + this.config.bitrateReportIntervalMs, + () => this._sendBitrateCountsToServer()); + + this._startPlaybackTimer(); + this._startSegmentTimer(); + + this._logInfo('STREAM', 'All timers started - streaming in real-time'); + + } catch (err) { + this._logError('STREAM', `Fatal error: ${err.message}`, + { stack: err.stack }); + this._stopPlaybackTimer(); + this._stopSegmentTimer(); + if (platform.renderer) await platform.renderer.stop(); + platform.lifecycle.exit(1); + } + } + + /** + * Finalize: drain, upload, write metrics, release scratch, exit. + * + * Idempotent AND awaitable: repeated calls return the same promise, so a + * caller that arrives during shutdown waits for the real completion instead + * of racing past it. The original returned undefined on the second call, + * which meant nothing could reliably await the upload of a partial run. + */ + finish() { + if (this._finishPromise) return this._finishPromise; + this.finishing = true; + this._finishPromise = this._finish(); + return this._finishPromise; + } + + async _finish() { + const { platform } = this; + this._logInfo('STREAM', '=== FINISHING STREAM ==='); + this._stopSegmentTimer(); + this._stopPlaybackTimer(); + + if (this.pendingDownloads.size > 0 + && !this.metrics.renderSummary.earlyWindowClose) { + this._logInfo('STREAM', + `Waiting for ${this.pendingDownloads.size} pending downloads...`); + await Promise.all(this.pendingDownloads.values()); + } else if (this.pendingDownloads.size > 0) { + this._logInfo('STREAM', + `Discarding ${this.pendingDownloads.size} in-flight downloads ` + + 'after window close'); + } + + if (this.manifestTimer) platform.clock.cancel(this.manifestTimer); + if (this.bitrateReportTimer) platform.clock.cancel(this.bitrateReportTimer); + this.manifestTimer = null; + this.bitrateReportTimer = null; + + await this._sendBitrateCountsToServer(); + await this._enqueueRenderTraceUpload(true); + await this.renderUploadPromise; + if (platform.renderer) await platform.renderer.stop(); + + // Detach first, so a late frame can never write to a closing stream. + const closing = [this.renderTraceStream, this.decodeEventStream] + .filter(Boolean); + this.renderTraceStream = null; + this.decodeEventStream = null; + for (const stream of closing) await stream.close(); + + const bufferManager = this.abr.getBufferManager(); + const { finalStallMetrics, frozenMetrics } = finalizeMetrics({ + metrics: this.metrics, + bufferManager, + totalSegments: this.currentSegmentId, + totalPlaybackTime: this.playbackState.totalPlaybackTime, + totalWallTime: this._elapsed() / 1000, + estimatedBandwidth: this.bandwidth.estimate, + bandwidthSamples: this.bandwidth.samples, + renderTraceFile: this.interactive ? this.config.renderTraceName : null, + decodeEventFile: this.interactive ? this.config.decodeEventName : null + }); + + await platform.storage.writeResult(JSON.stringify(this.metrics, null, 2)); + + try { + await platform.transport.postJson('/api/results', this.metrics); + } catch (err) { + this._logError('RESULTS', `Failed to send: ${err.message}`); + } + + if (this.scratchRoot) { + try { + await platform.storage.release(this.scratchRoot); + } catch (err) { + this._logWarn('CACHE', + `Could not remove ${this.scratchRoot}: ${err.message}`); + } + } + + this._logInfo('STREAM', '=== STREAM COMPLETE ===', { + totalSegments: this.currentSegmentId, + playbackTimeSec: this.playbackState.totalPlaybackTime.toFixed(2), + wallTimeSec: (this._elapsed() / 1000).toFixed(2), + totalStallsSec: (this.metrics.summary.totalStallDuration / 1000).toFixed(2), + stallRatio: (this.playbackState.totalPlaybackTime > 0 + ? (finalStallMetrics.totalStallTime + / this.playbackState.totalPlaybackTime) * 100 + : 0).toFixed(2) + '%', + totalFrozenSec: frozenMetrics.sumFrozenTime.toFixed(2), + frozenSegments: frozenMetrics.frozenSegments, + skippedDownloads: this.metrics.summary.skippedDownloads || 0, + lateDownloads: this.metrics.summary.lateDownloads, + fallbacks: this.metrics.summary.fallbackCount, + finalBWEstimate: this.bandwidth.estimate.toFixed(2) + }); + + platform.lifecycle.exit(0); + } +} + +/** + * Per-object viewpoint priorities as the ABR expects them. + * + * Defaults matter: an object the viewpoint feed says nothing about is treated + * as visible at middling priority rather than dropped, so a partial feed + * degrades the weighting instead of silently culling the scene. + */ +function extractViewpointPriorities(viewpointData) { + const priorities = {}; + if (viewpointData?.objects) { + for (const [objName, objData] of Object.entries(viewpointData.objects)) { + priorities[objName] = { + inFOV: objData.inFOV ?? true, + priority: objData.priority ?? 3, + distance: objData.distance ?? 5 + }; + } + } + return priorities; +} + +module.exports = { StreamingClient, extractViewpointPriorities, DEFAULT_CONFIG }; diff --git a/open4d/streaming/system/ClientCore/testing/fake-platform.js b/open4d/streaming/system/ClientCore/testing/fake-platform.js new file mode 100644 index 00000000..0dd9de37 --- /dev/null +++ b/open4d/streaming/system/ClientCore/testing/fake-platform.js @@ -0,0 +1,362 @@ +'use strict'; + +/** + * In-memory ClientPlatform for tests, and the reference for what a real adapter + * has to do. + * + * Two jobs: + * 1. Let the client core be tested with no server, no disk, no GPU and no + * real time — `clock.advance(ms)` drives segment ticks deterministically, + * so a 40-segment run is a millisecond of test time instead of 80 seconds. + * 2. Pin the contract by example. When writing the browser adapter, read this + * alongside ClientCore/platform.js: anything this fake does, that adapter + * has to do for real. + * + * Not a mock framework: every capability is a working implementation over + * ordinary JS structures, so a test asserts on observable state + * (`storage.files`, `transport.requests`, `renderer.staged`) rather than on + * call expectations. + */ + +const { validatePlatform } = require('../platform'); + +// -------------------------------------------------------------------------- +// Clock: virtual time. +// -------------------------------------------------------------------------- +class FakeClock { + constructor(start = 1_700_000_000_000) { + this.t = start; + this._seq = 0; + this._timers = new Map(); + } + + now() { return this.t; } + + every(ms, fn) { + if (!(ms > 0)) throw new RangeError('every() needs a positive interval'); + const id = ++this._seq; + this._timers.set(id, { kind: 'interval', ms, next: this.t + ms, fn }); + return id; + } + + delay(ms) { + return new Promise(resolve => { + const id = ++this._seq; + this._timers.set(id, { kind: 'timeout', next: this.t + ms, fn: resolve }); + }); + } + + cancel(handle) { this._timers.delete(handle); } + + /** Timers still armed — a leak check for finishStream(). */ + get pending() { return this._timers.size; } + + /** + * Advance virtual time, firing every timer that comes due in order and + * letting each one's microtasks drain before the next fires. Async because + * the core's timer callbacks are async (a segment tick awaits a manifest). + */ + async advance(ms) { + const target = this.t + ms; + // Bounded so a pathological interval cannot hang the suite. + for (let guard = 0; guard < 100_000; guard++) { + let earliest = null; + let earliestId = null; + for (const [id, timer] of this._timers) { + if (timer.next <= target && (earliest === null || timer.next < earliest.next)) { + earliest = timer; + earliestId = id; + } + } + if (earliest === null) break; + + this.t = earliest.next; + if (earliest.kind === 'interval') { + earliest.next = this.t + earliest.ms; + } else { + this._timers.delete(earliestId); + } + await earliest.fn(); + await Promise.resolve(); // drain microtasks queued by the callback + } + this.t = target; + } +} + +// -------------------------------------------------------------------------- +// Transport: routed, recording, no sockets. +// -------------------------------------------------------------------------- +class FakeTransport { + constructor({ baseUrl = 'http://fake-server:3000' } = {}) { + this.baseUrl = baseUrl; + this.requests = []; // { method, path, body } + this.assetRequests = []; // { url, handle } + this._routes = []; // { method, pattern, handler } + /** + * Per-URL asset behaviour. Keys are matched as substrings so a test can + * fail "every geometry file of dancer" with one entry. Default: succeed. + */ + this.assetRules = []; // { match, success, size, timeMs } + this.defaultAsset = { success: true, size: 50_000, timeMs: 5 }; + } + + on(method, pattern, handler) { + this._routes.push({ method, pattern, handler }); + return this; + } + + onGet(pattern, handler) { return this.on('GET', pattern, handler); } + onPost(pattern, handler) { return this.on('POST', pattern, handler); } + + /** Fail or slow specific assets: failAsset('dancer_fr0003') */ + failAsset(match) { + this.assetRules.push({ match, success: false, size: 0, timeMs: 1 }); + return this; + } + + setAsset(match, result) { + this.assetRules.push({ match, ...result }); + return this; + } + + _resolve(method, path) { + for (const route of this._routes) { + if (route.method !== method) continue; + if (route.pattern === path) return route.handler; + } + // Prefix fallback, so '/api/segment/' catches '/api/segment/17'. + for (const route of this._routes) { + if (route.method !== method) continue; + if (typeof route.pattern === 'string' && path.startsWith(route.pattern)) { + return route.handler; + } + } + return null; + } + + async _call(method, path, body) { + this.requests.push({ method, path, body }); + const handler = this._resolve(method, path); + if (!handler) return { ok: false, status: 404, body: null }; + const result = await handler(body, path); + if (result && typeof result === 'object' && 'ok' in result) return result; + return { ok: true, status: 200, body: result ?? null }; + } + + getJson(path) { return this._call('GET', path, null); } + postJson(path, body) { return this._call('POST', path, body); } + + assetUrl(assetPath) { + if (!assetPath) return null; + if (/^https?:\/\//.test(assetPath)) return assetPath; + if (assetPath.startsWith('/files/')) return `${this.baseUrl}${assetPath}`; + const match = assetPath.match(/files\/(.+)/); + return match ? `${this.baseUrl}/files/${match[1]}` : null; + } + + async fetchAsset(url, handle) { + this.assetRequests.push({ url, handle }); + const rule = this.assetRules.find(r => url.includes(r.match)); + const { success, size, timeMs } = rule || this.defaultAsset; + return success + ? { success: true, size, timeMs } + : { success: false, size: 0, timeMs, error: 'fake asset failure' }; + } + + /** Paths requested, for order assertions. */ + pathsFor(method) { + return this.requests.filter(r => r.method === method).map(r => r.path); + } + + bodyFor(method, path) { + const hit = [...this.requests].reverse().find( + r => r.method === method && r.path === path); + return hit ? hit.body : null; + } +} + +// -------------------------------------------------------------------------- +// Storage: a Map with handle composition. +// -------------------------------------------------------------------------- +class FakeStorage { + constructor() { + this.files = new Map(); // name -> text + this.streams = new Map(); // name -> string[] + this.result = null; // writeResult payload + this.scratch = null; + this.released = []; + this._scratchCount = 0; + } + + handle(...parts) { return parts.filter(p => p != null).join('/'); } + + async createScratch() { + this.scratch = `/fake-scratch/run-${++this._scratchCount}`; + return this.scratch; + } + + async release(handle) { + // Tolerating an absent handle is part of the contract: an object whose + // download failed may never have had a directory created. + this.released.push(handle); + } + + async writeText(name, text) { this.files.set(name, text); } + + async writeResult(text) { this.result = text; } + + async openAppendStream(name) { + const lines = []; + this.streams.set(name, lines); + let closed = false; + return { + write: line => { + if (closed) throw new Error(`write after close on ${name}`); + lines.push(line); + }, + close: async () => { closed = true; } + }; + } + + /** Parsed JSONL for a telemetry stream. */ + linesOf(name) { + return (this.streams.get(name) || []).map(l => JSON.parse(l)); + } +} + +// -------------------------------------------------------------------------- +// Renderer: records what it was told, emits events on demand. +// -------------------------------------------------------------------------- +class FakeRenderer { + constructor({ initialCamera = { position: [0, 1.6, 3] } } = {}) { + this.latestCamera = initialCamera; + this.started = null; + this.stopped = false; + this.staged = []; // { segmentId, objects } + this.playbackStates = []; // { segmentId, objects } + this.callbacks = {}; + } + + /** The adapter takes event callbacks at construction; mirror that. */ + setCallbacks(callbacks) { this.callbacks = callbacks || {}; } + + async start(opts) { this.started = opts; } + async stop() { this.stopped = true; } + stageSegment(segmentId, objects) { this.staged.push({ segmentId, objects }); } + setPlaybackState(segmentId, objects) { + this.playbackStates.push({ segmentId, objects }); + } + + /** Drive the decode-completion path the core credits buffers from. */ + emitObjectReady(segmentId, objectName, repId, extra = {}) { + this.callbacks.onObjectReady?.({ + type: 'object_ready', segmentId, objectName, repId, decodeMs: 12, ...extra + }); + } + + emitObjectError(segmentId, objectName, repId, message = 'fake decode failure', + extra = {}) { + this.callbacks.onObjectReady?.({ + type: 'object_error', segmentId, objectName, repId, message, ...extra + }); + } + + emitFrame(frame = {}) { + this.callbacks.onFrame?.({ camera: this.latestCamera, ...frame }); + } + + emitClosed(event = {}) { this.callbacks.onClosed?.(event); } + + moveCamera(camera) { this.latestCamera = camera; } +} + +// -------------------------------------------------------------------------- +// Lifecycle +// -------------------------------------------------------------------------- +class FakeLifecycle { + constructor() { + this.exitCode = null; + this.exitCalls = 0; + this._shutdownHandlers = []; + } + + exit(code) { this.exitCode = code; this.exitCalls++; } + + onShutdownRequest(fn) { this._shutdownHandlers.push(fn); } + + /** Simulate SIGINT / the browser Stop control. */ + requestShutdown() { + for (const fn of this._shutdownHandlers) fn(); + } +} + +// -------------------------------------------------------------------------- +// Logger +// -------------------------------------------------------------------------- +class FakeLogger { + constructor({ echo = false } = {}) { + this.lines = []; // { level, line, data } + this.echo = echo; + } + + emit(level, line, data) { + this.lines.push({ level, line, data }); + if (this.echo) console.log(line, data ? JSON.stringify(data) : ''); + } + + /** Lines at a level, for asserting that a warning actually fired. */ + at(level) { return this.lines.filter(l => l.level === level); } + + /** True when any line contains `text`. */ + saw(text) { return this.lines.some(l => l.line.includes(text)); } +} + +// -------------------------------------------------------------------------- +// Assembly +// -------------------------------------------------------------------------- + +/** + * A complete, contract-valid platform for tests. + * + * @param {object} [options] + * @param {boolean} [options.withRenderer=true] false models simulated mode + * @param {Array<{filename: string, data: object}>} [options.viewpoints] + * @param {boolean} [options.echoLogs=false] print lines while debugging a test + * @returns {import('../platform').ClientPlatform & { + * clock: FakeClock, transport: FakeTransport, storage: FakeStorage, + * renderer: FakeRenderer|null, lifecycle: FakeLifecycle, logger: FakeLogger }} + */ +function createFakePlatform({ + withRenderer = true, + viewpoints = [{ filename: 'view_00.json', data: { objects: {} } }], + echoLogs = false +} = {}) { + const renderer = withRenderer ? new FakeRenderer() : null; + const platform = { + transport: new FakeTransport(), + storage: new FakeStorage(), + viewpoints: { + list: async () => { + if (!viewpoints || viewpoints.length === 0) { + throw new Error('no viewpoints available'); + } + return viewpoints; + } + }, + clock: new FakeClock(), + logger: new FakeLogger({ echo: echoLogs }), + renderer, + lifecycle: new FakeLifecycle() + }; + return validatePlatform(platform, { requireRenderer: withRenderer }); +} + +module.exports = { + createFakePlatform, + FakeClock, + FakeTransport, + FakeStorage, + FakeRenderer, + FakeLifecycle, + FakeLogger +}; diff --git a/open4d/streaming/system/Server/manifest.mpd.json b/open4d/streaming/system/Server/manifest.mpd.json new file mode 100644 index 00000000..37bfb7a2 --- /dev/null +++ b/open4d/streaming/system/Server/manifest.mpd.json @@ -0,0 +1,321 @@ +{ + "type": "object-dash-like", + "version": 1, + "segment": { + "t": 0, + "fps": 30, + "n_frames": 30, + "duration_s": 1.0 + }, + "objects": { + "dancer": { + "start_number": 1, + "representations": [ + { + "id": "r_res240_crf36_qp8", + "predicted": { + "bitrate_mbps": 13.82957935333252, + "quality": 24.361909866333008 + }, + "knobs": { + "resolution": 240, + "crf": 36, + "qp": 8 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res240_crf36_qp8", + "geometry_drc_pattern": "dancer_fr%04d_qp8.drc", + "texture_mp4": "files/t_00/dancer/r_res240_crf36_qp8/dancer_w240_crf36.mp4" + } + }, + { + "id": "r_res720_crf27_qp12", + "predicted": { + "bitrate_mbps": 22.377296447753906, + "quality": 36.574283599853516 + }, + "knobs": { + "resolution": 720, + "crf": 27, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res720_crf27_qp12", + "geometry_drc_pattern": "dancer_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/dancer/r_res720_crf27_qp12/dancer_w720_crf27.mp4" + } + }, + { + "id": "r_res720_crf25_qp12", + "predicted": { + "bitrate_mbps": 23.186378479003906, + "quality": 37.07225799560547 + }, + "knobs": { + "resolution": 720, + "crf": 25, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res720_crf25_qp12", + "geometry_drc_pattern": "dancer_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/dancer/r_res720_crf25_qp12/dancer_w720_crf25.mp4" + } + }, + { + "id": "r_res960_crf27_qp12", + "predicted": { + "bitrate_mbps": 24.421104431152344, + "quality": 37.67633056640625 + }, + "knobs": { + "resolution": 960, + "crf": 27, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res960_crf27_qp12", + "geometry_drc_pattern": "dancer_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/dancer/r_res960_crf27_qp12/dancer_w960_crf27.mp4" + } + }, + { + "id": "r_res720_crf22_qp12", + "predicted": { + "bitrate_mbps": 24.6586971282959, + "quality": 37.849430084228516 + }, + "knobs": { + "resolution": 720, + "crf": 22, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res720_crf22_qp12", + "geometry_drc_pattern": "dancer_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/dancer/r_res720_crf22_qp12/dancer_w720_crf22.mp4" + } + }, + { + "id": "r_res720_crf22_qp15", + "predicted": { + "bitrate_mbps": 26.6396484375, + "quality": 39.31940841674805 + }, + "knobs": { + "resolution": 720, + "crf": 22, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/dancer/r_res720_crf22_qp15", + "geometry_drc_pattern": "dancer_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/dancer/r_res720_crf22_qp15/dancer_w720_crf22.mp4" + } + } + ] + }, + "basketball_player": { + "start_number": 1, + "representations": [ + { + "id": "r_res240_crf36_qp8", + "predicted": { + "bitrate_mbps": 13.68151569366455, + "quality": 25.350805282592773 + }, + "knobs": { + "resolution": 240, + "crf": 36, + "qp": 8 + }, + "paths": { + "base_dir": "files/t_00/basketball_player/r_res240_crf36_qp8", + "geometry_drc_pattern": "basketball_player_fr%04d_qp8.drc", + "texture_mp4": "files/t_00/basketball_player/r_res240_crf36_qp8/basketball_player_w240_crf36.mp4" + } + } + ] + }, + "mitch": { + "start_number": 1, + "representations": [ + { + "id": "r_res240_crf36_qp8", + "predicted": { + "bitrate_mbps": 10.475520133972168, + "quality": 25.584638595581055 + }, + "knobs": { + "resolution": 240, + "crf": 36, + "qp": 8 + }, + "paths": { + "base_dir": "files/t_00/mitch/r_res240_crf36_qp8", + "geometry_drc_pattern": "mitch_fr%04d_qp8.drc", + "texture_mp4": "files/t_00/mitch/r_res240_crf36_qp8/mitch_w240_crf36.mp4" + } + }, + { + "id": "r_res360_crf27_qp12", + "predicted": { + "bitrate_mbps": 15.285351753234863, + "quality": 35.30508041381836 + }, + "knobs": { + "resolution": 360, + "crf": 27, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/mitch/r_res360_crf27_qp12", + "geometry_drc_pattern": "mitch_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/mitch/r_res360_crf27_qp12/mitch_w360_crf27.mp4" + } + } + ] + }, + "thomas": { + "start_number": 618, + "representations": [ + { + "id": "r_res240_crf36_qp8", + "predicted": { + "bitrate_mbps": 10.435447692871094, + "quality": 24.907878875732422 + }, + "knobs": { + "resolution": 240, + "crf": 36, + "qp": 8 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res240_crf36_qp8", + "geometry_drc_pattern": "thomas_fr%04d_qp8.drc", + "texture_mp4": "files/t_00/thomas/r_res240_crf36_qp8/thomas_w240_crf36.mp4" + } + }, + { + "id": "r_res720_crf22_qp12", + "predicted": { + "bitrate_mbps": 15.779021263122559, + "quality": 38.89402389526367 + }, + "knobs": { + "resolution": 720, + "crf": 22, + "qp": 12 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res720_crf22_qp12", + "geometry_drc_pattern": "thomas_fr%04d_qp12.drc", + "texture_mp4": "files/t_00/thomas/r_res720_crf22_qp12/thomas_w720_crf22.mp4" + } + }, + { + "id": "r_res960_crf27_qp14", + "predicted": { + "bitrate_mbps": 17.117748260498047, + "quality": 41.28874206542969 + }, + "knobs": { + "resolution": 960, + "crf": 27, + "qp": 14 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res960_crf27_qp14", + "geometry_drc_pattern": "thomas_fr%04d_qp14.drc", + "texture_mp4": "files/t_00/thomas/r_res960_crf27_qp14/thomas_w960_crf27.mp4" + } + }, + { + "id": "r_res960_crf27_qp15", + "predicted": { + "bitrate_mbps": 17.29946517944336, + "quality": 41.737125396728516 + }, + "knobs": { + "resolution": 960, + "crf": 27, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res960_crf27_qp15", + "geometry_drc_pattern": "thomas_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/thomas/r_res960_crf27_qp15/thomas_w960_crf27.mp4" + } + }, + { + "id": "r_res960_crf25_qp15", + "predicted": { + "bitrate_mbps": 17.720029830932617, + "quality": 42.24899673461914 + }, + "knobs": { + "resolution": 960, + "crf": 25, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res960_crf25_qp15", + "geometry_drc_pattern": "thomas_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/thomas/r_res960_crf25_qp15/thomas_w960_crf25.mp4" + } + }, + { + "id": "r_res720_crf20_qp15", + "predicted": { + "bitrate_mbps": 17.91791343688965, + "quality": 42.4348258972168 + }, + "knobs": { + "resolution": 720, + "crf": 20, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res720_crf20_qp15", + "geometry_drc_pattern": "thomas_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/thomas/r_res720_crf20_qp15/thomas_w720_crf20.mp4" + } + }, + { + "id": "r_res960_crf22_qp15", + "predicted": { + "bitrate_mbps": 18.28986358642578, + "quality": 43.088539123535156 + }, + "knobs": { + "resolution": 960, + "crf": 22, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res960_crf22_qp15", + "geometry_drc_pattern": "thomas_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/thomas/r_res960_crf22_qp15/thomas_w960_crf22.mp4" + } + }, + { + "id": "r_res960_crf20_qp15", + "predicted": { + "bitrate_mbps": 18.864965438842773, + "quality": 43.39103698730469 + }, + "knobs": { + "resolution": 960, + "crf": 20, + "qp": 15 + }, + "paths": { + "base_dir": "files/t_00/thomas/r_res960_crf20_qp15", + "geometry_drc_pattern": "thomas_fr%04d_qp15.drc", + "texture_mp4": "files/t_00/thomas/r_res960_crf20_qp15/thomas_w960_crf20.mp4" + } + } + ] + } + } +} diff --git a/open4d/streaming/system/Server/package-lock.json b/open4d/streaming/system/Server/package-lock.json new file mode 100644 index 00000000..417eb438 --- /dev/null +++ b/open4d/streaming/system/Server/package-lock.json @@ -0,0 +1,782 @@ +{ + "name": "Server", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "dependencies": { + "cors": "^2.8.6", + "express": "^5.2.1" + } + }, + "node_modules/accepts": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", + "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", + "dependencies": { + "mime-types": "^3.0.0", + "negotiator": "^1.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/body-parser": { + "version": "2.2.2", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", + "integrity": "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA==", + "dependencies": { + "bytes": "^3.1.2", + "content-type": "^1.0.5", + "debug": "^4.4.3", + "http-errors": "^2.0.0", + "iconv-lite": "^0.7.0", + "on-finished": "^2.4.1", + "qs": "^6.14.1", + "raw-body": "^3.0.1", + "type-is": "^2.0.1" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/content-disposition": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.0.1.tgz", + "integrity": "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q==", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", + "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", + "engines": { + "node": ">=6.6.0" + } + }, + "node_modules/cors": { + "version": "2.8.6", + "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", + "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", + "dependencies": { + "object-assign": "^4", + "vary": "^1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", + "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==" + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/express": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", + "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", + "dependencies": { + "accepts": "^2.0.0", + "body-parser": "^2.2.1", + "content-disposition": "^1.0.0", + "content-type": "^1.0.5", + "cookie": "^0.7.1", + "cookie-signature": "^1.2.1", + "debug": "^4.4.0", + "depd": "^2.0.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "finalhandler": "^2.1.0", + "fresh": "^2.0.0", + "http-errors": "^2.0.0", + "merge-descriptors": "^2.0.0", + "mime-types": "^3.0.0", + "on-finished": "^2.4.1", + "once": "^1.4.0", + "parseurl": "^1.3.3", + "proxy-addr": "^2.0.7", + "qs": "^6.14.0", + "range-parser": "^1.2.1", + "router": "^2.2.0", + "send": "^1.1.0", + "serve-static": "^2.2.0", + "statuses": "^2.0.1", + "type-is": "^2.0.1", + "vary": "^1.1.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/finalhandler": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", + "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", + "dependencies": { + "debug": "^4.4.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "on-finished": "^2.4.1", + "parseurl": "^1.3.3", + "statuses": "^2.0.1" + }, + "engines": { + "node": ">= 18.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", + "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", + "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.2.tgz", + "integrity": "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==" + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/is-promise": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", + "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==" + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/media-typer": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.0.tgz", + "integrity": "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/merge-descriptors": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", + "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==" + }, + "node_modules/negotiator": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", + "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-to-regexp": { + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.3.0.tgz", + "integrity": "sha512-7jdwVIRtsP8MYpdXSwOS0YdD0Du+qOoF/AEPIt88PcCFrZCzx41oxku1jD88hZBwbNUIEfpqvuhjFaMAqMTWnA==", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", + "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/qs": { + "version": "6.14.1", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.1.tgz", + "integrity": "sha512-4EK3+xJl8Ts67nLYNwqw/dsFVnCf+qR7RgXSK9jEEm9unao3njwMDdmsdvoKBKHzxd7tCYz5e5M+SnMjdtXGQQ==", + "dependencies": { + "side-channel": "^1.1.0" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", + "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.7.0", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/router": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", + "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", + "dependencies": { + "debug": "^4.4.0", + "depd": "^2.0.0", + "is-promise": "^4.0.0", + "parseurl": "^1.3.3", + "path-to-regexp": "^8.0.0" + }, + "engines": { + "node": ">= 18" + } + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==" + }, + "node_modules/send": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", + "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", + "dependencies": { + "debug": "^4.4.3", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "fresh": "^2.0.0", + "http-errors": "^2.0.1", + "mime-types": "^3.0.2", + "ms": "^2.1.3", + "on-finished": "^2.4.1", + "range-parser": "^1.2.1", + "statuses": "^2.0.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/serve-static": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", + "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", + "dependencies": { + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "parseurl": "^1.3.3", + "send": "^1.2.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==" + }, + "node_modules/side-channel": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", + "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.3", + "side-channel-list": "^1.0.0", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz", + "integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/type-is": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", + "integrity": "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==", + "dependencies": { + "content-type": "^1.0.5", + "media-typer": "^1.1.0", + "mime-types": "^3.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==" + } + } +} diff --git a/open4d/streaming/system/Server/package.json b/open4d/streaming/system/Server/package.json new file mode 100644 index 00000000..be9accea --- /dev/null +++ b/open4d/streaming/system/Server/package.json @@ -0,0 +1,6 @@ +{ + "dependencies": { + "cors": "^2.8.6", + "express": "^5.2.1" + } +} diff --git a/open4d/streaming/system/Server/public/index.html b/open4d/streaming/system/Server/public/index.html new file mode 100644 index 00000000..e69de29b diff --git a/open4d/streaming/system/Server/quest-launch.json b/open4d/streaming/system/Server/quest-launch.json new file mode 100644 index 00000000..cede99b4 --- /dev/null +++ b/open4d/streaming/system/Server/quest-launch.json @@ -0,0 +1,31 @@ +{ + "pipeline": "mesh", + "baselinePort": 12345, + "baselineDatasetManifest": "/media/frozzzen/DataDrive/ORBIT_datasets_rgbd/level_1/manifest.json", + "baselineTileCatalog": "/media/frozzzen/DataDrive/ORBIT_vivo_tiles/catalog.json", + "baselineConnectionTimeoutSeconds": 10.0, + "baselineAbrBandwidthMbps": 80.0, + "fullBandwidthMode": false, + "ablationVariant": "", + "geometryTextureAdaptationEnabled": true, + "globalAllocationEnabled": true, + "objectSchedulingEnabled": true, + "fastSwitchingEnabled": true, + "frameBufferEnabled": true, + "viewportPredictionEnabled": false, + "viewportHistoryWindowSec": 0.5, + "viewportPredictionWindowSec": 0.5, + "viewportSampleRateHz": 36.0, + "viewportTraceEnabled": true, + "orbitViewCullingEnabled": false, + "orbitCullMaxDistanceMetres": 50.0, + "orbitCullFovMarginDegrees": 30.0, + "pointSplatSize": 0.006, + "maxBaselineReceiveBufferBytes": 268435456, + "scene": "", + "sceneObjects": [], + "skybox": "", + "studyTrial": "", + "studyMethod": "", + "studyReady": true +} diff --git a/open4d/streaming/system/Server/quest-trace-player.js b/open4d/streaming/system/Server/quest-trace-player.js new file mode 100644 index 00000000..abdb4c47 --- /dev/null +++ b/open4d/streaming/system/Server/quest-trace-player.js @@ -0,0 +1,370 @@ +#!/usr/bin/env node +// Shape this server's egress NIC according to a CSV bandwidth trace. A root +// token-bucket is deliberately used instead of a destination u32 filter: the +// latter can silently miss traffic on a multiqueue NIC, as the previous Quest +// runs demonstrated. This covers ORBIT HTTP and every baseline TCP media path. +const { execFile, execFileSync } = require('child_process'); +const fs = require('fs'); +const http = require('http'); +let activeCleanup = async () => {}; +const TBF_BURST_FLOOR_BYTES = 128 * 1024; + +function usage(exitCode = 1) { + console.error(`Usage: + sudo node quest-trace-player.js --quest-ip [options] + +Options: + --interface Server egress NIC (auto-detected by default) + --wait-for-broadcast Start the trace clock when Quest presses A + --node-url Node URL used by --wait-for-broadcast + (default: http://127.0.0.1:3000) + --scale Multiply every CSV bandwidth (default: 1) + --hold Keep the final limit until Ctrl-C + --validate Validate the CSV without installing tc rules + --help + +CSV format: time,bandwidth, where time is seconds and bandwidth is Mbps. + +The old environment variables QUEST_IP, SERVER_INTERFACE, VS4D_TRACE_SCALE, +VS4D_TRACE_HOLD, and VS4D_TRACE_WAIT_FOR_BROADCAST remain supported.`); + process.exit(exitCode); +} + +function parseArgs(argv) { + const options = { + questIp: process.env.QUEST_IP || '', + interfaceName: process.env.SERVER_INTERFACE || '', + scale: Number(process.env.VS4D_TRACE_SCALE || 1), + hold: process.env.VS4D_TRACE_HOLD === '1', + validate: false, + waitForBroadcast: process.env.VS4D_TRACE_WAIT_FOR_BROADCAST === '1', + nodeUrl: process.env.VS4D_NODE_URL || 'http://127.0.0.1:3000', + traceFile: '', + }; + for (let index = 0; index < argv.length; index++) { + const value = argv[index]; + const requiredValue = name => { + if (index + 1 >= argv.length) throw new Error(`${name} requires a value`); + return argv[++index]; + }; + if (value === '--quest-ip') options.questIp = requiredValue(value); + else if (value === '--interface') options.interfaceName = requiredValue(value); + else if (value === '--scale') options.scale = Number(requiredValue(value)); + else if (value === '--node-url') options.nodeUrl = requiredValue(value); + else if (value === '--wait-for-broadcast') options.waitForBroadcast = true; + else if (value === '--hold') options.hold = true; + else if (value === '--validate') options.validate = true; + else if (value === '--help' || value === '-h') usage(0); + else if (value.startsWith('-')) throw new Error(`unknown option ${value}`); + else if (options.traceFile) throw new Error('only one trace CSV may be supplied'); + else options.traceFile = value; + } + return options; +} + +function validateIpv4(value) { + const octets = value.split('.'); + return octets.length === 4 && octets.every(part => /^\d+$/.test(part) + && Number(part) >= 0 && Number(part) <= 255); +} + +function loadTrace(filename, scale) { + const points = []; + const ignored = []; + const lines = fs.readFileSync(filename, 'utf8').split(/\r?\n/); + for (let index = 0; index < lines.length; index++) { + const line = lines[index].trim(); + if (!line) continue; + const values = line.split(',').map(value => Number(value.trim())); + if (values.length < 2 || !Number.isFinite(values[0]) + || !Number.isFinite(values[1])) { + // The normal header and a trailing duplicate header in an old lte.csv + // are harmless, but report other malformed rows instead of silently + // changing an experiment. + if (line.toLowerCase() !== 'time,bandwidth') ignored.push(index + 1); + continue; + } + const point = { time: values[0], bandwidth: values[1] * scale }; + if (point.time < 0 || point.bandwidth <= 0) + throw new Error(`invalid trace value on line ${index + 1}: ${line}`); + if (points.length && point.time < points[points.length - 1].time) + throw new Error(`trace time moves backwards on line ${index + 1}`); + points.push(point); + } + if (ignored.length) + console.warn(`[quest-trace] ignored malformed CSV line(s): ${ignored.join(', ')}`); + if (!points.length) throw new Error(`trace contains no bandwidth points: ${filename}`); + // The first sample is the rate at t=0, even when a source trace labels it + // with a non-zero absolute timestamp. + const origin = points[0].time; + return points.map(point => ({ ...point, time: point.time - origin })); +} + +function command(program, args) { + return new Promise((resolve, reject) => execFile(program, args, + (error, stdout, stderr) => error + ? reject(new Error(`${program} ${args.join(' ')}: ${(stderr || error.message).trim()}`)) + : resolve(stdout))); +} + +function tbfArgs(mbps) { + const kbit = `${Math.max(1, Math.round(mbps * 1000))}kbit`; + // The bucket must fit the largest skb presented to the qdisc, not merely an + // Ethernet MTU. TSO/GSO is enabled on the experiment NIC and commonly hands + // Linux a ~64 KiB TCP skb. With the former 16-33 KiB bucket, one such skb at + // the head could never accumulate enough tokens to leave: overlimits kept + // rising while egress stayed at exactly zero for the rest of the run. + // 128 KiB safely admits a normal GSO skb while remaining far below the old + // multi-megabyte/10 ms bursts that distorted short ViVo transfers. + const burstBytes = Math.max(TBF_BURST_FLOOR_BYTES, Math.ceil(mbps * 125)); + return ['handle', '1:', 'tbf', 'rate', kbit, 'burst', `${burstBytes}b`, + 'latency', '250ms']; +} + +function tcByteCount(value, suffix = '') { + const scale = suffix.toLowerCase() === 'k' ? 1024 + : suffix.toLowerCase() === 'm' ? 1024 * 1024 + : suffix.toLowerCase() === 'g' ? 1024 * 1024 * 1024 : 1; + return Number(value) * scale; +} + +async function readTbfStats(interfaceName) { + const output = await command('tc', ['-s', 'qdisc', 'show', 'dev', interfaceName]); + const block = output.split(/\n(?=qdisc )/) + .find(value => /^qdisc tbf 1: root\b/m.test(value)); + if (!block) + throw new Error(`root TBF is not active on ${interfaceName}; tc output: ${output.trim()}`); + const sent = block.match(/\bSent\s+(\d+)\s+bytes\s+(\d+)\s+pkt\b/); + if (!sent) + throw new Error(`cannot read root TBF counters on ${interfaceName}: ${block.trim()}`); + const limits = block.match(/\bdropped\s+(\d+),\s+overlimits\s+(\d+)\b/); + const backlog = block.match(/\bbacklog\s+(\d+(?:\.\d+)?)([KMG]?)b\s+(\d+)p\b/i); + const requeues = block.match(/\brequeues\s+(\d+)\b/); + return { + bytes: Number(sent[1]), + packets: Number(sent[2]), + dropped: limits ? Number(limits[1]) : 0, + overlimits: limits ? Number(limits[2]) : 0, + backlogBytes: backlog ? tcByteCount(backlog[1], backlog[2]) : 0, + backlogPackets: backlog ? Number(backlog[3]) : 0, + requeues: requeues ? Number(requeues[1]) : 0, + }; +} + +function tbfIsWedged(previous, current) { + return current.bytes === previous.bytes + && current.overlimits > previous.overlimits; +} + +function sleep(milliseconds) { + return new Promise(resolve => setTimeout(resolve, milliseconds)); +} + +function getHealth(nodeUrl) { + const url = new URL('/api/health', nodeUrl); + return new Promise((resolve, reject) => { + const request = http.get(url, { timeout: 1500 }, response => { + let body = ''; + response.setEncoding('utf8'); + response.on('data', chunk => { body += chunk; }); + response.on('end', () => { + if (response.statusCode < 200 || response.statusCode >= 300) + return reject(new Error(`Node health returned HTTP ${response.statusCode}`)); + try { resolve(JSON.parse(body)); } catch (error) { reject(error); } + }); + }); + request.on('timeout', () => request.destroy(new Error('Node health timed out'))); + request.on('error', reject); + }); +} + +function publishGroundTruthBandwidth(nodeUrl, broadcastId, elapsedSeconds, + bandwidthMbps) { + const url = new URL('/api/ground-truth-bandwidth', nodeUrl); + const body = JSON.stringify({ broadcastId, elapsedSeconds, bandwidthMbps }); + return new Promise((resolve, reject) => { + const request = http.request(url, { + method: 'POST', timeout: 1500, + headers: { + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(body), + }, + }, response => { + let responseBody = ''; + response.setEncoding('utf8'); + response.on('data', chunk => { responseBody += chunk; }); + response.on('end', () => { + if (response.statusCode < 200 || response.statusCode >= 300) + return reject(new Error( + `Node ground-truth endpoint returned HTTP ${response.statusCode}: ${responseBody}`)); + resolve(); + }); + }); + request.on('timeout', () => request.destroy( + new Error('Node ground-truth bandwidth publish timed out'))); + request.on('error', reject); + request.end(body); + }); +} + +async function waitForNextBroadcast(nodeUrl) { + let health; + try { health = await getHealth(nodeUrl); } + catch (error) { + throw new Error(`cannot use --wait-for-broadcast: ${error.message}`); + } + const previous = health.currentBroadcastId || null; + console.log(`[quest-trace] armed; waiting for Quest A on ${nodeUrl} ` + + `(current broadcast: ${previous || 'none'})`); + for (;;) { + await sleep(100); + health = await getHealth(nodeUrl); + const current = health.currentBroadcastId || null; + if (current && current !== previous) { + console.log(`[quest-trace] broadcast ${current} started; trace clock is running`); + return current; + } + } +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + if (!options.traceFile || !fs.existsSync(options.traceFile)) usage(); + if (!Number.isFinite(options.scale) || options.scale <= 0) + throw new Error('--scale must be positive'); + const points = loadTrace(options.traceFile, options.scale); + if (options.validate) { + const rates = points.map(point => point.bandwidth); + console.log(`[quest-trace] valid: ${points.length} points, ` + + `${points[points.length - 1].time.toFixed(1)} s, ` + + `${Math.min(...rates).toFixed(2)}-${Math.max(...rates).toFixed(2)} Mbps`); + return; + } + if (!validateIpv4(options.questIp)) + throw new Error('--quest-ip must be a valid IPv4 address'); + if (process.getuid && process.getuid() !== 0) + throw new Error('tc requires root; run this command through sudo'); + const interfaceName = options.interfaceName || (() => { + const route = execFileSync('ip', ['route', 'get', options.questIp], { encoding: 'utf8' }); + const match = route.match(/\bdev\s+(\S+)/); + if (!match) throw new Error('cannot determine the routed interface for the Quest'); + return match[1]; + })(); + if (!/^[A-Za-z0-9_.:-]+$/.test(interfaceName)) + throw new Error(`invalid interface name ${JSON.stringify(interfaceName)}`); + + let ownsQdisc = false; + const cleanup = async () => { + if (!ownsQdisc) return; + ownsQdisc = false; + try { await command('tc', ['qdisc', 'del', 'dev', interfaceName, 'root']); } + catch (_) { /* Cleanup is best effort, including after interface loss. */ } + }; + activeCleanup = cleanup; + // This script owns the root qdisc for the duration of one experiment. + try { await command('tc', ['qdisc', 'del', 'dev', interfaceName, 'root']); } + catch (_) { /* No replaceable root qdisc is a normal starting state. */ } + await command('tc', ['qdisc', 'add', 'dev', interfaceName, 'root', + ...tbfArgs(points[0].bandwidth)]); + ownsQdisc = true; + let previousStats = await readTbfStats(interfaceName); + let previousStatsAt = process.hrtime.bigint(); + + const stop = async exitCode => { await cleanup(); process.exit(exitCode); }; + process.on('SIGINT', () => { void stop(0); }); + process.on('SIGTERM', () => { void stop(0); }); + process.on('uncaughtException', error => { + console.error(`[quest-trace] ${error.stack || error.message}`); + void stop(1); + }); + + console.log(`[quest-trace] ${options.traceFile} -> Quest on ${interfaceName}`); + console.log('[quest-trace] root TBF verified; shaping ALL egress on this interface ' + + '(ORBIT HTTP and baseline TCP)'); + console.log('[quest-trace] do not run unrelated bulk transfers on this interface during the experiment'); + const broadcastId = options.waitForBroadcast + ? await waitForNextBroadcast(options.nodeUrl) : null; + + previousStats = await readTbfStats(interfaceName); + previousStatsAt = process.hrtime.bigint(); + const started = previousStatsAt; + let counterBaseBytes = 0; + if (broadcastId) + await publishGroundTruthBandwidth( + options.nodeUrl, broadcastId, 0, points[0].bandwidth); + console.log(`[quest-trace][0.0s] target ${points[0].bandwidth.toFixed(2)} Mbps; ` + + `burst ${(Math.max(TBF_BURST_FLOOR_BYTES, + Math.ceil(points[0].bandwidth * 125)) / 1024).toFixed(1)} KiB`); + for (let index = 1; index < points.length; index++) { + const dueNanoseconds = BigInt(Math.round(points[index].time * 1e9)); + for (;;) { + const remaining = dueNanoseconds - (process.hrtime.bigint() - started); + if (remaining <= 0) break; + await sleep(Math.min(100, Math.max(1, Number(remaining / 1000000n)))); + } + const now = process.hrtime.bigint(); + const stats = await readTbfStats(interfaceName); + const seconds = Number(now - previousStatsAt) / 1e9; + const byteDelta = stats.bytes >= previousStats.bytes + ? stats.bytes - previousStats.bytes : 0; + const overlimitDelta = stats.overlimits >= previousStats.overlimits + ? stats.overlimits - previousStats.overlimits : 0; + const droppedDelta = stats.dropped >= previousStats.dropped + ? stats.dropped - previousStats.dropped : 0; + const measuredMbps = seconds > 0 ? byteDelta * 8 / seconds / 1e6 : 0; + const wedged = tbfIsWedged(previousStats, stats); + if (wedged) { + // A qdisc whose counter is stationary while overlimits rises has an skb + // it cannot dequeue. Replacing it drops that queued head packet; TCP + // retransmits it, which is preferable to losing the entire remaining + // experiment. Preserve the old byte counter for cumulative diagnostics. + counterBaseBytes += stats.bytes; + console.warn(`[quest-trace][${points[index].time.toFixed(1)}s] ` + + `TBF stalled with ${stats.backlogBytes} queued bytes and ` + + `${overlimitDelta} new overlimits; rebuilding qdisc`); + await command('tc', ['qdisc', 'replace', 'dev', interfaceName, 'root', + ...tbfArgs(points[index].bandwidth)]); + } else { + await command('tc', ['qdisc', 'change', 'dev', interfaceName, 'root', + ...tbfArgs(points[index].bandwidth)]); + } + if (broadcastId) + await publishGroundTruthBandwidth( + options.nodeUrl, broadcastId, + Number(process.hrtime.bigint() - started) / 1e9, + points[index].bandwidth); + previousStats = await readTbfStats(interfaceName); + previousStatsAt = process.hrtime.bigint(); + const elapsed = Number(process.hrtime.bigint() - started) / 1e9; + console.log(`[quest-trace][${elapsed.toFixed(1)}s] ` + + `target ${points[index].bandwidth.toFixed(2)} Mbps; prior cap ` + + `${points[index - 1].bandwidth.toFixed(2)}, egress ${measuredMbps.toFixed(2)} Mbps, ` + + `overlimits +${overlimitDelta}, dropped +${droppedDelta}, ` + + `backlog ${(stats.backlogBytes / 1024).toFixed(1)} KiB; ` + + `shaped ${((counterBaseBytes + (wedged ? 0 : stats.bytes)) / 1e6).toFixed(1)} MB`); + } + + if (options.hold) { + console.log('[quest-trace] trace complete; final rate remains active (Ctrl-C clears it)'); + await new Promise(() => {}); + } else { + console.log('[quest-trace] trace complete; removing traffic shaping'); + await cleanup(); + } +} + +if (require.main === module) { + main().catch(async error => { + console.error(`[quest-trace] ${error.stack || error.message}`); + await activeCleanup(); + process.exit(1); + }); +} + +module.exports = { + TBF_BURST_FLOOR_BYTES, + loadTrace, + tbfArgs, + tcByteCount, + tbfIsWedged, +}; diff --git a/open4d/streaming/system/Server/server.js b/open4d/streaming/system/Server/server.js new file mode 100644 index 00000000..980b9e11 --- /dev/null +++ b/open4d/streaming/system/Server/server.js @@ -0,0 +1,2539 @@ +// server.js +const express = require("express"); +const cors = require("cors"); +const fs = require("fs"); +const net = require("net"); +const path = require("path"); +const { spawn, execFile } = require("child_process"); +const readline = require("readline"); // add here + +// -------------------- Logging -------------------- +// One line per event: [YYYY-MM-DD HH:MM:SS.mmm][LEVEL][COMPONENT] message +function logTimestamp() { + return new Date().toISOString().replace("T", " ").replace("Z", ""); +} +function log(level, component, message) { + const line = `[${logTimestamp()}][${level}][${component}] ${message}`; + if (level === "ERROR") console.error(line); + else console.log(line); +} +const logInfo = (component, message) => log("INFO", component, message); +const logWarn = (component, message) => log("WARN", component, message); +const logError = (component, message) => log("ERROR", component, message); + +const app = express(); +app.use(cors({ origin: "*" })); +app.use(express.json({ limit: "200mb" })); + +// -------------------- Paths -------------------- +// The server lives at /system/Server; everything is derived from the +// repo root and can be overridden with the same VS4D_* env vars the Python +// side (vstream/config.py) uses. +const REPO_ROOT = path.resolve(__dirname, "..", ".."); +const FILES_ROOT = process.env.VS4D_FILES_ROOT || path.join(REPO_ROOT, "files"); + +// The packaged media cache under FILES_ROOT/media is keyed by representation +// id -- resolution, CRF and geometry QP -- and NOT by codec. So an HEVC and an +// H.264 corpus produce byte-different files at identical paths, and pointing +// VS4D_COMPRESSED_ROOT at a different codec while reusing a populated cache +// serves the OLD codec's bytes under the new corpus's manifest. Nothing errors: +// the ladder's predicted bitrates come from the new models while the client +// downloads the old encodes, so every rate the run reports is wrong by the +// difference between two codecs, and a browser that can decode one but not the +// other appears to succeed or fail at random. +// +// So the cache records which corpus filled it and refuses to be reused by +// another. Use a separate VS4D_FILES_ROOT per corpus. +const COMPRESSED_ROOT = process.env.VS4D_COMPRESSED_ROOT || ""; +function enforceCorpusStamp() { + const stampPath = path.join(FILES_ROOT, "media", ".corpus"); + const mediaDir = path.join(FILES_ROOT, "media"); + const corpus = COMPRESSED_ROOT + ? path.resolve(COMPRESSED_ROOT) : "(config default)"; + let existing = null; + try { existing = fs.readFileSync(stampPath, "utf8").trim(); } catch (_) {} + + const populated = (() => { + try { return fs.readdirSync(mediaDir).some(n => !n.startsWith(".")); } + catch (_) { return false; } + })(); + const advice = "Point VS4D_FILES_ROOT at a directory of its own for this " + + `corpus, or delete ${mediaDir}.`; + + if (populated && existing === null) { + // Unknown provenance is as dangerous as a known mismatch, and must not be + // resolved by assuming it matches: that is how the stamp would certify a + // cache it never checked. + throw new Error( + `${mediaDir} already holds packaged media but carries no .corpus stamp, ` + + `so which corpus encoded it cannot be determined. ${advice}`); + } + if (populated && existing !== corpus) { + throw new Error( + `${mediaDir} was packaged from ${existing} but VS4D_COMPRESSED_ROOT is ` + + `now ${corpus}. Representation ids carry no codec, so reusing this ` + + `cache would serve the old corpus's encodes under the new corpus's ` + + `manifest. ${advice}`); + } + try { + fs.mkdirSync(mediaDir, { recursive: true }); + fs.writeFileSync(stampPath, corpus + "\n"); + } catch (error) { + logInfo("FILES", `could not stamp the media cache: ${error.message}`); + } +} + +// Reproducibility without endpoint or identity leakage. This captures all +// experiment knobs (including ladder QP/texture caps) while excluding values +// that can contain an IP, path, credential, or machine-specific endpoint. +const PRIVATE_CONFIG_KEY = /(host|url|token|secret|password|key|root|path|dir|(^|_)ip($|_)|ipaddress)/i; +const isPrivateConfigKey = key => PRIVATE_CONFIG_KEY.test(key) || /Ip$/.test(key); +function studySafeEnvironment() { + return Object.fromEntries(Object.entries(process.env) + .filter(([key]) => key.startsWith("VS4D_") && !isPrivateConfigKey(key)) + .sort(([left], [right]) => left.localeCompare(right))); +} +function privacySafeObject(value) { + if (Array.isArray(value)) return value.map(privacySafeObject); + if (!value || typeof value !== "object") return value; + return Object.fromEntries(Object.entries(value) + .filter(([key]) => !isPrivateConfigKey(key) && !/controller/i.test(key)) + .map(([key, item]) => [key, privacySafeObject(item)])); +} +const redactIpAddresses = value => String(value ?? "") + .replace(/(? res.set("Cache-Control", "no-store") +})); +// Vega's exported VGS splat frames for the browser viewer. A separate root +// from /files because these are a baseline's offline-evaluation assets, not +// ladder media; produce them with orbitvega.export_quest. +// NeVo's pre-rendered ReRF/NeVo/reference frames for the browser viewer. +// Produced by orbitnevo/render_frames.py; NeVo cannot be rendered client-side, +// so the page plays these images. +const NEVO_ASSETS_ROOT = process.env.VS4D_NEVO_WEB_ROOT + || path.join(process.env.HOME || REPO_ROOT, "nevo_output"); +app.use("/nevo-assets", express.static(NEVO_ASSETS_ROOT, { + setHeaders: res => res.set("Cache-Control", "no-store") +})); + +// The prepared ViVo tile corpus, shared by the ViVo and NAVA servers. Only +// read here to report whether those baselines *could* run; the server process +// is started by hand. +const VIVO_TILES_ROOT = process.env.VS4D_VIVO_TILES_ROOT + || "/media/frozzzen/DataDrive/ORBIT_vivo_tiles"; + +const VEGA_ASSETS_ROOT = process.env.VS4D_VEGA_WEB_ROOT + || path.join(REPO_ROOT, "results/vega-web"); +app.use("/vega-assets", express.static(VEGA_ASSETS_ROOT, { + setHeaders: res => res.set("Cache-Control", "no-store") +})); + +app.get("/web", (req, res) => { + if (!fs.existsSync(path.join(WEB_CLIENT_DIST, "index.html"))) { + return res.status(503).type("text/plain").send( + "Browser client is not built.\n\n" + + " cd system/WebClient && npm install && node build.js\n"); + } + res.sendFile(path.join(WEB_CLIENT_DIST, "index.html")); +}); + +// A bulk, allocation-bounded response gives the Quest a stable end-to-end +// HTTP capacity sample. Short media files otherwise measure request/connection +// overhead more than link capacity. +const BANDWIDTH_PROBE_CHUNK = Buffer.alloc(1024 * 1024, 0x5a); +app.get("/api/bandwidth-probe", async (req, res) => { + const requested = Number.parseInt(req.query.bytes, 10); + const bytes = Math.max(1, Math.min(64 * 1000 * 1000, + Number.isFinite(requested) ? requested : 32 * 1000 * 1000)); + res.set({ + "Content-Type": "application/octet-stream", + "Content-Length": String(bytes), + "Cache-Control": "no-store, no-transform", + }); + let remaining = bytes; + while (remaining > 0 && !res.destroyed) + { + const chunk = remaining >= BANDWIDTH_PROBE_CHUNK.length + ? BANDWIDTH_PROBE_CHUNK : BANDWIDTH_PROBE_CHUNK.subarray(0, remaining); + remaining -= chunk.length; + if (!res.write(chunk)) await new Promise(resolve => res.once("drain", resolve)); + } + if (!res.destroyed) res.end(); +}); + +// Stream one object's Draco frames in a length-prefixed container. This keeps +// the exact compressed media bytes but avoids sixty small HTTP transactions. +// The headset's decoded-geometry cache decides whether a segment costs a +// 60-frame Draco decode or a disk read, and which storage root it resolved to +// is invisible from here otherwise. Reporting it means a run's cache state is +// on the server console instead of only in a volatile adb ring buffer. +app.post("/api/geometry-cache", (req, res) => { + const b = req.body || {}; + const used = Number.isFinite(b.cachedBytes) && Number.isFinite(b.maxBytes) + ? ` used=${(b.cachedBytes / 1e9).toFixed(2)}/${(b.maxBytes / 1e9).toFixed(1)}GB` + : ""; + const root = b.root ? ` root=${b.root}` : ""; + const entries = Number.isFinite(b.entryCount) ? ` entries=${b.entryCount}` : ""; + if (b.event === "warmup-progress" || b.event === "warmup-finished") { + const finished = b.event === "warmup-finished"; + logInfo("CACHE", `${finished ? "warm-up finished" : "warm-up"}` + + ` ${b.done ?? 0}/${b.total ?? 0}: cached=${b.cached ?? 0}` + + ` decoded=${b.decoded ?? 0} failed=${b.failed ?? 0}${entries}${used}` + // Repeat the root on the closing line: when a warm-up ends having + // written nothing, where it was writing is the whole diagnosis. + + (finished ? root : "") + + (b.error ? ` error=${b.error}` : "")); + } else { + logInfo("CACHE", `${b.event || "report"}:${root}${entries}${used}` + + (b.error ? ` error=${b.error}` : "")); + } + return res.json({ status: "ok" }); +}); + +// Enumerates every (object, QP, segment start) the headset can be asked to +// play, so it can decode them all once instead of paying a 60-frame Draco +// decode whenever the ladder picks a QP it has not seen. Written by +// scripts/warm_geometry_links.py; regenerate that after changing the object +// catalogue or the QP sweep. +app.get("/api/geometry-warmup-plan", (req, res) => { + const planFile = path.join(FILES_ROOT, "geometry-warmup-plan.json"); + if (!fs.existsSync(planFile)) { + return res.status(404).json({ + error: "no geometry warm-up plan published", + hint: "run: python -m scripts.warm_geometry_links", + }); + } + try { + const plan = JSON.parse(fs.readFileSync(planFile, "utf8")); + logInfo("WARMUP", `Sent warm-up plan: ${plan.entries?.length ?? 0} entries`); + return res.json(plan); + } catch (error) { + return res.status(500).json({ error: `unreadable warm-up plan: ${error.message}` }); + } +}); + +app.get("/api/geometry-bundle", async (req, res) => { + try { + const pattern = String(req.query.pattern || ""); + const start = Number.parseInt(req.query.start, 10); + const count = Math.max(1, Math.min(600, Number.parseInt(req.query.count, 10))); + if (!pattern.startsWith("/files/") || !pattern.includes("%04d") + || !Number.isFinite(start) || !Number.isFinite(count)) { + return res.status(400).json({ error: "invalid geometry bundle request" }); + } + const rootPrefix = path.resolve(FILES_ROOT) + path.sep; + const files = []; + for (let index = 0; index < count; index++) { + const frame = String(start + index).padStart(4, "0"); + const relative = pattern.replace("%04d", frame).substring("/files/".length); + const filename = path.resolve(FILES_ROOT, relative); + if (!filename.startsWith(rootPrefix)) + return res.status(400).json({ error: "geometry path escapes files root" }); + const stat = fs.statSync(filename); + if (!stat.isFile() || stat.size > 0xffffffff) + return res.status(404).json({ error: `invalid geometry frame ${frame}` }); + files.push({ filename, size: stat.size }); + } + const headerBytes = 12 + files.length * 4; + const contentLength = headerBytes + files.reduce((sum, file) => sum + file.size, 0); + res.set({ + "Content-Type": "application/x-vs4d-geometry-bundle", + "Content-Length": String(contentLength), + "Cache-Control": "no-store, no-transform", + }); + const header = Buffer.alloc(headerBytes); + header.write("V4DB", 0, "ascii"); + header.writeUInt32LE(1, 4); + header.writeUInt32LE(files.length, 8); + files.forEach((file, index) => header.writeUInt32LE(file.size, 12 + index * 4)); + res.write(header); + for (const file of files) { + if (res.destroyed) return; + await new Promise((resolve, reject) => { + const input = fs.createReadStream(file.filename); + input.once("error", reject); + input.once("end", resolve); + input.pipe(res, { end: false }); + }); + } + if (!res.destroyed) res.end(); + } catch (error) { + if (!res.headersSent) res.status(404).json({ error: error.message }); + else res.destroy(error); + } +}); + +const PORT = process.env.PORT || 3000; + +function numberFromEnv(name, fallback) { + const value = Number(process.env[name]); + return Number.isFinite(value) && value > 0 ? value : fallback; +} + +function integerFromEnv(name, fallback) { + const value = Number.parseInt(process.env[name], 10); + return Number.isFinite(value) && value > 0 ? value : fallback; +} + +function booleanSetting(value, fallback, name) { + if (value === undefined || value === null || value === "") return fallback; + if (typeof value === "boolean") return value; + const normalized = String(value).trim().toLowerCase(); + if (["1", "true", "yes", "on"].includes(normalized)) return true; + if (["0", "false", "no", "off"].includes(normalized)) return false; + throw new Error(`${name} must be true or false`); +} + +// Must match vstream/config.py FRAMES_PER_SEG / FPS. +const SEGMENT_DURATION = numberFromEnv("VS4D_SEGMENT_DURATION", 2.0); +const FRAMES_PER_SEGMENT = integerFromEnv("VS4D_FRAMES_PER_SEGMENT", 60); +const SEGMENT_INTERVAL = integerFromEnv( + "VS4D_SEGMENT_INTERVAL_MS", + Math.round(SEGMENT_DURATION * 1000) +); +// The default user-study window is 20 seconds at two seconds per segment. +const TOTAL_SEGMENTS = integerFromEnv("VS4D_TOTAL_SEGMENTS", 10); +const UPDATE_INTERVAL_SEGMENTS = integerFromEnv("VS4D_UPDATE_INTERVAL_SEGMENTS", 1); +const STREAM_CONFIG = { + segmentDuration: SEGMENT_DURATION, + framesPerSegment: FRAMES_PER_SEGMENT, + segmentIntervalMs: SEGMENT_INTERVAL, + totalSegments: TOTAL_SEGMENTS, + updateIntervalSegments: UPDATE_INTERVAL_SEGMENTS, +}; + +// Quest selects its pipeline from this server-owned launch profile whenever A +// is pressed. The file is read per request so switching experiments does not +// require an APK rebuild, adb, or even a Node restart. Environment variables +// take precedence for scripted runs. +const QUEST_LAUNCH_OVERRIDE = String(process.env.VS4D_QUEST_LAUNCH || "").trim(); +const QUEST_LAUNCH_FILE = QUEST_LAUNCH_OVERRIDE + ? path.resolve(QUEST_LAUNCH_OVERRIDE) + : path.join(__dirname, "quest-launch.json"); +// Vega and NeVo are explicit 30-frame offline-only paths. They are deliberately +// absent from scripts/user_study.py's condition set. +const QUEST_PIPELINES = new Set(["mesh", "metastream", "deltastream", "vivo", "nava", "livo", "vega", "nevo"]); + +function questLaunchProfile(requestHost) { + let file = {}; + if (QUEST_LAUNCH_OVERRIDE && !fs.existsSync(QUEST_LAUNCH_FILE)) { + throw new Error( + `VS4D_QUEST_LAUNCH does not exist: ${QUEST_LAUNCH_FILE}`); + } + if (fs.existsSync(QUEST_LAUNCH_FILE)) { + file = JSON.parse(fs.readFileSync(QUEST_LAUNCH_FILE, "utf8")); + if (!file || Array.isArray(file) || typeof file !== "object") + throw new Error("quest launch profile must be a JSON object"); + } + const pipeline = String(process.env.VS4D_QUEST_PIPELINE || file.pipeline || "mesh") + .trim().toLowerCase(); + if (!QUEST_PIPELINES.has(pipeline)) + throw new Error(`unsupported Quest pipeline '${pipeline}'`); + const baselinePort = Number.parseInt( + process.env.VS4D_BASELINE_PORT ?? file.baselinePort ?? 12345, 10); + if (!Number.isInteger(baselinePort) || baselinePort <= 0 || baselinePort > 65535) + throw new Error("baselinePort must be an integer from 1 through 65535"); + const abr = Number(process.env.VS4D_BASELINE_ABR_BANDWIDTH_MBPS + ?? file.baselineAbrBandwidthMbps ?? 100); + if (!Number.isFinite(abr) || abr <= 0) + throw new Error("baselineAbrBandwidthMbps must be positive"); + const connectionTimeout = Number(file.baselineConnectionTimeoutSeconds ?? 10); + if (!Number.isFinite(connectionTimeout) || connectionTimeout <= 0) + throw new Error("baselineConnectionTimeoutSeconds must be positive"); + const splatSize = Number(file.pointSplatSize ?? 0.006); + if (!Number.isFinite(splatSize) || splatSize <= 0) + throw new Error("pointSplatSize must be positive"); + const receiveBytes = Number.parseInt( + file.maxBaselineReceiveBufferBytes ?? 268435456, 10); + if (!Number.isInteger(receiveBytes) || receiveBytes < 1024 * 1024) + throw new Error("maxBaselineReceiveBufferBytes must be at least 1048576"); + const fullBandwidthMode = booleanSetting( + process.env.VS4D_FULL_BANDWIDTH_MODE ?? file.fullBandwidthMode, + true, "fullBandwidthMode"); + const groundTruthBandwidthMode = booleanSetting( + process.env.VS4D_GROUND_TRUTH_BANDWIDTH_MODE + ?? file.groundTruthBandwidthMode, + false, "groundTruthBandwidthMode"); + if (groundTruthBandwidthMode && pipeline !== "mesh") + throw new Error("groundTruthBandwidthMode is only supported by pipeline=mesh"); + const ablationVariant = String(file.ablationVariant || "").trim(); + if (ablationVariant && pipeline !== "mesh") + throw new Error("ablationVariant is only supported by pipeline=mesh"); + const geometryTextureAdaptationEnabled = booleanSetting( + file.geometryTextureAdaptationEnabled, true, + "geometryTextureAdaptationEnabled"); + const globalAllocationEnabled = booleanSetting( + file.globalAllocationEnabled, true, "globalAllocationEnabled"); + const objectSchedulingEnabled = booleanSetting( + file.objectSchedulingEnabled, true, "objectSchedulingEnabled"); + const fastSwitchingEnabled = booleanSetting( + file.fastSwitchingEnabled, true, "fastSwitchingEnabled"); + const frameBufferEnabled = booleanSetting( + file.frameBufferEnabled, true, "frameBufferEnabled"); + const viewportPredictionEnabled = booleanSetting( + file.viewportPredictionEnabled, true, "viewportPredictionEnabled"); + const viewportTraceEnabled = booleanSetting( + file.viewportTraceEnabled, true, "viewportTraceEnabled"); + const orbitViewCullingEnabled = booleanSetting( + file.orbitViewCullingEnabled, true, "orbitViewCullingEnabled"); + const offlineBenchmarkMode = booleanSetting( + file.offlineBenchmarkMode, false, "offlineBenchmarkMode"); + const visualBaselineStreamingMode = booleanSetting( + file.visualBaselineStreamingMode, false, "visualBaselineStreamingMode"); + if (visualBaselineStreamingMode && pipeline !== "vega" && pipeline !== "nevo") + throw new Error("visualBaselineStreamingMode requires pipeline=vega or pipeline=nevo"); + if ((pipeline === "vega" || pipeline === "nevo") && !offlineBenchmarkMode) + throw new Error(`the ${pipeline} Quest pipeline is restricted to offlineBenchmarkMode=true`); + const sceneObjects = Array.isArray(file.sceneObjects) + ? file.sceneObjects.map(value => String(value).trim()).filter(Boolean) : []; + if (new Set(sceneObjects).size !== sceneObjects.length) + throw new Error("sceneObjects must not contain duplicates"); + const viewpointNumber = (name, fallback, allowZero = false) => { + const value = Number(file[name] ?? fallback); + if (!Number.isFinite(value) || (allowZero ? value < 0 : value <= 0)) + throw new Error(`${name} must be ${allowZero ? "non-negative" : "positive"}`); + return value; + }; + return { + pipeline, + // The host from the Quest's HTTP request is the Node machine address it + // can actually reach, making an explicit baselineHost optional. + baselineHost: String(process.env.VS4D_BASELINE_HOST + || file.baselineHost || requestHost || "").trim(), + baselinePort, + baselineDatasetManifest: String(file.baselineDatasetManifest || ""), + baselineTileCatalog: String(file.baselineTileCatalog || ""), + baselineConnectionTimeoutSeconds: connectionTimeout, + baselineAbrBandwidthMbps: abr, + pointSplatSize: splatSize, + maxBaselineReceiveBufferBytes: receiveBytes, + fullBandwidthMode, + groundTruthBandwidthMode, + ablationVariant, + geometryTextureAdaptationEnabled, + globalAllocationEnabled, + objectSchedulingEnabled, + fastSwitchingEnabled, + frameBufferEnabled, + viewportPredictionEnabled, + viewportHistoryWindowSec: viewpointNumber("viewportHistoryWindowSec", 0.5), + viewportPredictionWindowSec: viewpointNumber( + "viewportPredictionWindowSec", 0.5, true), + viewportSampleRateHz: viewpointNumber("viewportSampleRateHz", 36), + viewportTraceEnabled, + orbitViewCullingEnabled, + orbitCullMaxDistanceMetres: viewpointNumber("orbitCullMaxDistanceMetres", 20), + orbitCullFovMarginDegrees: viewpointNumber( + "orbitCullFovMarginDegrees", 5, true), + scene: String(file.scene || "").trim(), + sceneObjects, + skybox: String(file.skybox || "").trim(), + studyTrial: String(file.studyTrial || "").trim(), + studyMethod: String(file.studyMethod || "").trim(), + participantId: String(file.participantId || "").trim(), + studyTrialIndex: Number(file.studyTrialIndex || 0), + studyTotalTrials: Number(file.studyTotalTrials || 0), + studyProtocolVersion: String(file.studyProtocolVersion || "").trim(), + plannedDurationSeconds: Number(file.plannedDurationSeconds || 0), + studyReady: booleanSetting(file.studyReady, true, "studyReady"), + offlineBenchmarkMode, + visualBaselineStreamingMode, + offlineTrajectoryBroadcastId: String( + file.offlineTrajectoryBroadcastId || "").trim(), + source: fs.existsSync(QUEST_LAUNCH_FILE) + ? path.basename(QUEST_LAUNCH_FILE) : "server defaults", + }; +} + +function getManifestPath(segId) { + return path.resolve( + path.join(FILES_ROOT, `t_${String(segId).padStart(2, "0")}`), + "menu.json" + ); +} +function getStatePath(broadcastId) { + const id = broadcastId || "default"; + const dir = path.join(STATE_DIR); + ensureDir(dir); + return dir; // directory, not a file +} + + +function ensureManifestDir(segId) { + const outPath = getManifestPath(segId); + const dir = path.dirname(outPath); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); + return outPath; +} + + +const SERVER_VIEWPOINTS_DIR = path.resolve(__dirname, "server_viewpoints"); +const SERVER_SELECTIONS_DIR = path.resolve(__dirname, "server_selections"); +const SERVER_RESULTS_DIR = path.resolve(__dirname, "server_results"); // NEW +const VIEWPORT_PLOT_SCRIPT = path.join( + REPO_ROOT, "system", "QuestClient", "Tools", "plot_viewport_traces.py" +); +const STATE_DIR = FILES_ROOT; + +// Run-scoped evaluation plots and CSVs are intentionally separate from the +// media tree. This mount makes the generated PNGs viewable without exposing +// arbitrary host paths returned by the debugging API. +app.use("/server-results", express.static(SERVER_RESULTS_DIR)); + +// All per-run artifacts (metrics, bitrate counts, QoE, download telemetry) +// live in server_results// so each run stays self-contained. +function broadcastResultsDir(broadcastId) { + const id = broadcastId || currentBroadcastId || "no-broadcast"; + const dir = path.join(SERVER_RESULTS_DIR, id); + ensureDir(dir); + return dir; +} +function broadcastArtifactDir(broadcastId, name) { + const dir = path.join(broadcastResultsDir(broadcastId), name); + ensureDir(dir); + return dir; +} +function intervalsFilePath(broadcastId) { + return path.join(broadcastResultsDir(broadcastId), "bitrate_intervals.jsonl"); +} + +ensureDir(STATE_DIR); + +const PYTHON_BIN = process.env.PYTHON_BIN || "python"; +// Importing matplotlib and redrawing three figures is intentionally kept out +// of the request path, but doing it for every two-second Quest trace upload can +// still keep a CPU busy continuously alongside the ladder process. Ten seconds +// keeps the dashboard live without making evaluation compete with streaming. +const VIEWPORT_PLOT_INTERVAL_MS = integerFromEnv( + "VS4D_VIEWPORT_PLOT_INTERVAL_MS", 10000 +); + +// -------------------- State -------------------- +const broadcasts = new Map(); +const groundTruthBandwidthByBroadcast = new Map(); +const selectionLog = []; +const viewportPlotJobs = new Map(); + +// Keep a startup/default menu available as before. Starting a broadcast resets +// this to -1, and that broadcast's first viewpoint force-regenerates segment 0 +// before the client fetches it, so stale/uniform weights are never consumed by +// an active run. +let latestManifestSegId = fs.existsSync(getManifestPath(0)) ? 0 : -1; +let ladderBusy = false; +let pendingSegId = null; +let manifestGeneration = 0; +const manifestOwners = new Map(); + +// NEW: Track current broadcast for results naming +let currentBroadcastId = null; +let currentTraceStartTime = null; +let currentScene = ""; +let currentSceneObjects = []; +let currentSkybox = ""; +let lastQuestLaunchLogKey = ""; + +// Latest bandwidth estimate reported by the client (Mbps); forwarded to the +// ladder service as the cap on expected served bitrate (C_cap). +let latestClientBandwidthMbps = null; + +// -------------------- Ensure dirs -------------------- +function ensureDir(p) { + if (!fs.existsSync(p)) fs.mkdirSync(p, { recursive: true }); +} +ensureDir(SERVER_VIEWPOINTS_DIR); +ensureDir(SERVER_SELECTIONS_DIR); +ensureDir(SERVER_RESULTS_DIR); // NEW +ensureDir(path.dirname(getManifestPath(0))); + +function viewportTracePath(broadcastId) { + return path.join(broadcastResultsDir(broadcastId), "viewport_trace.csv"); +} + +function viewportEvaluationDir(broadcastId) { + return path.join(broadcastResultsDir(broadcastId), "viewport_evaluation"); +} + +// Regenerate at most one plot set per broadcast at a time. Uploads arriving +// during a render mark it dirty and cause exactly one follow-up render. +function scheduleViewportPlots(broadcastId) { + let state = viewportPlotJobs.get(broadcastId); + if (!state) { + state = { + running: false, dirty: false, timer: null, lastError: null, lastStartedAt: 0, + }; + viewportPlotJobs.set(broadcastId, state); + } + state.dirty = true; + if (state.running || state.timer) return; + const earliestStart = state.lastStartedAt + VIEWPORT_PLOT_INTERVAL_MS; + const delay = Math.max(750, earliestStart - Date.now()); + state.timer = setTimeout(() => { + state.timer = null; + state.running = true; + state.dirty = false; + state.lastError = null; + state.lastStartedAt = Date.now(); + const trace = viewportTracePath(broadcastId); + const output = viewportEvaluationDir(broadcastId); + ensureDir(output); + const summary = path.join(output, "viewport_metrics.json"); + const child = spawn(PYTHON_BIN, [ + VIEWPORT_PLOT_SCRIPT, + trace, + "--out", output, + "--streaming-only", + "--summary-json", summary, + ], { + cwd: REPO_ROOT, + env: { ...process.env, MPLCONFIGDIR: path.join(output, ".matplotlib") }, + }); + let diagnostics = ""; + const collect = chunk => { + diagnostics = (diagnostics + chunk.toString()).slice(-8000); + }; + child.stdout.on("data", collect); + child.stderr.on("data", collect); + child.on("error", error => { + state.lastError = error.message; + logWarn("VIEWPORT", `Plot launch failed for ${broadcastId}: ${error.message}`); + }); + child.on("close", code => { + state.running = false; + if (code === 0) { + logInfo("VIEWPORT", `Evaluation plots updated: /viewport-evaluation/${broadcastId}`); + } else { + state.lastError = diagnostics.trim() || `plotter exited ${code}`; + logWarn("VIEWPORT", `Plot update failed for ${broadcastId}: ${state.lastError}`); + } + if (state.dirty) scheduleViewportPlots(broadcastId); + }); + }, delay); +} + +// -------------------- Manifest helpers -------------------- +function atomicWriteJson(outPath, obj) { + const tmp = outPath + ".tmp"; + fs.writeFileSync(tmp, JSON.stringify(obj, null, 2)); + fs.renameSync(tmp, outPath); +} + +function loadManifest(segId = latestManifestSegId) { + const mp = getManifestPath(segId); + if (!fs.existsSync(mp)) return null; + const content = fs.readFileSync(mp, "utf-8"); + return JSON.parse(content); +} +///////////////////// + +let ladderProc = null; +let rl = null; +let pending = []; + +function startLadderService() { + if (ladderProc) return; + + ladderProc = spawn(PYTHON_BIN, ["-m", "vstream.ladder.ladder_service"], { + cwd: REPO_ROOT, + env: process.env, + stdio: ["pipe", "pipe", "pipe"], + }); + + rl = readline.createInterface({ input: ladderProc.stdout }); + + rl.on("line", (line) => { + const s = line.trim(); + + // Ignore non-JSON lines (Python logs) + if (!s.startsWith("{")) { + logInfo("LADDER-PY", s); + return; + } + + const item = pending.shift(); + if (!item) return; + + try { + const resp = JSON.parse(s); + item.resolve(resp); + } catch (e) { + item.reject(new Error(`Bad JSON from ladder service: ${e.message}\nLine: ${line}`)); + } + }); + + + logInfo("LADDER", `Started ladder service: ${PYTHON_BIN} -m vstream.ladder.ladder_service (pid ${ladderProc.pid})`); + + ladderProc.stderr.on("data", (d) => { + // keep it for debugging + const text = d.toString().trimEnd(); + if (text) logWarn("LADDER-PY", text); + }); + + ladderProc.on("exit", (code) => { + logError("LADDER", `Ladder service exited with code ${code}; ${pending.length} pending request(s) failed`); + ladderProc = null; + if (rl) rl.close(); + rl = null; + // fail all pending requests + while (pending.length) pending.shift().reject(new Error("ladder_service died")); + }); +} + +// -------------------- Python runner -------------------- +// viewSegId names the viewpoint file that must weight this solve. It cannot be +// derived from segId: the per-segment path below solves segId+1 from the pose it +// just wrote at segId, while the viewpoint bootstrap solves segId from the pose +// at segId. The service used to guess `segment_{segId}.json`, which - because +// SERVER_VIEWPOINTS_DIR is shared by every broadcast and never cleared - always +// resolved to a leftover file from an earlier run. Pass null when no client pose +// exists yet (the startup default manifest) and the ladder weights uniformly. +function runLadderPython(segId, broadcastId, viewSegId, sceneObjects) { + startLadderService(); + + return new Promise((resolve, reject) => { + const lastSegId = Math.max(0, latestManifestSegId); + logInfo("LADDER", `Requesting ladder: seg=${segId} last_seg=${lastSegId} ` + + `view_seg=${viewSegId ?? "none"} broadcast=${broadcastId || "default"}`); + + // Object catalogs (mesh/texture patterns, start frames) are NOT sent: + // the ladder service defaults to vstream/config.py, the single source + // of truth for the active object set. + const req = { + current_seg_id: segId, + last_seg_id: lastSegId, + view_dir: SERVER_VIEWPOINTS_DIR, + requests_path: intervalsFilePath(broadcastId), + state_path: getStatePath(broadcastId), + view_seg_id: Number.isFinite(viewSegId) ? viewSegId : null, + files_root: FILES_ROOT, // optional + // client's bandwidth estimate caps the ladder's expected served bitrate + bandwidth_mbps: latestClientBandwidthMbps, + objects: Array.isArray(sceneObjects) && sceneObjects.length + ? sceneObjects : undefined, + }; + + pending.push({ resolve, reject }); + + ladderProc.stdin.write(JSON.stringify(req) + "\n"); + }); +} + + +// -------------------- Ladder update queue -------------------- +let pendingUpdate = null; +let ladderDrainPromise = null; + +function requestLadderUpdate(segId, broadcastId, reason = "", force = false, + viewSegId = null) { + // viewSegId travels with the update so that coalescing keeps the pose and the + // segment consistent: collapsing to the newest segId also keeps that request's + // pose, which is the freshest one on disk. + const update = { + segId, broadcastId, reason, force, viewSegId, + generation: manifestGeneration, + sceneObjects: [...currentSceneObjects], + }; + // Coalesce ordinary updates to the newest segment. A forced viewpoint + // bootstrap wins because it establishes the state for a new broadcast. + if (!pendingUpdate || force || (!pendingUpdate.force && segId >= pendingUpdate.segId)) { + pendingUpdate = update; + pendingSegId = segId; + } + if (!ladderDrainPromise) { + ladderDrainPromise = drainLadderUpdates().finally(() => { ladderDrainPromise = null; }); + } + return ladderDrainPromise; +} + +async function drainLadderUpdates() { + ladderBusy = true; + try { + while (pendingUpdate !== null) { + const update = pendingUpdate; + const seg = update.segId; + pendingUpdate = null; + pendingSegId = null; + + // A faster later run may already cover an ordinary update. The first + // viewpoint of a broadcast deliberately replaces segment 0 on disk. + if (!update.force && seg <= latestManifestSegId) continue; + + try { + logInfo("LADDER", `Generating ladder for seg=${seg}${update.reason ? ` (${update.reason})` : ""}`); + + const resp = await runLadderPython( + seg, update.broadcastId, update.viewSegId, update.sceneObjects); + + if (resp.status === "error") { + throw new Error(resp.error + (resp.traceback ? `\n${resp.traceback}` : "")); + } + + // Python should return the path to the generated manifest/menu.json + const manifestPath = resp.mpd_path || resp.manifestPath || resp.manifest_path; + if (!manifestPath) { + throw new Error(`ladder service returned no mpd_path. resp=${JSON.stringify(resp)}`); + } + + // (optional) sanity check file exists + if (!fs.existsSync(manifestPath)) { + throw new Error(`Manifest not found at ${manifestPath}`); + } + + // A solve from the preceding user-study trial may finish after the next + // broadcast has reset the manifest clock. Its Python output can remain + // on disk for diagnostics/cache reuse, but must never become eligible + // as this broadcast's temporal revision. + if (update.generation !== manifestGeneration) { + logWarn("LADDER", `Discarded stale-broadcast manifest seg=${seg}`); + continue; + } + + latestManifestSegId = seg; + manifestOwners.set(seg, update.broadcastId || null); + if (update.broadcastId && broadcasts.has(update.broadcastId)) { + fs.appendFileSync( + path.join(broadcastResultsDir(update.broadcastId), "manifest_events.jsonl"), + JSON.stringify({ + event: "published", segmentId: seg, reason: update.reason || null, + elapsedSeconds: resp.elapsed_sec ?? null, + timingsMs: resp.timings_ms ?? null, + publishedAt: new Date().toISOString(), + }) + "\n" + ); + } + logInfo( + "LADDER", + `Manifest ready: seg=${seg} path=${manifestPath}` + + (resp.elapsed_sec != null ? ` elapsed=${resp.elapsed_sec.toFixed(3)}s` : "") + ); + if (resp.elapsed_sec != null && resp.elapsed_sec > SEGMENT_DURATION) { + logWarn("LADDER", `Ladder generation slower than one segment: ${resp.elapsed_sec.toFixed(3)}s > ${SEGMENT_DURATION}s`); + } + + // (optional) if you want to warm-load it / validate JSON: + // const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf-8")); + + } catch (err) { + logError("LADDER", `Ladder generation failed for seg=${seg}: ${err.message}`); + if (update.broadcastId && broadcasts.has(update.broadcastId)) { + fs.appendFileSync( + path.join(broadcastResultsDir(update.broadcastId), "manifest_events.jsonl"), + JSON.stringify({ + event: "failed", segmentId: seg, reason: update.reason || null, + error: err.message, failedAt: new Date().toISOString(), + }) + "\n" + ); + } + } + } + } finally { + ladderBusy = false; + } +} + + +// -------------------- Routes -------------------- + +app.get("/api/manifest", async (req, res) => { + const segQ = req.query.seg; + const seg = + segQ !== undefined && segQ !== null && segQ !== "" + ? parseInt(segQ, 10) + : latestManifestSegId; + + if (!Number.isInteger(seg) || seg < 0) + return res.status(400).json({ error: "invalid manifest segment", seg: segQ }); + + // A numbered request is a temporal-content contract, not a request for the + // latest quality menu. If the normal one-segment-ahead update is still being + // solved, join that drain; if it was lost/failed, demand the exact revision. + // Returning the latest revision here made logical segment N silently replay + // an older source segment whenever ladder generation exceeded two seconds. + // Files from a previous broadcast remain on disk for diagnostics. Only a + // revision published in the current broadcast (tracked by the resettable + // latestManifestSegId) is eligible for delivery. + const broadcastId = String(req.query.broadcastId || ""); + const ownedByRequest = segQ === undefined || !broadcastId + || manifestOwners.get(seg) === broadcastId; + let manifest = seg <= latestManifestSegId && ownedByRequest ? loadManifest(seg) : null; + if (!manifest && segQ !== undefined && broadcasts.has(broadcastId)) { + try { + await requestLadderUpdate(seg, broadcastId, "exact manifest demand", false, + Math.max(0, seg - 1)); + } catch (error) { + logError("MANIFEST", `Exact manifest generation failed for seg=${seg}: ${error.message}`); + } + manifest = seg <= latestManifestSegId && manifestOwners.get(seg) === broadcastId + ? loadManifest(seg) : null; + } + if (!manifest) { + logWarn("MANIFEST", `Manifest not found for seg=${seg}`); + return res.status(503).json({ error: "Manifest not ready", seg }); + } + + const actual = Number(manifest?.segment?.t); + if (segQ !== undefined && actual !== seg) { + logError("MANIFEST", `Refusing mismatched manifest: requested=${seg} actual=${actual}`); + return res.status(409).json({ error: "Manifest temporal mismatch", requested: seg, actual }); + } + + if (broadcastId && broadcasts.has(broadcastId)) { + const archive = path.join( + broadcastArtifactDir(broadcastId, "manifests"), + `segment_${String(seg).padStart(4, "0")}.json` + ); + if (!fs.existsSync(archive)) atomicWriteJson(archive, manifest); + } + + logInfo("MANIFEST", `Sent manifest seg=${seg} objects=${Object.keys(manifest.objects || {}).length}`); + res.json(manifest); +}); + +// Viewpoint index for the browser client's simulated mode. +// +// Interactive clients send their own live pose, but a simulated run replays +// canned poses and the browser has no filesystem to read them from. These are +// the same Open3D PinholeCameraParameters files the desktop client loads from +// system/Client/viewpoints, served as one document so the page needs a single +// request. `?dir=` selects a sibling set (the directory holds several). +// Is something listening? Used to tell "the corpus exists" from "a server is +// actually up", which need opposite responses from whoever is reading the +// chooser. Short timeout: this runs inline in a request handler. +function portIsOpen(port, host = "127.0.0.1", timeoutMs = 250) { + return new Promise(resolve => { + const socket = new net.Socket(); + const done = (value) => { + socket.destroy(); + resolve(value); + }; + socket.setTimeout(timeoutMs); + socket.once("connect", () => done(true)); + socket.once("timeout", () => done(false)); + socket.once("error", () => done(false)); + socket.connect(port, host); + }); +} + +// MetaStream, DeltaStream and LiVo are deliberately absent from the demo. +// They index the real capture rig while serving (`obj.cameras[camera_index]`) +// and read the source RGB-D frames, so the prepared tiles cannot substitute +// and the absent corpus is missing data rather than missing metadata. Listing +// them as permanently unavailable was noise in a chooser whose job is to say +// what you can look at. +// +// The point-cloud baselines that can run from the prepared tiles, and the +// ports scripts/serve_pointcloud_baseline.sh gives each by default. Both can +// be up at once, which is the point: switching baseline is then a click. +const POINTCLOUD_BASELINES = (process.env.VS4D_POINTCLOUD_PORTS + || "vivo:8790:12345,nava:8791:12346").split(",").map(entry => { + const [id, bridgePort, serverPort] = entry.split(":"); + return { id, bridgePort: Number(bridgePort), serverPort: Number(serverPort) }; + }); + +// The shaped link rate, read from tc. +// +// Without this the demo cannot show what it exists to show. Adaptation is a +// response to a changing link, and on an unshaped LAN all three adaptive +// systems sit on one operating point forever -- NAVA held quality level 5 for +// thirteen consecutive segments here. Reporting the rate the kernel is +// actually enforcing, beside each client's own estimate, is what makes a +// representation switch legible as cause and effect rather than noise. +// +// Read-only and unprivileged: `tc qdisc show` needs no root, only installing +// rules does. +const SHAPED_INTERFACE = process.env.VS4D_SHAPED_INTERFACE || "eth0"; + +// Start (or restart) a point-cloud baseline with a chosen object set. +// +// A baseline fixes its scene at startup -- `--objects` on the Python server -- +// so unlike every other system here, choosing objects means replacing the +// process. That is why this exists rather than a query parameter. +// +// This endpoint spawns processes and the server listens on 0.0.0.0, so the +// input is constrained hard rather than trusted: +// * the baseline id must be one of the two configured ones, never a module +// name from the request; +// * every object name must appear in the tile catalogue, so a name cannot +// reach the command line unless the corpus already contains it; +// * the child is spawned with an argv array and no shell, so nothing in the +// request is interpretable as syntax. +const runningPointcloud = new Map(); // id -> objects[] currently served +const pointcloudChildren = new Map(); // id -> ChildProcess + +function stopPointcloud(id) { + const child = pointcloudChildren.get(id); + if (!child) return; + pointcloudChildren.delete(id); + runningPointcloud.delete(id); + // The supervisor script traps TERM and takes its bridge and server with it. + try { process.kill(-child.pid, "SIGTERM"); } + catch (_) { try { child.kill("SIGTERM"); } catch (_) { /* already gone */ } } +} + +app.post("/api/pointcloud/start", async (req, res) => { + const id = String(req.body?.id || ""); + const entry = POINTCLOUD_BASELINES.find(value => value.id === id); + if (!entry) { + return res.status(400).json({ error: `unknown baseline ${JSON.stringify(id)}` }); + } + const catalogPath = path.join(VIVO_TILES_ROOT, "catalog.json"); + let known; + try { + known = new Set((JSON.parse(fs.readFileSync(catalogPath, "utf8")).objects || []) + .map(value => String(value.name))); + } catch (error) { + return res.status(409).json({ + error: `no tile catalogue at ${catalogPath}: ${error.message}` }); + } + const requested = Array.isArray(req.body?.objects) + ? req.body.objects.map(value => String(value).trim()).filter(Boolean) : []; + if (!requested.length) { + return res.status(400).json({ error: "choose at least one object" }); + } + const unknown = requested.filter(name => !known.has(name)); + if (unknown.length) { + return res.status(400).json({ + error: `not in the tile catalogue: ${unknown.join(", ")}` }); + } + + stopPointcloud(id); + const script = path.join(REPO_ROOT, "scripts/serve_pointcloud_baseline.sh"); + const child = spawn("bash", [script, id, requested.join(",")], { + cwd: REPO_ROOT, + detached: true, // its own group, so one kill stops all + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, PYTHON_BIN: PYTHON_BIN } + }); + child.stdout.on("data", chunk => logInfo(id.toUpperCase(), String(chunk).trim())); + child.stderr.on("data", chunk => logInfo(id.toUpperCase(), String(chunk).trim())); + child.on("exit", code => { + if (pointcloudChildren.get(id) === child) { + pointcloudChildren.delete(id); + runningPointcloud.delete(id); + } + logInfo(id.toUpperCase(), `supervisor exited (${code})`); + }); + pointcloudChildren.set(id, child); + runningPointcloud.set(id, requested); + + // Report only once it is actually accepting connections; the page navigates + // on this response, and arriving before the bridge is up reads as a failure. + const deadline = Date.now() + 40000; + while (Date.now() < deadline) { + if (!pointcloudChildren.has(id)) { + return res.status(500).json({ error: `${id} supervisor exited during startup` }); + } + // BOTH ports, not just the bridge. The bridge listens as soon as the + // supervisor starts it and stays up even when the baseline behind it has + // crashed and is being restarted in a loop, so probing it alone reports + // "serving" over a dead baseline -- observed when the spawned process had + // the wrong interpreter. + if (await portIsOpen(entry.bridgePort) + && await portIsOpen(entry.serverPort)) { + logInfo(id.toUpperCase(), `serving ${requested.join(", ")}`); + return res.json({ + id, objects: requested, bridgePort: entry.bridgePort, + bridge: `ws://${req.hostname}:${entry.bridgePort}` + }); + } + await new Promise(resolve => setTimeout(resolve, 500)); + } + const bridgeUp = await portIsOpen(entry.bridgePort); + stopPointcloud(id); + res.status(504).json({ + error: bridgeUp + // Naming the port that failed matters: a live bridge with a dead + // baseline means the Python side could not start (wrong interpreter, + // unreadable corpus), which is a different fix from a dead bridge. + ? `${id}: the bridge is up on ${entry.bridgePort} but the baseline ` + + `server never listened on ${entry.serverPort} — check the corpus and ` + + "PYTHON_BIN in the server log" + : `${id} did not start listening on ${entry.bridgePort} within 40s` }); +}); + +// Leaving orphaned Python servers and bridges behind would hold their ports +// and silently serve a stale scene to the next run. +for (const signal of ["SIGINT", "SIGTERM"]) { + process.on(signal, () => { + for (const id of [...pointcloudChildren.keys()]) stopPointcloud(id); + process.exit(0); + }); +} + +app.get("/api/shaping", (req, res) => { + execFile("tc", ["qdisc", "show", "dev", SHAPED_INTERFACE], + { timeout: 2000 }, (error, stdout) => { + res.set("Cache-Control", "no-store"); + if (error) { + return res.json({ + interface: SHAPED_INTERFACE, shaped: false, + detail: `could not read tc on ${SHAPED_INTERFACE}: ${error.message}` + }); + } + // The trace player installs a root TBF; anything else means unshaped. + const root = stdout.split(/\n(?=qdisc )/) + .find(block => /^qdisc tbf \S+ root\b/.test(block.trim())); + if (!root) { + return res.json({ + interface: SHAPED_INTERFACE, shaped: false, + detail: "no root TBF installed — the link is unshaped, so the " + + "adaptive systems will hold one operating point. Install a trace " + + "with scripts/shape_web_demo.sh" + }); + } + // `rate 12500Kbit`, `rate 1Gbit`, `rate 950Mbit` + const match = /\brate (\d+(?:\.\d+)?)([KMG])?bit\b/.exec(root); + const unit = { K: 1e-3, M: 1, G: 1e3 }; + const mbps = match + ? Number(match[1]) * (unit[match[2]] ?? 1e-6) + : null; + res.json({ + interface: SHAPED_INTERFACE, shaped: true, + rateMbps: mbps === null ? null : Number(mbps.toFixed(2)), + detail: root.trim().split("\n")[0] + }); + }); +}); + +// What the comparison pages can actually run right now. +// +// Every baseline needs assets or a process that may simply not be there: Vega +// needs an export, NeVo needs pre-rendered frames, the point-cloud baselines +// need a Python server plus a WebSocket bridge. Without this the pages fail at +// fetch time with a 404, which reads as a bug in the viewer rather than as +// missing input. The chooser asks here first and says which is which. +// +// The mesh entry also carries the per-object ladder cost, because the scene +// size sets an irreducible bitrate floor (the ladder must publish at least one +// representation per object) and that floor, not the ABR, is what decides +// whether a link can carry the scene. +app.get("/api/systems", async (req, res) => { + const dirHasAny = (dir, predicate) => { + try { return fs.readdirSync(dir).some(predicate); } catch (_) { return false; } + }; + + // The catalog comes from the packaged media tree, not from the last + // manifest. A manifest is a run artifact: after a run restricted to three + // objects it names only those three, and a picker built from it could then + // never offer the other six back. Costs still come from the manifest, so an + // object outside the current one is selectable with its cost reported as + // unknown rather than as zero. + const manifest = loadManifest(); + const published = manifest?.objects || {}; + const packaged = (() => { + try { + return fs.readdirSync(path.join(FILES_ROOT, "media"), { withFileTypes: true }) + .filter(entry => entry.isDirectory()).map(entry => entry.name); + } catch (_) { return []; } + })(); + const names = [...new Set([...packaged, ...Object.keys(published)])].sort(); + + const objects = names.map(name => { + const entry = published[name]; + const reps = Array.isArray(entry?.representations) ? entry.representations : []; + const rates = reps + .map(rep => Number(rep?.predicted?.bitrate_mbps)) + .filter(Number.isFinite); + return { + name, + inManifest: Boolean(entry), + weight: entry ? Number(entry.weight) || 0 : null, + representations: reps.length, + floorMbps: rates.length ? Math.min(...rates) : null, + ceilingMbps: rates.length ? Math.max(...rates) : null + }; + }); + const priced = objects.filter(o => o.floorMbps !== null); + const floorMbps = priced.reduce((sum, o) => sum + o.floorMbps, 0); + + // Vega's own catalogue, read for the same reason: so the chooser can offer + // its objects instead of the page taking all of them by default. + const vega = (() => { + try { + const catalog = JSON.parse(fs.readFileSync( + path.join(VEGA_ASSETS_ROOT, "catalog.json"), "utf8")); + const entries = (catalog.objects || []).map(entry => ({ + name: String(entry.name), + bytes: (entry.frames || []).reduce( + (sum, frame) => sum + (Number(frame.exportBytes) || 0), 0), + points: Number(entry.frames?.[0]?.points) || 0 + })); + return { + objects: entries, + bytes: entries.reduce((sum, entry) => sum + entry.bytes, 0) + }; + } catch (_) { return { objects: [], bytes: 0 }; } + })(); + + // One entry per point-cloud baseline, each probed independently. Both can + // be up at once on different ports, so choosing between ViVo and NAVA is a + // click rather than a server restart. + const haveTiles = fs.existsSync(path.join(VIVO_TILES_ROOT, "catalog.json")); + // The tile catalogue's object list, so the chooser can offer a selection. + // Unlike the other systems this is not a URL parameter: a baseline fixes its + // object set at startup, so choosing one means restarting the process. + const tileObjects = (() => { + if (!haveTiles) return []; + try { + const catalog = JSON.parse(fs.readFileSync( + path.join(VIVO_TILES_ROOT, "catalog.json"), "utf8")); + return (catalog.objects || []).map(entry => ({ + name: String(entry.name), + points: Math.max(0, ...(entry.sequence_tiles || []) + .map(tile => Number(tile.max_point_count) || 0)) + })); + } catch (_) { return []; } + })(); + const NAMES = { vivo: "ViVo", nava: "NAVA" }; + const pointcloud = await Promise.all(POINTCLOUD_BASELINES.map(async entry => { + const live = haveTiles && await portIsOpen(entry.bridgePort); + return { + id: entry.id, + name: NAMES[entry.id] || entry.id, + page: "/web/baseline.html", + bridge: `ws://${req.hostname}:${entry.bridgePort}`, + // These do adapt, but the decision is the Python server's, driven by the + // client's pose and goodput feedback -- not the browser's. Conflating + // that with our own client-side ABR would misrepresent the comparison. + adaptive: false, + adaptsServerSide: true, + ready: live, + objects: tileObjects, + // What the running process was started with, so the chooser can tell a + // restart is needed rather than silently connecting to the wrong scene. + servingObjects: runningPointcloud.get(entry.id) || null, + restartable: haveTiles, + detail: !haveTiles + ? `no tile corpus at ${VIVO_TILES_ROOT} (prepare it with ` + + "baselines.ViVo.orbitvivo.prepare)" + : live + ? `serving on port ${entry.bridgePort}` + : `nothing listening on ${entry.bridgePort} — start it with ` + + `scripts/serve_pointcloud_baseline.sh ${entry.id}` + }; + })); + + res.set("Cache-Control", "no-store"); + res.json({ + schemaVersion: 1, + systems: [ + { + id: "mesh", + name: "Ours", + page: "/web/", + adaptive: true, + ready: objects.length > 0, + detail: objects.length + ? `${objects.length} objects · ladder floor ${floorMbps.toFixed(0)} Mbps` + : "no packaged media and no manifest — nothing to stream yet", + objects, + sceneFloorMbps: priced.length ? floorMbps : null + }, + ...pointcloud, + { + id: "vega", + name: "Vega", + page: "/web/vega.html", + adaptive: false, + ready: vega.objects.length > 0, + // Vega's page preloads whole clips, so per-object size is the number + // that decides what is worth opening: the full nine-object export is + // ~345 MB, which is over a minute of loading on a normal link. + detail: vega.objects.length + ? `${vega.objects.length} objects · ${(vega.bytes / 1e6).toFixed(0)} MB ` + + "total, preloaded per clip" + : `no export at ${VEGA_ASSETS_ROOT} ` + + "(produce it with baselines.Vega.orbitvega.export_quest)", + objects: vega.objects + }, + { + id: "nevo", + name: "NeVo", + page: "/web/nevo.html", + adaptive: false, + ready: dirHasAny(NEVO_ASSETS_ROOT, f => f.startsWith("g_")), + detail: dirHasAny(NEVO_ASSETS_ROOT, f => f.startsWith("g_")) + ? "pre-rendered comparison panels" + : `no renders at ${NEVO_ASSETS_ROOT} ` + + "(produce them with orbitnevo/render_frames.py)" + } + ] + }); +}); + +app.get("/api/viewpoint-index", (req, res) => { + const requested = String(req.query.dir || ""); + if (requested && !/^[A-Za-z0-9._-]+$/.test(requested)) { + return res.status(400).json({ error: "invalid viewpoint directory" }); + } + const root = path.join(SYSTEM_ROOT_CLIENT, "viewpoints"); + const directory = requested ? path.join(root, requested) : root; + if (!path.resolve(directory).startsWith(path.resolve(root))) { + return res.status(400).json({ error: "viewpoint path escapes the client dir" }); + } + if (!fs.existsSync(directory)) { + return res.status(404).json({ error: `no viewpoint directory ${directory}` }); + } + try { + const files = fs.readdirSync(directory) + .filter(f => f.startsWith("view_") && f.endsWith(".json")) + .sort(); + if (files.length === 0) { + return res.status(404).json({ error: "no view_*.json files found" }); + } + const viewpoints = files.map(filename => ({ + filename, + data: JSON.parse(fs.readFileSync(path.join(directory, filename), "utf8")) + })); + res.set("Cache-Control", "no-store"); + res.json({ schemaVersion: 1, count: viewpoints.length, viewpoints }); + } catch (error) { + res.status(409).json({ error: `could not read viewpoints: ${error.message}` }); + } +}); + +app.get("/api/config", (req, res) => { + res.json(STREAM_CONFIG); +}); + +app.get("/api/quest-launch", (req, res) => { + try { + const profile = questLaunchProfile(req.hostname); + const logKey = JSON.stringify(profile); + if (logKey !== lastQuestLaunchLogKey) { + lastQuestLaunchLogKey = logKey; + logInfo("QUEST", `Launch profile: pipeline=${profile.pipeline}` + + (profile.pipeline === "mesh" ? "" : ` mediaPort=${profile.baselinePort}`) + + ` prediction=${profile.viewportPredictionEnabled}` + + ` culling=${profile.orbitViewCullingEnabled}` + + ` scene=${profile.scene || "default"}` + + ` objects=${profile.sceneObjects.length}` + + ` trial=${profile.studyTrial || "manual"}` + + ` method=${profile.studyMethod || "manual"}` + + ` ablation=${profile.ablationVariant || "none"}` + + ` ready=${profile.studyReady}` + + ` source=${profile.source}`); + } + res.set("Cache-Control", "no-store"); + res.json(profile); + } catch (error) { + logError("QUEST", `Invalid launch profile: ${error.message}`); + res.status(500).json({ error: error.message }); + } +}); + +// Texture-decode knobs, read fresh on every request so editing the file takes +// effect on the next run with no server restart. +// +// These live here rather than only in the client because quest-client.json is a +// Unity Resource baked into the APK: changing it means a rebuild and a redeploy, +// and the headset is not the machine anyone is sitting at. The client applies +// whatever this returns just before it starts streaming, and logs what took +// effect. Omit a field, or delete the file, to leave the client's own value +// alone - note that 0 and -1 are meaningful values for these keys, so "absent" +// has to mean absent rather than zero. +const DECODE_TUNING_FILE = + process.env.VS4D_DECODE_TUNING || path.join(__dirname, "decode-tuning.json"); +// `benchmark` is a one-shot: non-zero makes the client sweep the decoder for +// capacity numbers and stop, instead of streaming. Left in the same file so a +// measurement run is selected the same way a tuning variant is. +const DECODE_TUNING_KEYS = [ + "maxVideoDecoders", "videoOperatingRate", "videoPriority", "benchmark", +]; + +app.get("/api/decode-tuning", (req, res) => { + if (!fs.existsSync(DECODE_TUNING_FILE)) { + logInfo("TUNING", `No ${path.basename(DECODE_TUNING_FILE)}; client keeps its own decode settings`); + return res.json({}); + } + try { + const raw = JSON.parse(fs.readFileSync(DECODE_TUNING_FILE, "utf8")); + // Whitelisted so a stray key cannot look like it was applied when the + // client would have ignored it. + const tuning = {}; + for (const key of DECODE_TUNING_KEYS) { + if (Number.isInteger(raw[key])) tuning[key] = raw[key]; + } + const ignored = Object.keys(raw).filter((k) => !(k in tuning)); + if (ignored.length > 0) { + logWarn("TUNING", `Ignored non-integer or unknown keys: ${ignored.join(", ")}`); + } + logInfo("TUNING", `Sent decode tuning: ${JSON.stringify(tuning)}`); + return res.json(tuning); + } catch (error) { + logError("TUNING", `Unreadable ${DECODE_TUNING_FILE}: ${error.message}`); + // Not fatal: a typo here must not stop a run, it just means the client + // keeps its baked settings. + return res.json({}); + } +}); + +app.put("/api/manifest", (req, res) => { + const segQ = req.query.seg; + const seg = + segQ !== undefined && segQ !== null && segQ !== "" + ? parseInt(segQ, 10) + : latestManifestSegId; + + const manifest = req.body; + const mp = ensureManifestDir(seg); + atomicWriteJson(mp, manifest); + + latestManifestSegId = seg; + manifestOwners.set(seg, currentBroadcastId || null); + + logInfo("MANIFEST", `Manifest updated via PUT: seg=${seg} path=${mp}`); + res.json({ status: "ok", message: "Manifest updated", seg, path: mp }); +}); + +app.post("/api/manifest", (req, res) => { + const segQ = req.query.seg; + const seg = + segQ !== undefined && segQ !== null && segQ !== "" + ? parseInt(segQ, 10) + : latestManifestSegId; + + const manifest = req.body; + const mp = ensureManifestDir(seg); + atomicWriteJson(mp, manifest); + + latestManifestSegId = seg; + manifestOwners.set(seg, currentBroadcastId || null); + + logInfo("MANIFEST", `Manifest updated via POST: seg=${seg} path=${mp}`); + res.json({ status: "ok", message: "Manifest updated", seg, path: mp }); +}); + +// Human-readable broadcast id: _YYYYMMDD-HHMMSS[_n]. +// The client may send { label, algorithm } in the body; label wins. +function makeBroadcastId(body) { + const raw = (body && (body.label || body.algorithm)) || "broadcast"; + const name = + String(raw).replace(/[^a-zA-Z0-9._-]+/g, "-").replace(/^-+|-+$/g, "") || + "broadcast"; + const d = new Date(); + const pad = (n) => String(n).padStart(2, "0"); + const stamp = + `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}` + + `-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`; + let id = `${name}_${stamp}`; + for (let n = 2; broadcasts.has(id); n++) id = `${name}_${stamp}_${n}`; + return id; +} + +app.post("/api/broadcast/start", (req, res) => { + const requestedObjects = Array.isArray(req.body?.sceneObjects) + ? req.body.sceneObjects.map(value => String(value).trim()).filter(Boolean) : []; + if (new Set(requestedObjects).size !== requestedObjects.length) + return res.status(400).json({ error: "sceneObjects must not contain duplicates" }); + const broadcastId = makeBroadcastId(req.body); + currentScene = String(req.body?.scene || "").trim(); + currentSceneObjects = requestedObjects; + currentSkybox = String(req.body?.skybox || "").trim(); + const studyTrial = String(req.body?.studyTrial || "").trim(); + const studyMethod = String(req.body?.studyMethod || "").trim(); + const participantId = String(req.body?.participantId || "").trim(); + if (participantId && !/^[a-zA-Z0-9._-]{1,64}$/.test(participantId)) + return res.status(400).json({ error: "participantId must be a pseudonymous label" }); + const studyTrialIndex = Number(req.body?.studyTrialIndex || 0); + const studyTotalTrials = Number(req.body?.studyTotalTrials || 0); + const plannedDurationSeconds = Number(req.body?.plannedDurationSeconds || 0); + const studyProtocolVersion = String(req.body?.studyProtocolVersion || "").trim(); + const clientConfiguration = privacySafeObject( + req.body?.clientConfiguration || {}); + if (studyProtocolVersion.startsWith("vs4d-mesh-ablation-")) { + const variant = String(clientConfiguration.ablationVariant || "").trim(); + const policies = { + full: [], + "no-gt-adaptation": ["geometryTextureAdaptationEnabled"], + "no-global-allocation": ["globalAllocationEnabled"], + "no-object-scheduling": ["objectSchedulingEnabled"], + "no-fast-switching": ["fastSwitchingEnabled"], + "no-frame-buffer": ["frameBufferEnabled"], + }; + const flags = [ + "geometryTextureAdaptationEnabled", "globalAllocationEnabled", + "objectSchedulingEnabled", "fastSwitchingEnabled", "frameBufferEnabled", + ]; + if (!Object.prototype.hasOwnProperty.call(policies, variant)) + return res.status(400).json({ + error: `ablation APK/config mismatch: unsupported variant '${variant}'`, + }); + const disabled = new Set(policies[variant]); + const mismatch = flags.filter(name => + typeof clientConfiguration[name] !== "boolean" + || clientConfiguration[name] !== !disabled.has(name)); + if (mismatch.length) + return res.status(400).json({ + error: "ablation APK/config mismatch; rebuild the Quest client: " + + mismatch.join(", "), + }); + } + const groundTruthBandwidthMode = req.body?.pipeline === "mesh" + && clientConfiguration.groundTruthBandwidthMode === true; + const configuredSafetyFactor = Number(clientConfiguration.bandwidthSafetyFactor); + const groundTruthBandwidthSafetyFactor = Number.isFinite(configuredSafetyFactor) + ? Math.max(0.25, Math.min(1, configuredSafetyFactor)) : 1; + broadcasts.set(broadcastId, { + id: broadcastId, startTime: Date.now(), scene: currentScene, + sceneObjects: [...currentSceneObjects], skybox: currentSkybox, + studyTrial, studyMethod, participantId, studyTrialIndex, studyTotalTrials, + groundTruthBandwidthMode, groundTruthBandwidthSafetyFactor, + }); + + // NEW: Track current broadcast for results + currentBroadcastId = broadcastId; + currentTraceStartTime = Date.now(); + // New trace = new network conditions; don't carry over the old estimate + latestClientBandwidthMbps = null; + manifestGeneration++; + manifestOwners.clear(); + latestManifestSegId = -1; + + const runDir = broadcastResultsDir(broadcastId); + fs.writeFileSync( + path.join(runDir, "run.json"), + JSON.stringify( + { + broadcastId, + startedAt: new Date().toISOString(), + label: req.body?.label || null, + algorithm: req.body?.algorithm || null, + pipeline: req.body?.pipeline || "mesh", + baselinePort: req.body?.baselinePort || null, + datasetManifest: req.body?.datasetManifest || null, + scene: currentScene || null, + sceneObjects: currentSceneObjects, + skybox: currentSkybox || null, + studyTrial: studyTrial || null, + studyMethod: studyMethod || null, + participantId: participantId || null, + studyTrialIndex: Number.isInteger(studyTrialIndex) && studyTrialIndex > 0 + ? studyTrialIndex : null, + studyTotalTrials: Number.isInteger(studyTotalTrials) && studyTotalTrials > 0 + ? studyTotalTrials : null, + studyProtocolVersion: studyProtocolVersion || null, + ablationVariant: clientConfiguration.ablationVariant || null, + plannedDurationSeconds: Number.isFinite(plannedDurationSeconds) + && plannedDurationSeconds > 0 ? plannedDurationSeconds : null, + streamConfig: STREAM_CONFIG, + serverRuntime: { + nodeVersion: process.version, + platform: process.platform, + architecture: process.arch, + }, + studyEnvironment: studySafeEnvironment(), + client: privacySafeObject(req.body?.client || {}), + clientConfiguration, + privacy: { + pseudonymousParticipantOnly: true, + storesRawIpAddresses: false, + storesControllerTrajectories: false, + storesNamesOrBirthDates: false, + headPoseStoredForSelectionReplay: true, + }, + }, + null, + 2 + ) + ); + + logInfo("BROADCAST", `Broadcast started: ${broadcastId}` + + ` scene=${currentScene || "default"}` + + ` objects=${currentSceneObjects.length}` + + ` skybox=${currentSkybox || "client-default"}`); + res.json({ broadcastId }); +}); + +// The trace shaper is the authority for the currently installed TBF rate. In +// oracle experiments it publishes every rate change here; the Quest reads it +// immediately before an MCKP decision. Normal runs retain the value only as +// diagnostic provenance and continue using their causal estimator. +app.post("/api/ground-truth-bandwidth", (req, res) => { + const broadcastId = String(req.body?.broadcastId || "").trim(); + const bandwidthMbps = Number(req.body?.bandwidthMbps); + const elapsedSeconds = Number(req.body?.elapsedSeconds); + if (!broadcastId || broadcastId !== currentBroadcastId + || !broadcasts.has(broadcastId)) + return res.status(409).json({ error: "ground-truth bandwidth broadcast mismatch" }); + if (!Number.isFinite(bandwidthMbps) || bandwidthMbps <= 0 + || !Number.isFinite(elapsedSeconds) || elapsedSeconds < 0) + return res.status(400).json({ error: "invalid ground-truth bandwidth sample" }); + const sample = { + broadcastId, bandwidthMbps, elapsedSeconds, + serverReceivedAt: new Date().toISOString(), + }; + groundTruthBandwidthByBroadcast.set(broadcastId, sample); + const broadcast = broadcasts.get(broadcastId); + if (broadcast.groundTruthBandwidthMode) + latestClientBandwidthMbps = bandwidthMbps + * broadcast.groundTruthBandwidthSafetyFactor; + fs.appendFileSync( + path.join(broadcastResultsDir(broadcastId), "ground_truth_bandwidth.jsonl"), + JSON.stringify(sample) + "\n"); + res.json({ status: "ok" }); +}); + +app.get("/api/ground-truth-bandwidth/:broadcastId", (req, res) => { + const broadcastId = String(req.params.broadcastId || "").trim(); + if (!broadcasts.has(broadcastId)) + return res.status(404).json({ error: "Broadcast not found" }); + const sample = groundTruthBandwidthByBroadcast.get(broadcastId); + if (!sample) + return res.status(503).json({ error: "ground-truth bandwidth is not available yet" }); + res.set("Cache-Control", "no-store"); + res.json(sample); +}); + +// Baseline media bypasses the ladder but keeps this HTTP control plane for +// provenance and telemetry. The TCP header supplies the calibration hash an +// exact replay must match. +app.post("/api/baseline/session", (req, res) => { + const body = req.body || {}; + if (!broadcasts.has(body.broadcastId)) + return res.status(404).json({ error: "Broadcast not found" }); + const filepath = path.join(broadcastResultsDir(body.broadcastId), "baseline_session.json"); + const safeBody = privacySafeObject(body); + fs.writeFileSync(filepath, JSON.stringify({ + ...safeBody, + serverReceivedAt: new Date().toISOString(), + }, null, 2)); + const runPath = path.join(broadcastResultsDir(body.broadcastId), "run.json"); + try { + const run = JSON.parse(fs.readFileSync(runPath, "utf8")); + run.baseline = safeBody; + atomicWriteJson(runPath, run); + } catch (error) { + logWarn("BASELINE", `Could not merge session metadata into run.json: ${error.message}`); + } + logInfo("BASELINE", `Session ${body.broadcastId}: pipeline=${body.pipeline}` + + ` streams=${body.streamCount} calibration=${body.calibrationHash}`); + res.json({ status: "ok", saved: filepath }); +}); + +app.post("/api/baseline/frames", (req, res) => { + const { broadcastId, frames } = req.body || {}; + if (!broadcasts.has(broadcastId)) + return res.status(404).json({ error: "Broadcast not found" }); + if (!Array.isArray(frames) || frames.length > 1000) + return res.status(400).json({ error: "expected a bounded frames array" }); + const filepath = path.join(broadcastResultsDir(broadcastId), "baseline_frames.jsonl"); + if (frames.length) + fs.appendFileSync(filepath, frames.map(frame => JSON.stringify(frame)).join("\n") + "\n"); + res.json({ status: "ok", count: frames.length }); +}); + +app.post("/api/viewpoint", async (req, res) => { + const { broadcastId, viewpoint, segId } = req.body; + + if (!broadcasts.has(broadcastId)) { + return res.status(404).json({ error: "Broadcast not found" }); + } + + const s = Number.isFinite(segId) ? segId : 0; + const filename = `segment_${String(s).padStart(4, "0")}.json`; + const filepath = path.join(SERVER_VIEWPOINTS_DIR, filename); + + fs.writeFileSync(filepath, JSON.stringify(viewpoint, null, 2)); + fs.writeFileSync( + path.join(broadcastArtifactDir(broadcastId, "viewpoints"), filename), + JSON.stringify(viewpoint, null, 2) + ); + logInfo("VIEWPOINT", `Viewpoint received: seg=${s} file=${filename}`); + + // Always bootstrap this broadcast from its real segment-0 viewpoint. Await + // completion so the client's immediately-following manifest GET cannot race + // and consume a stale, uniformly weighted menu from an earlier/default run. + if (latestManifestSegId < 0) { + logInfo("LADDER", "Bootstrapping seg=0 ladder from the client viewpoint"); + // This path labels the solve with the same id as the pose it just wrote. + await requestLadderUpdate(s, broadcastId, "viewpoint bootstrap", true, s); + } + return res.json({ status: "ok", segId: s, filename }); +}); + +app.post("/api/segment/:id", async (req, res) => { + const segId = parseInt(req.params.id, 10); + const { broadcastId, viewpoint, selection } = req.body; + if (!broadcasts.has(broadcastId)) { + logWarn("SEGMENT", `Rejected seg=${segId}: unknown broadcast ${broadcastId}`); + return res.status(404).json({ error: "Broadcast not found" }); + } + + // The ladder plans its published floor against what the client will actually + // commit (vstream/config.py CLIENT_BUDGET_MULTIPLIER_STRUGGLING), so a client + // that keeps its own headroom must report that budget or the floor lands + // above what it will spend and it freezes objects. The JS client applies a + // multiplier of 1 and reports only the raw estimate, so the fallback is + // unchanged for it. + const estBW = req.body.bandwidthBudget + ?? req.body.estimatedBandwidth ?? selection?.estimatedBandwidth; + if (typeof estBW === "number" && estBW > 0) { + latestClientBandwidthMbps = estBW; + } + + const received = []; + if (viewpoint) { + const filename = `segment_${String(segId).padStart(4, "0")}.json`; + const viewpointFile = path.join(SERVER_VIEWPOINTS_DIR, filename); + fs.writeFileSync(viewpointFile, JSON.stringify(viewpoint, null, 2)); + fs.writeFileSync( + path.join(broadcastArtifactDir(broadcastId, "viewpoints"), filename), + JSON.stringify(viewpoint, null, 2) + ); + received.push("viewpoint"); + } + + if (selection) { + selectionLog.push(selection); + const selectionFile = path.join( + SERVER_SELECTIONS_DIR, + `selection_${String(segId).padStart(4, "0")}.json` + ); + fs.writeFileSync(selectionFile, JSON.stringify(selection, null, 2)); + fs.writeFileSync( + path.join( + broadcastArtifactDir(broadcastId, "selections"), + `selection_${String(segId).padStart(4, "0")}.json` + ), + JSON.stringify(selection, null, 2) + ); + received.push("selection"); + } + + const stats = []; + if (selection) { + if (typeof selection.minBufferLevel === "number") stats.push(`buffer=${selection.minBufferLevel.toFixed(1)}s`); + if (typeof selection.totalBitrate === "number") stats.push(`bitrate=${selection.totalBitrate.toFixed(1)}Mbps`); + if (typeof selection.totalQuality === "number") stats.push(`quality=${selection.totalQuality.toFixed(1)}`); + if (typeof selection.estimatedBandwidth === "number") stats.push(`estBW=${selection.estimatedBandwidth.toFixed(1)}Mbps`); + if (typeof selection.segmentStallDurationSec === "number" && selection.segmentStallDurationSec > 0) { + stats.push(`stall=${selection.segmentStallDurationSec.toFixed(2)}s`); + } + if (typeof selection.missingCount === "number" && selection.missingCount > 0) { + stats.push(`missing=${selection.missingCount}[${(selection.missingObjects || []).join(",")}]`); + } + if (typeof selection.frozenCount === "number" && selection.frozenCount > 0) { + stats.push(`frozen=${selection.frozenCount}[${(selection.frozenObjects || []).join(",")}]`); + } + if (selection.clientOverloadSkip) stats.push("client-overload-skip"); + } + logInfo( + "SEGMENT", + `seg=${segId} received=[${received.join(",") || "none"}]${stats.length ? " " + stats.join(" ") : ""}` + ); + // Solve for the segment the client will fetch NEXT, not the one it just + // reported. A client asks for the manifest before it reports, so a solve + // labelled with the reported id is always published too late for that + // segment and gets picked up one segment later. Mid stream that was + // invisible - every fetch still got a manifest one content step on - but + // segment 0 and segment 1 both landed on t_00, because report(0) was + // skipped entirely and there was nothing newer to fetch. The client played + // the first segment's media twice before moving on. + // + // The pose is unaffected: last_seg_id tracks latestManifestSegId rather + // than segId, and the viewpoint read is the one this report just wrote, so + // a manifest is still solved one segment ahead of the frame it weights. + // Only the content step it is labelled with changes, which is what makes + // logical segment N line up with t_N. + const needUpdate = Number.isFinite(segId) && segId >= 0 + && segId % UPDATE_INTERVAL_SEGMENTS === 0 && Boolean(viewpoint); + // Solve segId+1, weighted by the pose written for segId just above - the one + // this report carried, which is the client's predicted pose at its + // viewportPredictionWindowSec lead. + if (needUpdate) { + requestLadderUpdate(segId + 1, broadcastId, "update", false, segId) + .catch(()=>{}); + } + + // latestManifestSegId lets the client log manifest staleness (telemetry only) + res.json({ segId, status: "ok", latestManifestSegId }); +}); + +// Download-completion reports from the client (bandwidth/buffer telemetry). +// Appended to a JSONL so QoE analysis can correlate with selections. +app.post("/api/segment/:id/download-complete", (req, res) => { + const segId = parseInt(req.params.id, 10); + const record = { + receivedAt: new Date().toISOString(), + segId, + ...req.body, + }; + const downloadsFile = path.join(broadcastResultsDir(), "download_complete.jsonl"); + fs.appendFileSync(downloadsFile, JSON.stringify(record) + "\n"); + + const parts = []; + if (typeof req.body.downloadTimeMs === "number") parts.push(`time=${req.body.downloadTimeMs}ms`); + if (typeof req.body.downloadSizeBytes === "number") parts.push(`size=${(req.body.downloadSizeBytes / 1024 / 1024).toFixed(2)}MB`); + if (typeof req.body.measuredBandwidthMbps === "number") parts.push(`bw=${req.body.measuredBandwidthMbps.toFixed(1)}Mbps`); + if (typeof req.body.prepareTimeMs === "number") parts.push(`prepare=${req.body.prepareTimeMs}ms`); + if (typeof req.body.stagedObjects === "number" && typeof req.body.requestedObjects === "number") { + parts.push(`staged=${req.body.stagedObjects}/${req.body.requestedObjects}`); + } + if (typeof req.body.geometryCacheHits === "number" + && typeof req.body.geometryCacheRequested === "number") { + const used = typeof req.body.geometryCacheBytes === "number" + ? ` used=${(req.body.geometryCacheBytes / 1e9).toFixed(2)}GB` : ""; + parts.push(`geometry-cache=${req.body.geometryCacheHits}/${req.body.geometryCacheRequested}${used}`); + if (Array.isArray(req.body.geometryCacheUncached) + && req.body.geometryCacheUncached.length > 0) + parts.push(`uncached=[${req.body.geometryCacheUncached.join(",")}]`); + } + if (req.body.isLate) parts.push("late"); + if (req.body.usedFallback) parts.push("fallback"); + logInfo("DOWNLOAD", `seg=${segId} complete${parts.length ? " " + parts.join(" ") : ""}`); + + res.json({ status: "ok", segId }); +}); + +// Per-presented-frame truth from the interactive renderer. The client batches +// records to avoid a request per frame; JSONL keeps long runs streamable by the +// full-frame evaluator. +app.post("/api/render-frames", (req, res) => { + const { broadcastId, frames } = req.body || {}; + if (!broadcasts.has(broadcastId)) { + return res.status(404).json({ error: "Broadcast not found" }); + } + if (!Array.isArray(frames) || frames.length === 0) { + return res.status(400).json({ error: "frames must be a non-empty array" }); + } + const renderFile = path.join(broadcastResultsDir(broadcastId), "render_frames.jsonl"); + fs.appendFileSync(renderFile, frames.map((frame) => JSON.stringify(frame)).join("\n") + "\n"); + const summary = summarizeRenderFrames(frames); + if (summary.stuck) logWarn("RENDER", summary.line); + else logInfo("RENDER", summary.line); + res.json({ status: "ok", count: frames.length, saved: renderFile }); +}); + +// Matured actual/predicted 6DoF pairs from the Quest. CSV is kept as the +// canonical artifact so the same offline plotter and server-side plotter score +// exactly the same samples. +app.post("/api/viewport-trace", (req, res) => { + const { broadcastId, header, rows } = req.body || {}; + if (!broadcasts.has(broadcastId)) + return res.status(404).json({ error: "Broadcast not found" }); + if (typeof header !== "string" || !header.startsWith("t_s,") + || !Array.isArray(rows) || rows.length === 0 || rows.length > 1000 + || rows.some(row => typeof row !== "string" || row.length > 4096 + || row.includes("\n") || row.includes("\r"))) { + return res.status(400).json({ error: "invalid bounded viewport CSV batch" }); + } + const filepath = viewportTracePath(broadcastId); + if (!fs.existsSync(filepath)) { + fs.writeFileSync(filepath, header + "\n"); + } else { + const existingHeader = fs.readFileSync(filepath, "utf8").split(/\r?\n/, 1)[0]; + if (existingHeader !== header) + return res.status(409).json({ error: "viewport trace header changed during run" }); + } + fs.appendFileSync(filepath, rows.join("\n") + "\n"); + scheduleViewportPlots(broadcastId); + logInfo("VIEWPORT", `Trace batch: run=${broadcastId} rows=${rows.length}`); + res.json({ + status: "ok", + count: rows.length, + saved: filepath, + evaluationUrl: `/viewport-evaluation/${broadcastId}`, + }); +}); + +function offlineTrajectorySamples(csvText, expectedBroadcastId = "") { + const lines = String(csvText || "").trim().split(/\r?\n/); + const header = (lines.shift() || "").split(","); + const required = ["playback_s", "x", "y", "z", "yaw", "pitch", "roll"]; + const columns = Object.fromEntries(required.map(name => [name, header.indexOf(name)])); + const broadcastColumn = header.indexOf("broadcast_id"); + if (required.some(name => columns[name] < 0)) { + const error = new Error("trajectory predates offline playback-aligned pose logging"); + error.statusCode = 409; + throw error; + } + + // A stale upload batch from an earlier run used to be posted under the next + // run's HTTP path. Keep the CSV as evidence, but select only rows whose own + // embedded broadcast id matches the capture named by source.json. + const epochs = [[]]; + let sampleCount = 0; + for (const line of lines) { + if (!line) continue; + const values = line.split(","); + if (expectedBroadcastId && broadcastColumn >= 0 + && values[broadcastColumn] !== expectedBroadcastId) continue; + const raw = required.map(name => values[columns[name]]); + // Number("") is zero in JavaScript. Pre-playback rows intentionally leave + // playback_s blank, so reject blanks before numeric conversion. + if (raw.some(value => value === undefined || value.trim() === "")) continue; + const sample = raw.map(Number); + if (!sample.every(Number.isFinite) || sample[0] < 0) continue; + let epoch = epochs[epochs.length - 1]; + if (epoch.length && sample[0] < epoch[epoch.length - 1][0]) { + epoch = []; + epochs.push(epoch); + } + epoch.push(sample); + if (++sampleCount > 10000) { + const error = new Error("offline trajectory exceeds 10000 samples"); + error.statusCode = 413; + throw error; + } + } + // If playback restarted within one broadcast, use the epoch with the greatest + // time coverage. Sorting all rows would splice physically unrelated paths. + const samples = epochs.reduce((best, value) => { + const duration = value.length > 1 ? value[value.length - 1][0] - value[0][0] : 0; + const bestDuration = best.length > 1 ? best[best.length - 1][0] - best[0][0] : 0; + return duration > bestDuration || (duration === bestDuration && value.length > best.length) + ? value : best; + }, []); + if (samples.length < 2) { + const error = new Error("offline trajectory has fewer than two playback samples"); + error.statusCode = 409; + throw error; + } + return samples; +} + +function offlineTrajectoryPayload(csvText, preferredBroadcastId = "") { + try { + return { + sourceBroadcastId: preferredBroadcastId, + samples: offlineTrajectorySamples(csvText, preferredBroadcastId), + }; + } catch (error) { + if (!preferredBroadcastId + || error.message !== "offline trajectory has fewer than two playback samples") throw error; + // Resume metadata from the first implementation used the placeholder + // "archived". Recover the actual source id from the rows themselves and + // choose the id with the most playback-aligned samples. + const lines = String(csvText || "").trim().split(/\r?\n/); + const header = (lines.shift() || "").split(","); + const broadcastColumn = header.indexOf("broadcast_id"); + const playbackColumn = header.indexOf("playback_s"); + if (broadcastColumn < 0 || playbackColumn < 0) throw error; + const counts = new Map(); + for (let lineIndex = 0; lineIndex < lines.length; lineIndex++) { + const line = lines[lineIndex]; + const values = line.split(","); + const id = values[broadcastColumn]; + const playback = values[playbackColumn]; + if (!id || !playback || !Number.isFinite(Number(playback))) continue; + const previous = counts.get(id) || { count: 0, last: -1 }; + counts.set(id, { count: previous.count + 1, last: lineIndex }); + } + const recovered = [...counts.entries()].sort( + (left, right) => right[1].count - left[1].count || right[1].last - left[1].last + )[0]?.[0]; + if (!recovered || recovered === preferredBroadcastId) throw error; + return { + sourceBroadcastId: recovered, + samples: offlineTrajectorySamples(csvText, recovered), + }; + } +} + +function sceneRelativeTrajectoryPayload(value, preferredBroadcastId = "") { + if (!value || value.schemaVersion !== 2 + || value.coordinateSpace !== "open3d-camera-extrinsic-column-major") { + const error = new Error("unsupported scene-relative offline trajectory"); + error.statusCode = 409; + throw error; + } + const samples = Array.isArray(value.samples) ? value.samples : []; + if (samples.length < 2 || samples.length > 10000) { + const error = new Error("scene-relative trajectory needs 2-10000 samples"); + error.statusCode = 409; + throw error; + } + let previous = -Infinity; + for (const sample of samples) { + if (!Array.isArray(sample) || sample.length !== 17 + || !sample.every(Number.isFinite) || sample[0] < 0 || sample[0] < previous) { + const error = new Error("invalid scene-relative trajectory sample"); + error.statusCode = 409; + throw error; + } + previous = sample[0]; + } + return { + schemaVersion: 2, + coordinateSpace: value.coordinateSpace, + sourceBroadcastId: String(value.sourceBroadcastId || preferredBroadcastId), + samples, + }; +} + +// A fair offline comparison records one physical head trajectory and replays +// it for the remaining methods in the same scene/network block. Retained +// cross-session runs carry scene-relative Open3D matrices in trajectory.json; +// older in-session runs fall back to playback-aligned Unity poses in the CSV. +app.get("/api/offline-trajectory/:broadcastId", (req, res) => { + const broadcastId = String(req.params.broadcastId || ""); + if (!/^[A-Za-z0-9._-]+$/.test(broadcastId)) + return res.status(400).json({ error: "invalid trajectory broadcast id" }); + const filepath = path.join(SERVER_RESULTS_DIR, broadcastId, "viewport_trace.csv"); + if (!fs.existsSync(filepath)) + return res.status(404).json({ error: "offline trajectory not found" }); + let sourceBroadcastId = broadcastId; + const sourceFile = path.join(SERVER_RESULTS_DIR, broadcastId, "source.json"); + if (fs.existsSync(sourceFile)) { + try { + sourceBroadcastId = String( + JSON.parse(fs.readFileSync(sourceFile, "utf8")).sourceBroadcastId || broadcastId); + } catch (error) { + return res.status(409).json({ error: `invalid trajectory source metadata: ${error.message}` }); + } + } + try { + const sceneRelativeFile = path.join( + SERVER_RESULTS_DIR, broadcastId, "trajectory.json"); + if (fs.existsSync(sceneRelativeFile)) { + const payload = sceneRelativeTrajectoryPayload( + JSON.parse(fs.readFileSync(sceneRelativeFile, "utf8")), sourceBroadcastId); + res.set("Cache-Control", "no-store"); + return res.json(payload); + } + const payload = offlineTrajectoryPayload( + fs.readFileSync(filepath, "utf8"), sourceBroadcastId); + res.set("Cache-Control", "no-store"); + res.json({ schemaVersion: 1, ...payload }); + } catch (error) { + res.status(error.statusCode || 409).json({ error: error.message }); + } +}); + +// Condenses one uploaded batch into the per-object playback truth. +// +// `State` alone is not enough. In the textured path the presented frame is the +// video decoder's last delivered frame, so an object keeps reporting "ok" while +// SourceFrame never moves - a freeze that is invisible to both the state field +// and the client's buffer accounting. Counting how often SourceFrame changes is +// what separates "playing" from "holding one frame". +function summarizeRenderFrames(frames) { + const objects = new Map(); + for (const frame of frames) { + const presented = frame.objects || frame.Objects || {}; + for (const [name, value] of Object.entries(presented)) { + if (!objects.has(name)) { + objects.set(name, { + states: new Map(), advanced: 0, steps: 0, last: null, segment: null, + playback: null, parts: new Set(), + }); + } + const entry = objects.get(name); + const state = String(value?.State ?? value?.state ?? "unknown"); + const reason = value?.FreezeReason ?? value?.freezeReason ?? null; + const key = state === "ok" ? "ok" : reason ? `${state}:${reason}` : state; + entry.states.set(key, (entry.states.get(key) || 0) + 1); + entry.segment = value?.MediaSegmentId ?? value?.mediaSegmentId ?? entry.segment; + const playback = value?.PlaybackFrame ?? value?.playbackFrame ?? null; + if (playback !== null && playback !== undefined) entry.playback = playback; + const part = value?.Part ?? value?.part ?? null; + if (part !== null && part !== undefined && state !== "ok") entry.parts.add(part); + const source = value?.AbsoluteSourceFrame ?? value?.absoluteSourceFrame + ?? value?.SourceFrame ?? value?.sourceFrame ?? null; + if (source === null || source === undefined) continue; + if (entry.last !== null) { + entry.steps++; + if (source !== entry.last) entry.advanced++; + } + entry.last = source; + } + } + const trouble = []; + let healthy = 0; + for (const [name, entry] of objects) { + const allOk = entry.states.size === 1 && entry.states.has("ok"); + const advancing = entry.steps === 0 || entry.advanced > 0; + if (allOk && advancing) { + healthy++; + continue; + } + const states = [...entry.states.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([key, count]) => `${key}=${count}`) + .join(","); + // `part` is what a stalled object is waiting on, and it is derived from the + // playback clock rather than the delivered frame. Reading the stall location + // off SourceFrame instead points at the wrong temporal part entirely. + const parts = [...entry.parts].sort().join("/"); + trouble.push( + `${name}[${states} advanced=${entry.advanced}/${entry.steps}` + + ` seg=${entry.segment} frame=${entry.last}` + + (entry.playback === null ? "" : ` clock=${entry.playback}`) + + (parts ? ` part=${parts}` : "") + + `]` + ); + } + const line = + `frames=${frames.length} playing=${healthy}/${objects.size}` + + (trouble.length ? ` ${trouble.join(" ")}` : ""); + return { line, stuck: trouble.length > 0 }; +} + +// The headset has one USB-C port and it carries ethernet during a run, so +// `adb logcat` is unavailable while streaming. The client relays its own Unity +// log here instead; this is the only view of renderer-side diagnostics while a +// run is live. Deliberately does not require a known broadcast: startup and +// warm-up failures happen before one exists. +app.post("/api/client-log", (req, res) => { + const { broadcastId, entries, lost } = req.body || {}; + if (!Array.isArray(entries) || entries.length === 0) { + return res.status(400).json({ error: "entries must be a non-empty array" }); + } + const safeEntries = entries.map(entry => ({ + ...entry, + message: redactIpAddresses(entry.message), + stack: entry.stack ? redactIpAddresses(entry.stack) : entry.stack, + })); + const logFile = path.join(broadcastResultsDir(broadcastId), "client_log.jsonl"); + fs.appendFileSync(logFile, safeEntries.map((entry) => JSON.stringify(entry)).join("\n") + "\n"); + for (const entry of safeEntries) { + const level = String(entry.level || "INFO").toUpperCase(); + const message = String(entry.message || "").replace(/\s+/g, " ").trim(); + const stack = entry.stack ? ` | ${String(entry.stack).replace(/\s+/g, " ").trim()}` : ""; + log(level === "ERROR" || level === "WARN" ? level : "INFO", "CLIENT", message + stack); + } + if (lost > 0) logWarn("CLIENT", `${lost} client log entries were dropped before upload`); + res.json({ status: "ok", count: safeEntries.length }); +}); + +// ==================== NEW ENDPOINTS ==================== + +// NEW: Receive 10-second interval bitrate counts per object +app.post("/api/bitrate-counts-interval", (req, res) => { + if (!req.body || typeof req.body !== "object") { + return res.status(400).json({ error: "expected JSON body" }); + } + const { segmentId, timestamp, intervalMs, countsPerObject } = req.body; + + const data = { + receivedAt: new Date().toISOString(), + broadcastId: currentBroadcastId, + segmentId, + timestamp, + intervalMs, + countsPerObject + }; + // Append to JSONL file (one JSON object per line) + const intervalsFile = intervalsFilePath(); + fs.appendFileSync(intervalsFile, JSON.stringify(data) + "\n"); + + // Log summary + const objectCount = Object.keys(countsPerObject || {}).length; + const totalSelections = Object.values(countsPerObject || {}).reduce( + (sum, obj) => sum + Object.values(obj).reduce((s, c) => s + c, 0), 0 + ); + + logInfo( + "BITRATE", + `Interval received: objects=${objectCount} selections=${totalSelections} t=${(timestamp / 1000).toFixed(1)}s` + ); + + res.json({ status: "ok", saved: intervalsFile }); +}); + +// NEW: Receive final bitrate counts per object (end of trace) +app.post("/api/bitrate-counts-final", (req, res) => { + if (!req.body || typeof req.body !== "object") { + logWarn("BITRATE", "Rejected final counts: body is not JSON (missing Content-Type: application/json?)"); + return res.status(400).json({ error: "expected JSON body" }); + } + const { totalSegments, bitrateRequestCounts, bitrateRequestCountsPerObject } = req.body; + + const filepath = path.join(broadcastResultsDir(), "bitrate_final.json"); + + const data = { + savedAt: new Date().toISOString(), + broadcastId: currentBroadcastId, + totalSegments, + bitrateRequestCounts, + bitrateRequestCountsPerObject + }; + + fs.writeFileSync(filepath, JSON.stringify(data, null, 2)); + + // Log per-object summary + logInfo("BITRATE", `Final counts received: totalSegments=${totalSegments} saved=${filepath}`); + if (bitrateRequestCountsPerObject) { + for (const [objName, counts] of Object.entries(bitrateRequestCountsPerObject)) { + const sorted = Object.entries(counts).sort((a, b) => b[1] - a[1]); + const top3 = sorted.slice(0, 3).map(([rep, cnt]) => `${rep}:${cnt}`).join(", "); + logInfo("BITRATE", ` ${objName}: ${top3}`); + } + } + + res.json({ status: "ok", saved: filepath }); +}); + +// NEW: Receive complete metrics JSON (end of trace) +app.post("/api/results", (req, res) => { + const metrics = req.body; + if (!metrics || typeof metrics !== "object") { + logWarn("RESULTS", "Rejected metrics upload: body is not JSON (missing Content-Type: application/json?)"); + return res.status(400).json({ error: "expected JSON body" }); + } + + const resultBroadcastId = + broadcasts.has(metrics.broadcastId) ? metrics.broadcastId : currentBroadcastId; + const filepath = path.join(broadcastResultsDir(resultBroadcastId), "metrics.json"); + + // Add server-side metadata + metrics.serverReceivedAt = new Date().toISOString(); + metrics.serverBroadcastId = resultBroadcastId; + try { + metrics.runMetadata = JSON.parse(fs.readFileSync( + path.join(broadcastResultsDir(resultBroadcastId), "run.json"), "utf8")); + } catch (_) { /* Older/non-broadcast uploads have no run metadata. */ } + + fs.writeFileSync(filepath, JSON.stringify(metrics, null, 2)); + + // Log summary + const summary = metrics.summary || {}; + logInfo( + "RESULTS", + `Metrics received: segments=${summary.totalSegments || "N/A"} ` + + `rebuffers=${summary.rebuffers || 0} ` + + `stall=${((summary.totalStallDuration || 0) / 1000).toFixed(2)}s saved=${filepath}` + ); + + res.json({ status: "ok", saved: filepath }); +}); + +const QUESTIONNAIRE_RATINGS = [ + "visualQuality", "geometryDepthFidelity", "temporalSmoothness", + "overallQualityOfExperience", +]; +const QUESTIONNAIRE_ARTIFACTS = new Set([ + "surface_geometry_degradation", "missing_parts", "quality_flicker", "freezing", + "mixed_quality_parts", "client_low_fps", "view_dependent_degradation", "none", +]); +const QUESTIONNAIRE_MAX_ARTIFACTS = 2; + +app.post("/api/questionnaire", (req, res) => { + const body = req.body || {}; + const broadcastId = String(body.broadcastId || ""); + const broadcast = broadcasts.get(broadcastId); + if (!broadcast) + return res.status(404).json({ error: "Broadcast not found" }); + const ratings = body.ratings; + if (!ratings || typeof ratings !== "object") + return res.status(400).json({ error: "ratings are required" }); + for (const key of QUESTIONNAIRE_RATINGS) { + if (!Number.isInteger(ratings[key]) || ratings[key] < 1 || ratings[key] > 5) + return res.status(400).json({ error: `${key} must be an integer from 1 through 5` }); + } + if (!Array.isArray(body.artifacts) || body.artifacts.length < 1) + return res.status(400).json({ error: "select artifacts or none" }); + const artifacts = [...new Set(body.artifacts.map(value => String(value)))]; + const invalid = artifacts.filter(value => !QUESTIONNAIRE_ARTIFACTS.has(value)); + if (invalid.length) + return res.status(400).json({ error: `unknown artifact(s): ${invalid.join(", ")}` }); + if (artifacts.includes("none") && artifacts.length !== 1) + return res.status(400).json({ error: "none cannot be combined with artifacts" }); + if (!artifacts.includes("none") && artifacts.length > QUESTIONNAIRE_MAX_ARTIFACTS) + return res.status(400).json({ + error: `select at most ${QUESTIONNAIRE_MAX_ARTIFACTS} most severe artifacts`, + }); + + const payload = { + broadcastId, + participantId: broadcast.participantId || null, + trialIndex: broadcast.studyTrialIndex || null, + totalTrials: broadcast.studyTotalTrials || null, + studyTrial: broadcast.studyTrial || null, + method: broadcast.studyMethod || null, + scene: broadcast.scene || null, + ratings: Object.fromEntries(QUESTIONNAIRE_RATINGS.map(key => [key, ratings[key]])), + artifacts, + artifactSelectionLimit: QUESTIONNAIRE_MAX_ARTIFACTS, + responseTiming: privacySafeObject(body.responseTiming || {}), + clientSubmittedAt: body.clientSubmittedAt || null, + serverReceivedAt: new Date().toISOString(), + }; + const filepath = path.join(broadcastResultsDir(broadcastId), "questionnaire.json"); + atomicWriteJson(filepath, payload); + logInfo("QUESTIONNAIRE", `Saved ${broadcastId}: method=${payload.method || "-"}` + + ` scene=${payload.scene || "-"} ratings=` + + QUESTIONNAIRE_RATINGS.map(key => `${key}=${payload.ratings[key]}`).join(",") + + ` artifacts=${artifacts.join(",")}`); + res.json({ status: "ok", saved: filepath }); +}); + +// NEW: Receive QoE results (from shell script) +// The QoE report is plain text (calculate-qoe.js output), uploaded as the raw +// body with trace/algorithm passed as query params. +app.post("/api/qoe", express.text({ type: "*/*", limit: "10mb" }), (req, res) => { + if (typeof req.body !== "string" || req.body.length === 0) { + logWarn("QOE", "Rejected QoE upload: empty or non-text body"); + return res.status(400).json({ error: "expected non-empty text body" }); + } + + const sanitize = (s) => String(s).replace(/[^a-zA-Z0-9._-]+/g, "-"); + const parts = ["qoe"]; + if (req.query.trace) parts.push(sanitize(req.query.trace)); + if (req.query.algorithm) parts.push(sanitize(req.query.algorithm)); + const filepath = path.join(broadcastResultsDir(), `${parts.join("_")}.txt`); + + fs.writeFileSync(filepath, req.body); + + logInfo("QOE", `QoE results saved: ${filepath}`); + + res.json({ status: "ok", saved: filepath }); +}); + +// NEW: Get all results (for debugging/dashboard) +// Results live one level down, in server_results//; loose files +// in the root (from older runs) are still listed with broadcast=null. +app.get("/api/results", (req, res) => { + const files = []; + for (const entry of fs.readdirSync(SERVER_RESULTS_DIR, { withFileTypes: true })) { + const entryPath = path.join(SERVER_RESULTS_DIR, entry.name); + if (entry.isDirectory()) { + for (const f of fs.readdirSync(entryPath)) { + const p = path.join(entryPath, f); + const st = fs.statSync(p); + if (st.isFile()) { + files.push({ broadcast: entry.name, filename: f, path: p, size: st.size, modified: st.mtime }); + } + } + } else if (entry.isFile()) { + const st = fs.statSync(entryPath); + files.push({ broadcast: null, filename: entry.name, path: entryPath, size: st.size, modified: st.mtime }); + } + } + files.sort((a, b) => b.modified - a.modified); + + res.json({ + resultsDir: SERVER_RESULTS_DIR, + fileCount: files.length, + files + }); +}); + +function validViewportBroadcastId(value) { + return typeof value === "string" && /^[a-zA-Z0-9._-]+$/.test(value) + && fs.existsSync(path.join(SERVER_RESULTS_DIR, value)); +} + +function latestViewportBroadcastId() { + if (validViewportBroadcastId(currentBroadcastId)) return currentBroadcastId; + return fs.readdirSync(SERVER_RESULTS_DIR, { withFileTypes: true }) + .filter(entry => entry.isDirectory()) + .map(entry => ({ + id: entry.name, + path: path.join(SERVER_RESULTS_DIR, entry.name, "viewport_trace.csv"), + })) + .filter(entry => fs.existsSync(entry.path)) + .sort((a, b) => fs.statSync(b.path).mtimeMs - fs.statSync(a.path).mtimeMs)[0]?.id || null; +} + +function viewportEvaluationData(broadcastId) { + const output = viewportEvaluationDir(broadcastId); + const trace = viewportTracePath(broadcastId); + const metricPath = path.join(output, "viewport_metrics.json"); + let metrics = null; + try { metrics = JSON.parse(fs.readFileSync(metricPath, "utf8")); } + catch (_) { /* The first plot may still be rendering. */ } + const plotNames = [ + "viewport_tracking.png", "viewport_error_cdf.png", "viewport_trajectory.png", + ]; + const plots = Object.fromEntries(plotNames.map(name => [name, { + ready: fs.existsSync(path.join(output, name)), + url: `/server-results/${encodeURIComponent(broadcastId)}` + + `/viewport_evaluation/${name}`, + }])); + const job = viewportPlotJobs.get(broadcastId); + return { + broadcastId, + traceReady: fs.existsSync(trace), + traceBytes: fs.existsSync(trace) ? fs.statSync(trace).size : 0, + plotting: Boolean(job?.running || job?.timer), + plotError: job?.lastError || null, + metrics, + plots, + }; +} + +app.get("/api/viewport-evaluation/:broadcastId", (req, res) => { + if (!validViewportBroadcastId(req.params.broadcastId)) + return res.status(404).json({ error: "unknown broadcast" }); + res.json(viewportEvaluationData(req.params.broadcastId)); +}); + +function renderViewportEvaluation(req, res) { + const broadcastId = req.params.broadcastId || latestViewportBroadcastId(); + if (!validViewportBroadcastId(broadcastId)) { + return res.status(404).type("html").send( + "

No viewport evaluation yet

Start a Quest mesh run and wait for the first trace batch.

" + ); + } + const data = viewportEvaluationData(broadcastId); + const metric = value => Number.isFinite(value) ? value.toFixed(4) : "—"; + const featureRows = Object.entries(data.metrics?.features || {}).map(([name, value]) => + `${name}${value.unit}${value.n}` + + `${metric(value.mae)}` + + `${metric(value.rmse)}` + + `${metric(value.p95)}` + ).join(""); + const cards = Object.entries(data.plots).map(([name, plot]) => + `

${name.replace("viewport_", "").replace(".png", "").replaceAll("_", " ")}

` + + (plot.ready + ? `${name}` + : "

Waiting for the first server-side render…

") + + "
" + ).join(""); + res.type("html").send(` + + +Viewport evaluation — ${broadcastId} + +

Viewport predictor evaluation

+
Run: ${broadcastId} · trace ${(data.traceBytes / 1024).toFixed(1)} KiB · ` + + `${data.plotting ? "updating plots" : data.plotError ? "plot failed" : "plots current"} · auto-refresh 5 s
+${data.plotError ? `

Plot error: ${data.plotError}

` : ""} +${featureRows ? `${featureRows}
FeatureUnitNMAERMSEP95
` : "

Waiting for metrics…

"} +
${cards}
+`); +} + +app.get("/viewport-evaluation", renderViewportEvaluation); +app.get("/viewport-evaluation/:broadcastId", renderViewportEvaluation); + +// NEW: Get latest interval data (for live monitoring) +app.get("/api/bitrate-counts-interval/latest", (req, res) => { + const intervalsFile = intervalsFilePath(); + + if (!fs.existsSync(intervalsFile)) { + return res.json({ latest: null, message: "No interval data yet" }); + } + + const lines = fs.readFileSync(intervalsFile, 'utf-8').trim().split('\n'); + if (lines.length === 0) { + return res.json({ latest: null, message: "No interval data yet" }); + } + + try { + const latest = JSON.parse(lines[lines.length - 1]); + res.json({ latest, totalIntervals: lines.length }); + } catch (e) { + res.status(500).json({ error: "Failed to parse latest interval" }); + } +}); + +// NEW: Clear interval data (for new trace) +app.delete("/api/bitrate-counts-interval", (req, res) => { + const intervalsFile = intervalsFilePath(); + + if (fs.existsSync(intervalsFile)) { + fs.unlinkSync(intervalsFile); + logInfo("BITRATE", "Interval data cleared"); + } + + res.json({ status: "ok", message: "Interval data cleared" }); +}); + +// ==================== END NEW ENDPOINTS ==================== + +app.get("/api/health", (req, res) => { + res.json({ + status: "ok", + active: broadcasts.size, + segments: selectionLog.length, + ladderBusy, + pendingSegId, + latestManifestSegId, + latestManifestPath: getManifestPath(latestManifestSegId), + streamConfig: STREAM_CONFIG, + currentBroadcastId, // NEW + currentScene, + currentSceneObjects, + currentSkybox, + resultsDir: SERVER_RESULTS_DIR // NEW + }); +}); +app.use((req, res, next) => { res.header('Access-Control-Allow-Origin','*'); res.header('Access-Control-Allow-Headers','Content-Type'); next(); }); +app.get("/api/selections", (req, res) => { + res.json(selectionLog); +}); + +// -------------------- Startup -------------------- +async function bootstrap() { + logInfo("SERVER", "Viewpoint logger + ladder generator starting"); + logInfo("SERVER", ` port: ${PORT}`); + logInfo("SERVER", ` manifest: ${getManifestPath(0)} (initial)`); + logInfo("SERVER", ` viewpoints: ${SERVER_VIEWPOINTS_DIR}`); + logInfo("SERVER", ` selections: ${SERVER_SELECTIONS_DIR}`); + logInfo("SERVER", ` results: ${SERVER_RESULTS_DIR}`); + logInfo("SERVER", ` ladder: ${PYTHON_BIN} -m vstream.ladder.ladder_service (cwd ${REPO_ROOT})`); + logInfo("SERVER", ` static: /files -> ${FILES_ROOT}`); + logInfo( + "SERVER", + ` config: segments=${TOTAL_SEGMENTS} interval=${SEGMENT_INTERVAL}ms frames=${FRAMES_PER_SEGMENT} updateEvery=${UPDATE_INTERVAL_SEGMENTS}` + ); + try { + const launch = questLaunchProfile(""); + logInfo("SERVER", ` quest: pipeline=${launch.pipeline}` + + (launch.pipeline === "mesh" ? "" + : ` mediaPort=${launch.baselinePort}`) + + ` bandwidth=${launch.groundTruthBandwidthMode + ? "ground-truth-trace" + : launch.fullBandwidthMode ? "fixed-probe" : "adaptive"}` + + ` prediction=${launch.viewportPredictionEnabled}` + + ` culling=${launch.orbitViewCullingEnabled}` + + ` (${launch.source})`); + } catch (error) { + logError("SERVER", ` quest: invalid launch profile: ${error.message}`); + } + // Spawn the ladder service now so its heavy context (models, meshes, + // raycast scenes, cached predictions) loads before the first request. + startLadderService(); + + if (latestManifestSegId < 0) { + logInfo("SERVER", "No manifest on disk - generating default seg=0 manifest"); + requestLadderUpdate(0, null, "default manifest").catch(() => {}); + } else { + logInfo("SERVER", "Initial segment-0 manifest already exists"); + } + logInfo("SERVER", "First broadcast viewpoint will refresh segment 0 weights"); +} + +if (require.main === module) { + // Before the port opens: a mismatched media cache must stop the server, not + // produce a run whose reported bitrates belong to a different codec. + try { + enforceCorpusStamp(); + } catch (error) { + logError("SERVER", error.message); + process.exit(1); + } + if (COMPRESSED_ROOT) logInfo("SERVER", `corpus ${COMPRESSED_ROOT}`); + app.listen(PORT, "0.0.0.0", () => { + bootstrap().catch((e) => { + logError("SERVER", `Bootstrap failed: ${e.message}`); + }); + }); +} + +module.exports = { + app, questLaunchProfile, offlineTrajectorySamples, offlineTrajectoryPayload, + sceneRelativeTrajectoryPayload, +}; diff --git a/open4d/streaming/system/Server/test_offline_trajectory.js b/open4d/streaming/system/Server/test_offline_trajectory.js new file mode 100644 index 00000000..42d32bd4 --- /dev/null +++ b/open4d/streaming/system/Server/test_offline_trajectory.js @@ -0,0 +1,60 @@ +#!/usr/bin/env node +const assert = require("assert"); +const { + offlineTrajectorySamples, offlineTrajectoryPayload, sceneRelativeTrajectoryPayload, +} = require("./server"); + +const header = "t_s,playback_s,presentation_slot,scene,streaming,broadcast_id,x,y,z,yaw,pitch,roll"; +const csv = [ + header, + "1,,,-,1,old,0,0,0,0,0,0", + "2,18.7,560,-,1,old,99,0,0,0,0,0", + "3,19.9,598,-,1,old,99,0,0,0,0,0", + "4,0.01,0,-,1,capture,1,0,0,0,0,0", + "5,1.01,30,-,1,capture,3,0,0,0,0,0", +].join("\n"); + +assert.deepStrictEqual(offlineTrajectorySamples(csv, "capture"), [ + [0.01, 1, 0, 0, 0, 0, 0], + [1.01, 3, 0, 0, 0, 0, 0], +]); + +const restarted = [ + header, + "1,18.7,560,-,1,capture,99,0,0,0,0,0", + "2,19.9,598,-,1,capture,99,0,0,0,0,0", + "3,0.01,0,-,1,capture,1,0,0,0,0,0", + "4,10.01,300,-,1,capture,2,0,0,0,0,0", +].join("\n"); +assert.deepStrictEqual(offlineTrajectorySamples(restarted, "capture"), [ + [0.01, 1, 0, 0, 0, 0, 0], + [10.01, 2, 0, 0, 0, 0, 0], +]); + +assert.deepStrictEqual(offlineTrajectoryPayload(csv, "archived"), { + sourceBroadcastId: "capture", + samples: [ + [0.01, 1, 0, 0, 0, 0, 0], + [1.01, 3, 0, 0, 0, 0, 0], + ], +}); + +const identityExtrinsic = [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]; +assert.deepStrictEqual(sceneRelativeTrajectoryPayload({ + schemaVersion: 2, + coordinateSpace: "open3d-camera-extrinsic-column-major", + sourceBroadcastId: "p00-capture", + samples: [[0, ...identityExtrinsic], [1, ...identityExtrinsic]], +}), { + schemaVersion: 2, + coordinateSpace: "open3d-camera-extrinsic-column-major", + sourceBroadcastId: "p00-capture", + samples: [[0, ...identityExtrinsic], [1, ...identityExtrinsic]], +}); +assert.throws(() => sceneRelativeTrajectoryPayload({ + schemaVersion: 2, + coordinateSpace: "open3d-camera-extrinsic-column-major", + samples: [[1, ...identityExtrinsic], [0, ...identityExtrinsic]], +}), /invalid scene-relative/); + +console.log("offline trajectory tests passed"); diff --git a/open4d/streaming/system/Server/test_quest_trace_player.js b/open4d/streaming/system/Server/test_quest_trace_player.js new file mode 100644 index 00000000..10e4f0f4 --- /dev/null +++ b/open4d/streaming/system/Server/test_quest_trace_player.js @@ -0,0 +1,31 @@ +#!/usr/bin/env node +const assert = require('assert'); +const { + TBF_BURST_FLOOR_BYTES, + tbfArgs, + tcByteCount, + tbfIsWedged, +} = require('./quest-trace-player'); + +function argumentAfter(args, name) { + const index = args.indexOf(name); + assert.notStrictEqual(index, -1, `missing ${name}`); + return args[index + 1]; +} + +const normalBurst = Number(argumentAfter(tbfArgs(200), 'burst').replace(/b$/, '')); +assert.strictEqual(normalBurst, TBF_BURST_FLOOR_BYTES); +assert(normalBurst >= 2 * 65536, 'bucket must fit a normal GSO skb with headroom'); + +const veryFastBurst = Number(argumentAfter(tbfArgs(2000), 'burst').replace(/b$/, '')); +assert.strictEqual(veryFastBurst, 250000, 'one-millisecond rate burst still applies above floor'); + +assert.strictEqual(tcByteCount('64', 'K'), 65536); +assert.strictEqual(tcByteCount('1.5', 'M'), 1572864); + +const before = { bytes: 761000000, overlimits: 1000 }; +assert.strictEqual(tbfIsWedged(before, { bytes: 761000000, overlimits: 1001 }), true); +assert.strictEqual(tbfIsWedged(before, { bytes: 761000001, overlimits: 1001 }), false); +assert.strictEqual(tbfIsWedged(before, { bytes: 761000000, overlimits: 1000 }), false); + +console.log('quest trace player tests passed'); diff --git a/open4d/streaming/system/WebClient/README.md b/open4d/streaming/system/WebClient/README.md new file mode 100644 index 00000000..3b9f9b74 --- /dev/null +++ b/open4d/streaming/system/WebClient/README.md @@ -0,0 +1,342 @@ +# WebClient + +Four browser pages, served by `system/Server` at `/web`, one per system under +comparison. + +Start at **`/web/compare.html`** — the chooser. It asks `/api/systems` which +systems have their assets present, and for our own system it lists each +object's published ladder cost so you can pick a scene the link can actually +carry before launching. Every page carries a `switch` link back to it. + +| Page | System | Runs the ABR? | +|---|---|---| +| `/web/compare.html` | the chooser — availability, scene picker, and which systems adapt | — | +| `/web/` | **ours** — adaptive mesh ladder | yes, in the browser | +| `/web/baseline.html` | ViVo / NAVA point clouds (`?bridge=` selects which) | yes, server-side | +| `/web/vega.html` | Vega (3D Gaussian splatting) | no, fixed quality | +| `/web/nevo.html` | NeVo (ReRF neural volumetric) | no, pre-rendered | + +The first page is the important one: it runs the **same** `system/ClientCore` +as the Node desktop client — same MCKP ABR, same segment loop, same bandwidth +estimator, same metrics — behind a browser implementation of the +`ClientPlatform` contract. So a desktop/browser difference is a real difference +in the system under test, not a difference between two hand-written clients. + +The other three do **not** use ClientCore, deliberately. ClientCore models an +HTTP segment loop over a published ladder; the baselines push frames at their +own cadence and adapt server-side. Wrapping them in a segment loop would +measure the wrapper. + +## Build and run + +```bash +cd system/WebClient && npm install && node build.js # --watch to rebuild on change +cd ../.. && PYTHON_BIN= scripts/run_web_demo.sh +``` + +That starts everything — the server on the **H.264** corpus plus a supervised +ViVo and NAVA — and prints the URLs. Start at +`http://:3000/web/compare.html`. Ctrl-C stops all of it. + +Two defaults in that script are deliberate. It serves H.264 because HEVC is +Firefox-no and Chrome-only-with-hardware, and it sets 300 segments because the +server's own default of 10 is a *trial* length: the page reaches "run complete" +in 20 s, before you have finished looking at it. + +To make adaptation visible rather than merely running, replay a trace in +another shell (needs root, and shapes the whole host): + +```bash +sudo scripts/shape_web_demo.sh cascade-20 +``` + +## Page parameters + +Query strings, mirroring how the Node client takes environment variables. + +**`/web/` (ours)** + +| parameter | default | meaning | +|---|---|---| +| `server` | page origin | server base URL, for a cross-host run | +| `mode` | `interactive` | `interactive` \| `simulated` | +| `storage` | `memory` | `memory` \| `opfs` asset store | +| `viewpoints` | — | viewpoint-index JSON; **required** for `simulated` | +| `decodeBudget` | `512` | decoded-clip cache budget, MB | +| `concurrency` / `inflight` | `10` / `2` | parallel asset requests / concurrent segments | +| `label` | `web-client` | run label recorded on the server | +| `objects` | server's full catalog | comma-separated object subset | + +`simulated` has no default viewpoint file on purpose: solving the first ladder +against an arbitrary pose would silently invalidate the viewpoint-aware +comparison. + +`objects` is the parameter that decides whether a run demonstrates adaptation +or a permanent deficit. The ladder publishes at least one representation per +object in the scene, so the scene size sets an irreducible floor: all nine +ORBIT objects floor at **116 Mbps**, which no ordinary link carries, and the +MCKP then buys the few highest-weighted objects at their cheapest rung and +freezes the rest for the whole run. Three objects floor near **21 Mbps**. An +absent list resets the server to its full catalog, so a run never inherits the +previous one's scene. The Node client reads the same subset from +`VS4D_SCENE_OBJECTS`. + +**`/web/baseline.html`** — `bridge` (`ws://:8790`), `pointSize` (`0.012` m), +`strict` (`0`; `1` aborts on a frame gap instead of resynchronising). + +Which point-cloud baselines can actually run depends on the corpus, and the two +cases need opposite responses: + +`?bridge=` is what selects the baseline: the page is identical either way, and +`/api/systems` probes each port so the chooser marks a baseline ready only when +something is actually listening on it (which is different from the tile corpus +merely existing -- the two need opposite fixes). Override the port map with +`VS4D_POINTCLOUD_PORTS=vivo:8790:12345,nava:8791:12346`. + +- **ViVo and NAVA** run from the prepared tile corpus with + `--tile-catalog-ladder`. Everything they read at serve time is in + `catalog.json`, and the cameras they put in the connection header are + synthetic per-tile identities, so the absent RGB-D source costs them nothing. + See [`baselines/ViVo/orbitvivo/tile_ladder.py`](../../baselines/ViVo/orbitvivo/tile_ladder.py). +- **MetaStream, DeltaStream and LiVo** index the real capture rig while serving + (`obj.cameras[camera_index]`) and read the source RGB-D frames. For them the + absent corpus is missing *data*, and no adapter can substitute for it. + +**`/web/vega.html`** — `assets` (`/vega-assets`), `objects` (all nine), +`frame` (`object`\|`all`), `splatMode` (`isotropic`\|`anisotropic`), +`splatScale` (`1.0`). + +The Vega page **loads the whole clip before playing** and then loops it from +memory; expect ~15 s of visible progress on a 35 Mbps link for the 64 MB +two-object export. It is not streamed, and cannot be: a frame is ~1.1 MB at +30 fps, so two objects in real time would need **509 Mbps**. The full +nine-object export is ~345 MB, so use `?objects=` (or the chooser's picker, +which defaults to the two smallest clips) rather than opening all of them. The earlier +sliding-window prefetch only worked when served from the same machine — over a +real link it issued requests faster than they completed and, because it checked +only the decoded cache, re-issued each pending frame every 33 ms tick until +Chrome's socket pool gave out and every fetch returned a bare +`TypeError: Failed to fetch` with no backoff. Fetches are now deduplicated by +`(object, frame)`, bounded to 6 in flight, and a failed frame is remembered +rather than retried. + +**`/web/nevo.html`** — `assets` (`/nevo-assets`), `object` (`g_dancer`), +`fps` (`8`), `nevoOnly` (`0`). + +### Getting the baselines to serve something + +```bash +# point clouds: raw TCP, which a browser cannot open, so bridge it. This runs +# the bridge and supervises the baseline server, because a baseline serves one +# connection from frame zero and then exits -- correct for a measured trial, +# but it means an unsupervised server survives exactly one page load. +# Run both: they take different default ports (vivo 8790, nava 8791), so the +# chooser offers either and switching is a click rather than a restart. +PYTHON_BIN= scripts/serve_pointcloud_baseline.sh vivo dancer,thomas & +PYTHON_BIN= scripts/serve_pointcloud_baseline.sh nava dancer,thomas & +# -> /web/baseline.html?bridge=ws://:8790 (vivo) +# -> /web/baseline.html?bridge=ws://:8791 (nava) + +# vega: export the trained bitstream to the portable VGS format once +python -m baselines.Vega.orbitvega.export_quest \ + --prepared-dir results/vega-gaussian/prepared-final --output-dir results/vega-web \ + --dataset-root --objects dancer thomas +# -> /web/vega.html?objects=dancer (override location: VS4D_VEGA_WEB_ROOT) + +# nevo: renders come from orbitnevo/render_frames.py +# -> /web/nevo.html?object=g_dancer (override location: VS4D_NEVO_WEB_ROOT) +``` + +## Files + +| File | Role | +|---|---| +| `src/browser-platform.js` | The `ClientPlatform` capabilities: transport, storage, viewpoints, clock, logger, lifecycle | +| `src/webgl-renderer.js` | Renderer capability: Three.js scene, frame presentation, decode orchestration | +| `src/decode-cache.js` | Decoded-clip LRU under a byte budget, keyed `(objectName, repId)` | +| `src/texture-decoder.js` | MP4 demux (mp4box) + WebCodecs `VideoDecoder` | +| `src/draco-worker.js` | Draco mesh and point-cloud decode, off the main thread | +| `src/camera-pose.js` | Three.js camera → Open3D `PinholeCameraParameters` | +| `src/main.js` | Entry point for `/web/` | +| `src/chooser.js` | The chooser: `/api/systems`, the scene picker, and the floor-vs-capacity check | +| `bridge/v4ds-bridge.js` | WebSocket ↔ TCP proxy for the point-cloud baselines; reframes only | +| `src/v4ds-protocol.js` | V4DS CONNECTION/FRAME decode, FEEDBACK encode | +| `src/point-reconstruction.js` | MetaStream/DeltaStream delta reconstruction | +| `src/point-renderer.js` | `THREE.Points` per object | +| `src/vgs-format.js` | VGS1 decoder — positions, scales, rotations, opacity, colour | +| `src/splat-renderer.js` | Instanced splat quads, depth sort, GLSL3 shaders | +| `src/nevo-manifest.js` | Conditions, frame filenames, panel layout, captions (pure) | +| `src/nevo-client.js` | Preload, subject crop, 2D-canvas compositing | +| `src/{baseline,vega,nevo}-client.js`, `*-main.js` | Per-page logic and wiring | +| `public/*.html` | Canvas, status line, log pane, artifact links | +| `vendor/draco/` | Official Draco JS/WASM decoder, unmodified | + +## Pitfalls + +Each of these cost real debugging time. Most fail *silently* — plausible output +rather than an error — which is why they are written down. + +**Measurement integrity** + +- Media and API fetches are `cache: 'no-store'`. A segment served from the HTTP + cache turns a shaped-bandwidth measurement into fiction; a cached manifest + pins the client to a stale ladder with no error. +- Stop finalizes, it does not navigate. The final `POST /api/results` happens + during shutdown, so unloading would cancel it. `pagehide` finalizes too. +- Telemetry writes never block: 30 Hz render telemetry through a synchronous + write per frame starved the renderer in Node. +- Only decodes are cached, never downloads. A 1920-wide frame is `w*h*1.5` = + 5.5 MB decoded however it was stored, so a 60-frame clip × 5 objects is + 1.66 GB. The cache pins the clip on screen, since evicting *that* looks like + a frame reverting mid-playback rather than an error. +- Bandwidth shaping stays `tc` on the Linux host, exactly as + `system/Client/run-client-experiments.sh` does it — the browser is just + another process behind the same qdisc. DevTools throttling is not scriptable + enough to be an experimental control. +- **Without shaping the demo cannot show adaptation**, only run it. On an + unshaped LAN every adaptive system settles on one operating point: NAVA held + quality level 5 for thirteen consecutive segments while its DP solved each + one. Replay a trace with `sudo scripts/shape_web_demo.sh cascade-20` (a + 12.5 → 25 → 50 → 100 → 175 Mbps staircase in 20 s steps) and both `/web/` and + `/web/baseline.html` show the enforced rate on their `link:` line beside the + client's own estimate, so a representation switch reads as cause and effect. + `GET /api/shaping` is the unprivileged source — reading `tc` needs no root, + only installing rules does. Note the trace player installs a **root** token + bucket, so it shapes every outbound flow on the host, not just the demo; + that is inherited from the Quest methodology because a destination `u32` + filter silently misses traffic on a multiqueue NIC, and these NICs are + multiqueue. + +**Formats and coordinates** + +- The pose POSTed to `/api/viewpoint` must be Open3D `PinholeCameraParameters`; + the server writes it straight to disk and reads it with + `o3d.io.read_pinhole_camera_parameters`. Anything else leaves the ladder + unable to parse it, and it returns **equal** object weights with no error — + silently disabling the whole point of the system. `camera-pose.js` handles + both conversions: Three.js is Y-up/−Z-forward, Open3D is Y-down/+Z-forward, + and translations are in **millimetres** (`VIEW_RAYCAST_UNITS_PER_METER`). +- V4DS is **big-endian**; VGS1 is **little-endian**. Same client decodes both. +- Streamed baseline points are in **camera** space; `camera_to_world` arrives + **row-major**, and `THREE.Matrix4.fromArray` is column-major, so handing it + over directly transposes every camera. +- V4DS timestamps are epoch **nanoseconds** (~1.8×10¹⁸, ~200× `MAX_SAFE_INTEGER`) + and need BigInt. Frame ids stay Numbers; `sourceTimestampMs` is derived. +- VGS1 quaternion components are **signed** bytes. Read unsigned, every + rotation mirrors into a still-plausible blob of Gaussians. + +**Rendering** + +- The camera auto-frames the content. ORBIT subjects are baked into venue world + coordinates with three floor tiers (`scene_layout.json`), so a mesh can sit + metres above the origin — the first `dancer` frame centres at y=3.03. Framing + stops as soon as the viewer touches the camera; their pose is the input. +- **Fitting the whole scene is the wrong default for a two-object viewer.** The + same venue layout puts `dancer` at z −1.8..−0.9 and `thomas` at z +8.0..+8.4, + ten metres apart, so a camera that fits both retreats to z 18.5 and each + subject covers **0.43%** of the canvas — indistinguishable from an empty + player. Vega therefore frames one object (**3.6–5.1%** coverage), names the + others in its log, and offers a `Focus` button to cycle; `?frame=all` + restores the whole-venue fit for inspecting the layout itself. +- Geometry gaps hold, they do not shift: a null frame re-presents the previous + one, which is why `ClientCore/download-plan.js` preallocates a null slot per + frame rather than collecting only successes. +- Splats: data lives in a texture and only a 16-bit-depth counting sort over an + index attribute is reordered (230 KB instead of ~3 MB per sort). Depth test + and depth write are both off — in 3DGS the ordering *is* the depth resolution. +- **A `SplatObject` must be constructed at its sequence's real splat count.** + Three caches `_maxInstanceCount` from the instanced attributes the first time + it binds a geometry, and draws + `min(geometry.instanceCount, _maxInstanceCount)`. Swapping in a bigger + `splatIndex` attribute afterwards does not reliably reset that cache, so a + geometry first bound at a placeholder capacity keeps drawing **one** instance + however large `instanceCount` is — 4 triangles instead of 213,224, an + empty-looking canvas with every other diagnostic reading healthy. Whether it + happened depended on whether the render loop bound the mesh before the first + frame arrived, so it failed on roughly three page loads in four while the + stats panel still reported 108,342 splats and zero decode failures. + `_allocate` now throws on post-draw growth instead of capping silently, the + mesh stays hidden until it has data, and `VegaClient` takes each capacity + from the catalogue. +- Vega colour is **baked**, not view-dependent: `export_quest.py` quantises a + view-independent sample per splat. Geometry and opacity are exact; appearance + is the same approximation the offline evaluator uses, so it is comparable + with those numbers but is not the paper's full appearance. +- NeVo has no camera control because it cannot run client-side: a frame takes + ~0.5 s to ray-march on a workstation GPU and its entropy decoder is CUDA + + Python 3.8. The page plays pre-rendered frames — plain ReRF, NeVo's + visibility-filtered output, and the captured camera at the same instant and + viewpoint, each captioned with its kept voxel fraction. All renders preload + before playback (a condition one frame behind would be indistinguishable from + a real difference) and the crop is measured once and applied to every panel. + +## Known limitations + +- **Texture codec.** The decoder is codec-agnostic: it reads `track.codec` and + whichever of avcC/hvcC the file carries, so it decodes H.264 + (`avc1.64001f`) and HEVC (`hvc1.1.6.L90.90`) through the same path. **Serve + the H.264 corpus for a browser audience** — HEVC is Safari-yes, Firefox-no, + Chrome-only-where-the-OS-provides-hardware-decode, and + `analysis/compare_codec_rd.py` measures H.264 at just **1.003x** the bitrate + of HEVC at matched quality across all nine objects, so the ladder's floor and + ceiling barely move. Where the served codec cannot be decoded the page + renders untextured geometry and logs the reason once per object + (`texture-decoder.js: probeCodecSupport`). Never transcode at serve time: + the quality/bitrate models are codec-specific, so re-encoding changes + delivered bitrate out from under the ladder. + + +- **`splatMode=anisotropic` is untested.** Isotropic is the default because + Vega's Gaussians are near-isotropic (log scales within ~0.3) and because a + vertex shader that merely *contains* the covariance projection draws nothing + under headless SwiftShader, even when `gl_Position` ignores the result. Every + input was verified independently, which points at the driver. Try it on real + GPU hardware. +- **MetaStream, DeltaStream and LiVo are not in the demo.** They index the real + capture rig while serving (`obj.cameras[camera_index]`) and read the source + RGB-D frames, which are absent with no surviving copy — missing data, not + missing metadata, so no adapter can substitute. LiVo additionally needs + client-side depth unprojection for its `LIVO_SEGMENT` colour+depth messages; + the protocol decoder recognises the type and says so rather than failing + silently. +- **WebCodecs texture decode has never succeeded here**, for the HEVC reason + above; only the degradation path is exercised. OrbitControls interaction is + untested (headless has no pointer). +- **The multi-view NeVo sprite path** (`export_quest.py`, 8 yaw views/frame) + needs `rerf_render.py --render_views 8` under the ReRF CUDA environment. + Those renders do not exist, so the viewer is single-view. + +## Tests + +```bash +node --test tests/test_browser_platform.js tests/test_decode_cache.js \ + tests/test_camera_pose.js tests/test_v4ds_protocol.js \ + tests/test_v4ds_bridge.js tests/test_point_reconstruction.js \ + tests/test_vgs_format.js tests/test_nevo_manifest.js \ + tests/test_web_client_bundle.js +``` + +The binary decoders are checked against the authoritative Python +implementations on real data: `test_v4ds_protocol.js` against golden bytes from +the Python encoder plus a live round-trip, `test_point_reconstruction.js` +point-for-point, `test_vgs_format.js` on a real exported frame, +`test_camera_pose.js` against a real captured viewpoint file. The +`fixtures_*.gen.py` scripts regenerate the golden files. +`test_v4ds_bridge.js` covers adversarial TCP chunk boundaries and a 3 MiB +message through real sockets. `test_web_client_bundle.js` loads the **built** +`dist/main.js` in a stubbed DOM and drives a complete simulated run, so run +`node build.js` first. + +### Debugging in a real browser + +`window.__vs4d.inspect()` returns live renderer state — object visibility, +vertex counts, camera pose, decode-cache stats — which is the fastest way to +tell "nothing is drawn" from "drawn off-screen". Two headless traps: + +1. `--virtual-time-budget` deadlocks the Draco worker: `importScripts` is + synchronous and the worker goes silent with no error. Drive the page with + puppeteer-core against the installed Chrome instead. +2. `page.screenshot()` comes out black — SwiftShader does not composite the + WebGL surface. Call `renderer.render()` and `gl.readPixels()` in the *same* + synchronous task, since `preserveDrawingBuffer: false` clears after + compositing. diff --git a/open4d/streaming/system/WebClient/bridge/v4ds-bridge.js b/open4d/streaming/system/WebClient/bridge/v4ds-bridge.js new file mode 100644 index 00000000..03865bc3 --- /dev/null +++ b/open4d/streaming/system/WebClient/bridge/v4ds-bridge.js @@ -0,0 +1,280 @@ +#!/usr/bin/env node +'use strict'; + +/** + * WebSocket <-> TCP bridge for the V4DS baseline protocol. + * + * The five point-cloud baselines (MetaStream, DeltaStream, ViVo, NAVA, LiVo) + * serve over a raw TCP socket, which a browser cannot open. This proxies one + * WebSocket client onto one TCP baseline server. + * + * It is deliberately a DUMB proxy in everything except framing. It never + * decodes a message, never rewrites a field, and never invents traffic — so a + * browser run and a Quest run reach the baseline server with identical bytes, + * and any difference in their results is a real difference rather than a + * bridging artifact. + * + * What it DOES do is reframe: + * + * TCP -> browser strip the 4-byte big-endian length prefix, deliver each + * complete message as one binary WebSocket message + * browser -> TCP prepend the length prefix to each WebSocket message + * + * That is worth the small amount of state because TCP is a byte stream: without + * it the browser would have to reassemble partial messages itself, and a + * point-cloud keyframe can easily span many TCP segments. WebSocket is already + * message-framed, so this puts the boundary in the one place that has to know + * about it. + * + * One client at a time, because the baseline servers themselves `listen(1)` and + * `accept()` exactly once. A second browser is refused with a clear reason + * rather than being silently queued behind the first. + * + * Usage: + * node bridge/v4ds-bridge.js --baseline-port 12345 [options] + * + * --listen-port N WebSocket port to serve (default 8790) + * --listen-host H WebSocket bind address (default 0.0.0.0) + * --baseline-host H baseline TCP host (default 127.0.0.1) + * --baseline-port N baseline TCP port (required) + * --max-message-bytes refuse larger frames (default 256 MiB) + * --verbose log every message + */ + +const net = require('net'); +const { WebSocketServer } = require('ws'); + +const MAGIC = Buffer.from('V4DS'); +const PREFIX_BYTES = 4; +const DEFAULT_MAX_MESSAGE_BYTES = 256 * 1024 * 1024; + +function parseArgs(argv) { + const options = { + listenHost: '0.0.0.0', + listenPort: 8790, + baselineHost: '127.0.0.1', + baselinePort: null, + maxMessageBytes: DEFAULT_MAX_MESSAGE_BYTES, + verbose: false + }; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + const next = () => { + const value = argv[++i]; + if (value === undefined) throw new Error(`${arg} needs a value`); + return value; + }; + switch (arg) { + case '--listen-host': options.listenHost = next(); break; + case '--listen-port': options.listenPort = Number(next()); break; + case '--baseline-host': options.baselineHost = next(); break; + case '--baseline-port': options.baselinePort = Number(next()); break; + case '--max-message-bytes': + options.maxMessageBytes = Number(next()); break; + case '--verbose': options.verbose = true; break; + case '--help': case '-h': options.help = true; break; + default: throw new Error(`unknown argument: ${arg}`); + } + } + return options; +} + +/** + * Reassembles length-prefixed V4DS messages from a TCP byte stream. + * + * Kept as a class with no I/O so it can be tested directly against adversarial + * chunk boundaries — one byte at a time, several messages in one chunk, a prefix + * split across chunks. Those are exactly the cases that work by luck on + * localhost and fail on a real link. + */ +class MessageFramer { + constructor({ maxMessageBytes = DEFAULT_MAX_MESSAGE_BYTES } = {}) { + this.maxMessageBytes = maxMessageBytes; + this._buffer = Buffer.alloc(0); + } + + /** @returns {Buffer[]} every complete message now available */ + push(chunk) { + this._buffer = this._buffer.length === 0 + ? chunk : Buffer.concat([this._buffer, chunk]); + + const messages = []; + for (;;) { + if (this._buffer.length < PREFIX_BYTES) break; + const length = this._buffer.readUInt32BE(0); + if (length > this.maxMessageBytes) { + throw new Error( + `message length ${length} exceeds the ${this.maxMessageBytes} limit`); + } + if (this._buffer.length < PREFIX_BYTES + length) break; + messages.push( + this._buffer.subarray(PREFIX_BYTES, PREFIX_BYTES + length)); + this._buffer = this._buffer.subarray(PREFIX_BYTES + length); + } + return messages; + } + + /** Bytes held back waiting for the rest of a message. */ + get pendingBytes() { return this._buffer.length; } +} + +/** Prepend the 4-byte big-endian length prefix. */ +function frame(payload) { + const prefix = Buffer.alloc(PREFIX_BYTES); + prefix.writeUInt32BE(payload.length, 0); + return Buffer.concat([prefix, payload]); +} + +function looksLikeV4ds(message) { + return message.length >= 8 && message.subarray(0, 4).equals(MAGIC); +} + +function timestamp() { + return new Date().toISOString().replace('T', ' ').replace('Z', ''); +} + +function log(level, message) { + const line = `[${timestamp()}][${level}][BRIDGE] ${message}`; + if (level === 'ERROR') console.error(line); + else console.log(line); +} + +function createBridge(options) { + const server = new WebSocketServer({ + host: options.listenHost, + port: options.listenPort + }); + let active = null; + + server.on('listening', () => { + log('INFO', `WebSocket on ws://${options.listenHost}:${options.listenPort}` + + ` -> tcp://${options.baselineHost}:${options.baselinePort}`); + }); + + server.on('connection', (socket, request) => { + const peer = request.socket.remoteAddress; + + // The baseline servers accept exactly one connection, so admitting a + // second browser would leave it waiting forever with no explanation. + if (active) { + log('WARN', `refusing ${peer}: a client is already connected`); + socket.close(1013, 'bridge already has a client'); + return; + } + + log('INFO', `client ${peer} connected; dialling the baseline server`); + const tcp = net.createConnection( + { host: options.baselineHost, port: options.baselinePort }); + tcp.setNoDelay(true); + const framer = new MessageFramer(options); + active = { socket, tcp }; + + let fromBaseline = 0; + let toBaseline = 0; + + const shutdown = (reason, code = 1000) => { + if (active?.socket !== socket) return; + active = null; + log('INFO', `closing (${reason}); ${fromBaseline} messages down, ` + + `${toBaseline} up`); + try { tcp.destroy(); } catch (_) { /* already gone */ } + try { socket.close(code, reason.slice(0, 120)); } catch (_) { /* ditto */ } + }; + + tcp.on('connect', () => { + log('INFO', 'baseline connected'); + }); + + tcp.on('data', chunk => { + let messages; + try { + messages = framer.push(chunk); + } catch (err) { + log('ERROR', `framing failed: ${err.message}`); + shutdown('framing error', 1011); + return; + } + for (const message of messages) { + fromBaseline++; + if (options.verbose) { + log('DEBUG', `down #${fromBaseline} type=` + + `${looksLikeV4ds(message) ? message.readUInt8(6) : '?'}` + + ` ${message.length}B`); + } + if (socket.readyState === socket.OPEN) socket.send(message); + } + // Backpressure: if the browser cannot keep up, stop reading from the + // baseline rather than growing an unbounded buffer in this process. + if (socket.bufferedAmount > 32 * 1024 * 1024) { + tcp.pause(); + const resume = () => { + if (socket.bufferedAmount < 8 * 1024 * 1024) tcp.resume(); + else setTimeout(resume, 20); + }; + setTimeout(resume, 20); + } + }); + + tcp.on('error', err => { + log('ERROR', `baseline socket: ${err.message}`); + shutdown(`baseline error: ${err.message}`, 1011); + }); + tcp.on('close', () => shutdown('baseline closed the connection')); + + socket.on('message', (data, isBinary) => { + if (!isBinary) { + log('WARN', 'ignoring a text frame; V4DS is binary'); + return; + } + const payload = Buffer.isBuffer(data) ? data : Buffer.from(data); + if (!looksLikeV4ds(payload)) { + log('WARN', `dropping a ${payload.length}B upstream message ` + + 'without the V4DS magic'); + return; + } + toBaseline++; + if (options.verbose) { + log('DEBUG', `up #${toBaseline} type=${payload.readUInt8(6)}` + + ` ${payload.length}B`); + } + if (!tcp.destroyed) tcp.write(frame(payload)); + }); + + socket.on('close', () => shutdown('client disconnected')); + socket.on('error', err => { + log('ERROR', `client socket: ${err.message}`); + shutdown(`client error: ${err.message}`, 1011); + }); + }); + + server.on('error', err => log('ERROR', `WebSocket server: ${err.message}`)); + return server; +} + +function main(argv) { + let options; + try { + options = parseArgs(argv); + } catch (err) { + console.error(`${err.message}\n`); + console.error(require('fs').readFileSync(__filename, 'utf8') + .split('\n').slice(2, 44).join('\n')); + process.exit(2); + } + if (options.help || !options.baselinePort) { + console.log(require('fs').readFileSync(__filename, 'utf8') + .split('\n').slice(2, 44).join('\n')); + process.exit(options.help ? 0 : 2); + } + const server = createBridge(options); + for (const signal of ['SIGINT', 'SIGTERM']) { + process.on(signal, () => { + log('INFO', `received ${signal}, shutting down`); + server.close(() => process.exit(0)); + }); + } +} + +if (require.main === module) main(process.argv.slice(2)); + +module.exports = { MessageFramer, frame, createBridge, parseArgs, looksLikeV4ds }; diff --git a/open4d/streaming/system/WebClient/build.js b/open4d/streaming/system/WebClient/build.js new file mode 100644 index 00000000..42728149 --- /dev/null +++ b/open4d/streaming/system/WebClient/build.js @@ -0,0 +1,76 @@ +#!/usr/bin/env node +'use strict'; + +/** + * Bundle the browser client. + * + * Two entry points, because the Draco decode runs in a worker and a worker + * needs its own script: `main.js` and `draco-worker.js`. ClientCore is + * CommonJS and shared with the Node client, which esbuild bundles for the + * browser unchanged — that is the point, since the whole aim is for both + * clients to run the same streaming logic. + * + * node build.js one-shot build + * node build.js --watch rebuild on change + */ + +const esbuild = require('esbuild'); +const fs = require('fs'); +const path = require('path'); + +const ROOT = __dirname; +const OUT = path.join(ROOT, 'dist'); +const watch = process.argv.includes('--watch'); + +const options = { + entryPoints: { + main: path.join(ROOT, 'src/main.js'), + 'draco-worker': path.join(ROOT, 'src/draco-worker.js'), + baseline: path.join(ROOT, 'src/baseline-main.js'), + vega: path.join(ROOT, 'src/vega-main.js'), + nevo: path.join(ROOT, 'src/nevo-main.js'), + chooser: path.join(ROOT, 'src/chooser.js') + }, + outdir: OUT, + bundle: true, + format: 'iife', + platform: 'browser', + target: ['chrome110', 'safari16', 'firefox115'], + sourcemap: true, + minify: !watch, + logLevel: 'info', + // ClientCore is CommonJS; browsers have no `process`, and some transitive + // code sniffs for it. + define: { 'process.env.NODE_ENV': '"production"' } +}; + +function copyStatic() { + fs.mkdirSync(OUT, { recursive: true }); + for (const page of ['index.html', 'baseline.html', 'vega.html', + 'nevo.html', 'compare.html']) { + fs.copyFileSync(path.join(ROOT, 'public', page), path.join(OUT, page)); + } + const vendorOut = path.join(OUT, 'vendor/draco'); + fs.mkdirSync(vendorOut, { recursive: true }); + for (const file of ['draco_decoder.js', 'draco_decoder.wasm']) { + fs.copyFileSync(path.join(ROOT, 'vendor/draco', file), + path.join(vendorOut, file)); + } +} + +async function run() { + copyStatic(); + if (watch) { + const context = await esbuild.context(options); + await context.watch(); + console.log('watching for changes …'); + return; + } + await esbuild.build(options); + const sizes = fs.readdirSync(OUT) + .filter(f => f.endsWith('.js')) + .map(f => `${f} ${(fs.statSync(path.join(OUT, f)).size / 1024).toFixed(0)} kB`); + console.log(`built -> ${path.relative(process.cwd(), OUT)}: ${sizes.join(', ')}`); +} + +run().catch(err => { console.error(err); process.exit(1); }); diff --git a/open4d/streaming/system/WebClient/package-lock.json b/open4d/streaming/system/WebClient/package-lock.json new file mode 100644 index 00000000..ef845090 --- /dev/null +++ b/open4d/streaming/system/WebClient/package-lock.json @@ -0,0 +1,522 @@ +{ + "name": "vs4d-web-client", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "vs4d-web-client", + "version": "1.0.0", + "dependencies": { + "mp4box": "^2.4.1", + "three": "^0.170.0", + "ws": "^8.21.3" + }, + "devDependencies": { + "esbuild": "^0.24.0" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.24.2.tgz", + "integrity": "sha512-thpVCb/rhxE/BnMLQ7GReQLLN8q9qbHmI55F4489/ByVg2aQaQ6kbcLb6FHkocZzQhxc4gx0sCk0tJkKBFzDhA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.24.2.tgz", + "integrity": "sha512-tmwl4hJkCfNHwFB3nBa8z1Uy3ypZpxqxfTQOcHX+xRByyYgunVbZ9MzUUfb0RxaHIMnbHagwAxuTL+tnNM+1/Q==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.24.2.tgz", + "integrity": "sha512-cNLgeqCqV8WxfcTIOeL4OAtSmL8JjcN6m09XIgro1Wi7cF4t/THaWEa7eL5CMoMBdjoHOTh/vwTO/o2TRXIyzg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.24.2.tgz", + "integrity": "sha512-B6Q0YQDqMx9D7rvIcsXfmJfvUYLoP722bgfBlO5cGvNVb5V/+Y7nhBE3mHV9OpxBf4eAS2S68KZztiPaWq4XYw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.24.2.tgz", + "integrity": "sha512-kj3AnYWc+CekmZnS5IPu9D+HWtUI49hbnyqk0FLEJDbzCIQt7hg7ucF1SQAilhtYpIujfaHr6O0UHlzzSPdOeA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.24.2.tgz", + "integrity": "sha512-WeSrmwwHaPkNR5H3yYfowhZcbriGqooyu3zI/3GGpF8AyUdsrrP0X6KumITGA9WOyiJavnGZUwPGvxvwfWPHIA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.24.2.tgz", + "integrity": "sha512-UN8HXjtJ0k/Mj6a9+5u6+2eZ2ERD7Edt1Q9IZiB5UZAIdPnVKDoG7mdTVGhHJIeEml60JteamR3qhsr1r8gXvg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.24.2.tgz", + "integrity": "sha512-TvW7wE/89PYW+IevEJXZ5sF6gJRDY/14hyIGFXdIucxCsbRmLUcjseQu1SyTko+2idmCw94TgyaEZi9HUSOe3Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.24.2.tgz", + "integrity": "sha512-n0WRM/gWIdU29J57hJyUdIsk0WarGd6To0s+Y+LwvlC55wt+GT/OgkwoXCXvIue1i1sSNWblHEig00GBWiJgfA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.24.2.tgz", + "integrity": "sha512-7HnAD6074BW43YvvUmE/35Id9/NB7BeX5EoNkK9obndmZBUk8xmJJeU7DwmUeN7tkysslb2eSl6CTrYz6oEMQg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.24.2.tgz", + "integrity": "sha512-sfv0tGPQhcZOgTKO3oBE9xpHuUqguHvSo4jl+wjnKwFpapx+vUDcawbwPNuBIAYdRAvIDBfZVvXprIj3HA+Ugw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.24.2.tgz", + "integrity": "sha512-CN9AZr8kEndGooS35ntToZLTQLHEjtVB5n7dl8ZcTZMonJ7CCfStrYhrzF97eAecqVbVJ7APOEe18RPI4KLhwQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.24.2.tgz", + "integrity": "sha512-iMkk7qr/wl3exJATwkISxI7kTcmHKE+BlymIAbHO8xanq/TjHaaVThFF6ipWzPHryoFsesNQJPE/3wFJw4+huw==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.24.2.tgz", + "integrity": "sha512-shsVrgCZ57Vr2L8mm39kO5PPIb+843FStGt7sGGoqiiWYconSxwTiuswC1VJZLCjNiMLAMh34jg4VSEQb+iEbw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.24.2.tgz", + "integrity": "sha512-4eSFWnU9Hhd68fW16GD0TINewo1L6dRrB+oLNNbYyMUAeOD2yCK5KXGK1GH4qD/kT+bTEXjsyTCiJGHPZ3eM9Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.24.2.tgz", + "integrity": "sha512-S0Bh0A53b0YHL2XEXC20bHLuGMOhFDO6GN4b3YjRLK//Ep3ql3erpNcPlEFed93hsQAjAQDNsvcK+hV90FubSw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.24.2.tgz", + "integrity": "sha512-8Qi4nQcCTbLnK9WoMjdC9NiTG6/E38RNICU6sUNqK0QFxCYgoARqVqxdFmWkdonVsvGqWhmm7MO0jyTqLqwj0Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.24.2.tgz", + "integrity": "sha512-wuLK/VztRRpMt9zyHSazyCVdCXlpHkKm34WUyinD2lzK07FAHTq0KQvZZlXikNWkDGoT6x3TD51jKQ7gMVpopw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.24.2.tgz", + "integrity": "sha512-VefFaQUc4FMmJuAxmIHgUmfNiLXY438XrL4GDNV1Y1H/RW3qow68xTwjZKfj/+Plp9NANmzbH5R40Meudu8mmw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.24.2.tgz", + "integrity": "sha512-YQbi46SBct6iKnszhSvdluqDmxCJA+Pu280Av9WICNwQmMxV7nLRHZfjQzwbPs3jeWnuAhE9Jy0NrnJ12Oz+0A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.24.2.tgz", + "integrity": "sha512-+iDS6zpNM6EnJyWv0bMGLWSWeXGN/HTaF/LXHXHwejGsVi+ooqDfMCCTerNFxEkM3wYVcExkeGXNqshc9iMaOA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.24.2.tgz", + "integrity": "sha512-hTdsW27jcktEvpwNHJU4ZwWFGkz2zRJUz8pvddmXPtXDzVKTTINmlmga3ZzwcuMpUvLw7JkLy9QLKyGpD2Yxig==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.24.2.tgz", + "integrity": "sha512-LihEQ2BBKVFLOC9ZItT9iFprsE9tqjDjnbulhHoFxYQtQfai7qfluVODIYxt1PgdoyQkz23+01rzwNwYfutxUQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.24.2.tgz", + "integrity": "sha512-q+iGUwfs8tncmFC9pcnD5IvRHAzmbwQ3GPS5/ceCyHdjXubwQWI12MKWSNSMYLJMq23/IUCvJMS76PDqXe1fxA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.24.2.tgz", + "integrity": "sha512-7VTgWzgMGvup6aSqDPLiW5zHaxYJGTO4OokMjIlrCtf+VpEL+cXKtCvg723iguPYI5oaUNdS+/V7OU2gvXVWEg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/esbuild": { + "version": "0.24.2", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.24.2.tgz", + "integrity": "sha512-+9egpBW8I3CD5XPe0n6BfT5fxLzxrlDzqydF3aviG+9ni1lDC/OvMHcxqEFV0+LANZG5R1bFMWfUrjVsdwxJvA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.24.2", + "@esbuild/android-arm": "0.24.2", + "@esbuild/android-arm64": "0.24.2", + "@esbuild/android-x64": "0.24.2", + "@esbuild/darwin-arm64": "0.24.2", + "@esbuild/darwin-x64": "0.24.2", + "@esbuild/freebsd-arm64": "0.24.2", + "@esbuild/freebsd-x64": "0.24.2", + "@esbuild/linux-arm": "0.24.2", + "@esbuild/linux-arm64": "0.24.2", + "@esbuild/linux-ia32": "0.24.2", + "@esbuild/linux-loong64": "0.24.2", + "@esbuild/linux-mips64el": "0.24.2", + "@esbuild/linux-ppc64": "0.24.2", + "@esbuild/linux-riscv64": "0.24.2", + "@esbuild/linux-s390x": "0.24.2", + "@esbuild/linux-x64": "0.24.2", + "@esbuild/netbsd-arm64": "0.24.2", + "@esbuild/netbsd-x64": "0.24.2", + "@esbuild/openbsd-arm64": "0.24.2", + "@esbuild/openbsd-x64": "0.24.2", + "@esbuild/sunos-x64": "0.24.2", + "@esbuild/win32-arm64": "0.24.2", + "@esbuild/win32-ia32": "0.24.2", + "@esbuild/win32-x64": "0.24.2" + } + }, + "node_modules/mp4box": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/mp4box/-/mp4box-2.4.1.tgz", + "integrity": "sha512-0HGX7nXoDIX6FKLVl4a3wtYjBlwqsN3xuQC3GXzNtKp98FXUOhDSq623azsz8DG5ptd9ZXcXodDkgbdMZOjWvw==", + "license": "BSD-3-Clause", + "engines": { + "node": ">=20.8.1" + } + }, + "node_modules/three": { + "version": "0.170.0", + "resolved": "https://registry.npmjs.org/three/-/three-0.170.0.tgz", + "integrity": "sha512-FQK+LEpYc0fBD+J8g6oSEyyNzjp+Q7Ks1C568WWaoMRLW+TkNNWmenWeGgJjV105Gd+p/2ql1ZcjYvNiPZBhuQ==", + "license": "MIT" + }, + "node_modules/ws": { + "version": "8.21.3", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz", + "integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + } + } +} diff --git a/open4d/streaming/system/WebClient/package.json b/open4d/streaming/system/WebClient/package.json new file mode 100644 index 00000000..57eee95a --- /dev/null +++ b/open4d/streaming/system/WebClient/package.json @@ -0,0 +1,18 @@ +{ + "name": "vs4d-web-client", + "version": "1.0.0", + "private": true, + "description": "Browser client for 4DVideoStreaming; runs the shared ClientCore", + "scripts": { + "build": "node build.js", + "watch": "node build.js --watch" + }, + "devDependencies": { + "esbuild": "^0.24.0" + }, + "dependencies": { + "mp4box": "^2.4.1", + "three": "^0.170.0", + "ws": "^8.21.3" + } +} diff --git a/open4d/streaming/system/WebClient/public/baseline.html b/open4d/streaming/system/WebClient/public/baseline.html new file mode 100644 index 00000000..a0ba17c1 --- /dev/null +++ b/open4d/streaming/system/WebClient/public/baseline.html @@ -0,0 +1,88 @@ + + + + + +4DVideoStreaming — baseline viewer + + + + + + + + diff --git a/open4d/streaming/system/WebClient/public/compare.html b/open4d/streaming/system/WebClient/public/compare.html new file mode 100644 index 00000000..6c362bb1 --- /dev/null +++ b/open4d/streaming/system/WebClient/public/compare.html @@ -0,0 +1,62 @@ + + + + + +4DVideoStreaming + + + +
+

Pick a system

+

loading …

+
+
+
+ Shaping: sudo scripts/shape_web_demo.sh cascade-20. +
+ + + diff --git a/open4d/streaming/system/WebClient/public/index.html b/open4d/streaming/system/WebClient/public/index.html new file mode 100644 index 00000000..83f7809a --- /dev/null +++ b/open4d/streaming/system/WebClient/public/index.html @@ -0,0 +1,85 @@ + + + + + +4DVideoStreaming — browser client + + + + + + + + diff --git a/open4d/streaming/system/WebClient/public/nevo.html b/open4d/streaming/system/WebClient/public/nevo.html new file mode 100644 index 00000000..e4eb06f3 --- /dev/null +++ b/open4d/streaming/system/WebClient/public/nevo.html @@ -0,0 +1,82 @@ + + + + + +4DVideoStreaming — NeVo viewer + + + + + + + + diff --git a/open4d/streaming/system/WebClient/public/vega.html b/open4d/streaming/system/WebClient/public/vega.html new file mode 100644 index 00000000..8d007f53 --- /dev/null +++ b/open4d/streaming/system/WebClient/public/vega.html @@ -0,0 +1,84 @@ + + + + + +4DVideoStreaming — Vega splat viewer + + + + + + + + diff --git a/open4d/streaming/system/WebClient/src/baseline-client.js b/open4d/streaming/system/WebClient/src/baseline-client.js new file mode 100644 index 00000000..e0e9e3c0 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/baseline-client.js @@ -0,0 +1,397 @@ +'use strict'; + +/** + * Browser client for the V4DS point-cloud baselines. + * + * MetaStream, DeltaStream, ViVo, NAVA and LiVo all speak the same protocol, so + * one client serves all five; the mode arrives in the CONNECTION header rather + * than being configured here. + * + * This does NOT use `ClientCore`, and that is the right call rather than an + * omission. ClientCore implements *our* system: an HTTP segment loop, a + * published ladder, an MCKP selector choosing one representation per object per + * segment. A baseline pushes frames at its own cadence over a socket and makes + * its own adaptation decisions server-side. Wrapping that in a segment loop + * would model the baselines as something they are not, and the comparison would + * measure the wrapper. + * + * What it does: + * 1. connect to the bridge, which proxies the baseline's TCP socket + * 2. decode CONNECTION -> calibrations, stream mode, tile catalogue + * 3. per FRAME: decode every record's Draco point cloud in the worker, fold + * it through `ReconstructionState`, hand the world clouds to the renderer + * 4. send FEEDBACK at the content rate: displayed frame, measured fps, + * measured goodput, and the live camera + */ + +const { + decodeConnection, decodeFrame, encodeFeedback, messageType, MessageType +} = require('./v4ds-protocol'); +const { ReconstructionState, PointCloud } = require('./point-reconstruction'); +const { PointRenderer } = require('./point-renderer'); + +/** Rolling goodput estimate over the received WebSocket bytes. */ +class GoodputMeter { + constructor({ windowMs = 2000, now = () => performance.now() } = {}) { + this.windowMs = windowMs; + this._now = now; + this._samples = []; // { at, bytes } + } + + record(bytes) { + const at = this._now(); + this._samples.push({ at, bytes }); + const cutoff = at - this.windowMs; + while (this._samples.length && this._samples[0].at < cutoff) { + this._samples.shift(); + } + } + + /** Mbps over the window, or null before there is enough to divide by. */ + get mbps() { + if (this._samples.length < 2) return null; + const span = this._samples[this._samples.length - 1].at - this._samples[0].at; + if (span <= 0) return null; + const bytes = this._samples.reduce((sum, s) => sum + s.bytes, 0); + return (bytes * 8) / span / 1000; + } +} + +class BaselineClient { + /** + * @param {object} args + * @param {string} args.bridgeUrl ws:// address of v4ds-bridge + * @param {HTMLCanvasElement} args.canvas + * @param {string} [args.workerUrl] + * @param {string} [args.vendorBase] + * @param {number} [args.feedbackIntervalMs] + * @param {boolean} [args.strictOrder] throw on a frame gap (default false + * in the browser: a lossy link is normal, and resynchronising on the next + * keyframe beats aborting the run) + * @param {(event: object) => void} [args.onEvent] + */ + constructor({ + bridgeUrl, canvas, workerUrl = '/web/draco-worker.js', + vendorBase = '/web/vendor/draco', feedbackIntervalMs = 200, + strictOrder = false, onEvent = null, pointSize = 0.012 + }) { + this.bridgeUrl = bridgeUrl; + this.workerUrl = workerUrl; + this.vendorBase = vendorBase; + this.feedbackIntervalMs = feedbackIntervalMs; + this.strictOrder = strictOrder; + this._onEvent = onEvent; + + this.renderer = new PointRenderer({ canvas, pointSize }); + this.header = null; + this.state = null; + this.mode = null; + + this.stats = { + framesReceived: 0, framesReconstructed: 0, framesDropped: 0, + bytesReceived: 0, resyncs: 0, decodeFailures: 0, + lastFrameId: -1, displayedFrameId: -1, measuredFps: 0 + }; + + this._socket = null; + this._worker = null; + this._workerSeq = 0; + this._workerWaiters = new Map(); + this._feedbackTimer = null; + this._goodput = new GoodputMeter(); + this._frameTimestamps = []; + this._busy = false; + this._pendingFrame = null; + this._awaitingKeyframe = false; + } + + _emit(type, detail = {}) { + this._onEvent?.({ type, ...detail }); + } + + async start() { + this.renderer.start(); + this._worker = new Worker(this.workerUrl); + this._worker.onmessage = event => { + const waiter = this._workerWaiters.get(event.data.id); + this._workerWaiters.delete(event.data.id); + if (!waiter) return; + if (event.data.error) waiter.reject(new Error(event.data.error)); + else waiter.resolve(event.data.frames); + }; + this._worker.onerror = err => + this._emit('error', { message: `draco worker: ${err.message}` }); + + await this._connect(); + this._feedbackTimer = setInterval( + () => this._sendFeedback(), this.feedbackIntervalMs); + } + + _connect() { + return new Promise((resolve, reject) => { + const socket = new WebSocket(this.bridgeUrl); + socket.binaryType = 'arraybuffer'; + this._socket = socket; + + socket.onopen = () => { + this._emit('open', { url: this.bridgeUrl }); + resolve(); + }; + socket.onerror = () => { + const message = `could not reach the bridge at ${this.bridgeUrl}`; + this._emit('error', { message }); + reject(new Error(message)); + }; + socket.onclose = event => this._emit('closed', { + code: event.code, + reason: event.reason || '(no reason given)' + }); + socket.onmessage = event => this._onMessage(event.data); + }); + } + + async _onMessage(buffer) { + this.stats.bytesReceived += buffer.byteLength; + this._goodput.record(buffer.byteLength); + + let type; + try { + type = messageType(buffer); + } catch (err) { + this._emit('error', { message: `bad message: ${err.message}` }); + return; + } + + if (type === MessageType.CONNECTION) { + this._onConnectionHeader(buffer); + return; + } + if (type === MessageType.FRAME) { + // Only one frame is reconstructed at a time; a newer frame replaces + // any frame still waiting, because showing the freshest content + // matters more than showing every frame of a backlog. + if (this._busy) { + if (this._pendingFrame) this.stats.framesDropped++; + this._pendingFrame = buffer; + return; + } + await this._drainFrames(buffer); + return; + } + if (type === MessageType.LIVO_SEGMENT) { + // LiVo ships HEVC colour+depth rather than point clouds; it needs + // the texture decoder and an unprojection step, not this path. + this._emit('unsupported', { + message: 'LiVo segments need RGB-D unprojection, not implemented' + }); + return; + } + this._emit('error', { message: `unexpected message type ${type}` }); + } + + _onConnectionHeader(buffer) { + try { + this.header = decodeConnection(buffer); + } catch (err) { + this._emit('error', { message: `bad CONNECTION: ${err.message}` }); + return; + } + this.mode = this.header.mode; + this.state = new ReconstructionState(this.header); + this._emit('header', { + mode: this.header.mode, + objects: this.header.objects.map(o => ({ + objectId: o.objectId, name: o.name, + cameras: o.cameras.length, loopFrames: o.loopFrames + })), + fps: this.header.fps, + source: `${this.header.width}x${this.header.height}`, + blockSize: this.header.blockSize, + tiled: Boolean(this.header.tileAbr) + }); + } + + async _drainFrames(first) { + this._busy = true; + let buffer = first; + try { + while (buffer) { + await this._handleFrame(buffer); + buffer = this._pendingFrame; + this._pendingFrame = null; + } + } finally { + this._busy = false; + } + } + + async _handleFrame(buffer) { + if (!this.state) { + this._emit('error', { message: 'FRAME arrived before CONNECTION' }); + return; + } + let frame; + try { + frame = decodeFrame(buffer); + } catch (err) { + this._emit('error', { message: `bad FRAME: ${err.message}` }); + return; + } + this.stats.framesReceived++; + this.stats.lastFrameId = frame.frameId; + + // After a gap the delta chain is broken; wait for a keyframe rather + // than compounding the error into visible corruption. + if (this._awaitingKeyframe) { + if (frame.frameType !== 'keyframe') return; + this._awaitingKeyframe = false; + this.state.reset(); + } + + const clouds = await this._decodeRecords(frame); + if (!clouds) return; + + try { + const worlds = this.state.apply(frame, blob => { + const cloud = clouds.get(blobKey(blob)); + return cloud || PointCloud.empty(); + }, { strictOrder: this.strictOrder }); + this.renderer.update(worlds); + this.stats.framesReconstructed++; + this.stats.displayedFrameId = frame.frameId; + this._recordPresentation(); + } catch (err) { + this.stats.resyncs++; + this._awaitingKeyframe = true; + this._emit('resync', { message: err.message, frameId: frame.frameId }); + } + } + + /** + * Decode every record's Draco payload in one worker round trip. + * + * `ReconstructionState.apply` wants a synchronous decoder, so the payloads + * are decoded up front and looked up by identity. One round trip per frame + * also beats one per record: a nine-object scene with four cameras is 36 + * payloads, and 36 postMessage round trips would not fit in a frame budget. + */ + async _decodeRecords(frame) { + const withPayload = frame.records.filter(r => r.draco && r.draco.length); + if (withPayload.length === 0) return new Map(); + + // COPY, do not transfer. Transferring `record.draco.buffer` detaches it, + // after which `record.draco.length` reads 0 — and + // ReconstructionState.apply uses exactly that length to decide whether a + // record has a payload. It would silently treat every record as empty + // and then fail the point-count check. A memcpy of a few hundred KB is + // negligible beside the Draco decode it feeds. + const buffers = withPayload.map(r => r.draco.slice().buffer); + const id = ++this._workerSeq; + let decoded; + try { + decoded = await new Promise((resolve, reject) => { + this._workerWaiters.set(id, { resolve, reject }); + this._worker.postMessage( + { id, buffers, vendorBase: this.vendorBase }, buffers); + }); + } catch (err) { + this.stats.decodeFailures++; + this._emit('error', { message: `draco decode: ${err.message}` }); + return null; + } + + const clouds = new Map(); + withPayload.forEach((record, index) => { + const frameData = decoded[index]; + if (!frameData || frameData.kind !== 'cloud') { + this.stats.decodeFailures++; + clouds.set(blobKey(record.draco), PointCloud.empty()); + return; + } + clouds.set(blobKey(record.draco), new PointCloud( + frameData.positions, + frameData.colors || new Uint8Array(frameData.positions.length))); + }); + return clouds; + } + + _recordPresentation() { + const now = performance.now(); + this._frameTimestamps.push(now); + while (this._frameTimestamps.length + && this._frameTimestamps[0] < now - 2000) { + this._frameTimestamps.shift(); + } + if (this._frameTimestamps.length >= 2) { + const span = now - this._frameTimestamps[0]; + this.stats.measuredFps = + ((this._frameTimestamps.length - 1) * 1000) / span; + } + } + + _sendFeedback() { + if (!this._socket || this._socket.readyState !== WebSocket.OPEN) return; + if (!this.header) return; + const viewer = this.renderer.viewerState(); + const bandwidth = this._goodput.mbps; + + // The frustum tier requires the extended tier, and both are all-or- + // nothing; sending a partial one is a protocol error the server rejects. + const payload = { + displayedFrameId: Math.max(0, this.stats.displayedFrameId), + measuredFps: this.stats.measuredFps || this.header.fps, + repeatedFrames: this.stats.framesDropped + }; + if (bandwidth !== null) { + Object.assign(payload, { + viewPosition: viewer.position, + viewForward: viewer.forward, + bandwidthMbps: bandwidth, + viewUp: viewer.up, + verticalFovDegrees: viewer.verticalFovDegrees, + viewAspect: viewer.aspect, + viewNear: viewer.near, + viewFar: viewer.far + }); + } + try { + this._socket.send(encodeFeedback(payload)); + } catch (err) { + this._emit('error', { message: `feedback: ${err.message}` }); + } + } + + stop() { + if (this._feedbackTimer) clearInterval(this._feedbackTimer); + this._feedbackTimer = null; + try { this._socket?.close(1000, 'client stopped'); } catch (_) { /* closed */ } + this._worker?.terminate(); + this._worker = null; + this.renderer.stop(); + } + + inspect() { + return { + mode: this.mode, + header: this.header && { + objects: this.header.objects.length, + fps: this.header.fps, + tiled: Boolean(this.header.tileAbr) + }, + stats: { ...this.stats, goodputMbps: this._goodput.mbps }, + renderer: this.renderer.stats, + viewer: this.header ? this.renderer.viewerState() : null, + awaitingKeyframe: this._awaitingKeyframe + }; + } +} + +/** Identity key for a payload, so the sync decoder callback can look it up. */ +let blobCounter = 0; +const blobKeys = new WeakMap(); +function blobKey(blob) { + if (!blobKeys.has(blob)) blobKeys.set(blob, ++blobCounter); + return blobKeys.get(blob); +} + +module.exports = { BaselineClient, GoodputMeter }; diff --git a/open4d/streaming/system/WebClient/src/baseline-main.js b/open4d/streaming/system/WebClient/src/baseline-main.js new file mode 100644 index 00000000..4dc96a1c --- /dev/null +++ b/open4d/streaming/system/WebClient/src/baseline-main.js @@ -0,0 +1,147 @@ +'use strict'; + +/** + * Entry point for the V4DS baseline viewer. + * + * Separate from `main.js` because the two are genuinely different clients: that + * one drives our HTTP segment ladder through ClientCore, this one consumes a + * pushed socket stream. Sharing an entry would mean a mode flag that changes + * almost everything. + * + * /web/baseline.html?bridge=ws://host:8790 + * + * | parameter | default | meaning | + * |-----------|----------------------------|--------------------------------| + * | bridge | ws://:8790 | v4ds-bridge WebSocket address | + * | pointSize | 0.012 | point size in metres | + * | strict | 0 | 1 = abort on a frame gap | + */ + +const { BaselineClient } = require('./baseline-client'); +const { mountLinkRate } = require('./link-rate'); + +function readConfig() { + const params = new URLSearchParams(window.location.search); + const host = window.location.hostname || '127.0.0.1'; + const number = (name, fallback) => { + const value = Number(params.get(name)); + return params.get(name) !== null && Number.isFinite(value) ? value : fallback; + }; + return { + bridgeUrl: params.get('bridge') || `ws://${host}:8790`, + pointSize: number('pointSize', 0.012), + strictOrder: params.get('strict') === '1' + }; +} + +function createUi() { + const status = document.getElementById('status'); + const logPane = document.getElementById('log'); + const statsPane = document.getElementById('stats'); + return { + setStatus: text => { status.textContent = text; }, + log(level, message) { + const row = document.createElement('div'); + row.className = `log-line log-${level}`; + row.textContent = `[${new Date().toISOString().slice(11, 23)}] ${message}`; + logPane.appendChild(row); + while (logPane.childElementCount > 300) { + logPane.removeChild(logPane.firstChild); + } + logPane.scrollTop = logPane.scrollHeight; + }, + setStats(lines) { statsPane.textContent = lines.join('\n'); } + }; +} + +async function main() { + const config = readConfig(); + const ui = createUi(); + const stopLinkRate = mountLinkRate(document.getElementById('link')); + ui.setStatus(`connecting to ${config.bridgeUrl} …`); + + const client = new BaselineClient({ + bridgeUrl: config.bridgeUrl, + canvas: document.getElementById('view'), + pointSize: config.pointSize, + strictOrder: config.strictOrder, + onEvent: event => { + switch (event.type) { + case 'open': + ui.log('info', `bridge connected: ${event.url}`); + ui.setStatus('waiting for the stream header …'); + break; + case 'header': + ui.log('info', `${event.mode.toUpperCase()} · ` + + `${event.objects.length} objects · ${event.fps} fps · ` + + `source ${event.source}` + + (event.tiled ? ' · tiled/ABR' : '')); + for (const o of event.objects) { + ui.log('info', ` object ${o.objectId} ${o.name}: ` + + `${o.cameras} cameras, ${o.loopFrames} frames`); + } + ui.setStatus(`streaming ${event.mode}`); + break; + case 'resync': + ui.log('warn', `resynchronising: ${event.message}`); + break; + case 'unsupported': + ui.log('warn', event.message); + break; + case 'error': + ui.log('error', event.message); + break; + case 'closed': + ui.log('warn', `bridge closed (${event.code}): ${event.reason}`); + ui.setStatus('disconnected'); + break; + default: + break; + } + } + }); + + window.__vs4dBaseline = client; + + document.getElementById('stop').addEventListener('click', () => { + stopLinkRate(); + client.stop(); + ui.setStatus('stopped'); + document.getElementById('stop').disabled = true; + }); + + setInterval(() => { + const info = client.inspect(); + const s = info.stats; + // Four lines. Frame/resync/drop counters and byte totals were useful + // while bringing the protocol up and are noise now; they remain on + // window.__vs4dBaseline. Faults appear only when non-zero, so a clean + // run stays quiet and a broken one still says so. + const faults = [ + s.decodeFailures ? `${s.decodeFailures} decode` : null, + s.resyncs ? `${s.resyncs} resync` : null, + s.framesDropped ? `${s.framesDropped} dropped` : null + ].filter(Boolean); + ui.setStats([ + `system ${info.mode ?? '-'}`, + `rate ${s.measuredFps.toFixed(1)} fps`, + `goodput ${s.goodputMbps ? s.goodputMbps.toFixed(1) + ' Mbps' : '-'}`, + `points ${info.renderer.objects + .reduce((sum, o) => sum + o.points, 0).toLocaleString()}`, + ...(faults.length ? [`FAULTS ${faults.join(', ')}`] : []) + ]); + }, 500); + + try { + await client.start(); + } catch (err) { + ui.setStatus(`could not connect: ${err.message}`); + ui.log('error', 'Is the bridge running? ' + + 'node system/WebClient/bridge/v4ds-bridge.js --baseline-port '); + } +} + +main().catch(err => { + document.getElementById('status').textContent = `fatal: ${err.message}`; + console.error(err); +}); diff --git a/open4d/streaming/system/WebClient/src/browser-platform.js b/open4d/streaming/system/WebClient/src/browser-platform.js new file mode 100644 index 00000000..b88a4986 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/browser-platform.js @@ -0,0 +1,494 @@ +'use strict'; + +/** + * Browser implementation of the ClientPlatform contract + * (../../ClientCore/platform.js). + * + * Six of the seven capabilities live here; the renderer is ./webgl-renderer.js + * because it is much larger and needs a GPU. All streaming logic is in + * ClientCore and is shared verbatim with the Node desktop client. + * + * Browser globals are reached through an injected `env` rather than captured at + * module scope, so this file can be exercised in Node against stubs — which is + * how tests/test_browser_platform.js verifies the cache policy, the OPFS paths + * and the non-blocking telemetry buffer without a browser. + */ + +/** Default environment: the real browser globals. */ +function defaultEnv() { + return { + fetch: typeof fetch === 'function' ? fetch.bind(globalThis) : undefined, + navigator: typeof navigator !== 'undefined' ? navigator : undefined, + document: typeof document !== 'undefined' ? document : undefined, + console: typeof console !== 'undefined' ? console : undefined, + setInterval: globalThis.setInterval?.bind(globalThis), + clearInterval: globalThis.clearInterval?.bind(globalThis), + setTimeout: globalThis.setTimeout?.bind(globalThis), + now: () => Date.now(), + URL: globalThis.URL, + Blob: globalThis.Blob + }; +} + +// -------------------------------------------------------------------------- +// Asset stores +// -------------------------------------------------------------------------- + +/** + * Downloaded media held as compressed bytes in memory. + * + * This is the default, and it is the right default: what blows up a browser's + * memory is DECODED frames (a 1920-wide texture frame is 5.5 MB), not the + * compressed segment, which is tens of MB for a whole scene. The decode cache + * is the renderer's problem; this only has to hold what arrived off the wire + * until the renderer has consumed it. + */ +class MemoryAssetStore { + constructor() { + this.buffers = new Map(); // handle -> ArrayBuffer + } + + async put(handle, buffer) { this.buffers.set(handle, buffer); } + + /** The renderer reads bytes back out by handle. */ + get(handle) { return this.buffers.get(handle) || null; } + + has(handle) { return this.buffers.has(handle); } + + /** Drop everything under a handle prefix (an object-segment directory). */ + async releasePrefix(prefix) { + for (const key of [...this.buffers.keys()]) { + if (key === prefix || key.startsWith(`${prefix}/`)) { + this.buffers.delete(key); + } + } + } + + get byteLength() { + let total = 0; + for (const buffer of this.buffers.values()) total += buffer.byteLength || 0; + return total; + } +} + +/** + * Downloaded media written to the Origin Private File System. + * + * Slower than memory but survives a reload and does not compete with the + * decoder for heap. Note that `createSyncAccessHandle` is worker-only; this uses + * the async writable-stream API so it works on the main thread. + */ +class OpfsAssetStore { + constructor(rootDirectory) { + this.root = rootDirectory; + } + + static async create(env, name) { + const opfsRoot = await env.navigator.storage.getDirectory(); + const directory = await opfsRoot.getDirectoryHandle(name, { create: true }); + return new OpfsAssetStore(directory); + } + + async _dirFor(parts, { create }) { + let directory = this.root; + for (const part of parts) { + directory = await directory.getDirectoryHandle(part, { create }); + } + return directory; + } + + async put(handle, buffer) { + const parts = handle.split('/').filter(Boolean); + const filename = parts.pop(); + const directory = await this._dirFor(parts, { create: true }); + const fileHandle = await directory.getFileHandle(filename, { create: true }); + const writable = await fileHandle.createWritable(); + await writable.write(buffer); + await writable.close(); + } + + async get(handle) { + const parts = handle.split('/').filter(Boolean); + const filename = parts.pop(); + try { + const directory = await this._dirFor(parts, { create: false }); + const fileHandle = await directory.getFileHandle(filename); + return await (await fileHandle.getFile()).arrayBuffer(); + } catch (_) { + return null; + } + } + + async releasePrefix(prefix) { + const parts = prefix.split('/').filter(Boolean); + const name = parts.pop(); + try { + const directory = await this._dirFor(parts, { create: false }); + await directory.removeEntry(name, { recursive: true }); + } catch (_) { + // Contract: release must tolerate a handle that was never created. + } + } +} + +// -------------------------------------------------------------------------- +// Transport +// -------------------------------------------------------------------------- + +function createTransport({ serverUrl, assetStore, env }) { + async function call(method, apiPath, body) { + const res = await env.fetch(`${serverUrl}${apiPath}`, { + method, + // API responses must never come from the HTTP cache: the manifest + // changes every segment and a cached menu would silently pin the + // client to a stale ladder. + cache: 'no-store', + ...(body === undefined || body === null ? {} : { + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body) + }) + }); + let parsed = null; + try { + parsed = await res.json(); + } catch (_) { + // Not every endpoint answers with JSON; telemetry posts do not. + } + return { ok: res.ok, status: res.status, body: parsed }; + } + + return { + getJson: apiPath => call('GET', apiPath, null), + postJson: (apiPath, body) => call('POST', apiPath, body), + + assetUrl(assetPath) { + if (!assetPath) return null; + if (/^https?:\/\//.test(assetPath)) return assetPath; + if (assetPath.startsWith('/files/')) return `${serverUrl}${assetPath}`; + const match = assetPath.match(/files\/(.+)/); + return match ? `${serverUrl}/files/${match[1]}` : null; + }, + + /** + * Fetch one media file. Never rejects: a failed asset is normal and is + * judged by download-plan's 90% rules. + * + * `cache: 'no-store'` is REQUIRED, not a nicety. The segment must keep + * generating real network load or the shaped-bandwidth experiment stops + * meaning anything — the same reason system/Client/decode_cache.py + * caches decodes but deliberately never caches downloads. A browser + * quietly serving a segment from its HTTP cache turns a bandwidth + * measurement into fiction. + */ + async fetchAsset(url, handle) { + const start = env.now(); + try { + const res = await env.fetch(url, { cache: 'no-store' }); + if (!res.ok) { + return { + success: false, size: 0, timeMs: 0, error: `HTTP ${res.status}` + }; + } + const buffer = await res.arrayBuffer(); + // A null handle means "count the bytes, keep nothing". + if (handle) await assetStore.put(handle, buffer); + return { + success: true, + size: buffer.byteLength, + timeMs: env.now() - start + }; + } catch (err) { + return { + success: false, size: 0, + timeMs: env.now() - start, error: err.message + }; + } + } + }; +} + +// -------------------------------------------------------------------------- +// Storage +// -------------------------------------------------------------------------- + +/** + * Buffered line sink. + * + * `write` MUST NOT block: render telemetry arrives at 30 Hz, and in Node a + * synchronous write per frame stalled the event loop badly enough to starve the + * renderer. Lines accumulate in an array and are flushed on an interval, so the + * hot path is one array push. + */ +class BufferedLineSink { + constructor({ name, flush, env, flushIntervalMs = 2000 }) { + this.name = name; + this.lines = []; + this._pending = []; + this._flush = flush; + this._env = env; + this._closed = false; + this._timer = flush + ? env.setInterval(() => { this._drain(); }, flushIntervalMs) + : null; + } + + write(line) { + if (this._closed) throw new Error(`write after close on ${this.name}`); + this.lines.push(line); + this._pending.push(line); + } + + _drain() { + if (this._pending.length === 0) return Promise.resolve(); + const batch = this._pending.splice(0, this._pending.length); + // The try/catch is load-bearing: a SYNCHRONOUS throw from the flush + // (an OPFS quota error, say) escapes before Promise.resolve can wrap + // it, so a trailing .catch alone would let it reject close() — which + // finish() awaits, taking down the metrics upload with it. Telemetry + // must never take the run down. + try { + return Promise.resolve(this._flush(batch)).catch(() => {}); + } catch (_) { + return Promise.resolve(); + } + } + + async close() { + this._closed = true; + if (this._timer) this._env.clearInterval(this._timer); + await this._drain(); + } + + text() { return this.lines.join('\n') + (this.lines.length ? '\n' : ''); } +} + +/** + * @param {object} args + * @param {MemoryAssetStore|OpfsAssetStore} args.assetStore + * @param {object} args.env + * @param {(filename: string, text: string) => void} [args.onArtifact] + * Called with the run's result and telemetry so the page can offer them as + * downloads. The browser has nowhere to "write a file" unprompted. + */ +function createStorage({ assetStore, env, onArtifact = null }) { + const artifacts = new Map(); // filename -> text + const sinks = new Map(); // name -> BufferedLineSink + let scratchCount = 0; + + return { + artifacts, + sinks, + + handle: (...parts) => parts.filter(p => p != null).join('/'), + + async createScratch() { + return `run-${++scratchCount}`; + }, + + async release(handle) { + await assetStore.releasePrefix(handle); + }, + + async writeText(name, text) { + artifacts.set(name, text); + onArtifact?.(name, text); + }, + + async writeResult(text) { + artifacts.set('metrics.json', text); + onArtifact?.('metrics.json', text); + }, + + async openAppendStream(name) { + const sink = new BufferedLineSink({ + name, + env, + // Keep the accumulated text available as a downloadable + // artifact; flushing to OPFS as well would double-store it for + // no benefit at these sizes. + flush: () => { artifacts.set(name, sink.text()); } + }); + sinks.set(name, sink); + return sink; + } + }; +} + +// -------------------------------------------------------------------------- +// Viewpoints +// -------------------------------------------------------------------------- + +/** + * Where the initial camera pose comes from. + * + * `initialPose` short-circuits the fetch, which is what an interactive page + * does: the user's camera is the pose, and the canned list only matters in + * simulated mode. + * + * Rejects when nothing is available, per the contract — a run with no initial + * pose would solve the first ladder against a default camera and silently + * invalidate the viewpoint-aware comparison. + */ +function createViewpoints({ serverUrl, indexPath, initialPose, env }) { + return { + async list() { + if (initialPose) { + return [{ filename: 'browser-initial-pose', data: initialPose }]; + } + if (!indexPath) { + throw new Error( + 'no viewpoints available: pass initialPose or viewpointIndexPath'); + } + const res = await env.fetch(`${serverUrl}${indexPath}`, + { cache: 'no-store' }); + if (!res.ok) { + throw new Error(`viewpoint index fetch failed: HTTP ${res.status}`); + } + const body = await res.json(); + const list = Array.isArray(body) ? body : body?.viewpoints; + if (!Array.isArray(list) || list.length === 0) { + throw new Error('viewpoint index contained no poses'); + } + return list.map((entry, index) => ({ + filename: entry.filename || `view_${String(index).padStart(2, '0')}.json`, + data: entry.data || entry + })); + } + }; +} + +// -------------------------------------------------------------------------- +// Clock, logger, lifecycle +// -------------------------------------------------------------------------- + +function createClock(env) { + return { + now: () => env.now(), + every: (ms, fn) => env.setInterval(fn, ms), + cancel: handle => env.clearInterval(handle), + delay: ms => new Promise(resolve => env.setTimeout(resolve, ms)) + }; +} + +/** + * @param {object} args + * @param {(entry: {level: string, line: string, data: object|null}) => void} [args.onLine] + * Page sink, e.g. an on-screen log pane. + * @param {number} [args.keep] ring-buffer size held for inspection + */ +function createLogger({ env, onLine = null, keep = 500 }) { + const lines = []; + return { + lines, + emit(level, line, data) { + lines.push({ level, line, data }); + if (lines.length > keep) lines.shift(); + if (data) env.console?.log(line, data); + else env.console?.log(line); + onLine?.({ level, line, data }); + } + }; +} + +/** + * Page lifecycle. + * + * `exit` MUST NOT navigate away. The final POST /api/results happens during + * shutdown, and unloading the page cancels it — the run's metrics would be lost + * exactly when they matter. So this resolves a promise the page can await and + * leaves the document alone. + */ +function createLifecycle({ env } = { env: defaultEnv() }) { + let resolveDone; + const done = new Promise(resolve => { resolveDone = resolve; }); + const handlers = []; + + return { + /** Resolves with the exit code once the run has finalized. */ + done, + exitCode: null, + + exit(code) { + this.exitCode = code; + resolveDone(code); + }, + + onShutdownRequest(fn) { + handlers.push(fn); + }, + + /** Wire a Stop control / pagehide to the registered finalizers. */ + requestShutdown() { + for (const fn of handlers) fn(); + }, + + get handlerCount() { return handlers.length; } + }; +} + +// -------------------------------------------------------------------------- +// Assembly +// -------------------------------------------------------------------------- + +/** + * Build the browser platform. + * + * @param {object} args + * @param {string} args.serverUrl + * @param {object|null} [args.renderer] a RendererAdapter, or null for simulated mode + * @param {object} [args.initialPose] + * @param {string} [args.viewpointIndexPath] + * @param {'memory'|'opfs'} [args.storageMode='memory'] + * @param {Function} [args.onArtifact] + * @param {Function} [args.onLogLine] + * @param {object} [args.env] injected globals, for tests + * @returns {Promise} the platform, plus `assetStore` and `lifecycle` + */ +async function createBrowserPlatform({ + serverUrl, + renderer = null, + initialPose = null, + viewpointIndexPath = null, + storageMode = 'memory', + onArtifact = null, + onLogLine = null, + env = defaultEnv() +}) { + if (!serverUrl) throw new Error('serverUrl is required'); + if (typeof env.fetch !== 'function') { + throw new Error('this environment has no fetch()'); + } + + const assetStore = storageMode === 'opfs' + ? await OpfsAssetStore.create(env, 'vs4d-client') + : new MemoryAssetStore(); + + return { + transport: createTransport({ serverUrl, assetStore, env }), + storage: createStorage({ assetStore, env, onArtifact }), + viewpoints: createViewpoints({ + serverUrl, indexPath: viewpointIndexPath, initialPose, env + }), + clock: createClock(env), + logger: createLogger({ env, onLine: onLogLine }), + renderer, + lifecycle: createLifecycle({ env }), + // Not part of the contract; the renderer needs to read downloaded bytes + // back out by handle. + assetStore + }; +} + +module.exports = { + createBrowserPlatform, + createTransport, + createStorage, + createViewpoints, + createClock, + createLogger, + createLifecycle, + MemoryAssetStore, + OpfsAssetStore, + BufferedLineSink, + defaultEnv +}; diff --git a/open4d/streaming/system/WebClient/src/camera-pose.js b/open4d/streaming/system/WebClient/src/camera-pose.js new file mode 100644 index 00000000..fb1b0689 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/camera-pose.js @@ -0,0 +1,169 @@ +'use strict'; + +/** + * Camera pose conversion: WebGL/Three.js camera -> Open3D + * `PinholeCameraParameters`. + * + * This is not a convenience. What the client POSTs to `/api/viewpoint` is + * written to disk verbatim and read back by + * `o3d.io.read_pinhole_camera_parameters` (see + * `vstream/ladder/ladder_service.py` and `create_ladder.py`), so the JSON must + * be exactly that schema or the ladder cannot parse the pose at all. When it + * cannot, `normalize_weights` necessarily returns equal weights and the ladder + * silently stops being viewpoint-aware — which is the entire contribution being + * measured. A wrong pose here does not crash anything; it quietly invalidates + * the experiment. + * + * Two conventions have to be bridged: + * + * 1. AXES. Three.js cameras look down -Z with +Y up. Open3D's camera frame is + * OpenCV-style: +X right, +Y DOWN, +Z FORWARD. So the world->camera matrix + * gets its Y and Z rows negated. The captured viewpoint files show this + * directly — their extrinsic diagonal is roughly (+1, -1, -1). + * + * 2. UNITS. `vstream/config.py` is explicit: "Baked OBJ/Draco coordinates are + * metres, while Open3D camera extrinsics and the quality model's + * mean-distance feature are millimetres", and + * VIEW_RAYCAST_UNITS_PER_METER defaults to 1000. The captured files agree — + * their translations are in the thousands. A browser scene in metres must + * therefore scale its translation by 1000, or the server raycasts from + * ~4 mm away, every ray misses, and the weights come back uniform. + * + * Only the translation scales. The extrinsic maps world points to camera + * space as `p_cam = R*p_world + t` with `t = -R*C`, so re-expressing the + * same pose in a millimetre world gives `t_mm = 1000 * t_m` and leaves R + * untouched. + */ + +/** Open3D stores the principal point at the pixel-grid centre. */ +function principalPoint(size) { + return (size - 1) / 2; +} + +/** + * Vertical focal length in pixels for a vertical field of view. + * + * Cross-check against the captured corpus: a 60 degree vertical FOV at 1920 px + * gives 960 / tan(30 deg) = 1662.7687752661222, which is exactly the value in + * `system/Client/viewpoints/view_00.json`. + */ +function focalLengthPx(fovDegrees, height) { + const fovRadians = (fovDegrees * Math.PI) / 180; + return (height / 2) / Math.tan(fovRadians / 2); +} + +/** + * Build an Open3D `PinholeCameraParameters` object. + * + * @param {object} args + * @param {number[]} args.viewMatrix world->camera matrix, COLUMN-major, 16 + * elements. In Three.js this is `camera.matrixWorldInverse.elements`, which is + * already column-major. + * @param {number} args.fovDegrees vertical field of view + * @param {number} args.width viewport width in pixels + * @param {number} args.height viewport height in pixels + * @param {number} [args.unitsPerMeter=1000] world units per metre on the + * SERVER side; must match config.VIEW_RAYCAST_UNITS_PER_METER + * @returns {object} JSON-ready PinholeCameraParameters + */ +function toOpen3DCameraParameters({ + viewMatrix, fovDegrees, width, height, unitsPerMeter = 1000 +}) { + if (!Array.isArray(viewMatrix) && !(viewMatrix instanceof Float32Array) + && !(viewMatrix instanceof Float64Array)) { + throw new TypeError('viewMatrix must be an array of 16 numbers'); + } + if (viewMatrix.length !== 16) { + throw new RangeError(`viewMatrix must have 16 elements, got ${viewMatrix.length}`); + } + if (!(width > 0) || !(height > 0)) { + throw new RangeError('width and height must be positive'); + } + if (!(fovDegrees > 0) || fovDegrees >= 180) { + throw new RangeError(`fovDegrees out of range: ${fovDegrees}`); + } + + // Column-major indexing: element(row, col) = viewMatrix[col * 4 + row]. + // Negating rows 1 and 2 applies diag(1, -1, -1, 1) on the left, which is + // the Y-up/-Z-forward -> Y-down/+Z-forward change of basis. + const extrinsic = new Array(16); + for (let col = 0; col < 4; col++) { + for (let row = 0; row < 4; row++) { + const index = col * 4 + row; + const flip = (row === 1 || row === 2) ? -1 : 1; + // `+ 0` normalizes -0 away. Negating a zero row entry yields -0, + // which is numerically harmless but does not survive a JSON round + // trip, so it would make the emitted pose fail an equality check + // against itself. + extrinsic[index] = flip * viewMatrix[index] + 0; + } + } + // Translation is the last column; scale it into the server's world units. + extrinsic[12] *= unitsPerMeter; + extrinsic[13] *= unitsPerMeter; + extrinsic[14] *= unitsPerMeter; + + const focal = focalLengthPx(fovDegrees, height); + + return { + class_name: 'PinholeCameraParameters', + extrinsic, + intrinsic: { + height, + // Column-major 3x3: [fx, 0, 0, 0, fy, 0, cx, cy, 1]. + // Square pixels: Three.js takes a vertical FOV and derives the + // horizontal extent from the aspect ratio, so fx == fy. + intrinsic_matrix: [ + focal, 0, 0, + 0, focal, 0, + principalPoint(width), principalPoint(height), 1 + ], + width + }, + version_major: 1, + version_minor: 0 + }; +} + +/** + * Convenience wrapper for a Three.js PerspectiveCamera. + * + * `updateMatrixWorld` then `matrixWorldInverse` rather than trusting whatever + * the render loop last computed: the pose is read on the segment tick, which is + * not synchronised with a frame. + */ +function fromThreeCamera(camera, { width, height, unitsPerMeter = 1000 }) { + camera.updateMatrixWorld(); + camera.updateProjectionMatrix(); + return toOpen3DCameraParameters({ + viewMatrix: Array.from(camera.matrixWorldInverse.elements), + fovDegrees: camera.fov, + width, + height, + unitsPerMeter + }); +} + +/** + * Map a world point into the camera frame described by these parameters. + * Used by the tests to assert the axis convention: a point the camera is + * looking at must land at POSITIVE z. + */ +function projectToCameraSpace(parameters, worldPointMetres) { + const e = parameters.extrinsic; + const scale = 1000; // parameters are in the server's units + const [x, y, z] = worldPointMetres.map(v => v * scale); + return [ + e[0] * x + e[4] * y + e[8] * z + e[12], + e[1] * x + e[5] * y + e[9] * z + e[13], + e[2] * x + e[6] * y + e[10] * z + e[14] + ]; +} + +module.exports = { + toOpen3DCameraParameters, + fromThreeCamera, + focalLengthPx, + principalPoint, + projectToCameraSpace +}; diff --git a/open4d/streaming/system/WebClient/src/chooser.js b/open4d/streaming/system/WebClient/src/chooser.js new file mode 100644 index 00000000..505ae0ac --- /dev/null +++ b/open4d/streaming/system/WebClient/src/chooser.js @@ -0,0 +1,227 @@ +'use strict'; + +/** + * The launcher: one row per system, so the list stays readable as methods are + * added. + * + * Each row is a link. Selecting objects is behind a per-row toggle rather than + * inline, because the object list is the one thing here that does not scale — + * nine rows of checkboxes under every method would bury the list it belongs to. + * + * Why an object picker exists at all: the ladder must publish at least one + * representation per object in the scene, so the scene sets an irreducible + * bitrate floor. All nine ORBIT objects floor at ~116 Mbps, and under that + * floor the MCKP can only buy the few highest-weighted objects at their + * cheapest rung and freezes the rest — the page then looks permanently starved + * however good the link is. Three objects floor near 21 Mbps. So each picker + * defaults to the cheapest few rather than leaving a viewer to discover the + * deficit, and Vega's defaults to the smallest clips because that page + * preloads whole clips before it plays them. + */ + +// One line per system, carrying the one thing a screenshot cannot show: +// whether it adapts, and where. A fixed-quality player and an adaptive one +// look identical on a fast link. +const DESCRIPTION = { + mesh: 'Textured meshes. Adapts in this browser, one representation per ' + + 'object per segment.', + vivo: 'Point clouds. Adapts server-side, per spatial tile.', + nava: 'Point clouds. Adapts server-side, one quality per object per segment.', + vega: '3D Gaussian splats. Fixed quality, no adaptation.', + nevo: 'Neural volumetric. Pre-rendered comparison panels.' +}; + +function el(tag, props = {}, children = []) { + const node = document.createElement(tag); + for (const [key, value] of Object.entries(props)) { + if (key === 'class') node.className = value; + else if (key === 'text') node.textContent = value; + else if (key.startsWith('on')) node.addEventListener(key.slice(2), value); + else node.setAttribute(key, value); + } + for (const child of [].concat(children)) if (child) node.appendChild(child); + return node; +} + +function mbps(value) { + return Number.isFinite(value) ? `${value.toFixed(1)} Mbps` : '—'; +} + +/** Where a row points, including whatever the page needs to find its data. */ +function launchUrl(system, objects) { + const url = new URL(system.page, window.location.origin); + if (objects && objects.length) { + url.searchParams.set('objects', objects.join(',')); + } + // Each point-cloud baseline has its own bridge, so the address is what + // selects ViVo vs NAVA — the page itself is the same. + if (system.bridge) url.searchParams.set('bridge', system.bridge); + return url.toString(); +} + +/** + * Which objects a system offers, and a sensible default selection. + * `cost` labels the single numeric column: bitrate for a ladder, size for a + * preloading viewer. + */ +function objectChoice(system) { + const objects = [...(system.objects || [])]; + if (!objects.length) return null; + + if (system.id === 'vega') { // sized by download, not by bitrate + return { + objects, + head: 'clip', + cell: o => `${(o.bytes / 1e6).toFixed(0)} MB`, + defaults: [...objects].sort((a, b) => a.bytes - b.bytes) + .slice(0, 2).map(o => o.name) + }; + } + const priced = objects.filter(o => o.floorMbps !== null); + return { + objects: objects.sort((a, b) => (b.weight || 0) - (a.weight || 0) + || a.name.localeCompare(b.name)), + head: 'floor', + cell: o => mbps(o.floorMbps), + defaults: (priced.length ? priced : objects).slice() + .sort((a, b) => (a.floorMbps ?? Infinity) - (b.floorMbps ?? Infinity)) + .slice(0, 3).map(o => o.name) + }; +} + +/** + * Ask the server to (re)start a point-cloud baseline with this object set. + * + * Needed because these baselines fix their scene with `--objects` at startup, + * so unlike the others the selection cannot be a query parameter — it is a new + * process. The server validates every name against the tile catalogue and + * waits until both the bridge and the baseline are listening before replying, + * so a resolved promise means the page can actually connect. + */ +async function restartBaseline(system, objects) { + const response = await fetch('/api/pointcloud/start', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + cache: 'no-store', + body: JSON.stringify({ id: system.id, objects }) + }); + const body = await response.json().catch(() => ({})); + if (!response.ok) throw new Error(body.error || `HTTP ${response.status}`); + return body; +} + +/** One row: name, description, and an optional collapsed object picker. */ +function renderRow(system) { + const choice = objectChoice(system); + const chosen = new Set(choice ? choice.defaults : []); + const status = document.getElementById('status'); + + const link = el('a', { + class: 'name', + text: system.name, + // `detail` explains a system that cannot run. It is not on the row — + // that is what keeps the list scannable — but it is one hover away, + // and /api/systems still reports `ready`. + title: system.detail || '' + }); + const sync = () => { link.href = launchUrl(system, [...chosen]); }; + sync(); + + if (system.restartable) { + // Starting the process takes ~15 s, so navigating first would land on + // a page that cannot connect yet. Hold the click, start it, then go. + link.addEventListener('click', async (event) => { + if (event.metaKey || event.ctrlKey || event.button !== 0) return; + event.preventDefault(); + const objects = [...chosen]; + if (!objects.length) { + status.textContent = `${system.name}: choose at least one object`; + status.className = 'bad'; + return; + } + link.classList.add('busy'); + status.className = ''; + status.textContent = `starting ${system.name} with ` + + `${objects.join(', ')} …`; + try { + const started = await restartBaseline(system, objects); + window.location.href = launchUrl( + { ...system, bridge: started.bridge }, []); + } catch (error) { + status.textContent = `${system.name}: ${error.message}`; + status.className = 'bad'; + link.classList.remove('busy'); + } + }); + } + + const row = el('li', { class: 'row' }, [ + link, + el('span', { class: 'desc', text: DESCRIPTION[system.id] || '' }) + ]); + if (!choice) return row; + + const picker = el('div', { class: 'picker', hidden: 'hidden' }); + const toggle = el('button', { + class: 'toggle', + text: `${chosen.size} of ${choice.objects.length}`, + onclick: () => { + const open = picker.hasAttribute('hidden'); + if (open) picker.removeAttribute('hidden'); + else picker.setAttribute('hidden', 'hidden'); + toggle.classList.toggle('open', open); + } + }); + row.appendChild(toggle); + row.appendChild(picker); + + const table = el('table', {}, [el('tbody')]); + const body = table.querySelector('tbody'); + for (const object of choice.objects) { + const box = el('input', { type: 'checkbox' }); + box.checked = chosen.has(object.name); + const tr = el('tr', {}, [ + el('td', {}, box), + el('td', { text: object.name }), + el('td', { class: 'num', text: choice.cell(object) }) + ]); + box.addEventListener('change', () => { + if (box.checked) chosen.add(object.name); + else chosen.delete(object.name); + tr.classList.toggle('off', !box.checked); + toggle.textContent = `${chosen.size} of ${choice.objects.length}`; + sync(); + }); + tr.classList.toggle('off', !box.checked); + body.appendChild(tr); + } + picker.appendChild(table); + return row; +} + +async function main() { + const root = document.getElementById('systems'); + const status = document.getElementById('status'); + let payload; + try { + const response = await fetch('/api/systems', { cache: 'no-store' }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + payload = await response.json(); + } catch (error) { + status.textContent = `could not reach /api/systems: ${error.message}`; + status.className = 'bad'; + return; + } + status.textContent = ''; + + const list = el('ul', { class: 'systems' }); + for (const system of payload.systems || []) list.appendChild(renderRow(system)); + root.appendChild(list); +} + +main().catch(error => { + const status = document.getElementById('status'); + if (status) status.textContent = `fatal: ${error.message}`; + // eslint-disable-next-line no-console + console.error(error); +}); diff --git a/open4d/streaming/system/WebClient/src/decode-cache.js b/open4d/streaming/system/WebClient/src/decode-cache.js new file mode 100644 index 00000000..b67dac8f --- /dev/null +++ b/open4d/streaming/system/WebClient/src/decode-cache.js @@ -0,0 +1,166 @@ +'use strict'; + +/** + * Decoded-clip cache for the browser renderer. + * + * The browser analogue of system/Client/decode_cache.py, and it exists for the + * same measured reason. The desktop client originally re-ran the Draco decoder + * over 60 frames plus a texture decode for EVERY object of EVERY segment — an + * order of magnitude more work than a 2 s segment budget allows. Objects then + * never became playable and were reported missing. The browser has strictly + * less decode headroom, so this is built in from the start rather than added + * after the first starved run. + * + * `(objectName, repId)` is a COMPLETE cache key. Media is a fixed + * FRAMES_PER_SEGMENT-frame loop published once under + * `files/media///` and shared by every logical segment + * (ladderlib.build_mpd writes `source_segment: 0, loop: true`), so a clip + * decoded during segment 3 is still correct in segment 88. + * + * What is deliberately NOT cached: the downloads. The segment must keep + * generating real network load or the shaped-bandwidth experiment stops meaning + * anything. Only the decode is amortized. + * + * Eviction is by byte budget, least-recently-used first, because decoded frames + * are what actually exhaust a browser tab: a 1920-wide texture frame is + * w*h*1.5 = 5.5 MB however it is stored, so one 60-frame clip for five objects + * is 1.66 GB. `vstream/config.py` caps published texture width for the same + * reason. + */ + +const DEFAULT_BUDGET_BYTES = 512 * 1024 * 1024; + +/** `(objectName, repId)` -> cache key. */ +function clipKey(objectName, repId) { + return `${objectName}::${repId}`; +} + +class DecodeCache { + /** + * @param {object} [options] + * @param {number} [options.budgetBytes] evict once decoded bytes exceed this + * @param {(clip: object) => void} [options.onEvict] release GPU resources + * @param {() => number} [options.now] clock, for LRU ordering and tests + */ + constructor({ + budgetBytes = DEFAULT_BUDGET_BYTES, + onEvict = null, + now = () => Date.now() + } = {}) { + this.budgetBytes = budgetBytes; + this._onEvict = onEvict; + this._now = now; + /** key -> { clip, bytes, lastUsed, pinned } */ + this._entries = new Map(); + /** key -> Promise, so concurrent requests share one decode. */ + this._inFlight = new Map(); + this.stats = { hits: 0, misses: 0, shared: 0, evictions: 0 }; + } + + get bytes() { + let total = 0; + for (const entry of this._entries.values()) total += entry.bytes; + return total; + } + + get size() { return this._entries.size; } + + has(objectName, repId) { return this._entries.has(clipKey(objectName, repId)); } + + /** + * Fetch a decoded clip, decoding it at most once. + * + * Concurrent callers for the same key await the SAME decode rather than + * starting a second one. Without this, two segments selecting the same + * representation would each decode 60 frames — the exact duplication that + * `decodeShared` reports in the desktop client's telemetry. + * + * @param {string} objectName + * @param {string} repId + * @param {() => Promise<{clip: object, bytes: number}>} decode + * @returns {Promise<{clip: object, cacheHit: boolean, decodeShared: boolean}>} + */ + async get(objectName, repId, decode) { + const key = clipKey(objectName, repId); + + const entry = this._entries.get(key); + if (entry) { + entry.lastUsed = this._now(); + this.stats.hits++; + return { clip: entry.clip, cacheHit: true, decodeShared: false }; + } + + const pending = this._inFlight.get(key); + if (pending) { + this.stats.shared++; + const clip = await pending; + return { clip, cacheHit: false, decodeShared: true }; + } + + this.stats.misses++; + const work = (async () => { + const { clip, bytes } = await decode(); + this._entries.set(key, { + clip, bytes, lastUsed: this._now(), pinned: false + }); + this._evictToBudget(); + return clip; + })(); + this._inFlight.set(key, work); + try { + const clip = await work; + return { clip, cacheHit: false, decodeShared: false }; + } finally { + this._inFlight.delete(key); + } + } + + /** + * Protect a clip from eviction while it is on screen. + * + * Without pinning, a large scene can evict the clip the renderer is in the + * middle of presenting, which shows up as a frame reverting mid-playback + * rather than as an error. + */ + pin(objectName, repId) { + const entry = this._entries.get(clipKey(objectName, repId)); + if (entry) entry.pinned = true; + } + + unpin(objectName, repId) { + const entry = this._entries.get(clipKey(objectName, repId)); + if (entry) entry.pinned = false; + } + + /** Unpin every clip of an object except the one now in use. */ + pinOnly(objectName, repId) { + for (const [key, entry] of this._entries) { + if (key.startsWith(`${objectName}::`)) { + entry.pinned = (key === clipKey(objectName, repId)); + } + } + } + + _evictToBudget() { + if (this.bytes <= this.budgetBytes) return; + // Least-recently-used first, pinned clips last-resort only. + const candidates = [...this._entries.entries()] + .filter(([, entry]) => !entry.pinned) + .sort((a, b) => a[1].lastUsed - b[1].lastUsed); + + for (const [key, entry] of candidates) { + if (this.bytes <= this.budgetBytes) break; + this._entries.delete(key); + this.stats.evictions++; + this._onEvict?.(entry.clip); + } + } + + /** Drop everything, releasing GPU resources. */ + clear() { + for (const entry of this._entries.values()) this._onEvict?.(entry.clip); + this._entries.clear(); + } +} + +module.exports = { DecodeCache, clipKey, DEFAULT_BUDGET_BYTES }; diff --git a/open4d/streaming/system/WebClient/src/draco-worker.js b/open4d/streaming/system/WebClient/src/draco-worker.js new file mode 100644 index 00000000..04e0ade2 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/draco-worker.js @@ -0,0 +1,191 @@ +'use strict'; + +/** + * Draco decode worker. + * + * A segment is up to 60 Draco meshes per object. Decoding them on the main + * thread would compete with the render loop for exactly the window in which + * frames must be presented, so every decode happens here and only typed arrays + * cross back — transferred, not copied. + * + * Loads the official Draco JS/WASM decoder vendored under ../vendor/draco. + * That build is upstream's, unmodified. + * + * Handles BOTH Draco geometry kinds, because the two pipelines differ: + * + * TRIANGULAR_MESH the mesh ladder (`compress_geometry.py` -> textured OBJs) + * POINT_CLOUD every V4DS baseline (`draco_encoder -point_cloud -qp 11 + * -qg 8`), carrying POSITION plus 8-bit COLOR + * + * The kind is read from the payload rather than configured, so one worker + * serves both the mesh client and the baseline client. + * + * Protocol: + * in { id, buffers: ArrayBuffer[] } frames in order + * out { id, frames: [frame|null] } | { id, error } + * mesh frame = { kind: 'mesh', positions, normals, uvs, indices } + * cloud frame = { kind: 'cloud', positions, colors } + * + * A frame that fails to decode comes back as null in the array rather than + * failing the whole clip: download-plan already tolerates gaps, and the + * renderer holds the previous frame for one. + */ + +let decoderModulePromise = null; + +function loadDecoder(vendorBase) { + if (decoderModulePromise) return decoderModulePromise; + decoderModulePromise = new Promise((resolve, reject) => { + try { + self.importScripts(`${vendorBase}/draco_decoder.js`); + } catch (err) { + reject(new Error(`could not load draco_decoder.js: ${err.message}`)); + return; + } + // The emscripten build resolves its .wasm through locateFile. + self.DracoDecoderModule({ + locateFile: file => `${vendorBase}/${file}` + }).then(resolve, reject); + }); + return decoderModulePromise; +} + +/** + * Decode one Draco buffer into plain typed arrays. + * + * Every Draco object must be explicitly destroyed: the WASM heap is not + * garbage-collected from JS, and leaking one mesh per frame at 30 fps exhausts + * it within a minute. + */ +function decodeGeometry(draco, decoder, buffer) { + const dracoBuffer = new draco.DecoderBuffer(); + dracoBuffer.Init(new Int8Array(buffer), buffer.byteLength); + + let geometry = null; + try { + const geometryType = decoder.GetEncodedGeometryType(dracoBuffer); + + if (geometryType === draco.TRIANGULAR_MESH) { + geometry = new draco.Mesh(); + const status = decoder.DecodeBufferToMesh(dracoBuffer, geometry); + if (!status.ok() || geometry.ptr === 0) { + throw new Error(status.error_msg() || 'mesh decode failed'); + } + return { + kind: 'mesh', + positions: readFloatAttribute( + draco, decoder, geometry, draco.POSITION, 3), + normals: readFloatAttribute( + draco, decoder, geometry, draco.NORMAL, 3), + uvs: readFloatAttribute( + draco, decoder, geometry, draco.TEX_COORD, 2), + indices: readIndices(draco, decoder, geometry) + }; + } + + if (geometryType === draco.POINT_CLOUD) { + geometry = new draco.PointCloud(); + const status = decoder.DecodeBufferToPointCloud(dracoBuffer, geometry); + if (!status.ok() || geometry.ptr === 0) { + throw new Error(status.error_msg() || 'point cloud decode failed'); + } + return { + kind: 'cloud', + positions: readFloatAttribute( + draco, decoder, geometry, draco.POSITION, 3), + // Colours are quantised to 8 bits by the encoder's -qg 8, so + // read them as bytes. Reading them as floats yields 0..255 + // values that then have to be rediscovered downstream. + colors: readUint8Attribute( + draco, decoder, geometry, draco.COLOR, 3) + }; + } + + throw new Error(`unsupported draco geometry type ${geometryType}`); + } finally { + if (geometry) draco.destroy(geometry); + draco.destroy(dracoBuffer); + } +} + +function readFloatAttribute(draco, decoder, geometry, attributeType, components) { + const id = decoder.GetAttributeId(geometry, attributeType); + if (id < 0) return null; + const attribute = decoder.GetAttribute(geometry, id); + const count = geometry.num_points(); + const array = new draco.DracoFloat32Array(); + try { + decoder.GetAttributeFloatForAllPoints(geometry, attribute, array); + const out = new Float32Array(count * components); + for (let i = 0; i < out.length; i++) out[i] = array.GetValue(i); + return out; + } finally { + draco.destroy(array); + } +} + +function readUint8Attribute(draco, decoder, geometry, attributeType, components) { + const id = decoder.GetAttributeId(geometry, attributeType); + if (id < 0) return null; + const attribute = decoder.GetAttribute(geometry, id); + const count = geometry.num_points(); + const array = new draco.DracoUInt8Array(); + try { + decoder.GetAttributeUInt8ForAllPoints(geometry, attribute, array); + const out = new Uint8Array(count * components); + for (let i = 0; i < out.length; i++) out[i] = array.GetValue(i); + return out; + } finally { + draco.destroy(array); + } +} + +function readIndices(draco, decoder, mesh) { + const faceCount = mesh.num_faces(); + const out = new Uint32Array(faceCount * 3); + const face = new draco.DracoInt32Array(); + try { + for (let i = 0; i < faceCount; i++) { + decoder.GetFaceFromMesh(mesh, i, face); + out[i * 3] = face.GetValue(0); + out[i * 3 + 1] = face.GetValue(1); + out[i * 3 + 2] = face.GetValue(2); + } + return out; + } finally { + draco.destroy(face); + } +} + +self.onmessage = async event => { + const { id, buffers, vendorBase } = event.data; + try { + const draco = await loadDecoder(vendorBase || '/web/vendor/draco'); + const decoder = new draco.Decoder(); + const frames = []; + const transfer = []; + try { + for (const buffer of buffers) { + if (!buffer) { frames.push(null); continue; } + try { + const frame = decodeGeometry(draco, decoder, buffer); + frames.push(frame); + // Transfer rather than copy: a 60-frame clip is tens of MB. + for (const key of ['positions', 'normals', 'uvs', 'indices', + 'colors']) { + if (frame[key]) transfer.push(frame[key].buffer); + } + } catch (_) { + // One bad frame is a gap the renderer can hold through, not + // a reason to discard the whole object-segment. + frames.push(null); + } + } + } finally { + draco.destroy(decoder); + } + self.postMessage({ id, frames }, transfer); + } catch (err) { + self.postMessage({ id, error: err.message }); + } +}; diff --git a/open4d/streaming/system/WebClient/src/link-rate.js b/open4d/streaming/system/WebClient/src/link-rate.js new file mode 100644 index 00000000..59f3d7b7 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/link-rate.js @@ -0,0 +1,55 @@ +'use strict'; + +/** + * Live readout of the shaped link rate, polled from /api/shaping. + * + * The point of showing it is causality. A representation switch on its own is + * unreadable — it could be the ABR working or the ABR thrashing. Beside the + * rate the kernel is enforcing, the same switch becomes evidence. And when the + * link is unshaped this says so, which matters because an unshaped run looks + * like a broken adaptive system: every system just holds one operating point. + */ +function mountLinkRate(element, { fetchImpl = fetch, intervalMs = 2000 } = {}) { + if (!element) return () => {}; + let stopped = false; + + // Short text on screen, full detail in the tooltip. The distinctions still + // matter -- "cannot tell" must never read as "flat link" -- but they belong + // on hover rather than in the viewer's way. + const render = (state) => { + if (state.error) { + element.textContent = 'link: unknown'; + element.title = `could not read the shaped rate: ${state.error}`; + element.className = 'link unknown'; + return; + } + if (!state.shaped) { + element.textContent = 'link: unshaped'; + element.title = 'no root TBF installed, so nothing to adapt to — ' + + 'replay a trace with scripts/shape_web_demo.sh'; + element.className = 'link unshaped'; + return; + } + element.textContent = state.rateMbps === null + ? 'link: shaped' + : `link: ${state.rateMbps.toFixed(1)} Mbps`; + element.title = `enforced by tc on ${state.interface}`; + element.className = 'link shaped'; + }; + + const tick = async () => { + try { + const response = await fetchImpl('/api/shaping', { cache: 'no-store' }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + render(await response.json()); + } catch (error) { + render({ error: error.message }); + } + }; + + tick(); + const handle = setInterval(() => { if (!stopped) tick(); }, intervalMs); + return () => { stopped = true; clearInterval(handle); }; +} + +module.exports = { mountLinkRate }; diff --git a/open4d/streaming/system/WebClient/src/main.js b/open4d/streaming/system/WebClient/src/main.js new file mode 100644 index 00000000..5e7b0fb0 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/main.js @@ -0,0 +1,224 @@ +'use strict'; + +/** + * Browser client entry point. + * + * The counterpart of system/Client/client.js: read configuration, assemble the + * platform, hand it to the shared `StreamingClient`. No streaming logic here. + * + * Configuration comes from the query string so a run can be launched from a URL + * without a rebuild, mirroring how the Node client takes environment variables: + * + * /web/?server=http://host:3000&mode=interactive&segments=20 + * + * | parameter | default | meaning | + * |--------------|----------------------|--------------------------------------| + * | server | the page's own origin| server base URL | + * | mode | interactive | interactive \| simulated | + * | storage | memory | memory \| opfs asset store | + * | viewpoints | (none) | path to a viewpoint index JSON | + * | decodeBudget | 512 | decoded-clip cache budget, MB | + * | concurrency | 10 | parallel asset requests | + * | inflight | 2 | max concurrent segment downloads | + * | objects | (server's full scene)| comma-separated object subset | + * + * `?objects=` is the one parameter that changes what the experiment can show. + * The ladder publishes at least one representation per object in the scene, so + * nine ORBIT objects floor at ~116 Mbps; under that the MCKP can only buy the + * few highest-weighted objects at the cheapest rung and freezes the rest, and + * the page looks permanently starved however good the link is. Three objects + * floor near 21 Mbps, which leaves an ordinary link enough headroom to climb + * the ladder — which is the behaviour worth watching. + */ + +const { StreamingClient } = require('../../ClientCore/streaming-client'); +const { mountLinkRate } = require('./link-rate'); +const { createBrowserPlatform } = require('./browser-platform'); +const { WebGLRenderer } = require('./webgl-renderer'); + +function readConfig() { + const params = new URLSearchParams(window.location.search); + const number = (name, fallback) => { + const raw = params.get(name); + const value = Number(raw); + return raw !== null && Number.isFinite(value) ? value : fallback; + }; + return { + serverUrl: (params.get('server') || window.location.origin) + .replace(/\/+$/, ''), + mode: (params.get('mode') || 'interactive').toLowerCase(), + storageMode: (params.get('storage') || 'memory').toLowerCase(), + viewpointIndexPath: params.get('viewpoints'), + decodeBudgetBytes: number('decodeBudget', 512) * 1024 * 1024, + downloadConcurrency: number('concurrency', 10), + maxInflightSegments: number('inflight', 2), + runLabel: params.get('label') || 'web-client', + sceneObjects: (params.get('objects') || '') + .split(',').map(name => name.trim()).filter(Boolean) + }; +} + +/** Minimal page chrome: status line, log pane, artifact downloads. */ +function createUi() { + const status = document.getElementById('status'); + const logPane = document.getElementById('log'); + const artifactList = document.getElementById('artifacts'); + const stopButton = document.getElementById('stop'); + + return { + stopButton, + setStatus(text) { status.textContent = text; }, + appendLog({ level, line }) { + // DEBUG is for the console, not the pane. Lines like + // "mitch superseded" fire several times a segment and bury the + // INFO/WARN lines that actually tell you what the run is doing. + if (level.toUpperCase() === 'DEBUG') return; + const row = document.createElement('div'); + row.className = `log-line log-${level.toLowerCase()}`; + row.textContent = line; + logPane.appendChild(row); + while (logPane.childElementCount > 400) { + logPane.removeChild(logPane.firstChild); + } + logPane.scrollTop = logPane.scrollHeight; + }, + /** + * A browser cannot write a file unprompted, so every artifact the core + * "writes" is surfaced as a download link instead. + */ + offerArtifact(name, text) { + let link = artifactList.querySelector(`[data-name="${name}"]`); + if (!link) { + link = document.createElement('a'); + link.dataset.name = name; + link.textContent = name; + link.download = name; + artifactList.appendChild(link); + } + if (link.href) URL.revokeObjectURL(link.href); + link.href = URL.createObjectURL( + new Blob([text], { type: 'application/json' })); + } + }; +} + +async function main() { + const config = readConfig(); + const ui = createUi(); + // Kept so it can be stopped: a poller that outlives the run keeps hitting + // /api/shaping after Stop, and in a test harness the live interval stops + // the process exiting at all. + const stopLinkRate = mountLinkRate(document.getElementById('link')); + ui.setStatus(`connecting to ${config.serverUrl} …`); + + const interactive = config.mode === 'interactive'; + if (!['interactive', 'simulated'].includes(config.mode)) { + throw new Error( + `?mode must be "interactive" or "simulated", got "${config.mode}"`); + } + // Simulated mode replays canned poses, so it has no camera of its own. There + // is deliberately no default: solving the first ladder against an arbitrary + // pose would silently invalidate the viewpoint-aware comparison, which is + // the whole point of the experiment. + if (!interactive && !config.viewpointIndexPath) { + throw new Error( + 'simulated mode needs ?viewpoints=; ' + + 'interactive mode uses the live camera instead'); + } + let renderer = null; + + // The platform is built first so the renderer can read downloaded bytes + // back out of the asset store by handle. + const platform = await createBrowserPlatform({ + serverUrl: config.serverUrl, + storageMode: config.storageMode, + viewpointIndexPath: config.viewpointIndexPath, + // Interactive mode's pose IS the live camera, so no canned list is + // needed; the renderer's own pose is adopted immediately after start. + initialPose: interactive && !config.viewpointIndexPath + ? { objects: {} } : null, + onArtifact: (name, text) => ui.offerArtifact(name, text), + onLogLine: entry => ui.appendLog(entry) + }); + + if (interactive) { + renderer = new WebGLRenderer({ + canvas: document.getElementById('view'), + readAsset: handle => platform.assetStore.get(handle), + decodeBudgetBytes: config.decodeBudgetBytes + }); + platform.renderer = renderer; + } + + const client = new StreamingClient({ + platform, + config: { + clientMode: interactive ? 'interactive' : 'simulated', + downloadConcurrency: config.downloadConcurrency, + maxInflightSegments: config.maxInflightSegments, + runLabel: config.runLabel, + sceneObjects: config.sceneObjects + } + }); + + // Stop must finalize the run, not unload the page: the final + // POST /api/results happens during shutdown and navigating away cancels it. + ui.stopButton.addEventListener('click', () => { + ui.stopButton.disabled = true; + ui.setStatus('finishing …'); + platform.lifecycle.requestShutdown(); + }); + window.addEventListener('pagehide', () => platform.lifecycle.requestShutdown()); + + // Debug handle. Deliberate and documented: without it the only way to + // inspect a live run is to add logging and rebuild, and the renderer's + // state (what is on screen, where the camera is) is exactly what you need + // when the page looks wrong but every log line looks right. + window.__vs4d = { + client, platform, renderer, + inspect() { + const scene = renderer?._three?.scene; + const camera = renderer?._three?.camera; + return { + objects: [...(renderer?._objects || new Map())].map(([name, e]) => ({ + name, + visible: e.mesh.visible, + vertices: e.mesh.geometry.attributes.position?.count ?? 0, + frames: e.clip?.frameCount ?? 0, + textured: Boolean(e.material.map), + centre: e.mesh.geometry.boundingSphere + ? e.mesh.geometry.boundingSphere.center.toArray() + .map(v => Number(v.toFixed(2))) + : null + })), + playback: renderer?._playback, + camera: camera ? { + position: camera.position.toArray().map(v => Number(v.toFixed(2))), + target: renderer._three.controls.target.toArray() + .map(v => Number(v.toFixed(2))), + fov: camera.fov, near: camera.near, far: camera.far + } : null, + sceneChildren: scene?.children?.length ?? 0, + stats: renderer?.stats + }; + } + }; + + await client.run(); + ui.setStatus(`streaming · broadcast ${client.broadcastId ?? '(none)'}`); + + const code = await platform.lifecycle.done; + ui.setStatus(code === 0 + ? 'run complete — metrics.json is ready below' + : `run failed (exit ${code}) — see the log`); + ui.stopButton.disabled = true; + stopLinkRate(); + if (renderer) await renderer.stop(); +} + +main().catch(err => { + const status = document.getElementById('status'); + if (status) status.textContent = `fatal: ${err.message}`; + // eslint-disable-next-line no-console + console.error(err); +}); diff --git a/open4d/streaming/system/WebClient/src/nevo-client.js b/open4d/streaming/system/WebClient/src/nevo-client.js new file mode 100644 index 00000000..2623e318 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/nevo-client.js @@ -0,0 +1,274 @@ +'use strict'; + +/** + * NeVo viewer: plays pre-rendered ReRF / NeVo / captured-camera panels. + * + * A 2D canvas, not WebGL, because there is no geometry to draw — the frames are + * images. See nevo-manifest.js for why NeVo cannot be rendered client-side at + * all, and therefore why there is no camera control here. + */ + +const { + resolveConditions, frameFiles, frameFile, layoutPanels, conditionCaption, + clipSummary +} = require('./nevo-manifest'); + +class NevoClient { + /** + * @param {object} args + * @param {string} args.assetBase URL prefix serving the render output root + * @param {HTMLCanvasElement} args.canvas + * @param {string} args.object clip directory name, e.g. g_dancer + * @param {boolean} [args.nevoOnly] + * @param {number} [args.fps] + * @param {(event: object) => void} [args.onEvent] + */ + constructor({ + assetBase, canvas, object, nevoOnly = false, fps = 8, onEvent = null + }) { + this.assetBase = assetBase.replace(/\/+$/, ''); + this.canvas = canvas; + this.object = object; + this.nevoOnly = nevoOnly; + this.fps = fps; + this._onEvent = onEvent; + + this.manifest = null; + this.conditions = []; + this.images = new Map(); // file -> HTMLImageElement + this.crop = null; + this.frameIndex = 0; + this.playing = false; + + this.stats = { + imagesLoaded: 0, imagesFailed: 0, bytesUnknown: true, + framesPresented: 0, lastDrawMs: 0 + }; + this._context = canvas.getContext('2d'); + this._lastAdvance = 0; + } + + _emit(type, detail = {}) { this._onEvent?.({ type, ...detail }); } + + get clipBase() { return `${this.assetBase}/${this.object}`; } + + async start() { + const response = await fetch(`${this.clipBase}/manifest.json`, + { cache: 'no-store' }); + if (!response.ok) { + throw new Error( + `manifest fetch failed for ${this.object}: HTTP ${response.status}`); + } + this.manifest = await response.json(); + this.conditions = resolveConditions(this.manifest, + { nevoOnly: this.nevoOnly }); + this._emit('manifest', clipSummary(this.manifest, this.conditions)); + + this._installResize(); + await this._preload(); + + this.playing = true; + this._lastAdvance = performance.now(); + this._loop(); + } + + /** + * Load every image before playing. + * + * A clip is 30 images at most, so loading them all up front costs a second + * and removes any chance of the comparison showing one condition a frame + * behind another — which would be indistinguishable from a real difference + * between the conditions. + */ + async _preload() { + const entries = frameFiles(this.manifest, this.conditions); + await Promise.all(entries.map(entry => new Promise(resolve => { + const image = new Image(); + image.onload = () => { + this.images.set(entry.file, image); + this.stats.imagesLoaded++; + resolve(); + }; + image.onerror = () => { + this.stats.imagesFailed++; + this._emit('error', { message: `missing render ${entry.file}` }); + resolve(); + }; + image.src = `${this.clipBase}/${entry.file}`; + }))); + this.crop = this._contentCrop(); + this._emit('ready', { + loaded: this.stats.imagesLoaded, failed: this.stats.imagesFailed, + crop: this.crop + }); + } + + /** + * Crop to the subject, so three 4:3 panels side by side are not mostly + * empty background. + * + * Measured once from the captured-camera frame and then applied IDENTICALLY + * to every panel — identically is the point, because the conditions have to + * stay pixel-aligned or the comparison stops meaning anything. This mirrors + * what live_demo.py does on the server side. + * + * The renders are a subject over pure white, so "content" is anything below + * the white threshold. + */ + _contentCrop(pad = 0.06, threshold = 246) { + const reference = this.conditions.find(c => c.isReference) + || this.conditions[0]; + const image = this.images.get( + frameFile(reference.prefix, this.manifest.frames[0])); + if (!image) return null; + + const probe = document.createElement('canvas'); + probe.width = image.naturalWidth; + probe.height = image.naturalHeight; + const context = probe.getContext('2d', { willReadFrequently: true }); + context.drawImage(image, 0, 0); + let data; + try { + data = context.getImageData(0, 0, probe.width, probe.height).data; + } catch (_) { + return null; // tainted canvas; fall back to the full frame + } + + let minX = probe.width, minY = probe.height, maxX = -1, maxY = -1; + for (let y = 0; y < probe.height; y++) { + for (let x = 0; x < probe.width; x++) { + const i = (y * probe.width + x) * 4; + if (data[i] < threshold || data[i + 1] < threshold + || data[i + 2] < threshold) { + if (x < minX) minX = x; + if (x > maxX) maxX = x; + if (y < minY) minY = y; + if (y > maxY) maxY = y; + } + } + } + if (maxX < 0) return null; + + const padX = (maxX - minX) * pad; + const padY = (maxY - minY) * pad; + const left = Math.max(0, Math.round(minX - padX)); + const top = Math.max(0, Math.round(minY - padY)); + const right = Math.min(probe.width, Math.round(maxX + padX + 1)); + const bottom = Math.min(probe.height, Math.round(maxY + padY + 1)); + return { x: left, y: top, width: right - left, height: bottom - top }; + } + + _installResize() { + const apply = () => { + const ratio = Math.min(window.devicePixelRatio || 1, 2); + const width = this.canvas.clientWidth || 1280; + const height = this.canvas.clientHeight || 720; + this.canvas.width = Math.max(1, Math.round(width * ratio)); + this.canvas.height = Math.max(1, Math.round(height * ratio)); + this._draw(); + }; + apply(); + if (typeof ResizeObserver === 'function') { + this._resizeObserver = new ResizeObserver(apply); + this._resizeObserver.observe(this.canvas); + } else { + window.addEventListener('resize', apply); + } + } + + _draw() { + if (!this.manifest || !this._context) return; + const started = performance.now(); + const context = this._context; + const { width, height } = this.canvas; + + context.fillStyle = '#101014'; + context.fillRect(0, 0, width, height); + + const frame = this.manifest.frames[this.frameIndex]; + const first = this.images.get( + frameFile(this.conditions[0].prefix, frame)); + if (!first) return; + + const crop = this.crop + || { x: 0, y: 0, width: first.naturalWidth, height: first.naturalHeight }; + const layout = layoutPanels({ + panelCount: this.conditions.length, + sourceWidth: crop.width, + sourceHeight: crop.height, + canvasWidth: width, + canvasHeight: height, + labelHeight: Math.max(18, Math.round(height * 0.03)) + }); + + context.textBaseline = 'top'; + context.font = `${Math.max(10, Math.round(layout.labelHeight * 0.6))}px ` + + 'ui-monospace, Menlo, monospace'; + + this.conditions.forEach((condition, index) => { + const panel = layout.panels[index]; + const image = this.images.get(frameFile(condition.prefix, frame)); + if (image) { + context.drawImage(image, + crop.x, crop.y, crop.width, crop.height, + panel.x, panel.y, panel.width, panel.height); + } else { + context.fillStyle = '#1a1a22'; + context.fillRect(panel.x, panel.y, panel.width, panel.height); + } + // The captured camera is ground truth, so mark it differently from + // the two reconstructions it is there to judge. + context.fillStyle = condition.isReference ? '#8b8b98' : '#e6e6ea'; + context.fillText(conditionCaption(condition), + panel.x + 2, panel.labelY + 2, panel.width - 4); + }); + + this.stats.lastDrawMs = performance.now() - started; + this.stats.framesPresented++; + } + + _loop() { + this._rafHandle = requestAnimationFrame(() => this._loop()); + if (!this.playing || !this.manifest) return; + const interval = 1000 / this.fps; + const now = performance.now(); + if (now - this._lastAdvance >= interval) { + const steps = Math.floor((now - this._lastAdvance) / interval); + this._lastAdvance += steps * interval; + this.frameIndex = + (this.frameIndex + steps) % this.manifest.frames.length; + this._draw(); + } + } + + setPlaying(playing) { + this.playing = playing; + this._lastAdvance = performance.now(); + } + + stop() { + this.playing = false; + if (this._rafHandle) cancelAnimationFrame(this._rafHandle); + this._resizeObserver?.disconnect(); + } + + inspect() { + return { + object: this.object, + frameIndex: this.frameIndex, + frameCount: this.manifest ? this.manifest.frames.length : 0, + playing: this.playing, + conditions: this.conditions.map(c => ({ + prefix: c.prefix, label: c.label, + keptFraction: c.kept_fraction ?? null + })), + summary: this.manifest + ? clipSummary(this.manifest, this.conditions) : null, + crop: this.crop, + stats: { ...this.stats }, + canvas: { width: this.canvas.width, height: this.canvas.height } + }; + } +} + +module.exports = { NevoClient }; diff --git a/open4d/streaming/system/WebClient/src/nevo-main.js b/open4d/streaming/system/WebClient/src/nevo-main.js new file mode 100644 index 00000000..b1270d08 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/nevo-main.js @@ -0,0 +1,95 @@ +'use strict'; + +/** + * Entry point for the NeVo viewer. + * + * /web/nevo.html?object=g_dancer + * + * | parameter | default | meaning | + * |-----------|---------------|--------------------------------------------| + * | assets | /nevo-assets | URL prefix serving the render output root | + * | object | g_dancer | clip directory name | + * | fps | 8 | playback rate | + * | nevoOnly | 0 | 1 shows only NeVo's own filtered output | + */ + +const { NevoClient } = require('./nevo-client'); + +async function main() { + const params = new URLSearchParams(window.location.search); + const status = document.getElementById('status'); + const logPane = document.getElementById('log'); + const statsPane = document.getElementById('stats'); + + const log = (level, message) => { + const row = document.createElement('div'); + row.className = `log-line log-${level}`; + row.textContent = message; + logPane.appendChild(row); + while (logPane.childElementCount > 200) { + logPane.removeChild(logPane.firstChild); + } + logPane.scrollTop = logPane.scrollHeight; + }; + + const fps = Number(params.get('fps')); + const client = new NevoClient({ + assetBase: params.get('assets') || '/nevo-assets', + canvas: document.getElementById('view'), + object: params.get('object') || 'g_dancer', + nevoOnly: params.get('nevoOnly') === '1', + fps: Number.isFinite(fps) && fps > 0 ? fps : 8, + onEvent: event => { + if (event.type === 'manifest') { + log('info', `${event.name} · ${event.representation}`); + log('info', `${event.frames} frames · ${event.source} · ` + + `view ${event.view}` + + (event.viewInTrainingSet + ? ' (in the training set)' : ' (held out)')); + for (const kept of event.keptFractions) { + log('info', ` ${kept.label}: kept ` + + `${(kept.keptFraction * 100).toFixed(1)}% of voxels`); + } + status.textContent = 'loading renders …'; + } else if (event.type === 'ready') { + log('info', `${event.loaded} renders loaded` + + (event.failed ? `, ${event.failed} missing` : '')); + status.textContent = 'playing pre-rendered frames'; + } else if (event.type === 'error') { + log('error', event.message); + } + } + }); + window.__vs4dNevo = client; + + const playButton = document.getElementById('play'); + playButton.addEventListener('click', () => { + client.setPlaying(!client.playing); + playButton.textContent = client.playing ? 'Pause' : 'Play'; + }); + + setInterval(() => { + const info = client.inspect(); + statsPane.textContent = [ + `clip ${info.object}`, + `frame ${info.frameIndex + 1} / ${info.frameCount}`, + ...(info.stats.imagesFailed + ? [`MISSING ${info.stats.imagesFailed} renders`] : []) + ].join('\n'); + }, 500); + + try { + await client.start(); + } catch (error) { + status.textContent = `could not start: ${error.message}`; + log('error', error.message); + log('info', 'Renders come from orbitnevo/render_frames.py; point ' + + '?assets= at its output root (default ~/nevo_output, served at ' + + '/nevo-assets).'); + } +} + +main().catch(error => { + document.getElementById('status').textContent = `fatal: ${error.message}`; + console.error(error); +}); diff --git a/open4d/streaming/system/WebClient/src/nevo-manifest.js b/open4d/streaming/system/WebClient/src/nevo-manifest.js new file mode 100644 index 00000000..24c44392 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/nevo-manifest.js @@ -0,0 +1,158 @@ +'use strict'; + +/** + * NeVo clip manifest handling: conditions, frame filenames, panel layout. + * + * Pure, so the parts that decide WHAT is shown can be tested without a browser + * (see tests/test_nevo_manifest.js). The drawing itself is in nevo-client.js. + * + * NeVo is the one baseline that cannot run client-side at all. It is a NeRF + * (ReRF feature voxels) whose frames take roughly half a second each to + * ray-march on a workstation GPU, and its entropy decoder is CUDA and Python + * 3.8. So the browser shows PRE-RENDERED frames — exactly what + * `orbitnevo/live_demo.py` does, and its own docstring says the same. There is + * therefore no camera control here, and pretending otherwise would be + * misleading rather than convenient. + * + * What the panel does show is the comparison the baseline exists for: plain + * ReRF against NeVo's visibility-filtered reconstruction at the same instant + * and viewpoint, with the captured camera alongside as ground truth. + */ + +/** The captured-camera condition, appended the way live_demo.py appends it. */ +function referenceCondition(manifest) { + return { + name: 'capture', + prefix: 'reference', + label: `captured camera ${manifest.view}`, + threshold: null, + kept_fraction: null, + isReference: true + }; +} + +/** + * Resolve which conditions to show, in display order. + * + * @param {object} manifest parsed manifest.json + * @param {object} [options] + * @param {boolean} [options.nevoOnly] show only NeVo's own output — the + * conditions with a threshold — dropping plain ReRF and the captured camera, + * which are the comparison rather than the output. + * @param {boolean} [options.withReference=true] + */ +function resolveConditions(manifest, { nevoOnly = false, withReference = true } = {}) { + const declared = Array.isArray(manifest.conditions) ? manifest.conditions : []; + if (nevoOnly) { + return declared.filter(condition => condition.threshold !== null + && condition.threshold !== undefined); + } + const resolved = [...declared]; + if (withReference) resolved.push(referenceCondition(manifest)); + return resolved; +} + +/** `{prefix}_{frame:03d}.png`, matching what render_frames.py wrote. */ +function frameFile(prefix, frame) { + return `${prefix}_${String(frame).padStart(3, '0')}.png`; +} + +/** Every image a clip needs, so they can be preloaded before playback. */ +function frameFiles(manifest, conditions) { + const files = []; + for (const frame of manifest.frames) { + for (const condition of conditions) { + files.push({ frame, condition, file: frameFile(condition.prefix, frame) }); + } + } + return files; +} + +/** + * Lay out the condition panels in a row. + * + * The source renders are 1280x960 with the subject a small part of the frame, + * so a crop is applied identically to every panel — identically, because the + * conditions must stay pixel-aligned for the comparison to mean anything. + * + * @param {object} args + * @param {number} args.panelCount + * @param {number} args.sourceWidth width after cropping + * @param {number} args.sourceHeight height after cropping + * @param {number} args.canvasWidth space available + * @param {number} args.canvasHeight + * @param {number} [args.labelHeight] + * @param {number} [args.gap] + */ +function layoutPanels({ + panelCount, sourceWidth, sourceHeight, canvasWidth, canvasHeight, + labelHeight = 26, gap = 6 +}) { + if (panelCount <= 0) throw new RangeError('panelCount must be positive'); + if (!(sourceWidth > 0) || !(sourceHeight > 0)) { + throw new RangeError('source dimensions must be positive'); + } + const totalGap = gap * (panelCount - 1); + const aspect = sourceWidth / sourceHeight; + + // Fit by width, then shrink if the resulting height does not fit. Both + // constraints matter: a wide window is width-bound, a tall narrow one is + // height-bound, and picking only one leaves panels clipped. + let panelWidth = Math.max(1, Math.floor((canvasWidth - totalGap) / panelCount)); + let panelHeight = Math.round(panelWidth / aspect); + const available = canvasHeight - labelHeight; + if (panelHeight > available && available > 0) { + panelHeight = available; + panelWidth = Math.max(1, Math.round(panelHeight * aspect)); + } + + const rowWidth = panelWidth * panelCount + totalGap; + const originX = Math.max(0, Math.round((canvasWidth - rowWidth) / 2)); + const originY = Math.max(0, Math.round( + (canvasHeight - (panelHeight + labelHeight)) / 2)); + + return { + panelWidth, panelHeight, labelHeight, gap, rowWidth, originX, originY, + panels: Array.from({ length: panelCount }, (_, index) => ({ + index, + x: originX + index * (panelWidth + gap), + y: originY, + width: panelWidth, + height: panelHeight, + labelY: originY + panelHeight + })) + }; +} + +/** A caption line for a condition: what it is, and how much it kept. */ +function conditionCaption(condition) { + if (condition.kept_fraction === null || condition.kept_fraction === undefined) { + return condition.label; + } + const percent = (condition.kept_fraction * 100).toFixed(1); + return `${condition.label} · ${percent}% of voxels`; +} + +/** Headline facts about a clip, for the page's status area. */ +function clipSummary(manifest, conditions) { + const filtered = conditions.filter( + c => c.threshold !== null && c.threshold !== undefined); + return { + name: manifest.name, + representation: manifest.representation, + frames: manifest.frames.length, + view: manifest.view, + viewInTrainingSet: manifest.view_in_training_set === true, + source: `${manifest.width}x${manifest.height}`, + seconds: manifest.seconds, + conditions: conditions.length, + keptFractions: filtered.map(c => ({ + label: c.label, threshold: c.threshold, keptFraction: c.kept_fraction + })) + }; +} + +module.exports = { + resolveConditions, referenceCondition, frameFile, frameFiles, + layoutPanels, conditionCaption, clipSummary +}; diff --git a/open4d/streaming/system/WebClient/src/point-reconstruction.js b/open4d/streaming/system/WebClient/src/point-reconstruction.js new file mode 100644 index 00000000..661f1ca0 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/point-reconstruction.js @@ -0,0 +1,259 @@ +'use strict'; + +/** + * MetaStream / DeltaStream point-cloud reconstruction. + * + * A faithful port of `ReconstructionState` in + * `baselines/DeltaStream/orbitstream/reconstruction.py`, which is the + * authority. `tests/test_point_reconstruction.js` compares this against that + * implementation's output on the same input, because the delta model is fiddly + * enough that "looks about right on screen" is not evidence. + * + * The streamed points are in CAMERA space. The server encodes + * `cloud.positions` unchanged and only uses `camera_to_world` for tile + * assignment, so placing the points is the client's job — see `worldClouds()`. + * + * The delta model, per (object, camera) stream: + * + * keyframe the payload IS the whole cloud; replace. + * delta project the previous cloud back into the source image to recover + * each point's block, then + * 1. copy the points inside each motion's source block and + * translate them by that motion's delta, + * 2. drop every point whose block appears in removal_blocks, + * 3. append the residual payload. + * A motion source may overlap a removal block; the translated copy + * survives, matching the reference desktop client. + */ + +/** Camera-space cloud: interleaved xyz floats and rgb bytes. */ +class PointCloud { + constructor(positions, colors) { + this.positions = positions; // Float32Array, 3N + this.colors = colors; // Uint8Array, 3N + } + + static empty() { + return new PointCloud(new Float32Array(0), new Uint8Array(0)); + } + + get pointCount() { return this.positions.length / 3; } +} + +/** Concatenate clouds in order. */ +function concatClouds(clouds) { + const total = clouds.reduce((sum, c) => sum + c.pointCount, 0); + if (total === 0) return PointCloud.empty(); + const positions = new Float32Array(total * 3); + const colors = new Uint8Array(total * 3); + let offset = 0; + for (const cloud of clouds) { + positions.set(cloud.positions, offset); + colors.set(cloud.colors, offset); + offset += cloud.positions.length; + } + return new PointCloud(positions, colors); +} + +class ReconstructionState { + /** @param {object} header decoded CONNECTION message */ + constructor(header) { + this.header = header; + this.calibrations = new Map(); + for (const object of header.objects) { + for (const camera of object.cameras) { + this.calibrations.set(`${object.objectId}:${camera.cameraId}`, + { ...camera, objectId: object.objectId }); + } + } + this._cameraClouds = new Map(); // "obj:cam" -> PointCloud + this.lastFrameId = -1; + } + + reset() { + this._cameraClouds.clear(); + this.lastFrameId = -1; + } + + /** + * Fold one FRAME into the state and return the per-object world clouds. + * + * @param {object} frame decoded FRAME message + * @param {(draco: Uint8Array) => PointCloud} decodeDraco + * @param {{strictOrder?: boolean}} [options] + * strictOrder mirrors the Python implementation, which refuses a gap in + * the frame sequence. A browser over a real link may legitimately miss a + * frame, in which case the stream cannot be reconstructed and the caller + * must resynchronise on the next keyframe rather than render nonsense. + */ + apply(frame, decodeDraco, { strictOrder = true } = {}) { + if (strictOrder && frame.frameId !== this.lastFrameId + 1) { + throw new Error( + `dependency frame out of order: expected ${this.lastFrameId + 1}, ` + + `received ${frame.frameId}`); + } + if (frame.frameId === 0 && frame.frameType !== 'keyframe') { + throw new Error('a run must begin with a keyframe'); + } + + const seen = new Set(); + for (const record of frame.records) { + const key = `${record.objectId}:${record.cameraId}`; + if (seen.has(key)) throw new Error(`duplicate record for stream ${key}`); + if (!this.calibrations.has(key)) { + throw new Error(`unknown stream ${key}`); + } + seen.add(key); + + const residual = record.draco && record.draco.length + ? decodeDraco(record.draco) : PointCloud.empty(); + if (residual.pointCount !== record.pointCount) { + throw new Error( + `decoded point count differs for ${key}: ` + + `${residual.pointCount} != ${record.pointCount}`); + } + + if (frame.frameType === 'keyframe') { + if (record.removalBlocks.length || record.motions.length) { + throw new Error('keyframes cannot contain delta operations'); + } + this._cameraClouds.set(key, residual); + } else { + if (!this._cameraClouds.has(key)) { + throw new Error(`delta precedes keyframe for stream ${key}`); + } + this._cameraClouds.set(key, this._applyDelta( + this._cameraClouds.get(key), residual, record)); + } + } + + this.lastFrameId = frame.frameId; + return this.worldClouds(); + } + + /** Streams the header declares but this frame omitted. */ + missingStreams(seenKeys) { + return [...this.calibrations.keys()].filter(key => !seenKeys.has(key)); + } + + _applyDelta(previous, residual, record) { + const calibration = this.calibrations.get( + `${record.objectId}:${record.cameraId}`); + const { fx, fy, cx, cy } = calibration; + const blockSize = this.header.blockSize; + const blocksPerRow = Math.floor(this.header.width / blockSize); + const count = previous.pointCount; + const points = previous.positions; + + // Project each previous point back into its source image to recover the + // block it belongs to. A point behind the camera or outside the frame + // has no block, and must be excluded rather than clamped. + const u = new Float32Array(count); + const v = new Float32Array(count); + const inImage = new Uint8Array(count); + const blockIndices = new Int32Array(count).fill(-1); + + for (let i = 0; i < count; i++) { + const z = points[i * 3 + 2]; + if (!(z > 0)) { u[i] = -1; v[i] = -1; continue; } + const pu = (points[i * 3] / z) * fx + cx; + const pv = (points[i * 3 + 1] / z) * fy + cy; + u[i] = pu; + v[i] = pv; + if (pu >= 0 && pv >= 0 && pu < this.header.width + && pv < this.header.height) { + inImage[i] = 1; + blockIndices[i] = Math.floor(pv / blockSize) * blocksPerRow + + Math.floor(pu / blockSize); + } + } + + // 1. Motion: translated copies of the points in each source block. + const moved = []; + for (const motion of record.motions) { + const x0 = motion.sourceX - blockSize / 2; + const y0 = motion.sourceY - blockSize / 2; + const selected = []; + for (let i = 0; i < count; i++) { + if (!inImage[i]) continue; + if (u[i] >= x0 && u[i] < x0 + blockSize + && v[i] >= y0 && v[i] < y0 + blockSize) { + selected.push(i); + } + } + if (selected.length === 0) continue; + const positions = new Float32Array(selected.length * 3); + const colors = new Uint8Array(selected.length * 3); + selected.forEach((source, target) => { + positions[target * 3] = points[source * 3] + motion.deltaX; + positions[target * 3 + 1] = points[source * 3 + 1] + motion.deltaY; + positions[target * 3 + 2] = points[source * 3 + 2] + motion.deltaZ; + colors[target * 3] = previous.colors[source * 3]; + colors[target * 3 + 1] = previous.colors[source * 3 + 1]; + colors[target * 3 + 2] = previous.colors[source * 3 + 2]; + }); + moved.push(new PointCloud(positions, colors)); + } + + // 2. Removal: drop points whose block was republished. + let kept; + if (record.removalBlocks.length) { + const removed = new Set(Array.from(record.removalBlocks)); + const keepIndices = []; + for (let i = 0; i < count; i++) { + if (!removed.has(blockIndices[i])) keepIndices.push(i); + } + const positions = new Float32Array(keepIndices.length * 3); + const colors = new Uint8Array(keepIndices.length * 3); + keepIndices.forEach((source, target) => { + positions.set(points.subarray(source * 3, source * 3 + 3), target * 3); + colors.set(previous.colors.subarray(source * 3, source * 3 + 3), + target * 3); + }); + kept = new PointCloud(positions, colors); + } else { + kept = previous; + } + + // 3. Order matches the Python implementation: moved, residual, kept. + const parts = [...moved]; + if (residual.pointCount) parts.push(residual); + if (kept.pointCount) parts.push(kept); + return concatClouds(parts); + } + + /** + * Per-object clouds in world space. + * + * `cameraToWorldRowMajor` is row-major (the Python encoder writes + * `for row in matrix for v in row`), so the rotation is rows 0..2 columns + * 0..2 and the translation is column 3 — indices 3, 7, 11. + */ + worldClouds() { + const byObject = new Map(); + for (const [key, cloud] of this._cameraClouds) { + const calibration = this.calibrations.get(key); + const m = calibration.cameraToWorldRowMajor; + const count = cloud.pointCount; + const positions = new Float32Array(count * 3); + for (let i = 0; i < count; i++) { + const x = cloud.positions[i * 3]; + const y = cloud.positions[i * 3 + 1]; + const z = cloud.positions[i * 3 + 2]; + positions[i * 3] = m[0] * x + m[1] * y + m[2] * z + m[3]; + positions[i * 3 + 1] = m[4] * x + m[5] * y + m[6] * z + m[7]; + positions[i * 3 + 2] = m[8] * x + m[9] * y + m[10] * z + m[11]; + } + const transformed = new PointCloud(positions, cloud.colors); + const objectId = calibration.objectId; + byObject.set(objectId, byObject.has(objectId) + ? concatClouds([byObject.get(objectId), transformed]) + : transformed); + } + return byObject; + } + + get streamCount() { return this._cameraClouds.size; } +} + +module.exports = { ReconstructionState, PointCloud, concatClouds }; diff --git a/open4d/streaming/system/WebClient/src/point-renderer.js b/open4d/streaming/system/WebClient/src/point-renderer.js new file mode 100644 index 00000000..ee6d8373 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/point-renderer.js @@ -0,0 +1,246 @@ +'use strict'; + +/** + * Point-cloud renderer for the V4DS baselines. + * + * One `THREE.Points` per object, fed world-space clouds from + * `point-reconstruction.js`. Shared by MetaStream, DeltaStream, ViVo, NAVA and + * LiVo, which is the point: they differ in what they choose to send, not in how + * a received cloud is drawn, so putting the drawing in one place keeps the + * comparison about the algorithms. + * + * Deliberately separate from `webgl-renderer.js`. That one implements the + * ClientPlatform renderer contract for the mesh ladder — decode caching, + * per-frame clip playback, `object_ready` crediting. None of that applies here: + * a baseline pushes a fresh cloud whenever it likes, there is no segment + * buffer, and there is no ABR asking whether an object became playable. Forcing + * both through one class would mean a contract that fits neither. + * + * Geometry is reallocated only when a cloud outgrows its buffer. Point counts + * swing frame to frame (a DeltaStream residual adds a few thousand, a keyframe + * replaces everything), and allocating a new BufferGeometry per frame at 30 Hz + * makes the garbage collector the bottleneck. + */ + +const THREE = require('three'); +const { OrbitControls } = require('three/examples/jsm/controls/OrbitControls.js'); + +/** Extra headroom when growing a buffer, so small growth is not a realloc. */ +const GROWTH_FACTOR = 1.5; + +class PointRenderer { + /** + * @param {object} args + * @param {HTMLCanvasElement} args.canvas + * @param {number} [args.pointSize] world-space point size, metres + * @param {object} [args.scene] { background } + */ + constructor({ canvas, pointSize = 0.012, scene: sceneOptions = {} }) { + this.canvas = canvas; + this.pointSize = pointSize; + this._sceneOptions = sceneOptions; + this._objects = new Map(); // objectId -> { points, geometry, capacity } + this._userMovedCamera = false; + this._framedObjectCount = 0; + this._running = false; + this._presented = 0; + this._three = null; + } + + start() { + this._three = this._buildScene(); + this._running = true; + this._loop(); + return this; + } + + stop() { + this._running = false; + if (this._rafHandle) cancelAnimationFrame(this._rafHandle); + this._resizeObserver?.disconnect(); + for (const entry of this._objects.values()) { + entry.geometry.dispose(); + entry.points.material.dispose(); + } + this._objects.clear(); + this._three?.renderer.dispose(); + } + + _buildScene() { + const renderer = new THREE.WebGLRenderer({ canvas: this.canvas, antialias: false }); + const scene = new THREE.Scene(); + scene.background = new THREE.Color(this._sceneOptions.background ?? 0x101014); + const camera = new THREE.PerspectiveCamera(60, 1, 0.05, 500); + camera.position.set(0, 1.6, 4); + + const controls = new OrbitControls(camera, this.canvas); + controls.target.set(0, 1.0, 0); + controls.enableDamping = true; + controls.update(); + controls.addEventListener('start', () => { this._userMovedCamera = true; }); + + const apply = () => { + const width = this.canvas.clientWidth || this.canvas.width || 1280; + const height = this.canvas.clientHeight || this.canvas.height || 720; + if (width <= 0 || height <= 0) return; + renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2)); + renderer.setSize(width, height, false); + camera.aspect = width / height; + camera.updateProjectionMatrix(); + }; + apply(); + if (typeof ResizeObserver === 'function') { + this._resizeObserver = new ResizeObserver(apply); + this._resizeObserver.observe(this.canvas); + } else { + window.addEventListener('resize', apply); + } + + return { renderer, scene, camera, controls }; + } + + _entry(objectId, requiredPoints) { + let entry = this._objects.get(objectId); + if (!entry) { + const geometry = new THREE.BufferGeometry(); + const capacity = Math.max(1, Math.ceil(requiredPoints * GROWTH_FACTOR)); + geometry.setAttribute('position', + new THREE.BufferAttribute(new Float32Array(capacity * 3), 3)); + geometry.setAttribute('color', + new THREE.BufferAttribute(new Uint8Array(capacity * 3), 3, true)); + const material = new THREE.PointsMaterial({ + size: this.pointSize, sizeAttenuation: true, vertexColors: true + }); + const points = new THREE.Points(geometry, material); + points.frustumCulled = false; // bounds change every frame + this._three.scene.add(points); + entry = { points, geometry, capacity }; + this._objects.set(objectId, entry); + return entry; + } + if (requiredPoints > entry.capacity) { + const capacity = Math.ceil(requiredPoints * GROWTH_FACTOR); + entry.geometry.setAttribute('position', + new THREE.BufferAttribute(new Float32Array(capacity * 3), 3)); + entry.geometry.setAttribute('color', + new THREE.BufferAttribute(new Uint8Array(capacity * 3), 3, true)); + entry.capacity = capacity; + } + return entry; + } + + /** + * Show the per-object world clouds produced by a reconstruction step. + * + * @param {Map} clouds + */ + update(clouds) { + for (const [objectId, cloud] of clouds) { + const count = cloud.pointCount; + const entry = this._entry(objectId, count); + const position = entry.geometry.getAttribute('position'); + const color = entry.geometry.getAttribute('color'); + position.array.set(cloud.positions); + color.array.set(cloud.colors); + // drawRange, not a resize: the buffers are oversized on purpose and + // only the first `count` points are valid this frame. + entry.geometry.setDrawRange(0, count); + position.needsUpdate = true; + color.needsUpdate = true; + entry.points.visible = count > 0; + entry.lastCount = count; + } + // An object the stream stopped sending should disappear rather than + // freeze at its last cloud, which would read as a live object. + for (const [objectId, entry] of this._objects) { + if (!clouds.has(objectId)) entry.points.visible = false; + } + if (!this._userMovedCamera && this._objects.size !== this._framedObjectCount) { + this._framedObjectCount = this._objects.size; + this.frameScene(clouds); + } + } + + /** + * Point the camera at the content. + * + * Computed from the cloud data rather than Three.js bounding spheres: the + * buffers are oversized, so the unused tail would drag the bounds towards + * the origin. The ORBIT corpus is also baked into venue world coordinates + * with raised floor tiers, so a fixed pose frames empty air. + */ + frameScene(clouds) { + if (this._userMovedCamera || !this._three) return; + let minX = Infinity, minY = Infinity, minZ = Infinity; + let maxX = -Infinity, maxY = -Infinity, maxZ = -Infinity; + let total = 0; + for (const cloud of clouds.values()) { + for (let i = 0; i < cloud.pointCount; i++) { + const x = cloud.positions[i * 3]; + const y = cloud.positions[i * 3 + 1]; + const z = cloud.positions[i * 3 + 2]; + if (x < minX) minX = x; + if (y < minY) minY = y; + if (z < minZ) minZ = z; + if (x > maxX) maxX = x; + if (y > maxY) maxY = y; + if (z > maxZ) maxZ = z; + total++; + } + } + if (total === 0 || !Number.isFinite(minX)) return; + + const centre = new THREE.Vector3( + (minX + maxX) / 2, (minY + maxY) / 2, (minZ + maxZ) / 2); + const radius = Math.max( + 0.5, Math.hypot(maxX - minX, maxY - minY, maxZ - minZ) / 2); + const { camera, controls } = this._three; + const distance = (radius / Math.tan((camera.fov * Math.PI) / 360)) * 1.6; + controls.target.copy(centre); + camera.position.set(centre.x, centre.y + radius * 0.15, centre.z + distance); + camera.near = Math.max(radius / 20, 0.05); + camera.far = distance + radius * 30; + camera.updateProjectionMatrix(); + controls.update(); + } + + _loop() { + if (!this._running) return; + this._rafHandle = requestAnimationFrame(() => this._loop()); + this._three.controls.update(); + this._three.renderer.render(this._three.scene, this._three.camera); + this._presented++; + } + + /** Camera pose in the shape V4DS FEEDBACK wants. */ + viewerState() { + const { camera, controls } = this._three; + camera.updateMatrixWorld(); + const forward = new THREE.Vector3(); + camera.getWorldDirection(forward); + return { + position: camera.position.toArray(), + forward: forward.toArray(), + up: camera.up.toArray(), + target: controls.target.toArray(), + verticalFovDegrees: camera.fov, + aspect: camera.aspect, + near: camera.near, + far: camera.far + }; + } + + get stats() { + return { + framesPresented: this._presented, + objects: [...this._objects].map(([objectId, entry]) => ({ + objectId, + points: entry.lastCount ?? 0, + capacity: entry.capacity, + visible: entry.points.visible + })) + }; + } +} + +module.exports = { PointRenderer }; diff --git a/open4d/streaming/system/WebClient/src/splat-renderer.js b/open4d/streaming/system/WebClient/src/splat-renderer.js new file mode 100644 index 00000000..7a4a33a2 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/splat-renderer.js @@ -0,0 +1,422 @@ +'use strict'; + +/** + * 3D Gaussian splat renderer for the Vega baseline. + * + * Draws anisotropic Gaussians the way 3DGS does: project each splat's 3D + * covariance into a screen-space 2D covariance, size a camera-facing quad to + * its eigenvectors, and evaluate the Gaussian falloff in the fragment shader + * with premultiplied-alpha blending, back to front. + * + * Two design choices worth knowing: + * + * 1. SPLAT DATA LIVES IN A TEXTURE, and the only per-instance attribute is an + * index into it. Correct 3DGS needs back-to-front ordering, which changes + * every time the camera moves. Reordering the attribute buffers themselves + * would mean copying 14 floats per splat per sort — about 3 MB for a 58k + * frame. Reordering a single index attribute is 230 KB. + * + * 2. SORTING IS A COUNTING SORT over quantised depth, not a comparison sort. + * At 58k splats a comparison sort costs milliseconds of the frame budget; + * a 16-bit bucket pass is a few hundred microseconds and the ordering error + * within a bucket is far below what the blending can show. + * + * Depth testing is off and depth writing is off, as 3DGS requires: the ordering + * IS the depth resolution. Turning them on produces hard edges where splats + * should blend. + */ + +const THREE = require('three'); + +/** Texels per splat in the data texture: 4 x RGBA32F = 16 floats. */ +const TEXELS_PER_SPLAT = 4; +const TEXTURE_WIDTH = 1024; +/** Depth buckets for the counting sort. */ +const SORT_BUCKETS = 1 << 16; + +const VERTEX_SHADER = /* glsl */` +precision highp float; +precision highp int; + +// RawShaderMaterial does NOT inject Three.js's built-in uniforms, so they are +// declared here. GLSL 3 is required for transpose(), which GLSL ES 1.00 lacks. +uniform mat4 modelMatrix; +uniform mat4 viewMatrix; +uniform mat4 projectionMatrix; + +uniform sampler2D splatData; +uniform vec2 splatTextureSize; +uniform vec2 viewport; +uniform float splatScale; + +in vec2 quadPosition; // corner in [-1, 1] +in float splatIndex; + +out vec4 vColor; +out vec2 vGaussian; + +vec4 fetch(float texel) { + float index = splatIndex * ${TEXELS_PER_SPLAT}.0 + texel; + float x = mod(index, splatTextureSize.x); + float y = floor(index / splatTextureSize.x); + return texture(splatData, (vec2(x, y) + 0.5) / splatTextureSize); +} + +mat3 quaternionToMatrix(vec4 q) { + float x = q.x, y = q.y, z = q.z, w = q.w; + return mat3( + 1.0 - 2.0 * (y * y + z * z), 2.0 * (x * y + w * z), 2.0 * (x * z - w * y), + 2.0 * (x * y - w * z), 1.0 - 2.0 * (x * x + z * z), 2.0 * (y * z + w * x), + 2.0 * (x * z + w * y), 2.0 * (y * z - w * x), 1.0 - 2.0 * (x * x + y * y) + ); +} + +void main() { + vec4 centerAndOpacity = fetch(0.0); + vec4 logScale = fetch(1.0); + vec4 rotation = fetch(2.0); + vec4 color = fetch(3.0); + + vec3 center = centerAndOpacity.xyz; + float opacity = centerAndOpacity.w; + + mat4 modelView = viewMatrix * modelMatrix; + vec4 cam = modelView * vec4(center, 1.0); + vec4 clip = projectionMatrix * cam; + // Behind the camera: park the vertex outside the clip volume rather than + // letting a divide by a near-zero w throw the quad across the screen. + if (clip.w <= 0.0) { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + vColor = vec4(0.0); + vGaussian = vec2(0.0); + return; + } + + vec3 scale = exp(logScale.xyz) * splatScale; + +#ifdef ISOTROPIC_SPLATS + // Isotropic path: size the quad from the mean scale projected through the + // focal length, with no covariance projection at all. + // + // Two reasons this is the default rather than a fallback. First, Vega's + // Gaussians are near-isotropic — the exported log scales sit within about + // 0.3 of each other — so the anisotropic projection buys very little here. + // Second, the full covariance path (mat3 transposes and products in the + // vertex stage) does not render under software GL: measured in headless + // Chrome with SwiftShader, a shader that merely CONTAINS that math draws + // nothing even when gl_Position does not use its result, while the same + // shader without it draws correctly. That is a driver-level failure, not a + // logic error, but it makes the anisotropic path unverifiable here. + float fxIso = projectionMatrix[0][0] * viewport.x * 0.5; + float fyIso = projectionMatrix[1][1] * viewport.y * 0.5; + float meanScale = (scale.x + scale.y + scale.z) / 3.0; + float depth = max(-cam.z, 1e-4); + // Two sigma, with a half-pixel floor so a distant splat still marks a pixel. + vec2 radiusPx = vec2(max(2.0 * fxIso * meanScale / depth, 0.5), + max(2.0 * fyIso * meanScale / depth, 0.5)); + vec2 offset = quadPosition * radiusPx; + vColor = vec4(color.rgb, opacity); + vGaussian = quadPosition * 2.0; + gl_Position = vec4( + clip.xy / clip.w + offset / viewport * 2.0, + clip.z / clip.w, 1.0); +#else + mat3 rotationMatrix = quaternionToMatrix(rotation); + mat3 scaled = mat3( + rotationMatrix[0] * scale.x, + rotationMatrix[1] * scale.y, + rotationMatrix[2] * scale.z); + mat3 covariance3d = scaled * transpose(scaled); + + // Focal lengths in pixels, recovered from the projection matrix so this + // follows whatever FOV and viewport the camera currently has. + float fx = projectionMatrix[0][0] * viewport.x * 0.5; + float fy = projectionMatrix[1][1] * viewport.y * 0.5; + + // Jacobian of the perspective projection at this splat's camera position. + mat3 jacobian = mat3( + fx / cam.z, 0.0, -(fx * cam.x) / (cam.z * cam.z), + 0.0, fy / cam.z, -(fy * cam.y) / (cam.z * cam.z), + 0.0, 0.0, 0.0); + mat3 world = transpose(mat3(modelView)); + mat3 transform = world * jacobian; + mat3 covariance2d = transpose(transform) * covariance3d * transform; + + // A low-pass term keeps sub-pixel splats from vanishing entirely, which is + // what 3DGS calls the dilation filter. + float a = covariance2d[0][0] + 0.3; + float b = covariance2d[0][1]; + float c = covariance2d[1][1] + 0.3; + float mid = 0.5 * (a + c); + float radius = length(vec2(0.5 * (a - c), b)); + float lambda1 = mid + radius; + float lambda2 = max(mid - radius, 0.1); + // A splat smaller than a pixel contributes nothing; skip its quad. + if (lambda1 < 0.02) { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + vColor = vec4(0.0); + vGaussian = vec2(0.0); + return; + } + + // The eigenvector of the 2D covariance, guarded against the isotropic case. + // + // This guard is essential, not defensive. For a near-isotropic splat the + // off-diagonal b tends to 0 AND lambda1 tends to a, so the unguarded + // normalize(vec2(b, lambda1 - a)) is normalize(vec2(0, 0)) = NaN. A NaN + // gl_Position produces no primitive at all, silently: every intermediate + // value reads correct, 100k triangles are submitted, and not one pixel is + // shaded. Vega's Gaussians are close to isotropic (log scales within 0.3 of + // each other), so this is the common case here, not a rare one. + // + // When the covariance really is isotropic, any orthonormal basis is a + // correct pair of axes, so falling back to the x-axis loses nothing. + vec2 eigenDirection = vec2(b, lambda1 - a); + float eigenLength = length(eigenDirection); + vec2 majorAxis = eigenLength > 1e-6 + ? eigenDirection / eigenLength + : vec2(1.0, 0.0); + // Clamped so one degenerate splat cannot ask for a screen-filling quad. + vec2 axis1 = min(sqrt(2.0 * lambda1), 1024.0) * majorAxis; + vec2 axis2 = min(sqrt(2.0 * lambda2), 1024.0) * vec2(majorAxis.y, -majorAxis.x); + + vec2 offset = quadPosition.x * axis1 + quadPosition.y * axis2; + vColor = vec4(color.rgb, opacity); + // Two sigma across the quad, matching the 4.0 cutoff in the fragment stage. + vGaussian = quadPosition * 2.0; + + gl_Position = vec4( + clip.xy / clip.w + offset / viewport * 2.0, + clip.z / clip.w, 1.0); +#endif +} +`; + +const FRAGMENT_SHADER = /* glsl */` +precision highp float; + +in vec4 vColor; +in vec2 vGaussian; + +out vec4 fragColor; + +void main() { + float power = -dot(vGaussian, vGaussian); + // Beyond two sigma the contribution is under 2%; discarding there saves + // most of the fill cost with no visible change. + if (power < -4.0) discard; + float alpha = exp(0.5 * power) * vColor.a; + if (alpha < 1.0 / 255.0) discard; + // Premultiplied alpha, to match the ONE / ONE_MINUS_SRC_ALPHA blend. + fragColor = vec4(vColor.rgb * alpha, alpha); +} +`; + +/** + * One object's splat cloud: a data texture plus a sortable index attribute. + */ +class SplatObject { + /** + * @param {object} args + * @param {number} args.capacity + * @param {'isotropic'|'anisotropic'} [args.mode] quad sizing. Isotropic is + * the default; see the ISOTROPIC_SPLATS note in the vertex shader. + */ + constructor({ capacity, mode = 'isotropic' }) { + this.mode = mode; + this._bound = false; + this.capacity = 0; + this.count = 0; + this.geometry = new THREE.InstancedBufferGeometry(); + + // A unit quad, two triangles, shared by every instance. + this.geometry.setAttribute('quadPosition', + new THREE.BufferAttribute(new Float32Array([ + -1, -1, 1, -1, 1, 1, -1, 1 + ]), 2)); + this.geometry.setIndex([0, 1, 2, 0, 2, 3]); + + this.material = new THREE.RawShaderMaterial({ + vertexShader: VERTEX_SHADER, + fragmentShader: FRAGMENT_SHADER, + glslVersion: THREE.GLSL3, + defines: mode === 'isotropic' ? { ISOTROPIC_SPLATS: '' } : {}, + uniforms: { + splatData: { value: null }, + splatTextureSize: { value: new THREE.Vector2(1, 1) }, + viewport: { value: new THREE.Vector2(1, 1) }, + splatScale: { value: 1.0 } + }, + transparent: true, + // 3DGS blending: the ordering carries the depth information, so + // depth test and write must both be off or splats hard-clip each + // other instead of blending. + depthTest: false, + depthWrite: false, + blending: THREE.CustomBlending, + blendSrc: THREE.OneFactor, + blendDst: THREE.OneMinusSrcAlphaFactor, + blendSrcAlpha: THREE.OneFactor, + blendDstAlpha: THREE.OneMinusSrcAlphaFactor + }); + + this.mesh = new THREE.Mesh(this.geometry, this.material); + this.mesh.frustumCulled = false; // bounds change every frame + // Hidden until setFrame supplies data. Not cosmetic: a visible mesh is + // bound by the render loop, and being bound at a placeholder capacity + // is what caps the instance count forever (see _allocate). + this.mesh.visible = false; + this._allocate(Math.max(capacity, 1)); + + // Sort scratch, reused across frames. + this._depths = new Float32Array(0); + this._counts = new Uint32Array(SORT_BUCKETS); + this._sorted = new Float32Array(0); + } + + /** + * Grow the data texture and the sort index to hold `capacity` splats. + * + * Growing AFTER the geometry has been drawn once is a trap. Three caches + * `_maxInstanceCount` from the instanced attributes the first time it sets + * up the vertex bindings, and the draw count is + * `min(geometry.instanceCount, _maxInstanceCount)`. Swapping in a larger + * `splatIndex` attribute later does not always reset that cache, so a + * geometry first bound at capacity 1 keeps drawing ONE instance no matter + * what `instanceCount` says -- two triangles instead of two hundred + * thousand, which looks exactly like an empty canvas. + * + * Measured: identical draw calls, 213,224 triangles when the growth + * happened before the first bind and 4 when it happened after, varying + * per page load. Callers therefore pass the real capacity up front (see + * `VegaClient`), and the mesh stays hidden until it has a frame so it + * cannot be bound at a placeholder size. + */ + _allocate(capacity) { + if (capacity <= this.capacity) return; + if (this._bound) { + // Not reachable when the caller sized this correctly, and a loud + // failure beats silently rendering one splat out of sixty thousand. + throw new Error( + `SplatObject grew from ${this.capacity} to ${capacity} after it ` + + 'was first drawn; construct it with the maximum splat count ' + + 'for the sequence instead'); + } + this.capacity = capacity; + const texels = capacity * TEXELS_PER_SPLAT; + const height = Math.ceil(texels / TEXTURE_WIDTH); + this._textureData = new Float32Array(TEXTURE_WIDTH * height * 4); + this._texture?.dispose(); + this._texture = new THREE.DataTexture( + this._textureData, TEXTURE_WIDTH, height, + THREE.RGBAFormat, THREE.FloatType); + this._texture.needsUpdate = true; + this.material.uniforms.splatData.value = this._texture; + this.material.uniforms.splatTextureSize.value.set(TEXTURE_WIDTH, height); + + this._indexAttribute = new THREE.InstancedBufferAttribute( + new Float32Array(capacity), 1); + this._indexAttribute.setUsage(THREE.DynamicDrawUsage); + this.geometry.setAttribute('splatIndex', this._indexAttribute); + this._depths = new Float32Array(capacity); + this._sorted = new Float32Array(capacity); + } + + /** Upload a decoded VGS frame. */ + setFrame(frame) { + this._allocate(frame.count); + this.count = frame.count; + const data = this._textureData; + for (let i = 0; i < frame.count; i++) { + const base = i * TEXELS_PER_SPLAT * 4; + data[base] = frame.positions[i * 3]; + data[base + 1] = frame.positions[i * 3 + 1]; + data[base + 2] = frame.positions[i * 3 + 2]; + data[base + 3] = frame.opacities[i]; + + data[base + 4] = frame.scales[i * 3]; + data[base + 5] = frame.scales[i * 3 + 1]; + data[base + 6] = frame.scales[i * 3 + 2]; + data[base + 7] = 0; + + data[base + 8] = frame.rotations[i * 4]; + data[base + 9] = frame.rotations[i * 4 + 1]; + data[base + 10] = frame.rotations[i * 4 + 2]; + data[base + 11] = frame.rotations[i * 4 + 3]; + + data[base + 12] = frame.colors[i * 3]; + data[base + 13] = frame.colors[i * 3 + 1]; + data[base + 14] = frame.colors[i * 3 + 2]; + data[base + 15] = 0; + } + this._texture.needsUpdate = true; + this._positions = frame.positions; + this.geometry.instanceCount = frame.count; + this._sortedForKey = null; + this.mesh.visible = frame.count > 0; + // From here the geometry may be bound at this capacity, so growth is + // no longer safe. + this._bound = this._bound || frame.count > 0; + } + + /** + * Order splats back to front for the given camera. + * + * Counting sort over depth quantised to 16 bits. Re-sorting only when the + * camera or the frame actually changed keeps a static view free. + */ + sort(camera) { + if (this.count === 0 || !this._positions) return false; + const matrix = camera.matrixWorldInverse.elements; + // Third row of the view matrix gives camera-space z directly. + const m2 = matrix[2], m6 = matrix[6], m10 = matrix[10], m14 = matrix[14]; + const key = `${m2.toFixed(5)},${m6.toFixed(5)},${m10.toFixed(5)},` + + `${m14.toFixed(3)},${this.count}`; + if (this._sortedForKey === key) return false; + this._sortedForKey = key; + + const depths = this._depths; + let min = Infinity; + let max = -Infinity; + for (let i = 0; i < this.count; i++) { + const z = m2 * this._positions[i * 3] + + m6 * this._positions[i * 3 + 1] + + m10 * this._positions[i * 3 + 2] + m14; + depths[i] = z; + if (z < min) min = z; + if (z > max) max = z; + } + const span = max - min || 1; + const counts = this._counts.fill(0); + const scale = (SORT_BUCKETS - 1) / span; + // Camera-space z is negative in front of the camera, so ASCENDING z is + // far to near — exactly the back-to-front order the blend needs. + for (let i = 0; i < this.count; i++) { + counts[((depths[i] - min) * scale) | 0]++; + } + let running = 0; + for (let bucket = 0; bucket < SORT_BUCKETS; bucket++) { + const value = counts[bucket]; + counts[bucket] = running; + running += value; + } + const order = this._indexAttribute.array; + for (let i = 0; i < this.count; i++) { + order[counts[((depths[i] - min) * scale) | 0]++] = i; + } + this._indexAttribute.needsUpdate = true; + this._indexAttribute.updateRanges = [{ start: 0, count: this.count }]; + return true; + } + + dispose() { + this.geometry.dispose(); + this.material.dispose(); + this._texture?.dispose(); + } +} + +module.exports = { + SplatObject, VERTEX_SHADER, FRAGMENT_SHADER, + TEXELS_PER_SPLAT, TEXTURE_WIDTH, SORT_BUCKETS +}; diff --git a/open4d/streaming/system/WebClient/src/texture-decoder.js b/open4d/streaming/system/WebClient/src/texture-decoder.js new file mode 100644 index 00000000..c72da281 --- /dev/null +++ b/open4d/streaming/system/WebClient/src/texture-decoder.js @@ -0,0 +1,208 @@ +'use strict'; + +/** + * Texture clip decode: MP4 demux (mp4box) + WebCodecs VideoDecoder. + * + * WebCodecs rather than an HTMLVideoElement on purpose. Geometry arrives as one + * Draco mesh per frame, and a presented frame must pair frame N of the texture + * with frame N of the geometry. A `