sensing: add the SENSING SG8A SIPL camera plugin

CMakeLists.txt

@@ -71,7 +71,7 @@ option(BUILD_PLUGIN_NOITOM_MOCAP "Build Noitom mocap plugin (downloads MocapApi
 option(BUILD_PLUGIN_OGLO "Build OGLO tactile glove plugin (BLE, Linux only; fetches SimpleBLE + nlohmann/json)" OFF)
 option(BUILD_PLUGIN_WUJI_GLOVE "Build Wuji glove plugin (requires the wuji_sdk C SDK)" OFF)
 option(BUILD_PLUGIN_SHARPA_AVATAR "Build Sharpa AvatarExo Glove plugin (Linux; skips if the Avatar SDK is missing)" OFF)
-option(BUILD_PLUGIN_SENSING "Build SENSING GMSL camera plugin (Jetson only)" OFF)
+option(BUILD_PLUGIN_SENSING "Build SENSING GMSL camera plugin (Jetson only; requires SIPL + NVENC)" OFF)
 option(BUILD_EXAMPLES "Build examples" ON)
 option(BUILD_EXAMPLE_TELEOP_ROS2 "Build only the teleop_ros2 ROS 2 reference integration (e.g. for Docker)" OFF)
 option(BUILD_TESTING "Build unit tests" ON)

docs/source/device/camera/sensing.rst

@@ -10,6 +10,11 @@ through NVIDIA SIPL. Choose either setup below to preview both cameras with
 terminal on the Jetson desktop, at the Isaac Capture repository root. The
 preview appears on the Jetson monitor.
 
+The ``camera_plugin_sensing`` C++ plugin captures from the same cameras,
+encoding H.264 on the Jetson's V4L2 engine or handing frames to another
+process as CUDA device memory. Source and scripts live in
+:code-dir:`src/plugins/sensing`.
+
 Prerequisites
 -------------
 
@@ -113,6 +118,172 @@ For custom images, :code-file:`docker/Dockerfile.example
 ``--json`` emits the same flags as a JSON array. These runtime helpers do not
 install the SDK headers needed to compile SIPL applications.
 
+Build prerequisites
+~~~~~~~~~~~~~~~~~~~
+
+Building ``camera_plugin_sensing`` needs two Jetson SDKs that the setup scripts do not install.
+Take the versions matching the L4T release in ``/etc/nv_tegra_release`` (the examples use
+R39.2.1). CMake stops with these same instructions when either is missing.
+
+- **Jetson SIPL API headers** — ``Jetson_SIPL_API_R<rel>_aarch64.tbz2``, for example
+  `R39.2.1
+  <https://developer.nvidia.com/downloads/embedded/L4T/r39_Release_v2.1/release/Jetson_SIPL_API_R39.2.1_aarch64.tbz2>`_.
+  Unpack it with ``sudo tar xf Jetson_SIPL_API_R39.2.1_aarch64.tbz2 -C /``, which creates
+  ``/usr/src/jetson_sipl_api``.
+- **Jetson Multimedia API** — the ``nvidia-l4t-jetson-multimedia-api_<rel>-*_arm64.deb`` package
+  from the `Jetson package pool
+  <https://repo.download.nvidia.com/jetson/common/pool/main/n/nvidia-l4t-jetson-multimedia-api/>`_.
+  Unpack it without installing it:
+  ``dpkg-deb -x nvidia-l4t-jetson-multimedia-api_<rel>-*_arm64.deb /tmp/mmapi`` then
+  ``sudo cp -a /tmp/mmapi/usr/src/jetson_multimedia_api /usr/src/``.
+
+If they live elsewhere, pass ``-DSIPL_API_DIR=PATH`` and ``-DJETSON_MMAPI_DIR=PATH`` to CMake.
+
+Building and running
+--------------------
+
+.. code-block:: bash
+
+   cmake -B build -DBUILD_PLUGIN_SENSING=ON
+   cmake --build build --target camera_plugin_sensing
+
+The platform config is vendored at ``src/plugins/sensing/configs/shw5g.json``
+and resolved relative to the executable, so the plugin runs without the vendor
+package on disk; ``--platform-config=PATH`` points it at a newer vendor drop.
+
+.. note::
+
+   The published package has **no** ``SHW5G_2`` config — only the mixed
+   populations. SENSING sends ``shw5g.json`` directly to customers with a
+   2× SHW5G rig, so both the plugin and the reference-tool commands above use the
+   tracked copy.
+
+Start by asking the platform config what exists. This needs no hardware — the
+query API only parses the driver database and the JSON, so it works with the
+cameras unplugged:
+
+.. code-block:: bash
+
+   ./build/src/plugins/sensing/app/camera_plugin_sensing --list-sensors
+
+.. code-block:: text
+
+   SHW5G_2: 2 sensor(s)
+     sensor=0  SHW5G  2560x1984 @ 60 fps
+     sensor=1  SHW5G  2560x1984 @ 60 fps
+
+.. warning::
+
+   ``sensor=N`` is the **SIPL pipeline index**, and for ``SHW5G_2`` it is not
+   the number the JSON appears to name. The GMSL link indices, the CSI virtual
+   channels and the JSON's ``sensorInfo.id`` are all **2 and 3**; the pipeline
+   indices are **0 and 1**. Always take the number from ``--list-sensors``.
+
+   In the vendor's ``S56C_1_SHF3L_2`` config the two coincide, which is exactly
+   why this is worth stating.
+
+Geometry comes from the platform config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is no ``--width``, ``--height``, ``--fps`` or ``--sensor-mode``. SIPL has
+no runtime mode index; resolution and frame rate are properties of the virtual
+channel in the platform config, and the plugin sizes its buffers from what the
+query reports.
+
+Encoding at 5 MP60
+~~~~~~~~~~~~~~~~~~
+
+Two sensors at 2560×1984 @ 60 is 4.9× the pixel rate the 1080p30 defaults were
+chosen for, so the codec settings moved with it:
+
+.. list-table::
+   :widths: 25 20 55
+   :header-rows: 1
+
+   * - Setting
+     - Value
+     - Why
+   * - level
+     - **5.2**
+     - 1,190,400 MB/s exceeds Level 5.1's 983,040 by 21%
+   * - ``--bitrate``
+     - 40 Mbps
+     - 0.13 bits/pixel; 20 Mbps was 0.066
+   * - ``--peak-bitrate``
+     - 60 Mbps
+     - VBR ceiling; pass ``0`` to select CBR
+
+80 Mbps for the pair, about 36 GB/hour.
+
+Live streaming with camera_viz
+------------------------------
+
+``ipc=<socket>`` serves a sensor's frames to another process as **CUDA device
+memory** — RGBA8, no encode, no host round-trip — and
+:code-file:`camera_viz <examples/camera_viz/README.md>` consumes it with
+``type: cuda_ipc``. It is independent of ``output=``; an ``ipc``-only stream
+never starts an encoder.
+
+.. code-block:: bash
+
+   # terminal 1 — producer
+   ./build/src/plugins/sensing/app/camera_plugin_sensing \
+       --add-stream=sensor=0,ipc=/tmp/sensing0.sock \
+       --add-stream=sensor=1,ipc=/tmp/sensing1.sock
+
+   # terminal 2 — viewer
+   cd examples/camera_viz
+   ./camera_viz.sh setup                                     # one-time
+   ./camera_viz.sh run configs/cuda_ipc.yaml --mode window   # desktop window
+   ./camera_viz.sh run configs/cuda_ipc.yaml                 # XR headset
+
+:code-file:`configs/cuda_ipc.yaml <examples/camera_viz/configs/cuda_ipc.yaml>`
+needs only the socket path and the frame size:
+
+.. code-block:: yaml
+
+   cameras:
+     - name: left
+       type: cuda_ipc
+       socket: /tmp/sensing0.sock   # must match the producer's ipc= path
+       width: 2560                  # must match what the producer serves
+       height: 1984
+
+Order does not matter — the source retries until the socket appears and survives
+the producer restarting under it. ``width``/``height`` are checked during the
+handshake and a mismatch is refused rather than rendered at the wrong stride.
+**One consumer at a time**: the producer serves whoever connected most recently,
+so a second viewer silently takes the feed from the first.
+
+Pairing the two eyes
+~~~~~~~~~~~~~~~~~~~~
+
+Both sensors are driven from one deserializer fsync generator, and every frame
+carries SIPL's ``frameCaptureTSC`` on a timebase shared across the whole rig.
+It is recorded in the MCAP metadata as ``capture_tsc_ns``.
+
+.. important::
+
+   Pair on ``capture_tsc_ns``, not on the wrapper timestamps. The wrapper stamps
+   are ``CLOCK_MONOTONIC`` taken after per-sensor conversion, so they carry each
+   sensor's own queueing jitter; the TSC does not.
+
+Testing without a camera
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+``sensing_ipc_testsrc`` publishes an animated pattern over the same protocol. It
+needs CUDA only — no SIPL, no encoder — so the viewer can be developed with
+nothing attached:
+
+.. code-block:: bash
+
+   cmake --build build --target sensing_ipc_testsrc
+   ./build/src/plugins/sensing/tools/sensing_ipc_testsrc --socket=/tmp/sensing0.sock \
+       --width=2560 --height=1984 --fps=60
+
+Each frame carries its 16-bit frame number as a binary bar across the top, so a
+stale or torn frame is visible rather than merely suspected.
+
 Troubleshooting
 ---------------
 
@@ -155,3 +326,11 @@ on the side that fails for a read-only runtime report:
    * - Low FPS or dropped frames
      - Check the Jetson power mode. Maximum performance can be selected with
        ``sudo nvpmodel -m 0 && sudo jetson_clocks``.
+   * - ``sensor=N is not a pipeline in '<config>'``
+     - Use the pipeline index from ``--list-sensors``, not the link index or JSON id.
+   * - ``ISP0 reconciled to colour standard ...``
+     - The ISP refused BT.601; see the note in ``sipl_camera.cpp``.
+
+The plugin clears ``DISPLAY`` before its first EGL call so headless capture
+selects the Tegra driver. EGL is required to map SIPL buffers into CUDA even
+when the plugin does not render.

docs/source/references/camera_streaming.rst

@@ -365,7 +365,8 @@ The source kind is selected by the ``type`` field of each entry in the YAML ``ca
        ``nvargus-daemon``. Build setup with ``--with-argus`` first.
    * - ``cuda_ipc``
      - RGBA8 frames mapped straight out of another process's CUDA memory — no encode, no host
-       copy. See :ref:`camera-streaming-cuda-ipc`.
+       copy. Pairs with the :doc:`sensing plugin </device/camera/sensing>`; see
+       :ref:`camera-streaming-cuda-ipc`.
    * - ``video``
      - Recorded camera data — video-file replay (anything OpenCV's FFmpeg backend reads). Loops
        by default; ``stereo: true`` splits side-by-side recordings into eyes (viewer only). See
@@ -377,7 +378,7 @@ The source kind is selected by the ``type`` field of each entry in the YAML ``ca
 .. _camera-streaming-cuda-ipc:
 
 Frames from another process (``cuda_ipc``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 A producer on the same machine publishes captured frames as CUDA device memory; camera_viz maps
 that memory once and renders from it. Nothing is encoded and nothing crosses host RAM, so the
@@ -398,9 +399,17 @@ restarting under it. ``width`` and ``height`` are checked during the handshake,
 refused rather than rendered at the wrong stride. ``stereo: true`` is rejected, because each
 producer serves one sensor — declare one entry per socket.
 
-Any process can be the producer: the wire format is a 24-byte message and a CUDA VMM handle over
-``SCM_RIGHTS``. An animated test pattern ships with it, so the consumer can be developed with no
-camera and no capture SDK:
+The producer is the sensing plugin. Geometry comes from its platform config, not from flags, and
+``sensor=`` is the **SIPL pipeline index** — take it from ``--list-sensors``, not from the vendor
+JSON, where the same cameras are numbered differently:
+
+.. code-block:: bash
+
+   camera_plugin_sensing --add-stream=sensor=0,ipc=/tmp/s0.sock \
+                         --add-stream=sensor=1,ipc=/tmp/s1.sock
+
+To develop against this without a camera, the plugin ships an animated test pattern that speaks
+the same protocol:
 
 .. code-block:: bash
 

src/core/schema/fbs/sensing.fbs

@@ -0,0 +1,39 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+include "timestamp.fbs";
+
+namespace core;
+
+// Per-frame metadata pushed by the SENSING camera plugin (one per stream).
+// Streams are keyed by SIPL pipeline index rather than an enum: the populated
+// set varies with the platform config the rig is running.
+// All fields are always present when the parent Tracked/Record wrapper's data is non-null.
+table FrameMetadataSensing {
+  // SIPL pipeline index, as INvSIPLCameraQuery reports it for the active
+  // platform config. This is NOT the GMSL link index, the CSI virtual channel,
+  // or the JSON sensorInfo.id -- for SHW5G_2 those are 2 and 3 while this is
+  // 0 and 1.
+  sensor_id: uint32 (id: 0);
+
+  // Per-stream counter of records emitted for this sensor. Not a SIPL capture
+  // index: a stream with an encoder counts bitstream units, one per captured
+  // frame, and an ipc-only stream counts published frames.
+  sequence_number: uint64 (id: 1);
+
+  // SIPL frameCaptureTSC: capture time on the Tegra TSC timebase, which every
+  // sensor on the rig shares because the deserializer drives them from one
+  // fsync generator. This is the field that lets a consumer pair the two eyes
+  // of a stereo rig; the wrapper timestamps cannot, since they are stamped
+  // after per-sensor conversion. Zero when the encoder dropped the frame this
+  // record's stamp came from.
+  capture_tsc_ns: uint64 (id: 2);
+}
+
+// MCAP recording wrapper for FrameMetadataSensing.
+table FrameMetadataSensingRecord {
+    data: FrameMetadataSensing (id: 0);
+    timestamp: DeviceDataTimestamp (id: 1);
+}
+
+root_type FrameMetadataSensingRecord;

src/plugins/CMakeLists.txt

@@ -43,7 +43,7 @@ endif()
 if(BUILD_PLUGIN_SHARPA_AVATAR)
     add_subdirectory(sharpa_avatar)
 endif()
-# Gated: Jetson only -- the Tegra CUDA and NVENC entry points are L4T-only.
+# Gated: Jetson only -- SIPL and the Tegra NVENC entry point are L4T-only.
 if(BUILD_PLUGIN_SENSING)
     add_subdirectory(sensing)
 endif()

src/plugins/sensing/AGENTS.md

@@ -21,3 +21,5 @@ Container preview must carry the desktop display and authorization into the imag
 
 Support SSH launch with a local DISPLAY: when XAUTHORITY is unset, check the
 current user's GDM authority file as well as ~/.Xauthority.
+
+Include CUDA `.cu` and `.cuh` files in clang-format-14 checks for this plugin.