Interactive seg and stereo docs

client/README.md

@@ -21,8 +21,9 @@ npm run serve
 npm run build:web
 
 # Electron
-npm run serve:electron
+npm run serve:electron   # alias: npm run dev:electron
 npm run build:electron
+npm run build:electron:dir   # unpacked build (no installer) for local testing
 
 # lint and test
 npm run lint

client/platform/desktop/README.md

@@ -49,8 +49,20 @@ After `electron-vite build`, **`electron-builder --config electron-builder.json`
 
 ### npm scripts (in `client/package.json`)
 
-* **`serve:electron`** — `electron-vite dev`: compiles main/preload, serves the renderer from Vite, opens Electron with `ELECTRON_ENTRY=.electron/main/background.js` (and related env).
+* **`serve:electron`** / **`dev:electron`** — `electron-vite dev`: compiles main/preload, serves the renderer from Vite, opens Electron with `ELECTRON_ENTRY=.electron/main/background.js` (and related env). Use this to develop interactive segmentation and stereo features locally.
 * **`build:electron`** — `electron-vite build` then `electron-builder --config electron-builder.json`.
+* **`build:electron:dir`** — same as `build:electron` but produces an unpacked directory instead of an installer (faster iteration for testing).
+
+## Interactive service (segmentation + stereo)
+
+Desktop-only interactive annotation runs through a single persistent Python subprocess managed by `backend/native/interactive.ts`. It hosts:
+
+* **Interactive segmentation** — point-click mask prediction (`backend/native/segmentation.ts`, IPC via `ipcService.ts`)
+* **Interactive stereo** — line transfer, length measurement, dense disparity (`backend/native/stereo.ts`)
+
+Models load lazily on first use. The renderer calls into the service through `frontend/api.ts`; shared UI types live in `dive-common/apispec.ts`. The segmentation recipe is `dive-common/recipes/segmentationpointclick.ts`; stereo wiring is in `platform/desktop/frontend/components/ViewerLoader.vue` and `dive-common/use/useModeManager.ts`.
+
+User documentation: [Interactive Annotation](../../docs/Interactive-Annotation.md).
 
 ## Main process, preload, and renderer
 

docs/Annotation-QuickStart.md

@@ -132,3 +132,24 @@ Every track is required to have a bounding box, but a polygon region may be adde
 ### Polygon Demo
 
 ![Polygon Demo](videos/CreationModes/PolygonAnnotation.gif)
+
+### Multi-polygon regions with holes
+
+DIVE supports multiple polygon regions on one detection and polygons with interior holes (for example, annotating a ring-shaped object). Holes are preserved in DIVE JSON and [VIAME CSV export](DataFormats.md#viame-csv-polygons-and-length) using `(poly)` and `(hole)` columns.
+
+## Interactive Segmentation (Desktop)
+
+[Interactive point-click segmentation](Interactive-Annotation.md#interactive-segmentation) is available in DIVE Desktop only. It helps you create polygon annotations by clicking include/exclude points instead of placing every vertex manually.
+
+1. Create or select a track/detection (**right-click** to edit).
+2. Click ==Segment== in the [editing bar](UI-Navigation-Editing-Bar.md) or press ++s++.
+3. **Left-click** on the object to add foreground points (green).
+4. **Shift+click** or **middle-click** on areas to exclude (red background points).
+5. **Right-click** or press ++enter++ to confirm and commit the polygon.
+6. Press ++escape++ to cancel; use ==Reset Points== to clear points without leaving segment mode.
+
+![Segmentation mode](images/CreationMode/SegmentationMode.png)
+
+On stereo datasets, confirmed segmentations can be warped to the other camera when [interactive stereo auto-compute](Multicamera-data.md#interactive-stereo-desktop) is enabled.
+
+See [Interactive Annotation](Interactive-Annotation.md) for multi-frame behavior, continuous detection mode, and troubleshooting.

docs/Annotation-User-Interface-Overview.md

@@ -10,6 +10,7 @@ This documentation section provides a reference guide to the annotation interfac
 * **[Track List](UI-Track-List.md)** - List of all the tracks as well as providing a way to perform editing functions on those tracks.
 * **[Timeline](UI-Timeline.md)** - timeline view of tracks and detections, as well as an interface to control the current frame along the video/image-sequence
 * **[Attributes](UI-Attributes.md)** - Attributes panel used to assign attributes to individual tracks or detections.
+* **[Interactive Annotation](Interactive-Annotation.md)** (Desktop) - Point-click segmentation and interactive stereo tools for faster polygon and multicam annotation.
 * **Context Sidebar** - Open from the [editing bar](UI-Navigation-Editing-Bar.md#context-sidebar-web) (Web). The right sidebar has several view modes selected from a dropdown at the top of the panel.
     * **[Dataset Info](UI-DatasetInfo.md)** - View dataset properties and attach custom dataset-level metadata that is shown while annotating and included in CSV export.
     * **[Revision History](Web-Version.md#revision-history)** (Web) - Inspect and check out past saved annotation states.

docs/Architecture-For-New-Developers.md

@@ -418,8 +418,11 @@ Optional methods on **`Api`** (for example `getTiles`, `getTileURL`, `getLastCal
 - **Renderer:** `main.ts`, `frontend/` — same viewer stack as web
 - **Backend:** `backend/server.ts` — Express routes mirroring Girder
 - **Native jobs:** `backend/native/` — VIAME pipelines, training, media jobs
+- **Interactive service:** `backend/native/interactive.ts` — unified segmentation + stereo subprocess (desktop only)
 - **Serializers:** `backend/serializers/` — format conversion (viame, coco, kpf, dive, …)
-- **Build:** `npm run build:electron` → `dist_electron/`
+- **Build:** `npm run build:electron` → `dist_electron/`; `npm run dev:electron` for local development
+
+Shared segmentation recipe: `dive-common/recipes/segmentationpointclick.ts`. User docs: [Interactive Annotation](Interactive-Annotation.md).
 
 More detail: [client/platform/desktop/README.md](https://github.com/Kitware/dive/blob/main/client/platform/desktop/README.md).
 

docs/DataFormats.md

@@ -212,6 +212,32 @@ entry keyed `dataset_info`:
   annotations to an external database, travels with the exported annotations without
   renaming files. See the [Dataset Info panel](UI-DatasetInfo.md) for how to populate it.
 
+### VIAME CSV polygons and length
+
+DIVE extends standard VIAME CSV with additional geometry and measurement fields:
+
+**Polygons** — one or more `(poly)` columns per row, each followed by flat `x y` coordinate pairs:
+
+```
+0,1.png,0,100,100,500,500,1.0,-1,fish,1.0,(poly) 100 100 200 100 200 200 100 200
+```
+
+**Multiple polygons** — additional `(poly)` columns on the same row:
+
+```
+..., (poly) 100 100 200 100 200 200 100 200, (poly) 300 300 400 300 400 400 300 400
+```
+
+**Holes** — `(hole)` columns follow the outer `(poly)` they belong to:
+
+```
+..., (poly) 100 100 500 100 500 500 100 500, (hole) 200 200 400 200 400 400 200 400
+```
+
+Multiple holes are supported with additional `(hole)` columns.
+
+**Length measurements** — the standard VIAME length column (8th numeric field) stores stereo fish-length values. DIVE also reads and writes a `length` entry in detection attributes; on export, the resolved value is written to both the column and attributes when present. Interactive stereo in [DIVE Desktop](Interactive-Annotation.md) populates these values during annotation.
+
 ## KWIVER Packet Format (KPF)
 
 DIVE supports [MEVA KPF](https://mevadata.org/)

docs/Dive-Desktop.md

@@ -38,6 +38,15 @@ DIVE Desktop supports single- and multi-camera datasets.
 
 Shared viewer behavior (camera selection, linked tracks, MultiCamera Tools, and pipelines) is documented in [Multicamera and Stereo Data](Multicamera-data.md). The [web version](Web-Version.md#uploading-multicamera-and-stereo-data) supports the same import types with a few desktop-only advanced options.
 
+### Interactive annotation (Desktop only)
+
+DIVE Desktop can run VIAME-backed **interactive segmentation** and **interactive stereo** tools while you annotate. These require a local VIAME install with interactive service support and are not available in the web annotator.
+
+* **Interactive segmentation** — click foreground/background points to generate polygon masks (++s++ or the Segment button). See [Interactive Annotation](Interactive-Annotation.md).
+* **Interactive stereo** — on stereo datasets, auto-warp annotations between cameras and recompute head/tail length measurements. Configure from the **Stereo Settings** section in the track list creation settings menu.
+
+Both features share one lazy-loaded Python interactive service subprocess. GPU is recommended for model initialization.
+
 ### Importing Datasets
 
 Click either ==Open Image Sequence :material-folder-open:== or ==Open Video :material-file-video:== to begin a single camera default import.  Click the ==:material-chevron-down:== dropdown button to show additional import options.
@@ -158,4 +167,8 @@ If you experience problems or have questions about DIVE Desktop, [contact us](Su
 
 It's also helpful to look in the debug console.  Press ++ctrl+shift+i++ to launch the Dev Tools and look under the console tab.  Errors and warnings will appear in red and yellow.  You can right-click in the console area and click "Save As" to save the log file and [open a support ticket](Support.md)
 
+> Interactive segmentation or stereo fails to load
+
+See [Interactive Annotation troubleshooting](Interactive-Annotation.md#troubleshooting). Verify the VIAME install path, confirm VIAME includes interactive service support, and check the debug console for subprocess errors (message: "Unable to load the interactive service").
+
 ![Debugging Desktop](images/General/desktop-debug.png)

docs/FAQ.md

@@ -74,3 +74,7 @@ There are caveats with this approach.
 
 * It does not handle padding properly.  If a video begins or ends with padding, you may see a black screen in DIVE, but kwiver will wait for the first true frame to use as the representative for the bucket.
 * It does not handle variable width frames properly.  If a video has variable width frames, the assumptions about the locations of true frame boundaries do not hold and kwiver training may have alignment issues.
+
+## Does DIVE support interactive segmentation or stereo while annotating?
+
+Yes, on **[DIVE Desktop](Dive-Desktop.md)** only. Interactive point-click segmentation and interactive stereo (auto-warp between cameras, length recompute) require a local VIAME install with interactive service support. See [Interactive Annotation](Interactive-Annotation.md).

docs/Interactive-Annotation.md

@@ -0,0 +1,111 @@
+# Interactive Annotation (Desktop)
+
+DIVE Desktop can run **interactive point-click segmentation** and **interactive stereo** tools backed by a local VIAME Python service. Both features are **desktop-only** and require a working [VIAME installation](Dive-Desktop.md#desktop-settings).
+
+| Feature | Description |
+|---------|-------------|
+| **Interactive segmentation** | Click foreground/background points to generate polygon masks (SAM-style models when available). |
+| **Interactive stereo** | Warp annotations between stereo camera views and recompute length measurements from head/tail lines. |
+
+See also: [Multicamera and Stereo Data](Multicamera-data.md#interactive-stereo-desktop), [Annotation Quickstart](Annotation-QuickStart.md#interactive-segmentation-desktop), [Keyboard shortcuts](Mouse-Keyboard-Shortcuts.md).
+
+## Requirements
+
+* [DIVE Desktop](Dive-Desktop.md) with a valid **VIAME Install Path**
+* For stereo features: a **stereo dataset** imported with a calibration `.npz` file
+* GPU recommended for model load times; CPU fallback may be available depending on VIAME build
+
+On first use, DIVE spawns a single persistent `viame.core.interactive_service` subprocess that loads segmentation and stereo models lazily.
+
+## Interactive Segmentation
+
+![Segmentation mode](images/CreationMode/SegmentationMode.png)
+
+Point-click segmentation helps you draw precise polygon regions faster than placing vertices by hand. It is available on single-camera and multicamera datasets.
+
+### Activating segmentation
+
+1. Create or select a track/detection and enter edit mode (**right-click** the annotation).
+2. Click the ==Segment== button (magic-wand icon) in the [editing bar](UI-Navigation-Editing-Bar.md), or press ++s++.
+3. On first activation, DIVE loads the segmentation model. A loading indicator appears in the status area.
+
+The status indicator shows instructions while segmentation is active:
+
+* **Left click** — add a foreground (include) point
+* **Shift+click** or **middle click** — add a background (exclude) point
+* **Right click** or ++enter++ — confirm and commit the polygon
+* ++escape++ — cancel and clear points (restores the previous polygon if you were editing)
+
+Green dots are foreground points; red dots are background points.
+
+### Resetting points
+
+While in Point (segmentation) mode, click ==Reset Points :material-undo:== in the editing bar to clear prompt points and revert to the pre-segmentation polygon without leaving segmentation mode.
+
+### Multi-frame and video
+
+* Points are tracked **per frame**. Switching frames saves the current frame's points and shows only the active frame's prompt dots.
+* Confirming commits all frames that have valid polygons.
+* On video datasets, the service receives the frame timestamp so models can seek the correct frame.
+
+### Continuous detection mode
+
+In **Detection** mode with **Continuous** enabled, each foreground click can start a new detection. Background clicks do not spawn new detections.
+
+### Stereo datasets
+
+On stereo datasets, confirming a segmentation on one camera can warp the resulting polygon to the paired camera when [interactive stereo](#interactive-stereo) auto-compute is enabled.
+
+!!! note
+
+    If the primary segmentation model fails to load, DIVE may fall back to a GrabCut-based method with reduced accuracy. Check the desktop debug console (++ctrl+shift+i++) for details.
+
+## Interactive Stereo
+
+Interactive stereo assists annotation on **two-camera stereo datasets** with a calibration file. It is controlled from the **Stereo Settings** section of the [creation settings menu](UI-Track-List.md) (gear icon in the track list or bottom panel).
+
+![Stereo settings](images/CreationMode/StereoSettings.png)
+
+| Setting | Behavior |
+|---------|----------|
+| **Update lengths when modified** | When you edit a head/tail line on a detection linked across both cameras, DIVE recomputes stereo measurements (length, midpoint, range, RMS). |
+| **Auto-compute location on other camera** | When you draw an annotation on one camera and the other camera has no detection yet, DIVE warps it to the other view using dense stereo disparity. |
+
+Enabling either toggle loads the interactive stereo service. A loading dialog appears while models initialize.
+
+### Warped annotations
+
+Stereo-warped head/tail lines become **normal editable line annotations**. If you manually edit a warped line, DIVE locks it from automatic re-warping so your corrections are preserved.
+
+Auto-created head/tail connector lines use the standard head/tail line key and behave like manually drawn lines.
+
+### Multicamera workflow improvements
+
+* **One-click cross-camera edit** — selecting a linked track on one camera selects it on all camera views and enables editing.
+
+![Cross-camera edit](images/MultiCam/CrossCameraEdit.png)
+
+* **Create on any camera** — start a new linked detection from any camera view without breaking the multicam workflow.
+* **Line aspect ratio** — auto-warped line bounding boxes are capped at a 6:1 aspect ratio for readability.
+
+### Measurements and export
+
+Computed lengths are stored as detection attributes (for example `length`) and round-trip through [VIAME CSV export](DataFormats.md#viame-csv-polygons-and-length). The legacy VIAME length column is also populated when a length value is present.
+
+## Troubleshooting
+
+> **Unable to load the interactive service**
+
+Verify your **VIAME Install Path** in Desktop Settings. Ensure VIAME is installed with interactive segmentation/stereo support (see [VIAME installation docs](https://github.com/viame/VIAME#installations)). Open the debug console (++ctrl+shift+i++) and look for stderr from the interactive subprocess.
+
+> **Segmentation button missing or disabled**
+
+Interactive segmentation is only available in DIVE Desktop, not the web annotator.
+
+> **Stereo settings not visible**
+
+Stereo settings appear only on stereo datasets (two cameras + calibration) in DIVE Desktop.
+
+> **Stereo warp did not appear on the other camera**
+
+Confirm **Auto-compute location on other camera** is enabled, the track is linked across cameras, and the other camera has no existing detection at that frame. Human-edited lines are not overwritten.

docs/Mouse-Keyboard-Shortcuts.md

@@ -49,6 +49,18 @@ Most editing controls are available when a track or detection is selected.
 | ++g++ | create new group including the selected track |
 | ++shift+enter++ | focus class select/text box on selected track in track list.  Press ++arrow-down++ to open all options.  Pres ++enter++ twice to accept an option.  Press ++escape++ to unfocus. |
 
+## Interactive Segmentation (Desktop only)
+
+Available when a track or detection is in edit mode. See [Interactive Annotation](Interactive-Annotation.md).
+
+| control    | description |
+| ---------- | ------------|
+| ++s++ | activate **Segment** (point-click segmentation) mode |
+| Left Click | add foreground (include) point |
+| ++shift++ + Left Click or Middle Click | add background (exclude) point |
+| Right Click or ++enter++ | confirm segmentation and commit polygon |
+| ++escape++ | cancel segmentation; clear points and restore previous polygon when editing |
+
 ## Adding new shortcuts
 
 If you think a new shortcut or hotkey would be useful, [please send us feedback](Support.md).