Appearance-weighted tracking cost and per-particle pulsatile gates

.github/workflows/cibuildwheel.yml

@@ -75,7 +75,7 @@ jobs:
           # regression tests unsuited to wheel smoke.
           CIBW_TEST_COMMAND: >
             python {project}/scripts/ci_test_setup.py {project} &&
-            python -c "import openptv2.algorithms.track_kernels_tracking, openptv2.algorithms.track3d; print('openmp import OK')" &&
+            python -c "import openptv2.algorithms.track_kernels_corr, openptv2.algorithms.track3d; print('openmp import OK')" &&
             pytest
             tests/unit/test_vec_utils.py
             tests/unit/test_correspondences.py

docs/index.md

@@ -13,6 +13,8 @@ Welcome to the openptv2 documentation.
 ### User Documentation
 
 - [Tracking Pipeline & Results Guide](tracking_guide.md) - Pipeline workflow, parameter guide, multi-pass tracking, and ptv_is.# output format
+- [Particle Trackers](trackers.md) - Which tracker to use and why: basics, all eight engines, caveats, tips, upstream credits
+- [Two-Phase Tracking](two-phase-tracking.md) - Two-Phase usage and parameters in depth
 - [Lid-Driven Cavity Flow Tutorial](tutorials/cavity_flow_tutorial.md) - End-to-end 3D-PTV case study: Autocalibration, Tracer Shaking, Warmup, and 3D Trajectories
 - [Aortic Pulsatile Flow Tutorial](aorta_tutorial.md) - Cloud-native 3D-PTV on complex aortic flow
 - [Auto-Calibration with `openptv warmup`](tutorials/warmup_tutorial.md) - Standalone parameter/engine auto-tuning before tracking

docs/plans/2026-09-20-tracking-kernels-dedup.md

@@ -0,0 +1,89 @@
+# Tracking-kernels dedup (ponytail-audit items 1-9)
+
+Branch: `refactor/dedup-tracking-kernels` (off `main`). Committed and pushed (0e1200a0, 1b83cfa6); no PR yet.
+Goal: remove duplicated / orphaned code in `src/openptv2/algorithms/track_kernels_*.py`
+without changing behavior.
+
+## What was found
+
+The same Cython `cdef`/`nogil` kernels were copy-pasted across sibling modules
+(cross-module C calls need a `.pxd`, so people copied instead). Bodies were
+identical or differed only in decorators/comments. `track_kernels_pixel.pxd` and
+`track_kernels_position.pxd` already existed as the sharing mechanism, so I
+extended them rather than inventing anything.
+
+## Done
+
+Items 1-5, dedup. Owners:
+- `track_kernels_pixel.py`: `_multimed_r_nlay_1layer` (added `exceptval(check=False)` + pxd entry),
+  `_point_to_pixel_out`, `_candsearch_in_pix_rest_nogil`, `_pixel_to_metric_out`,
+  `_dist_to_flat_out`, `_sorted_candidates_fast_out_nogil`, `candsearch_in_pix_fast_nogil`
+  (the pixel copy is the newer one with `max_cands`/`out_dists`; search's was a stale, dead copy).
+- `track_kernels_position.py`: `_ray_tracing_out` (added exceptval + pxd), `_angle_acc_out`
+  (moved here from geom/corr, added exceptval + pxd), `_point_position_out`,
+  `assess_new_position_fast_nogil`.
+- Copies deleted from geom, search, transform, corr. Importers use the existing
+  `if cython.compiled: cimport ... else: from .x import ...` pattern.
+  Import direction is acyclic: pixel <- position <- geom/transform/corr/batch.
+
+Item 6: deleted the `track_kernels_tracking.py` shim. Repointed `track_kernels.py`,
+`setup.py` (`ALGORITHMS_MODULES`), `.github/workflows/cibuildwheel.yml` (import smoke test),
+and `tests/unit/test_track_kernels_tracking_coverage.py` (now imports from owners; shim
+constants defined locally; `_mod` = the pixel module, which keeps the old inert-patch behavior).
+
+Item 9: deleted 16 wrappers with no non-test callers, plus their tests and the two
+re-exports in `track_kernels.py`:
+`angle_acc_fast`, `_ray_tracing_fast`, `pixel_to_metric_fast`, `dist_to_flat_fast`,
+`metric_to_pixel_fast`, `_metric_to_pixel_out`, `_flat_image_coord_fast`, `_img_coord_fast`,
+`img_coord_batch_fast`, `flat_image_coord_batch_fast`, `point_position_fast`,
+`ray_tracing_batch_fast`, `point_position_batch_fast`, `pixel_to_metric_batch_fast`,
+`metric_to_pixel_batch_fast`, `sort_candidates_by_freq_fast`.
+Also dropped two search test classes that only tested the deleted dead copies.
+
+Net so far: roughly -2,200 lines from dedup, about -840 from item 9, plus tests.
+
+## Verification status (updated 2026-09-21)
+
+Clean Cython rebuild of the final tree: OK.
+- Hot-path tests (`test_track`, `test_track3d`, `test_correspondences`, `test_track4be`): 49 passed = baseline.
+- Full suite `uv run --no-sync pytest tests`: 2017 passed, 86 skipped, 39 deselected (12 min).
+- Pure-Python fallback, kernel coverage files only
+  (`test_track_kernels_*_coverage.py`, 6 files): 266 passed.
+- Pure-Python fallback over the whole `tests/unit/test_*_coverage.py` glob: 1227 passed, 24 failed
+  (38 min; the glob now matches 29 files, not the 16 CLAUDE.md mentions, so it is slow).
+  The failures I inspected (`test_epi_coverage`, `test_correspondences_coverage`, 14 of the 24)
+  are all `Coord2d.__init__() got an unexpected keyword argument 'pnr'` /
+  `Candidate.__init__() ... 'pnr'`: interpreted-mode constructor mismatch in modules this
+  branch does not touch. Not confirmed on `main`; the other 10 were not inspected.
+- Running the suite rewrites tracked `test_data/test_cavity/img/*_targets`; `git checkout -- test_data` before committing.
+
+## Remaining
+
+1. Optional: confirm the 24 fallback failures also occur on `main` (build `main`, run
+   `tests/unit/test_epi_coverage.py tests/unit/test_correspondences_coverage.py` interpreted).
+2. Optional perf sanity: `_angle_acc_out` was `ccall inline` inside corr and is now a cross-module
+   C call (lost inlining). Time a tracking run before/after; if it regressed, keep a private copy in corr.
+3. Cosmetic: stray banner comments in `test_track_kernels_batch_coverage.py` (about lines 84-102) and
+   `test_track_kernels_transform_coverage.py` (lines 22, 352).
+4. Open a PR.
+
+## Deliberately skipped
+
+- **Item 7** (merge forward/backward tracking loops in `track_kernels_corr.py`): not a clean
+  dedup. Normalised diff of `trackcorr_loop_fast` vs `trackback_loop_fast` shows about 670 of about 900
+  lines differ; forward uses the `_trackcorr_particle_fast` worker, backward is inline. Merging
+  changes core tracking logic and there is no golden-output regression data. Only attempt
+  with a recorded before/after trajectory comparison on a real dataset.
+- **Item 8** (delete 4BE tracker: `track4be_loop_fast`, `track4be.py`, `plugins/four_be_tracking.py`):
+  it is registered in `tracking_registry.py` and benchmarked in about 10 scripts/notebooks
+  (`bench_*`, `benchmark_*`, `tracker_tutorial_dashboard.py`). Needs an explicit product decision.
+  Do not delete without Alex saying so.
+
+## Gotchas
+
+- Cimported names are not importable from Python: tests and non-cimporting modules must import
+  the `cpdef` names from the owner module (pixel/position), not from geom/transform.
+- Cython pure mode + `.pxd`: `noexcept nogil` in the pxd requires `@cython.exceptval(check=False)`
+  on the `def`, or the signatures mismatch.
+- Shell cwd drift: `cd` to the repo root at the start of every command.
+- Use `git grep`, not `grep -r`, at the repo root (huge build dirs; a plain grep timed out).

docs/plans/2026-09-24-cloud-wp1-twophase-run-plan.md

@@ -0,0 +1,118 @@
+# Plan: Cloud Run of wp1 with Bidirectional Two-Phase Tracker & Postprocessing
+
+**Date:** 2026-09-24  
+**Target Dataset:** `C:\Users\alex\Downloads\HiDImaging\CompleteTest\wp1\test`  
+**GCP Environment:** Project `iucc-alex-liberzon`, Region `europe-west3`, Buckets `openptv-uploads` / `openptv-results`  
+**Objective:** Re-run tracking on the full 5,005 frames directly in the cloud using the new bidirectional Two-Phase tracker, execute flowtracks repair and smoothing, generate Eulerian and phase-averaged Zarr datasets, and fetch trajectories and results locally for analysis against `res_orig` (3dptv.exe `xuap`).
+
+---
+
+## Background & Rationale
+
+1. **Why rerun in Cloud:**
+   The full correspondences and target intermediate files for the 5,005 frames already reside in `openptv-results` on GCS (`CompleteTest-track-20260915-065041/wp1/test/res.tar`). Running tracking on Cloud Run directly where the correspondence data is located avoids pulling tens of gigabytes across the internet and executes in parallel.
+
+2. **Why the new Two-Phase Bidirectional Tracker:**
+   - Previous runs suffered from overly conservative kinematic limits (`dacc: 0.4`, `dvxmax: 1.4`), which gate-dropped fast systolic bursts in the aorta.
+   - The new Two-Phase tracker with **Forward-First Bidirectional Tracking** (`bidirectional: true`):
+     - Forward pass with $v_{\max} = 2.2\text{ mm}$ establishes the clean diastolic core.
+     - Backward pass with wider reach ($v_{\max}^{\text{bwd}} = 3.5\text{ mm}$) recovers high-velocity systolic bursts without risking swaps in dense regions.
+     - Closed 75% of the link gap to 4-frame trackcorr on wp1 (reaching 97.48% exact recall in 3.0 s vs 15.0 s).
+     - Incorporates canonical `TargetArray.sort_y()` and auto-grid safety.
+
+---
+
+## Step-by-Step Execution Plan
+
+### Step 1: Merge PR #39 in `openptv2`
+- **Current Status:** PR #39 (`refactor/dedup-tracking-kernels` -> `main`) is open, mergeable, and all 8 GitHub Actions CI checks are **GREEN** (cross-platform wheels on Linux/macOS/Windows, ruff linting, multi-Python tests).
+- **Action:**
+  ```bash
+  gh pr merge 39 --merge
+  git switch main
+  git pull --ff-only
+  ```
+
+### Step 2: Build & Push Docker Image in `openptv-cloud`
+- Update `openptv-cloud` dependency reference to include the merged `openptv2` `main`.
+- Build the container image and push to Google Artifact Registry:
+  ```bash
+  cd C:\Users\alex\projects\openptv-cloud
+  uv run openptv-cloud build
+  ```
+- Image target:
+  `europe-west3-docker.pkg.dev/iucc-alex-liberzon/openptv/openptv-cloud-job:latest`
+
+### Step 3: Verify & Configure `parameters_twophase.yaml`
+- Parameter file already prepared at:
+  `C:\Users\alex\Downloads\HiDImaging\CompleteTest\wp1\test\parameters_twophase.yaml`
+- Config details:
+  ```yaml
+  plugins:
+    selected_sequence: default
+    selected_tracking: two_phase
+
+  track:
+    selected_tracking: two_phase
+    preset: two_phase
+    use_velocity: true
+    cost_mode: projected
+    leaf_weight: 1.0
+    v_max: 2.2            # Diastolic core search radius
+    bidirectional: true   # Forward-first locked bidirectional tracking
+    bwd_v_max: 3.5        # Systolic burst recovery radius
+    max_gap: 2            # Missing-frame bridge capacity
+    allow_shared: true    # Occlusion clustering
+    share_tol: 1.0
+    confirm_tol: 1.9      # Kinematic acceleration gate matching dacc=1.9
+    confirm_ends: false
+    max_group_size: 128
+    postprocess: false
+
+  sequence:
+    first: 1
+    last: 5005
+  ```
+
+### Step 4: Configure `experiment.yaml` for wp1 Single-Folder Run
+In `C:\Users\alex\Downloads\HiDImaging\CompleteTest\experiment.yaml`:
+- Scope run folders strictly to `wp1/test`:
+  ```yaml
+  runs:
+    folders: [wp1/test]
+
+  frames:
+    first: 1
+    last: 5005
+
+  trajectories:
+    repair: true          # flowtracks.repair (cut bad links, join pieces across gaps)
+    smoothing_window: 7   # Savitzky-Golay filter
+    smoothing_order: 2
+  ```
+
+### Step 5: Launch Tracking on GCP Cloud Run
+- Run the cloud pipeline:
+  ```bash
+  uv run openptv-cloud run C:\Users\alex\Downloads\HiDImaging\CompleteTest
+  ```
+- Monitor progress:
+  ```bash
+  uv run openptv-cloud status C:\Users\alex\Downloads\HiDImaging\CompleteTest --watch
+  ```
+
+### Step 6: Fetch Trajectories & Postprocess Locally
+- Download the resulting trajectory table and phase-binned Zarr datasets:
+  ```bash
+  uv run openptv-cloud fetch C:\Users\alex\Downloads\HiDImaging\CompleteTest
+  ```
+- Confirm output files in `C:\Users\alex\Downloads\HiDImaging\CompleteTest\wp1\test\res\`:
+  - `run.zarr/trajectories` (repaired & smoothed positions, velocities, accelerations)
+  - `run.zarr/eulerian` (Eulerian velocity, TKE, MKE fields)
+  - `phase_binned.zarr` (24 cardiac phase bins)
+
+### Step 7: Validation against `res_orig/xuap`
+- Run comparison script between newly generated trajectories in `res/run.zarr` vs ground-truth `res_orig`:
+  - Track length distribution (mean length $\ge 14$ frames).
+  - Trajectory coverage and sample counts.
+  - Phase-averaged velocity profiles in peak systole vs diastole.