Dev

.gitattributes

@@ -14,5 +14,5 @@
 # committing it without the exclusion would force every `git clone` to
 # download 2.65 GB and spend the repo's LFS bandwidth quota. Fetch it via the
 # "BiRefNet setup" page in the app, or:
-#   git lfs pull --include="weights/birefnet_leaf.pth"
+#   git lfs pull -X "" -I "weights/birefnet_leaf.pth"
 weights/*.pth filter=lfs diff=lfs merge=lfs -text

.github/workflows/ci.yml

@@ -26,3 +26,18 @@ jobs:
           python -m pip install pytest
       - name: Run tests
         run: python -m pytest tests/ -q
+
+  pre-cleanup-integration:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install image test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install --no-deps -e .
+          python -m pip install pytest numpy opencv-python-headless
+      - name: Run pre-cleanup integration tests
+        run: python -m pytest tests/test_pre_cleanup_exports.py -q -rs

.lfsconfig

@@ -12,9 +12,9 @@
 # Opt in via the "BiRefNet setup" page in the MATS app, `mats fetch-weights
 # --only birefnet --source lfs`, or by hand:
 #
-#     git lfs pull --include="weights/birefnet_leaf.pth"
+#     git lfs pull -X "" -I "weights/birefnet_leaf.pth"
 #
-# Note that a bare `git lfs pull` will NOT fetch it -- only -I/--include
-# overrides fetchexclude. That's deliberate: no accidental 2.65 GB pulls.
-# [lfs]
-#	fetchexclude = weights/birefnet_leaf.pth
+# A bare pull and an include-only pull both retain fetchexclude. `-X ""`
+# clears that exclusion for this invocation; `-I` limits the pull to BiRefNet.
+[lfs]
+	fetchexclude = weights/birefnet_leaf.pth

AGENTS.md

@@ -75,8 +75,9 @@ other system libraries (QR codes are decoded with OpenCV).
 > fresh checkout fails to detect markers, and it looks like a model problem
 > rather than a setup problem. Verify with `ls -l weights/rf_detr_marker.pth`
 > (~134 MB, not ~134 bytes) or `mats doctor`; repair with
-> `git lfs install && git lfs pull --exclude="weights/birefnet_leaf.pth"` — the `--exclude` matters: a bare `git lfs pull` can also
-> fetch the 2.65 GB BiRefNet checkpoint.
+> `git lfs install && git lfs pull --exclude="weights/birefnet_leaf.pth"` — the `--exclude` keeps the repair to RF-DETR. With
+> `.lfsconfig` active, a bare `git lfs pull` also leaves the 2.65 GB BiRefNet checkpoint out; fetch it only with
+> `git lfs pull -X "" -I "weights/birefnet_leaf.pth"` (or `mats fetch-weights --only birefnet --source lfs`).
 
 ```bash
 git lfs install             # one-time, per machine, BEFORE cloning
@@ -128,6 +129,23 @@ Currently true, and worth knowing because users ask:
   `--input-dir`).
 - Defaults that surprise people: `--mask-method threshold`, `--threshold-level
   auto`, `--csv-schema full`, `--results-unit cm`, `--output-mode masks`.
+- `--threshold-level` takes a preset (`auto`, `low`, `medium`, `high`) or an
+  integer cutoff `1`–`255`; the GUI's **custom** level is the same thing.
+- `--mask-method both` (two checked methods in the GUI) measures each image with
+  Otsu and BiRefNet and writes one CSV and failure log per method
+  (`*_threshold.csv`, `*_birefnet.csv`); a single-method run keeps the
+  unsuffixed names and identical output.
+- `--measure-pre-cleanup` is not "every raw pixel": it clears a `--clean-margin`
+  band along the target-box edge (default 1% of the shorter side, where the
+  printed box outline lands), keeps the largest remaining object (the leaf),
+  and drops pieces that touch the band or lie beyond `--stray-gap` (default
+  `0.25` × the leaf's bounding-box diagonal) — see `mask_cleanup.clean_raw_mask`.
+  `--clean-size` (pixels, default `0` = off, pre-cleanup only) then removes
+  specks and fills holes below that inscribed radius — the app's **Clean size**,
+  via `mask_cleanup.raw_measurement_mask`; Adjust's Overwrite saves it for Otsu
+  specimens. `--clean-margin` is the edge margin, not the clean size.
+  Clean image and Remove flashfill use the same margin. `--export pre-cleanup`
+  still writes the untouched raw mask. The default cleaned path is unaffected.
 - `mats fetch-weights` with no flag fetches **RF-DETR only**; BiRefNet needs
   `--only birefnet` or `--all`. `--source {auto,hf,lfs}` picks the channel —
   `lfs` is the one to use on networks that block huggingface.co. After a
@@ -202,8 +220,8 @@ Rules that keep this repo working:
   module-level import of torch, rfdetr, or streamlit in an imported path breaks
   CI even when it works locally.
 - **`paths.py` stays import-light** — standard library only. Models load lazily
-  inside `core.py`; `samples.py`, `dimensions.py`, and `scaling.py` follow the
-  same contract.
+  inside `core.py`; `samples.py`, `dimensions.py`, `scaling.py`, and
+  `mask_settings.py` follow the same contract.
 - **The CLI and the GUI share one execution path** (`run_leaf_morpho_batch`).
   Never fork pipeline logic between them — divergence would mean the two
   interfaces report different measurements.

CHANGELOG.md

@@ -6,13 +6,85 @@ All notable changes to MATs are documented here. This project adheres to
 ## [Unreleased]
 
 ### Changed
+- Adjust now lists specimens in a searchable measurement table with sortable
+  columns. Clicking a row (its **View** button) previews that specimen; for Otsu
+  runs, ticking **Marked for Adjustment** marks it, and marks persist across
+  searches. Its duplicate
+  measurement cards and saved-image viewer have been removed, leaving the
+  controls and live preview.
+- **Remove flashfill from this preview** now sits under **Clean size** inside
+  **Explore and adjust output**.
+- Adjust saves marked specimens with its own **Overwrite all marked specimens
+  (N)** button, below **Overwrite this specimen** (the specimen in View),
+  replacing the "Apply these adjustments to all marked specimens" checkbox.
+- Analyze now makes the Classic thresholding/BiRefNet comparison switch explicit
+  and keeps each method's table selection separate, so the specimen viewer
+  always follows the active method's table.
+- Setup now places threshold level directly under Classic thresholding and
+  labels the default method without the Otsu parenthetical.
+- The Analyze measurement table and selected-specimen viewer occupy separate
+  full-width blocks. Interactive controls now live in the Adjust tab.
+- Analyze shows only the specimen selected in the measurement table. Adjust
+  carries that selection into its preview and overwrite controls.
+- The Analyze sample viewer now shows the raw or cleaned mask used for the
+  completed run, even when that image was not selected for export.
+- The app's segmentation choice is now two checkboxes. **Threshold level**
+  appears only while Otsu is checked, and a single **Pre-cleanup masks** export
+  covers every checked method.
+- The workbench now uses Setup, Analyze, Adjust, and Export, with Diagnostics
+  in the sidebar. Image exports are selected before a run; downloads include
+  only files recorded for that run.
+- Target boxes and cleaned masks remain default exports but can be disabled.
 - Documentation now matches the shipped weights-delivery behavior: Git LFS is
   listed as an install prerequisite (a clone without it yields a 134-byte
   pointer stub, not the model), RF-DETR is documented as arriving *with* the
   clone rather than needing `mats fetch-weights`, and `mats fetch-weights` is
   described as the repair path it is.
 
 ### Added
+- An interactive threshold preview for the selected Otsu sample in Adjust.
+  Dragging updates its raw mask, and a color panel of the masked leaf beside
+  it, immediately; releasing applies the existing
+  cleanup when cleaned measurements are selected. A chosen cutoff can be used
+  for the next run or overwrite the selected specimen's mask, measurement row,
+  and dependent images with one button. Marked specimens can receive the same
+  settings in one bulk save, each using its own calibration.
+- A **Clean size** slider in Adjust's Explore and adjust box, for Otsu and
+  BiRefNet specimens. Above 0 it live-previews Clean image, which drops small
+  white specks and fills small enclosed holes without flash-filling the leaf.
+  For Otsu specimens, **Overwrite this specimen** and **Overwrite all marked
+  specimens** save that mask, re-measure it, and record the clean size;
+  BiRefNet specimens preview it only.
+- A run-wide clean size for pre-cleanup measurements: `--clean-size PX` in the
+  CLI, **Clean size (px)** in Setup (default 0, off). It removes specks and
+  fills enclosed holes whose inscribed radius is below that many pixels before
+  measuring, exactly as the Adjust slider previews, and is recorded in each
+  results CSV's `.meta.json`. It requires `--measure-pre-cleanup`.
+- Optional measurements from pre-cleanup binary masks in the app and CLI
+  (`--measure-pre-cleanup`), with per-CSV metadata recording the source.
+  A band along the target-box edge (`--clean-margin`, **Edge margin** in the
+  app; default 1% of the box's shorter side) is cleared first, so the
+  template's printed box outline can never outweigh and replace a small leaf.
+  The largest remaining object then anchors the measurement: pieces touching
+  that band, or farther from the leaf than `--stray-gap` (**Stray-piece
+  distance**; default 0.25 × the leaf's bounding-box diagonal), are dropped so
+  printed box lines and distant debris no longer stretch width and length.
+  Clean image (a clean size above 0), Remove flashfill, and the pre-cleanup
+  threshold explorer apply
+  the same cleanup; the default cleaned measurements are unchanged. In Setup
+  these settings are grayed out unless pre-cleanup measurement is checked; each
+  specimen's explorer can adjust them for its own preview and overwrite.
+- Measure with Otsu and BiRefNet in one run: `--mask-method both`, or check both
+  segmentation methods in the app. Markers are detected once per image; each
+  method writes its own results CSV and failure log (`_threshold`/`_birefnet`
+  suffixes) in the single-method schema, and the Analyze view can switch
+  between them. Single-method runs write the same files as before.
+- Separate pre-cleanup binary mask exports for threshold/Otsu and BiRefNet,
+  including both methods in one run while measurements use the selected method.
+- Optional overlay, cutout, and measurement-axis exports in the CLI and app.
+- A custom grayscale threshold: `--threshold-level` accepts an integer cutoff
+  `1`–`255`, and the app's **custom** threshold level shows a slider with the
+  low/medium/high presets marked.
 - A **Robust QR setup** sidebar page that explains the optional pyzbar and
   QReader fallbacks, reports their usable status, and keeps Conda optional.
 - A **Help** page in the app (sidebar) with a quick start, a photography guide

README.md

@@ -1,4 +1,4 @@
-# MATS — Morphometric Analysis Toolbox
+# MATS — Morphometric Analysis Toolbox for Segmentation
 
 Measure leaf **area, length, and width** in real-world units from a photo of
 leaves laid on a printed calibration template.
@@ -68,8 +68,7 @@ ls -l weights/rf_detr_marker.pth
 
 **Already cloned without Git LFS?** No need to start over — install Git LFS as
 above, then repair the checkout in place. The `--exclude` keeps this to the
-~134 MB RF-DETR file; a bare `git lfs pull` can also fetch the 2.65 GB BiRefNet
-checkpoint:
+~134 MB RF-DETR file, avoiding the optional 2.65 GB BiRefNet checkpoint:
 
 ```bash
 git lfs install && git lfs pull --exclude="weights/birefnet_leaf.pth"
@@ -148,12 +147,14 @@ checkpoint or install the pyzbar/QReader robust-QR fallbacks.
 
 The required ~134 MB RF-DETR marker checkpoint is different: it is mandatory for
 every run, so it ships **in the clone** via Git LFS and needs no separate
-download step. If it is ever missing — a clone made without Git LFS, or an
-install outside a Git checkout — MATS fetches it once on first use and prints
-`Fetching weights/rf_detr_marker.pth via Git LFS ...` while it does. Set
-`MATS_NO_AUTO_FETCH=1` to turn that off and require pre-staged weights instead
-(recommended on HPC login nodes). The app never does this silently: a missing
-RF-DETR checkpoint is a blocking Preflight error.
+download step. If it is missing from a Git checkout — for example, after cloning
+without Git LFS — MATS can fetch it once on first use and prints
+`Fetching weights/rf_detr_marker.pth via Git LFS ...` while it does. An install
+outside a Git checkout must use a pre-staged checkpoint (or a separately
+configured Hugging Face source). Set `MATS_NO_AUTO_FETCH=1` to turn automatic
+fetching off and require pre-staged weights instead (recommended on HPC login
+nodes). The app never does this silently: a missing RF-DETR checkpoint is a
+blocking Preflight error.
 
 This keeps the initial network and disk footprint predictable, avoids native
 `zbar` failures on managed machines, and works better on HPC systems and
@@ -235,10 +236,14 @@ Common options (full reference in [docs/cli.md](docs/cli.md)):
 | `-r, --results_path` | Measurement CSV path | `./leaf_morpho_results.csv` |
 | `--sheet-dimensions` | Finished Template Creator sheet size, `<w>x<h><unit>` | read from QR |
 | `-t, --template_dimensions` | Legacy/custom marker-centre calibration area | unused |
-| `--mask-method` | `birefnet` (accurate, GPU) or `threshold` (fast) | `threshold` |
-| `--threshold-level` | `auto` (Otsu) / `low` / `medium` / `high` | `auto` |
+| `--mask-method` | `birefnet` (accurate, GPU), `threshold` (fast), or `both` | `threshold` |
+| `--threshold-level` | `auto` (Otsu) / `low` / `medium` / `high`, or a custom cutoff `1`–`255` | `auto` |
 | `--csv-schema` | `full` (area/width/length + per-axis pixels-per-selected-unit) or `compact` | `full` |
 | `--results-unit` | Measurement-output unit: `mm`, `cm`, or `in` | `cm` |
+| `--measure-pre-cleanup` | Measure from raw binary masks before cleanup, minus the edge margin and stray pieces | off |
+| `--clean-margin` | With `--measure-pre-cleanup`: band cleared along the target-box edge, as a percent of its shorter side | `1` |
+| `--stray-gap` | With `--measure-pre-cleanup`: how far a piece may lie from the leaf, as a fraction of its bounding-box diagonal | `0.25` |
+| `--clean-size` | With `--measure-pre-cleanup`: remove specks and fill enclosed holes smaller than this radius in pixels (the app's **Clean size**) | `0` (off) |
 | `-w, --workers` | Parallel workers (threshold path only) | auto |
 | `--save-axes` | Also save length/width overlay images for QC | off |
 
@@ -247,7 +252,9 @@ no GPU, no extra download, and good for clean, high-contrast backgrounds where
 a leaf sits on plain white. `birefnet` is more accurate on cluttered or
 low-contrast backgrounds and uses a GPU when available (CPU works but is
 slow), at the cost of the ~2.65 GB checkpoint — fetch it once with
-`mats fetch-weights --only birefnet`.
+`mats fetch-weights --only birefnet`. To compare them, `--mask-method both`
+(or checking both methods in the app) measures every image with each method
+and writes one results CSV per method.
 
 ---
 
@@ -258,6 +265,29 @@ Per image, in the output folder:
 - `{sample_id}_target_box.jpg` — the perspective-corrected observation box
 - `{sample_id}_mask.png` — the leaf segmentation mask
 
+Both image exports are on by default and can be disabled with `--no-target-boxes`
+or `--no-masks`. Pre-cropped target-box inputs are not copied. Optional
+`--export pre-cleanup` saves each measurement method's binary mask before cleanup;
+`--pre-cleanup-methods both` runs Otsu/threshold and BiRefNet and saves their
+pre-cleanup masks separately. Only a `--mask-method` method determines
+measurements. BiRefNet exports require its checkpoint to be installed locally.
+With `--mask-method both`, each method's masks and QC images end in
+`_threshold` or `_birefnet` (for example `{sample_id}_mask_birefnet.png`).
+`--export overlay`, `--export cutout`, and `--export axes` add QC images; repeat
+the flag to request multiple kinds. The app offers the same choices in Setup.
+Use `--measure-pre-cleanup` (or **Measure from pre-cleanup masks** in the app)
+to calculate measurements from the raw binary segmentation. A thin band along
+the target-box edge, where the template's printed box outline lands, is cleared
+first (`--clean-margin`). The largest remaining object is taken as the leaf;
+other pieces that touch that band, or lie farther from the leaf than
+`--stray-gap` times its bounding-box diagonal, are dropped. Area counts the remaining foreground pixels, and width and length span
+their extent, so specks near the leaf can still affect the result; `--clean-size`
+(for example `--clean-size 3`) removes specks and fills enclosed holes whose
+inscribed radius is below that many pixels, without flash-filling the leaf. This choice
+is independent of `--export pre-cleanup`, which saves the mask with every piece.
+Each results CSV has a `.meta.json` companion that records the measurement source
+and, for pre-cleanup runs, these settings.
+
 Plus a measurements CSV. Choose `mm`, `cm` (the default), or `in` with
 `--results-unit` in the CLI or the **Result units** control in the app. The
 selection changes results, dashboard labels, and unit-bearing CSV column names;
@@ -279,7 +309,11 @@ it does not change calibration math. Two schemas:
   millimeters or inches selected, `cm` is replaced consistently in the
   measurement column names.
 
-A `leaf_morpho_failures.csv` records per-image warnings and failures.
+A `leaf_morpho_failures.csv` records per-image warnings and failures. With
+`--mask-method both` (or both methods checked in the app), each method writes its
+own results CSV and failure log with a method suffix, for example
+`leaf_morpho_results_birefnet.csv` and `leaf_morpho_failures_birefnet.csv`, in
+the same schema as a single-method run.
 
 > **Migration note:** earlier versions reported three isotropic scale
 > conventions (`*_meanscale`, `*_widthscale`, `*_heightscale`). Old CSVs remain

deploy/ondemand/mats/README.md

@@ -13,12 +13,16 @@ compute node and exposes it through the Open OnDemand reverse proxy.
    ```
    Then set `CONDA_ENV` in `template/script.sh.erb` to that env name (`mats`).
    (A plain virtualenv works too — the default install needs no system libs.)
-2. **The model checkpoints.** Fetch them once, ideally to a shared location:
+2. **The model checkpoints.** Pre-stage them once in a shared location. From a
+   Git checkout with Git LFS installed, materialize the files, then copy them to
+   the shared directory:
    ```bash
    export MATS_WEIGHTS_DIR=/shared/models/mats
-   mats fetch-weights --all   # RF-DETR + BiRefNet (the GUI defaults to Otsu, but a
-                              # GPU-backed OOD app is the typical BiRefNet use case)
-   mats doctor            # confirm they resolve
+   mkdir -p "$MATS_WEIGHTS_DIR"
+   mats fetch-weights --all   # writes RF-DETR + BiRefNet to the checkout's weights/
+   cp weights/rf_detr_marker.pth "$MATS_WEIGHTS_DIR/"
+   cp weights/birefnet_leaf.pth "$MATS_WEIGHTS_DIR/"
+   mats doctor                 # confirm they resolve
    ```
    Point the same `MATS_WEIGHTS_DIR` at that path in `template/script.sh.erb`.
 3. *(Optional)* Enhanced QR reading (`pip install -e ".[qr]"`) adds the `pyzbar`