v0.5.0 + v0.6.0: measured faces, one engine, search the whole archive by what it shows

.github/workflows/search-eval.yml

@@ -0,0 +1,62 @@
+name: Search quality
+
+# Typed search with the real vision-language weights: does a query find the photos
+# it should? Compares SigLIP with SigLIP 2, one phrasing with several, and English
+# with Persian queries, on public labelled images. See scripts/eval_search.py.
+on:
+  workflow_dispatch:
+  push:
+    branches: ["claude/**", "release/**"]
+    paths:
+      - "scripts/eval_search.py"
+      - "scripts/calibrate.py"
+      - "src/vie/models.py"
+      - "src/vie/default.yaml"
+      - ".github/workflows/search-eval.yml"
+
+jobs:
+  eval:
+    name: ${{ matrix.model }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    strategy:
+      fail-fast: false
+      matrix:
+        model:
+          - google/siglip-so400m-patch14-384
+          - google/siglip2-so400m-patch14-384
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install the CPU stack
+        run: |
+          python -m pip install --upgrade pip
+          pip install "torch==2.5.1" "torchvision==0.20.1" \
+            --index-url https://download.pytorch.org/whl/cpu
+          pip install -e ".[models,cpu]"
+          pip freeze | grep -iE "^(torch|transformers|tokenizers|sentencepiece)="
+      - name: Cache model weights
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: hf-${{ matrix.model }}-v1
+      - name: Fetch the labelled set
+        run: python scripts/fetch_calibration_data.py benchmarks/search-data --per-label 4
+      - name: Evaluate
+        # Whole frame only: the multi-crop layout is 6x the crops, too slow for a CPU runner.
+        run: python scripts/eval_search.py benchmarks/search-data --model ${{ matrix.model }} --layout 1
+      - name: Food and species thresholds on this model
+        # The food, species and food-gate thresholds were calibrated on SigLIP; this
+        # reports precision and recall at the configured values for each model.
+        run: |
+          python scripts/calibrate.py benchmarks/search-data --model ${{ matrix.model }} \
+            --out benchmarks/calibration-${{ strategy.job-index }}.json
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: search-eval-${{ strategy.job-index }}
+          path: |
+            benchmarks/search-eval-*.json
+            benchmarks/calibration-*.json

.github/workflows/showcase.yml

@@ -0,0 +1,60 @@
+name: Showcase
+
+# Builds the search showcase (food, species, free text, Persian, photos-and-videos
+# concept search) with the real weights on public data, and commits the cards to
+# assets/showcase/search on the same branch. See scripts/make_search_showcase.py.
+on:
+  workflow_dispatch:
+  push:
+    branches: ["claude/**"]
+    paths:
+      - "scripts/make_search_showcase.py"
+      - ".github/workflows/showcase.yml"
+
+permissions:
+  contents: write
+
+jobs:
+  showcase:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref_name }}
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install the CPU stack
+        run: |
+          sudo apt-get update && sudo apt-get install -y libgl1 libglib2.0-0 libfribidi0 libraqm0
+          python -m pip install --upgrade pip
+          pip install "torch==2.5.1" "torchvision==0.20.1" \
+            --index-url https://download.pytorch.org/whl/cpu
+          pip install -e ".[models,cpu,video]" arabic-reshaper python-bidi
+      - name: Cache model weights
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/huggingface
+            ~/.cache/torch
+            ~/.insightface
+          key: showcase-models-v1
+      - name: Fetch public data
+        run: |
+          python scripts/fetch_calibration_data.py data/public --per-label 2
+          mkdir -p data/videos
+          for name in classroom face-demographics-walking-and-pause; do
+            curl -sSL -o "data/videos/$name.mp4" \
+              "https://raw.githubusercontent.com/intel-iot-devkit/sample-videos/master/$name.mp4"
+          done
+      - name: Build the showcase
+        run: python scripts/make_search_showcase.py --data data/public --videos data/videos
+      - name: Commit the cards
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add assets/showcase/search
+          git commit -m "docs: search showcase built with the real models" || exit 0
+          git pull --rebase origin "${{ github.ref_name }}"
+          git push origin "HEAD:${{ github.ref_name }}"

CHANGELOG.md

@@ -2,6 +2,89 @@
 
 Each version is a git tag. Check one out to get that state of the project.
 
+## v0.6.0: search the whole archive by what it shows
+
+**One concept search over photos and videos.** `Engine.search_everything("a theatre
+performance")` ranks photos and video moments together on SigLIP's calibrated
+probability that the description fits, keeps the strongest moments in view, and
+draws video tiles with their timestamp. The notebook has a section for it.
+
+**Food and unknown species no longer touch pixels at query time.** Each detected
+animal and dish is encoded at index time, padded exactly as search pads it, in the
+same batched pass as the scene regions (schema v4). A food query, or a species
+outside ImageNet, used to decode every candidate image and run the image tower on
+its crops, so it took minutes on thousands of photos; it is now a matrix product
+over stored vectors through the paths' own ranking rule. An older index is still
+served by cropping.
+
+**Persian queries and SigLIP 2.** `google/siglip2-so400m-patch14-384` is supported
+(its tokenizer declares no length, so the 64 tokens both models trained on are now
+explicit) and understands Persian. A query is averaged over several phrasings
+(`text.query_templates`). `scripts/eval_search.py` and the *Search quality* workflow
+measure both choices with the real weights.
+
+**Measured defaults.** SigLIP 2 is the default vision-language model: equal on English
+queries (mAP 1.0 both), 0.945 against 0.738 on Persian, and with thresholds calibrated
+for it (`model_overrides`, food 0.92, species 0.84) level with SigLIP on food and species
+(precision 0.90–0.94 against 0.89 over two runs; species 0.95 both).
+Each model's thresholds follow it automatically, and a threshold set explicitly still
+wins. A Persian query is encoded as typed, since wrapping it in phrasings never helped.
+`text.min_probability: 0.001` lets an absent concept return nothing. Changing the model
+means `vie index --rebuild`; the index refuses to mix models.
+
+**Video faces at a fixed cost per person.** Each person keeps up to four exemplar
+views (the most typical, then the least alike) and their on-screen intervals,
+instead of a vector for every sighting; face search returns all of a person's
+intervals.
+
+## v0.5.0: one engine, more accurate faces, output worth showing
+
+**Faces, measured against student-face-gate.** `scripts/face_benchmark.py` puts ArcFace
+(this repo) and SFace (the gate) on identical aligned crops. ArcFace wins on every measure
+(d′ 8.08 against 5.86; 96.9% against 83.6% of hard head poses at each model's own
+threshold, with no wrong person accepted by either), so it stays. Two ideas came over from
+the gate, each measured here:
+
+* **Mirror averaging**, on by default (`face.flip_tta`): every face is embedded as seen and
+  mirrored. Hard poses 96.9% -> 97.6%, d′ 8.08 -> 8.43. One extra 112x112 pass per face.
+* **Several query photos**: `vie search face --image a.jpg --image b.jpg`. Each indexed face
+  keeps its best score over the query photos.
+
+Turning the recipe on re-indexes only the images that have faces, not the whole archive.
+
+**`vie.api.Engine`.** Every capability behind one object that loads each model once:
+incremental image and video indexing, the four searches, video search, result cards and
+annotated video. It refuses to prune when most of the archive vanishes at once, which is a
+disconnected drive far more often than a real deletion.
+
+**Output.** `vie.report.showcase` renders a search as a presentation-grade image, cropped
+around the matched subject. `vie.video.render` writes an annotated MP4: tracked faces,
+names from reference photos (with a margin over the runner-up, as in the gate) or grouped
+people, shot labels and a presence timeline, plus a card of who appeared for how long.
+
+**Found in a strict review of the whole codebase, and fixed:**
+
+* **Face indexing was doing 2.2x the work it needed.** InsightFace runs every model in
+  its pack on every face by default, including two landmark models and age/gender that
+  nothing here reads. Loading only detection and recognition: 1144 -> 527 ms on a
+  four-face 1080p frame, with identical faces and vectors.
+* **Unchanged files could be re-hashed forever.** A file whose content matched but
+  whose mtime had moved (a copy, a Drive sync, an extracted zip) was skipped without the
+  new mtime being stored, so every later run read it in full again. A gallery that
+  moved kept pointing at its old paths. Both are now recorded on the first such run.
+* **Pruning is guarded in the pipeline,** so `vie index` is protected too, not only the
+  notebook: if fewer than half of the indexed files are on disk, nothing is deleted.
+* **The annotated video ignored rotation flags** (portrait phone videos came out
+  sideways) and gave up on a damaged file; it now uses the analysis decoder for both.
+* **Persian names and file names** are shaped right to left on cards and videos.
+* Comments that still described CLIP, MobileNetV3 and a 768-d space now describe what
+  runs: SigLIP so400m, ConvNeXt-Tiny and the margin rule.
+
+**`notebooks/0_run_everything.ipynb`.** The whole system with one *Run all* in Colab. The
+index lives on Google Drive, so the first run reads the whole dataset and later runs read
+only new or changed files: 37 photos took 18.4 s, and with one photo added the next run
+took 1.1 s; two videos took 67.8 s the first time and 0.0 s the next. Every stage reports its own failure and the rest carry on.
+
 ## v0.4.0: video understanding
 
 New `vie.video` package and `vie video` commands: shot detection, keyframes, zero-shot

README.md

@@ -26,30 +26,56 @@ Faces, animals, food, plain-English descriptions, and moments inside a video, al
 
 | Ask with | Example | You get |
 |:--|:--|:--|
-| A face | `vie search face --image person.jpg` | Every photo of that person |
+| A face | `vie search face --image a.jpg --image b.jpg` | Every photo of that person; more photos find more |
 | An animal | `vie search animal --query "red panda"` | Photos with the animal boxed |
 | A dish | `vie search food --query "ghormeh sabzi"` | Photos of it, even with no detector class for it |
 | A description | `vie search text --query "a man with a blue backpack"` | The closest matches |
+| A concept, everywhere | `Engine.search_everything("a theatre performance")` | Photos and video moments together, ranked on one scale |
 | A video | `vie video analyze clip.mp4 --html report.html` | Shots, topics, people, speech and a summary |
+| An annotated video | `Engine.render_video(...)` | An MP4 with every face tracked, named or grouped, on a timeline |
 | A moment | `vie video search --query "a goal being scored"` | Timestamps across all your videos |
 
 ## See it work
 
+<div align="center">
+
+![Two enrolled students named live in a video they were never enrolled from](assets/showcase/desk_named.gif)
+
+<sub>Named from three reference photos each, taken from <i>other</i> recordings. Corridor and classroom: [MP4](assets/showcase/corridor_people.mp4) · [MP4](assets/showcase/classroom_people.mp4)</sub>
+
+</div>
+
+| Face search with two query photos | Every face in a classroom, grouped |
+|:--|:--|
+| ![One person found across cameras and head poses](assets/showcase/face-search.jpg) | ![Classroom, four people tracked and grouped](assets/showcase/classroom_people.gif) |
+| **Who was in the video, and for how long** | **Corridor camera** |
+| ![People card for the front desk video](assets/showcase/desk_named.png) | ![People card for the corridor video](assets/showcase/corridor_people.png) |
+
+<sub>Real output of the current code, regenerated by <a href="scripts/make_showcase.py"><code>scripts/make_showcase.py</code></a>. Footage: <a href="https://github.com/intel-iot-devkit/sample-videos">intel-iot-devkit/sample-videos</a>, CC BY 4.0.</sub>
+
+<details><summary>Screenshots from the original notebooks</summary>
+
 | Face search | Species search |
 |:--|:--|
 | ![Face search results](assets/demo/face-search.jpg) | ![Species search results](assets/demo/animal-search.jpg) |
 | **Food search** | **Free-text search** |
 | ![Food search results](assets/demo/food-search.jpg) | ![Free-text search results](assets/demo/text-search.jpg) |
 
-<sub>Screenshots from the original notebooks, so the scores shown predate the current scoring.</sub>
+</details>
+
+## Run everything in Colab
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ParsaVictor/visual-intelligence-engine/blob/main/notebooks/0_run_everything.ipynb)
+
+[`notebooks/0_run_everything.ipynb`](notebooks/0_run_everything.ipynb) runs the whole system with one *Run all*: index, all four searches, video analysis, annotated output videos and presentation cards. The index lives on Google Drive, so **the first run reads the whole dataset and every later run reads only new or changed files**. Measured in the notebook itself on a 4-core CPU: indexing 37 photos took 18.4 s, and after one more photo was added the next run took 1.1 s; two videos took 67.8 s to analyse the first time and 0.0 s the next.
 
 ## Architecture
 
 ![Files flow through analysis into a SQLite index, which search then filters and compares](assets/diagrams/architecture.png)
 
 Three decisions carry most of the design:
 
-- **Index once, query cheaply.** A species the classifier knows is answered from stored predictions, with no model loaded at all.
+- **Index once, query cheaply.** A species the classifier knows is answered from stored predictions, with no model loaded at all. Every detected animal and dish also has its vector stored, so a food query or an unknown species is a matrix product, not a pass over the pixels.
 - **Ask against rivals.** Instead of thresholding a raw similarity, each query competes with prompts like "a different kind of animal", which gives the model a way to say no.
 - **Measure, then decide.** Thresholds and the search backend come from measurements on real data, including where they overturned the original assumptions.
 
@@ -67,6 +93,8 @@ More in [the architecture](docs/architecture.md), [the design decisions](docs/de
 | **[Audit it honestly](docs/AUDIT.md)** | Found 25 defects behind 198 green tests, then fixed them all |
 | **Test with real weights** | End-to-end tests, plus a CI job running every real model |
 | **Understand video** | Shots, people, speech, timelines and search by moment |
+| **One engine** | Measured face upgrade, annotated output video, one-click Colab notebook |
+| **Search everything** | Concepts across photos and videos, Persian queries, food and species from stored vectors |
 
 Next up: on-screen text search, natural-language summaries and a REST API. See the [roadmap](ROADMAP.md) and [changelog](CHANGELOG.md).
 
@@ -95,7 +123,44 @@ Every search also takes `--json` for scripts and `--html report.html` for a shar
 
 ## Measured
 
-Precision on 72 public labelled images with the default model, at 100% recall. The set is small and easy, so read this as a comparison between versions, not as expected accuracy on your archive.
+**Faces.** ArcFace (this repo) against SFace (the recogniser in
+[student-face-gate](https://github.com/ParsaVictor/student-face-gate)), on identical aligned crops:
+776 faces, 17,956 same-person pairs (3,716 with a head turn of 30° or more) and 18,472
+different-person pairs. Each model at the threshold its own project ships with.
+
+| Recogniser | Equal error rate | Separation d′ | Same person accepted | Hard poses accepted | Wrong person accepted |
+|:--|--:|--:|--:|--:|--:|
+| **ArcFace + mirror (default)** | **0.0%** | **8.43** | **98.9%** | **97.6%** | **0** |
+| ArcFace | 0.0% | 8.08 | 98.6% | 96.9% | 0 |
+| SFace + mirror | 0.21% | 6.07 | 91.2% | 86.9% | 0 |
+| SFace | 0.20% | 5.86 | 89.8% | 83.6% | 0 |
+
+ArcFace separates people about 40% more widely, so it stays the recogniser. From the gate
+come two measured ideas: averaging each face with its mirror image (now on by default) and
+searching with several photos of a person. This is a pose stress test on public footage, not
+LFW; reproduce it with [`scripts/face_benchmark.py`](scripts/face_benchmark.py), results in
+[`results/face_benchmark.json`](results/face_benchmark.json).
+
+**Typed search, SigLIP vs SigLIP 2** (`scripts/eval_search.py` in the *Search quality*
+workflow, real weights, 14 labelled queries over 92 public photos):
+
+| | SigLIP so400m | **SigLIP 2 so400m (default)** |
+|:--|--:|--:|
+| English query, mAP | 1.000 | 1.000 |
+| Persian query, mAP | 0.738 | **0.945** |
+| Food precision / recall, at each model's thresholds | 0.89 / 1.00 | 0.90–0.94 / 0.98–1.00 |
+| Unknown-species precision / recall | 0.95 / 1.00 | 0.95 / 0.95–1.00 |
+| Best right image of a present concept, calibrated probability (worst case) | 0.023 | 0.009 |
+| Best image of an absent concept (median of 8) | 0.00002 | 0.00001 |
+
+The last two rows set `text.min_probability: 0.001`: a concept that is not in the archive
+returns nothing rather than its least-bad five. SigLIP 2 ranges are over two calibration
+runs whose samples differed slightly (44 and 48 dish images): on food and species the two
+models are level within that noise, and Persian is where SigLIP 2 is clearly ahead. The English set is too easy to separate the
+models or to show what averaging phrasings adds; both score 1.0 either way. Results in
+[`results/search_eval.json`](results/search_eval.json).
+
+**Species and food.** Precision on 72 public labelled images with the default model, at 100% recall. The set is small and easy, so read this as a comparison between versions, not as expected accuracy on your archive.
 
 | Search | v0.2 | Now |
 |:--|--:|--:|
@@ -104,7 +169,7 @@ Precision on 72 public labelled images with the default model, at 100% recall. T
 
 Vector search over 100,000 vectors on 4 CPU cores takes **10 ms**, and returns the exact answer.
 
-Not yet measured: accuracy on a real archive, face recognition quality, and anything on a GPU. Reproduce the numbers above with [`scripts/calibrate.py`](scripts/calibrate.py) and [`scripts/benchmark.py`](scripts/benchmark.py).
+Not yet measured: accuracy on a real archive, and anything on a GPU. Reproduce the numbers above with [`scripts/calibrate.py`](scripts/calibrate.py) and [`scripts/benchmark.py`](scripts/benchmark.py).
 
 ## Privacy