Skip to content

v0.5.0 + v0.6.0: measured faces, one engine, search the whole archive by what it shows - #40

Draft
ParsaVictor wants to merge 21 commits into
mainfrom
claude/image-processing-optimization-7abxze
Draft

v0.5.0 + v0.6.0: measured faces, one engine, search the whole archive by what it shows#40
ParsaVictor wants to merge 21 commits into
mainfrom
claude/image-processing-optimization-7abxze

Conversation

@ParsaVictor

@ParsaVictor ParsaVictor commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Two versions in one branch:

  • v0.5.0 ends at c3ef80c. Tagging was not possible from the session; the notebook accepts that commit as REPO_REF.
  • v0.6.0 is everything after it.

v0.6.0: search the whole archive by what it shows

Measured with the real weights in the Search quality workflow: 14 labelled queries over about 90 public photos, see results/search_eval.json. SigLIP 2 ranges come from two calibration runs whose samples differed slightly.

SigLIP so400m SigLIP 2 so400m (new default)
English query, mAP 1.000 1.000
Persian query, mAP 0.738 0.945
Food precision / recall, each model's thresholds 0.89 / 1.00 0.90–0.94 / 0.98–1.00
Unknown species precision / recall 0.95 / 1.00 0.95 / 0.95–1.00

The two models are level on food and species; Persian decides.

  • SigLIP 2 by default. Its thresholds live under model_overrides and follow the selected model (SigLIP's thresholds lost a tenth of SigLIP 2's right answers). A threshold set explicitly still wins. The 64 text tokens are now explicit, because SigLIP 2's tokenizer declares no length.
  • "Nothing found". text.min_probability: 0.001, from the measured gap: present concepts scored at least 0.009, and 7 of 8 absent concepts scored under 0.001.
  • Persian queries are encoded as typed. Wrapping them in phrasings never helped and cost SigLIP 0.32 mAP. English queries are averaged over text.query_templates; the set is too easy to show a gain from that.
  • Food and unknown species are answered from vectors stored at index time.
    • Schema v4: object_embeddings, encoded in the same batched pass, padded exactly as search pads.
    • A query is a matrix product through the paths' own ranking rule, with no image decoded.
    • Tests check score parity with the crop path. An older index is still served by cropping.
  • One concept search over photos and video moments. Engine.search_everything ranks both on SigLIP's calibrated probability and tags video tiles with their timestamp. The notebook has a section for it.
  • Video faces at a fixed cost per person. Up to 4 exemplar views (farthest-point sampling) plus video_person_spans. Face search returns all of a person's intervals.

v0.5.0: measured faces, one engine, presentable output

  • ArcFace vs SFace (student-face-gate), on identical crops. Hard poses accepted: 97.6% vs 83.6%. d′: 8.43 vs 5.86. Mirror averaging on by default. Multi-photo query.
  • Speed. Only detection and recognition are loaded: 2.2× faster per face.
  • vie.api.Engine. Incremental indexing:
    • records new mtimes and paths of unchanged files;
    • refuses to prune most of the index at once.
  • Output. Presentation cards, annotated MP4 (rotation-aware, Persian names shaped), people cards, and a one-click Colab notebook with the index on Drive.

Testing

  • pytest: the full suite passes on Linux and Windows, Python 3.10 and 3.12. ruff is clean.
  • Search quality workflow: real SigLIP and SigLIP 2 weights; evaluation plus two calibration runs.
  • Face benchmark and showcase: real InsightFace weights.
  • Notebook: executed end to end with nbclient (real InsightFace, fake SigLIP), every stage green.
  • Not yet run: the notebook with real SigLIP 2 on a Colab GPU.

🤖 Generated with Claude Code

https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq

claude added 14 commits November 6, 2026 13:05
@claude
feat: measured face upgrade, mirror averaging and several query photos
0dfee3c 
scripts/face_benchmark.py compares ArcFace (this repo) with SFace (student-face-gate)
on identical aligned crops from public CC BY 4.0 footage. ArcFace wins on every
measure (d' 8.08 vs 5.86, hard poses 96.9% vs 83.6%), so it stays, and two ideas
from the gate come over:

- face.flip_tta (default on): embed each face and its mirror, sum, normalise.
  Hard poses 96.9% -> 97.6%, d' 8.08 -> 8.43, one extra 112x112 pass per face.
- vie search face --image a.jpg --image b.jpg: each face keeps its best score.

The index records the face recipe; changing it re-indexes only images with faces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
feat: Engine API, presentation cards and annotated output video
7323437 
- vie.api.Engine: every capability behind one object that loads each model once,
  with incremental indexing that refuses to prune when the archive seems to vanish.
- vie.report.showcase: a search rendered as one image, cropped around the subject.
- vie.video.render: an annotated MP4 with tracked faces, names from reference photos
  (with a margin over the runner-up) or grouped people, shot labels and a timeline.
- index_videos takes on_analysed, so reports are written in the same pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: one-click Colab notebook for the whole system, v0.5.0 notes
6ba756e 
notebooks/0_run_everything.ipynb keeps the index on Google Drive, so the first run
reads the whole dataset and later runs read only new or changed files. Executed
end to end here with real face models and faked SigLIP weights: one photo added to
37 took 2.7 s instead of 49.7 s, and unchanged videos 0.1 s instead of 216 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: real showcase output, regenerated by scripts/make_showcase.py
53c37ba 
Face search card, three annotated videos with GIF previews and people cards, all
from public CC BY 4.0 footage (intel-iot-devkit/sample-videos). People-card
thumbnails now take the sharpest large view, since the largest alone was motion blur.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
fix: halve face cost, keep incremental runs incremental, guard pruning
487c57c 
Found in a strict review of the whole codebase:

- FaceAnalysis ran all five models of buffalo_l on every face, including two
  landmark models and age/gender that nothing reads. Detection and recognition
  only: 1144 -> 527 ms on a four-face 1080p frame, identical faces and vectors.
- A file with unchanged content but a moved mtime (copy, Drive sync, unzip) was
  skipped without storing the new mtime, so it was re-hashed on every later run;
  a moved gallery kept its old paths. Both are now recorded on the first such run.
- Pruning is refused inside build_index when fewer than half the indexed files
  are on disk, so vie index is protected too, not only the notebook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
fix: upright phone videos, Persian text, honest score captions
025cdaa 
- render: decode through vie.video.decode, so rotation flags and damaged tails
  are handled as in analysis; faststart is a muxer option, not a codec one; the
  frame rate keeps 29.97; timeline lanes keep their order; people are listed in
  order of first appearance; a backed watermark; balanced people-card rows.
- showcase: Persian and Arabic shaped right to left (raqm, else arabic-reshaper);
  fonts cached; a per-path score caption and footnote (a face score is a cosine,
  not a probability); file names shortened in the middle, where they differ.
- api: the automatic face query is a crop of the chosen face; video search takes
  exactly one of query or image and keeps speech hits first, as the CLI does;
  shot labels use video.label_threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: comments describe what runs, not what used to
a6fdf3a 
Module docstrings still described CLIP, MobileNetV3 and a 768-d space; they now
describe SigLIP so400m, ConvNeXt-Tiny and the margin rule. The face threshold
comment cites the benchmark it rests on. Benchmark FAR/FRR are computed with
searchsorted, O(n log n) instead of a loop per threshold, with identical results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: notebook hardening and regenerated showcase
c3ef80c 
Notebook: the install is verified in a fresh process, with a repair for a broken
OpenCV; a zipped dataset keeps its files' times, so unchanged photos are skipped
unread; the index is copied to Drive via a local snapshot and an atomic rename;
the automatic face query is a crop. Re-measured on 4 CPU cores: 37 photos 18.4 s,
then 1.1 s after adding one; two videos 67.8 s, then 0.0 s. The face card shows
twelve results, including tilted heads and the second camera.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
feat: SigLIP 2 support, query ensembling, and a real-model search eva…
3d3d67c 
…luation

- SigLIP 2's Gemma tokenizer declares no maximum length, so padding to max_length
  was unbounded; the 64 tokens both models trained on are now explicit.
- encode_query averages a query over text.query_templates (prompt ensembling);
  text and video search use it. A Persian query on an English-only model warns.
- scripts/eval_search.py ranks labelled photos for every label as a query, with
  the product's multi-crop scoring, and reports mAP, top-1 and R-precision for
  one phrasing vs an ensemble and English vs Persian, plus the calibrated
  probabilities a 'nothing found' floor would have to separate.
- .github/workflows/search-eval.yml runs it for SigLIP and SigLIP 2 on CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
perf: answer food and unknown-species queries from vectors stored at …
c41b2ce 
…index time

Every food and out-of-vocabulary species query used to decode each candidate
image and run the image tower on its crops, so its cost grew with the archive:
minutes per query on thousands of photos. Each detected animal and dish is now
encoded at index time, padded exactly as the search paths pad it, in the same
batched pass as the scene regions (schema v4: object_embeddings and
gallery_meta.objects_embedded).

At query time those vectors go through the paths' own ranking functions via a
scorer over stored vectors, so the decision rule is unchanged; confidence and
class floors are applied to the stored boxes. A gate-admitted image with no box
above the floor is scored on its stored whole-frame vector. Images indexed before
this are still served by cropping, so an old index keeps working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
feat: one concept search over photos and video moments
a12fa6d 
Engine.search_everything ranks photos (best region) and video moments (keyframes)
together on SigLIP's calibrated probability that the description fits, keeping the
strongest few moments even when photos outscore them, since a keyframe carries a
whole-frame vector only. Each moment's first frame is grabbed through the analysis
decoder, upright for phone video, and the result card tags video tiles with their
timestamp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
perf: keep a few exemplar face views per person per video
c475792 
Every sighting of every face was stored with its 2 KB vector, up to 240 keyframes
per video, and every face search loaded all of them. Each person now keeps up to
video.face_exemplars views (default 4): the most typical, then the least alike
(farthest-point sampling, so a turned head or a change of light is kept), and their
on-screen intervals are stored as plain times in video_person_spans. A face search
matches a person on any exemplar and returns all their intervals. An index from
before keeps working through the old per-sighting path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: v0.6.0 notes, design decisions, notebook concept search
27ebf32 
The notebook gains a concept search over photos and video moments, a multilingual
model option for Persian queries, and REPO_REF accepting a commit, so v0.5.0
(c3ef80c) stays reachable without a tag. The GPU default model is provisional until
the Search quality workflow reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
ci: make the search evaluation fit a CPU runner
3b38d0a 
At about 6 s per 384 px crop for an 877M-parameter model on a CPU runner, six crops
per image for 134 images did not finish inside the job limit. The workflow now
embeds whole frames only (--layout 1) on four images per dish, prints progress, and
has 120 minutes; the comparison it exists for (SigLIP vs SigLIP 2, one phrasing vs
several, English vs Persian) is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@ParsaVictor ParsaVictor changed the title v0.5.0: one engine, measured face upgrade, annotated video, one-click Colab notebook v0.5.0 + v0.6.0: measured faces, one engine, search the whole archive by what it shows Nov 6, 2026
claude and others added 7 commits November 6, 2026 15:18
@claude
feat: measured search defaults: Persian as typed, a 'nothing found' f…
29ce762 
…loor

From the Search quality run on SigLIP and SigLIP 2 (14 labelled queries, 92 photos):

- English queries: mAP 1.0 on both models, one phrasing or several.
- Persian: SigLIP 0.738, SigLIP 2 0.945 mAP. Wrapping a Persian query in phrasings
  never helped and cost SigLIP 0.32, so a right-to-left query is encoded as typed.
- Calibrated probability of the best right image for a present concept: at least
  0.009; best image for 7 of 8 absent concepts: under 0.001 (median 0.00001).
  text.min_probability: 0.001 now hides results below that, in text search and in
  the photos-and-videos concept search, so an absent concept returns nothing.

The workflow also runs calibrate.py per model, since the food and species
thresholds were calibrated on SigLIP before SigLIP 2 becomes a default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: record the Search quality results for SigLIP and SigLIP 2
81e537e 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
feat: SigLIP 2 by default, with thresholds calibrated for it
a61f831 
Measured in the Search quality workflow with the real weights:

                              SigLIP so400m   SigLIP 2 so400m
  English query mAP                1.000           1.000
  Persian query mAP                0.738           0.945
  food precision / recall      0.889 / 1.00    0.935 / 0.896  at SigLIP's 0.98
                                               0.94  / 0.98   at 0.92
  species precision / recall   0.952 / 1.00    1.00  / 0.90   at SigLIP's 0.91
                                               0.952 / 1.00   at 0.84

SigLIP 2's logit margins run narrower, so SigLIP's thresholds kept precision and
lost a tenth of the right answers. model_overrides holds each model's thresholds
and Config applies the selected model's automatically; a threshold the caller sets
explicitly still wins. The notebook picks SigLIP 2 on a GPU, so Persian queries work
by default, with 'english' for the previous model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
docs: SigLIP 2 is level on food and species, not ahead
8760681 
A confirmation run at the new thresholds, on a slightly different sample (44 dish
images instead of 48), gave food precision 0.898 / recall 1.0 and species
0.95 / 0.95, against 0.94 / 0.98 and 0.95 / 1.0 in the first. Over the two runs SigLIP
2 is level with SigLIP on food (0.89) and species (0.95); the choice of SigLIP 2
rests on Persian (0.945 vs 0.738 mAP). README, changelog, config comments, notebook
and results now say that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@claude
ci: build the search showcase with the real models
e35693f 
scripts/make_search_showcase.py indexes a small public archive (Food-101 dishes,
cats and dogs, unrelated objects, classroom stills and one classroom video) with
SigLIP 2, ConvNeXt, MegaDetector, RT-DETR and InsightFace through vie.api.Engine,
and saves a card for food, species (in and out of the classifier's vocabulary),
free text, and photos-and-videos concept searches in English and Persian, plus one
concept that is absent, to show the 'nothing found' floor. The Showcase workflow
runs it on a CPU runner and commits the cards to assets/showcase/search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
@github-actions
docs: search showcase built with the real models
6c270d7
@claude
fix: a card with one result keeps the normal tile size
707c6d6 
Fixing the column count to the number of results made a single match one tile the
full width of the card, as the real-model 'cat' card showed. The count now stays at
the configured four.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018b4X4tyoVUYiQo4mDBZBoq
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ParsaVictor @claude