Skip to content

Add JSON Lines results store and image-level evaluation - #6

Merged
Ray3R3 merged 1 commit into
mainfrom
feat/results-store-metrics
Nov 19, 2026
Merged

Ray3R3 merged 1 commit into
mainfrom
feat/results-store-metrics

Conversation

@Ray3R3

@Ray3R3 Ray3R3 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • vista.store: JSON Lines results store, one InspectionResult per line, one file per run (as discussed in Inspection Results: JSON files or Postgres for inspection results? #5). Lines are validated on load and errors report file:line.
  • vista.evaluation:
    • roc_auc via the Mann-Whitney U statistic, ties counted as half, no extra dependencies
    • threshold_for_fpr picks the threshold on held-out good images to hit a target false-positive rate, so the test set is never used to choose it (ADR 0001)
    • evaluate_images reports AUROC plus precision, recall, F1 and FPR at that threshold, with ground truth read from MVTec-style image ids

Pixel-level metrics (pixel AUROC, PRO) and latency come next.

Testing

pytest (28 passed). AUROC is checked against the pairwise definition, and threshold selection against ties and a zero-FPR target.

Results are stored one InspectionResult per line, one file per run, and
validated on load. Evaluation reports AUROC (Mann-Whitney U, ties as half)
and precision/recall/F1 at a threshold chosen on held-out good images for a
target false-positive rate, so the test set never sets the threshold.

Co-authored-by: Sam Oteng <98087809+samoteng@users.noreply.github.com>
@Ray3R3
Ray3R3 merged commit 637dedf into main Nov 19, 2026
2 checks passed
@Ray3R3
Ray3R3 deleted the feat/results-store-metrics branch November 19, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant