Skip to content

Fingerprint: the plan and its spec (FP0-FP13) - #121

Closed
shinyobjectz wants to merge 1 commit into
mainfrom
fingerprint
Closed

shinyobjectz wants to merge 1 commit into
mainfrom
fingerprint

Conversation

@shinyobjectz

Copy link
Copy Markdown
Owner

Fingerprint is a person's or a team's sense of good work, learned on their own device. It is a taxonomy of axes (soft picture, dead motion, noise under the voice, a cut that drags) with one small head per axis over frozen encoders that run on consumer hardware, a person's weighting over the axes, reference profiles, and the evidence ledger the heads learn from. Teams share evidence, never media, and each machine retrains in milliseconds.

This PR is docs only:

  • docs/fingerprint.md: the one plan. It covers the stack (what runs at score time and what's only for training), how it trains, the CLI, teams, keeping agents honest, earshot as the reference design, the measurements, the build order with confidence, and the decisions taken by default.
  • docs/fingerprint.feature: the rask tasks. FP0a-g are done (the 11-12 measurements). FP1-FP13 are the build, with plan tables, and tools/rask.sh parses them.

What the measurements say

  • Notes alone don't predict where the next note goes. Held out by film, every encoder scores at or below chance (AUC 0.40-0.53). The notes aren't good/bad labels: 98 of 205 name a defect, 74 are directions and 3 are approvals.
  • A named axis is learned from 4-32 examples. With 32 examples, blur, freeze, noise and clipping score 0.98-1.00, and dark and dropout 0.95-0.97. Glow needs its own look columns.
  • CLAP scores sound axes from their words alone, with no labels: noise 0.97 and clipping 0.98. Its learned heads match or beat YAMNet's on every sound axis from 16 labels up. It costs 33 ms a clip on the CPU, and the 8-bit model is 161 MB.
  • Taxonomy first, then a person's weighting. This scored 0.84 / 0.73 / 0.95, against 0.73 / 0.64 for one head per person. For teams, pooling evidence (0.76) beats averaging heads (0.74).
  • A click on the problem teaches "where" from 8 frames, with the head's top patch in the right place 0.85-0.96 of the time. Today's pins don't: they anchor a note near the frame's middle and don't mark the thing.
  • PE-AV is dropped. It can't run on consumer hardware.

Stack

All encoders run in Rust on ort (raea-sys-ml): SigLIP 2, CLAP and YAMNet. SigLIP 2 moves to MLX on the Mac once mlx.feature MX5 lands. EmbeddingGemma 2 serves scripts, boards and stills through the Library's rows. EdgeTAM and Depth Anything 3 serve footage-only checks.

earshot (github.com/Leonxlnx/earshot, MIT) is the reference for these methods, not for code:

  • zero-shot calibrated word scores;
  • reference profiles;
  • a realness axis;
  • summary, JSON, sheet and timeline outputs;
  • a resident model process;
  • checks as the arbiter.

Not in the repo

The spike is Python in a scratch venv under ~/films/fingerprint-spike, and it stays out by design (bash, not Python, for repo scripts). Client footage and the golden references' media aren't in git either.

…l heads over frozen on-device encoders (SigLIP 2, CLAP, YAMNet on ort; SigLIP on MLX once MX5 lands), trained from notes, marks, approvals and reference profiles, shared as evidence. FP0a-g hold the 11-12 measurements; FP1-FP13 are the build, with earshot (MIT) as the reference design
@shinyobjectz

Copy link
Copy Markdown
Owner Author

The plan and spec landed on main as e2c5104, and FP1-FP13 are built on main for 0.1.2. Where things ended up differently from this plan:

  • Realness (FP2f) was measured and not shipped: SAO foley held out by generator scored 0.57 against a bar of 0.85.
  • EmbeddingGemma now embeds scripts and other text itself rather than reading the Library's rows, and stills get the picture columns.
  • Patch heads (FP12a) say where in the frame blur, glitch and flat surfaces are, from local defects, at 0.88 to 0.92 held out by source.
  • Waiting on real labels: notes train nothing until 50 readings are checked (FP5b), so FP5c's proof, the grounder (FP13b) and pointing on real marks (FP12b) aren't measured yet.

Each decision is written in docs/fingerprint.feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant