Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/fingerprint.md
Original file line number Diff line number Diff line change
Expand Up @@ -353,3 +353,14 @@ notes and is worth shipping without it: a person gets word scores, checks and bo
5. **CLAP joins the stack** (Shane, 11-12: "include clap if we can if its not too much work"). It is about 300 lines
of Rust on ort: the 48 kHz log-mel front end and the audio session (200), and the text tower with its tokenizer
(100), run only when an axis's words change. Both are checked against the spike's vectors at cosine 0.999.
6. **Pro and Studio include checks and content intelligence** (Shane, 11-20). Each month's pool (about 8,000 frames
checked on Pro, ten times that on Studio) stops when it's spent and is never billed past. It pays for two things:
- **clef-omni answering the fingerprint's questions** about a render's frames (`POST /v1/clef/ask`), so the agent
reads answers in words and opens only the frames they flag. Its verdicts are another eye's (FP6d), and it serves
the axes with no labels yet.
- **content intelligence for references** (`GET /v1/intel/...`): public posts, videos, transcripts and ads, to find
a videoform's references by what performs and keep them current (docs/videoform.md). A source's views and
engagement are another eye's evidence too, never the person's taste. A reference found this way is still read
once, as VF1 reads any link, and VF1's rights confirmation still applies.

Without a pool, everything here still works locally, and clef runs on the person's own Cloudflare account.
28 changes: 22 additions & 6 deletions docs/videoform.feature
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ Feature: Videoform: the kind of video a person makes again and again, read once
references, its vocabulary, its template and its rules. It's found, inherited, taught and shared as a
fingerprint is.
2. Each part flows into the part of raea that does that job already: its ranges into its profile (FP6e), its rules
into a controller's rule fields (DR4, DR8), its structure into a NAME.script.ts template, its references into
brief.ts's `references` shape.
into the section's `rules`, each in a taste line's shape (words, a check, its values, a level; DR4, DR8), its
structure into a NAME.script.ts template, its references into brief.ts's `references` shape.
3. A reference is read once, from the local drive or streamed from a URL, and never needed again: its per-shot rows,
its vocabulary's vectors, its ranges, its structure and its rules are all that following and checking read.
4. What stays the same across the references is the videoform; what varies is content. A rule's strength is its
Expand All @@ -23,13 +23,26 @@ Feature: Videoform: the kind of video a person makes again and again, read once
6. Checks start from the composition: a videoform's rules run in `raea check PROJECT` before any render, and
embeddings only for shots that changed.

Start here (the order of work, and where it touches what exists):
- The order: VF1 and VF2 (a reference in, a videoform made), VF3 (each reference read once into the kept rows), then
VF4 with VF12 (the ranges, judged on the test bench). That pair is the gate: it passes on at least five of the
person's sets before VF1b's platform pages or anything in VF9-VF11 is built.
- New crates hold the work: raea-app-videoform (apps/videoform: ingest, rights, reading, the verbs, the bench),
raea-sys-videoform (systems/videoform: the maths over kept rows) and raea-io-resolve (io/resolve, VF1b). The
videoform section's types are core/brief's videoform.rs.
- Three small hooks into what exists: fingerprint.ts's optional `videoform` field (systems/fingerprint taxonomy.rs,
VF2), `raea context` printing a project's videoform (engine/cli guide.rs, VF2), and `raea check PROJECT` holding
the rules of the fingerprint that judges the piece (apps/direction check.rs, VF8).
- The bench (VF12) runs after every step, on every set, from the kept files alone; its numbers are the progress.

Background:
Given the readers raea has: footage perception (systems/footage), Fingerprint's columns (systems/columns: SigLIP 2,
CLAP, YAMNet, cached by content hash), Parakeet's words, Beat This!, HTDemucs and Silero VAD (systems/song), OCR,
Depth Anything 3, and raea check's one decode of the picture (raea-app-check-read once.rs)
And Fingerprint's fingerprint.ts (`about`, `extends`, `applies`, axes, encoder pins), its profiles (FP6e: ranges by
measure name, critique, leave-one-out), `which` (FP14c) and evidence rows (FP2c)
And controllers' `rule` fields and raea/check's `.ts` checks over the world (DR4), run by `raea check PROJECT`
And taste lines' checks (raea's six forms, a check by name, a `.ts` check written with raea/check over the world;
DR4), run by `raea check PROJECT`
And the crate limits (a file at most 500 lines, a crate at most 2,000): raea-sys-fingerprint, -fingerprint-use and
raea-app-fingerprint are each within 150 lines of theirs, so a videoform's code is its own crates, and the
fingerprint crates grow only by a field and a map
Expand Down Expand Up @@ -149,12 +162,15 @@ Feature: Videoform: the kind of video a person makes again and again, read once
And the template is a NAME.script.ts a project made to the videoform starts from, checked by `raea spec`

@needs:VF6,VF7
Scenario: VF8. Rules, with their strength, into a controller
Scenario: VF8. Rules, with their strength, in the videoform's rules
Given the plan
| crate | file | lines | job |
| raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written as rule fields of the controller `videoform.rules` names |
| raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written into the section's `rules`, each a taste line with its check and values |
| raea-app-direction | apps/direction/src/check.rs | 15 | tastes(): the rules of the fingerprint `which` names for the project, after the person's, the workspace's and the project's taste |
Then a rule that holds in most references (a punch-in when a number is said; a close-up of the counters in every
kitchen) is a rule field with its strength and its check
kitchen) is a line in the section's `rules`, with its strength and its check
And `raea check PROJECT` holds a piece to the rules of the fingerprint that judges it (raea-sys-fingerprint-use
`which`), beside the person's, the workspace's and the project's taste
And the person confirms the strongest rules once, on a contact sheet of shot-kind stills, and each answer is
evidence in the videoform

Expand Down
15 changes: 7 additions & 8 deletions docs/videoform.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,10 @@ not only what was corrected. Notes still matter. They teach what the numbers can
| Fingerprint: `fingerprint.ts` with `about`, `extends`, `applies` kinds and tags; `which` finds one by project, workspace and account; a child inherits its parent's evidence, its own counted more; notes teach the one that judged (FP14a-e) | the home: a videoform is one of these | the fields a kind of video needs |
| Fingerprint's frozen encoders (SigLIP 2, CLAP, YAMNet, EmbeddingGemma 2, Depth Anything 3), columns cached by content hash, word and learned heads, evidence in f16 rows (FP1-FP4) | the per-shot vectors and measures, and where they're kept | no shot vocabulary, no structure |
| Fingerprint profiles: ranges from admired work, critique against them, a leave-one-out test (FP6e, `raea fingerprint profile`) | a videoform's ranges, and its validation | few measures (loudness, tilt, cut rate, motion, light, colour, sharpness) |
| Controllers' `rule` fields, a check by name or a `.ts` check on raea/check's world (DR4) | rules that `raea check PROJECT` and every agent hold work to | none written from references |
| Notes become checks, and climb to project, workspace and taste (DR5-DR8, `raea notes keep`) | the path a correction takes to becoming a rule | a videoform's rules start from evidence, not notes |
| `raea check`: findings and the record (DR3); who can fix what, carried findings, one decode (raea-app-check-cause, PR #205) | the check every pass runs | a videoform's rules join it as more rules |
| `NAME.script.ts` (scenes, beats, lines, cut); `brief.ts` (`references`, `look`, `mode`, `deliver`) | a storyboard spec, and a brief that names references | no template made from references; references named, never read |
| `taste.ts`, six checked forms | the person's standing lines | the forms are fixed: mined rules go to controllers, not here |
| `taste.ts` lines: words, and a check that holds them (one of six forms, a check by name, or a `.ts` check on raea/check's world; DR4) | the person's, the workspace's and the project's standing lines, which `raea check PROJECT` and every agent hold work to | none written from references |
| Footage perception (`systems/footage`: shots, cuts, camera, faces, words on screen, scene labels, performance, facts.json) | what each reference shot is | read for raw footage, not for references |
| Transcription (Parakeet), beats, stems and voice activity (`raea song`, HTDemucs, Silero VAD), OCR | words, pauses, cuts against beats, music under voice, captions | not gathered into a videoform |
| Index and libraries (SigLIP over a person's media; turbopuffer in the cloud) | finding footage by meaning; vector storage for teams | — |
Expand Down Expand Up @@ -83,7 +82,7 @@ Each part of a videoform flows into the part of raea that already does that job:
|---|---|---|
| ranges (pace, pauses, shot lengths, loudness, colour, captions) | its own profile (FP6e) | `raea fingerprint score --profile` critiques a cut against them, and leave-one-out is already built |
| the taste taught later | its own evidence and heads (FP2, FP14b-d) | a note on a piece it judged teaches it, never the person's general fingerprint unless they move it up |
| rules (coverage, order, event rules, limits) | a controller's `rule` fields, which `videoform.rules` names | `raea check PROJECT` runs them, and they're edited in the app like any controller |
| rules (coverage, order, event rules, limits) | the section's `rules`, each in a taste line's shape (words, check, values, level) | `raea check PROJECT` runs them for a piece the videoform judges (`which`), beside the taste |
| structure (slots, their order and lengths) | a `NAME.script.ts` template, which `videoform.template` names | a project made to the videoform starts from it, and agents plan from it as from any storyboard |
| references | `brief.ts`'s existing `references` shape, `{ target, note }`, with the source kept by content hash | a reference is named the way a brief already names one |

Expand Down Expand Up @@ -161,7 +160,7 @@ Following and checking need four kinds of thing from the references, and a video
| per-shot rows | each reference shot's SigLIP 2 vector (one pooled vector, a few frames averaged), its CLAP vector over the shot's sound, and its measures (length, scale, camera move, caption boxes, light and colour, words, pauses at its edges, its place in the video) | the shot vocabulary; matching new footage to shot kinds; the embedding checks |
| the vocabulary's vectors | each term's phrasings through the text towers, as word heads store them (FP2e) | naming shot kinds; finding domain things in new footage |
| the profile | the ranges of every measure across the references (FP6e) | the critique, and composition-first range checks |
| the structure and the rules | the slots with their lengths and shot kinds; the event and coverage rules with their strength | the template, and the controller rules `raea check` runs |
| the structure and the rules | the slots with their lengths and shot kinds; the event and coverage rules with their strength | the template, and the rules `raea check` runs |

Also kept: one small still for each shot kind (a thumbnail, a few kilobytes), so the person can see what a kind is when
confirming rules, and the evidence the person adds over time. The reference videos aren't kept.
Expand Down Expand Up @@ -226,8 +225,8 @@ working as it was. This is the one time a reference is needed again, and it's th
(open, sections, close), each with its length range, its shot kinds and what's on screen. This becomes the script
template.
5. **The event rules.** How often one thing comes with another across the references ("a punch-in when a number is
said", "a close-up of the counters in every kitchen"). Those that hold in most references become controller rules,
each stating how strongly it holds.
said", "a close-up of the counters in every kitchen"). Those that hold in most references become the videoform's
rules, each stating how strongly it holds.
6. **The person confirms the strongest rules once,** on a contact sheet ("countertop close-up in every kitchen: 9 of
10"). Each yes or no is evidence in the videoform, as any label is in a fingerprint.

Expand All @@ -247,7 +246,7 @@ working as it was. This is the one time a reference is needed again, and it's th
- shot lengths and pauses (the voice's cut points are in the composition);
- caption places and sizes.

These are the controller rules, run by `raea check PROJECT` as today.
These are the videoform's rules, run by `raea check PROJECT` with the taste.
2. **On the render manifest and measures, in seconds:** loudness, light and colour against the profile (`raea
fingerprint score --profile`).
3. **On embeddings, only for shots that changed:** each shot against its slot's shot kind (one SigLIP frame a shot);
Expand Down Expand Up @@ -298,7 +297,7 @@ The sets, their answers and their verdicts are kept as fixtures, so each step of
| | VF5 | the vocabulary loop with the person's AI, as word heads | medium-high | word heads already score from text; the report is new |
| | VF6 | the shot vocabulary (clusters named from the vocabulary) | medium | clustering is plain; naming quality varies by domain |
| | VF7 | the structure template | medium | alignment across varied videos is the hard part |
| | VF8 | event rules into controller rules, with strength, confirmed by the person | medium | co-occurrence is plain; telling habit from style needs the person |
| | VF8 | event rules into the videoform's rules, with strength, confirmed by the person; `raea check` holds them | medium | co-occurrence is plain; telling habit from style needs the person |
| | VF8b | when it doesn't hold together: too few, mixed kinds (split into videoforms), traits that vary, outliers | medium-high | groups and spreads come from the kept rows |
| | VF8c | adding and removing references from kept rows; encoder pins and `refresh` | high | Fingerprint keeps evidence across upgrades the same way (FP8c) |
| 3, use | VF9 | following: context, coverage gaps, planning to the template | medium-high | context and scripts exist |
Expand Down
Loading