diff --git a/docs/fingerprint.md b/docs/fingerprint.md index 189dfb6a..4dedb411 100644 --- a/docs/fingerprint.md +++ b/docs/fingerprint.md @@ -353,3 +353,14 @@ notes and is worth shipping without it: a person gets word scores, checks and bo 5. **CLAP joins the stack** (Shane, 11-12: "include clap if we can if its not too much work"). It is about 300 lines of Rust on ort: the 48 kHz log-mel front end and the audio session (200), and the text tower with its tokenizer (100), run only when an axis's words change. Both are checked against the spike's vectors at cosine 0.999. +6. **Pro and Studio include checks and content intelligence** (Shane, 11-20). Each month's pool (about 8,000 frames + checked on Pro, ten times that on Studio) stops when it's spent and is never billed past. It pays for two things: + - **clef-omni answering the fingerprint's questions** about a render's frames (`POST /v1/clef/ask`), so the agent + reads answers in words and opens only the frames they flag. Its verdicts are another eye's (FP6d), and it serves + the axes with no labels yet. + - **content intelligence for references** (`GET /v1/intel/...`): public posts, videos, transcripts and ads, to find + a videoform's references by what performs and keep them current (docs/videoform.md). A source's views and + engagement are another eye's evidence too, never the person's taste. A reference found this way is still read + once, as VF1 reads any link, and VF1's rights confirmation still applies. + + Without a pool, everything here still works locally, and clef runs on the person's own Cloudflare account. diff --git a/docs/videoform.feature b/docs/videoform.feature index 78931928..c81c3cb7 100644 --- a/docs/videoform.feature +++ b/docs/videoform.feature @@ -12,8 +12,8 @@ Feature: Videoform: the kind of video a person makes again and again, read once references, its vocabulary, its template and its rules. It's found, inherited, taught and shared as a fingerprint is. 2. Each part flows into the part of raea that does that job already: its ranges into its profile (FP6e), its rules - into a controller's rule fields (DR4, DR8), its structure into a NAME.script.ts template, its references into - brief.ts's `references` shape. + into the section's `rules`, each in a taste line's shape (words, a check, its values, a level; DR4, DR8), its + structure into a NAME.script.ts template, its references into brief.ts's `references` shape. 3. A reference is read once, from the local drive or streamed from a URL, and never needed again: its per-shot rows, its vocabulary's vectors, its ranges, its structure and its rules are all that following and checking read. 4. What stays the same across the references is the videoform; what varies is content. A rule's strength is its @@ -23,13 +23,26 @@ Feature: Videoform: the kind of video a person makes again and again, read once 6. Checks start from the composition: a videoform's rules run in `raea check PROJECT` before any render, and embeddings only for shots that changed. + Start here (the order of work, and where it touches what exists): + - The order: VF1 and VF2 (a reference in, a videoform made), VF3 (each reference read once into the kept rows), then + VF4 with VF12 (the ranges, judged on the test bench). That pair is the gate: it passes on at least five of the + person's sets before VF1b's platform pages or anything in VF9-VF11 is built. + - New crates hold the work: raea-app-videoform (apps/videoform: ingest, rights, reading, the verbs, the bench), + raea-sys-videoform (systems/videoform: the maths over kept rows) and raea-io-resolve (io/resolve, VF1b). The + videoform section's types are core/brief's videoform.rs. + - Three small hooks into what exists: fingerprint.ts's optional `videoform` field (systems/fingerprint taxonomy.rs, + VF2), `raea context` printing a project's videoform (engine/cli guide.rs, VF2), and `raea check PROJECT` holding + the rules of the fingerprint that judges the piece (apps/direction check.rs, VF8). + - The bench (VF12) runs after every step, on every set, from the kept files alone; its numbers are the progress. + Background: Given the readers raea has: footage perception (systems/footage), Fingerprint's columns (systems/columns: SigLIP 2, CLAP, YAMNet, cached by content hash), Parakeet's words, Beat This!, HTDemucs and Silero VAD (systems/song), OCR, Depth Anything 3, and raea check's one decode of the picture (raea-app-check-read once.rs) And Fingerprint's fingerprint.ts (`about`, `extends`, `applies`, axes, encoder pins), its profiles (FP6e: ranges by measure name, critique, leave-one-out), `which` (FP14c) and evidence rows (FP2c) - And controllers' `rule` fields and raea/check's `.ts` checks over the world (DR4), run by `raea check PROJECT` + And taste lines' checks (raea's six forms, a check by name, a `.ts` check written with raea/check over the world; + DR4), run by `raea check PROJECT` And the crate limits (a file at most 500 lines, a crate at most 2,000): raea-sys-fingerprint, -fingerprint-use and raea-app-fingerprint are each within 150 lines of theirs, so a videoform's code is its own crates, and the fingerprint crates grow only by a field and a map @@ -149,12 +162,15 @@ Feature: Videoform: the kind of video a person makes again and again, read once And the template is a NAME.script.ts a project made to the videoform starts from, checked by `raea spec` @needs:VF6,VF7 - Scenario: VF8. Rules, with their strength, into a controller + Scenario: VF8. Rules, with their strength, in the videoform's rules Given the plan | crate | file | lines | job | - | raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written as rule fields of the controller `videoform.rules` names | + | raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written into the section's `rules`, each a taste line with its check and values | + | raea-app-direction | apps/direction/src/check.rs | 15 | tastes(): the rules of the fingerprint `which` names for the project, after the person's, the workspace's and the project's taste | Then a rule that holds in most references (a punch-in when a number is said; a close-up of the counters in every - kitchen) is a rule field with its strength and its check + kitchen) is a line in the section's `rules`, with its strength and its check + And `raea check PROJECT` holds a piece to the rules of the fingerprint that judges it (raea-sys-fingerprint-use + `which`), beside the person's, the workspace's and the project's taste And the person confirms the strongest rules once, on a contact sheet of shot-kind stills, and each answer is evidence in the videoform diff --git a/docs/videoform.md b/docs/videoform.md index b5eef4e7..8cb27e8f 100644 --- a/docs/videoform.md +++ b/docs/videoform.md @@ -49,11 +49,10 @@ not only what was corrected. Notes still matter. They teach what the numbers can | Fingerprint: `fingerprint.ts` with `about`, `extends`, `applies` kinds and tags; `which` finds one by project, workspace and account; a child inherits its parent's evidence, its own counted more; notes teach the one that judged (FP14a-e) | the home: a videoform is one of these | the fields a kind of video needs | | Fingerprint's frozen encoders (SigLIP 2, CLAP, YAMNet, EmbeddingGemma 2, Depth Anything 3), columns cached by content hash, word and learned heads, evidence in f16 rows (FP1-FP4) | the per-shot vectors and measures, and where they're kept | no shot vocabulary, no structure | | Fingerprint profiles: ranges from admired work, critique against them, a leave-one-out test (FP6e, `raea fingerprint profile`) | a videoform's ranges, and its validation | few measures (loudness, tilt, cut rate, motion, light, colour, sharpness) | -| Controllers' `rule` fields, a check by name or a `.ts` check on raea/check's world (DR4) | rules that `raea check PROJECT` and every agent hold work to | none written from references | | Notes become checks, and climb to project, workspace and taste (DR5-DR8, `raea notes keep`) | the path a correction takes to becoming a rule | a videoform's rules start from evidence, not notes | | `raea check`: findings and the record (DR3); who can fix what, carried findings, one decode (raea-app-check-cause, PR #205) | the check every pass runs | a videoform's rules join it as more rules | | `NAME.script.ts` (scenes, beats, lines, cut); `brief.ts` (`references`, `look`, `mode`, `deliver`) | a storyboard spec, and a brief that names references | no template made from references; references named, never read | -| `taste.ts`, six checked forms | the person's standing lines | the forms are fixed: mined rules go to controllers, not here | +| `taste.ts` lines: words, and a check that holds them (one of six forms, a check by name, or a `.ts` check on raea/check's world; DR4) | the person's, the workspace's and the project's standing lines, which `raea check PROJECT` and every agent hold work to | none written from references | | Footage perception (`systems/footage`: shots, cuts, camera, faces, words on screen, scene labels, performance, facts.json) | what each reference shot is | read for raw footage, not for references | | Transcription (Parakeet), beats, stems and voice activity (`raea song`, HTDemucs, Silero VAD), OCR | words, pauses, cuts against beats, music under voice, captions | not gathered into a videoform | | Index and libraries (SigLIP over a person's media; turbopuffer in the cloud) | finding footage by meaning; vector storage for teams | — | @@ -83,7 +82,7 @@ Each part of a videoform flows into the part of raea that already does that job: |---|---|---| | ranges (pace, pauses, shot lengths, loudness, colour, captions) | its own profile (FP6e) | `raea fingerprint score --profile` critiques a cut against them, and leave-one-out is already built | | the taste taught later | its own evidence and heads (FP2, FP14b-d) | a note on a piece it judged teaches it, never the person's general fingerprint unless they move it up | -| rules (coverage, order, event rules, limits) | a controller's `rule` fields, which `videoform.rules` names | `raea check PROJECT` runs them, and they're edited in the app like any controller | +| rules (coverage, order, event rules, limits) | the section's `rules`, each in a taste line's shape (words, check, values, level) | `raea check PROJECT` runs them for a piece the videoform judges (`which`), beside the taste | | structure (slots, their order and lengths) | a `NAME.script.ts` template, which `videoform.template` names | a project made to the videoform starts from it, and agents plan from it as from any storyboard | | references | `brief.ts`'s existing `references` shape, `{ target, note }`, with the source kept by content hash | a reference is named the way a brief already names one | @@ -161,7 +160,7 @@ Following and checking need four kinds of thing from the references, and a video | per-shot rows | each reference shot's SigLIP 2 vector (one pooled vector, a few frames averaged), its CLAP vector over the shot's sound, and its measures (length, scale, camera move, caption boxes, light and colour, words, pauses at its edges, its place in the video) | the shot vocabulary; matching new footage to shot kinds; the embedding checks | | the vocabulary's vectors | each term's phrasings through the text towers, as word heads store them (FP2e) | naming shot kinds; finding domain things in new footage | | the profile | the ranges of every measure across the references (FP6e) | the critique, and composition-first range checks | -| the structure and the rules | the slots with their lengths and shot kinds; the event and coverage rules with their strength | the template, and the controller rules `raea check` runs | +| the structure and the rules | the slots with their lengths and shot kinds; the event and coverage rules with their strength | the template, and the rules `raea check` runs | Also kept: one small still for each shot kind (a thumbnail, a few kilobytes), so the person can see what a kind is when confirming rules, and the evidence the person adds over time. The reference videos aren't kept. @@ -226,8 +225,8 @@ working as it was. This is the one time a reference is needed again, and it's th (open, sections, close), each with its length range, its shot kinds and what's on screen. This becomes the script template. 5. **The event rules.** How often one thing comes with another across the references ("a punch-in when a number is - said", "a close-up of the counters in every kitchen"). Those that hold in most references become controller rules, - each stating how strongly it holds. + said", "a close-up of the counters in every kitchen"). Those that hold in most references become the videoform's + rules, each stating how strongly it holds. 6. **The person confirms the strongest rules once,** on a contact sheet ("countertop close-up in every kitchen: 9 of 10"). Each yes or no is evidence in the videoform, as any label is in a fingerprint. @@ -247,7 +246,7 @@ working as it was. This is the one time a reference is needed again, and it's th - shot lengths and pauses (the voice's cut points are in the composition); - caption places and sizes. - These are the controller rules, run by `raea check PROJECT` as today. + These are the videoform's rules, run by `raea check PROJECT` with the taste. 2. **On the render manifest and measures, in seconds:** loudness, light and colour against the profile (`raea fingerprint score --profile`). 3. **On embeddings, only for shots that changed:** each shot against its slot's shot kind (one SigLIP frame a shot); @@ -298,7 +297,7 @@ The sets, their answers and their verdicts are kept as fixtures, so each step of | | VF5 | the vocabulary loop with the person's AI, as word heads | medium-high | word heads already score from text; the report is new | | | VF6 | the shot vocabulary (clusters named from the vocabulary) | medium | clustering is plain; naming quality varies by domain | | | VF7 | the structure template | medium | alignment across varied videos is the hard part | -| | VF8 | event rules into controller rules, with strength, confirmed by the person | medium | co-occurrence is plain; telling habit from style needs the person | +| | VF8 | event rules into the videoform's rules, with strength, confirmed by the person; `raea check` holds them | medium | co-occurrence is plain; telling habit from style needs the person | | | VF8b | when it doesn't hold together: too few, mixed kinds (split into videoforms), traits that vary, outliers | medium-high | groups and spreads come from the kept rows | | | VF8c | adding and removing references from kept rows; encoder pins and `refresh` | high | Fingerprint keeps evidence across upgrades the same way (FP8c) | | 3, use | VF9 | following: context, coverage gaps, planning to the template | medium-high | context and scripts exist |