|
| 1 | +@spec |
| 2 | +Feature: Videoform: the kind of video a person makes again and again, read once from their own finished videos, followed and checked on every cut |
| 3 | + The person, 2026-11-19: an agency or creator "making the same type of video over and over again can drop in a bunch |
| 4 | + of those videos, and raea codifies that style so the user can then drop a bucket of raw footage and raea with great |
| 5 | + accuracy and intent edits the raw footage into the exact style it has learned ... and it all needs to happen FAST". |
| 6 | + Then: the embedding route, on the models raea ships; "the point is to embed them, store the embeds, and be able to |
| 7 | + repeat the style without having to go back to the references themselves"; a videoform is "a kind of fingerprint, |
| 8 | + flowing into what has already been built". The plan, its decisions and why: docs/videoform.md. |
| 9 | + |
| 10 | + The model this file sets: |
| 11 | + 1. A videoform is a fingerprint (docs/fingerprint.feature FP14) with a `videoform` section: its context, its |
| 12 | + references, its vocabulary, its template and its rules. It's found, inherited, taught and shared as a |
| 13 | + fingerprint is. |
| 14 | + 2. Each part flows into the part of raea that does that job already: its ranges into its profile (FP6e), its rules |
| 15 | + into a controller's rule fields (DR4, DR8), its structure into a NAME.script.ts template, its references into |
| 16 | + brief.ts's `references` shape. |
| 17 | + 3. A reference is read once, from the local drive or streamed from a URL, and never needed again: its per-shot rows, |
| 18 | + its vocabulary's vectors, its ranges, its structure and its rules are all that following and checking read. |
| 19 | + 4. What stays the same across the references is the videoform; what varies is content. A rule's strength is its |
| 20 | + consistency, and every videoform is tested leave one out. |
| 21 | + 5. What ships runs on the models raea ships, on the person's machine. The domain vocabulary is written by the |
| 22 | + person's own AI. No paid model API is in the product. |
| 23 | + 6. Checks start from the composition: a videoform's rules run in `raea check PROJECT` before any render, and |
| 24 | + embeddings only for shots that changed. |
| 25 | + |
| 26 | + Background: |
| 27 | + Given the readers raea has: footage perception (systems/footage), Fingerprint's columns (systems/columns: SigLIP 2, |
| 28 | + CLAP, YAMNet, cached by content hash), Parakeet's words, Beat This!, HTDemucs and Silero VAD (systems/song), OCR, |
| 29 | + Depth Anything 3, and raea check's one decode of the picture (raea-app-check-read once.rs) |
| 30 | + And Fingerprint's fingerprint.ts (`about`, `extends`, `applies`, axes, encoder pins), its profiles (FP6e: ranges by |
| 31 | + measure name, critique, leave-one-out), `which` (FP14c) and evidence rows (FP2c) |
| 32 | + And controllers' `rule` fields and raea/check's `.ts` checks over the world (DR4), run by `raea check PROJECT` |
| 33 | + And the crate limits (a file at most 500 lines, a crate at most 2,000): raea-sys-fingerprint, -fingerprint-use and |
| 34 | + raea-app-fingerprint are each within 150 lines of theirs, so a videoform's code is its own crates, and the |
| 35 | + fingerprint crates grow only by a field and a map |
| 36 | + And the layers: a systems crate depends sideways only on a facility, so reading (which uses footage, columns and |
| 37 | + song) is apps' (raea-app-videoform), and the maths over kept rows is systems' (raea-sys-videoform) |
| 38 | + |
| 39 | + Rule: VF1. Videos in: from the local drive, or streamed from a URL, read once |
| 40 | + |
| 41 | + @needs:VF2 |
| 42 | + Scenario: VF1. A reference from the drive or a URL, with nothing downloaded |
| 43 | + Given the plan |
| 44 | + | crate | file | lines | job | |
| 45 | + | raea-app-videoform | apps/videoform/src/ingest.rs | 220 | paths, folders and URLs to sources: content hash, source URL, title, creator, date; a source read before is recognised and not read again | |
| 46 | + | raea-app-videoform | apps/videoform/src/rights.rs | 80 | the rights prompt, once per account, and its answer kept with the account | |
| 47 | + When the person runs `raea videoform add NAME PATH|URL...` |
| 48 | + Then a file or a folder on the drive is read where it is, with no copy made |
| 49 | + And a direct link to a media file is decoded straight from the network, and a platform page, a playlist or a |
| 50 | + channel (`--last N`) is resolved (VF1b) into stream URLs that are decoded the same way |
| 51 | + And nothing but the readings is written: no video file is kept, in a cache or anywhere else |
| 52 | + And each source is known by the content hash of its decoded picture and sound, beside its URL, title, creator |
| 53 | + and date, so adding it again reads nothing |
| 54 | + And before the first source from a URL, once per account, the person confirms they have the right to use what |
| 55 | + they add; raea says a videoform keeps measurements and vectors, never the video, and that a platform's terms |
| 56 | + are the person's to keep |
| 57 | + And a login-only or private link is refused with the site's reason, and streams are read at 1080p at most |
| 58 | + |
| 59 | + @needs:VF1 |
| 60 | + Scenario: VF1b. The resolver: yt-dlp, kept current by raea |
| 61 | + Given the plan |
| 62 | + | crate | file | lines | job | |
| 63 | + | raea-io-resolve | io/resolve/src/lib.rs | 60 | a page URL to its videos' stream URLs and headers | |
| 64 | + | raea-io-resolve | io/resolve/src/fetch.rs | 220 | the standalone yt-dlp for this platform fetched into ~/.raea/bin, checked against the release's SHA-256 sums; the pinned version, the daily move forward, the one before kept | |
| 65 | + | raea-io-resolve | io/resolve/src/formats.rs | 140 | `yt-dlp -J` read: the best stream at 1080p or under, its URL and headers; a playlist's or channel's entries | |
| 66 | + Then yt-dlp (Unlicense) is used only to resolve, never to download, and is fetched on first use as FFmpeg is on |
| 67 | + Windows |
| 68 | + And each raea release pins the version it was tested with; between releases raea checks the latest at most once |
| 69 | + a day when a URL is added, checks it against its sums, and keeps the version before |
| 70 | + And when resolving fails with an extractor error, raea fetches the latest and tries once more; if the new one |
| 71 | + fails where the old one worked, it goes back and says which site failed |
| 72 | + And a site that needs JavaScript run (YouTube) gets a runtime raea ships or fetches, named to yt-dlp |
| 73 | + And `raea doctor` prints the yt-dlp in use, its date and the last site that failed |
| 74 | + |
| 75 | + Rule: VF2. A videoform is a kind of fingerprint |
| 76 | + |
| 77 | + Scenario: VF2. `videoform` in fingerprint.ts, and `raea videoform new` |
| 78 | + Given the plan |
| 79 | + | crate | file | lines | job | |
| 80 | + | raea-core-brief | core/brief/src/videoform.rs | 140 | the videoform section's types (context, references, vocabulary, template, rules), with their TypeScript | |
| 81 | + | raea-sys-fingerprint | systems/fingerprint/src/taxonomy.rs | 20 | the optional `videoform` field on a fingerprint | |
| 82 | + | raea-app-videoform | apps/videoform/src/lib.rs | 120 | `raea videoform`: its usage, flags and the verbs' dispatch | |
| 83 | + | raea-app-videoform | apps/videoform/src/verbs.rs | 240 | new, add, report and refresh | |
| 84 | + When the person runs `raea videoform new NAME --context "…" [--extends FP] [--kinds K] [--tags T] [SOURCE...]` |
| 85 | + Then a fingerprint folder is made with a `videoform` section, `extends` the person's fingerprint by default, and |
| 86 | + `applies` from --kinds and --tags, where fingerprints are kept (the project's, the workspace's or the person's) |
| 87 | + And `raea fingerprint which PATH` finds it for a piece as it finds any fingerprint, and says why |
| 88 | + And `raea context` prints a project's videoform with its brief: its context, template, rules and ranges |
| 89 | + And its references use brief.ts's `references` shape, `{ target, note }`, the target the source's hash and URL |
| 90 | + |
| 91 | + Rule: VF3. Each reference read once, into the rows a videoform keeps |
| 92 | + |
| 93 | + @needs:VF1 |
| 94 | + Scenario: VF3. One pass, every reader on its share |
| 95 | + Given the plan |
| 96 | + | crate | file | lines | job | |
| 97 | + | raea-app-videoform | apps/videoform/src/pass.rs | 300 | one decode of a file or a stream: shot bounds from the same pass, each shot's frames held until it ends, each reader fed | |
| 98 | + | raea-app-videoform | apps/videoform/src/readers.rs | 260 | perception, columns, words, the voice's pauses, beats, stems, OCR and depth into per-shot rows | |
| 99 | + | raea-sys-videoform | systems/videoform/src/rows.rs | 160 | the kept rows: per shot its SigLIP 2 vector, its CLAP vector and its measures, f16, with the encoders' pins | |
| 100 | + Then each reference is decoded once, whether from the drive or a stream, and every reader takes its share |
| 101 | + And each shot gets its SigLIP 2 vector (a few frames pooled), its CLAP vector over its sound, and its measures: |
| 102 | + length, scale, camera move, caption boxes, light and colour, words, pauses at its edges, its place in the video |
| 103 | + And reading runs at about real time or faster, measured on the test sets |
| 104 | + |
| 105 | + @needs:VF3 |
| 106 | + Scenario: VF3b. What's kept is enough |
| 107 | + Then a videoform keeps its per-shot rows, its vocabulary's vectors, its profile, its structure and rules, a small |
| 108 | + still for each shot kind, and the person's evidence, and no reference video |
| 109 | + And ten one-minute references come to about 1.5 MB, small enough to load at once and keep in git |
| 110 | + And every test on the test sets (VF12) reads a set's references once, then runs on the kept files alone, with the |
| 111 | + references moved away, and passes the same |
| 112 | + |
| 113 | + Rule: VF4-VF8. Writing the videoform from the kept rows |
| 114 | + |
| 115 | + @needs:VF3 |
| 116 | + Scenario: VF4. The ranges, in the videoform's profile |
| 117 | + Given the plan |
| 118 | + | crate | file | lines | job | |
| 119 | + | raea-sys-videoform | systems/videoform/src/ranges.rs | 220 | the measures across the references: shot lengths, cuts a minute, pauses inside a line, between lines and between sections, words a minute, cuts against beats and sentence ends, music under voice, caption size, place and words at once, talking head against cutaway | |
| 120 | + | raea-sys-fingerprint-use | systems/fingerprint-use/src/profile.rs | 30 | a profile's measures beyond the fixed ones, by name, ranged and critiqued like the rest | |
| 121 | + Then the videoform's profile holds every measure's range across the references, leave one out as FP6e checks it |
| 122 | + And `raea fingerprint score --profile NAME` critiques a cut against the new measures as against the old |
| 123 | + And a measure whose range is wide is recorded as free, not as part of the videoform |
| 124 | + |
| 125 | + @needs:VF2,VF3 |
| 126 | + Scenario: VF5. The vocabulary, written by the person's AI |
| 127 | + Given the plan |
| 128 | + | crate | file | lines | job | |
| 129 | + | raea-sys-videoform | systems/videoform/src/vocab.rs | 180 | vocabulary.ts's terms against the kept rows: never fires, fires everywhere, confused pairs, recurring shots no term names | |
| 130 | + When the person's AI writes vocabulary.ts from the context (places, things in them with several phrasings, |
| 131 | + near-misses, OCR patterns for text on screen, things to ignore) |
| 132 | + Then its terms become the videoform's word heads (FP2e), embedded once by the text towers |
| 133 | + And `raea videoform report NAME` says what each term did on the references, and the AI revises it from that |
| 134 | + |
| 135 | + @needs:VF5 |
| 136 | + Scenario: VF6. The shot vocabulary |
| 137 | + Given the plan |
| 138 | + | crate | file | lines | job | |
| 139 | + | raea-sys-videoform | systems/videoform/src/kinds.rs | 220 | the reference shots clustered on vectors and measures; each cluster named from the vocabulary, its ranges and how many references have it | |
| 140 | + Then every reference shot belongs to a shot kind, named in the vocabulary's words, with its scale, move and length |
| 141 | + ranges and the share of references that have it |
| 142 | + |
| 143 | + @needs:VF6 |
| 144 | + Scenario: VF7. The structure, as a script template |
| 145 | + Given the plan |
| 146 | + | crate | file | lines | job | |
| 147 | + | raea-sys-videoform | systems/videoform/src/structure.rs | 300 | the references aligned by their run of shot kinds and their script's meaning, into slots with their lengths and kinds; the template as NAME.script.ts | |
| 148 | + Then the videoform's slots (open, sections, close) each have a length range, their shot kinds and what's on screen |
| 149 | + And the template is a NAME.script.ts a project made to the videoform starts from, checked by `raea spec` |
| 150 | + |
| 151 | + @needs:VF6,VF7 |
| 152 | + Scenario: VF8. Rules, with their strength, into a controller |
| 153 | + Given the plan |
| 154 | + | crate | file | lines | job | |
| 155 | + | raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written as rule fields of the controller `videoform.rules` names | |
| 156 | + Then a rule that holds in most references (a punch-in when a number is said; a close-up of the counters in every |
| 157 | + kitchen) is a rule field with its strength and its check |
| 158 | + And the person confirms the strongest rules once, on a contact sheet of shot-kind stills, and each answer is |
| 159 | + evidence in the videoform |
| 160 | + |
| 161 | + @needs:VF4,VF6 |
| 162 | + Scenario: VF8b. When it doesn't hold together |
| 163 | + Given the plan |
| 164 | + | crate | file | lines | job | |
| 165 | + | raea-sys-videoform | systems/videoform/src/groups.rs | 180 | too few references, references in groups, traits that vary, outliers | |
| 166 | + Then under five references every rule and range is weak, and nothing gates on it |
| 167 | + And references that fall into clear groups are reported, with a videoform offered for each, never averaged |
| 168 | + And one reference far from the others on several measures is named, for the person to keep or drop |
| 169 | + |
| 170 | + @needs:VF3 |
| 171 | + Scenario: VF8c. References added and removed; encoders upgraded |
| 172 | + Given the plan |
| 173 | + | crate | file | lines | job | |
| 174 | + | raea-sys-videoform | systems/videoform/src/pins.rs | 100 | the encoders' pins on the kept rows; what carries over an upgrade and what waits for refresh | |
| 175 | + Then adding a reference reads only it and makes the videoform again from the kept rows, in seconds; removing one |
| 176 | + drops its rows; the person's evidence stays through both |
| 177 | + And a new encoder in raea leaves the videoform on its rows' pins, working as it was; the measures carry over; `raea |
| 178 | + videoform refresh NAME SOURCE...` reads the references again for the new encoder, the one time they're needed |
| 179 | + |
| 180 | + Rule: VF9-VF11. Following it, and checking it fast |
| 181 | + |
| 182 | + @needs:VF7,VF8 |
| 183 | + Scenario: VF9. Following: the footage matched, what's missing said, the cut planned to the template |
| 184 | + Given the plan |
| 185 | + | crate | file | lines | job | |
| 186 | + | raea-app-videoform | apps/videoform/src/follow.rs | 200 | new footage read and matched to shot kinds; what the footage lacks; the plan slot by slot | |
| 187 | + Then new raw footage is read as references are, and each shot matched to a shot kind against the kept rows |
| 188 | + And before any cut the agent says what the footage lacks ("no close-up of the counters, which every kitchen in the |
| 189 | + references has"), a shot list for the person |
| 190 | + And the agent plans the cut from the template, slot by slot, and edits to the ranges |
| 191 | + |
| 192 | + @needs:VF8 |
| 193 | + Scenario: VF10. A videoform's rules checked on the composition, before any render |
| 194 | + Given the plan |
| 195 | + | crate | file | lines | job | |
| 196 | + | raea-app-videoform | apps/videoform/src/checks.rs | 220 | the videoform's checks by name over the world: order, slot lengths, coverage, shot lengths, pauses, caption places and sizes | |
| 197 | + | raea-app-direction | apps/direction/src/check.rs | 15 | the videoform's checks run with the project's rules | |
| 198 | + Then `raea check PROJECT` runs the videoform's rules with raea's floor and the person's taste, in milliseconds |
| 199 | + And each finding is routed as raea-app-check-cause routes any: a footage's own fault a note, a carried one said |
| 200 | + |
| 201 | + @needs:VF6,VF10 |
| 202 | + Scenario: VF11. Embedding checks, only for shots that changed |
| 203 | + Then each changed shot is held against its slot's shot kind, one SigLIP 2 frame a shot, and the run of shots |
| 204 | + against the structure |
| 205 | + And a shot that didn't change since the last check is not embedded again |
| 206 | + |
| 207 | + Rule: VF12-VF13. Judged on the person's test sets |
| 208 | + |
| 209 | + @needs:VF4 |
| 210 | + Scenario: VF12. The test bench |
| 211 | + Given the plan |
| 212 | + | crate | file | lines | job | |
| 213 | + | raea-app-videoform | apps/videoform/src/bench.rs | 200 | the person's sets as fixtures: leave one out, telling sets apart, edit to it, the verdicts kept | |
| 214 | + Given the person's sets, 8 to 10 videos each from one creator or one format, some with raw footage |
| 215 | + Then for each set, a videoform built from all but one reference holds the one left out (its rules pass, its |
| 216 | + measures in range) more often than other sets' videos do, and rejects other sets' videos |
| 217 | + And where raw footage for a held-out reference exists, the agent edits it to the videoform, set beside the original |
| 218 | + And the bench runs after every step of this file, on every set, from the kept files alone |
| 219 | + And the first proof that matters: on at least five sets, each videoform built leave one out holds its left-out |
| 220 | + video's pacing and shot lengths and rejects other sets' videos, more often than chance by a clear margin |
| 221 | + |
| 222 | + @needs:VF12 |
| 223 | + Scenario: VF13. The person's blind judging, kept as evidence |
| 224 | + Given the plan |
| 225 | + | crate | file | lines | job | |
| 226 | + | raea-app-videoform | apps/videoform/src/judge.rs | 150 | a pair shown unlabelled in a random order; the pick, the scores and the words kept, and as evidence | |
| 227 | + Then the person sees the original and raea's edit side by side, unlabelled, in a random order, picks which they'd |
| 228 | + post, scores pacing, shot choice, structure, graphics, sound and overall feel from 1 to 5, and says what's wrong |
| 229 | + And each verdict is kept in the bench's record and is evidence in the videoform: a pick a pair (FP6b), a note a |
| 230 | + label (FP5) |
| 231 | + |
| 232 | + Rule: Deferred |
| 233 | + Scenario: Not in this file yet |
| 234 | + Then an automatic judge for the test bench (a video-capable language model, paid by the person, in the bench only, |
| 235 | + never shipped: fidelity, pairs, and gaps between what the references show and what the videoform holds) |
| 236 | + And the order of the test sets |
0 commit comments