Skip to content

Commit db53845

Browse files
authored
Merge pull request #207 from wackyza/styles-plan
Videoform: the plan and its tasks (VF1-VF13)
2 parents 3ffe34b + 0706bae commit db53845

2 files changed

Lines changed: 555 additions & 0 deletions

File tree

‎docs/videoform.feature‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
@spec
2+
Feature: Videoform: the kind of video a person makes again and again, read once from their own finished videos, followed and checked on every cut
3+
The person, 2026-11-19: an agency or creator "making the same type of video over and over again can drop in a bunch
4+
of those videos, and raea codifies that style so the user can then drop a bucket of raw footage and raea with great
5+
accuracy and intent edits the raw footage into the exact style it has learned ... and it all needs to happen FAST".
6+
Then: the embedding route, on the models raea ships; "the point is to embed them, store the embeds, and be able to
7+
repeat the style without having to go back to the references themselves"; a videoform is "a kind of fingerprint,
8+
flowing into what has already been built". The plan, its decisions and why: docs/videoform.md.
9+
10+
The model this file sets:
11+
1. A videoform is a fingerprint (docs/fingerprint.feature FP14) with a `videoform` section: its context, its
12+
references, its vocabulary, its template and its rules. It's found, inherited, taught and shared as a
13+
fingerprint is.
14+
2. Each part flows into the part of raea that does that job already: its ranges into its profile (FP6e), its rules
15+
into a controller's rule fields (DR4, DR8), its structure into a NAME.script.ts template, its references into
16+
brief.ts's `references` shape.
17+
3. A reference is read once, from the local drive or streamed from a URL, and never needed again: its per-shot rows,
18+
its vocabulary's vectors, its ranges, its structure and its rules are all that following and checking read.
19+
4. What stays the same across the references is the videoform; what varies is content. A rule's strength is its
20+
consistency, and every videoform is tested leave one out.
21+
5. What ships runs on the models raea ships, on the person's machine. The domain vocabulary is written by the
22+
person's own AI. No paid model API is in the product.
23+
6. Checks start from the composition: a videoform's rules run in `raea check PROJECT` before any render, and
24+
embeddings only for shots that changed.
25+
26+
Background:
27+
Given the readers raea has: footage perception (systems/footage), Fingerprint's columns (systems/columns: SigLIP 2,
28+
CLAP, YAMNet, cached by content hash), Parakeet's words, Beat This!, HTDemucs and Silero VAD (systems/song), OCR,
29+
Depth Anything 3, and raea check's one decode of the picture (raea-app-check-read once.rs)
30+
And Fingerprint's fingerprint.ts (`about`, `extends`, `applies`, axes, encoder pins), its profiles (FP6e: ranges by
31+
measure name, critique, leave-one-out), `which` (FP14c) and evidence rows (FP2c)
32+
And controllers' `rule` fields and raea/check's `.ts` checks over the world (DR4), run by `raea check PROJECT`
33+
And the crate limits (a file at most 500 lines, a crate at most 2,000): raea-sys-fingerprint, -fingerprint-use and
34+
raea-app-fingerprint are each within 150 lines of theirs, so a videoform's code is its own crates, and the
35+
fingerprint crates grow only by a field and a map
36+
And the layers: a systems crate depends sideways only on a facility, so reading (which uses footage, columns and
37+
song) is apps' (raea-app-videoform), and the maths over kept rows is systems' (raea-sys-videoform)
38+
39+
Rule: VF1. Videos in: from the local drive, or streamed from a URL, read once
40+
41+
@needs:VF2
42+
Scenario: VF1. A reference from the drive or a URL, with nothing downloaded
43+
Given the plan
44+
| crate | file | lines | job |
45+
| raea-app-videoform | apps/videoform/src/ingest.rs | 220 | paths, folders and URLs to sources: content hash, source URL, title, creator, date; a source read before is recognised and not read again |
46+
| raea-app-videoform | apps/videoform/src/rights.rs | 80 | the rights prompt, once per account, and its answer kept with the account |
47+
When the person runs `raea videoform add NAME PATH|URL...`
48+
Then a file or a folder on the drive is read where it is, with no copy made
49+
And a direct link to a media file is decoded straight from the network, and a platform page, a playlist or a
50+
channel (`--last N`) is resolved (VF1b) into stream URLs that are decoded the same way
51+
And nothing but the readings is written: no video file is kept, in a cache or anywhere else
52+
And each source is known by the content hash of its decoded picture and sound, beside its URL, title, creator
53+
and date, so adding it again reads nothing
54+
And before the first source from a URL, once per account, the person confirms they have the right to use what
55+
they add; raea says a videoform keeps measurements and vectors, never the video, and that a platform's terms
56+
are the person's to keep
57+
And a login-only or private link is refused with the site's reason, and streams are read at 1080p at most
58+
59+
@needs:VF1
60+
Scenario: VF1b. The resolver: yt-dlp, kept current by raea
61+
Given the plan
62+
| crate | file | lines | job |
63+
| raea-io-resolve | io/resolve/src/lib.rs | 60 | a page URL to its videos' stream URLs and headers |
64+
| raea-io-resolve | io/resolve/src/fetch.rs | 220 | the standalone yt-dlp for this platform fetched into ~/.raea/bin, checked against the release's SHA-256 sums; the pinned version, the daily move forward, the one before kept |
65+
| raea-io-resolve | io/resolve/src/formats.rs | 140 | `yt-dlp -J` read: the best stream at 1080p or under, its URL and headers; a playlist's or channel's entries |
66+
Then yt-dlp (Unlicense) is used only to resolve, never to download, and is fetched on first use as FFmpeg is on
67+
Windows
68+
And each raea release pins the version it was tested with; between releases raea checks the latest at most once
69+
a day when a URL is added, checks it against its sums, and keeps the version before
70+
And when resolving fails with an extractor error, raea fetches the latest and tries once more; if the new one
71+
fails where the old one worked, it goes back and says which site failed
72+
And a site that needs JavaScript run (YouTube) gets a runtime raea ships or fetches, named to yt-dlp
73+
And `raea doctor` prints the yt-dlp in use, its date and the last site that failed
74+
75+
Rule: VF2. A videoform is a kind of fingerprint
76+
77+
Scenario: VF2. `videoform` in fingerprint.ts, and `raea videoform new`
78+
Given the plan
79+
| crate | file | lines | job |
80+
| raea-core-brief | core/brief/src/videoform.rs | 140 | the videoform section's types (context, references, vocabulary, template, rules), with their TypeScript |
81+
| raea-sys-fingerprint | systems/fingerprint/src/taxonomy.rs | 20 | the optional `videoform` field on a fingerprint |
82+
| raea-app-videoform | apps/videoform/src/lib.rs | 120 | `raea videoform`: its usage, flags and the verbs' dispatch |
83+
| raea-app-videoform | apps/videoform/src/verbs.rs | 240 | new, add, report and refresh |
84+
When the person runs `raea videoform new NAME --context "…" [--extends FP] [--kinds K] [--tags T] [SOURCE...]`
85+
Then a fingerprint folder is made with a `videoform` section, `extends` the person's fingerprint by default, and
86+
`applies` from --kinds and --tags, where fingerprints are kept (the project's, the workspace's or the person's)
87+
And `raea fingerprint which PATH` finds it for a piece as it finds any fingerprint, and says why
88+
And `raea context` prints a project's videoform with its brief: its context, template, rules and ranges
89+
And its references use brief.ts's `references` shape, `{ target, note }`, the target the source's hash and URL
90+
91+
Rule: VF3. Each reference read once, into the rows a videoform keeps
92+
93+
@needs:VF1
94+
Scenario: VF3. One pass, every reader on its share
95+
Given the plan
96+
| crate | file | lines | job |
97+
| raea-app-videoform | apps/videoform/src/pass.rs | 300 | one decode of a file or a stream: shot bounds from the same pass, each shot's frames held until it ends, each reader fed |
98+
| raea-app-videoform | apps/videoform/src/readers.rs | 260 | perception, columns, words, the voice's pauses, beats, stems, OCR and depth into per-shot rows |
99+
| raea-sys-videoform | systems/videoform/src/rows.rs | 160 | the kept rows: per shot its SigLIP 2 vector, its CLAP vector and its measures, f16, with the encoders' pins |
100+
Then each reference is decoded once, whether from the drive or a stream, and every reader takes its share
101+
And each shot gets its SigLIP 2 vector (a few frames pooled), its CLAP vector over its sound, and its measures:
102+
length, scale, camera move, caption boxes, light and colour, words, pauses at its edges, its place in the video
103+
And reading runs at about real time or faster, measured on the test sets
104+
105+
@needs:VF3
106+
Scenario: VF3b. What's kept is enough
107+
Then a videoform keeps its per-shot rows, its vocabulary's vectors, its profile, its structure and rules, a small
108+
still for each shot kind, and the person's evidence, and no reference video
109+
And ten one-minute references come to about 1.5 MB, small enough to load at once and keep in git
110+
And every test on the test sets (VF12) reads a set's references once, then runs on the kept files alone, with the
111+
references moved away, and passes the same
112+
113+
Rule: VF4-VF8. Writing the videoform from the kept rows
114+
115+
@needs:VF3
116+
Scenario: VF4. The ranges, in the videoform's profile
117+
Given the plan
118+
| crate | file | lines | job |
119+
| raea-sys-videoform | systems/videoform/src/ranges.rs | 220 | the measures across the references: shot lengths, cuts a minute, pauses inside a line, between lines and between sections, words a minute, cuts against beats and sentence ends, music under voice, caption size, place and words at once, talking head against cutaway |
120+
| raea-sys-fingerprint-use | systems/fingerprint-use/src/profile.rs | 30 | a profile's measures beyond the fixed ones, by name, ranged and critiqued like the rest |
121+
Then the videoform's profile holds every measure's range across the references, leave one out as FP6e checks it
122+
And `raea fingerprint score --profile NAME` critiques a cut against the new measures as against the old
123+
And a measure whose range is wide is recorded as free, not as part of the videoform
124+
125+
@needs:VF2,VF3
126+
Scenario: VF5. The vocabulary, written by the person's AI
127+
Given the plan
128+
| crate | file | lines | job |
129+
| raea-sys-videoform | systems/videoform/src/vocab.rs | 180 | vocabulary.ts's terms against the kept rows: never fires, fires everywhere, confused pairs, recurring shots no term names |
130+
When the person's AI writes vocabulary.ts from the context (places, things in them with several phrasings,
131+
near-misses, OCR patterns for text on screen, things to ignore)
132+
Then its terms become the videoform's word heads (FP2e), embedded once by the text towers
133+
And `raea videoform report NAME` says what each term did on the references, and the AI revises it from that
134+
135+
@needs:VF5
136+
Scenario: VF6. The shot vocabulary
137+
Given the plan
138+
| crate | file | lines | job |
139+
| raea-sys-videoform | systems/videoform/src/kinds.rs | 220 | the reference shots clustered on vectors and measures; each cluster named from the vocabulary, its ranges and how many references have it |
140+
Then every reference shot belongs to a shot kind, named in the vocabulary's words, with its scale, move and length
141+
ranges and the share of references that have it
142+
143+
@needs:VF6
144+
Scenario: VF7. The structure, as a script template
145+
Given the plan
146+
| crate | file | lines | job |
147+
| raea-sys-videoform | systems/videoform/src/structure.rs | 300 | the references aligned by their run of shot kinds and their script's meaning, into slots with their lengths and kinds; the template as NAME.script.ts |
148+
Then the videoform's slots (open, sections, close) each have a length range, their shot kinds and what's on screen
149+
And the template is a NAME.script.ts a project made to the videoform starts from, checked by `raea spec`
150+
151+
@needs:VF6,VF7
152+
Scenario: VF8. Rules, with their strength, into a controller
153+
Given the plan
154+
| crate | file | lines | job |
155+
| raea-sys-videoform | systems/videoform/src/rules.rs | 240 | coverage, order and event rules from the kept rows, each with how strongly it holds; written as rule fields of the controller `videoform.rules` names |
156+
Then a rule that holds in most references (a punch-in when a number is said; a close-up of the counters in every
157+
kitchen) is a rule field with its strength and its check
158+
And the person confirms the strongest rules once, on a contact sheet of shot-kind stills, and each answer is
159+
evidence in the videoform
160+
161+
@needs:VF4,VF6
162+
Scenario: VF8b. When it doesn't hold together
163+
Given the plan
164+
| crate | file | lines | job |
165+
| raea-sys-videoform | systems/videoform/src/groups.rs | 180 | too few references, references in groups, traits that vary, outliers |
166+
Then under five references every rule and range is weak, and nothing gates on it
167+
And references that fall into clear groups are reported, with a videoform offered for each, never averaged
168+
And one reference far from the others on several measures is named, for the person to keep or drop
169+
170+
@needs:VF3
171+
Scenario: VF8c. References added and removed; encoders upgraded
172+
Given the plan
173+
| crate | file | lines | job |
174+
| raea-sys-videoform | systems/videoform/src/pins.rs | 100 | the encoders' pins on the kept rows; what carries over an upgrade and what waits for refresh |
175+
Then adding a reference reads only it and makes the videoform again from the kept rows, in seconds; removing one
176+
drops its rows; the person's evidence stays through both
177+
And a new encoder in raea leaves the videoform on its rows' pins, working as it was; the measures carry over; `raea
178+
videoform refresh NAME SOURCE...` reads the references again for the new encoder, the one time they're needed
179+
180+
Rule: VF9-VF11. Following it, and checking it fast
181+
182+
@needs:VF7,VF8
183+
Scenario: VF9. Following: the footage matched, what's missing said, the cut planned to the template
184+
Given the plan
185+
| crate | file | lines | job |
186+
| raea-app-videoform | apps/videoform/src/follow.rs | 200 | new footage read and matched to shot kinds; what the footage lacks; the plan slot by slot |
187+
Then new raw footage is read as references are, and each shot matched to a shot kind against the kept rows
188+
And before any cut the agent says what the footage lacks ("no close-up of the counters, which every kitchen in the
189+
references has"), a shot list for the person
190+
And the agent plans the cut from the template, slot by slot, and edits to the ranges
191+
192+
@needs:VF8
193+
Scenario: VF10. A videoform's rules checked on the composition, before any render
194+
Given the plan
195+
| crate | file | lines | job |
196+
| raea-app-videoform | apps/videoform/src/checks.rs | 220 | the videoform's checks by name over the world: order, slot lengths, coverage, shot lengths, pauses, caption places and sizes |
197+
| raea-app-direction | apps/direction/src/check.rs | 15 | the videoform's checks run with the project's rules |
198+
Then `raea check PROJECT` runs the videoform's rules with raea's floor and the person's taste, in milliseconds
199+
And each finding is routed as raea-app-check-cause routes any: a footage's own fault a note, a carried one said
200+
201+
@needs:VF6,VF10
202+
Scenario: VF11. Embedding checks, only for shots that changed
203+
Then each changed shot is held against its slot's shot kind, one SigLIP 2 frame a shot, and the run of shots
204+
against the structure
205+
And a shot that didn't change since the last check is not embedded again
206+
207+
Rule: VF12-VF13. Judged on the person's test sets
208+
209+
@needs:VF4
210+
Scenario: VF12. The test bench
211+
Given the plan
212+
| crate | file | lines | job |
213+
| raea-app-videoform | apps/videoform/src/bench.rs | 200 | the person's sets as fixtures: leave one out, telling sets apart, edit to it, the verdicts kept |
214+
Given the person's sets, 8 to 10 videos each from one creator or one format, some with raw footage
215+
Then for each set, a videoform built from all but one reference holds the one left out (its rules pass, its
216+
measures in range) more often than other sets' videos do, and rejects other sets' videos
217+
And where raw footage for a held-out reference exists, the agent edits it to the videoform, set beside the original
218+
And the bench runs after every step of this file, on every set, from the kept files alone
219+
And the first proof that matters: on at least five sets, each videoform built leave one out holds its left-out
220+
video's pacing and shot lengths and rejects other sets' videos, more often than chance by a clear margin
221+
222+
@needs:VF12
223+
Scenario: VF13. The person's blind judging, kept as evidence
224+
Given the plan
225+
| crate | file | lines | job |
226+
| raea-app-videoform | apps/videoform/src/judge.rs | 150 | a pair shown unlabelled in a random order; the pick, the scores and the words kept, and as evidence |
227+
Then the person sees the original and raea's edit side by side, unlabelled, in a random order, picks which they'd
228+
post, scores pacing, shot choice, structure, graphics, sound and overall feel from 1 to 5, and says what's wrong
229+
And each verdict is kept in the bench's record and is evidence in the videoform: a pick a pair (FP6b), a note a
230+
label (FP5)
231+
232+
Rule: Deferred
233+
Scenario: Not in this file yet
234+
Then an automatic judge for the test bench (a video-capable language model, paid by the person, in the bench only,
235+
never shipped: fidelity, pairs, and gaps between what the references show and what the videoform holds)
236+
And the order of the test sets

0 commit comments

Comments
 (0)