diff --git a/directory/skills/raea/3d.md b/directory/skills/raea/3d.md index 0ce0aa6..3d899f6 100644 --- a/directory/skills/raea/3d.md +++ b/directory/skills/raea/3d.md @@ -17,6 +17,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s | setting up the camera, camera moves, entities, 2D under and over 3D | [3d/scene.md](3d/scene.md) | | lighting: sun, bulbs, spots, ambient, shadows, brightness | [3d/lights.md](3d/lights.md) | | surfaces: colour, metal, glass, glow, clearcoat, picture maps, toon (cel) shading and ink outlines (`ToonMaterial`), wear (`WearExtension`), decals (`Decal`), a model's materials re-covered (a material inside its `Model`) | [3d/materials.md](3d/materials.md) | +| weathering with a cause: where rain runs off and stains, what stays sheltered and sooty, where hands and feet wear a model, baked once from its own shape as cause maps over a second UV set (`raea wear bake REQUEST.json`; the recipes that draw them come next, until then `WearExtension`) | [3d/materials.md](3d/materials.md) | | a whole scene to build and make believable: brief, routing (which tool makes which object: solid, field, generator, Tripo HD or Smart Mesh, stock), build order, review sheets (`raea render --review`), the judge, stall rules | [3d/scenes.md](3d/scenes.md) | | the words to paste for a whole scene: the brief template, the builder prompt, the fresh-context judge prompt | [3d/prompts.md](3d/prompts.md) | | shapes: primitives, Manifold solids, blended organic bodies (`sdf`), trees, terrain, rocks, plants, lofts (`grow`), Text3D | [3d/meshes.md](3d/meshes.md) | @@ -35,6 +36,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s | liquids and goo: coffee dumped from a mug, a tap's stream, honey, things melting (``, sph or mpm, baked) | [3d/liquids.md](3d/liquids.md) | | water: seas, lakes and pools (Gerstner waves in a `ShaderExtension`, Fresnel, refraction, absorption, caustics, foam), and water under Solari | [3d/water.md](3d/water.md) | | ray-traced light (`SolariLighting`, `Pathtracer`): what it lights and traces, glowing lamps, rooms that don't leak, cuts, metering | [3d/solari.md](3d/solari.md) | +| which light path a scene needs: each scene drawn on raster, traced AO, Solari and GI against its path-traced reference, the distance and frame time of each (`raea light check SCENES.json -o DIR`) | [3d/solari.md](3d/solari.md) | | bloom, fog, tone mapping, exposure, fisheye and lens distortion, motion blur, fire (``), frame looks (pixel, ink, painted illustration ``, cartoon ``) | [3d/post.md](3d/post.md) | | three.js work: an R3F scene in `` (useFrame, drei, postprocessing), a plain three.js page in `` (its own renderer and loop), TSL node materials, three's shaders, and what doesn't draw yet | [3d/r3f.md](3d/r3f.md) | | budgets, render-equals-play rules, what isn't there yet | [3d/limits.md](3d/limits.md) | diff --git a/directory/skills/raea/3d/characters.md b/directory/skills/raea/3d/characters.md index 62ca1e1..ecc1454 100644 --- a/directory/skills/raea/3d/characters.md +++ b/directory/skills/raea/3d/characters.md @@ -98,17 +98,28 @@ its hands, feet, gaze and seat on the scene's real things, said in the scene's c - `raea clothes wear assets/characters/dressed trousers tee --body assets/characters/NAME --head HEAD_DIR` writes a dressed body folder. Register that folder with `registerBody`, and give each garment's glb to `` as one of its `parts`. -- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words on this Mac with - `raea motion`, Kimodo on MLX (no cloud GPU): +- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words with `raea motion`: + Kimodo, run on the machine itself (no cloud GPU). On a Mac it runs on MLX. On Linux and Windows it runs upstream + Kimodo on PyTorch. Both write the same clip file. - `raea motion "A person waves at the camera" --keys stand.json --seconds 4 --seed 2 -o wave.clip.json` takes about - 8 seconds. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or places - on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and ends - where the walks around it can join it. + 8 seconds on a Mac. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or + places on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and + ends where the walks around it can join it. - Each seed is another take, and a take can stand still through its line. `bun sdk/body/tools/makedo.ts SKELETON LIBDIR OUTDIR "wave at the camera" ...` makes a stage's `do` lines with three seeds each, and keeps the take - that passes the body's checks and moves most. - - The first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB). A line's - words are read once and kept. + that passes the body's checks and moves most. A seed is a different take on MLX and on PyTorch, because their + noise is drawn differently. + - On a Mac, the first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB). + - On Linux and Windows, run `raea setup motion` once first. It needs uv (https://docs.astral.sh/uv) on PATH. It + makes a Python environment in raea's models folder with PyTorch and upstream Kimodo, then fetches the weights, + about 26 GB in all. With an NVIDIA GPU the motion runs on CUDA. The text encoder (16 GB in bf16) joins it on a + GPU of 18 GB or more; on a smaller card it runs on the CPU and needs about 18 GB of memory. Without an NVIDIA GPU + everything runs on the CPU, which is slower. `raea setup motion --check` says what is set up and where each part + runs. + - A line's words are read once and kept, so the text encoder loads once a line. + - Licences: Kimodo's code is Apache-2.0 and its weights are under NVIDIA's Open Model License (commercial use + allowed). The text encoder is Meta Llama 3 8B Instruct (Meta Llama 3 Community License) with LLM2Vec's adapters + (MIT). ```tsx const perf = perform(lib, { at: [0.4, 0, 2], facing: Math.PI / 2, frames: 450, steps: [ diff --git a/directory/skills/raea/motion/compositions.md b/directory/skills/raea/motion/compositions.md index 3ce2824..8fc8942 100644 --- a/directory/skills/raea/motion/compositions.md +++ b/directory/skills/raea/motion/compositions.md @@ -8,6 +8,7 @@ - `id` uses letters, digits and hyphens only (no spaces, no underscores) and is unique in the root. - `width`, `height` and `durationInFrames` are positive whole numbers, and `fps` is a positive number. - `defaultProps` (optional) is a plain object of JSON values, which the component receives as its props. + - `controls` (optional) says which of those props a person may tune: see "Controls" below. - Size the composition from the shape the brief asks for: 1080x1920 (9:16), 1080x1350 (4:5), 1080x1080 (1:1) or 1920x1080 (16:9). Use 30 fps unless the footage or brief says otherwise. Set the length as `Math.round(seconds * fps)`. @@ -22,6 +23,36 @@ 7. the main component, which places the clips, parts and sounds in `Sequence`s; 8. `Root` and `registerRoot(Root)`. +## Controls: anything a person might tune + +Anything a person might tune goes in `controls`: a colour, a warmth, a speed, a size, a title's words, which model +or take is used, whether it rains. Make it a prop with its value in `defaultProps`, read it in the component (as a +prop, or with `useControl("name")` in a part deep in the tree), and declare it in `controls` by the same key. The +feed's Controls panel lists them beside the timeline, and moving one changes the live picture at once, then the +render around it; a value written in the middle of the code is one the person can only ask you to change. + +- Each control names a `defaultProps` key, and its `kind` fits that value: `number` (with `min`, `max`, `step` and + `unit` when it has a range), `switch` (true or false), `choice` (one of `options`), `colour` (a hex string, + `"#rgb"` to `"#rrggbbaa"`) or `text` (`maxLength`, `multiline`). The registry refuses one that doesn't fit, naming + the control. +- `group` puts related controls together ("Grade", "Camera", "Words"); `label` and `about` are what the panel shows. +- `cost` says how a change reaches the picture. `live` (the default) is drawn at once, for values the running world + takes frame by frame: a place, a turn, a size, a colour, roughness and metalness, a glow, opacity, a light's + strength, the camera, any 2D style or text. `reopen` is for what a 3D world keeps from its first frame: a shape's + geometry, a model's file, an instance count, a picture map, a shader. `resim` is for state carried from frame to + frame: physics, particles, a game's fold. + +```tsx + +``` + ## Why The renderer finds compositions by walking the root's element tree without rendering it. It never calls a @@ -40,6 +71,10 @@ anyway, because a guessed size or length is how a 16:9 brief comes out vertical. - Calling `registerRoot` twice, or once per composition. Register one root that returns several ``s. - A `Root` that calls `useCurrentFrame()` or `useVideoConfig()`. Hooks belong in the composition's component. - `defaultProps` holding a function, a Date or a class instance. Only JSON values are allowed. +- A tunable value as a literal in a part (`const WARMTH = 0.2`, `color="#d04a3a"` in a mesh): the person can't move + it. Make it a prop in `defaultProps` and a control. +- A control for a key `defaultProps` doesn't have, or a `number` control whose value is outside its `min` and `max`: + registering fails, naming the control. - Writing a second file for "the parts" and importing it. That works (modules resolve from the importing file), but one self-contained file is easier for the next pass to read and is how the approved pieces are written. @@ -67,11 +102,14 @@ export default function EndCard({ line }: { line: string }) { ); } +/** What a person may tune: the line's words. */ +const controls = { line: { kind: "text", maxLength: 32, group: "Words" } } as const; + function Root() { return ( <> - - + + ); } diff --git a/directory/skills/raea/reference/cli.md b/directory/skills/raea/reference/cli.md index f9767c3..d956f08 100644 --- a/directory/skills/raea/reference/cli.md +++ b/directory/skills/raea/reference/cli.md @@ -142,7 +142,7 @@ Flags: `--body` (1), `--colour` (1), `--hard`, `--head` (1), `--over` (1), `--pa Access: HEADLESS. Arguments besides flags: 1. ``` -raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this Mac by Kimodo (NVIDIA, on MLX): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads; the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters); a line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT) +raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this machine by Kimodo (NVIDIA): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads. On a Mac it runs on MLX (Apple silicon's GPU): the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters). On Linux and Windows it runs upstream Kimodo on PyTorch, set up once by raea setup motion: on an NVIDIA GPU with CUDA (the text encoder, 16 GB in bf16, on the CPU unless the card has 18 GB or more), else on the CPU alone; reading a new line on the CPU takes about 18 GB of memory. A line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT, Kimodo's code Apache-2.0) ``` Flags: `--frames` (1), `--keys` (1), `--seconds` (1), `--seed` (1), `--steps` (1), `-o` (1). @@ -271,6 +271,7 @@ Access: HEADLESS. Arguments besides flags: 1. ``` raea setup blender [--check] [--download] (find Blender or download the LTS into ~/.raea/tools/blender, and install raea's add-on and the MCP add-on) +raea setup motion [--check] (what raea motion runs on, made where missing (--check only says): on a Mac the MLX models in raea's models folder (the motion model 1.1 GB, the text encoder 8.8 GB); on Linux and Windows, in raea's models folder's kimodo-torch, a Python 3.11 environment made by uv (which it needs on PATH: https://docs.astral.sh/uv) with PyTorch (its CUDA build where there is an NVIDIA driver, about 8 GB, else CPU only, about 2 GB) and upstream Kimodo (nv-tlabs/kimodo at e1c219c, Apache-2.0), then the weights: Kimodo-SOMA-RP-v1 (1.1 GB, NVIDIA Open Model License), LLM2Vec's two adapters (0.3 GB, MIT) and Meta Llama 3 8B Instruct in bf16 (16 GB, Meta Llama 3 Community License, from NousResearch's ungated copy, checked file by file against Meta's SHA-256); about 26 GB in all. It says what this machine has and where each network runs: the denoiser on CUDA when there is an NVIDIA GPU, else the CPU; the text encoder beside it on a GPU of 18 GB or more, else the CPU) raea setup godot [--check] (download Godot 4.6.1 with GodotJS, its TypeScript bindings (MIT), into ~/.raea/tools/godot: JavaScriptCore on the Mac, QuickJS on Windows and Linux; raea's own build (the Mac's notarized), else GodotJS's, each checked against its pinned sha256) ``` @@ -496,6 +497,16 @@ raea physics bake REQUEST.json [-o OUT.json] | physics goo SPEC.json | physics p Flags: `-o` (1). +## raea wear + +Access: HEADLESS. Arguments besides flags: 2. + +``` +raea wear bake REQUEST.json [-o OUT.json] (a .glb's weathering causes for raea/wear: the model with a second UV set and its cause maps, made once in the asset store) +``` + +Flags: `-o` (1). + ## raea splat Access: HEADLESS. Arguments besides flags: 2. @@ -535,6 +546,16 @@ raea neural mlxdlss DIR -o OUT [--model M.dlssmodel] [--intensity X] [--profile Flags: `--intensity` (1), `--local-structure` (1), `--model` (1), `--profile` (1), `--skin-structure` (1), `--steady` (1), `-o` (1). +## raea light + +Access: HEADLESS. Arguments besides flags: 2. + +``` +raea light check SCENES.json -o DIR [--scene NAME] [--paths raster,traced-ao,solari,gi] [--spp N] [--fresh] (each scene on each light path against its path-traced reference: distance in stops, the worst tile, a moving shot's flicker, a frame's cost; DIR/report.json) +``` + +Flags: `--fresh`, `--paths` (1), `--scene` (1), `--spp` (1), `-o` (1). + ## raea mezz Access: HEADLESS. Arguments besides flags: any number. diff --git a/directory/skills/raea/reference/sdk.md b/directory/skills/raea/reference/sdk.md index 0f8c363..2261856 100644 --- a/directory/skills/raea/reference/sdk.md +++ b/directory/skills/raea/reference/sdk.md @@ -7,7 +7,7 @@ with a capital, hooks with `use`. A component's props are in its file. ## `raea` (sdk/raea/src/index.tsx) -`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word` +`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useControl`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word` ## `raea/3d` (sdk/raea/src/3d.tsx) @@ -19,7 +19,7 @@ with a capital, hooks with `use`. A component's props are in its file. ## `raea/app` (sdk/raea/src/app.ts) -`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot` +`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot`, `useControl`, `validControls` ## `raea/asset` (sdk/raea/src/asset.ts) diff --git a/skills/raea/3d.md b/skills/raea/3d.md index 0ce0aa6..3d899f6 100644 --- a/skills/raea/3d.md +++ b/skills/raea/3d.md @@ -17,6 +17,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s | setting up the camera, camera moves, entities, 2D under and over 3D | [3d/scene.md](3d/scene.md) | | lighting: sun, bulbs, spots, ambient, shadows, brightness | [3d/lights.md](3d/lights.md) | | surfaces: colour, metal, glass, glow, clearcoat, picture maps, toon (cel) shading and ink outlines (`ToonMaterial`), wear (`WearExtension`), decals (`Decal`), a model's materials re-covered (a material inside its `Model`) | [3d/materials.md](3d/materials.md) | +| weathering with a cause: where rain runs off and stains, what stays sheltered and sooty, where hands and feet wear a model, baked once from its own shape as cause maps over a second UV set (`raea wear bake REQUEST.json`; the recipes that draw them come next, until then `WearExtension`) | [3d/materials.md](3d/materials.md) | | a whole scene to build and make believable: brief, routing (which tool makes which object: solid, field, generator, Tripo HD or Smart Mesh, stock), build order, review sheets (`raea render --review`), the judge, stall rules | [3d/scenes.md](3d/scenes.md) | | the words to paste for a whole scene: the brief template, the builder prompt, the fresh-context judge prompt | [3d/prompts.md](3d/prompts.md) | | shapes: primitives, Manifold solids, blended organic bodies (`sdf`), trees, terrain, rocks, plants, lofts (`grow`), Text3D | [3d/meshes.md](3d/meshes.md) | @@ -35,6 +36,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s | liquids and goo: coffee dumped from a mug, a tap's stream, honey, things melting (``, sph or mpm, baked) | [3d/liquids.md](3d/liquids.md) | | water: seas, lakes and pools (Gerstner waves in a `ShaderExtension`, Fresnel, refraction, absorption, caustics, foam), and water under Solari | [3d/water.md](3d/water.md) | | ray-traced light (`SolariLighting`, `Pathtracer`): what it lights and traces, glowing lamps, rooms that don't leak, cuts, metering | [3d/solari.md](3d/solari.md) | +| which light path a scene needs: each scene drawn on raster, traced AO, Solari and GI against its path-traced reference, the distance and frame time of each (`raea light check SCENES.json -o DIR`) | [3d/solari.md](3d/solari.md) | | bloom, fog, tone mapping, exposure, fisheye and lens distortion, motion blur, fire (``), frame looks (pixel, ink, painted illustration ``, cartoon ``) | [3d/post.md](3d/post.md) | | three.js work: an R3F scene in `` (useFrame, drei, postprocessing), a plain three.js page in `` (its own renderer and loop), TSL node materials, three's shaders, and what doesn't draw yet | [3d/r3f.md](3d/r3f.md) | | budgets, render-equals-play rules, what isn't there yet | [3d/limits.md](3d/limits.md) | diff --git a/skills/raea/3d/characters.md b/skills/raea/3d/characters.md index 62ca1e1..ecc1454 100644 --- a/skills/raea/3d/characters.md +++ b/skills/raea/3d/characters.md @@ -98,17 +98,28 @@ its hands, feet, gaze and seat on the scene's real things, said in the scene's c - `raea clothes wear assets/characters/dressed trousers tee --body assets/characters/NAME --head HEAD_DIR` writes a dressed body folder. Register that folder with `registerBody`, and give each garment's glb to `` as one of its `parts`. -- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words on this Mac with - `raea motion`, Kimodo on MLX (no cloud GPU): +- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words with `raea motion`: + Kimodo, run on the machine itself (no cloud GPU). On a Mac it runs on MLX. On Linux and Windows it runs upstream + Kimodo on PyTorch. Both write the same clip file. - `raea motion "A person waves at the camera" --keys stand.json --seconds 4 --seed 2 -o wave.clip.json` takes about - 8 seconds. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or places - on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and ends - where the walks around it can join it. + 8 seconds on a Mac. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or + places on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and + ends where the walks around it can join it. - Each seed is another take, and a take can stand still through its line. `bun sdk/body/tools/makedo.ts SKELETON LIBDIR OUTDIR "wave at the camera" ...` makes a stage's `do` lines with three seeds each, and keeps the take - that passes the body's checks and moves most. - - The first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB). A line's - words are read once and kept. + that passes the body's checks and moves most. A seed is a different take on MLX and on PyTorch, because their + noise is drawn differently. + - On a Mac, the first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB). + - On Linux and Windows, run `raea setup motion` once first. It needs uv (https://docs.astral.sh/uv) on PATH. It + makes a Python environment in raea's models folder with PyTorch and upstream Kimodo, then fetches the weights, + about 26 GB in all. With an NVIDIA GPU the motion runs on CUDA. The text encoder (16 GB in bf16) joins it on a + GPU of 18 GB or more; on a smaller card it runs on the CPU and needs about 18 GB of memory. Without an NVIDIA GPU + everything runs on the CPU, which is slower. `raea setup motion --check` says what is set up and where each part + runs. + - A line's words are read once and kept, so the text encoder loads once a line. + - Licences: Kimodo's code is Apache-2.0 and its weights are under NVIDIA's Open Model License (commercial use + allowed). The text encoder is Meta Llama 3 8B Instruct (Meta Llama 3 Community License) with LLM2Vec's adapters + (MIT). ```tsx const perf = perform(lib, { at: [0.4, 0, 2], facing: Math.PI / 2, frames: 450, steps: [ diff --git a/skills/raea/motion/compositions.md b/skills/raea/motion/compositions.md index 3ce2824..8fc8942 100644 --- a/skills/raea/motion/compositions.md +++ b/skills/raea/motion/compositions.md @@ -8,6 +8,7 @@ - `id` uses letters, digits and hyphens only (no spaces, no underscores) and is unique in the root. - `width`, `height` and `durationInFrames` are positive whole numbers, and `fps` is a positive number. - `defaultProps` (optional) is a plain object of JSON values, which the component receives as its props. + - `controls` (optional) says which of those props a person may tune: see "Controls" below. - Size the composition from the shape the brief asks for: 1080x1920 (9:16), 1080x1350 (4:5), 1080x1080 (1:1) or 1920x1080 (16:9). Use 30 fps unless the footage or brief says otherwise. Set the length as `Math.round(seconds * fps)`. @@ -22,6 +23,36 @@ 7. the main component, which places the clips, parts and sounds in `Sequence`s; 8. `Root` and `registerRoot(Root)`. +## Controls: anything a person might tune + +Anything a person might tune goes in `controls`: a colour, a warmth, a speed, a size, a title's words, which model +or take is used, whether it rains. Make it a prop with its value in `defaultProps`, read it in the component (as a +prop, or with `useControl("name")` in a part deep in the tree), and declare it in `controls` by the same key. The +feed's Controls panel lists them beside the timeline, and moving one changes the live picture at once, then the +render around it; a value written in the middle of the code is one the person can only ask you to change. + +- Each control names a `defaultProps` key, and its `kind` fits that value: `number` (with `min`, `max`, `step` and + `unit` when it has a range), `switch` (true or false), `choice` (one of `options`), `colour` (a hex string, + `"#rgb"` to `"#rrggbbaa"`) or `text` (`maxLength`, `multiline`). The registry refuses one that doesn't fit, naming + the control. +- `group` puts related controls together ("Grade", "Camera", "Words"); `label` and `about` are what the panel shows. +- `cost` says how a change reaches the picture. `live` (the default) is drawn at once, for values the running world + takes frame by frame: a place, a turn, a size, a colour, roughness and metalness, a glow, opacity, a light's + strength, the camera, any 2D style or text. `reopen` is for what a 3D world keeps from its first frame: a shape's + geometry, a model's file, an instance count, a picture map, a shader. `resim` is for state carried from frame to + frame: physics, particles, a game's fold. + +```tsx + +``` + ## Why The renderer finds compositions by walking the root's element tree without rendering it. It never calls a @@ -40,6 +71,10 @@ anyway, because a guessed size or length is how a 16:9 brief comes out vertical. - Calling `registerRoot` twice, or once per composition. Register one root that returns several ``s. - A `Root` that calls `useCurrentFrame()` or `useVideoConfig()`. Hooks belong in the composition's component. - `defaultProps` holding a function, a Date or a class instance. Only JSON values are allowed. +- A tunable value as a literal in a part (`const WARMTH = 0.2`, `color="#d04a3a"` in a mesh): the person can't move + it. Make it a prop in `defaultProps` and a control. +- A control for a key `defaultProps` doesn't have, or a `number` control whose value is outside its `min` and `max`: + registering fails, naming the control. - Writing a second file for "the parts" and importing it. That works (modules resolve from the importing file), but one self-contained file is easier for the next pass to read and is how the approved pieces are written. @@ -67,11 +102,14 @@ export default function EndCard({ line }: { line: string }) { ); } +/** What a person may tune: the line's words. */ +const controls = { line: { kind: "text", maxLength: 32, group: "Words" } } as const; + function Root() { return ( <> - - + + ); } diff --git a/skills/raea/reference/cli.md b/skills/raea/reference/cli.md index f9767c3..d956f08 100644 --- a/skills/raea/reference/cli.md +++ b/skills/raea/reference/cli.md @@ -142,7 +142,7 @@ Flags: `--body` (1), `--colour` (1), `--hard`, `--head` (1), `--over` (1), `--pa Access: HEADLESS. Arguments besides flags: 1. ``` -raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this Mac by Kimodo (NVIDIA, on MLX): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads; the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters); a line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT) +raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this machine by Kimodo (NVIDIA): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads. On a Mac it runs on MLX (Apple silicon's GPU): the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters). On Linux and Windows it runs upstream Kimodo on PyTorch, set up once by raea setup motion: on an NVIDIA GPU with CUDA (the text encoder, 16 GB in bf16, on the CPU unless the card has 18 GB or more), else on the CPU alone; reading a new line on the CPU takes about 18 GB of memory. A line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT, Kimodo's code Apache-2.0) ``` Flags: `--frames` (1), `--keys` (1), `--seconds` (1), `--seed` (1), `--steps` (1), `-o` (1). @@ -271,6 +271,7 @@ Access: HEADLESS. Arguments besides flags: 1. ``` raea setup blender [--check] [--download] (find Blender or download the LTS into ~/.raea/tools/blender, and install raea's add-on and the MCP add-on) +raea setup motion [--check] (what raea motion runs on, made where missing (--check only says): on a Mac the MLX models in raea's models folder (the motion model 1.1 GB, the text encoder 8.8 GB); on Linux and Windows, in raea's models folder's kimodo-torch, a Python 3.11 environment made by uv (which it needs on PATH: https://docs.astral.sh/uv) with PyTorch (its CUDA build where there is an NVIDIA driver, about 8 GB, else CPU only, about 2 GB) and upstream Kimodo (nv-tlabs/kimodo at e1c219c, Apache-2.0), then the weights: Kimodo-SOMA-RP-v1 (1.1 GB, NVIDIA Open Model License), LLM2Vec's two adapters (0.3 GB, MIT) and Meta Llama 3 8B Instruct in bf16 (16 GB, Meta Llama 3 Community License, from NousResearch's ungated copy, checked file by file against Meta's SHA-256); about 26 GB in all. It says what this machine has and where each network runs: the denoiser on CUDA when there is an NVIDIA GPU, else the CPU; the text encoder beside it on a GPU of 18 GB or more, else the CPU) raea setup godot [--check] (download Godot 4.6.1 with GodotJS, its TypeScript bindings (MIT), into ~/.raea/tools/godot: JavaScriptCore on the Mac, QuickJS on Windows and Linux; raea's own build (the Mac's notarized), else GodotJS's, each checked against its pinned sha256) ``` @@ -496,6 +497,16 @@ raea physics bake REQUEST.json [-o OUT.json] | physics goo SPEC.json | physics p Flags: `-o` (1). +## raea wear + +Access: HEADLESS. Arguments besides flags: 2. + +``` +raea wear bake REQUEST.json [-o OUT.json] (a .glb's weathering causes for raea/wear: the model with a second UV set and its cause maps, made once in the asset store) +``` + +Flags: `-o` (1). + ## raea splat Access: HEADLESS. Arguments besides flags: 2. @@ -535,6 +546,16 @@ raea neural mlxdlss DIR -o OUT [--model M.dlssmodel] [--intensity X] [--profile Flags: `--intensity` (1), `--local-structure` (1), `--model` (1), `--profile` (1), `--skin-structure` (1), `--steady` (1), `-o` (1). +## raea light + +Access: HEADLESS. Arguments besides flags: 2. + +``` +raea light check SCENES.json -o DIR [--scene NAME] [--paths raster,traced-ao,solari,gi] [--spp N] [--fresh] (each scene on each light path against its path-traced reference: distance in stops, the worst tile, a moving shot's flicker, a frame's cost; DIR/report.json) +``` + +Flags: `--fresh`, `--paths` (1), `--scene` (1), `--spp` (1), `-o` (1). + ## raea mezz Access: HEADLESS. Arguments besides flags: any number. diff --git a/skills/raea/reference/sdk.md b/skills/raea/reference/sdk.md index 0f8c363..2261856 100644 --- a/skills/raea/reference/sdk.md +++ b/skills/raea/reference/sdk.md @@ -7,7 +7,7 @@ with a capital, hooks with `use`. A component's props are in its file. ## `raea` (sdk/raea/src/index.tsx) -`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word` +`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useControl`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word` ## `raea/3d` (sdk/raea/src/3d.tsx) @@ -19,7 +19,7 @@ with a capital, hooks with `use`. A component's props are in its file. ## `raea/app` (sdk/raea/src/app.ts) -`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot` +`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot`, `useControl`, `validControls` ## `raea/asset` (sdk/raea/src/asset.ts)