Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions directory/skills/raea/3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s
| setting up the camera, camera moves, entities, 2D under and over 3D | [3d/scene.md](3d/scene.md) |
| lighting: sun, bulbs, spots, ambient, shadows, brightness | [3d/lights.md](3d/lights.md) |
| surfaces: colour, metal, glass, glow, clearcoat, picture maps, toon (cel) shading and ink outlines (`ToonMaterial`), wear (`WearExtension`), decals (`Decal`), a model's materials re-covered (a material inside its `Model`) | [3d/materials.md](3d/materials.md) |
| weathering with a cause: where rain runs off and stains, what stays sheltered and sooty, where hands and feet wear a model, baked once from its own shape as cause maps over a second UV set (`raea wear bake REQUEST.json`; the recipes that draw them come next, until then `WearExtension`) | [3d/materials.md](3d/materials.md) |
| a whole scene to build and make believable: brief, routing (which tool makes which object: solid, field, generator, Tripo HD or Smart Mesh, stock), build order, review sheets (`raea render --review`), the judge, stall rules | [3d/scenes.md](3d/scenes.md) |
| the words to paste for a whole scene: the brief template, the builder prompt, the fresh-context judge prompt | [3d/prompts.md](3d/prompts.md) |
| shapes: primitives, Manifold solids, blended organic bodies (`sdf`), trees, terrain, rocks, plants, lofts (`grow`), Text3D | [3d/meshes.md](3d/meshes.md) |
Expand All @@ -35,6 +36,7 @@ the topic files in `3d/` are copies of raea's craft guides, made by its `tools/s
| liquids and goo: coffee dumped from a mug, a tap's stream, honey, things melting (`<Goo>`, sph or mpm, baked) | [3d/liquids.md](3d/liquids.md) |
| water: seas, lakes and pools (Gerstner waves in a `ShaderExtension`, Fresnel, refraction, absorption, caustics, foam), and water under Solari | [3d/water.md](3d/water.md) |
| ray-traced light (`SolariLighting`, `Pathtracer`): what it lights and traces, glowing lamps, rooms that don't leak, cuts, metering | [3d/solari.md](3d/solari.md) |
| which light path a scene needs: each scene drawn on raster, traced AO, Solari and GI against its path-traced reference, the distance and frame time of each (`raea light check SCENES.json -o DIR`) | [3d/solari.md](3d/solari.md) |
| bloom, fog, tone mapping, exposure, fisheye and lens distortion, motion blur, fire (`<Fire>`), frame looks (pixel, ink, painted illustration `<Look mode="paint">`, cartoon `<Look mode="cartoon">`) | [3d/post.md](3d/post.md) |
| three.js work: an R3F scene in `<R3F>` (useFrame, drei, postprocessing), a plain three.js page in `<Three program>` (its own renderer and loop), TSL node materials, three's shaders, and what doesn't draw yet | [3d/r3f.md](3d/r3f.md) |
| budgets, render-equals-play rules, what isn't there yet | [3d/limits.md](3d/limits.md) |
Expand Down
27 changes: 19 additions & 8 deletions directory/skills/raea/3d/characters.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,17 +98,28 @@ its hands, feet, gaze and seat on the scene's real things, said in the scene's c
- `raea clothes wear assets/characters/dressed trousers tee --body assets/characters/NAME --head HEAD_DIR` writes
a dressed body folder. Register that folder with `registerBody`, and give each garment's glb to `<Character>`
as one of its `parts`.
- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words on this Mac with
`raea motion`, Kimodo on MLX (no cloud GPU):
- An act no capture holds (wave at the camera, shrug, look at a watch) is made from its words with `raea motion`:
Kimodo, run on the machine itself (no cloud GPU). On a Mac it runs on MLX. On Linux and Windows it runs upstream
Kimodo on PyTorch. Both write the same clip file.
- `raea motion "A person waves at the camera" --keys stand.json --seconds 4 --seed 2 -o wave.clip.json` takes about
8 seconds. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or places
on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and ends
where the walks around it can join it.
8 seconds on a Mac. Say who moves ("A person ..."), as Kimodo's captions do. `--keys` holds the body to poses or
places on given frames: `bun sdk/body/tools/between.ts` writes standing keys at both ends, so the act starts and
ends where the walks around it can join it.
- Each seed is another take, and a take can stand still through its line. `bun sdk/body/tools/makedo.ts SKELETON
LIBDIR OUTDIR "wave at the camera" ...` makes a stage's `do` lines with three seeds each, and keeps the take
that passes the body's checks and moves most.
- The first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB). A line's
words are read once and kept.
that passes the body's checks and moves most. A seed is a different take on MLX and on PyTorch, because their
noise is drawn differently.
- On a Mac, the first use fetches the motion model (1.1 GB), and the first new line the text encoder (8.8 GB).
- On Linux and Windows, run `raea setup motion` once first. It needs uv (https://docs.astral.sh/uv) on PATH. It
makes a Python environment in raea's models folder with PyTorch and upstream Kimodo, then fetches the weights,
about 26 GB in all. With an NVIDIA GPU the motion runs on CUDA. The text encoder (16 GB in bf16) joins it on a
GPU of 18 GB or more; on a smaller card it runs on the CPU and needs about 18 GB of memory. Without an NVIDIA GPU
everything runs on the CPU, which is slower. `raea setup motion --check` says what is set up and where each part
runs.
- A line's words are read once and kept, so the text encoder loads once a line.
- Licences: Kimodo's code is Apache-2.0 and its weights are under NVIDIA's Open Model License (commercial use
allowed). The text encoder is Meta Llama 3 8B Instruct (Meta Llama 3 Community License) with LLM2Vec's adapters
(MIT).

```tsx
const perf = perform(lib, { at: [0.4, 0, 2], facing: Math.PI / 2, frames: 450, steps: [
Expand Down
42 changes: 40 additions & 2 deletions directory/skills/raea/motion/compositions.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- `id` uses letters, digits and hyphens only (no spaces, no underscores) and is unique in the root.
- `width`, `height` and `durationInFrames` are positive whole numbers, and `fps` is a positive number.
- `defaultProps` (optional) is a plain object of JSON values, which the component receives as its props.
- `controls` (optional) says which of those props a person may tune: see "Controls" below.
- Size the composition from the shape the brief asks for: 1080x1920 (9:16), 1080x1350 (4:5), 1080x1080 (1:1) or
1920x1080 (16:9). Use 30 fps unless the footage or brief says otherwise. Set the length as
`Math.round(seconds * fps)`.
Expand All @@ -22,6 +23,36 @@
7. the main component, which places the clips, parts and sounds in `Sequence`s;
8. `Root` and `registerRoot(Root)`.

## Controls: anything a person might tune

Anything a person might tune goes in `controls`: a colour, a warmth, a speed, a size, a title's words, which model
or take is used, whether it rains. Make it a prop with its value in `defaultProps`, read it in the component (as a
prop, or with `useControl("name")` in a part deep in the tree), and declare it in `controls` by the same key. The
feed's Controls panel lists them beside the timeline, and moving one changes the live picture at once, then the
render around it; a value written in the middle of the code is one the person can only ask you to change.

- Each control names a `defaultProps` key, and its `kind` fits that value: `number` (with `min`, `max`, `step` and
`unit` when it has a range), `switch` (true or false), `choice` (one of `options`), `colour` (a hex string,
`"#rgb"` to `"#rrggbbaa"`) or `text` (`maxLength`, `multiline`). The registry refuses one that doesn't fit, naming
the control.
- `group` puts related controls together ("Grade", "Camera", "Words"); `label` and `about` are what the panel shows.
- `cost` says how a change reaches the picture. `live` (the default) is drawn at once, for values the running world
takes frame by frame: a place, a turn, a size, a colour, roughness and metalness, a glow, opacity, a light's
strength, the camera, any 2D style or text. `reopen` is for what a 3D world keeps from its first frame: a shape's
geometry, a model's file, an instance count, a picture map, a shader. `resim` is for state carried from frame to
frame: physics, particles, a game's fold.

```tsx
<Composition id="mayor" component={Film} width={1920} height={1080} fps={30} durationInFrames={2700}
controls={{
warmth: { kind: "number", min: -1, max: 1, step: 0.05, group: "Grade" },
model: { kind: "choice", options: ["goose", "mayor"], cost: "reopen" },
rain: { kind: "switch", cost: "resim" },
title: { kind: "text", maxLength: 40, group: "Words" },
}}
defaultProps={{ warmth: 0.2, model: "goose", rain: false, title: "Swamp-Side Mayor" }} />
```

## Why

The renderer finds compositions by walking the root's element tree without rendering it. It never calls a
Expand All @@ -40,6 +71,10 @@ anyway, because a guessed size or length is how a 16:9 brief comes out vertical.
- Calling `registerRoot` twice, or once per composition. Register one root that returns several `<Composition>`s.
- A `Root` that calls `useCurrentFrame()` or `useVideoConfig()`. Hooks belong in the composition's component.
- `defaultProps` holding a function, a Date or a class instance. Only JSON values are allowed.
- A tunable value as a literal in a part (`const WARMTH = 0.2`, `color="#d04a3a"` in a mesh): the person can't move
it. Make it a prop in `defaultProps` and a control.
- A control for a key `defaultProps` doesn't have, or a `number` control whose value is outside its `min` and `max`:
registering fails, naming the control.
- Writing a second file for "the parts" and importing it. That works (modules resolve from the importing file), but
one self-contained file is easier for the next pass to read and is how the approved pieces are written.

Expand Down Expand Up @@ -67,11 +102,14 @@ export default function EndCard({ line }: { line: string }) {
);
}

/** What a person may tune: the line's words. */
const controls = { line: { kind: "text", maxLength: 32, group: "Words" } } as const;

function Root() {
return (
<>
<Composition id="end-9x16" component={EndCard} width={1080} height={1920} fps={30} durationInFrames={90} defaultProps={{ line: "Try it free" }} />
<Composition id="end-16x9" component={EndCard} width={1920} height={1080} fps={30} durationInFrames={90} defaultProps={{ line: "Try it free" }} />
<Composition id="end-9x16" component={EndCard} width={1080} height={1920} fps={30} durationInFrames={90} defaultProps={{ line: "Try it free" }} controls={controls} />
<Composition id="end-16x9" component={EndCard} width={1920} height={1080} fps={30} durationInFrames={90} defaultProps={{ line: "Try it free" }} controls={controls} />
</>
);
}
Expand Down
23 changes: 22 additions & 1 deletion directory/skills/raea/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ Flags: `--body` (1), `--colour` (1), `--hard`, `--head` (1), `--over` (1), `--pa
Access: HEADLESS. Arguments besides flags: 1.

```
raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this Mac by Kimodo (NVIDIA, on MLX): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads; the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters); a line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT)
raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this machine by Kimodo (NVIDIA): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads. On a Mac it runs on MLX (Apple silicon's GPU): the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters). On Linux and Windows it runs upstream Kimodo on PyTorch, set up once by raea setup motion: on an NVIDIA GPU with CUDA (the text encoder, 16 GB in bf16, on the CPU unless the card has 18 GB or more), else on the CPU alone; reading a new line on the CPU takes about 18 GB of memory. A line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT, Kimodo's code Apache-2.0)
```

Flags: `--frames` (1), `--keys` (1), `--seconds` (1), `--seed` (1), `--steps` (1), `-o` (1).
Expand Down Expand Up @@ -271,6 +271,7 @@ Access: HEADLESS. Arguments besides flags: 1.

```
raea setup blender [--check] [--download] (find Blender or download the LTS into ~/.raea/tools/blender, and install raea's add-on and the MCP add-on)
raea setup motion [--check] (what raea motion runs on, made where missing (--check only says): on a Mac the MLX models in raea's models folder (the motion model 1.1 GB, the text encoder 8.8 GB); on Linux and Windows, in raea's models folder's kimodo-torch, a Python 3.11 environment made by uv (which it needs on PATH: https://docs.astral.sh/uv) with PyTorch (its CUDA build where there is an NVIDIA driver, about 8 GB, else CPU only, about 2 GB) and upstream Kimodo (nv-tlabs/kimodo at e1c219c, Apache-2.0), then the weights: Kimodo-SOMA-RP-v1 (1.1 GB, NVIDIA Open Model License), LLM2Vec's two adapters (0.3 GB, MIT) and Meta Llama 3 8B Instruct in bf16 (16 GB, Meta Llama 3 Community License, from NousResearch's ungated copy, checked file by file against Meta's SHA-256); about 26 GB in all. It says what this machine has and where each network runs: the denoiser on CUDA when there is an NVIDIA GPU, else the CPU; the text encoder beside it on a GPU of 18 GB or more, else the CPU)
raea setup godot [--check] (download Godot 4.6.1 with GodotJS, its TypeScript bindings (MIT), into ~/.raea/tools/godot: JavaScriptCore on the Mac, QuickJS on Windows and Linux; raea's own build (the Mac's notarized), else GodotJS's, each checked against its pinned sha256)
```

Expand Down Expand Up @@ -496,6 +497,16 @@ raea physics bake REQUEST.json [-o OUT.json] | physics goo SPEC.json | physics p

Flags: `-o` (1).

## raea wear

Access: HEADLESS. Arguments besides flags: 2.

```
raea wear bake REQUEST.json [-o OUT.json] (a .glb's weathering causes for raea/wear: the model with a second UV set and its cause maps, made once in the asset store)
```

Flags: `-o` (1).

## raea splat

Access: HEADLESS. Arguments besides flags: 2.
Expand Down Expand Up @@ -535,6 +546,16 @@ raea neural mlxdlss DIR -o OUT [--model M.dlssmodel] [--intensity X] [--profile

Flags: `--intensity` (1), `--local-structure` (1), `--model` (1), `--profile` (1), `--skin-structure` (1), `--steady` (1), `-o` (1).

## raea light

Access: HEADLESS. Arguments besides flags: 2.

```
raea light check SCENES.json -o DIR [--scene NAME] [--paths raster,traced-ao,solari,gi] [--spp N] [--fresh] (each scene on each light path against its path-traced reference: distance in stops, the worst tile, a moving shot's flicker, a frame's cost; DIR/report.json)
```

Flags: `--fresh`, `--paths` (1), `--scene` (1), `--spp` (1), `-o` (1).

## raea mezz

Access: HEADLESS. Arguments besides flags: any number.
Expand Down
4 changes: 2 additions & 2 deletions directory/skills/raea/reference/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ with a capital, hooks with `use`. A component's props are in its file.

## `raea` (sdk/raea/src/index.tsx)

`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word`
`ACTION_KEY`, `AbsoluteFill`, `Animate`, `Audio`, `Clock`, `Composition`, `EFFECTS`, `EFFECT_FNS`, `Easing`, `Effect`, `FEED_KEY`, `Freeze`, `HOLD_TIMEOUT_MS`, `Img`, `InputRegion`, `Loop`, `MEDIA_ROOT`, `Mix`, `Nudge`, `Overlay`, `Region`, `SPACE_KEY`, `Sequence`, `Series`, `SongPlot`, `Stagger`, `Transition`, `Video`, `actionsIn`, `applyActions`, `cancelRender`, `continueRender`, `curve`, `curveFn`, `delay`, `delayRender`, `effectAt`, `effectFilter`, `effectPartsError`, `fade`, `feedIn`, `fill`, `findPhrase`, `firstDecider`, `fish`, `getCompositions`, `gridDelay`, `inState`, `interpolate`, `isGeneratedAsset`, `keyEdges`, `lyria`, `machine`, `measureSpring`, `move`, `moves`, `overlays`, `parallel`, `partsFilter`, `pop`, `proofBoxes`, `propsFingerprint`, `pushIn`, `regionBand`, `registerRoot`, `replayDecider`, `reverse`, `rise`, `sampleAnimation`, `sao`, `sequence`, `sequenceGap`, `slam`, `slide`, `song`, `songAt`, `speedLength`, `speedPoints`, `speedSource`, `spring`, `stagger`, `stateOf`, `staticFile`, `stepAll`, `strike`, `tick`, `tickClocks`, `timeline`, `toMove`, `track`, `transitionOf`, `useControl`, `useCurrentFrame`, `useGameEvents`, `useGameState`, `useInput`, `useRegion`, `useSong`, `useVideoConfig`, `useWords`, `wipe`, `word`

## `raea/3d` (sdk/raea/src/3d.tsx)

Expand All @@ -19,7 +19,7 @@ with a capital, hooks with `use`. A component's props are in its file.

## `raea/app` (sdk/raea/src/app.ts)

`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot`
`App`, `BevyApp`, `Composition`, `First`, `FixedFirst`, `FixedLast`, `FixedPostUpdate`, `FixedPreUpdate`, `FixedUpdate`, `Last`, `PluginGroup`, `PostStartup`, `PostUpdate`, `PreStartup`, `PreUpdate`, `RunFixedMainLoop`, `SpawnScene`, `Startup`, `StateTransition`, `Update`, `clearRegistry`, `conditions`, `definePlugin`, `getCompositions`, `registerRoot`, `useControl`, `validControls`

## `raea/asset` (sdk/raea/src/asset.ts)

Expand Down
Loading
Loading