diff --git a/directory/skills/raea/3d/games.md b/directory/skills/raea/3d/games.md
index f0ceb91..3a04dd6 100644
--- a/directory/skills/raea/3d/games.md
+++ b/directory/skills/raea/3d/games.md
@@ -236,10 +236,12 @@ function Hop() {
registerRoot(() => );
```
-## Sound: one-shots on transitions, a bed under the voice
+## Sound: one-shots on transitions, music ducked under the voice
-A game's sound is Tone.js on raea (`motion/music` has the whole of it). Build the instruments at the top level, give
-the bed and the sounds their buses, and fire each one-shot on the frame its event happens:
+A game's sound is Tone.js on raea (`motion/music` has the whole of it). Every input and outcome gets a sound, the
+world's own sounds (machines, footsteps, a room's tone) sit where they are, and music, when the game has some, is a
+loop written to come round, not a pad. Build the instruments at the top level, give the music and the sounds their
+buses, and fire each one-shot on the frame its event happens:
- `useGameEvents(INITIAL, fold, (before, after) => machine.state(after.hero) === "land" && machine.state(before.hero) !== "land")`
gives the frames where a transition happened. ` thump.triggerAttackRelease("C2", "8n", t)} />`
@@ -267,4 +269,5 @@ function Game() {
}
```
-sdk/react/bench/tone/game is a working game with all of this (a machine, a ducked bed, an SAO coin, a voice line).
+sdk/react/bench/tone/game is a working game with all of this (a machine, a chiptune loop that asks and answers,
+ducked under a voice line, an SAO coin).
diff --git a/directory/skills/raea/craft/floor.md b/directory/skills/raea/craft/floor.md
index 7e93c92..120ac9f 100644
--- a/directory/skills/raea/craft/floor.md
+++ b/directory/skills/raea/craft/floor.md
@@ -61,6 +61,8 @@ direction asks for it by name, and the direction then says why in a line.
- **Template captions.** White bold text with one yellow word, an emoji pop, a zoom on every sentence, a whoosh on
every entrance.
- **The same move everywhere.** One entrance, one zoom or one sound on every beat makes every beat weaker.
+- **Music by default.** A bed under every second, or a held pad and drone standing in for a score. Decide what each
+ scene is heard as, and compose what it needs (`motion/music`).
- **Copy that sounds like an ad.** Buzzwords, "Not X. Y." cadences, aphorisms, a label above every headline.
## Never
diff --git a/directory/skills/raea/craft/modes/story.md b/directory/skills/raea/craft/modes/story.md
index db6a9d9..2c7c43f 100644
--- a/directory/skills/raea/craft/modes/story.md
+++ b/directory/skills/raea/craft/modes/story.md
@@ -33,8 +33,9 @@ they want the whole film.
moves. The easy default is choosing a look shot by shot from references, and that ends up as several films.
- **Each shot's subject.** Choose the person, object or action each shot is about. The easy default is a wide
establishing shot, so question every one.
-- **What the music does.** It either leads, and you cut to its map, or it sits under the picture as a score. The easy
- default is a cut on every bar.
+- **What the music does.** It either leads, and you cut to its map, or it scores the picture, written to the cut, or a
+ scene has none and belongs to its own sound or to silence. The easy defaults are a cut on every bar and a score
+ that is one held chord.
- **The poster frame.** Choose it before anything renders, and build the edit toward it.
## The rut
diff --git a/directory/skills/raea/craft/rubric.md b/directory/skills/raea/craft/rubric.md
index eef3827..602b074 100644
--- a/directory/skills/raea/craft/rubric.md
+++ b/directory/skills/raea/craft/rubric.md
@@ -25,14 +25,32 @@ piece's modes isn't asked.
- land (pitch, story, explain): Does the end land: something this audience can do next, a payoff, or a return to the opening?
- pace (all): Do shot lengths change with what they hold, with no stretch sitting still longer than it earns, and cuts falling on words or beats?
- read (all): Is every word that matters large, sharp against what is behind it, and on long enough to read?
-- sound (all): Is the voice clear over everything else, with music and effects marking moments rather than filling space?
-- sound-silent (silent): With no voice, do the music and the effects carry it: cuts and moments landing on the beat, each effect marking a moment rather than filling space, nothing muddy? (asked in place of sound when no one speaks)
+- sound (all): Is the voice clear over everything else, with music and effects marking moments rather than filling space, and any music written for this cut (a tune, harmony that moves, a rhythm, an ending on a cut) rather than a pad or drone held under it?
+- sound-silent (silent): With no voice, do the music and the effects carry it: cuts and moments landing on the beat, the music composed (a tune, moving harmony, a rhythm, sections that change with the picture), each effect marking a moment rather than filling space, nothing muddy? (asked in place of sound when no one speaks)
- world (all): Do colour, type, motion and sound hold one direction from the first frame to the last?
- own (all): With the name covered, could this only be this project, rather than any product in its category?
- feel (play): Does the player's input change the picture right away and in a way they can read?
- again (play): After a failure, is it clear what went wrong and is the next attempt one press away?
- task (tool): Can the task the screen exists for be done without stopping to work out where to look?
+## Music that sounds generated
+
+Listen for these before the decision model does; each is how a made score gives itself away, and the fix
+(`motion/music`):
+
+- **One held texture.** A pad, a sine sub and a long reverb under everything, no attack anywhere. Write a motif, chords
+ that move and a part with rhythm; reverb per instrument, nothing on the bus.
+- **Harmony that stands still.** One chord for four bars or more in a cue that's meant to move. Change it every bar or two.
+- **A loop, not a cue.** The same two bars for thirty seconds. Vary the motif, change the arrangement every four to
+ eight bars, and turn where the edit turns.
+- **Played by a machine.** Every note at one velocity, exactly on the grid. Spread the dynamics; move played parts a
+ few milliseconds; use a recorded groove.
+- **No ending.** A fade-out. End on a chord or a hit on the last cut.
+- **Everywhere.** Music under every scene because the piece "needs music". Some scenes belong to silence or to their
+ own sound.
+- **The wrong source.** A whole score from TokenSynth takes (organ-like sustains, buzz) or from an SAO "ambient
+ music" prompt. Recorded instruments play the parts; TokenSynth fills a timbre no library has; SAO makes effects.
+
## Ranking
| | Means | Do |
diff --git a/directory/skills/raea/craft/verbs/quieter.md b/directory/skills/raea/craft/verbs/quieter.md
index c46f1c4..179ce89 100644
--- a/directory/skills/raea/craft/verbs/quieter.md
+++ b/directory/skills/raea/craft/verbs/quieter.md
@@ -18,7 +18,7 @@ note names. Keep the point of view: a quieter piece still has one.
- Light: fewer sources pulsing, shorter pulses, lower bloom. Glow only where the eye should go.
- Colour: lower saturation on what's around the accent, not on the accent.
- Motion: fewer moving things at once, smaller moves, no bounce.
-- Sound: duck the bed further under the voice; drop layers before lowering levels.
+- Sound: duck the music further under the voice; drop parts and layers before lowering levels.
- Never solve quiet with blur, fog or grain that softens the work itself.
## Verify
diff --git a/directory/skills/raea/craft/verbs/remix.md b/directory/skills/raea/craft/verbs/remix.md
index 4a65ab3..b43bb39 100644
--- a/directory/skills/raea/craft/verbs/remix.md
+++ b/directory/skills/raea/craft/verbs/remix.md
@@ -5,18 +5,23 @@ sound: voice, music, foley and the master.
## Assess
- Listen once for the voice alone: is every word clear over everything else?
-- Then listen for what's filling space rather than marking moments: a bed at one level under everything, a whoosh on
+- Then listen for what's filling space rather than marking moments: music at one level under everything, a whoosh on
every entrance.
+- Then for what the music is: a tune you could hum, chords that move, a part with rhythm, an ending on a cut; or one
+ held texture (a pad, a drone) under a long reverb.
- With a song, check the cuts and hits against its map.
## Decide
+- What each scene is heard as: silence, its own sound, source music, score. Some scenes need no music.
- What leads: the voice, or the music. Everything else sits under it.
- Which moments get a sound. A few marked moments beat a sound on every one.
## Change
- The voice first; duck music and foley under it.
- Sound marks moments: a hit on the cut that matters, silence before the drop. Remove what only fills.
-- Music: a library instrument first; TokenSynth takes are curated, not trusted. Under a voice, a sparse score.
+- Music: composed for the cut (`motion/music`) on recorded instruments first; TokenSynth takes are curated, not
+ trusted. Under a voice, thin the arrangement and let the tune answer between lines.
+- Effects: layered (a transient, a body, a tail), with repeats varied (`motion/sound`).
- Master at -14 LUFS integrated, true peak at or below -1 dBTP.
## Verify
@@ -24,6 +29,8 @@ sound: voice, music, foley and the master.
## When it goes wrong
- The voice is clear on headphones and buried on a phone's speaker.
-- The music pumps as it ducks under the voice. Use a slower release, or a lower bed.
+- The music pumps as it ducks under the voice. Use a slower release, or lower music.
+- The score has turned into a drone: one chord held under a long reverb. Move the harmony, add a part with rhythm,
+ and take the reverb off the bus.
- Clipping, or a true peak over the limit. Measure, and don't trust the meter by eye.
- The loudness is off the delivery's spec.
diff --git a/directory/skills/raea/motion.md b/directory/skills/raea/motion.md
index bdb59fd..c964e8d 100644
--- a/directory/skills/raea/motion.md
+++ b/directory/skills/raea/motion.md
@@ -18,7 +18,8 @@ Read the files a task needs with `raea guide motion/NAME`, one or two at a time,
| planning how shots cut into each other: hooks on the thirds, the eye carried through each cut | `motion/match-cuts` |
| moving anything: entrances, exits, counters, push-ins | `motion/animating` |
| cutting takes, framing a face, trimming, clip sound | `motion/footage` |
-| music beds, foley, gains, fades, the master | `motion/sound` (the music itself: `motion/music`) |
+| what each scene is heard as, effects and foley, gains, fades, the master | `motion/sound` (the music itself: `motion/music`) |
+| where sounds come from: the local models and their setup, sample libraries, effect packs, MIDI | `motion/sources` |
| captions, or anything cued to what someone says | `motion/words` |
| putting parts over a face, platform zones, the proof | `motion/placement` |
| cards, stacks, type sizes, fonts | `motion/layout` |
@@ -63,9 +64,11 @@ Read the files a task needs with `raea guide motion/NAME`, one or two at a time,
(`text-brand-accent`, `bg-brand-surface`, `font-brand`, `text-title`, `rounded-brand` and the rest in AGENTS.md),
or `var(--brand-*)` where a class can't reach. Never write a hex palette of your own. A role the kit leaves empty
is yours to decide from the brand's own colours (`motion/brand`).
-8. **Sound is part of the edit.** Each sound goes in its own ``. Use `gainDb` for
- levels, Tone.js for music (`motion/music`: TokenSynth or sample-library instruments in a Sampler) and `sao()` for
- foley, and put one ` ` in the composition. Put the sounds in the tree in time order.
+8. **Sound is part of the edit.** Decide what each scene is heard as (silence, its own sound, source music, score,
+ effects) before placing any; music is composed for the cut, never a bed by default. Each sound goes in its own
+ ``. Use `gainDb` for levels, Tone.js for music (`motion/music`: sample-library or
+ TokenSynth instruments in a Sampler, playing a motif, chords that move and a rhythm) and `sao()` for foley, and
+ put one ` ` in the composition. Put the sounds in the tree in time order.
9. **Keep animated styles present on every frame.** The engine keeps a style property that React stops sending.
Write `transform: scale(1)` and `opacity: 1` rather than leaving them out when an animation settles. Always set
`fontFamily`, because the engine's default face is a serif.
@@ -80,13 +83,19 @@ unreadable text, the wrong loudness) and bans (invented claims, generated imager
## A whole composition, small
```tsx
-import { AbsoluteFill, Composition, Easing, Mix, Sequence, Audio, interpolate, registerRoot, sao, useCurrentFrame, useVideoConfig } from "raea";
+import { AbsoluteFill, Composition, Easing, Mix, Sequence, Audio, interpolate, registerRoot, sao, staticFile, useCurrentFrame, useVideoConfig } from "raea";
import * as Tone from "raea/tone";
import { css } from "./styles.gen";
-// the bed: one TokenSynth instrument in a Sampler, a bar of notes on a loop (motion/music)
-const keys = new Tone.Sampler({ urls: Tone.tokensynth("a soft felt piano, warm, intimate", ["C3", "C4"], { seconds: 2 }), volume: -14 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => keys.triggerAttackRelease(n, "4n", t), ["C4", "G3", "E4", "G3"], "4n").start(0);
+// the music: a chord on the title and a two-note lift after it, on a library piano (`raea sounds get
+// salamander/piano`: roots a minor third apart, D#4 as Ds4.mp3); a sting, not a bed (motion/music)
+const ROOTS = ["C4", "D#4", "F#4", "A4", "C5", "D#5"];
+const piano = new Tone.Sampler({ urls: Object.fromEntries(ROOTS.map((n) => [n, staticFile(`sounds/salamander-piano/${n.replace("#", "s")}.mp3`)])), release: 1, volume: -10 }).connect(Tone.buses.music);
+new Tone.Part((t, n) => piano.triggerAttackRelease(n.notes, n.dur, t, n.vel), [
+ { time: "0:0", notes: ["C4", "E4", "G4", "C5"], dur: "2n", vel: 0.8 },
+ { time: "0:2", notes: ["D5"], dur: "8n", vel: 0.55 },
+ { time: "0:2:2", notes: ["E5"], dur: "4n", vel: 0.65 },
+]).start(0);
Tone.getTransport().start();
const WHOOSH = sao("soft airy whoosh, short, clean, no music", { seconds: 0.8 });
diff --git a/directory/skills/raea/motion/assets.md b/directory/skills/raea/motion/assets.md
index 8a063fa..77c5012 100644
--- a/directory/skills/raea/motion/assets.md
+++ b/directory/skills/raea/motion/assets.md
@@ -15,7 +15,8 @@
and `Tone.tokensynth(timbre, notes)` for an instrument's notes in a `Tone.Sampler` (`motion/music`).
- Recorded instruments and drum hits come from the open sample libraries: `raea sounds WORDS` finds one, `raea sounds
get PACK/INSTRUMENT` copies its files into `media/sounds/PACK-INSTRUMENT/` with a `LICENCE.txt`, and prints the
- `staticFile` paths and the Sampler that plays them.
+ `staticFile` paths and the Sampler that plays them. Orchestras, drum kits, effect packs and MIDI grooves fetched by
+ hand, and how to set up SAO and TokenSynth, are in `motion/sources`.
- Show stills with ` `. The frame waits for the image to load, and a missing one fails the render, naming
`src`.
- Keep pictures real: the person's footage, stock already in the object's media, or things you build (components,
@@ -28,9 +29,9 @@ error and so is a URL (`https://...`). `staticFile` checks both. A module import
fails loudly with the line that has it instead of rendering a black clip.
Generated imagery is banned outright (floor): no AI-generated images or video, ever. Generated sound is allowed by
-policy, and made on this computer: TokenSynth for an instrument's notes (music), SAO for foley, Pocket TTS
-(``) for narration over the picture. The sample libraries are recordings under open licences (CC0, CC-BY, public domain);
-a CC-BY one is credited where the piece is shown.
+policy, and made on this computer: TokenSynth for an instrument's notes where no library has the timbre, SAO for
+foley, Pocket TTS (``) for narration over the picture. The sample libraries are recordings under open licences
+(CC0, CC-BY, public domain); a CC-BY one is credited where the piece is shown.
## Common mistakes
diff --git a/directory/skills/raea/motion/gen.md b/directory/skills/raea/motion/gen.md
index 8c7ef16..67900cd 100644
--- a/directory/skills/raea/motion/gen.md
+++ b/directory/skills/raea/motion/gen.md
@@ -3,7 +3,9 @@
## What to do
- Generate a picture, a clip, music, an effect, a voice, a 3D model or words with `raea gen` when the piece needs
- something that isn't in the footage, the stock or the scene. `-o` is the folder; each command prints the files it wrote; use it like any
+ something that isn't in the footage, the stock or the scene. Music only when the person asks for a generated
+ track: a score is composed on instruments for the cut (`motion/music`), and a generated track can't be cut on the
+ bar or edited note by note. `-o` is the folder; each command prints the files it wrote; use it like any
other media (` `, ``, ``, ``).
- Quote first. `raea gen plan FLOW` and `--plan` on a one-shot cost nothing and say where each step runs and what it
costs in cents. A run past its cap is refused before anything is spent: `--max-cents N`, else
diff --git a/directory/skills/raea/motion/music.md b/directory/skills/raea/motion/music.md
index 7cf6ff9..0c3e872 100644
--- a/directory/skills/raea/motion/music.md
+++ b/directory/skills/raea/motion/music.md
@@ -1,5 +1,26 @@
# Music and sound effects: Tone.js on raea (raea/tone)
+## Decide what the picture needs before writing a note
+
+Music is never a default. A bed under every second makes every piece sound like every other one, and a held pad or
+drone under a long reverb is what generated music sounds like. Before any instrument goes in:
+
+- Name what each stretch is heard as: silence; the scene's own sound (diegetic: footsteps, a room, a radio, a game's
+ machines, anything the people on screen could hear); source music (music playing in the scene); score (music only
+ the audience hears); effects; a voice. Several at once is fine, and none is a choice too: a pitch with a voice may
+ need only effects and a four-second sting on the end card. Write the decision in the storyboard or DIRECTION.md,
+ one line a scene.
+- Spot the cut the way a music editor does: number each cue (1m1, 1m2...) with the frame it starts on, the moments
+ it has to hit (the hook, a reveal, the logo), where it turns (where the edit changes idea) and where it stops. A
+ cue ends on a hit, the "button" (the last chord on the last cut or the logo), not on a fade. Half a bar of silence
+ before a reveal is what makes the reveal.
+- Take the tempo from the cut. Frames per beat = fps × 60 / bpm: at 30 fps, 12 frames at 150 bpm, 15 at 120, 20 at
+ 90. Pick the bpm that puts the main cuts on beats and the biggest on downbeats, or set it so they do
+ (`bpm.value = (bars * 4 * 60 * fps) / framesToTheCut`). Shots of 1 to 1.5 s suit 120 to 160 bpm; shots of 3 to 5 s
+ suit 70 to 100, or a half-time feel. With a song, cut to its map instead (`motion/timing`).
+- A sustained texture (a held pad, a drone, a reverb wash) goes in only when the scene asks for one by name ("dread",
+ "the ship's hum") and the direction says why in a line. Even then something over it moves.
+
## What to do
- Write music and sound effects in Tone.js, as its docs write them: `import * as Tone from "raea/tone"`, then
@@ -7,53 +28,102 @@
`Part`, `Sequence` and `Pattern`. What you know of Tone works; the differences are below. It's a DAW in code: a
track is an instrument and its effects into a bus, a clip is a `Part` or `Sequence`, the arrangement is where each
starts and stops on the Transport.
-- Make music from instruments, not from a generated track. Each instrument is a `Tone.Sampler`; fill it one of two
- ways, both local and free:
- - **TokenSynth** for an instrument described in words: `Tone.tokensynth("a warm upright bass, plucked", ["C2",
- "C3", "C4"], { seconds: 2 })` renders those root notes once on this computer, and the Sampler pitches between
- them. Describe the timbre the way you'd describe a patch ("a soft wooden marimba, felt mallet, warm"; "a bright
- analog synth lead, saw, short decay"). Three roots an octave apart cover a part; a note's `seconds` is how long
- it's held (under 5). Each root takes about 5 s to make the first time, then it's cached.
- - **A sample library** for a real recorded instrument: `raea sounds piano` searches the open libraries (the
- Salamander grand piano, the 128 General MIDI instruments, tonejs-instruments' orchestra, the Versilian library's
- orchestral, world and percussion instruments); `raea sounds get salamander/piano` copies its notes into
- `media/sounds/` and prints the `Sampler` that plays them. Drums and hits come as takes for `Tone.Player`s.
- - Tone's own synths (`Synth`, `PolySynth`, `MembraneSynth`, `MetalSynth`, `NoiseSynth`, `FMSynth`) are there for
- pads, kicks, risers and anything synthetic.
-- Make sound effects and foley with SAO: `Tone.sao("a bright arcade coin chime, short", { seconds: 1 })` in a
- Sampler or Player, or `sao(...)` in an `` (`motion/sound`). Narration over the picture is
- `` from `raea/speech` (Pocket TTS, local; `motion/sound`).
+- Make music from instruments playing notes, not from a generated track. Each instrument is a `Tone.Sampler` (a kit
+ is `Tone.Players`); fill it one of three ways:
+ - **A sample library** for a real instrument, first: `raea sounds cello` searches the open libraries raea carries
+ (the Salamander grand piano; FluidR3's 128 General MIDI instruments, every note; tonejs-instruments' strings,
+ brass, woodwinds and guitars; the Versilian library's keyboards, harps, mallets, drums and percussion), and
+ `raea sounds get tonejs-instruments/cello` copies its notes into `media/sounds/` and prints the `Sampler` that
+ plays them. Drums and hits come as takes for `Tone.Player`s. Whole orchestras, multi-velocity kits and drummers'
+ grooves are fetched by hand (`motion/sources`).
+ - **TokenSynth** for a timbre no library has: `Tone.tokensynth("a bowed saw, close, eerie", ["C3", "E3", "G#3",
+ "C4", "E4", "G#4", "C5"], { seconds: 2 })` renders those root notes, a major third apart, once on this computer,
+ and the Sampler pitches between them. Describe
+ the timbre the way you'd describe a patch ("a soft wooden marimba, felt mallet, warm"; "a bright analog synth
+ lead, saw, short decay"); a note's `seconds` is how long it's held (under 5). Its takes need curating (below).
+ - Tone's own synths (`Synth`, `PolySynth`, `MonoSynth`, `MembraneSynth`, `MetalSynth`, `NoiseSynth`, `FMSynth`) for
+ parts that are synthetic on purpose: an 808 kick, an analog bass, a chiptune lead, a riser.
+- Make sound effects and foley with SAO or a recorded pack: `Tone.sao("a bright arcade coin chime, short", { seconds:
+ 1 })` in a Sampler or Player, or `sao(...)` in an ``, built in layers (`motion/sound`). Narration over the
+ picture is `` from `raea/speech` (Pocket TTS, local; `motion/sound`).
- Build the graph at the top level of the file: every instrument, effect and channel, and how they connect. The graph
is the one the composition has when its first frame renders. Schedule notes with the Transport (or with explicit
times). Never trigger notes from inside a component's render: a game's one-shots go through ``.
+- Set the tempo before building any Part: a Part reads its times at the tempo when it's made.
- Time 0 is the composition's first frame. `Tone.now()` is 0 at the top level and the event's time inside a callback,
so `Tone.getTransport().start()` at the top level starts the music at frame 0. Every note lands on its own sample,
so a render and a play are the same.
- Route through the buses: `Tone.buses.music`, `.sfx`, `.voice`, `.ambience`, `.ui`. Connect Tone nodes to them, and
give `` and `` a `bus="voice"` (or another bus) prop. Each bus is a Tone `Channel`: `volume`, `pan`,
`mute`, `solo`, `send`/`receive`.
-- Duck the bed under the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`; do the same for
- ambience. Start at -10 to -14 dB, 40 ms attack and 300 to 400 ms release. Then listen.
+- Duck the music under the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`; do the same for
+ ambience. Start at -10 to -14 dB, 40 ms attack and 300 to 400 ms release; under a sparse cue, where pumping shows,
+ -6 to -8 dB, 80 ms and about 0.9 s. Then listen.
- Keep one ` ` per composition. The master measures the whole core, Tone included.
- In a game, fire a one-shot on the exact frame something happens: ` ...} />` with
- `when` from `useGameEvents` or a machine's transition. Give one-shots `Tone.vary(instrument, { pitch: 0.3,
- volume: 2 })` so repeats don't sound copied (the variation is seeded by the frame, so a replay is the same). Cap
- their voices with `maxPolyphony` (the oldest is stolen, faded in 5 ms).
+ `when` true on that frame only (a frame from `useGameEvents`, or a machine's transition). Give one-shots
+ `Tone.vary(instrument, { pitch: 0.3, volume: 2 })` so repeats don't sound copied (the variation is seeded by the
+ frame, so a replay is the same). Cap their voices with `maxPolyphony` (the oldest is stolen, faded in 5 ms).
- In a timeline, cut on the music: `Tone.Time("1m").toFrames(fps)` is a bar in frames, and `Tone.Time("4n")` a beat.
-- To hear a sound before writing it in, `raea sao "PROMPT" --seconds 1 -o media/sfx/try.wav` or `raea tokensynth
- "TIMBRE" --notes C3 -o media/try/` makes it now; the edit's own call with the same words and settings then plays
- what was made instead of making it again.
+- To hear a sound before writing it in, `raea sao "PROMPT" --seconds 1 --dir EDIT_DIR -o EDIT_DIR/media/sfx/try.wav`
+ or `raea tokensynth "TIMBRE" --notes C3 --dir EDIT_DIR -o EDIT_DIR/media/try/` makes it now and keeps it in the
+ edit's asset store (`--dir` is the edit's folder; without it, the folder you run it from), so the edit's own call
+ with the same words and settings then plays what was made instead of making it again. Another `--seed` makes
+ another take; play a seed's file by its path (`staticFile("sfx/try.wav")`), since `sao()` in the edit has no seed.
+- When SAO or TokenSynth isn't installed yet, `raea setup-audio` installs it (`motion/sources` says what it fetches
+ and what to do when it fails).
+
+## Writing music for picture
+
+Write it as a composer would, for this cut:
+
+- **A motif first.** Three to seven notes over two to four bars, with a rhythm of its own, before any chord. Then
+ vary it rather than loop it: answer it, fragment it, move it to another instrument or register, put new chords
+ under it. End the cue on it.
+- **Harmony that moves.** In an active cue, change chord every bar or every two beats; keep chords of two to four bars
+ for a slow passage. Move the inner voices by step or hold a common tone, set the bass against the tune (contrary
+ motion, inversions so it walks), and resolve a pedal when it ends. Keep the low end open: no thirds below about C3,
+ fifths down to about C2, only octaves under that.
+- **Rhythm and groove.** At least one part moves in eighths or sixteenths: an ostinato, a pizzicato, an arpeggio, a
+ kit, hand percussion. Accent where the picture moves. Spread velocities (0.35 to 0.95, not all 0.8), and put a
+ played part a few milliseconds off the grid (5 to 15 ms, written into the note's time: `humanize` is ignored here).
+ A drummer's recorded groove beats one typed in (Groove MIDI, `motion/sources`).
+- **Orchestrate by register.** One job each: a bass (E1 to C3), sparse harmony (C3 to C5), one lead (C4 to C6), the
+ rhythm, colour on top (glockenspiel, celesta, harp). Hand the tune to a new instrument each section, and use the
+ articulations the library has (staccato, legato, pizzicato, tremolo, a swell). A pad, where the scene names one,
+ is glue 12 dB or more under the lead, never the part.
+- **The arrangement follows the edit.** Sections change where the edit changes idea: build by adding parts and by
+ switching sections, on bar lines; peak, then strip back; drop out before the reveal; end on the button. A sting
+ lasts 1 to 4 s, sits in the cue's key and lands on a beat.
+- **Write around the voice.** The tune answers in the gaps between lines, and the arrangement thins under speech.
+- **Each instrument its own room.** A short reverb per instrument or per section (decay 1.2 to 2.5 s, wet 0.15 to
+ 0.3), never one long wash over the bus: a 6 s reverb and a low-pass on the whole music bus erase every attack, and
+ that is how a score turns into a drone.
+- **Source music lives in the scene.** A radio, a band in the room, a phone: narrow it to the device (a high-pass
+ around 300 Hz and a low-pass around 3.5 kHz for a small speaker), pan it to where it is (`Tone.Panner`), put it in
+ the room's reverb, and let the scene's own sound sit with it. Crossing over is strong: a radio song that opens out
+ to full range as it becomes the score, a score hit that lands on an action in the scene.
+- Leave silence. Let the music start after a chapter card clears and stop before the next; let a card sit on one low
+ note.
+- Write for this picture's feeling, not a genre's surface: a stock walking bass and brushes under everything reads as
+ pastiche as surely as a pad does.
+- Period source music can be real: US sound recordings published before 1926 and compositions published before 1931
+ are public domain in the US (2026). archive.org's 78rpm transfers give the period's own sound; play them as
+ ``, narrowed and placed as source music, ducked with the rest.
+- Synth drums (`MembraneSynth` kicks, noise "brushes") sound synthetic on anything acoustic; use a library's recorded
+ kit (`raea sounds kick`, `snare`, `hihat`).
## Instruments from TokenSynth: build a library, don't trust a take
A TokenSynth note is one sample from a model, and a lot of takes are bad. In a 35-minute documentary score (2026-10-20),
12 of 30 single-seed root notes failed: hiss and buzz in place of a horn, a sax or accordion an octave (or 34
semitones) off, a piano note still at full level when its file ends, so every held note stopped with a click. That is
-the "vanilla and abrasive" sound. So:
+the "vanilla and abrasive" sound. It learned from 5-second notes at four velocities, so it gives one dynamic and can
+miss the pitch asked for. So:
- Reach for an open sample library first for a standard instrument (`raea sounds piano`, `sounds get
- salamander/piano`): recorded, velocity-layered, no gamble. Use TokenSynth for what no library has, or a timbre you
- can describe better than find.
+ salamander/piano`): a recording, no gamble. Use TokenSynth for what no library has, or a timbre you can describe
+ better than find.
- Declare each instrument as a candidate, not a call: its timbre, roots every 3 to 4 semitones over the range it plays
(a Sampler pitched further than that starts to sound fake), 3 or more seeds, and `seconds` long enough for the
sound to die away on its own (4 for piano, 6 for sustained strings).
@@ -67,24 +137,7 @@ the "vanilla and abrasive" sound. So:
- Prefer timbres TokenSynth does well (felt or soft piano, sustained cello section, bowed or plucked bass, vibraphone,
clarinet in its low register) to the ones it does badly (muted trumpet, tenor sax, accordion: breath and reed
noise come out as hiss).
-
-## Writing music for picture
-
-- Write for the picture's mood, not a genre's surface. A walking bass, brushes and block chords on every bar reads as
- MIDI pastiche. Under narration, a drone score works: a chord every four bars, bass and a sine sub swelling under it,
- a sustained section with a slow attack, and a piano playing single notes off any grid, with a motif returning in
- fragments. Sustained textures also hide what's left of a sample's flaws.
-- Leave silence. Let the music start after a chapter card clears and stop before the next; let a card sit on one low
- note.
-- Soften the whole bed: a 24 dB/octave low-pass around 2.5 kHz into a long reverb (decay 6 to 7 s) on the music bus.
- The voice owns the presence range.
-- Duck gently under a voice over a drone (-6 to -8 dB, 80 ms attack, about 0.9 s release); the -12 dB duck suits a
- rhythmic bed, where pumping is less audible.
-- Period source music can be real: US sound recordings published before 1926 and compositions published before 1931
- are public domain in the US (2026). archive.org's 78rpm transfers give the period's own sound; play them as
- `` under the score's drones, ducked with the rest.
-- Synth drums (`MembraneSynth` kicks, noise "brushes") sound synthetic on anything acoustic; leave them out or use a
- library's recorded kit.
+- For dynamics, render a soft and a loud velocity as two Samplers and pick one by each note's velocity.
## Why
@@ -96,24 +149,38 @@ So the graph goes at the top level, and in-game sounds go through ``.
Generated sources are made once per their inputs: the prompt, the length, a TokenSynth note's pitch, velocity, seed
and precision. They're normalised, cached as PCM in the object's `assets/`, and played like any file. TokenSynth and
-SAO run on this computer and install themselves the first time they're used (a few minutes, once). Music written as
-notes on instruments can be edited note by note, cut on the bar and ducked per part; a generated track can't.
+SAO run on this computer and install themselves the first time they're used (minutes, once; `raea setup-audio` does
+it ahead of time). Music written as notes on instruments can be edited note by note, cut on the bar and ducked per
+part; a generated track can't.
-Policy: music is Tone.js instruments (TokenSynth, the sample libraries, synths), foley and effects are SAO, narration
-is Pocket TTS (``), all local. No other generation: no generated images, video or remote sound effects. (`lyria()`, a remote music model, is
-retired: an older edit keeps the take it made, and a new one is refused.)
+Policy: music is notes on Tone.js instruments (sample libraries first, TokenSynth for a timbre no library has, synths
+for synthetic parts), effects and foley are SAO or recorded packs (`motion/sources`), narration is Pocket TTS
+(``), all local. Any other generation is a `raea gen` step, quoted and capped (`motion/gen`). (`lyria()`, a
+remote music model, is retired: an older edit keeps the take it made, and a new one is refused.)
## Common mistakes
+- A pad and a sine sub through a low-pass into a long reverb as the score. That's a drone, not music: write a motif,
+ chords that move and a part with rhythm.
+- One chord every four bars for a minute, or a two-bar loop for thirty seconds. Change the harmony every bar or two
+ and the arrangement every four to eight.
+- Every note at the same velocity, on the grid. Spread the dynamics and move played parts a few milliseconds.
+- A cue that fades out at the end. End it on the last cut, on a chord or a hit.
+- Music under every scene because the piece "needs music". Decide per scene; silence and the scene's own sound count.
- `synth.triggerAttackRelease(...)` in a component body. It fires on every render; use ``.
+- `` on a flag the state holds for many frames: it fires on every one. Give it the
+ frames the event happens on (`useGameEvents`).
- `new Tone.Synth()` inside a component. Make instruments at the top level.
- A `https://` URL for a Sampler or a Player. Sources are media files or generated assets, and the render has no
network: `raea sounds get` puts a library's files in `media/` first.
-- One TokenSynth root for a whole part. A Sampler stretched three octaves from one note sounds chipmunked or
- muddy; give it a root every octave (or `raea sounds get` a library's roots, a minor third apart).
+- One root for a whole part, or one an octave. A Sampler stretched far from its root sounds chipmunked or muddy;
+ give it a root every 3 to 4 semitones over the range the part plays: a library's own roots (`raea sounds get`
+ with no `--notes` takes them a minor third apart, or every note the library sampled if it's sparser), or as many
+ TokenSynth roots.
- Rewording a `tokensynth()` timbre on every pass: each new wording makes its notes again. Settle the timbre first.
-- A sample library's licence ignored. CC-BY ones (FluidR3, Salamander) need their credit where the piece is shown;
- `LICENCE.txt` beside the files has the line. VCSL is CC0.
+- A sample library's licence ignored. CC-BY ones (FluidR3, Salamander, tonejs-instruments) need their credit where
+ the piece is shown; `LICENCE.txt` beside the files has the line. VCSL is CC0.
+- `part.loop = true` after `part.start()`: the Part plays once. Set `loop` and `loopEnd` first.
- `Tone.Draw.schedule` for visuals. There's no draw loop: read the frame (`useCurrentFrame`) and the same times.
- Cutting on `Tone.Time("1m").toFrames(fps)` across a tempo ramp. `Tone.getTransport().bpm.rampTo(120, 4)` bends the
grid, as in Tone (the notes come closer together; an exponential ramp runs linear), but `Time(...)` is at the tempo
@@ -123,44 +190,112 @@ retired: an older edit keeps the take it made, and a new one is refused.)
.connect(filter.frequency)`).
- A Sequence or Loop with no `Tone.getTransport().start()`. Nothing plays until the Transport starts, as in Tone.
-## Example: a bed from TokenSynth instruments and a library piano, cut on the bar
+## Example: a cue with a motif, chords that move and a kit, cut on the bar
+
+Spotted as one cue over nine one-bar cards: A (bars 1 to 4) piano and a pizzicato ostinato, the flute's motif from
+bar 3; B (bars 5 to 8) the ostinato drops out, bass and kit come in, the motif climbs; the button on the end card's
+cut. The instruments came from these commands, each printing the roots it fetched (with no `--notes`, a minor third
+apart from C1 to C7, or every note the library sampled when that's sparser):
-`raea sounds get salamander/piano --notes C3,C4,C5` put the piano's notes in `media/sounds/salamander-piano/`.
+```sh
+raea sounds get salamander/piano
+raea sounds get fluid/pizzicato_strings
+raea sounds get tonejs-instruments/flute
+raea sounds get fluid/acoustic_bass
+raea sounds get vcsl/bass-drum-1 && raea sounds get vcsl/snare-drum-modern-1 && raea sounds get vcsl/hi-hat-cymbal
+```
```tsx
import { AbsoluteFill, Composition, Mix, Series, registerRoot, staticFile } from "raea";
import * as Tone from "raea/tone";
-Tone.getTransport().bpm.value = 92;
-const BAR = Tone.Time("1m").toFrames(30); // 78 frames at 92 bpm in 4/4
-const room = new Tone.Reverb({ decay: 2.2, wet: 0.25 }).connect(Tone.buses.music);
-const bass = new Tone.Sampler({ urls: Tone.tokensynth("a warm upright bass, plucked, round", ["C2", "C3"], { seconds: 1.5 }), release: 0.3 }).connect(Tone.buses.music);
-const keys = new Tone.Sampler({
- urls: { C3: staticFile("sounds/salamander-piano/C3.mp3"), C4: staticFile("sounds/salamander-piano/C4.mp3"), C5: staticFile("sounds/salamander-piano/C5.mp3") },
- release: 1.2,
- volume: -8,
-}).connect(room);
-const kick = new Tone.MembraneSynth({ octaves: 5, volume: -8 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => kick.triggerAttackRelease(n, "8n", t), ["C1", null, null, null], "4n").start(0);
-new Tone.Part((t, n) => bass.triggerAttackRelease(n, "4n", t), [["0:0", "A2"], ["0:2", "E2"], ["1:0", "F2"], ["1:2", "C3"]]).start(0);
-new Tone.Part((t, c) => keys.triggerAttackRelease(c, "2n", t), [["0:0", ["A3", "C4", "E4"]], ["1:0", ["F3", "A3", "C4"]]]).start(0);
+Tone.getTransport().bpm.value = 96; // first: a Part reads its times at the tempo when it's made
+const BAR = Tone.Time("1m").toFrames(30); // 75 frames
+
+// the roots each part's range needs, of those `raea sounds get` printed (D#3 is the file Ds3.mp3)
+const notes = (dir: string, names: string[]) => Object.fromEntries(names.map((n) => [n, staticFile(`sounds/${dir}/${n.replace("#", "s")}.mp3`)]));
+const room = new Tone.Reverb({ decay: 1.6, wet: 0.2 }).connect(Tone.buses.music);
+const piano = new Tone.Sampler({ urls: notes("salamander-piano", ["A2", "C3", "D#3", "F#3", "A3", "C4", "D#4", "F#4"]), release: 1, volume: -8 }).connect(room);
+const pizz = new Tone.Sampler({ urls: notes("fluid-pizzicato_strings", ["D#2", "F#2", "A2", "C3", "D#3", "F#3", "A3", "C4"]), release: 0.2, volume: -12 }).connect(room);
+const flute = new Tone.Sampler({ urls: notes("tonejs-instruments-flute", ["E4", "A4", "C5", "E5", "A5", "C6"]), release: 0.4, volume: -6 }).connect(room);
+const bass = new Tone.Sampler({ urls: notes("fluid-acoustic_bass", ["D#1", "F#1", "A1", "C2", "D#2"]), release: 0.3, volume: -6 }).connect(Tone.buses.music);
+const kit = new Tone.Players({
+ kick: staticFile("sounds/vcsl-bass-drum-1/BDrumNew_hit_v5_rr1_Sum.wav"),
+ snare: staticFile("sounds/vcsl-snare-drum-modern-1/Snare2_HitSN_v5_rr1_Mid.wav"),
+ hat: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v3_rr1_Mid.wav"),
+ ghost: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v1_rr1_Mid.wav"), // the soft take, for the off-beats
+}).connect(Tone.buses.music);
+
+// a chord a bar, voiced so the inner notes move by step: Am F C G, then F G Am E
+const CHORDS = [["A3", "C4", "E4"], ["A3", "C4", "F4"], ["G3", "C4", "E4"], ["G3", "B3", "D4"], ["A3", "C4", "F4"], ["G3", "B3", "D4"], ["A3", "C4", "E4"], ["G#3", "B3", "E4"]];
+// comped, not held: the downbeat and the and-of-two, the second softer
+new Tone.Part((t, c) => piano.triggerAttackRelease(c.notes, "8n", t, c.vel), CHORDS.flatMap((chord, bar) => [
+ { time: `${bar}:0`, notes: chord, vel: 0.62 },
+ { time: `${bar}:1:2`, notes: chord, vel: 0.42 },
+])).start(0);
+
+// A: an eighth-note ostinato on each chord's root (root, fifth, octave, fifth), accented on the beat
+new Tone.Part((t, n) => pizz.triggerAttackRelease(n.note, "16n", t, n.vel), ["A2", "F2", "C3", "G2"].flatMap((root, bar) =>
+ [0, 7, 12, 7, 0, 7, 12, 7].map((up, i) => ({ time: `${bar}:${i >> 1}:${(i & 1) * 2}`, note: Tone.Frequency(root).transpose(up).toNote(), vel: i % 2 ? 0.45 : 0.75 })),
+)).start(0);
+
+// the motif (E D C D, then B A G) from bar 3, sequenced up and answered in B; each note a few ms off the grid
+const TUNE: [string, string, string, number][] = [
+ ["2:0", "E5", "4n", 0.7], ["2:1", "D5", "8n", 0.55], ["2:1:2", "C5", "8n", 0.6], ["2:2", "D5", "2n", 0.75],
+ ["3:0", "B4", "4n.", 0.65], ["3:1:2", "A4", "8n", 0.5], ["3:2", "G4", "2n", 0.6],
+ ["4:0", "A5", "4n", 0.8], ["4:1", "G5", "8n", 0.6], ["4:1:2", "F5", "8n", 0.65], ["4:2", "G5", "2n", 0.8],
+ ["5:0", "D5", "4n.", 0.7], ["5:1:2", "B4", "8n", 0.55], ["5:2", "D5", "2n", 0.7],
+ ["6:0", "E5", "4n", 0.75], ["6:1", "C5", "8n", 0.6], ["6:1:2", "E5", "8n", 0.7], ["6:2", "A5", "2n", 0.9],
+ ["7:0", "G#5", "2n.", 0.8], ["7:3", "B4", "4n", 0.6],
+];
+new Tone.Part((t, n) => flute.triggerAttackRelease(n.note, n.dur, t + n.late, n.vel),
+ TUNE.map(([time, note, dur, vel], i) => ({ time, note, dur, vel, late: ((i * 5) % 3 - 1) * 0.007 }))).start(0);
+
+// B: the bass walks up under the climb (F G A, then E to come home), root on one and fifth on three
+new Tone.Part((t, n) => bass.triggerAttackRelease(n.note, "4n.", t, n.vel), [["F1", "C2"], ["G1", "D2"], ["A1", "E2"], ["E1", "B1"]].flatMap(([root, fifth], i) => [
+ { time: `${4 + i}:0`, note: root, vel: 0.85 },
+ { time: `${4 + i}:2`, note: fifth, vel: 0.6 },
+])).start(0);
+
+// B: the kit, four bars of sixteenth-note lanes (null rests)
+const _ = null;
+for (const steps of [
+ ["kick", _, _, _, _, _, "kick", _, "kick", _, _, _, _, _, _, _],
+ [_, _, _, _, "snare", _, _, _, _, _, _, _, "snare", _, _, _],
+ ["hat", _, "ghost", _, "hat", _, "ghost", "ghost", "hat", _, "ghost", _, "hat", _, "ghost", _],
+]) {
+ const lane = new Tone.Sequence((t, hit) => kit.player(hit).start(t), steps, "16n");
+ lane.loop = 4; // set before start: four bars, then it stops
+ lane.start("4m");
+}
+
+// the button: everything on the end card's downbeat, then nothing
+Tone.getTransport().schedule((t) => {
+ piano.triggerAttackRelease(["A2", "A3", "C4", "E4"], "2n", t, 0.85);
+ bass.triggerAttackRelease("A1", "2n", t, 0.9);
+ flute.triggerAttackRelease("A5", "4n", t, 0.8);
+ kit.player("kick").start(t);
+}, "8m");
Tone.getTransport().start();
-// each card is one bar long, so every cut lands on a downbeat
+// each card is one bar, so every cut lands on a downbeat; the ninth is the end card
const Edit = () => (
-
- {[0, 1, 2, 3].map((i) => {/* card i */} )}
+
+ {Array.from({ length: 9 }, (_, i) => {/* card i */} )}
);
-registerRoot(() => );
+registerRoot(() => );
```
+FluidR3, Salamander and tonejs-instruments (the flute) are CC-BY 3.0, so the piece credits all three (the line is in
+each `LICENCE.txt`); VCSL is CC0.
+
## Example: a game's sound
```tsx
-import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, staticFile, useCurrentFrame, useGameEvents, useGameState } from "raea";
+import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, staticFile, useCurrentFrame, useGameEvents } from "raea";
import * as Tone from "raea/tone";
// the game: its state and step (motion/games); only what the sound reads is shown
@@ -168,33 +303,49 @@ type State = { hits: number; coinTaken: boolean };
const INITIAL: State = { hits: 0, coinTaken: false };
const fold = (s: State): State => s;
+Tone.getTransport().bpm.value = 112;
Tone.buses.music.duck(Tone.buses.voice, { amount: -12 });
const thump = Tone.vary(new Tone.MembraneSynth({ octaves: 4, maxPolyphony: 4, poly: true }), { pitch: 0.5, volume: 2 }).connect(Tone.buses.sfx);
const coin = new Tone.Sampler({ urls: { C5: Tone.sao("a bright arcade coin chime, short", { seconds: 1 }) } }).connect(Tone.buses.sfx);
-const marimba = new Tone.Sampler({ urls: Tone.tokensynth("a soft wooden marimba, felt mallet, warm", ["C3", "C4", "C5"], { seconds: 2 }), volume: -10 }).connect(Tone.buses.music);
-const tune = new Tone.Part((t, n) => marimba.triggerAttackRelease(n, "8n", t), [["0:0", "C4"], ["0:1", "E4"], ["0:2", "G4"], ["1:0", "A4"], ["1:2", "G4"]]);
+// a recorded marimba (`raea sounds get vcsl/marimba`: CC0, every note VCSL sampled, four to seven semitones apart)
+const marimba = new Tone.Sampler({
+ urls: Object.fromEntries(["F3", "C4", "G4", "B4", "F5"].map((n) => [n, staticFile(`sounds/vcsl-marimba/${n}.wav`)])),
+ release: 0.4,
+ volume: -10,
+}).connect(new Tone.Reverb({ decay: 1.2, wet: 0.15 }).connect(Tone.buses.music));
+// a loop that wants to come round: a question (bars 1-2) and its answer (bars 3-4), rhythm before pitch
+const tune = new Tone.Part((t, n) => marimba.triggerAttackRelease(n.note, "8n", t, n.vel), [
+ { time: "0:0", note: "C4", vel: 0.8 }, { time: "0:0:2", note: "E4", vel: 0.5 }, { time: "0:1:2", note: "G4", vel: 0.6 }, { time: "0:3", note: "A4", vel: 0.7 },
+ { time: "1:0", note: "G4", vel: 0.8 }, { time: "1:1:2", note: "E4", vel: 0.5 }, { time: "1:2", note: "D4", vel: 0.6 },
+ { time: "2:0", note: "C4", vel: 0.8 }, { time: "2:0:2", note: "E4", vel: 0.5 }, { time: "2:1:2", note: "G4", vel: 0.6 }, { time: "2:3", note: "C5", vel: 0.75 },
+ { time: "3:0", note: "B4", vel: 0.7 }, { time: "3:1", note: "G4", vel: 0.55 }, { time: "3:2", note: "C4", vel: 0.85 },
+]);
tune.loop = true;
-tune.loopEnd = "2m";
+tune.loopEnd = "4m";
tune.start(0);
Tone.getTransport().start();
function Game() {
const frame = useCurrentFrame();
- const g = useGameState(INITIAL, fold);
const hits = useGameEvents(INITIAL, fold, (b, a) => a.hits > b.hits);
+ const coins = useGameEvents(INITIAL, fold, (b, a) => a.coinTaken && !b.coinTaken);
return (
{/* ...the game's picture... */}
thump.triggerAttackRelease("C2", "8n", t)} />
- coin.triggerAttackRelease("C5", "8n", t)} />
+ coin.triggerAttackRelease("C5", "8n", t)} />
);
}
+
+registerRoot(() => );
```
`raea/music` is the shorthand for notes on named instruments (`new Part("bass", [...])`, ``). It plays
-through the same Tone layer, so the two mix.
+through the same Tone layer, so the two mix. Its named instruments are Tone synth presets (`pad` is a PolySynth of
+fat saws, `kick` a MembraneSynth), good for a sketch or a test, not a score: write the score on sampled instruments,
+as above. It also reads and writes MIDI files (`parseMidi`, `toMidi`; `motion/sources`).
diff --git a/directory/skills/raea/motion/parts.md b/directory/skills/raea/motion/parts.md
index 53ee3df..7f6d5bf 100644
--- a/directory/skills/raea/motion/parts.md
+++ b/directory/skills/raea/motion/parts.md
@@ -44,8 +44,8 @@ registerRoot(() => `,
+A screen recording or b-roll has no words to cue on, so it's cut on its music: the person's own track (the host's
+`bed`), never a pad made to fill the silence. The host cuts the shots on the track's beats and gives `BEATS` (each beat's frame in the cut) and `SHOTS` (where each shot looks). Pass ``,
and cue each part by beat:
- `beat`: the beat it lands on, counted from 0. A part on a shot's first beat lands on its cut.
@@ -100,4 +100,4 @@ line has one to three.
- **Captions are always on.** They move under a card, hide under a hook, a strike or the ask, and go big at the top when the band is empty. Leave some beats empty so the face and the captions carry them.
- **The thesis gets Phrases.** The thesis is the first line where the speaker says what the brief's `believe:` line means (the believe line finds it; its own wording never goes on screen): the line the piece is about, often the turn after the numbers, and often said as "not this, but that" ("so the fix isn't / another *discount* / it's a *fast reply*"). It reads phrase by phrase: one `` over it, a phrase per breath, the word that carries each phrase in the accent. Captions alone show it three words at a time, lit word by word, which reads as a caption, not a statement.
- **About one part every three to five seconds.** A beat with nothing to show stays empty.
-- The sound is done for you. Each part brings its own hit, and `` plus ` ` finish it. Don't add more foley.
+- The sound is done for you. Each part brings its own hit, and ` ` finishes it, with `` when the person gave a track (their own track, or none: never a pad made to fill it). Don't add more foley.
diff --git a/directory/skills/raea/motion/sound.md b/directory/skills/raea/motion/sound.md
index 479c9a3..2a6fd6f 100644
--- a/directory/skills/raea/motion/sound.md
+++ b/directory/skills/raea/motion/sound.md
@@ -2,6 +2,10 @@
## What to do
+- Decide what each scene is heard as before placing anything: silence, the scene's own sound (footsteps, a room, a
+ machine: what the people on screen would hear), source music, score, effects, a voice. Music is a decision per
+ scene, never a bed by default (`motion/music` has the spotting and the writing; `motion/sources` where each sound
+ can come from).
- Put each sound in its own `` with an `` inside. The Sequence's
`from` is when the sound starts, and its `durationInFrames` (optional) cuts it off. `layout="none"` because a sound
draws nothing.
@@ -9,17 +13,26 @@
values above 1 boost). Either one can be a function of the sound's own frame (0 at its first frame) for a duck or
a swell. `gainDb` wins when both are given.
- Fade with `fadeIn` and `fadeOut`, in seconds. They ramp from and to silence at the start and end of where the sound
- plays, at the audio's own rate, and multiply with the gain. Use about 1 to 1.4 s at the end of a bed, and
- 0.012 s at a cut.
-- Make foley and sound effects with `sao(prompt, {seconds})` (2 s by default; up to 11.9 s) and pass the result to
- ``. Write prompts the way the approved pieces do: the sound, its character and what to leave out ("deep
- punchy cinematic impact hit, tight, no reverb tail"; "bubbly UI pop click, short, dry"). Ask for the length you
- need.
-- Make the music bed in Tone.js on `Tone.buses.music` (`motion/music`): instruments in a `Sampler` (TokenSynth notes
- or a sample library's), playing a `Part` or `Sequence`. Duck it under the voice with
- `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`, give the voice ``, and set the bed's
- level on the bus: `Tone.buses.music.volume.value = -21`, then `Tone.buses.music.volume.rampTo(-12, 0.3, t)` to lift
- it at `t` seconds for the end card.
+ plays, at the audio's own rate, and multiply with the gain. Use about 1 to 1.4 s where an ambience or a source cue
+ leaves the scene, and 0.012 s at a cut. A score ends on its last hit instead (`motion/music`).
+- Make foley and sound effects with `sao(prompt, {seconds})` (2 s by default; up to 11.9 s, and a longer ask comes
+ back 11.9 s long) and pass the result to ``, or take a recorded one from a pack (`motion/sources`).
+ Write prompts the way the approved pieces do: the sound, its character and what to leave out ("deep punchy
+ cinematic impact hit, tight, no reverb tail"; "bubbly UI pop click, short, dry"). Ask for the length you need.
+- Design an effect in layers, each its own short prompt or take, laid on the same frame: a transient (the first 5 to
+ 30 ms: a snap, a click), a body that says what the thing is (50 to 500 ms), weight for a big hit only (under
+ 120 Hz, short), and a tail that puts it in its room (0.3 to 3 s). Vary a sound that repeats: 3 to 5 takes (another seed or another file), a semitone or two and a couple of dB apart.
+- Give a place its room tone and keep it across that place's cuts: no digital silence under picture, an ambience
+ that changes when the place does, and the next scene's sound starting about half a second before its cut. Foley
+ is three passes, each on the frame of contact: steps, cloth, props.
+- Place what's in the scene where it is: pan it to its side of the frame (`Tone.Panner`, a Channel's `pan`), give
+ it the room's reverb, and narrow a device (a phone, a radio) with a high-pass and a low-pass. In 3D, `` places it and the camera hears it move.
+- When the piece has music, write it in Tone.js on `Tone.buses.music` (`motion/music`): instruments in a `Sampler`
+ (a sample library's notes, or TokenSynth's), parts in a `Part` or `Sequence`, composed for the cut. Duck it under
+ the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`, give the voice ``,
+ and set the music's level on the bus: `Tone.buses.music.volume.value = -21`, then
+ `Tone.buses.music.volume.rampTo(-12, 0.3, t)` to lift it at `t` seconds for the end card.
- Narration over the picture, when the footage has no voice for it, is
` ` from `raea/speech` (Pocket TTS, on this computer; `voice` picks a
Pocket voice or a WAV of one).
@@ -32,12 +45,15 @@
| sound | with a voice | without one (promo, deck, game) |
|---|---|---|
| the voice clips | about 0 dB | none |
- | music bed | -21 dB, rising to about -12 dB once the voice stops (the end card) | -12 to -15 dB |
+ | music | -21 dB, rising to about -12 dB once the voice stops (the end card) | -12 to -15 dB |
| foley hits, whooshes, pops | -16 to -22 dB | -1 to -6 dB |
These are starting points; listen and decide.
-- `raea sao "PROMPT" --seconds 1.2 -o media/sfx/hit.wav` makes a sound now, to check it before it goes in; the edit's
- `sao("PROMPT", { seconds: 1.2 })` then plays that one rather than making another.
+- `raea sao "PROMPT" --seconds 1.2 --dir EDIT_DIR -o EDIT_DIR/media/sfx/hit.wav` makes a sound now, to check it
+ before it goes in, and keeps it in the edit's asset store (`--dir` is the edit's folder; without it, the folder you
+ run it from), so the edit's `sao("PROMPT", { seconds: 1.2 })` then plays that one rather than making another.
+ `--seed 2` makes another take of the same prompt; play that file by its path (`staticFile("sfx/hit-2.wav")`), since
+ `sao()` has no seed.
## Why
@@ -46,14 +62,15 @@ full-frame box that paints over the parts beneath it.
`sao()` returns an asset `{kind, prompt, seconds, id, src}` whose id is a stable hash of the kind, the
prompt and the length. The renderer makes each asset once and caches it in the object's `assets/`. SAO runs on this
-computer (Stable Audio Open, a few seconds a sound; it installs itself the first time). The same call always reuses
+computer (Stable Audio Open Small, a few seconds a sound; it installs itself the first time, and `motion/sources`
+says what that fetches). It does better on effects and field recordings than on music. The same call always reuses
the same sound, and changing one word of the prompt makes a new one. So settle the prompt before you iterate on
gains.
-Policy: music is Tone.js instruments (TokenSynth, the sample libraries, synths), SAO makes foley, and Pocket TTS
-(``) says narration over the picture, all on this computer. Any other generation is a `raea gen` step
-(`motion/gen`). (`lyria()`, a remote music model, is retired: a
-new take is refused, and a bed of instruments can be cut on the bar and edited note by note anyway.)
+Policy: music is notes on Tone.js instruments (sample libraries first, TokenSynth, synths), SAO and recorded packs
+make effects and foley, and Pocket TTS (``) says narration over the picture, all on this computer. Any other
+generation is a `raea gen` step (`motion/gen`). (`lyria()`, a remote music model, is retired: a new take is refused,
+and music written as notes can be cut on the bar and edited note by note anyway.)
Sound sells the cut, but one move everywhere is a lint: a whoosh on every entrance flattens all of them. Give
foley to the moments that are physical (a slam, a strike-through, a card landing on the beat) and let others be
@@ -61,22 +78,23 @@ silent.
## Check the sound before you call it done
-Frames get looked at; sound has to be measured, because a broken bed is easy to miss on one listen and obvious to
+Frames get looked at; sound has to be measured, because a broken mix is easy to miss on one listen and obvious to
everyone else. Two checks, both cheap:
- **The render says when a sound runs out.** `WARN sound X runs out N s before its stretch ends` means the
source is shorter than its Sequence and still sounding (or asked for a fadeOut) at its end: it stops dead and the
- fade never plays, so a 10 s `sao()` bed under a 12 s cue cuts off mid-line. `loop` alone doesn't help: a sound
- loops only between `startFrom` and `endAt`, and that loop is a hard jump. Tile it instead: copies of the take end to
- end, each in its own Sequence, crossfading over 1.5 s (`fadeIn`/`fadeOut`), or make the take as long as the span.
+ fade never plays, so a 10 s `sao()` ambience under a 12 s scene cuts off mid-line. `loop` alone doesn't help: a
+ sound loops only between `startFrom` and `endAt`, and that loop is a hard jump. Tile it instead: copies of the take
+ end to end, each in its own Sequence, crossfading over 1.5 s (`fadeIn`/`fadeOut`), or make the take as long as the
+ span.
- **Check the finished file.** `raea check FILM.mp4 --words words.json` (the voice's words on the film's clock)
- reports, by timecode, dead stops (a steady sound that ends in one 10 ms step: fade a bed out over 100 ms or more;
+ reports, by timecode, dead stops (a steady sound that ends in one 10 ms step: fade it out over 100 ms or more;
a voice ending its word isn't one), stretches with no voice louder than
- the voice, a bed closer than 12 dB to the voice over it, and a true peak over -1 dBTP. Fix every line it prints, or
+ the voice, music closer than 12 dB to the voice over it, and a true peak over -1 dBTP. Fix every line it prints, or
say why it stands.
Set levels from measurements, not by ear on one pass. Measure each source's integrated loudness (`ffmpeg -i F -af
-ebur128 -f null -`) and set its gain against the narration's: beds 15-20 LU under the voice; a film's own sound,
+ebur128 -f null -`) and set its gain against the narration's: music 15-20 LU under the voice; a film's own sound,
where the narration stops, about 3 LU under it with its loudest 3 s no more than 2 LU over; never boost a near-silent
take more than +3 dB (its hiss comes up with it). A flat gain on every archive clip is how one film's main title
ends up shouting over the narrator while another is inaudible.
@@ -85,18 +103,21 @@ ends up shouting over the narrator while another is inaudible.
- An `` without a Sequence, or in a Sequence without `layout="none"`.
- `volume={-6}` meaning decibels. That throws, because volume is a non-negative linear gain. Write `gainDb={-6}`.
-- A bed at -6 dB under the voice. The voice must stay intelligible over everything (floor).
-- `loop` on an `` without `endAt` to stretch a short bed: nothing loops, it plays once and stops dead (the
- render warns). With `endAt` it repeats with a hard jump at each loop; tile copies with crossfades instead.
+- Music at -6 dB under the voice. The voice must stay intelligible over everything (floor).
+- A pad, a drone or one `sao()` take of "ambient music" laid under the whole piece as its score. Decide what each
+ scene needs, and write the music (`motion/music`).
+- An effect from one take with nothing under it: a click that should be a door. Layer it.
+- `loop` on an `` without `endAt` to stretch a short ambience: nothing loops, it plays once and stops dead
+ (the render warns). With `endAt` it repeats with a hard jump at each loop; tile copies with crossfades instead.
- Two `` in one composition: it throws.
- No ``: the sum plays as it is, at whatever loudness it adds up to.
-- Rewording a `sao()` prompt on every pass. Each new prompt makes a new sound .
+- Rewording a `sao()` prompt on every pass. Each new prompt makes a new sound.
- `style` on an ``: Audio has no picture and takes no style.
- Hand-placing dozens of game sounds from a log. Cue them from `useGameEvents` (`motion/games`).
- Writing music note by note in `` files, or a game's sounds by hand. Write them in Tone.js (`motion/music`):
instruments, the Transport, buses with ducking, and `` for one-shots.
-## Example: a bed that lifts for the end card, a voice, foley on two beats
+## Example: a cue that lands on the end card, a voice, a layered hit
```tsx
import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, sao, staticFile } from "raea";
@@ -106,24 +127,44 @@ import { css } from "./styles.gen";
const FRAMES = 300;
const END_CARD = 252; // the voice stops here
-// the bed: a TokenSynth pulse and a pad on the music bus, ducked under the voice, lifted for the end card
-Tone.getTransport().bpm.value = 100;
+// the music (motion/music): a sampled piano, chords that change every bar under a motif that answers the voice,
+// ducked under it, lifted for the end card, and ending on a chord on that cut. The tempo is set from the cut:
+// four bars to the end card, so its cut is a downbeat.
+Tone.getTransport().bpm.value = (4 * 4 * 60 * 30) / END_CARD; // about 114 bpm
Tone.buses.music.duck(Tone.buses.voice, { amount: -12 });
Tone.buses.music.volume.value = -21;
Tone.buses.music.volume.rampTo(-12, 0.33, END_CARD / 30);
-const pluck = new Tone.Sampler({ urls: Tone.tokensynth("a soft muted electric piano, warm, short decay", ["C3", "C4"], { seconds: 1 }) }).connect(Tone.buses.music);
-const pad = new Tone.PolySynth(Tone.Synth, { oscillator: { type: "fatsine" }, envelope: { attack: 0.6, release: 1.5 }, volume: -12 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => pluck.triggerAttackRelease(n, "8n", t), ["A3", "E4", "C4", "E4"], "8n").start(0);
-new Tone.Part((t, c) => pad.triggerAttackRelease(c, "2m", t), [["0:0", ["A3", "C4", "E4"]], ["2:0", ["F3", "A3", "C4"]]]).start(0);
+// `raea sounds get salamander/piano` printed its roots a minor third apart (D#2 is the file Ds2.mp3); these cover
+// F2 to G5, what the cue plays
+const ROOTS = ["D#2", "F#2", "A2", "C3", "D#3", "F#3", "A3", "C4", "D#4", "F#4", "A4", "C5", "D#5", "F#5"];
+const piano = new Tone.Sampler({
+ urls: Object.fromEntries(ROOTS.map((n) => [n, staticFile(`sounds/salamander-piano/${n.replace("#", "s")}.mp3`)])),
+ release: 1,
+}).connect(new Tone.Reverb({ decay: 1.4, wet: 0.2 }).connect(Tone.buses.music));
+const CHORDS = [["A2", "E3", "C4"], ["F2", "C3", "A3"], ["C3", "G3", "E4"], ["G2", "D3", "B3"]];
+new Tone.Part((t, c) => piano.triggerAttackRelease(c.notes, "4n", t, c.vel), CHORDS.flatMap((chord, bar) => [
+ { time: `${bar}:0`, notes: chord, vel: 0.55 },
+ { time: `${bar}:2:2`, notes: chord, vel: 0.4 },
+])).start(0);
+new Tone.Part((t, n) => piano.triggerAttackRelease(n.note, "8n", t, n.vel), [
+ { time: "1:1", note: "E5", vel: 0.7 }, { time: "1:1:2", note: "D5", vel: 0.5 }, { time: "1:2", note: "C5", vel: 0.6 }, { time: "1:3", note: "A4", vel: 0.65 },
+ { time: "3:1", note: "G5", vel: 0.75 }, { time: "3:1:2", note: "F5", vel: 0.5 }, { time: "3:2", note: "E5", vel: 0.6 }, { time: "3:3", note: "D5", vel: 0.7 },
+]).start(0);
+Tone.getTransport().schedule((t) => piano.triggerAttackRelease(["A2", "E3", "A3", "C4", "E4"], "1m", t, 0.8), "4m");
Tone.getTransport().start();
+// one hit in three layers, on the same frame: the snap that starts it, the body that says what it is, the room's tail
+const SNAP = sao("sharp wooden snap, close, very short, dry", { seconds: 0.3 });
const HIT = sao("deep punchy impact hit, tight, no reverb tail", { seconds: 1.2 });
+const TAIL = sao("the reverb tail of one impact in a large hall, no attack", { seconds: 2.5 });
const POP = sao("bubbly UI pop click, short, dry", { seconds: 0.6 });
const VOICE = staticFile("voice/vo01.wav"); // the person's narration, as a file
/** Every sound, in time order. */
const SOUNDS = [
+ { id: "hit-snap", from: 0, gainDb: -20, src: SNAP, fade: 0, bus: "sfx" },
{ id: "hit", from: 0, gainDb: -16, src: HIT, fade: 0, bus: "sfx" },
+ { id: "hit-tail", from: 0, gainDb: -24, src: TAIL, fade: 0, bus: "sfx" },
{ id: "voice", from: 15, to: END_CARD, gainDb: 0, src: VOICE, fade: 0.012, bus: "voice" },
{ id: "pop", from: END_CARD + 2, gainDb: -17, src: POP, fade: 0, bus: "sfx" },
] as const;
diff --git a/directory/skills/raea/motion/sources.md b/directory/skills/raea/motion/sources.md
new file mode 100644
index 0000000..f76d5e5
--- /dev/null
+++ b/directory/skills/raea/motion/sources.md
@@ -0,0 +1,238 @@
+# Sources: the local models, sample libraries, effect packs and MIDI, and how to get each onto the Mac
+
+## What to do
+
+- Take each sound from the source that fits its job, and know its licence before it goes in:
+
+ | the job | the source |
+ |---|---|
+ | an instrument playing a part: piano, strings, brass, woodwinds, mallets, guitars, a drum kit | a sampled library: `raea sounds` first, then the packs below |
+ | a timbre no library has ("a bowed saw, close") | TokenSynth, curated (`motion/music`) |
+ | an effect, foley, an ambience | a recorded pack or the person's own recordings first; SAO for what none has, in layers (`motion/sound`) |
+ | a voice over the picture | `` (Pocket TTS, local); a voice model is a `raea gen speech` step, Fish's with the person's own FISH_API_KEY (`motion/gen`) |
+ | a drummer's feel | a Groove MIDI performance played on a sampled kit |
+ | a melody or harmony to arrange from the public domain | Mutopia, OpenScore |
+
+- Use what raea's licence rule takes: CC0 and public domain, CC-BY (credited), MIT and Apache, and free licences that
+ allow a commercial film (Sonniss's, Pixabay's, Mixkit's for its effects). Never a non-commercial (NC),
+ no-derivatives (ND), Sampling+, "personal use", share-alike (SA) or GPL source: share-alike and GPL terms can reach
+ the film or game made with it. A pack's licence is per pack and sometimes per sound; read its readme and keep it
+ beside the files.
+- Credit each CC-BY source where the piece is shown (its description or end credits). The line is in the
+ `LICENCE.txt` that `raea sounds get` writes, or in the pack's readme.
+- Keep a pack fetched by hand in one folder on the Mac, outside any project (`~/Music/raea-sounds/PACK/` with its
+ licence file), and copy the files a piece uses into its `media/` (`media/sounds/PACK/`): a render reads only its
+ own media, never a URL. With the raea app open, `raea library new sounds` and `raea library add sounds
+ ~/Music/raea-sounds` index that folder once the person says yes in the app; then `raea library search "metal door
+ slam" --kind audio --sound` finds a sound by what it sounds like.
+- raea reads WAV, MP3, FLAC and AAC (.m4a). Convert Ogg (most of Kenney's packs) and Opus to WAV first (`ffmpeg -i
+ in.ogg out.wav`).
+
+## Getting SAO and TokenSynth onto the Mac
+
+Both run on this computer and install themselves the first time a render, `raea sao` or `raea tokensynth` asks for
+a sound. To do it ahead of time and see where they are:
+
+```sh
+raea setup-audio --check # what's installed, where, and how much disk
+raea setup-audio # installs what's missing: TokenSynth, then SAO
+raea setup-audio sao # one of them
+```
+
+- **What it fetches.** Each model has a script (`TOOLS/sao/setup.sh` and `TOOLS/tokensynth/setup.sh`; TOOLS is
+ `tools/` in a raea checkout, `share/raea/tools/` beside an installed raea) that installs into `~/.cache/raea-sao`
+ and `~/.cache/raea-tokensynth` (RAEA_SAO and RAEA_TOKENSYNTH move them):
+ - SAO, about 2 GB: Stable Audio Open Small's Core ML conversion (the DiT, the VAE decoder and the number embedder)
+ from a Hugging Face mirror at a pinned revision, each file checked against its SHA-256; t5-base's tokenizer; a
+ small Python environment; and raea's own fp32 T5 encoder, converted once in a throwaway environment whose torch
+ and t5-base weights (about 900 MB) are deleted after.
+ - TokenSynth, about 6 GB: its code from GitHub, a Python environment with torch (about 1.1 GB), the model and its
+ unconditional twin (713 MB each), DAC (307 MB) and CLAP (2.35 GB) from Hugging Face, the text encoders CLAP
+ fetches, then a converted fp16 runtime and Core ML models.
+- **What it needs.** A network connection, `uv` (Astral's Python manager) and, for TokenSynth, `git`, which the
+ scripts take from the PATH. On a Mac without them, `TOOLS/ensure.sh` gets raea's pinned builds (uv 0.12.24 and git
+ 2.56.0, under `~/.cache/raea-tools`, each download checked against its SHA-256) and prints the program's path. Put
+ both on the PATH for the setup:
+
+ ```sh
+ B=$(dirname "$(realpath "$(command -v raea)")"); T=$B/../share/raea/tools; [ -d "$T" ] || T=$B/../../tools # an install, or a checkout
+ PATH="$(dirname "$(bash "$T/ensure.sh" uv)"):$(dirname "$(bash "$T/ensure.sh" git)"):$PATH" raea setup-audio
+ ```
+
+- **The licences.** TokenSynth's code and weights are MIT. Stable Audio Open Small is under Stability AI's Community
+ License (stability.ai/community-license-agreement), which isn't an open licence. As between the person and
+ Stability, the person owns the sounds it makes, as far as the law allows; the sounds may not be used to create or
+ improve another generative AI model, and Stability's acceptable use policy applies to them. The licence's other
+ duties fall first on raea, which hands the model on: a copy of the agreement and a notice file beside the model,
+ "Powered by Stability AI" where raea documents it, and registering raea's own commercial use. raea hasn't settled
+ those yet, nor what is left for the person (docs/sound.feature SN2), so don't tell the person what the licence asks
+ of them: before the first SAO sound goes into a commercial piece of theirs, say the model is under that licence,
+ give them the link and let them decide, or take that sound from a recorded pack instead.
+- **When it fails.** The error names the script and the folder; run that script with `sh` to see its own output.
+ `exit status: 127` is uv not found. On a Mac without the Command Line Tools, `/usr/bin/git` is only a stub: it
+ opens an install dialog, and the setup stops with status 1 and `xcrun: error: invalid active developer path`. Use
+ the git `ensure.sh` installs (above). `75` is another setup running in the same folder: wait for it, or remove the
+ folder's `.setup-lock` once nothing is running. RAEA_OFFLINE refuses the download. RAEA_AUDIO_AUTOSETUP=0 turns
+ the self-install off, so a render names the command instead; it also stops `raea setup-audio` and `ensure.sh`, so
+ with it set, run the model's script yourself (`sh TOOLS/sao/setup.sh`). Running setup-audio again picks up where a
+ broken install stopped: finished files are kept.
+- A render that meets a model not yet installed waits while it installs (minutes, once). Run setup-audio before the
+ first render, not inside one.
+
+## What `raea sounds` fetches for you
+
+`raea sounds WORDS` searches four libraries with no network (`kick`, `hihat` and the other words producers use find
+their parts). `raea sounds get PACK/INSTRUMENT` downloads one instrument's files once (into `~/.cache/raea-sounds`),
+copies them into `media/sounds/PACK-INSTRUMENT/` with a `LICENCE.txt`, and prints the `Sampler` (or the `Player`s)
+that plays them.
+
+| pack | licence | what it has |
+|---|---|---|
+| `salamander` | CC-BY 3.0 (credit) | a grand piano, sampled every minor third |
+| `fluid` | CC-BY 3.0 (credit) | the 128 General MIDI instruments of the FluidR3 font, every note A0 to C8; no drum kit |
+| `tonejs-instruments` | CC-BY 3.0 (credit) | 20, recorded from VSCO 2, the University of Iowa, Karoryfer and Freesound: cello, violin, contrabass, French horn, trumpet, trombone, tuba, saxophone, flute, clarinet, bassoon, harp, three guitars (acoustic, electric, nylon), electric bass, organ, harmonium, piano, xylophone |
+| `vcsl` | CC0 | 104 from the Versilian Community Sample Library: pianos, harpsichords, harps, organs, recorders, marimba, vibraphone, glockenspiel, kalimbas; bass drums, snares, toms, hi-hat, cymbals, claps, shakers, congas, timpani; a siren, a ratchet, an anvil and other effects |
+
+The ids are exact: `raea sounds kick` lists `vcsl/bass-drum-1`, and `vcsl/bass-drum` isn't one. A pitched
+instrument comes as one take a note, at its middle dynamic: with no `--notes`, roots at least a minor third apart from
+C1 to C7, the ones a Sampler needs, named as notes with sharps written `s` (`Ds3.mp3` is D#3). A drum or a hit comes
+as its first 12 takes (`--all` for every one). VCSL names a hit's dynamic and round robin in the file (a higher `_vN`
+is a harder hit, `_rr1`, `_rr2` the round robins), so a part can pick a softer take for a softer note.
+
+## Packs to fetch by hand
+
+Each goes into its own folder in `~/Music/raea-sounds/`, with its licence file beside it; the files a piece uses are
+then copied into its `media/`. These have a stable address, so one command fetches each (`git` is the one
+`TOOLS/ensure.sh git` prints on a Mac without the Command Line Tools):
+
+```sh
+P=~/Music/raea-sounds
+# VSCO 2 CE, the orchestra with its SFZ maps: 3.2 GB, CC0 (LICENSE inside)
+git clone --depth 1 --branch SFZ https://github.com/sgossner/VSCO-2-CE "$P/vsco2-ce"
+# VCSL, every dynamic and round robin, with SFZ maps: 6.2 GB, CC0 (the sfz branch has no LICENSE, so it's fetched beside)
+git clone --depth 1 --branch sfz https://github.com/sgossner/VCSL "$P/vcsl" && curl -fsSL -o "$P/vcsl/LICENSE" https://raw.githubusercontent.com/sgossner/VCSL/master/LICENSE
+# Salamander Grand Piano V3, 16 velocities: 0.75 GB, CC-BY 3.0 (LICENSE inside)
+git clone --depth 1 https://github.com/sfzinstruments/SalamanderGrandPiano "$P/salamander-v3"
+# Groove MIDI, the MIDI only: 3.1 MB, CC-BY 4.0, checked against the SHA-256 its page gives
+mkdir -p "$P/groove" && curl -fL -o "$P/groove/groove-midionly.zip" https://storage.googleapis.com/magentadata/datasets/groove/groove-v1.0.0-midionly.zip \
+ && echo "651cbc524ffb891be1a3e46d89dc82a1cecb09a57c748c7b45b844c4841dcc1e $P/groove/groove-midionly.zip" | shasum -a 256 -c \
+ && unzip -q "$P/groove/groove-midionly.zip" -d "$P/groove" \
+ && echo "Groove MIDI Dataset (Google Magenta), CC-BY 4.0: magenta.tensorflow.org/datasets/groove" > "$P/groove/LICENCE.txt"
+# MuseScore General, a General MIDI bank with drum kits: 40 MB as SF3, MIT, with its licence and notices
+mkdir -p "$P/musescore-general" && curl -fL --output-dir "$P/musescore-general" \
+ -O https://ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/MuseScore_General.sf3 \
+ -O https://ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/MuseScore_General_License.md
+```
+
+The rest are picked a pack or a sound at a time on their own pages. `raea sounds get` reaching all of them is filed
+as docs/sound.feature SN4.
+
+| source | licence | where | what it has |
+|---|---|---|---|
+| VSCO 2 Community Edition | CC0 | github.com/sgossner/VSCO-2-CE (its `SFZ` branch has the maps) | an orchestra: solo and section strings with articulations, woodwinds, brass, percussion, keys; WAV with SFZ maps |
+| VCSL, the whole library | CC0 | github.com/sgossner/VCSL (its `sfz` branch has the maps) | every dynamic and round robin of what `raea sounds` gives one take of |
+| Salamander Grand Piano V3 | CC-BY 3.0 | github.com/sfzinstruments/SalamanderGrandPiano | the same piano at 16 velocities, 48 kHz, SFZ |
+| sfzinstruments' lists | per pack, on the list | sfzinstruments.github.io (pianos, strings, drums) | take the CC0, public domain and CC-BY ones: Karoryfer's free packs (Big Rusty, Swirly and Unruly drums, Frankensnare, Meatbass, Bigcat Cello; mostly CC0, older ones CC-BY 3.0: read each readme), SM Drums (public domain), Splendid Grand Piano (public domain), DrumGizmo's DRS and Muldjord kits and Naked Drums (CC-BY 4.0) |
+| University of Iowa Musical Instrument Samples | "may be downloaded and used for any projects, without restrictions" | theremin.music.uiowa.edu/mis.html | single notes of orchestral instruments at several dynamics, recorded dry: give them a room's reverb |
+| FreePats | per bank: CC0, CC-BY or GPL; take the CC0 and CC-BY banks only | freepats.zenvoid.org | General MIDI families, world instruments, percussion; SFZ and SF2 |
+| MuseScore General | MIT (keep its notices) | ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/ | a General MIDI bank with drum kits, SF2 (216 MB) or SF3 (40 MB), for playing a MIDI file as it is |
+| Kenney's audio packs | CC0 | kenney.nl/assets/category:Audio | Impact Sounds, Interface Sounds, UI Audio, Sci-fi Sounds, RPG Audio, Casino Audio, Digital Audio, Music Jingles, voice-over packs; mostly Ogg |
+| OpenGameArt | per asset: take CC0, CC-BY and OGA-BY | opengameart.org | game effects and loops, such as "512 Sound Effects (8-bit style)" (CC0) |
+| Freesound | per sound: take CC0 and CC-BY only | freesound.org (a free account to download) | field recordings, foley and effects of every kind |
+| Sonniss #GameAudioGDC bundles | royalty-free for commercial use, no credit; never passed on as sounds (a library, a pack, a template) and never used to train a model | sonniss.com/gameaudiogdc | over 200 GB of professional foley, impacts, ambiences and field recordings, a bundle a year |
+| Pixabay sound effects | Pixabay's Content License: free, commercial use, no credit; never sold or shared on their own | pixabay.com/sound-effects | effects and ambiences |
+| Mixkit sound effects | Mixkit's Sound Effects Free License: free, commercial use (games and broadcast too), no credit; never passed on on their own (as stock, in a tool or a template) and never registered with a rights service such as Content ID | mixkit.co/free-sound-effects | effects, whooshes, interface sounds, ambiences |
+| Groove MIDI Dataset | CC-BY 4.0 (credit Google Magenta) | magenta.tensorflow.org/datasets/groove | 1,150 MIDI performances by 10 drummers: 13.6 hours of beats and fills in 18 styles |
+| Mutopia Project | per piece: public domain or CC-BY (skip the BY-SA ones) | mutopiaproject.org | 2,100 classical pieces as MIDI and LilyPond |
+| OpenScore Lieder | CC0 | github.com/OpenScore/Lieder | 1,350 art songs as MuseScore scores (MuseScore exports MIDI) |
+
+Not these, whatever they sound like: Sonatina Symphonic Orchestra and the Virtual Playing Orchestra (Sampling+ and
+mixed terms: no use in an ad), the Philharmonia's samples (terms unclear), Spitfire LABS (no samples to take), the
+MusyngKite and FatBoy fonts beside FluidR3 (CC BY-SA 3.0: share-alike), FreePats' GPL banks (copyleft), Tone.js's
+own drum kits (CC BY-NC-SA), the BBC's sound effects (personal and research use), Zapsplat (its own terms), Mixkit's
+stock music (not in games or broadcasts, and no remixing), Salamander Drumkit and other CC-BY-SA kits, and Lakh MIDI
+(copyrighted songs). Freesound's API is free only for non-commercial use, so raea doesn't search Freesound for the
+person: they download its sounds themselves.
+
+## Loading a pack in a composition
+
+- **One-shots** (an effect pack, a field recording): the file in `media/sfx/PACK/`, then ` ` in a Sequence at its frame (`motion/sound`), or a `Tone.Player`
+ for a game's one-shot.
+- **A multisampled instrument** (the SFZ packs): raea has no SFZ reader yet. The `.sfz` file is text: each ``
+ names a `sample=` file, its `pitch_keycenter` (the note it is), `lovel`/`hivel` (the velocities it plays) and
+ `seq_position` (its round robin). Copy the files for the notes and dynamics the part needs, map them by note, and
+ give each dynamic its own Sampler, picked by the note's velocity:
+
+ ```tsx
+ // SOFT and LOUD: { C3: staticFile("sounds/vsco-cello/..."), ... }, written from the .sfz's regions
+ const soft = new Tone.Sampler({ urls: SOFT, release: 0.6 }).connect(Tone.buses.music);
+ const loud = new Tone.Sampler({ urls: LOUD, release: 0.6 }).connect(Tone.buses.music);
+ const cello = (note: string, dur: string, t: number, vel: number) => (vel < 0.6 ? soft : loud).triggerAttackRelease(note, dur, t, vel);
+ ```
+
+- **A MIDI file** (a Groove MIDI performance, a Mutopia piece): turned into JSON and played on Samplers and Players
+ (below).
+- **A SoundFont** (`.sf2`, `.sf3`): raea can't play one. To hear a MIDI file the person hands over as it is, render
+ it with FluidSynth (`brew install fluid-synth`, then `fluidsynth -ni -F song.wav -r 48000 MuseScore_General.sf3
+ song.mid`) and play the WAV as ``; to write parts, use the sampled libraries instead.
+
+## A MIDI file in a composition
+
+A Groove MIDI performance or a Mutopia piece comes as a `.mid`. raea/music reads one (`parseMidi`, in
+sdk/audio/src/midi.ts), and the raea app imports a `.mid` into a listen item's Part, but a composition can't import
+a `.mid` yet (docs/sound.feature SN6). Turn it into JSON in raea/music's `MidiFile` shape (`tracks[].notes[]` with
+`midi`, `time` and `duration` in seconds, and `velocity` 0 to 1), each note from its note-on to its note-off, one
+track per channel; a composition imports that, and its code stays the same when a `.mid` can be imported:
+
+```sh
+uv run -q --no-project --with mido python3 -I - media/grooves/funk-92.mid media/grooves/funk-92.json <<'PY'
+import json, sys, mido
+f = mido.MidiFile(sys.argv[1])
+tempo, ticks, secs, held, tracks, tempos, sigs = 500000, 0, 0.0, {}, {}, [], []
+def end(ch, note, at, t, v): # one note, from its note-on to now, as raea/music's MidiNote
+ tracks.setdefault(ch, []).append({"midi": note, "time": round(at, 4), "duration": round(secs - at, 4), "velocity": round(v / 127, 3), "ticks": t, "durationTicks": ticks - t})
+for m in mido.merge_tracks(f.tracks):
+ ticks += m.time
+ secs += mido.tick2second(m.time, f.ticks_per_beat, tempo)
+ if m.type == "set_tempo":
+ tempo = m.tempo
+ tempos.append({"ticks": ticks, "bpm": round(mido.tempo2bpm(m.tempo), 3)})
+ elif m.type == "time_signature":
+ sigs.append({"ticks": ticks, "timeSignature": [m.numerator, m.denominator]})
+ elif m.type == "note_on" and m.velocity:
+ held.setdefault((m.channel, m.note), []).append((secs, ticks, m.velocity))
+ elif m.type in ("note_on", "note_off") and held.get((m.channel, m.note)):
+ end(m.channel, m.note, *held[(m.channel, m.note)].pop(0))
+for (ch, note), left in held.items(): # still sounding when the file ends
+ for n in left:
+ end(ch, note, *n)
+out = [{"name": f"channel {ch + 1}", "channel": ch, "notes": sorted(ns, key=lambda n: n["time"])} for ch, ns in sorted(tracks.items())]
+json.dump({"header": {"ppq": f.ticks_per_beat, "tempos": tempos, "timeSignatures": sigs, "name": ""}, "tracks": out, "duration": round(secs, 4)}, open(sys.argv[2], "w"))
+PY
+```
+
+A drum track plays on a kit of Players by note number. Groove MIDI was recorded on a Roland kit, whose map differs
+from General MIDI's: its hi-hat edge hits are 22 (closed) and 26 (open), notes General MIDI doesn't have, so a
+General MIDI map drops many of its hi-hats:
+
+```tsx
+import groove from "./media/grooves/funk-92.json"; // seconds, at the drummer's 92 bpm
+// Roland's drum map (the dataset's page lists the toms and cymbals): 36 kick; 37, 38 and 40 snare (cross-stick,
+// head, rim); 42, 22 and 44 closed hi-hat (bow, edge, pedal); 46 and 26 open hi-hat (bow, edge)
+const KIT: Record = { 36: "kick", 37: "snare", 38: "snare", 40: "snare", 42: "hat", 22: "hat", 44: "hat", 46: "open", 26: "open" };
+const kit = new Tone.Players({
+ kick: staticFile("sounds/vcsl-bass-drum-1/BDrumNew_hit_v5_rr1_Sum.wav"),
+ snare: staticFile("sounds/vcsl-snare-drum-modern-1/Snare2_HitSN_v5_rr1_Mid.wav"),
+ hat: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v3_rr1_Mid.wav"),
+ ghost: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v1_rr1_Mid.wav"),
+ open: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitO_rr1_Mid.wav"),
+}).connect(Tone.buses.music);
+const stretch = 92 / Tone.getTransport().bpm.value; // the drummer's tempo onto the cue's
+const hits = groove.tracks.flatMap((track) => track.notes).filter((n) => n.midi in KIT);
+// a soft closed hat plays the soft take
+new Tone.Part((t, n) => kit.player(KIT[n.midi] === "hat" && n.velocity < 0.45 ? "ghost" : KIT[n.midi]).start(t), hits.map((n) => ({ ...n, time: n.time * stretch }))).start("4m");
+```
+
+A pitched track (a Mutopia piece) plays on a Sampler, one Part a track, each note at its own length:
+`piano.triggerAttackRelease(Tone.Frequency(n.midi, "midi").toNote(), n.duration * stretch, t, n.velocity)`.
diff --git a/directory/skills/raea/reference/cli.md b/directory/skills/raea/reference/cli.md
index 301bff2..f9767c3 100644
--- a/directory/skills/raea/reference/cli.md
+++ b/directory/skills/raea/reference/cli.md
@@ -132,10 +132,30 @@ Flags: `--as` (1), `--ask` (1), `--choice` (1), `--clip` (1), `--done` (1), `--i
Access: HEADLESS. Arguments besides flags: any number.
```
-raea clothes [make NAME ["WHAT IT IS"] [--plain] [--set PATH=VALUE,...] [--colour #RRGGBB] | settings | cover NAME --parts PART[:SHARE],... --colour #RRGGBB [--stand MM] [--over UNDER] | wear OUT_DIR GARMENT... [--body DIR] [--head HEAD_DIR]] (clothes for SOMA's body from words: make has an image model (flux_2_pro, one picture a view, at most N cents each, 10 unless said) dress SOMA's grey mannequin from the front, back, left and right, then measures the pictures onto SOMA's own vertices as assets/clothes/NAME.json, its pictures in assets/clothes/NAME/; --over draws it over garments already made (innermost first), as the next layer, a garment made by procedure drawn on the mannequin first; remeasure measures a garment again from its kept pictures (after raea's measuring changed), nothing drawn or paid for; cover makes one by procedure in the same file, no picture: the skin of the parts named (hips, waist, torso, chest, neck, head, upperArm, foreArm, hand, thigh, shin, foot, toes; a limb's SHARE is how far along it from its top), in one colour, MM off the skin (6 unless said); wear puts garments on a body, innermost first (each GARMENT a name or its .json): OUT_DIR/NAME.glb each, with OUT_DIR/body.glb the body without the skin wholly under cloth; --body DIR another SOMA body folder (a headless one for a head part); --head HEAD_DIR the head that joins it (head.json), whose skin the body's is graded to at the collar; alone, the project's garments. The first use fetches SOMA's body (NVlabs/SOMA-X, Apache-2.0) into raea's models folder and needs Blender: raea setup blender)
+raea clothes [make NAME ["WHAT IT IS"] [--plain] [--set PATH=VALUE,...] [--colour #RRGGBB] | settings | cover NAME --parts PART[:SHARE],... --colour #RRGGBB [--stand MM] [--over UNDER] [--hard [--sole MM] [--sole-colour #RRGGBB] [--panels #RRGGBB,...]] | wear OUT_DIR GARMENT... [--body DIR] [--head HEAD_DIR]] (clothes for SOMA's body from words: make sews a garment: a sewing pattern (GarmentCode, MIT) from the words, or the plain base with --plain, its settings changed by --set PATH=VALUE (settings lists them and what each may be), cut to SOMA's measurements, draped by raea's own cloth and baked, as assets/clothes/NAME.json and its .glb, in --colour or the palette's next; the first make sets up GarmentCode and its Python in raea's models folder; cover makes one by procedure in the same file, no picture: the skin of the parts named (hips, waist, torso, chest, neck, head, upperArm, foreArm, hand, thigh, shin, foot, toes; a limb's SHARE is how far along it from its top), in one colour, MM off the skin (6 unless said), --over a garment already made as the one under it; wear puts garments on a body, innermost first (each GARMENT a name or its .json): OUT_DIR/NAME.glb each, with OUT_DIR/body.glb the body without the skin wholly under cloth; --body DIR another SOMA body folder (a headless one for a head part); --head HEAD_DIR the head that joins it (head.json), whose skin the body's is graded to at the collar; alone, the project's garments. The first use fetches SOMA's body (NVlabs/SOMA-X, Apache-2.0) into raea's models folder and needs Blender: raea setup blender)
```
-Flags: `--body` (1), `--colour` (1), `--head` (1), `--over` (1), `--parts` (1), `--plain`, `--set` (1), `--stand` (1).
+Flags: `--body` (1), `--colour` (1), `--hard`, `--head` (1), `--over` (1), `--panels` (1), `--parts` (1), `--plain`, `--set` (1), `--sole` (1), `--sole-colour` (1), `--stand` (1).
+
+## raea motion
+
+Access: HEADLESS. Arguments besides flags: 1.
+
+```
+raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this Mac by Kimodo (NVIDIA, on MLX): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads; the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters); a line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT)
+```
+
+Flags: `--frames` (1), `--keys` (1), `--seconds` (1), `--seed` (1), `--steps` (1), `-o` (1).
+
+## raea bug
+
+Access: HEADLESS. Arguments besides flags: any number.
+
+```
+raea bug TITLE... --text TEXT [--command CMD] [--error TEXT] [--agent NAME] [--dry-run] (a bug in raea itself, not in your own composition, sent to raea's server with raea's version and the OS, no account needed: words only, paths, emails and keys taken out before it goes; five a day from a machine, the same bug again counted on the first report; --dry-run prints what would go and sends nothing)
+```
+
+Flags: `--agent` (1), `--command` (1), `--dry-run`, `--error` (1), `--text` (1).
## raea spec
diff --git a/directory/skills/raea/reference/feed.md b/directory/skills/raea/reference/feed.md
index f898d21..ef0aa58 100644
--- a/directory/skills/raea/reference/feed.md
+++ b/directory/skills/raea/reference/feed.md
@@ -25,6 +25,7 @@ Post an item to the project's feed: a render to watch (.mp4), a still to look at
| `changed` | optional | with replaces: one short line on what this version changed ("title bigger, cut the second shot"); the card shows it beside the new picture |
| `answers` | optional | with replaces: the ids of the person's comments on that item this version settles; each shows done, linked to this version |
| `export` | optional | true when this is a deliverable (an export for the client or a platform): it shows under Exports in the project's Table |
+| `final` | optional | true when this version is ready to share with reviewers, as you'd send it: it's tagged final and offered first when the person makes a review link. Say so only when it is, after the person's notes on it are settled |
| `tags` | optional | tags; key:value tags (platform:tiktok, aspect:9x16, cut:15s, market:uk) become the Table's columns, so use the same keys across a project's exports |
| `uses` | optional | the library files it was made from (footage, stills, music) |
| `controls` | optional | play: the game's controls |
@@ -59,7 +60,7 @@ Mark the person's comments on an item as you work them: each edit comment they s
Access: PAGE.
-Set the tags of an item already in the project's feed (feed_items lists them with their tags), to give the project's Table its structure: key:value tags (platform:tiktok, aspect:9x16) are its columns, and the tag export makes the item a deliverable. Replaces the item's own tags.
+Set the tags of an item already in the project's feed (feed_items lists them with their tags), to give the project's Table its structure: key:value tags (platform:tiktok, aspect:9x16) are its columns, the tag export makes the item a deliverable, and the tag final says this version is ready to share (it's offered first for a review link; put it on the newest version only, once it is). Replaces the item's own tags.
| input | | |
|---|---|---|
@@ -323,6 +324,20 @@ Change a live review link: its title, how many more days it works (from now), do
| `parent` | optional | the comment it answers |
| `resolve` | optional | a comment's id to mark resolved |
+## bug_report
+
+Access: HOST.
+
+Tell raea about a bug you found in raea itself: a raea command that fails or hangs, a wrong result, help or a guide that says one thing while raea does another. Not for a bug in your own composition. Words only: paths, emails and keys are taken out before it's sent, and the same bug again is counted on the first report. Then carry on with a workaround.
+
+| input | | |
+|---|---|---|
+| `title` | required | what's wrong, in a line |
+| `text` | required | what you did, what you expected, what happened |
+| `command` | optional | the raea command that went wrong, when there was one |
+| `error` | optional | what it printed |
+| `dry_run` | optional | show what would be sent, and send nothing |
+
## storage_status
Access: HOST.
diff --git a/directory/skills/raea/reference/files.md b/directory/skills/raea/reference/files.md
index cef7b14..78d85f0 100644
--- a/directory/skills/raea/reference/files.md
+++ b/directory/skills/raea/reference/files.md
@@ -64,9 +64,10 @@ that moves each (when there is one) and the files that name them.
| `~/.raea/cache` | `RAEA_EMOJI3D_CACHE` | apps/app/host/emoji3d.ts, apps/app/host/take.ts |
| `~/.raea/env` | `RAEA_ENV_FILE` | apps/app/host/director.ts, apps/app/host/plugins.ts, io/gen-core/src/fal.rs, io/gen-core/src/fish.rs, io/gen-core/src/made.rs, io/gen-core/src/modal.rs, io/gen-core/src/openrouter.rs, io/gen-core/src/replicate.rs, io/gen-core/src/tripo.rs, io/home/src/lib.rs |
| `~/.raea/evals/skills` | | apps/app/host/evals/skills.ts |
-| `~/.raea/feed` | | apps/app/host/feed-items.ts, apps/app/host/fingerprint.ts, apps/app/host/storage/owners.ts |
+| `~/.raea/feed` | | apps/app/host/feed-items.ts, apps/app/host/fingerprint.ts, apps/app/host/rules-asks.ts, apps/app/host/rules-train.ts, apps/app/host/storage/owners.ts |
+| `~/.raea/feed/i-taste-` | | apps/app/host/rules-asks.ts |
| `~/.raea/feed/media` | | apps/app/host/paths.ts |
-| `~/.raea/fingerprints` | `RAEA_FINGERPRINTS` | apps/app/host/fingerprint.ts |
+| `~/.raea/fingerprints` | `RAEA_FINGERPRINTS` | apps/app/host/fingerprint.ts, apps/app/host/rules-train.ts, apps/app/host/taste.ts |
| `~/.raea/fingerprints/default` | | apps/app/host/fingerprint.ts |
| `~/.raea/inbox` | `RAEA_INBOX_DIR` | apps/app/host/inbox.ts, engine/cli/src/show.ts |
| `~/.raea/inbox/INBOX.jsonl` | | engine/cli/src/cli.rs |
@@ -87,12 +88,13 @@ that moves each (when there is one) and the files that name them.
| `~/.raea/plugins.json` | `RAEA_PLUGINS_FILE` | apps/app/host/plugins.ts |
| `~/.raea/publish/drive.json` | `RAEA_DRIVE_PICKS` | apps/app/host/publish-source.ts |
| `~/.raea/publish/ready.json` | `RAEA_PUBLISH_READY` | apps/app/host/publish-ready.ts |
-| `~/.raea/sessions` | `RAEA_SESSIONS_DIR` | apps/app/host/sessions.ts |
+| `~/.raea/renders` | | apps/app/host/renderProgress.ts, runtime/encode/src/progress.rs |
+| `~/.raea/sessions` | `RAEA_SESSIONS_DIR` | apps/app/host/fingerprint-uses.ts, apps/app/host/sessions.ts |
| `~/.raea/storage-log.jsonl` | `RAEA_STORAGE_LOG` | apps/app/host/storage/trash.ts |
| `~/.raea/storage-used.json` | `RAEA_STORAGE_USED` | apps/app/host/storage/back.ts |
| `~/.raea/storage.json` | `RAEA_STORAGE_FILE` | apps/app/host/storage/plan.ts, apps/app/host/storage/service.ts |
-| `~/.raea/taste` | `RAEA_TASTE` | apps/app/host/taste.ts, engine/cli/src/spec.rs |
-| `~/.raea/taste/corrections.jsonl` | | apps/app/host/taste.ts |
+| `~/.raea/taste` | `RAEA_TASTE` | apps/app/host/rules.ts, apps/app/host/taste.ts, engine/cli/src/spec.rs |
+| `~/.raea/taste/corrections.jsonl` | | apps/app/host/rules-asks.ts, apps/app/host/rules-file.ts, apps/app/host/taste.ts |
| `~/.raea/taste/taste.ts` | | engine/cli/src/guide.rs |
| `~/.raea/trash` | `RAEA_TRASH_DIR` | apps/app/host/storage/trash.ts, apps/app/host/storage/usage.ts |
| `~/.raea/update` | `RAEA_UPDATE_DIR` | apps/app/host/onboard.ts, apps/app/host/update.ts |
diff --git a/skills/raea/3d/games.md b/skills/raea/3d/games.md
index f0ceb91..3a04dd6 100644
--- a/skills/raea/3d/games.md
+++ b/skills/raea/3d/games.md
@@ -236,10 +236,12 @@ function Hop() {
registerRoot(() => );
```
-## Sound: one-shots on transitions, a bed under the voice
+## Sound: one-shots on transitions, music ducked under the voice
-A game's sound is Tone.js on raea (`motion/music` has the whole of it). Build the instruments at the top level, give
-the bed and the sounds their buses, and fire each one-shot on the frame its event happens:
+A game's sound is Tone.js on raea (`motion/music` has the whole of it). Every input and outcome gets a sound, the
+world's own sounds (machines, footsteps, a room's tone) sit where they are, and music, when the game has some, is a
+loop written to come round, not a pad. Build the instruments at the top level, give the music and the sounds their
+buses, and fire each one-shot on the frame its event happens:
- `useGameEvents(INITIAL, fold, (before, after) => machine.state(after.hero) === "land" && machine.state(before.hero) !== "land")`
gives the frames where a transition happened. ` thump.triggerAttackRelease("C2", "8n", t)} />`
@@ -267,4 +269,5 @@ function Game() {
}
```
-sdk/react/bench/tone/game is a working game with all of this (a machine, a ducked bed, an SAO coin, a voice line).
+sdk/react/bench/tone/game is a working game with all of this (a machine, a chiptune loop that asks and answers,
+ducked under a voice line, an SAO coin).
diff --git a/skills/raea/craft/floor.md b/skills/raea/craft/floor.md
index 7e93c92..120ac9f 100644
--- a/skills/raea/craft/floor.md
+++ b/skills/raea/craft/floor.md
@@ -61,6 +61,8 @@ direction asks for it by name, and the direction then says why in a line.
- **Template captions.** White bold text with one yellow word, an emoji pop, a zoom on every sentence, a whoosh on
every entrance.
- **The same move everywhere.** One entrance, one zoom or one sound on every beat makes every beat weaker.
+- **Music by default.** A bed under every second, or a held pad and drone standing in for a score. Decide what each
+ scene is heard as, and compose what it needs (`motion/music`).
- **Copy that sounds like an ad.** Buzzwords, "Not X. Y." cadences, aphorisms, a label above every headline.
## Never
diff --git a/skills/raea/craft/modes/story.md b/skills/raea/craft/modes/story.md
index db6a9d9..2c7c43f 100644
--- a/skills/raea/craft/modes/story.md
+++ b/skills/raea/craft/modes/story.md
@@ -33,8 +33,9 @@ they want the whole film.
moves. The easy default is choosing a look shot by shot from references, and that ends up as several films.
- **Each shot's subject.** Choose the person, object or action each shot is about. The easy default is a wide
establishing shot, so question every one.
-- **What the music does.** It either leads, and you cut to its map, or it sits under the picture as a score. The easy
- default is a cut on every bar.
+- **What the music does.** It either leads, and you cut to its map, or it scores the picture, written to the cut, or a
+ scene has none and belongs to its own sound or to silence. The easy defaults are a cut on every bar and a score
+ that is one held chord.
- **The poster frame.** Choose it before anything renders, and build the edit toward it.
## The rut
diff --git a/skills/raea/craft/rubric.md b/skills/raea/craft/rubric.md
index eef3827..602b074 100644
--- a/skills/raea/craft/rubric.md
+++ b/skills/raea/craft/rubric.md
@@ -25,14 +25,32 @@ piece's modes isn't asked.
- land (pitch, story, explain): Does the end land: something this audience can do next, a payoff, or a return to the opening?
- pace (all): Do shot lengths change with what they hold, with no stretch sitting still longer than it earns, and cuts falling on words or beats?
- read (all): Is every word that matters large, sharp against what is behind it, and on long enough to read?
-- sound (all): Is the voice clear over everything else, with music and effects marking moments rather than filling space?
-- sound-silent (silent): With no voice, do the music and the effects carry it: cuts and moments landing on the beat, each effect marking a moment rather than filling space, nothing muddy? (asked in place of sound when no one speaks)
+- sound (all): Is the voice clear over everything else, with music and effects marking moments rather than filling space, and any music written for this cut (a tune, harmony that moves, a rhythm, an ending on a cut) rather than a pad or drone held under it?
+- sound-silent (silent): With no voice, do the music and the effects carry it: cuts and moments landing on the beat, the music composed (a tune, moving harmony, a rhythm, sections that change with the picture), each effect marking a moment rather than filling space, nothing muddy? (asked in place of sound when no one speaks)
- world (all): Do colour, type, motion and sound hold one direction from the first frame to the last?
- own (all): With the name covered, could this only be this project, rather than any product in its category?
- feel (play): Does the player's input change the picture right away and in a way they can read?
- again (play): After a failure, is it clear what went wrong and is the next attempt one press away?
- task (tool): Can the task the screen exists for be done without stopping to work out where to look?
+## Music that sounds generated
+
+Listen for these before the decision model does; each is how a made score gives itself away, and the fix
+(`motion/music`):
+
+- **One held texture.** A pad, a sine sub and a long reverb under everything, no attack anywhere. Write a motif, chords
+ that move and a part with rhythm; reverb per instrument, nothing on the bus.
+- **Harmony that stands still.** One chord for four bars or more in a cue that's meant to move. Change it every bar or two.
+- **A loop, not a cue.** The same two bars for thirty seconds. Vary the motif, change the arrangement every four to
+ eight bars, and turn where the edit turns.
+- **Played by a machine.** Every note at one velocity, exactly on the grid. Spread the dynamics; move played parts a
+ few milliseconds; use a recorded groove.
+- **No ending.** A fade-out. End on a chord or a hit on the last cut.
+- **Everywhere.** Music under every scene because the piece "needs music". Some scenes belong to silence or to their
+ own sound.
+- **The wrong source.** A whole score from TokenSynth takes (organ-like sustains, buzz) or from an SAO "ambient
+ music" prompt. Recorded instruments play the parts; TokenSynth fills a timbre no library has; SAO makes effects.
+
## Ranking
| | Means | Do |
diff --git a/skills/raea/craft/verbs/quieter.md b/skills/raea/craft/verbs/quieter.md
index c46f1c4..179ce89 100644
--- a/skills/raea/craft/verbs/quieter.md
+++ b/skills/raea/craft/verbs/quieter.md
@@ -18,7 +18,7 @@ note names. Keep the point of view: a quieter piece still has one.
- Light: fewer sources pulsing, shorter pulses, lower bloom. Glow only where the eye should go.
- Colour: lower saturation on what's around the accent, not on the accent.
- Motion: fewer moving things at once, smaller moves, no bounce.
-- Sound: duck the bed further under the voice; drop layers before lowering levels.
+- Sound: duck the music further under the voice; drop parts and layers before lowering levels.
- Never solve quiet with blur, fog or grain that softens the work itself.
## Verify
diff --git a/skills/raea/craft/verbs/remix.md b/skills/raea/craft/verbs/remix.md
index 4a65ab3..b43bb39 100644
--- a/skills/raea/craft/verbs/remix.md
+++ b/skills/raea/craft/verbs/remix.md
@@ -5,18 +5,23 @@ sound: voice, music, foley and the master.
## Assess
- Listen once for the voice alone: is every word clear over everything else?
-- Then listen for what's filling space rather than marking moments: a bed at one level under everything, a whoosh on
+- Then listen for what's filling space rather than marking moments: music at one level under everything, a whoosh on
every entrance.
+- Then for what the music is: a tune you could hum, chords that move, a part with rhythm, an ending on a cut; or one
+ held texture (a pad, a drone) under a long reverb.
- With a song, check the cuts and hits against its map.
## Decide
+- What each scene is heard as: silence, its own sound, source music, score. Some scenes need no music.
- What leads: the voice, or the music. Everything else sits under it.
- Which moments get a sound. A few marked moments beat a sound on every one.
## Change
- The voice first; duck music and foley under it.
- Sound marks moments: a hit on the cut that matters, silence before the drop. Remove what only fills.
-- Music: a library instrument first; TokenSynth takes are curated, not trusted. Under a voice, a sparse score.
+- Music: composed for the cut (`motion/music`) on recorded instruments first; TokenSynth takes are curated, not
+ trusted. Under a voice, thin the arrangement and let the tune answer between lines.
+- Effects: layered (a transient, a body, a tail), with repeats varied (`motion/sound`).
- Master at -14 LUFS integrated, true peak at or below -1 dBTP.
## Verify
@@ -24,6 +29,8 @@ sound: voice, music, foley and the master.
## When it goes wrong
- The voice is clear on headphones and buried on a phone's speaker.
-- The music pumps as it ducks under the voice. Use a slower release, or a lower bed.
+- The music pumps as it ducks under the voice. Use a slower release, or lower music.
+- The score has turned into a drone: one chord held under a long reverb. Move the harmony, add a part with rhythm,
+ and take the reverb off the bus.
- Clipping, or a true peak over the limit. Measure, and don't trust the meter by eye.
- The loudness is off the delivery's spec.
diff --git a/skills/raea/motion.md b/skills/raea/motion.md
index bdb59fd..c964e8d 100644
--- a/skills/raea/motion.md
+++ b/skills/raea/motion.md
@@ -18,7 +18,8 @@ Read the files a task needs with `raea guide motion/NAME`, one or two at a time,
| planning how shots cut into each other: hooks on the thirds, the eye carried through each cut | `motion/match-cuts` |
| moving anything: entrances, exits, counters, push-ins | `motion/animating` |
| cutting takes, framing a face, trimming, clip sound | `motion/footage` |
-| music beds, foley, gains, fades, the master | `motion/sound` (the music itself: `motion/music`) |
+| what each scene is heard as, effects and foley, gains, fades, the master | `motion/sound` (the music itself: `motion/music`) |
+| where sounds come from: the local models and their setup, sample libraries, effect packs, MIDI | `motion/sources` |
| captions, or anything cued to what someone says | `motion/words` |
| putting parts over a face, platform zones, the proof | `motion/placement` |
| cards, stacks, type sizes, fonts | `motion/layout` |
@@ -63,9 +64,11 @@ Read the files a task needs with `raea guide motion/NAME`, one or two at a time,
(`text-brand-accent`, `bg-brand-surface`, `font-brand`, `text-title`, `rounded-brand` and the rest in AGENTS.md),
or `var(--brand-*)` where a class can't reach. Never write a hex palette of your own. A role the kit leaves empty
is yours to decide from the brand's own colours (`motion/brand`).
-8. **Sound is part of the edit.** Each sound goes in its own ``. Use `gainDb` for
- levels, Tone.js for music (`motion/music`: TokenSynth or sample-library instruments in a Sampler) and `sao()` for
- foley, and put one ` ` in the composition. Put the sounds in the tree in time order.
+8. **Sound is part of the edit.** Decide what each scene is heard as (silence, its own sound, source music, score,
+ effects) before placing any; music is composed for the cut, never a bed by default. Each sound goes in its own
+ ``. Use `gainDb` for levels, Tone.js for music (`motion/music`: sample-library or
+ TokenSynth instruments in a Sampler, playing a motif, chords that move and a rhythm) and `sao()` for foley, and
+ put one ` ` in the composition. Put the sounds in the tree in time order.
9. **Keep animated styles present on every frame.** The engine keeps a style property that React stops sending.
Write `transform: scale(1)` and `opacity: 1` rather than leaving them out when an animation settles. Always set
`fontFamily`, because the engine's default face is a serif.
@@ -80,13 +83,19 @@ unreadable text, the wrong loudness) and bans (invented claims, generated imager
## A whole composition, small
```tsx
-import { AbsoluteFill, Composition, Easing, Mix, Sequence, Audio, interpolate, registerRoot, sao, useCurrentFrame, useVideoConfig } from "raea";
+import { AbsoluteFill, Composition, Easing, Mix, Sequence, Audio, interpolate, registerRoot, sao, staticFile, useCurrentFrame, useVideoConfig } from "raea";
import * as Tone from "raea/tone";
import { css } from "./styles.gen";
-// the bed: one TokenSynth instrument in a Sampler, a bar of notes on a loop (motion/music)
-const keys = new Tone.Sampler({ urls: Tone.tokensynth("a soft felt piano, warm, intimate", ["C3", "C4"], { seconds: 2 }), volume: -14 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => keys.triggerAttackRelease(n, "4n", t), ["C4", "G3", "E4", "G3"], "4n").start(0);
+// the music: a chord on the title and a two-note lift after it, on a library piano (`raea sounds get
+// salamander/piano`: roots a minor third apart, D#4 as Ds4.mp3); a sting, not a bed (motion/music)
+const ROOTS = ["C4", "D#4", "F#4", "A4", "C5", "D#5"];
+const piano = new Tone.Sampler({ urls: Object.fromEntries(ROOTS.map((n) => [n, staticFile(`sounds/salamander-piano/${n.replace("#", "s")}.mp3`)])), release: 1, volume: -10 }).connect(Tone.buses.music);
+new Tone.Part((t, n) => piano.triggerAttackRelease(n.notes, n.dur, t, n.vel), [
+ { time: "0:0", notes: ["C4", "E4", "G4", "C5"], dur: "2n", vel: 0.8 },
+ { time: "0:2", notes: ["D5"], dur: "8n", vel: 0.55 },
+ { time: "0:2:2", notes: ["E5"], dur: "4n", vel: 0.65 },
+]).start(0);
Tone.getTransport().start();
const WHOOSH = sao("soft airy whoosh, short, clean, no music", { seconds: 0.8 });
diff --git a/skills/raea/motion/assets.md b/skills/raea/motion/assets.md
index 8a063fa..77c5012 100644
--- a/skills/raea/motion/assets.md
+++ b/skills/raea/motion/assets.md
@@ -15,7 +15,8 @@
and `Tone.tokensynth(timbre, notes)` for an instrument's notes in a `Tone.Sampler` (`motion/music`).
- Recorded instruments and drum hits come from the open sample libraries: `raea sounds WORDS` finds one, `raea sounds
get PACK/INSTRUMENT` copies its files into `media/sounds/PACK-INSTRUMENT/` with a `LICENCE.txt`, and prints the
- `staticFile` paths and the Sampler that plays them.
+ `staticFile` paths and the Sampler that plays them. Orchestras, drum kits, effect packs and MIDI grooves fetched by
+ hand, and how to set up SAO and TokenSynth, are in `motion/sources`.
- Show stills with ` `. The frame waits for the image to load, and a missing one fails the render, naming
`src`.
- Keep pictures real: the person's footage, stock already in the object's media, or things you build (components,
@@ -28,9 +29,9 @@ error and so is a URL (`https://...`). `staticFile` checks both. A module import
fails loudly with the line that has it instead of rendering a black clip.
Generated imagery is banned outright (floor): no AI-generated images or video, ever. Generated sound is allowed by
-policy, and made on this computer: TokenSynth for an instrument's notes (music), SAO for foley, Pocket TTS
-(``) for narration over the picture. The sample libraries are recordings under open licences (CC0, CC-BY, public domain);
-a CC-BY one is credited where the piece is shown.
+policy, and made on this computer: TokenSynth for an instrument's notes where no library has the timbre, SAO for
+foley, Pocket TTS (``) for narration over the picture. The sample libraries are recordings under open licences
+(CC0, CC-BY, public domain); a CC-BY one is credited where the piece is shown.
## Common mistakes
diff --git a/skills/raea/motion/gen.md b/skills/raea/motion/gen.md
index 8c7ef16..67900cd 100644
--- a/skills/raea/motion/gen.md
+++ b/skills/raea/motion/gen.md
@@ -3,7 +3,9 @@
## What to do
- Generate a picture, a clip, music, an effect, a voice, a 3D model or words with `raea gen` when the piece needs
- something that isn't in the footage, the stock or the scene. `-o` is the folder; each command prints the files it wrote; use it like any
+ something that isn't in the footage, the stock or the scene. Music only when the person asks for a generated
+ track: a score is composed on instruments for the cut (`motion/music`), and a generated track can't be cut on the
+ bar or edited note by note. `-o` is the folder; each command prints the files it wrote; use it like any
other media (` `, ``, ``, ``).
- Quote first. `raea gen plan FLOW` and `--plan` on a one-shot cost nothing and say where each step runs and what it
costs in cents. A run past its cap is refused before anything is spent: `--max-cents N`, else
diff --git a/skills/raea/motion/music.md b/skills/raea/motion/music.md
index 7cf6ff9..0c3e872 100644
--- a/skills/raea/motion/music.md
+++ b/skills/raea/motion/music.md
@@ -1,5 +1,26 @@
# Music and sound effects: Tone.js on raea (raea/tone)
+## Decide what the picture needs before writing a note
+
+Music is never a default. A bed under every second makes every piece sound like every other one, and a held pad or
+drone under a long reverb is what generated music sounds like. Before any instrument goes in:
+
+- Name what each stretch is heard as: silence; the scene's own sound (diegetic: footsteps, a room, a radio, a game's
+ machines, anything the people on screen could hear); source music (music playing in the scene); score (music only
+ the audience hears); effects; a voice. Several at once is fine, and none is a choice too: a pitch with a voice may
+ need only effects and a four-second sting on the end card. Write the decision in the storyboard or DIRECTION.md,
+ one line a scene.
+- Spot the cut the way a music editor does: number each cue (1m1, 1m2...) with the frame it starts on, the moments
+ it has to hit (the hook, a reveal, the logo), where it turns (where the edit changes idea) and where it stops. A
+ cue ends on a hit, the "button" (the last chord on the last cut or the logo), not on a fade. Half a bar of silence
+ before a reveal is what makes the reveal.
+- Take the tempo from the cut. Frames per beat = fps × 60 / bpm: at 30 fps, 12 frames at 150 bpm, 15 at 120, 20 at
+ 90. Pick the bpm that puts the main cuts on beats and the biggest on downbeats, or set it so they do
+ (`bpm.value = (bars * 4 * 60 * fps) / framesToTheCut`). Shots of 1 to 1.5 s suit 120 to 160 bpm; shots of 3 to 5 s
+ suit 70 to 100, or a half-time feel. With a song, cut to its map instead (`motion/timing`).
+- A sustained texture (a held pad, a drone, a reverb wash) goes in only when the scene asks for one by name ("dread",
+ "the ship's hum") and the direction says why in a line. Even then something over it moves.
+
## What to do
- Write music and sound effects in Tone.js, as its docs write them: `import * as Tone from "raea/tone"`, then
@@ -7,53 +28,102 @@
`Part`, `Sequence` and `Pattern`. What you know of Tone works; the differences are below. It's a DAW in code: a
track is an instrument and its effects into a bus, a clip is a `Part` or `Sequence`, the arrangement is where each
starts and stops on the Transport.
-- Make music from instruments, not from a generated track. Each instrument is a `Tone.Sampler`; fill it one of two
- ways, both local and free:
- - **TokenSynth** for an instrument described in words: `Tone.tokensynth("a warm upright bass, plucked", ["C2",
- "C3", "C4"], { seconds: 2 })` renders those root notes once on this computer, and the Sampler pitches between
- them. Describe the timbre the way you'd describe a patch ("a soft wooden marimba, felt mallet, warm"; "a bright
- analog synth lead, saw, short decay"). Three roots an octave apart cover a part; a note's `seconds` is how long
- it's held (under 5). Each root takes about 5 s to make the first time, then it's cached.
- - **A sample library** for a real recorded instrument: `raea sounds piano` searches the open libraries (the
- Salamander grand piano, the 128 General MIDI instruments, tonejs-instruments' orchestra, the Versilian library's
- orchestral, world and percussion instruments); `raea sounds get salamander/piano` copies its notes into
- `media/sounds/` and prints the `Sampler` that plays them. Drums and hits come as takes for `Tone.Player`s.
- - Tone's own synths (`Synth`, `PolySynth`, `MembraneSynth`, `MetalSynth`, `NoiseSynth`, `FMSynth`) are there for
- pads, kicks, risers and anything synthetic.
-- Make sound effects and foley with SAO: `Tone.sao("a bright arcade coin chime, short", { seconds: 1 })` in a
- Sampler or Player, or `sao(...)` in an `` (`motion/sound`). Narration over the picture is
- `` from `raea/speech` (Pocket TTS, local; `motion/sound`).
+- Make music from instruments playing notes, not from a generated track. Each instrument is a `Tone.Sampler` (a kit
+ is `Tone.Players`); fill it one of three ways:
+ - **A sample library** for a real instrument, first: `raea sounds cello` searches the open libraries raea carries
+ (the Salamander grand piano; FluidR3's 128 General MIDI instruments, every note; tonejs-instruments' strings,
+ brass, woodwinds and guitars; the Versilian library's keyboards, harps, mallets, drums and percussion), and
+ `raea sounds get tonejs-instruments/cello` copies its notes into `media/sounds/` and prints the `Sampler` that
+ plays them. Drums and hits come as takes for `Tone.Player`s. Whole orchestras, multi-velocity kits and drummers'
+ grooves are fetched by hand (`motion/sources`).
+ - **TokenSynth** for a timbre no library has: `Tone.tokensynth("a bowed saw, close, eerie", ["C3", "E3", "G#3",
+ "C4", "E4", "G#4", "C5"], { seconds: 2 })` renders those root notes, a major third apart, once on this computer,
+ and the Sampler pitches between them. Describe
+ the timbre the way you'd describe a patch ("a soft wooden marimba, felt mallet, warm"; "a bright analog synth
+ lead, saw, short decay"); a note's `seconds` is how long it's held (under 5). Its takes need curating (below).
+ - Tone's own synths (`Synth`, `PolySynth`, `MonoSynth`, `MembraneSynth`, `MetalSynth`, `NoiseSynth`, `FMSynth`) for
+ parts that are synthetic on purpose: an 808 kick, an analog bass, a chiptune lead, a riser.
+- Make sound effects and foley with SAO or a recorded pack: `Tone.sao("a bright arcade coin chime, short", { seconds:
+ 1 })` in a Sampler or Player, or `sao(...)` in an ``, built in layers (`motion/sound`). Narration over the
+ picture is `` from `raea/speech` (Pocket TTS, local; `motion/sound`).
- Build the graph at the top level of the file: every instrument, effect and channel, and how they connect. The graph
is the one the composition has when its first frame renders. Schedule notes with the Transport (or with explicit
times). Never trigger notes from inside a component's render: a game's one-shots go through ``.
+- Set the tempo before building any Part: a Part reads its times at the tempo when it's made.
- Time 0 is the composition's first frame. `Tone.now()` is 0 at the top level and the event's time inside a callback,
so `Tone.getTransport().start()` at the top level starts the music at frame 0. Every note lands on its own sample,
so a render and a play are the same.
- Route through the buses: `Tone.buses.music`, `.sfx`, `.voice`, `.ambience`, `.ui`. Connect Tone nodes to them, and
give `` and `` a `bus="voice"` (or another bus) prop. Each bus is a Tone `Channel`: `volume`, `pan`,
`mute`, `solo`, `send`/`receive`.
-- Duck the bed under the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`; do the same for
- ambience. Start at -10 to -14 dB, 40 ms attack and 300 to 400 ms release. Then listen.
+- Duck the music under the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`; do the same for
+ ambience. Start at -10 to -14 dB, 40 ms attack and 300 to 400 ms release; under a sparse cue, where pumping shows,
+ -6 to -8 dB, 80 ms and about 0.9 s. Then listen.
- Keep one ` ` per composition. The master measures the whole core, Tone included.
- In a game, fire a one-shot on the exact frame something happens: ` ...} />` with
- `when` from `useGameEvents` or a machine's transition. Give one-shots `Tone.vary(instrument, { pitch: 0.3,
- volume: 2 })` so repeats don't sound copied (the variation is seeded by the frame, so a replay is the same). Cap
- their voices with `maxPolyphony` (the oldest is stolen, faded in 5 ms).
+ `when` true on that frame only (a frame from `useGameEvents`, or a machine's transition). Give one-shots
+ `Tone.vary(instrument, { pitch: 0.3, volume: 2 })` so repeats don't sound copied (the variation is seeded by the
+ frame, so a replay is the same). Cap their voices with `maxPolyphony` (the oldest is stolen, faded in 5 ms).
- In a timeline, cut on the music: `Tone.Time("1m").toFrames(fps)` is a bar in frames, and `Tone.Time("4n")` a beat.
-- To hear a sound before writing it in, `raea sao "PROMPT" --seconds 1 -o media/sfx/try.wav` or `raea tokensynth
- "TIMBRE" --notes C3 -o media/try/` makes it now; the edit's own call with the same words and settings then plays
- what was made instead of making it again.
+- To hear a sound before writing it in, `raea sao "PROMPT" --seconds 1 --dir EDIT_DIR -o EDIT_DIR/media/sfx/try.wav`
+ or `raea tokensynth "TIMBRE" --notes C3 --dir EDIT_DIR -o EDIT_DIR/media/try/` makes it now and keeps it in the
+ edit's asset store (`--dir` is the edit's folder; without it, the folder you run it from), so the edit's own call
+ with the same words and settings then plays what was made instead of making it again. Another `--seed` makes
+ another take; play a seed's file by its path (`staticFile("sfx/try.wav")`), since `sao()` in the edit has no seed.
+- When SAO or TokenSynth isn't installed yet, `raea setup-audio` installs it (`motion/sources` says what it fetches
+ and what to do when it fails).
+
+## Writing music for picture
+
+Write it as a composer would, for this cut:
+
+- **A motif first.** Three to seven notes over two to four bars, with a rhythm of its own, before any chord. Then
+ vary it rather than loop it: answer it, fragment it, move it to another instrument or register, put new chords
+ under it. End the cue on it.
+- **Harmony that moves.** In an active cue, change chord every bar or every two beats; keep chords of two to four bars
+ for a slow passage. Move the inner voices by step or hold a common tone, set the bass against the tune (contrary
+ motion, inversions so it walks), and resolve a pedal when it ends. Keep the low end open: no thirds below about C3,
+ fifths down to about C2, only octaves under that.
+- **Rhythm and groove.** At least one part moves in eighths or sixteenths: an ostinato, a pizzicato, an arpeggio, a
+ kit, hand percussion. Accent where the picture moves. Spread velocities (0.35 to 0.95, not all 0.8), and put a
+ played part a few milliseconds off the grid (5 to 15 ms, written into the note's time: `humanize` is ignored here).
+ A drummer's recorded groove beats one typed in (Groove MIDI, `motion/sources`).
+- **Orchestrate by register.** One job each: a bass (E1 to C3), sparse harmony (C3 to C5), one lead (C4 to C6), the
+ rhythm, colour on top (glockenspiel, celesta, harp). Hand the tune to a new instrument each section, and use the
+ articulations the library has (staccato, legato, pizzicato, tremolo, a swell). A pad, where the scene names one,
+ is glue 12 dB or more under the lead, never the part.
+- **The arrangement follows the edit.** Sections change where the edit changes idea: build by adding parts and by
+ switching sections, on bar lines; peak, then strip back; drop out before the reveal; end on the button. A sting
+ lasts 1 to 4 s, sits in the cue's key and lands on a beat.
+- **Write around the voice.** The tune answers in the gaps between lines, and the arrangement thins under speech.
+- **Each instrument its own room.** A short reverb per instrument or per section (decay 1.2 to 2.5 s, wet 0.15 to
+ 0.3), never one long wash over the bus: a 6 s reverb and a low-pass on the whole music bus erase every attack, and
+ that is how a score turns into a drone.
+- **Source music lives in the scene.** A radio, a band in the room, a phone: narrow it to the device (a high-pass
+ around 300 Hz and a low-pass around 3.5 kHz for a small speaker), pan it to where it is (`Tone.Panner`), put it in
+ the room's reverb, and let the scene's own sound sit with it. Crossing over is strong: a radio song that opens out
+ to full range as it becomes the score, a score hit that lands on an action in the scene.
+- Leave silence. Let the music start after a chapter card clears and stop before the next; let a card sit on one low
+ note.
+- Write for this picture's feeling, not a genre's surface: a stock walking bass and brushes under everything reads as
+ pastiche as surely as a pad does.
+- Period source music can be real: US sound recordings published before 1926 and compositions published before 1931
+ are public domain in the US (2026). archive.org's 78rpm transfers give the period's own sound; play them as
+ ``, narrowed and placed as source music, ducked with the rest.
+- Synth drums (`MembraneSynth` kicks, noise "brushes") sound synthetic on anything acoustic; use a library's recorded
+ kit (`raea sounds kick`, `snare`, `hihat`).
## Instruments from TokenSynth: build a library, don't trust a take
A TokenSynth note is one sample from a model, and a lot of takes are bad. In a 35-minute documentary score (2026-10-20),
12 of 30 single-seed root notes failed: hiss and buzz in place of a horn, a sax or accordion an octave (or 34
semitones) off, a piano note still at full level when its file ends, so every held note stopped with a click. That is
-the "vanilla and abrasive" sound. So:
+the "vanilla and abrasive" sound. It learned from 5-second notes at four velocities, so it gives one dynamic and can
+miss the pitch asked for. So:
- Reach for an open sample library first for a standard instrument (`raea sounds piano`, `sounds get
- salamander/piano`): recorded, velocity-layered, no gamble. Use TokenSynth for what no library has, or a timbre you
- can describe better than find.
+ salamander/piano`): a recording, no gamble. Use TokenSynth for what no library has, or a timbre you can describe
+ better than find.
- Declare each instrument as a candidate, not a call: its timbre, roots every 3 to 4 semitones over the range it plays
(a Sampler pitched further than that starts to sound fake), 3 or more seeds, and `seconds` long enough for the
sound to die away on its own (4 for piano, 6 for sustained strings).
@@ -67,24 +137,7 @@ the "vanilla and abrasive" sound. So:
- Prefer timbres TokenSynth does well (felt or soft piano, sustained cello section, bowed or plucked bass, vibraphone,
clarinet in its low register) to the ones it does badly (muted trumpet, tenor sax, accordion: breath and reed
noise come out as hiss).
-
-## Writing music for picture
-
-- Write for the picture's mood, not a genre's surface. A walking bass, brushes and block chords on every bar reads as
- MIDI pastiche. Under narration, a drone score works: a chord every four bars, bass and a sine sub swelling under it,
- a sustained section with a slow attack, and a piano playing single notes off any grid, with a motif returning in
- fragments. Sustained textures also hide what's left of a sample's flaws.
-- Leave silence. Let the music start after a chapter card clears and stop before the next; let a card sit on one low
- note.
-- Soften the whole bed: a 24 dB/octave low-pass around 2.5 kHz into a long reverb (decay 6 to 7 s) on the music bus.
- The voice owns the presence range.
-- Duck gently under a voice over a drone (-6 to -8 dB, 80 ms attack, about 0.9 s release); the -12 dB duck suits a
- rhythmic bed, where pumping is less audible.
-- Period source music can be real: US sound recordings published before 1926 and compositions published before 1931
- are public domain in the US (2026). archive.org's 78rpm transfers give the period's own sound; play them as
- `` under the score's drones, ducked with the rest.
-- Synth drums (`MembraneSynth` kicks, noise "brushes") sound synthetic on anything acoustic; leave them out or use a
- library's recorded kit.
+- For dynamics, render a soft and a loud velocity as two Samplers and pick one by each note's velocity.
## Why
@@ -96,24 +149,38 @@ So the graph goes at the top level, and in-game sounds go through ``.
Generated sources are made once per their inputs: the prompt, the length, a TokenSynth note's pitch, velocity, seed
and precision. They're normalised, cached as PCM in the object's `assets/`, and played like any file. TokenSynth and
-SAO run on this computer and install themselves the first time they're used (a few minutes, once). Music written as
-notes on instruments can be edited note by note, cut on the bar and ducked per part; a generated track can't.
+SAO run on this computer and install themselves the first time they're used (minutes, once; `raea setup-audio` does
+it ahead of time). Music written as notes on instruments can be edited note by note, cut on the bar and ducked per
+part; a generated track can't.
-Policy: music is Tone.js instruments (TokenSynth, the sample libraries, synths), foley and effects are SAO, narration
-is Pocket TTS (``), all local. No other generation: no generated images, video or remote sound effects. (`lyria()`, a remote music model, is
-retired: an older edit keeps the take it made, and a new one is refused.)
+Policy: music is notes on Tone.js instruments (sample libraries first, TokenSynth for a timbre no library has, synths
+for synthetic parts), effects and foley are SAO or recorded packs (`motion/sources`), narration is Pocket TTS
+(``), all local. Any other generation is a `raea gen` step, quoted and capped (`motion/gen`). (`lyria()`, a
+remote music model, is retired: an older edit keeps the take it made, and a new one is refused.)
## Common mistakes
+- A pad and a sine sub through a low-pass into a long reverb as the score. That's a drone, not music: write a motif,
+ chords that move and a part with rhythm.
+- One chord every four bars for a minute, or a two-bar loop for thirty seconds. Change the harmony every bar or two
+ and the arrangement every four to eight.
+- Every note at the same velocity, on the grid. Spread the dynamics and move played parts a few milliseconds.
+- A cue that fades out at the end. End it on the last cut, on a chord or a hit.
+- Music under every scene because the piece "needs music". Decide per scene; silence and the scene's own sound count.
- `synth.triggerAttackRelease(...)` in a component body. It fires on every render; use ``.
+- `` on a flag the state holds for many frames: it fires on every one. Give it the
+ frames the event happens on (`useGameEvents`).
- `new Tone.Synth()` inside a component. Make instruments at the top level.
- A `https://` URL for a Sampler or a Player. Sources are media files or generated assets, and the render has no
network: `raea sounds get` puts a library's files in `media/` first.
-- One TokenSynth root for a whole part. A Sampler stretched three octaves from one note sounds chipmunked or
- muddy; give it a root every octave (or `raea sounds get` a library's roots, a minor third apart).
+- One root for a whole part, or one an octave. A Sampler stretched far from its root sounds chipmunked or muddy;
+ give it a root every 3 to 4 semitones over the range the part plays: a library's own roots (`raea sounds get`
+ with no `--notes` takes them a minor third apart, or every note the library sampled if it's sparser), or as many
+ TokenSynth roots.
- Rewording a `tokensynth()` timbre on every pass: each new wording makes its notes again. Settle the timbre first.
-- A sample library's licence ignored. CC-BY ones (FluidR3, Salamander) need their credit where the piece is shown;
- `LICENCE.txt` beside the files has the line. VCSL is CC0.
+- A sample library's licence ignored. CC-BY ones (FluidR3, Salamander, tonejs-instruments) need their credit where
+ the piece is shown; `LICENCE.txt` beside the files has the line. VCSL is CC0.
+- `part.loop = true` after `part.start()`: the Part plays once. Set `loop` and `loopEnd` first.
- `Tone.Draw.schedule` for visuals. There's no draw loop: read the frame (`useCurrentFrame`) and the same times.
- Cutting on `Tone.Time("1m").toFrames(fps)` across a tempo ramp. `Tone.getTransport().bpm.rampTo(120, 4)` bends the
grid, as in Tone (the notes come closer together; an exponential ramp runs linear), but `Time(...)` is at the tempo
@@ -123,44 +190,112 @@ retired: an older edit keeps the take it made, and a new one is refused.)
.connect(filter.frequency)`).
- A Sequence or Loop with no `Tone.getTransport().start()`. Nothing plays until the Transport starts, as in Tone.
-## Example: a bed from TokenSynth instruments and a library piano, cut on the bar
+## Example: a cue with a motif, chords that move and a kit, cut on the bar
+
+Spotted as one cue over nine one-bar cards: A (bars 1 to 4) piano and a pizzicato ostinato, the flute's motif from
+bar 3; B (bars 5 to 8) the ostinato drops out, bass and kit come in, the motif climbs; the button on the end card's
+cut. The instruments came from these commands, each printing the roots it fetched (with no `--notes`, a minor third
+apart from C1 to C7, or every note the library sampled when that's sparser):
-`raea sounds get salamander/piano --notes C3,C4,C5` put the piano's notes in `media/sounds/salamander-piano/`.
+```sh
+raea sounds get salamander/piano
+raea sounds get fluid/pizzicato_strings
+raea sounds get tonejs-instruments/flute
+raea sounds get fluid/acoustic_bass
+raea sounds get vcsl/bass-drum-1 && raea sounds get vcsl/snare-drum-modern-1 && raea sounds get vcsl/hi-hat-cymbal
+```
```tsx
import { AbsoluteFill, Composition, Mix, Series, registerRoot, staticFile } from "raea";
import * as Tone from "raea/tone";
-Tone.getTransport().bpm.value = 92;
-const BAR = Tone.Time("1m").toFrames(30); // 78 frames at 92 bpm in 4/4
-const room = new Tone.Reverb({ decay: 2.2, wet: 0.25 }).connect(Tone.buses.music);
-const bass = new Tone.Sampler({ urls: Tone.tokensynth("a warm upright bass, plucked, round", ["C2", "C3"], { seconds: 1.5 }), release: 0.3 }).connect(Tone.buses.music);
-const keys = new Tone.Sampler({
- urls: { C3: staticFile("sounds/salamander-piano/C3.mp3"), C4: staticFile("sounds/salamander-piano/C4.mp3"), C5: staticFile("sounds/salamander-piano/C5.mp3") },
- release: 1.2,
- volume: -8,
-}).connect(room);
-const kick = new Tone.MembraneSynth({ octaves: 5, volume: -8 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => kick.triggerAttackRelease(n, "8n", t), ["C1", null, null, null], "4n").start(0);
-new Tone.Part((t, n) => bass.triggerAttackRelease(n, "4n", t), [["0:0", "A2"], ["0:2", "E2"], ["1:0", "F2"], ["1:2", "C3"]]).start(0);
-new Tone.Part((t, c) => keys.triggerAttackRelease(c, "2n", t), [["0:0", ["A3", "C4", "E4"]], ["1:0", ["F3", "A3", "C4"]]]).start(0);
+Tone.getTransport().bpm.value = 96; // first: a Part reads its times at the tempo when it's made
+const BAR = Tone.Time("1m").toFrames(30); // 75 frames
+
+// the roots each part's range needs, of those `raea sounds get` printed (D#3 is the file Ds3.mp3)
+const notes = (dir: string, names: string[]) => Object.fromEntries(names.map((n) => [n, staticFile(`sounds/${dir}/${n.replace("#", "s")}.mp3`)]));
+const room = new Tone.Reverb({ decay: 1.6, wet: 0.2 }).connect(Tone.buses.music);
+const piano = new Tone.Sampler({ urls: notes("salamander-piano", ["A2", "C3", "D#3", "F#3", "A3", "C4", "D#4", "F#4"]), release: 1, volume: -8 }).connect(room);
+const pizz = new Tone.Sampler({ urls: notes("fluid-pizzicato_strings", ["D#2", "F#2", "A2", "C3", "D#3", "F#3", "A3", "C4"]), release: 0.2, volume: -12 }).connect(room);
+const flute = new Tone.Sampler({ urls: notes("tonejs-instruments-flute", ["E4", "A4", "C5", "E5", "A5", "C6"]), release: 0.4, volume: -6 }).connect(room);
+const bass = new Tone.Sampler({ urls: notes("fluid-acoustic_bass", ["D#1", "F#1", "A1", "C2", "D#2"]), release: 0.3, volume: -6 }).connect(Tone.buses.music);
+const kit = new Tone.Players({
+ kick: staticFile("sounds/vcsl-bass-drum-1/BDrumNew_hit_v5_rr1_Sum.wav"),
+ snare: staticFile("sounds/vcsl-snare-drum-modern-1/Snare2_HitSN_v5_rr1_Mid.wav"),
+ hat: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v3_rr1_Mid.wav"),
+ ghost: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v1_rr1_Mid.wav"), // the soft take, for the off-beats
+}).connect(Tone.buses.music);
+
+// a chord a bar, voiced so the inner notes move by step: Am F C G, then F G Am E
+const CHORDS = [["A3", "C4", "E4"], ["A3", "C4", "F4"], ["G3", "C4", "E4"], ["G3", "B3", "D4"], ["A3", "C4", "F4"], ["G3", "B3", "D4"], ["A3", "C4", "E4"], ["G#3", "B3", "E4"]];
+// comped, not held: the downbeat and the and-of-two, the second softer
+new Tone.Part((t, c) => piano.triggerAttackRelease(c.notes, "8n", t, c.vel), CHORDS.flatMap((chord, bar) => [
+ { time: `${bar}:0`, notes: chord, vel: 0.62 },
+ { time: `${bar}:1:2`, notes: chord, vel: 0.42 },
+])).start(0);
+
+// A: an eighth-note ostinato on each chord's root (root, fifth, octave, fifth), accented on the beat
+new Tone.Part((t, n) => pizz.triggerAttackRelease(n.note, "16n", t, n.vel), ["A2", "F2", "C3", "G2"].flatMap((root, bar) =>
+ [0, 7, 12, 7, 0, 7, 12, 7].map((up, i) => ({ time: `${bar}:${i >> 1}:${(i & 1) * 2}`, note: Tone.Frequency(root).transpose(up).toNote(), vel: i % 2 ? 0.45 : 0.75 })),
+)).start(0);
+
+// the motif (E D C D, then B A G) from bar 3, sequenced up and answered in B; each note a few ms off the grid
+const TUNE: [string, string, string, number][] = [
+ ["2:0", "E5", "4n", 0.7], ["2:1", "D5", "8n", 0.55], ["2:1:2", "C5", "8n", 0.6], ["2:2", "D5", "2n", 0.75],
+ ["3:0", "B4", "4n.", 0.65], ["3:1:2", "A4", "8n", 0.5], ["3:2", "G4", "2n", 0.6],
+ ["4:0", "A5", "4n", 0.8], ["4:1", "G5", "8n", 0.6], ["4:1:2", "F5", "8n", 0.65], ["4:2", "G5", "2n", 0.8],
+ ["5:0", "D5", "4n.", 0.7], ["5:1:2", "B4", "8n", 0.55], ["5:2", "D5", "2n", 0.7],
+ ["6:0", "E5", "4n", 0.75], ["6:1", "C5", "8n", 0.6], ["6:1:2", "E5", "8n", 0.7], ["6:2", "A5", "2n", 0.9],
+ ["7:0", "G#5", "2n.", 0.8], ["7:3", "B4", "4n", 0.6],
+];
+new Tone.Part((t, n) => flute.triggerAttackRelease(n.note, n.dur, t + n.late, n.vel),
+ TUNE.map(([time, note, dur, vel], i) => ({ time, note, dur, vel, late: ((i * 5) % 3 - 1) * 0.007 }))).start(0);
+
+// B: the bass walks up under the climb (F G A, then E to come home), root on one and fifth on three
+new Tone.Part((t, n) => bass.triggerAttackRelease(n.note, "4n.", t, n.vel), [["F1", "C2"], ["G1", "D2"], ["A1", "E2"], ["E1", "B1"]].flatMap(([root, fifth], i) => [
+ { time: `${4 + i}:0`, note: root, vel: 0.85 },
+ { time: `${4 + i}:2`, note: fifth, vel: 0.6 },
+])).start(0);
+
+// B: the kit, four bars of sixteenth-note lanes (null rests)
+const _ = null;
+for (const steps of [
+ ["kick", _, _, _, _, _, "kick", _, "kick", _, _, _, _, _, _, _],
+ [_, _, _, _, "snare", _, _, _, _, _, _, _, "snare", _, _, _],
+ ["hat", _, "ghost", _, "hat", _, "ghost", "ghost", "hat", _, "ghost", _, "hat", _, "ghost", _],
+]) {
+ const lane = new Tone.Sequence((t, hit) => kit.player(hit).start(t), steps, "16n");
+ lane.loop = 4; // set before start: four bars, then it stops
+ lane.start("4m");
+}
+
+// the button: everything on the end card's downbeat, then nothing
+Tone.getTransport().schedule((t) => {
+ piano.triggerAttackRelease(["A2", "A3", "C4", "E4"], "2n", t, 0.85);
+ bass.triggerAttackRelease("A1", "2n", t, 0.9);
+ flute.triggerAttackRelease("A5", "4n", t, 0.8);
+ kit.player("kick").start(t);
+}, "8m");
Tone.getTransport().start();
-// each card is one bar long, so every cut lands on a downbeat
+// each card is one bar, so every cut lands on a downbeat; the ninth is the end card
const Edit = () => (
-
- {[0, 1, 2, 3].map((i) => {/* card i */} )}
+
+ {Array.from({ length: 9 }, (_, i) => {/* card i */} )}
);
-registerRoot(() => );
+registerRoot(() => );
```
+FluidR3, Salamander and tonejs-instruments (the flute) are CC-BY 3.0, so the piece credits all three (the line is in
+each `LICENCE.txt`); VCSL is CC0.
+
## Example: a game's sound
```tsx
-import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, staticFile, useCurrentFrame, useGameEvents, useGameState } from "raea";
+import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, staticFile, useCurrentFrame, useGameEvents } from "raea";
import * as Tone from "raea/tone";
// the game: its state and step (motion/games); only what the sound reads is shown
@@ -168,33 +303,49 @@ type State = { hits: number; coinTaken: boolean };
const INITIAL: State = { hits: 0, coinTaken: false };
const fold = (s: State): State => s;
+Tone.getTransport().bpm.value = 112;
Tone.buses.music.duck(Tone.buses.voice, { amount: -12 });
const thump = Tone.vary(new Tone.MembraneSynth({ octaves: 4, maxPolyphony: 4, poly: true }), { pitch: 0.5, volume: 2 }).connect(Tone.buses.sfx);
const coin = new Tone.Sampler({ urls: { C5: Tone.sao("a bright arcade coin chime, short", { seconds: 1 }) } }).connect(Tone.buses.sfx);
-const marimba = new Tone.Sampler({ urls: Tone.tokensynth("a soft wooden marimba, felt mallet, warm", ["C3", "C4", "C5"], { seconds: 2 }), volume: -10 }).connect(Tone.buses.music);
-const tune = new Tone.Part((t, n) => marimba.triggerAttackRelease(n, "8n", t), [["0:0", "C4"], ["0:1", "E4"], ["0:2", "G4"], ["1:0", "A4"], ["1:2", "G4"]]);
+// a recorded marimba (`raea sounds get vcsl/marimba`: CC0, every note VCSL sampled, four to seven semitones apart)
+const marimba = new Tone.Sampler({
+ urls: Object.fromEntries(["F3", "C4", "G4", "B4", "F5"].map((n) => [n, staticFile(`sounds/vcsl-marimba/${n}.wav`)])),
+ release: 0.4,
+ volume: -10,
+}).connect(new Tone.Reverb({ decay: 1.2, wet: 0.15 }).connect(Tone.buses.music));
+// a loop that wants to come round: a question (bars 1-2) and its answer (bars 3-4), rhythm before pitch
+const tune = new Tone.Part((t, n) => marimba.triggerAttackRelease(n.note, "8n", t, n.vel), [
+ { time: "0:0", note: "C4", vel: 0.8 }, { time: "0:0:2", note: "E4", vel: 0.5 }, { time: "0:1:2", note: "G4", vel: 0.6 }, { time: "0:3", note: "A4", vel: 0.7 },
+ { time: "1:0", note: "G4", vel: 0.8 }, { time: "1:1:2", note: "E4", vel: 0.5 }, { time: "1:2", note: "D4", vel: 0.6 },
+ { time: "2:0", note: "C4", vel: 0.8 }, { time: "2:0:2", note: "E4", vel: 0.5 }, { time: "2:1:2", note: "G4", vel: 0.6 }, { time: "2:3", note: "C5", vel: 0.75 },
+ { time: "3:0", note: "B4", vel: 0.7 }, { time: "3:1", note: "G4", vel: 0.55 }, { time: "3:2", note: "C4", vel: 0.85 },
+]);
tune.loop = true;
-tune.loopEnd = "2m";
+tune.loopEnd = "4m";
tune.start(0);
Tone.getTransport().start();
function Game() {
const frame = useCurrentFrame();
- const g = useGameState(INITIAL, fold);
const hits = useGameEvents(INITIAL, fold, (b, a) => a.hits > b.hits);
+ const coins = useGameEvents(INITIAL, fold, (b, a) => a.coinTaken && !b.coinTaken);
return (
{/* ...the game's picture... */}
thump.triggerAttackRelease("C2", "8n", t)} />
- coin.triggerAttackRelease("C5", "8n", t)} />
+ coin.triggerAttackRelease("C5", "8n", t)} />
);
}
+
+registerRoot(() => );
```
`raea/music` is the shorthand for notes on named instruments (`new Part("bass", [...])`, ``). It plays
-through the same Tone layer, so the two mix.
+through the same Tone layer, so the two mix. Its named instruments are Tone synth presets (`pad` is a PolySynth of
+fat saws, `kick` a MembraneSynth), good for a sketch or a test, not a score: write the score on sampled instruments,
+as above. It also reads and writes MIDI files (`parseMidi`, `toMidi`; `motion/sources`).
diff --git a/skills/raea/motion/parts.md b/skills/raea/motion/parts.md
index 53ee3df..7f6d5bf 100644
--- a/skills/raea/motion/parts.md
+++ b/skills/raea/motion/parts.md
@@ -44,8 +44,8 @@ registerRoot(() => `,
+A screen recording or b-roll has no words to cue on, so it's cut on its music: the person's own track (the host's
+`bed`), never a pad made to fill the silence. The host cuts the shots on the track's beats and gives `BEATS` (each beat's frame in the cut) and `SHOTS` (where each shot looks). Pass ``,
and cue each part by beat:
- `beat`: the beat it lands on, counted from 0. A part on a shot's first beat lands on its cut.
@@ -100,4 +100,4 @@ line has one to three.
- **Captions are always on.** They move under a card, hide under a hook, a strike or the ask, and go big at the top when the band is empty. Leave some beats empty so the face and the captions carry them.
- **The thesis gets Phrases.** The thesis is the first line where the speaker says what the brief's `believe:` line means (the believe line finds it; its own wording never goes on screen): the line the piece is about, often the turn after the numbers, and often said as "not this, but that" ("so the fix isn't / another *discount* / it's a *fast reply*"). It reads phrase by phrase: one `` over it, a phrase per breath, the word that carries each phrase in the accent. Captions alone show it three words at a time, lit word by word, which reads as a caption, not a statement.
- **About one part every three to five seconds.** A beat with nothing to show stays empty.
-- The sound is done for you. Each part brings its own hit, and `` plus ` ` finish it. Don't add more foley.
+- The sound is done for you. Each part brings its own hit, and ` ` finishes it, with `` when the person gave a track (their own track, or none: never a pad made to fill it). Don't add more foley.
diff --git a/skills/raea/motion/sound.md b/skills/raea/motion/sound.md
index 479c9a3..2a6fd6f 100644
--- a/skills/raea/motion/sound.md
+++ b/skills/raea/motion/sound.md
@@ -2,6 +2,10 @@
## What to do
+- Decide what each scene is heard as before placing anything: silence, the scene's own sound (footsteps, a room, a
+ machine: what the people on screen would hear), source music, score, effects, a voice. Music is a decision per
+ scene, never a bed by default (`motion/music` has the spotting and the writing; `motion/sources` where each sound
+ can come from).
- Put each sound in its own `` with an `` inside. The Sequence's
`from` is when the sound starts, and its `durationInFrames` (optional) cuts it off. `layout="none"` because a sound
draws nothing.
@@ -9,17 +13,26 @@
values above 1 boost). Either one can be a function of the sound's own frame (0 at its first frame) for a duck or
a swell. `gainDb` wins when both are given.
- Fade with `fadeIn` and `fadeOut`, in seconds. They ramp from and to silence at the start and end of where the sound
- plays, at the audio's own rate, and multiply with the gain. Use about 1 to 1.4 s at the end of a bed, and
- 0.012 s at a cut.
-- Make foley and sound effects with `sao(prompt, {seconds})` (2 s by default; up to 11.9 s) and pass the result to
- ``. Write prompts the way the approved pieces do: the sound, its character and what to leave out ("deep
- punchy cinematic impact hit, tight, no reverb tail"; "bubbly UI pop click, short, dry"). Ask for the length you
- need.
-- Make the music bed in Tone.js on `Tone.buses.music` (`motion/music`): instruments in a `Sampler` (TokenSynth notes
- or a sample library's), playing a `Part` or `Sequence`. Duck it under the voice with
- `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`, give the voice ``, and set the bed's
- level on the bus: `Tone.buses.music.volume.value = -21`, then `Tone.buses.music.volume.rampTo(-12, 0.3, t)` to lift
- it at `t` seconds for the end card.
+ plays, at the audio's own rate, and multiply with the gain. Use about 1 to 1.4 s where an ambience or a source cue
+ leaves the scene, and 0.012 s at a cut. A score ends on its last hit instead (`motion/music`).
+- Make foley and sound effects with `sao(prompt, {seconds})` (2 s by default; up to 11.9 s, and a longer ask comes
+ back 11.9 s long) and pass the result to ``, or take a recorded one from a pack (`motion/sources`).
+ Write prompts the way the approved pieces do: the sound, its character and what to leave out ("deep punchy
+ cinematic impact hit, tight, no reverb tail"; "bubbly UI pop click, short, dry"). Ask for the length you need.
+- Design an effect in layers, each its own short prompt or take, laid on the same frame: a transient (the first 5 to
+ 30 ms: a snap, a click), a body that says what the thing is (50 to 500 ms), weight for a big hit only (under
+ 120 Hz, short), and a tail that puts it in its room (0.3 to 3 s). Vary a sound that repeats: 3 to 5 takes (another seed or another file), a semitone or two and a couple of dB apart.
+- Give a place its room tone and keep it across that place's cuts: no digital silence under picture, an ambience
+ that changes when the place does, and the next scene's sound starting about half a second before its cut. Foley
+ is three passes, each on the frame of contact: steps, cloth, props.
+- Place what's in the scene where it is: pan it to its side of the frame (`Tone.Panner`, a Channel's `pan`), give
+ it the room's reverb, and narrow a device (a phone, a radio) with a high-pass and a low-pass. In 3D, `` places it and the camera hears it move.
+- When the piece has music, write it in Tone.js on `Tone.buses.music` (`motion/music`): instruments in a `Sampler`
+ (a sample library's notes, or TokenSynth's), parts in a `Part` or `Sequence`, composed for the cut. Duck it under
+ the voice with `Tone.buses.music.duck(Tone.buses.voice, { amount: -12 })`, give the voice ``,
+ and set the music's level on the bus: `Tone.buses.music.volume.value = -21`, then
+ `Tone.buses.music.volume.rampTo(-12, 0.3, t)` to lift it at `t` seconds for the end card.
- Narration over the picture, when the footage has no voice for it, is
` ` from `raea/speech` (Pocket TTS, on this computer; `voice` picks a
Pocket voice or a WAV of one).
@@ -32,12 +45,15 @@
| sound | with a voice | without one (promo, deck, game) |
|---|---|---|
| the voice clips | about 0 dB | none |
- | music bed | -21 dB, rising to about -12 dB once the voice stops (the end card) | -12 to -15 dB |
+ | music | -21 dB, rising to about -12 dB once the voice stops (the end card) | -12 to -15 dB |
| foley hits, whooshes, pops | -16 to -22 dB | -1 to -6 dB |
These are starting points; listen and decide.
-- `raea sao "PROMPT" --seconds 1.2 -o media/sfx/hit.wav` makes a sound now, to check it before it goes in; the edit's
- `sao("PROMPT", { seconds: 1.2 })` then plays that one rather than making another.
+- `raea sao "PROMPT" --seconds 1.2 --dir EDIT_DIR -o EDIT_DIR/media/sfx/hit.wav` makes a sound now, to check it
+ before it goes in, and keeps it in the edit's asset store (`--dir` is the edit's folder; without it, the folder you
+ run it from), so the edit's `sao("PROMPT", { seconds: 1.2 })` then plays that one rather than making another.
+ `--seed 2` makes another take of the same prompt; play that file by its path (`staticFile("sfx/hit-2.wav")`), since
+ `sao()` has no seed.
## Why
@@ -46,14 +62,15 @@ full-frame box that paints over the parts beneath it.
`sao()` returns an asset `{kind, prompt, seconds, id, src}` whose id is a stable hash of the kind, the
prompt and the length. The renderer makes each asset once and caches it in the object's `assets/`. SAO runs on this
-computer (Stable Audio Open, a few seconds a sound; it installs itself the first time). The same call always reuses
+computer (Stable Audio Open Small, a few seconds a sound; it installs itself the first time, and `motion/sources`
+says what that fetches). It does better on effects and field recordings than on music. The same call always reuses
the same sound, and changing one word of the prompt makes a new one. So settle the prompt before you iterate on
gains.
-Policy: music is Tone.js instruments (TokenSynth, the sample libraries, synths), SAO makes foley, and Pocket TTS
-(``) says narration over the picture, all on this computer. Any other generation is a `raea gen` step
-(`motion/gen`). (`lyria()`, a remote music model, is retired: a
-new take is refused, and a bed of instruments can be cut on the bar and edited note by note anyway.)
+Policy: music is notes on Tone.js instruments (sample libraries first, TokenSynth, synths), SAO and recorded packs
+make effects and foley, and Pocket TTS (``) says narration over the picture, all on this computer. Any other
+generation is a `raea gen` step (`motion/gen`). (`lyria()`, a remote music model, is retired: a new take is refused,
+and music written as notes can be cut on the bar and edited note by note anyway.)
Sound sells the cut, but one move everywhere is a lint: a whoosh on every entrance flattens all of them. Give
foley to the moments that are physical (a slam, a strike-through, a card landing on the beat) and let others be
@@ -61,22 +78,23 @@ silent.
## Check the sound before you call it done
-Frames get looked at; sound has to be measured, because a broken bed is easy to miss on one listen and obvious to
+Frames get looked at; sound has to be measured, because a broken mix is easy to miss on one listen and obvious to
everyone else. Two checks, both cheap:
- **The render says when a sound runs out.** `WARN sound X runs out N s before its stretch ends` means the
source is shorter than its Sequence and still sounding (or asked for a fadeOut) at its end: it stops dead and the
- fade never plays, so a 10 s `sao()` bed under a 12 s cue cuts off mid-line. `loop` alone doesn't help: a sound
- loops only between `startFrom` and `endAt`, and that loop is a hard jump. Tile it instead: copies of the take end to
- end, each in its own Sequence, crossfading over 1.5 s (`fadeIn`/`fadeOut`), or make the take as long as the span.
+ fade never plays, so a 10 s `sao()` ambience under a 12 s scene cuts off mid-line. `loop` alone doesn't help: a
+ sound loops only between `startFrom` and `endAt`, and that loop is a hard jump. Tile it instead: copies of the take
+ end to end, each in its own Sequence, crossfading over 1.5 s (`fadeIn`/`fadeOut`), or make the take as long as the
+ span.
- **Check the finished file.** `raea check FILM.mp4 --words words.json` (the voice's words on the film's clock)
- reports, by timecode, dead stops (a steady sound that ends in one 10 ms step: fade a bed out over 100 ms or more;
+ reports, by timecode, dead stops (a steady sound that ends in one 10 ms step: fade it out over 100 ms or more;
a voice ending its word isn't one), stretches with no voice louder than
- the voice, a bed closer than 12 dB to the voice over it, and a true peak over -1 dBTP. Fix every line it prints, or
+ the voice, music closer than 12 dB to the voice over it, and a true peak over -1 dBTP. Fix every line it prints, or
say why it stands.
Set levels from measurements, not by ear on one pass. Measure each source's integrated loudness (`ffmpeg -i F -af
-ebur128 -f null -`) and set its gain against the narration's: beds 15-20 LU under the voice; a film's own sound,
+ebur128 -f null -`) and set its gain against the narration's: music 15-20 LU under the voice; a film's own sound,
where the narration stops, about 3 LU under it with its loudest 3 s no more than 2 LU over; never boost a near-silent
take more than +3 dB (its hiss comes up with it). A flat gain on every archive clip is how one film's main title
ends up shouting over the narrator while another is inaudible.
@@ -85,18 +103,21 @@ ends up shouting over the narrator while another is inaudible.
- An `` without a Sequence, or in a Sequence without `layout="none"`.
- `volume={-6}` meaning decibels. That throws, because volume is a non-negative linear gain. Write `gainDb={-6}`.
-- A bed at -6 dB under the voice. The voice must stay intelligible over everything (floor).
-- `loop` on an `` without `endAt` to stretch a short bed: nothing loops, it plays once and stops dead (the
- render warns). With `endAt` it repeats with a hard jump at each loop; tile copies with crossfades instead.
+- Music at -6 dB under the voice. The voice must stay intelligible over everything (floor).
+- A pad, a drone or one `sao()` take of "ambient music" laid under the whole piece as its score. Decide what each
+ scene needs, and write the music (`motion/music`).
+- An effect from one take with nothing under it: a click that should be a door. Layer it.
+- `loop` on an `` without `endAt` to stretch a short ambience: nothing loops, it plays once and stops dead
+ (the render warns). With `endAt` it repeats with a hard jump at each loop; tile copies with crossfades instead.
- Two `` in one composition: it throws.
- No ``: the sum plays as it is, at whatever loudness it adds up to.
-- Rewording a `sao()` prompt on every pass. Each new prompt makes a new sound .
+- Rewording a `sao()` prompt on every pass. Each new prompt makes a new sound.
- `style` on an ``: Audio has no picture and takes no style.
- Hand-placing dozens of game sounds from a log. Cue them from `useGameEvents` (`motion/games`).
- Writing music note by note in `` files, or a game's sounds by hand. Write them in Tone.js (`motion/music`):
instruments, the Transport, buses with ducking, and `` for one-shots.
-## Example: a bed that lifts for the end card, a voice, foley on two beats
+## Example: a cue that lands on the end card, a voice, a layered hit
```tsx
import { AbsoluteFill, Audio, Composition, Mix, Sequence, registerRoot, sao, staticFile } from "raea";
@@ -106,24 +127,44 @@ import { css } from "./styles.gen";
const FRAMES = 300;
const END_CARD = 252; // the voice stops here
-// the bed: a TokenSynth pulse and a pad on the music bus, ducked under the voice, lifted for the end card
-Tone.getTransport().bpm.value = 100;
+// the music (motion/music): a sampled piano, chords that change every bar under a motif that answers the voice,
+// ducked under it, lifted for the end card, and ending on a chord on that cut. The tempo is set from the cut:
+// four bars to the end card, so its cut is a downbeat.
+Tone.getTransport().bpm.value = (4 * 4 * 60 * 30) / END_CARD; // about 114 bpm
Tone.buses.music.duck(Tone.buses.voice, { amount: -12 });
Tone.buses.music.volume.value = -21;
Tone.buses.music.volume.rampTo(-12, 0.33, END_CARD / 30);
-const pluck = new Tone.Sampler({ urls: Tone.tokensynth("a soft muted electric piano, warm, short decay", ["C3", "C4"], { seconds: 1 }) }).connect(Tone.buses.music);
-const pad = new Tone.PolySynth(Tone.Synth, { oscillator: { type: "fatsine" }, envelope: { attack: 0.6, release: 1.5 }, volume: -12 }).connect(Tone.buses.music);
-new Tone.Sequence((t, n) => pluck.triggerAttackRelease(n, "8n", t), ["A3", "E4", "C4", "E4"], "8n").start(0);
-new Tone.Part((t, c) => pad.triggerAttackRelease(c, "2m", t), [["0:0", ["A3", "C4", "E4"]], ["2:0", ["F3", "A3", "C4"]]]).start(0);
+// `raea sounds get salamander/piano` printed its roots a minor third apart (D#2 is the file Ds2.mp3); these cover
+// F2 to G5, what the cue plays
+const ROOTS = ["D#2", "F#2", "A2", "C3", "D#3", "F#3", "A3", "C4", "D#4", "F#4", "A4", "C5", "D#5", "F#5"];
+const piano = new Tone.Sampler({
+ urls: Object.fromEntries(ROOTS.map((n) => [n, staticFile(`sounds/salamander-piano/${n.replace("#", "s")}.mp3`)])),
+ release: 1,
+}).connect(new Tone.Reverb({ decay: 1.4, wet: 0.2 }).connect(Tone.buses.music));
+const CHORDS = [["A2", "E3", "C4"], ["F2", "C3", "A3"], ["C3", "G3", "E4"], ["G2", "D3", "B3"]];
+new Tone.Part((t, c) => piano.triggerAttackRelease(c.notes, "4n", t, c.vel), CHORDS.flatMap((chord, bar) => [
+ { time: `${bar}:0`, notes: chord, vel: 0.55 },
+ { time: `${bar}:2:2`, notes: chord, vel: 0.4 },
+])).start(0);
+new Tone.Part((t, n) => piano.triggerAttackRelease(n.note, "8n", t, n.vel), [
+ { time: "1:1", note: "E5", vel: 0.7 }, { time: "1:1:2", note: "D5", vel: 0.5 }, { time: "1:2", note: "C5", vel: 0.6 }, { time: "1:3", note: "A4", vel: 0.65 },
+ { time: "3:1", note: "G5", vel: 0.75 }, { time: "3:1:2", note: "F5", vel: 0.5 }, { time: "3:2", note: "E5", vel: 0.6 }, { time: "3:3", note: "D5", vel: 0.7 },
+]).start(0);
+Tone.getTransport().schedule((t) => piano.triggerAttackRelease(["A2", "E3", "A3", "C4", "E4"], "1m", t, 0.8), "4m");
Tone.getTransport().start();
+// one hit in three layers, on the same frame: the snap that starts it, the body that says what it is, the room's tail
+const SNAP = sao("sharp wooden snap, close, very short, dry", { seconds: 0.3 });
const HIT = sao("deep punchy impact hit, tight, no reverb tail", { seconds: 1.2 });
+const TAIL = sao("the reverb tail of one impact in a large hall, no attack", { seconds: 2.5 });
const POP = sao("bubbly UI pop click, short, dry", { seconds: 0.6 });
const VOICE = staticFile("voice/vo01.wav"); // the person's narration, as a file
/** Every sound, in time order. */
const SOUNDS = [
+ { id: "hit-snap", from: 0, gainDb: -20, src: SNAP, fade: 0, bus: "sfx" },
{ id: "hit", from: 0, gainDb: -16, src: HIT, fade: 0, bus: "sfx" },
+ { id: "hit-tail", from: 0, gainDb: -24, src: TAIL, fade: 0, bus: "sfx" },
{ id: "voice", from: 15, to: END_CARD, gainDb: 0, src: VOICE, fade: 0.012, bus: "voice" },
{ id: "pop", from: END_CARD + 2, gainDb: -17, src: POP, fade: 0, bus: "sfx" },
] as const;
diff --git a/skills/raea/motion/sources.md b/skills/raea/motion/sources.md
new file mode 100644
index 0000000..f76d5e5
--- /dev/null
+++ b/skills/raea/motion/sources.md
@@ -0,0 +1,238 @@
+# Sources: the local models, sample libraries, effect packs and MIDI, and how to get each onto the Mac
+
+## What to do
+
+- Take each sound from the source that fits its job, and know its licence before it goes in:
+
+ | the job | the source |
+ |---|---|
+ | an instrument playing a part: piano, strings, brass, woodwinds, mallets, guitars, a drum kit | a sampled library: `raea sounds` first, then the packs below |
+ | a timbre no library has ("a bowed saw, close") | TokenSynth, curated (`motion/music`) |
+ | an effect, foley, an ambience | a recorded pack or the person's own recordings first; SAO for what none has, in layers (`motion/sound`) |
+ | a voice over the picture | `` (Pocket TTS, local); a voice model is a `raea gen speech` step, Fish's with the person's own FISH_API_KEY (`motion/gen`) |
+ | a drummer's feel | a Groove MIDI performance played on a sampled kit |
+ | a melody or harmony to arrange from the public domain | Mutopia, OpenScore |
+
+- Use what raea's licence rule takes: CC0 and public domain, CC-BY (credited), MIT and Apache, and free licences that
+ allow a commercial film (Sonniss's, Pixabay's, Mixkit's for its effects). Never a non-commercial (NC),
+ no-derivatives (ND), Sampling+, "personal use", share-alike (SA) or GPL source: share-alike and GPL terms can reach
+ the film or game made with it. A pack's licence is per pack and sometimes per sound; read its readme and keep it
+ beside the files.
+- Credit each CC-BY source where the piece is shown (its description or end credits). The line is in the
+ `LICENCE.txt` that `raea sounds get` writes, or in the pack's readme.
+- Keep a pack fetched by hand in one folder on the Mac, outside any project (`~/Music/raea-sounds/PACK/` with its
+ licence file), and copy the files a piece uses into its `media/` (`media/sounds/PACK/`): a render reads only its
+ own media, never a URL. With the raea app open, `raea library new sounds` and `raea library add sounds
+ ~/Music/raea-sounds` index that folder once the person says yes in the app; then `raea library search "metal door
+ slam" --kind audio --sound` finds a sound by what it sounds like.
+- raea reads WAV, MP3, FLAC and AAC (.m4a). Convert Ogg (most of Kenney's packs) and Opus to WAV first (`ffmpeg -i
+ in.ogg out.wav`).
+
+## Getting SAO and TokenSynth onto the Mac
+
+Both run on this computer and install themselves the first time a render, `raea sao` or `raea tokensynth` asks for
+a sound. To do it ahead of time and see where they are:
+
+```sh
+raea setup-audio --check # what's installed, where, and how much disk
+raea setup-audio # installs what's missing: TokenSynth, then SAO
+raea setup-audio sao # one of them
+```
+
+- **What it fetches.** Each model has a script (`TOOLS/sao/setup.sh` and `TOOLS/tokensynth/setup.sh`; TOOLS is
+ `tools/` in a raea checkout, `share/raea/tools/` beside an installed raea) that installs into `~/.cache/raea-sao`
+ and `~/.cache/raea-tokensynth` (RAEA_SAO and RAEA_TOKENSYNTH move them):
+ - SAO, about 2 GB: Stable Audio Open Small's Core ML conversion (the DiT, the VAE decoder and the number embedder)
+ from a Hugging Face mirror at a pinned revision, each file checked against its SHA-256; t5-base's tokenizer; a
+ small Python environment; and raea's own fp32 T5 encoder, converted once in a throwaway environment whose torch
+ and t5-base weights (about 900 MB) are deleted after.
+ - TokenSynth, about 6 GB: its code from GitHub, a Python environment with torch (about 1.1 GB), the model and its
+ unconditional twin (713 MB each), DAC (307 MB) and CLAP (2.35 GB) from Hugging Face, the text encoders CLAP
+ fetches, then a converted fp16 runtime and Core ML models.
+- **What it needs.** A network connection, `uv` (Astral's Python manager) and, for TokenSynth, `git`, which the
+ scripts take from the PATH. On a Mac without them, `TOOLS/ensure.sh` gets raea's pinned builds (uv 0.12.24 and git
+ 2.56.0, under `~/.cache/raea-tools`, each download checked against its SHA-256) and prints the program's path. Put
+ both on the PATH for the setup:
+
+ ```sh
+ B=$(dirname "$(realpath "$(command -v raea)")"); T=$B/../share/raea/tools; [ -d "$T" ] || T=$B/../../tools # an install, or a checkout
+ PATH="$(dirname "$(bash "$T/ensure.sh" uv)"):$(dirname "$(bash "$T/ensure.sh" git)"):$PATH" raea setup-audio
+ ```
+
+- **The licences.** TokenSynth's code and weights are MIT. Stable Audio Open Small is under Stability AI's Community
+ License (stability.ai/community-license-agreement), which isn't an open licence. As between the person and
+ Stability, the person owns the sounds it makes, as far as the law allows; the sounds may not be used to create or
+ improve another generative AI model, and Stability's acceptable use policy applies to them. The licence's other
+ duties fall first on raea, which hands the model on: a copy of the agreement and a notice file beside the model,
+ "Powered by Stability AI" where raea documents it, and registering raea's own commercial use. raea hasn't settled
+ those yet, nor what is left for the person (docs/sound.feature SN2), so don't tell the person what the licence asks
+ of them: before the first SAO sound goes into a commercial piece of theirs, say the model is under that licence,
+ give them the link and let them decide, or take that sound from a recorded pack instead.
+- **When it fails.** The error names the script and the folder; run that script with `sh` to see its own output.
+ `exit status: 127` is uv not found. On a Mac without the Command Line Tools, `/usr/bin/git` is only a stub: it
+ opens an install dialog, and the setup stops with status 1 and `xcrun: error: invalid active developer path`. Use
+ the git `ensure.sh` installs (above). `75` is another setup running in the same folder: wait for it, or remove the
+ folder's `.setup-lock` once nothing is running. RAEA_OFFLINE refuses the download. RAEA_AUDIO_AUTOSETUP=0 turns
+ the self-install off, so a render names the command instead; it also stops `raea setup-audio` and `ensure.sh`, so
+ with it set, run the model's script yourself (`sh TOOLS/sao/setup.sh`). Running setup-audio again picks up where a
+ broken install stopped: finished files are kept.
+- A render that meets a model not yet installed waits while it installs (minutes, once). Run setup-audio before the
+ first render, not inside one.
+
+## What `raea sounds` fetches for you
+
+`raea sounds WORDS` searches four libraries with no network (`kick`, `hihat` and the other words producers use find
+their parts). `raea sounds get PACK/INSTRUMENT` downloads one instrument's files once (into `~/.cache/raea-sounds`),
+copies them into `media/sounds/PACK-INSTRUMENT/` with a `LICENCE.txt`, and prints the `Sampler` (or the `Player`s)
+that plays them.
+
+| pack | licence | what it has |
+|---|---|---|
+| `salamander` | CC-BY 3.0 (credit) | a grand piano, sampled every minor third |
+| `fluid` | CC-BY 3.0 (credit) | the 128 General MIDI instruments of the FluidR3 font, every note A0 to C8; no drum kit |
+| `tonejs-instruments` | CC-BY 3.0 (credit) | 20, recorded from VSCO 2, the University of Iowa, Karoryfer and Freesound: cello, violin, contrabass, French horn, trumpet, trombone, tuba, saxophone, flute, clarinet, bassoon, harp, three guitars (acoustic, electric, nylon), electric bass, organ, harmonium, piano, xylophone |
+| `vcsl` | CC0 | 104 from the Versilian Community Sample Library: pianos, harpsichords, harps, organs, recorders, marimba, vibraphone, glockenspiel, kalimbas; bass drums, snares, toms, hi-hat, cymbals, claps, shakers, congas, timpani; a siren, a ratchet, an anvil and other effects |
+
+The ids are exact: `raea sounds kick` lists `vcsl/bass-drum-1`, and `vcsl/bass-drum` isn't one. A pitched
+instrument comes as one take a note, at its middle dynamic: with no `--notes`, roots at least a minor third apart from
+C1 to C7, the ones a Sampler needs, named as notes with sharps written `s` (`Ds3.mp3` is D#3). A drum or a hit comes
+as its first 12 takes (`--all` for every one). VCSL names a hit's dynamic and round robin in the file (a higher `_vN`
+is a harder hit, `_rr1`, `_rr2` the round robins), so a part can pick a softer take for a softer note.
+
+## Packs to fetch by hand
+
+Each goes into its own folder in `~/Music/raea-sounds/`, with its licence file beside it; the files a piece uses are
+then copied into its `media/`. These have a stable address, so one command fetches each (`git` is the one
+`TOOLS/ensure.sh git` prints on a Mac without the Command Line Tools):
+
+```sh
+P=~/Music/raea-sounds
+# VSCO 2 CE, the orchestra with its SFZ maps: 3.2 GB, CC0 (LICENSE inside)
+git clone --depth 1 --branch SFZ https://github.com/sgossner/VSCO-2-CE "$P/vsco2-ce"
+# VCSL, every dynamic and round robin, with SFZ maps: 6.2 GB, CC0 (the sfz branch has no LICENSE, so it's fetched beside)
+git clone --depth 1 --branch sfz https://github.com/sgossner/VCSL "$P/vcsl" && curl -fsSL -o "$P/vcsl/LICENSE" https://raw.githubusercontent.com/sgossner/VCSL/master/LICENSE
+# Salamander Grand Piano V3, 16 velocities: 0.75 GB, CC-BY 3.0 (LICENSE inside)
+git clone --depth 1 https://github.com/sfzinstruments/SalamanderGrandPiano "$P/salamander-v3"
+# Groove MIDI, the MIDI only: 3.1 MB, CC-BY 4.0, checked against the SHA-256 its page gives
+mkdir -p "$P/groove" && curl -fL -o "$P/groove/groove-midionly.zip" https://storage.googleapis.com/magentadata/datasets/groove/groove-v1.0.0-midionly.zip \
+ && echo "651cbc524ffb891be1a3e46d89dc82a1cecb09a57c748c7b45b844c4841dcc1e $P/groove/groove-midionly.zip" | shasum -a 256 -c \
+ && unzip -q "$P/groove/groove-midionly.zip" -d "$P/groove" \
+ && echo "Groove MIDI Dataset (Google Magenta), CC-BY 4.0: magenta.tensorflow.org/datasets/groove" > "$P/groove/LICENCE.txt"
+# MuseScore General, a General MIDI bank with drum kits: 40 MB as SF3, MIT, with its licence and notices
+mkdir -p "$P/musescore-general" && curl -fL --output-dir "$P/musescore-general" \
+ -O https://ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/MuseScore_General.sf3 \
+ -O https://ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/MuseScore_General_License.md
+```
+
+The rest are picked a pack or a sound at a time on their own pages. `raea sounds get` reaching all of them is filed
+as docs/sound.feature SN4.
+
+| source | licence | where | what it has |
+|---|---|---|---|
+| VSCO 2 Community Edition | CC0 | github.com/sgossner/VSCO-2-CE (its `SFZ` branch has the maps) | an orchestra: solo and section strings with articulations, woodwinds, brass, percussion, keys; WAV with SFZ maps |
+| VCSL, the whole library | CC0 | github.com/sgossner/VCSL (its `sfz` branch has the maps) | every dynamic and round robin of what `raea sounds` gives one take of |
+| Salamander Grand Piano V3 | CC-BY 3.0 | github.com/sfzinstruments/SalamanderGrandPiano | the same piano at 16 velocities, 48 kHz, SFZ |
+| sfzinstruments' lists | per pack, on the list | sfzinstruments.github.io (pianos, strings, drums) | take the CC0, public domain and CC-BY ones: Karoryfer's free packs (Big Rusty, Swirly and Unruly drums, Frankensnare, Meatbass, Bigcat Cello; mostly CC0, older ones CC-BY 3.0: read each readme), SM Drums (public domain), Splendid Grand Piano (public domain), DrumGizmo's DRS and Muldjord kits and Naked Drums (CC-BY 4.0) |
+| University of Iowa Musical Instrument Samples | "may be downloaded and used for any projects, without restrictions" | theremin.music.uiowa.edu/mis.html | single notes of orchestral instruments at several dynamics, recorded dry: give them a room's reverb |
+| FreePats | per bank: CC0, CC-BY or GPL; take the CC0 and CC-BY banks only | freepats.zenvoid.org | General MIDI families, world instruments, percussion; SFZ and SF2 |
+| MuseScore General | MIT (keep its notices) | ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/ | a General MIDI bank with drum kits, SF2 (216 MB) or SF3 (40 MB), for playing a MIDI file as it is |
+| Kenney's audio packs | CC0 | kenney.nl/assets/category:Audio | Impact Sounds, Interface Sounds, UI Audio, Sci-fi Sounds, RPG Audio, Casino Audio, Digital Audio, Music Jingles, voice-over packs; mostly Ogg |
+| OpenGameArt | per asset: take CC0, CC-BY and OGA-BY | opengameart.org | game effects and loops, such as "512 Sound Effects (8-bit style)" (CC0) |
+| Freesound | per sound: take CC0 and CC-BY only | freesound.org (a free account to download) | field recordings, foley and effects of every kind |
+| Sonniss #GameAudioGDC bundles | royalty-free for commercial use, no credit; never passed on as sounds (a library, a pack, a template) and never used to train a model | sonniss.com/gameaudiogdc | over 200 GB of professional foley, impacts, ambiences and field recordings, a bundle a year |
+| Pixabay sound effects | Pixabay's Content License: free, commercial use, no credit; never sold or shared on their own | pixabay.com/sound-effects | effects and ambiences |
+| Mixkit sound effects | Mixkit's Sound Effects Free License: free, commercial use (games and broadcast too), no credit; never passed on on their own (as stock, in a tool or a template) and never registered with a rights service such as Content ID | mixkit.co/free-sound-effects | effects, whooshes, interface sounds, ambiences |
+| Groove MIDI Dataset | CC-BY 4.0 (credit Google Magenta) | magenta.tensorflow.org/datasets/groove | 1,150 MIDI performances by 10 drummers: 13.6 hours of beats and fills in 18 styles |
+| Mutopia Project | per piece: public domain or CC-BY (skip the BY-SA ones) | mutopiaproject.org | 2,100 classical pieces as MIDI and LilyPond |
+| OpenScore Lieder | CC0 | github.com/OpenScore/Lieder | 1,350 art songs as MuseScore scores (MuseScore exports MIDI) |
+
+Not these, whatever they sound like: Sonatina Symphonic Orchestra and the Virtual Playing Orchestra (Sampling+ and
+mixed terms: no use in an ad), the Philharmonia's samples (terms unclear), Spitfire LABS (no samples to take), the
+MusyngKite and FatBoy fonts beside FluidR3 (CC BY-SA 3.0: share-alike), FreePats' GPL banks (copyleft), Tone.js's
+own drum kits (CC BY-NC-SA), the BBC's sound effects (personal and research use), Zapsplat (its own terms), Mixkit's
+stock music (not in games or broadcasts, and no remixing), Salamander Drumkit and other CC-BY-SA kits, and Lakh MIDI
+(copyrighted songs). Freesound's API is free only for non-commercial use, so raea doesn't search Freesound for the
+person: they download its sounds themselves.
+
+## Loading a pack in a composition
+
+- **One-shots** (an effect pack, a field recording): the file in `media/sfx/PACK/`, then ` ` in a Sequence at its frame (`motion/sound`), or a `Tone.Player`
+ for a game's one-shot.
+- **A multisampled instrument** (the SFZ packs): raea has no SFZ reader yet. The `.sfz` file is text: each ``
+ names a `sample=` file, its `pitch_keycenter` (the note it is), `lovel`/`hivel` (the velocities it plays) and
+ `seq_position` (its round robin). Copy the files for the notes and dynamics the part needs, map them by note, and
+ give each dynamic its own Sampler, picked by the note's velocity:
+
+ ```tsx
+ // SOFT and LOUD: { C3: staticFile("sounds/vsco-cello/..."), ... }, written from the .sfz's regions
+ const soft = new Tone.Sampler({ urls: SOFT, release: 0.6 }).connect(Tone.buses.music);
+ const loud = new Tone.Sampler({ urls: LOUD, release: 0.6 }).connect(Tone.buses.music);
+ const cello = (note: string, dur: string, t: number, vel: number) => (vel < 0.6 ? soft : loud).triggerAttackRelease(note, dur, t, vel);
+ ```
+
+- **A MIDI file** (a Groove MIDI performance, a Mutopia piece): turned into JSON and played on Samplers and Players
+ (below).
+- **A SoundFont** (`.sf2`, `.sf3`): raea can't play one. To hear a MIDI file the person hands over as it is, render
+ it with FluidSynth (`brew install fluid-synth`, then `fluidsynth -ni -F song.wav -r 48000 MuseScore_General.sf3
+ song.mid`) and play the WAV as ``; to write parts, use the sampled libraries instead.
+
+## A MIDI file in a composition
+
+A Groove MIDI performance or a Mutopia piece comes as a `.mid`. raea/music reads one (`parseMidi`, in
+sdk/audio/src/midi.ts), and the raea app imports a `.mid` into a listen item's Part, but a composition can't import
+a `.mid` yet (docs/sound.feature SN6). Turn it into JSON in raea/music's `MidiFile` shape (`tracks[].notes[]` with
+`midi`, `time` and `duration` in seconds, and `velocity` 0 to 1), each note from its note-on to its note-off, one
+track per channel; a composition imports that, and its code stays the same when a `.mid` can be imported:
+
+```sh
+uv run -q --no-project --with mido python3 -I - media/grooves/funk-92.mid media/grooves/funk-92.json <<'PY'
+import json, sys, mido
+f = mido.MidiFile(sys.argv[1])
+tempo, ticks, secs, held, tracks, tempos, sigs = 500000, 0, 0.0, {}, {}, [], []
+def end(ch, note, at, t, v): # one note, from its note-on to now, as raea/music's MidiNote
+ tracks.setdefault(ch, []).append({"midi": note, "time": round(at, 4), "duration": round(secs - at, 4), "velocity": round(v / 127, 3), "ticks": t, "durationTicks": ticks - t})
+for m in mido.merge_tracks(f.tracks):
+ ticks += m.time
+ secs += mido.tick2second(m.time, f.ticks_per_beat, tempo)
+ if m.type == "set_tempo":
+ tempo = m.tempo
+ tempos.append({"ticks": ticks, "bpm": round(mido.tempo2bpm(m.tempo), 3)})
+ elif m.type == "time_signature":
+ sigs.append({"ticks": ticks, "timeSignature": [m.numerator, m.denominator]})
+ elif m.type == "note_on" and m.velocity:
+ held.setdefault((m.channel, m.note), []).append((secs, ticks, m.velocity))
+ elif m.type in ("note_on", "note_off") and held.get((m.channel, m.note)):
+ end(m.channel, m.note, *held[(m.channel, m.note)].pop(0))
+for (ch, note), left in held.items(): # still sounding when the file ends
+ for n in left:
+ end(ch, note, *n)
+out = [{"name": f"channel {ch + 1}", "channel": ch, "notes": sorted(ns, key=lambda n: n["time"])} for ch, ns in sorted(tracks.items())]
+json.dump({"header": {"ppq": f.ticks_per_beat, "tempos": tempos, "timeSignatures": sigs, "name": ""}, "tracks": out, "duration": round(secs, 4)}, open(sys.argv[2], "w"))
+PY
+```
+
+A drum track plays on a kit of Players by note number. Groove MIDI was recorded on a Roland kit, whose map differs
+from General MIDI's: its hi-hat edge hits are 22 (closed) and 26 (open), notes General MIDI doesn't have, so a
+General MIDI map drops many of its hi-hats:
+
+```tsx
+import groove from "./media/grooves/funk-92.json"; // seconds, at the drummer's 92 bpm
+// Roland's drum map (the dataset's page lists the toms and cymbals): 36 kick; 37, 38 and 40 snare (cross-stick,
+// head, rim); 42, 22 and 44 closed hi-hat (bow, edge, pedal); 46 and 26 open hi-hat (bow, edge)
+const KIT: Record = { 36: "kick", 37: "snare", 38: "snare", 40: "snare", 42: "hat", 22: "hat", 44: "hat", 46: "open", 26: "open" };
+const kit = new Tone.Players({
+ kick: staticFile("sounds/vcsl-bass-drum-1/BDrumNew_hit_v5_rr1_Sum.wav"),
+ snare: staticFile("sounds/vcsl-snare-drum-modern-1/Snare2_HitSN_v5_rr1_Mid.wav"),
+ hat: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v3_rr1_Mid.wav"),
+ ghost: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitC_v1_rr1_Mid.wav"),
+ open: staticFile("sounds/vcsl-hi-hat-cymbal/HiHat_HitO_rr1_Mid.wav"),
+}).connect(Tone.buses.music);
+const stretch = 92 / Tone.getTransport().bpm.value; // the drummer's tempo onto the cue's
+const hits = groove.tracks.flatMap((track) => track.notes).filter((n) => n.midi in KIT);
+// a soft closed hat plays the soft take
+new Tone.Part((t, n) => kit.player(KIT[n.midi] === "hat" && n.velocity < 0.45 ? "ghost" : KIT[n.midi]).start(t), hits.map((n) => ({ ...n, time: n.time * stretch }))).start("4m");
+```
+
+A pitched track (a Mutopia piece) plays on a Sampler, one Part a track, each note at its own length:
+`piano.triggerAttackRelease(Tone.Frequency(n.midi, "midi").toNote(), n.duration * stretch, t, n.velocity)`.
diff --git a/skills/raea/reference/cli.md b/skills/raea/reference/cli.md
index 301bff2..f9767c3 100644
--- a/skills/raea/reference/cli.md
+++ b/skills/raea/reference/cli.md
@@ -132,10 +132,30 @@ Flags: `--as` (1), `--ask` (1), `--choice` (1), `--clip` (1), `--done` (1), `--i
Access: HEADLESS. Arguments besides flags: any number.
```
-raea clothes [make NAME ["WHAT IT IS"] [--plain] [--set PATH=VALUE,...] [--colour #RRGGBB] | settings | cover NAME --parts PART[:SHARE],... --colour #RRGGBB [--stand MM] [--over UNDER] | wear OUT_DIR GARMENT... [--body DIR] [--head HEAD_DIR]] (clothes for SOMA's body from words: make has an image model (flux_2_pro, one picture a view, at most N cents each, 10 unless said) dress SOMA's grey mannequin from the front, back, left and right, then measures the pictures onto SOMA's own vertices as assets/clothes/NAME.json, its pictures in assets/clothes/NAME/; --over draws it over garments already made (innermost first), as the next layer, a garment made by procedure drawn on the mannequin first; remeasure measures a garment again from its kept pictures (after raea's measuring changed), nothing drawn or paid for; cover makes one by procedure in the same file, no picture: the skin of the parts named (hips, waist, torso, chest, neck, head, upperArm, foreArm, hand, thigh, shin, foot, toes; a limb's SHARE is how far along it from its top), in one colour, MM off the skin (6 unless said); wear puts garments on a body, innermost first (each GARMENT a name or its .json): OUT_DIR/NAME.glb each, with OUT_DIR/body.glb the body without the skin wholly under cloth; --body DIR another SOMA body folder (a headless one for a head part); --head HEAD_DIR the head that joins it (head.json), whose skin the body's is graded to at the collar; alone, the project's garments. The first use fetches SOMA's body (NVlabs/SOMA-X, Apache-2.0) into raea's models folder and needs Blender: raea setup blender)
+raea clothes [make NAME ["WHAT IT IS"] [--plain] [--set PATH=VALUE,...] [--colour #RRGGBB] | settings | cover NAME --parts PART[:SHARE],... --colour #RRGGBB [--stand MM] [--over UNDER] [--hard [--sole MM] [--sole-colour #RRGGBB] [--panels #RRGGBB,...]] | wear OUT_DIR GARMENT... [--body DIR] [--head HEAD_DIR]] (clothes for SOMA's body from words: make sews a garment: a sewing pattern (GarmentCode, MIT) from the words, or the plain base with --plain, its settings changed by --set PATH=VALUE (settings lists them and what each may be), cut to SOMA's measurements, draped by raea's own cloth and baked, as assets/clothes/NAME.json and its .glb, in --colour or the palette's next; the first make sets up GarmentCode and its Python in raea's models folder; cover makes one by procedure in the same file, no picture: the skin of the parts named (hips, waist, torso, chest, neck, head, upperArm, foreArm, hand, thigh, shin, foot, toes; a limb's SHARE is how far along it from its top), in one colour, MM off the skin (6 unless said), --over a garment already made as the one under it; wear puts garments on a body, innermost first (each GARMENT a name or its .json): OUT_DIR/NAME.glb each, with OUT_DIR/body.glb the body without the skin wholly under cloth; --body DIR another SOMA body folder (a headless one for a head part); --head HEAD_DIR the head that joins it (head.json), whose skin the body's is graded to at the collar; alone, the project's garments. The first use fetches SOMA's body (NVlabs/SOMA-X, Apache-2.0) into raea's models folder and needs Blender: raea setup blender)
```
-Flags: `--body` (1), `--colour` (1), `--head` (1), `--over` (1), `--parts` (1), `--plain`, `--set` (1), `--stand` (1).
+Flags: `--body` (1), `--colour` (1), `--hard`, `--head` (1), `--over` (1), `--panels` (1), `--parts` (1), `--plain`, `--set` (1), `--sole` (1), `--sole-colour` (1), `--stand` (1).
+
+## raea motion
+
+Access: HEADLESS. Arguments besides flags: 1.
+
+```
+raea motion "LINE" [--keys KEYS.json] [--seconds 5 | --frames N] [--seed 1] [--steps 100] [-o CLIP.json] (body motion made on this Mac by Kimodo (NVIDIA, on MLX): a SOMA clip from a line of words and, with --keys, Kimodo constraints (full-body keyframes, hand or foot targets, a root path, as sdk/body/tools/between.ts writes them), as the clip sdk/body/tools/fromkimodo.ts reads; the first use fetches the motion model (1.1 GB) into raea's models folder, and the first new line its text encoder (8.8 GB: Llama 3 8B in 8 bits with LLM2Vec's adapters); a line's words are read once and kept. Licences: Kimodo's weights NVIDIA's Open Model License, Meta Llama 3 the Meta Llama 3 Community License, LLM2Vec's adapters MIT)
+```
+
+Flags: `--frames` (1), `--keys` (1), `--seconds` (1), `--seed` (1), `--steps` (1), `-o` (1).
+
+## raea bug
+
+Access: HEADLESS. Arguments besides flags: any number.
+
+```
+raea bug TITLE... --text TEXT [--command CMD] [--error TEXT] [--agent NAME] [--dry-run] (a bug in raea itself, not in your own composition, sent to raea's server with raea's version and the OS, no account needed: words only, paths, emails and keys taken out before it goes; five a day from a machine, the same bug again counted on the first report; --dry-run prints what would go and sends nothing)
+```
+
+Flags: `--agent` (1), `--command` (1), `--dry-run`, `--error` (1), `--text` (1).
## raea spec
diff --git a/skills/raea/reference/feed.md b/skills/raea/reference/feed.md
index f898d21..ef0aa58 100644
--- a/skills/raea/reference/feed.md
+++ b/skills/raea/reference/feed.md
@@ -25,6 +25,7 @@ Post an item to the project's feed: a render to watch (.mp4), a still to look at
| `changed` | optional | with replaces: one short line on what this version changed ("title bigger, cut the second shot"); the card shows it beside the new picture |
| `answers` | optional | with replaces: the ids of the person's comments on that item this version settles; each shows done, linked to this version |
| `export` | optional | true when this is a deliverable (an export for the client or a platform): it shows under Exports in the project's Table |
+| `final` | optional | true when this version is ready to share with reviewers, as you'd send it: it's tagged final and offered first when the person makes a review link. Say so only when it is, after the person's notes on it are settled |
| `tags` | optional | tags; key:value tags (platform:tiktok, aspect:9x16, cut:15s, market:uk) become the Table's columns, so use the same keys across a project's exports |
| `uses` | optional | the library files it was made from (footage, stills, music) |
| `controls` | optional | play: the game's controls |
@@ -59,7 +60,7 @@ Mark the person's comments on an item as you work them: each edit comment they s
Access: PAGE.
-Set the tags of an item already in the project's feed (feed_items lists them with their tags), to give the project's Table its structure: key:value tags (platform:tiktok, aspect:9x16) are its columns, and the tag export makes the item a deliverable. Replaces the item's own tags.
+Set the tags of an item already in the project's feed (feed_items lists them with their tags), to give the project's Table its structure: key:value tags (platform:tiktok, aspect:9x16) are its columns, the tag export makes the item a deliverable, and the tag final says this version is ready to share (it's offered first for a review link; put it on the newest version only, once it is). Replaces the item's own tags.
| input | | |
|---|---|---|
@@ -323,6 +324,20 @@ Change a live review link: its title, how many more days it works (from now), do
| `parent` | optional | the comment it answers |
| `resolve` | optional | a comment's id to mark resolved |
+## bug_report
+
+Access: HOST.
+
+Tell raea about a bug you found in raea itself: a raea command that fails or hangs, a wrong result, help or a guide that says one thing while raea does another. Not for a bug in your own composition. Words only: paths, emails and keys are taken out before it's sent, and the same bug again is counted on the first report. Then carry on with a workaround.
+
+| input | | |
+|---|---|---|
+| `title` | required | what's wrong, in a line |
+| `text` | required | what you did, what you expected, what happened |
+| `command` | optional | the raea command that went wrong, when there was one |
+| `error` | optional | what it printed |
+| `dry_run` | optional | show what would be sent, and send nothing |
+
## storage_status
Access: HOST.
diff --git a/skills/raea/reference/files.md b/skills/raea/reference/files.md
index cef7b14..78d85f0 100644
--- a/skills/raea/reference/files.md
+++ b/skills/raea/reference/files.md
@@ -64,9 +64,10 @@ that moves each (when there is one) and the files that name them.
| `~/.raea/cache` | `RAEA_EMOJI3D_CACHE` | apps/app/host/emoji3d.ts, apps/app/host/take.ts |
| `~/.raea/env` | `RAEA_ENV_FILE` | apps/app/host/director.ts, apps/app/host/plugins.ts, io/gen-core/src/fal.rs, io/gen-core/src/fish.rs, io/gen-core/src/made.rs, io/gen-core/src/modal.rs, io/gen-core/src/openrouter.rs, io/gen-core/src/replicate.rs, io/gen-core/src/tripo.rs, io/home/src/lib.rs |
| `~/.raea/evals/skills` | | apps/app/host/evals/skills.ts |
-| `~/.raea/feed` | | apps/app/host/feed-items.ts, apps/app/host/fingerprint.ts, apps/app/host/storage/owners.ts |
+| `~/.raea/feed` | | apps/app/host/feed-items.ts, apps/app/host/fingerprint.ts, apps/app/host/rules-asks.ts, apps/app/host/rules-train.ts, apps/app/host/storage/owners.ts |
+| `~/.raea/feed/i-taste-` | | apps/app/host/rules-asks.ts |
| `~/.raea/feed/media` | | apps/app/host/paths.ts |
-| `~/.raea/fingerprints` | `RAEA_FINGERPRINTS` | apps/app/host/fingerprint.ts |
+| `~/.raea/fingerprints` | `RAEA_FINGERPRINTS` | apps/app/host/fingerprint.ts, apps/app/host/rules-train.ts, apps/app/host/taste.ts |
| `~/.raea/fingerprints/default` | | apps/app/host/fingerprint.ts |
| `~/.raea/inbox` | `RAEA_INBOX_DIR` | apps/app/host/inbox.ts, engine/cli/src/show.ts |
| `~/.raea/inbox/INBOX.jsonl` | | engine/cli/src/cli.rs |
@@ -87,12 +88,13 @@ that moves each (when there is one) and the files that name them.
| `~/.raea/plugins.json` | `RAEA_PLUGINS_FILE` | apps/app/host/plugins.ts |
| `~/.raea/publish/drive.json` | `RAEA_DRIVE_PICKS` | apps/app/host/publish-source.ts |
| `~/.raea/publish/ready.json` | `RAEA_PUBLISH_READY` | apps/app/host/publish-ready.ts |
-| `~/.raea/sessions` | `RAEA_SESSIONS_DIR` | apps/app/host/sessions.ts |
+| `~/.raea/renders` | | apps/app/host/renderProgress.ts, runtime/encode/src/progress.rs |
+| `~/.raea/sessions` | `RAEA_SESSIONS_DIR` | apps/app/host/fingerprint-uses.ts, apps/app/host/sessions.ts |
| `~/.raea/storage-log.jsonl` | `RAEA_STORAGE_LOG` | apps/app/host/storage/trash.ts |
| `~/.raea/storage-used.json` | `RAEA_STORAGE_USED` | apps/app/host/storage/back.ts |
| `~/.raea/storage.json` | `RAEA_STORAGE_FILE` | apps/app/host/storage/plan.ts, apps/app/host/storage/service.ts |
-| `~/.raea/taste` | `RAEA_TASTE` | apps/app/host/taste.ts, engine/cli/src/spec.rs |
-| `~/.raea/taste/corrections.jsonl` | | apps/app/host/taste.ts |
+| `~/.raea/taste` | `RAEA_TASTE` | apps/app/host/rules.ts, apps/app/host/taste.ts, engine/cli/src/spec.rs |
+| `~/.raea/taste/corrections.jsonl` | | apps/app/host/rules-asks.ts, apps/app/host/rules-file.ts, apps/app/host/taste.ts |
| `~/.raea/taste/taste.ts` | | engine/cli/src/guide.rs |
| `~/.raea/trash` | `RAEA_TRASH_DIR` | apps/app/host/storage/trash.ts, apps/app/host/storage/usage.ts |
| `~/.raea/update` | `RAEA_UPDATE_DIR` | apps/app/host/onboard.ts, apps/app/host/update.ts |