Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions apps/tools/src/sound.rs
Original file line number Diff line number Diff line change
Expand Up @@ -253,11 +253,11 @@ pub fn setup_audio(args: &[String]) -> R<String> {
}
let now = raea_io_assets::installed(m, &dir);
missing += !now as usize;
let what = if m == "sao" { "foley (Stable Audio Open Small, Core ML)" } else { "music: an instrument's notes (TokenSynth)" };
let what = if m == "sao" { "foley (Stable Audio Open Small: Core ML on a Mac, else ONNX)" } else { "music: an instrument's notes (TokenSynth)" };
let state = match (was, now) {
(true, _) => "installed".to_string(),
(false, true) => "installed now".to_string(),
(false, false) => format!("not installed: {} installs it", raea_io_assets::setup_script(m).display()),
(false, false) => format!("not installed: `raea setup-audio` or {} installs it", raea_io_assets::setup_how(&raea_io_assets::setup_script(m))),
};
lines.push(format!(" {m:<10} {state}; {what}\n {} ({})", dir.display(), if now { raea_io_home::disk_use(&dir, false) } else { "-".into() }));
}
Expand Down
44 changes: 22 additions & 22 deletions docs/manual/reference/cli/setup-audio.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,34 +14,34 @@ raea setup-audio --check
raea setup-audio tokensynth
```

Each model's script (`share/raea/tools/sao/setup.sh` and `share/raea/tools/tokensynth/setup.sh` beside an installed
raea) installs it into `~/.cache/raea-sao` or `~/.cache/raea-tokensynth` (`RAEA_SAO` and `RAEA_TOKENSYNTH` move them):

- **SAO, about 2 GB:** Stable Audio Open Small's Core ML models from a Hugging Face mirror at a pinned revision, each
file checked against its SHA-256, t5-base's tokenizer, a small Python environment, and a T5 encoder converted once.
Each model's setup (`share/raea/tools/sao/setup.py` and `share/raea/tools/tokensynth/setup.py` beside an installed
raea, each with a `setup.sh` that runs it on a Mac or Linux) installs it into `~/.cache/raea-sao` or
`~/.cache/raea-tokensynth` (`RAEA_SAO` and `RAEA_TOKENSYNTH` move them):

- **SAO, about 2 GB:** on a Mac, Stable Audio Open Small's Core ML models from a Hugging Face mirror at a pinned
revision, each file checked against its SHA-256, t5-base's tokenizer, a small Python environment, and a T5 encoder
converted once. On Windows and Linux, the same weights as three ONNX graphs (each checked against its SHA-256) on
ONNX Runtime: with CUDA where `nvidia-smi` finds an NVIDIA GPU (its CUDA libraries add a few GB; `RAEA_SAO_CUDA=0`
keeps the CPU build), else on the CPU.
Its licence is Stability AI's Community License
([stability.ai/community-license-agreement](https://stability.ai/community-license-agreement)), not an open one.
As between you and Stability, you own the sounds it makes, as far as the law allows; they may not be used to
create or improve another generative AI model, and Stability's acceptable use policy applies to them. What else
the licence asks, of raea and of a commercial user, raea hasn't written down yet: read it before an SAO sound goes
into a commercial piece.
- **TokenSynth, about 6 GB:** its code from GitHub, a Python environment with torch, and its weights, DAC and CLAP
from Hugging Face, then a converted fp16 runtime. MIT.

Both need a network connection and [uv](https://docs.astral.sh/uv/) on the PATH; TokenSynth also needs `git`. On a
Mac without them, `share/raea/tools/ensure.sh uv` and `ensure.sh git` install raea's pinned builds (uv 0.12.24 and
git 2.56.0, each download checked against its SHA-256) and print where; put those folders on the PATH:

```sh
B=$(dirname "$(realpath "$(command -v raea)")"); T=$B/../share/raea/tools; [ -d "$T" ] || T=$B/../../tools # an install, or a checkout
PATH="$(dirname "$(bash "$T/ensure.sh" uv)"):$(dirname "$(bash "$T/ensure.sh" git)"):$PATH" raea setup-audio
```

When a setup fails, the error names its script: run that with `sh` to see why. Exit status 127 is uv not found; 1
with `xcrun: error: invalid active developer path` is macOS's stand-in git on a Mac without the Command Line Tools
(use the git `ensure.sh` installs); 75 is another setup running in the same folder. `RAEA_OFFLINE` refuses the
download. `RAEA_AUDIO_AUTOSETUP=0` stops a render from installing a model itself, and stops `raea setup-audio` and
`ensure.sh` too: with it set, run the model's script yourself (`sh share/raea/tools/sao/setup.sh`).
- **TokenSynth, about 6 GB:** its code from GitHub at a pinned commit (an archive: no git), a Python environment
with torch, and its weights, DAC and CLAP from Hugging Face, then a converted fp16 runtime (and, on a Mac, Core ML
models for `--precision coreml`). MIT.

Both need only a network connection, on a Mac, Windows or Linux: no Homebrew, git or FFmpeg. raea runs each setup
through [uv](https://docs.astral.sh/uv/), the one on the PATH or else its own (uv 0.12.24, fetched once into
`~/.cache/raea-tools/uv` and checked against its SHA-256; `RAEA_TOOLS_CACHE` moves it), and uv brings the Python.

When a setup fails, the error names the command: run it yourself to see why (`sh share/raea/tools/sao/setup.sh` on
a Mac or Linux; on Windows `uv run --no-project --python 3.11 python -I share\raea\tools\sao\setup.py`, with uv
from the PATH or `%USERPROFILE%\.cache\raea-tools\uv`). Exit status 75 is another setup running in the same folder.
`RAEA_OFFLINE` refuses the download. `RAEA_AUDIO_AUTOSETUP=0` stops a render from installing a model itself, and
stops `raea setup-audio` and `ensure.sh` too: with it set, run the model's setup yourself.

<!-- generated:begin by tools/manual-ref/cli.ts from `raea help --json`; edit only the flag descriptions -->

Expand Down
4 changes: 3 additions & 1 deletion docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ command asks for it (a few minutes, once), or ahead of time with `raea setup-aud
|---|---|
| `RAEA_TOKENSYNTH` | Where TokenSynth lives (default `~/.cache/raea-tokensynth`, about 6 GB): an instrument described in words, one note at a time, for a Tone.js `Sampler` |
| `RAEA_SAO` | Where Stable Audio Open Small lives (default `~/.cache/raea-sao`, about 2 GB, Core ML): foley and sound effects |
| `RAEA_AUDIO_AUTOSETUP=0` | Don't install on first use; a missing model's error names the command (`tools/sao/setup.sh`, `tools/tokensynth/setup.sh`) |
| `RAEA_AUDIO_AUTOSETUP=0` | Don't install on first use; a missing model's error names the command (`tools/sao/setup.sh` on a Mac or Linux, `uv run --no-project --python 3.11 python -I tools\sao\setup.py` on Windows) |
| `RAEA_SAO_CUDA` | SAO off a Mac: `1` installs ONNX Runtime with CUDA, `0` the CPU build; unset, an NVIDIA GPU (`nvidia-smi`) decides |
| `RAEA_TOOLS_CACHE` | Where raea keeps the uv it fetches when there's none on the PATH (default `~/.cache/raea-tools`; uv 0.12.24, checked against its SHA-256); every local model's setup runs through it |

From the command line, `raea sao "PROMPT" -o hit.wav` and `raea tokensynth "TIMBRE" --notes C3,C4 -o
notes/` make sounds through the same cache a composition's render uses, and `raea sounds` searches open sample
Expand Down
5 changes: 3 additions & 2 deletions docs/sound.feature
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ Feature: Sound for picture: the local models on any Mac, open libraries and pack
result in drones and pads and not actually design real complex music, scoring, sound effects, diegetic and
non-diegetic."

Where it stands (2026-11-16): `raea setup-audio` installs both models through tools/sao/setup.sh and
tools/tokensynth/setup.sh, and `raea sounds` reaches four libraries (sdk/raea/library/sounds/catalog.json:
Where it stands (2026-11-19): `raea setup-audio` installs both models through tools/sao/setup.py and
tools/tokensynth/setup.py, run by uv on a Mac, Windows or Linux with nothing installed first (no Homebrew, git
or FFmpeg; raea fetches uv itself when the PATH has none), and `raea sounds` reaches four libraries (sdk/raea/library/sounds/catalog.json:
salamander, fluid, tonejs-instruments, vcsl). The guides teach deciding what each scene is heard as, spotting the
cut and composing for it (sdk/guides/craft/motion/music.md, sound.md), and list every vetted source with its licence
and how to fetch it (motion/sources.md). What those guides had to write down as a gap is below.
Expand Down
27 changes: 11 additions & 16 deletions io/assets/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
//! seed, what tests use), the two local audio models, `tokensynth` (an instrument's notes, for music) and `sao` (Stable
//! Audio Open on Core ML, foley). Every model runs on this computer. `lyria` (music) and `fish` (narration, through
//! raea's API) are retired: a take made before plays from the index, and a new one is refused. Narration is Pocket TTS
//! (raea/speech), and Fish Audio is a `raea gen` backend like fal or Replicate (`fish:<model>`, the user's own key). The local models install themselves on first use (tools/sao/setup.sh,
//! tools/tokensynth/setup.sh). Each made asset gets a line in `assets/made.jsonl` under its cache key, a hash of
//! (raea/speech), and Fish Audio is a `raea gen` backend like fal or Replicate (`fish:<model>`, the user's own key). The local models install themselves on first use (tools/sao/setup.py,
//! tools/tokensynth/setup.py, run through uv on any OS: setup.rs). Each made asset gets a line in `assets/made.jsonl` under its cache key, a hash of
//! (model, prompt, seed, params); `made` reads only that index, so a render or a check that resolves a make() never
//! generates.

Expand All @@ -24,6 +24,8 @@ pub use splat::{ensure_splat, splat_dir, splat_params};

mod local;
use local::*;
mod setup;
pub use setup::{run_setup, setup_how, uv, venv_python};
#[cfg(test)]
mod tests;

Expand Down Expand Up @@ -192,11 +194,11 @@ pub fn setup_script(model: &str) -> PathBuf {
dunce::canonicalize(&p).unwrap_or(p)
}

/// Whether a local model can run from `dir`. SAO: the venv and either run.py with setup.sh's `.ready`, or the
/// Whether a local model can run from `dir`. SAO: the venv and either run.py with its setup's `.ready`, or the
/// hand-built spike's coreml_run.py (~/.cache/raea-sao before tools/sao existed, still served as it is). TokenSynth:
/// the venv, run.py, and setup.sh's `.ready` or the fp16 runtime that every finished install has.
/// the venv, run.py, and its setup's `.ready` or the fp16 runtime that every finished install has.
pub fn installed(model: &str, dir: &Path) -> bool {
let py = dir.join(".venv/bin/python").exists();
let py = venv_python(dir).exists();
match model {
"sao" => py && ((dir.join("run.py").exists() && dir.join(".ready").exists()) || dir.join("coreml_run.py").exists()),
"tokensynth" => py && dir.join("run.py").exists() && (dir.join(".ready").exists() || dir.join("weights-fp16/token_synth_aug.pt").exists()),
Expand All @@ -205,7 +207,7 @@ pub fn installed(model: &str, dir: &Path) -> bool {
}
}

/// A local model's install, set up first if it isn't there: setup.sh runs once, its progress on stderr (a render's
/// A local model's install, set up first if it isn't there: its setup runs once, its progress on stderr (a render's
/// log, the CLI's terminal), unless $RAEA_AUDIO_AUTOSETUP=0, when the error names the one command that installs it.
pub fn ensure_installed(model: &str) -> Result<PathBuf, String> {
let dir = install_dir(model);
Expand All @@ -218,7 +220,7 @@ pub fn ensure_installed(model: &str) -> Result<PathBuf, String> {
"matchmove" => "about 250 MB",
_ => "about 6 GB",
};
let how = format!("{} (installs it into {}, {size}, once)", script.display(), dir.display());
let how = format!("{} (installs it into {}, {size}, once)", setup_how(&script), dir.display());
if std::env::var("RAEA_AUDIO_AUTOSETUP").as_deref() == Ok("0") {
return Err(format!("{model} isn't installed: run {how}"));
}
Expand All @@ -227,14 +229,7 @@ pub fn ensure_installed(model: &str) -> Result<PathBuf, String> {
return Err(format!("{model} isn't installed in {}, and its setup script isn't at {}", dir.display(), script.display()));
}
eprintln!("raea: {model} isn't installed yet; setting it up now with {how}");
let env = install_var(model).0;
let status = std::process::Command::new("sh")
.arg(&script)
.env(env, &dir)
// its progress on stderr: stdout is the caller's answer (a render's APPLIED line, the CLI's)
.stdout(std::io::stderr())
.status()
.map_err(|e| format!("{model} setup: {e}"))?;
let status = run_setup(&script, install_var(model).0, &dir).map_err(|e| format!("{model} setup: {e}"))?;
if !status.success() || !installed(model, &dir) {
return Err(format!("{model} setup didn't finish ({status}): run {how}"));
}
Expand Down Expand Up @@ -307,7 +302,7 @@ impl Store {
std::fs::create_dir_all(&self.dir).map_err(|e| format!("{}: {e}", self.dir.display()))?;
let out = self.dir.join(format!(".matchmove-{key}.part"));
let _ = std::fs::remove_dir_all(&out);
let mut cmd = std::process::Command::new(dir.join(".venv/bin/python"));
let mut cmd = std::process::Command::new(venv_python(&dir));
// the repo's run.py when raea's files are here (so an edit to it is used without reinstalling), else the installed copy
let repo = setup_script("matchmove").with_file_name("run.py");
cmd.arg("-I").arg(if repo.exists() { repo } else { dir.join("run.py") }).arg("--video").arg(clip).arg("--out").arg(&out);
Expand Down
4 changes: 2 additions & 2 deletions io/assets/src/local.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ pub(crate) fn sao(prompt: &str, seed: u64, params: &J, part: &Path) -> Result<Ve
// the runner's ffmpeg picks the container from the extension
let wav = part.with_extension("wav");
let runner = if dir.join("run.py").exists() && dir.join(".ready").exists() { "run.py" } else { "coreml_run.py" };
let mut cmd = std::process::Command::new(dir.join(".venv/bin/python"));
let mut cmd = std::process::Command::new(venv_python(&dir));
// the spike's runner imports its sibling sao_test.py, so it can't run isolated (-I drops the script's folder)
if runner == "run.py" {
cmd.arg("-I");
Expand Down Expand Up @@ -51,7 +51,7 @@ pub(crate) fn tokensynth(prompt: &str, seed: u64, params: &J, part: &Path) -> Re
}
let dir = ensure_installed("tokensynth")?;
let wav = part.with_extension("wav");
let out = std::process::Command::new(dir.join(".venv/bin/python"))
let out = std::process::Command::new(venv_python(&dir))
.arg("-I")
.arg(dir.join("run.py"))
.args(["--timbre", prompt, "--seed", &seed.to_string()])
Expand Down
Loading
Loading