[mparrett]

Magenta RealTime 2 is Google DeepMind's open-weights live music model for Apple Silicon. It plays whatever MIDI notes are held in each 40 ms frame, in the style of a text prompt. That maps onto Slab's pattern clips: the clip's notes go in, and a performance in a described style comes out ("solo grand piano, warm", "west coast hip hop, deep bass"). I'm opening this as a draft so we can agree on the direction first. The design is in docs/42-mrt2.md, which ends with three open questions.

What it adds

• Render with MRT2… in the arrangement's clip menu, for pattern clips with notes. A dialog asks for a prompt and a seed. The clip's notes go out as MIDI through the tempo map, the render shows its progress and can be cancelled, and the WAV lands on a new track under the clip at the clip's start, as one undo step. The same notes, prompt, and seed give byte-identical audio.
• Clip.mrt2(prompt, seed) in slabkit, so song scripts can do the same. It uses the same timing as the menu item and caches renders by their inputs, so re-running a script doesn't render again.
• The MRT2 pack (packs/mrt2.pack.json, tools/mrt2/mrt2.py). Its importer downloads the mrt2_small weights and resources from Hugging Face at a pinned revision, checks each file's SHA-256, then builds mrt_render against upstream's C++ engine (magentart::core). MLX comes prebuilt from the mlx-metal wheel, pinned to 0.31.2 because 0.32 can't load the published exports (RuntimeError: [import_function] Invalid string size when loading mrt2_small.mlxfn magenta/magenta-realtime#103). On a base M2 the first build took about 16 minutes and put 2 GB in <home>/Cache, mostly TFLite.
• mrt_render (tools/mrt2/native/): about 240 lines of C++ that turn a MIDI file, a prompt, and a seed into a WAV.

Slab runs mrt_render as a child process, the same way pack importers run. That keeps MLX, TFLite, and SentencePiece out of build.zig, so people who never install the pack build exactly what they build today. A model failure also can't take down the audio thread. Linking the engine in, which live playing would need, is in docs/42 under Later.

Licensing

The weights are CC-BY-4.0 and are downloaded, not shipped; the pack's README.txt carries the attribution. magentart::core is Apache-2.0 and is built on the user's machine. Google claims no rights in the output.

Verification

• zig build test: 506 passed and 2 skipped, plus the 2 bench tests. One skip is a new opt-in end-to-end test (SLAB_MRT2_TEST=1) that runs Slab's render job through an installed pack: clip to MIDI to mrt_render to WAV, with the log followed. It passes when enabled. The other is an existing test that depends on what's installed locally.
• New unit tests: MIDI timing through the tempo map, with notes cut at the clip's end; the mrt2 manifest parses and has an importer.
• The importer, run end to end into a scratch home: weights checked, mrt_render built, its one-second check render passing, and models/ installed last.
• In the app: rendered a pattern clip from demos/synthpop_8bar and got the new track under the clip; the pack card shows as installed.
• slabkit: a script with a tempo change under the clip renders onto the right track with the right length, and a second run reuses the cache.
• Overlapping notes of one pitch: a held C4 with a shorter C4 inside it renders byte-identical to the same C4 re-struck, so the inner note's release no longer cuts the held one.
• Concurrent slabkit renders: two scripts with different chords, rendering at once into one cache, produce the same WAVs as each rendered alone.
• Notes steer the output: on a four-bar Am F C E test score, the strongest pitch classes in each bar matched the chord tones in 11 of 12 bars, against 4 of 12 with no notes.

Notes

• mrt2_small renders at roughly real time on a base M2 (measured 2026-10-17). MLX wheels start at macOS 14, so MRT2 needs macOS 14 or later while Slab supports 13.
• Unrelated to this change: Homebrew's Zig is now 0.17, which removed b.args, so build.zig and fy/build.zig no longer configure with it. I built with Zig 0.16.

🤖 Generated with Claude Code