Skip to content

Commit ea5fc2e

Browse files
committed
MX1, MX2: raea-sys-mlx, MLX on Apple silicon's GPU beside raea-sys-ml's ONNX Runtime (docs/mlx.feature)
On mlx-rs =0.34.0 (MIT OR Apache-2.0), which builds MLX v0.32.2 from source with cmake at build time. The dependency is macOS-only, so the crate is empty on Linux and Windows. A model's MLX path is plain functions over its safetensors, read by the checkpoint's own tensor names (weights.rs), with f32 tensors in and out (tensor.rs) and layers that do PyTorch's maths (nn.rs): a Linear runs in its weight's dtype; LayerNorm, GELU and SiLU run in f32; attention is MLX's fused kernel. Tests check each layer against the same maths in plain Rust on the CPU. f32 matmuls run in TF32 on the GPU by default (within 3e-3; 1e-5 with MLX_ENABLE_TF32=0). use_metallib points MLX at a fetched metallib (MX4). No model uses it yet: IDOL, which was to be first, is held.
1 parent d415e0c commit ea5fc2e

10 files changed

Lines changed: 543 additions & 7 deletions

File tree

‎Cargo.lock‎

Lines changed: 133 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ members = [
7878
"systems/tone",
7979
"systems/mix",
8080
"systems/ml",
81+
"systems/mlx",
8182
"systems/speech",
8283
"systems/layout",
8384
"systems/vision",

‎docs/mlx.feature‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
@spec
2+
Feature: MLX: local models on Apple silicon's GPU (systems/mlx, raea-sys-mlx)
3+
Shane, 2026-11-10, told IDOL could run on MLX: "we should just do mlx anyway makes no sense not to support mlx
4+
generally going forward". Then, holding IDOL: "just have mlx support". On this M5 Pro Mac, ort's CPU kernels are the
5+
fastest ONNX path and too slow for big models. ort's Core ML provider has been slower still, and Core ML's GPU
6+
computes some models wrong (SigLIP: cosine 0.855 against the CPU's 0.98). MLX runs on the GPU directly.
7+
8+
So raea-sys-mlx sits beside raea-sys-ml's ONNX Runtime:
9+
- Each model gets an MLX path on the Mac.
10+
- ONNX stays the path on Windows and Linux, and the cross-platform reference.
11+
- The facility reaches users only once a model and a verb use it.
12+
13+
Background:
14+
Given mlx-rs =0.34.0 (crates.io, MIT OR Apache-2.0, maintained), which builds MLX v0.32.2 from source at build
15+
time. Its cmake clones MLX from GitHub, and Xcode's Metal toolchain compiles the kernels.
16+
And a model's MLX path is plain functions over its safetensors, keyed by the checkpoint's own tensor names, so it
17+
reads the same files as its PyTorch and ONNX paths
18+
19+
Rule: the facility builds where MLX runs, and nowhere else
20+
@done
21+
Scenario: MX1. raea-sys-mlx links MLX on the Mac and is empty on Linux and Windows
22+
Then on macOS the crate links MLX and runs on the GPU (`on_gpu`)
23+
And on Linux and Windows it compiles to nothing and brings no MLX in: the dependency is macOS-only
24+
And 11-10, measured on the M5 Pro with renders sharing the GPU: a 4096^3 matmul runs at 9.2 TFLOPS in f32 and
25+
11.5 TFLOPS in f16
26+
27+
Rule: its layers do PyTorch's maths, so a model's numbers can be matched
28+
@done
29+
Scenario: MX2. Each layer matches the same maths written out on the CPU
30+
Then linear, layer norm, attention, GELU (tanh) and SiLU agree with plain Rust on the CPU
31+
(systems/mlx/src/tests.rs)
32+
And the tolerance is 1e-5, or 3e-3 for an f32 matmul or attention, which MLX runs in TF32 on the GPU unless
33+
MLX_ENABLE_TF32=0 (with it set, all of them agree to 1e-5)
34+
And a Linear runs in its weight's dtype and a LayerNorm in f32, as PyTorch's do, so a model's mixed precision
35+
can be mirrored
36+
And weights load from safetensors by name, and a missing name is an error that names it
37+
38+
Scenario: MX3. A model's MLX path is checked against its reference before it becomes the Mac default
39+
Given a model with a PyTorch or ONNX path
40+
Then each of its networks is compared with the reference on the same inputs (cosine and max error), against a bar
41+
written in the model's own feature file
42+
And the end-to-end output is compared the same way
43+
And the Mac default is picked by parity first, then speed, with both times stated
44+
45+
Rule: the metallib reaches the user's Mac
46+
Scenario: MX4. A Mac without Xcode runs an MLX op from the signed raea
47+
Given MLX loads mlx.metallib (184 MB, 59 MB gzipped) at run time. It looks, in order: the path set by
48+
`use_metallib`, beside the binary, Resources/, then the build machine's path (~/.mlx/lib/<key>)
49+
Then with the first model, the metallib either:
50+
- arrives with that model's files on first use, sha256-pinned and from a models release, then `use_metallib`
51+
- or ships beside raea in the Mac tarball and the app
52+
And release.yml's Mac job builds the crate
53+
54+
Rule: models use it
55+
Scenario: MX5. The first model runs on MLX on the Mac
56+
# IDOL (one image to a 3D Gaussian splat) was to be the first. It was held 11-10 (docs/handoff/idol.md, "Parked
57+
# 11-10"), with its six-network seam agreed.
58+
# Next candidates:
59+
# - SigLIP (perception's embeddings), which Core ML's GPU computes wrong on this Mac
60+
# - any local model whose ONNX run on the Mac's CPU is too slow to use

‎systems/mlx/Cargo.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[package]
2+
name = "raea-sys-mlx"
3+
version = "0.1.0"
4+
edition = "2024"
5+
publish = false
6+
description = "raea's MLX facility: models run on Apple silicon's GPU through MLX (mlx-rs), beside raea-sys-ml's ONNX Runtime"
7+
license = "FSL-1.1-ALv2"
8+
9+
[dependencies]
10+
anyhow = "1"
11+
12+
# MLX is Apple silicon's: on other systems the crate is empty and pulls nothing in
13+
[target.'cfg(target_os = "macos")'.dependencies]
14+
mlx-rs = "=0.34.0"
15+

‎systems/mlx/src/lib.rs‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
//! raea's MLX facility (docs/mlx.feature): models run on Apple silicon's GPU through MLX (mlx-rs, MIT OR Apache-2.0),
2+
//! beside raea-sys-ml's ONNX Runtime, which stays the path on Windows and Linux and the cross-platform reference.
3+
//!
4+
//! A model's MLX path is written as plain functions over a [`weights::Weights`] map keyed by the checkpoint's own
5+
//! tensor names, so it reads the same safetensors files its PyTorch and ONNX paths do, with the layers in [`nn`]
6+
//! matching PyTorch's numerics. No model uses it yet (IDOL, its first, is held: docs/handoff/idol.md).
7+
//!
8+
//! Precision: MLX runs f32 matmuls (and attention) on the GPU in TF32 unless MLX_ENABLE_TF32=0 is in the environment
9+
//! when the process makes its first matmul, process-wide (like PyTorch's `allow_tf32`; about 1e-3 relative). A model
10+
//! whose parity bar needs exact f32 says so; most run in f16 or bf16 anyway.
11+
//!
12+
//! The crate is empty on every OS but macOS.
13+
#![cfg(target_os = "macos")]
14+
15+
pub mod nn;
16+
pub mod tensor;
17+
pub mod weights;
18+
19+
#[cfg(test)]
20+
mod tests;
21+
22+
/// Points MLX at the metallib to load, ahead of one next to the binary and the path baked in at build time (raea is
23+
/// to fetch it with a model's files on first use, docs/mlx.feature MX4). Call before the first MLX operation.
24+
pub fn use_metallib(path: &std::path::Path) -> anyhow::Result<()> {
25+
mlx_rs::metal::set_metallib_path(path.to_string_lossy())
26+
.map_err(|e| anyhow::anyhow!("mlx: can't use metallib {}: {e}", path.display()))
27+
}
28+
29+
/// Whether MLX runs on the GPU here (it falls back to the CPU only where Metal is missing).
30+
pub fn on_gpu() -> bool {
31+
mlx_rs::Device::default() == mlx_rs::Device::gpu()
32+
}

0 commit comments

Comments
 (0)