diff --git a/mseries/OS9-HANDOFF.md b/mseries/OS9-HANDOFF.md index 743fbbc1..e8c6e843 100644 --- a/mseries/OS9-HANDOFF.md +++ b/mseries/OS9-HANDOFF.md @@ -1,3 +1,11 @@ +## macOS MLX profile scope correction — 2026-10-30 + +The engine scoreboard now displays chip/OS-labelled dot plots and includes matching M1 Pro macOS MLX Qwen3.5 2B and 4B, Liquid LFM3 and dense Granite 1B short/long receipts, including SHA-1/SHA-256, kernel and local run timestamps. The installed TensorFold b3ab9ba short and long receipts are now represented as their own Linux rows; its long divergence is preserved. M4 rows remain absent until measured. Short Qwen2B, Liquid and Granite matched available Linux MLX text; long Qwen2B and short Qwen4B differed. Apple/Linux timing boundaries and numerical fidelity remain explicit. [Scoreboard](local-ai/ENGINE-SCOREBOARD.md). + +TensorFold batching update: public [PR #1561](https://github.com/ashhart/TensorFold/pull/1561) is closed and unmerged; its exact one-commit diff failed a clean apply check against installed `b3ab9ba` in two files. Earlier "unpublished" wording is historical. No patch was applied or tested; see [source audit](local-ai/receipts/2026-10-30-macos-metal-profile/tensorfold-batching-audit.md). + +Captured Apple Metal profiles on M1 Pro for Qwen3.5 4B, Qwen3 4B and Qwen3.5 9B at 512/4096 synthetic prompt tokens. These are exploratory Apple MLX results, not the TensorFold Vulkan 2B benchmark; no matching Linux or TensorFold run occurred. The current engine scoreboard also includes Liquid LFM3 1.7B and dense Granite 4.0 1B. Exact pins, traces, logs, context and proposed matched Mac work are in [the receipt](local-ai/receipts/2026-10-30-macos-metal-profile/README.md). Raw trace bundles remain local because they contain model buffers or system metadata. No project runtime or default changed. + ## Warren local AI and eGPU updates — 2026-10-30 Installed `mseries-nvidia-open-source` `595.112.07+apple.1-1` (SHA-256 `a397b68a...b209ab23`) containing the three prepared NVIDIA fixes and license at `/usr/src/mseries-nvidia-open-595.112.07`. This is source only; no NVIDIA module or matching CUDA userspace is installed. The [source package script](nvidia-egpu/package-source.sh) verifies the exact prepared diff before packaging. diff --git a/mseries/ROADMAP.md b/mseries/ROADMAP.md index e3dbed33..7413332f 100644 --- a/mseries/ROADMAP.md +++ b/mseries/ROADMAP.md @@ -1,3 +1,9 @@ +## macOS MLX reference and TensorFold comparison — 2026-10-30 + +The scoreboard now shows separate chip/OS dot plots and includes all four completed M1 Pro Mac scoreboard model references. Add M4 data only after real runs and investigate the observed 2B-long and 4B-short cross-platform text differences. + +Next: compare the recorded macOS references for the four `engine-compare.py` model pins with matching Linux in-process receipts, preserving the short/long prompt settings and separate in-process versus server timing boundaries. The exploratory 4B/9B Metal captures are not TensorFold or Linux baselines. See [the scope and receipts](local-ai/receipts/2026-10-30-macos-metal-profile/README.md). + ## WD22TB4 and wip4 candidate status — 2026-10-28 Current `dev` is `57b143c4afd78b0a4a0f9905e48e39632d1ecc3c`. Pages deployment diff --git a/mseries/local-ai/ENGINE-SCOREBOARD.md b/mseries/local-ai/ENGINE-SCOREBOARD.md index d9944345..8a18fbc5 100644 --- a/mseries/local-ai/ENGINE-SCOREBOARD.md +++ b/mseries/local-ai/ENGINE-SCOREBOARD.md @@ -1,55 +1,123 @@ # Local AI engine scoreboard -One table for every engine we run or watch on the M1 Pro. ★ marks the most functional setup per model, length and load: the shortest typical reply (first token plus 200 tokens) whose output matched every other engine and whose correctness gate did not fail. Regenerate with `python3 mseries/local-ai/scoreboard.py` after adding a receipt. Tracked in the "Local AI engine scoreboard" issue. - -![First token, streaming speed and typical reply time by engine](engine-scoreboard.svg) - -Each block is one model and load; each panel has its own scale. **First token** is the wait before text appears, **streaming** is decode speed after that, and **typical reply** is first token plus 200 tokens at that speed (for several requests at once, one request's share). Blue is the best value in each panel; a bar ending in › is clipped at three times the block's median. - -## Measured on this machine - -| ★ | Model | Engine | Settings | Tokens | Decode tok/s | TTFT s | Total s | Runs | Same output as MLX default | Gate | Kernel | Driver | Date | Receipt | -|---|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---| -| ★ | Granite 4.0 1B MLX 4-bit (IBM, dense) | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **53.77** | 0.23 | 0.81 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | -| | Granite 4.0 1B MLX 4-bit (IBM, dense) | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 52.17 | 0.29 | 0.88 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | -| | Granite 4.0 1B MLX 4-bit (IBM, dense) | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 44.96 | 0.30 | 0.99 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | -| | Granite 4.0 1B MLX 4-bit (IBM, dense) | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 16.51 | 0.39 | 2.27 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | -| ★ | LFM3-1.7B MLX 8-bit | **MLX (omarchy-mlx, in-process)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **27.49** | 0.30 | 1.44 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | -| | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 22.63 | 0.33 | 1.69 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | -| | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 16.90 | 0.39 | 2.23 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short.json](receipts/2026-10-28-engine-compare-lfm3-short.json) | -| | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 15.27 | 0.40 | 2.38 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short.json](receipts/2026-10-28-engine-compare-lfm3-short.json) | -| ★ | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | **57.69** | 2.97 | 4.44 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| ★ | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **57.18** | 0.20 | 0.75 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | -| ★ | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | **57.04** | 0.77 | 2.24 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-2B MLX 4-bit | experiment: serve-nolp-120k + Rapid-MLX singleton module loaded from its checkout | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 56.85 | 0.58 | 1.12 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | -| | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 56.39 | 0.45 | 1.00 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | -| | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 49.87 | 0.19 | 0.82 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | -| | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | 49.16 | 1.93 | 5.21 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 48.86 | 0.46 | 1.10 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | -| | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 47.19 | 0.46 | 1.13 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | -| | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 42.58 | 1.15 | 3.01 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 42.42 | 0.19 | 0.92 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | -| | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve + skip-logprob-read patch only (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 42.27 | 0.59 | 1.31 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | -| | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 36.02 | 0.51 | 1.37 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | -| | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 33.75 | 0.61 | 1.54 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | -| | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | 20.30 | 8.08 | 12.61 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 20.24 | 4.08 | 6.32 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1517 (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 32 | 2.40 | 2.00 | 14.92 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | -| ★ | Qwen3.5-2B MLX 4-bit | **MLX (omarchy-mlx, in-process)** | `MLX_OMARCHY_BATCH_WORK=120000` | 256 | **49.28** | 0.56 | 5.74 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | -| | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 256 | 43.17 | 0.40 | 6.27 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | -| | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 256 | 37.19 | 0.55 | 7.41 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | -| | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1517 (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 256 | 2.77 | 3.12 | 95.26 | 3 | no: diverges after 880 of 1301 chars | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | -| | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 28.01 | 0.39 | 1.50 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | -| ★ | Qwen3.5-4B MLX 4-bit | **Rapid-MLX 1d09cdb (HTTP server)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **27.53** | 0.18 | 1.30 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | -| | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 27.01 | 0.64 | 1.78 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | -| ★ | Qwen3.5-4B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | **25.77** | 2.04 | 4.97 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 24.79 | 0.71 | 1.96 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | -| | Qwen3.5-4B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 23.44 | 0.24 | 1.56 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | -| | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 23.14 | 0.65 | 1.99 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | -| | Qwen3.5-4B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 21.18 | 2.25 | 6.04 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | -| | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 21.00 | 0.85 | 2.34 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | -| | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 18.55 | 0.73 | 2.38 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | -| | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 12.76 | 6.27 | 10.03 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +Chip and OS labels distinguish measurements. The Linux table’s ★ marks the most functional setup per model, length and load: the shortest typical reply (first token plus 200 tokens) whose output matched every other engine and whose correctness gate did not fail. Regenerate with `python3 mseries/local-ai/scoreboard.py` after adding a receipt. Tracked in the "Local AI engine scoreboard" issue. + +![Dot plots of first token, decode speed and reply time by chip, operating system and engine](engine-scoreboard.svg) + +Each dot is a measured engine configuration. Chip, OS and driver are identified; macOS and Aurora have different runtime versions and conditions. The plots group the same model revision, weight hash, prompt hash and token limit, but do not establish token or numerical equivalence across systems. First token and estimated reply time are lower-is-better; decode is higher-is-better. The 200-token value extrapolates from measured decode speed. HTTP and in-process timing boundaries differ. No dot implies a winner across systems. The Linux-only star and gates below are unchanged. + +## Relative in-process performance + +![Dot plot of Aurora decode and first-token time relative to macOS](relative-scoreboard.svg) + +Ratios are separate-run medians on the same named chip, model revision, weight hash, prompt hash and token limit. The same chip name does not guarantee the same GPU configuration. 1× means the macOS reference; higher values mean more Aurora time. Decode time uses macOS decode tok/s divided by Aurora decode tok/s; first-token time uses Aurora seconds divided by macOS seconds. Only in-process MLX runs with at least three trials appear. The runs were not paired across OS boots, and runtimes, load and thermal state differ; these ratios do not identify a cause or prove a regression. Text-match flags are limited to these prompts and do not establish numerical fidelity. + +| Chip | Model | Token limit | Mac OS / runtime | Aurora OS / kernel / driver | Aurora budget | Decode time vs Mac | First token vs Mac | Text | Runs Mac / Aurora | Receipts | +|---|---|---:|---|---|---|---:|---:|---|---|---| +| M1 Pro | Granite 4.0 1B MLX 4-bit (IBM, dense) | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | MLX_OMARCHY_BATCH_WORK=120000 | 2.09× | 1.73× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-granite1b-short.json) · [Aurora](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| M1 Pro | Granite 4.0 1B MLX 4-bit (IBM, dense) | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.43× | 1.78× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-granite1b-short.json) · [Aurora](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| M1 Pro | LFM3-1.7B MLX 8-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | MLX_OMARCHY_BATCH_WORK=120000 | 4.03× | 2.76× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-lfm3-short.json) · [Aurora](receipts/2026-10-28-engine-compare-lfm3-short.json) | +| M1 Pro | LFM3-1.7B MLX 8-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 4.46× | 2.78× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-lfm3-short.json) · [Aurora](receipts/2026-10-28-engine-compare-lfm3-short.json) | +| M1 Pro | LFM3-1.7B MLX 8-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | MLX_OMARCHY_BATCH_WORK=120000 | 2.48× | 2.12× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-lfm3-short.json) · [Aurora](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | +| M1 Pro | LFM3-1.7B MLX 8-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 3.01× | 2.30× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-lfm3-short.json) · [Aurora](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | MLX_OMARCHY_BATCH_WORK=120000 | 2.18× | 1.87× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-short.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.85× | 2.05× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-short.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | MLX_OMARCHY_BATCH_WORK=120000 | 1.82× | 1.83× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.10× | 1.86× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip5` / Honeykrisp Mesa 9323171c12 | MLX_OMARCHY_BATCH_WORK=120000 | 1.89× | 1.71× | match | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) · [Aurora](receipts/2026-10-30-tensorfold-b3ab9ba/compare-short.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 256 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | MLX_OMARCHY_BATCH_WORK=120000 | 2.06× | 1.87× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-long.json) · [Aurora](receipts/2026-10-28-engine-compare-long.json) | +| M1 Pro | Qwen3.5-2B MLX 4-bit | 256 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.73× | 1.84× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen2b-long.json) · [Aurora](receipts/2026-10-28-engine-compare-long.json) | +| M1 Pro | Qwen3.5-4B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | MLX_OMARCHY_BATCH_WORK=120000 | 2.16× | 2.17× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen4b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen4b-short.json) | +| M1 Pro | Qwen3.5-4B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 7faf04c065 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.89× | 2.24× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen4b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen4b-short.json) | +| M1 Pro | Qwen3.5-4B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | MLX_OMARCHY_BATCH_WORK=120000 | 1.98× | 1.97× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen4b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | +| M1 Pro | Qwen3.5-4B MLX 4-bit | 32 | macOS 27.2 build 26B123 / MLX 0.31.2; mlx-lm 0.31.3 | Aurora Linux / `7.2.11-elementary-aurora72-wip4` / Honeykrisp Mesa 9323171c12 | defaults (MLX_OMARCHY_BATCH_WORK fallback 5000) | 2.32× | 2.00× | different | 3 / 3 | [Mac](receipts/2026-10-30-macos-scoreboard-qwen4b-short.json) · [Aurora](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | + +## GPU submission counter evidence + +These are separate historical M1 Pro Aurora Qwen3.5 2B work-budget experiments, not counters from the macOS or engine-comparison rows. Medians are per measured request, including prompt prefill and generation, not per token. The 80k long experiment was interrupted after two pairs. The counters describe activity and do not measure individual kernel time or prove a cause; the experiment also checked GTK callbacks, not display presentation or input latency. See the [method and limitations](receipts/2026-10-27-batch-work.md). + +| Receipt | Status | Generated tokens | Work budget | Runs | Vulkan submissions | Compute dispatches | Barriers emitted | Decode tok/s | Min free GiB | Max °C | Max fan RPM | Token IDs | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---| +| [2026-10-27-batch-work-120k-long.json](receipts/2026-10-27-batch-work-120k-long.json) | complete | 256 | 5000 | 3 | 3139 | 63403 | 63184 | 37.12 | 9.10 | 52.88 | 2522 | identical within receipt | +| [2026-10-27-batch-work-120k-long.json](receipts/2026-10-27-batch-work-120k-long.json) | complete | 256 | 120000 | 3 | 263 | 63411 | 59768 | 47.77 | 8.76 | 53.24 | 2521 | identical within receipt | +| [2026-10-27-batch-work-20k-ab.json](receipts/2026-10-27-batch-work-20k-ab.json) | complete | 32 | 5000 | 3 | 429 | 8931 | 8695 | 36.98 | 10.71 | 50.63 | 2522 | identical within receipt | +| [2026-10-27-batch-work-20k-ab.json](receipts/2026-10-27-batch-work-20k-ab.json) | complete | 32 | 20000 | 3 | 141 | 8931 | 8335 | 45.98 | 10.22 | 50.62 | 2522 | identical within receipt | +| [2026-10-27-batch-work-20k-desktop.json](receipts/2026-10-27-batch-work-20k-desktop.json) | complete | 32 | 5000 | 3 | 429 | 8931 | 8695 | 36.48 | 9.57 | 48.85 | 2515 | identical within receipt | +| [2026-10-27-batch-work-20k-desktop.json](receipts/2026-10-27-batch-work-20k-desktop.json) | complete | 32 | 20000 | 3 | 141 | 8931 | 8335 | 46.36 | 9.55 | 48.33 | 2515 | identical within receipt | +| [2026-10-27-batch-work-40k-ab.json](receipts/2026-10-27-batch-work-40k-ab.json) | complete | 32 | 5000 | 3 | 429 | 8931 | 8695 | 37.84 | 9.34 | 52.13 | 2522 | identical within receipt | +| [2026-10-27-batch-work-40k-ab.json](receipts/2026-10-27-batch-work-40k-ab.json) | complete | 32 | 40000 | 3 | 104 | 8931 | 8297 | 47.11 | 9.11 | 51.37 | 2522 | identical within receipt | +| [2026-10-27-batch-work-80k-desktop.json](receipts/2026-10-27-batch-work-80k-desktop.json) | complete | 32 | 5000 | 3 | 429 | 8931 | 8695 | 37.01 | 9.61 | 47.45 | 2528 | identical within receipt | +| [2026-10-27-batch-work-80k-desktop.json](receipts/2026-10-27-batch-work-80k-desktop.json) | complete | 32 | 80000 | 3 | 69 | 8931 | 8261 | 48.77 | 9.38 | 47.30 | 2515 | identical within receipt | +| [2026-10-27-batch-work-80k-long.json](receipts/2026-10-27-batch-work-80k-long.json) | interrupted | 256 | 5000 | 2 | 3139 | 63403 | 63184 | 35.82 | 8.89 | 50.25 | 2523 | identical within receipt | +| [2026-10-27-batch-work-80k-long.json](receipts/2026-10-27-batch-work-80k-long.json) | interrupted | 256 | 80000 | 2 | 519 | 63411 | 60024 | 46.21 | 8.79 | 49.01 | 2515 | identical within receipt | +| [2026-10-27-batch-work-ab.json](receipts/2026-10-27-batch-work-ab.json) | complete | 32 | 5000 | 3 | 429 | 8931 | 8695 | 36.99 | 10.48 | 50.27 | 2526 | identical within receipt | +| [2026-10-27-batch-work-ab.json](receipts/2026-10-27-batch-work-ab.json) | complete | 32 | 10000 | 3 | 248 | 8931 | 8449 | 43.24 | 10.52 | 51.20 | 2531 | identical within receipt | + +Apple Metal traces captured shader pipeline names but no shader timing or GPU counter set; the available evidence cannot rank individual kernels or attribute the cross-OS gap. Fresh paired profiles, matched correctness checks, sustained thermals and desktop responsiveness remain next steps. + +## Aurora Linux measurements + +| ★ | Chip | OS | Model | Engine | Settings | Token limit | Decode tok/s | TTFT s | Total s | Runs | Same output as reference in receipt | Gate | Kernel | Driver | Date | Receipt | +|---|---|---|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---| +| ★ | M1 Pro | Aurora Linux | Granite 4.0 1B MLX 4-bit (IBM, dense) | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **53.77** | 0.23 | 0.81 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| | M1 Pro | Aurora Linux | Granite 4.0 1B MLX 4-bit (IBM, dense) | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 52.17 | 0.29 | 0.88 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| | M1 Pro | Aurora Linux | Granite 4.0 1B MLX 4-bit (IBM, dense) | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 44.96 | 0.30 | 0.99 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| | M1 Pro | Aurora Linux | Granite 4.0 1B MLX 4-bit (IBM, dense) | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 16.51 | 0.39 | 2.27 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-29 | [2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json](receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json) | +| ★ | M1 Pro | Aurora Linux | LFM3-1.7B MLX 8-bit | **MLX (omarchy-mlx, in-process)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **27.49** | 0.30 | 1.44 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | +| | M1 Pro | Aurora Linux | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 22.63 | 0.33 | 1.69 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json) | +| | M1 Pro | Aurora Linux | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 16.90 | 0.39 | 2.23 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short.json](receipts/2026-10-28-engine-compare-lfm3-short.json) | +| | M1 Pro | Aurora Linux | LFM3-1.7B MLX 8-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 15.27 | 0.40 | 2.38 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-lfm3-short.json](receipts/2026-10-28-engine-compare-lfm3-short.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | **57.69** | 2.97 | 4.44 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **57.18** | 0.20 | 0.75 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | **57.04** | 0.77 | 2.24 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | experiment: serve-nolp-120k + Rapid-MLX singleton module loaded from its checkout | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 56.85 | 0.58 | 1.12 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 56.39 | 0.45 | 1.00 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 54.51 | 0.42 | 0.99 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip5` | Honeykrisp Mesa 9323171c12 | 2026-10-30 | [compare-short.json](receipts/2026-10-30-tensorfold-b3ab9ba/compare-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 49.87 | 0.19 | 0.82 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | 49.16 | 1.93 | 5.21 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 48.86 | 0.46 | 1.10 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 47.19 | 0.46 | 1.13 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 42.58 | 1.15 | 3.01 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 42.42 | 0.19 | 0.92 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve + skip-logprob-read patch only (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 42.27 | 0.59 | 1.31 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 36.02 | 0.51 | 1.37 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 33.75 | 0.61 | 1.54 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s)` | 32 | 20.30 | 8.08 | 12.61 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 20.24 | 4.08 | 6.32 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1544 b3ab9ba (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 32 | 2.89 | 1.91 | 12.64 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip5` | Honeykrisp Mesa 9323171c12 | 2026-10-30 | [compare-short.json](receipts/2026-10-30-tensorfold-b3ab9ba/compare-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1517 (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 32 | 2.40 | 2.00 | 14.92 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-short.json](receipts/2026-10-28-engine-compare-short.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | **MLX (omarchy-mlx, in-process)** | `MLX_OMARCHY_BATCH_WORK=120000` | 256 | **56.19** | 0.50 | 5.03 | 1 | reference | PASS | `7.2.11-elementary-aurora72-wip5` | Honeykrisp Mesa 9323171c12 | 2026-10-30 | [compare-long.json](receipts/2026-10-30-tensorfold-b3ab9ba/compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 256 | 49.28 | 0.56 | 5.74 | 3 | yes | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 256 | 43.17 | 0.40 | 6.27 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 256 | 37.19 | 0.55 | 7.41 | 3 | reference | PASS | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1544 b3ab9ba (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 256 | 2.79 | 2.97 | 94.48 | 1 | no: diverges after 880 of 1301 chars | not gated | `7.2.11-elementary-aurora72-wip5` | Honeykrisp Mesa 9323171c12 | 2026-10-30 | [compare-long.json](receipts/2026-10-30-tensorfold-b3ab9ba/compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-2B MLX 4-bit | TensorFold Vulkan PR 1517 (HTTP server) | `serve --backend vulkan --context 1024 --no-thinking` | 256 | 2.77 | 3.12 | 95.26 | 3 | no: diverges after 880 of 1301 chars | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-long.json](receipts/2026-10-28-engine-compare-long.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 28.01 | 0.39 | 1.50 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | **Rapid-MLX 1d09cdb (HTTP server)** | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | **27.53** | 0.18 | 1.30 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 27.01 | 0.64 | 1.78 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | +| ★ | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | **mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model)** | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | **25.77** | 2.04 | 4.97 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 24.79 | 0.71 | 1.96 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 23.44 | 0.24 | 1.56 | 3 | yes | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 23.14 | 0.65 | 1.99 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | Rapid-MLX 1d09cdb (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 21.18 | 2.25 | 6.04 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000` | 32 | 21.00 | 0.85 | 2.34 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | MLX (omarchy-mlx, in-process) | `defaults (MLX_OMARCHY_BATCH_WORK fallback 5000)` | 32 | 18.55 | 0.73 | 2.38 | 3 | reference | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 7faf04c065 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short.json](receipts/2026-10-28-engine-compare-qwen4b-short.json) | +| | M1 Pro | Aurora Linux | Qwen3.5-4B MLX 4-bit | mlx-omarchy-serve, mlx-lm backend, stock (HTTP server) | `MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s)` | 32 | 12.76 | 6.27 | 10.03 | 3 | yes (engines in this run agree) | not gated | `7.2.11-elementary-aurora72-wip4` | Honeykrisp Mesa 9323171c12 | 2026-10-28 | [2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json](receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json) | + +## Native macOS measurements + +| Chip | OS | Model | Token limit | Decode tok/s | TTFT s | Total s | Runs | Linux MLX text | Start (local) | End (local) | Model SHA-1 | Kernel | Receipt | +|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---| +| M1 Pro | macOS 27.2 build 26B123 | Granite 4.0 1B MLX 4-bit (IBM, dense) | 32 | 109.17 | 0.17 | 0.45 | 3 | text hash matches 2/2 Linux MLX rows | 2026-10-30 20:15:11 EDT-0400 | 2026-10-30 20:15:48 EDT-0400 | `362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-granite1b-short.json](receipts/2026-10-30-macos-scoreboard-granite1b-short.json) | +| M1 Pro | macOS 27.2 build 26B123 | Granite 4.0 1B MLX 4-bit (IBM, dense) | 256 | 106.60 | 0.18 | 2.58 | 3 | no comparable Linux text hash | 2026-10-30 20:16:04 EDT-0400 | 2026-10-30 20:16:54 EDT-0400 | `362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-granite1b-long.json](receipts/2026-10-30-macos-scoreboard-granite1b-long.json) | +| M1 Pro | macOS 27.2 build 26B123 | LFM3-1.7B MLX 8-bit | 32 | 68.04 | 0.14 | 0.59 | 3 | text hash matches 4/4 Linux MLX rows | 2026-10-30 20:18:38 EDT-0400 | 2026-10-30 20:19:20 EDT-0400 | `3b46079dd265f521a232051c54c6218398c5e4b6` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-lfm3-short.json](receipts/2026-10-30-macos-scoreboard-lfm3-short.json) | +| M1 Pro | macOS 27.2 build 26B123 | LFM3-1.7B MLX 8-bit | 256 | 76.15 | 0.18 | 3.53 | 3 | no comparable Linux text hash | 2026-10-30 20:19:36 EDT-0400 | 2026-10-30 20:20:35 EDT-0400 | `3b46079dd265f521a232051c54c6218398c5e4b6` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-lfm3-long.json](receipts/2026-10-30-macos-scoreboard-lfm3-long.json) | +| M1 Pro | macOS 27.2 build 26B123 | Qwen3.5-2B MLX 4-bit | 32 | 102.78 | 0.25 | 0.56 | 3 | text hash matches 5/5 Linux MLX rows | 2026-10-30 20:11:33 EDT-0400 | 2026-10-30 20:12:15 EDT-0400 | `e2219e3a77ad2fe2fed64b3ea7892d524e66d606` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-qwen2b-short.json](receipts/2026-10-30-macos-scoreboard-qwen2b-short.json) | +| M1 Pro | macOS 27.2 build 26B123 | Qwen3.5-2B MLX 4-bit | 256 | 101.38 | 0.30 | 2.82 | 3 | text hash matches 0/3 Linux MLX rows | 2026-10-30 20:12:31 EDT-0400 | 2026-10-30 20:13:26 EDT-0400 | `e2219e3a77ad2fe2fed64b3ea7892d524e66d606` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-qwen2b-long.json](receipts/2026-10-30-macos-scoreboard-qwen2b-long.json) | +| M1 Pro | macOS 27.2 build 26B123 | Qwen3.5-4B MLX 4-bit | 32 | 53.61 | 0.33 | 0.90 | 3 | text hash matches 0/4 Linux MLX rows | 2026-10-30 20:23:11 EDT-0400 | 2026-10-30 20:23:55 EDT-0400 | `8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-qwen4b-short.json](receipts/2026-10-30-macos-scoreboard-qwen4b-short.json) | +| M1 Pro | macOS 27.2 build 26B123 | Qwen3.5-4B MLX 4-bit | 256 | 55.18 | 0.43 | 5.04 | 3 | no comparable Linux text hash | 2026-10-30 20:24:11 EDT-0400 | 2026-10-30 20:25:20 EDT-0400 | `8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5` | `Darwin 27.2.0 RELEASE_ARM64_T6000` | [2026-10-30-macos-scoreboard-qwen4b-long.json](receipts/2026-10-30-macos-scoreboard-qwen4b-long.json) | + +These rows are Apple MLX in-process measurements. Text-hash matches cover these prompts only and do not establish token identity or numerical fidelity. LFM3 used the project-pinned injected implementation; its tokenizer emitted a custom-code prompt and continued without approval, and Transformers reported a model-type compatibility warning. See its receipts. M4 appears only when measured receipts identify an M4 chip and OS. ## Watched or reported elsewhere (not comparable to the table above) @@ -58,7 +126,7 @@ Each block is one model and load; each panel has its own scale. **First token** | Qwen3.8-2B (Q4_K_M GGUF, staged) | ANE (omarchy-ane staged decoder) | fp16 / int8 staged programs | 12-16 | reported: 13-inch M1 (H13), not run here | Needs the compiled 38-program H13 bundle and stride-aware layout binding; six incompatible packed surfaces found locally | UZU-AND-ANE-EVALUATION-2026-10-27.md; receipts/2026-10-27-ane-binding-handoff.md | | Qwen3.8-2B (Q4_K_M GGUF, staged) | ANE (ane-linux-experiments decoder preflight) | 38 ANE submissions per token | 8.23 | reported: M1, peak RSS 8.38 GB, not run here | Memory: does not fit beside the 6 GiB desktop reserve on our 32 GB M1 Pro today | receipts/2026-10-27-ane-qwen-preflight.md | | Parakeet TDT (dictation encoder) | ANE (whole encoder) | CPU-cluster boost held | 138 ms/encode (GPU 113 ms) | reported: 13-inch M1, not run here | Worth it for CPU and power, not speed | ane-linux-experiments README; UZU-AND-ANE-EVALUATION-2026-10-27.md | -| Qwen3.5-2B MLX 4-bit | TensorFold Vulkan + submit batching | unpublished follow-up to PR 1544 (batching build a8b888dd) | ~10.6 (M1 Max), ~6.9 (M1) | reported upstream, code not public | Watched in people-watch/branch-watch; re-run engine-compare when it lands | ashhart/TensorFold issue 1420 | +| Qwen3.5-2B MLX 4-bit | TensorFold Vulkan + submit batching | Public PR 1561, closed and unmerged (earlier batching build a8b888dd) | ~10.6 (M1 Max), ~6.9 (M1) | reported upstream; not tested here | Patch needs adaptation to installed b3ab9ba and M1 Pro fidelity/performance gates | [TensorFold PR 1561](https://github.com/ashhart/TensorFold/pull/1561) | | Qwen3.5 0.8B (Mirai format) | uzu (CPU backend) | backend-cpu, stable Rust | 0.36 | measured here 2026-10-27 | No Linux GPU backend; Metal only | UZU-AND-ANE-EVALUATION-2026-10-27.md | | LFM3-1.7B | Locals (Generated-Computers) batched server | 8 requests at once | 620 total (2.99x mlx-lm) | reported: M2 Pro 16 GB, macOS; not run here | Metal custom kernels and mlx-lm 0.32 (our MLX stack is 0.31.3); single-request gain is only 1.07x | Generated-Computers/Locals README | | Granite 4.0 H 1B MLX 4-bit (IBM, Mamba2 hybrid) | MLX (omarchy-mlx, in-process) | mlx-community/granite-4.0-h-1b-4bit | | left out; dense Granite 4.0 1B measured instead | GPU passes over several tokens at once are wrong for this hybrid (CPU reference PPL 1.217, GPU one token at a time 1.217, GPU 4-token passes 5,860); the dense build matches the CPU reference (PPL 1.24176 vs 1.24177, 10 prompts x 512 tokens) | measured here 2026-10-28 (v0.7.34 model gate for omarchy-mlx PR #296) | @@ -72,7 +140,7 @@ The engines above move fast, so this page is a living record on `dev`, not a one 3. **Re-test.** Build it in a private prefix (never system-wide), run its own gates first (`tensorfold-vulkan.sh test` / `gpu`, `mlx-gate.py sweep`), then `engine-compare.py` with the same models, prompts and guards. 4. **Publish.** Commit the new receipt JSON, run `scoreboard.py`, and open a PR to `dev` with the regenerated CSV, SVG and this page. Rows are replaced by the newest receipt; old receipts stay for history. 5. **Adopt.** A setting becomes our default only when it holds ★ *and* passes the MLX gate; change defaults in a separate PR. -6. **Version steps follow upstream.** We move mlx-lm (now 0.31.3) and the omarchy-mlx pin (now v0.7.33, `33630f9c`) only when omarchy-mlx moves its own `install.sh` default or cuts a release. Its 0.32 line (mlx-lm `94cdcae`, used by its oMLX packaging) is tested as a separate engine, not adopted as our default. `install-omlx.sh` installs oMLX that way (its own venv under `~/.local/share/omlx-linux`) for engine-compare's `omlx-120k` arm. +6. **Version steps follow upstream.** Check the current installer pins and deployment receipt; qualify any newer runtime in an isolated environment before changing the default. Rules: everything is documented in our own repo only; we do not post to upstream projects. Rows under "reported elsewhere" are other people's numbers on other hardware and are never ranked against ours. diff --git a/mseries/local-ai/engine-scoreboard.csv b/mseries/local-ai/engine-scoreboard.csv index 4473c1d4..04787c12 100644 --- a/mseries/local-ai/engine-scoreboard.csv +++ b/mseries/local-ai/engine-scoreboard.csv @@ -1,48 +1,60 @@ -best,model,engine,settings,tokens,decode_tok_s,ttft_s,total_s,runs,same_output,gate,kernel,driver,date,status,source,blocker -,"Granite 4.0 1B MLX 4-bit (IBM, dense)","MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,52.17,0.29,0.88,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, -,"Granite 4.0 1B MLX 4-bit (IBM, dense)","MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,44.96,0.30,0.99,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, -,"Granite 4.0 1B MLX 4-bit (IBM, dense)","mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,16.51,0.39,2.27,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, -★,"Granite 4.0 1B MLX 4-bit (IBM, dense)",mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,53.77,0.23,0.81,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, -,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,16.90,0.39,2.23,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short.json, -★,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,27.49,0.30,1.44,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json, -,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,15.27,0.40,2.38,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short.json, -,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,22.63,0.33,1.69,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json, -,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,47.19,0.46,1.13,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, -,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,56.39,0.45,1.00,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json, -★,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,256,49.28,0.56,5.74,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, -,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,36.02,0.51,1.37,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, -,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,48.86,0.46,1.10,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json, -,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),256,37.19,0.55,7.41,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, -,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,42.42,0.19,0.92,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, -,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,49.87,0.19,0.82,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, -,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,42.58,1.15,3.01,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,49.16,1.93,5.21,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,256,43.17,0.40,6.27,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, -,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,33.75,0.61,1.54,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, -,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,20.24,4.08,6.32,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,20.30,8.08,12.61,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + skip-logprob-read patch only (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,42.27,0.59,1.31,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, -★,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,57.18,0.20,0.75,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, -★,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,57.04,0.77,2.24,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -★,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,57.69,2.97,4.44,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-2B MLX 4-bit,experiment: serve-nolp-120k + Rapid-MLX singleton module loaded from its checkout,MLX_OMARCHY_BATCH_WORK=120000,32,56.85,0.58,1.12,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, -,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1517 (HTTP server),serve --backend vulkan --context 1024 --no-thinking,32,2.40,2.00,14.92,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, -,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1517 (HTTP server),serve --backend vulkan --context 1024 --no-thinking,256,2.77,3.12,95.26,3,no: diverges after 880 of 1301 chars,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, -,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,24.79,0.71,1.96,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, -,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,27.01,0.64,1.78,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json, -,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,18.55,0.73,2.38,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, -,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,23.14,0.65,1.99,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json, -,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,23.44,0.24,1.56,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, -★,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,27.53,0.18,1.30,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, -,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,21.18,2.25,6.04,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-4B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,21.00,0.85,2.34,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, -,Qwen3.5-4B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,12.76,6.27,10.03,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,Qwen3.5-4B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,28.01,0.39,1.50,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, -★,Qwen3.5-4B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,25.77,2.04,4.97,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, -,"Qwen3.8-2B (Q4_K_M GGUF, staged)",ANE (omarchy-ane staged decoder),fp16 / int8 staged programs,,12-16,,,,,,,,,"reported: 13-inch M1 (H13), not run here",UZU-AND-ANE-EVALUATION-2026-10-27.md; receipts/2026-10-27-ane-binding-handoff.md,Needs the compiled 38-program H13 bundle and stride-aware layout binding; six incompatible packed surfaces found locally -,"Qwen3.8-2B (Q4_K_M GGUF, staged)",ANE (ane-linux-experiments decoder preflight),38 ANE submissions per token,,8.23,,,,,,,,,"reported: M1, peak RSS 8.38 GB, not run here",receipts/2026-10-27-ane-qwen-preflight.md,Memory: does not fit beside the 6 GiB desktop reserve on our 32 GB M1 Pro today -,Parakeet TDT (dictation encoder),ANE (whole encoder),CPU-cluster boost held,,138 ms/encode (GPU 113 ms),,,,,,,,,"reported: 13-inch M1, not run here",ane-linux-experiments README; UZU-AND-ANE-EVALUATION-2026-10-27.md,"Worth it for CPU and power, not speed" -,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan + submit batching,unpublished follow-up to PR 1544 (batching build a8b888dd),,"~10.6 (M1 Max), ~6.9 (M1)",,,,,,,,,"reported upstream, code not public",ashhart/TensorFold issue 1420,Watched in people-watch/branch-watch; re-run engine-compare when it lands -,Qwen3.5 0.8B (Mirai format),uzu (CPU backend),"backend-cpu, stable Rust",,0.36,,,,,,,,,measured here 2026-10-27,UZU-AND-ANE-EVALUATION-2026-10-27.md,No Linux GPU backend; Metal only -,LFM3-1.7B,Locals (Generated-Computers) batched server,8 requests at once,,620 total (2.99x mlx-lm),,,,,,,,,"reported: M2 Pro 16 GB, macOS; not run here",Generated-Computers/Locals README,Metal custom kernels and mlx-lm 0.32 (our MLX stack is 0.31.3); single-request gain is only 1.07x -,"Granite 4.0 H 1B MLX 4-bit (IBM, Mamba2 hybrid)","MLX (omarchy-mlx, in-process)",mlx-community/granite-4.0-h-1b-4bit,,,,,,,,,,,left out; dense Granite 4.0 1B measured instead,measured here 2026-10-28 (v0.7.34 model gate for omarchy-mlx PR #296),"GPU passes over several tokens at once are wrong for this hybrid (CPU reference PPL 1.217, GPU one token at a time 1.217, GPU 4-token passes 5,860); the dense build matches the CPU reference (PPL 1.24176 vs 1.24177, 10 prompts x 512 tokens)" +best,chip,os,model_reference,weights_sha1,weights_sha256,prompt_sha256,start_time,end_time,model,engine,settings,tokens,decode_tok_s,ttft_s,total_s,runs,same_output,gate,kernel,driver,date,status,source,blocker +,M1 Pro,Aurora Linux,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,"Granite 4.0 1B MLX 4-bit (IBM, dense)","MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,52.17,0.29,0.88,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, +,M1 Pro,Aurora Linux,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,"Granite 4.0 1B MLX 4-bit (IBM, dense)","MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,44.96,0.30,0.99,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, +,M1 Pro,Aurora Linux,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,"Granite 4.0 1B MLX 4-bit (IBM, dense)","mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,16.51,0.39,2.27,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, +★,M1 Pro,Aurora Linux,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,"Granite 4.0 1B MLX 4-bit (IBM, dense)",mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,53.77,0.23,0.81,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-29,measured here,receipts/2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json, +,M1 Pro,Aurora Linux,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,16.90,0.39,2.23,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short.json, +★,M1 Pro,Aurora Linux,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,27.49,0.30,1.44,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,15.27,0.40,2.38,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short.json, +,M1 Pro,Aurora Linux,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,LFM3-1.7B MLX 8-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,22.63,0.33,1.69,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-lfm3-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,47.19,0.46,1.13,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,56.39,0.45,1.00,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,54.51,0.42,0.99,3,reference,PASS,7.2.11-elementary-aurora72-wip5,Honeykrisp Mesa 9323171c12,2026-10-30,measured here,receipts/2026-10-30-tensorfold-b3ab9ba/compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,256,49.28,0.56,5.74,3,yes,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,256,56.19,0.50,5.03,1,reference,PASS,7.2.11-elementary-aurora72-wip5,Honeykrisp Mesa 9323171c12,2026-10-30,measured here,receipts/2026-10-30-tensorfold-b3ab9ba/compare-long.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,36.02,0.51,1.37,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,48.86,0.46,1.10,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),256,37.19,0.55,7.41,3,reference,PASS,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,42.42,0.19,0.92,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,49.87,0.19,0.82,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,42.58,1.15,3.01,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,49.16,1.93,5.21,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,256,43.17,0.40,6.27,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,33.75,0.61,1.54,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,20.24,4.08,6.32,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,20.30,8.08,12.61,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + skip-logprob-read patch only (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,42.27,0.59,1.31,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,57.18,0.20,0.75,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,57.04,0.77,2.24,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s),32,57.69,2.97,4.44,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-c8-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,experiment: serve-nolp-120k + Rapid-MLX singleton module loaded from its checkout,MLX_OMARCHY_BATCH_WORK=120000,32,56.85,0.58,1.12,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen2b-short-hk9323171c12-serve-nolp-120k+serve-single-120k+rapid-120k.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1517 (HTTP server),serve --backend vulkan --context 1024 --no-thinking,32,2.40,2.00,14.92,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1544 b3ab9ba (HTTP server),serve --backend vulkan --context 1024 --no-thinking,32,2.89,1.91,12.64,3,yes,not gated,7.2.11-elementary-aurora72-wip5,Honeykrisp Mesa 9323171c12,2026-10-30,measured here,receipts/2026-10-30-tensorfold-b3ab9ba/compare-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1517 (HTTP server),serve --backend vulkan --context 1024 --no-thinking,256,2.77,3.12,95.26,3,no: diverges after 880 of 1301 chars,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-long.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,,,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan PR 1544 b3ab9ba (HTTP server),serve --backend vulkan --context 1024 --no-thinking,256,2.79,2.97,94.48,1,no: diverges after 880 of 1301 chars,not gated,7.2.11-elementary-aurora72-wip5,Honeykrisp Mesa 9323171c12,2026-10-30,measured here,receipts/2026-10-30-tensorfold-b3ab9ba/compare-long.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,24.79,0.71,1.96,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",MLX_OMARCHY_BATCH_WORK=120000,32,27.01,0.64,1.78,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,18.55,0.73,2.38,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"MLX (omarchy-mlx, in-process)",defaults (MLX_OMARCHY_BATCH_WORK fallback 5000),32,23.14,0.65,1.99,3,reference,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,23.44,0.24,1.56,3,yes,not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 7faf04c065,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000,32,27.53,0.18,1.30,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,Rapid-MLX 1d09cdb (HTTP server),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,21.18,2.25,6.04,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000,32,21.00,0.85,2.34,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,"mlx-omarchy-serve, mlx-lm backend, stock (HTTP server)",MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,12.76,6.27,10.03,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000,32,28.01,0.39,1.50,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-serve-120k+serve-patched-120k+rapid-120k-round4.json, +★,M1 Pro,Aurora Linux,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,,,Qwen3.5-4B MLX 4-bit,mlx-omarchy-serve + our four server patches + batching of 4 (HTTP server; runs the everyday model),MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s),32,25.77,2.04,4.97,3,yes (engines in this run agree),not gated,7.2.11-elementary-aurora72-wip4,Honeykrisp Mesa 9323171c12,2026-10-28,measured here,receipts/2026-10-28-engine-compare-qwen4b-short-hk9323171c12-c4-serve-120k+serve-patched-120k+rapid-120k-round2.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,2026-10-30 20:15:11 EDT-0400,2026-10-30 20:15:48 EDT-0400,"Granite 4.0 1B MLX 4-bit (IBM, dense)",Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,32,109.17,0.17,0.45,3,text hash matches 2/2 Linux MLX rows,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-granite1b-short.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f,362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb,b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,2026-10-30 20:16:04 EDT-0400,2026-10-30 20:16:54 EDT-0400,"Granite 4.0 1B MLX 4-bit (IBM, dense)",Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,256,106.60,0.18,2.58,3,no comparable Linux text hash,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-granite1b-long.json, +,M1 Pro,macOS 27.2 build 26B123,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,3b46079dd265f521a232051c54c6218398c5e4b6,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,2026-10-30 20:18:38 EDT-0400,2026-10-30 20:19:20 EDT-0400,LFM3-1.7B MLX 8-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,32,68.04,0.14,0.59,3,text hash matches 4/4 Linux MLX rows,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-lfm3-short.json, +,M1 Pro,macOS 27.2 build 26B123,LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7,3b46079dd265f521a232051c54c6218398c5e4b6,2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,2026-10-30 20:19:36 EDT-0400,2026-10-30 20:20:35 EDT-0400,LFM3-1.7B MLX 8-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,256,76.15,0.18,3.53,3,no comparable Linux text hash,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-lfm3-long.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,e2219e3a77ad2fe2fed64b3ea7892d524e66d606,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,2026-10-30 20:11:33 EDT-0400,2026-10-30 20:12:15 EDT-0400,Qwen3.5-2B MLX 4-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,32,102.78,0.25,0.56,3,text hash matches 5/5 Linux MLX rows,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-qwen2b-short.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3,e2219e3a77ad2fe2fed64b3ea7892d524e66d606,713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,2026-10-30 20:12:31 EDT-0400,2026-10-30 20:13:26 EDT-0400,Qwen3.5-2B MLX 4-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,256,101.38,0.30,2.82,3,text hash matches 0/3 Linux MLX rows,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-qwen2b-long.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926,2026-10-30 20:23:11 EDT-0400,2026-10-30 20:23:55 EDT-0400,Qwen3.5-4B MLX 4-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,32,53.61,0.33,0.90,3,text hash matches 0/4 Linux MLX rows,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-qwen4b-short.json, +,M1 Pro,macOS 27.2 build 26B123,mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3,8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5,5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db,2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921,2026-10-30 20:24:11 EDT-0400,2026-10-30 20:25:20 EDT-0400,Qwen3.5-4B MLX 4-bit,Apple MLX (in-process),MLX 0.31.2; mlx-lm 0.31.3,256,55.18,0.43,5.04,3,no comparable Linux text hash,not gated,Darwin 27.2.0 RELEASE_ARM64_T6000,Apple Metal,2026-10-30,measured here,receipts/2026-10-30-macos-scoreboard-qwen4b-long.json, +,,,,,,,,,"Qwen3.8-2B (Q4_K_M GGUF, staged)",ANE (omarchy-ane staged decoder),fp16 / int8 staged programs,,12-16,,,,,,,,,"reported: 13-inch M1 (H13), not run here",UZU-AND-ANE-EVALUATION-2026-10-27.md; receipts/2026-10-27-ane-binding-handoff.md,Needs the compiled 38-program H13 bundle and stride-aware layout binding; six incompatible packed surfaces found locally +,,,,,,,,,"Qwen3.8-2B (Q4_K_M GGUF, staged)",ANE (ane-linux-experiments decoder preflight),38 ANE submissions per token,,8.23,,,,,,,,,"reported: M1, peak RSS 8.38 GB, not run here",receipts/2026-10-27-ane-qwen-preflight.md,Memory: does not fit beside the 6 GiB desktop reserve on our 32 GB M1 Pro today +,,,,,,,,,Parakeet TDT (dictation encoder),ANE (whole encoder),CPU-cluster boost held,,138 ms/encode (GPU 113 ms),,,,,,,,,"reported: 13-inch M1, not run here",ane-linux-experiments README; UZU-AND-ANE-EVALUATION-2026-10-27.md,"Worth it for CPU and power, not speed" +,,,,,,,,,Qwen3.5-2B MLX 4-bit,TensorFold Vulkan + submit batching,"Public PR 1561, closed and unmerged (earlier batching build a8b888dd)",,"~10.6 (M1 Max), ~6.9 (M1)",,,,,,,,,reported upstream; not tested here,[TensorFold PR 1561](https://github.com/ashhart/TensorFold/pull/1561),Patch needs adaptation to installed b3ab9ba and M1 Pro fidelity/performance gates +,,,,,,,,,Qwen3.5 0.8B (Mirai format),uzu (CPU backend),"backend-cpu, stable Rust",,0.36,,,,,,,,,measured here 2026-10-27,UZU-AND-ANE-EVALUATION-2026-10-27.md,No Linux GPU backend; Metal only +,,,,,,,,,LFM3-1.7B,Locals (Generated-Computers) batched server,8 requests at once,,620 total (2.99x mlx-lm),,,,,,,,,"reported: M2 Pro 16 GB, macOS; not run here",Generated-Computers/Locals README,Metal custom kernels and mlx-lm 0.32 (our MLX stack is 0.31.3); single-request gain is only 1.07x +,,,,,,,,,"Granite 4.0 H 1B MLX 4-bit (IBM, Mamba2 hybrid)","MLX (omarchy-mlx, in-process)",mlx-community/granite-4.0-h-1b-4bit,,,,,,,,,,,left out; dense Granite 4.0 1B measured instead,measured here 2026-10-28 (v0.7.34 model gate for omarchy-mlx PR #296),"GPU passes over several tokens at once are wrong for this hybrid (CPU reference PPL 1.217, GPU one token at a time 1.217, GPU 4-token passes 5,860); the dense build matches the CPU reference (PPL 1.24176 vs 1.24177, 10 prompts x 512 tokens)" diff --git a/mseries/local-ai/engine-scoreboard.svg b/mseries/local-ai/engine-scoreboard.svg index 977c501d..58f09e7c 100644 --- a/mseries/local-ai/engine-scoreboard.svg +++ b/mseries/local-ai/engine-scoreboard.svg @@ -1 +1 @@ -First token, streaming speed and reply time by engine on Apple M1 ProGranite 4.0 1B MLX 4-bit (IBM, dense) · 32-token test · one requestFirst token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerUpstream server, patched (ships) ★Upstream server, patched (ships): First token, s 0.23 s0.23 sUpstream server, patched (ships): Streaming, tok/s 53.853.8Upstream server, patched (ships): Typical reply (200 tokens), s 4.0 s4.0 sMLX in-process, tunedMLX in-process, tuned: First token, s 0.29 s0.29 sMLX in-process, tuned: Streaming, tok/s 52.252.2MLX in-process, tuned: Typical reply (200 tokens), s 4.1 s4.1 sMLX in-process, defaultMLX in-process, default: First token, s 0.30 s0.30 sMLX in-process, default: Streaming, tok/s 45.045.0MLX in-process, default: Typical reply (200 tokens), s 4.7 s4.7 sUpstream server, stockUpstream server, stock: First token, s 0.39 s0.39 sUpstream server, stock: Streaming, tok/s 16.516.5Upstream server, stock: Typical reply (200 tokens), s 12.5 s12.5 sLFM3-1.7B MLX 8-bit · 32-token test · one requestFirst token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerMLX in-process, tuned ★MLX in-process, tuned: First token, s 0.30 s0.30 sMLX in-process, tuned: Streaming, tok/s 27.527.5MLX in-process, tuned: Typical reply (200 tokens), s 7.6 s7.6 sMLX in-process, defaultMLX in-process, default: First token, s 0.33 s0.33 sMLX in-process, default: Streaming, tok/s 22.622.6MLX in-process, default: Typical reply (200 tokens), s 9.2 s9.2 sMLX in-process, tuned · old driverMLX in-process, tuned · old driver: First token, s 0.39 s0.39 sMLX in-process, tuned · old driver: Streaming, tok/s 16.916.9MLX in-process, tuned · old driver: Typical reply (200 tokens), s 12.2 s12.2 sMLX in-process, default · old driverMLX in-process, default · old driver: First token, s 0.40 s0.40 sMLX in-process, default · old driver: Streaming, tok/s 15.315.3MLX in-process, default · old driver: Typical reply (200 tokens), s 13.5 s13.5 sQwen3.5-2B MLX 4-bit · 32-token test · one requestFirst token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerUpstream server, patched (ships) ★Upstream server, patched (ships): First token, s 0.20 s0.20 sUpstream server, patched (ships): Streaming, tok/s 57.257.2Upstream server, patched (ships): Typical reply (200 tokens), s 3.7 s3.7 sMLX in-process, tunedMLX in-process, tuned: First token, s 0.45 s0.45 sMLX in-process, tuned: Streaming, tok/s 56.456.4MLX in-process, tuned: Typical reply (200 tokens), s 4.0 s4.0 sUpstream + singleton (trial)Upstream + singleton (trial): First token, s 0.58 s0.58 sUpstream + singleton (trial): Streaming, tok/s 56.856.8Upstream + singleton (trial): Typical reply (200 tokens), s 4.1 s4.1 sRapid-MLX serverRapid-MLX server: First token, s 0.19 s0.19 sRapid-MLX server: Streaming, tok/s 49.949.9Rapid-MLX server: Typical reply (200 tokens), s 4.2 s4.2 sMLX in-process, defaultMLX in-process, default: First token, s 0.46 s0.46 sMLX in-process, default: Streaming, tok/s 48.948.9MLX in-process, default: Typical reply (200 tokens), s 4.6 s4.6 sMLX in-process, tuned · old driverMLX in-process, tuned · old driver: First token, s 0.46 s0.46 sMLX in-process, tuned · old driver: Streaming, tok/s 47.247.2MLX in-process, tuned · old driver: Typical reply (200 tokens), s 4.7 s4.7 sRapid-MLX server · old driverRapid-MLX server · old driver: First token, s 0.19 s0.19 sRapid-MLX server · old driver: Streaming, tok/s 42.442.4Rapid-MLX server · old driver: Typical reply (200 tokens), s 4.9 s4.9 sUpstream server + logprob patchUpstream server + logprob patch: First token, s 0.59 s0.59 sUpstream server + logprob patch: Streaming, tok/s 42.342.3Upstream server + logprob patch: Typical reply (200 tokens), s 5.3 s5.3 sMLX in-process, default · old driverMLX in-process, default · old driver: First token, s 0.51 s0.51 sMLX in-process, default · old driver: Streaming, tok/s 36.036.0MLX in-process, default · old driver: Typical reply (200 tokens), s 6.1 s6.1 sUpstream server, stockUpstream server, stock: First token, s 0.61 s0.61 sUpstream server, stock: Streaming, tok/s 33.733.7Upstream server, stock: Typical reply (200 tokens), s 6.5 s6.5 sTensorFold Vulkan · old driverTensorFold Vulkan · old driver: First token, s 2.00 s› 2.00 sTensorFold Vulkan · old driver: Streaming, tok/s 2.42.4TensorFold Vulkan · old driver: Typical reply (200 tokens), s 85.3 s› 85.3 sQwen3.5-2B MLX 4-bit · 32-token test · 4 requests at once (streaming = total)First token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerUpstream server, patched (ships) ★Upstream server, patched (ships): First token, s 0.77 s0.77 sUpstream server, patched (ships): Streaming, tok/s 57.057.0Upstream server, patched (ships): Typical reply (200 tokens), s 14.8 s14.8 sRapid-MLX serverRapid-MLX server: First token, s 1.15 s1.15 sRapid-MLX server: Streaming, tok/s 42.642.6Rapid-MLX server: Typical reply (200 tokens), s 19.9 s19.9 sUpstream server, stockUpstream server, stock: First token, s 4.08 s› 4.08 sUpstream server, stock: Streaming, tok/s 20.220.2Upstream server, stock: Typical reply (200 tokens), s 43.6 s43.6 sQwen3.5-2B MLX 4-bit · 32-token test · 8 requests at once (streaming = total)First token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerUpstream server, patched (ships) ★Upstream server, patched (ships): First token, s 2.97 s2.97 sUpstream server, patched (ships): Streaming, tok/s 57.757.7Upstream server, patched (ships): Typical reply (200 tokens), s 30.7 s30.7 sRapid-MLX serverRapid-MLX server: First token, s 1.93 s1.93 sRapid-MLX server: Streaming, tok/s 49.249.2Rapid-MLX server: Typical reply (200 tokens), s 34.5 s34.5 sUpstream server, stockUpstream server, stock: First token, s 8.08 s8.08 sUpstream server, stock: Streaming, tok/s 20.320.3Upstream server, stock: Typical reply (200 tokens), s 86.9 s86.9 sQwen3.5-2B MLX 4-bit · 256-token test · one requestFirst token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerMLX in-process, tuned · old driver ★MLX in-process, tuned · old driver: First token, s 0.56 s0.56 sMLX in-process, tuned · old driver: Streaming, tok/s 49.349.3MLX in-process, tuned · old driver: Typical reply (200 tokens), s 4.6 s4.6 sRapid-MLX server · old driverRapid-MLX server · old driver: First token, s 0.40 s0.40 sRapid-MLX server · old driver: Streaming, tok/s 43.243.2Rapid-MLX server · old driver: Typical reply (200 tokens), s 5.0 s5.0 sMLX in-process, default · old driverMLX in-process, default · old driver: First token, s 0.55 s0.55 sMLX in-process, default · old driver: Streaming, tok/s 37.237.2MLX in-process, default · old driver: Typical reply (200 tokens), s 5.9 s5.9 sTensorFold Vulkan · old driverTensorFold Vulkan · old driver: First token, s 3.12 s› 3.12 sTensorFold Vulkan · old driver: Streaming, tok/s 2.82.8TensorFold Vulkan · old driver: Typical reply (200 tokens), s 75.4 s› 75.4 sQwen3.5-4B MLX 4-bit · 32-token test · one requestFirst token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerRapid-MLX server ★Rapid-MLX server: First token, s 0.18 s0.18 sRapid-MLX server: Streaming, tok/s 27.527.5Rapid-MLX server: Typical reply (200 tokens), s 7.4 s7.4 sUpstream server, patched (ships)Upstream server, patched (ships): First token, s 0.39 s0.39 sUpstream server, patched (ships): Streaming, tok/s 28.028.0Upstream server, patched (ships): Typical reply (200 tokens), s 7.5 s7.5 sMLX in-process, tunedMLX in-process, tuned: First token, s 0.64 s0.64 sMLX in-process, tuned: Streaming, tok/s 27.027.0MLX in-process, tuned: Typical reply (200 tokens), s 8.0 s8.0 sRapid-MLX server · old driverRapid-MLX server · old driver: First token, s 0.24 s0.24 sRapid-MLX server · old driver: Streaming, tok/s 23.423.4Rapid-MLX server · old driver: Typical reply (200 tokens), s 8.8 s8.8 sMLX in-process, tuned · old driverMLX in-process, tuned · old driver: First token, s 0.71 s0.71 sMLX in-process, tuned · old driver: Streaming, tok/s 24.824.8MLX in-process, tuned · old driver: Typical reply (200 tokens), s 8.8 s8.8 sMLX in-process, defaultMLX in-process, default: First token, s 0.65 s0.65 sMLX in-process, default: Streaming, tok/s 23.123.1MLX in-process, default: Typical reply (200 tokens), s 9.3 s9.3 sUpstream server, stockUpstream server, stock: First token, s 0.85 s0.85 sUpstream server, stock: Streaming, tok/s 21.021.0Upstream server, stock: Typical reply (200 tokens), s 10.4 s10.4 sMLX in-process, default · old driverMLX in-process, default · old driver: First token, s 0.73 s0.73 sMLX in-process, default · old driver: Streaming, tok/s 18.618.6MLX in-process, default · old driver: Typical reply (200 tokens), s 11.5 s11.5 sQwen3.5-4B MLX 4-bit · 32-token test · 4 requests at once (streaming = total)First token, s · lower is betterStreaming, tok/s · higher is betterTypical reply (200 tokens), s · lowerUpstream server, patched (ships) ★Upstream server, patched (ships): First token, s 2.04 s2.04 sUpstream server, patched (ships): Streaming, tok/s 25.825.8Upstream server, patched (ships): Typical reply (200 tokens), s 33.1 s33.1 sRapid-MLX serverRapid-MLX server: First token, s 2.25 s2.25 sRapid-MLX server: Streaming, tok/s 21.221.2Rapid-MLX server: Typical reply (200 tokens), s 40.0 s40.0 sUpstream server, stockUpstream server, stock: First token, s 6.27 s6.27 sUpstream server, stock: Streaming, tok/s 12.812.8Upstream server, stock: Typical reply (200 tokens), s 69.0 s69.0 s +Local AI model timings by chip, operating system and enginemacOSAurora Linux — labels identify chip and engine; driver in tooltipGranite 4.0 1B MLX 4-bit (IBM, dense) · rev 29f108a4 · weights b7a43ebb · prompt af125b15 · 32-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · MLX in-process, tunedM1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.29; 3 runs0.29M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 52.2; 3 runs52.2M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.1; 3 runs4.1M1 Pro · Aurora wip4 · MLX in-process, defaultM1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.30; 3 runs0.30M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 45.0; 3 runs45.0M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 4.7; 3 runs4.7M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.39; 3 runs0.39M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 16.5; 3 runs16.5M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 12.5; 3 runs12.5M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.23; 3 runs0.23M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 53.8; 3 runs53.8M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.0; 3 runs4.0M1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.17; 3 runs0.17M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 109.2; 3 runs109.2M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 2.0; 3 runs2.0Granite 4.0 1B MLX 4-bit (IBM, dense) · rev 29f108a4 · weights b7a43ebb · prompt 2f23962c · 256-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.18; 3 runs0.18M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 106.6; 3 runs106.6M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 2.1; 3 runs2.1LFM3-1.7B MLX 8-bit · rev ee7667f3 · weights 2b06cea9 · prompt af125b15 · 32-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · MLX in-process, tuned · old driverM1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.39; 3 runs0.39M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 16.9; 3 runs16.9M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 12.2; 3 runs12.2M1 Pro · Aurora wip4 · MLX in-process, tunedM1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.30; 3 runs0.30M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 27.5; 3 runs27.5M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 7.6; 3 runs7.6M1 Pro · Aurora wip4 · MLX in-process, default · old driverM1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.40; 3 runs0.40M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 15.3; 3 runs15.3M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 13.5; 3 runs13.5M1 Pro · Aurora wip4 · MLX in-process, defaultM1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.33; 3 runs0.33M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 22.6; 3 runs22.6M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 9.2; 3 runs9.2M1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.14; 3 runs0.14M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 68.0; 3 runs68.0M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 3.1; 3 runs3.1LFM3-1.7B MLX 8-bit · rev ee7667f3 · weights 2b06cea9 · prompt 2f23962c · 256-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.18; 3 runs0.18M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 76.1; 3 runs76.1M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 2.8; 3 runs2.8Qwen3.5-2B MLX 4-bit · rev 93760be4 · weights 713fe7e5 · prompt af125b15 · 32-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · MLX in-process, tuned · old driverM1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.46; 3 runs0.46M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 47.2; 3 runs47.2M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.7; 3 runs4.7M1 Pro · Aurora wip4 · MLX in-process, tunedM1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.45; 3 runs0.45M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 56.4; 3 runs56.4M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.0; 3 runs4.0M1 Pro · Aurora wip5 · MLX in-process, tunedM1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.42; 3 runs0.42M1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 54.5; 3 runs54.5M1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.1; 3 runs4.1M1 Pro · Aurora wip4 · MLX in-process, default · old driverM1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.51; 3 runs0.51M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 36.0; 3 runs36.0M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 6.1; 3 runs6.1M1 Pro · Aurora wip4 · MLX in-process, defaultM1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.46; 3 runs0.46M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 48.9; 3 runs48.9M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 4.6; 3 runs4.6M1 Pro · Aurora wip4 · Rapid-MLX server · old driverM1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.19; 3 runs0.19M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 42.4; 3 runs42.4M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.9; 3 runs4.9M1 Pro · Aurora wip4 · Rapid-MLX serverM1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.19; 3 runs0.19M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 49.9; 3 runs49.9M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.2; 3 runs4.2M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.61; 3 runs0.61M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 33.7; 3 runs33.7M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 6.5; 3 runs6.5M1 Pro · Aurora wip4 · Upstream server + logprob patchM1 Pro · Aurora wip4 · Upstream server + logprob patch; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.59; 3 runs0.59M1 Pro · Aurora wip4 · Upstream server + logprob patch; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 42.3; 3 runs42.3M1 Pro · Aurora wip4 · Upstream server + logprob patch; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 5.3; 3 runs5.3M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.20; 3 runs0.20M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 57.2; 3 runs57.2M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 3.7; 3 runs3.7M1 Pro · Aurora wip4 · Upstream + singleton (trial)M1 Pro · Aurora wip4 · Upstream + singleton (trial); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.58; 3 runs0.58M1 Pro · Aurora wip4 · Upstream + singleton (trial); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 56.8; 3 runs56.8M1 Pro · Aurora wip4 · Upstream + singleton (trial); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.1; 3 runs4.1M1 Pro · Aurora wip4 · TensorFold Vulkan · old driverM1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; First token (s): lower: 2.00; 3 runs2.00M1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; Decode (tok/s): higher: 2.4; 3 runs2.4M1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; 200-token estimate (s): lower: 85.3; 3 runs85.3M1 Pro · Aurora wip5 · TensorFold VulkanM1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; First token (s): lower: 1.91; 3 runs1.91M1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; Decode (tok/s): higher: 2.9; 3 runs2.9M1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; 200-token estimate (s): lower: 71.2; 3 runs71.2M1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.25; 3 runs0.25M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 102.8; 3 runs102.8M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 2.2; 3 runs2.2Qwen3.5-2B MLX 4-bit · rev 93760be4 · weights 713fe7e5 · prompt af125b15 · 32-token limit · 4 requests (aggregate decode)First token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · Rapid-MLX serverM1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 1.15; 3 runs1.15M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 42.6; 3 runs42.6M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 19.9; 3 runs19.9M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 4.08; 3 runs4.08M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 20.2; 3 runs20.2M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 43.6; 3 runs43.6M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 0.77; 3 runs0.77M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 57.0; 3 runs57.0M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 14.8; 3 runs14.8Qwen3.5-2B MLX 4-bit · rev 93760be4 · weights 713fe7e5 · prompt af125b15 · 32-token limit · 8 requests (aggregate decode)First token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · Rapid-MLX serverM1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); First token (s): lower: 1.93; 3 runs1.93M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); Decode (tok/s): higher: 49.2; 3 runs49.2M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); 200-token estimate (s): lower: 34.5; 3 runs34.5M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); First token (s): lower: 8.08; 3 runs8.08M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); Decode (tok/s): higher: 20.3; 3 runs20.3M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); 200-token estimate (s): lower: 86.9; 3 runs86.9M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); First token (s): lower: 2.97; 3 runs2.97M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); Decode (tok/s): higher: 57.7; 3 runs57.7M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 8 requests at once (decode = total tok/s); 200-token estimate (s): lower: 30.7; 3 runs30.7Qwen3.5-2B MLX 4-bit · rev 93760be4 · weights 713fe7e5 · prompt 2f23962c · 256-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · MLX in-process, tuned · old driverM1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.56; 3 runs0.56M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 49.3; 3 runs49.3M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.6; 3 runs4.6M1 Pro · Aurora wip5 · MLX in-process, tunedM1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.50; 1 runs0.50M1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 56.2; 1 runs56.2M1 Pro · Aurora wip5 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 4.1; 1 runs4.1M1 Pro · Aurora wip4 · MLX in-process, default · old driverM1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.55; 3 runs0.55M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 37.2; 3 runs37.2M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 5.9; 3 runs5.9M1 Pro · Aurora wip4 · Rapid-MLX server · old driverM1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.40; 3 runs0.40M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 43.2; 3 runs43.2M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 5.0; 3 runs5.0M1 Pro · Aurora wip4 · TensorFold Vulkan · old driverM1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; First token (s): lower: 3.12; 3 runs3.12M1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; Decode (tok/s): higher: 2.8; 3 runs2.8M1 Pro · Aurora wip4 · TensorFold Vulkan · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; serve --backend vulkan --context 1024 --no-thinking; 200-token estimate (s): lower: 75.4; 3 runs75.4M1 Pro · Aurora wip5 · TensorFold VulkanM1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; First token (s): lower: 2.97; 1 runs2.97M1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; Decode (tok/s): higher: 2.8; 1 runs2.8M1 Pro · Aurora wip5 · TensorFold Vulkan; 7.2.11-elementary-aurora72-wip5; Honeykrisp Mesa 9323171c12; serve --backend vulkan --context 1024 --no-thinking; 200-token estimate (s): lower: 74.7; 1 runs74.7M1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.30; 3 runs0.30M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 101.4; 3 runs101.4M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 2.3; 3 runs2.3Qwen3.5-4B MLX 4-bit · rev 32f3e8ec · weights 5fb9acd0 · prompt af125b15 · 32-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · MLX in-process, tuned · old driverM1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.71; 3 runs0.71M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 24.8; 3 runs24.8M1 Pro · Aurora wip4 · MLX in-process, tuned · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 8.8; 3 runs8.8M1 Pro · Aurora wip4 · MLX in-process, tunedM1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.64; 3 runs0.64M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 27.0; 3 runs27.0M1 Pro · Aurora wip4 · MLX in-process, tuned; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 8.0; 3 runs8.0M1 Pro · Aurora wip4 · MLX in-process, default · old driverM1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.73; 3 runs0.73M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 18.6; 3 runs18.6M1 Pro · Aurora wip4 · MLX in-process, default · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 11.5; 3 runs11.5M1 Pro · Aurora wip4 · MLX in-process, defaultM1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); First token (s): lower: 0.65; 3 runs0.65M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); Decode (tok/s): higher: 23.1; 3 runs23.1M1 Pro · Aurora wip4 · MLX in-process, default; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); 200-token estimate (s): lower: 9.3; 3 runs9.3M1 Pro · Aurora wip4 · Rapid-MLX server · old driverM1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.24; 3 runs0.24M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 23.4; 3 runs23.4M1 Pro · Aurora wip4 · Rapid-MLX server · old driver; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 7faf04c065; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 8.8; 3 runs8.8M1 Pro · Aurora wip4 · Rapid-MLX serverM1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.18; 3 runs0.18M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 27.5; 3 runs27.5M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 7.4; 3 runs7.4M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.85; 3 runs0.85M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 21.0; 3 runs21.0M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 10.4; 3 runs10.4M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; First token (s): lower: 0.39; 3 runs0.39M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; Decode (tok/s): higher: 28.0; 3 runs28.0M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000; 200-token estimate (s): lower: 7.5; 3 runs7.5M1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.33; 3 runs0.33M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 53.6; 3 runs53.6M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 4.1; 3 runs4.1Qwen3.5-4B MLX 4-bit · rev 32f3e8ec · weights 5fb9acd0 · prompt af125b15 · 32-token limit · 4 requests (aggregate decode)First token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · Aurora wip4 · Rapid-MLX serverM1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 2.25; 3 runs2.25M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 21.2; 3 runs21.2M1 Pro · Aurora wip4 · Rapid-MLX server; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 40.0; 3 runs40.0M1 Pro · Aurora wip4 · Upstream server, stockM1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 6.27; 3 runs6.27M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 12.8; 3 runs12.8M1 Pro · Aurora wip4 · Upstream server, stock; 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 69.0; 3 runs69.0M1 Pro · Aurora wip4 · Upstream server, patched (ships)M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); First token (s): lower: 2.04; 3 runs2.04M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); Decode (tok/s): higher: 25.8; 3 runs25.8M1 Pro · Aurora wip4 · Upstream server, patched (ships); 7.2.11-elementary-aurora72-wip4; Honeykrisp Mesa 9323171c12; MLX_OMARCHY_BATCH_WORK=120000 · 4 requests at once (decode = total tok/s); 200-token estimate (s): lower: 33.1; 3 runs33.1Qwen3.5-4B MLX 4-bit · rev 32f3e8ec · weights 5fb9acd0 · prompt 2f23962c · 256-token limit · one requestFirst token (s): lowerDecode (tok/s): higher200-token estimate (s): lowerM1 Pro · macOS 27.2 · Apple MLX in-processM1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; First token (s): lower: 0.43; 3 runs0.43M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; Decode (tok/s): higher: 55.2; 3 runs55.2M1 Pro · macOS 27.2 · Apple MLX in-process; Darwin 27.2.0 RELEASE_ARM64_T6000; Apple Metal; MLX 0.31.2; mlx-lm 0.31.3; 200-token estimate (s): lower: 4.1; 3 runs4.1 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/README.md b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/README.md new file mode 100644 index 00000000..ba9c0e89 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/README.md @@ -0,0 +1,42 @@ +# macOS MLX measurements and TensorFold comparison scope — 2026-10-30 + +## Scope correction + +These Mac measurements ran Apple MLX, not the TensorFold engine. They cover Qwen3.5 4B, Qwen3 4B Instruct and Qwen3.5 9B; they do **not** test Qwen3.5 2B, Liquid LFM3, Granite, Linux Vulkan, or TensorFold. The 4B model overlaps the project engine scoreboard, but the Mac runs used random synthetic 512/4096-token prompts and 128-token outputs, not the scoreboard's same-text 32/256-token workloads. They cannot be inserted into the scoreboard as matched comparisons. No Linux runtime, kernel, driver or service was changed. + +## The intended project benchmarks + +The current [`engine-compare.py`](../../engine-compare.py) has four named local scoreboard model configurations: + +| Selector | Pinned model | Revision | Model weights SHA-256 | Current comparison arms | +|---|---|---|---|---| +| `qwen2b` | `mlx-community/Qwen3.5-2B-MLX-4bit` | `93760be4f1f69842a46bc13dbdc0f19e291392a3` | `713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5` | MLX, servers, Rapid, oMLX, **TensorFold** | +| `qwen4b` | `mlx-community/Qwen3.5-4B-MLX-4bit` | `32f3e8ecf65426fc3306969496342d504bfa13f3` | `5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db` | MLX, servers, Rapid, oMLX | +| `lfm3` | `LiquidAI/LFM3-1.7B-MLX-8bit` | `ee7667f3cfccb2a41725f91820531823fcabc6b7` | `2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad` | MLX 5k and 120k | +| `granite1b` | `mlx-community/granite-4.0-1b-4bit` (dense) | `29f108a4e2ff99079a173004871604c2f1e12c1f` | `b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7` | MLX, Rapid and selected servers | + +The dense Granite selection is intentional; the hybrid Granite 4.0 H 1B is excluded because the GPU multi-token gate found a severe correctness failure. LFM3 used a separately pinned mlx-lm model implementation; check compatibility and pin its implementation when establishing a Mac reference. The project also has separate exploratory Qwen3.8-2B and ANE work; it is not one of these four `engine-compare.py` configurations. See [scoreboard](../../ENGINE-SCOREBOARD.md) and [Qwen3.8 receipt](../2026-10-27-qwen38-mac-rapid.md). + +The benchmark harness uses short and long fixed rainbow prompts, greedy decoding, 32 or 256 generated tokens, fresh processes, one warmup and three rotated rounds. MLX in-process and server HTTP have different timing boundaries. The Linux experiments enforce the shared worker lock, memory reserve and temperature guard; use matched measurement boundaries when comparing macOS and Linux. + +TensorFold's installed opt-in Vulkan engine is pinned to PR 1544 head `b3ab9ba217a80c63fcc48f7643ddfe339a8f4bdc`, and its current compatible scoreboard arm is Qwen3.5 2B. The [October 30 engine receipt](../2026-10-30-tensorfold-b3ab9ba/README.md) records a 32-token TensorFold loopback median of 2.89 tok/s and installed MLX in-process 54.51 tok/s with different timing boundaries; it does not justify a direct speed ratio. Long free-running text diverged after 880 characters, and strict teacher-forced exact-ID checks failed on near-ties in some cases. Do not claim full fidelity or a submit-batching gain. The [package receipt](../2026-10-30-tensorfold-package/README.md) records the installed opt-in package. + +## What we actually measured on this Mac + +[Model and shape report](model-shape-comparison.md) records six instrumented one-trial runs and their limits. Earlier three-trial native Apple MLX runs are retained in `*-baseline.log`. Separate synthetic 512/4096 prompt runs have their raw stdout in `*-p512.log` and `*-p4096.log`; the interrupted first Qwen3 4096 trace-save log is retained as `qwen3-4b-p4096-interrupted.log`. All six completed trace targets exited 0. The Mac's model files matched the hashes listed in the shape report; no correctness/fidelity comparison to TensorFold was performed. The first 4B baseline wrapper printed all results but failed when recording its shell exit status; its target exit status was not retained. + +The original 4B System Trace contained 4,978 target-process active compute intervals over warmup and the measured trial; overlapping intervals are not a per-kernel dispatch count or elapsed GPU time. The Xcode Metal System Trace preflight also completed a synthetic 1024×1024 MLX matrix multiplication on the M1 Pro. Instruments did not collect shader timing or counters. Its shader pipeline lists are not dispatch counts or evidence that any named kernel dominates. The longer rolling captures include only the end of the workload. The separately captured warmed decode trace includes model buffers, and the system traces may contain private process metadata. The approximately 2.7 GiB trace bundles are retained locally on the Mac, not committed; this repository contains the synthetic logs and interpretation, not the original trace bundles or weights. `decode-capture.log` records the one-step capture result. These artifacts do not prove a Linux bottleneck or support a TensorFold speedup. + +A later read-only source check found a public but closed and unmerged [batching PR #1561](https://github.com/ashhart/TensorFold/pull/1561). Its one-commit patch does not apply cleanly to the installed revision; see [the applicability audit](tensorfold-batching-audit.md). No candidate was applied or tested. + +## Matched macOS scoreboard runs + +The four scoreboard models were subsequently run on the M1 Pro Mac for both 32- and 256-token limits with three fresh-process trials per workload and the project’s prompts, greedy sampling and same-prompt warmup. Eight synthetic receipts in the parent receipts directory record run timestamps, model SHA-1/SHA-256, token IDs, hardware, macOS build, Darwin kernel and MLX versions. See [the scoreboard](../../ENGINE-SCOREBOARD.md) and [cross-platform text-hash evidence](linux-text-comparison.json). All three runs within each Mac group returned identical text and token IDs. + +Short Qwen2B, dense Granite and LFM3 text matched available Linux MLX reference text. Long Qwen2B diverged after 633 characters; short Qwen4B diverged after 94. These are text comparisons on these prompts, not proof of token or numerical fidelity, a defect diagnosis, or a TensorFold comparison. LFM3 required the project-pinned injected model implementation; its tokenizer emitted a custom-code prompt and continued without approval, and Transformers reported a model-type warning. No fan/temperature telemetry or Linux-style continuous reserve guard was recorded. The Mac harness only checked disk and macOS memory pressure at admission. No Linux benchmark was run in this session. + +The original session harness copies are retained in `harness/` for audit. Their paths and consolidator hardware/version fields are hardcoded to this M1 session: **do not use the consolidator to label M4 or later runs**. A future M4 run needs machine-detected chip, OS, kernel, runtime and hashes; use the same prompt and sampling settings. Model files were removed after the runs. + +## Next comparison + +Use the four scoreboard model pins above, starting with Qwen3.5 2B for the TensorFold port. Compare the completed macOS MLX results against matching Linux in-process receipts, keeping timing boundaries, output differences and runtime versions visible. Capture the exact tokenizer/chat-template setup and token IDs, runtime versions and timing boundaries. Compare Apple MLX in-process to Linux MLX in-process first; compare HTTP servers only with equivalent HTTP servers. Then evaluate the TensorFold Vulkan 2B arm and preserve its separate numerical and text-fidelity gates. Repeat on M4 only with exact chip, OS and runtime recorded. A fresh Linux benchmark requires booting the M1 Pro into elementary OS; do not claim a fresh paired run from the historical results. diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/decode-capture.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/decode-capture.log new file mode 100644 index 00000000..c0a6adbf --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/decode-capture.log @@ -0,0 +1,2 @@ +prefill and three generation steps complete +capture complete diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/consolidate.py b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/consolidate.py new file mode 100644 index 00000000..c537f25c --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/consolidate.py @@ -0,0 +1,11 @@ +import json,hashlib,glob,os,sys,statistics +name=sys.argv[1]; label=sys.argv[2]; modelids={'qwen2b':'mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3','qwen4b':'mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3','lfm3':'LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7','granite1b':'mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f'} +root='outputs/mac-scoreboard'; paths=sorted(glob.glob(f'{root}/{name}-{label}-[123].json')); runs=[] +for i,path in enumerate(paths,1): + x=json.load(open(path));x['arm']='mac-mlx';x['round']=i;x['decode_tps']=x['decode_estimate_tps'];x['text_sha256']=hashlib.sha256(x['text'].encode()).hexdigest();x['token_ids_sha256']=hashlib.sha256(json.dumps(x['token_ids']).encode()).hexdigest();runs.append(x) +def hashfile(alg): + p=f'{root}/{name}-{alg}.txt';return open(p).read().split()[0] if os.path.exists(p) else None +prompt=('Explain in plain language how a rainbow forms. Give four distinct sentences.' if label=='short' else 'Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.') +out={'prompt':prompt,'prompt_sha256':hashlib.sha256(prompt.encode()).hexdigest(),'schema':'mac-scoreboard-v1','status':'complete' if len(runs)==3 else 'incomplete','model_key':name,'model':modelids[name],'weights_sha1':hashfile('sha1'),'weights_sha256':hashfile('sha256'),'os':'macOS 27.2 build 26B123','kernel':'Darwin 27.2.0 RELEASE_ARM64_T6000','hardware':'MacBookPro18,3 Apple M1 Pro 32 GB','driver':'Apple Metal','mlx_version':'0.31.2','mlx_lm_version':'0.31.3','max_tokens':32 if label=='short' else 256,'arms':['mac-mlx'],'concurrency':1,'method':'Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.','runs':runs} +if runs:out['summary']={'mac-mlx':{'ok_runs':len(runs),'median_ttft_s':statistics.median(x['ttft_s'] for x in runs),'median_total_s':statistics.median(x['total_s'] for x in runs),'median_decode_tps':statistics.median(x['decode_tps'] for x in runs)}} +path=f'{root}/{name}-{label}-receipt.json';open(path,'w').write(json.dumps(out,indent=2)+'\n');print(path) diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/driver.py b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/driver.py new file mode 100644 index 00000000..050cc8e9 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/driver.py @@ -0,0 +1,17 @@ +import os,sys,time,subprocess,json,shutil,re +name=sys.argv[1];folder=os.path.abspath('work/mac-scoreboard/'+name);target='outputs/mac-scoreboard';lfm=os.path.abspath('work/mac-scoreboard/lfm3.py') if name=='lfm3' else '-' +prompts=[('short','Explain in plain language how a rainbow forms. Give four distinct sentences.',32),('long','Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.',256)] +for label,prompt,n in prompts: + for i in range(1,4): + free=shutil.disk_usage('.').free + memory=subprocess.run(['memory_pressure'],capture_output=True,text=True).stdout + match=re.search(r'System-wide memory free percentage: (\d+)%',memory) + avail=int(match.group(1)) if match else None + print(f'{name} {label} {i}: disk {free/2**30:.1f} GiB memory free {avail}%',flush=True) + if free<6*2**30 or avail is None or avail < 25: + print('Stopped before admission: disk or memory headroom below guard',flush=True);sys.exit(3) + out=f'{target}/{name}-{label}-{i}.json' + env=dict(os.environ,HF_HUB_OFFLINE='1',OMP_NUM_THREADS='2',OPENBLAS_NUM_THREADS='2') + run=subprocess.run(['nice','-n','10',sys.executable,'work/mac-scoreboard/worker.py',folder,name,prompt,str(n),out,lfm],env=env,timeout=300) + if run.returncode:print('failed',run.returncode,flush=True);sys.exit(run.returncode) + time.sleep(15) diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/worker.py b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/worker.py new file mode 100644 index 00000000..5290959e --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/worker.py @@ -0,0 +1,21 @@ +import sys,os,json,time,hashlib,importlib.util +from pathlib import Path +model_dir,which,prompt,max_tokens,out,lfm_file=sys.argv[1:] +if lfm_file!='-': + p=Path(lfm_file);expected='a4096cddc8f0d3f7923008b2e6afb896dabc9aef963a5ebe7d76d13a6069b0c4' + if hashlib.sha256(p.read_bytes()).hexdigest()!=expected: raise RuntimeError('LFM3 implementation checksum mismatch') + spec=importlib.util.spec_from_file_location('mlx_lm.models.lfm3',str(p));mod=importlib.util.module_from_spec(spec);sys.modules[spec.name]=mod;spec.loader.exec_module(mod) +from mlx_lm import load,stream_generate +from mlx_lm.sample_utils import make_sampler +import mlx.core as mx +wall_start=time.strftime('%Y-%m-%d %H:%M:%S %Z%z'); started=time.monotonic(); model,tok=load(model_dir) +prompt=tok.apply_chat_template([{'role':'user','content':prompt}],tokenize=False,add_generation_prompt=True,enable_thinking=False) +ids=tok.encode(prompt); sampler=make_sampler(temp=0) +for _ in stream_generate(model,tok,prompt,max_tokens=int(max_tokens),sampler=sampler):pass +mx.synchronize(); ready=time.monotonic()-started;start=time.monotonic();first=None;tokens=[];text='' +for r in stream_generate(model,tok,prompt,max_tokens=int(max_tokens),sampler=sampler): + if first is None:first=time.monotonic()-start + tokens.append(r.token);text+=r.text +mx.synchronize();total=time.monotonic()-start +res={'model':which,'prompt_tokens':len(ids),'prompt_token_sha256':hashlib.sha256(json.dumps(ids).encode()).hexdigest(),'ready_s':ready,'ttft_s':first,'total_s':total,'completion_tokens':len(tokens),'decode_estimate_tps':(len(tokens)-1)/(total-first) if first is not None and len(tokens)>1 and total>first else None,'token_ids':tokens,'text':text,'start_time':wall_start,'end_time':time.strftime('%Y-%m-%d %H:%M:%S %Z%z'),'peak_memory_bytes':mx.get_peak_memory()} +Path(out).write_text(json.dumps(res,indent=2)+'\n');print(json.dumps({k:v for k,v in res.items() if k not in ('text','token_ids')}),flush=True) diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/linux-text-comparison.json b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/linux-text-comparison.json new file mode 100644 index 00000000..320c3d2e --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/linux-text-comparison.json @@ -0,0 +1,138 @@ +{ + "method": "Compare synthetic macOS text hashes to existing Linux mlx-5k completed receipts with same model selector, exact prompt SHA-256 and max_tokens. Different platform and runtime revisions; evidence is text only, not a numerical/fidelity gate.", + "rows": [ + { + "model": "qwen2b", + "max_tokens": 32, + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926", + "linux_baselines": [ + { + "receipt": "2026-10-28-engine-compare-qwen2b-short-hk9323171c12.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e" + ], + "mac_text_sha256": "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e", + "same_text_for_all": true, + "common_prefix_chars": 138 + }, + { + "receipt": "2026-10-28-engine-compare-short.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e" + ], + "mac_text_sha256": "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e", + "same_text_for_all": true, + "common_prefix_chars": 138 + } + ] + }, + { + "model": "qwen2b", + "max_tokens": 256, + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921", + "linux_baselines": [ + { + "receipt": "2026-10-28-engine-compare-long.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "3f3ac70c5d72eda0b31229d8da4d90ca6fce34ca69e6ebc24a3d6502fc816990" + ], + "mac_text_sha256": "e0c3922c7d3fd430bdec1f3c211d48da56a805e582f2d885b639ffc20878086e", + "same_text_for_all": false, + "common_prefix_chars": 633 + } + ] + }, + { + "model": "qwen4b", + "max_tokens": 32, + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926", + "linux_baselines": [ + { + "receipt": "2026-10-28-engine-compare-qwen4b-short-hk9323171c12.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "bdb76d67e763d4e8ae7176616718338bafdab9ecab8cea1f66e0a73e706abe84" + ], + "mac_text_sha256": "e74285cad2b17b0b4991f3aa68d669d7ae2442c37bf93e366dc901ec1bb2f476", + "same_text_for_all": false, + "common_prefix_chars": 94 + }, + { + "receipt": "2026-10-28-engine-compare-qwen4b-short.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "bdb76d67e763d4e8ae7176616718338bafdab9ecab8cea1f66e0a73e706abe84" + ], + "mac_text_sha256": "e74285cad2b17b0b4991f3aa68d669d7ae2442c37bf93e366dc901ec1bb2f476", + "same_text_for_all": false, + "common_prefix_chars": 94 + } + ] + }, + { + "model": "qwen4b", + "max_tokens": 256, + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921", + "linux_baselines": [] + }, + { + "model": "lfm3", + "max_tokens": 32, + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926", + "linux_baselines": [ + { + "receipt": "2026-10-28-engine-compare-lfm3-short-hk9323171c12.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b" + ], + "mac_text_sha256": "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b", + "same_text_for_all": true, + "common_prefix_chars": 150 + }, + { + "receipt": "2026-10-28-engine-compare-lfm3-short.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b" + ], + "mac_text_sha256": "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b", + "same_text_for_all": true, + "common_prefix_chars": 150 + } + ] + }, + { + "model": "lfm3", + "max_tokens": 256, + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921", + "linux_baselines": [] + }, + { + "model": "granite1b", + "max_tokens": 32, + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926", + "linux_baselines": [ + { + "receipt": "2026-10-29-engine-compare-granite1b-short-hk9323171c12-mlx-5k+mlx-120k+rapid-120k+serve-120k+serve-patched-120k.json", + "linux_runs": 3, + "linux_text_sha256s": [ + "81588d18f3846cfeedeae583a5c7771c04a5568c3bec5aece292542a8adc7741" + ], + "mac_text_sha256": "81588d18f3846cfeedeae583a5c7771c04a5568c3bec5aece292542a8adc7741", + "same_text_for_all": true, + "common_prefix_chars": 141 + } + ] + }, + { + "model": "granite1b", + "max_tokens": 256, + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921", + "linux_baselines": [] + } + ] +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/model-shape-comparison.md b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/model-shape-comparison.md new file mode 100644 index 00000000..5d8c875b --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/model-shape-comparison.md @@ -0,0 +1,36 @@ +# M1 Pro Metal profiles: model and prompt shapes + +**Scope correction:** These are exploratory Apple MLX profiles, not TensorFold Vulkan benchmark results. They do not cover the TensorFold 2B checkpoint or the full engine scoreboard. See [README](README.md) for the exact relevant model set and next comparison. + +Captured October 30, 2026 on the M1 Pro Mac, macOS 27.2, MLX 0.31.2, mlx-lm 0.31.3. Each row is one instrumented measured trial after a warmup, batch 1, 128 generated tokens, prefill step 256, offline and two OMP/OpenBLAS threads. These are profiling runs under variable background load, not three-trial controlled baselines. Tracing can affect timing. + +| Model | Prompt tokens | Prompt tok/s | Generation tok/s | Peak MLX GiB | Trace | +|---|---:|---:|---:|---:|---| +| Qwen3.5 4B MLX 4-bit | 512 | 275.781 | 53.441 | 3.088 | Complete System Trace, full run | +| Qwen3.5 4B MLX 4-bit | 4096 | 283.301 | 48.802 | 3.273 | Complete System Trace, last 3 seconds | +| Qwen3 4B Instruct 2507 4-bit | 512 | 283.930 | 52.096 | 2.856 | Complete System Trace, full run | +| Qwen3 4B Instruct 2507 4-bit | 4096 | 255.632 | 43.227 | 3.333 | Complete retry, last 3 seconds | +| Qwen3.5 9B MLX 4-bit | 512 | 151.570 | 30.633 | 5.680 | Complete System Trace, last 10 seconds | +| Qwen3.5 9B MLX 4-bit | 4096 | 153.487 | 30.322 | 5.826 | Complete System Trace, last 3 seconds | + +The Qwen3 4096 initial run produced 256.876 / 41.495 tok/s; its trace save was stopped when disk space fell below 5 GiB. That interrupted attempt's log (`qwen3-4b/p4096.log`) is retained and is not a completed trace; the table uses the separately completed bounded-window retry. A rolling window covers only the end of its run and cannot establish full-run kernel counts or prefill coverage. The Instruments table of contents recorded successful target exit for all six completed traces. Trace data may include system process metadata; keep it local or review it before sharing. + +## Model and shape clues + +The Qwen3 4B 512 trace shader list includes quantized QMV/QMM and `sdpa_vector_bfloat16_t_128_128...`, plus steel attention kernels. The Qwen3.5 4B 512 and Qwen3.5 9B lists include quantized QMV/QMM, a gated-delta step, and `sdpa_vector_bfloat16_t_256_256...`. The 9B 4096 trace's shader list also includes `sdpa_vector_2pass_1_bfloat16_t_256_256...` and `sdpa_vector_2pass_2_bfloat16_t_256`; those names were not present in the exported 9B 512 list. The rolling windows differ, so absence from a list does not prove a kernel never ran. Shader-list entries are pipeline metadata, not kernel time rankings, invocation counts, or proof of the source of the Linux gap. No GPU counters or shader timeline were enabled in these recordings. + +## Linux work to target + +- Capture the matching three models and 512/4096 shapes on bare-metal elementary OS with the existing memory, fan, temperature and quiet guards. Record actual kernel dispatch names, shapes, counts and GPU times. Use the same model revisions, prefill step and token budget; compare instrumented profiles separately from uninstrumented speed trials. +- Check whether Linux selects fast 4-bit group-64 quantized matrix-vector/matrix kernels for these shapes; distinguish single-token decode from prompt prefill. Measure before changing launch dimensions, fusion, memory access or dispatch behavior. +- For Qwen3.5, check the real gated-delta path and the attention route, including whether longer-context inputs trigger a two-pass equivalent. Compare Qwen3 attention shapes independently rather than assuming the 4B models execute alike. +- Establish route reachability before testing candidate row-reduce or GDN switches. Preserve existing fidelity and deterministic token gates, alternative-off controls, and at least three alternating uninstrumented baseline/candidate trials. The existing Honeykrisp candidate compiler regression remains a blocker to promotion. +- Preserve Linux/Mac hardware, runtime, ICD/driver and load metadata. A macOS profile cannot establish an elementary OS bottleneck or guarantee the same effect on M4. + +## Model provenance + +- Qwen3.5 4B revision `32f3e8ecf65426fc3306969496342d504bfa13f3`; model SHA-256 `5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db` matched the project catalog. +- Qwen3 4B revision `50d427756c6b1b2fe0c0a10f67fbda1fc8e82c1b`; SHA-256 `2a73c6c248601ab904e035548abd8e6abb65ea27dcb5f342fb0a8910eb44173f` matched the previous benchmark download. +- Qwen3.5 9B revision `938d8919941c6e7efd3c7150eff7fe9d12afa631`; shard hashes `a68b87558c6ef43f74c2bd63ce7e9092ceddc3101f3def0030774bae5f42aadd` and `b0a770bf8469c7f3f18756a0e0283f1c1174344a83e059a4e483f6af4907352d` matched the earlier benchmark. + +Sanitized raw logs are committed alongside this report. The large trace bundles and warmed single-step `.gputrace` remain local only; they contain system process metadata or model buffers and are not committed. Temporary downloaded model directories were removed after profiling. diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-baseline.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-baseline.log new file mode 100644 index 00000000..9bad849a --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-baseline.log @@ -0,0 +1,6 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=273.549, generation_tps=53.688, peak_memory=2.856, total_time=4.367 +Trial 2: prompt_tps=261.933, generation_tps=54.210, peak_memory=2.856, total_time=4.516 +Trial 3: prompt_tps=273.002, generation_tps=54.311, peak_memory=2.856, total_time=4.330 +Averages: prompt_tps=269.494, generation_tps=54.069, peak_memory=2.856 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096-interrupted.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096-interrupted.log new file mode 100644 index 00000000..66b237db --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096-interrupted.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=4096, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=256.876, generation_tps=41.495, peak_memory=3.333, total_time=19.148 +Averages: prompt_tps=256.876, generation_tps=41.495, peak_memory=3.333 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096.log new file mode 100644 index 00000000..51ead112 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p4096.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=4096, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=255.632, generation_tps=43.227, peak_memory=3.333, total_time=19.097 +Averages: prompt_tps=255.632, generation_tps=43.227, peak_memory=3.333 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p512.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p512.log new file mode 100644 index 00000000..3fb2f209 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen3-4b-p512.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=283.930, generation_tps=52.096, peak_memory=2.856, total_time=4.377 +Averages: prompt_tps=283.930, generation_tps=52.096, peak_memory=2.856 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-baseline.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-baseline.log new file mode 100644 index 00000000..8c61e2ef --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-baseline.log @@ -0,0 +1,6 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=266.333, generation_tps=54.402, peak_memory=3.088, total_time=4.476 +Trial 2: prompt_tps=268.575, generation_tps=53.177, peak_memory=3.089, total_time=4.528 +Trial 3: prompt_tps=265.946, generation_tps=53.715, peak_memory=3.089, total_time=4.493 +Averages: prompt_tps=266.951, generation_tps=53.765, peak_memory=3.089 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p4096.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p4096.log new file mode 100644 index 00000000..4e20832d --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p4096.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=4096, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=283.301, generation_tps=48.802, peak_memory=3.273, total_time=17.279 +Averages: prompt_tps=283.301, generation_tps=48.802, peak_memory=3.273 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p512.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p512.log new file mode 100644 index 00000000..1fab8aa7 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-4b-p512.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=275.781, generation_tps=53.441, peak_memory=3.088, total_time=4.449 +Averages: prompt_tps=275.781, generation_tps=53.441, peak_memory=3.088 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-baseline.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-baseline.log new file mode 100644 index 00000000..5de792ea --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-baseline.log @@ -0,0 +1,6 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=152.934, generation_tps=33.356, peak_memory=5.680, total_time=7.373 +Trial 2: prompt_tps=152.249, generation_tps=33.375, peak_memory=5.680, total_time=7.414 +Trial 3: prompt_tps=150.203, generation_tps=33.247, peak_memory=5.681, total_time=7.442 +Averages: prompt_tps=151.795, generation_tps=33.326, peak_memory=5.680 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p4096.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p4096.log new file mode 100644 index 00000000..bd59365d --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p4096.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=4096, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=153.487, generation_tps=30.322, peak_memory=5.826, total_time=31.107 +Averages: prompt_tps=153.487, generation_tps=30.322, peak_memory=5.826 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p512.log b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p512.log new file mode 100644 index 00000000..f01f32e3 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/qwen35-9b-p512.log @@ -0,0 +1,4 @@ +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=151.570, generation_tps=30.633, peak_memory=5.680, total_time=7.781 +Averages: prompt_tps=151.570, generation_tps=30.633, peak_memory=5.680 diff --git a/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/tensorfold-batching-audit.md b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/tensorfold-batching-audit.md new file mode 100644 index 00000000..adb2793b --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-metal-profile/tensorfold-batching-audit.md @@ -0,0 +1,17 @@ +# TensorFold Vulkan batching source check — 2026-10-30 + +## Discovery + +[TensorFold PR #1561](https://github.com/ashhart/TensorFold/pull/1561) exposes the previously described batching work in a public fork. As checked on October 30, the PR is **closed, unmerged and draft**; its author says it was opened early and the batching work is intended to follow the stacked engine PRs. It is not a merged release or a tested downstream update. The older references describing it as unpublished reflect the prior observation, not its current visibility. + +The public PR head was `a86eff2561ce1970d4edfdf67ff970e2785dcb88`; its immediate parent was `3992d9a` (a probe timer), whose parent was engine head `f280f25`. The batching commit changes five source files: engine/backend, command handling, dispatch and greedy. Upstream describes replacing approximately 496 submissions and fence waits per decode token with one command buffer and one fence wait, retaining per-dispatch descriptor sets and distinct staging slices. Treat the PR's performance and byte-identity claims as upstream-reported on its tested M1, M1 Max and M2 Max, not an M1 Pro result. Its reported full host matrix included 37 failures in an existing host snapshot test group, also reproduced on its own base; three Vulkan host groups passed. The server was not tested over that change. + +## Compatibility with the installed downstream pin + +The installed opt-in package is built from `b3ab9ba217a80c63fcc48f7643ddfe339a8f4bdc`. GitHub's compare of that revision and the public batching head reports divergent history (35 ahead, 165 behind, merge base `dd0a9cc5`); ahead/behind are graph counts, not patch equivalence. A direct tree comparison includes many non-batching differences, so switching to the old head would replace more than a batching change. + +A local **read-only** applicability check exported the exact one-commit batching diff `3992d9a..a86eff2` and ran `git apply --check` against a clean archive of installed source `b3ab9ba`. The patch did not apply cleanly: `engine.zig` and `greedy.zig` failed to match; individual checks for `backend.zig`, `commands.zig` and `dispatch.zig` applied. The unbatched source shows substantial documentation/comment reflow between these histories; textual rejection alone does not prove a semantic conflict. No patch was applied, compiled, installed or run, and no Git branch, GPU, kernel, service or default was changed. + +## Next qualification gate + +Review the exact upstream batching commit against `b3ab9ba`, including comment/context drift and any functional differences, then prepare an isolated candidate while retaining the pinned installed build. Validate descriptor lifetimes, staging slices, barriers, reset/poison and failure handling with host tests and the project's GPU stage/contract gates. Run matched Qwen3.5 2B old/new tests with the same prompt and measurement boundary, alternating at least three pairs, and retain token IDs, teacher-forced logits, submission counts, timing, memory/thermal observations and desktop responsiveness. Compare with the existing exact-ID failures and near-tie findings; do not substitute upstream M1/M1 Max numbers for M1 Pro results. No promotion before those gates pass. diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-long.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-long.json new file mode 100644 index 00000000..bc56dbd2 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-long.json @@ -0,0 +1,863 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "granite1b", + "model": "mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f", + "weights_sha1": "362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb", + "weights_sha256": "b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 256, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "granite1b", + "prompt_tokens": 57, + "prompt_token_sha256": "1809926bfb9c3bfef677d35cd52a5225fbdaa6033b483452173eedc21421bcae", + "ready_s": 3.049706417001289, + "ttft_s": 0.18441408400030923, + "total_s": 2.535268375002488, + "completion_tokens": 256, + "decode_estimate_tps": 108.47120596797706, + "token_ids": [ + 32, + 48713, + 374, + 264, + 43395, + 20441, + 5933, + 25885, + 430, + 13980, + 994, + 40120, + 84261, + 449, + 3090, + 7118, + 90592, + 304, + 279, + 16975, + 13, + 578, + 18488, + 315, + 264, + 48713, + 18065, + 3892, + 1401, + 11618, + 25, + 2098, + 16597, + 11, + 22599, + 11, + 86712, + 11, + 323, + 20705, + 9392, + 13, + 9062, + 315, + 1521, + 11618, + 11335, + 264, + 16996, + 3560, + 304, + 6968, + 279, + 34076, + 1358, + 315, + 8146, + 584, + 23846, + 304, + 264, + 48713, + 382, + 4032, + 16597, + 374, + 279, + 1176, + 3094, + 304, + 279, + 18488, + 315, + 264, + 48713, + 13, + 3277, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 11, + 433, + 374, + 19914, + 22955, + 11, + 477, + 30280, + 11, + 4245, + 311, + 279, + 2349, + 304, + 279, + 17915, + 315, + 3805, + 323, + 3090, + 13, + 1115, + 58218, + 315, + 3177, + 374, + 9057, + 555, + 279, + 6811, + 304, + 279, + 19914, + 3104, + 1963, + 1990, + 3805, + 323, + 3090, + 13, + 1666, + 279, + 3177, + 29933, + 279, + 7118, + 11053, + 11, + 433, + 70270, + 1523, + 323, + 93897, + 4245, + 311, + 279, + 2349, + 304, + 11298, + 13, + 578, + 19914, + 3104, + 1963, + 315, + 3090, + 374, + 5190, + 1109, + 430, + 315, + 3805, + 11, + 14718, + 279, + 3177, + 311, + 37920, + 7119, + 279, + 4725, + 320, + 1820, + 1584, + 77933, + 311, + 279, + 7479, + 570, + 1115, + 58218, + 315, + 3177, + 374, + 3967, + 439, + 2098, + 16597, + 382, + 791, + 19914, + 22955, + 3177, + 1243, + 35292, + 1555, + 279, + 3090, + 7118, + 11053, + 11, + 323, + 439, + 433, + 43186, + 11, + 433, + 37771, + 288, + 2500, + 2098, + 16597, + 13, + 1115, + 2132, + 2098, + 16597, + 11384, + 279, + 3177, + 311, + 834, + 82344, + 1139, + 1202, + 3777, + 8146, + 13, + 578, + 2204, + 8146, + 315, + 3177, + 617, + 2204, + 93959, + 11, + 323, + 439, + 814, + 4974, + 279, + 7118, + 11053, + 11, + 814, + 527, + 19914, + 22955, + 1578, + 11, + 14718, + 1124, + 311, + 8821, + 4726, + 13, + 1115, + 1920, + 374, + 3967, + 439, + 86712 + ], + "text": "A rainbow is a visually stunning natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere. The formation of a rainbow involves several key processes: refraction, reflection, dispersion, and viewing angle. Each of these processes plays a crucial role in creating the vibrant array of colors we observe in a rainbow.\n\nRefraction is the first step in the formation of a rainbow. When sunlight enters a water droplet, it is refracted, or bent, due to the change in the density of air and water. This bending of light is caused by the difference in the refractive index between air and water. As the light enters the droplet, it slows down and bends due to the change in medium. The refractive index of water is higher than that of air, causing the light to bend towards the normal (the line perpendicular to the surface). This bending of light is known as refraction.\n\nThe refracted light then travels through the water droplet, and as it exits, it undergoes another refraction. This second refraction causes the light to disperse into its component colors. The different colors of light have different wavelengths, and as they exit the droplet, they are refracted again, causing them to separate further. This process is known as dispersion", + "start_time": "2026-10-30 20:16:04 EDT-0400", + "end_time": "2026-10-30 20:16:10 EDT-0400", + "peak_memory_bytes": 1123734084, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 108.47120596797706, + "text_sha256": "6f891669d3a0ab4818ea7b2c7dd3cb7e9784162482fb662dd2bd77d243cfdf28", + "token_ids_sha256": "cfea8492702f1b83fcf00ac1cdeac674d4e9f497144b5728f33b989639f60a3a" + }, + { + "model": "granite1b", + "prompt_tokens": 57, + "prompt_token_sha256": "1809926bfb9c3bfef677d35cd52a5225fbdaa6033b483452173eedc21421bcae", + "ready_s": 3.048402749998786, + "ttft_s": 0.1708140420014388, + "total_s": 2.643278832998476, + "completion_tokens": 256, + "decode_estimate_tps": 103.13594795304229, + "token_ids": [ + 32, + 48713, + 374, + 264, + 43395, + 20441, + 5933, + 25885, + 430, + 13980, + 994, + 40120, + 84261, + 449, + 3090, + 7118, + 90592, + 304, + 279, + 16975, + 13, + 578, + 18488, + 315, + 264, + 48713, + 18065, + 3892, + 1401, + 11618, + 25, + 2098, + 16597, + 11, + 22599, + 11, + 86712, + 11, + 323, + 20705, + 9392, + 13, + 9062, + 315, + 1521, + 11618, + 11335, + 264, + 16996, + 3560, + 304, + 6968, + 279, + 34076, + 1358, + 315, + 8146, + 584, + 23846, + 304, + 264, + 48713, + 382, + 4032, + 16597, + 374, + 279, + 1176, + 3094, + 304, + 279, + 18488, + 315, + 264, + 48713, + 13, + 3277, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 11, + 433, + 374, + 19914, + 22955, + 11, + 477, + 30280, + 11, + 4245, + 311, + 279, + 2349, + 304, + 279, + 17915, + 315, + 3805, + 323, + 3090, + 13, + 1115, + 58218, + 315, + 3177, + 374, + 9057, + 555, + 279, + 6811, + 304, + 279, + 19914, + 3104, + 1963, + 1990, + 3805, + 323, + 3090, + 13, + 1666, + 279, + 3177, + 29933, + 279, + 7118, + 11053, + 11, + 433, + 70270, + 1523, + 323, + 93897, + 4245, + 311, + 279, + 2349, + 304, + 11298, + 13, + 578, + 19914, + 3104, + 1963, + 315, + 3090, + 374, + 5190, + 1109, + 430, + 315, + 3805, + 11, + 14718, + 279, + 3177, + 311, + 37920, + 7119, + 279, + 4725, + 320, + 1820, + 1584, + 77933, + 311, + 279, + 7479, + 570, + 1115, + 58218, + 315, + 3177, + 374, + 3967, + 439, + 2098, + 16597, + 382, + 791, + 19914, + 22955, + 3177, + 1243, + 35292, + 1555, + 279, + 3090, + 7118, + 11053, + 11, + 323, + 439, + 433, + 43186, + 11, + 433, + 37771, + 288, + 2500, + 2098, + 16597, + 13, + 1115, + 2132, + 2098, + 16597, + 11384, + 279, + 3177, + 311, + 834, + 82344, + 1139, + 1202, + 3777, + 8146, + 13, + 578, + 2204, + 8146, + 315, + 3177, + 617, + 2204, + 93959, + 11, + 323, + 439, + 814, + 4974, + 279, + 7118, + 11053, + 11, + 814, + 527, + 19914, + 22955, + 1578, + 11, + 14718, + 1124, + 311, + 8821, + 4726, + 13, + 1115, + 1920, + 374, + 3967, + 439, + 86712 + ], + "text": "A rainbow is a visually stunning natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere. The formation of a rainbow involves several key processes: refraction, reflection, dispersion, and viewing angle. Each of these processes plays a crucial role in creating the vibrant array of colors we observe in a rainbow.\n\nRefraction is the first step in the formation of a rainbow. When sunlight enters a water droplet, it is refracted, or bent, due to the change in the density of air and water. This bending of light is caused by the difference in the refractive index between air and water. As the light enters the droplet, it slows down and bends due to the change in medium. The refractive index of water is higher than that of air, causing the light to bend towards the normal (the line perpendicular to the surface). This bending of light is known as refraction.\n\nThe refracted light then travels through the water droplet, and as it exits, it undergoes another refraction. This second refraction causes the light to disperse into its component colors. The different colors of light have different wavelengths, and as they exit the droplet, they are refracted again, causing them to separate further. This process is known as dispersion", + "start_time": "2026-10-30 20:16:27 EDT-0400", + "end_time": "2026-10-30 20:16:32 EDT-0400", + "peak_memory_bytes": 1123750468, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 103.13594795304229, + "text_sha256": "6f891669d3a0ab4818ea7b2c7dd3cb7e9784162482fb662dd2bd77d243cfdf28", + "token_ids_sha256": "cfea8492702f1b83fcf00ac1cdeac674d4e9f497144b5728f33b989639f60a3a" + }, + { + "model": "granite1b", + "prompt_tokens": 57, + "prompt_token_sha256": "1809926bfb9c3bfef677d35cd52a5225fbdaa6033b483452173eedc21421bcae", + "ready_s": 3.0807002500005183, + "ttft_s": 0.18291162500099745, + "total_s": 2.5750470839993795, + "completion_tokens": 256, + "decode_estimate_tps": 106.59931444968078, + "token_ids": [ + 32, + 48713, + 374, + 264, + 43395, + 20441, + 5933, + 25885, + 430, + 13980, + 994, + 40120, + 84261, + 449, + 3090, + 7118, + 90592, + 304, + 279, + 16975, + 13, + 578, + 18488, + 315, + 264, + 48713, + 18065, + 3892, + 1401, + 11618, + 25, + 2098, + 16597, + 11, + 22599, + 11, + 86712, + 11, + 323, + 20705, + 9392, + 13, + 9062, + 315, + 1521, + 11618, + 11335, + 264, + 16996, + 3560, + 304, + 6968, + 279, + 34076, + 1358, + 315, + 8146, + 584, + 23846, + 304, + 264, + 48713, + 382, + 4032, + 16597, + 374, + 279, + 1176, + 3094, + 304, + 279, + 18488, + 315, + 264, + 48713, + 13, + 3277, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 11, + 433, + 374, + 19914, + 22955, + 11, + 477, + 30280, + 11, + 4245, + 311, + 279, + 2349, + 304, + 279, + 17915, + 315, + 3805, + 323, + 3090, + 13, + 1115, + 58218, + 315, + 3177, + 374, + 9057, + 555, + 279, + 6811, + 304, + 279, + 19914, + 3104, + 1963, + 1990, + 3805, + 323, + 3090, + 13, + 1666, + 279, + 3177, + 29933, + 279, + 7118, + 11053, + 11, + 433, + 70270, + 1523, + 323, + 93897, + 4245, + 311, + 279, + 2349, + 304, + 11298, + 13, + 578, + 19914, + 3104, + 1963, + 315, + 3090, + 374, + 5190, + 1109, + 430, + 315, + 3805, + 11, + 14718, + 279, + 3177, + 311, + 37920, + 7119, + 279, + 4725, + 320, + 1820, + 1584, + 77933, + 311, + 279, + 7479, + 570, + 1115, + 58218, + 315, + 3177, + 374, + 3967, + 439, + 2098, + 16597, + 382, + 791, + 19914, + 22955, + 3177, + 1243, + 35292, + 1555, + 279, + 3090, + 7118, + 11053, + 11, + 323, + 439, + 433, + 43186, + 11, + 433, + 37771, + 288, + 2500, + 2098, + 16597, + 13, + 1115, + 2132, + 2098, + 16597, + 11384, + 279, + 3177, + 311, + 834, + 82344, + 1139, + 1202, + 3777, + 8146, + 13, + 578, + 2204, + 8146, + 315, + 3177, + 617, + 2204, + 93959, + 11, + 323, + 439, + 814, + 4974, + 279, + 7118, + 11053, + 11, + 814, + 527, + 19914, + 22955, + 1578, + 11, + 14718, + 1124, + 311, + 8821, + 4726, + 13, + 1115, + 1920, + 374, + 3967, + 439, + 86712 + ], + "text": "A rainbow is a visually stunning natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere. The formation of a rainbow involves several key processes: refraction, reflection, dispersion, and viewing angle. Each of these processes plays a crucial role in creating the vibrant array of colors we observe in a rainbow.\n\nRefraction is the first step in the formation of a rainbow. When sunlight enters a water droplet, it is refracted, or bent, due to the change in the density of air and water. This bending of light is caused by the difference in the refractive index between air and water. As the light enters the droplet, it slows down and bends due to the change in medium. The refractive index of water is higher than that of air, causing the light to bend towards the normal (the line perpendicular to the surface). This bending of light is known as refraction.\n\nThe refracted light then travels through the water droplet, and as it exits, it undergoes another refraction. This second refraction causes the light to disperse into its component colors. The different colors of light have different wavelengths, and as they exit the droplet, they are refracted again, causing them to separate further. This process is known as dispersion", + "start_time": "2026-10-30 20:16:49 EDT-0400", + "end_time": "2026-10-30 20:16:54 EDT-0400", + "peak_memory_bytes": 1123750468, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 106.59931444968078, + "text_sha256": "6f891669d3a0ab4818ea7b2c7dd3cb7e9784162482fb662dd2bd77d243cfdf28", + "token_ids_sha256": "cfea8492702f1b83fcf00ac1cdeac674d4e9f497144b5728f33b989639f60a3a" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.18291162500099745, + "median_total_s": 2.5750470839993795, + "median_decode_tps": 106.59931444968078 + } + }, + "prompt": "Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.", + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-short.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-short.json new file mode 100644 index 00000000..6e1f593f --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-granite1b-short.json @@ -0,0 +1,191 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "granite1b", + "model": "mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f", + "weights_sha1": "362c6a1c5e1e54e39c91d7dc600b0b0944ed8ceb", + "weights_sha256": "b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 32, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "granite1b", + "prompt_tokens": 45, + "prompt_token_sha256": "31db491109686ce8eb19ecaa4022f013f8b9312a6c183f45ac10400b51409f08", + "ready_s": 1.0261594580006204, + "ttft_s": 0.16696862500248244, + "total_s": 0.45092750000185333, + "completion_tokens": 32, + "decode_estimate_tps": 109.17073819252411, + "token_ids": [ + 32, + 48713, + 7739, + 994, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 304, + 279, + 3805, + 11, + 14718, + 433, + 311, + 19914, + 533, + 323, + 8881, + 4871, + 279, + 7118, + 11053, + 13, + 1666, + 279, + 3177, + 19914, + 11613, + 11 + ], + "text": "A rainbow forms when sunlight enters a water droplet in the air, causing it to refract and reflect inside the droplet. As the light refracts,", + "start_time": "2026-10-30 20:15:11 EDT-0400", + "end_time": "2026-10-30 20:15:12 EDT-0400", + "peak_memory_bytes": 1083838996, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 109.17073819252411, + "text_sha256": "81588d18f3846cfeedeae583a5c7771c04a5568c3bec5aece292542a8adc7741", + "token_ids_sha256": "2fca36c156ff9d40da6a926296ac5e6bca3f4e4b7c51b1360165939312a51dd8" + }, + { + "model": "granite1b", + "prompt_tokens": 45, + "prompt_token_sha256": "31db491109686ce8eb19ecaa4022f013f8b9312a6c183f45ac10400b51409f08", + "ready_s": 0.9465352090010128, + "ttft_s": 0.1607938750021276, + "total_s": 0.4402760830016632, + "completion_tokens": 32, + "decode_estimate_tps": 110.91940421499572, + "token_ids": [ + 32, + 48713, + 7739, + 994, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 304, + 279, + 3805, + 11, + 14718, + 433, + 311, + 19914, + 533, + 323, + 8881, + 4871, + 279, + 7118, + 11053, + 13, + 1666, + 279, + 3177, + 19914, + 11613, + 11 + ], + "text": "A rainbow forms when sunlight enters a water droplet in the air, causing it to refract and reflect inside the droplet. As the light refracts,", + "start_time": "2026-10-30 20:15:29 EDT-0400", + "end_time": "2026-10-30 20:15:30 EDT-0400", + "peak_memory_bytes": 1083838996, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 110.91940421499572, + "text_sha256": "81588d18f3846cfeedeae583a5c7771c04a5568c3bec5aece292542a8adc7741", + "token_ids_sha256": "2fca36c156ff9d40da6a926296ac5e6bca3f4e4b7c51b1360165939312a51dd8" + }, + { + "model": "granite1b", + "prompt_tokens": 45, + "prompt_token_sha256": "31db491109686ce8eb19ecaa4022f013f8b9312a6c183f45ac10400b51409f08", + "ready_s": 1.1828392089992121, + "ttft_s": 0.17547537499922328, + "total_s": 0.4614042499997595, + "completion_tokens": 32, + "decode_estimate_tps": 108.4185708769073, + "token_ids": [ + 32, + 48713, + 7739, + 994, + 40120, + 29933, + 264, + 3090, + 7118, + 11053, + 304, + 279, + 3805, + 11, + 14718, + 433, + 311, + 19914, + 533, + 323, + 8881, + 4871, + 279, + 7118, + 11053, + 13, + 1666, + 279, + 3177, + 19914, + 11613, + 11 + ], + "text": "A rainbow forms when sunlight enters a water droplet in the air, causing it to refract and reflect inside the droplet. As the light refracts,", + "start_time": "2026-10-30 20:15:47 EDT-0400", + "end_time": "2026-10-30 20:15:48 EDT-0400", + "peak_memory_bytes": 1083838996, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 108.4185708769073, + "text_sha256": "81588d18f3846cfeedeae583a5c7771c04a5568c3bec5aece292542a8adc7741", + "token_ids_sha256": "2fca36c156ff9d40da6a926296ac5e6bca3f4e4b7c51b1360165939312a51dd8" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.16696862500248244, + "median_total_s": 0.45092750000185333, + "median_decode_tps": 109.17073819252411 + } + }, + "prompt": "Explain in plain language how a rainbow forms. Give four distinct sentences.", + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-long.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-long.json new file mode 100644 index 00000000..e8e31b10 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-long.json @@ -0,0 +1,868 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "lfm3", + "model": "LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7", + "weights_sha1": "3b46079dd265f521a232051c54c6218398c5e4b6", + "weights_sha256": "2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 256, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "lfm3", + "prompt_tokens": 37, + "prompt_token_sha256": "9dbb476f910146c702306094caf4089d17800e70462b147a034ecd5ba0f54e0f", + "ready_s": 5.08111966600336, + "ttft_s": 0.1849441670019587, + "total_s": 3.5337495420026244, + "completion_tokens": 256, + "decode_estimate_tps": 76.14655718830758, + "token_ids": [ + 962, + 3522, + 267, + 56445, + 59718, + 748, + 334, + 116632, + 90978, + 64521, + 41, + 52514, + 355, + 267, + 33835, + 20, + 18709, + 16896, + 302, + 22831, + 2292, + 395, + 7204, + 296, + 278, + 11622, + 1219, + 7624, + 22, + 810, + 2485, + 487, + 278, + 8465, + 3366, + 302, + 7575, + 1390, + 10977, + 88056, + 7575, + 41625, + 20219, + 88056, + 309, + 7575, + 64302, + 6514, + 962, + 302, + 18961, + 2137, + 1382, + 48875, + 22, + 2488, + 2345, + 278, + 6900, + 20, + 522, + 1946, + 1360, + 278, + 3008, + 302, + 1394, + 35664, + 405, + 435, + 26205, + 20, + 293, + 1663, + 664, + 20, + 309, + 94443, + 267, + 40252, + 964, + 549, + 1526, + 20, + 309, + 1256, + 13556, + 1030, + 278, + 7687, + 22118, + 309, + 10776, + 437, + 5522, + 467, + 278, + 41880, + 63908, + 64515, + 14066, + 229, + 25, + 22, + 7072, + 4601, + 309, + 278, + 28920, + 302, + 278, + 7072, + 8, + 35243, + 4601, + 355, + 267, + 13703, + 302, + 1196, + 42626, + 342, + 97322, + 17, + 302, + 9566, + 2292, + 22, + 2644, + 278, + 4478, + 355, + 2419, + 383, + 278, + 27150, + 20, + 851, + 23592, + 16519, + 278, + 48875, + 302, + 1382, + 296, + 278, + 10513, + 22, + 440, + 11630, + 462, + 643, + 278, + 4478, + 355, + 3036, + 278, + 27150, + 24694, + 278, + 3572, + 302, + 278, + 52514, + 383, + 278, + 3578, + 63908, + 64515, + 14066, + 229, + 26, + 22, + 12119, + 10977, + 342, + 42, + 3573, + 17, + 462, + 278, + 5206, + 42438, + 8, + 3952, + 267, + 2292, + 23935, + 26205, + 267, + 1382, + 4494, + 10530, + 20, + 435, + 58741, + 1707, + 309, + 293, + 2285, + 6568, + 278, + 3880, + 342, + 1319, + 2216, + 43801, + 310, + 278, + 3755, + 690, + 969, + 45204, + 355, + 2087, + 7575, + 1390, + 10977, + 88050, + 7656, + 278, + 115788, + 5935, + 302, + 1382, + 355, + 3084, + 898, + 395, + 302, + 2576, + 20, + 278, + 23935, + 3306, + 6084, + 22, + 440, + 3184, + 302, + 45204, + 10113, + 383, + 278, + 32358, + 302, + 278, + 2292, + 34, + 17913, + 42626, + 342, + 24475, + 20, + 56374, + 17 + ], + "text": "**How a Rainbow Forms \u2013 A Detailed Explanation**\n\nA rainbow is a spectacular, circular arc of colored light that appears in the sky after rain. It results from the combined effects of **refraction**, **internal reflection**, and **dispersion** of sunlight within water droplets. To understand the formation, we must follow the path of each photon as it enters, bounces, and exits a spherical raindrop, and then examine how the resulting angles and colors are observed by the viewer.\n\n---\n\n### 1. Sunlight and the Role of the Sun\n\nSunlight is a mixture of many wavelengths (colors) of visible light. When the sun is low on the horizon, its rays strike the droplets of water in the atmosphere. The angle at which the sun is above the horizon determines the position of the rainbow on the ground.\n\n---\n\n### 2. Refraction (Bending) at the First Surface\n\nWhen a light ray enters a water droplet, it slows down and bends toward the normal (the line perpendicular to the surface). This bending is called **refraction**. Because the refractive index of water is higher than that of air, the ray changes direction. The amount of bending depends on the wavelength of the light: shorter wavelengths (blue, violet)", + "start_time": "2026-10-30 20:19:36 EDT-0400", + "end_time": "2026-10-30 20:19:45 EDT-0400", + "peak_memory_bytes": 1945685414, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 76.14655718830758, + "text_sha256": "4c7cacd96bba54875c2b03652347284fd426697f6ea38a666e5d153e41e92a08", + "token_ids_sha256": "5e80e90f42530063ae3e40cc0b561be50136acc246f786c489c973e043a3fb23" + }, + { + "model": "lfm3", + "prompt_tokens": 37, + "prompt_token_sha256": "9dbb476f910146c702306094caf4089d17800e70462b147a034ecd5ba0f54e0f", + "ready_s": 5.0342072090024885, + "ttft_s": 0.1694671670011303, + "total_s": 3.528595209001651, + "completion_tokens": 256, + "decode_estimate_tps": 75.9125573100022, + "token_ids": [ + 962, + 3522, + 267, + 56445, + 59718, + 748, + 334, + 116632, + 90978, + 64521, + 41, + 52514, + 355, + 267, + 33835, + 20, + 18709, + 16896, + 302, + 22831, + 2292, + 395, + 7204, + 296, + 278, + 11622, + 1219, + 7624, + 22, + 810, + 2485, + 487, + 278, + 8465, + 3366, + 302, + 7575, + 1390, + 10977, + 88056, + 7575, + 41625, + 20219, + 88056, + 309, + 7575, + 64302, + 6514, + 962, + 302, + 18961, + 2137, + 1382, + 48875, + 22, + 2488, + 2345, + 278, + 6900, + 20, + 522, + 1946, + 1360, + 278, + 3008, + 302, + 1394, + 35664, + 405, + 435, + 26205, + 20, + 293, + 1663, + 664, + 20, + 309, + 94443, + 267, + 40252, + 964, + 549, + 1526, + 20, + 309, + 1256, + 13556, + 1030, + 278, + 7687, + 22118, + 309, + 10776, + 437, + 5522, + 467, + 278, + 41880, + 63908, + 64515, + 14066, + 229, + 25, + 22, + 7072, + 4601, + 309, + 278, + 28920, + 302, + 278, + 7072, + 8, + 35243, + 4601, + 355, + 267, + 13703, + 302, + 1196, + 42626, + 342, + 97322, + 17, + 302, + 9566, + 2292, + 22, + 2644, + 278, + 4478, + 355, + 2419, + 383, + 278, + 27150, + 20, + 851, + 23592, + 16519, + 278, + 48875, + 302, + 1382, + 296, + 278, + 10513, + 22, + 440, + 11630, + 462, + 643, + 278, + 4478, + 355, + 3036, + 278, + 27150, + 24694, + 278, + 3572, + 302, + 278, + 52514, + 383, + 278, + 3578, + 63908, + 64515, + 14066, + 229, + 26, + 22, + 12119, + 10977, + 342, + 42, + 3573, + 17, + 462, + 278, + 5206, + 42438, + 8, + 3952, + 267, + 2292, + 23935, + 26205, + 267, + 1382, + 4494, + 10530, + 20, + 435, + 58741, + 1707, + 309, + 293, + 2285, + 6568, + 278, + 3880, + 342, + 1319, + 2216, + 43801, + 310, + 278, + 3755, + 690, + 969, + 45204, + 355, + 2087, + 7575, + 1390, + 10977, + 88050, + 7656, + 278, + 115788, + 5935, + 302, + 1382, + 355, + 3084, + 898, + 395, + 302, + 2576, + 20, + 278, + 23935, + 3306, + 6084, + 22, + 440, + 3184, + 302, + 45204, + 10113, + 383, + 278, + 32358, + 302, + 278, + 2292, + 34, + 17913, + 42626, + 342, + 24475, + 20, + 56374, + 17 + ], + "text": "**How a Rainbow Forms \u2013 A Detailed Explanation**\n\nA rainbow is a spectacular, circular arc of colored light that appears in the sky after rain. It results from the combined effects of **refraction**, **internal reflection**, and **dispersion** of sunlight within water droplets. To understand the formation, we must follow the path of each photon as it enters, bounces, and exits a spherical raindrop, and then examine how the resulting angles and colors are observed by the viewer.\n\n---\n\n### 1. Sunlight and the Role of the Sun\n\nSunlight is a mixture of many wavelengths (colors) of visible light. When the sun is low on the horizon, its rays strike the droplets of water in the atmosphere. The angle at which the sun is above the horizon determines the position of the rainbow on the ground.\n\n---\n\n### 2. Refraction (Bending) at the First Surface\n\nWhen a light ray enters a water droplet, it slows down and bends toward the normal (the line perpendicular to the surface). This bending is called **refraction**. Because the refractive index of water is higher than that of air, the ray changes direction. The amount of bending depends on the wavelength of the light: shorter wavelengths (blue, violet)", + "start_time": "2026-10-30 20:20:01 EDT-0400", + "end_time": "2026-10-30 20:20:10 EDT-0400", + "peak_memory_bytes": 1943151270, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 75.9125573100022, + "text_sha256": "4c7cacd96bba54875c2b03652347284fd426697f6ea38a666e5d153e41e92a08", + "token_ids_sha256": "5e80e90f42530063ae3e40cc0b561be50136acc246f786c489c973e043a3fb23" + }, + { + "model": "lfm3", + "prompt_tokens": 37, + "prompt_token_sha256": "9dbb476f910146c702306094caf4089d17800e70462b147a034ecd5ba0f54e0f", + "ready_s": 4.717450750002172, + "ttft_s": 0.18223179200140294, + "total_s": 3.4234361670023645, + "completion_tokens": 256, + "decode_estimate_tps": 78.67445878044155, + "token_ids": [ + 962, + 3522, + 267, + 56445, + 59718, + 748, + 334, + 116632, + 90978, + 64521, + 41, + 52514, + 355, + 267, + 33835, + 20, + 18709, + 16896, + 302, + 22831, + 2292, + 395, + 7204, + 296, + 278, + 11622, + 1219, + 7624, + 22, + 810, + 2485, + 487, + 278, + 8465, + 3366, + 302, + 7575, + 1390, + 10977, + 88056, + 7575, + 41625, + 20219, + 88056, + 309, + 7575, + 64302, + 6514, + 962, + 302, + 18961, + 2137, + 1382, + 48875, + 22, + 2488, + 2345, + 278, + 6900, + 20, + 522, + 1946, + 1360, + 278, + 3008, + 302, + 1394, + 35664, + 405, + 435, + 26205, + 20, + 293, + 1663, + 664, + 20, + 309, + 94443, + 267, + 40252, + 964, + 549, + 1526, + 20, + 309, + 1256, + 13556, + 1030, + 278, + 7687, + 22118, + 309, + 10776, + 437, + 5522, + 467, + 278, + 41880, + 63908, + 64515, + 14066, + 229, + 25, + 22, + 7072, + 4601, + 309, + 278, + 28920, + 302, + 278, + 7072, + 8, + 35243, + 4601, + 355, + 267, + 13703, + 302, + 1196, + 42626, + 342, + 97322, + 17, + 302, + 9566, + 2292, + 22, + 2644, + 278, + 4478, + 355, + 2419, + 383, + 278, + 27150, + 20, + 851, + 23592, + 16519, + 278, + 48875, + 302, + 1382, + 296, + 278, + 10513, + 22, + 440, + 11630, + 462, + 643, + 278, + 4478, + 355, + 3036, + 278, + 27150, + 24694, + 278, + 3572, + 302, + 278, + 52514, + 383, + 278, + 3578, + 63908, + 64515, + 14066, + 229, + 26, + 22, + 12119, + 10977, + 342, + 42, + 3573, + 17, + 462, + 278, + 5206, + 42438, + 8, + 3952, + 267, + 2292, + 23935, + 26205, + 267, + 1382, + 4494, + 10530, + 20, + 435, + 58741, + 1707, + 309, + 293, + 2285, + 6568, + 278, + 3880, + 342, + 1319, + 2216, + 43801, + 310, + 278, + 3755, + 690, + 969, + 45204, + 355, + 2087, + 7575, + 1390, + 10977, + 88050, + 7656, + 278, + 115788, + 5935, + 302, + 1382, + 355, + 3084, + 898, + 395, + 302, + 2576, + 20, + 278, + 23935, + 3306, + 6084, + 22, + 440, + 3184, + 302, + 45204, + 10113, + 383, + 278, + 32358, + 302, + 278, + 2292, + 34, + 17913, + 42626, + 342, + 24475, + 20, + 56374, + 17 + ], + "text": "**How a Rainbow Forms \u2013 A Detailed Explanation**\n\nA rainbow is a spectacular, circular arc of colored light that appears in the sky after rain. It results from the combined effects of **refraction**, **internal reflection**, and **dispersion** of sunlight within water droplets. To understand the formation, we must follow the path of each photon as it enters, bounces, and exits a spherical raindrop, and then examine how the resulting angles and colors are observed by the viewer.\n\n---\n\n### 1. Sunlight and the Role of the Sun\n\nSunlight is a mixture of many wavelengths (colors) of visible light. When the sun is low on the horizon, its rays strike the droplets of water in the atmosphere. The angle at which the sun is above the horizon determines the position of the rainbow on the ground.\n\n---\n\n### 2. Refraction (Bending) at the First Surface\n\nWhen a light ray enters a water droplet, it slows down and bends toward the normal (the line perpendicular to the surface). This bending is called **refraction**. Because the refractive index of water is higher than that of air, the ray changes direction. The amount of bending depends on the wavelength of the light: shorter wavelengths (blue, violet)", + "start_time": "2026-10-30 20:20:26 EDT-0400", + "end_time": "2026-10-30 20:20:35 EDT-0400", + "peak_memory_bytes": 1942885398, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 78.67445878044155, + "text_sha256": "4c7cacd96bba54875c2b03652347284fd426697f6ea38a666e5d153e41e92a08", + "token_ids_sha256": "5e80e90f42530063ae3e40cc0b561be50136acc246f786c489c973e043a3fb23" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.18223179200140294, + "median_total_s": 3.528595209001651, + "median_decode_tps": 76.14655718830758 + } + }, + "model_implementation": { + "source": "https://github.com/techwritescode/mlx-lm/blob/9027ac5b062175fba890c612c3905d5d8d5b5294/mlx_lm/models/lfm3.py", + "sha256": "a4096cddc8f0d3f7923008b2e6afb896dabc9aef963a5ebe7d76d13a6069b0c4", + "note": "Pinned project model implementation loaded by module injection. Tokenizer emitted a custom-code prompt and continued without approval; transformers emitted a model-type compatibility warning. No correctness qualification." + }, + "prompt": "Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.", + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-short.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-short.json new file mode 100644 index 00000000..a07bb129 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-lfm3-short.json @@ -0,0 +1,196 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "lfm3", + "model": "LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7", + "weights_sha1": "3b46079dd265f521a232051c54c6218398c5e4b6", + "weights_sha256": "2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 32, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "lfm3", + "prompt_tokens": 25, + "prompt_token_sha256": "c076f3e59eae5e410fd2b2779d07c9a26cb486f3c40d9799055912b69e9fe653", + "ready_s": 2.8555167089980387, + "ttft_s": 0.1452582079982676, + "total_s": 0.6669637919985689, + "completion_tokens": 32, + "decode_estimate_tps": 59.4204872455076, + "token_ids": [ + 41, + 52514, + 355, + 3821, + 902, + 18961, + 16997, + 1125, + 1382, + 48875, + 296, + 278, + 2576, + 309, + 293, + 2285, + 342, + 1390, + 42531, + 608, + 1256, + 20914, + 1111, + 278, + 1391, + 302, + 1394, + 4494, + 10530, + 309, + 293, + 2285 + ], + "text": "A rainbow is created when sunlight passes through water droplets in the air and bends (refracts), then reflects off the back of each droplet and bends", + "start_time": "2026-10-30 20:18:38 EDT-0400", + "end_time": "2026-10-30 20:18:42 EDT-0400", + "peak_memory_bytes": 1971546866, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 59.4204872455076, + "text_sha256": "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b", + "token_ids_sha256": "9c5849d50ab2f4303948a9ff812ee3fc6b3c91543093aa2976c3e6f085ad4e09" + }, + { + "model": "lfm3", + "prompt_tokens": 25, + "prompt_token_sha256": "c076f3e59eae5e410fd2b2779d07c9a26cb486f3c40d9799055912b69e9fe653", + "ready_s": 1.9093678329991235, + "ttft_s": 0.14269441699798335, + "total_s": 0.5776229170005536, + "completion_tokens": 32, + "decode_estimate_tps": 71.27608330982405, + "token_ids": [ + 41, + 52514, + 355, + 3821, + 902, + 18961, + 16997, + 1125, + 1382, + 48875, + 296, + 278, + 2576, + 309, + 293, + 2285, + 342, + 1390, + 42531, + 608, + 1256, + 20914, + 1111, + 278, + 1391, + 302, + 1394, + 4494, + 10530, + 309, + 293, + 2285 + ], + "text": "A rainbow is created when sunlight passes through water droplets in the air and bends (refracts), then reflects off the back of each droplet and bends", + "start_time": "2026-10-30 20:18:58 EDT-0400", + "end_time": "2026-10-30 20:19:01 EDT-0400", + "peak_memory_bytes": 1966569574, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 71.27608330982405, + "text_sha256": "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b", + "token_ids_sha256": "9c5849d50ab2f4303948a9ff812ee3fc6b3c91543093aa2976c3e6f085ad4e09" + }, + { + "model": "lfm3", + "prompt_tokens": 25, + "prompt_token_sha256": "c076f3e59eae5e410fd2b2779d07c9a26cb486f3c40d9799055912b69e9fe653", + "ready_s": 1.961616791999404, + "ttft_s": 0.13646341699859477, + "total_s": 0.5920898330005002, + "completion_tokens": 32, + "decode_estimate_tps": 68.03819732846736, + "token_ids": [ + 41, + 52514, + 355, + 3821, + 902, + 18961, + 16997, + 1125, + 1382, + 48875, + 296, + 278, + 2576, + 309, + 293, + 2285, + 342, + 1390, + 42531, + 608, + 1256, + 20914, + 1111, + 278, + 1391, + 302, + 1394, + 4494, + 10530, + 309, + 293, + 2285 + ], + "text": "A rainbow is created when sunlight passes through water droplets in the air and bends (refracts), then reflects off the back of each droplet and bends", + "start_time": "2026-10-30 20:19:17 EDT-0400", + "end_time": "2026-10-30 20:19:20 EDT-0400", + "peak_memory_bytes": 1966304262, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 68.03819732846736, + "text_sha256": "b8cc5b9478a05082c75e5ec40e9d3a528f77c91c5ac0d639b827ff94b25a609b", + "token_ids_sha256": "9c5849d50ab2f4303948a9ff812ee3fc6b3c91543093aa2976c3e6f085ad4e09" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.14269441699798335, + "median_total_s": 0.5920898330005002, + "median_decode_tps": 68.03819732846736 + } + }, + "model_implementation": { + "source": "https://github.com/techwritescode/mlx-lm/blob/9027ac5b062175fba890c612c3905d5d8d5b5294/mlx_lm/models/lfm3.py", + "sha256": "a4096cddc8f0d3f7923008b2e6afb896dabc9aef963a5ebe7d76d13a6069b0c4", + "note": "Pinned project model implementation loaded by module injection. Tokenizer emitted a custom-code prompt and continued without approval; transformers emitted a model-type compatibility warning. No correctness qualification." + }, + "prompt": "Explain in plain language how a rainbow forms. Give four distinct sentences.", + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-long.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-long.json new file mode 100644 index 00000000..3066bea0 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-long.json @@ -0,0 +1,863 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "qwen2b", + "model": "mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3", + "weights_sha1": "e2219e3a77ad2fe2fed64b3ea7892d524e66d606", + "weights_sha256": "713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 256, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "qwen2b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 4.468082041999878, + "ttft_s": 0.2997159590013325, + "total_s": 2.815021541999158, + "completion_tokens": 256, + "decode_estimate_tps": 101.37933208738893, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 5629, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 303, + 279, + 16078, + 11, + 6611, + 264, + 19142, + 314, + 7736, + 421, + 567, + 1436, + 303, + 279, + 12515, + 13, + 1061, + 27374, + 2376, + 369, + 264, + 6150, + 916, + 1315, + 314, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 82655, + 11, + 321, + 279, + 19489, + 8937, + 13, + 1946, + 3418, + 1204, + 264, + 46026, + 7346, + 11, + 799, + 1902, + 1118, + 2814, + 279, + 5706, + 314, + 3000, + 321, + 279, + 7465, + 314, + 2919, + 6756, + 86412, + 13, + 271, + 760, + 1817, + 11690, + 440, + 279, + 4202, + 314, + 37728, + 1083, + 264, + 2919, + 6756, + 10502, + 13, + 7806, + 4009, + 369, + 22687, + 314, + 660, + 9155, + 89661, + 314, + 279, + 63358, + 19142, + 11, + 23245, + 494, + 2438, + 310, + 76990, + 13, + 1577, + 1439, + 42655, + 3599, + 279, + 6756, + 10502, + 11, + 781, + 6103, + 1441, + 321, + 35600, + 11, + 264, + 1817, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 18766, + 519, + 506, + 9781, + 2086, + 25111, + 13, + 3605, + 3000, + 682, + 279, + 21354, + 43796, + 321, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 682, + 279, + 37847, + 43796, + 321, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 7736, + 369, + 2512, + 82655, + 13, + 28477, + 15783, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 17754, + 89600, + 8574, + 279, + 4472, + 1500, + 318, + 276, + 48271, + 1500, + 74206, + 310, + 279, + 7099, + 553, + 271, + 23705, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 35456, + 287, + 264, + 2018, + 1985, + 15722, + 430, + 424, + 40693, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 13, + 4213, + 11, + 279, + 1752, + 369, + 524, + 11040, + 13094, + 26, + 279, + 3000, + 17754, + 35456, + 287, + 264, + 2702, + 5138, + 21172, + 506, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 1518, + 22482, + 13, + 1061, + 21172, + 369 + ], + "text": "A rainbow is a spectacular natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere, creating a spectrum of colors that we see in the sky. This optical effect is a complex interplay of physics involving refraction, reflection, dispersion, and the viewing angle. To understand how a rainbow forms, one must first consider the properties of light and the behavior of water droplets.\n\nThe process begins with the entry of sunlight into a water droplet. Sunlight is composed of all visible wavelengths of the electromagnetic spectrum, ranging from red to violet. As these rays enter the droplet, they slow down and bend, a process known as refraction. Crucially, different wavelengths of light refract at slightly different angles. Red light has the longest wavelength and bends the least, while violet light has the shortest wavelength and bends the most. This separation of colors is called dispersion. Upon entering the droplet, the light ray bends toward the normal line (an imaginary line perpendicular to the surface).\n\nInside the droplet, the light undergoes a second refraction as it exits the back of the droplet. However, the path is not entirely linear; the light ray undergoes a total internal reflection at the back of the droplet before emerging. This reflection is", + "start_time": "2026-10-30 20:12:31 EDT-0400", + "end_time": "2026-10-30 20:12:39 EDT-0400", + "peak_memory_bytes": 1191309564, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 101.37933208738893, + "text_sha256": "e0c3922c7d3fd430bdec1f3c211d48da56a805e582f2d885b639ffc20878086e", + "token_ids_sha256": "45d0c42ed269ccd376013e3db385ae9a80033af2d55f7553734903b5e1edb7e3" + }, + { + "model": "qwen2b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 4.5881985840023845, + "ttft_s": 0.2899765000001935, + "total_s": 2.8236517920013284, + "completion_tokens": 256, + "decode_estimate_tps": 100.64430939711977, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 5629, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 303, + 279, + 16078, + 11, + 6611, + 264, + 19142, + 314, + 7736, + 421, + 567, + 1436, + 303, + 279, + 12515, + 13, + 1061, + 27374, + 2376, + 369, + 264, + 6150, + 916, + 1315, + 314, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 82655, + 11, + 321, + 279, + 19489, + 8937, + 13, + 1946, + 3418, + 1204, + 264, + 46026, + 7346, + 11, + 799, + 1902, + 1118, + 2814, + 279, + 5706, + 314, + 3000, + 321, + 279, + 7465, + 314, + 2919, + 6756, + 86412, + 13, + 271, + 760, + 1817, + 11690, + 440, + 279, + 4202, + 314, + 37728, + 1083, + 264, + 2919, + 6756, + 10502, + 13, + 7806, + 4009, + 369, + 22687, + 314, + 660, + 9155, + 89661, + 314, + 279, + 63358, + 19142, + 11, + 23245, + 494, + 2438, + 310, + 76990, + 13, + 1577, + 1439, + 42655, + 3599, + 279, + 6756, + 10502, + 11, + 781, + 6103, + 1441, + 321, + 35600, + 11, + 264, + 1817, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 18766, + 519, + 506, + 9781, + 2086, + 25111, + 13, + 3605, + 3000, + 682, + 279, + 21354, + 43796, + 321, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 682, + 279, + 37847, + 43796, + 321, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 7736, + 369, + 2512, + 82655, + 13, + 28477, + 15783, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 17754, + 89600, + 8574, + 279, + 4472, + 1500, + 318, + 276, + 48271, + 1500, + 74206, + 310, + 279, + 7099, + 553, + 271, + 23705, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 35456, + 287, + 264, + 2018, + 1985, + 15722, + 430, + 424, + 40693, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 13, + 4213, + 11, + 279, + 1752, + 369, + 524, + 11040, + 13094, + 26, + 279, + 3000, + 17754, + 35456, + 287, + 264, + 2702, + 5138, + 21172, + 506, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 1518, + 22482, + 13, + 1061, + 21172, + 369 + ], + "text": "A rainbow is a spectacular natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere, creating a spectrum of colors that we see in the sky. This optical effect is a complex interplay of physics involving refraction, reflection, dispersion, and the viewing angle. To understand how a rainbow forms, one must first consider the properties of light and the behavior of water droplets.\n\nThe process begins with the entry of sunlight into a water droplet. Sunlight is composed of all visible wavelengths of the electromagnetic spectrum, ranging from red to violet. As these rays enter the droplet, they slow down and bend, a process known as refraction. Crucially, different wavelengths of light refract at slightly different angles. Red light has the longest wavelength and bends the least, while violet light has the shortest wavelength and bends the most. This separation of colors is called dispersion. Upon entering the droplet, the light ray bends toward the normal line (an imaginary line perpendicular to the surface).\n\nInside the droplet, the light undergoes a second refraction as it exits the back of the droplet. However, the path is not entirely linear; the light ray undergoes a total internal reflection at the back of the droplet before emerging. This reflection is", + "start_time": "2026-10-30 20:12:55 EDT-0400", + "end_time": "2026-10-30 20:13:03 EDT-0400", + "peak_memory_bytes": 1191375100, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 100.64430939711977, + "text_sha256": "e0c3922c7d3fd430bdec1f3c211d48da56a805e582f2d885b639ffc20878086e", + "token_ids_sha256": "45d0c42ed269ccd376013e3db385ae9a80033af2d55f7553734903b5e1edb7e3" + }, + { + "model": "qwen2b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 4.464824167000188, + "ttft_s": 0.30224804100362235, + "total_s": 2.7173295830034476, + "completion_tokens": 256, + "decode_estimate_tps": 105.58649700450505, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 5629, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 303, + 279, + 16078, + 11, + 6611, + 264, + 19142, + 314, + 7736, + 421, + 567, + 1436, + 303, + 279, + 12515, + 13, + 1061, + 27374, + 2376, + 369, + 264, + 6150, + 916, + 1315, + 314, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 82655, + 11, + 321, + 279, + 19489, + 8937, + 13, + 1946, + 3418, + 1204, + 264, + 46026, + 7346, + 11, + 799, + 1902, + 1118, + 2814, + 279, + 5706, + 314, + 3000, + 321, + 279, + 7465, + 314, + 2919, + 6756, + 86412, + 13, + 271, + 760, + 1817, + 11690, + 440, + 279, + 4202, + 314, + 37728, + 1083, + 264, + 2919, + 6756, + 10502, + 13, + 7806, + 4009, + 369, + 22687, + 314, + 660, + 9155, + 89661, + 314, + 279, + 63358, + 19142, + 11, + 23245, + 494, + 2438, + 310, + 76990, + 13, + 1577, + 1439, + 42655, + 3599, + 279, + 6756, + 10502, + 11, + 781, + 6103, + 1441, + 321, + 35600, + 11, + 264, + 1817, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 18766, + 519, + 506, + 9781, + 2086, + 25111, + 13, + 3605, + 3000, + 682, + 279, + 21354, + 43796, + 321, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 682, + 279, + 37847, + 43796, + 321, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 7736, + 369, + 2512, + 82655, + 13, + 28477, + 15783, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 17754, + 89600, + 8574, + 279, + 4472, + 1500, + 318, + 276, + 48271, + 1500, + 74206, + 310, + 279, + 7099, + 553, + 271, + 23705, + 279, + 6756, + 10502, + 11, + 279, + 3000, + 35456, + 287, + 264, + 2018, + 1985, + 15722, + 430, + 424, + 40693, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 13, + 4213, + 11, + 279, + 1752, + 369, + 524, + 11040, + 13094, + 26, + 279, + 3000, + 17754, + 35456, + 287, + 264, + 2702, + 5138, + 21172, + 506, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 1518, + 22482, + 13, + 1061, + 21172, + 369 + ], + "text": "A rainbow is a spectacular natural phenomenon that occurs when sunlight interacts with water droplets in the atmosphere, creating a spectrum of colors that we see in the sky. This optical effect is a complex interplay of physics involving refraction, reflection, dispersion, and the viewing angle. To understand how a rainbow forms, one must first consider the properties of light and the behavior of water droplets.\n\nThe process begins with the entry of sunlight into a water droplet. Sunlight is composed of all visible wavelengths of the electromagnetic spectrum, ranging from red to violet. As these rays enter the droplet, they slow down and bend, a process known as refraction. Crucially, different wavelengths of light refract at slightly different angles. Red light has the longest wavelength and bends the least, while violet light has the shortest wavelength and bends the most. This separation of colors is called dispersion. Upon entering the droplet, the light ray bends toward the normal line (an imaginary line perpendicular to the surface).\n\nInside the droplet, the light undergoes a second refraction as it exits the back of the droplet. However, the path is not entirely linear; the light ray undergoes a total internal reflection at the back of the droplet before emerging. This reflection is", + "start_time": "2026-10-30 20:13:19 EDT-0400", + "end_time": "2026-10-30 20:13:26 EDT-0400", + "peak_memory_bytes": 1191375100, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 105.58649700450505, + "text_sha256": "e0c3922c7d3fd430bdec1f3c211d48da56a805e582f2d885b639ffc20878086e", + "token_ids_sha256": "45d0c42ed269ccd376013e3db385ae9a80033af2d55f7553734903b5e1edb7e3" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.2997159590013325, + "median_total_s": 2.815021541999158, + "median_decode_tps": 101.37933208738893 + } + }, + "prompt": "Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.", + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-short.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-short.json new file mode 100644 index 00000000..61580cba --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen2b-short.json @@ -0,0 +1,191 @@ +{ + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "qwen2b", + "model": "mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3", + "weights_sha1": "e2219e3a77ad2fe2fed64b3ea7892d524e66d606", + "weights_sha256": "713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 32, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "qwen2b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 3.0237954589974834, + "ttft_s": 0.24648666599750868, + "total_s": 0.54696524999963, + "completion_tokens": 32, + "decode_estimate_tps": 103.16875028864334, + "token_ids": [ + 29108, + 4009, + 33060, + 1472, + 279, + 16078, + 321, + 5055, + 18766, + 21479, + 318, + 65, + 305, + 8, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 198, + 760, + 3000, + 1179, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756 + ], + "text": "Sunlight travels through the atmosphere and gets refracted (bent) as it enters a raindrop.\nThe light then reflects off the back of the dro", + "start_time": "2026-10-30 20:11:33 EDT-0400", + "end_time": "2026-10-30 20:11:36 EDT-0400", + "peak_memory_bytes": 1200510234, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 103.16875028864334, + "text_sha256": "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e", + "token_ids_sha256": "29e2f99787987b01d25007569e73c73812c9d0bd8c190b11dbda0cd30eadd098" + }, + { + "model": "qwen2b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 2.2403570000024047, + "ttft_s": 0.25593333299912047, + "total_s": 0.5575564999999187, + "completion_tokens": 32, + "decode_estimate_tps": 102.77725119144431, + "token_ids": [ + 29108, + 4009, + 33060, + 1472, + 279, + 16078, + 321, + 5055, + 18766, + 21479, + 318, + 65, + 305, + 8, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 198, + 760, + 3000, + 1179, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756 + ], + "text": "Sunlight travels through the atmosphere and gets refracted (bent) as it enters a raindrop.\nThe light then reflects off the back of the dro", + "start_time": "2026-10-30 20:11:53 EDT-0400", + "end_time": "2026-10-30 20:11:56 EDT-0400", + "peak_memory_bytes": 1200510426, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 102.77725119144431, + "text_sha256": "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e", + "token_ids_sha256": "29e2f99787987b01d25007569e73c73812c9d0bd8c190b11dbda0cd30eadd098" + }, + { + "model": "qwen2b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 2.212231542001973, + "ttft_s": 0.24776562500119326, + "total_s": 0.5580588750017341, + "completion_tokens": 32, + "decode_estimate_tps": 99.9054926265588, + "token_ids": [ + 29108, + 4009, + 33060, + 1472, + 279, + 16078, + 321, + 5055, + 18766, + 21479, + 318, + 65, + 305, + 8, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 198, + 760, + 3000, + 1179, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756 + ], + "text": "Sunlight travels through the atmosphere and gets refracted (bent) as it enters a raindrop.\nThe light then reflects off the back of the dro", + "start_time": "2026-10-30 20:12:12 EDT-0400", + "end_time": "2026-10-30 20:12:15 EDT-0400", + "peak_memory_bytes": 1200510426, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 99.9054926265588, + "text_sha256": "ffbf23cb27bb50106bf151750276ab30f61683b0bec1ceb077bab1d61844393e", + "token_ids_sha256": "29e2f99787987b01d25007569e73c73812c9d0bd8c190b11dbda0cd30eadd098" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.24776562500119326, + "median_total_s": 0.5575564999999187, + "median_decode_tps": 102.77725119144431 + } + }, + "prompt": "Explain in plain language how a rainbow forms. Give four distinct sentences.", + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926" +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-long.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-long.json new file mode 100644 index 00000000..50e9b892 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-long.json @@ -0,0 +1,863 @@ +{ + "prompt": "Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.", + "prompt_sha256": "2f23962cfdcb7ce0c2b482c4dd71e415f72113b963cfe6594fd5dc1fd4347921", + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "qwen4b", + "model": "mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3", + "weights_sha1": "8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5", + "weights_sha256": "5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 256, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "qwen4b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 6.772894207999343, + "ttft_s": 0.4182184170022083, + "total_s": 5.039851124998677, + "completion_tokens": 256, + "decode_estimate_tps": 55.175306241621584, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 27374, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 20950, + 303, + 279, + 16078, + 11, + 6611, + 264, + 31883, + 15123, + 314, + 7736, + 13, + 1061, + 5629, + 2940, + 369, + 524, + 8613, + 539, + 279, + 6797, + 4924, + 9722, + 1829, + 11, + 694, + 4598, + 539, + 279, + 6150, + 916, + 1315, + 314, + 3000, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 321, + 82655, + 2785, + 13280, + 61984, + 2919, + 6756, + 86412, + 13, + 1946, + 3418, + 1204, + 411, + 13273, + 11, + 799, + 1902, + 11319, + 279, + 11285, + 314, + 264, + 3074, + 17754, + 314, + 4022, + 37728, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 271, + 760, + 1817, + 11690, + 440, + 2972, + 1062, + 15722, + 159034, + 1577, + 14835, + 42655, + 314, + 4022, + 37728, + 5313, + 264, + 10849, + 6508, + 11, + 781, + 12787, + 279, + 3929, + 7099, + 314, + 279, + 6756, + 10502, + 13, + 8938, + 2919, + 369, + 37564, + 261, + 1056, + 3594, + 11, + 279, + 3000, + 66828, + 1441, + 321, + 89600, + 430, + 424, + 27891, + 279, + 2919, + 13, + 1061, + 55207, + 314, + 3000, + 369, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 2785, + 279, + 4022, + 22690, + 35600, + 539, + 9781, + 2086, + 14287, + 13, + 3605, + 3000, + 11, + 3322, + 279, + 21354, + 43796, + 11, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 11, + 440, + 279, + 37847, + 43796, + 11, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 4022, + 3000, + 1083, + 1141, + 71545, + 54100, + 7736, + 369, + 2512, + 2972, + 28821, + 1279, + 159034, + 271, + 12162, + 279, + 3000, + 369, + 74087, + 4613, + 279, + 6756, + 10502, + 11, + 424, + 33060, + 8574, + 279, + 1142, + 314, + 279, + 24561, + 13, + 28477, + 18176, + 279, + 8906, + 7099, + 11, + 279, + 3000, + 35456, + 287, + 2972, + 10168, + 21172, + 159034, + 43274, + 11, + 279, + 3000, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 11 + ], + "text": "A rainbow is a spectacular optical phenomenon that occurs when sunlight interacts with water droplets suspended in the atmosphere, creating a vibrant arc of colors. This natural display is not caused by the sun itself changing color, but rather by the complex interplay of light physics involving refraction, reflection, and dispersion within tiny spherical water droplets. To understand how this occurs, one must trace the journey of a single ray of white sunlight as it enters a raindrop.\n\nThe process begins with **refraction**. As parallel rays of white sunlight approach a raindrop, they strike the front surface of the droplet. Because water is denser than air, the light slows down and bends as it enters the water. This bending of light is known as refraction. Crucially, different wavelengths of light within the white beam bend by slightly different amounts. Red light, having the longest wavelength, bends the least, while violet light, with the shortest wavelength, bends the most. This separation of white light into its constituent spectral colors is called **dispersion**.\n\nOnce the light is dispersed inside the droplet, it travels toward the back of the sphere. Upon reaching the inner surface, the light undergoes **internal reflection**. Specifically, the light reflects off the back of the droplet,", + "start_time": "2026-10-30 20:24:11 EDT-0400", + "end_time": "2026-10-30 20:24:23 EDT-0400", + "peak_memory_bytes": 2575561928, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 55.175306241621584, + "text_sha256": "234f678cc2d4ba00d9893643fd127cb3ce066a186f1814b82ae46117a03c4ef5", + "token_ids_sha256": "ebb3cb671a0d8a37947cdec60f557a15846a4827185f1d1c1cec68bcbedf4f3e" + }, + { + "model": "qwen4b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 6.860008458002994, + "ttft_s": 0.4338122079971072, + "total_s": 5.2179280829986965, + "completion_tokens": 256, + "decode_estimate_tps": 53.301384553089505, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 27374, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 20950, + 303, + 279, + 16078, + 11, + 6611, + 264, + 31883, + 15123, + 314, + 7736, + 13, + 1061, + 5629, + 2940, + 369, + 524, + 8613, + 539, + 279, + 6797, + 4924, + 9722, + 1829, + 11, + 694, + 4598, + 539, + 279, + 6150, + 916, + 1315, + 314, + 3000, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 321, + 82655, + 2785, + 13280, + 61984, + 2919, + 6756, + 86412, + 13, + 1946, + 3418, + 1204, + 411, + 13273, + 11, + 799, + 1902, + 11319, + 279, + 11285, + 314, + 264, + 3074, + 17754, + 314, + 4022, + 37728, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 271, + 760, + 1817, + 11690, + 440, + 2972, + 1062, + 15722, + 159034, + 1577, + 14835, + 42655, + 314, + 4022, + 37728, + 5313, + 264, + 10849, + 6508, + 11, + 781, + 12787, + 279, + 3929, + 7099, + 314, + 279, + 6756, + 10502, + 13, + 8938, + 2919, + 369, + 37564, + 261, + 1056, + 3594, + 11, + 279, + 3000, + 66828, + 1441, + 321, + 89600, + 430, + 424, + 27891, + 279, + 2919, + 13, + 1061, + 55207, + 314, + 3000, + 369, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 2785, + 279, + 4022, + 22690, + 35600, + 539, + 9781, + 2086, + 14287, + 13, + 3605, + 3000, + 11, + 3322, + 279, + 21354, + 43796, + 11, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 11, + 440, + 279, + 37847, + 43796, + 11, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 4022, + 3000, + 1083, + 1141, + 71545, + 54100, + 7736, + 369, + 2512, + 2972, + 28821, + 1279, + 159034, + 271, + 12162, + 279, + 3000, + 369, + 74087, + 4613, + 279, + 6756, + 10502, + 11, + 424, + 33060, + 8574, + 279, + 1142, + 314, + 279, + 24561, + 13, + 28477, + 18176, + 279, + 8906, + 7099, + 11, + 279, + 3000, + 35456, + 287, + 2972, + 10168, + 21172, + 159034, + 43274, + 11, + 279, + 3000, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 11 + ], + "text": "A rainbow is a spectacular optical phenomenon that occurs when sunlight interacts with water droplets suspended in the atmosphere, creating a vibrant arc of colors. This natural display is not caused by the sun itself changing color, but rather by the complex interplay of light physics involving refraction, reflection, and dispersion within tiny spherical water droplets. To understand how this occurs, one must trace the journey of a single ray of white sunlight as it enters a raindrop.\n\nThe process begins with **refraction**. As parallel rays of white sunlight approach a raindrop, they strike the front surface of the droplet. Because water is denser than air, the light slows down and bends as it enters the water. This bending of light is known as refraction. Crucially, different wavelengths of light within the white beam bend by slightly different amounts. Red light, having the longest wavelength, bends the least, while violet light, with the shortest wavelength, bends the most. This separation of white light into its constituent spectral colors is called **dispersion**.\n\nOnce the light is dispersed inside the droplet, it travels toward the back of the sphere. Upon reaching the inner surface, the light undergoes **internal reflection**. Specifically, the light reflects off the back of the droplet,", + "start_time": "2026-10-30 20:24:39 EDT-0400", + "end_time": "2026-10-30 20:24:51 EDT-0400", + "peak_memory_bytes": 2575529160, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 53.301384553089505, + "text_sha256": "234f678cc2d4ba00d9893643fd127cb3ce066a186f1814b82ae46117a03c4ef5", + "token_ids_sha256": "ebb3cb671a0d8a37947cdec60f557a15846a4827185f1d1c1cec68bcbedf4f3e" + }, + { + "model": "qwen4b", + "prompt_tokens": 41, + "prompt_token_sha256": "c3d1a6c5d63a2ecfb567f72f4d3b4857bf53ec60b21b8d20f7e450cf17948da1", + "ready_s": 6.671107916998153, + "ttft_s": 0.42590149999887217, + "total_s": 5.042005915998743, + "completion_tokens": 256, + "decode_estimate_tps": 55.24138473041142, + "token_ids": [ + 32, + 46026, + 369, + 264, + 26811, + 27374, + 24057, + 421, + 13273, + 948, + 37728, + 80291, + 440, + 2919, + 6756, + 86412, + 20950, + 303, + 279, + 16078, + 11, + 6611, + 264, + 31883, + 15123, + 314, + 7736, + 13, + 1061, + 5629, + 2940, + 369, + 524, + 8613, + 539, + 279, + 6797, + 4924, + 9722, + 1829, + 11, + 694, + 4598, + 539, + 279, + 6150, + 916, + 1315, + 314, + 3000, + 20665, + 15394, + 1985, + 15722, + 11, + 21172, + 11, + 321, + 82655, + 2785, + 13280, + 61984, + 2919, + 6756, + 86412, + 13, + 1946, + 3418, + 1204, + 411, + 13273, + 11, + 799, + 1902, + 11319, + 279, + 11285, + 314, + 264, + 3074, + 17754, + 314, + 4022, + 37728, + 430, + 424, + 27891, + 264, + 10849, + 6508, + 13, + 271, + 760, + 1817, + 11690, + 440, + 2972, + 1062, + 15722, + 159034, + 1577, + 14835, + 42655, + 314, + 4022, + 37728, + 5313, + 264, + 10849, + 6508, + 11, + 781, + 12787, + 279, + 3929, + 7099, + 314, + 279, + 6756, + 10502, + 13, + 8938, + 2919, + 369, + 37564, + 261, + 1056, + 3594, + 11, + 279, + 3000, + 66828, + 1441, + 321, + 89600, + 430, + 424, + 27891, + 279, + 2919, + 13, + 1061, + 55207, + 314, + 3000, + 369, + 3750, + 430, + 1985, + 15722, + 13, + 72451, + 32365, + 11, + 2086, + 89661, + 314, + 3000, + 2785, + 279, + 4022, + 22690, + 35600, + 539, + 9781, + 2086, + 14287, + 13, + 3605, + 3000, + 11, + 3322, + 279, + 21354, + 43796, + 11, + 89600, + 279, + 3140, + 11, + 1345, + 76990, + 3000, + 11, + 440, + 279, + 37847, + 43796, + 11, + 89600, + 279, + 1379, + 13, + 1061, + 23956, + 314, + 4022, + 3000, + 1083, + 1141, + 71545, + 54100, + 7736, + 369, + 2512, + 2972, + 28821, + 1279, + 159034, + 271, + 12162, + 279, + 3000, + 369, + 74087, + 4613, + 279, + 6756, + 10502, + 11, + 424, + 33060, + 8574, + 279, + 1142, + 314, + 279, + 24561, + 13, + 28477, + 18176, + 279, + 8906, + 7099, + 11, + 279, + 3000, + 35456, + 287, + 2972, + 10168, + 21172, + 159034, + 43274, + 11, + 279, + 3000, + 25133, + 974, + 279, + 1142, + 314, + 279, + 6756, + 10502, + 11 + ], + "text": "A rainbow is a spectacular optical phenomenon that occurs when sunlight interacts with water droplets suspended in the atmosphere, creating a vibrant arc of colors. This natural display is not caused by the sun itself changing color, but rather by the complex interplay of light physics involving refraction, reflection, and dispersion within tiny spherical water droplets. To understand how this occurs, one must trace the journey of a single ray of white sunlight as it enters a raindrop.\n\nThe process begins with **refraction**. As parallel rays of white sunlight approach a raindrop, they strike the front surface of the droplet. Because water is denser than air, the light slows down and bends as it enters the water. This bending of light is known as refraction. Crucially, different wavelengths of light within the white beam bend by slightly different amounts. Red light, having the longest wavelength, bends the least, while violet light, with the shortest wavelength, bends the most. This separation of white light into its constituent spectral colors is called **dispersion**.\n\nOnce the light is dispersed inside the droplet, it travels toward the back of the sphere. Upon reaching the inner surface, the light undergoes **internal reflection**. Specifically, the light reflects off the back of the droplet,", + "start_time": "2026-10-30 20:25:08 EDT-0400", + "end_time": "2026-10-30 20:25:20 EDT-0400", + "peak_memory_bytes": 2575529160, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 55.24138473041142, + "text_sha256": "234f678cc2d4ba00d9893643fd127cb3ce066a186f1814b82ae46117a03c4ef5", + "token_ids_sha256": "ebb3cb671a0d8a37947cdec60f557a15846a4827185f1d1c1cec68bcbedf4f3e" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.42590149999887217, + "median_total_s": 5.042005915998743, + "median_decode_tps": 55.175306241621584 + } + } +} diff --git a/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-short.json b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-short.json new file mode 100644 index 00000000..7800a832 --- /dev/null +++ b/mseries/local-ai/receipts/2026-10-30-macos-scoreboard-qwen4b-short.json @@ -0,0 +1,191 @@ +{ + "prompt": "Explain in plain language how a rainbow forms. Give four distinct sentences.", + "prompt_sha256": "af125b15dfba55caedb541e89e59fe1252c0f13553e5ce6bc181e237fada9926", + "schema": "mac-scoreboard-v1", + "status": "complete", + "model_key": "qwen4b", + "model": "mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3", + "weights_sha1": "8dd392f6cbaaf73322ce289dcf8f8c62d82fa0a5", + "weights_sha256": "5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db", + "os": "macOS 27.2 build 26B123", + "kernel": "Darwin 27.2.0 RELEASE_ARM64_T6000", + "hardware": "MacBookPro18,3 Apple M1 Pro 32 GB", + "driver": "Apple Metal", + "mlx_version": "0.31.2", + "mlx_lm_version": "0.31.3", + "max_tokens": 32, + "arms": [ + "mac-mlx" + ], + "concurrency": 1, + "method": "Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.", + "runs": [ + { + "model": "qwen4b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 2.9788147079998453, + "ttft_s": 0.3438798749994021, + "total_s": 0.9221773329991265, + "completion_tokens": 32, + "decode_estimate_tps": 53.605630754847226, + "token_ids": [ + 32, + 46026, + 7346, + 948, + 37728, + 12611, + 10849, + 87398, + 303, + 279, + 3594, + 321, + 5055, + 6507, + 1083, + 1141, + 7840, + 7736, + 11, + 1691, + 1040, + 264, + 90387, + 13, + 8618, + 10849, + 6508, + 13653, + 1040, + 264, + 13280, + 17796 + ], + "text": "A rainbow forms when sunlight hits raindrops in the air and gets split into its seven colors, much like a prism. Each raindrop acts like a tiny lens", + "start_time": "2026-10-30 20:23:11 EDT-0400", + "end_time": "2026-10-30 20:23:15 EDT-0400", + "peak_memory_bytes": 2561906800, + "arm": "mac-mlx", + "round": 1, + "decode_tps": 53.605630754847226, + "text_sha256": "e74285cad2b17b0b4991f3aa68d669d7ae2442c37bf93e366dc901ec1bb2f476", + "token_ids_sha256": "8c73aeaf94801998c85ff327664840ea2ef6472c4cd7a6d4783951cb8eed505b" + }, + { + "model": "qwen4b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 2.577901374999783, + "ttft_s": 0.3200703330003307, + "total_s": 0.9047864169988316, + "completion_tokens": 32, + "decode_estimate_tps": 53.01718363553598, + "token_ids": [ + 32, + 46026, + 7346, + 948, + 37728, + 12611, + 10849, + 87398, + 303, + 279, + 3594, + 321, + 5055, + 6507, + 1083, + 1141, + 7840, + 7736, + 11, + 1691, + 1040, + 264, + 90387, + 13, + 8618, + 10849, + 6508, + 13653, + 1040, + 264, + 13280, + 17796 + ], + "text": "A rainbow forms when sunlight hits raindrops in the air and gets split into its seven colors, much like a prism. Each raindrop acts like a tiny lens", + "start_time": "2026-10-30 20:23:31 EDT-0400", + "end_time": "2026-10-30 20:23:35 EDT-0400", + "peak_memory_bytes": 2561988720, + "arm": "mac-mlx", + "round": 2, + "decode_tps": 53.01718363553598, + "text_sha256": "e74285cad2b17b0b4991f3aa68d669d7ae2442c37bf93e366dc901ec1bb2f476", + "token_ids_sha256": "8c73aeaf94801998c85ff327664840ea2ef6472c4cd7a6d4783951cb8eed505b" + }, + { + "model": "qwen4b", + "prompt_tokens": 27, + "prompt_token_sha256": "bd0ce9e2d453b2a87178b02ddf49bdae7058508e0c5a13066e1137dcb2be433a", + "ready_s": 2.518986874998518, + "ttft_s": 0.32595424999817624, + "total_s": 0.9036256670005969, + "completion_tokens": 32, + "decode_estimate_tps": 53.66372489201781, + "token_ids": [ + 32, + 46026, + 7346, + 948, + 37728, + 12611, + 10849, + 87398, + 303, + 279, + 3594, + 321, + 5055, + 6507, + 1083, + 1141, + 7840, + 7736, + 11, + 1691, + 1040, + 264, + 90387, + 13, + 8618, + 10849, + 6508, + 13653, + 1040, + 264, + 13280, + 17796 + ], + "text": "A rainbow forms when sunlight hits raindrops in the air and gets split into its seven colors, much like a prism. Each raindrop acts like a tiny lens", + "start_time": "2026-10-30 20:23:51 EDT-0400", + "end_time": "2026-10-30 20:23:55 EDT-0400", + "peak_memory_bytes": 2561955952, + "arm": "mac-mlx", + "round": 3, + "decode_tps": 53.66372489201781, + "text_sha256": "e74285cad2b17b0b4991f3aa68d669d7ae2442c37bf93e366dc901ec1bb2f476", + "token_ids_sha256": "8c73aeaf94801998c85ff327664840ea2ef6472c4cd7a6d4783951cb8eed505b" + } + ], + "summary": { + "mac-mlx": { + "ok_runs": 3, + "median_ttft_s": 0.32595424999817624, + "median_total_s": 0.9047864169988316, + "median_decode_tps": 53.605630754847226 + } + } +} diff --git a/mseries/local-ai/relative-scoreboard.svg b/mseries/local-ai/relative-scoreboard.svg new file mode 100644 index 00000000..d5b9a84e --- /dev/null +++ b/mseries/local-ai/relative-scoreboard.svg @@ -0,0 +1 @@ +Relative Aurora and macOS in-process MLX timingsRelative time: macOS = 1× · same chip/model/prompt · in-process MLX · ratios of separate run mediansDecode time (× Mac) · lower is betterFirst token time (× Mac) · lower is better1×1×M1 Pro · Granite 1B · 32 · wip4/9323171 · 120k · text matchGranite 4.0 1B MLX 4-bit (IBM, dense); macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 2.09×2.09×Granite 4.0 1B MLX 4-bit (IBM, dense); macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.73×1.73×M1 Pro · Granite 1B · 32 · wip4/9323171 · 5k · text matchGranite 4.0 1B MLX 4-bit (IBM, dense); macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.43×2.43×Granite 4.0 1B MLX 4-bit (IBM, dense); macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 1.78×1.78×M1 Pro · Liquid LFM3 1.7B · 32 · wip4/7faf04c · 120k · text matchLFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 4.03×4.03×LFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 2.76×2.76×M1 Pro · Liquid LFM3 1.7B · 32 · wip4/7faf04c · 5k · text matchLFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 4.46×4.46×LFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.78×2.78×M1 Pro · Liquid LFM3 1.7B · 32 · wip4/9323171 · 120k · text matchLFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 2.48×2.48×LFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 2.12×2.12×M1 Pro · Liquid LFM3 1.7B · 32 · wip4/9323171 · 5k · text matchLFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 3.01×3.01×LFM3-1.7B MLX 8-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.30×2.30×M1 Pro · Qwen3.5 2B · 32 · wip4/7faf04c · 120k · text matchQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 2.18×2.18×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.87×1.87×M1 Pro · Qwen3.5 2B · 32 · wip4/7faf04c · 5k · text matchQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.85×2.85×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.05×2.05×M1 Pro · Qwen3.5 2B · 32 · wip4/9323171 · 120k · text matchQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.82×1.82×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.83×1.83×M1 Pro · Qwen3.5 2B · 32 · wip4/9323171 · 5k · text matchQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 2.10×2.10×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text match; ratio 1.86×1.86×M1 Pro · Qwen3.5 2B · 32 · wip5/9323171 · 120k · text matchQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip5, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.89×1.89×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip5, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text match; ratio 1.71×1.71×M1 Pro · Qwen3.5 2B · 256 · wip4/7faf04c · 120k · text differentQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 2.06×2.06×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 1.87×1.87×M1 Pro · Qwen3.5 2B · 256 · wip4/7faf04c · 5k · text differentQwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 2.73×2.73×Qwen3.5-2B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 1.84×1.84×M1 Pro · Qwen3.5 4B · 32 · wip4/7faf04c · 120k · text differentQwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 2.16×2.16×Qwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 2.17×2.17×M1 Pro · Qwen3.5 4B · 32 · wip4/7faf04c · 5k · text differentQwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 2.89×2.89×Qwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 7faf04c065, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 2.24×2.24×M1 Pro · Qwen3.5 4B · 32 · wip4/9323171 · 120k · text differentQwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 1.98×1.98×Qwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, MLX_OMARCHY_BATCH_WORK=120000; text different; ratio 1.97×1.97×M1 Pro · Qwen3.5 4B · 32 · wip4/9323171 · 5k · text differentQwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 2.32×2.32×Qwen3.5-4B MLX 4-bit; macOS 27.2 build 26B123, Darwin 27.2.0 RELEASE_ARM64_T6000, MLX 0.31.2; mlx-lm 0.31.3; Aurora Linux, 7.2.11-elementary-aurora72-wip4, Honeykrisp Mesa 9323171c12, defaults (MLX_OMARCHY_BATCH_WORK fallback 5000); text different; ratio 2.00×2.00× diff --git a/mseries/local-ai/scoreboard-notes.json b/mseries/local-ai/scoreboard-notes.json index 1f0e77ec..138b14a7 100644 --- a/mseries/local-ai/scoreboard-notes.json +++ b/mseries/local-ai/scoreboard-notes.json @@ -31,11 +31,11 @@ { "model": "Qwen3.5-2B MLX 4-bit", "engine": "TensorFold Vulkan + submit batching", - "settings": "unpublished follow-up to PR 1544 (batching build a8b888dd)", + "settings": "Public PR 1561, closed and unmerged (earlier batching build a8b888dd)", "decode_tok_s": "~10.6 (M1 Max), ~6.9 (M1)", - "status": "reported upstream, code not public", - "blocker": "Watched in people-watch/branch-watch; re-run engine-compare when it lands", - "source": "ashhart/TensorFold issue 1420" + "status": "reported upstream; not tested here", + "blocker": "Patch needs adaptation to installed b3ab9ba and M1 Pro fidelity/performance gates", + "source": "[TensorFold PR 1561](https://github.com/ashhart/TensorFold/pull/1561)" }, { "model": "Qwen3.5 0.8B (Mirai format)", diff --git a/mseries/local-ai/scoreboard.py b/mseries/local-ai/scoreboard.py index cef0816c..b53b430c 100644 --- a/mseries/local-ai/scoreboard.py +++ b/mseries/local-ai/scoreboard.py @@ -3,10 +3,11 @@ Reads receipts/*engine-compare-*.json (engine-compare.py), receipts/*mlx-gate-sweep*.json (mlx-gate.py) and scoreboard-notes.json (rows measured elsewhere or reported by others), -then writes engine-scoreboard.csv and ENGINE-SCOREBOARD.md next to this script. -The newest receipt wins for each (model, engine, length). Run it after every new receipt. +plus completed macOS scoreboard receipts and batch-counter trials, then writes CSV, SVG and Markdown next to this script. +The newest receipt wins within each model revision, chip, OS, kernel, engine, driver and workload. """ -import csv, json, statistics +import csv, json, math, re, statistics +from html import escape from pathlib import Path HERE = Path(__file__).resolve().parent @@ -22,10 +23,10 @@ 'tensorfold': ('TensorFold Vulkan (HTTP server)', 'serve --backend vulkan --context 1024 --no-thinking', None), } # Receipts record the TensorFold revision they ran; name the PR it came from. -TF_PRS = {'670902196c47f3e70e3f56ab933805713be31aa8': 'PR 1517', '92bb49db5c313c105cbf4b44638d1e1896615174': 'PR 1544'} +TF_PRS = {'670902196c47f3e70e3f56ab933805713be31aa8': 'PR 1517', '92bb49db5c313c105cbf4b44638d1e1896615174': 'PR 1544', 'b3ab9ba217a80c63fcc48f7643ddfe339a8f4bdc': 'PR 1544 b3ab9ba'} MODEL_NAMES = {'qwen2b': 'Qwen3.5-2B MLX 4-bit', 'qwen4b': 'Qwen3.5-4B MLX 4-bit', 'lfm3': 'LFM3-1.7B MLX 8-bit', 'granite1b': 'Granite 4.0 1B MLX 4-bit (IBM, dense)'} -FIELDS = ['best', 'model', 'engine', 'settings', 'tokens', 'decode_tok_s', 'ttft_s', 'total_s', 'runs', +FIELDS = ['best', 'chip', 'os', 'model_reference', 'weights_sha1', 'weights_sha256', 'prompt_sha256', 'start_time', 'end_time', 'model', 'engine', 'settings', 'tokens', 'decode_tok_s', 'ttft_s', 'total_s', 'runs', 'same_output', 'gate', 'kernel', 'driver', 'date', 'status', 'source'] @@ -65,26 +66,42 @@ def reply_s(x): return x['ttft_s'] + REPLY_TOKENS / per +def chip_name(r, historical=False): + explicit = r.get('chip') or r.get('hardware') or '' + match = re.search(r'\bM[0-9]+(?:\s+(?:Pro|Max|Ultra))?\b', explicit, re.IGNORECASE) + if match: + return match.group(0) + return explicit or ('M1 Pro' if historical else 'Unknown chip') + + def measured_rows(): gates = gate_results() latest = {} - for f in sorted(RECEIPTS.glob('*engine-compare-*.json'), key=lambda f: f.stem): # stem: '-round2' sorts after its base + files = list(RECEIPTS.glob('*engine-compare-*.json')) + # The installed TensorFold qualification stores its receipts in a subdirectory. + files += [RECEIPTS / '2026-10-30-tensorfold-b3ab9ba' / f'compare-{size}.json' for size in ('short', 'long')] + for f in sorted((p for p in files if p.exists()), key=lambda p: (p.parent.name[:10] if p.parent != RECEIPTS else p.name[:10], p.stem)): r = json.loads(f.read_text()) if r.get('status') != 'complete': continue model = r.get('model_key', 'qwen2b') drv = r.get('driver', 'Honeykrisp Mesa 7faf04c065') conc = r.get('concurrency', 1) + # Existing receipts belong to the documented M1 Pro; future unlabelled hardware stays unknown. + date = f.parent.name[:10] if f.parent != RECEIPTS else f.name[:10] + chip = chip_name(r, historical=date <= '2026-10-30') + os_name = r.get('os') or ('Aurora Linux' if date <= '2026-10-30' else 'Unknown OS') for arm in r['arms']: - latest[(model, arm, r['max_tokens'], drv, conc)] = (f, r) + latest[(model, arm, r['max_tokens'], drv, conc, chip, os_name, r.get('kernel', ''), r.get('model', 'unknown revision'), r.get('prompt_sha256', 'unknown prompt'), r.get('weights_sha256') or 'unknown weights')] = (f, r) rows = [] - for (model, arm, tokens, drv, conc), (f, r) in sorted(latest.items()): + for (model, arm, tokens, drv, conc, chip, os_name, _kernel, model_ref, prompt_hash, weights_hash), (f, r) in sorted(latest.items()): ok = [x for x in r['runs'] if x['arm'] == arm and 'text' in x and not x.get('error')] if not ok: continue texts = {x['text_sha256'] for x in ok} - base = {x['text_sha256'] for x in r['runs'] if x['arm'] == 'mlx-5k' and 'text_sha256' in x} - if arm == 'mlx-5k': + base_arm = 'mlx-5k' if 'mlx-5k' in r['arms'] else ('mlx-120k' if 'mlx-120k' in r['arms'] else None) + base = {x['text_sha256'] for x in r['runs'] if x['arm'] == base_arm and 'text_sha256' in x} + if arm == base_arm: same = 'reference' elif not base: every = {x['text_sha256'] for x in r['runs'] if 'text_sha256' in x} @@ -92,7 +109,7 @@ def measured_rows(): elif texts == base: same = 'yes' else: - mt = next(x['text'] for x in r['runs'] if x['arm'] == 'mlx-5k' and 'text' in x) + mt = next(x['text'] for x in r['runs'] if x['arm'] == base_arm and 'text' in x) at = ok[0]['text'] i = next((i for i, (u, v) in enumerate(zip(mt, at)) if u != v), min(len(mt), len(at))) same = f'no: diverges after {i} of {len(mt)} chars' @@ -109,108 +126,286 @@ def measured_rows(): 'total_s': med([x['total_s'] for x in ok]), 'runs': len(ok), 'same_output': same if len(texts) == 1 else f'{same}; {len(texts)} variants across its own runs', 'gate': gate or 'not gated', 'kernel': r.get('kernel', ''), - 'driver': drv, 'date': f.name[:10], - 'status': 'measured here', 'source': f'receipts/{f.name}', '_model': model, '_conc': conc, '_arm': arm}) + 'driver': drv, 'date': f.parent.name[:10] if f.parent != RECEIPTS else f.name[:10], + 'status': 'measured here', 'source': str(f.relative_to(HERE)), '_model': model, '_conc': conc, '_arm': arm, + 'chip': chip, 'os': os_name, 'model_reference': model_ref, 'prompt_sha256': prompt_hash, + 'weights_sha256': weights_hash, '_text_hashes': texts}) # Most functional per model, length and load: the shortest typical reply (first token plus # REPLY_TOKENS at its streaming speed) among rows whose output matched and whose gate did not fail. - for key in {(x['_model'], x['tokens'], x['_conc']) for x in rows}: - group = [x for x in rows if (x['_model'], x['tokens'], x['_conc']) == key and x['gate'] != 'FAIL' + for key in {(x['_model'], x['tokens'], x['_conc'], x['chip'], x['os'], x['model_reference'], x['prompt_sha256'], x['weights_sha256']) for x in rows}: + group = [x for x in rows if (x['_model'], x['tokens'], x['_conc'], x['chip'], x['os'], x['model_reference'], x['prompt_sha256'], x['weights_sha256']) == key and x['gate'] != 'FAIL' and x['same_output'].startswith(('yes', 'reference')) and x['decode_tok_s']] if group: min(group, key=lambda x: reply_s(x) or 1e9)['best'] = '★' return rows +def mac_rows(): + """Read completed native Apple MLX receipts; do not assign Linux gate verdicts.""" + latest = {} + for f in sorted(RECEIPTS.glob('*-macos-scoreboard-*.json')): + r = json.loads(f.read_text()) + if r.get('status') != 'complete' or r.get('concurrency', 1) != 1: + continue + runs = [x for x in r.get('runs', []) if x.get('arm') == 'mac-mlx' and not x.get('error')] + if len(runs) < 3 or any(x.get('decode_tps') is None or x.get('ttft_s') is None for x in runs): + continue + model = r['model_key'] + chip = chip_name(r) + os_name = r.get('os', 'macOS') + latest[(model, r['max_tokens'], chip, os_name, r.get('hardware', ''), r.get('kernel', ''), r.get('mlx_version', ''), r.get('mlx_lm_version', ''), r.get('model', 'unknown revision'), r.get('prompt_sha256', 'unknown prompt'), r.get('weights_sha256') or 'unknown weights')] = (f, r, runs, chip, os_name) + rows = [] + for (model, tokens, _, _, _hardware, _kernel, _mlx, _mlx_lm, model_ref, prompt_hash, weights_hash), (f, r, runs, chip, os_name) in sorted(latest.items()): + rows.append({'model': MODEL_NAMES.get(model, model), 'engine': 'Apple MLX (in-process)', + 'settings': f"MLX {r.get('mlx_version', '?')}; mlx-lm {r.get('mlx_lm_version', '?')}", + 'tokens': tokens, 'decode_tok_s': med([x['decode_tps'] for x in runs]), + 'ttft_s': med([x['ttft_s'] for x in runs]), 'total_s': med([x['total_s'] for x in runs]), + 'runs': len(runs), 'same_output': 'not compared with Linux', 'gate': 'not gated', + 'kernel': r.get('kernel', ''), 'driver': r.get('driver', 'Apple Metal'), + 'date': f.name[:10], 'status': 'measured here', 'source': f'receipts/{f.name}', + '_model': model, '_conc': 1, '_arm': 'mac-mlx', 'chip': chip, 'os': os_name, + 'model_reference': model_ref, 'prompt_sha256': prompt_hash, 'weights_sha1': r.get('weights_sha1', ''), + 'weights_sha256': weights_hash, 'start_time': runs[0].get('start_time', ''), + 'end_time': runs[-1].get('end_time', ''), + '_text_hashes': ({x['text_sha256'] for x in runs} if all(x.get('text_sha256') for x in runs) else set())}) + return rows + + +def annotate_cross_platform(mac, linux): + """Compare synthetic text hashes only for same model, prompt, length and in-process boundary.""" + for row in mac: + refs = [x for x in linux if x['_arm'] in ('mlx-5k', 'mlx-120k') and x['_conc'] == 1 + and x['_model'] == row['_model'] and x['tokens'] == row['tokens'] + and x['model_reference'] == row['model_reference'] + and x['prompt_sha256'] == row['prompt_sha256'] + and x['weights_sha256'] == row['weights_sha256'] + and x['weights_sha256'] != 'unknown weights' + and x['prompt_sha256'] != 'unknown prompt' and len(x['_text_hashes']) == 1] + if refs and len(row['_text_hashes']) == 1: + matches = sum(x['_text_hashes'] == row['_text_hashes'] for x in refs) + row['same_output'] = f'text hash matches {matches}/{len(refs)} Linux MLX rows' + else: + row['same_output'] = 'no comparable Linux text hash' + + def chart(rows): - """Small multiples, no plotting deps: per (model, length, load) block, one row per engine and three - panels on their own scales - first token (lower is better), streaming speed (higher is better) and the - wait for a typical reply. Two measures never share an axis. Blue is the best value in its panel; - ★ by a name is the block's shortest typical reply among rows that matched and passed the gate.""" + """Dot plots grouped by model, output length and load; separate scales per metric.""" groups = {} - for x in sorted(rows, key=lambda x: (x['_model'], x['tokens'], x['_conc'], reply_s(x) or 1e9)): - groups.setdefault((x['model'], x['tokens'], x['_conc']), []).append(x) - label_w, panel_w, gap, row_h = 250, 190, 34, 22 - w = label_w + 3 * (panel_w + gap) - panels = [('First token, s · lower is better', lambda x: x['ttft_s'], '{:.2f} s'), - ('Streaming, tok/s · higher is better', lambda x: x['decode_tok_s'], '{:.1f}'), - (f'Typical reply ({REPLY_TOKENS} tokens), s · lower', reply_s, '{:.1f} s')] - y = 24; parts = [] - for (model, tokens, conc), xs in groups.items(): - head = f'{model} · {tokens}-token test' + (f' · {conc} requests at once (streaming = total)' if conc > 1 else ' · one request') - parts.append(f'{head}'); y += 8 + for x in sorted(rows, key=lambda x: (x['_model'], x['tokens'], x['_conc'], x['chip'], x['os'], x['_arm'], x['driver'])): + groups.setdefault((x['model'], x['tokens'], x['_conc'], x.get('model_reference', 'unknown revision'), x.get('prompt_sha256', 'unknown prompt'), x.get('weights_sha256', 'unknown weights')), []).append(x) + label_w, panel_w, gap, row_h = 340, 205, 24, 22 + w = label_w + 3 * (panel_w + gap) + 20 + panels = [('First token (s): lower', lambda x: x['ttft_s'], '{:.2f}'), + ('Decode (tok/s): higher', lambda x: x['decode_tok_s'], '{:.1f}'), + (f'{REPLY_TOKENS}-token estimate (s): lower', reply_s, '{:.1f}')] + y = 26 + parts = ['macOSAurora Linux — labels identify chip and engine; driver in tooltip'] + for (model, tokens, conc, model_ref, prompt_hash, weights_hash), xs in groups.items(): + revision = model_ref.rsplit('@', 1)[-1][:8] if '@' in model_ref else 'revision unknown' + head = f'{model} · rev {revision} · weights {weights_hash[:8]} · prompt {prompt_hash[:8]} · {tokens}-token limit' + (f' · {conc} requests (aggregate decode)' if conc > 1 else ' · one request') + parts.append(f'{escape(head)}'); y += 22 for i, (title, _, _) in enumerate(panels): - parts.append(f'{title}') - y += 18 - tops, bests = [], [] + parts.append(f'{escape(title)}') + y += 21 + tops = [] for _, f, _ in panels: - vals = sorted(v for v in (f(x) for x in xs) if v) - # Cap at 3x the median so one very slow engine does not flatten the rest; clipped bars end in '›'. - tops.append(min(vals[-1], 3 * vals[len(vals) // 2]) if vals else 1) - bests.append(None) - for i, (title, f, _) in enumerate(panels): - vals = [f(x) for x in xs if f(x)] - if vals: - bests[i] = max(vals) if 'higher' in title else min(vals) + vals = [v for v in (f(x) for x in xs) if v is not None and v > 0] + tops.append(max(vals) if vals else 1) base_y = y for x in xs: - arm = x['_arm'] - name = SHORT.get(arm, arm) + (' · old driver' if '7faf' in x['driver'] else '') - weight = ' class="l s"' if arm == 'serve-patched-120k' else ' class="l"' - parts.append(f'{name}{" ★" if x.get("best") else ""}') + is_mac = x['_arm'] == 'mac-mlx' + os_short = x['os'].split(' build')[0] if is_mac else ('Aurora' if 'Aurora' in x['os'] else x['os']) + if not is_mac and 'wip' in x['kernel']: + os_short += ' wip' + x['kernel'].split('wip')[-1].split('-')[0] + engine = 'Apple MLX in-process' if is_mac else SHORT.get(x['_arm'], x['_arm']) + if not is_mac and '7faf' in x['driver']: + engine += ' · old driver' + label = f"{x['chip']} · {os_short} · {engine}" + parts.append(f'{escape(label)}') for i, (title, f, fmt_) in enumerate(panels): v = f(x); x0 = label_w + i * (panel_w + gap) if v is None: parts.append(f'n/a'); continue - clipped = v > tops[i] - bw = max(2, (panel_w - 52) * min(v, tops[i]) / tops[i]) - cls = 'best' if v == bests[i] else 'bar' - tip = f'{name}: {title.split(" ·")[0]} {fmt_.format(v)}' - parts.append(f'{tip}' - f'{"› " if clipped else ""}{fmt_.format(v)}') + dx = (panel_w - 52) * max(0, v) / tops[i] + cls = 'mac' if is_mac else 'linux' + tip = escape(f"{label}; {x['kernel']}; {x['driver']}; {x['settings']}; {title}: {fmt_.format(v)}; {x['runs']} runs") + parts.append(f'{tip}' + f'{fmt_.format(v)}') y += row_h for i in range(3): x0 = label_w + i * (panel_w + gap) parts.append(f'') - y += 22 - style = ('') + y += 20 + style = ('') return (f'' - f'First token, streaming speed and reply time by engine on Apple M1 Pro' - f'{style}{"".join(parts)}\n') + f'Local AI model timings by chip, operating system and engine{style}{"".join(parts)}\n') + +def relative_rows(mac, linux): + """Ratios of median in-process measurements only; no server or unlabelled chip pairing.""" + out = [] + for m in mac: + if m['runs'] < 3 or m['chip'] in ('Unknown chip', ''): + continue + for l in linux: + if (l['_arm'] not in ('mlx-5k', 'mlx-120k') or l['_conc'] != 1 or l['runs'] < 3 + or l['chip'] != m['chip'] or 'Aurora' not in l['os'] or m['_conc'] != 1 + or l['_model'] != m['_model'] or l['tokens'] != m['tokens'] + or l['model_reference'] != m['model_reference'] + or l['weights_sha256'] != m['weights_sha256'] or l['weights_sha256'] == 'unknown weights' + or l['prompt_sha256'] != m['prompt_sha256'] or l['prompt_sha256'] == 'unknown prompt' + or not all(isinstance(v, (int, float)) and math.isfinite(v) and v > 0 + for v in (l['decode_tok_s'], m['decode_tok_s'], l['ttft_s'], m['ttft_s']))): + continue + if len(l['_text_hashes']) == len(m['_text_hashes']) == 1: + output = 'match' if l['_text_hashes'] == m['_text_hashes'] else 'different' + else: + output = 'unknown' + out.append({'model': m['model'], 'tokens': m['tokens'], 'chip': m['chip'], + 'mac_os': m['os'], 'mac_kernel': m['kernel'], 'mac_settings': m['settings'], + 'linux_os': l['os'], 'kernel': l['kernel'], 'driver': l['driver'], + 'settings': l['settings'], 'arm': l['_arm'], 'mac_source': m['source'], + 'linux_source': l['source'], 'mac_runs': m['runs'], 'linux_runs': l['runs'], + 'decode_ratio': m['decode_tok_s'] / l['decode_tok_s'], + 'ttft_ratio': l['ttft_s'] / m['ttft_s'], 'text': output}) + return sorted(out, key=lambda r: (r['model'], r['tokens'], r['chip'], r['kernel'], r['driver'], r['arm'])) + + +def relative_chart(rows): + """Relative times, with 1× representing the matched macOS result.""" + width, label_w, panel_w, gap, row_h = 990, 455, 230, 25, 25 + height = 66 + len(rows) * row_h + top_decode = max([1.0] + [r['decode_ratio'] for r in rows]) * 1.1 + top_ttft = max([1.0] + [r['ttft_ratio'] for r in rows]) * 1.1 + parts = ['Relative time: macOS = 1× · same chip/model/prompt · in-process MLX · ratios of separate run medians', + f'Decode time (× Mac) · lower is better', + f'First token time (× Mac) · lower is better'] + for i, maximum in enumerate((top_decode, top_ttft)): + x0 = label_w + i * (panel_w + gap) + ref = x0 + (panel_w - 52) / maximum + parts.append(f'1×') + for i, r in enumerate(rows): + y = 80 + i * row_h + model = {'Qwen3.5-2B MLX 4-bit': 'Qwen3.5 2B', 'Qwen3.5-4B MLX 4-bit': 'Qwen3.5 4B', + 'LFM3-1.7B MLX 8-bit': 'Liquid LFM3 1.7B', 'Granite 4.0 1B MLX 4-bit (IBM, dense)': 'Granite 1B'}.get(r['model'], r['model']) + kernel = ('wip' + r['kernel'].split('wip')[-1].split('-')[0]) if 'wip' in r['kernel'] else r['kernel'] + driver = r['driver'].split()[-1][:7] + budget = '120k' if r['arm'] == 'mlx-120k' else '5k' + label = f"{r['chip']} · {model} · {r['tokens']} · {kernel}/{driver} · {budget} · text {r['text']}" + parts.append(f'{escape(label)}') + for j, (value, maximum) in enumerate(((r['decode_ratio'], top_decode), (r['ttft_ratio'], top_ttft))): + x0 = label_w + j * (panel_w + gap); px = x0 + (panel_w - 52) * value / maximum + tip = escape(f"{r['model']}; {r['mac_os']}, {r['mac_kernel']}, {r['mac_settings']}; {r['linux_os']}, {r['kernel']}, {r['driver']}, {r['settings']}; text {r['text']}; ratio {value:.2f}×") + parts.append(f'{tip}{value:.2f}×') + if not rows: + parts.append('No matching platform pairs have been measured.') + height = 100 + style = ('') + return f'Relative Aurora and macOS in-process MLX timings{style}{"".join(parts)}\n' +def batch_counter_rows(): + """Historical guarded Qwen2B paired experiments; counters include prompt and generation.""" + out = [] + for f in sorted(RECEIPTS.glob('2026-10-27-batch-work*.json')): + r = json.loads(f.read_text()) + runs = [x for x in r.get('runs', []) if x.get('exit') == 0 and not x.get('stopped') + and x.get('result', {}).get('counter_delta') and x.get('result', {}).get('tokens')] + if not runs: + continue + sequences = {tuple(x['result']['tokens']) for x in runs} + for budget in sorted({x['budget'] for x in runs}): + subset = [x for x in runs if x['budget'] == budget] + def counter(name): + return med([x['result']['counter_delta'].get(name) for x in subset]) + samples = [sample for x in subset for sample in x.get('samples', [])] + available = [x['available_bytes'] / (1024 ** 3) for x in samples if x.get('available_bytes') is not None] + temperatures = [x['max_temp_c'] for x in samples if x.get('max_temp_c') is not None] + fans = [x['max_fan_rpm'] for x in samples if x.get('max_fan_rpm') is not None] + out.append({'source': str(f.relative_to(HERE)), 'status': r.get('status', 'unknown'), + 'budget': budget, 'tokens': len(subset[0]['result']['tokens']), 'runs': len(subset), + 'submissions': counter('vk_submissions'), 'dispatches': counter('vk_compute_dispatches'), + 'barriers': counter('barriers_emitted'), + 'decode': med([x['result'].get('generation_tps') for x in subset]), + 'min_available_gib': min(available) if available else None, + 'max_temp_c': max(temperatures) if temperatures else None, + 'max_fan_rpm': max(fans) if fans else None, + 'ids': ('only one run' if len(runs) == 1 else + 'identical within receipt' if len(sequences) == 1 else 'differ within receipt')}) + return out + def main(): rows = measured_rows() + mac = mac_rows() + annotate_cross_platform(mac, rows) + relative = relative_rows(mac, rows) + counters = batch_counter_rows() notes = json.loads((HERE / 'scoreboard-notes.json').read_text())['rows'] with (HERE / 'engine-scoreboard.csv').open('w', newline='') as fh: - w = csv.DictWriter(fh, fieldnames=FIELDS + ['blocker'], extrasaction='ignore') + w = csv.DictWriter(fh, fieldnames=FIELDS + ['blocker'], extrasaction='ignore', lineterminator='\n') w.writeheader() for x in rows: w.writerow({k: (fmt(v) if isinstance(v, float) else v) for k, v in x.items()}) + for x in mac: + w.writerow({k: (fmt(v) if isinstance(v, float) else v) for k, v in x.items()}) for n in notes: w.writerow({**n, 'best': ''}) md = ['# Local AI engine scoreboard', '', - 'One table for every engine we run or watch on the M1 Pro. ★ marks the most functional setup per model, length and load: the shortest typical reply ' + 'Chip and OS labels distinguish measurements. The Linux table’s ★ marks the most functional setup per model, length and load: the shortest typical reply ' '(first token plus 200 tokens) whose output matched every other engine and whose correctness gate did not fail. ' 'Regenerate with `python3 mseries/local-ai/scoreboard.py` after adding a receipt. Tracked in the ' '"Local AI engine scoreboard" issue.', '', - '![First token, streaming speed and typical reply time by engine](engine-scoreboard.svg)', '', - f'Each block is one model and load; each panel has its own scale. **First token** is the wait before text ' - f'appears, **streaming** is decode speed after that, and **typical reply** is first token plus {REPLY_TOKENS} ' - 'tokens at that speed (for several requests at once, one request\'s share). Blue is the best value in each ' - 'panel; a bar ending in › is clipped at three times the block\'s median.', '', - '## Measured on this machine', '', - '| ★ | Model | Engine | Settings | Tokens | Decode tok/s | TTFT s | Total s | Runs | Same output as MLX default | Gate | Kernel | Driver | Date | Receipt |', - '|---|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---|'] + '![Dot plots of first token, decode speed and reply time by chip, operating system and engine](engine-scoreboard.svg)', '', + 'Each dot is a measured engine configuration. Chip, OS and driver are identified; macOS and Aurora have different runtime versions and conditions. ' + 'The plots group the same model revision, weight hash, prompt hash and token limit, but do not establish token or numerical equivalence across systems. ' + 'First token and estimated reply time are lower-is-better; decode is higher-is-better. The 200-token value extrapolates from measured decode speed. HTTP and in-process timing boundaries differ. ' + 'No dot implies a winner across systems. The Linux-only star and gates below are unchanged.', '', + '## Relative in-process performance', '', + '![Dot plot of Aurora decode and first-token time relative to macOS](relative-scoreboard.svg)', '', + 'Ratios are separate-run medians on the same named chip, model revision, weight hash, prompt hash and token limit. The same chip name does not guarantee the same GPU configuration. ' + '1× means the macOS reference; higher values mean more Aurora time. Decode time uses macOS decode tok/s divided by Aurora decode tok/s; ' + 'first-token time uses Aurora seconds divided by macOS seconds. Only in-process MLX runs with at least three trials appear. ' + 'The runs were not paired across OS boots, and runtimes, load and thermal state differ; these ratios do not identify a cause or prove a regression. ' + 'Text-match flags are limited to these prompts and do not establish numerical fidelity.', '', + '| Chip | Model | Token limit | Mac OS / runtime | Aurora OS / kernel / driver | Aurora budget | Decode time vs Mac | First token vs Mac | Text | Runs Mac / Aurora | Receipts |', + '|---|---|---:|---|---|---|---:|---:|---|---|---|'] + for r in relative: + md.append(f"| {r['chip']} | {r['model']} | {r['tokens']} | {r['mac_os']} / {r['mac_settings']} | {r['linux_os']} / `{r['kernel']}` / {r['driver']} | {r['settings']} | {r['decode_ratio']:.2f}× | {r['ttft_ratio']:.2f}× | {r['text']} | {r['mac_runs']} / {r['linux_runs']} | [Mac]({r['mac_source']}) · [Aurora]({r['linux_source']}) |") + md += ['', '## GPU submission counter evidence', '', + 'These are separate historical M1 Pro Aurora Qwen3.5 2B work-budget experiments, not counters from the macOS or engine-comparison rows. ' + 'Medians are per measured request, including prompt prefill and generation, not per token. The 80k long experiment was interrupted after two pairs. ' + 'The counters describe activity and do not measure individual kernel time or prove a cause; the experiment also checked GTK callbacks, not display presentation or input latency. ' + 'See the [method and limitations](receipts/2026-10-27-batch-work.md).', '', + '| Receipt | Status | Generated tokens | Work budget | Runs | Vulkan submissions | Compute dispatches | Barriers emitted | Decode tok/s | Min free GiB | Max °C | Max fan RPM | Token IDs |', + '|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|'] + for c in counters: + md.append(f"| [{Path(c['source']).name}]({c['source']}) | {c['status']} | {c['tokens']} | {c['budget']} | {c['runs']} | {fmt(c['submissions'], 0)} | {fmt(c['dispatches'], 0)} | {fmt(c['barriers'], 0)} | {fmt(c['decode'])} | {fmt(c['min_available_gib'])} | {fmt(c['max_temp_c'])} | {fmt(c['max_fan_rpm'], 0)} | {c['ids']} |") + md += ['', 'Apple Metal traces captured shader pipeline names but no shader timing or GPU counter set; ' + 'the available evidence cannot rank individual kernels or attribute the cross-OS gap. ' + 'Fresh paired profiles, matched correctness checks, sustained thermals and desktop responsiveness remain next steps.', '', + '## Aurora Linux measurements', '', + '| ★ | Chip | OS | Model | Engine | Settings | Token limit | Decode tok/s | TTFT s | Total s | Runs | Same output as reference in receipt | Gate | Kernel | Driver | Date | Receipt |', + '|---|---|---|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---|'] for x in sorted(rows, key=lambda x: (x['_model'], x['tokens'], -(x['decode_tok_s'] or 0))): b = '**' if x.get('best') else '' - md.append(f"| {x.get('best', '')} | {x['model']} | {b}{x['engine']}{b} | `{x['settings']}` | {x['tokens']} | " + md.append(f"| {x.get('best', '')} | {x['chip']} | {x['os']} | {x['model']} | {b}{x['engine']}{b} | `{x['settings']}` | {x['tokens']} | " f"{b}{fmt(x['decode_tok_s'])}{b} | {fmt(x['ttft_s'])} | {fmt(x['total_s'])} | {x['runs']} | {x['same_output']} | " f"{x['gate']} | `{x['kernel']}` | {x['driver']} | {x['date']} | [{Path(x['source']).name}]({x['source'].replace('receipts/', 'receipts/')}) |") - md += ['', '## Watched or reported elsewhere (not comparable to the table above)', '', + md += ['', '## Native macOS measurements', '', + '| Chip | OS | Model | Token limit | Decode tok/s | TTFT s | Total s | Runs | Linux MLX text | Start (local) | End (local) | Model SHA-1 | Kernel | Receipt |', + '|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|---|'] + for x in sorted(mac, key=lambda x: (x['_model'], x['tokens'], x['chip'], x['os'])): + md.append(f"| {x['chip']} | {x['os']} | {x['model']} | {x['tokens']} | {fmt(x['decode_tok_s'])} | {fmt(x['ttft_s'])} | {fmt(x['total_s'])} | {x['runs']} | {x['same_output']} | {x['start_time']} | {x['end_time']} | `{x['weights_sha1']}` | `{x['kernel']}` | [{Path(x['source']).name}]({x['source']}) |") + md += ['', 'These rows are Apple MLX in-process measurements. Text-hash matches cover these prompts only and do not establish token identity or numerical fidelity. ' + 'LFM3 used the project-pinned injected implementation; its tokenizer emitted a custom-code prompt and continued without approval, ' + 'and Transformers reported a model-type compatibility warning. See its receipts. M4 appears only when measured receipts identify an M4 chip and OS.', '', + '## Watched or reported elsewhere (not comparable to the table above)', '', '| Model | Engine | Settings | Decode | Status | Blocker | Source |', '|---|---|---|---|---|---|---|'] for n in notes: md.append(f"| {n['model']} | {n['engine']} | {n['settings']} | {n['decode_tok_s']} | {n['status']} | {n['blocker']} | {n['source']} |") @@ -229,9 +424,7 @@ def main(): 'CSV, SVG and this page. Rows are replaced by the newest receipt; old receipts stay for history.', '5. **Adopt.** A setting becomes our default only when it holds ★ *and* passes the MLX gate; ' 'change defaults in a separate PR.', - '6. **Version steps follow upstream.** We move mlx-lm (now 0.31.3) and the omarchy-mlx pin (now v0.7.33, ' - '`33630f9c`) only when omarchy-mlx moves its own `install.sh` default or cuts a release. Its 0.32 line ' - '(mlx-lm `94cdcae`, used by its oMLX packaging) is tested as a separate engine, not adopted as our default. `install-omlx.sh` installs oMLX that way (its own venv under `~/.local/share/omlx-linux`) for engine-compare\'s `omlx-120k` arm.', '', + '6. **Version steps follow upstream.** Check the current installer pins and deployment receipt; qualify any newer runtime in an isolated environment before changing the default.', '', 'Rules: everything is documented in our own repo only; we do not post to upstream projects. ' 'Rows under "reported elsewhere" are other people\'s numbers on other hardware and are never ranked against ours.', '', '## Reproduce', '', @@ -251,9 +444,10 @@ def main(): './tensorfold-vulkan.sh build && ./tensorfold-vulkan.sh gpu', 'python3 scoreboard.py', '```', ''] - (HERE / 'engine-scoreboard.svg').write_text(chart(rows)) + (HERE / 'engine-scoreboard.svg').write_text(chart(rows + mac)) + (HERE / 'relative-scoreboard.svg').write_text(relative_chart(relative)) (HERE / 'ENGINE-SCOREBOARD.md').write_text('\n'.join(md)) - print(f'{len(rows)} measured rows, {len(notes)} noted rows') + print(f'{len(rows)} Linux rows, {len(mac)} macOS rows, {len(notes)} noted rows') if __name__ == '__main__': diff --git a/mseries/local-ai/test_scoreboard_platforms.py b/mseries/local-ai/test_scoreboard_platforms.py new file mode 100644 index 00000000..1ff24297 --- /dev/null +++ b/mseries/local-ai/test_scoreboard_platforms.py @@ -0,0 +1,111 @@ +"""Regression checks for provenance-preserving platform scoreboard ingestion.""" +import json +import tempfile +import unittest +import xml.etree.ElementTree as ET +from pathlib import Path +from unittest.mock import patch + +import scoreboard + + +class PlatformScoreboardTest(unittest.TestCase): + def test_macs_do_not_overwrite_other_chips_or_revisions(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + base = {'status': 'complete', 'concurrency': 1, 'model_key': 'qwen2b', + 'model': 'example/model@revision1', 'max_tokens': 32, + 'hardware': 'Apple M1 Pro', 'os': 'macOS 27.2', 'kernel': 'Darwin', + 'runs': [{'arm': 'mac-mlx', 'decode_tps': n, 'ttft_s': 0.2, 'total_s': 0.6} for n in (10, 20, 30)]} + for index, overrides in enumerate([{}, {'hardware': 'Apple M4'}, {'model': 'example/model@revision2'}, {'mlx_version': 'new'}, {'status': 'partial'}]): + (root / f'2026-10-30-macos-scoreboard-{index}.json').write_text(json.dumps({**base, **overrides})) + with patch.object(scoreboard, 'RECEIPTS', root): + rows = scoreboard.mac_rows() + self.assertEqual(len(rows), 4) + self.assertEqual({x['model_reference'] for x in rows}, {'example/model@revision1', 'example/model@revision2'}) + self.assertIn('M4', {x['chip'] for x in rows}) + self.assertTrue(all(x['decode_tok_s'] == 20 for x in rows)) + self.assertTrue(all(x['gate'] == 'not gated' for x in rows)) + + def test_linux_chip_and_kernel_are_not_overwritten(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + receipt = {'status': 'complete', 'model_key': 'qwen2b', 'model': 'example/model@abc', + 'max_tokens': 32, 'concurrency': 1, 'driver': 'test driver', 'os': 'Aurora Linux', + 'arms': ['mlx-120k'], 'runs': [{'arm': 'mlx-120k', 'text': 'synthetic', + 'text_sha256': 'x', 'decode_tps': 42, 'ttft_s': .2, 'total_s': 1}]} + for index, chip in enumerate(['M1 Pro', 'M4']): + (root / f'2026-11-01-engine-compare-test-{index}.json').write_text(json.dumps({**receipt, 'chip': chip, 'kernel': 'Aurora test'})) + with patch.object(scoreboard, 'RECEIPTS', root), patch.object(scoreboard, 'HERE', root), patch.object(scoreboard, 'gate_results', return_value={}): + rows = scoreboard.measured_rows() + self.assertEqual({row['chip'] for row in rows}, {'M1 Pro', 'M4'}) + self.assertTrue(all(row['os'] == 'Aurora Linux' for row in rows)) + + def test_text_match_requires_the_same_prompt_and_model(self): + mac = {'_model': 'qwen2b', 'tokens': 32, 'model_reference': 'example@abc', + 'prompt_sha256': 'prompt', 'weights_sha256': 'weights', '_text_hashes': {'same'}} + linux = {'_model': 'qwen2b', 'tokens': 32, 'model_reference': 'example@abc', + 'prompt_sha256': 'prompt', 'weights_sha256': 'weights', '_text_hashes': {'same'}, '_arm': 'mlx-120k', '_conc': 1} + scoreboard.annotate_cross_platform([mac], [linux, {**linux, 'prompt_sha256': 'other'}, + {**linux, 'model_reference': 'example@other'}]) + self.assertEqual(mac['same_output'], 'text hash matches 1/1 Linux MLX rows') + scoreboard.annotate_cross_platform([mac], [{**linux, '_text_hashes': {'different'}}]) + self.assertEqual(mac['same_output'], 'text hash matches 0/1 Linux MLX rows') + scoreboard.annotate_cross_platform([mac], [{**linux, 'prompt_sha256': 'other'}]) + self.assertEqual(mac['same_output'], 'no comparable Linux text hash') + scoreboard.annotate_cross_platform([mac], [{**linux, 'weights_sha256': 'different weights'}]) + self.assertEqual(mac['same_output'], 'no comparable Linux text hash') + + def test_chart_separates_model_revisions_and_escapes_labels(self): + row = {'_model': 'qwen2b', '_arm': 'mac-mlx', '_conc': 1, 'model': 'model & test', + 'model_reference': 'example@abcd1234', 'tokens': 32, 'chip': 'M1 Pro', 'os': 'macOS', + 'ttft_s': 0.2, 'decode_tok_s': 25, 'driver': 'Apple Metal', 'settings': 'test', 'runs': 3, 'kernel': 'Darwin'} + output = scoreboard.chart([row, {**row, 'chip': 'M4', 'model_reference': 'example@different'}]) + ET.fromstring(output) + self.assertIn('model & test', output) + self.assertIn('abcd1234', output) + self.assertIn('differen', output) + self.assertIn('M4 · macOS', output) + + def test_relative_ratios_require_matching_workload_and_valid_measurements(self): + base = {'_model': 'qwen2b', 'model': 'Qwen', 'tokens': 32, 'chip': 'M1 Pro', + 'model_reference': 'model@rev', 'weights_sha256': 'weights', 'prompt_sha256': 'prompt', + '_text_hashes': {'same'}, '_conc': 1, 'runs': 3, 'decode_tok_s': 100, + 'ttft_s': 1, 'source': 'receipt', 'kernel': 'kernel', 'driver': 'driver', + 'settings': 'settings', 'os': 'macOS'} + linux = {**base, '_arm': 'mlx-120k', 'os': 'Aurora Linux', 'decode_tok_s': 50, 'ttft_s': 3} + result = scoreboard.relative_rows([base], [linux]) + self.assertEqual((result[0]['decode_ratio'], result[0]['ttft_ratio'], result[0]['text']), (2, 3, 'match')) + self.assertEqual(scoreboard.relative_rows([base], [{**linux, '_text_hashes': {'other'}}])[0]['text'], 'different') + for change in ({'chip': 'M4'}, {'model_reference': 'model@other'}, {'weights_sha256': 'other'}, + {'prompt_sha256': 'other'}, {'runs': 2}, {'_arm': 'tensorfold'}, + {'decode_tok_s': float('nan')}, {'ttft_s': -1}, {'_conc': 2}): + with self.subTest(change=change): + self.assertEqual(scoreboard.relative_rows([base], [{**linux, **change}]), []) + ET.fromstring(scoreboard.relative_chart(result)) + + def test_batch_counters_exclude_stopped_runs_and_preserve_partial_status(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + run = {'exit': 0, 'budget': 5000, 'result': {'tokens': [1, 2], 'generation_tps': 10, + 'counter_delta': {'vk_submissions': 100, 'vk_compute_dispatches': 200, 'barriers_emitted': 300}}, + 'samples': []} + receipt = {'status': 'interrupted', 'runs': [run, {**run, 'stopped': True}]} + (root / '2026-10-27-batch-work-test.json').write_text(json.dumps(receipt)) + with patch.object(scoreboard, 'RECEIPTS', root), patch.object(scoreboard, 'HERE', root): + rows = scoreboard.batch_counter_rows() + self.assertEqual(len(rows), 1) + self.assertEqual((rows[0]['status'], rows[0]['runs'], rows[0]['submissions']), ('interrupted', 1, 100)) + self.assertEqual(rows[0]['ids'], 'only one run') + self.assertIsNone(rows[0]['max_temp_c']) + + def test_installed_tensorfold_receipt_is_preserved(self): + rows = scoreboard.measured_rows() + matches = [r for r in rows if r['_arm'] == 'tensorfold' and 'b3ab9ba' in r['engine']] + self.assertEqual({r['tokens'] for r in matches}, {32, 256}) + self.assertTrue(any('diverges after 880' in r['same_output'] for r in matches)) + self.assertTrue(all('/2026-10-30-tensorfold-b3ab9ba/' in r['source'] for r in matches)) + + +if __name__ == '__main__': + unittest.main()