Skip to content
This repository was archived by the owner on Nov 12, 2026. It is now read-only.

Mac mini M4 macOS (Metal) baseline for the engine scoreboard - #709

Closed
techmore wants to merge 1 commit into
devfrom
m4-mini-macos-baseline
Closed

techmore wants to merge 1 commit into
devfrom
m4-mini-macos-baseline

Conversation

@techmore

Copy link
Copy Markdown
Owner

What

The Apple-native half of a same-machine comparison: every model × length × load from the M1 Pro scoreboard, run under macOS 27.0 (26A5421a) on the Mac mini M4 (Mac16,10, 10-core GPU, 16 GB) with the repo's own engine-compare.py, so the same commands can later run under Linux on this mini and compare row for row.

  • engine-compare.py: macOS arms mlx-metal (in-process, same worker), lm-metal (stock mlx_lm.server) and tensorfold-metal (TensorFold v1.1.9 release, pinned sha256, --backend mlx --no-drafts). Models come from the HF cache at the pinned revisions (weights hash-checked as before). New env knobs also let the Linux side on the mini use HF snapshots and the packaged venv: MSERIES_HF_SNAPSHOTS, MSERIES_MLX_PY, MSERIES_TF_VULKAN_BIN, MSERIES_LFM3_MODULE, plus --reserve-gib.
  • machine.py: macOS memory (kern.memorystatus_level) and thermal (pmset -g therm) guard; receipts record load, top CPU users, limactl list, power and swap at start and end.
  • scoreboard.py: a separate M4 section, engine-scoreboard-m4-mini-macos.csv / .svg, never ranked against the M1 Pro, with a "Same output as M1 Pro (Linux)" column and a "How to run the same on Linux on the M4 mini" note (points at omarchy-mlx m4-mini-bench / omarchy-mlx-m4-bench). The M1 Pro table regenerates byte-for-byte unchanged.
  • Receipts: receipts/2026-10-31-m4-mini-macos/ (8 runs, the three macOS legs of m4_mini_bench.py, venv freeze, README).

Pins: Python 3.14.3, mlx 0.32.3, mlx-lm 0.31.3, transformers 5.16.1 (the pins m4-mini-bench names for macOS).

Results (median of 3, 32 tokens unless noted)

Model In-process MLX (Metal) mlx_lm.server TensorFold Metal
Granite 4.0 1B 4-bit 58.6 tok/s, TTFT 0.22 s 43.0, 0.22 s –
LFM3-1.7B 8-bit 31.5, 0.24 s – –
Qwen3.5-2B 4-bit 47.7, 0.38 s 42.4, 0.42 s 39.7, 0.07 s
Qwen3.5-2B, 256 tokens 51.3, 0.52 s 40.6, 0.55 s 36.4, 0.12 s
Qwen3.5-2B, 4 / 8 at once (total) – 41.6 / failed 36.8 / 35.4
Qwen3.5-4B 4-bit (4 at once) 29.0, 0.39 s 25.2 (29.2) –

Outputs match the M1 Pro's Linux text except Qwen3.5-2B at 256 tokens (MLX/server on Metal diverge after 1,207 of 1,301 chars; TensorFold matches) and Qwen3.5-4B in-process (after 55 chars; the server matches). mlx_lm.server at 8 simultaneous requests had 1–2 connections reset each round (listen backlog of 5 on macOS); reported under the table.

Caveat: the Lima kbuild CI VM was busy throughout (390–590 % CPU, load average 15–23, 10 of 16 GB held). It was not touched; these are a loaded-machine floor. Compare against a Linux run under similar load, or re-run both quiet.

🤖 Generated with Claude Code

engine-compare.py gains three macOS arms (mlx-metal in-process, stock
mlx_lm.server, TensorFold v1.1.9 Metal release) that run the same models,
revisions, prompts, token counts and loads as the M1 Pro table, with models
from the Hugging Face cache at the pinned revisions. machine.py reads memory
and thermal state on macOS and records host conditions (load, the busy kbuild
VM, power) in each receipt. scoreboard.py writes a separate M4 section, CSV
and chart, never ranked against the M1 Pro, with a column comparing each
output to the M1 Pro's Linux text, and a note on running the same on Linux
on the mini (MSERIES_HF_SNAPSHOTS, MSERIES_MLX_PY, MSERIES_TF_VULKAN_BIN).

Receipts: receipts/2026-10-31-m4-mini-macos/ (8 engine-compare runs, the
three macOS legs of omarchy-mlx's m4_mini_bench.py, venv freeze, README).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@techmore

Copy link
Copy Markdown
Owner Author

Superseded by the archived ElementaryOS experiment and the active Aurora M4 repositories.

@techmore techmore closed this Nov 12, 2026
@techmore
techmore deleted the m4-mini-macos-baseline branch November 12, 2026 19:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant