Document macOS MLX profiles and TensorFold benchmark scope

mseries/OS9-HANDOFF.md

@@ -1,3 +1,11 @@
+## macOS MLX profile scope correction — 2026-10-30
+
+The engine scoreboard now displays chip/OS-labelled dot plots and includes matching M1 Pro macOS MLX Qwen3.5 2B and 4B, Liquid LFM3 and dense Granite 1B short/long receipts, including SHA-1/SHA-256, kernel and local run timestamps. The installed TensorFold b3ab9ba short and long receipts are now represented as their own Linux rows; its long divergence is preserved. M4 rows remain absent until measured. Short Qwen2B, Liquid and Granite matched available Linux MLX text; long Qwen2B and short Qwen4B differed. Apple/Linux timing boundaries and numerical fidelity remain explicit. [Scoreboard](local-ai/ENGINE-SCOREBOARD.md).
+
+TensorFold batching update: public [PR #1561](https://github.com/ashhart/TensorFold/pull/1561) is closed and unmerged; its exact one-commit diff failed a clean apply check against installed `b3ab9ba` in two files. Earlier "unpublished" wording is historical. No patch was applied or tested; see [source audit](local-ai/receipts/2026-10-30-macos-metal-profile/tensorfold-batching-audit.md).
+
+Captured Apple Metal profiles on M1 Pro for Qwen3.5 4B, Qwen3 4B and Qwen3.5 9B at 512/4096 synthetic prompt tokens. These are exploratory Apple MLX results, not the TensorFold Vulkan 2B benchmark; no matching Linux or TensorFold run occurred. The current engine scoreboard also includes Liquid LFM3 1.7B and dense Granite 4.0 1B. Exact pins, traces, logs, context and proposed matched Mac work are in [the receipt](local-ai/receipts/2026-10-30-macos-metal-profile/README.md). Raw trace bundles remain local because they contain model buffers or system metadata. No project runtime or default changed.
+
 ## Warren local AI and eGPU updates — 2026-10-30
 
 Installed `mseries-nvidia-open-source` `595.112.07+apple.1-1` (SHA-256 `a397b68a...b209ab23`) containing the three prepared NVIDIA fixes and license at `/usr/src/mseries-nvidia-open-595.112.07`. This is source only; no NVIDIA module or matching CUDA userspace is installed. The [source package script](nvidia-egpu/package-source.sh) verifies the exact prepared diff before packaging.

mseries/ROADMAP.md

@@ -1,3 +1,9 @@
+## macOS MLX reference and TensorFold comparison — 2026-10-30
+
+The scoreboard now shows separate chip/OS dot plots and includes all four completed M1 Pro Mac scoreboard model references. Add M4 data only after real runs and investigate the observed 2B-long and 4B-short cross-platform text differences.
+
+Next: compare the recorded macOS references for the four `engine-compare.py` model pins with matching Linux in-process receipts, preserving the short/long prompt settings and separate in-process versus server timing boundaries. The exploratory 4B/9B Metal captures are not TensorFold or Linux baselines. See [the scope and receipts](local-ai/receipts/2026-10-30-macos-metal-profile/README.md).
+
 ## WD22TB4 and wip4 candidate status — 2026-10-28
 
 Current `dev` is `57b143c4afd78b0a4a0f9905e48e39632d1ecc3c`. Pages deployment

mseries/local-ai/receipts/2026-10-30-macos-metal-profile/README.md

@@ -0,0 +1,42 @@
+# macOS MLX measurements and TensorFold comparison scope — 2026-10-30
+
+## Scope correction
+
+These Mac measurements ran Apple MLX, not the TensorFold engine. They cover Qwen3.5 4B, Qwen3 4B Instruct and Qwen3.5 9B; they do **not** test Qwen3.5 2B, Liquid LFM3, Granite, Linux Vulkan, or TensorFold. The 4B model overlaps the project engine scoreboard, but the Mac runs used random synthetic 512/4096-token prompts and 128-token outputs, not the scoreboard's same-text 32/256-token workloads. They cannot be inserted into the scoreboard as matched comparisons. No Linux runtime, kernel, driver or service was changed.
+
+## The intended project benchmarks
+
+The current [`engine-compare.py`](../../engine-compare.py) has four named local scoreboard model configurations:
+
+| Selector | Pinned model | Revision | Model weights SHA-256 | Current comparison arms |
+|---|---|---|---|---|
+| `qwen2b` | `mlx-community/Qwen3.5-2B-MLX-4bit` | `93760be4f1f69842a46bc13dbdc0f19e291392a3` | `713fe7e5d3c3965f7106b0d0ee17615f7869c23c8d327996df8c1196fbcf07d5` | MLX, servers, Rapid, oMLX, **TensorFold** |
+| `qwen4b` | `mlx-community/Qwen3.5-4B-MLX-4bit` | `32f3e8ecf65426fc3306969496342d504bfa13f3` | `5fb9acd0246866381cf8c5c354c6db1019f6498eec4ccb4f5edcc71ffeacb2db` | MLX, servers, Rapid, oMLX |
+| `lfm3` | `LiquidAI/LFM3-1.7B-MLX-8bit` | `ee7667f3cfccb2a41725f91820531823fcabc6b7` | `2b06cea98d75527c248a0b04383f7cf0a1c72fb4ed7b45096618fc4e59d015ad` | MLX 5k and 120k |
+| `granite1b` | `mlx-community/granite-4.0-1b-4bit` (dense) | `29f108a4e2ff99079a173004871604c2f1e12c1f` | `b7a43ebbc86517b8ecfc4cff265a787ea31620f7eb356984c08abe14646617c7` | MLX, Rapid and selected servers |
+
+The dense Granite selection is intentional; the hybrid Granite 4.0 H 1B is excluded because the GPU multi-token gate found a severe correctness failure. LFM3 used a separately pinned mlx-lm model implementation; check compatibility and pin its implementation when establishing a Mac reference. The project also has separate exploratory Qwen3.8-2B and ANE work; it is not one of these four `engine-compare.py` configurations. See [scoreboard](../../ENGINE-SCOREBOARD.md) and [Qwen3.8 receipt](../2026-10-27-qwen38-mac-rapid.md).
+
+The benchmark harness uses short and long fixed rainbow prompts, greedy decoding, 32 or 256 generated tokens, fresh processes, one warmup and three rotated rounds. MLX in-process and server HTTP have different timing boundaries. The Linux experiments enforce the shared worker lock, memory reserve and temperature guard; use matched measurement boundaries when comparing macOS and Linux.
+
+TensorFold's installed opt-in Vulkan engine is pinned to PR 1544 head `b3ab9ba217a80c63fcc48f7643ddfe339a8f4bdc`, and its current compatible scoreboard arm is Qwen3.5 2B. The [October 30 engine receipt](../2026-10-30-tensorfold-b3ab9ba/README.md) records a 32-token TensorFold loopback median of 2.89 tok/s and installed MLX in-process 54.51 tok/s with different timing boundaries; it does not justify a direct speed ratio. Long free-running text diverged after 880 characters, and strict teacher-forced exact-ID checks failed on near-ties in some cases. Do not claim full fidelity or a submit-batching gain. The [package receipt](../2026-10-30-tensorfold-package/README.md) records the installed opt-in package.
+
+## What we actually measured on this Mac
+
+[Model and shape report](model-shape-comparison.md) records six instrumented one-trial runs and their limits. Earlier three-trial native Apple MLX runs are retained in `*-baseline.log`. Separate synthetic 512/4096 prompt runs have their raw stdout in `*-p512.log` and `*-p4096.log`; the interrupted first Qwen3 4096 trace-save log is retained as `qwen3-4b-p4096-interrupted.log`. All six completed trace targets exited 0. The Mac's model files matched the hashes listed in the shape report; no correctness/fidelity comparison to TensorFold was performed. The first 4B baseline wrapper printed all results but failed when recording its shell exit status; its target exit status was not retained.
+
+The original 4B System Trace contained 4,978 target-process active compute intervals over warmup and the measured trial; overlapping intervals are not a per-kernel dispatch count or elapsed GPU time. The Xcode Metal System Trace preflight also completed a synthetic 1024×1024 MLX matrix multiplication on the M1 Pro. Instruments did not collect shader timing or counters. Its shader pipeline lists are not dispatch counts or evidence that any named kernel dominates. The longer rolling captures include only the end of the workload. The separately captured warmed decode trace includes model buffers, and the system traces may contain private process metadata. The approximately 2.7 GiB trace bundles are retained locally on the Mac, not committed; this repository contains the synthetic logs and interpretation, not the original trace bundles or weights. `decode-capture.log` records the one-step capture result. These artifacts do not prove a Linux bottleneck or support a TensorFold speedup.
+
+A later read-only source check found a public but closed and unmerged [batching PR #1561](https://github.com/ashhart/TensorFold/pull/1561). Its one-commit patch does not apply cleanly to the installed revision; see [the applicability audit](tensorfold-batching-audit.md). No candidate was applied or tested.
+
+## Matched macOS scoreboard runs
+
+The four scoreboard models were subsequently run on the M1 Pro Mac for both 32- and 256-token limits with three fresh-process trials per workload and the project’s prompts, greedy sampling and same-prompt warmup. Eight synthetic receipts in the parent receipts directory record run timestamps, model SHA-1/SHA-256, token IDs, hardware, macOS build, Darwin kernel and MLX versions. See [the scoreboard](../../ENGINE-SCOREBOARD.md) and [cross-platform text-hash evidence](linux-text-comparison.json). All three runs within each Mac group returned identical text and token IDs.
+
+Short Qwen2B, dense Granite and LFM3 text matched available Linux MLX reference text. Long Qwen2B diverged after 633 characters; short Qwen4B diverged after 94. These are text comparisons on these prompts, not proof of token or numerical fidelity, a defect diagnosis, or a TensorFold comparison. LFM3 required the project-pinned injected model implementation; its tokenizer emitted a custom-code prompt and continued without approval, and Transformers reported a model-type warning. No fan/temperature telemetry or Linux-style continuous reserve guard was recorded. The Mac harness only checked disk and macOS memory pressure at admission. No Linux benchmark was run in this session.
+
+The original session harness copies are retained in `harness/` for audit. Their paths and consolidator hardware/version fields are hardcoded to this M1 session: **do not use the consolidator to label M4 or later runs**. A future M4 run needs machine-detected chip, OS, kernel, runtime and hashes; use the same prompt and sampling settings. Model files were removed after the runs.
+
+## Next comparison
+
+Use the four scoreboard model pins above, starting with Qwen3.5 2B for the TensorFold port. Compare the completed macOS MLX results against matching Linux in-process receipts, keeping timing boundaries, output differences and runtime versions visible. Capture the exact tokenizer/chat-template setup and token IDs, runtime versions and timing boundaries. Compare Apple MLX in-process to Linux MLX in-process first; compare HTTP servers only with equivalent HTTP servers. Then evaluate the TensorFold Vulkan 2B arm and preserve its separate numerical and text-fidelity gates. Repeat on M4 only with exact chip, OS and runtime recorded. A fresh Linux benchmark requires booting the M1 Pro into elementary OS; do not claim a fresh paired run from the historical results.

mseries/local-ai/receipts/2026-10-30-macos-metal-profile/decode-capture.log

@@ -0,0 +1,2 @@
+prefill and three generation steps complete
+capture complete

mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/consolidate.py

@@ -0,0 +1,11 @@
+import json,hashlib,glob,os,sys,statistics
+name=sys.argv[1]; label=sys.argv[2]; modelids={'qwen2b':'mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3','qwen4b':'mlx-community/Qwen3.5-4B-MLX-4bit@32f3e8ecf65426fc3306969496342d504bfa13f3','lfm3':'LiquidAI/LFM3-1.7B-MLX-8bit@ee7667f3cfccb2a41725f91820531823fcabc6b7','granite1b':'mlx-community/granite-4.0-1b-4bit@29f108a4e2ff99079a173004871604c2f1e12c1f'}
+root='outputs/mac-scoreboard'; paths=sorted(glob.glob(f'{root}/{name}-{label}-[123].json')); runs=[]
+for i,path in enumerate(paths,1):
+ x=json.load(open(path));x['arm']='mac-mlx';x['round']=i;x['decode_tps']=x['decode_estimate_tps'];x['text_sha256']=hashlib.sha256(x['text'].encode()).hexdigest();x['token_ids_sha256']=hashlib.sha256(json.dumps(x['token_ids']).encode()).hexdigest();runs.append(x)
+def hashfile(alg):
+ p=f'{root}/{name}-{alg}.txt';return open(p).read().split()[0] if os.path.exists(p) else None
+prompt=('Explain in plain language how a rainbow forms. Give four distinct sentences.' if label=='short' else 'Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.')
+out={'prompt':prompt,'prompt_sha256':hashlib.sha256(prompt.encode()).hexdigest(),'schema':'mac-scoreboard-v1','status':'complete' if len(runs)==3 else 'incomplete','model_key':name,'model':modelids[name],'weights_sha1':hashfile('sha1'),'weights_sha256':hashfile('sha256'),'os':'macOS 27.2 build 26B123','kernel':'Darwin 27.2.0 RELEASE_ARM64_T6000','hardware':'MacBookPro18,3 Apple M1 Pro 32 GB','driver':'Apple Metal','mlx_version':'0.31.2','mlx_lm_version':'0.31.3','max_tokens':32 if label=='short' else 256,'arms':['mac-mlx'],'concurrency':1,'method':'Fresh process per run, same prompt warmup, greedy, in-process, 15-second delay; decode=(completion_tokens-1)/(total_s-ttft_s). CPU thread env set to 2, nice 10. No fan/temperature telemetry.','runs':runs}
+if runs:out['summary']={'mac-mlx':{'ok_runs':len(runs),'median_ttft_s':statistics.median(x['ttft_s'] for x in runs),'median_total_s':statistics.median(x['total_s'] for x in runs),'median_decode_tps':statistics.median(x['decode_tps'] for x in runs)}}
+path=f'{root}/{name}-{label}-receipt.json';open(path,'w').write(json.dumps(out,indent=2)+'\n');print(path)

mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/driver.py

@@ -0,0 +1,17 @@
+import os,sys,time,subprocess,json,shutil,re
+name=sys.argv[1];folder=os.path.abspath('work/mac-scoreboard/'+name);target='outputs/mac-scoreboard';lfm=os.path.abspath('work/mac-scoreboard/lfm3.py') if name=='lfm3' else '-'
+prompts=[('short','Explain in plain language how a rainbow forms. Give four distinct sentences.',32),('long','Explain in detail how a rainbow forms, including refraction, reflection, color and viewing angle. Write at least 350 words.',256)]
+for label,prompt,n in prompts:
+ for i in range(1,4):
+  free=shutil.disk_usage('.').free
+  memory=subprocess.run(['memory_pressure'],capture_output=True,text=True).stdout
+  match=re.search(r'System-wide memory free percentage: (\d+)%',memory)
+  avail=int(match.group(1)) if match else None
+  print(f'{name} {label} {i}: disk {free/2**30:.1f} GiB memory free {avail}%',flush=True)
+  if free<6*2**30 or avail is None or avail < 25:
+   print('Stopped before admission: disk or memory headroom below guard',flush=True);sys.exit(3)
+  out=f'{target}/{name}-{label}-{i}.json'
+  env=dict(os.environ,HF_HUB_OFFLINE='1',OMP_NUM_THREADS='2',OPENBLAS_NUM_THREADS='2')
+  run=subprocess.run(['nice','-n','10',sys.executable,'work/mac-scoreboard/worker.py',folder,name,prompt,str(n),out,lfm],env=env,timeout=300)
+  if run.returncode:print('failed',run.returncode,flush=True);sys.exit(run.returncode)
+  time.sleep(15)

mseries/local-ai/receipts/2026-10-30-macos-metal-profile/harness/worker.py

@@ -0,0 +1,21 @@
+import sys,os,json,time,hashlib,importlib.util
+from pathlib import Path
+model_dir,which,prompt,max_tokens,out,lfm_file=sys.argv[1:]
+if lfm_file!='-':
+ p=Path(lfm_file);expected='a4096cddc8f0d3f7923008b2e6afb896dabc9aef963a5ebe7d76d13a6069b0c4'
+ if hashlib.sha256(p.read_bytes()).hexdigest()!=expected: raise RuntimeError('LFM3 implementation checksum mismatch')
+ spec=importlib.util.spec_from_file_location('mlx_lm.models.lfm3',str(p));mod=importlib.util.module_from_spec(spec);sys.modules[spec.name]=mod;spec.loader.exec_module(mod)
+from mlx_lm import load,stream_generate
+from mlx_lm.sample_utils import make_sampler
+import mlx.core as mx
+wall_start=time.strftime('%Y-%m-%d %H:%M:%S %Z%z'); started=time.monotonic(); model,tok=load(model_dir)
+prompt=tok.apply_chat_template([{'role':'user','content':prompt}],tokenize=False,add_generation_prompt=True,enable_thinking=False)
+ids=tok.encode(prompt); sampler=make_sampler(temp=0)
+for _ in stream_generate(model,tok,prompt,max_tokens=int(max_tokens),sampler=sampler):pass
+mx.synchronize(); ready=time.monotonic()-started;start=time.monotonic();first=None;tokens=[];text=''
+for r in stream_generate(model,tok,prompt,max_tokens=int(max_tokens),sampler=sampler):
+ if first is None:first=time.monotonic()-start
+ tokens.append(r.token);text+=r.text
+mx.synchronize();total=time.monotonic()-start
+res={'model':which,'prompt_tokens':len(ids),'prompt_token_sha256':hashlib.sha256(json.dumps(ids).encode()).hexdigest(),'ready_s':ready,'ttft_s':first,'total_s':total,'completion_tokens':len(tokens),'decode_estimate_tps':(len(tokens)-1)/(total-first) if first is not None and len(tokens)>1 and total>first else None,'token_ids':tokens,'text':text,'start_time':wall_start,'end_time':time.strftime('%Y-%m-%d %H:%M:%S %Z%z'),'peak_memory_bytes':mx.get_peak_memory()}
+Path(out).write_text(json.dumps(res,indent=2)+'\n');print(json.dumps({k:v for k,v in res.items() if k not in ('text','token_ids')}),flush=True)