Repository navigation

backends/cuda: a torch reference backend — edge0-8b and edge0-35b generate text on an NVIDIA GPU - #19

Closed

agourakis82 wants to merge 34 commits into

Edge0-AI:mainfrom

agourakis82:cuda-backend

Closed

backends/cuda: a torch reference backend — edge0-8b and edge0-35b generate text on an NVIDIA GPU#19
agourakis82 wants to merge 34 commits into
Edge0-AI:mainfrom
agourakis82:cuda-backend

Commits on Sep 11, 2026
- backends/cuda: skeleton facade (torch reference impl) + EDGE0_BACKEND=cuda dispatch
  
  Demetrios Chiuratto Agourakis
  authored and
  agourakis82
  committed
  850cd6f
- docs: document running on NVIDIA GPUs via mlx[cuda12] (no code changes)
  
  Demetrios Chiuratto Agourakis
  authored and
  agourakis82
  committed
  497c032
- backends/cuda/io.py: implement load_model (dispatch verified against real checkpoints)
  
  Demetrios Chiuratto Agourakis
  authored and
  agourakis82
  committed
  a4e1796
- backends/cuda: fix dispatch + key-prefix bugs found against REAL checkpoints; add MoESpec instances
  
  Demetrios Chiuratto Agourakis
  authored and
  agourakis82
  committed
  4d67e08
- backends/cuda/quant.py: mark int4 packing as VERIFIED against real mx.quantize output
  
  Demetrios Chiuratto Agourakis
  authored and
  agourakis82
  committed
  1c4ee13
- backends/cuda/quant.py: follow mx.gather_qmm broadcasting; parity tests vs real MLX
  
  agourakis82
  and
  claude
  committed
  0c89f90
- backends/cuda, docs, tests: fix what the patch series got wrong
  
  agourakis82
  and
  claude
  committed
  ac0053a
- backends: make routing and sampling run on the torch backend, same answers as MLX
  
  agourakis82
  and
  claude
  committed
  716d977
- streaming: run StreamingSwitchGLU on the torch backend; fix hot-stack out-of-bounds gather
  
  agourakis82
  and
  claude
  committed
  2740f7a
- prerouter, adapters: reach the backend only through the facade
  
  agourakis82
  and
  claude
  committed
  6aa6ee8
- streaming/install: install streaming experts into transformers models
  
  agourakis82
  and
  claude
  committed
  4c1fa30
- backends/cuda/io.py: load_model loads the real MLX checkpoints
  
  agourakis82
  and
  claude
  committed
  46a34aa
- engine: import without MLX; refuse non-MLX backends up front
  
  agourakis82
  and
  claude
  committed
  6e13276
- docs/nvidia.md: what the torch backend has, how it is checked, what is left
  
  agourakis82
  and
  claude
  committed
  9e16abb
- backends/cuda: torch port of bailing_hybrid (edge0-8b), checked layer by layer against MLX
  
  agourakis82
  and
  claude
  committed
  8fbe20f
- engine: run the edge0-8b engine on the torch backend; bound its memory
  
  agourakis82
  and
  claude
  committed
  8af2e6b
- backends/cuda: torch port of qwen3_5_moe (edge0-35b); the 35b engine runs on torch
  
  agourakis82
  and
  claude
  committed
  b0e4d4d
- docs/nvidia.md: cuInit 999 on our Spark was a host device policy, not the driver
  
  agourakis82
  and
  claude
  committed
  7ab23fc
- backends/cuda: parameters never require grad (the prefill held its whole autograd graph)
  
  agourakis82
  and
  claude
  committed
  2597d09
- backends/cuda: run on an accelerator; load_model moves init-time buffers to DEVICE
  
  agourakis82
  and
  claude
  committed
  750a5b9
Commits on Sep 12, 2026
- backends/cuda: gather quantized embedding rows on an int32 view (CUDA has no uint32 index)
  
  agourakis82
  and
  claude
  committed
  f6276a8
- tests: check the qwen3_5_moe port against MLX on the real edge0-35b weights
  
  agourakis82
  and
  claude
  committed
  0903837
- docs/nvidia.md: edge0-35b runs on the GB10, on the real weights
  
  agourakis82
  and
  claude
  committed
  af697c4
- docs/nvidia.md: edge0-35b on the GB10, layer by layer against MLX
  
  agourakis82
  and
  claude
  committed
  b2b7a8a
Commits on Sep 13, 2026
- backends/cuda: optional dequantized-weight cache (2.5x a decode step on a GB10)
  
  agourakis82
  and
  claude
  committed
  eb5ef48
Commits on Sep 14, 2026
- Merge upstream main into cuda-backend
  
  agourakis82
  and
  claude
  committed
  890f39e
Commits on Sep 15, 2026
- cuda: stop paying a host sync per expert in gather_qmm
  
  agourakis82
  and
  claude
  committed
  800c306
Commits on Sep 28, 2026
- Merge remote-tracking branch 'origin/main' into cuda-backend
  agourakis82
  committed
  8e532d1
- tests: incremental staged stack matches on both backends
  
  agourakis82
  and
  claude
  committed
  7e5656f
Commits on Oct 2, 2026
- review: remote code opt-in, optional cuda extra, README install path
  
  agourakis82
  and
  claude
  committed
  72d58fc
- review: take() result shape, Bailing wrapper args, honest smoke tests
  
  agourakis82
  and
  claude
  committed
  50317a2
- Merge upstream main into cuda-backend (adopt the python/ layout)
  
  agourakis82
  and
  claude
  committed
  5153694
Commits on Oct 9, 2026
- docs: clarify the existing CUDA reference backend
  agourakis82
  committed
  96ee715
Commits on Oct 15, 2026
- Merge current main into CUDA backend contribution
  agourakis82
  committed
  4929986

Commits

Commits on Sep 11, 2026

Commits on Sep 12, 2026

Commits on Sep 13, 2026

Commits on Sep 14, 2026

Commits on Sep 15, 2026

Commits on Sep 28, 2026

Commits on Oct 2, 2026

Commits on Oct 9, 2026

Commits on Oct 15, 2026