Your A100 has FP8. Your T4 doesn't. Neither does your 2080 Ti, your 3060, or the P40 collecting dust in your homelab. softfp8 is a Triton GEMV for batch-1 decoding that stores weights as packed E5M2 bytes and turns them back into fp32 inside registers, inside the K-loop — the memory bus never sees an fp16 copy of your weights, so decode gets ~2× faster on hardware that can't do FP8 at all. Software FP8, as in: the emulation is the feature.
(one int32 load per four weights; bytes peel off into FMAs — nothing ever materializes as an expanded fp16 matrix)
Decode is not a compute problem. At batch 1 your arithmetic intensity is ~2 FLOP per byte and even a 3060's ridge point is two orders of magnitude away. So the whole game is bytes moved:
| weight format | bytes streamed per token (7B) | predicted tok/s, RTX 3090 @ 65% BW |
|---|---|---|
| fp16 | 14.0 GB | 43.5 |
| E5M2 (this repo) | 7.0 GB | 86.9 |
| theoretical 4-bit | 3.5 GB | 173.9 |
E5M2 is the format with the cheapest software decode: its exponent field
lines up with fp16's after a shift, so a 0x4b0000__ trick plus exp2
converts a packed byte to a float in a couple of integer ops — no
conversion tables, no ldgsts, nothing SM-version-specific. Four codes ride
in every int32, so one 128-bit load feeds four FMAs.
The kernel decode is bit-exact against torch.float8_e5m2: every code,
sign / exponent / mantissa reconstructed by integer bit-twiddling, checked
against PyTorch's hardware cast in test_cast_roundtrip_exact. The fused
GEMV then matches the dequant-then-matmul reference to 1.6e-6 max abs
error, plotted across 3,840 outputs so it isn't a cherry-picked shape:
Bandwidth-model check — the speedup claim as a graph, per GPU, not a table you have to trust:
| what it needs | what it trades | |
|---|---|---|
| TensorRT-Model-Optimizer / TorchAO FP8 | Hopper/Ada FP8 units | free lunch, but only on new cards |
| bitsandbytes NF4/INT8 | own kernels, fp16 compute | good, but 8-bit is the floor |
| softfp8 | any GPU Triton compiles for | E5M2's 2-bit mantissa: 4.9% output rel-err |
| QLoRA NF4 | 4-bit, block scales | requires dequant ladder; fp8-exact |
The niche: you already own an old GPU, you decode at batch 1, and your quality loss at 8 bits is already under your noise floor — so why is your memory traffic still fp16-shaped?
softfp8/
├── softfp8/core.py # pack/decode/quantize + the Triton kernel
├── tests/test_core.py # 6: bit-exactness vs torch cast, parity, ratio algebra
├── demos/demo.py # writes receipts/receipt.json
└── demos/make_figures.py # regenerates every figure above from live computation
pip install -e ".[dev]"
pytest tests/ -q && python demos/demo.py && python demos/make_figures.pyCI badge: see the shields above — same commands, GitHub-hosted.
- GEMV only (M=1). The packed-decode idea ports to tiled GEMM but the win shrinks as M grows — it's a bandwidth trick, and bandwidth only dominates at small M. Say so in the paper and in the code.
- 4.9% output rel-err is E5M2, not this kernel: any fp8 weight format pays that. Activation precision is untouched (fp32 in this path).
- Predicted tok/s is bytes ÷ measured-bandwidth × 0.65, with 0.65 from
public copy-kernel literature. Run
gembench-style measurement on your card for the real coefficient.