feat: add FP8 blockwise weight quantization and FP8(E4M3) KV cache support

[shsaihdsaiudh]

Summary

• New FP8Blockwise quantization scheme (csrc/layers/quantization/fp8_blockwise.{cpp,hpp}, QuantScheme::FP8_W8A16): loads FP8 E4M3 checkpoints with 128x128 block scales (quant_method: "fp8", weight_block_size: [128,128] — Qwen3-FP8 series, DeepSeek-style) and runs them without materializing BF16 weights.
• Decode routing in FP8Blockwise::forward: M <= 32 dispatches to InfiniCore's fused fp8_blockwise_gemm (SIMT for M <= 8, tensor-core mma for 9 <= M <= 32); larger M and prefill use dequantize + cuBLAS. INFINILM_FP8_FUSED_GEMM=0 forces the fallback.
• Opt-in Marlin conversion path (INFINILM_FP8_MARLIN=1) in fp8_blockwise.cpp — currently blocked by an InfiniCore sm_120 kernel defect, see "Notes for Reviewers".
• FP8(E4M3) KV cache for paged attention: KVQuantAlgo::FP8 (csrc/config/quant_config.*), F8 cache allocation plus per-layer F32 k_scale/v_scale (csrc/cache/kv_cache.*, csrc/global_state/forward_context.hpp, csrc/models/infinilm_model.cpp), scale passing in csrc/layers/attention/backends/paged_attn.cpp. Other attention backends reject FP8 KV explicitly at construction.
• kv_cache_dtype exposed through EngineConfig, LLM, and the server entry points (python/infinilm/config/engine_config.py, python/infinilm/llm/llm.py, python/infinilm/server/inference_server.py); "fp8" handled in modeling_utils.py::parse_dtype.
• FP8 checkpoint loading in python/infinilm/modeling_utils.py: _resolve_preserve_config generalizes dtype-preservation to any config with quantization_config (keeps float8 dtypes and weight_scale/weight_scale_inv tensors as stored); _cast_fp8_scales_to_fp32.
• New test test/models/qwen3_fp8/test_fp8_checkpoint_loading.py.
• Bundled small fixes (called out per checklist; can be split if preferred): declare runtime dependencies and server/multimodal extras in pyproject.toml; log expected shutdown exceptions as warnings, not fatal, in csrc/engine/rank_worker.cpp; rounding carry in F16 scale conversion for the Marlin path (csrc/layers/quantization/gptq_marlin.cpp).
• Depends on InfiniCore PR feat: add FP8 blockwise dequantize/GEMM operators and FP8(E4M3) KV cache support in paged attention InfiniCore#1568 — InfiniLM CI builds against InfiniCore main, so CI here can only go green after that PR merges.

Motivation

InfiniLM already supports GPTQ/AWQ (Marlin), MXFP4 and INT8 KV cache, but had no FP8 weight quantization at all — the release format of DeepSeek-V3 main weights and the Qwen3 FP8 series (e.g. Qwen/Qwen3-8B-FP8, dynamic activation quantization). This PR adds the full path end to end, plus FP8 KV cache (halves cache capacity and decode attention read bandwidth). This is a course project submission (LLAISYS 2026 spring camp, project #4 — quantization and low-precision inference); design doc fp8_quant_proposal.md, weekly reports and all raw artifacts (logs, JSON, scripts) referenced below as dev_fp8/... live on the fork branch backup/fp8-lm-full-history, intentionally kept out of this PR.

Before/after, decode throughput (Qwen3-8B-FP8 vs Qwen3-8B, RTX 5090 sm_120, examples/bench.py --enable-paged-attn --warmup, input 128 / output 256, greedy; methodology in dev_fp8/w3~w5, w8 reports):

decode	FP8 naive (before fused routing)	FP8 this PR	BF16 baseline	FP8 this PR vs BF16
bs=1	28.49 tok/s	142.96 tok/s (ITL 7.00 ms)	92.34 tok/s	155% (5.02x vs naive; 91% of vLLM FP8's 157.02)
bs=4	110.21	354.32	350.64	101%
bs=8	218.1	408.7 (SIMT, M=8)	686.52	~60%
bs=16	427.0	1794.6 (mma)	1304.1	1.38x (4.20x vs naive, 4.73x vs SIMT 379.7)
bs=32	817.6	2702.0 (mma)	—	3.30x vs naive

Accuracy / memory (targets from the project proposal: PPL deviation < 1%, C-Eval/MMLU within 0.5 pt of BF16, weight memory <= 60% of BF16):

Metric	FP8	BF16	Delta	Target
wikitext2 PPL (chunk 512, full test set, 293457 tokens)	18.8771	18.8617	+0.082%	< 1% ✅
C-Eval val (1346 questions, logprob-MCQ)	74.74%	74.37%	+0.37 pt	±0.5 pt ✅
MMLU test (14042 questions, logprob-MCQ)	69.47%	69.68%	-0.21 pt	±0.5 pt ✅
Weight memory / file size	8.9 GB	16 GB	55.6%	<= 60% ✅
Weight load time	2.4–2.7 s	5.0 s	~1.9x faster	—

FP8 KV cache (same FP8 weights; baseline is paged BF16 KV): PPL 18.8724 vs 18.8772 (-0.026%); C-Eval 74.29% vs 74.67% (-0.37 pt); MMLU 69.54% vs 69.32% (+0.22 pt) — all within the ±0.5 pt noise band. KV bytes per token (Qwen3-8B): 147456 B -> 73728 B + 2304 B scales = ~1.94x compression.

Closes # — no tracking issue; course submission, see above. Depends on InfiniTensor/InfiniCore#1568.

Type of Change

• feat — new feature / new model
• fix — bug fix (rank_worker shutdown logging; Marlin F16 scale rounding carry)
• build / ci — build system or CI configuration (pyproject.toml runtime dependencies/extras)

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Platform: NVIDIA RTX 5090 (sm_120, 32 GB), driver 610.43.02, CUDA 12.8/13.3. Model: Qwen/Qwen3-8B-FP8 (snapshot 220b46e3); baseline Qwen/Qwen3-8B (b968826d); vLLM 0.28.0 cross-check (torch 2.13.0+cu130).

• Offline performance (examples/bench.py): tables in "Benchmark / Performance Impact" below; raw logs in dev_fp8/results/perf_bench.log, perf_bf16_bs48.log, W6/W7/W8 logs (dev_fp8/results/w6_perf3.log, w8_chain.log, ...).
• Accuracy: PPL / C-Eval / MMLU tables in "Motivation" above; raw JSON in dev_fp8/results/{ppl,ceval,mmlu}_*.json, runners fp8_ppl_eval.py, fp8_mcq_eval.py, run_accuracy_evals.sh. vLLM parity: BF16 C-Eval identical between engines (74.37%), FP8 MMLU within 0.11 pt.
• Checkpoint loading unit test: python -m unittest test.models.qwen3_fp8.test_fp8_checkpoint_loading — developed and run on RTX 5090 during the W3 phase; not re-run on the final rebased branch (no NVIDIA GPU on the machine preparing this submission).
• E2E greedy parity: fused vs naive token-identical (3 prompts x 64 tokens); FP8 KV vs BF16 KV token-identical (4020-token prompt + 48 tokens, split-kv active, dev_fp8/results/w7_e2e_parity.py); bs=16 mma vs naive coherent with identical openings.
• Single-request test (examples/test_infer.py) and service test: not re-run on the final rebased branch (same reason as above); see Checklist for status.

Other platforms: no behavior change on non-NVIDIA platforms (FP8 paths are opt-in; FP8 KV on non-paged backends fails explicitly at construction). Untested on non-NVIDIA hardware — no device access; see Checklist/Testing.

Benchmark / Performance Impact

Harness: examples/bench.py --enable-paged-attn --warmup, input 128 / output 256, greedy, RTX 5090 (sm_120). Reproduce with dev_fp8/results/fp8_perf_bench.sh (build: fp8_build_core.sh / fp8_build_lm.sh, env: fp8_env.sh). Decode tables are in "Motivation" above.

FP8 KV cache decode ITL (ms, output 128; same-session BF16 KV baseline; dev_fp8/results/w7_splitkv_bench.sh):

Scenario	BF16 KV	FP8 KV (split-kv auto)	FP8/BF16	FP8 (split-kv off)
bs=1, in=1024	7.85	7.27	0.93x	8.35
bs=1, in=4096	10.29	7.96	0.77x	11.92
bs=1, in=16384	20.58	11.93	0.58x	26.47
bs=8, in=1024	20.84	22.84	1.10x	22.00
bs=8, in=4096	25.31	29.09	1.15x	29.76

(bs=8/in=16384 OOMs at bench warmup for both dtypes — pre-existing bench behavior, unrelated to KV dtype.) Prefill TTFT overhead of FP8 KV: +0.5%~+3.0% (gather-dequant scratch). INFINIOP_FLASH_DECODE_SPLITKV=0 regression column matches the pre-split-kv numbers point-for-point, confirming the gain comes from split-kv alone.

Remaining gaps (recorded honestly): bs=8 decode stays on the SIMT path (~60% of BF16); prefill still uses dequant + cuBLAS (bs=1 41.5% of BF16, bs=16 90.5%); vLLM FP8 is faster at bs>=8 (1224/2327 tok/s at bs=8/16) — the GEMM part of that gap at bs>=16 was closed by the mma path (382 -> 1794.6 tok/s), the remainder is attention/runtime, not re-measured against vLLM.

Notes for Reviewers

• Dependency: requires InfiniCore PR feat: add FP8 blockwise dequantize/GEMM operators and FP8(E4M3) KV cache support in paged attention InfiniCore#1568 (new operators fp8_blockwise_dequantize, fp8_blockwise_gemm, FP8 KV in paged attention operators). CI here builds against InfiniCore main, so CI must be (re)triggered after that merges.
• Marlin path: conversion code is included but opt-in (INFINILM_FP8_MARLIN=1) and currently unusable on sm_120: the InfiniCore Marlin kernel deadlocks (CUDA 12.8) or produces wrong output for all bit widths (CUDA 13.2). Minimal repro and investigation: dev_fp8/marlin_sm120_issue.md (+ dev_fp8/marlin_sm120_repro/). The default path does not touch Marlin.
• Routing thresholds are measured, not arbitrary: SIMT fused wins for M <= 8, mma for 9 <= M <= 32, naive above (W4/W8 reports). Fallback switches: INFINILM_FP8_FUSED_GEMM=0, INFINIOP_FP8_GEMM_MMA=0.
• FP8 KV limitations: paged-attention backend only (FLASH_ATTN/STATIC, MLA hd576/512, videonsa/deepseek custom allocation paths reject FP8 KV explicitly); KV connector (PD disaggregation) does not transfer scales — that combination is unavailable; INT8 static-attn path untouched.
• Evaluation protocol note: test/bench/test_benchmark.py generative C-Eval scoring breaks on Qwen3 chat output (answer extraction inspects only the first 2 characters; --max-new-tokens 8 truncates), so accuracy was measured with logprob-MCQ (lm-eval-harness protocol, identical prompts/data/scoring on both engines) — see dev_fp8/w5_eval_report.md §1.
• Bundled fixes (pyproject.toml, rank_worker.cpp, gptq_marlin.cpp rounding) are small and listed in Summary; happy to split into separate PRs if reviewers prefer.

CI / ChatOps

CI runs on this PR automatically; it builds against InfiniCore main, so a green run requires the InfiniCore PR linked above to merge first. Will retrigger (or ask a maintainer for /retest) after that lands.

---

Checklist

Title, Branch, and Commits

• PR title follows Conventional Commits (feat: ...).
• Branch name follows <type>/xxx-yyyy-zzzz — feat/fp8-blockwise-quantization.
• Each commit message follows Conventional Commits (feat: / test: / build: / fix:).
• Large PR: every commit is meaningful and independently reviewable — 4 commits: feature, unit test, and the two small bundled fixes as their own commits.
• No stray merge commits from main — branch is linear and freshly based on current main.
• No fixup! / squash! / wip commits remain.
• N/A — no legacy issue format involved.

Scope and Design

• Changes are minimal — the only non-FP8 changes are the three small fixes listed in Summary (can be split on request).
• No dead code, commented-out blocks, debug prints, or ownerless TODOs — the INFINILM_FP8_* env switches are documented fallback toggles, not debug leftovers.
• No unrelated formatting churn (scripts/format.py).
• Public API changes are intentional and documented — EngineConfig.kv_cache_dtype, LLM(...) / inference_server parameters with docstrings; KVQuantAlgo::FP8.

General Code Hygiene (applies to all languages)

• Comments only where the why is non-obvious (FP8 bit-layout decode, routing thresholds, Marlin blockage).
• Every modified or added file ends with a single trailing newline.
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages wrapped in backticks.
• All comments and error messages in English.
• Comments and error messages are complete sentences.

C++ Specific (if C++ files changed)

• Google C++ Style Guide.
• LLVM-style error/warning wording.
• Constructor initializer list order matches member declaration order.
• No raw new/delete; RAII / existing allocators.
• Changed files formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• PEP 8 compliant.
• Complete-sentence English comments with backticks.
• PEP 257 docstrings.
• Changed files formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• Platforms that could not be tested: non-NVIDIA platforms — no hardware access; FP8 KV is gated to the paged-attention backend and other backends reject it explicitly, so risk of silent regression elsewhere is low.
• Passed single request test (examples/test_infer.py) — not re-run on the final rebased branch (no NVIDIA GPU on the machine preparing this submission); during development parity was verified at the InferEngine level with the scripts in dev_fp8/results/ (E2E greedy parity above), and kv_cache_dtype has since been added to the LLM entry points.
• Passed offline performance test (examples/bench.py) — extensively, see Benchmark section.
• Passed sanity test (test/bench/test_benchmark.py) — skipped with reason: its generative C-Eval scoring cannot parse Qwen3 chat answers (see Notes for Reviewers); accuracy covered instead by full logprob-MCQ evals (C-Eval val 1346 / MMLU test 14042) with raw data in dev_fp8/results/.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py) — not run; server-side change is limited to kv_cache_dtype parameter passthrough, covered by the config-level wiring.

Build, CI, and Tooling

• Builds cleanly from a fresh directory — server builds ran from clean, md5-verified trees (dev_fp8/results/fp8_build_core.sh, fp8_build_lm.sh; W8 build log w8_build2.log).
• CI green — runs on PR open; builds against InfiniCore main, so it can only pass after feat: add FP8 blockwise dequantize/GEMM operators and FP8(E4M3) KV cache support in paged attention InfiniCore#1568 merges; will retrigger then.

Documentation

• Docs updated — design + weekly reports in fp8_quant_proposal.md and dev_fp8/*.md (on the fork branch backup/fp8-lm-full-history); kv_cache_dtype documented in EngineConfig/LLM/server docstrings. README.md/MODELS.md unchanged (no new model family — this enables existing Qwen3 FP8 checkpoints).
• No user-visible breaking change — FP8 weights/KV are opt-in; defaults are unchanged.

Security and Safety

• No secrets, tokens, or personal identifiers committed (dev_fp8/ is not part of this PR; its results/fp8_ssh.py is a password-auth wrapper and the password is never stored in the repo).
• No new third-party code.
• Kernel-side bounds: tail blocks zero-filled on load and discarded on store; GEMM shape constraints (K % 128 == 0, block_k % 128 == 0, block_n % 16 == 0) are enforced at InfiniCore descriptor creation, so mis-shapes fail loudly instead of miscomputing.