🔨 feat(quants): use the importance matrix when encoding NVFP4 - #66
Open
worthant wants to merge 1 commit into
Open
🔨 feat(quants): use the importance matrix when encoding NVFP4#66worthant wants to merge 1 commit into
worthant wants to merge 1 commit into
Conversation
quantize_nvfp4 discarded quant_weights, so an imatrix changed nothing for this type while every K and IQ type uses it. Encode each sub-block by searching the ue4m3 scale grid around amax/6 under an importance-weighted error, refitting the scale against its own assignment. Blocks whose importance is entirely zero, which happens for experts the corpus never routed to, fall back to magnitude-only weights so the search stays well posed. Without an imatrix the reference encoder is still used, so existing output is unchanged. On Ling-3.0-flash the mean KL divergence against bf16 drops from 0.05602 to 0.05363 and top-1 agreement rises from 94.721% to 94.865%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream
quantize_nvfp4takesquant_weightsand immediately doesGGML_UNUSED(quant_weights), so an importance matrix has no effect on this type while every K and IQ type uses one.The scale of an NVFP4 sub-block is stored as UE4M3, so neighbouring scales sit ~12.5% apart. Rounding to
amax/6picks one point on that grid and it is rarely the best one: a slightly smaller scale clips the single largest weight but resolves the other fifteen better. This walks eight codes around that point, and for each one refits the scale against its own assignment by weighted least squares, keeping whichever candidate has the lowest importance-weighted error.Blocks with all-zero importance, which is what experts the calibration corpus never routed to look like, fall back to magnitude-only weights. Without that the weighted error is degenerate and the search picks an arbitrary scale; an earlier revision that missed this case measured a mean KLD of 2.76.
Measurements
Ling-3.0-flash, mean KL divergence against the bf16 baseline, 92 chunks of held-out text at 4096 tokens, 4x RTX PRO 6000:
A 4.3% reduction in KLD for no change in file size, format or inference cost; the work is paid once at quantization time. The measurement is one model on one corpus, and the spread between the encoder variants tried along the way (0.05363 to 0.05392) is close to the +-0.001 error bar, so treat the ranking among those as unresolved. The gap to the unweighted encoder is about 2.4x the error bar.
Raw logs:
AtomicChat/Ling-3.0-flash-GGUF-metrics.Not done yet
Before this is worth proposing upstream it needs a sha256 check that output without an imatrix is byte-identical to before, an explicit all-zero imatrix test, numbers on a dense model such as Llama-3.1-8B on wikitext2, and the quantization-time cost of the search.