Gemma-4-31B-it-QAT-MX46A_S-GGUF

MX46A_S (sensitivity-tiered adaptive NVFP4/MXFP6 mix, ~4.4 bpw average) quantization of Google's Gemma 4 31B It QAT — the half-precision checkpoint extracted from Google's quantization-aware-training pipeline, which makes it markedly more robust at 4-bit precision than post-training quantization of the standard release.

⚠️ Requires a custom llama.cpp fork

The MX46A / MX46AS tensor types (GGML types 43/44) do not exist in mainline llama.cpp. This GGUF will not load in stock llama.cpp, LM Studio, Ollama, or Jan. Build the fork here: https://github.com/AcerThyRacer/llama.cpp/tree/mxfp6-adaptive (commit b0e5c2a24), with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 for RTX 50-series.

Files

File	Size	Purpose
gemma-4-31B-it-qat-MX46A_S.gguf	17.8 GB	Main model, MX46A_S tiered quantization
gemma-4-31B-it-mmproj.gguf	1.2 GB	Vision projector (F16, from Google's official GGUF release) — required for image input
gemma-4-31B-it-F16-MTP.gguf	955 MB	MTP head for --spec-type draft-mtp speculative decoding (optional, big decode speedup)

What even is MX46A_S?

MX46A_S is a per-tensor-category mix built from two custom superblock formats plus established ones, assigning bits where the signal actually lives:

Tensor category	Type	Why
attn_v, attn_output	MX46A (5.0 bpw)	Highest quantization sensitivity; 256-weight superblocks where the chunks with the largest imatrix-weighted error get promoted from FP4 (E2M1) to MXFP6 (E2M3)
ffn_down (outer layers)	MX46A	First/last layers carry disproportionate signal
ffn_gate, ffn_up	IQ4_XS (4.25 bpw, imatrix)	Bulk of parameters, lowest sensitivity
everything else (attn_q/k, inner ffn_down)	MX46AS (4.625 bpw)	Lighter sibling: MXFP4-style E8M0 base, exactly 1-of-8 chunks promoted to FP6
output, token_embd	Q6_K	Standard practice for fragile tensors

All FP4/FP6 element formats and E8M0/UE4M3 scale formats follow the OCP Microscaling (MX) and NVIDIA NVFP4 conventions and decode through native CUDA kernels (MMVQ vec-dot for generation, MMQ with Blackwell-specific SRAM layouts for prefill on supported types).

Quantization provenance

• Source: google/gemma-4-31B-it-qat-q4_0-unquantized (BF16 QAT checkpoint), streamed remotely and converted to Q8_0 GGUF (convert_hf_to_gguf.py --remote --outtype q8_0). The Q8_0 intermediate hop is effectively transparent (per-element RMSE ≈ 7e-5) and was required because a 62 GB BF16 intermediate did not fit the build machine's disk.
• Importance matrix: 120 chunks × 512 tokens of WikiText-2 train, computed on the Q8_0 model (llama-imatrix). The imatrix guides both MX46A chunk promotion and the IQ4_XS ffn tiers.
• Quantized with llama-quantize ... MX46A_S at fork commit b0e5c2a24.
• Build/eval hardware: RTX 5080 16 GB (Blackwell, CC 12.0), CUDA 13.3, driver 595.80.

Measured quality & speed

Why KL-divergence and not wikitext perplexity? This model is a thinking-instruct QAT artifact: raw untemplated text is far outside its post-training distribution, and its raw wikitext perplexity lands in the thousands — including Google's own official Q4_0 GGUF (verified on 2026-06-10; the model is healthy: templated long-context comprehension is flawless). Do not panic if you measure it yourself. Quantization fidelity is instead measured as KL-divergence of this quant's token distributions against the Q8_0 reference on identical inputs.

All numbers measured on RTX 5080 16 GB (partial offload -ngl 30, FA on, fork commit 47fa11b36+fixes), same Q8_0 reference, same text, same machine:

Metric	This release (MX46A_S)	UD-Q4_K_XL (pure Q4_0)
File size / bpw	17.8 GB / 4.64	17.3 GB / 4.50
Mean KLD vs Q8_0	0.142 (median 0.069)	0.011
Mean Δp on reference tokens	−0.54 %	—
pp512 t/s	486	515
tg64 t/s	2.15	3.30

Honest positioning — read this before using

This checkpoint was QAT-trained against Q4_0's exact quantization grid: its weights sit on Q4_0 lattice points by construction, so plain Q4_0 quantizes it near-losslessly (KLD 0.011) and any other 4-bit grid — including this one — moves weights off their QAT-optimal positions. For everyday use of this particular model on this class of hardware, Google's official Q4_0 (or a UD-Q4_K_XL) is the better choice: smaller, more faithful, and faster under partial offload, where decode is bound by CPU-resident layers and Q4_0's mature AVX2 kernels win.

What this release is for instead:

• Format research: a complete, working, end-to-end MX46A/MX46AS/NVFP4/MXFP6 pipeline on a real 31B model — reference quality numbers for adaptive FP4/FP6 superblock formats included.
• Full-GPU Blackwell scenarios: the native FP4/FP6 MMQ/MMVQ paths pay off when all layers are GPU-resident (24 GB+ cards, or the smaller Gemma-4 variants); on 16 GB with a 31B they are masked by the CPU-side bottleneck.
• Non-QAT models: against ordinary BF16 checkpoints the imatrix-guided FP6 promotion competes on fidelity-per-bit; against a Q4_0-QAT checkpoint nothing beats Q4_0 — by design.

VRAM note: a 31B dense model at ~4.4 bpw is ~17 GB of weights — on a 16 GB card expect partial offload (≈30/61 layers on an RTX 5080 alongside a desktop session). Full-GPU residency on 16 GB requires the smaller Gemma-4 variants or a lower-bpw mix.

Run it

# text + vision
./build/bin/llama-server \
  -m  gemma-4-31B-it-qat-MX46A_S.gguf \
  --mmproj gemma-4-31B-it-mmproj.gguf \
  -ngl 30 -fa on -c 8192 -ctk q8_0 -ctv q8_0 --port 8080

# with MTP speculative decoding (recommended; the head is trained with the model)
./build/bin/llama-server \
  -m  gemma-4-31B-it-qat-MX46A_S.gguf \
  --spec-type draft-mtp \
  --spec-draft-model gemma-4-31B-it-F16-MTP.gguf \
  -ngl 30 -fa on -c 8192 -ctk q8_0 -ctv q8_0 --port 8080

Known limitations

• Fork-only format — see the warning above. Track upstreaming status at https://github.com/AcerThyRacer/llama.cpp/tree/mxfp6-adaptive.
• The MXFP6 direct-FP8 activation decode path (GGML_CUDA_MXFP6_FP8_ACT=1) trades ~2× matmul NMSE for speed and is off by default; this release's tier mix does not use standalone MXFP6 tensors.
• imatrix calibration is English-centric (WikiText-2); multilingual quality may shift slightly.
• Vision projector is kept at F16 by design; quantizing it is not supported in this release.

License

Subject to the Gemma Terms of Use. This repository redistributes a quantized derivative of a Google Gemma model; you must comply with the Gemma license, including its use restrictions, when using these files.