kingjones777/Ling-3.0-flash-NVFP4-SGLang-MTP

Ling-3.0-flash — NVFP4, single GPU, MTP working, SwiGLU clamp fixed

Ling-3.0-flash quantized to NVFP4 so it runs on one 128GB-class GPU under SGLang, with NEXTN multi-token prediction working and the SwiGLU clamp actually applied.

Upstream ships a 4×141GB reference deployment. This runs on one card.

Two things make this different from a plain conversion, and I explain both below so you can reproduce them rather than trust me:

1. The MTP layer is preserved and correctly declared, so speculative decoding works.
2. SGLang silently drops Ling's SwiGLU clamp on the NVFP4 path. Without the fix in sglang_patch/, this model — or any NVFP4 build of it — corrupts roughly two thirds of the code it writes. The weights are fine. The serving stack is what needs patching.

---

⚠️ Read this first: apply sglang_patch/ or you get a broken model

Ling is trained with clamped SwiGLU on its late layers. The official config carries the limits:

expert_swiglu_limit_list        layers 35–41 = 4      (all other layers 0)
share_expert_swiglu_limit_list  layers 34–39 = 5, layers 40–41 = 7

SGLang reads those keys and implements the clamp — but the NVFP4 MoE scheme throws it away:

# sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4_moe.py
gemm1_clamp_limit=None,     # hard-coded

Every other MoE runner in SGLang (triton, marlin, ascend, deep_gemm, mxint4) forwards the real value. Only the NVFP4 scheme discards it. With no clamp, activations in the last seven layers can run away, and a junk token wins a step:

stack.append(c)ulp
high = len(arr) -  Advisory
elif arr[mid] < x Advisory:
seen.add(culp)          # should be seen.add(c)

It looks random and it recovers immediately, which is what makes it easy to misdiagnose as a bad quantization. It is not. It is a missing clamp.

What I measured, before and after the patch

Detection is an AST check for identifiers that are loaded but never bound. Use that, not a non-ASCII scan — the most common junk tokens (ulp, oly, culp, Advisory) are plain ASCII and a charset filter sails straight past them.

sampling	before patch	after patch
temp 0, thinking off — 15 tasks	4/15	15/15
temp 0, thinking off — 10 tasks	2/10	10/10
temp 0.6 / top_p 0.95 / top_k 20, thinking on (upstream recipe)	0/10	10/10
temp 0.6 / top_p 0.95 / top_k 20, thinking off	1/10	10/10
generated code executed, assertions checked	—	8/8

The upstream-recommended sampling going from 0/10 to 10/10 is the clearest signal. Before the patch that configuration was the worst one; after it, it is perfect.

Applying it

cd <sglang>/python/sglang/srt
patch -p1 < sglang_patch/01-flashinfer_cutlass.patch
patch -p1 < sglang_patch/02-compressed_tensors_w4a4_nvfp4_moe.patch

Or bind-mount the two patched files over the image. Three edits total:

• layers/moe/moe_runner/flashinfer_cutlass.py — add a swiglu_limit field to FlashInferCutlassMoeQuantInfo and pass it into the fused-MoE call. The CUTLASS kernel already accepts swiglu_limit; the sibling mxfp4 quant info passes it and the NVFP4 one simply never did. Both call the same flashinfer_cutlass_fused_moe.
• .../compressed_tensors_w4a4_nvfp4_moe.py — build the clamp tensor from moe_runner_config.gemm1_clamp_limit (Bailing sets that, not swiglu_limit) and pass it. The same file's TRTLLM branch gets the hard-coded None fixed too.

Verify the clamp is live

The patch logs once per layer. You want to see this:

[swiglu-clamp-patch] cutlass nvfp4 limit=None    <- layers 0–34
[swiglu-clamp-patch] cutlass nvfp4 limit=4       <- layers 35–41

If you never see limit=4, the clamp is not being applied and your output will be corrupted. Check that first before blaming anything else.

Do not use the TRTLLM MoE backend with this model

--moe-runner-backend flashinfer_trtllm cannot run Ling at all. Bailing never sets routing_method_type, and once you supply it (RoutingMethodType.DeepSeekV3 — Ling's routing is sigmoid + expert bias, n_group 8, topk_group 4, top_k 8) the routing kernel still refuses the shape:

Routing kernel expects #experts per group <= warp size (32),
got 512 experts / 8 groups = 64 experts per group

512 experts over 8 groups is 64 per group. That is a compiled CUDA limit, not a config problem. Use the default CUTLASS backend — which is where the clamp fix lives anyway.

---

MTP (NEXTN) — how it works here and what to watch

Ling ships one MTP layer (num_nextn_predict_layers: 1, layer 42). It drafts tokens that the main model then verifies in a single batched pass, so you pay one forward pass for several accepted tokens.

Layer 42 is BF16 in this checkpoint — 0 packed tensors, 1552 plain. It must be excluded from the quantization targets. config.json here puts re:.*layers\.42\..* first in ignore.

This matters more than it sounds. If a blanket re:.*\.experts\..* target claims layer 42's experts are FP4 while the tensors are BF16, the draft head loads under the wrong scheme and drafts garbage. You get:

accept len: 1.00, accept rate: 0.00

and speculation becomes pure overhead — it gets worse the deeper you draft (I measured 0.94× at k=1, 0.80× at k=2, 0.71× at k=3 in that broken state). An acceptance rate of 0.00 means your draft head is broken, not that MTP is a bad idea for this model. Read the acceptance number before you conclude anything about speculative decoding.

With layer 42 loading correctly, acceptance is 0.81–0.85 and k=2 is the optimum:

configuration	tok/s	vs baseline	accept len	accept rate
speculation off	22.60	1.000×	—	—
NEXTN k=1	40.19	1.778×	1.85–1.95	0.85–0.95
NEXTN k=2	45.1	~2.0×	2.62–2.70	0.81–0.85
NEXTN k=3	40.48	1.791×	2.58–2.90	0.53–0.63

Deeper is not better. k=3 drafts more but accepts less, and the extra verify work eats the gain.

Speculation is not byte-identical

Under greedy, speculation-off is 5/5 self-reproducible, but NEXTN k=2 matches it byte-for-byte on only 1/5 prompts. The differences I saw were paraphrases, not corruption — floating-point non-associativity in the batched verify pass picking a different token at a near-tie.

With --enable-deterministic-inference that rises to 4/5 at 38.26 tok/s. That mode needs one more edit: raise MIN_DEEPGEMM_DIM in sglang/srt/batch_invariant_ops/batch_invariant_ops.py from 16 to 64, because the guard is N >= MIN_DEEPGEMM_DIM and Ling's N=32 MoE router otherwise dies with DeepGEMM failed for matrix shapes M=1, N=32, K=2560.

---

Quantization layout

component	precision
routed experts (mlp.experts.*, layers 0–41)	NVFP4 — e2m1 weights, FP8-e4m3 block scales, tensor_group, group size 16
attention q/k/v/o_proj, f_proj, g_proj, b_proj	BF16
q_conv1d, k_conv1d, v_conv1d, o_norm, A_log, dt_bias	BF16
MLA kv_a_proj_with_mqa, kv_b_proj, attention.dense	BF16
shared experts, mlp.gate (router), dense MLP (layers 0–1)	BF16
lm_head, word_embeddings, all norms	BF16
layer 42 (NEXTN/MTP), all 1552 tensors	BF16

f_proj and g_proj are the KDA decay and output gates. I keep them at BF16 — a 4-bit gate feeds an exponential decay whose error compounds along the sequence.

Weights 71.8 GB resident, 76 GB on disk, ~300 s to load.

input_global_scale tensors are included — do not strip them

SGLang's CompressedTensorsW4A4Nvfp4MoE allocates w13_input_global_scale and w2_input_global_scale with torch.empty() and fills them only from the checkpoint. A checkpoint without them loads with no error and no warning, leaves those parameters holding uninitialized memory, and computes g_alphas = (1 / w13_input_global_scale) * w13_weight_scale_2 from garbage. The result is NaN logits and an output of nothing but ! characters.

This repo ships 61,440 of them, set to 1.0 (float32), one beside every expert weight_global_scale. With dynamic: "local" activations the per-block scale is computed at runtime and the kernel applies alpha = 1/(g_a·g_w), so g_a = 1 leaves s_a_block = amax/6, comfortably inside FP8 e4m3 range.

These are uncalibrated. A calibrated activation scale may do better; I have not measured that.

Why W4A4 and not W4A16

SGLang implements exactly one NVFP4 MoE path — _is_fp4a4_nvfp4 → CompressedTensorsW4A4Nvfp4MoE. There is no weight-only branch: get_moe_scheme falls through _is_wNa16_group_channel (INT-typed only) to the w8a8 predicates and raises. A W4A16 checkpoint (input_activations: null) hits AttributeError: 'NoneType' object has no attribute 'num_bits' in _is_static_tensor_w8a8 — those predicates dereference input_quant with no None guard.

So config.json declares input_activations (4-bit, dynamic: "local", tensor_group) to select the W4A4 path. That is why the community W4A16 NVFP4 builds of this model will not serve on SGLang.

Serving

Requires an SGLang build with bailing_moe_v3 support, plus sglang_patch/.

python3 -m sglang.launch_server \
  --model-path <this-repo> --trust-remote-code \
  --host 0.0.0.0 --port 30013 \
  --tp-size 1 --ep-size 1 \
  --mem-fraction-static 0.75 \
  --context-length 32768 \
  --max-running-requests 1 \
  --chunked-prefill-size 2048 \
  --disable-shared-experts-fusion \
  --tool-call-parser glm45 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3

--disable-shared-experts-fusion is required — the shared experts are BF16 while the fused MoE expects FP4.

Do not pass --reasoning-parser. With one active, any request sending enable_thinking: false comes back with content: "" and the answer stranded in reasoning_content: the model emits no <think>…</think> wrapper, so the parser never sees a terminator and files the whole output as reasoning. Agentic harnesses that disable thinking will see empty replies.

Native context is 262144 (rope_scaling: null); 32768 above is just what I run.

Provenance

Quantized from the official BF16 release with llmcompressor 0.12.0.1 / compressed_tensors 0.17.1 via model_free_ptq, scheme NVFP4A16.

Sequential oneshot was not usable — compressed_tensors' from_accelerate path asserts on disk/meta tensors on this hardware. The Bailing MoE layout also requires the fused expert maps to be emptied, with MLA kept out through the ignore list instead.

Counts after quantization: 61,440 quantized expert tensors; 16 MLA tensors and all 1,552 layer-42 tensors left unpacked at BF16.

Credit

All credit to InclusionAI for Ling-3.0-flash. The SwiGLU clamp behaviour was first documented for the GGUF conversion path by raulvidis/Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF; the fix here is the equivalent for SGLang's NVFP4 CUTLASS path.

License

MIT, inherited from inclusionAI/Ling-3.0-flash.