JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-MXFP6mix-vllm

Qwopus3.6-35B-A3B-v1 — PrismaQuant 5.06-bit MXFP6mix (compressed-tensors, vLLM)

Mixed-precision quantization of Jackrong/Qwopus3.6-35B-A3B-v1 (Qwen3.6 MoE, 35B total / ~3B active, hybrid linear+full attention, in-checkpoint MTP head) produced with PrismaQuant (spark-improvements fork, streaming production-render pipeline).

This is a experimental build with MXFP6 and exists only for testing and demostration purpose.

Same 5.06-bit budget as the sibling PrismaQuant-5.06bit-vllm, but with MXFP6_E2M3 added to the allocator menu (NVFP4 / MXFP6_E2M3 / MXFP8_E4M3 / BF16). On measured per-Linear error the allocator displaced MXFP8 entirely — fp6 dominates fp8 as the escalation tier at this budget.

• Selected point: 4.981 bpp (slightly under the 5.06 target)
• Mix (packed MoE expert groups): 168 NVFP4 / 22 MXFP6_E2M3 / 2 BF16; MTP head preserved in BF16; visual encoder uniform NVFP4
• Calibration: diverse-v1 (256 rows x ~4096 tok; 40% prose / 20% code / 20% math / 20% multilingual), GPTQ + static act order + joint NVFP4 fused-sibling global scales
• Size: 26 GB (vs 72 GB BF16 source)
• Quality (same harness, kv fp8, MTP k=3): tools 6/6; MTP draft acceptance 67.8-68.9% vs 67.2% for the MXFP8-escalation sibling — acceptance scales with expert precision across this artifact family

Serving — requires the PrismaQuant MXFP6 vLLM overlay

Stock vLLM (as of 0.22.1) has no CUDA compressed-tensors scheme for mxfp6-*-pack-quantized and fails with No compressed-tensors compatible scheme was found. Serve with the vllm_patches/mxfp6/ overlay from the PrismaQuant fork (a Python-only, anchor-verified patch + derived Docker image):

# Tier-2 (recommended): fp6 expert groups stay packed on GPU (~6.5 bpp
# residency for the fp6 share) and dequantize inline in Triton fused-MoE
# kernels. Bit-identical math to Tier-1; faster and smaller.
PRISMAQUANT_MXFP6_TIER2=1 vllm serve \
  JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-MXFP6mix-vllm \
  --quantization compressed-tensors --kv-cache-dtype fp8 \
  --trust-remote-code --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

# Without the env knob the overlay serves Tier-1: fp6 groups are
# dequantized to BF16 once at load (BF16-class memory for that share).

Measured on a DGX Spark (GB10 / Blackwell sm_121, 121 GB unified memory), aggregate decode tok/s at concurrency 1/4/8:

serving mode	resident weights	TPS 1/4/8	MTP acc
Tier-1 (load-time dequant)	~31 GiB	57.9 / 128.9 / 187.8	67.8%
Tier-2 (packed + Triton inline dequant)	25.2 GiB	58.7 / 135.8 / 211.1	68.3%

The NVFP4 share runs on vLLM's native kernels in both modes; only the 22 fp6 expert groups change path.