btbtyler09/Qwen3.6-35B-A3B-GPTQ-8bit

Qwen3.6-35B-A3B GPTQ 8-bit

GPTQ 8-bit quantization of Qwen/Qwen3.6-35B-A3B, a 35B-parameter Mixture-of-Experts (MoE) multimodal model with 3B activated parameters per token.

Includes full vision encoder and MTP (Multi-Token Prediction) module for image understanding and speculative decoding support.

Model Overview

• Architecture: Qwen3_5MoeForConditionalGeneration (multimodal: text + vision; same architecture as Qwen3.5)
• Total parameters: ~35B
• Activated parameters: ~3B per token (8 of 256 experts selected per token)
• Layers: 40 (30 linear attention + 10 full attention, repeating 3:1 pattern)
• Experts: 256 per layer + 1 shared expert per layer
• Context length: 262,144 tokens
• Vision encoder: 27-block ViT (1152 hidden, 16x16 patches), BF16
• MTP module: 1-layer speculative decoding head, BF16

Quantization Details

All 30,720 MoE expert modules (256 experts x 3 projections x 40 layers) are quantized to INT8 using GPTQ. Non-expert modules (including the full vision encoder and MTP module) remain at BF16/FP16 for quality preservation.

Component	Precision	Notes
MoE experts (gate_proj, up_proj, down_proj)	INT8 (GPTQ)	30,720 modules quantized
Full attention (q_proj, k_proj, v_proj, o_proj)	FP16	Every 4th layer
Linear attention (in_proj_qkv, in_proj_z, out_proj)	FP16	Full precision
Shared experts	FP16	Full precision
Vision encoder (model.visual.*)	BF16	333 tensors, full precision
MTP module (mtp.*)	BF16	785 tensors, full precision
Embeddings, LM head, norms	FP16	Full precision

GPTQ configuration:

• Bits: 8
• Group size: 32
• Symmetric: Yes
• desc_act: No
• true_sequential: Yes
• act_group_aware: Yes
• Failsafe: RTN for poorly-calibrated rare experts (1,305 of 30,720 modules, ~4.25%)

Calibration

• Dataset: Mixed - evol-codealpaca-v1 (code) + C4 (general text)
• Samples: 2,048
• Quantizer: GPTQModel v5.7.1

Model Size

Version	Size	Compression
BF16 (original)	67 GB	-
GPTQ 8-bit	40 GB	1.7x
GPTQ 4-bit	25 GB	2.7x

Perplexity

Evaluated on wikitext-2-raw-v1 (test set), seq_len=2048, stride=512:

Model	Perplexity	Degradation
BF16 (original)	6.0567	-
GPTQ 8-bit	6.0543	-0.04% (within noise; effectively lossless)
GPTQ 4-bit	6.1070	+0.83%

Usage

vLLM (Recommended for Serving)

vllm serve btbtyler09/Qwen3.6-35B-A3B-GPTQ-8bit \
  --gpu-memory-utilization 0.95 \
  --max-model-len 256000 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --dtype float16 \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image": 2}'

Parameter	Description
--gpu-memory-utilization 0.95	Use 95% of GPU VRAM for KV cache + weights
--max-model-len 256000	Full 256K context window support
--tensor-parallel-size 4	Shard across 4 GPUs (adjust to your setup)
--reasoning-parser qwen3	Enable thinking/reasoning token parsing
--enable-auto-tool-choice --tool-call-parser qwen3_coder	Enable tool/function calling
--dtype float16	Run in FP16 (required for ROCm GPTQ kernels)
--skip-mm-profiling	Skip multimodal memory profiling at startup
--limit-mm-per-prompt '{"image": 2}'	Allow up to 2 images per request

vLLM bug workaround (may apply): Up through at least vLLM 0.15.2, Qwen3_5MoeTextConfig defines ignore_keys_at_rope_validation as a list instead of a set, causing a TypeError during config parsing. Since Qwen3.6 reuses the same qwen3_5_moe model_type, the same fix applies. Apply this patch before serving if you hit the error:

python3 -c "
for f in [
    '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5_moe.py',
    '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5.py',
]:
    t = open(f).read()
    t = t.replace(
        'ignore_keys_at_rope_validation\"] = [\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        ]',
        'ignore_keys_at_rope_validation\"] = {\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        }')
    open(f,'w').write(t)
    print('Patched', f)
"

Vision Example (via OpenAI API)

import base64, requests

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "btbtyler09/Qwen3.6-35B-A3B-GPTQ-8bit",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        {"type": "text", "text": "Describe what you see in this image."},
    ]}],
    "max_tokens": 1024,
})
print(response.json()["choices"][0]["message"]["content"])

GPTQModel / transformers

Note: Neither GPTQModel nor transformers can currently load this model directly. GPTQModel's Qwen3_5MoeGPTQ class expects the text-only weight prefix (model.layers.*) and does not support the multimodal architecture (model.language_model.layers.*). The transformers GPTQ path delegates to optimum, which does not handle the fused-expert architecture. Use vLLM for inference.

Technical Notes

Qwen3.6-35B-A3B is architecturally identical to Qwen3.5-35B-A3B (same Qwen3_5MoeForConditionalGeneration class, model_type=qwen3_5_moe, 256 experts/layer, 40 layers, hybrid linear+full attention pattern). The same GPTQModel definition and expert converter handle both checkpoints without modification.

MoE expert weights are stored as fused 3D nn.Parameter tensors in the source model rather than individual nn.Linear modules. During quantization, GPTQModel's MODULE_CONVERTER_MAP converts these to individual quantizable nn.Linear layers. This same conversion must also run during model loading for the quantized kernels to be applied correctly.

The vision encoder (27-block ViT) and MTP speculative decoding module are preserved at full BF16 precision from the original model. Only the text model's MoE expert weights are quantized.

Credits

• Base Model: Qwen - Qwen3.6-35B-A3B
• Quantization: GPTQ via GPTQModel v5.7.1
• Expert Converter: convert_qwen3_5_moe_expert_converter for fused 3D expert weights (shared with Qwen3.5)
• Quantized by: btbtyler09

License

This model inherits the Apache 2.0 license from the base model.