caiovicentino1/Qwen3.5-9B-EOQ-v2

Qwen3.5-9B EOQ v2 (AWQ + Mixed-Bit + BitPacked)

5 GB download | PPL 7.05 | GPU dequant 4s | 10s total load | 45.8 tok/s

EOQ v2 combines four techniques for maximum compression with near-FP16 quality:

1. AWQ pre-scaling - protects important weight channels (scales saved per tensor)
2. Mixed-bit allocation - Q3 for MLP gate/up, Q4 for MLP down, Q5 for QKV, Q6 for O-proj
3. Bit-packing - actual N-bit storage (not int8)
4. GPU-accelerated dequant - 4s load on GPU (100x faster than CPU)

Benchmark (RTX PRO 6000 Blackwell)

Metric	FP16	EOQ v1	EOQ v2
Download	17.9 GB	4.93 GB	~5 GB
Load time	~53s	~27s	10s
PPL	6.37	7.31 (+0.94)	7.05 (+0.68)
tok/s	45.7	45.8	45.8
VRAM	17.9 GB	17.9 GB	17.9 GB

AWQ reduced the PPL delta by 28% vs v1 (0.94 to 0.68). GPU dequant is 100x faster than CPU numpy (4s vs 437s). Load time is 5x faster than FP16 (10s vs 53s).

Quick Start

from huggingface_hub import snapshot_download
import sys

local = snapshot_download("caiovicentino1/Qwen3.5-9B-EOQ-v2")
sys.path.insert(0, local)
from eoq_loader import load_eoq_model

model, tokenizer = load_eoq_model("caiovicentino1/Qwen3.5-9B-EOQ-v2")
# Download ~5 GB -> GPU dequant 4s -> Ready in 10s!

inputs = tokenizer("Hello!", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

How It Works

Quantization (one-time)

FP16 weights -> AWQ scale (protect important channels)
             -> Mixed-bit quantize (Q3-Q6 by tensor type)
             -> Bit-pack (actual N-bit storage)
             -> Save codes + quant scales + AWQ scales

Loading (every run)

Download ~5 GB -> Load safetensors to GPU
              -> Unpack N-bit codes (GPU, 100x faster than CPU)
              -> Dequantize: codes * quant_scales
              -> Undo AWQ: divide by AWQ scales
              -> FP16 model ready in 10s

Bit Allocation

Tensor Type	Bits	Params	Share
MLP gate/up	Q3	3,221M	36%
MLP down	Q4	1,611M	18%
Attn Q/K/V + embed	Q5	2,567M	29%
Attn O + lm_head	Q6	1,554M	17%
Norms/biases	FP16	0.3M	0%

Links

• GitHub: https://github.com/caiovicentino/eoq-quantization
• Base model: Qwen/Qwen3.5-9B
• v1 (without AWQ): Qwen3.5-9B-EOQ-Dynamic-BitPacked