minicpm5-1b-optiq-5bpw-mlx

MLX quantization of openbmb/MiniCPM5-1B for Apple Silicon.

Variant: OptiQ mixed-precision (target 5.0 bpw)
Disk size: 819 MB
Quantized by: sahilchachra

About this quantization

Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

1. A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
2. The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
3. The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

• Method: optiq_mixed_precision (mlx-optiq)
• Target bits/weight: 5.0
• Achieved bits/weight: 5.279
• Candidate bits: [3, 4, 6, 8]
• Group size: 64
• Sensitivity reference: bf16
• Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Per-layer bit allocation

169 model components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Bits	Components	Share
8-bit	17	10.1%
6-bit	47	27.8%
4-bit	103	60.9%
3-bit	2	1.2%
Total	169	100.0%

Components kept at 8-bit (most sensitive to quantization):

• lm_head
• model.layers.23.mlp.up_proj
• model.layers.23.mlp.down_proj
• model.layers.23.mlp.gate_proj
• model.layers.23.self_attn.o_proj
• model.layers.23.self_attn.v_proj
• model.layers.23.self_attn.k_proj
• model.layers.23.self_attn.q_proj
• model.layers.2.self_attn.v_proj
• model.layers.1.self_attn.v_proj
• model.layers.0.mlp.up_proj
• model.layers.0.mlp.down_proj
• model.layers.0.mlp.gate_proj
• model.layers.0.self_attn.o_proj
• model.layers.0.self_attn.v_proj
• model.layers.0.self_attn.k_proj
• model.layers.0.self_attn.q_proj

Notice the pattern: lm_head, the first transformer block, and the last transformer block — these layers carry the most information that downstream tokens depend on, so OptiQ preserves them at high precision while compressing the middle of the network more aggressively.

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	Naive 8-bit	FP16 baseline
Decode tok/s (steady-state)	329.21	243.83	144.04
Prefill tok/s (steady-state)	1322.74	1297.85	1005.67
Decode tok/s (avg, long traces)	294.99	87.97	143.39
Peak memory (GB)	1.188	1.528	2.537
Disk size (MB)	819	1105	2071

Warmed, short-prompt, chat-templated, thinking disabled. Represents steady-state decode for typical chat use; long thinking traces will be slower due to KV-cache growth.

Quality

Benchmark	This model	Naive 8-bit	FP16 baseline	n
MATH-500 (math reasoning)	36.7% (answered 13/30)	60.0% (answered 22/30)	70.0% (answered 24/30)	30
IFEval (instruction following)	68.2%	70.5%	72.7%	44
HumanEval (code, pass@1)	66.7%	83.3%	83.3%	30

MATH-500 per-level accuracy

Level	This model	Naive 8-bit	FP16 baseline
level 1	16.7%	83.3%	83.3%
level 2	50.0%	83.3%	83.3%
level 3	50.0%	33.3%	50.0%
level 4	50.0%	66.7%	66.7%
level 5	16.7%	33.3%	66.7%

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	322.4
~256 tokens	319.0
~512 tokens	320.2
~1024 tokens	313.6

Limitations — degraded math reasoning

On MATH-500 with thinking enabled, this variant scores 36.7% vs 70.0% on the FP16 baseline. The model still produces real answers (answer-rate 43.3%), but the math-reasoning quality is noticeably lower than the FP16 reference. Code generation and instruction following are closer to baseline.

This is the standard tradeoff when targeting low bits-per-weight on a hybrid-thinking model: code and instruction-following are robust under quantization, but multi-step reasoning chains accumulate logit drift that costs accuracy.

Recommended usage:

• ✅ Code generation, chat assistant, instruction following
• ✅ Short single-step Q&A
• ⚠️ Heavy math reasoning — prefer the FP16 source or the naive 8-bit variant for that workload

To run in no-think mode (preferred for chat / non-reasoning use):

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "..."}],
    add_generation_prompt=True, enable_thinking=False, tokenize=False,
)

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/minicpm5-1b-optiq-5bpw-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256, verbose=True)

All variants in this collection

Model	Variant
sahilchachra/minicpm5-1b-8bit-mlx	Affine int8
sahilchachra/minicpm5-1b-optiq-5bpw-mlx	OptiQ mixed-precision (target 5.0 bpw) ← this model

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M5 Pro, 24 GB unified memory
• License: see openbmb/MiniCPM5-1B for the original model's license

Original model

See openbmb/MiniCPM5-1B for full model details and intended use.