File size: 4,157 Bytes
87acb41 bc0af79 87acb41 bc0af79 87acb41 bc0af79 87acb41 07be91a 87acb41 07be91a 87acb41 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
---
license: other
license_name: modified-mit
license_link: https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905/blob/main/LICENSE
library_name: mlx
tags:
- mlx
pipeline_tag: text-generation
base_model: moonshotai/Kimi-K2-Instruct-0905
---
. . .
# UPLOADING FILES ...
---
# mlx-community/Kimi-K2-Instruct-0905-mlx-DQ3_K_M
This model [mlx-community/Kimi-K2-Instruct-0905-mlx-DQ3_K_M](https://huggingface.co/mlx-community/Kimi-K2-Instruct-0905-mlx-DQ3_K_M) was
converted to MLX format from [moonshotai/Kimi-K2-Instruct-0905](https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905)
using mlx-lm version **0.26.3**.
This is created for people using a single Apple Mac Studio M3 Ultra with 512 GB. The 4-bit version of Kimi K2 does not fit. Using research results, we aim to get 4-bit performance from a slightly smaller and smarter quantization. It should also not be so large that it leaves no memory for a useful context window.
```bash
pip install mlx-lm
mlx_lm.generate --model mlx-community/Kimi-K2-Instruct-0905-mlx-DQ3_K_M --temp 0.6 --min-p 0.01 --max-tokens 4096 --trust-remote-code --prompt "Hallo"
```
---
## What is this DQ3_K_M?
In the Arxiv paper [Quantitative Analysis of Performance Drop in DeepSeek Model Quantization](https://arxiv.org/abs/2505.02390) the authors write,
> We further propose `DQ3_K_M`, a dynamic 3-bit quantization method that significantly outperforms traditional `Q3_K_M` variant on various benchmarks, which is also comparable with 4-bit quantization (`Q4_K_M`) approach in most tasks.
and
> dynamic 3-bit quantization method (`DQ3_K_M`) that outperforms the 3-bit quantization implementation in `llama.cpp` and achieves performance comparable to 4-bit quantization across multiple benchmarks.
The resulting multi-bitwidth quantization has been well tested and documented.
---
## How can you create your own DQ3_K_M quants?
In the `convert.py` file of mlx-lm on your system ( [you can see the original code here](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/convert.py) ), replace the code inside `def mixed_quant_predicate()` with something like
```python
index = (
int(path.split(".")[layer_location])
if len(path.split(".")) > layer_location
else 0
)
# Build a mixed quant like "DQ3" of Arxiv paper https://arxiv.org/abs/2505.02390
# Quantitative Analysis of Performance Drop in DeepSeek Model Quantization
q_bits = 4
if "lm_head" in path:
q_bits = 6
#if "tokens" in path:
# q_bits = 4
if "attn.kv" in path:
q_bits = 6
#if "o_proj" in path:
# q_bits = 4
#if "attn.q" in path:
# q_bits = 4
# For all "mlp" and "shared experts"
if "down_proj" in path:
q_bits = 6
#if "up_proj" in path:
# q_bits = 4
#if "gate_proj" in path:
# q_bits = 4
# For "switch experts"
if "switch_mlp.up_proj" in path:
q_bits = 3
if "switch_mlp.gate_proj" in path:
q_bits = 3
if "switch_mlp.down_proj" in path:
q_bits = 3
# Blocks up to 5 are higher quality
if index < 5:
q_bits = 6
# Every 5th block is "medium" quality
if (index % 5) == 0:
q_bits = 4
#print("path:", path, "index:", index, "q_bits:", q_bits)
return {"group_size": group_size, "bits": q_bits}
```
Should you wish to squeeze more out of your quant, and you do not need to use a larger context window, you can change the last part of the above code to
```python
if "switch_mlp.down_proj" in path:
q_bits = 4
# Blocks up to 5 are higher quality
if index < 5:
q_bits = 6
#print("path:", path, "index:", index, "q_bits:", q_bits)
return {"group_size": group_size, "bits": q_bits}
```
Then create your DQ3_K_M quant with
```bash
mlx_lm.convert --hf-path moonshotai/Kimi-K2-Instruct-0905 --mlx-path your-model-DQ3_K_M -q --quant-predicate mixed_3_4 --trust-remote-code
```
---
Enjoy!
|