Update README.md

README.md

@@ -1,5 +1,123 @@
----
-license: other
-license_name: modified-mit
-license_link: https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905/blob/main/LICENSE
----
+---
+license: other
+license_name: modified-mit
+license_link: https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905/blob/main/LICENSE
+library_name: mlx
+tags:
+- mlx
+pipeline_tag: text-generation
+base_model: moonshotai/Kimi-K2-Instruct-0905
+---
+
+. . .
+
+# UPLOADING FILES ...
+
+---
+
+# mlx-community/moonshotai_Kimi-K2-Instruct-0905-mlx-DQ3_K_M
+
+This model [mlx-community/moonshotai_Kimi-K2-Instruct-0905-mlx-DQ3_K_M](https://huggingface.co/mlx-community/moonshotai_Kimi-K2-Instruct-0905-mlx-DQ3_K_M) was
+converted to MLX format from [moonshotai/Kimi-K2-Instruct-0905](https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905)
+using mlx-lm version **0.26.3**.
+
+---
+
+## Who is this for?
+
+This is created for people using a single Apple Mac Studio M3 Ultra with 512 GB. The 4-bit version of Kimi K2 does not fit. Using research results, we aim to get 4-bit performance from a slightly smaller and smarter quantization. It should also not be so large that it leaves no memory for a useful context window.
+
+---
+
+## Use this model with mlx
+
+```bash
+pip install mlx-lm
+
+mlx_lm.generate --model mlx-community/moonshotai_Kimi-K2-Instruct-0905-mlx-DQ3_K_M --temp 0.6 --min-p 0.01 --max-tokens 4096 --trust-remote-code --prompt "Hallo"
+```
+
+---
+
+## What is this DQ3_K_M?
+
+In the Arxiv paper [Quantitative Analysis of Performance Drop in DeepSeek Model Quantization](https://arxiv.org/abs/2505.02390) the authors write,
+
+> We further propose `DQ3_K_M`, a dynamic 3-bit quantization method that significantly outperforms traditional `Q3_K_M` variant on various benchmarks, which is also comparable with 4-bit quantization (`Q4_K_M`) approach in most tasks.
+
+and
+
+> dynamic 3-bit quantization method (`DQ3_K_M`) that outperforms the 3-bit quantization implementation in `llama.cpp` and achieves performance comparable to 4-bit quantization across multiple benchmarks.
+
+The resulting multi-bitwidth quantization has been well tested and documented.
+
+---
+
+## How can you create your own DQ3_K_M quants?
+
+In the `convert.py` file of mlx-lm on your system ( [you can see the original code here](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/convert.py) ), replace the code inside `def mixed_quant_predicate()` with something like
+
+```python
+        index = (
+            int(path.split(".")[layer_location])
+            if len(path.split(".")) > layer_location
+            else 0
+        )
+        # Build a mixed quant like "DQ3" of Arxiv paper https://arxiv.org/abs/2505.02390
+        #    Quantitative Analysis of Performance Drop in DeepSeek Model Quantization
+        q_bits = 4       
+        if "lm_head" in path:
+           q_bits = 6
+        #if "tokens" in path:
+        #   q_bits = 4
+        if "attn.kv" in path:
+           q_bits = 6
+        #if "o_proj" in path:
+        #   q_bits = 4
+        #if "attn.q" in path:
+        #   q_bits = 4
+        # For all "mlp" and "shared experts"
+        if "down_proj" in path:
+           q_bits = 6
+        #if "up_proj" in path:
+        #   q_bits = 4
+        #if "gate_proj" in path:
+        #   q_bits = 4
+        # For "switch experts"
+        if "switch_mlp.up_proj" in path:
+           q_bits = 3
+        if "switch_mlp.gate_proj" in path:
+           q_bits = 3
+        if "switch_mlp.down_proj" in path:
+           q_bits = 3
+           # Blocks 3 and 4 are higher quality
+           if (index == 3) or (index == 4):
+              q_bits = 6
+           # Every 5th block is "medium" quality
+           if (index % 5) == 0:
+              q_bits = 4
+        #print("path:", path, "index:", index, "q_bits:", q_bits)
+        return {"group_size": group_size, "bits": q_bits}
+```
+
+Should you wish to squeeze more out of your quant, and you do not need to use a larger context window, you can change the last part of the above code to
+
+```python
+        if "switch_mlp.down_proj" in path:
+           q_bits = 4
+           # Blocks 3 and 4 are higher quality
+           if (index == 3) or (index == 4):
+              q_bits = 6
+        #print("path:", path, "index:", index, "q_bits:", q_bits)
+        return {"group_size": group_size, "bits": q_bits}
+```
+
+Then create your DQ3_K_M quant with
+
+```bash
+mlx_lm.convert --hf-path moonshotai/Kimi-K2-Instruct-0905 --mlx-path your-model-DQ3_K_M -q --quant-predicate mixed_3_4 --trust-remote-code
+```
+
+---
+
+Enjoy!