Update README.md

README.md

@@ -10,28 +10,122 @@ tags:
   - exl3
 ---
 
-exllamav3 quantizations of [MiniMaxAI/MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5).
+[exllamav3](https://github.com/turboderp-org/exllamav3/) quantizations of [MiniMaxAI/MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5).
 
-### Optimized quants
+| Quant | Size | KLD | PPL |
+| --- | --- | --- | --- |
+| [2.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.00bpw_H6) | 61.054 GiB | 0.42365 | 9.31452 |
+| [2.10 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.10bpw_H6) (optimized) | 57.292 GiB | 0.36355 | 9.20850 |
+| [2.50 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.50bpw_H6) (optimized) | 67.838 GiB | 0.30152 | 8.88802 |
+| [3.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/3.00bpw_H6) | 81.613 GiB | 0.17263 | 8.58626 |
+| [3.06 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/3.06bpw_H6) (optimized) | 82.656 GiB | 0.15648 | 8.66856 |
+| [4.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/4.00bpw_H6) | 108.087 GiB | 0.07882 | 8.45404 |
+| [5.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/5.00bpw_H6) | 134.561 GiB | - | - |
 
-[2.10 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.10bpw_H6) 57.292 GiB    
-[2.50 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.50bpw_H6) 67.838 GiB    
-[3.06 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/3.06bpw_H6) 82.656 GiB    
+### K/L-D and PPL graphs
 
-### Straight quants
+![KLD Chart](MinMax25kld.png)
+![PPL Chart](MinMax25ppl.png)
 
-As the charts below will show, a 4bpw or 5bpw is still better than an optimized 3.06 or 2.5bpw quant.
+### How to create optimized quants
 
-[2.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/2.00bpw_H6) 61.054 GiB    
-[3.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/3.00bpw_H6) 81.613 GiB    
-[4.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/4.00bpw_H6) 108.087 GiB     
-[5.00 bpw h6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/tree/5.00bpw_H6) 134.561 GiB    
+It's possible to produce quants that are better for a given size than the ones you get by performing a quant directly to a given target bitrate. The process involves comparing two quants, measuring which modules are more affected by the quantization process, and selecting those modules first when targeting some in-between bitrate.
 
-### K/L-D and PPL graphs
+<details>
+<summary>Expand for more details</summary>
 
-![KLD Chart](MinMax25kld.png)
-![PPL Chart](MinMax25ppl.png)
+exllamav3 includes a measurement script `util/measure.py` that will compare two exllamav3 models module by module against the original model. The goal is to see which modules are the most affected by the decrease in precision involved in going from a larger quant to a smaller quant.
+
+The command is: 
+```bash
+python util/measure.py -l [level] -d [device] -ms [max_sys_memory] -i [path/to/quant1] [path/to/quant2] -r [path/to/original_model] -o [path/to/measurement.json]
+```
+Where:
+* `level` is an integer between 0 and 3 that determines the resolution of the measurement. 0 is fastest but least granular, 2 is default, 3 is most granular and slowest.
+* `device` is the index of the CUDA device that will perform the work
+* `max_sys_memory` is the amount of memory that can be used for state data to speed things up, in GiB
+* `path/to/quant1` and `path/to/quant2` are the paths to the two quants to compare
+* `path/to/original_model` is the path to the original model
+* `path/to/measurement.json` is the path to the resulting json measurement file
+
+The masurement fie I created above compared my 2.0bpw_H6 and my 3.0bpw_H6 quants.
+
+You can then feed this measurement file, along with the two quants, to `util/optimize.py` to create optimized quants that draw modules from both quants where appropriate to get the best result for a given bitrate.
+
+The command is:
+```bash
+python util/optimize.py -i [path/to/quant1] [path/to/quant2] -o [path/to/resulting_model] -m [path/to/measurement.json] -b [target_bitrate]
+```
+Where:
+* `path/to/quant1` and `path/to/quant2` are paths to the two source models
+* `path/to/resulting_model` is the output path
+* `target_bitrate` is the target bitrate as a number a decimal point
+
+You can use a measurement script from one pair of quants with another pair of quants of the same model. When I tried to use 2.0bpw and 4.0bpw quants to create a 2.25bpw quant, the size of the resulting model was larger than requested because of the substitution at 2.48 bpw, but it was still an improvement over a straight 2.48bpw quant. An explicitly-requested 2.48bpw quant drawing from the 2.0bpw and 3.0bpw quants proved to be even better (in terms of k/l divergence). Finally, I tried creating a 3.25bpw quant from 3.0bpw and 4.0bpw quants, still using my 2.0-vs-3.0 measurement file. This was not as successful as the optimized 2.25bpw quant, and may have benefitted from a 'correct' measurement file that matched the two actual sources.
+</details>
 
 [measurement.json - 2.0bpw_H6 vs 3.0bpw_H6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/blob/main/measurement_MiniMaxAI_MiniMax-M2.5-2.0-3.0.json)    
 [measurement.json - 3.0bpw_H6 vs 4.0bpw_H6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/blob/main/measurement_MiniMaxAI_MiniMax-M2.5-3.0-4.0.json)    
 [measurement.json - 4.0bpw_H6 vs 5.0bpw_H6](https://huggingface.co/MikeRoz/MiniMax-M2.5-exl3/blob/main/measurement_MiniMaxAI_MiniMax-M2.5-4.0-5.0.json)    
+
+### How to measure Perplexity and KL Divergence
+
+<details>
+<summary>Expand for details</summary>
+Measuring KL/D is a process that involves comparing the outputs of the quantized model to outputs of the original model. If the original model is too large for your hardware to load without quantization, you can run a script to generate logits which can then be passed into the comparison script, sparing you the need to load the whole source model.
+
+First, you'll need to create a dataset spec file. I based mine on `eval/spec/wiki2_llama3_large.json`.
+```json
+{
+    "tokenize_fn": "transformers",
+    "tokenizer_dir": "path/to/full_model",
+    "dataset": "wiki2",
+    "eval_stride": 512,
+    "eval_len": 2048,
+    "max_rows": 100
+}
+```
+
+I passed this into `eval/compare_q_logits.py` as follows:
+
+```bash
+python eval/compare_q_logits.py -m [path/to/full_model] -o [path/to/output_logits.safetensors] -d [path/to/dataset_spec.json] -rpb [rows_per_batch] -dev [device_index]
+```
+Where:
+* `path/to/full_model` is the path to the model
+* `path/to/output_logits.safetensors` is the path to the output logits file
+* `path/to/dataset_spec.json` is the path to the dataset spec file described above
+* `rows_per_batch` - I would run out of memory without this parameter. I set it to 32768.
+* `device_index` - optional CUDA device index
+
+Next, you'll need a model spec file that describes all the quants you want in the graph. You'll need to be able to load any model you'd like compared. Here's a sample of the one I used for these quants:
+```json
+[
+    {
+        "load_fn": "exllamav3",
+        "fwd_fn": "exllamav3",
+        "label": "EXL3 2.0bpw H6",
+        "model_dir": "path/to/MiniMaxAI_MiniMax-M2.5-2.0bpw-h6-exl3"
+    },
+    {
+        "load_fn": "exllamav3",
+        "fwd_fn": "exllamav3",
+        "label": "EXL3 2.1bpw H6 (optimized)",
+        "model_dir": "path/to/MiniMaxAI_MiniMax-M2.5-2.1bpw-h6-exl3"
+    }
+]
+```
+
+This spec file can be passed in to the following command:
+```bash
+python eval/compare_q.py -d [path/to/dataset_spec.json] -m [path/to/model_spec.json] -lf [path/to/logits.safetensors] -p [-kld] -t [chart_title]
+```
+Where:
+* `path/to/dataset_spec.json` is the path to the dataset spec file described above
+* `path/to/model_spec.json` is the path to the model spec file described above
+* `path/to/logits.safetensors` is the path to the full model's logits, created above
+* `-kld` the script creates a perplexity chart by default, add this if you want K/L-d instead
+* `chart_title` the chart title in the resulting plot
+
+Results are cached, so if the process crashes after processing one or more models, you just need to restart the script until every model has been tested (don't use the argument that clears the cache). Also note that if you're running this via SSH like me, you may not see anything - the script uses `plt.show()`. I hacked in an extra arg and a `plt.savefig()` install instead.
+</details>