initial commit

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+imatrix-*.dat filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,93 @@
+---
+quantized_by: ubergarm
+pipeline_tag: text-generation
+base_model: MiniMaxAI/MiniMax-M2.5
+base_model_relation: quantized
+license_name: modified-mit
+license_link: https://github.com/MiniMax-AI/MiniMax-M2.5/blob/main/LICENSE
+tags:
+- imatrix
+- conversational
+- minimax_m2
+- ik_llama.cpp
+---
+
+## WIP
+
+- [x] download and convert_hf_to_gguyf.py (automatically casts fp8 to bf16)
+- [ ] imatrix computation on full upcast bf16 gguf
+- [ ] upload imatrix dat for others to do custom quantizations
+- [ ] upload some logs/ files so others can see commands and early perplexity values for partial runs
+- [ ] cook q8_0 and test perplexity
+- [ ] fugure out initial custom quantization recipes
+- [ ] release IQ5_K
+- [ ] release smaller quants prioritizing based on what discussions are opened
+- [ ] release graph of perplexity across this quant collection
+
+## `ik_llama.cpp` imatrix Quantizations of MiniMaxAI/MiniMax-M2.5
+*NOTE* `ik_llama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.
+
+Some of ik's new quants are supported with [Nexesenex/croco.cpp](https://github.com/Nexesenex/croco.cpp) fork of KoboldCPP with Windows builds for CUDA 12.9. Also check for [Windows builds by Thireus here.](https://github.com/Thireus/ik_llama.cpp/releases) which have been CUDA 12.8.
+
+These quants provide best in class perplexity for the given memory footprint.
+
+## Big Thanks
+Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)!  **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!!
+
+Also thanks to all the folks in the quanting and inferencing community on [BeaverAI Club Discord](https://huggingface.co/BeaverAI) and on [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for tips and tricks helping each other run, test, and benchmark all the fun new models! Thanks to huggingface for hosting all these big quants!
+
+Finally, I *really* appreciate the support from [aifoundry.org](https://aifoundry.org) so check out their open source RISC-V based solutions!
+
+## Quant Collection
+Perplexity computed against *wiki.test.raw*. (lower is "better")
+
+![Perplexity Chart](images/perplexity.png "Chart showing Perplexity vs Model Size.")
+
+These two are just a test quants for baseline perplexity comparison and not available for download here:
+* `BF16` TODO
+  - TODO
+* `Q8_0` TODO
+  - TODO
+
+*NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine!
+
+## IQ5_K TODO
+TODO
+
+<details>
+
+<summary>👈 Secret Recipe</summary>
+
+```bash
+echo TODO
+```
+
+</details>
+
+## Quick Start
+
+```bash
+# Clone and checkout
+$ git clone https://github.com/ikawrakow/ik_llama.cpp
+$ cd ik_llama.cpp
+
+# Build for hybrid CPU+CUDA
+$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
+$ cmake --build build --config Release -j $(nproc)
+
+# Hybrid CPU and Single GPU
+echo TODO
+
+# Hybrid CPU and Multi GPU
+echo TODO
+
+# CPU-Only
+echo TODO
+```
+
+For tool use you can always bring your own template with `--chat-template-file myTemplate.jinja` and might need `--special` etc.
+
+## References
+* [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)
+* [Getting Started Guide (already out of date lol)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)
+* [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584)