Configure the model in Pi
# Install Pi:
npm install -g @mariozechner/pi-coding-agent# Add to ~/.pi/agent/models.json:
{
"providers": {
"llama-cpp": {
"baseUrl": "http://localhost:8080/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [
{
"id": "aj9o9/Ling-3.0-flash-GGUF:Q4_K_M"
}
]
}
}
}Run Pi
# Start Pi in your project directory:
piLing-3.0-flash - GGUF (Q4_K_M)
GGUF quantization of inclusionAI/Ling-3.0-flash, a 124B parameter Mixture-of-Experts model with 5.1B active parameters per token.
A note from me
This is my first quantization, and my first time patching llama.cpp. I did it to learn, and I learned a lot - including that I got part of it wrong (see the warning below). I am sharing it because a working GGUF of this model did not exist when I started, not because I think it is authoritative.
Please report anything you find. Bad output, crashes, wrong metadata, better flags for a particular card - open a discussion on this repo or reach me at @ItsmeAjayKV. Suggestions are as welcome as bug reports.
On the llama.cpp side: I am not planning to submit this fork upstream. The architecture support was written with heavy AI assistance and the model is very new, so PR #26608 is the right place for upstream support to come from - it is independent work by someone else and, at least on the KDA gate, more correct than mine. My branch exists so these files have something to run on, and so I could understand how the pieces fit together.
This will not load in upstream llama.cpp. Ling 3.0 uses a hybrid KDA + MLA attention architecture that upstream does not yet support. These files declare the architecture
bailing-hybrid, which currently exists only in the fork linked below. Loading them with a standard llama.cpp build fails withunknown model architecture.
Known correctness issue - fix in progress. The KDA forget gate in this build uses the wrong branch of the reference implementation. Ling 3.0 sets
kda_safe_gate: true, which selectslower_bound * sigmoid(exp(A_log) * (g + bias)), but this build applies-exp(A_log) * softplus(g + bias)followed by a hard clamp. That affects 35 of the 42 layers.Output is fluent and coherent, so the problem is not obvious from casual use, but quality is likely degraded relative to the reference implementation. Treat this as a preview. PR #26608 implements the gate correctly.
Files
| File | Size | Quant | Bits/weight |
|---|---|---|---|
Ling-3.0-flash-Q4_K_M-00001-of-00002.gguf |
41.5 GB | Q4_K_M | 4.84 |
Ling-3.0-flash-Q4_K_M-00002-of-00002.gguf |
37.4 GB | Q4_K_M | 4.84 |
Total 75.3 GB. Download both shards into the same directory and point -m at the first one -
llama.cpp finds the second automatically. Do not rename them.
Quantized from bf16 without an importance matrix (imatrix).
Usage
Build the fork:
git clone -b ling-3.0-bailing-hybrid https://github.com/Ajay9o9/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-completion
Run it. The model is 70 GiB, so unless you have that much VRAM, offload the experts to CPU with
--n-cpu-moe - only 8 of 512 experts are read per token, so this costs less than it sounds:
./build/bin/llama-completion \
-m Ling-3.0-flash-Q4_K_M-00001-of-00002.gguf \
-p "Write a haiku about linear attention." \
-ngl 99 --n-cpu-moe 42 -n 100
Lower the --n-cpu-moe number to keep more experts on the GPU if you have VRAM spare.
On a 96 GB card with everything resident, this quant runs at roughly 113 tok/s.
Hardware notes
I made this quant with consumer cards in mind - specifically a 3060 (12 GB) and a 3090 (24 GB), paired with 64 GB of system RAM. Because only 8 of 512 experts run per token, pushing the expert weights to CPU costs far less throughput than the size suggests.
| Setup | Expect |
|---|---|
| 3090 (24 GB) + 64 GB RAM | Workable. Use --n-cpu-moe 42, lower it if VRAM allows. |
| 3060 (12 GB) + 64 GB RAM | Tight but runs. Keep --n-cpu-moe 42 and expect a slow prompt phase. |
| 64 GB RAM, no GPU | Possible via mmap, but slow. |
At 70 GiB the file is larger than 64 GB of RAM, so some of it is served from disk through mmap regardless. An SSD makes a real difference here; on a spinning disk it will crawl. Keep the model on your fastest drive.
If you have less RAM than this, a smaller quant (IQ3 or below) would suit you better than this one. I have not made those yet.
About the model
- 42 layers: 35 KDA (linear attention, recurrent state) interleaved 5:1 with 7 gated MLA layers
- 512 routed experts + 1 shared, 8 experts active per token, group-limited
noaux_tcrouting - 256K context. Only the 7 MLA layers keep a KV cache, and they store a 576-value compressed latent per token rather than full per-head keys and values, so the cache stays small at long context.
See the original model card for benchmarks and training details. Those numbers were measured on the full bf16 model with the reference implementation and do not describe this quantization.
Links
- Original model: inclusionAI/Ling-3.0-flash
- Fork required to run this: Ajay9o9/llama.cpp @ ling-3.0-bailing-hybrid
- Upstream request: ggml-org/llama.cpp#26590
- Alternative implementation: ggml-org/llama.cpp#26608
- Reference implementation: inclusionAI/vllm-ling-v3
License
MIT, inherited from the original model.
- Downloads last month
- 6
4-bit
Model tree for aj9o9/Ling-3.0-flash-GGUF
Base model
inclusionAI/Ling-3.0-flash
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp# Start a local OpenAI-compatible server: llama serve -hf aj9o9/Ling-3.0-flash-GGUF:Q4_K_M