Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64

KAT-Coder-V2.5-Dev — colibri int4 gs64 container (group-scaled)

colibri container for Kwaipilot/KAT-Coder-V2.5-Dev, a coding model that shares the Qwen3.6-35B-A3B architecture (qwen3_5_moe, 35B total / 3B active, 40 layers, 256 experts top-8, hybrid Gated Attention + Gated DeltaNet). Experts are quantized to group-scaled int4 (expert_gs=64, one f32 scale per 64 input elements per row, about 22 GB), the higher-accuracy variant of colibri's int4 format: on the sibling base model the same pipeline cut first-token logit error by ~44 % vs per-row scaling (see qwen36-35b-a3b-colibri-i4-gs64). Self-contained: bundled tokenizer.json, flat config.json.

This runs a 35B-class coding MoE interactively on one or two 8 GB GPUs plus ~30–40 GB RAM (see numbers below).

⚠️ Two KAT-Coder-V2.5 versions exist — don't mix up the scores

Kwaipilot's paper evaluates the commercial flagship KAT-Coder-V2.5; the open weights (and therefore this container) are KAT-Coder-V2.5-Dev, a separate open release. Reported numbers differ substantially:

	KAT-Coder-V2.5 (commercial flagship)	KAT-Coder-V2.5-Dev (open weights, this container)
Source	paper evaluation	HuggingFace open weights
Architecture	MoE (undisclosed)	MoE (qwen3_5_moe)
Parameters	undisclosed	35B total / 3B active
Base model	undisclosed	Qwen3.6-35B-A3B
Modality	includes vision/multimodal	text-only
Context	unspecified	262,144 tokens (YaRN up to ~1M)
License	commercial, closed	Apache 2.0
SWE-Bench Pro	65.2	45.96
KAT Code Bench	53.1	46.21
PinchBench	94.9	93.43
Terminal-Bench 2.1	60.7	41.02
SciCode	50.3	44.20
SWE-Bench Verified	not reported	69.40
SWE-Bench Multilingual	not reported	63.00

(Scores as reported by Kwaipilot for the respective version.)

Which commits you need

The qwen3.6 engine is not merged upstream yet; pick the branch matching your setup (all on github.com/kreuzzelg/colibri):

Setup	Branch / commit	Upstream status
CPU, gs64 (this container)	gs64-ab (2f9aab2) — engine + group-scaled GEMV matmul_q_gs	engine submitted as #712
CUDA VRAM expert tier + gs64 (recommended, 1–2 GPUs)	gs64-gpu (de5dde7) = gs64 tier upload (90450ec) + upstream-dev backend incl. the async grouped-int4 fix	tier submitted as #713; backend fix as #762
Per-row containers on the plain engine	upstream-pr1-qwen36-engine (16b945a)	= #712

The backend fix #762 is required for correct gs64 output on the GPU tier — without it the async decode path silently applies per-row scales to grouped containers. Validated on this exact combination: CPU-vs-GPU-tier logits cosine 1.0 over 24 greedy tokens, token-identical output.

Run it

git clone -b gs64-gpu https://github.com/kreuzzelg/colibri && cd colibri/c
# GPU tier (CUDA):
nvcc -O3 -std=c++17 -arch=native -c backend_cuda.cu -o backend_cuda.o
gcc -O3 -march=x86-64-v3 -fopenmp -pthread qwen36.c qwen36_tier.c vulkan_gemv.c \
    backend_cuda.o -o qwen36 -lm -lcudart -lstdc++

SNAP=$(python -c "from huggingface_hub import snapshot_download; \
print(snapshot_download('Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64'))")
printf 'Write a Python function that returns the n-th Fibonacci number using memoization.\n' > prompt.txt
SNAP=$SNAP N_NEW=300 COLI_CUDA=1 HEAT_FILE=heat.bin \
  OMP_NUM_THREADS=<physical cores> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
  ./qwen36 256 4 prompt.txt
# CPU only: build from the gs64-ab branch with `make -C c qwen36` and drop COLI_CUDA.

The engine reads expert_gs from qwen36_meta.json and picks the group-scaled GEMV automatically. HEAT_FILE persists expert-routing heat: the second run starts with the right experts in VRAM.

Measured (Threadripper PRO 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)

	2×8 GB GPUs	1×8 GB GPU (Quadro)
tok/s cold / warm	10.4 / 11.3	9.3 / 10.4
VRAM hit rate cold / warm	75 % / 100 %	39 % / 97 %
VRAM used	13.2 GiB total	6.6 GiB
peak RSS	30 GB	40 GB
TTFT (49-token prompt)	1.9 s	2.1 s

Verified qualitatively: correct, clean Python (memoization, docstring, type hints, edge cases) on the Fibonacci prompt; coherent 200-token continuations under the benchmark prompt.

Caveats

• Dev checkpoint: for serious use, handle the chat template and EOS yourself — in raw-completion mode the model keeps writing after the code.
• The 44 % accuracy A/B was measured on the sibling Qwen3.6-35B-A3B base with the same converter/engine; no separate A/B was run for the KAT finetune.

Credits

• Model: Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0), a finetune of Qwen/Qwen3.6-35B-A3B
• colibri engine & concept: JustVugg/colibri
• Original qwen36 engine + converter: @minne100 (PR #602); group-scaled quantization, CUDA VRAM tier and the grouped-int4 backend fix added on top.