mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

GLM-5.2 — colibri int4 g64 container (grouped scales, int8 MTP head)

GLM-5.2 (744B MoE) converted for the colibri streaming engine: fmt=4 grouped int4 (group size 64) expert weights with an int8 MTP head for speculative decoding.

Status (2026-07-31): fully GPU-validated; use colibri v1.3.0. The fmt=4 CUDA path merged upstream (colibri #298), the engine validates this container token-exact against the transformers oracle (32/32), and an independent quality A/B measured grouped int4 ahead of per-row int4 (hellaswag acc_norm 87.0% vs 83.5%, n=200 — colibri #326). This is the reference grouped-int4 container for the engine.

Independently reproduced as the fix for the reasoning-loop / EOS-starvation failure (colibri #455): a third-party 5-cell capture matrix — same engine, same prompts, same flags, only the container swapped from per-row int4 to this one — went 5/5 clean, including the TEMP=0.9 NUCLEUS=0.95 setting that was catastrophic on per-row. On this container the sampling knobs stop being safety-critical.

Contents

• out-00000..00140.safetensors — 141 shards, 390.5 GB, int4 gs=64 experts (per-group f32 scales), int8 embed/lm_head, f32 norms
• out-mtp-00000.safetensors — 9.3 GB, MTP head at int8 (int4 MTP heads give ~0% draft acceptance; int8 measures 39–59%)
• config.json, tokenizer.json, tokenizer_config.json, generation_config.json

Why g64 over the per-row int4 container

Per-row scales let one outlier poison a whole row's precision and are implicated in the repetition-attractor reports (colibri #225/#307). Grouped scales (one per 64 elements) cost ~12% more disk and measurably improve quality: hellaswag acc_norm 87.0% vs 83.5% per-row in the #326 A/B.

The same difference shows up as stopping reliability, not just accuracy. In the #455 container-swap matrix (5 sampling cells, engine and prompts held fixed), per-row int4 produced reasoning loops that never emitted EOS and ran to the token budget; this container stopped cleanly in all 5, worst case included. Run colibri v1.3.0 or later alongside it — v1.2.0 carried the upstream fix for the CUDA-path EOS-starvation bug and v1.3.0 adds the grouped-scale attention fix, and container plus engine together are what make the stop behaviour reliable.

Requirements

• colibri v1.3.0 (01abef3) recommended. The hard floor is commit c98e5f8 (the #298 merge, 2026-07-20) — earlier builds either exit without loading fmt=4, silently fall back to CPU, or — with CUDA_DENSE=1 — apply grouped scales per-row and produce garbage. Two later fixes matter enough to make v1.3.0 the version to use: v1.2.0 fixed the thinking-loop / EOS-starvation bug, and v1.3.0 fixed a ragged-attention kernel that applied per-row scales to this container's group scales, producing fluent-then-collapsing gibberish under COLI_CUDA_ATTN=1 in served mode (colibri #685 / #692). v1.3.0 also fixes an expert-cache corruption where slots could be evicted mid-computation.
• On any build before v1.3.0, do not set COLI_CUDA_ATTN=1 with this container. That is the ragged-attention path above. The failure is quiet and easy to misread — it hits prefill tiles, so answers begin as plausible text and then degrade into random tokens. Confirmed independently on 4×A100 and H100. Fixed in v1.3.0; harmless there.
• ~400 GB of local NVMe — on one drive, or split across several: COLI_MODEL_DIRS=<dir2>[;<dir3>...] (in dev since #469) reads a distinct subset of the shards from each drive, so two 250 GB drives can host it together. Metadata stays in the primary model dir.
• 16 GB+ RAM minimum; RAM is the dominant performance lever (see below).

Recommended config & measured performance

Reference box: RTX 5080 (sm_120, 16 GB) / Core Ultra 9 285K / 128 GB DDR5 / Windows 11, full K=8, greedy. Full 12-cell benchmark record: colibri #467 (raw logs linked there).

COLI_CUDA=1  CUDA_DENSE=1  CUDA_EXPERT_GB=4  CUDA_RESERVE_GB=1  PIPE=1  DIRECT=1
# balanced (recommended): + CACHE_ROUTE=1 ROUTE_J=3 ROUTE_M=16 ROUTE_ALPHA=0.5
# speed:                   + CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=24 ROUTE_ALPHA=0.5

config	decode tok/s	expert hit	notes
speed (CR M24)	1.41	97%	swap 23% — experimental routing, label results accordingly
balanced (CR J3/M16)	1.31	93%	swap 18%, route_agree 82% — recommended default
faithful (no CR, greedy-identical)	0.89–0.96	79–87%	0.96 with PILOT_EVICT_GUARD=0
same, without DIRECT=1/overlap	0.67	84%
true CPU (--gpu none), 128 GB RAM	0.62	77%
any compute, RAM capped to 25 GB	0.31–0.38	16–33%

The two CR rows were measured on colibri v1.3.0 (01abef3), warm-up run discarded, two replicates for balanced (1.29 / 1.32) and one for speed; run-to-run tok/s noise on this box is ~±0.04. Fidelity is unchanged from v1.1.1 (route_kl 2.90–2.98, route_agree ~82% on both). The lower rows are from the earlier #467 matrix (dev ≥ 2026-07-21: #495 tier fix, #497 eviction-guard fix, #477 AVX-512 MLA) and have not been re-run.

For reference, the same two cells measured 1.27 balanced / 1.38 speed on v1.1.1 with the same prompt and config, so v1.3.0 is at parity or marginally ahead — there is no version regression on this path. CACHE_ROUTE is the opt-in cache-aware routing from colibri #161 — not greedy-identical to true top-8 (quality evidence: MMLU A/B n.s. at n=200, colibri #161); label CR results as experimental.

Notes from the measurements:

• RAM dominates everything — until the working set fits, then it saturates. Starving RAM is catastrophic: capped to 25 GB this box runs 0.31–0.38 tok/s at 16–33% expert hit, against 1.31 at ~93% with the full 128 GB. That is the ~150% swing, and GPU is worth ~30–44% at full RAM and close to nothing when RAM-starved. But past the point where the hot working set fits, extra budget buys nothing: sweeping RAM_GB at 115 / auto (120.9) / 122 moved the resident set 2,291 → 2,431 → 2,456 experts and expert hit rate by 0.3 points (93.0 / 93.3 / 93.1%). Give it enough RAM to hold the working set; there is no reward for more.

  Taken together with the VRAM-tier rows above, expert hit rate on this container sits at ~93% across every residency knob — a 4.4× VRAM tier range and a 165-expert RAM range all land at 93.0–93.8%. Routing is concentrated enough that the marginal expert is essentially never requested, so the remaining ~7% are genuinely cold and no cache size reaches them. That tail is the ceiling on a streaming host, and it is what the fully-resident hosts (DGX Spark, ~11 tok/s) are buying their way past.

• DIRECT=1 is drive-dependent: +34% on this box's NVMe (DRAM-cached); QLC/DRAM-less or virtualised disks can be neutral to negative. Measure it.

• Keep CUDA_DENSE=1. A bigger VRAM expert tier is not the lever it looks like. Measured on this box, warm, same prompt, balanced config:

  config	VRAM expert tier	decode tok/s	expert hit
  CUDA_DENSE=1 CUDA_EXPERT_GB=4 (recommended)	156 experts, 3.31 GB	1.29	93.8%
  CUDA_DENSE=1 CUDA_EXPERT_GB=auto	156, 3.31 GB	1.30	93.3%
  CUDA_DENSE=0 CUDA_EXPERT_GB=13	612, 13.0 GB	1.15 (−11%)	93.0%
  CUDA_DENSE=0 CUDA_EXPERT_GB=auto	692, 14.69 GB	0.57 (−56%)	93.5%

  Hit rate is flat at 93.0–93.8% while the VRAM tier varies 4.4×. That is the whole result: a RAM-resident expert already counts as a hit, so enlarging the VRAM tier does not raise residency — it only moves some hits closer, and it costs you the dense weights. The principle is that VRAM should hold what is touched on every token (the ~10.9 GB of dense tensors), not what is touched occasionally (individual experts). Expert residency belongs in RAM, which is cheap and plentiful. CUDA_EXPERT_GB=auto sizes to 3.3 GB here and is within noise of the explicit 4 — the auto-sizing is already right.

  The last row is a hazard worth stating plainly: with dense off and the tier sized to the whole card, 14.69 GB of a 17.1 GB card leaves almost nothing for activations, prefill blows out (36 s vs 15 s for the same 19 tokens) and decode more than halves — at unchanged hit rate, so it is memory pressure, not routing. Do not chase a large expert tier on a 16 GB card.

• CUDA_RELEASE_HOST=1 scales with your VRAM tier, not with the flag. It frees the duplicate host copy of every VRAM-tier expert so the RAM tier can use that memory (colibri #686). On this box the tier is only 3.31 GB and the effect is within noise — balanced 1.29/1.32 default vs 1.33/1.30 with it on, a 0.01 gap against a ±0.04 band. On a single GPU with a large tier it is transformative: 1.10 → 4.18–5.37 tok/s measured upstream on 1×H200/235 GB, and it matters most on unified-memory hosts (GB10) where the duplicate copy competes for the same physical pool. v1.3.0 enables it by default when PIN_GB is large. Rule of thumb: worth setting if your VRAM expert tier is a meaningful fraction of system RAM; ignore it if it isn't.

• Speculation under CUDA — leave it off on streaming boxes. Measured on this box at the balanced config (v1.1.1, warm, NGEN=96, TEMP=0): baseline 1.28 tok/s → n-gram PLD DRAFT=3 0.67 (−48%) → MTP head COLI_CUDA_MTP=1 DRAFT=3 0.61 (−52%). Drafting itself works (PLD accepts 2.46 tokens/forward at 49% acceptance); what kills it is that the multi-token verify forward widens the per-forward expert union, so hit rate falls 92.5% → 86% and each forward costs >2× on disk. The deciding variable is expert residency, not the speculator: the same PLD lever measures +66% on a ~100%-resident host (DGX Spark, colibri #161). Break-even sits between ~92% and ~99.9% hit — above it speculation pays, below it it costs. On CPU, MTP is on by default and helps.

• Residency at the recommended config: 12.48 GB dense (952 tensors) + 109 hot experts (2.31 GB) in VRAM; no TDR, zero fallbacks.

Benchmarking this container: two traps

Both of these cost real time to discover, so they are worth stating.

.coli_usage is shared mutable state between your benchmark arms. The engine appends expert-selection history to <model>/.coli_usage on every turn, and PIN=auto derives the pinned tier from it. So each run silently changes the placement the next run inherits. Comparing configs back-to-back therefore compares config plus an evolving pin history, and the effect is not small: in one session here a control arm at identical settings drifted from 1.29 to 0.81 tok/s across roughly twenty-five runs. If you are measuring anything finer than a ~20% effect, snapshot .coli_usage before the sweep and restore it before each arm. An unexplained "this version got slower" is very often this.

Warm-up matters more than usual, and single runs mislead. Prefill and decode both improve substantially between a cold first run and a warm one. Discard a warm-up run, then take at least two replicates per arm: on this box the run-to-run noise is ~±0.04 tok/s in a settled session, but arms that change the per-forward expert union (any speculation) are far noisier — one such arm here spread 0.66 → 1.04 tok/s while its acceptance and hit-rate metrics replicated tightly to within a point. The speculation behaviour is stable; the wall-clock is not, because the variance lives in disk service time.

Windows notes

Current dev auto-enables CUDA on a bare run when a GPU is present (no flags needed). To force a true CPU run use --gpu none, or COLI_CUDA=0 (honoured since #468 — on older builds it was silently overridden by the auto-enable, so pre-#468 Windows "CPU" benchmarks taken on GPU boxes were GPU-assisted).

Provenance

Converted from the official FP8 checkpoint with colibri's c/tools/convert_fp8_to_int4.py (--group-size 64, ebits 4, io-bits 8) plus a corrected local MTP pass (int8; see colibri #355 for the converter footgun and fix). License follows the base model (zai-org/GLM-5.2, MIT).

Author

Container built and tested by Mohamed Mastouri on the colibri issue-306 reference box, at the request of the colibri maintainers. Related upstream work by the same author: the #298 fmt=4 correctness validation, the COLI_MODEL_DIRS N-drive split (#469), the Windows COLI_CUDA=0 fix (#468), and the resource-matrix benchmark record (#467).