Instructions to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="willfalco/GLM-5.2-EXL3-TR3-3.25bpw")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.25bpw")
model = AutoModelForCausalLM.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.25bpw", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Trellis

How to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "willfalco/GLM-5.2-EXL3-TR3-3.25bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.25bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.25bpw

SGLang

How to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "willfalco/GLM-5.2-EXL3-TR3-3.25bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.25bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "willfalco/GLM-5.2-EXL3-TR3-3.25bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.25bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use willfalco/GLM-5.2-EXL3-TR3-3.25bpw with Docker Model Runner:
```
docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.25bpw
```

GLM-5.2 EXL3 TR3 3.25 bpw Coder

This was superseded by 3.42bpw Coder vesrion

Can still be used with start up options from 3.42 bpw vesrion to get more KV

with Coding expert allignments from NF3

This is a TP4, rank-sliced EXL3 build of zai-org/GLM-5.2, optimized for four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-78 use EXL3 Trellis weights targeting 3.0/4.0 bits per weight — includes the MTP (layer 78) routed experts. Accuracy-sensitive and dense components remain in BF16. Combined 3.0bpw from brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw The repository payload is 315.9 GiB. This format requires the custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model. The config.json retains ModelOpt/NVFP4 compatibility metadata used by the conversion pipeline, but the routed weights are EXL3 and the required launch flag is --quantization exl3. NVFP4 in the supplied runtime refers to the KV cache, not the routed-expert weight format.

Weights        | KV format                 | KLD
───────────────────────────────────────────────────────────────────────
NF3            | Dynamic NVFP4 + RoPE8     | 0.139036 ± 0.002010 (n=3)
NF3            | Standard FP8 + BF16 RoPE  | 0.1263†
EXL3 3.0-bpw   | Dynamic NVFP4 + RoPE8     | 0.119525
EXL3 3.0-bpw   | Standard FP8 + BF16 RoPE  | 0.102508
EXL3 3.25-bpw  | Dynamic NVFP4 + RoPE8     | 0.095971
EXL3 3.25-bpw  | Standard FP8 + BF16 RoPE  | 0.087711

services:
  g52h:
    image: voipmonitor/vllm:gilded-gnosis-v20-vllm749050e-si8110e3e-fi801d57a-cu132-20260730-r14
    container_name: g52h
    ports:
      - "0.0.0.0:8000:8000"
    gpus: all
    shm_size: "32g"
    ipc: "host"
    ulimits:
      memlock: -1
      nofile: 1048576
    environment:
      - CUDA_VISIBLE_DEVICES=2,3,4,5
      - CUDA_DEVICE_MAX_CONNECTIONS=32
      - CUTE_DSL_ARCH=sm_120a
      - OMP_NUM_THREADS=16
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - SAFETENSORS_FAST_GPU=1
      - NCCL_IB_DISABLE=1
      - NCCL_P2P_LEVEL=SYS
      - NCCL_PROTO=LL,LL128,Simple
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_B12X_FP8_GEMM=0  # -kld
      - VLLM_USE_B12X_SPARSE_INDEXER=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - VLLM_PCIE_ALLREDUCE_BACKEND=b12x
      - VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=64KB
      - VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE=84KB
      - B12X_PCIE_DMA_FP8=0  # -kld
      - B12X_DENSE_SPLITK_TURBO=1
      - B12X_W4A16_TC_DECODE=1
      - B12X_MOE_FORCE_A16=1
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_BREAKABLE_CUDAGRAPH=0
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
      - VLLM_USE_B12X_MHC=1
      - B12X_MHC_MAX_TOKENS=16384
      - VLLM_USE_B12X_WO_PROJECTION=1
      - B12X_MLA_SM120_UNIFIED=1
      - VLLM_CACHE_DIR=/cache/jit/vllm
      - TRITON_CACHE_DIR=/cache/jit/triton
      - TORCH_EXTENSIONS_DIR=/cache/jit/torch_extensions
      - TORCHINDUCTOR_CACHE_DIR=/cache/jit/torchinductor
      - FLASHINFER_WORKSPACE_BASE=/cache/jit/flashinfer
      - XDG_CACHE_HOME=/cache/jit
      - TVM_FFI_CACHE_DIR=/cache/jit/tvm-ffi
      - GLOO_SOCKET_IFNAME=lo
      - NCCL_SOCKET_IFNAME=lo
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - VLLM_PCIE_DMA_MIN_BYTES=6MB
      - VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0  # +pp +kld
      - VLLM_B12X_MLA_SPEC_DECODE_MAX_Q=8
      - VLLM_USE_B12X_DCP_A2A=1
      - VLLM_DCP_A2A_MAX_TOKENS=16
      - VLLM_DCP_A2A_LARGE_BACKEND=ag_rs
      - VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MIN_TOKENS=512  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=16384  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_DCP_QUERY_SPLIT=1  # r14
      - VLLM_MEMORY_PROFILE_INCLUDE_ATTN=1
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - TORCH_CUDA_ARCH_LIST=12.0a
      - FLASHINFER_CUDA_ARCH_LIST=12.0f
      - FLASHINFER_DISABLE_VERSION_CHECK=1
      - VLLM_USE_B12X_MOE=1
      - VLLM_CPP_AR_1STAGE_NCCL_CUTOFF=56KB
      - VLLM_CPP_AR_IGNORE_CUTOFF_MAX_ROWS=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD_END_BARRIER=0
      - VLLM_DISABLE_SHARED_EXPERTS_STREAM=0  # v20 
      - VLLM_DISABLED_KERNELS=MarlinFP8ScaledMMLinearKernel
      - VLLM_DCP_GLOBAL_TOPK=1
      - VLLM_DCP_SHARD_DRAFT=1
      - VLLM_DCP_QUERY_SPLIT=0
      - VLLM_EXL3_TRELLIS_MIN_M=1
      - VLLM_EXL3_TRELLIS_MAX_M=48
      - VLLM_EXL3_TRELLIS_BLOCK_M=8
      - VLLM_EXL3_PREFILL_CHUNK=128
      - KV_FP8_ROPE=0  # -kld
      - VLLM_B12X_ABSORB_BMM=0
    volumes:
      - /data1/GLM-5.2-EXL3-TR3-3.25bpw:/model:ro
      - /data1/GLM-5.2-EXL3-TR3-3.25bpw.cache:/cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.25bpw.cache:/root/.cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.25bpw/patches:/patches:ro
    entrypoint:
      - /bin/sh
      - -c
      - "unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS && exec vllm serve \"$@\""
      - --
    command:
      - /model
      - --served-model-name=g52h
      - --trust-remote-code
      - --tensor-parallel-size=4
      - --decode-context-parallel-size=4
      - --dcp-comm-backend=a2a
      - --dcp-kv-cache-interleave-size=1
      - --quantization=exl3
      - --kv-cache-dtype=fp8
      - --attention-backend=B12X_MLA_SPARSE
      - --moe-backend=b12x
      - --load-format=safetensors
      - '--compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48],"custom_ops":["all"],"pass_config":{"fuse_allreduce_rms":true}}'
      - --gpu-memory-utilization=0.972
      - --max-model-len=256256
      - --max-num-seqs=16
      - --max-num-batched-tokens=2048
      - --max-cudagraph-capture-size=48
      - --enable-auto-tool-choice
      - --tool-call-parser=glm47
      - --reasoning-parser=glm45
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --no-async-scheduling
      - --enable-flashinfer-autotune
      - '--default-chat-template-kwargs={"reasoning_effort":"high"}'
      - '--hf-overrides={"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'
      - '--speculative-config={"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton","draft_sample_method":"greedy"}'
#      - '--override-generation-config={"top_p":0.95,"repetition_penalty":1.18}'  # for temp=0.1 MMLU-Pro
      - --host=0.0.0.0
      - --port=8000

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Downloads last month: 406

Safetensors

Model size

169B params

Tensor type

BF16

F16

I16

F32

Model tree for willfalco/GLM-5.2-EXL3-TR3-3.25bpw

Base model

zai-org/GLM-5.2

Quantized

(134)

this model

Paper for willfalco/GLM-5.2-EXL3-TR3-3.25bpw

GLM-5: from Vibe Coding to Agentic Engineering

Paper • 2602.15763 • Published Feb 17 • 205