supakorn-emch/typhoon2.5-qwen3-4b-GGUF

Typhoon 2.5 Qwen3 4B - GGUF Quantized

GGUF conversion of typhoon-ai/typhoon2.5-qwen3-4b for use with llama.cpp, Ollama, and other GGUF-compatible inference engines.

About the Model

Original Model	typhoon-ai/typhoon2.5-qwen3-4b
Architecture	Qwen3 (decoder-only)
Parameters	4B
Context Length	256K tokens
License	Apache 2.0
Tokenizer	Qwen-style (<|im_start|>, <|im_end|>, <tool_call>, <tool_response>)

Available Quantizations

Quant	Size	BPW	Notes
Q2_K	~1.6 GB	2.96	Smallest, lower quality
Q3_K_S	~1.8 GB	3.41
Q3_K_M	~1.9 GB	3.74
Q3_K_L	~2.1 GB	4.03
Q4_0	~2.2 GB	4.34	Legacy
Q4_1	~2.4 GB	4.78	Legacy
Q4_K_S	~2.2 GB	4.37
Q4_K_M	~2.3 GB	4.58	⭐ Recommended (balanced)
Q5_0	~2.6 GB	5.21	Legacy
Q5_1	~2.8 GB	5.65	Legacy
Q5_K_S	~2.6 GB	5.21
Q5_K_M	~2.7 GB	5.33
Q6_K	~3.1 GB	6.14	Near-perfect quality
Q8_0	~4.0 GB	8.50	Near-FP16 quality
IQ3_S	~1.8 GB	3.44
IQ3_M	~1.8 GB	3.66
IQ4_NL	~2.2 GB	4.50	Non-linear
IQ4_XS	~1.7 GB	4.25	Non-linear

Usage

llama.cpp

./llama-cli -m typhoon2.5-qwen3-4b-Q4_K_M.gguf -p "สวัสดีครับ" -n 256

Ollama

# Create Modelfile:
FROM ./typhoon2.5-qwen3-4b-Q4_K_M.gguf

# Import:
ollama create typhoon2.5-4b:q4_k_m -f Modelfile
ollama run typhoon2.5-4b:q4_k_m

Conversion Details

• Converter: convert_hf_to_gguf.py from llama.cpp
• Quantizer: llama-quantize
• FP16 size: 7.5 GB (not included in this repo)
• Source commit: llama.cpp build 9843 (86b94708f)

Credits

• Original model by SCB DataX (Typhoon-AI)
• GGUF conversion using llama.cpp