Inconsistent results when using fp8?
vllm serve /share5/projects/llm/models/weight/Kimi-K2-Instruct-0905 \
--distributed-executor-backend ray \
--tensor-parallel-size 16 \
--host 0.0.0.0 --port 8080 \
--served-model-name kimi-k2-instruct-0905 \
--trust-remote-code \
--max-model-len 131072 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.95 \
--quantization fp8 \
--kv-cache-dtype fp8 \
--calculate-kv-scales \
--enable-auto-tool-choice \
--tool-call-parser kimi_k2
I served kimi-k2-instruct-0905 on 16 h100 gpus. when I inference with the endpoint, I got some inconsitent reuslts. Any clues? Is my hosting the model correct.
The original prompt (11,701 tokens) consistently fails with kimi-k2:
5/5 attempts returned empty response
stop_reason: 163586 (appears to be an internal error code)
completion_tokens: 1 (only generates 1 token before stopping)
Comparison:
Prompt Type Tokens kimi-k2 Claude GPT-5
Simple (same question) 136 ✅ Works ✅ Works ✅ Works
Original complex 11,701 ❌ Empty ✅ Works ✅ Works
Investigation with prompt length:
======================================================================
FINDING KIMI-K2 TOKEN THRESHOLD (8K-15K Range)
✅ 1,299 prompt tokens | completion: 12 | The meeting lasts 35 minutes and 25 seconds.
✅ 2,499 prompt tokens | completion: 35 | The meeting lasts 35 minutes and 25 seconds, c
✅ 3,699 prompt tokens | completion: 22 | The meeting lasts 35 minutes and 25 seconds, i
✅ 4,899 prompt tokens | completion: 35 | The meeting lasts 35 minutes and 25 seconds, c
✅ 6,099 prompt tokens | completion: 22 | The meeting lasts 35 minutes and 25 seconds, i
✅ 7,299 prompt tokens | completion: 30 | The meeting lasts 35 minutes and 25 seconds, c
✅ 8,499 prompt tokens | completion: 24 | The meeting lasts 35 minutes and 25 seconds, for a
✅ 9,699 prompt tokens | completion: 34 | The meeting lasts 35 minutes and 25 seconds, c
======================================================================
BINARY SEARCH FOR KIMI-K2 TOKEN THRESHOLD
❌ 12,099 prompt tokens | completion: 1 | (empty)
❌ 14,099 prompt tokens | completion: 1 | (empty)
✅ 16,099 prompt tokens | completion: 12 | The meeting lasts **35 minutes and 25 se
❌ 18,099 prompt tokens | completion: 1 | (empty)
✅ 20,099 prompt tokens | completion: 12 | The meeting lasts **35 minutes and 25 se
❌ 22,099 prompt tokens | completion: 1 | (empty)
✅ 24,099 prompt tokens | completion: 21 | The meeting lasts **35 minutes and 25 se