bleysg/DeepSeek-V4-Flash-IQ2XXS-Q2K-FP8-120GB-target

DeepSeek-V4-Flash · IQ2_XXS + Q2_K + FP8 hybrid for vLLM

A 2-bit hybrid quantization of deepseek-ai/DeepSeek-V4-Flash, packaged for vLLM serving on a single DGX Spark (GB10 / SM121, 119 GiB unified memory).

The recipe is identical to the one used by antirez/ds4, the C+Metal reference implementation that this conversion was validated against. Every routed-MoE expert is stored at ~2 bits per weight; dense / attention layers stay at FP8 with UE8M0 block scales; embeddings, lm_head, and norms remain in BF16.

Component	Format	bpw
Routed experts: gate / up	IQ2_XXS	~2.06
Routed experts: down	Q2_K	~2.62
Dense linears + attention	FP8 E4M3, block-128, UE8M0 scales	8
Embeddings, lm_head, norms, scalars	BF16	16

Total on-disk: ~85 GiB across 17 safetensors shards. Resident usage during serving is ~110 GiB (model + workspace + KV cache pool). Plan for ≥120 GiB system memory with headroom for the OS — that's the meaning of the -120GB-target suffix.

Hardware

This checkpoint is specifically targeted at consumer Blackwell (SM12x), where shared memory per SM is ~99 KiB rather than the 228 KiB on datacenter Blackwell. It has been validated end-to-end on NVIDIA DGX Spark (GB10).

It will work on other GPUs with FP8 + Triton support (H100, B200, etc.) but the surrounding vLLM build was tuned for SM121. There is one SM12x-specific decode-kernel workaround required at serve time — see Serving below.

Serving

The fastest path is the one-shot installer:

curl -sSL https://raw.githubusercontent.com/Entrpi/ds4-spark-vllm/main/install.sh | bash

It will pull this checkpoint, the lmxxf/vllm-deepseek-v4-dgx-spark base image, the small overlay package from Entrpi/ds4-spark-vllm, and start vllm serve on localhost:8000.

If you want to run the docker command directly:

docker run -d --gpus all --name vllm-ds4 --network host \
  -v "$HOME/models":/models -v "$HOME/ds4-spark-vllm":/work \
  -v "$HOME/logs":/logs -v "$HOME/extras":/extras \
  -e DG_LOCAL=/extras/DeepGEMM \
  -e LD_PRELOAD=/usr/local/cuda/lib64/libnvrtc.so \
  -e VLLM_TRITON_MLA_SPARSE_MATMUL_DECODE=0 \
  --entrypoint bash lmxxf/vllm-deepseek-v4-dgx-spark:latest \
  -c "bash /work/eugr_mod/mods/ds4-2bit-deepseek-v4-flash/run-on-lmxxf.sh > /logs/serve-mod.log 2>&1 && \
      vllm serve /models/DeepSeek-V4-Flash-IQ2XXS-Q2K-FP8-120GB-target \
        --served-model-name dsv4 --quantization deepseek_v4_hybrid_iq2 \
        --port 8000 --host 0.0.0.0 \
        --max-model-len 16384 --gpu-memory-utilization 0.86 \
        --kv-cache-dtype fp8 --attention-backend FLASHINFER \
        --enforce-eager 2>&1 | tee /logs/serve.log"

Two correctness-critical knobs:

• --quantization deepseek_v4_hybrid_iq2 — registered by the ds4_hybrid_quant plugin (the Entrpi/ds4-spark-vllm overlay).
• VLLM_TRITON_MLA_SPARSE_MATMUL_DECODE=0 — required on SM121. The default triton compressed-decode kernel (matmul_sparse_mla_attention_with_sink) produces wrong output on consumer Blackwell for layers with compress_ratio≥4. This env flag switches to the working fp8ds_global_paged_sparse_mla_attention_with_sink_multihead path. Without it, the model emits one correct token then degenerates.

See docs/DSV4_FLASH_2BIT_SPARK_REPORT.md in the overlay repo for the full bring-up writeup, including how those two bugs were found via layer-by-layer hidden-state bisection against the antirez/ds4 reference.

Validation

Layer-by-layer hidden-state cosine similarity vs the C+Metal antirez/ds4 reference, on the canonical "The capital of France is" prompt:

Layer	input cos	output cos
0	0.9991	0.9975
21	0.9924	0.9909
42	0.9590	0.9304

Mean input cosine across all 43 layers: 0.9875.

Single-call generation (max_tokens=10, temperature=0):

prompt:    "The capital of France is"
ds4 ref:   ' We are asked: "...'   → tokens [2581, 477, 4869, 28, 582, ...]
this ckpt: ' We are asked: "The capital of France is'

First five generated tokens match the reference exactly.

Files

config.json                         vLLM config (model_type: deepseek_v4)
generation_config.json              default sampling params (overridden at request time)
tokenizer.json, tokenizer_config.json
model.safetensors.index.json        17-shard index
model-00000.safetensors  .. model-00016.safetensors
SHA256SUMS                          integrity manifest for all of the above
LICENSE                             MIT

To verify integrity after download:

hf download bleysg/DeepSeek-V4-Flash-IQ2XXS-Q2K-FP8-120GB-target --local-dir ./dsv4-q2
( cd ./dsv4-q2 && sha256sum -c SHA256SUMS )

Conversion provenance

This checkpoint was converted from antirez's GGUF release (DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf) via scripts/build-ds4-2bit-checkpoint.py in the overlay repo. The conversion is deterministic: re-running on the same GGUF produces a byte-identical safetensors set.

License

MIT, the same as upstream deepseek-ai/DeepSeek-V4-Flash. The LICENSE file in this repo carries both DeepSeek's original copyright and the redistribution copyright on the quantized derivative.

The conversion logic, vLLM overlay (registers the deepseek_v4_hybrid_iq2 quantization method), serving fix-ups, and bring-up scripts are also MIT — see the Entrpi/ds4-spark-vllm repo.

Citation

If you use this checkpoint, please cite both DeepSeek and antirez's reference implementation:

@misc{deepseekv4flash,
  title  = {DeepSeek-V4-Flash},
  author = {DeepSeek},
  year   = {2026},
  url    = {https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash},
}
@misc{antirezds4,
  title  = {ds4: a DeepSeek-V4-Flash inference engine},
  author = {Sanfilippo, Salvatore},
  year   = {2026},
  url    = {https://github.com/antirez/ds4},
}