Instructions to use unsloth/DeepSeek-V4-Flash-0731-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

LM Studio
Jan
Ollama
How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Ollama:
```
ollama run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.DeepSeek-V4-Flash-0731-GGUF-UD-Q4_K_XL

List all available models

lemonade list

dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf FAILS to load in the latest llama-server

#26

by Nerdsking - opened about 12 hours ago

Discussion

Nerdsking

about 12 hours ago

Using latest llama (https://github.com/ggml-org/llama.cpp/releases/tag/b10240), it gives this message:

"0.00.708.410 W srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model"

Hardware: Ryzen 3700x 128GB RAM + 20GB VRAM (modded RTX 3080) - Windows 11

Command: llama-server.exe --model "c:\AI_Models\DeepSeek-V4-Flash-0731-UD-IQ4_NL-00001-of-00004.gguf" --model-draft "c:\AI_Models\dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf" --spec-type draft-dspark --spec-draft-n-max 2 --top-p 0.95 --temp 1.0 --seed 42 --threads 8 -Cr 0-7 --ctx-size 50000 --host 0.0.0.0 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --parallel 1 --batch-size 512 --ubatch-size 128 --offline --verbosity 4 --load-mode none

danielhanchen

Unsloth AI org about 10 hours ago

Using latest llama (https://github.com/ggml-org/llama.cpp/releases/tag/b10240), it gives this message:

"0.00.708.410 W srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model"

Hardware: Ryzen 3700x 128GB RAM + 20GB VRAM (modded RTX 3080) - Windows 11

Command: llama-server.exe --model "c:\AI_Models\DeepSeek-V4-Flash-0731-UD-IQ4_NL-00001-of-00004.gguf" --model-draft "c:\AI_Models\dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf" --spec-type draft-dspark --spec-draft-n-max 2 --top-p 0.95 --temp 1.0 --seed 42 --threads 8 -Cr 0-7 --ctx-size 50000 --host 0.0.0.0 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --parallel 1 --batch-size 512 --ubatch-size 128 --offline --verbosity 4 --load-mode none

Still experimental, we're going to announce once everything is ready! Thanks for the error report btw we will investigate

LaikaFramework

about 9 hours ago

try: --flash-attn auto

Mikeczyk

about 5 hours ago

•

edited about 5 hours ago

It has been working ok for me. I was using the Q8_K_XL model but tried the IQ4_ML model and seems to be working there as well. Acceptance rate is a bit low so it actually doesn't speed up my generation but it works.
My llama-server was build from:

commit 3581ba0cf591b3f772fbb002de0f70e294bc0396 (HEAD -> master, real-upstream/master)
Author: Aman Gupta <[email protected]>
Date:   Sun Aug 2 23:16:31 2026 +0800

    convert: add option to create separate dspark GGUF (#26452)
    
    * convert: add option to create separate dspark GGUF
    
    * add --no-nextn
    
    * fix convert bug

root@llama:/opt/llama.cpp# ./llama-server --rpc pve.lan:8083 -m  /AI/DeepSeek-V4-Flash-0731-UD-IQ4_NL-00001-of-00004.gguf --host 0.0.0.0 --port 8081 -ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 -c 131072 --threads 2 --threads-batch 4 --top-p 0.95 --temp .9 --repeat-penalty 1 --jinja -np 1 -fa on --dynatemp-range 0.2 --reasoning-budget 3072 --load-mode none --reasoning-format deepseek --reasoning-budget-message "Therefore, the final concise conclusion is:"  --ubatch-size 1024 --batch-size 2048 --cache-prompt --tensor-split 2.4,0,1.8,3,3,3,3,3 --spec-type draft-dspark  --spec-draft-n-max 1 --spec-draft-model /AI2/DeepseekV4-Flash-20260731-DSpark.gguf -devd CUDA0,CUDA6 -ctvd q8_0 -ctkd q8_0
0.00.078.299 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.01.357.090 W srv  llama_server: -----------------
0.01.357.094 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.01.357.095 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.01.357.095 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.01.357.095 W srv  llama_server: -----------------
0.01.358.576 I srv    load_model: loading model '/AI/DeepSeek-V4-Flash-0731-UD-IQ4_NL-00001-of-00004.gguf'
0.01.746.194 E llama_init_from_model: failed to initialize the context: dflash requires ctx_other to be set (this warning is normal during memory fitting)
0.01.775.476 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
13.55.481.933 I common_speculative_init_result: loading draft model '/AI2/DeepseekV4-Flash-20260731-DSpark.gguf'
14.52.491.668 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'
14.52.491.717 I common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
14.52.491.726 I common_speculative_impl_draft_dflash: - n_max=1, n_min=0, p_min=0.00
14.52.491.727 I common_speculative_impl_draft_dflash: - block_size=5, mask_token_id=128799, n_extract=3
14.52.668.641 I srv  llama_server: model loaded
14.52.668.649 I srv  llama_server: listening on http://0.0.0.0:8081
15.22.652.057 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
17.26.798.328 I slot launch_slot_: id  0 | task 415 | processing task, is_child = 0
17.33.759.155 I slot print_timing: id  0 | task 415 | prompt processing, n_tokens =    441, progress = 0.99, t =   6.96 s / 63.35 tokens per second
17.49.723.415 I slot print_timing: id  0 | task 415 | n_decoded =    100, tg =   6.44 t/s, tg_3s =   6.44 t/s
17.52.964.167 I slot print_timing: id  0 | task 415 | n_decoded =    123, tg =   6.55 t/s, tg_3s =   7.10 t/s
17.56.203.860 I slot print_timing: id  0 | task 415 | n_decoded =    145, tg =   6.59 t/s, tg_3s =   6.79 t/s
17.59.443.249 I slot print_timing: id  0 | task 415 | n_decoded =    165, tg =   6.53 t/s, tg_3s =   6.17 t/s
18.02.704.079 I slot print_timing: id  0 | task 415 | n_decoded =    188, tg =   6.59 t/s, tg_3s =   7.05 t/s
18.05.813.277 I slot print_timing: id  0 | task 415 | n_decoded =    208, tg =   6.58 t/s, tg_3s =   6.43 t/s
18.09.052.295 I slot print_timing: id  0 | task 415 | n_decoded =    229, tg =   6.57 t/s, tg_3s =   6.48 t/s
18.12.293.780 I slot print_timing: id  0 | task 415 | n_decoded =    252, tg =   6.61 t/s, tg_3s =   7.10 t/s
18.15.532.153 I slot print_timing: id  0 | task 415 | n_decoded =    273, tg =   6.60 t/s, tg_3s =   6.48 t/s
18.17.421.771 I slot print_timing: id  0 | task 415 | prompt eval time =    7392.84 ms /   445 tokens (   16.61 ms per token,    60.19 tokens per second)
18.17.421.782 I slot print_timing: id  0 | task 415 |        eval time =   43230.29 ms /   286 tokens (  151.15 ms per token,     6.62 tokens per second)
18.17.421.784 I slot print_timing: id  0 | task 415 |       total time =   50623.13 ms /   731 tokens
18.17.421.785 I slot print_timing: id  0 | task 415 |    graphs reused =        556
18.17.421.788 I slot print_timing: id  0 | task 415 | draft acceptance = 0.79874 (  127 accepted /   159 generated), mean len =  1.80
18.17.421.886 I slot      release: id  0 | task 415 | stop processing: n_tokens = 731, truncated = 0

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment