Instructions to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Use Docker

docker model run hf.co/ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

LM Studio
Jan

vLLM

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ggml-org/DeepSeek-V4-Flash-0731-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ggml-org/DeepSeek-V4-Flash-0731-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Ollama
How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Ollama:
```
ollama run hf.co/ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
```

Unsloth Studio

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ggml-org/DeepSeek-V4-Flash-0731-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ggml-org/DeepSeek-V4-Flash-0731-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ggml-org/DeepSeek-V4-Flash-0731-GGUF to start chatting

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Docker Model Runner:
```
docker model run hf.co/ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16
```

Lemonade

How to use ggml-org/DeepSeek-V4-Flash-0731-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ggml-org/DeepSeek-V4-Flash-0731-GGUF:BF16

Run and chat with the model

lemonade run user.DeepSeek-V4-Flash-0731-GGUF-BF16

List all available models

lemonade list

Spec draft models are missing sliding_window_pattern

by idgaf1234 - opened about 23 hours ago

Discussion

idgaf1234

about 23 hours ago

•

edited about 23 hours ago

When running with llama-server, getting this error

0.02.688.369 E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: dflash.attention.sliding_window_pattern
0.02.688.374 E llama_model_load_from_file_impl: failed to load model

sliding_window is defined but sliding_window_pattern is not. Using this command to start the server

./build/bin/llama-server -m ./models/DeepSeek-V4-Flash-0731-MXFP4.gguf -c 262144 --host 0.0.0.0 --port 9000 --tools all --no-mmap --spec-draft-model ./models/dspark-DeepSeek-V4-Flash-0731-BF16.gguf

Also happens when loading the MXFP4 version

ggerganov

ggml-org org about 23 hours ago

•

edited about 23 hours ago

Add --spec-type draft-dspark to your command.

idgaf1234

about 23 hours ago

•

edited about 23 hours ago

Add --spec-type draft-dspark to your command.

same error with this command

./build/bin/llama-server -m ./models/DeepSeek-V4-Flash-0731-MXFP4.gguf -c 262144 --host 0.0.0.0 --port 9000 --tools all --no-mmap --spec-type draft-dspark --spec-draft-model ./models/dspark-DeepSeek-V4-Flash-0731-BF16.gguf

ggerganov

ggml-org org about 22 hours ago

It works on my end. Make sure you are using the latest llama.cpp.

What is the output of this command:

./build/bin/llama-server --version

idgaf1234

about 22 hours ago

•

edited about 22 hours ago

I just pulled the latest version, so running that gives me

version: 10236 (1464c62d8)
built with GNU 13.3.0 for Linux x86_64

Running now with the same command yields

E llama_init_from_model: failed to initialize the context: dflash requires ctx_other to be set (this warning is normal during memory fitting)
0.02.918.154 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
/home/am/Documents/llama.cpp/ggml/src/ggml-backend.cpp:1356: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
[New LWP 475622]
[New LWP 475621]
[New LWP 475620]
[New LWP 475619]
[New LWP 475618]
[New LWP 475617]
[New LWP 475616]
[New LWP 475615]
[New LWP 475614]
[New LWP 475613]
[New LWP 475612]
[New LWP 475611]
[New LWP 475610]
[New LWP 475609]
[New LWP 475608]
[New LWP 475607]
[New LWP 475606]
[New LWP 475605]
[New LWP 475604]
[New LWP 475603]
[New LWP 475602]
[New LWP 475601]
[New LWP 475600]
[New LWP 475599]
[New LWP 475598]
[New LWP 475597]
[New LWP 475596]
[New LWP 475595]
[New LWP 475594]
[New LWP 475593]
[New LWP 475592]
[New LWP 475591]
[New LWP 475583]
[New LWP 475578]
[New LWP 475573]
[New LWP 475568]
[New LWP 475558]
[New LWP 475555]
[New LWP 475554]
[New LWP 475549]
[New LWP 475512]
[New LWP 475511]
[New LWP 475510]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000716e21310913 in __GI___wait4 (pid=475632, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000716e21310913 in __GI___wait4 (pid=475632, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000716e2154d683 in ggml_print_backtrace () from /home/am/Documents/llama.cpp/build/bin/libggml-base.so.0
#2  0x0000716e2154d82b in ggml_abort () from /home/am/Documents/llama.cpp/build/bin/libggml-base.so.0
#3  0x0000716e2156a687 in ggml_backend_sched_split_graph () from /home/am/Documents/llama.cpp/build/bin/libggml-base.so.0
#4  0x0000716e208f61a2 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /home/am/Documents/llama.cpp/build/bin/libllama.so.0
#5  0x0000716e208f676a in llama_context::resolve_fused_ops(llama_memory_context_i const*, unsigned int) () from /home/am/Documents/llama.cpp/build/bin/libllama.so.0
#6  0x0000716e208f76d4 in llama_context::sched_reserve() () from /home/am/Documents/llama.cpp/build/bin/libllama.so.0
#7  0x0000716e208fada9 in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/am/Documents/llama.cpp/build/bin/libllama.so.0
#8  0x0000716e208fc079 in llama_init_from_model () from /home/am/Documents/llama.cpp/build/bin/libllama.so.0
#9  0x0000716e20e547cd in common_get_device_memory_data_impl(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from /home/am/Documents/llama.cpp/build/bin/libllama-common.so.0
#10 0x0000716e20e55943 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from /home/am/Documents/llama.cpp/build/bin/libllama-common.so.0
#11 0x0000716e20e59802 in common_fit_params(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from /home/am/Documents/llama.cpp/build/bin/libllama-common.so.0
#12 0x0000716e20e26927 in common_init_result::common_init_result(common_params&, bool) () from /home/am/Documents/llama.cpp/build/bin/libllama-common.so.0
#13 0x0000716e20e27b43 in common_init_from_params(common_params&, bool) () from /home/am/Documents/llama.cpp/build/bin/libllama-common.so.0
#14 0x0000716e21bbdede in server_context_impl::load_model(common_params&) () from /home/am/Documents/llama.cpp/build/bin/libllama-server-impl.so
#15 0x0000716e21af151f in llama_server(common_params&, int, char**) () from /home/am/Documents/llama.cpp/build/bin/libllama-server-impl.so
#16 0x0000716e21af391f in llama_server(int, char**) () from /home/am/Documents/llama.cpp/build/bin/libllama-server-impl.so
#17 0x0000716e2122a1ca in __libc_start_call_main (main=main@entry=0x583d50ed5270 <main>, argc=argc@entry=16, argv=argv@entry=0x7ffe760ca7b8) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58	../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#18 0x0000716e2122a28b in __libc_start_main_impl (main=0x583d50ed5270 <main>, argc=16, argv=0x7ffe760ca7b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe760ca7a8) at ../csu/libc-start.c:360
warning: 360	../csu/libc-start.c: No such file or directory
#19 0x0000583d50ed52a5 in _start ()
[Inferior 1 (process 475504) detached]
Aborted (core dumped)

What version are you using? Maybe I can try that

ggerganov

ggml-org org about 10 hours ago

There was an issue which should be now resolved. Try pulling the latest master of llama.cpp and let me know if the issue persists.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment