Instructions to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Use Docker

docker model run hf.co/ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

LM Studio
Jan

vLLM

How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/DeepSeek-V3.2-Speciale-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/DeepSeek-V3.2-Speciale-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Ollama
How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with Ollama:
```
ollama run hf.co/ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
```

Unsloth Studio

How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-V3.2-Speciale-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-V3.2-Speciale-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ubergarm/DeepSeek-V3.2-Speciale-GGUF to start chatting

Docker Model Runner
How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K
```

Lemonade

How to use ubergarm/DeepSeek-V3.2-Speciale-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/DeepSeek-V3.2-Speciale-GGUF:Q2_K

Run and chat with the model

lemonade run user.DeepSeek-V3.2-Speciale-GGUF-Q2_K

List all available models

lemonade list

Atomic Chat

ubergarm commited on Jan 9

Commit

5d793ef

1 Parent(s): 088dbe8

add some perplexity data and logs

Browse files

Files changed (4) hide show

README.md +1 -1
logs/perplexity-DeepSeek-V3.2-Speciale-IQ3_K.log +265 -0
logs/perplexity-DeepSeek-V3.2-Speciale-IQ5_K.log +263 -0
logs/perplexity-DeepSeek-V3.2-Speciale-Q8_0.log +253 -0

README.md CHANGED Viewed

@@ -34,7 +34,7 @@ Perplexity computed against *wiki.test.raw*.
 Tis one is just a test quants for baseline perplexity comparison:
 * `Q8_0` 664.295 GiB (8.504 BPW)
-  - Final estimate: PPL over 561 chunks for n_ctx=512 = TODO
 *NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine!

 Tis one is just a test quants for baseline perplexity comparison:
 * `Q8_0` 664.295 GiB (8.504 BPW)
+  - Final estimate: PPL over 561 chunks for n_ctx=512 = 3.2082 +/- 0.01742
 *NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine!

logs/perplexity-DeepSeek-V3.2-Speciale-IQ3_K.log ADDED Viewed

	@@ -0,0 +1,265 @@

+numactl -N "$SOCKET" -m "$SOCKET" \
+./build/bin/llama-perplexity \
+    -m "$model" \
+    -f wiki.test.raw \
+    --seed 1337 \
+    --ctx-size 512 \
+    -mla 3 \
+    -ub 4096 -b 4096 \
+    --numa numactl \
+    --threads 96 \
+    --threads-batch 128 \
+    --validate-quants \
+    --no-mmap
+SOCKET is set to: 1
+main: build = 4099 (145e4f4e)
+main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
+main: seed  = 1337
+CPU: using device CPU - 0 MiB free
+llama_model_loader: additional 7 GGUFs metadata loaded.
+llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /mnt/raid/hf/DeepSeek-V3.2-Speciale-GGUF/IQ3_K/DeepSeek-V3.2-Speciale-IQ3_K-00001-of-00008.gguf (version GGUF V3 (latest))
+llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
+llama_model_loader: - kv   1:                               general.type str              = model
+llama_model_loader: - kv   2:                     general.sampling.top_p f32              = 0.950000
+llama_model_loader: - kv   3:                      general.sampling.temp f32              = 1.000000
+llama_model_loader: - kv   4:                               general.name str              = DeepSeek V3.2 Speciale Bf16 Safetensors
+llama_model_loader: - kv   5:                           general.finetune str              = safetensors
+llama_model_loader: - kv   6:                           general.basename str              = DeepSeek-V3.2-Speciale
+llama_model_loader: - kv   7:                         general.size_label str              = 256x20B
+llama_model_loader: - kv   8:                      deepseek2.block_count u32              = 61
+llama_model_loader: - kv   9:                   deepseek2.context_length u32              = 163840
+llama_model_loader: - kv  10:                 deepseek2.embedding_length u32              = 7168
+llama_model_loader: - kv  11:              deepseek2.feed_forward_length u32              = 18432
+llama_model_loader: - kv  12:             deepseek2.attention.head_count u32              = 128
+llama_model_loader: - kv  13:          deepseek2.attention.head_count_kv u32              = 1
+llama_model_loader: - kv  14:                deepseek2.rope.scaling.type str              = yarn
+llama_model_loader: - kv  15:              deepseek2.rope.scaling.factor f32              = 40.000000
+llama_model_loader: - kv  16: deepseek2.rope.scaling.original_context_length u32              = 4096
+llama_model_loader: - kv  17:      deepseek2.rope.scaling.yarn_beta_fast f32              = 32.000000
+llama_model_loader: - kv  18:      deepseek2.rope.scaling.yarn_beta_slow f32              = 1.000000
+llama_model_loader: - kv  19:                   deepseek2.rope.freq_base f32              = 10000.000000
+llama_model_loader: - kv  20: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
+llama_model_loader: - kv  21:                deepseek2.expert_used_count u32              = 8
+llama_model_loader: - kv  22:               deepseek2.expert_group_count u32              = 8
+llama_model_loader: - kv  23:          deepseek2.expert_group_used_count u32              = 4
+llama_model_loader: - kv  24:               deepseek2.expert_gating_func u32              = 2
+llama_model_loader: - kv  25:                          general.file_type u32              = 139
+llama_model_loader: - kv  26:        deepseek2.leading_dense_block_count u32              = 3
+llama_model_loader: - kv  27:                       deepseek2.vocab_size u32              = 129280
+llama_model_loader: - kv  28:            deepseek2.attention.q_lora_rank u32              = 1536
+llama_model_loader: - kv  29:           deepseek2.attention.kv_lora_rank u32              = 512
+llama_model_loader: - kv  30:             deepseek2.attention.key_length u32              = 576
+llama_model_loader: - kv  31:           deepseek2.attention.value_length u32              = 512
+llama_model_loader: - kv  32:         deepseek2.attention.key_length_mla u32              = 192
+llama_model_loader: - kv  33:       deepseek2.attention.value_length_mla u32              = 128
+llama_model_loader: - kv  34:       deepseek2.expert_feed_forward_length u32              = 2048
+llama_model_loader: - kv  35:                     deepseek2.expert_count u32              = 256
+llama_model_loader: - kv  36:              deepseek2.expert_shared_count u32              = 1
+llama_model_loader: - kv  37:             deepseek2.expert_weights_scale f32              = 2.500000
+llama_model_loader: - kv  38:              deepseek2.expert_weights_norm bool             = true
+llama_model_loader: - kv  39:             deepseek2.rope.dimension_count u32              = 64
+llama_model_loader: - kv  40: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
+llama_model_loader: - kv  41:               general.quantization_version u32              = 2
+llama_model_loader: - kv  42:                       tokenizer.ggml.model str              = gpt2
+llama_model_loader: - kv  43:                         tokenizer.ggml.pre str              = deepseek-v3
+llama_model_loader: - kv  44:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
+llama_model_loader: - kv  45:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
+llama_model_loader: - kv  46:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
+llama_model_loader: - kv  47:                tokenizer.ggml.bos_token_id u32              = 0
+llama_model_loader: - kv  48:                tokenizer.ggml.eos_token_id u32              = 1
+llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 1
+llama_model_loader: - kv  50:               tokenizer.ggml.add_bos_token bool             = true
+llama_model_loader: - kv  51:               tokenizer.ggml.add_eos_token bool             = false
+llama_model_loader: - kv  52:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
+llama_model_loader: - kv  53:                      quantize.imatrix.file str              = /mnt/data/models/ubergarm/DeepSeek-V3...
+llama_model_loader: - kv  54:                   quantize.imatrix.dataset str              = ubergarm-imatrix-calibration-corpus-v...
+llama_model_loader: - kv  55:             quantize.imatrix.entries_count i32              = 782
+llama_model_loader: - kv  56:              quantize.imatrix.chunks_count i32              = 812
+llama_model_loader: - kv  57:                                   split.no u16              = 0
+llama_model_loader: - kv  58:                                split.count u16              = 8
+llama_model_loader: - kv  59:                        split.tensors.count i32              = 1086
+llama_model_loader: - type  f32:  361 tensors
+llama_model_loader: - type q8_0:  368 tensors
+llama_model_loader: - type iq3_k:  116 tensors
+llama_model_loader: - type iq4_k:    1 tensors
+llama_model_loader: - type iq6_k:  182 tensors
+llama_model_loader: - type iq4_kss:   58 tensors
+load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
+load: printing all EOG tokens:
+load:   - 1 ('<｜end▁of▁sentence｜>')
+load: special tokens cache size = 818
+load: token to piece cache size = 0.8222 MB
+llm_load_print_meta: format           = GGUF V3 (latest)
+llm_load_print_meta: arch             = deepseek2
+llm_load_print_meta: n_ctx_train      = 163840
+llm_load_print_meta: n_embd           = 7168
+llm_load_print_meta: n_layer          = 61
+llm_load_print_meta: n_head           = 128
+llm_load_print_meta: n_head_kv        = 128
+llm_load_print_meta: n_rot            = 64
+llm_load_print_meta: n_swa            = 0
+llm_load_print_meta: n_swa_pattern    = 1
+llm_load_print_meta: n_embd_head_k    = 192
+llm_load_print_meta: n_embd_head_v    = 128
+llm_load_print_meta: n_gqa            = 1
+llm_load_print_meta: n_embd_k_gqa     = 24576
+llm_load_print_meta: n_embd_v_gqa     = 16384
+llm_load_print_meta: f_norm_eps       = 0.0e+00
+llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
+llm_load_print_meta: f_clamp_kqv      = 0.0e+00
+llm_load_print_meta: f_max_alibi_bias = 0.0e+00
+llm_load_print_meta: f_logit_scale    = 0.0e+00
+llm_load_print_meta: n_ff             = 18432
+llm_load_print_meta: n_expert         = 256
+llm_load_print_meta: n_expert_used    = 8
+llm_load_print_meta: causal attn      = 1
+llm_load_print_meta: pooling type     = 0
+llm_load_print_meta: rope type        = 0
+llm_load_print_meta: rope scaling     = yarn
+llm_load_print_meta: freq_base_train  = 10000.0
+llm_load_print_meta: freq_scale_train = 0.025
+llm_load_print_meta: n_ctx_orig_yarn  = 4096
+llm_load_print_meta: rope_finetuned   = unknown
+llm_load_print_meta: ssm_d_conv       = 0
+llm_load_print_meta: ssm_d_inner      = 0
+llm_load_print_meta: ssm_d_state      = 0
+llm_load_print_meta: ssm_dt_rank      = 0
+llm_load_print_meta: model type       = 671B
+llm_load_print_meta: model ftype      = IQ3_K - 3.4325 bpw
+llm_load_print_meta: model params     = 671.026 B
+llm_load_print_meta: model size       = 290.897 GiB (3.724 BPW)
+llm_load_print_meta: repeating layers = 289.697 GiB (3.719 BPW, 669.173 B parameters)
+llm_load_print_meta: general.name     = DeepSeek V3.2 Speciale Bf16 Safetensors
+llm_load_print_meta: n_layer_dense_lead   = 3
+llm_load_print_meta: n_lora_q             = 1536
+llm_load_print_meta: n_lora_kv            = 512
+llm_load_print_meta: n_ff_exp             = 2048
+llm_load_print_meta: n_expert_shared      = 1
+llm_load_print_meta: expert_weights_scale = 2.5
+llm_load_print_meta: expert_weights_norm  = 1
+llm_load_print_meta: expert_gating_func   = sigmoid
+llm_load_print_meta: rope_yarn_log_mul    = 0.1000
+print_info: vocab type       = BPE
+print_info: n_vocab          = 129280
+print_info: n_merges         = 127741
+print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
+print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
+print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
+print_info: PAD token        = 1 '<｜end▁of▁sentence｜>'
+print_info: LF token         = 201 'Ċ'
+print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
+print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
+print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
+print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
+print_info: max token length = 256
+llm_load_tensors: ggml ctx size =    0.45 MiB
+llm_load_tensors: offloading 0 repeating layers to GPU
+llm_load_tensors: offloaded 0/62 layers to GPU
+llm_load_tensors:        CPU buffer size = 297878.97 MiB
+....................................................................................................
+============ llm_prepare_mla: need to compute 61 wkv_b tensors
+================= Adjusted mainline llama.cpp MLA tensors to ik_llama.cpp
+Computed blk.0.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.1.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.2.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.3.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.4.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.5.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.6.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.7.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.8.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.9.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.10.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.11.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.12.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.13.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.14.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.15.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.16.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.17.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.18.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.19.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.20.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.21.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.22.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.23.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.24.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.25.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.26.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.27.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.28.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.29.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.30.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.31.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.32.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.33.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.34.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.35.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.36.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.37.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.38.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.39.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.40.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.41.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.42.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.43.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.44.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.45.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.46.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.47.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.48.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.49.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.50.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.51.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.52.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.53.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.54.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.55.allama_new_context_with_model: n_ctx         = 4096
+llama_new_context_with_model: n_batch       = 4096
+llama_new_context_with_model: n_ubatch      = 4096
+llama_new_context_with_model: flash_attn    = 1
+llama_new_context_with_model: mla_attn      = 3
+llama_new_context_with_model: attn_max_b    = 0
+llama_new_context_with_model: fused_moe     = 1
+llama_new_context_with_model: grouped er    = 0
+llama_new_context_with_model: fused_up_gate = 1
+llama_new_context_with_model: fused_mmad    = 1
+llama_new_context_with_model: rope_cache    = 0
+llama_new_context_with_model: graph_reuse   = 1
+llama_new_context_with_model: k_cache_hadam = 0
+llama_new_context_with_model: split_mode_graph_scheduling = 0
+llama_new_context_with_model: split_mode_f16= 1
+llama_new_context_with_model: sched_async   = 0
+llama_new_context_with_model: ser           = -1, 0
+llama_new_context_with_model: freq_base     = 10000.0
+llama_new_context_with_model: freq_scale    = 0.025
+llama_kv_cache_init:        CPU KV buffer size =   274.50 MiB
+llama_new_context_with_model: KV self size  =  274.50 MiB, c^KV (f16):  274.50 MiB, kv^T: not used
+llama_new_context_with_model:        CPU  output buffer size =     3.95 MiB
+llama_new_context_with_model:        CPU compute buffer size =  2132.00 MiB
+llama_new_context_with_model: graph nodes  = 3248
+llama_new_context_with_model: graph splits = 1
+XXXXXXXXXXXXXXXXXXXXX Setting only active experts offload
+system_info: n_threads = 96 (n_threads_batch = 128) / 512 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
+perplexity: tokenizing the input ..
+perplexity: tokenization took 716.237 ms
+perplexity: calculating perplexity over 561 chunks, n_ctx=512, batch_size=4096, n_seq=8
+perplexity: 21.77 seconds per pass - ETA 25.43 minutes
+ttn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.56.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.57.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.58.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.59.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.60.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+======================================= HAVE_FANCY_SIMD is defined
+[1]2.3565,[2]3.1626,[3]2.3033,[4]1.9245,[5]1.7426,[6]1.6089,[7]1.5253,[8]1.4644,[9]1.4222,[10]1.3868,[11]1.3744,[12]1.3853,[13]1.4045,[14]1.5218,[15]1.6496,[16]1.7098,[17]1.8719,[18]1.9915,[19]1.9583,[20]1.9386,[21]2.0457,[22]2.0174,[23]1.9939,[24]2.0072,[25]1.9809,[26]1.9607,[27]2.0075,[28]2.0140,[29]2.0594,[30]2.0920,[31]2.1230,[32]2.1389,[33]2.1739,[34]2.2161,[35]2.2624,[36]2.3129,[37]2.3488,[38]2.3958,[39]2.4371,[40]2.4941,[41]2.5312,[42]2.5435,[43]2.5917,[44]2.6048,[45]2.6803,[46]2.7289,[47]2.6887,[48]2.6468,[49]2.6287,[50]2.6471,[51]2.6908,[52]2.7037,[53]2.7561,[54]2.7705,[55]2.8019,[56]2.8326,[57]2.8470,[58]2.8819,[59]2.8950,[60]2.9406,[61]2.9795,[62]3.0269,[63]3.0582,[64]3.0983,[65]3.1073,[66]3.0885,[67]3.0637,[68]3.0887,[69]3.0848,[70]3.0996,[71]3.1167,[72]3.1308,[73]3.1436,[74]3.1661,[75]3.1471,[76]3.1015,[77]3.0591,[78]3.0545,[79]3.0308,[80]3.0138,[81]2.9803,[82]2.9830,[83]2.9518,[84]2.9178,[85]2.8864,[86]2.8625,[87]2.8558,[88]2.8291,[89]2.8119,[90]2.7882,[91]2.7611,[92]2.7397,[93]2.7162,[94]2.6936,[95]2.6721,[96]2.6694,[97]2.6757,[98]2.6618,[99]2.6443,[100]2.6447,[101]2.6361,[102]2.6543,[103]2.6797,[104]2.6983,[105]2.6955,[106]2.7188,[107]2.7431,[108]2.7633,[109]2.7953,[110]2.8283,[111]2.8464,[112]2.8218,[113]2.8101,[114]2.7886,[115]2.7740,[116]2.7617,[117]2.7415,[118]2.7210,[119]2.7003,[120]2.6825,[121]2.6658,[122]2.6488,[123]2.6333,[124]2.6156,[125]2.5987,[126]2.5826,[127]2.5697,[128]2.5600,[129]2.5503,[130]2.5384,[131]2.5288,[132]2.5343,[133]2.5437,[134]2.5495,[135]2.5597,[136]2.5746,[137]2.5877,[138]2.5964,[139]2.6070,[140]2.6078,[141]2.6096,[142]2.6089,[143]2.6099,[144]2.6070,[145]2.5985,[146]2.5967,[147]2.6011,[148]2.6010,[149]2.6024,[150]2.5978,[151]2.5956,[152]2.5919,[153]2.5886,[154]2.5887,[155]2.5926,[156]2.5935,[157]2.5993,[158]2.6077,[159]2.6106,[160]2.6186,[161]2.6270,[162]2.6372,[163]2.6407,[164]2.6599,[165]2.6831,[166]2.7001,[167]2.7111,[168]2.7351,[169]2.7580,[170]2.7793,[171]2.8016,[172]2.7863,[173]2.7704,[174]2.7570,[175]2.7446,[176]2.7337,[177]2.7226,[178]2.7103,[179]2.6970,[180]2.7009,[181]2.7144,[182]2.7290,[183]2.7427,[184]2.7561,[185]2.7653,[186]2.7813,[187]2.7960,[188]2.8100,[189]2.8211,[190]2.8219,[191]2.8286,[192]2.8319,[193]2.8367,[194]2.8559,[195]2.8646,[196]2.8777,[197]2.8878,[198]2.8925,[199]2.8979,[200]2.8970,[201]2.9116,[202]2.9067,[203]2.9115,[204]2.9145,[205]2.9145,[206]2.9173,[207]2.9249,[208]2.9346,[209]2.9446,[210]2.9451,[211]2.9409,[212]2.9415,[213]2.9492,[214]2.9511,[215]2.9561,[216]2.9561,[217]2.9517,[218]2.9522,[219]2.9538,[220]2.9533,[221]2.9545,[222]2.9549,[223]2.9549,[224]2.9599,[225]2.9613,[226]2.9533,[227]2.9503,[228]2.9516,[229]2.9554,[230]2.9615,[231]2.9675,[232]2.9597,[233]2.9528,[234]2.9528,[235]2.9501,[236]2.9588,[237]2.9670,[238]2.9772,[239]2.9868,[240]2.9965,[241]3.0077,[242]3.0217,[243]3.0343,[244]3.0434,[245]3.0556,[246]3.0666,[247]3.0651,[248]3.0606,[249]3.0581,[250]3.0514,[251]3.0492,[252]3.0520,[253]3.0556,[254]3.0625,[255]3.0690,[256]3.0727,[257]3.0756,[258]3.0767,[259]3.0804,[260]3.0832,[261]3.0846,[262]3.0834,[263]3.0895,[264]3.0920,[265]3.0922,[266]3.0941,[267]3.0963,[268]3.0995,[269]3.1030,[270]3.1021,[271]3.1003,[272]3.0937,[273]3.0930,[274]3.0860,[275]3.0758,[276]3.0654,[277]3.0679,[278]3.0772,[279]3.0833,[280]3.0910,[281]3.0984,[282]3.1044,[283]3.1108,[284]3.1169,[285]3.1307,[286]3.1330,[287]3.1363,[288]3.1412,[289]3.1434,[290]3.1357,[291]3.1273,[292]3.1248,[293]3.1234,[294]3.1209,[295]3.1180,[296]3.1200,[297]3.1201,[298]3.1251,[299]3.1298,[300]3.1320,[301]3.1355,[302]3.1373,[303]3.1381,[304]3.1376,[305]3.1489,[306]3.1563,[307]3.1669,[308]3.1560,[309]3.1496,[310]3.1401,[311]3.1417,[312]3.1416,[313]3.1461,[314]3.1475,[315]3.1504,[316]3.1521,[317]3.1536,[318]3.1539,[319]3.1536,[320]3.1573,[321]3.1571,[322]3.1586,[323]3.1641,[324]3.1649,[325]3.1700,[326]3.1735,[327]3.1769,[328]3.1796,[329]3.1811,[330]3.1874,[331]3.1911,[332]3.1955,[333]3.1946,[334]3.1950,[335]3.1959,[336]3.1961,[337]3.1974,[338]3.1979,[339]3.2004,[340]3.2042,[341]3.2096,[342]3.2181,[343]3.2272,[344]3.2323,[345]3.2233,[346]3.2166,[347]3.2106,[348]3.2036,[349]3.1995,[350]3.1980,[351]3.2028,[352]3.2165,[353]3.2251,[354]3.2371,[355]3.2455,[356]3.2505,[357]3.2622,[358]3.2715,[359]3.2744,[360]3.2807,[361]3.2900,[362]3.2983,[363]3.3040,[364]3.3105,[365]3.3162,[366]3.3262,[367]3.3346,[368]3.3417,[369]3.3493,[370]3.3576,[371]3.3711,[372]3.3793,[373]3.3826,[374]3.3857,[375]3.3907,[376]3.4028,[377]3.4135,[378]3.4162,[379]3.4160,[380]3.4135,[381]3.4185,[382]3.4242,[383]3.4272,[384]3.4318,[385]3.4357,[386]3.4415,[387]3.4471,[388]3.4506,[389]3.4405,[390]3.4316,[391]3.4215,[392]3.4165,[393]3.4076,[394]3.3995,[395]3.3909,[396]3.3814,[397]3.3731,[398]3.3642,[399]3.3546,[400]3.3462,[401]3.3365,[402]3.3267,[403]3.3187,[404]3.3092,[405]3.3003,[406]3.2908,[407]3.2819,[408]3.2733,[409]3.2655,[410]3.2602,[411]3.2619,[412]3.2578,[413]3.2598,[414]3.2617,[415]3.2578,[416]3.2583,[417]3.2606,[418]3.2550,[419]3.2566,[420]3.2540,[421]3.2531,[422]3.2539,[423]3.2534,[424]3.2569,[425]3.2569,[426]3.2573,[427]3.2568,[428]3.2592,[429]3.2607,[430]3.2635,[431]3.2646,[432]3.2637,[433]3.2603,[434]3.2605,[435]3.2535,[436]3.2484,[437]3.2447,[438]3.2432,[439]3.2411,[440]3.2462,[441]3.2512,[442]3.2585,[443]3.2558,[444]3.2562,[445]3.2571,[446]3.2614,[447]3.2643,[448]3.2671,[449]3.2701,[450]3.2738,[451]3.2767,[452]3.2786,[453]3.2800,[454]3.2790,[455]3.2811,[456]3.2817,[457]3.2845,[458]3.2894,[459]3.2901,[460]3.2903,[461]3.2872,[462]3.2907,[463]3.2979,[464]3.3028,[465]3.2965,[466]3.2955,[467]3.2946,[468]3.2969,[469]3.2945,[470]3.2920,[471]3.2924,[472]3.2934,[473]3.2931,[474]3.2926,[475]3.2938,[476]3.2925,[477]3.2919,[478]3.2928,[479]3.2948,[480]3.2973,[481]3.2938,[482]3.2974,[483]3.2967,[484]3.3003,[485]3.3068,[486]3.3102,[487]3.3135,[488]3.3184,[489]3.3210,[490]3.3255,[491]3.3314,[492]3.3355,[493]3.3350,[494]3.3363,[495]3.3387,[496]3.3409,[497]3.3437,[498]3.3441,[499]3.3434,[500]3.3471,[501]3.3515,[502]3.3494,[503]3.3478,[504]3.3495,[505]3.3529,[506]3.3602,[507]3.3631,[508]3.3665,[509]3.3594,[510]3.3552,[511]3.3491,[512]3.3454,[513]3.3396,[514]3.3386,[515]3.3410,[516]3.3364,[517]3.3371,[518]3.3370,[519]3.3374,[520]3.3418,[521]3.3408,[522]3.3392,[523]3.3446,[524]3.3424,[525]3.3409,[526]3.3366,[527]3.3317,[528]3.3287,[529]3.3255,[530]3.3227,[531]3.3197,[532]3.3141,[533]3.3081,[534]3.3034,[535]3.3035,[536]3.3058,[537]3.3084,[538]3.3099,[539]3.3121,[540]3.3166,[541]3.3191,[542]3.3213,[543]3.3159,[544]3.3126,[545]3.3122,[546]3.3057,[547]3.2998,[548]3.2935,[549]3.2870,[550]3.2810,[551]3.2750,[552]3.2692,[553]3.2634,[554]3.2615,[555]3.2599,[556]3.2627,[557]3.2664,[558]3.2723,[559]3.2762,[560]3.2812,[561]3.2793,
+llama_print_timings:        load time =  109797.04 ms
+llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings: prompt eval time = 1301597.31 ms / 287232 tokens (    4.53 ms per token,   220.68 tokens per second)
+llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings:       total time = 1312944.55 ms / 287233 tokens
+Final estimate: PPL over 561 chunks for n_ctx=512 = 3.2793 +/- 0.01789

logs/perplexity-DeepSeek-V3.2-Speciale-IQ5_K.log ADDED Viewed

	@@ -0,0 +1,263 @@

+numactl -N "$SOCKET" -m "$SOCKET" \
+./build/bin/llama-perplexity \
+    -m "$model" \
+    -f wiki.test.raw \
+    --seed 1337 \
+    --ctx-size 512 \
+    -mla 3 \
+    -ub 4096 -b 4096 \
+    --numa numactl \
+    --threads 96 \
+    --threads-batch 128 \
+    --validate-quants \
+    --no-mmap
+SOCKET is set to: 0
+main: build = 4099 (145e4f4e)
+main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
+main: seed  = 1337
+CPU: using device CPU - 0 MiB free
+llama_model_loader: additional 11 GGUFs metadata loaded.
+llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /mnt/raid/hf/DeepSeek-V3.2-Speciale-GGUF/IQ5_K/DeepSeek-V3.2-Speciale-IQ5_K-00001-of-00012.gguf (version GGUF V3 (latest))
+llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
+llama_model_loader: - kv   1:                               general.type str              = model
+llama_model_loader: - kv   2:                     general.sampling.top_p f32              = 0.950000
+llama_model_loader: - kv   3:                      general.sampling.temp f32              = 1.000000
+llama_model_loader: - kv   4:                               general.name str              = DeepSeek V3.2 Speciale Bf16 Safetensors
+llama_model_loader: - kv   5:                           general.finetune str              = safetensors
+llama_model_loader: - kv   6:                           general.basename str              = DeepSeek-V3.2-Speciale
+llama_model_loader: - kv   7:                         general.size_label str              = 256x20B
+llama_model_loader: - kv   8:                      deepseek2.block_count u32              = 61
+llama_model_loader: - kv   9:                   deepseek2.context_length u32              = 163840
+llama_model_loader: - kv  10:                 deepseek2.embedding_length u32              = 7168
+llama_model_loader: - kv  11:              deepseek2.feed_forward_length u32              = 18432
+llama_model_loader: - kv  12:             deepseek2.attention.head_count u32              = 128
+llama_model_loader: - kv  13:          deepseek2.attention.head_count_kv u32              = 1
+llama_model_loader: - kv  14:                deepseek2.rope.scaling.type str              = yarn
+llama_model_loader: - kv  15:              deepseek2.rope.scaling.factor f32              = 40.000000
+llama_model_loader: - kv  16: deepseek2.rope.scaling.original_context_length u32              = 4096
+llama_model_loader: - kv  17:      deepseek2.rope.scaling.yarn_beta_fast f32              = 32.000000
+llama_model_loader: - kv  18:      deepseek2.rope.scaling.yarn_beta_slow f32              = 1.000000
+llama_model_loader: - kv  19:                   deepseek2.rope.freq_base f32              = 10000.000000
+llama_model_loader: - kv  20: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
+llama_model_loader: - kv  21:                deepseek2.expert_used_count u32              = 8
+llama_model_loader: - kv  22:               deepseek2.expert_group_count u32              = 8
+llama_model_loader: - kv  23:          deepseek2.expert_group_used_count u32              = 4
+llama_model_loader: - kv  24:               deepseek2.expert_gating_func u32              = 2
+llama_model_loader: - kv  25:                          general.file_type u32              = 141
+llama_model_loader: - kv  26:        deepseek2.leading_dense_block_count u32              = 3
+llama_model_loader: - kv  27:                       deepseek2.vocab_size u32              = 129280
+llama_model_loader: - kv  28:            deepseek2.attention.q_lora_rank u32              = 1536
+llama_model_loader: - kv  29:           deepseek2.attention.kv_lora_rank u32              = 512
+llama_model_loader: - kv  30:             deepseek2.attention.key_length u32              = 576
+llama_model_loader: - kv  31:           deepseek2.attention.value_length u32              = 512
+llama_model_loader: - kv  32:         deepseek2.attention.key_length_mla u32              = 192
+llama_model_loader: - kv  33:       deepseek2.attention.value_length_mla u32              = 128
+llama_model_loader: - kv  34:       deepseek2.expert_feed_forward_length u32              = 2048
+llama_model_loader: - kv  35:                     deepseek2.expert_count u32              = 256
+llama_model_loader: - kv  36:              deepseek2.expert_shared_count u32              = 1
+llama_model_loader: - kv  37:             deepseek2.expert_weights_scale f32              = 2.500000
+llama_model_loader: - kv  38:              deepseek2.expert_weights_norm bool             = true
+llama_model_loader: - kv  39:             deepseek2.rope.dimension_count u32              = 64
+llama_model_loader: - kv  40: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
+llama_model_loader: - kv  41:               general.quantization_version u32              = 2
+llama_model_loader: - kv  42:                       tokenizer.ggml.model str              = gpt2
+llama_model_loader: - kv  43:                         tokenizer.ggml.pre str              = deepseek-v3
+llama_model_loader: - kv  44:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
+llama_model_loader: - kv  45:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
+llama_model_loader: - kv  46:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
+llama_model_loader: - kv  47:                tokenizer.ggml.bos_token_id u32              = 0
+llama_model_loader: - kv  48:                tokenizer.ggml.eos_token_id u32              = 1
+llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 1
+llama_model_loader: - kv  50:               tokenizer.ggml.add_bos_token bool             = true
+llama_model_loader: - kv  51:               tokenizer.ggml.add_eos_token bool             = false
+llama_model_loader: - kv  52:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
+llama_model_loader: - kv  53:                      quantize.imatrix.file str              = /mnt/data/models/ubergarm/DeepSeek-V3...
+llama_model_loader: - kv  54:                   quantize.imatrix.dataset str              = ubergarm-imatrix-calibration-corpus-v...
+llama_model_loader: - kv  55:             quantize.imatrix.entries_count i32              = 782
+llama_model_loader: - kv  56:              quantize.imatrix.chunks_count i32              = 812
+llama_model_loader: - kv  57:                                   split.no u16              = 0
+llama_model_loader: - kv  58:                                split.count u16              = 12
+llama_model_loader: - kv  59:                        split.tensors.count i32              = 1086
+llama_model_loader: - type  f32:  361 tensors
+llama_model_loader: - type q8_0:  551 tensors
+llama_model_loader: - type iq5_k:  116 tensors
+llama_model_loader: - type iq6_k:   58 tensors
+load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
+load: printing all EOG tokens:
+load:   - 1 ('<｜end▁of▁sentence｜>')
+load: special tokens cache size = 818
+load: token to piece cache size = 0.8222 MB
+llm_load_print_meta: format           = GGUF V3 (latest)
+llm_load_print_meta: arch             = deepseek2
+llm_load_print_meta: n_ctx_train      = 163840
+llm_load_print_meta: n_embd           = 7168
+llm_load_print_meta: n_layer          = 61
+llm_load_print_meta: n_head           = 128
+llm_load_print_meta: n_head_kv        = 128
+llm_load_print_meta: n_rot            = 64
+llm_load_print_meta: n_swa            = 0
+llm_load_print_meta: n_swa_pattern    = 1
+llm_load_print_meta: n_embd_head_k    = 192
+llm_load_print_meta: n_embd_head_v    = 128
+llm_load_print_meta: n_gqa            = 1
+llm_load_print_meta: n_embd_k_gqa     = 24576
+llm_load_print_meta: n_embd_v_gqa     = 16384
+llm_load_print_meta: f_norm_eps       = 0.0e+00
+llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
+llm_load_print_meta: f_clamp_kqv      = 0.0e+00
+llm_load_print_meta: f_max_alibi_bias = 0.0e+00
+llm_load_print_meta: f_logit_scale    = 0.0e+00
+llm_load_print_meta: n_ff             = 18432
+llm_load_print_meta: n_expert         = 256
+llm_load_print_meta: n_expert_used    = 8
+llm_load_print_meta: causal attn      = 1
+llm_load_print_meta: pooling type     = 0
+llm_load_print_meta: rope type        = 0
+llm_load_print_meta: rope scaling     = yarn
+llm_load_print_meta: freq_base_train  = 10000.0
+llm_load_print_meta: freq_scale_train = 0.025
+llm_load_print_meta: n_ctx_orig_yarn  = 4096
+llm_load_print_meta: rope_finetuned   = unknown
+llm_load_print_meta: ssm_d_conv       = 0
+llm_load_print_meta: ssm_d_inner      = 0
+llm_load_print_meta: ssm_d_state      = 0
+llm_load_print_meta: ssm_dt_rank      = 0
+llm_load_print_meta: model type       = 671B
+llm_load_print_meta: model ftype      = IQ5_K - 5.5 bpw
+llm_load_print_meta: model params     = 671.026 B
+llm_load_print_meta: model size       = 464.467 GiB (5.946 BPW)
+llm_load_print_meta: repeating layers = 462.633 GiB (5.939 BPW, 669.173 B parameters)
+llm_load_print_meta: general.name     = DeepSeek V3.2 Speciale Bf16 Safetensors
+llm_load_print_meta: n_layer_dense_lead   = 3
+llm_load_print_meta: n_lora_q             = 1536
+llm_load_print_meta: n_lora_kv            = 512
+llm_load_print_meta: n_ff_exp             = 2048
+llm_load_print_meta: n_expert_shared      = 1
+llm_load_print_meta: expert_weights_scale = 2.5
+llm_load_print_meta: expert_weights_norm  = 1
+llm_load_print_meta: expert_gating_func   = sigmoid
+llm_load_print_meta: rope_yarn_log_mul    = 0.1000
+print_info: vocab type       = BPE
+print_info: n_vocab          = 129280
+print_info: n_merges         = 127741
+print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
+print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
+print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
+print_info: PAD token        = 1 '<｜end▁of▁sentence｜>'
+print_info: LF token         = 201 'Ċ'
+print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
+print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
+print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
+print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
+print_info: max token length = 256
+llm_load_tensors: ggml ctx size =    0.45 MiB
+llm_load_tensors: offloading 0 repeating layers to GPU
+llm_load_tensors: offloaded 0/62 layers to GPU
+llm_load_tensors:        CPU buffer size = 475613.97 MiB
+....................................................................................................
+============ llm_prepare_mla: need to compute 61 wkv_b tensors
+================= Adjusted mainline llama.cpp MLA tensors to ik_llama.cpp
+Computed blk.0.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.1.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.2.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.3.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.4.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.5.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.6.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.7.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.8.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.9.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.10.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.11.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.12.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.13.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.14.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.15.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.16.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.17.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.18.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.19.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.20.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.21.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.22.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.23.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.24.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.25.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.26.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.27.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.28.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.29.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.30.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.31.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.32.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.33.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.34.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.35.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.36.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.37.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.38.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.39.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.40.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.41.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.42.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.43.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.44.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.45.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.46.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.47.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.48.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.49.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.50.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.51.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.52.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.53.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.54.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.55.allama_new_context_with_model: n_ctx         = 4096
+llama_new_context_with_model: n_batch       = 4096
+llama_new_context_with_model: n_ubatch      = 4096
+llama_new_context_with_model: flash_attn    = 1
+llama_new_context_with_model: mla_attn      = 3
+llama_new_context_with_model: attn_max_b    = 0
+llama_new_context_with_model: fused_moe     = 1
+llama_new_context_with_model: grouped er    = 0
+llama_new_context_with_model: fused_up_gate = 1
+llama_new_context_with_model: fused_mmad    = 1
+llama_new_context_with_model: rope_cache    = 0
+llama_new_context_with_model: graph_reuse   = 1
+llama_new_context_with_model: k_cache_hadam = 0
+llama_new_context_with_model: split_mode_graph_scheduling = 0
+llama_new_context_with_model: split_mode_f16= 1
+llama_new_context_with_model: sched_async   = 0
+llama_new_context_with_model: ser           = -1, 0
+llama_new_context_with_model: freq_base     = 10000.0
+llama_new_context_with_model: freq_scale    = 0.025
+llama_kv_cache_init:        CPU KV buffer size =   274.50 MiB
+llama_new_context_with_model: KV self size  =  274.50 MiB, c^KV (f16):  274.50 MiB, kv^T: not used
+llama_new_context_with_model:        CPU  output buffer size =     3.95 MiB
+llama_new_context_with_model:        CPU compute buffer size =  2132.00 MiB
+llama_new_context_with_model: graph nodes  = 3248
+llama_new_context_with_model: graph splits = 1
+XXXXXXXXXXXXXXXXXXXXX Setting only active experts offload
+system_info: n_threads = 96 (n_threads_batch = 128) / 512 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
+perplexity: tokenizing the input ..
+perplexity: tokenization took 455.57 ms
+perplexity: calculating perplexity over 561 chunks, n_ctx=512, batch_size=4096, n_seq=8
+perplexity: 25.99 seconds per pass - ETA 30.37 minutes
+ttn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.56.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.57.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.58.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.59.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.60.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+======================================= HAVE_FANCY_SIMD is defined
+[1]2.3446,[2]3.1812,[3]2.3136,[4]1.9288,[5]1.7391,[6]1.6041,[7]1.5178,[8]1.4570,[9]1.4126,[10]1.3784,[11]1.3637,[12]1.3680,[13]1.3804,[14]1.4992,[15]1.6250,[16]1.6856,[17]1.8466,[18]1.9657,[19]1.9301,[20]1.9048,[21]2.0121,[22]1.9854,[23]1.9617,[24]1.9740,[25]1.9461,[26]1.9251,[27]1.9698,[28]1.9763,[29]2.0201,[30]2.0514,[31]2.0842,[32]2.0993,[33]2.1335,[34]2.1717,[35]2.2167,[36]2.2673,[37]2.3039,[38]2.3512,[39]2.3926,[40]2.4490,[41]2.4849,[42]2.4968,[43]2.5460,[44]2.5592,[45]2.6341,[46]2.6816,[47]2.6413,[48]2.5994,[49]2.5772,[50]2.5951,[51]2.6378,[52]2.6512,[53]2.7002,[54]2.7135,[55]2.7445,[56]2.7761,[57]2.7902,[58]2.8219,[59]2.8349,[60]2.8789,[61]2.9165,[62]2.9625,[63]2.9935,[64]3.0336,[65]3.0420,[66]3.0213,[67]2.9978,[68]3.0242,[69]3.0186,[70]3.0334,[71]3.0517,[72]3.0669,[73]3.0809,[74]3.1032,[75]3.0844,[76]3.0405,[77]3.0003,[78]2.9949,[79]2.9703,[80]2.9509,[81]2.9173,[82]2.9202,[83]2.8909,[84]2.8579,[85]2.8253,[86]2.7999,[87]2.7904,[88]2.7637,[89]2.7461,[90]2.7227,[91]2.6952,[92]2.6733,[93]2.6482,[94]2.6246,[95]2.6041,[96]2.6016,[97]2.6077,[98]2.5923,[99]2.5751,[100]2.5761,[101]2.5682,[102]2.5850,[103]2.6096,[104]2.6280,[105]2.6249,[106]2.6466,[107]2.6706,[108]2.6910,[109]2.7228,[110]2.7551,[111]2.7740,[112]2.7506,[113]2.7389,[114]2.7183,[115]2.7045,[116]2.6918,[117]2.6715,[118]2.6516,[119]2.6316,[120]2.6140,[121]2.5975,[122]2.5813,[123]2.5652,[124]2.5477,[125]2.5316,[126]2.5164,[127]2.5031,[128]2.4927,[129]2.4819,[130]2.4703,[131]2.4605,[132]2.4664,[133]2.4759,[134]2.4819,[135]2.4921,[136]2.5061,[137]2.5191,[138]2.5277,[139]2.5379,[140]2.5394,[141]2.5418,[142]2.5416,[143]2.5427,[144]2.5404,[145]2.5318,[146]2.5303,[147]2.5351,[148]2.5351,[149]2.5367,[150]2.5324,[151]2.5306,[152]2.5275,[153]2.5245,[154]2.5246,[155]2.5286,[156]2.5300,[157]2.5360,[158]2.5447,[159]2.5475,[160]2.5555,[161]2.5640,[162]2.5743,[163]2.5771,[164]2.5962,[165]2.6195,[166]2.6365,[167]2.6473,[168]2.6708,[169]2.6933,[170]2.7147,[171]2.7365,[172]2.7216,[173]2.7063,[174]2.6933,[175]2.6812,[176]2.6701,[177]2.6590,[178]2.6473,[179]2.6342,[180]2.6381,[181]2.6515,[182]2.6662,[183]2.6797,[184]2.6929,[185]2.7025,[186]2.7186,[187]2.7333,[188]2.7472,[189]2.7583,[190]2.7592,[191]2.7662,[192]2.7696,[193]2.7747,[194]2.7935,[195]2.8024,[196]2.8155,[197]2.8256,[198]2.8304,[199]2.8357,[200]2.8352,[201]2.8498,[202]2.8452,[203]2.8500,[204]2.8534,[205]2.8538,[206]2.8568,[207]2.8649,[208]2.8746,[209]2.8847,[210]2.8851,[211]2.8811,[212]2.8819,[213]2.8897,[214]2.8916,[215]2.8965,[216]2.8967,[217]2.8925,[218]2.8933,[219]2.8951,[220]2.8949,[221]2.8962,[222]2.8968,[223]2.8969,[224]2.9020,[225]2.9033,[226]2.8955,[227]2.8925,[228]2.8939,[229]2.8977,[230]2.9037,[231]2.9100,[232]2.9025,[233]2.8949,[234]2.8950,[235]2.8923,[236]2.9006,[237]2.9084,[238]2.9183,[239]2.9279,[240]2.9376,[241]2.9486,[242]2.9625,[243]2.9750,[244]2.9841,[245]2.9957,[246]3.0065,[247]3.0053,[248]3.0009,[249]2.9987,[250]2.9928,[251]2.9909,[252]2.9937,[253]2.9976,[254]3.0045,[255]3.0111,[256]3.0147,[257]3.0176,[258]3.0191,[259]3.0225,[260]3.0253,[261]3.0266,[262]3.0255,[263]3.0315,[264]3.0339,[265]3.0341,[266]3.0360,[267]3.0385,[268]3.0415,[269]3.0449,[270]3.0441,[271]3.0422,[272]3.0360,[273]3.0355,[274]3.0283,[275]3.0181,[276]3.0080,[277]3.0108,[278]3.0201,[279]3.0259,[280]3.0336,[281]3.0412,[282]3.0471,[283]3.0532,[284]3.0593,[285]3.0728,[286]3.0750,[287]3.0782,[288]3.0831,[289]3.0856,[290]3.0780,[291]3.0693,[292]3.0664,[293]3.0645,[294]3.0611,[295]3.0575,[296]3.0591,[297]3.0595,[298]3.0645,[299]3.0692,[300]3.0713,[301]3.0747,[302]3.0763,[303]3.0769,[304]3.0764,[305]3.0877,[306]3.0950,[307]3.1055,[308]3.0949,[309]3.0880,[310]3.0788,[311]3.0803,[312]3.0798,[313]3.0841,[314]3.0856,[315]3.0884,[316]3.0900,[317]3.0916,[318]3.0918,[319]3.0916,[320]3.0954,[321]3.0953,[322]3.0970,[323]3.1023,[324]3.1031,[325]3.1081,[326]3.1118,[327]3.1152,[328]3.1182,[329]3.1198,[330]3.1261,[331]3.1296,[332]3.1339,[333]3.1329,[334]3.1334,[335]3.1344,[336]3.1346,[337]3.1360,[338]3.1365,[339]3.1389,[340]3.1427,[341]3.1481,[342]3.1568,[343]3.1657,[344]3.1708,[345]3.1616,[346]3.1544,[347]3.1480,[348]3.1408,[349]3.1364,[350]3.1350,[351]3.1397,[352]3.1533,[353]3.1619,[354]3.1739,[355]3.1822,[356]3.1876,[357]3.1991,[358]3.2084,[359]3.2115,[360]3.2177,[361]3.2269,[362]3.2352,[363]3.2408,[364]3.2472,[365]3.2528,[366]3.2627,[367]3.2711,[368]3.2780,[369]3.2856,[370]3.2939,[371]3.3073,[372]3.3152,[373]3.3187,[374]3.3218,[375]3.3266,[376]3.3389,[377]3.3495,[378]3.3523,[379]3.3521,[380]3.3497,[381]3.3545,[382]3.3602,[383]3.3633,[384]3.3678,[385]3.3717,[386]3.3775,[387]3.3830,[388]3.3866,[389]3.3768,[390]3.3681,[391]3.3582,[392]3.3531,[393]3.3440,[394]3.3361,[395]3.3275,[396]3.3183,[397]3.3102,[398]3.3016,[399]3.2920,[400]3.2836,[401]3.2742,[402]3.2647,[403]3.2568,[404]3.2476,[405]3.2388,[406]3.2296,[407]3.2210,[408]3.2127,[409]3.2050,[410]3.1999,[411]3.2012,[412]3.1973,[413]3.1993,[414]3.2011,[415]3.1975,[416]3.1969,[417]3.1990,[418]3.1935,[419]3.1950,[420]3.1925,[421]3.1914,[422]3.1920,[423]3.1912,[424]3.1948,[425]3.1947,[426]3.1950,[427]3.1940,[428]3.1967,[429]3.1981,[430]3.2009,[431]3.2019,[432]3.2008,[433]3.1974,[434]3.1974,[435]3.1901,[436]3.1846,[437]3.1809,[438]3.1796,[439]3.1771,[440]3.1821,[441]3.1871,[442]3.1942,[443]3.1914,[444]3.1922,[445]3.1931,[446]3.1977,[447]3.2006,[448]3.2033,[449]3.2063,[450]3.2103,[451]3.2132,[452]3.2150,[453]3.2162,[454]3.2151,[455]3.2174,[456]3.2181,[457]3.2209,[458]3.2258,[459]3.2267,[460]3.2270,[461]3.2241,[462]3.2276,[463]3.2349,[464]3.2398,[465]3.2333,[466]3.2321,[467]3.2308,[468]3.2330,[469]3.2308,[470]3.2283,[471]3.2288,[472]3.2295,[473]3.2293,[474]3.2288,[475]3.2299,[476]3.2285,[477]3.2279,[478]3.2289,[479]3.2308,[480]3.2333,[481]3.2299,[482]3.2335,[483]3.2330,[484]3.2365,[485]3.2431,[486]3.2464,[487]3.2496,[488]3.2544,[489]3.2570,[490]3.2615,[491]3.2673,[492]3.2714,[493]3.2710,[494]3.2724,[495]3.2749,[496]3.2770,[497]3.2799,[498]3.2804,[499]3.2800,[500]3.2837,[501]3.2881,[502]3.2861,[503]3.2845,[504]3.2859,[505]3.2893,[506]3.2966,[507]3.2996,[508]3.3030,[509]3.2960,[510]3.2908,[511]3.2848,[512]3.2809,[513]3.2750,[514]3.2737,[515]3.2760,[516]3.2713,[517]3.2717,[518]3.2712,[519]3.2714,[520]3.2757,[521]3.2747,[522]3.2733,[523]3.2787,[524]3.2762,[525]3.2745,[526]3.2701,[527]3.2654,[528]3.2621,[529]3.2592,[530]3.2564,[531]3.2534,[532]3.2482,[533]3.2423,[534]3.2372,[535]3.2371,[536]3.2394,[537]3.2420,[538]3.2433,[539]3.2453,[540]3.2498,[541]3.2520,[542]3.2540,[543]3.2488,[544]3.2453,[545]3.2446,[546]3.2383,[547]3.2322,[548]3.2261,[549]3.2195,[550]3.2137,[551]3.2079,[552]3.2022,[553]3.1965,[554]3.1941,[555]3.1925,[556]3.1954,[557]3.1991,[558]3.2049,[559]3.2088,[560]3.2140,[561]3.2122,
+llama_print_timings:        load time =  174468.42 ms
+llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings: prompt eval time = 1524187.01 ms / 287232 tokens (    5.31 ms per token,   188.45 tokens per second)
+llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings:       total time = 1534925.89 ms / 287233 tokens
+Final estimate: PPL over 561 chunks for n_ctx=512 = 3.2122 +/- 0.01744

logs/perplexity-DeepSeek-V3.2-Speciale-Q8_0.log ADDED Viewed

	@@ -0,0 +1,253 @@

+numactl -N "$SOCKET" -m "$SOCKET" \
+./build/bin/llama-perplexity \
+    -m "$model" \
+    -f wiki.test.raw \
+    --seed 1337 \
+    --ctx-size 512 \
+    -mla 3 \
+    -ub 4096 -b 4096 \
+    --numa numactl \
+    --threads 96 \
+    --threads-batch 128 \
+    --validate-quants \
+    --no-mmap
+SOCKET is set to: 0
+main: build = 4099 (145e4f4e)
+main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
+main: seed  = 1337
+CPU: using device CPU - 0 MiB free
+llama_model_loader: loaded meta data with 53 key-value pairs and 1086 tensors from /mnt/data/models/ubergarm/DeepSeek-V3.2-Speciale-GGUF/DeepSeek-V3.2-Speciale-Q8_0.gguf (version GGUF V3 (latest))
+llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
+llama_model_loader: - kv   1:                               general.type str              = model
+llama_model_loader: - kv   2:                     general.sampling.top_p f32              = 0.950000
+llama_model_loader: - kv   3:                      general.sampling.temp f32              = 1.000000
+llama_model_loader: - kv   4:                               general.name str              = DeepSeek V3.2 Speciale Bf16 Safetensors
+llama_model_loader: - kv   5:                           general.finetune str              = safetensors
+llama_model_loader: - kv   6:                           general.basename str              = DeepSeek-V3.2-Speciale
+llama_model_loader: - kv   7:                         general.size_label str              = 256x20B
+llama_model_loader: - kv   8:                      deepseek2.block_count u32              = 61
+llama_model_loader: - kv   9:                   deepseek2.context_length u32              = 163840
+llama_model_loader: - kv  10:                 deepseek2.embedding_length u32              = 7168
+llama_model_loader: - kv  11:              deepseek2.feed_forward_length u32              = 18432
+llama_model_loader: - kv  12:             deepseek2.attention.head_count u32              = 128
+llama_model_loader: - kv  13:          deepseek2.attention.head_count_kv u32              = 1
+llama_model_loader: - kv  14:                deepseek2.rope.scaling.type str              = yarn
+llama_model_loader: - kv  15:              deepseek2.rope.scaling.factor f32              = 40.000000
+llama_model_loader: - kv  16: deepseek2.rope.scaling.original_context_length u32              = 4096
+llama_model_loader: - kv  17:      deepseek2.rope.scaling.yarn_beta_fast f32              = 32.000000
+llama_model_loader: - kv  18:      deepseek2.rope.scaling.yarn_beta_slow f32              = 1.000000
+llama_model_loader: - kv  19:                   deepseek2.rope.freq_base f32              = 10000.000000
+llama_model_loader: - kv  20: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
+llama_model_loader: - kv  21:                deepseek2.expert_used_count u32              = 8
+llama_model_loader: - kv  22:               deepseek2.expert_group_count u32              = 8
+llama_model_loader: - kv  23:          deepseek2.expert_group_used_count u32              = 4
+llama_model_loader: - kv  24:               deepseek2.expert_gating_func u32              = 2
+llama_model_loader: - kv  25:                          general.file_type u32              = 7
+llama_model_loader: - kv  26:        deepseek2.leading_dense_block_count u32              = 3
+llama_model_loader: - kv  27:                       deepseek2.vocab_size u32              = 129280
+llama_model_loader: - kv  28:            deepseek2.attention.q_lora_rank u32              = 1536
+llama_model_loader: - kv  29:           deepseek2.attention.kv_lora_rank u32              = 512
+llama_model_loader: - kv  30:             deepseek2.attention.key_length u32              = 576
+llama_model_loader: - kv  31:           deepseek2.attention.value_length u32              = 512
+llama_model_loader: - kv  32:         deepseek2.attention.key_length_mla u32              = 192
+llama_model_loader: - kv  33:       deepseek2.attention.value_length_mla u32              = 128
+llama_model_loader: - kv  34:       deepseek2.expert_feed_forward_length u32              = 2048
+llama_model_loader: - kv  35:                     deepseek2.expert_count u32              = 256
+llama_model_loader: - kv  36:              deepseek2.expert_shared_count u32              = 1
+llama_model_loader: - kv  37:             deepseek2.expert_weights_scale f32              = 2.500000
+llama_model_loader: - kv  38:              deepseek2.expert_weights_norm bool             = true
+llama_model_loader: - kv  39:             deepseek2.rope.dimension_count u32              = 64
+llama_model_loader: - kv  40: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
+llama_model_loader: - kv  41:               general.quantization_version u32              = 2
+llama_model_loader: - kv  42:                       tokenizer.ggml.model str              = gpt2
+llama_model_loader: - kv  43:                         tokenizer.ggml.pre str              = deepseek-v3
+llama_model_loader: - kv  44:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
+llama_model_loader: - kv  45:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
+llama_model_loader: - kv  46:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
+llama_model_loader: - kv  47:                tokenizer.ggml.bos_token_id u32              = 0
+llama_model_loader: - kv  48:                tokenizer.ggml.eos_token_id u32              = 1
+llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 1
+llama_model_loader: - kv  50:               tokenizer.ggml.add_bos_token bool             = true
+llama_model_loader: - kv  51:               tokenizer.ggml.add_eos_token bool             = false
+llama_model_loader: - kv  52:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
+llama_model_loader: - type  f32:  361 tensors
+llama_model_loader: - type q8_0:  725 tensors
+load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
+load: printing all EOG tokens:
+load:   - 1 ('<｜end▁of▁sentence｜>')
+load: special tokens cache size = 818
+load: token to piece cache size = 0.8222 MB
+llm_load_print_meta: format           = GGUF V3 (latest)
+llm_load_print_meta: arch             = deepseek2
+llm_load_print_meta: n_ctx_train      = 163840
+llm_load_print_meta: n_embd           = 7168
+llm_load_print_meta: n_layer          = 61
+llm_load_print_meta: n_head           = 128
+llm_load_print_meta: n_head_kv        = 128
+llm_load_print_meta: n_rot            = 64
+llm_load_print_meta: n_swa            = 0
+llm_load_print_meta: n_swa_pattern    = 1
+llm_load_print_meta: n_embd_head_k    = 192
+llm_load_print_meta: n_embd_head_v    = 128
+llm_load_print_meta: n_gqa            = 1
+llm_load_print_meta: n_embd_k_gqa     = 24576
+llm_load_print_meta: n_embd_v_gqa     = 16384
+llm_load_print_meta: f_norm_eps       = 0.0e+00
+llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
+llm_load_print_meta: f_clamp_kqv      = 0.0e+00
+llm_load_print_meta: f_max_alibi_bias = 0.0e+00
+llm_load_print_meta: f_logit_scale    = 0.0e+00
+llm_load_print_meta: n_ff             = 18432
+llm_load_print_meta: n_expert         = 256
+llm_load_print_meta: n_expert_used    = 8
+llm_load_print_meta: causal attn      = 1
+llm_load_print_meta: pooling type     = 0
+llm_load_print_meta: rope type        = 0
+llm_load_print_meta: rope scaling     = yarn
+llm_load_print_meta: freq_base_train  = 10000.0
+llm_load_print_meta: freq_scale_train = 0.025
+llm_load_print_meta: n_ctx_orig_yarn  = 4096
+llm_load_print_meta: rope_finetuned   = unknown
+llm_load_print_meta: ssm_d_conv       = 0
+llm_load_print_meta: ssm_d_inner      = 0
+llm_load_print_meta: ssm_d_state      = 0
+llm_load_print_meta: ssm_dt_rank      = 0
+llm_load_print_meta: model type       = 671B
+llm_load_print_meta: model ftype      = Q8_0
+llm_load_print_meta: model params     = 671.026 B
+llm_load_print_meta: model size       = 664.295 GiB (8.504 BPW)
+llm_load_print_meta: repeating layers = 662.461 GiB (8.504 BPW, 669.173 B parameters)
+llm_load_print_meta: general.name     = DeepSeek V3.2 Speciale Bf16 Safetensors
+llm_load_print_meta: n_layer_dense_lead   = 3
+llm_load_print_meta: n_lora_q             = 1536
+llm_load_print_meta: n_lora_kv            = 512
+llm_load_print_meta: n_ff_exp             = 2048
+llm_load_print_meta: n_expert_shared      = 1
+llm_load_print_meta: expert_weights_scale = 2.5
+llm_load_print_meta: expert_weights_norm  = 1
+llm_load_print_meta: expert_gating_func   = sigmoid
+llm_load_print_meta: rope_yarn_log_mul    = 0.1000
+print_info: vocab type       = BPE
+print_info: n_vocab          = 129280
+print_info: n_merges         = 127741
+print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
+print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
+print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
+print_info: PAD token        = 1 '<｜end▁of▁sentence｜>'
+print_info: LF token         = 201 'Ċ'
+print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
+print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
+print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
+print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
+print_info: max token length = 256
+llm_load_tensors: ggml ctx size =    0.45 MiB
+llm_load_tensors: offloading 0 repeating layers to GPU
+llm_load_tensors: offloaded 0/62 layers to GPU
+llm_load_tensors:        CPU buffer size = 680237.97 MiB
+....................................................................................................
+============ llm_prepare_mla: need to compute 61 wkv_b tensors
+================= Adjusted mainline llama.cpp MLA tensors to ik_llama.cpp
+Computed blk.0.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.1.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.2.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.3.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.4.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.5.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.6.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.7.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.8.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.9.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.10.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.11.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.12.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.13.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.14.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.15.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.16.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.17.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.18.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.19.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.20.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.21.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.22.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.23.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.24.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.25.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.26.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.27.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.28.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.29.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.30.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.31.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.32.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.33.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.34.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.35.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.36.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.37.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.38.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.39.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.40.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.41.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.42.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.43.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.44.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.45.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.46.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.47.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.48.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.49.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.50.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.51.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.52.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.53.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.54.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.55.allama_new_context_with_model: n_ctx         = 4096
+llama_new_context_with_model: n_batch       = 4096
+llama_new_context_with_model: n_ubatch      = 4096
+llama_new_context_with_model: flash_attn    = 1
+llama_new_context_with_model: mla_attn      = 3
+llama_new_context_with_model: attn_max_b    = 0
+llama_new_context_with_model: fused_moe     = 1
+llama_new_context_with_model: grouped er    = 0
+llama_new_context_with_model: fused_up_gate = 1
+llama_new_context_with_model: fused_mmad    = 1
+llama_new_context_with_model: rope_cache    = 0
+llama_new_context_with_model: graph_reuse   = 1
+llama_new_context_with_model: k_cache_hadam = 0
+llama_new_context_with_model: split_mode_graph_scheduling = 0
+llama_new_context_with_model: split_mode_f16= 1
+llama_new_context_with_model: sched_async   = 0
+llama_new_context_with_model: ser           = -1, 0
+llama_new_context_with_model: freq_base     = 10000.0
+llama_new_context_with_model: freq_scale    = 0.025
+llama_kv_cache_init:        CPU KV buffer size =   274.50 MiB
+llama_new_context_with_model: KV self size  =  274.50 MiB, c^KV (f16):  274.50 MiB, kv^T: not used
+llama_new_context_with_model:        CPU  output buffer size =     3.95 MiB
+llama_new_context_with_model:        CPU compute buffer size =  2132.00 MiB
+llama_new_context_with_model: graph nodes  = 3248
+llama_new_context_with_model: graph splits = 1
+XXXXXXXXXXXXXXXXXXXXX Setting only active experts offload
+system_info: n_threads = 96 (n_threads_batch = 128) / 512 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
+perplexity: tokenizing the input ..
+perplexity: tokenization took 450.855 ms
+perplexity: calculating perplexity over 561 chunks, n_ctx=512, batch_size=4096, n_seq=8
+perplexity: 25.76 seconds per pass - ETA 30.10 minutes
+ttn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.56.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.57.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.58.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.59.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+Computed blk.60.attn_kv_b.weight as 512 x 32768 and stored in buffer CPU
+======================================= HAVE_FANCY_SIMD is defined
+[1]2.3222,[2]3.1555,[3]2.3015,[4]1.9211,[5]1.7376,[6]1.6016,[7]1.5157,[8]1.4547,[9]1.4109,[10]1.3768,[11]1.3616,[12]1.3655,[13]1.3795,[14]1.4980,[15]1.6252,[16]1.6864,[17]1.8465,[18]1.9649,[19]1.9294,[20]1.9046,[21]2.0110,[22]1.9842,[23]1.9605,[24]1.9717,[25]1.9443,[26]1.9243,[27]1.9686,[28]1.9746,[29]2.0194,[30]2.0510,[31]2.0837,[32]2.0985,[33]2.1325,[34]2.1709,[35]2.2165,[36]2.2660,[37]2.3026,[38]2.3499,[39]2.3915,[40]2.4483,[41]2.4842,[42]2.4961,[43]2.5451,[44]2.5585,[45]2.6338,[46]2.6812,[47]2.6409,[48]2.5994,[49]2.5776,[50]2.5953,[51]2.6379,[52]2.6517,[53]2.7019,[54]2.7165,[55]2.7475,[56]2.7793,[57]2.7931,[58]2.8254,[59]2.8387,[60]2.8834,[61]2.9213,[62]2.9682,[63]2.9993,[64]3.0396,[65]3.0477,[66]3.0271,[67]3.0036,[68]3.0297,[69]3.0233,[70]3.0378,[71]3.0556,[72]3.0711,[73]3.0848,[74]3.1066,[75]3.0877,[76]3.0439,[77]3.0035,[78]2.9984,[79]2.9742,[80]2.9546,[81]2.9213,[82]2.9236,[83]2.8940,[84]2.8606,[85]2.8277,[86]2.8024,[87]2.7924,[88]2.7657,[89]2.7479,[90]2.7243,[91]2.6971,[92]2.6752,[93]2.6503,[94]2.6263,[95]2.6052,[96]2.6020,[97]2.6077,[98]2.5923,[99]2.5750,[100]2.5757,[101]2.5679,[102]2.5844,[103]2.6090,[104]2.6270,[105]2.6241,[106]2.6459,[107]2.6697,[108]2.6900,[109]2.7219,[110]2.7543,[111]2.7728,[112]2.7494,[113]2.7374,[114]2.7167,[115]2.7033,[116]2.6911,[117]2.6711,[118]2.6510,[119]2.6311,[120]2.6134,[121]2.5969,[122]2.5808,[123]2.5644,[124]2.5467,[125]2.5305,[126]2.5150,[127]2.5017,[128]2.4915,[129]2.4809,[130]2.4690,[131]2.4590,[132]2.4647,[133]2.4742,[134]2.4801,[135]2.4901,[136]2.5041,[137]2.5171,[138]2.5257,[139]2.5359,[140]2.5373,[141]2.5397,[142]2.5395,[143]2.5407,[144]2.5386,[145]2.5300,[146]2.5286,[147]2.5332,[148]2.5333,[149]2.5349,[150]2.5306,[151]2.5288,[152]2.5254,[153]2.5225,[154]2.5227,[155]2.5268,[156]2.5280,[157]2.5340,[158]2.5427,[159]2.5456,[160]2.5537,[161]2.5621,[162]2.5723,[163]2.5752,[164]2.5943,[165]2.6175,[166]2.6345,[167]2.6451,[168]2.6682,[169]2.6908,[170]2.7120,[171]2.7339,[172]2.7191,[173]2.7038,[174]2.6907,[175]2.6785,[176]2.6674,[177]2.6562,[178]2.6446,[179]2.6315,[180]2.6353,[181]2.6488,[182]2.6635,[183]2.6770,[184]2.6904,[185]2.6999,[186]2.7160,[187]2.7306,[188]2.7444,[189]2.7556,[190]2.7564,[191]2.7633,[192]2.7666,[193]2.7716,[194]2.7905,[195]2.7992,[196]2.8123,[197]2.8223,[198]2.8271,[199]2.8323,[200]2.8319,[201]2.8464,[202]2.8418,[203]2.8466,[204]2.8499,[205]2.8505,[206]2.8534,[207]2.8615,[208]2.8709,[209]2.8809,[210]2.8814,[211]2.8775,[212]2.8784,[213]2.8863,[214]2.8881,[215]2.8930,[216]2.8932,[217]2.8890,[218]2.8897,[219]2.8913,[220]2.8912,[221]2.8924,[222]2.8930,[223]2.8931,[224]2.8982,[225]2.8995,[226]2.8915,[227]2.8886,[228]2.8899,[229]2.8938,[230]2.8998,[231]2.9061,[232]2.8986,[233]2.8909,[234]2.8910,[235]2.8882,[236]2.8965,[237]2.9044,[238]2.9142,[239]2.9239,[240]2.9337,[241]2.9447,[242]2.9585,[243]2.9710,[244]2.9799,[245]2.9917,[246]3.0027,[247]3.0016,[248]2.9971,[249]2.9950,[250]2.9890,[251]2.9871,[252]2.9899,[253]2.9939,[254]3.0008,[255]3.0074,[256]3.0112,[257]3.0140,[258]3.0155,[259]3.0189,[260]3.0218,[261]3.0232,[262]3.0222,[263]3.0281,[264]3.0307,[265]3.0309,[266]3.0329,[267]3.0354,[268]3.0385,[269]3.0417,[270]3.0410,[271]3.0390,[272]3.0328,[273]3.0322,[274]3.0251,[275]3.0150,[276]3.0050,[277]3.0078,[278]3.0171,[279]3.0230,[280]3.0307,[281]3.0382,[282]3.0441,[283]3.0502,[284]3.0563,[285]3.0699,[286]3.0721,[287]3.0753,[288]3.0802,[289]3.0826,[290]3.0751,[291]3.0664,[292]3.0635,[293]3.0615,[294]3.0580,[295]3.0545,[296]3.0561,[297]3.0565,[298]3.0616,[299]3.0662,[300]3.0682,[301]3.0717,[302]3.0733,[303]3.0740,[304]3.0736,[305]3.0849,[306]3.0922,[307]3.1027,[308]3.0921,[309]3.0851,[310]3.0759,[311]3.0772,[312]3.0768,[313]3.0810,[314]3.0825,[315]3.0854,[316]3.0871,[317]3.0886,[318]3.0888,[319]3.0887,[320]3.0924,[321]3.0924,[322]3.0941,[323]3.0994,[324]3.1002,[325]3.1052,[326]3.1087,[327]3.1122,[328]3.1151,[329]3.1167,[330]3.1229,[331]3.1264,[332]3.1308,[333]3.1299,[334]3.1303,[335]3.1313,[336]3.1315,[337]3.1329,[338]3.1333,[339]3.1358,[340]3.1396,[341]3.1451,[342]3.1538,[343]3.1627,[344]3.1678,[345]3.1586,[346]3.1514,[347]3.1451,[348]3.1379,[349]3.1336,[350]3.1322,[351]3.1369,[352]3.1505,[353]3.1592,[354]3.1711,[355]3.1794,[356]3.1848,[357]3.1962,[358]3.2055,[359]3.2087,[360]3.2150,[361]3.2241,[362]3.2323,[363]3.2379,[364]3.2443,[365]3.2501,[366]3.2599,[367]3.2682,[368]3.2752,[369]3.2828,[370]3.2910,[371]3.3044,[372]3.3124,[373]3.3159,[374]3.3190,[375]3.3238,[376]3.3360,[377]3.3467,[378]3.3495,[379]3.3492,[380]3.3467,[381]3.3515,[382]3.3572,[383]3.3605,[384]3.3650,[385]3.3688,[386]3.3746,[387]3.3801,[388]3.3836,[389]3.3739,[390]3.3653,[391]3.3554,[392]3.3502,[393]3.3411,[394]3.3331,[395]3.3246,[396]3.3154,[397]3.3071,[398]3.2985,[399]3.2889,[400]3.2806,[401]3.2712,[402]3.2617,[403]3.2538,[404]3.2445,[405]3.2358,[406]3.2266,[407]3.2179,[408]3.2096,[409]3.2019,[410]3.1967,[411]3.1981,[412]3.1942,[413]3.1961,[414]3.1981,[415]3.1944,[416]3.1938,[417]3.1959,[418]3.1904,[419]3.1918,[420]3.1891,[421]3.1879,[422]3.1885,[423]3.1877,[424]3.1913,[425]3.1912,[426]3.1915,[427]3.1906,[428]3.1932,[429]3.1947,[430]3.1974,[431]3.1984,[432]3.1972,[433]3.1938,[434]3.1939,[435]3.1867,[436]3.1811,[437]3.1774,[438]3.1761,[439]3.1734,[440]3.1784,[441]3.1834,[442]3.1905,[443]3.1877,[444]3.1884,[445]3.1893,[446]3.1940,[447]3.1969,[448]3.1996,[449]3.2026,[450]3.2065,[451]3.2094,[452]3.2113,[453]3.2125,[454]3.2115,[455]3.2138,[456]3.2144,[457]3.2171,[458]3.2221,[459]3.2230,[460]3.2233,[461]3.2204,[462]3.2239,[463]3.2311,[464]3.2359,[465]3.2295,[466]3.2282,[467]3.2269,[468]3.2290,[469]3.2268,[470]3.2242,[471]3.2247,[472]3.2253,[473]3.2250,[474]3.2245,[475]3.2256,[476]3.2242,[477]3.2236,[478]3.2246,[479]3.2265,[480]3.2290,[481]3.2257,[482]3.2292,[483]3.2288,[484]3.2324,[485]3.2389,[486]3.2423,[487]3.2456,[488]3.2504,[489]3.2530,[490]3.2575,[491]3.2633,[492]3.2675,[493]3.2670,[494]3.2684,[495]3.2708,[496]3.2730,[497]3.2759,[498]3.2765,[499]3.2760,[500]3.2797,[501]3.2842,[502]3.2821,[503]3.2805,[504]3.2819,[505]3.2852,[506]3.2925,[507]3.2954,[508]3.2989,[509]3.2920,[510]3.2867,[511]3.2806,[512]3.2766,[513]3.2706,[514]3.2693,[515]3.2714,[516]3.2668,[517]3.2673,[518]3.2668,[519]3.2670,[520]3.2712,[521]3.2703,[522]3.2688,[523]3.2741,[524]3.2716,[525]3.2701,[526]3.2658,[527]3.2610,[528]3.2578,[529]3.2549,[530]3.2521,[531]3.2491,[532]3.2438,[533]3.2380,[534]3.2329,[535]3.2328,[536]3.2351,[537]3.2376,[538]3.2390,[539]3.2409,[540]3.2455,[541]3.2477,[542]3.2498,[543]3.2444,[544]3.2409,[545]3.2403,[546]3.2340,[547]3.2281,[548]3.2219,[549]3.2154,[550]3.2097,[551]3.2038,[552]3.1981,[553]3.1924,[554]3.1901,[555]3.1885,[556]3.1914,[557]3.1951,[558]3.2010,[559]3.2049,[560]3.2100,[561]3.2082,
+llama_print_timings:        load time =  232233.53 ms
+llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings: prompt eval time = 1638555.36 ms / 287232 tokens (    5.70 ms per token,   175.30 tokens per second)
+llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings:       total time = 1649429.62 ms / 287233 tokens
+Final estimate: PPL over 561 chunks for n_ctx=512 = 3.2082 +/- 0.01742