bekko-embedding-v1-a25m-GGUF

GGUF release of hotchpotch/bekko-embedding-v1-a25m, an efficient multilingual text embedding model with 24.9M active parameters.

llama.cpp

The GGUF contains a compatible Gemma2 SentencePiece vocabulary and runs on the standard llama.cpp runtime with ModernBERT embedding support:

# GPU or Apple Silicon: use BF16
llama-server \
  -hf hotchpotch/bekko-embedding-v1-a25m-GGUF:BF16 \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bekko","input":["What is the tallest mountain in Japan?","Mount Fuji is the tallest mountain in Japan."]}'

For CPU inference, replace :BF16 with :Q8_0. If the selector is unavailable in your llama.cpp version, download the desired GGUF file and pass it with --model:

hf download \
  hotchpotch/bekko-embedding-v1-a25m-GGUF \
  bekko-embedding-v1-a25m-Q8_0.gguf \
  --local-dir .

llama-server \
  --model ./bekko-embedding-v1-a25m-Q8_0.gguf \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

Ollama

The default Ollama tag is BF16 and is recommended for GPU and Apple Silicon users:

ollama pull hotchpotch/bekko-embedding-v1-a25m
curl http://localhost:11434/api/embed \
  -d '{"model":"hotchpotch/bekko-embedding-v1-a25m","input":"What is the tallest mountain in Japan?"}'

For CPU inference, select the Q8_0 tag explicitly:

ollama pull hotchpotch/bekko-embedding-v1-a25m:q8_0

The available tags are latest (BF16), bf16, f16, and q8_0. The registry artifact has been verified as an embedding model with 384 dimensional, L2-normalized output.

Model contract

• Architecture: ModernBERT encoder
• Context length: 8192 tokens
• Embedding dimension: 384
• Matryoshka dimensions: 256, 128, 64
• Pooling: mean
• Similarity: cosine or dot product after L2 normalization
• Query/document prefixes: none
• Global and local RoPE theta: 160000
• Languages: 100+

Use the same unprefixed encoding function for queries and documents.

Quantization quality

Q8_0 was compared with the revision-pinned SentenceTransformers source using the same unprefixed queries and documents:

Benchmark	HF source	Q8_0	Delta
NanoMIRACL-ja NDCG@10	0.734496	0.735662	+0.001165
NanoMIRACL-en NDCG@10	0.692223	0.692223	0.000000
NanoCoIR representative mean NDCG@10	0.912505	0.912170	-0.000335

The mean cosine similarity to the HF embeddings is 0.999717 over five ordinary multilingual samples. These retrieval differences are small enough to be treated as ranking noise.

Faster CPU inference

GGUF is intended for portable llama.cpp and Ollama deployment. On a fast x86 CPU, the optimized OpenVINO export is generally faster than llama.cpp GGUF and should be preferred when maximum throughput is the priority. It is available in the hotchpotch/bekko-embedding-v1-a25m source repository when maximum throughput is the priority. The source model also provides the original SentenceTransformers model and its full model card.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a25m",
    backend="openvino",
)
embeddings = model.encode(["What is the tallest mountain in Japan?"], normalize_embeddings=True)

Choosing a GGUF format

The repository publishes BF16, F16, and Q8_0. Use BF16 on GPUs and Apple Silicon; use Q8_0 on CPUs, especially CPUs without native BF16 arithmetic. F16 is provided for runtimes or hardware that prefer FP16 explicitly.

Q8_0 remains the best quantized portable trade-off found in our BF16/Q8_0/Q6_K/Q5_K_M/Q4_K_M/IQ4_XS comparison:

• Lower-bit K-quants provide little size reduction because the 384-wide tensors fall back to Q5_0 or Q8_0. Q4_K_M is only 5.4% smaller than Q8_0.
• Lower-bit embeddings have larger cosine drift. IQ4_XS loses 0.0113 mean NDCG@10 on the representative NanoCoIR tasks.
• Q8_0 is the fastest quality-preserving quantization on Raspberry Pi 5. Q4_K_M and Q5_K_M reach only 36% and 33% of Q8_0 throughput respectively.
• Q6_K has the same size, embeddings, benchmark scores, and effectively the same fallback tensor types as Q8_0, so publishing both would be redundant.

Requested quant	Size MiB	Cosine vs HF	NanoCoIR delta	Pi 5 speed vs Q8_0
Q8_0	130.65	0.999717	-0.000335	1.000x
Q6_K	130.65	0.999717	-0.000335	0.996x
Q5_K_M	124.81	0.996479	-0.002825	0.327x
Q4_K_M	123.64	0.995837	-0.000654	0.356x
IQ4_XS	118.85	0.981694	-0.011304	0.889x

Lower-bit quantization reduces numerical fidelity and, in the representative retrieval evaluation, does not provide a useful size/quality/speed trade-off.

Files and reproducibility

File	Purpose
bekko-embedding-v1-a25m-BF16.gguf	Recommended for GPU and Apple Silicon
bekko-embedding-v1-a25m-F16.gguf	FP16 compatibility option
bekko-embedding-v1-a25m-Q8_0.gguf	Recommended for CPU inference
conversion.json	Source revision, llama.cpp revision, checksums, and config checks

This repository is generated from an immutable source revision. See conversion.json for the exact revision, llama.cpp revision, SHA-256 hash, and release-critical config validation. The build refuses a source config unless both global and local RoPE theta are 160000.

For benchmark results, training details, license, and limitations, refer to the source model card.