sec-embedding

This is a fine-tuned version of BAAI/bge-m3 for CVE / cybersecurity dense retrieval.

It was trained on a CVE investigation-trajectory dataset with hard-negative mining from a local Qdrant collection (cve_kb, NVD/MITRE core chunks). It is not a raw copy of the base checkpoint.

Training

From notebooks/BGE_M3_Colab.ipynb:

Base	BAAI/bge-m3 via Unsloth FastSentenceTransformer (unsloth/bge-m3)
Role	Bi-encoder / dense retriever (1024-d, same geometry as bge-m3)
Adapter	LoRA, r=32, modules key, query, value, dense
Loss	CachedMultipleNegativesRankingLoss (InfoNCE, in-batch hard negatives)
Engine	sentence-transformers SentenceTransformerTrainer
Max sequence length	1024
Learning rate	2e-5, bf16

Each example is a (query, positive) pair:

• Query — CVE investigation trajectory (Vietnamese or English) over CVE-ID, CWE, product, severity, year, CAPEC / ATT&CK, filled from real KB metadata.
• Positive — matching CVE passage from local Qdrant cve_kb.
• Hard negatives — other CVE documents in the same mini-batch, all mined from that Qdrant index (near-miss CVEs: similar wording, wrong ID).

Dataset source field: Qdrant cve_kb (NVD/MITRE). Split: 40k train / 5k validation.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("DuyTa/sec-embedding")
query_emb = model.encode("CVE-2021-44228 impact on log4j", normalize_embeddings=True)
doc_emb = model.encode(passage, normalize_embeddings=True)

Rebuild the Qdrant index with this checkpoint. Mixing vectors with vanilla BAAI/bge-m3 drops recall.

Attribution & license

Derived from BAAI/bge-m3 (MIT License). Credit for the base model belongs to the BAAI authors.