bge-m3-medical-cn

Fine-tuned BGE-M3 on Chinese medical question-answer retrieval using hard negative mining and triple-path InfoNCE loss (dense + sparse + ColBERT).

Model Performance

Evaluated on cMedQA2 val set — 1,585 queries against a 196K answer corpus.

Model	Hit@10	Hit@1000	MRR@10	Acc@1024
coROM-medical-base (DAMO)	0.2055	0.5815	0.1437	0.3759
BGE-M3 (no fine-tuning)	0.2177	0.6290	0.1414	0.4158
BGE-M3 + in-batch negatives	0.2479	0.7148	0.1521	0.4662
bge-m3-medical-cn (ours)	0.3419	0.8839	0.2225	0.6145

Acc@1024: ranking accuracy with 1 positive + 1023 random negatives.

Usage

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("mingchen/bge-m3-medical-cn")
model = AutoModel.from_pretrained("mingchen/bge-m3-medical-cn").eval()

def encode(texts, max_len=256):
    enc = tokenizer(texts, padding=True, truncation=True,
                    max_length=max_len, return_tensors="pt")
    with torch.no_grad():
        out = model(**enc)
    return F.normalize(out.last_hidden_state[:, 0], p=2, dim=-1)

queries = ["头痛发烧怎么办"]
answers = ["建议多喝水，服用退烧药，若持续高烧请及时就医。"]

q_emb = encode(queries, max_len=128)
a_emb = encode(answers, max_len=256)
print((q_emb @ a_emb.T))

Dataset Construction

Training data is derived from cMedQA2 (226K real patient-doctor dialogues).

Step 1 — Deduplication

Raw data contains ~47% exact duplicates and many near-duplicates. Contrastive learning requires semantic diversity, so we run a 3-stage deduplication pipeline:

Stage	Method	Result
1	MD5 exact match	226K → 119K
2	MinHash LSH (Jaccard ≥ 0.80)	119K → 118K
3	BGE-M3 semantic (cosine ≥ 0.80)	118K → 79K

The semantic threshold 0.80 was determined empirically: an LLM judge (Baichuan-M3) labeled 300 sampled pairs across 5 similarity intervals and found that the duplicate rate drops sharply from 54% (sim ∈ [0.80, 0.85)) to 22% (sim ∈ [0.75, 0.80)) — a clear cliff at 0.80.

Step 2 — Hard Negative Mining

For each query, we use FAISS to retrieve the top-500 nearest neighbors and keep those with question similarity ∈ [0.50, 0.80) as hard negative candidates:

• Upper bound 0.80: deduplication already removes near-duplicates above this threshold
• Lower bound 0.50: below this, BGE-M3 already distinguishes easily — too easy to provide useful gradient signal
• LLM verification confirmed 100% of pairs in [0.50, 0.80) have different patient intents

Each training sample stores 99 hard negatives; training randomly samples 9 per step for diversity.

This strategy is the key contribution: switching from random in-batch negatives to hard negatives yields +14.8pp on Acc@1024 (0.4662 → 0.6145).

Training

• Loss: Equal-weight triple InfoNCE — L = L_dense + L_sparse + L_colbert
• Answer pool per step: 1280 (128 positives + 128 × 9 hard negatives)
• GradCache: enables large answer pools on a single GPU
• lr: 1e-5, warmup 1%, cosine decay
• Stopped: early stopping at epoch 6

License

MIT

Citation

@misc{bge-m3-medical-cn,
  author = {mingchen},
  title  = {bge-m3-medical-cn: BGE-M3 Fine-tuned for Chinese Medical Retrieval},
  year   = {2026},
  url    = {https://huggingface.co/mingchen/bge-m3-medical-cn}
}