llm-semantic-router/mmbert32k-factcheck-classifier-lora

mmBERT-32K Fact-Check Classifier (LoRA)

Binary classifier for determining if a query requires fact-checking, based on mmBERT-32K-YaRN.

Model Description

This model classifies queries to determine if they need fact-checking (factual questions) vs. creative/opinion queries that don't require verification.

Classes

• NO_FACT_CHECK_NEEDED (0): Creative, opinion, or brainstorming queries
• FACT_CHECK_NEEDED (1): Factual queries that should be verified

Base Model

• Base: llm-semantic-router/mmbert-32k-yarn
• Architecture: ModernBERT with YaRN RoPE scaling
• Context Length: 32,768 tokens
• Languages: 1800+ (multilingual fact-check classification)

Training Details

• Method: LoRA fine-tuning
• LoRA Rank: 16
• LoRA Alpha: 32
• Epochs: 5
• Training samples: 6,800
• Validation samples: 1,700
• Training time: ~4.5 minutes (GPU)

Training Data Sources

Factual (FACT_CHECK_NEEDED):

• SQuAD (500 samples)
• TriviaQA (500 samples)
• TruthfulQA (500 samples)
• HotpotQA (500 samples)
• CoQA (2,000 samples)
• HaluEval (500 samples)
• RAG datasets (500 samples)

Non-Factual (NO_FACT_CHECK_NEEDED):

• Dolly (1,666 samples - creative/brainstorming)
• WritingPrompts (1,666 samples)
• Alpaca (1,668 samples - non-factual instructions)

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
import torch

base_model = "llm-semantic-router/mmbert-32k-yarn"
adapter = "llm-semantic-router/mmbert32k-factcheck-classifier-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForSequenceClassification.from_pretrained(base_model, num_labels=2)
model = PeftModel.from_pretrained(model, adapter)

# Example queries
queries = [
    "What is the capital of France?",  # -> FACT_CHECK_NEEDED
    "Write me a poem about the ocean",  # -> NO_FACT_CHECK_NEEDED
]

for query in queries:
    inputs = tokenizer(query, return_tensors="pt", truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        prediction = torch.argmax(outputs.logits, dim=-1).item()
    label = "FACT_CHECK_NEEDED" if prediction == 1 else "NO_FACT_CHECK_NEEDED"
    print(f"{query} -> {label}")

Intended Use

• RAG (Retrieval-Augmented Generation) pipeline routing
• Query classification for LLM systems
• Fact-verification system triggers
• Knowledge base query optimization

License

Apache 2.0