legal-slm-500m-sft

A 528.5M-parameter instruction-tuned legal & financial assistant, fully fine-tuned from DeependraVerma/slm-500m-base — itself trained completely from scratch on a staged-schedule mix of seven sources: US case law, SEC filings (10-K/10-Q/8-K/S-1/20-F), the Code of Federal Regulations, the Federal Register, and educational web text. This is the scaled-up successor to DeependraVerma/legal-slm-125m-sft.

• Repo / full build: github.com/DeependraVerma/legal-slm-125M
• Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
• Base model: DeependraVerma/slm-500m-base
• Predecessor (125M): DeependraVerma/legal-slm-125m-sft

Read "Known limitations" below before using this model for anything. This revision fixes a real, tested usability bug (see below) and improves legal-benchmark performance, but it still hallucinates confidently on closed-book questions outside its training distribution. Never use its output as legal, financial, or factual advice.

What's new in this revision

The previous revision of this model would respond to plain conversational input ("Hi", "Thanks!", "How are you?") with unrelated, hallucinated legal-sounding text — confirmed directly, not hypothetically: it had zero conversational examples in its fine-tuning data, every training pair was a legal/financial task. This revision adds two small, deliberately-scoped slices to the SFT data (about 2.6% of the total):

• ~400 conversational pairs (greetings, small talk, "what can you do"), sourced from HuggingFaceTB/everyday-conversations-llama3.1-2k plus a small hand-written set targeting the exact failure observed, so the model responds naturally to plain conversational input instead of hallucinating.
• ~170 refusal pairs, synthesized by the same local Llama-3.1-70B teacher used for the rest of this dataset, covering five categories a small model genuinely cannot answer (future predictions, private individual data, invented/unverifiable legal specifics, real-time data, advice with no facts given) — teaching a calibrated decline instead of a confident, invented answer.

Result, tested directly:

• Conversational input now gets a sensible, on-topic response instead of hallucinated legal text (confirmed by direct generation, not just a benchmark proxy).
• LegalBench (13-task average) improved from 52.5% → 56.1% — now ahead of same-size-class Gemma-3-270M and within 0.7 points of OLMo-2-1B despite a quarter its parameters.
• A rigorous, leak-free paired comparison (scoring both this revision and the previous one on the same held-out questions that neither was trained on) found case-law closed-book QA and LEDGAR classification statistically unchanged — an earlier, naive before/after comparison had suggested a regression there, but that comparison turned out to be measuring two different, non-overlapping question samples, not a real capability change.
• The same clean comparison found a real, modest drop in CUAD extraction precision (Token-F1 0.886 → 0.818 on the matched, held-out subset) — reported here plainly rather than hidden, consistent with this project's practice throughout. If precise contract-clause extraction is your primary use case, weigh this against the conversational and LegalBench gains.

What this is

Same two-mode design as the 125M SFT model:

1. Open-book (recommended) — give it a contract excerpt or passage along with your question; it extracts/classifies/answers from that given text. This is the mode with real, measured reliability.
2. Closed-book — a general legal/financial question with no source text. Weaker and riskier — see Known limitations. Now also handles plain conversational input and declines gracefully on genuinely unanswerable questions (see "What's new" above).

Model description

Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder, 24 layers / 1152 hidden / 18 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 4,608
Vocabulary	16,384 byte-level BPE (same tokenizer as the 125M build, plus role tokens)
Context length	1,024 tokens
Base checkpoint	DeependraVerma/slm-500m-base (this author's own from-scratch pretrain)

Training data

Builds on the curated SFT dataset shared with the 125M model — CUAD contract clause extraction (CC BY 4.0), LEDGAR clause classification (CC BY 4.0), and open-book QA/summarization/extraction over case-law/SEC/educational-web passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher — plus the ~570 conversational/refusal pairs described above.

Train / val split	21,064 / 1,108
Total pairs	22,172 (21,602 original + 398 conversational + 172 refusal)
Method	full fine-tune (not LoRA), 2 epochs

Evaluation

Full held-out validation set (1,108 examples), same methodology as previous revisions — extraction/classification against ground truth, general QA judged by an independent local Meta-Llama-3.1-70B-Instruct:

Task	Previous revision	This revision
CUAD contract clause extraction (Token-F1)	0.786	0.798 (aggregate) / 0.818 (leak-free matched subset, n=25)
CUAD "clause not present" refusal accuracy	84.7%	84.2%
LEDGAR clause classification (exact-match)	79.1%	75.8% (aggregate) / 80.0% (leak-free matched subset, n=10 — unchanged)
Case law general Q&A (closed-book)	50.0%	43.5% (aggregate) / 28.6% (leak-free matched subset, n=14 — unchanged vs. previous on same subset)
SEC filings general Q&A (closed-book)	56.2%	54.9%
Refusal on genuinely unanswerable questions	n/a (not tested previously)	90.9%

A methodology note, stated plainly: the "aggregate" numbers above compare different, largely non-overlapping validation samples (re-curating the dataset to add the new pairs reshuffled which examples land in val), which makes some of the aggregate deltas look larger than the real effect. Where possible, the matched columns above instead compare both revisions on the exact same held-out questions that neither was trained on — that is the number to trust for case-law and LEDGAR specifically. CUAD's matched subsample does show a real, if modest, decline in extraction precision.

Known limitations — read before using

• Open-book contract tasks work well and are the trustworthy mode.
• Plain conversational input is now handled sensibly (tested directly), but this model was never trained for open-ended general-knowledge chat — ask it something outside law/finance (e.g. general trivia) and it will still confidently hallucinate, because its entire pretraining corpus is legal/financial/regulatory text. This is a pretraining-scope limitation that fine-tuning data cannot fix.
• Closed-book general legal/financial Q&A is still fundamentally limited. A 528.5M-parameter model can store at most ~2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — roughly 132MB of total compressible fact storage, shared across everything it knows. Combined with the fact that most specific facts in its training data appeared only once or twice, closed-book precision on rare facts is a hard capacity limitation, not something more training on this same recipe fixes.
• Confident fabrication on closed-book questions is real and was directly observed, not hypothetical.
• A modest CUAD extraction-precision regression was measured in this revision (see Evaluation) — worth weighing if precise contract-clause extraction is your primary use case.

Never use this model's output as legal, financial, or factual advice. Always treat specific claims as unverified until checked against a primary source, especially in closed-book use.

External benchmark evaluation

Run through lm-evaluation-harness (the same framework behind a widely used public LLM leaderboard) on five benchmark categories:

Benchmark	Result	Baseline	Read
HellaSwag / ARC-Easy / PIQA (general commonsense)	34.5% / 42.1% / 62.4% (acc_norm)	n/a	Normal range for a model this size — OLMo-2-1B (~2x params, general-purpose corpus) leads here as expected
MMLU professional_law / jurisprudence / international_law	24.5% / 27.8% / 24.8% (acc)	25% (4-choice)	Near the random-chance floor — no real legal knowledge memorized, as the caveats above already say
CaseHOLD (pick the correct legal holding, via LexGLUE)	20.0% (acc_norm)	20% (5-choice)	Right at random chance — real closed-book legal reasoning is not something this model can do
LegalBench — 13-task contract NLI/QA/CFR subset	56.1% mean acc	50% (binary)	Genuine positive signal, improved from the previous revision's 52.5% — now ahead of Gemma-3-270M (53.4%) and within 0.7 points of OLMo-2-1B (56.7%) despite a quarter the parameters

LegalBench isn't in lm-eval-harness's default task set, so a 13-task, contract-focused-plus-one-CFR-grounded subset was hand-configured and checked for training-data contamination before trusting the result (see the repo for the full contamination-scan methodology).

No comparably small domain-specific baseline exists. We looked for a small (<1B parameter), from-scratch, publicly-weighted generative legal- or financial-domain model to compare against and found none for either domain — this absence is itself a relevant data point about the current state of small, open, domain-specialized language models.

Full methodology, task configs, and raw logs: github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-500m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-500m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

Citation

@misc{verma2026legalslm500msft,
  author = {Deependra Verma},
  title  = {legal-slm-500m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 528.5M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-500m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face