README.md

---
license: mit
language:
- en
library_name: transformers
base_model: DeependraVerma/slm-500m-base
pipeline_tag: text-generation
tags:
- legal
- finance
- small-language-model
- llama
- from-scratch
- instruction-tuned
- question-answering
- contract-analysis
- sft
model-index:
- name: legal-slm-500m-sft
  results:
  - task:
      type: text-generation
      name: Open-book contract clause extraction (CUAD)
    dataset:
      name: legal-slm-500M SFT validation split, CUAD-sourced subset
      type: DeependraVerma/legal-slm-500M-sft-validation
    metrics:
    - type: f1
      value: 0.798
      name: Token-F1 (grounded extraction)
  - task:
      type: text-generation
      name: Open-book contract clause classification (LEDGAR)
    dataset:
      name: legal-slm-500M SFT validation split, LEDGAR-sourced subset
      type: DeependraVerma/legal-slm-500M-sft-validation
    metrics:
    - type: accuracy
      value: 0.758
      name: Exact-match accuracy (96 non-colliding clause categories)
  - task:
      type: text-generation
      name: General commonsense reasoning (HellaSwag)
    dataset:
      name: HellaSwag
      type: Rowan/hellaswag
    metrics:
    - type: acc_norm
      value: 0.3454
      name: acc_norm, zero-shot, via lm-evaluation-harness
  - task:
      type: text-generation
      name: General science QA (ARC-Easy)
    dataset:
      name: ARC-Easy
      type: allenai/ai2_arc
    metrics:
    - type: acc_norm
      value: 0.4205
      name: acc_norm, zero-shot, via lm-evaluation-harness
  - task:
      type: text-generation
      name: Physical commonsense reasoning (PIQA)
    dataset:
      name: PIQA
      type: ybisk/piqa
    metrics:
    - type: acc_norm
      value: 0.6235
      name: acc_norm, zero-shot, via lm-evaluation-harness
  - task:
      type: text-generation
      name: Legal knowledge (MMLU professional_law)
    dataset:
      name: MMLU — professional_law
      type: cais/mmlu
    metrics:
    - type: acc
      value: 0.2445
      name: acc, zero-shot — near the 25% random-chance floor for 4-choice, expected at this scale
  - task:
      type: text-generation
      name: Legal knowledge (MMLU jurisprudence)
    dataset:
      name: MMLU — jurisprudence
      type: cais/mmlu
    metrics:
    - type: acc
      value: 0.2778
      name: acc, zero-shot
  - task:
      type: text-generation
      name: Legal knowledge (MMLU international_law)
    dataset:
      name: MMLU — international_law
      type: cais/mmlu
    metrics:
    - type: acc
      value: 0.2479
      name: acc, zero-shot
  - task:
      type: text-generation
      name: Legal holding selection (CaseHOLD)
    dataset:
      name: CaseHOLD (via LexGLUE)
      type: coastalcph/lex_glue
    metrics:
    - type: acc_norm
      value: 0.2000
      name: acc_norm, zero-shot — at the 20% random-chance floor for 5-choice; already excluded from pretraining data
  - task:
      type: text-generation
      name: LegalBench — contract NLI + QA + CFR-grounded subset (13 tasks, contamination-checked)
    dataset:
      name: LegalBench
      type: nguha/legalbench
    metrics:
    - type: accuracy
      value: 0.5605
      name: mean acc across 13 tasks (50% chance floor) — see repo for per-task breakdown
---

# legal-slm-500m-sft

A **528.5M-parameter instruction-tuned legal & financial assistant**, fully
fine-tuned from [`DeependraVerma/slm-500m-base`](https://huggingface.co/DeependraVerma/slm-500m-base)
— itself trained completely from scratch on a staged-schedule mix of seven
sources: US case law, SEC filings (10-K/10-Q/8-K/S-1/20-F), the Code of
Federal Regulations, the Federal Register, and educational web text. This is
the scaled-up successor to
[`DeependraVerma/legal-slm-125m-sft`](https://huggingface.co/DeependraVerma/legal-slm-125m-sft).

- **Repo / full build:** [github.com/DeependraVerma/legal-slm-125M](https://github.com/DeependraVerma/legal-slm-125M)
- **Author:** [Deependra Verma](https://github.com/DeependraVerma) — Generative AI Researcher / AI Engineer ([Hugging Face](https://huggingface.co/DeependraVerma))
- **Base model:** [`DeependraVerma/slm-500m-base`](https://huggingface.co/DeependraVerma/slm-500m-base)
- **Predecessor (125M):** [`DeependraVerma/legal-slm-125m-sft`](https://huggingface.co/DeependraVerma/legal-slm-125m-sft)

> **Read "Known limitations" below before using this model for anything.**
> This revision fixes a real, tested usability bug (see below) and improves
> legal-benchmark performance, but it still hallucinates confidently on
> closed-book questions outside its training distribution. Never use its
> output as legal, financial, or factual advice.

## What's new in this revision

The previous revision of this model would respond to plain conversational
input ("Hi", "Thanks!", "How are you?") with unrelated, hallucinated
legal-sounding text — confirmed directly, not hypothetically: it had zero
conversational examples in its fine-tuning data, every training pair was a
legal/financial task. This revision adds two small, deliberately-scoped
slices to the SFT data (about 2.6% of the total):

- **~400 conversational pairs** (greetings, small talk, "what can you do"),
  sourced from `HuggingFaceTB/everyday-conversations-llama3.1-2k` plus a
  small hand-written set targeting the exact failure observed, so the model
  responds naturally to plain conversational input instead of hallucinating.
- **~170 refusal pairs**, synthesized by the same local Llama-3.1-70B teacher
  used for the rest of this dataset, covering five categories a small model
  genuinely cannot answer (future predictions, private individual data,
  invented/unverifiable legal specifics, real-time data, advice with no
  facts given) — teaching a calibrated decline instead of a confident,
  invented answer.

**Result, tested directly:**
- Conversational input now gets a sensible, on-topic response instead of
  hallucinated legal text (confirmed by direct generation, not just a
  benchmark proxy).
- LegalBench (13-task average) improved from 52.5% → **56.1%** — now ahead
  of same-size-class Gemma-3-270M and within 0.7 points of OLMo-2-1B despite
  a quarter its parameters.
- A rigorous, leak-free paired comparison (scoring both this revision and the
  previous one on the same held-out questions that neither was trained on)
  found case-law closed-book QA and LEDGAR classification **statistically
  unchanged** — an earlier, naive before/after comparison had suggested a
  regression there, but that comparison turned out to be measuring two
  different, non-overlapping question samples, not a real capability change.
- The same clean comparison found a **real, modest drop in CUAD extraction
  precision** (Token-F1 0.886 → 0.818 on the matched, held-out subset) —
  reported here plainly rather than hidden, consistent with this project's
  practice throughout. If precise contract-clause extraction is your primary
  use case, weigh this against the conversational and LegalBench gains.

## What this is

Same two-mode design as the 125M SFT model:

1. **Open-book (recommended)** — give it a contract excerpt or passage along
   with your question; it extracts/classifies/answers from that given text.
   This is the mode with real, measured reliability.
2. **Closed-book** — a general legal/financial question with no source text.
   Weaker and riskier — see Known limitations. Now also handles plain
   conversational input and declines gracefully on genuinely unanswerable
   questions (see "What's new" above).

## Model description

| | |
|---|---|
| Parameters | 528,538,752 (~528.5M, tied embeddings) |
| Architecture | Llama-style decoder, 24 layers / 1152 hidden / 18 heads (head dim 64, full MHA) |
| MLP | SwiGLU, intermediate size 4,608 |
| Vocabulary | 16,384 byte-level BPE (same tokenizer as the 125M build, plus role tokens) |
| Context length | 1,024 tokens |
| Base checkpoint | [`DeependraVerma/slm-500m-base`](https://huggingface.co/DeependraVerma/slm-500m-base) (this author's own from-scratch pretrain) |

## Training data

Builds on the curated SFT dataset shared with the 125M model — CUAD contract
clause extraction (CC BY 4.0), LEDGAR clause classification (CC BY 4.0), and
open-book QA/summarization/extraction over case-law/SEC/educational-web
passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher — plus
the ~570 conversational/refusal pairs described above.

| | |
|---|---|
| Train / val split | 21,064 / 1,108 |
| Total pairs | 22,172 (21,602 original + 398 conversational + 172 refusal) |
| Method | full fine-tune (not LoRA), 2 epochs |

## Evaluation

Full held-out validation set (1,108 examples), same methodology as previous
revisions — extraction/classification against ground truth, general QA
judged by an independent local Meta-Llama-3.1-70B-Instruct:

| Task | Previous revision | **This revision** |
|---|---|---|
| CUAD contract clause extraction (Token-F1) | 0.786 | 0.798 (aggregate) / **0.818** (leak-free matched subset, n=25) |
| CUAD "clause not present" refusal accuracy | 84.7% | 84.2% |
| LEDGAR clause classification (exact-match) | 79.1% | 75.8% (aggregate) / **80.0%** (leak-free matched subset, n=10 — unchanged) |
| Case law general Q&A (closed-book) | 50.0% | 43.5% (aggregate) / **28.6%** (leak-free matched subset, n=14 — unchanged vs. previous on same subset) |
| SEC filings general Q&A (closed-book) | 56.2% | 54.9% |
| Refusal on genuinely unanswerable questions | n/a (not tested previously) | **90.9%** |

**A methodology note, stated plainly:** the "aggregate" numbers above compare
different, largely non-overlapping validation samples (re-curating the
dataset to add the new pairs reshuffled which examples land in val), which
makes some of the aggregate deltas look larger than the real effect. Where
possible, the matched columns above instead compare both revisions on the
*exact same* held-out questions that neither was trained on — that is the
number to trust for case-law and LEDGAR specifically. CUAD's matched
subsample does show a real, if modest, decline in extraction precision.

## Known limitations — read before using

- **Open-book contract tasks work well and are the trustworthy mode.**
- **Plain conversational input is now handled sensibly** (tested directly),
  but this model was never trained for open-ended general-knowledge chat —
  ask it something outside law/finance (e.g. general trivia) and it will
  still confidently hallucinate, because its entire pretraining corpus is
  legal/financial/regulatory text. This is a pretraining-scope limitation
  that fine-tuning data cannot fix.
- **Closed-book general legal/financial Q&A is still fundamentally limited.**
  A 528.5M-parameter model can store at most ~2 bits of knowledge per
  parameter (Allen-Zhu & Li, ["Physics of Language Models: Knowledge Capacity
  Scaling Laws"](https://arxiv.org/abs/2404.05405)) — roughly 132MB of total
  compressible fact storage, shared across everything it knows. Combined with
  the fact that most specific facts in its training data appeared only once
  or twice, closed-book precision on rare facts is a hard capacity
  limitation, not something more training on this same recipe fixes.
- **Confident fabrication on closed-book questions is real and was directly
  observed**, not hypothetical.
- **A modest CUAD extraction-precision regression was measured** in this
  revision (see Evaluation) — worth weighing if precise contract-clause
  extraction is your primary use case.

**Never use this model's output as legal, financial, or factual advice.**
Always treat specific claims as unverified until checked against a primary
source, especially in closed-book use.

## External benchmark evaluation

Run through **`lm-evaluation-harness`** (the same framework behind a widely
used public LLM leaderboard) on five benchmark categories:

| Benchmark | Result | Baseline | Read |
|---|---|---|---|
| HellaSwag / ARC-Easy / PIQA (general commonsense) | 34.5% / 42.1% / 62.4% (acc_norm) | n/a | Normal range for a model this size — OLMo-2-1B (~2x params, general-purpose corpus) leads here as expected |
| MMLU professional_law / jurisprudence / international_law | 24.5% / 27.8% / 24.8% (acc) | 25% (4-choice) | Near the random-chance floor — no real legal knowledge memorized, as the caveats above already say |
| CaseHOLD (pick the correct legal holding, via LexGLUE) | 20.0% (acc_norm) | 20% (5-choice) | Right at random chance — real closed-book legal reasoning is not something this model can do |
| LegalBench — 13-task contract NLI/QA/CFR subset | **56.1% mean acc** | 50% (binary) | Genuine positive signal, improved from the previous revision's 52.5% — now ahead of Gemma-3-270M (53.4%) and within 0.7 points of OLMo-2-1B (56.7%) despite a quarter the parameters |

LegalBench isn't in `lm-eval-harness`'s default task set, so a 13-task,
contract-focused-plus-one-CFR-grounded subset was hand-configured and
checked for training-data contamination before trusting the result (see the
[repo](https://github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks)
for the full contamination-scan methodology).

**No comparably small domain-specific baseline exists.** We looked for a
small (<1B parameter), from-scratch, publicly-weighted *generative* legal- or
financial-domain model to compare against and found none for either domain —
this absence is itself a relevant data point about the current state of
small, open, domain-specialized language models.

Full methodology, task configs, and raw logs:
[github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks](https://github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks).

## How to use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-500m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-500m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))
```

## Citation

```bibtex
@misc{verma2026legalslm500msft,
  author = {Deependra Verma},
  title  = {legal-slm-500m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 528.5M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-500m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}
```

## Author

**Deependra Verma** — Generative AI Researcher / AI Engineer.
[GitHub](https://github.com/DeependraVerma) · [Hugging Face](https://huggingface.co/DeependraVerma)