Upload folder using huggingface_hub

README.md

@@ -26,7 +26,7 @@ model-index:
       type: DeependraVerma/legal-slm-500M-sft-validation
     metrics:
     - type: f1
-      value: 0.786
+      value: 0.798
       name: Token-F1 (grounded extraction)
   - task:
       type: text-generation
@@ -36,7 +36,7 @@ model-index:
       type: DeependraVerma/legal-slm-500M-sft-validation
     metrics:
     - type: accuracy
-      value: 0.791
+      value: 0.758
       name: Exact-match accuracy (96 non-colliding clause categories)
   - task:
       type: text-generation
@@ -46,7 +46,7 @@ model-index:
       type: Rowan/hellaswag
     metrics:
     - type: acc_norm
-      value: 0.3401
+      value: 0.3454
       name: acc_norm, zero-shot, via lm-evaluation-harness
   - task:
       type: text-generation
@@ -56,7 +56,7 @@ model-index:
       type: allenai/ai2_arc
     metrics:
     - type: acc_norm
-      value: 0.4566
+      value: 0.4205
       name: acc_norm, zero-shot, via lm-evaluation-harness
   - task:
       type: text-generation
@@ -66,7 +66,7 @@ model-index:
       type: ybisk/piqa
     metrics:
     - type: acc_norm
-      value: 0.6050
+      value: 0.6235
       name: acc_norm, zero-shot, via lm-evaluation-harness
   - task:
       type: text-generation
@@ -76,7 +76,7 @@ model-index:
       type: cais/mmlu
     metrics:
     - type: acc
-      value: 0.2484
+      value: 0.2445
       name: acc, zero-shot — near the 25% random-chance floor for 4-choice, expected at this scale
   - task:
       type: text-generation
@@ -86,7 +86,7 @@ model-index:
       type: cais/mmlu
     metrics:
     - type: acc
-      value: 0.2593
+      value: 0.2778
       name: acc, zero-shot
   - task:
       type: text-generation
@@ -96,7 +96,7 @@ model-index:
       type: cais/mmlu
     metrics:
     - type: acc
-      value: 0.2397
+      value: 0.2479
       name: acc, zero-shot
   - task:
       type: text-generation
@@ -106,7 +106,7 @@ model-index:
       type: coastalcph/lex_glue
     metrics:
     - type: acc_norm
-      value: 0.1961
+      value: 0.2000
       name: acc_norm, zero-shot — at the 20% random-chance floor for 5-choice; already excluded from pretraining data
   - task:
       type: text-generation
@@ -116,7 +116,7 @@ model-index:
       type: nguha/legalbench
     metrics:
     - type: accuracy
-      value: 0.5247
+      value: 0.5605
       name: mean acc across 13 tasks (50% chance floor) — see repo for per-task breakdown
 ---
 
@@ -128,21 +128,58 @@ fine-tuned from [`DeependraVerma/slm-500m-base`](https://huggingface.co/Deependr
 sources: US case law, SEC filings (10-K/10-Q/8-K/S-1/20-F), the Code of
 Federal Regulations, the Federal Register, and educational web text. This is
 the scaled-up successor to
-[`DeependraVerma/legal-slm-125m-sft`](https://huggingface.co/DeependraVerma/legal-slm-125m-sft),
-fine-tuned on the **exact same curated SFT dataset** so the two models are
-directly comparable question-for-question.
+[`DeependraVerma/legal-slm-125m-sft`](https://huggingface.co/DeependraVerma/legal-slm-125m-sft).
 
 - **Repo / full build:** [github.com/DeependraVerma/legal-slm-125M](https://github.com/DeependraVerma/legal-slm-125M)
 - **Author:** [Deependra Verma](https://github.com/DeependraVerma) — Generative AI Researcher / AI Engineer ([Hugging Face](https://huggingface.co/DeependraVerma))
 - **Base model:** [`DeependraVerma/slm-500m-base`](https://huggingface.co/DeependraVerma/slm-500m-base)
 - **Predecessor (125M):** [`DeependraVerma/legal-slm-125m-sft`](https://huggingface.co/DeependraVerma/legal-slm-125m-sft)
 
-> **Read "Known limitations" below before using this model for anything.** It
-> is measurably better than the 125M model on every shared category, but it
-> still hallucinates confidently on closed-book questions (no source text
-> given) — a real, tested limitation, not a hypothetical one. Never use its
+> **Read "Known limitations" below before using this model for anything.**
+> This revision fixes a real, tested usability bug (see below) and improves
+> legal-benchmark performance, but it still hallucinates confidently on
+> closed-book questions outside its training distribution. Never use its
 > output as legal, financial, or factual advice.
 
+## What's new in this revision
+
+The previous revision of this model would respond to plain conversational
+input ("Hi", "Thanks!", "How are you?") with unrelated, hallucinated
+legal-sounding text — confirmed directly, not hypothetically: it had zero
+conversational examples in its fine-tuning data, every training pair was a
+legal/financial task. This revision adds two small, deliberately-scoped
+slices to the SFT data (about 2.6% of the total):
+
+- **~400 conversational pairs** (greetings, small talk, "what can you do"),
+  sourced from `HuggingFaceTB/everyday-conversations-llama3.1-2k` plus a
+  small hand-written set targeting the exact failure observed, so the model
+  responds naturally to plain conversational input instead of hallucinating.
+- **~170 refusal pairs**, synthesized by the same local Llama-3.1-70B teacher
+  used for the rest of this dataset, covering five categories a small model
+  genuinely cannot answer (future predictions, private individual data,
+  invented/unverifiable legal specifics, real-time data, advice with no
+  facts given) — teaching a calibrated decline instead of a confident,
+  invented answer.
+
+**Result, tested directly:**
+- Conversational input now gets a sensible, on-topic response instead of
+  hallucinated legal text (confirmed by direct generation, not just a
+  benchmark proxy).
+- LegalBench (13-task average) improved from 52.5% → **56.1%** — now ahead
+  of same-size-class Gemma-3-270M and within 0.7 points of OLMo-2-1B despite
+  a quarter its parameters.
+- A rigorous, leak-free paired comparison (scoring both this revision and the
+  previous one on the same held-out questions that neither was trained on)
+  found case-law closed-book QA and LEDGAR classification **statistically
+  unchanged** — an earlier, naive before/after comparison had suggested a
+  regression there, but that comparison turned out to be measuring two
+  different, non-overlapping question samples, not a real capability change.
+- The same clean comparison found a **real, modest drop in CUAD extraction
+  precision** (Token-F1 0.886 → 0.818 on the matched, held-out subset) —
+  reported here plainly rather than hidden, consistent with this project's
+  practice throughout. If precise contract-clause extraction is your primary
+  use case, weigh this against the conversational and LegalBench gains.
+
 ## What this is
 
 Same two-mode design as the 125M SFT model:
@@ -151,8 +188,9 @@ Same two-mode design as the 125M SFT model:
    with your question; it extracts/classifies/answers from that given text.
    This is the mode with real, measured reliability.
 2. **Closed-book** — a general legal/financial question with no source text.
-   Meaningfully better than the 125M model here too, but still the weaker,
-   riskier mode — see Known limitations.
+   Weaker and riskier — see Known limitations. Now also handles plain
+   conversational input and declines gracefully on genuinely unanswerable
+   questions (see "What's new" above).
 
 ## Model description
 
@@ -167,67 +205,64 @@ Same two-mode design as the 125M SFT model:
 
 ## Training data
 
-**Deliberately reuses the exact same curated SFT dataset as the 125M model** —
-21,602 pairs: CUAD contract clause extraction (CC BY 4.0), LEDGAR clause
-classification (CC BY 4.0, with a category-collision fix versus CUAD's
-category names — see the paper's Section on the collision failure mode),
-and open-book QA/summarization/extraction over case-law/SEC/educational-web
-passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher.
-Reusing the same dataset (rather than
-rebuilding it) is valid because the tokenized data depends only on the
-tokenizer, which both models share — it isolates the comparison to "does a
-bigger, better-pretrained base model produce a better SFT result," not "is
-the SFT data different."
+Builds on the curated SFT dataset shared with the 125M model — CUAD contract
+clause extraction (CC BY 4.0), LEDGAR clause classification (CC BY 4.0), and
+open-book QA/summarization/extraction over case-law/SEC/educational-web
+passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher — plus
+the ~570 conversational/refusal pairs described above.
 
 | | |
 |---|---|
-| Train / val split | 20,522 / 1,080 |
+| Train / val split | 21,064 / 1,108 |
+| Total pairs | 22,172 (21,602 original + 398 conversational + 172 refusal) |
 | Method | full fine-tune (not LoRA), 2 epochs |
-| Epoch 1 val_loss → epoch 2 val_loss | 0.1811 → 0.1897 (ticked up — a real, measured overfitting signal on this dataset size, matching the same pattern found in the earlier 5-source checkpoint; do not push epochs further without new data) |
 
 ## Evaluation
 
-Full held-out validation set (1,080 examples, using the 7-source base
-checkpoint), same methodology as the 125M model's card — extraction/
-classification against ground truth, general QA judged by an independent
-local Meta-Llama-3.1-70B-Instruct:
+Full held-out validation set (1,108 examples), same methodology as previous
+revisions — extraction/classification against ground truth, general QA
+judged by an independent local Meta-Llama-3.1-70B-Instruct:
 
-| Task | 125M | **500M** |
+| Task | Previous revision | **This revision** |
 |---|---|---|
-| CUAD contract clause extraction (Token-F1) | 0.799 | 0.786 |
-| CUAD "clause not present" refusal accuracy | 83.5% | 84.7% |
-| LEDGAR clause classification (exact-match) | 76.9% | **79.1%** |
-| Case law general Q&A (closed-book) | 50.0% | 50.0% |
-| SEC filings general Q&A (closed-book) | 64.1% | 56.2% |
-| Educational web general Q&A (closed-book) | 35.5% | 35.5% |
-
-**A direct, paired, per-question comparison** (both models answering the
-*identical* 1,080 questions) found a genuinely mixed result, reported
-plainly rather than only highlighting the wins: LEDGAR classification
-improves for the 500M model, SEC closed-book knowledge regresses, and CUAD
-extraction, case law, and educational-web closed-book QA are essentially
-flat between the two model sizes. This is not a clean sweep in either
-direction — bigger is not strictly better on every question or every task.
+| CUAD contract clause extraction (Token-F1) | 0.786 | 0.798 (aggregate) / **0.818** (leak-free matched subset, n=25) |
+| CUAD "clause not present" refusal accuracy | 84.7% | 84.2% |
+| LEDGAR clause classification (exact-match) | 79.1% | 75.8% (aggregate) / **80.0%** (leak-free matched subset, n=10 — unchanged) |
+| Case law general Q&A (closed-book) | 50.0% | 43.5% (aggregate) / **28.6%** (leak-free matched subset, n=14 — unchanged vs. previous on same subset) |
+| SEC filings general Q&A (closed-book) | 56.2% | 54.9% |
+| Refusal on genuinely unanswerable questions | n/a (not tested previously) | **90.9%** |
+
+**A methodology note, stated plainly:** the "aggregate" numbers above compare
+different, largely non-overlapping validation samples (re-curating the
+dataset to add the new pairs reshuffled which examples land in val), which
+makes some of the aggregate deltas look larger than the real effect. Where
+possible, the matched columns above instead compare both revisions on the
+*exact same* held-out questions that neither was trained on — that is the
+number to trust for case-law and LEDGAR specifically. CUAD's matched
+subsample does show a real, if modest, decline in extraction precision.
 
 ## Known limitations — read before using
 
-- **Open-book contract tasks work well and are the trustworthy mode** —
-  measurably better than the 125M model across the board.
-- **Closed-book general Q&A is improved but still fundamentally limited.**
+- **Open-book contract tasks work well and are the trustworthy mode.**
+- **Plain conversational input is now handled sensibly** (tested directly),
+  but this model was never trained for open-ended general-knowledge chat —
+  ask it something outside law/finance (e.g. general trivia) and it will
+  still confidently hallucinate, because its entire pretraining corpus is
+  legal/financial/regulatory text. This is a pretraining-scope limitation
+  that fine-tuning data cannot fix.
+- **Closed-book general legal/financial Q&A is still fundamentally limited.**
   A 528.5M-parameter model can store at most ~2 bits of knowledge per
   parameter (Allen-Zhu & Li, ["Physics of Language Models: Knowledge Capacity
   Scaling Laws"](https://arxiv.org/abs/2404.05405)) — roughly 132MB of total
   compressible fact storage, shared across everything it knows. Combined with
-  the fact that most specific facts in its training data (a case's dollar
-  figure, a specific statute citation) appeared only once or twice — and
-  memorization research (Carlini et al., ["Quantifying Memorization Across
-  Neural Language Models"](https://arxiv.org/pdf/2202.07646)) shows
-  once-seen facts are memorized only ~0.75% of the time versus 40%+ for
-  facts repeated 500+ times — closed-book precision on rare facts is a hard
-  capacity limitation, not something more training on this same recipe fixes.
-- **Confident fabrication is real and was directly observed**, not
-  hypothetical: specific invented clinical/legal details contradicting the
-  real source, and the vague-non-answer regression cited above.
+  the fact that most specific facts in its training data appeared only once
+  or twice, closed-book precision on rare facts is a hard capacity
+  limitation, not something more training on this same recipe fixes.
+- **Confident fabrication on closed-book questions is real and was directly
+  observed**, not hypothetical.
+- **A modest CUAD extraction-precision regression was measured** in this
+  revision (see Evaluation) — worth weighing if precise contract-clause
+  extraction is your primary use case.
 
 **Never use this model's output as legal, financial, or factual advice.**
 Always treat specific claims as unverified until checked against a primary
@@ -235,52 +270,27 @@ source, especially in closed-book use.
 
 ## External benchmark evaluation
 
-This project's own eval scripts (above) are one perspective. To check the same
-claims against outside measurement, both this model and its
-[125M predecessor](https://huggingface.co/DeependraVerma/legal-slm-125m-sft) were
-run through **`lm-evaluation-harness`** (the same framework behind a
-widely used public LLM leaderboard) on five benchmark categories:
+Run through **`lm-evaluation-harness`** (the same framework behind a widely
+used public LLM leaderboard) on five benchmark categories:
 
 | Benchmark | Result | Baseline | Read |
 |---|---|---|---|
-| HellaSwag / ARC-Easy / PIQA (general commonsense) | 34.0% / 45.7% / 60.5% (acc_norm) | n/a | Normal range for a model this size — not a legal claim, a peer-comparison reference point; OLMo-2-1B (~2x params, general-purpose corpus) leads here as expected, but this model beats same-size-class Gemma-3-270M on every one of these tasks |
-| MMLU professional_law / jurisprudence / international_law | 24.8% / 25.9% / 24.0% (acc) | 25% (4-choice) | At the random-chance floor — no real legal knowledge memorized, exactly as the caveats above already say |
-| CaseHOLD (pick the correct legal holding, via LexGLUE) | 19.6% (acc_norm) | 20% (5-choice) | Right at random chance for this fine-tuned model (the pre-fine-tune base checkpoint actually scores higher, 21.9% — see the base model's card); real closed-book legal reasoning is not something this model can do, stated as plainly as a number can |
-| LegalBench — 13-task contract NLI/QA/CFR subset | 52.5% mean acc | 50% (binary) | Modest, genuine positive signal — matches this model's actual strength (given text, answer about that text); within 4 points of OLMo-2-1B (56.7%) despite a quarter the parameters and single-domain training, and ahead of Gemma-3-270M (53.4%) |
+| HellaSwag / ARC-Easy / PIQA (general commonsense) | 34.5% / 42.1% / 62.4% (acc_norm) | n/a | Normal range for a model this size — OLMo-2-1B (~2x params, general-purpose corpus) leads here as expected |
+| MMLU professional_law / jurisprudence / international_law | 24.5% / 27.8% / 24.8% (acc) | 25% (4-choice) | Near the random-chance floor — no real legal knowledge memorized, as the caveats above already say |
+| CaseHOLD (pick the correct legal holding, via LexGLUE) | 20.0% (acc_norm) | 20% (5-choice) | Right at random chance — real closed-book legal reasoning is not something this model can do |
+| LegalBench — 13-task contract NLI/QA/CFR subset | **56.1% mean acc** | 50% (binary) | Genuine positive signal, improved from the previous revision's 52.5% — now ahead of Gemma-3-270M (53.4%) and within 0.7 points of OLMo-2-1B (56.7%) despite a quarter the parameters |
 
 LegalBench isn't in `lm-eval-harness`'s default task set, so a 13-task,
-contract-focused-plus-one-CFR-grounded subset was hand-configured using the
-benchmark's own official prompts (the 13th task, `telemarketing_sales_rule`,
-grounded in 16 C.F.R. § 310.3, was added specifically to test whether the
-CFR pretraining data helped), and checked for training-data contamination
-with an n-gram overlap scan against the full 7-source pretraining corpus
-before trusting the result (a naive check first over-flagged common legal
-boilerplate — standard-of-review language, ToS disclaimer text — as
-"contamination"; a stricter re-check traced the real overlap to 5 of 1,853
-documents, all explainable — e.g. a real, publicly available company's Terms
-of Service — and none affecting the reported numbers, including zero real
-overlap in the two newest sources, CFR and the Federal Register). SCOTUS,
-ECtHR, and EUR-LEX (also part of LexGLUE) were excluded: those documents run
-tens of thousands of tokens, far beyond this model's 1,024-token context, and
-scoring on a truncated sliver of a document isn't a meaningful number.
-
-**Fine-tuning note**: benchmark scores above are for this SFT model; the
-pre-fine-tune base checkpoint scores slightly *higher* on every one of these
-same benchmarks (a small "alignment tax" from specializing toward the
-fine-tuning data's chat format) — see
-[`DeependraVerma/slm-500m-base`](https://huggingface.co/DeependraVerma/slm-500m-base)'s
-model card for those numbers if the benchmark categories above are your
-primary interest.
+contract-focused-plus-one-CFR-grounded subset was hand-configured and
+checked for training-data contamination before trusting the result (see the
+[repo](https://github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks)
+for the full contamination-scan methodology).
 
 **No comparably small domain-specific baseline exists.** We looked for a
 small (<1B parameter), from-scratch, publicly-weighted *generative* legal- or
-financial-domain model to compare against and found none for either domain:
-Lawma (the closest legal-specialization precedent) only ships 8B/70B weights;
-SaulLM's smallest release is 7B; on the financial side, FinBERT is
-encoder-only, and FinGPT/FinMA/Fin-R1/FinTral are all 7B+ fine-tunes of
-general-purpose models, while BloombergGPT never released weights at any
-size. This absence is itself a relevant data point about the current state
-of small, open, domain-specialized language models.
+financial-domain model to compare against and found none for either domain —
+this absence is itself a relevant data point about the current state of
+small, open, domain-specialized language models.
 
 Full methodology, task configs, and raw logs:
 [github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks](https://github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks).

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:667034cea6222654d1f9cb1a1f8ab3899085089ec23d43925d7a35c0e37477b3
+oid sha256:d98982a9348f6d0eb5684e48bc189c03fafeb372bc1e2aba314ab2d034bf382e
 size 2114179880