legal-colbert-clause-retriever

A small, open late-interaction (ColBERT) retriever fine-tuned for finding clauses in legal contracts — termination, assignment, limitation of liability, IP ownership, non-compete, governing law, and ~35 other common provision types. It maps queries and contract passages to sequences of 128-d token vectors and scores them with the MaxSim operator.

It is a continuation fine-tune of lightonai/GTE-ModernColBERT-v1 (149M params, ModernBERT-base backbone).

Results

Evaluated on the MLEB Contractual Clause Retrieval task (NDCG@10), the published benchmark for legal contract clause retrieval. Our evaluation reproduces the official leaderboard protocol exactly (BGE-M3 scores 0.7281 through our harness, matching the leaderboard to 4 decimals).

Metric	Score
NDCG@10	0.8374
MAP	0.7777
Recall@10	0.9444

Current release (2026-08-01). The NDCG@10 gain over the prior release (0.8338) is within this 45-query benchmark's noise band (bootstrap CI crosses zero), but the release weakly dominates its predecessor: it also improves adjudicated real-contract retrieval, clause-subtype discrimination, and held-out CUAD clause retrieval, with no per-query regressions.

Sibling model: for extracting clauses from real, filed contracts (rather than matching idealized template clauses), see legal-colbert-extractor — selected on a human-adjudicated blind SEC-contract gate, where it scores higher (0.7781 vs 0.7683).

At 149M parameters this is the best accuracy-per-parameter open model on the task — 3rd of 17 open-source models, ahead of Google's EmbeddingGemma (308M, 0.829) and the same-size legal peer Free Law ModernBERT (0.764), and behind only Qwen3-Embedding-4B/8B (which are 27–53× larger).

Usage

pip install pylate

from pylate import models, rank

model = models.ColBERT("kmad00/legal-colbert-clause-retriever")

# Describe the clause you want to find
queries = model.encode(
    ["This is a contractual provision that limits the maximum liability a party can incur."],
    is_query=True,
)

# Candidate contract passages
documents = model.encode(
    [
        "In no event shall either party's aggregate liability exceed the fees paid in the prior twelve months...",
        "This Agreement shall be governed by the laws of the State of Delaware...",
    ],
    is_query=False,
)

scores = rank.rerank(
    documents_ids=[["0", "1"]],
    queries_embeddings=queries,
    documents_embeddings=[documents],
)
print(scores)

Queries can be plain clause names ("governing law"), natural-language definitions, or questions — the model is robust to phrasing. Document length 300 tokens, query length 48, output dim 128, similarity = MaxSim.

Query phrasing matters

This model is trained on descriptive clause definitions and is measurably query-phrasing sensitive: terse keyword queries can flip rankings. Phrase queries as full descriptions of the provision you want:

• ✅ "Find contractual provisions governing termination rights, termination for cause, or the consequences of ending the agreement."
• ✅ "This is a contractual provision that limits the maximum liability a party can incur."
• ⚠️ "termination clause" — keyword-style queries underperform and can rank unrelated provisions above the target.

If you are building an application on top, map user inputs (clause-type names, keywords) to descriptive clause definitions before retrieval.

Supported clause types

Trained and evaluated on 41 CUAD clause categories plus ACORD drafting queries and LEDGAR provision labels, including: Cap on Liability / Uncapped Liability, IP Ownership Assignment, Joint IP Ownership, License Grant, Non-Compete, Anti-Assignment, Change of Control, Governing Law, Termination for Convenience, Renewal Term, Audit Rights, Insurance, Most Favored Nation, Exclusivity, Liquidated Damages, Source Code Escrow, ROFR/ROFO/ROFN, and more. As a retriever (not a fixed classifier) it also generalizes to clause types outside this set.

License

CC BY 4.0. This model is a derivative of CC BY 4.0 training data (CUAD, ACORD, LEDGAR) and an Apache 2.0 base model. You may use it commercially and non-commercially; attribution is required (see below). No share-alike obligation applies.

Base model

• lightonai/GTE-ModernColBERT-v1 — Apache 2.0 (← Alibaba-NLP/gte-modernbert-base ← answerdotai/ModernBERT-base)

Training data

Produced by a chain of light continuation fine-tunes. Across the full lineage it was trained on the following datasets (and no others):

• CUAD — CC BY 4.0. Hendrycks et al., "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review," NeurIPS 2021. The Atticus Project.
• ACORD — CC BY 4.0. The Atticus Project, "ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting," 2025.
• LEDGAR (LexGLUE ledgar config) — CC BY 4.0. Tuggener et al., "LEDGAR," LREC 2020; derived from public-domain US SEC EDGAR filings. Chalkidis et al., "LexGLUE," ACL 2022.

Hard negatives were mined with BM25 from each dataset's own corpus. No MLEB / isaacus/contractual-clause-retrieval data and no web-scraped data were used in training — MLEB is used only as an evaluation benchmark.

Limitations

• English-language commercial contracts (US-style); other jurisdictions/languages are out of distribution.
• Late-interaction (multi-vector) storage is heavier per document than single-vector embedders.
• The MLEB clause task is small (90 docs); treat ±1–2 points as noise.
• Trained on a narrow set of clause types; confidence is lower on provision types far from the training taxonomy.

Acknowledgments

• Training data: The Atticus Project (CUAD, ACORD); Tuggener et al. & coastalcph/LexGLUE (LEDGAR).
• Base model: LightOn (GTE-ModernColBERT-v1), built with PyLate.
• Benchmark: Isaacus (MLEB) — evaluation only, not training.

Full model architecture

ColBERT(
  (0): Transformer({'max_seq_length': 299, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'Identity'})
)