GPT-X2.5-135M is an improved 2.5th generation model in the GPT-X family built on the TX-3 architecture. 135M parameters, 75B tokens, custom 32K tokenizer, 30 layers, and our XGQA attention. Trained from scratch on a multi-source curriculum achieving near state-of-the-art performance on both natural language and mathematical reasoning benchmarks.
Results
GPT-X2.5-135M achieves competitive performance with leading models despite using significantly less training data. It reaches a 25.17 Intelligence Index—within 1.96 points of SmolLM2-135M while using about 27× fewer training tokens.
Benchmark results and Intelligence Index methodology follow the Open SLM Leaderboard. Training compute is estimated as 6 × parameters × training tokens.
| Rank | Organization | Model | Parameters | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 | Int Index | Training tokens |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | HuggingFace | SmolLM2-135M | 135M | 43.22% | 58.63% | 29.69% | 68.44% | 39.20% | 27.13 | 2T |
| 2 | HuggingFace | SmolLM-135M | 135M | 42.70% | 56.31% | 29.01% | 68.28% | 36.80% | 25.74 | 600B |
| 3 | Axiomic Labs | GPT-X2.5-135M | 135M | 40.57% | 51.81% | 29.18% | 69.42% | 38.40% | 25.17 | 75B |
| 4 | MobileLLM-R1-140M-base | 140M | 33.84% | 49.92% | 24.74% | 63.22% | 65.70% | 24.64 | 4.2T | |
| 5 | Axiomic Labs | GPT-X2-125M | 125M | 40.41% | 51.47% | 27.82% | 67.30% | 37.20% | 23.36 | 75B |
| 6 | BananaMind | BananaMind-2-Pro-Preview | 139M | 39.83% | 51.01% | 27.13% | 66.76% | 38.90% | 23.04 | 52B |
| 7 | Axiomic Labs | GPT-X-125M | 125M | 36.57% | 50.76% | 26.62% | 64.96% | 35.60% | 19.94 | 15B |
| 8 | SupraLabs | Supra2-100M-Base | 100.7M | 35.98% | 47.81% | 24.83% | 65.40% | 36.90% | 19.41 | 30B |
| 9 | OPT-125M | 125M | 31.31% | 40.28% | 22.70% | 62.24% | 35.20% | 13.80 | 180B | |
| 10 | OpenAI | GPT-2 (124M) | 124M | 31.26% | 39.35% | 22.35% | 62.08% | 35.70% | 13.58 | 10B |
The Intelligence Index chance-normalizes HellaSwag, combined ARC, PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.
The model also achieves an overall elo of 1104 on the Bananamind-Base-Bench-1.1 and can be viewed on the BananaMindBench Leaderboard.
What's New in 2.5?
| Change | GPT-X2 | GPT-X2.5 | Why |
|---|---|---|---|
| Parameters | 125.08M | 135.03M | More capacity is concentrated in the feed-forward layers |
| Feed-forward width | 1,536 | 1,728 | Expands each SwiGLU MLP from 2.67x to 3x the 576-wide residual stream |
| Context length | 1,024 | 8,192 | Training starts at 2K context and extends to 8K after 60B tokens |
| Attention refinement | QK-Norm | XSA projection; no QK-Norm | Projects out the component of each attention-head output parallel to its current-token value vector |
| Embedding scaling | Multiply by sqrt(576) | None | Matches the formulation used for the XSA training run |
| Tokenizer | 32,768-token custom BPE | 32,770 tokens (32K BPE + ChatML start/end) | Adds native chat-boundary tokens without replacing the base tokenizer |
| Training data | 75B tokens, 4-source curriculum | 75B tokens, expanded 11-source curriculum | Adds Cosmopedia, SmolTalk, and four FinePhrase subsets to the original mix |
Architecture
| Component | Details |
|---|---|
| Position encoding | RoPE (theta=100,000) |
| Normalization | RMSNorm (float32 upcast) |
| Feed-forward | SwiGLU (3-matrix gated MLP, 1,728 intermediate width, 3x ratio) |
| Attention | XGQA -- 9Q / 3KV (3:1) |
| Attention refinement | XSA projection; no QK-Norm |
| Bias | None (all layers bias-free) |
| Embedding | Weight tying, without embedding scaling |
| Depth | 30 layers x 576 hidden |
| Maximum context | Trained to 8,192 tokens |
Config
vocab_size = 32,770 (32K custom BPE + ChatML start/end tokens)
n_layer = 30
n_head = 9 (query heads)
n_kv_heads = 3 (key-value heads, 3:1 GQA)
n_embd = 576
head_dim = 64
intermediate = 1,728 (SwiGLU, 3x ratio)
block_size = 8,192
rope_theta = 100,000
xsa_projection = true
qk_norm = false
embedding_scale = false
total params = 135,032,256
Parameter Breakdown
| Component | Params |
|---|---|
| Token embeddings (32770 x 576) | 18,875,520 |
| Per block (x30): attention + SwiGLU + norms | 3,871,872 |
| 30 transformer blocks | 116,156,160 |
| Final RMSNorm | 576 |
| LM head (tied with embeddings) | 0 |
| Total | 135,032,256 |
Training
Data
Eleven sources are mixed per optimizer step with a progressive curriculum. The run begins with mostly educational and general web text, ramps math and code, introduces synthetic educational and conversational data, then switches to a FinePhrase-heavy final stage at 58B tokens.
| Source | Dataset | Purpose |
|---|---|---|
| FineWeb-Edu | HuggingFaceFW/fineweb-edu (sample-100BT) |
Primary educational web text |
| DCLM | mlfoundations/dclm-baseline-1.0 | High-quality diverse web text |
| FineMath | HuggingFaceTB/finemath (finemath-4plus) |
Mathematical reasoning (score >= 4) |
| NPset-2-Python-Edu | AxiomicLabs/NPset-2-Python-Edu | AST-normalized Python code |
| Cosmopedia | HuggingFaceTB/cosmopedia | Synthetic textbooks, stories, and educational content |
| ArithMark-2 | AxiomicLabs/ArithMark-2.0 | Focused arithmetic practice |
| SmolTalk | HuggingFaceTB/smol-smoltalk | Chat-formatted instruction and conversation data |
| FinePhrase | HuggingFaceFW/finephrase (FAQ, math, table, and tutorial subsets) | Structured explanatory and knowledge-dense text |
- Tokens: 75B
- Tokenizer: 32,770-token custom BPE (32K base vocabulary plus ChatML start/end tokens)
- Validation: held-out FineWeb-Edu; best recorded validation loss 2.671957
Progressive Data Curriculum
The configured weights change at discrete token thresholds. All percentages below come from the effective (second) CURRICULUM definition in the training script:
| Token range | FineWeb-Edu | DCLM | FineMath | Code | Cosmopedia | ArithMark-2 | SmolTalk | FinePhrase |
|---|---|---|---|---|---|---|---|---|
| 0 -- 18B | 58% | 40% | 1% | 1% | 0% | 0% | 0% | 0% |
| 18 -- 18.5B | 56% | 39% | 3% | 2% | 0% | 0% | 0% | 0% |
| 18.5 -- 19B | 54.5% | 37.5% | 5% | 3% | 0% | 0% | 0% | 0% |
| 19 -- 20B | 54% | 37.5% | 5% | 3.5% | 0% | 0% | 0% | 0% |
| 20 -- 40B | 54% | 36% | 6% | 4% | 0% | 0% | 0% | 0% |
| 40 -- 45B | 52% | 36% | 6% | 4% | 2% | 0% | 0% | 0% |
| 45 -- 48B | 50.8% | 36% | 6% | 3.7% | 3% | 0.5% | 0% | 0% |
| 48 -- 50B | 49.5% | 36% | 6% | 3.5% | 4.5% | 0.5% | 0% | 0% |
| 50 -- 53B | 48.5% | 35% | 6% | 2.5% | 7% | 1% | 0% | 0% |
| 53 -- 55B | 48% | 35% | 6% | 1.5% | 8.5% | 1% | 0% | 0% |
| 55 -- 58B | 46% | 32% | 6% | 1% | 9% | 1% | 5% | 0% |
| 58 -- 75B | 0% | 33% | 11.5% | 0% | 0% | 1% | 5% | 49.5% |
The final FinePhrase share is split into FAQ (14.85%), math (14.85%), table (7.425%), and tutorial (12.375%). The loader preserves the configured proportions across each optimizer step and shuffles source order; if a source is unavailable, its share is redistributed proportionally among the active sources.
Optimization
- Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.01)
- Learning rate: 1.5e-3 max, decays to 0
- Schedule: WSD -- 2,000-step warmup, stable through 90% of training, then linear decay to 0 over the final 10%
- Batch size: 524,288 tokens (micro_batch=4-1, seq_len=2048-8192, grad accum)
- Precision: bfloat16 mixed precision
- Gradient clipping: 1.0
Hardware
- 1x RTX 3080 Ti
- Training time: ~800 hours
Citation
@misc{gptx2_2025,
title={GPT-X2.5: Data-Efficient Language Modeling at 135M Scale},
author={Axiomic Labs},
year={2026},
howpublished={\url{https://huggingface.co/AxiomicLabs/GPT-X2.5-135M}},
note={Trained on 75B tokens with a progressive curriculum and custom tokenizer}
}
- Downloads last month
- -