babylm-2026-mtl-stage2

A 213M-parameter GPT-2-style causal language model for the multilingual track of the BabyLM 2026 Challenge (English, Dutch, Chinese), trained from scratch within the 100M English-equivalent-word budget.

This is the zero-shot-oriented variant. It is identical to babylm-2026-mtl-bigckpt except that part of the real text is replaced by higher-quality synthetic data, which raises zero-shot at the cost of fine-tuning.

Model details

Architecture	GPT-2-style decoder with multi-token prediction (MTP)
Parameters	212,790,272 (213M)
Layers / heads / hidden	14 / 16 / 1024
Context length	512
Tokenizer	byte-level BPE, 32,000 vocab
Training objective	next-token + MTP auxiliary heads ($k{=}4$, ramped $1\to4$ over epochs; only the first head is used at inference)
Optimizer	AdamW, peak LR $4.5\times10^{-4}$, 10 epochs
Track	BabyLM 2026 multilingual
Languages	English, Dutch, Chinese

Training data

Same as babylm-2026-mtl-bigckpt (BabyBabelLM + Tatoeba/OPUS-100 parallel text + task-family-targeted synthetic text from Qwen3.5-4B), plus higher-quality synthetic data (Qwen3.5-9B, ~14% of the corpus) replacing part of the real text to maximize zero-shot.

Published training data: 1ghub/babylm-2026-mtl-data

Note on budget: language shares were balanced with the track's Byte Premium; the corpus construction used a Chinese premium of 0.93 rather than the official 0.9894, so budget figures should be treated as provisional pending a recount with the official 2026 premiums.

Results

Development-set scores (percentages):

Metric	Score
Zero-shot (16-task mean, ZS16)	58.40 (higher than bigckpt)
Fine-tuning (22-task mean, FT22)	39.52
ZS — English / Dutch / Chinese	60.52 / 60.27 / 54.03

Relative to bigckpt, replacing real text with higher-quality synthetic data increases zero-shot (ZS16 57.82 → 58.40) and decreases fine-tuning (FT22 41.25 → 39.52). This model illustrates one pole of that trade-off.

Intended use and limitations

Research model for data-efficient multilingual language modeling under the BabyLM setting. Small model, developmentally plausible data budget, not intended for production use. Scores are on development sets and are subject to selection bias from evaluation reuse.

Teacher models

Synthetic training text was generated (not distilled) by Qwen3.5-4B and Qwen3.5-9B, approved teachers for the challenge. Generation prompts used task-level descriptions.

Citation

Companion paper: Language Allocation and Targeted Data Design for a Multilingual BabyLM (BabyLM 2026 Workshop, EMNLP 2026).