charliebrown2/wcb-gpt2-124m-from-scratch

GPT-2 124M (from scratch) fine-tuned on World Central Banks

Fine-tuned classifiers for the three tasks of the World Central Banks (WCB) benchmark from Words That Unite The World (NeurIPS 2025 Datasets & Benchmarks):

task	classes
stance	dovish / hawkish / irrelevant / neutral
temporal	forward looking / not forward looking
uncertainty	certain / uncertain

The backbone is a GPT-2 124M implemented from scratch (no transformers modelling code) and pretrained from random init on FineWeb-Edu. LoRA and QLoRA (blockwise NF4 with double-quantized scales) are also hand-written -- no peft, no bitsandbytes.

Checkpoints

file	task	mode	weighted F1	macro F1	trainable params	size
stance/full.pt	stance	full	0.667	0.669	124,478,976	498.0 MB
stance/lora.pt	stance	lora	0.657	0.657	445,440	1.8 MB
stance/qlora.pt	stance	qlora	0.664	0.659	445,440	1.8 MB
temporal/full.pt	temporal	full	0.863	0.862	124,477,440	498.0 MB
temporal/lora.pt	temporal	lora	0.868	0.866	443,904	1.8 MB
temporal/qlora.pt	temporal	qlora	0.864	0.863	443,904	1.8 MB
uncertainty/full.pt	uncertainty	full	0.833	0.772	124,477,440	498.0 MB
uncertainty/lora.pt	uncertainty	lora	0.839	0.768	443,904	1.8 MB
uncertainty/qlora.pt	uncertainty	qlora	0.835	0.764	443,904	1.8 MB

Metrics are on the WCB test split for seed config 5768. The published checkpoint is that seed; the numbers reported in the paper-comparison tables below are means over all three of the paper's seed configs (5768, 78516, 944601).

Base model: base/pretrain_fineweb_edu_9.9B.pt

GPT-2 124M written from scratch and pretrained from random init on one epoch of FineWeb-Edu's 9.9B-token sample, 3x RTX 6000 Ada with DistributedDataParallel, 10.89 h wall.

	this model	OpenAI GPT-2 124M	Karpathy llm.c @10B
val loss (FineWeb-Edu val)	3.0793	3.2924	~3.28
HellaSwag acc_norm	0.3035	0.2955	~0.305

Beating OpenAI's checkpoint on val loss is partly home-turf advantage: the loss is measured on FineWeb-Edu's own validation split, and FineWeb-Edu is a cleaner corpus than the WebText GPT-2 trained on. HellaSwag is the fairer comparison and the margin there is small (+0.008).

The LoRA and QLoRA checkpoints require this file -- they contain only adapters and a classifier head.

Loading

These are not transformers checkpoints -- AutoModel will not load them. The repo ships the model code:

pip install torch tiktoken
python load_example.py stance lora

model.py (GPT-2 + classifier head), peft.py (NF4 + LoRA) and metrics.py are included, and load_example.py shows the full path for each mode.

The classifier reads the hidden state of the last real token. Padding is on the right and attention is causal, so pad positions cannot influence that vector and no attention mask is needed.

Results in context

Weighted F1, mean +- std over the paper's three seeds:

	stance	temporal	uncertainty
majority class	0.181	0.404	0.677
TF-IDF + logistic regression	0.612	0.842	0.817
GPT-2 124M random init, full FT	0.601	0.839	0.823
GPT-2 124M this pretrain, full FT	0.668	0.862	0.838
GPT-2 124M this pretrain, LoRA r=8	0.652	0.862	0.843
GPT-2 124M this pretrain, QLoRA NF4 r=8	0.654	0.858	0.841
GPT-2 124M OpenAI weights, full FT	0.684	0.868	0.846
RoBERTa-Large, full FT (reproduced)	0.734	0.873	0.849
paper's best PLM	0.740	0.868	0.846
paper's best zero-shot LLM (70B-class)	0.620	0.864	0.816

Pretraining is worth +0.067 weighted F1 on stance and much less on the other two tasks, which are largely cued by surface lexis that 17.5k labelled examples teach directly. The reproduced BERT/RoBERTa baselines land within 0.006 of the paper's published numbers on identical splits, which is the evidence that these GPT-2 rows are measured on the same footing.

LoRA trains 0.36% of the weights and still matches full fine-tuning on temporal and beats it on uncertainty -- with 17.5k examples against 124M parameters, freezing the backbone regularizes.

Full evaluation record, including per-bank breakdowns and per-epoch histories for every run, is in results/.

Limitations

• 124M parameters: below the paper's fine-tuned encoder PLMs on stance (0.668 vs 0.740 for RoBERTa-Large).
• Near-useless zero-shot (0.218 calibrated on stance). These are fine-tuned classifiers, not instruction-followers.
• Trained on English-language central bank communications, 1996-2024. Stance labels encode the annotation guidelines of the source paper; do not treat outputs as financial advice.

Citation

@inproceedings{shah2025words,
  title     = {Words That Unite The World: A Unified Framework for Deciphering
               Central Bank Communications Globally},
  author    = {Shah, Agam and Sukhani, Siddhant and Pardawala, Huzaifa and others},
  booktitle = {NeurIPS 2025 Datasets and Benchmarks Track},
  year      = {2025}
}

The WCB dataset is released under CC-BY-NC-SA 4.0; these derived models carry the same terms.