Training a 2.7B MoE from scratch for $200, one GPU at a time

---

A few of us trained NanoColibri-Instruct — a 2.7B-parameter Mixture-of-Experts with 0.34B active params per token — from scratch, on rented GPUs, for about $180–260 total. No cluster: pretraining is sequential, so contributors took turns passing a baton, one GPU at a time. At its 5.4B-token budget it beats token-matched dense models on 5 of 7 zero-shot tasks with fewer active parameters than either baseline.

This post is the story, the numbers, the lessons (two of them embarrassing), and what's next. Everything is reproducible from the training repo: code, checkpoints, benchmark harness, and a public ledger of who trained what.

Why train a small MoE at all

The end goal isn't Nano — it's a class of models whose int4 containers are deliberately bigger than a consumer machine's memory, streaming experts from NVMe under a fixed RAM budget. A 16 GB laptop can't hold a 24–28B model in RAM; but in a fine-grained MoE, each token only touches the dense backbone, one always-resident shared expert, and a couple of 3–4 MB routed experts. Keep the resident stack pinned, cache experts with an LRU, and let the disk serve the misses — that's the bet.

Nano is the proof-of-loop: architecture, training infrastructure, int4 export, and a serving engine that streams — all exercised end-to-end at a price where mistakes don't hurt.

Relay pretraining: a compare-and-swap on the Hub

You can't split sequential pretraining across volunteers' GPUs the way you split a codebase — every step depends on the previous one. So the run lives in one HF model repo (weights + optimizer state + training_state.json + RELAY.json + LEDGER.md), and contributors take turns:

claim → pull → train a leg → push → release.

The part that has to be airtight is that two people never train the same leg — whoever pushes second silently destroys a GPU-day. So claiming the baton takes a lease, implemented as a real compare-and-swap against the Hub, not a polite convention: if two people claim at once, exactly one wins and the other is told so. Leases expire and are renewed by a heartbeat while training, so a preempted spot instance frees the baton on its own. A read-only dashboard on the training box (port 80) shows loss, throughput, expert balance, and who holds the baton.

It worked: 20,000 updates (~5.2B tokens of FineWeb-Edu) across contributors, then a 1,500-update chat SFT on smol-smoltalk, ~90 H100-hours all-in.

The architecture, and one router lesson

Nano is built in the serving engine's native architecture (HYV3): 24 layers (1 dense + 23 MoE), hidden 1024, 64 experts × 512 wide with top-2 routing, a fat 2048-wide shared expert that never leaves memory, GQA 4× attention with per-head QK-norm, and a sigmoid+bias router with aux-loss-free load balancing (the DeepSeek-V3 recipe: after each optimizer step, nudge each layer's expert bias += γ·sign(target − load); no auxiliary loss fighting the LM objective). The engine requires QK-norm and a shared expert — popular MoE architectures each miss one of them — so the int4 export drops into its fast path with zero conversion shims.

The lesson worth stealing: train top-2 even if you'll serve top-1. The router renormalizes selected gates, so at top_k=1 the routed weight collapses to a constant and the router's weight matrix gets essentially zero gradient from the LM loss — routing would stay frozen at its random init. top_k=2 restores gradient flow through relative competition between the two winners; at serve time the engine still runs one expert per token. Aux-loss-free balancing held up fine at $200 scale: dead-expert fraction stayed low, load stayed healthy, and it leaves the mild routing skew that a streaming cache likes.

Results: what 5.4B tokens buys

Fair comparison needs similar size and similar tokens. We benched everything with an in-repo, lm-evaluation-harness-compatible harness — same prompts, same scoring — and ran the dense references through the same code, rather than trusting published rows. Two token-matched dense baselines:

5 of 7 against both — ties LAMBADA against Pythia, and WinoGrande is within noise (±1.4 at n=1267). Honest caveats attached: the Pythia step-3000 checkpoint is only ~2% into its LR schedule, which handicaps it — that's why the fully-annealed Cerebras-GPT-256M row is there as the conservative comparison, and Nano wins that one too. The LAMBADA lag is at least partly data: FineWeb-Edu contains essentially no fiction, and LAMBADA is built from novels.

Reading: at this budget, a 2.7B-total MoE performs like a good dense model in the 300–600M class — the total parameters buy a real edge over token-matched dense at fewer active params per token. The gap to SmolLM2-class numbers is three orders of magnitude of data, not architecture.

Two embarrassing lessons, so you don't repeat them

The page-cache trap. We "measured streaming" by giving the engine sub-container RAM budgets on a rented server — and got suspiciously flat speeds all the way down to a budget 5× smaller than the container. Of course: the server's RAM dwarfed the 1.2 GB container, so the OS page cache silently backed every "miss." Those runs characterize the engine's cache bookkeeping (useful!), but they never touched the disk. True disk-pressure numbers need memory-capped cgroups on hardware where the cap is actually enforced — and honestly, a bigger container. The speed harness also grew a bench bug we've since found, so this post deliberately contains zero tokens/s claims. Measurement discipline is cheaper to learn at $200.

The spent cosine. A cosine LR schedule fixed to a step target means the schedule is spent when you reach it — there's no headroom to keep training on momentum. The successor runs use WSD (warmup–stable–decay): a long stable phase that's friendly to relay legs, decaying once, when we choose.

What's next

The plan (NEXT_MODEL.md) is two stages. Colibri-Micro (7B total / 1B active, 100B tokens, code-heavy mix, SFT → RLVR for code as the full-dress rehearsal) — and the first container actually big enough (3.8 GB int4) to measure real disk streaming on target laptops — then Colibri-Grande (24–28B total / ~2.4B active), whose container deliberately oversubscribes a 16 GB laptop's memory. Grande's exact expert count gets frozen only after Micro's measured miss/speed curves exist. Everything stays open end-to-end: weights, data recipe, training code, relay protocol, benchmark JSONs, and the ledger.

A note on compute

Nano was paid for out of pocket. The next stage needs a few thousand H100-hours (~$9–12k at market rates); I'm applying for compute grants, and if a cloud provider is interested in sponsoring open-source local-LLM development, I'd be glad to talk — everything the sponsorship produces ships open. Contact: [email protected] / vovaRL on huggingface, or open an issue on the repo.

Reproduce the numbers

git clone https://github.com/view321/NanoColibri && cd NanoColibri
pip install -e ".[data]"
python -m scripts.bench_nano --model vovaRL/NanoColibri-Instruct
python -m scripts.bench_nano --model EleutherAI/pythia-410m --revision step3000
python -m scripts.bench_nano --model cerebras/Cerebras-GPT-256M

Same harness, same prompts, full sets. If you rerun these and get different numbers, open an issue — the benchmark JSONs in the repo are the ones behind every table above.