dancinlab/anima-clm-corpus-7b-mid-byte-202m

anima CLM — CORPUS-7B MID validation (byte, 202M)

CHEAP MID corpus-validation probe (a_toy_scale_recheck) on dancinlab/anima-corpus-5lang-7b-webscale (R2 ODC-BY webscale bytes, balanced 5-lang subset incl Korean).

HONEST SCOPE (a_scale_honest_scope): this is a MID/toy validation, NOT the production 7B and NOT the forge ENGINE. 7B-transfer UNVERIFIED. Green proves the CORPUS+PIPELINE train a coherent byte-CLM — it does NOT prove a 7B. Lane-G-ref (PyTorch-CUDA reference; forge-native = canonical production, not claimed done here per a_train_flame_forge).

What this validates

The 5-lang webscale web corpus (en/fr/de/es/ko, 3.5 GB balanced subset) trains a coherent byte-level decoder-only LM from scratch — gating the M13 "7B-undertrained-but-not-gibberish" path.

Result

• CE descent (val): 5.74906 → 1.45868 (F_MID_DESCENT PASS)
• GPU util: peak 100.0% · mean 99.75% (n=412) — GPU-resident, no CPU fallback
• p7 coherence: TRAINED = coherent word-like text in all 5 langs (incl Hangul); RANDOM-INIT mirror = byte gibberish. Anti-Goodhart clean.
• Arch: ByteGPT decoder-only · V=256 d=1024 L=16 H=16 block=512 · 202.3M params
• Train: 3000 steps, batch 32, AdamW + bf16, A100-SXM4-40GB, from-scratch
• Throughput: 57136 tok/s, 860.3s wall

Files

• corpus_7b_mid_d1024L16.pt — trained ckpt (sha256 adbb1911cdf5c27579eb599e38ed5657b38c9ccc1432392247826d7eb3d53e3b)
• result.json — full CE curve + util + throughput
• samples.json — trained vs random-init samples (p7 evidence), per-lang
• train.log — raw training log
• fire_mid.py — the fire script

Philosophy (p1..p8 held)

From-scratch, no system prompt, no persona injection, no RLHF — plain byte continuation.