Zenyx V3 Base (1.5B Mixture-of-Experts)
Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.
This is a BASE model — it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue (
"The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.
Current checkpoint: step 64,800 · 36.5B tokens seen
Model Architecture
- Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 active per token, with a Sinkhorn transport-based gate.
- Multi-head Latent Attention (MLA): compresses the KV cache into a low-rank latent subspace, cutting HBM bandwidth and memory footprint.
- Hyper-Connections: Sinkhorn-normalised residual routing for gradient stability at scale.
- Multi-Token Prediction (MTP): one auxiliary prediction head during training.
- Context: pretrained at up to 4,096 tokens (progressive 2,048 → 4,096). YaRN and RoPE scaling factors are precomputed so context can be extended at inference time beyond the trained length.
| Total parameters | ~1.5B |
| Active parameters / token | ~0.4B |
| Layers | 16 (2 dense + 14 MoE) |
| Hidden size | 1,536 |
| Attention heads | 12 (head dim 128) |
| Vocabulary | 129,280 |
| Precision | bfloat16 |
Benchmarks — checkpoint step 64,800 (36.5B tokens)
All tasks are evaluated with the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is taken as the answer. Nothing is generated and no output parsing is involved, so the numbers do not depend on instruction-following ability. 0-shot, full evaluation sets, no subsampling.
acc_norm normalises each continuation's log-likelihood by its length in characters,
which removes the bias toward short answers; it is the headline metric wherever the
task has candidates of differing lengths.
| Benchmark | acc |
acc_norm |
Random | Δ | n | Description |
|---|---|---|---|---|---|---|
| HellaSwag | 29.74% ± 0.46 | 32.66% ± 0.47 | 25.0% | +7.7 | 10,042 | Commonsense sentence completion |
| ARC-Easy | 50.00% ± 1.03 | 45.08% ± 1.02 | 25.0% | +20.1 | 2,376 | Grade-school science questions |
| ARC-Challenge | 20.05% ± 1.17 | 25.51% ± 1.27 | 25.0% | +0.5 | 1,172 | Hard grade-school science questions |
| PIQA | 60.72% ± 1.14 | 59.85% ± 1.14 | 50.0% | +9.8 | 1,838 | Physical commonsense reasoning |
| WinoGrande | 50.51% ± 1.40 | — | 50.0% | +0.5 | 1,267 | Pronoun resolution / coreference |
| OpenBookQA | 17.80% ± 1.71 | 28.00% ± 2.01 | 25.0% | +3.0 | 500 | Elementary science with open book |
| BoolQ | 60.83% ± 0.85 | 62.17% ± 0.85 | 62.2% | -1.4 | 3,270 | Yes/no reading comprehension |
| SciQ | 76.10% ± 1.35 | 67.80% ± 1.48 | 25.0% | +51.1 | 1,000 | Science exam questions with support |
| LAMBADA (OpenAI) | 25.42% ± 0.61 | — | 0.0% | +25.4 | 5,153 | Long-range last-word prediction |
| MMLU (5-shot) | 26.71% ± 0.37 | — | 25.0% | +1.7 | 14,042 | 57 subjects of academic knowledge |
| RACE | 29.89% ± 0.65 | 32.77% ± 0.67 | 25.0% | +7.8 | 4,934 | Exam reading comprehension |
| CommonsenseQA | 27.76% ± 1.28 | 29.57% ± 1.31 | 20.0% | +9.6 | 1,221 | 5-choice commonsense (random = 20%) |
| COPA | 58.00% ± 4.94 | 59.00% ± 4.92 | 50.0% | +8.0 | 100 | Causal reasoning |
| LogiQA | 21.04% ± 1.60 | 25.19% ± 1.70 | 25.0% | +0.2 | 651 | Logical deduction |
| WSC273 | 52.38% ± 3.02 | — | 50.0% | +2.4 | 273 | Winograd coreference |
| TruthfulQA MC1 | 20.20% ± 1.40 | — | 22.8% | -2.6 | 817 | Resistance to common misconceptions |
| Arithmetic | 0.55% ± 0.06 | — | 0.0% | +0.5 | 14,000 | 2-5 digit add/sub/mul, generated in-harness |
Bold marks the metric that is conventional for that task — acc_norm for
HellaSwag, ARC, PIQA and OpenBookQA; acc for WinoGrande, BoolQ, SciQ and
LAMBADA. The convention is applied per task, not chosen per result: it lowers
the reported figure for ARC-Easy (44.53 rather than 49.54) and PIQA (59.79
rather than 60.83). Δ compares the bolded metric to the random baseline.
Both metrics
| Benchmark | acc |
acc_norm |
n |
|---|---|---|---|
| HellaSwag | 29.74% ± 0.46 | 32.66% ± 0.47 | 10,042 |
| ARC-Easy | 50.00% ± 1.03 | 45.08% ± 1.02 | 2,376 |
| ARC-Challenge | 20.05% ± 1.17 | 25.51% ± 1.27 | 1,172 |
| PIQA | 60.72% ± 1.14 | 59.85% ± 1.14 | 1,838 |
| WinoGrande | 50.51% ± 1.40 | — | 1,267 |
| OpenBookQA | 17.80% ± 1.71 | 28.00% ± 2.01 | 500 |
| BoolQ | 60.83% ± 0.85 | 62.17% ± 0.85 | 3,270 |
| SciQ | 76.10% ± 1.35 | 67.80% ± 1.48 | 1,000 |
| LAMBADA (OpenAI) | 25.42% ± 0.61 | — | 5,153 |
| MMLU (5-shot) | 26.71% ± 0.37 | — | 14,042 |
| RACE | 29.89% ± 0.65 | 32.77% ± 0.67 | 4,934 |
| CommonsenseQA | 27.76% ± 1.28 | 29.57% ± 1.31 | 1,221 |
| COPA | 58.00% ± 4.94 | 59.00% ± 4.92 | 100 |
| LogiQA | 21.04% ± 1.60 | 25.19% ± 1.70 | 651 |
| WSC273 | 52.38% ± 3.02 | — | 273 |
| TruthfulQA MC1 | 20.20% ± 1.40 | — | 817 |
| Arithmetic | 0.55% ± 0.06 | — | 14,000 |
Language modelling
| Corpus | Value | Metric |
|---|---|---|
| WikiText-2 (raw) | 25.79 | token-level perplexity |
| WikiText-2 (raw) | 48.19 | word-level perplexity |
| WikiText-2 (raw) | 1.0426 | bits per byte |
| LAMBADA | 36.19 | perplexity of the target word |
WikiText-2 is scored with a rolling 1024-token window at stride 512, so every counted token is predicted with at least 512 tokens of left context and each token is counted exactly once. (Scoring disjoint windows instead inflates these figures by ~15% because the leading tokens of each window are predicted from nothing.)
Method validation
SciQ places the correct answer at a fixed index (option 4), following lm-evaluation-harness. Measured at step 63,200. Log-likelihood scoring is position-blind in principle, so the suite was re-run with the option order shuffled per document as a control:
| SciQ variant | acc |
acc_norm |
|---|---|---|
| answer at fixed index | 75.10% | 68.50% |
| option order shuffled | 76.00% | 68.30% |
The two differ by 0.9 points against a standard error of 1.37, i.e. 0.7σ, confirming the score reflects answer content rather than position. The same comparison bounds the effect of MoE expert-capacity variation across batches at under one point, since both runs score an identical set of (context, continuation) pairs and differ only in batching order.
Does few-shot prompting help? (MMLU by shot count)
| Shots | step 63200 | step 64,800 | Shot source |
|---|---|---|---|
| 5 | 26.07% | 26.71% ± 0.37 | dev split, the published convention |
| 10 | 24.75% | 25.24% ± 0.37 | dev+validation, uniform across all 57 subjects |
| 25 | 25.57% | 26.16% ± 0.37 | dev+validation, 13-25 actual (sparse subjects cap out) |
No. More demonstrations do not help and the 5-shot result is the best of the three at both checkpoints, with 10-shot dropping to the 25% chance line (-1.47 points vs 5-shot at step 64,800, ~2.8 sigma). The same ordering appears independently at both checkpoints, so it is not a fluke of one run.
This is what a model without in-context learning looks like: using examples to infer a task is an ability that emerges later in training, and before it does, extra shots are just tokens competing for attention with the actual question. Practical consequence: prompt this model with a short direct prefix, not a long few-shot preamble.
Arithmetic
Exact-match on the answer, greedy decoding, GPT-3 prompt format
(Question: What is 47 plus 21? / Answer: 68).
| Operation | step 63200 | step 64,800 | n |
|---|---|---|---|
| 2-digit addition | 0.20% | 0.15% | 2,000 |
| 2-digit subtraction | 0.65% | 3.30% | 2,000 |
| 3-digit addition | 0.00% | 0.00% | 2,000 |
| 3-digit subtraction | 0.10% | 0.40% | 2,000 |
| 4-digit addition | 0.00% | 0.00% | 2,000 |
| 5-digit addition | 0.00% | 0.00% | 2,000 |
| 2-digit multiplication | 0.00% | 0.00% | 2,000 |
| overall | 0.136% | 0.550% | 14,000 |
The model essentially cannot do arithmetic — but two-digit subtraction moved from 0.65% to 3.30% between these two checkpoints (5.1x, ~6 sigma on identical problems), which is the signature of a capability just beginning to emerge. Note that 15.5% of the pretraining mix is mathematics, yet that has bought fluency in mathematical language rather than the ability to compute.
Items are generated in-harness from a fixed seed using this prompt format, because
EleutherAI/arithmeticis a loading script with no parquet branch and cannot be fetched underdatasets>=3. Both checkpoints see byte-identical problems, so the comparison is exact — but these numbers are not interchangeable with published EleutherAI/arithmetic results.
Language modelling by genre (The Pile)
Bits-per-byte on each Pile domain, lower is better, scored with the same rolling 1024-token window as WikiText-2 so the numbers are directly comparable to it. This is the clearest picture of what the model is actually good at, because it measures raw prediction rather than multiple-choice ability.
| Domain | bits/byte | perplexity | tokens | Δ vs prev |
|---|---|---|---|---|
| Github | 0.637 | 4.09 | 479,656 | -0.0150 |
| PubMed Central | 0.782 | 16.30 | 292,334 | -0.0048 |
| USPTO Backgrounds | 0.809 | 17.17 | 296,162 | -0.0038 |
| NIH ExPorter | 0.887 | 26.18 | 41,533 | -0.0051 |
| PubMed Abstracts | 0.910 | 21.91 | 306,636 | -0.0032 |
| ArXiv | 0.921 | 8.54 | 447,478 | -0.0050 |
| StackExchange | 0.987 | 14.30 | 387,038 | -0.0030 |
| FreeLaw | 1.001 | 21.78 | 339,260 | -0.0016 |
| Wikipedia (en) | 1.014 | 22.06 | 341,511 | +0.0006 |
| Pile-CC | 1.122 | 36.53 | 326,706 | -0.0027 |
| BookCorpus2 | 1.159 | 34.00 | 139,043 | +0.0003 |
| OpenWebText2 | 1.163 | 34.43 | 348,909 | -0.0057 |
| Enron Emails | 1.274 | 22.80 | 16,119 | -0.0050 |
| Gutenberg (PG-19) | 1.294 | 35.43 | 133,371 | +0.0017 |
| HackerNews | 1.326 | 44.76 | 57,843 | -0.0021 |
| Books3 | 1.353 | 35.17 | 396,623 | -0.0025 |
| OpenSubtitles | 1.363 | 31.92 | 234,514 | -0.0008 |
| PhilPapers | 1.394 | 55.19 | 38,263 | -0.0061 |
| DM Mathematics | 1.405 | 8.42 | 370,912 | +0.0069 |
| Ubuntu IRC | 1.813 | 46.39 | 14,407 | -0.0012 |
| YoutubeSubtitles | 1.934 | 147.78 | 51,428 | -0.0707 |
| EuroParl | 2.063 | 146.65 | 19,523 | -0.0331 |
The ordering here is a direct readout of the pretraining mix: code, papers and mathematics sit at the top because they are what the model has been fed most of.
Progress since the previous checkpoint
Same suite, same code, same full evaluation sets — only the checkpoint differs. Step 63,200 → 64,800 is +1.68B tokens.
| Benchmark | step 63,200 | step 64,800 | Δ | ±2σ needs |
|---|---|---|---|---|
HellaSwag (acc_norm) |
32.22% | 32.66% | +0.44 | ±0.66 |
ARC-Easy (acc_norm) |
44.53% | 45.08% | +0.55 | ±1.44 |
ARC-Challenge (acc_norm) |
25.77% | 25.51% | -0.26 | ±1.80 |
PIQA (acc_norm) |
59.79% | 59.85% | +0.05 | ±1.62 |
WinoGrande (acc) |
49.49% | 50.51% | +1.03 | ±1.99 |
OpenBookQA (acc_norm) |
30.00% | 28.00% | -2.00 | ±2.87 |
BoolQ (acc) |
60.55% | 60.83% | +0.28 | ±1.21 |
SciQ (acc) |
75.10% | 76.10% | +1.00 | ±1.92 |
LAMBADA (OpenAI) (acc) |
25.50% | 25.42% | -0.08 | ±0.86 |
MMLU (5-shot) (acc) |
26.07% | 26.71% | +0.63 | ±0.53 |
RACE (acc_norm) |
32.79% | 32.77% | -0.02 | ±0.95 |
CommonsenseQA (acc_norm) |
29.57% | 29.57% | +0.00 | ±1.85 |
COPA (acc) |
58.00% | 58.00% | +0.00 | ±6.98 |
LogiQA (acc_norm) |
25.81% | 25.19% | -0.61 | ±2.42 |
WSC273 (acc) |
54.95% | 52.38% | -2.56 | ±4.27 |
TruthfulQA MC1 (acc) |
19.22% | 20.20% | +0.98 | ±1.97 |
Arithmetic (acc) |
0.14% | 0.55% | +0.41 | ±0.07 |
| WikiText-2 perplexity | 26.47 | 25.79 | -0.674 | — |
| WikiText-2 bits/byte | 1.051 | 1.043 | -0.008274 | — |
| LAMBADA perplexity | 35.79 | 36.19 | +0.4003 | — |
Δ is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; anything unbolded is inside the noise floor and should not be read as movement. The quoted error treats the two runs as independent, which is conservative here — they score identical items, so the true paired error is smaller.
What actually changed. No individual multiple-choice benchmark moved significantly — the largest is MMLU at +0.63 points (1.2 sigma), and a sign test over all seventeen is not significant. That is the expected result: +1.68B on 34.8B is a 4.8% increase, and tasks with +/-0.5 to 5 point error bars cannot resolve it. Two continuous measures can, and both say the model improved:
- Arithmetic went from 19/14,000 to 77/14,000 correct, driven by two-digit subtraction rising 0.65% -> 3.30% (5.1x, ~6 sigma). Both checkpoints saw byte-identical generated problems, so this is a paired comparison. A capability crossing from absent to occasionally-present is worth far more than a fraction of a point on HellaSwag.
- The Pile: 18 of 22 domains improved in bits-per-byte (sign test p = 0.0022), token-weighted mean 1.0613 -> 1.0572. Each domain is a separate text distribution scored on identical token counts, so these are close to independent measurements agreeing. The largest gains are in the model's WORST domains -- YoutubeSubtitles -0.071 and EuroParl -0.033 -- i.e. it is filling in weaknesses faster than polishing strengths.
WikiText-2 perplexity (-2.55%) agrees with both. LAMBADA perplexity moved 1.12% the wrong way, but over 6,488 target tokens against WikiText's 287,596 and the Pile's 5,079,269, so it carries the least weight of the three.
Reading these numbers. This is a partially-trained 1.5B base model, so knowledge-heavy multiple-choice tasks sit close to their random baselines — that is expected at this scale and token count. The signal to watch is the language-modelling side: LAMBADA accuracy and WikiText perplexity measure whether the model has actually learned to predict text, and those improve steadily long before multiple-choice benchmarks move. Note also that BoolQ's majority-class baseline is 62.2%, so a score near that is not evidence of comprehension.
Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.
| Metric | Value | Notes |
|---|---|---|
| Decode speed | 68.5 tok/s | steady-state autoregressive decode |
| Warm prefill | ~20 ms | short prompt, shape already compiled |
| Checkpoint load | ~26 s | params → GPU, from local cache |
| Active VRAM | ~5.0 GB | of 24 GB |
Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.
Inference Example
from zenyx_v3_inference import ZenyxGenerator
generator = ZenyxGenerator(step=64800)
# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(generator.generate(
"The capital of France is",
max_new_tokens=80,
temperature=0.7,
repetition_penalty=1.15,
))
Evaluation Reproducibility
Benchmarks were produced by modal_base_evals.py on a single NVIDIA L4, scoring
continuations in batches with length-bucketed padding. Task formats follow the
lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions
and answer-key handling), so the numbers are broadly comparable to published
base-model results, though this is an independent implementation rather than a
harness run.
Limitations
- Pretraining is incomplete — the model will change substantially with more tokens.
- Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
- Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.