Tiny Qwen3-Next 3M

A 3,061,200-parameter Qwen3NextForCausalLM model trained from scratch for one epoch on TinyStories. It contains three Gated DeltaNet layers and one full-attention layer, each with eight routed experts (top-2) and a shared expert. This is a small architecture-test model, not an official Qwen release or an instruction-tuned assistant.

This release replaces the earlier 2,945,914-parameter checkpoint with a quantization-friendly layout. The tokenizer is unchanged, byte-for-byte: hf/tokenizer.json has SHA-256 773baf598902cda9e89c8cec2e1598cfe249bc9e763661c8904f006c2ccce59d. Its 1,003 token IDs, vocabulary, merges, and special-token IDs remain the same. The model weights are new and are not compatible with the old weights.

Files

Path Purpose
hf/ Float32 Hugging Face Safetensors model and original tokenizer
gguf/tinyqwen3next3m-Q4_K_M.gguf llama.cpp Q4_K_M release
convert_to_gguf.py Rebuild the GGUF using llama.cpp tools
example_generate.py Local Transformers generation example
artifact_metadata.json Training arguments, final metrics, routing usage
eval_text_generation.json Final Transformers sample generations
qwen3_next_config_dump.json Expanded model configuration

Optimizer checkpoints and the training log are not part of the distribution.

Architecture

architecture: Qwen3NextForCausalLM
model_type: qwen3_next
parameters: 3061200
vocab_size: 1024                 # 1003 tokenizer tokens; remaining rows reserved
hidden_size: 256
num_hidden_layers: 4
layer_types: [linear_attention, linear_attention, linear_attention, full_attention]
num_attention_heads: 8
num_key_value_heads: 1
head_dim: 32
linear_num_key_heads: 4
linear_key_head_dim: 32
linear_num_value_heads: 8
linear_value_head_dim: 32
intermediate_size: 256
num_experts: 8
num_experts_per_tok: 2
moe_intermediate_size: 64
shared_expert_intermediate_size: 64

The widths are multiples of 32 so llama.cpp can quantize important matrices with K-quants. The Q4_K_M preset is mixed precision by design: the GGUF has 26 actual Q4_K tensors, while some tensors use Q5_K, Q8_0, or F32. In particular, the expert down-projection tensors use Q8_0. It is not an all-Q4_K file. The quantized model was loaded and tested with llama.cpp.

This is Qwen3-Next, not the separate Qwen3MoeForCausalLM architecture.

Training

The original byte-level BPE tokenizer was loaded from the preceding release, not retrained. Special tokens are <s> (ID 1000, BOS/pad), </s> (ID 1001, EOS), and <|im_start|> (ID 1002). Text was packed as a continuous stream of BOS + story + EOS into 256-token blocks without padding.

The full run used 2,119,489 TinyStories rows: 2,098,294 training and 21,195 held-out validation stories. It used a batch size of 32, 76,284 optimizer steps (one epoch), AdamW, a 3e-4 peak learning rate, 1,000 warmup steps, cosine decay, and 1.0 gradient clipping. Weights and optimizer states were float32; CUDA forward passes used BF16 autocast. The run was performed on an NVIDIA GeForce RTX 5060 Ti, the GPU available in this environment.

Before the full run, a 5% data smoke test reached validation loss 1.9866 after 3,000 steps and produced recognizable English in both Transformers and the Q4_K_M llama.cpp build. The completed one-epoch run reached held-out validation loss 1.4624 (perplexity 4.3165); the final training language-model loss was 1.4523. Validation used 16 batches of packed 256-token blocks. Further metrics and samples are in artifact_metadata.json and eval_text_generation.json.

With sampling seed 0, the released GGUF produced this TinyStories-style opening from Once upon:

Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toy cars and trucks. One day, he found a big jar of jelly on the table.

The tokenizer IDs for a test prompt matched exactly between the HF and GGUF versions. Generations remain stochastic and can become inconsistent later.

Usage

Install requirements.txt, then run python example_generate.py from this package directory. The model requires a Transformers version with Qwen3NextForCausalLM; it was tested with Transformers 5.15.0.

For the GGUF build, install a llama.cpp version with Qwen3-Next conversion and inference support. For example:

llama-completion -m gguf/tinyqwen3next3m-Q4_K_M.gguf \
  -p 'Once upon' -n 100 --no-conversation

To rebuild the GGUF from the included HF checkpoint:

python convert_to_gguf.py

The script uses llama-quantize on PATH and the Debian/Ubuntu llama.cpp-tools converter path by default. Use --converter and --quantizer to point to another llama.cpp build. The temporary F16 GGUF is created beside the output and removed after conversion.

For a Hub copy, download the hf/ subdirectory and pass that local directory to AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained.

Limitations

This tiny English-only TinyStories model is useful for Qwen3-Next, Gated DeltaNet, MoE routing, tokenizer, and inference-engine tests. It is not suited to factual QA, instruction following, or production use. Sampling can produce repetition, contradictions, and unfinished stories. Validation loss is a compact diagnostic on TinyStories, not a general language benchmark.

MIT license. Copyright (c) 2026 Naoki Shibata.

Downloads last month
-
GGUF
Model size
3.06M params
Architecture
qwen3next
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Dataset used to train shibatch/tinyqwen3next3m