Instructions to use shibatch/tinyqwen3next3m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinyqwen3next3m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shibatch/tinyqwen3next3m")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinyqwen3next3m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use shibatch/tinyqwen3next3m with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinyqwen3next3m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinyqwen3next3m:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinyqwen3next3m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinyqwen3next3m:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf shibatch/tinyqwen3next3m:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf shibatch/tinyqwen3next3m:Q4_K_M
Use Docker
docker model run hf.co/shibatch/tinyqwen3next3m:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use shibatch/tinyqwen3next3m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shibatch/tinyqwen3next3m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen3next3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/shibatch/tinyqwen3next3m:Q4_K_M
- SGLang
How to use shibatch/tinyqwen3next3m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shibatch/tinyqwen3next3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen3next3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shibatch/tinyqwen3next3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen3next3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Ollama
How to use shibatch/tinyqwen3next3m with Ollama:
ollama run hf.co/shibatch/tinyqwen3next3m:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use shibatch/tinyqwen3next3m with Docker Model Runner:
docker model run hf.co/shibatch/tinyqwen3next3m:Q4_K_M
- Lemonade
How to use shibatch/tinyqwen3next3m with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull shibatch/tinyqwen3next3m:Q4_K_M
Run and chat with the model
lemonade run user.tinyqwen3next3m-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Tiny Qwen3-Next 3M
A 3,061,200-parameter Qwen3NextForCausalLM model trained from scratch
for one epoch on TinyStories. It contains three Gated DeltaNet layers and one
full-attention layer, each with eight routed experts (top-2) and a shared
expert. This is a small architecture-test model, not an official Qwen release
or an instruction-tuned assistant.
This release replaces the earlier 2,945,914-parameter checkpoint with a
quantization-friendly layout. The tokenizer is unchanged, byte-for-byte:
hf/tokenizer.json has SHA-256
773baf598902cda9e89c8cec2e1598cfe249bc9e763661c8904f006c2ccce59d.
Its 1,003 token IDs, vocabulary, merges, and special-token IDs remain the
same. The model weights are new and are not compatible with the old weights.
Files
| Path | Purpose |
|---|---|
hf/ |
Float32 Hugging Face Safetensors model and original tokenizer |
gguf/tinyqwen3next3m-Q4_K_M.gguf |
llama.cpp Q4_K_M release |
convert_to_gguf.py |
Rebuild the GGUF using llama.cpp tools |
example_generate.py |
Local Transformers generation example |
artifact_metadata.json |
Training arguments, final metrics, routing usage |
eval_text_generation.json |
Final Transformers sample generations |
qwen3_next_config_dump.json |
Expanded model configuration |
Optimizer checkpoints and the training log are not part of the distribution.
Architecture
architecture: Qwen3NextForCausalLM
model_type: qwen3_next
parameters: 3061200
vocab_size: 1024 # 1003 tokenizer tokens; remaining rows reserved
hidden_size: 256
num_hidden_layers: 4
layer_types: [linear_attention, linear_attention, linear_attention, full_attention]
num_attention_heads: 8
num_key_value_heads: 1
head_dim: 32
linear_num_key_heads: 4
linear_key_head_dim: 32
linear_num_value_heads: 8
linear_value_head_dim: 32
intermediate_size: 256
num_experts: 8
num_experts_per_tok: 2
moe_intermediate_size: 64
shared_expert_intermediate_size: 64
The widths are multiples of 32 so llama.cpp can quantize important matrices with K-quants. The Q4_K_M preset is mixed precision by design: the GGUF has 26 actual Q4_K tensors, while some tensors use Q5_K, Q8_0, or F32. In particular, the expert down-projection tensors use Q8_0. It is not an all-Q4_K file. The quantized model was loaded and tested with llama.cpp.
This is Qwen3-Next, not the separate Qwen3MoeForCausalLM architecture.
Training
The original byte-level BPE tokenizer was loaded from the preceding release,
not retrained. Special tokens are <s> (ID 1000, BOS/pad), </s> (ID 1001,
EOS), and <|im_start|> (ID 1002). Text was packed as a continuous stream
of BOS + story + EOS into 256-token blocks without padding.
The full run used 2,119,489 TinyStories rows: 2,098,294 training and 21,195 held-out validation stories. It used a batch size of 32, 76,284 optimizer steps (one epoch), AdamW, a 3e-4 peak learning rate, 1,000 warmup steps, cosine decay, and 1.0 gradient clipping. Weights and optimizer states were float32; CUDA forward passes used BF16 autocast. The run was performed on an NVIDIA GeForce RTX 5060 Ti, the GPU available in this environment.
Before the full run, a 5% data smoke test reached validation loss 1.9866
after 3,000 steps and produced recognizable English in both Transformers and
the Q4_K_M llama.cpp build. The completed one-epoch run reached held-out
validation loss 1.4624 (perplexity 4.3165); the final training
language-model loss was 1.4523. Validation used 16 batches of packed
256-token blocks. Further metrics and samples are in artifact_metadata.json
and eval_text_generation.json.
With sampling seed 0, the released GGUF produced this TinyStories-style
opening from Once upon:
Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toy cars and trucks. One day, he found a big jar of jelly on the table.
The tokenizer IDs for a test prompt matched exactly between the HF and GGUF versions. Generations remain stochastic and can become inconsistent later.
Usage
Install requirements.txt, then run python example_generate.py from this
package directory. The model requires a Transformers version with
Qwen3NextForCausalLM; it was tested with Transformers 5.15.0.
For the GGUF build, install a llama.cpp version with Qwen3-Next conversion and inference support. For example:
llama-completion -m gguf/tinyqwen3next3m-Q4_K_M.gguf \
-p 'Once upon' -n 100 --no-conversation
To rebuild the GGUF from the included HF checkpoint:
python convert_to_gguf.py
The script uses llama-quantize on PATH and the Debian/Ubuntu
llama.cpp-tools converter path by default. Use --converter and
--quantizer to point to another llama.cpp build. The temporary F16 GGUF is
created beside the output and removed after conversion.
For a Hub copy, download the hf/ subdirectory and pass that local directory
to AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained.
Limitations
This tiny English-only TinyStories model is useful for Qwen3-Next, Gated DeltaNet, MoE routing, tokenizer, and inference-engine tests. It is not suited to factual QA, instruction following, or production use. Sampling can produce repetition, contradictions, and unfinished stories. Validation loss is a compact diagnostic on TinyStories, not a general language benchmark.
MIT license. Copyright (c) 2026 Naoki Shibata.
- Downloads last month
- -
4-bit