README.md

---
license: apache-2.0
library_name: mlx
pipeline_tag: text-generation
base_model: Qwen/Qwen3-Next-80B-A3B-Thinking
tags:
  - mlx
  - qwen3_next
  - 4-bit
  - mxfp4
  - text-generation
quantization_config:
  bits: 4
  mode: mxfp4
  group_size: 32
model-index:
  - name: Qwen3-Next-80B-A3B-Thinking mxfp4 (MLX 4-bit)
    results: []
---

# Qwen3-Next-80B-A3B-Thinking — MLX 4-bit (mxfp4)

This repository provides an Apple MLX-optimized 4-bit mxfp4 quantized checkpoint of the base model
`Qwen/Qwen3-Next-80B-A3B-Thinking` for fast, memory‑efficient local inference on Apple Silicon.

Key details
- Format: MLX runtime, safetensors sharded weights
- Quantization: mxfp4, group_size=32 with selective 8‑bit gates
- Task: text generation / chat
- Tokenizer: provided via `tokenizer.json` (BPE) with `chat_template.jinja`

## Files
- `model-0000X-of-00009.safetensors` and `model.safetensors.index.json` — weights (LFS)
- `config.json` — architecture and quantization keys for MLX loaders
- `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`
- `chat_template.jinja` — chat formatting for multi‑turn prompts
- `generation_config.json` — sensible default generation params
- `LICENSE` — Apache-2.0

## Usage (MLX)
Install MLX-LM and run generation:

```bash
pip install mlx-lm
```

```python
from mlx_lm import load, generate
repo_id = "abnormalmapstudio/Qwen3-Next-80B-A3B-Thinking-mxfp4-mlx"

model, tokenizer = load(repo_id)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain what mxfp4 quantization is in one paragraph."},
]

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

out = generate(
    model,
    tokenizer,
    prompt,
    max_tokens=512,
    temp=0.7,
    top_p=0.95,
)
print(out)
```

CLI example:

```bash
mlx_lm.generate --model "abnormalmapstudio/Qwen3-Next-80B-A3B-Thinking-mxfp4-mlx" \
  --prompt "List 5 creative dinner ideas." --max-tokens 200
```

## Hardware Notes
- Apple Silicon recommended (M2/M3). The 4‑bit checkpoint reduces memory pressure significantly, but 80B‑class models still require substantial VRAM and system RAM. For best results, ensure macOS swap is available and limit `max_tokens` accordingly.

## Benchmarks
- Environment: Apple Silicon (isolated runs; one model in memory at a time).
- Script: `scripts/bench/qwen_mxfp4_vs_int4.py` with `--runs 1 --max-new 256`.
- Full JSON: [bench_results.json](./bench_results.json)
- Results (representative, single pass):
  - `abnormalmapstudio/Qwen3-Next-80B-A3B-Thinking-mxfp4-mlx`
    - gen_tok_s: ≈ 37.5 tok/s; ttft: ≈ 2.58 s; mem_active: ≈ 42.36 GB
  - `mlx-community/Qwen3-Next-80B-A3B-Thinking-4bit`
    - gen_tok_s: ≈ 33.8 tok/s; ttft: ≈ 3.10 s; mem_active: ≈ 44.84 GB

Notes
- Bench numbers vary with hardware, system load, and MLX version; treat as directional.
- The bench enforces single‑residency by default. Use `--no-isolate` only if you explicitly need concurrent in‑process loads.

## License
- License: Apache-2.0 (this quantized packaging). See `LICENSE`.
- Base model: `Qwen/Qwen3-Next-80B-A3B-Thinking` (see its model card for upstream license and usage terms). Ensure your use complies with the base model’s license.

## Attribution
If you use this model, please credit the upstream Qwen team and note that this is an MLX 4‑bit mxfp4 quantized derivative for Apple Silicon.