mlx-community/NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Read the write-up · All OptiQ quants · Docs

A 120-billion-parameter model that runs on a 36 GB Mac. This is a 2-bit mixed-precision MLX quant of NVIDIA's Nemotron-3-Super-120B-A12B (247 GB at bf16), produced by mlx-optiq. It is 47.5 GB on disk. While it generates, only ~14 GB sits in RAM: the Mamba blocks, attention, router and shared experts stay resident, and the 34 GB of routed mixture-of-experts weights stream off the SSD as the router selects them, through optiq serve --stream-experts.

Nemotron-3-Super is a hybrid: Mamba2 state-space blocks interleaved with attention and a 512-expert sparse MoE (22 active per token). Asked to write Flappy Bird in a single HTML file, the 2-bit model produced a complete, working game. Here it is playing it:

What it is

Property	Value
Base	NVIDIA-Nemotron-3-Super-120B-A12B (hybrid Mamba2 + attention + 512-expert MoE, 22 active)
Method	OptiQ static — structural per-layer bit allocation, no calibration
Bit-widths	4-bit on Mamba / attention / router / shared experts / edges, 2-bit on the routed experts
Achieved bits-per-weight	2.52
On disk	47.5 GB
Resident while running	~14 GB (routed experts streamed)
Decode speed	~3 tok/s on an M3 Max (36 GB)

For a model this large, exact calibration-driven sensitivity is impractical (it would run for days and needs the full model resident as a reference), so OptiQ's static method assigns bits from architecture alone. See the methods comparison.

Run it

This is a Nemotron hybrid (model_type: nemotron_h), so it needs mlx-lm from main and import optiq (install from git, not a version pin):

pip install -U mlx-optiq "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git"

Serve it with SSD expert streaming (auto-enabled for a MoE too big to fit resident; --stream-experts forces it):

optiq serve --model mlx-community/NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bit --stream-experts

Then open the Lab, ask for a game, and watch it render in the Canvas pane. Only the routed experts stream per token; the Mamba state, attention and shared experts stay resident, so the footprint stays ~14 GB no matter how large the model on disk is.

Notes

This is an extreme quant. 2-bit on the routed experts is lossy, and the point of this artifact is that a 120 B hybrid MoE runs at all on consumer Apple Silicon, with coherent output. For reference quality on this base, use the bf16 weights or a higher-bit quant. The full story is in the blog post.