Nemotron-3-Super-120B-A12B on Consumer Hardware (llama.cpp)

Companion repo — scripts + tested configs for running NVIDIA's Nemotron-3-Super-120B-A12B with llama.cpp on a 24 GB GPU + 64 GB RAM machine.

⚠️ This repo contains no model weights. The GGUF files are already hosted upstream by bartowski and unsloth. Download them from there — the scripts in this repo do that for you.

This is a configuration and tooling companion: drop-in scripts, an OpenAI-compatible API server, and benchmarked GPU_LAYERS / context-window combinations that actually fit in 24 GB of VRAM.

What's in this repo

File	Purpose
`run-nemotron.sh`	CLI: start the API server, interactive chat, or benchmark
`download-nemotron-3-super.sh`	Downloads GGUF weights (Q2_K, Q3_K_M, UD-Q3_K_M, IQ3_XXS, IQ4_XS)

Model Overview

Nemotron-3-Super-120B-A12B is a Mixture of Experts model from NVIDIA:

Property	Value
Total Parameters	120 billion
Active per Token	~12 billion (22 of 512 experts)
Architecture	Hybrid: Attention + SSM (Mamba-2) + MoE, Multi-Token Prediction
Layers	88 blocks (11 attention, 77 SSM)
Context Window	Up to 1,048,576 tokens
Vocabulary	131,072 tokens
Reasoning	Built-in thinking/reasoning support
License	NVIDIA Nemotron Open Model License

Because only ~12B parameters are active per token, generation speed is much faster than a dense 120B model.

Hardware Requirements

Tested: 24 GB GPU (two cards) + 64 GB RAM

Component	Specification
GPU	RTX 5060 Ti 16 GB (primary) + RTX 4060 Ti 8 GB (secondary) = 24 GB
System RAM	64 GB
Disk	60+ GB free for GGUF files

Minimum Requirements

CPU-only: 64 GB RAM, very slow (~1-2 t/s)
Partial GPU offload: 8+ GB VRAM + 64 GB RAM
Full GPU offload: 40+ GB VRAM (not required)

Prerequisites

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

ls build/bin/libggml-cuda.so  # Should exist

Or point the script at an existing build:

export LLAMA_CPP_DIR=/path/to/your/llama.cpp

2. Verify GPU drivers

nvidia-smi
# If "Driver/library version mismatch": reboot the system

3. Download the Model

./download-nemotron-3-super.sh
# Select option 1 (Q2_K, ~52 GB, 2 shards)

About split GGUF files: Large models are split into multiple shards. Always point to the first shard (-00001-of-*.gguf). llama.cpp automatically detects and loads all shards in the same directory.

Quick Start

# Start API server (default: 20 GPU layers, 4K context, port 8081)
./run-nemotron.sh server

# Interactive chat
./run-nemotron.sh chat

# Check if server is ready
curl http://localhost:8081/health

Recommended Configurations (24 GB GPU + 64 GB RAM)

The two knobs that matter:

GPU_LAYERS: how many of the 88 layers go on the GPU (more = faster, needs more VRAM)
CONTEXT_SIZE: context window in tokens (larger = more RAM, more VRAM for KV cache)

Because Nemotron-3-Super is a hybrid model (only 11 of 88 layers use attention), its KV cache is much smaller than a pure transformer of the same size:

Context Size	KV Cache (fp16)	Use Case
4,096	~90 MB	Short Q&A, code snippets
8,192	~180 MB	Single document analysis
16,384	~360 MB	Multi-turn conversations
32,768	~720 MB	Long document processing
34,816	~770 MB	Recommended for 24GB GPU + 64GB RAM
65,536	~1.4 GB	Book-length analysis
131,072	~2.9 GB	Very long contexts (needs more RAM)

34K Context (Recommended Default)

GPU_LAYERS=20 CONTEXT_SIZE=34816 ./run-nemotron.sh server

Component	VRAM (GPU)	RAM (CPU)
Model weights (20/88 layers on GPU)	~8.3 GB	~28 GB
KV cache (34K context)	~0.8 GB	-
Overhead	~0.5 GB	~2 GB
Total	~9.6 GB	~30 GB

Measured performance (4K ctx): 7 t/s prompt, **1.9 t/s generation** (see Performance Reference)

34K Context — Maximum GPU Offload

GPU_LAYERS=30 CONTEXT_SIZE=34816 ./run-nemotron.sh server

Component	VRAM (GPU)	RAM (CPU)
Model weights (30/88 layers on GPU)	~12.4 GB	~24 GB
KV cache (34K context)	~0.8 GB	-
Overhead	~0.5 GB	~2 GB
Total	~13.7 GB	~26 GB

Expected performance (estimate): faster generation than 20 layers, but not yet measured — verify with ./run-nemotron.sh bench.

Quick Reference

Context	GPU_LAYERS	VRAM Used	RAM Used	Speed (gen)	Best For
4K	20	~9 GB	~30 GB	1.9 t/s (measured)	Quick Q&A
16K	20	~9 GB	~30 GB	~1.9 t/s (est.)	Conversations
34K	20	~10 GB	~30 GB	~1.9 t/s (est.)	General use
34K	30	~14 GB	~26 GB	~2-3 t/s (est.)	Faster responses
65K	15	~8 GB	~32 GB	~1.5-2 t/s (est.)	Long documents
128K	10	~8 GB	~34 GB	~1-1.5 t/s (est.)	Maximum context

Speed note: Only the 4K / 20-layer row is measured (2026-08-05). All other rows are estimates — larger contexts add KV-cache overhead, more GPU layers speed things up. Verify any config on your own hardware with ./run-nemotron.sh bench.

Tip: Start with GPU_LAYERS=20 CONTEXT_SIZE=34816 and adjust from there.

Finding Your Optimal GPU Layers

Start with GPU_LAYERS=20
Increase by 5 until you get out of memory
Back off by 5

for layers in 20 25 30 35; do
    echo "Testing GPU_LAYERS=$layers"
    timeout 30 ./run-nemotron.sh server 2>&1 | grep -E "error|loaded|out of memory" || true
    pkill llama-server 2>/dev/null
done

CLI Reference

./run-nemotron.sh server    # Start API server (default)
./run-nemotron.sh chat      # Interactive chat
./run-nemotron.sh bench     # Performance benchmark
./run-nemotron.sh help      # Full help

Environment Variables

GPU_LAYERS=N                # Layers on GPU (0-88, default: 20)
CONTEXT_SIZE=N              # Context window (default: 4096)
PORT=N                      # API port (default: 8081)
HOST=addr                   # Bind address (default: 0.0.0.0)
THREADS=N                   # CPU threads (default: auto)
LLAMA_CPP_DIR=path          # llama.cpp location
MODEL_FILE=path             # Override model path
CUDA_VISIBLE_DEVICES=0,1    # Select GPUs
ALIAS=name                  # Model name alias in API responses

API Usage

Once the server is running, you have a full OpenAI-compatible API.

Chat Completions

curl http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain how transformers work in 3 sentences."}
    ],
    "max_tokens": 500,
    "temperature": 0.7
  }'

Nemotron-3-Super returns both reasoning_content (thinking process) and content (final answer).

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8081/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="nemotron",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a Python function to merge two sorted lists."}
    ],
    max_tokens=1000,
    temperature=0.2
)

print(response.choices[0].message.content)

Available Quantizations

Quant	Size	Shards	Quality	Notes
Q2_K	~52 GB	2	Lowest	Most compact, some quality loss
Q3_K_M	~65 GB	1	Good	Recommended quality/size balance
UD-Q3_K_M	~62 GB	3	Good	Unsloth dynamic quantization
IQ3_XXS	~53 GB	1	Fair	Imatrix calibrated
IQ4_XS	~62 GB	2	Better	Imatrix, higher quality

For 24 GB GPU + 64 GB RAM, Q2_K gives the most headroom. If you can fit Q3_K_M (needs ~65 GB disk and more RAM), it provides noticeably better output quality.

Performance Reference

Measured 2026-08-05 on RTX 5060 Ti (16 GB) + RTX 4060 Ti (8 GB), Q2_K quantization, 20 GPU layers, 4K context.

Measured via the API server (`mlock`, q8_0 KV cache)

Metric	Value
Prompt eval	7.22 t/s (40-token prompt)
Generation	1.88 t/s (200 tokens, ~532 ms/token)

Measured via llama-bench (`mmap`, f16 KV cache)

Test	Value
Prompt processing (pp512)	37.28 ± 0.35 t/s
Generation (tg128)	1.90 ± 0.05 t/s

How to reproduce

GPU_LAYERS=20 CONTEXT_SIZE=4096 ./run-nemotron.sh server

# Send a request and read the server log line:
# "eval time = X ms / 200 tokens (Y tokens per second)"
curl -s http://localhost:8081/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"nemotron","messages":[{"role":"user","content":"Write a 150-word essay."}],"max_tokens":200}'

Note on earlier figures: performance numbers published before 2026-08-05 (e.g. "~~5 t/s generation") were estimates and do not reproduce on this hardware. The values above are the measured ones. CPU-only numbers are still unmeasured estimates (~~1-2 t/s).

Troubleshooting

"cudaMalloc failed: out of memory"

Too many GPU layers for available VRAM.

GPU_LAYERS=10 CONTEXT_SIZE=34816 ./run-nemotron.sh server

"couldn't bind HTTP server socket, port: 8081"

Port already in use (likely Ollama on 8080).

PORT=8090 ./run-nemotron.sh server

Model generates blank lines or garbage

Use the /v1/chat/completions endpoint (handles prompt formatting)
Q2_K quantization may degrade quality — try Q3_K_M or higher

Server starts but `curl` returns "Connection refused"

Model is still loading (~2 minutes for 52 GB). Wait and retry:

watch -n 5 'curl -s http://localhost:8081/health'

License & Attribution

Model weights: The underlying model is governed by the NVIDIA Nemotron Open Model License. See NVIDIA's model card: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.
GGUF quantizations: hosted by bartowski/nvidia_Nemotron-3-Super-120B-A12B-GGUF and unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF. Download weights from these repos — do not repost them here.
Scripts in this repo: MIT license (see LICENSE). They only orchestrate llama.cpp and download weights from the URLs above; they contain no model weights or model code.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for impacte/nemotron-3-super-120b-24gb

Base model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

Quantized

bartowski/nvidia_Nemotron-3-Super-120B-A12B-GGUF

Finetuned

(1)

this model