Instructions to use AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant")
model = AutoModelForMultimodalLM.from_pretrained("AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant

SGLang

How to use AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant with Docker Model Runner:
```
docker model run hf.co/AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant
```

Gemma 4 31B DECKARD HERETIC Uncensored — NVFP4 SVDQuant

SVDQuant-quantized version of DavidAU/gemma-4-31B-it-The-DECKARD-HERETIC-UNCENSORED-Thinking. Quantized using NVIDIA ModelOpt 0.42.0 with SVDQuant (SVD decomposition + NVFP4) for maximum quality at 4-bit precision. Calibrated natively on NVIDIA B200 (Blackwell SM 12.0) for hardware-accurate FP4 scale factors.

See also: AWQ_FULL variant — same model quantized with AWQ_FULL instead of SVDQuant.

What is SVDQuant?

SVDQuant uses Singular Value Decomposition to separate weight matrices into two components before quantization:

Outlier channels — high-magnitude weight channels that cause large quantization error are extracted into a low-rank BF16 residual matrix
Cleaned weights — the remaining weights (with outliers removed) are quantized to NVFP4 (E2M1) with dramatically reduced quantization error

This produces higher quality than standard AWQ at the cost of a slightly larger model size (~20.9 GB vs ~20.5 GB) due to the low-rank residual matrices stored in BF16.

Original Weight Matrix W (BF16)
    |
    v
[SVD Decomposition]
    |
    ├── Low-rank residual R (BF16, rank=32) — captures outlier channels
    └── Cleaned weights W' = W - R
            |
            v
        [NVFP4 Quantization] — much lower error without outliers
            |
            v
        W'_quant (FP4 E2M1)

Inference: output = W'_quant @ x + R @ x

Model Details

Property	Value
Base Model	Gemma 4 31B-it DECKARD HERETIC
Architecture	Gemma 4 (Dense, 31B parameters)
Layers	60
Max Context	131,072 tokens
Hidden Size	5376
Intermediate Size	21,504
Attention Heads	32 (16 KV heads)
Vocabulary	262,144 tokens
Quantization	NVFP4 SVDQuant (ModelOpt format)
SVD Low-Rank	32
Model Size	~20.9 GB
Calibration Hardware	NVIDIA B200 (native Blackwell FP4)

Quantization Details

Gemma 4 31B DECKARD HERETIC (BF16, ~62 GB)
    |
    v
[NVFP4 SVDQuant on B200]
    - ModelOpt 0.42.0 with NVFP4_SVDQUANT_DEFAULT_CFG
    - Low-rank = 32 (BF16 residual matrices)
    - 2048 calibration samples (CNN DailyMail)
    - Native Blackwell FP4 hardware calibration (SM 12.0)
    - Excluded: vision tower, embed_vision, multi_modal_projector
    - Quantization time: ~69 minutes on B200
    |
    v
Gemma-4-31B-DECKARD-HERETIC-NVFP4-SVDQuant (~20.9 GB)

AWQ_FULL vs SVDQuant Comparison

Metric	AWQ_FULL	SVDQuant
Technique	Channel scaling + clipping optimization	SVD decomposition + low-rank residual
Model Size	~20.5 GB	~20.9 GB
Quant Time	~75 min	~69 min
Quality	Excellent	Potentially higher (preserves outliers in BF16)
Speed	Slightly faster (smaller)	Slightly slower (low-rank matmul overhead)
Best For	Maximum throughput	Maximum quality

Deployment

vLLM

vllm serve /path/to/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant \
  --served-model-name deckard-svdquant \
  --quantization modelopt \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.70 \
  --trust-remote-code \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes. Discrete-VRAM GPUs can run higher.

Docker Compose

services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    ports:
      - "8000:8000"
    volumes:
      - ./models:/models
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_ATTENTION_BACKEND=FLASHINFER
    command: >
      --model /models/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4-SVDQuant
      --served-model-name deckard-svdquant
      --quantization modelopt
      --dtype auto
      --kv-cache-dtype fp8
      --max-model-len 65536
      --max-num-seqs 8
      --gpu-memory-utilization 0.70
      --trust-remote-code
      --enable-chunked-prefill
      --enable-prefix-caching
      --enable-auto-tool-choice
      --tool-call-parser gemma4
      --reasoning-parser gemma4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

DGX Spark Performance Estimates

Configuration	Estimated tok/s
BF16 (no quantization)	~3-5
NVFP4 AWQ_FULL	~12-14
NVFP4 SVDQuant	~10-13

Key Deployment Flags

Flag	Purpose
`--quantization modelopt`	Required — tells vLLM to use ModelOpt NVFP4 format
`--kv-cache-dtype fp8`	Reduces KV cache memory by 2x for longer contexts
`--reasoning-parser gemma4`	Extracts `<think>` blocks for thinking/reasoning display
`--tool-call-parser gemma4`	Enables native function calling
`--enable-chunked-prefill`	Processes long prompts in chunks to avoid OOM
`--enable-prefix-caching`	Caches common prompt prefixes for faster responses

Related Models

GitHub repo: AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4 — deployment docs, Docker Compose, cross-model comparison
AWQ_FULL variant: AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4 — same base model, AWQ_FULL quantization
Gemma 4 MoE NVFP4: AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 — MoE variant, faster throughput
Base model: DavidAU/gemma-4-31B-it-The-DECKARD-HERETIC-UNCENSORED-Thinking

Advanced Techniques

Native B200 Calibration

Quantized on NVIDIA B200 with native FP4 hardware instructions (SM 12.0). The SVDQuant calibration measures actual FP4 rounding behavior on real hardware rather than simulating it, producing more accurate scale factors and SVD decomposition decisions than calibrating on non-FP4 hardware.

SVD Low-Rank Selection

The default low-rank of 32 was used, which provides an excellent balance between quality preservation and model size overhead. Each weight matrix has a 32-column BF16 residual that captures the most important outlier channels, keeping them at full precision while the remaining weights are safely quantized to FP4.

License

This model inherits the Gemma license from the base model.

Legal Disclaimer

THIS MODEL IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. The authors make no representations regarding accuracy, reliability, or fitness for any purpose. Use at your own risk. By downloading or using this model, you agree that the authors shall not be liable for any claims, damages, or losses arising from its use.