Instructions to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

SGLang

How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4
```

Gemma 4 E4B Uncensored NVFP4

EAGLE speculative decoding drafter for Gemma 4 26B-A4B Uncensored (TrevorJS) — a 42-layer E4B (EAGLE for Blackwell) model quantized to NVFP4 AWQ using NVIDIA ModelOpt 0.42.0.

Designed for EAGLE-based speculative decoding on NVIDIA DGX Spark (GB10, SM 12.1) and other Blackwell-architecture GPUs.

GitHub Repo — patches, deployment configs, Dockerfile

Model Details

Property	Value
Architecture	Gemma 4 (E4B EAGLE Drafter)
Target Model	TrevorJS/gemma-4-26B-A4B-it-uncensored (26B MoE)
Layers	42 (35 sliding-window + 7 full-attention)
Hidden Size	2560
Attention Heads	8 (2 KV heads), head_dim=256, global_head_dim=512
Sliding Window	512 tokens
Max Context	131,072 tokens
Quantization	NVFP4 AWQ (ModelOpt 0.42.0)
Model Size	9.6 GB
Vocabulary	262,144 tokens

Quick Start

Prerequisites

Target model — Any NVFP4-quantized Gemma 4 26B MoE (e.g., AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4)
This drafter model — Download below
Three vLLM patches — Required for Gemma 4 speculative decoding (see Required Patches)
Pre-built container — ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

1. Download both models

pip install -U huggingface-hub

# Target model (26B MoE)
huggingface-cli download AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/trevorjs-26b

# This drafter model (E4B)
huggingface-cli download AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/e4b-drafter

2. Get the patched vLLM files

Three patches to vLLM 0.19.1 are required for Gemma 4 speculative decoding. Download from the DECKARD 31B GitHub repo:

for f in eagle_patched.py serving_chat_patched.py modelopt_patched.py; do
  curl -LO https://raw.githubusercontent.com/AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4/main/$f
done

3. Launch with Docker Compose

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest
    container_name: vllm-trevorjs-26b-spec
    restart: unless-stopped
    network_mode: host
    volumes:
      - ~/models/trevorjs-26b:/models/target
      - ~/models/e4b-drafter:/models/e4b-drafter
      - ./modelopt_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
      - ./eagle_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py
    environment:
      - VLLM_TEST_FORCE_FP8_MARLIN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_USE_FLASHINFER_MOE_FP4=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - bash
      - -c
      - |
        exec vllm serve /models/target \
          --served-model-name trevorjs-26b \
          --quantization modelopt \
          --dtype auto \
          --kv-cache-dtype fp8 \
          --tensor-parallel-size 1 \
          --max-model-len 131072 \
          --max-num-seqs 4 \
          --gpu-memory-utilization 0.65 \
          --trust-remote-code \
          --host 0.0.0.0 --port 8000 \
          --enable-chunked-prefill \
          --enable-prefix-caching \
          --enable-auto-tool-choice \
          --tool-call-parser gemma4 \
          --reasoning-parser gemma4 \
          --speculative-config '{"method":"draft_model","model":"/models/e4b-drafter","num_speculative_tokens":5,"quantization":"modelopt"}'
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box, and a spec-decode drafter's verify buffers are not counted by the fraction so leave headroom (0.65 here). Discrete-VRAM GPUs can run higher.

Speculative Config Parameters

Parameter	Value	Description
`method`	`draft_model`	EAGLE-based draft model speculative decoding
`model`	`/models/e4b-drafter`	Path to this E4B drafter model
`num_speculative_tokens`	`5`	Number of tokens the drafter proposes per step
`quantization`	`modelopt`	Required — drafter uses NVFP4 ModelOpt format

Required vLLM Patches

Speculative decoding with Gemma 4 requires three patches to vLLM 0.19.1. Without these, the server will crash on startup.

Patch 1: `eagle_patched.py` — Gemma 4 spec decode support

File: vllm/v1/spec_decode/eagle.py

Three fixes are needed:

1a. Remove multimodal guard

vLLM 0.19.1 calls _raise_if_multimodal() which blocks ALL multimodal targets from speculative decoding, even when the drafter is text-only. Remove this call — the downstream code already handles text-only drafters with multimodal targets correctly.

# In initialize() method — REMOVE this line:
# self._raise_if_multimodal()

1b. Add Gemma4 to model whitelist

Gemma 4 uses image_token_id (258880) but NOT image_token_index. The spec decode framework needs an explicit mapping:

# In the model whitelist check, add Gemma4:
if self.get_model_name(target_model) in [
    "Qwen2_5_VLForConditionalGeneration",
    # ... existing models ...
    "Gemma4ForConditionalGeneration",  # ADD THIS
]:
    self.model.config.image_token_index = target_model.config.image_token_id

1c. Multi-group KV cache support

Gemma 4 uses heterogeneous attention: head_dim=256 for sliding-window layers and head_dim=512 for global attention layers. This creates two distinct KV cache groups. The spec decode framework assumes a single group.

Fix validate_same_kv_cache_group to log a warning instead of asserting, and rewrite initialize_attn_backend to key attention groups by (backend_class, kv_cache_group_id) instead of just backend_class, mapping each draft layer to its correct KV cache group.

Patch 2: `serving_chat_patched.py` — Non-streaming reasoning parser

File: vllm/entrypoints/openai/chat_completion/serving.py

Gemma 4's reasoning parser uses <|channel> (token 100) and <channel|> (token 101) delimiters. With skip_special_tokens=True (the default for non-streaming), these are stripped, causing extract_reasoning() to return None — thinking content lands in the content field.

The fix re-decodes from raw token_ids with skip_special_tokens=False when text-based extraction fails:

if reasoning is None and token_ids and hasattr(reasoning_parser, 'start_token_id'):
    token_ids_list = list(token_ids)
    if reasoning_parser.start_token_id in token_ids_list:
        full_text = reasoning_parser.model_tokenizer.decode(
            token_ids_list, skip_special_tokens=False
        )
        reasoning, content = reasoning_parser.extract_reasoning(full_text, request=request)
        if content:
            _tok = reasoning_parser.model_tokenizer
            content = _tok.decode(_tok.encode(content), skip_special_tokens=True)

Patch 3: `modelopt_patched.py` — NVFP4 AWQ support

File: vllm/model_executor/layers/quantization/modelopt.py

Three fixes:

FP8 NaN scrubbing — ModelOpt 0.42.0 produces ~60 FP8 NaN values (0x7F/0xFF) in weight_scale tensors. Scrubs to zero at load time.
NVFP4_AWQ quant_algo — Registers NVFP4_AWQ (upstream only handles NVFP4).
AWQ pre_quant_scale — Loads and applies per-channel pre_quant_scale tensors for AWQ weight redistribution.

Applying the Patches

Mount as volume binds in Docker Compose (shown above), or copy manually:

VLLM_PATH=$(python3 -c "import vllm; print(vllm.__path__[0])")
cp eagle_patched.py    $VLLM_PATH/v1/spec_decode/eagle.py
cp serving_chat_patched.py $VLLM_PATH/entrypoints/openai/chat_completion/serving.py
cp modelopt_patched.py $VLLM_PATH/model_executor/layers/quantization/modelopt.py

Heterogeneous Attention Architecture

This E4B drafter mirrors the Gemma 4 heterogeneous attention design:

35 sliding-window layers — head_dim=256, window of 512 tokens, default RoPE (theta=10000)
7 full-attention layers — head_dim=512, global attention, proportional RoPE (theta=1M, partial_rotary_factor=0.25)

This creates two distinct KV cache groups within the drafter, handled by the multi-group KV cache fix in eagle_patched.py.

Cross-Model Drafter Compatibility

This drafter was derived from the TrevorJS uncensored fine-tune. It can also be used with other Gemma 4 targets that share the same vocabulary (262K tokens):

Target Model	Expected Compatibility
TrevorJS 26B Uncensored	Best — same base model
DECKARD 31B Dense	Good — same vocab, different fine-tune
SuperGemma4 26B MoE	Good — same vocab, different abliteration
Official Gemma 4 26B	Moderate — base model, no uncensoring

Acceptance rate will be highest with the matching base model and lower with mismatched fine-tunes, but all combinations will function correctly.

Related Models

Model	Type	Size	Link
TrevorJS 26B MoE NVFP4 (target)	MoE NVFP4	16 GB	HuggingFace
DECKARD 31B AWQ_FULL	Dense NVFP4	20.5 GB	HuggingFace \| GitHub
DECKARD E4B Drafter	EAGLE NVFP4	9.6 GB	HuggingFace \| GitHub
SuperGemma4 26B MoE	MoE NVFP4	~14 GB	HuggingFace \| GitHub
vLLM AWQ Container	Docker	—	GHCR

Hardware Requirements

Target + Drafter combined: ~26 GB (16 GB target + 9.6 GB drafter)
Recommended: NVIDIA DGX Spark (128 GB unified memory) or any GPU with >= 40 GB VRAM
Required: Blackwell architecture (SM 10.0+) for native FP4

License

This model inherits the Gemma license from Google.

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

₿ Bitcoin (BTC)

_{bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4}

Ξ Ethereum (ETH)

_{0x1512667F6D61454ad531d2E45C0a5d1fd82D0500}

◎ Solana (SOL)

_{DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t}

ⓜ Monero (XMR)

_{836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd}

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.

Downloads last month: 524

Safetensors

Model size

6B params

Tensor type

BF16

F8_E4M3

Model tree for AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Finetuned

TrevorJS/gemma-4-26B-A4B-it-uncensored

Quantized

(20)

this model

Collection including AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4

Gemma 4 — DECKARD HERETIC, Multimodal & Speculators

Collection

Gemma 4 abliterated/quantized — DECKARD HERETIC 31B, SuperGemma4-26B multimodal, 26B-A4B MoE, plus EAGLE3/DFlash drafters. • 14 items • Updated Jul 2 • 8