gemma4-ushistory-sft-parityfix-GGUF

GGUF quantizations of ambrosfitz/gemma4-ushistory-sft-parityfix (the merged 16-bit parity-fix checkpoint), for local inference via llama.cpp / LM Studio. Two quants are provided:

File	Size	Notes
gemma4-ushistory-parityfix-Q4_K_M.gguf	7.38 GB	Verified clean 8/8 on structured-output smoke test
gemma4-ushistory-parityfix-Q5_K_M.gguf	8.55 GB	Verified clean 8/8 on structured-output smoke test

Both fit comfortably on a 12GB+ GPU with full offload and room for a 4096-token KV cache.

Required generation recipe

This model was trained and evaluated with a specific chat template and stop-token setup. Getting either wrong produces malformed or garbage output — this was confirmed directly during quantization verification (see below).

• Chat template: gemma-4, non-thinking mode. The correct jinja template is embedded in this GGUF's own metadata (tokenizer.chat_template) — do not override it with a runtime's built-in "gemma" preset. llama.cpp ships a built-in template named gemma that is for the classic Gemma 1/2/3 format (<start_of_turn>/<end_of_turn>) and is incompatible with this checkpoint's Gemma-4-unified format. Passing --chat-template gemma to llama-server silently overrides the correct embedded template and produces completely unrelated, garbage output (confirmed: 0/8 parseable outputs, all hit the token budget without stopping). Leave the chat template on its default (LM Studio: don't override it; llama.cpp: don't pass --chat-template) so the runtime uses the template embedded in this file.
• Stop token: <turn|> (vocab id 106). This is not encoded as an EOS/EOG token in the GGUF metadata — the only marked EOS token is <eos> (id 1), and llama.cpp's own automatic EOG detection at load time does not pick up <turn|> either. This mirrors the original HF Transformers inference recipe, which also had to pass it explicitly (eos_token_id=[tokenizer.eos_token_id, id_of("<turn|>")]). You must set <turn|> as an explicit stop string in your runtime (LM Studio: add it under the model's stop-string/sequence setting) — without this, generation will run past the intended turn boundary and repeat into garbage on longer outputs.

Verification

Both quants were served on-pod via llama-server and smoke-tested with 8 items spanning all 6 Bloom's-taxonomy levels (remember/understand/apply/analyze as multiple-choice, analyze/evaluate/create as open-ended — matching this model's trained item-type schema), using the project's exact prompt construction and an explicit <turn|> stop string.

Result: 8/8 parsed, 8/8 structurally valid, 8/8 stopped cleanly — on both quants. No malformed or garbage output on either quant once the chat-template issue above was fixed. Full per-item results and methodology in this repo's source project (handoff_A_verification.md).

A small parity sanity check (word-count spread across MCQ options, this project's max-min length-parity metric) on the n=5 MCQ smoke items per quant showed the same pattern already documented for the un-quantized model: understand-level items carry more residual parity failures than remember/apply. At this sample size this is noise-level, and there is no evidence quantization itself degrades parity beyond what the fp16 checkpoint already exhibits.

Recommended local config (tested target: RTX 4070, 12GB)

• Full GPU offload (-ngl 99 or LM Studio's "GPU offload: max")
• Context length: 4096 (matches training: passage + up to 1024 generated tokens)
• Chat template: leave on default / embedded (see above)
• Stop string: add <turn|> explicitly (see above)
• temperature: 0 for deterministic output, matching how this model was trained and evaluated throughout this project

Lineage

• Base: Gemma 4 (unsloth/gemma-4-12b)
• SFT: ambrosfitz/gemma4-ushistory-sft → parity-fix continued SFT → ambrosfitz/gemma4-ushistory-sft-parityfix (merged 16-bit) → this GGUF quantization