OhanaTract/gpt-oss-20b-q40

GPT-OSS 20B Q40 for Ohana/tract

This repository contains the preferred Ohana/tract NNEF export of openai/gpt-oss-20b, published as the short Ohana model id gpt-oss-20b_q40.

This is not a Transformers/safetensors checkpoint. It is a tract NNEF artifact intended for the Ohana local runtime and tract's GPT-OSS Q40 Metal path.

Contents

export_config.json
params.json
model/config.json
model/model.nnef.tgz
tokenizer/tokenizer_config.json
tokenizer/tokenizer.json
tokenizer/chat_template.jinja
ohana_manifest.json

Format

• Base model: openai/gpt-oss-20b
• License: Apache-2.0, inherited from the upstream model
• Runtime format: tract NNEF archive
• Ohana model id: gpt-oss-20b_q40
• Architecture: GPT-OSS MoE, 24 layers, 32 local experts, top-4 routing
• Context metadata: 131072 max positions, GPT-OSS sliding-window layers
• Embeddings: dense f16
• lm_head: tract Q40/Q4_0 (updated 2026-08-04; saves ~0.8 GB and ~5 ms per decoded token versus the previous f16 lm_head)
• MoE experts: tract Q40/Q4_0 linear-layout expert weights
• Tool calling: enabled (capabilities.tools = true)
• Export date: 2026-07-28, republished as the preferred artifact 2026-08-04

The export keeps GPT-OSS's per-layer sliding-window mask structure. This is important: earlier broken exports used one shared causal mask for all layers and degenerated on longer prompts.

Runtime requirement

Requires tract at or after rev 0f0e469db on feat/moe-ffn-operator (sonos/tract PR #2084 branch). Older tract Metal runtimes have a command-buffer synchronization defect that makes ANY GPT-OSS Q40 artifact degenerate to constant <|endofprompt|> past roughly 1024 tokens of context; the fix (MetalRoutedQ40MatMul::sync_after_dispatch, default on) ships in that rev together with a MoE routing-kernel occupancy fix.

Local Usage

In Ohana, this artifact is expected to appear as:

gpt-oss-20b_q40

The repository layout mirrors Ohana's local registry layout, so an Ohana/tract loader should consume the repository root as the model directory.

Validation

Validated locally in Ohana and tract on Apple Metal (M-series, 48 GB):

Check	Result
300-token raw generation	coherent
2800-token full MCP tool-schema prompt, Metal and CPU	coherent
Harmony tool calls through Ohana /api/chat	well-formed
Ohana two-turn Jensen-Shannon prompt	coherent, mathematically correct
WikiText-2 slice perplexity, relative harness	169.76
bench_decode, 74-token context, Metal	28.1 tok/s
bench_decode, 2800-token context, Metal	7.8 tok/s

The perplexity number is useful only for comparing the local export variants in the same harness (the previous f16-lm_head artifact scored 165.44; the lm_head Q40 delta was graded as holding quality). GPT-OSS is Harmony-format reasoning-tuned, so raw untemplated WikiText is far out of distribution.

Caveats

• This artifact is for Ohana/tract, not direct transformers loading.
• embed_tokens Q40 was tested and rejected because it degenerated despite shrinking the model.
• Decode throughput falls with context length on the current runtime (the KV cache concat is O(T^2) over a decode); an in-place KV cache is planned in tract.

Provenance

Exported from the local repaired/calibrated GPT-OSS 20B checkpoint:

quantized-pytorch-calib512-fullq40-lmheadq40-20260728

See export_config.json and ohana_manifest.json for the exact local export metadata recorded at publication time.