KyleHessling1/Qwopus3.6-27B-Fusion-BF16

▶ PLAY ORBITAL — a game this model built, live in your browser ◀

⬛ Full-precision BF16 weights (safetensors). This repo hosts the unquantized bf16 model in Hugging Face safetensors format (~55.6 GB, 27 shards) — the canonical source you can load in Transformers or use to roll your own quants (GGUF via llama.cpp, or GPTQ / AWQ / EXL2 / bitsandbytes). Just want a ready-to-run file? Pre-made GGUF quants (Q3_K_M, Q4_K_M, Q5_K_M, Q6_K) live in the companion repo → KyleHessling1/Qwopus3.6-27B-Fusion-GGUF.

Research preview. This model attempts to combine the reasoning capability of Qwopus3.6-27B-v2 with the code-execution capability of Qwopus3.6-27B-Coder — in a single 27B model, without significant loss to either skill. In practice it behaves like a production model with both capabilities fused, but it has not yet been through a full rigorous evaluation. If you find issues, please reach out on X — @kylehessling1.

• Format: safetensors, bf16 full precision (~55.6 GB, 27 shards) — load with 🤗 Transformers (Qwen3_5ForConditionalGeneration). For single-GPU inference, use a quantized GGUF from the companion repo.
• Architecture: qwen35 hybrid (linear-attention + periodic full-attention), 27B params, 64 layers.
• Native context: 262,144 tokens. Verified clean (needle + termination) to 60K.
• Modes: thinking (reasoning) on by default; strong agentic/coding behavior.
• Vision: these safetensors retain the base model's vision encoder (Qwen3_5ForConditionalGeneration, model.visual.*), which the GGUF quants drop. ⚠️ This release was evaluated on text/code only — the multimodal path is inherited and untested here.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ TL;DR

Base / anchor	Qwen/Qwen3.6-27B (both parents descend from it)
Parent A — reasoning	Qwopus3.6-27B-v2 — donates the merge base + embeddings/head/norms/MTP
Parent B — code	Qwopus3.6-27B-Coder — delta source, injected with depth-increasing weight
Merge math	W(L) = V2 + α(L)·(Coder − V2), α linear ramp 0.12 (early) → 0.48 (late)
Frozen (copied from A)	embed_tokens, lm_head, all norm, mtp/NextN, any vision tensors

Headline results (Q4_K_M, thinking-on unless noted):

Capability	Score	Stability	Score
HumanEval	94.5%	Termination @ temp 0.2 & 0.9	4/4 clean
MBPP	87.9%	Temp sweep 0.5–0.9	10/10 clean
GSM8K	95.0%	Seed variance (HumanEval)	std 0.0
MATH-500	62–64.7%	Long-context needle+term @ 60K	perfect
IFEval (strict)	82.8%	SWE-bench Verified (astropy, 0:15)	7/15 (47%)

Against the alternatives on the same astropy slice: Fusion 7/15 > equal-weight DARE-TIES 5/15 > DUS-38B frankenmerge 3/15.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Where it shines — the one-stop creative programming agent

Best used as a single, closed-loop agentic creative programmer. In a build → run → self-fix loop, Fusion is at its strongest as one agent that plans, writes, and debugs on its own — the reasoning⊕coding balance lets it close the loop by itself (see the ORBITAL / FLUX demos below, where it root-caused and fixed its own bugs from Playwright error reports).

This is a different niche than its parents. The Coder model is excellent for coding under active steering, or as a solid orchestration model driving other tools/agents. Fusion, by contrast, is the ideal one-stop creative builder — you hand it a goal and it carries the whole creative-programming loop end-to-end.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Why I chose to make this merge

Two finetunes of the same base diverge into complementary specialists: one gets better at multi-step reasoning, the other at writing code — but each pays for it. In particular the coding finetune becomes unstable in thinking mode (it loops / degrades at low temperature). The goal was a single model that keeps the reasoner's stability and the coder's skill.

My naive approaches, one of them eroneously proposed by Claude Fable, average the two, or run an equal-weight DARE-TIES merge from the common base. These alternatives underperformed and the geometry explains why.

▌ The lineage

Because the coder model descended from the v2 version, the coders additions can be evaluated against the v2 by analyzing the delta. Projecting the coding delta onto the reasoning delta reveals:

proj(ΔCoder · ΔV2 / |ΔV2|²)   ≈ 0.4  in early layers   →   ≈ 1.0  in late layers

That ramp to 1.0 means late layers of the coder already contain V2's full reasoning delta. And:

• The coding delta is ~1.8× the magnitude of the reasoning delta.
• In early layers the coding delta is anti-aligned with reasoning (cos < 0) — the coder partially reverted V2's early-layer reasoning/control representations. That anti-alignment is the measured source of the coder's thinking-mode instability.

An equal-weight merge (both task vectors added from the base) therefore commits two errors:

1. It double-counts V2's reasoning delta (the coder already contains it), and
2. It imports the full early-layer destabilizing coding delta at 100% strength.

▌ The fix

Because the coder sits "downstream" of V2, the correct anchor is V2 itself, not the shared base — this avoids the double-count. Then, instead of a constant fraction of the coding delta everywhere, inject it with a depth-increasing schedule:

• Early layers (α ≈ 0.12): inject little coding delta → protect V2's reasoning/control reps exactly where the coder damaged them.
• Late layers (α ≈ 0.48): inject more coding delta → add coding skill where the two parents already agree (proj → 1.0) and it's safe.

The shape of the α ramp is dictated by the geometry, not hand-tuning. Embeddings, LM head, norms, and the MTP head are identical across all three models (delta ≈ 0), so they're copied verbatim from V2 rather than merged.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ How it was built

Merge. The parents use Qwen's hybrid attention schedule (most layers linear/DeltaNet; every 4th layer full softmax attention). Off-the-shelf mergekit cannot merge this architecture — it templates linear_attn.* onto every layer and errors on the full-attention layers. So the merge is a custom streaming task-vector routine operating directly on the (identical) tensor keys, immune to arch templating. The full, reproducible script ships in this repo as merge_layerweighted.py:

def alpha(L, a_early, a_late):
    return a_early + (a_late - a_early) * (L / (NLAYERS - 1))   # smooth linear ramp by depth

# per decoder tensor at layer L:
out = (v2 + alpha(L, 0.12, 0.48) * (coder - v2)).to(torch.bfloat16)
# embeddings / lm_head / norms / mtp / vision: copied from V2 verbatim

This repo = the bf16 merge output (safetensors, 27 shards, 1199 tensors). It was then converted to GGUF and quantized (Q3_K_M…Q6_K) with llama.cpp for the companion quant repo — note the GGUF path drops the vision encoder, which these safetensors keep.

Evaluate. Every candidate was gated at both temp 0.2 (a termination stress test) and temp 0.9 (the finetune operating point) — "clean only at high temp" is riding sampling entropy, not stability. This model is clean at both.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Demonstrations — driving a local agentic coding loop

Every artifact below was generated by this model running as the backend of an agentic coding loop (OpenCode / Claude Code), building single-file HTML/JS apps end-to-end, then verified with headless Playwright. What stood out isn't just that it writes polished games — it's that when the Playwright harness handed back error reports, the same model root-caused and fixed its own bugs with precise, correct diagnoses (excerpts below).

🚀 ORBITAL — physics space-arcade (thinking-on)

Setup: "Build a complete, physics-driven, visually stunning arcade space game as a single self-contained index.html — gravity is the weapon." Outcome: built and runs cleanly, zero console errors, real 2-body gravity, particle trails, wave HUD, and it even self-balance-tuned its own difficulty. 1,150 lines.

▶ PLAY IT LIVE → huggingface.co/spaces/KyleHessling1/ORBITAL ◀

(runs entirely in your browser — the exact single file this model wrote)

Title screen	Live gameplay (Wave 1)

The same loop also produced three more complete, self-contained games — PRISM DEFENSE (neon tower defense, 1,074 lines), NEON SERPENT (synthwave Snake with smooth interpolated movement, 657 lines), and STARLIGHT SURVIVORS (survivors-style auto-shooter, 555 lines). Each was built, Playwright-tested, and where the harness surfaced a bug the model root-caused and fixed it itself (e.g. a tower-targeting bestDist = Infinity that should have been -Infinity; three weapon upgrades missing a fireTimer init that made fireTimer -= dt evaluate to NaN). The impressive part is the quality of the model's own debugging inside a build → error report → self-fix loop, not flawless one-shot output.

📋 FLUX — a full Trello-style Kanban dashboard (the strongest capability demo)

The most demanding build: a complete single-file Kanban app (~3,373 lines, no backend, localStorage persistence), assembled by the model over five steered phases, each a fresh session that read a spec + the current file and extended it — Playwright-verified between phases.

What works (all Playwright-driven against real interactions): boards / lists / cards CRUD, drag-and-drop cross-list moves with persistence, a markdown card modal with live preview, labels / due dates / checklists, keyboard shortcuts, and live search/filter — below, typing drag narrows to the matching card in real time:

Analysis. Over five phases the model needed only two corrections, each caught by Playwright and fixed by the model: an undeclared dragMoveOffsetX (1-line) and a search UI that was built but not yet wired to the filter logic.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Architecture details

Field	Value
general.architecture	qwen35 (hybrid linear + full attention)
Parameters	~27B (size label 27B)
Layers (block_count)	64
Hidden size	5120
FFN size	17408
Attention heads / KV heads	24 / 4 (GQA)
Full-attention interval	every 4th layer (full_attention_interval = 4); the rest are linear/DeltaNet
Linear-attn (SSM)	conv kernel 4, state 128, groups 16, inner size 6144, time-step rank 48
Key/Value length	256
Vocab	248,320
Native context	262,144
RoPE	freq base 1e7, mRoPE sections [11, 11, 10], partial rotary (dim 64)
MTP / NextN head	present (mtp.*, copied from V2)
Vision encoder	present (model.visual.*, inherited from base) — untested in this release
Precision	bf16 safetensors — 27 shards, 1199 tensors, unquantized

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Usage

These are the full-precision bf16 weights. On a single 24–32 GB GPU you'll want a quantized GGUF from the companion repo instead — the bf16 model is ~55 GB and needs multi-GPU or CPU offload.

▌ 🤗 Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "KyleHessling1/Qwopus3.6-27B-Fusion-BF16"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto",  # ~55GB across your devices
)
msgs = [{"role": "user", "content": "Write a single-file HTML gravity game."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=2048, temperature=0.9, top_p=0.9)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

Recommended sampling: temperature 0.85–1.0 top_p 0.9. Thinking mode is on by default.

▌ Make your own quants

This bf16 checkpoint is the source you quantize from:

# GGUF (llama.cpp): convert then quantize to any level
python convert_hf_to_gguf.py ./Qwopus3.6-27B-Fusion-BF16 --outtype bf16 --outfile fusion-bf16.gguf
llama-quantize fusion-bf16.gguf fusion-Q4_K_M.gguf Q4_K_M
# or feed the safetensors directly to GPTQ / AWQ / EXL2 / bitsandbytes

Temperature — run it hot. Best capability lands at temp 0.85–1.0 (the finetune's natural operating point, and consistent with the agentic/SWE results). Then the surprising part: the merge itself appears to have repaired much of the low-temperature instability of the parent finetunes — Fusion stays clean at temps where the bases loop and degrade (4/4 termination at both 0.2 and 0.9, 10/10 across 0.5–0.9). So, unlike its parents, you can run it cooler for more determinism if you want, without the looping. Exactly why merging improves temperature stability isn't fully understood, but it's consistent across every test we ran. High temperature still yields the best results. Thinking mode is on by default and is where the model is strongest.

▌ As a coding backend (Claude Code / OpenAI-compatible clients)

Served behind an OpenAI-compatible endpoint (vLLM / TGI for these bf16 weights, or llama-server for a GGUF quant), this model has driven multi-file app builds end-to-end. If you point Claude Code at it, use a proxy that does native Anthropic↔OpenAI tool translation (text-encoded tools stall the agent).

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Evaluation — full results

Benchmarks were run on the Q4_K_M quant (thinking-on unless noted); these bf16 safetensors are the full-precision source those results derive from and should meet or exceed them. All figures are text/code — the inherited vision path was not evaluated.

Capability: HumanEval 94.5% · MBPP 87.9% · GSM8K 95.0% · MATH-500 62% (64.7% subset) · IFEval-strict 82.8%.

Stability (the axis that separated this project's winners from losers):

Test	Result
Termination @ temp 0.2 / 0.9	4/4 / 4/4 clean
Temperature sweep 0.5–0.9	10/10 clean (9/10 at 0.1 & 1.0)
Seed variance — HumanEval / MATH	std 0.0 / 2.36
Long-context needle + termination	perfect to 60K tokens, zero looping

Agentic (SWE-bench Verified, astropy slice, thinking-on): Fusion 7/15 (47%) > equal-weight DARE-TIES 5/15 > DUS-38B frankenmerge 3/15.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Limitations

• Not yet rigorously evaluated. No large-scale safety alignment / RLHF / red-teaming. Inherits biases and limits of the Qwen base and both parent finetunes.
• The astropy +2 (7/15 vs 5/15) is within noise on n=15 — the geometry approach is validated as sound and directionally better, not a large agentic jump.
• On the hardest agentic instances the model can over-deliberate and hit the generation cap without submitting; capping max_tokens (~8K) mitigates this.
• Validated primarily at Q4_K_M; other quants and this bf16 source should behave the same or better, but aren't separately benchmarked.
• Vision is inherited, not tuned or tested. The model.visual.* encoder rides along from the base; the merge only touches the language tower, and no multimodal evaluation was done. Treat image input as experimental.

Found a problem? → please reach out on X: @kylehessling1.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Method credit & acknowledgments

The path to this model was not a straight line, and the credit is honest about that:

• Jackrong — base-model builder, collaborator, and good friend. This fusion stands entirely on his work: he built both parent finetunes (Qwopus3.6-27B-v2 and Qwopus3.6-27B-Coder) that this merge fuses. Beyond the models, his counsel and contributions throughout were priceless — this project simply does not exist without him. Go follow his work → huggingface.co/Jackrong.
• The initial naive approaches — a 38B depth-upscaled (DUS) frankenmerge and an equal-weight DARE-TIES merge from the shared base — were proposed by Fable. Both underdelivered. In hindsight, those plans conspicuously omitted the one nuance that turned out to be decisive: the lineage relationship between the parents (that the coder is a descendant of V2). Without that nuance, an equal-weight merge is essentially guaranteed to double-count V2's delta and import the destabilizing early-layer coding delta at full strength — which is exactly what went wrong.
• Grok 4.5 was instrumental in the final, working merge. Its analysis surfaced the parent→descendant lineage from the weight geometry and motivated the layer-weighted delta-scale recipe (protect early reasoning, inject coding late) that this model is built on.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Lineage / provenance

Qwen/Qwen3.6-27B  (shared base)
├── Qwopus3.6-27B-v2      (reasoning finetune)  ──┐
└── Qwopus3.6-27B-Coder   (code finetune,        │  ← descends from V2 (proj→1.0 in late layers)
     itself downstream of V2)                    │
                                                  ▼
        Qwopus3.6-27B-Fusion = V2 + α(L)·(Coder − V2),  α: 0.12 → 0.48 by depth

◆ License

A merge of derivatives of Qwen/Qwen3.6-27B; usage is governed by the Qwen base model license. The parent finetunes are personal research models. No third-party training data is redistributed here — only merged weights and the merge script. Comply with the upstream Qwen license.

◆ Reproduction

python merge_layerweighted.py --out ./Qwopus3.6-27B-Fusion --a-early 0.12 --a-late 0.48
# then convert + quantize with llama.cpp: convert_hf_to_gguf.py  →  llama-quantize ... Q4_K_M

If you build on the method (geometry-grounded, layer-weighted task-vector merging of a parent/descendant finetune pair), a mention of "FusionThink layer-weighted delta-scale merge" is appreciated.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Built and evaluated locally on a single RTX 5090. Model produced by Kyle Hessling (@kylehessling1).