Dipankar Sarkar's picture
🏗️ Building on HF

Dipankar Sarkar PRO

dipankarsarkar

AI & ML interests

Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.

Recent Activity

repliedto nightmedia's post about 1 hour ago
Qwen3.8-27B metrics It's hard to track all model cards where I post these, so I figured people would get more value out of seeing these in the open. ```brainwaves quant arc arc/e boolq hswag obkqa piqa wino mxfp8 0.591,0.782,0.896,0.746,0.448,0.801,0.711 q8-hi 0.602,0.779,0.896,0.747,0.446,0.793,0.703 q6-hi 0.602,0.775,0.895,0.748,0.448,0.795,0.710 q4-hi 0.604,0.780,0.898,0.744,0.454,0.795,0.708 mxfp4 0.581,0.771,0.889,0.738,0.442,0.798,0.713 1M mxfp8 0.590,0.787,0.897,0.744,0.446,0.801,0.709 Quant Perplexity Peak Memory Tokens/sec mxfp8 6.090 ± 0.054 34.74 GB 138 mxfp4 5.952 ± 0.051 21.30 GB 148 ``` The performance is as measured on a M4 MBP 128GB, speed may vary depending on your platform. These are all instruct metrics, generated by including this line in the jinja template: ``` {%- set enable_thinking = false %} ``` Then run the test suite to generate the metrics: ``` mlx_lm.evaluate --model MODEL --tasks winogrande boolq arc_challenge arc_easy hellaswag openbookqa piqa ``` This will generate the file: ``` eval_MODEL_0.4.9_winogrande_boolq_arc_challenge_arc_easy_hellaswag_openbookqa_piqa ``` This is a JSON containing all gathered metrics; for example the q4-hi: ``` "arc_challenge": { "alias": "arc_challenge", "acc,none": 0.5819112627986348, "acc_stderr,none": 0.014413988396996116, "acc_norm,none": 0.6040955631399317, "acc_norm_stderr,none": 0.01429122839353657 }, ``` I use the value of acc_norm for metrics, rounded to 3 decimals. As I get more quants tested, I will add them here. A complete test run for a single quant takes 7-9 hours depending on quant size, 10-12 hours for BF16 depending on perplexity: this is why you see on my model cards that I usually post the first three, that only take 2-3 hours :) -G
repliedto ManniX-ITA's post about 1 hour ago
🚀 Qwen3.6-27B-A3B-CoderX — the long-horizon sibling to A3B-Coder. Same 256→184 expert budget (~35B→27B, A3B active), different selection: our saliency map picks the keep-set, a REAP-style per-layer floor (p=24) protects the tail, and the 72 evicted experts per layer are folded DERN-style into the survivors instead of discarded. No fine-tuning, no distillation. 📊 Q6_K + imatrix, llama.cpp b9700, greedy, one pinned geometry per bench, same host — CoderX / A3B-Coder / unpruned 256e: ⚡ LiveCodeBench v6 (77q, 24k think) — 72.73 / 61.04 / 61.04 → +11.7pp over both ✅ HumanEval+ (164) — 96.95 / 95.12 / 93.90 → best of the three 🤝 MultiPL-E-100 (rs+java+js) — 88.67 / 89.00 / 91.00 ⚠️ Read that last row honestly: a same-basis repeat of MultiPL-E moved 1.0pp on batch-scheduling nondeterminism alone. The 0.33pp CoderX↔Coder gap is INSIDE that band — a tie. The 2.33pp gap to the base is outside it and real. CoderX takes Rust (0.85 vs 0.81), gives up JS (0.92 vs 0.96). 🎯 Ships top-8, and that was measured, not assumed: MBPP-full 78.4 / 79.0 at top-8 vs 73.2 / 73.0 at top-10. Opposite call from A3B-Coder, which bakes top-10. 🧠 It thinks long — LCB median completion ~15.8k tokens vs ~2.2k for Coder. The length is where the win comes from; give it context headroom rather than clamping it. 🔬 Not measured yet: the canonical 9-bench. GPQA / MATH-500 / IFEval are deliberately NOT quoted — treat the non-code profile as unknown. Coder remains the one with a published 9-bench table. 📦 bf16 safetensors (text-only) · 19 GGUF tiers, EVERY K/I-quant imatrix-built and verified by reading quantize.imatrix.* back out of each uploaded file · Ollama 39 tags (19 text + 19 vision-<tier> + :latest). MTP in every tier — draft_num_predict 3 gives 190→252 tok/s (+33%) on an RTX 5080. 🔗 https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-CoderX 🔗 https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-CoderX-MTP-GGUF 🔗 https://ollama.com/mannix/qwen3.6-27b-a3b-coderx
liked a model about 1 hour ago
ManniX-ITA/Qwen3.6-27B-A3B-CoderX
View all activity

Organizations

Skelf Research's profile picture Neul Labs's profile picture Cognisoc's profile picture Incredlabs's profile picture