kdirgul/Mamba3-177M-GQA-Hybrid_LAMBA_V1.0

LAMBA V1.0 — Mamba-3 + GQA Hybrid (EN + TR)

LAMBA V1.0 is a ~177M-parameter language model trained from scratch on a Mamba-3 (SISO) + Grouped-Query Attention hybrid architecture, for English (primary) and Turkish (secondary). It is a deliberate, honest first step: small enough to train on free/affordable compute (Google Colab A100), built end-to-end — tokenizer, data, pretraining, instruction tuning, and retrieval-augmented tuning — with no warm-start from any existing model.

TR: LAMBA V1.0, sıfırdan eğitilmiş ~177M parametreli, Mamba-3 + GQA hibrit mimarili, İngilizce (birincil) ve Türkçe (ikincil) bir dil modelidir. Hiçbir hazır modelden başlatılmadı; tokenizer'dan eğitime tüm hat baştan kuruldu.

Update (v1.1): now runs on CPU — a pure-PyTorch port (lamba_cpu.py, no Triton/CUDA) with a decode cache. Run anywhere with --device cpu; see How to run.

---

⚠️ Read this first — what LAMBA is and isn't

LAMBA V1.0 is a 177M model. At this scale it has fluent local language ability and real commonsense signal, but limited factual knowledge and weak abstract reasoning. It will hallucinate facts if asked open-ended questions from memory. Use it with retrieval (RAG) for any factual task — when given the answer in context, it reliably extracts it.

This is not a GPT-4 competitor. It is an open, reproducible, honestly-scoped small model and a foundation to grow from. Treat its free-form factual claims as unreliable.

---

Architecture

Total params	~177M (tied embeddings)
d_model	768
Layers	20 — 17 Mamba-3 (SISO) + 3 GQA (attention at layers 5, 11, 17; 5:1 hybrid)
Mamba-3	d_state=128, head_dim=64, complex-SSM, rope_fraction=0.5, no conv1d
GQA	12 query heads / 3 KV heads, QK-norm + RoPE
MLP	GatedMLP, inner 1500
Vocab	48,000 SentencePiece BPE (EN+TR, byte-fallback, digit-split)
Context	2048
Precision	bf16 (GPU) / fp32 (CPU)

The hybrid puts attention layers in the depth-middle so induction heads can form, with Mamba-3 handling the bulk linear-time sequence mixing.

Training pipeline

1. Pretraining — 12.0B tokens (EN 6.6B / TR 2.64B / code 1.56B / math 1.2B), WSD schedule, decontaminated against the eval suite. Final loss ~2.50 (PPL ~12).
2. SFT — bilingual instruction tuning. Pivot-language design: reasoning in English (Magpie CoT), Turkish direct-answer (Quardo Alpaca-GPT-4o). Replay mixing prevents forgetting.
3. RAG-aware SFT — extractive QA (SQuAD v2 EN + Turkish extractive QA), sentence-mode answers
   • abstain ("I don't know" when not in context). This is what makes RAG reliable.
4. DPO — attempted; at 177M the preference signal was marginal (acc ~0.57), so the SFT+RAG checkpoint is shipped as final. Kept for transparency.

Evaluation (zero-shot, multiple-choice log-likelihood, N=300)

Task	acc	acc_norm	random	Note
XCOPA-tr (causal)	0.590	0.540	0.50	clear signal
Belebele-tr (reading)	0.330	0.340	0.25	above random
HellaSwag-en (commonsense)	0.333	0.337	0.25	above random
XNLI-tr (inference)	0.330	0.337	0.333	≈ random
TurkishMMLU (academic)	0.177	0.207	0.20	≈ random

Honest read: commonsense / reading / causal reasoning carry real signal; abstract inference (NLI) and knowledge-heavy academic QA (MMLU) sit at chance — expected for 177M. This is exactly why LAMBA is designed to be used with retrieval.

Intended use & limitations

• Good for: retrieval-grounded QA, language-fluency tasks, on-context extraction, research on small SSM/hybrid models, Turkish+English experimentation.
• Not good for: standalone factual Q&A, math, multi-step reasoning, anything safety-critical.
• Bias/safety: trained on web/instruction data; may produce incorrect, biased, or unsafe content. Always keep a human in the loop. Do not use for medical/legal/financial decisions.
• Compute: runs on CPU (pure-PyTorch, --device cpu, no GPU/Triton needed — v1.1) or GPU (Triton Mamba-3 kernel, fastest). On a desktop CPU the decode-cached path does ~6 tok/s; GPU is recommended for interactive use.

How to run

The model is designed to be used with retrieval (RAG) — lamba_rag.py does document → embed → retrieve → grounded answer.

CPU — no GPU, no Triton (v1.1)

pip install torch sentencepiece sentence-transformers "huggingface_hub>=0.23"
# pure-PyTorch + decode-cache; NO wheel needed
python lamba_rag.py --device cpu \
  --ckpt checkpoints/lamba_v1.pt --tokenizer tokenizer/tokenizer.model \
  --demo --query "Türkiye'nin başkenti neresi?"

For raw generation without retrieval deps, lamba_cpu.py runs standalone:

python lamba_cpu.py --ckpt checkpoints/lamba_v1.pt \
  --tokenizer tokenizer/tokenizer.model --query "..."

Low-RAM / smaller footprint: add --int8 to either CPU command for dynamic int8 quantization — model memory drops ~708 MB → ~325 MB with effectively identical greedy output (~1.2× faster on CPU). Handy for laptops / Raspberry Pi. CPU only.

GPU — fastest (Triton Mamba-3 kernel)

pip install einops sentencepiece sentence-transformers "huggingface_hub>=0.23"
pip install --no-deps ./wheels/*.whl                    # Mamba-3 fork wheel (this repo)
python lamba_rag.py --device cuda \
  --ckpt checkpoints/lamba_v1.pt --tokenizer tokenizer/tokenizer.model \
  --demo --query "Türkiye'nin başkenti neresi?"
# your own documents:
python lamba_rag.py --device cuda --docs ./my_docs --query "..."

See LAMBA_Inference.ipynb for a one-click Colab notebook.

Files in this repo

• checkpoints/ — model weights (LAMBA V1.0 = the final SFT+RAG checkpoint)
• tokenizer/ — 48K SentencePiece model
• lamba_rag.py — retrieval-augmented inference (--device cpu or cuda)
• lamba_cpu.py — pure-PyTorch model + decode-cache (no Triton/CUDA; powers the CPU path)
• wheels/ — prebuilt Mamba-3 fork wheels (GPU path only — not needed for CPU)
• LAMBA_Inference.ipynb — Colab quickstart

License

TBD — to be finalized before release. Note: the model derives from the mamba-og (Mamba-3) codebase; its license affects redistribution of the weights/kernels. This will be settled and stated here explicitly.

💛 Support the project

LAMBA is built by an independent developer on self-funded compute (Colab Pro, storage). If you find it useful and want to help train bigger, better open LAMBA models, your support goes directly to GPU hours and data:

• Patreon: https://www.patreon.com/c/kdirgul/membership

(More options, including crypto, coming later.) Thank you. 🙏

Citation

@misc{lamba_v1_2026,
  title  = {LAMBA V1.0: A from-scratch Mamba-3 + GQA hybrid for English and Turkish},
  author = {Kadir Gül},
  year   = {2026},
  note   = {Hugging Face model card}
}

Acknowledgements

Architecture builds on Mamba-3 (state-spaces) and the mamba-og fork. The training recipe and the feasibility of the ~180M / 768-d / 12-layer band were directly informed by hrsvrn/mamba3-180m (the norm-free Mamba-3 project) — thank you. Tokenizer, data, training recipe, and all weights are original to this project.