| | --- |
| | license: apache-2.0 |
| | tags: |
| | - pytorch |
| | - transformer |
| | - mamba |
| | - hybrid |
| | - matryoshka |
| | - nanochat |
| | - adaptive-compute |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # 🌀 Adamba: Adaptive Mamba |
| |
|
| | > **Ad**aptive **Mamba**: Elastic compute with dynamic Matryoshka scaling |
| |
|
| | **Project location [unixsysdev/adamba](https://github.com/unixsysdev/adamba)** |
| |
|
| | ## Available Checkpoints |
| |
|
| | | Variant | Parameters | Dim | Features | Status | Download | |
| | |---------|------------|-----|----------|--------|----------| |
| | | phase1_6b_base | 6.4B | 2048 | mamba_integration | ✅ | [Download](./checkpoints/phase1_6b_base.pt) | |
| | | phase2_6b_matryoshka | 6.4B | 2048 | matryoshka, early_exit | ⏳ | — | |
| | | phase3_9b_matryoshka | 9.3B | 2560 | matryoshka, early_exit | ⏳ | — | |
| | | phase3_20b_matryoshka | 20B | 4096 | matryoshka, early_exit | ⏳ | — | |
| | | sft_20b | 20B | 4096 | matryoshka, early_exit, sft | ⏳ | — | |
| | | rl_20b | 20B | 4096 | matryoshka, early_exit, rl_agent | ⏳ | — | |
| | |
| | ## Architecture Overview |
| | |
| | Adamba combines three efficiency techniques: |
| | |
| | | Technique | Implementation | Purpose | |
| | |-----------|----------------|---------| |
| | | **Matryoshka (MRL)** | Width: 128 → 4096 per layer | Elastic compute | |
| | | **Early Exit** | ConfidenceGate per layer | Skip when confident | |
| | | **Static SSM** | Mamba at full dim | Stable memory backbone | |
| | |
| | ``` |
| | ┌─────────────────────────────────────────────────┐ |
| | │ PROMPT → LayerDimPredictor → [dim per layer] │ |
| | │ │ |
| | │ Attention + MLP: Dynamic (Matryoshka sliced) │ |
| | │ Mamba: Static (full dim) │ |
| | │ │ |
| | │ Gate > 0.95 → EXIT EARLY │ |
| | │ Gate < 0.50 → EXPAND remaining layers │ |
| | └─────────────────────────────────────────────────┘ |
| | ``` |
| | |
| | ## Training Pipeline |
| | |
| | ``` |
| | nanochat-d32 (1.9B) |
| | ↓ Surgery (add 32 Mamba layers) |
| | Phase 1: 6.4B (dim=2048) ← Mamba integration |
| | ↓ Enable Matryoshka |
| | Phase 2: 6.4B (dim=2048) ← Full training |
| | ↓ Progressive expand |
| | Phase 3: 9.3B → 20B (dim=4096) |
| | ↓ Fine-tuning |
| | SFT: Instruction tuning |
| | RL: Agent capabilities |
| | ``` |
| | |
| | ## Model Details |
| | |
| | - **Base**: [karpathy/nanochat-d32](https://huggingface.co/karpathy/nanochat-d32) |
| | - **Architecture**: 64 blocks (32 Attention + 32 Mamba interleaved) |
| | - **Vocabulary**: 65,536 tokens |
| | - **Matryoshka Dims**: [128, 256, 512, 1024, 2048, 4096] |
| | |
| | ## Usage |
| | |
| | ```python |
| | # Coming soon - inference code |
| | # See: https://github.com/unixsysdev/adamba |
| | ``` |
| | |
| | ## Links |
| | |
| | - 📂 **GitHub**: [unixsysdev/adamba](https://github.com/unixsysdev/adamba) |
| | - 📊 **Training**: [WandB](https://wandb.ai/dalletest123/nano-fractal) |
| | |
| | ## License |
| | |
| | Apache 2.0 |
| | |
