LGTEAMai/Nano-1.2-Lite-198M

LGTEAMai/Nano-1.2-Lite

A 198M parameter language model trained from scratch using a Decoder-only Transformer architecture (RoPE, GQA, SwiGLU, RMSNorm).

Specifications

• Architecture: Decoder-only Transformer
• Parameters: 198,077,440 (198M unique trainable parameters)
• Layers: 13
• Hidden Dimension: 1024
• Attention: Grouped Query Attention (16 Q-heads, 4 KV-heads)
• Feed-Forward: SwiGLU (2816)
• Position Embeddings: RoPE (theta=10000.0)
• Normalization: RMSNorm (Pre-Norm)
• Context Window: 1024 tokens
• Vocabulary Size: 50304

Quick Start (Python)

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LGTEAMai/Nano-1.2-Lite")
model = AutoModelForCausalLM.from_pretrained("LGTEAMai/Nano-1.2-Lite", trust_remote_code=True)

---

Описание на русском языке

Языковая модель на 198M параметров (198,077,440), обученная с нуля на базе архитектуры Transformer (Decoder-only: RoPE, GQA, SwiGLU, RMSNorm).

Характеристики

• Архитектура: Transformer (Decoder-only)
• Параметры: 198,077,440 (198M уникальных параметров)
• Слои: 13
• Скрытая размерность: 1024
• Внимание: GQA (16Q / 4KV)
• FFN: SwiGLU (2816)
• Позиционное кодирование: RoPE (theta=10000.0)
• Нормализация: RMSNorm (Pre-Norm)
• Длина контекста: 1024
• Словарь: 50304