Telugu GPT-2 (dravidian-gpt2-telugu)

A GPT-2 language model trained from scratch on Telugu text as part of the Dravidian LM project. The model is trained on a curated mix of CC100, Wikipedia, and Samanantar corpora, using a custom SentencePiece BPE tokenizer.

Model Details

Property	Value
Architecture	GPT-2
Embedding dim	768
Layers	12
Attention heads	12
FFN dim	3072
Context length	1024 tokens
Vocab size	32,000
Parameters	~124M

Training Data

Source	Description
CC100	Common Crawl Telugu web text
Wikipedia	Telugu Wikipedia (20231101 dump)
Samanantar	Telugu side of En-Te parallel corpus

Splits: 96% train / 2% validation / 2% test

Cleaning: Unicode NFC normalization, HTML/XML tag removal, URL and email removal.

Tokenizer

Property	Value
Type	SentencePiece BPE
Vocab size	32,000
Coverage	1.0 (full character coverage)
Normalization	NMT NFKC
Byte fallback	Yes

Training Configuration

Hyperparameter	Value
Epochs	3
Per-device batch size	4
Gradient accumulation	16
Effective batch size	64
Learning rate	1e-4
LR scheduler	Cosine
Warmup steps	4,000
Weight decay	0.01
Max grad norm	0.5
Precision	BF16

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "pulipakav-1/dravidian-gpt2-telugu"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.eval()

prompt = "తెలుగు భాష చాలా గొప్పది"

inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.3,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Repository Structure

config.json
generation_config.json
model.safetensors
training_args.bin
checkpoint-178965/
checkpoint-268446/
telugu.model
tokenizer.model
tokenizer.vocab

Limitations

• Trained primarily on web and Wikipedia text and may reflect biases present in those sources.
• May mix in English tokens on news-style or code-mixed prompts.
• Not fine-tuned for any downstream task; this is a base language model.

Citation

@misc{telugu_gpt2_2024,
  author    = {Pulipaka, V.},
  title     = {Telugu GPT-2: A Dravidian Language Model Trained from Scratch},
  year      = {2024},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/pulipakav-1/dravidian-gpt2-telugu}
}

Project

Part of the Dravidian LM project, training monolingual GPT-2 models for Telugu, Kannada, Tamil, and Malayalam from scratch.

GitHub: vishnup22/dravidian-lm-research