guaran-ia/llama-3.1-8b-guarani-initialized-mixed

Llama 3.1 8B Guarani Initialized Mixed

This repository contains the mixed-synthetic 2x2LS continued-pretraining checkpoint released as agustin-lucas/llama-3.1-8b-guarani-initialized-mixed.

The run started from the vocabulary-initialized checkpoint models/meta-llama_Meta-Llama-3.1-8B_260502-1736-guarani-initialized, which is based on meta-llama/Meta-Llama-3.1-8B.

Model Summary

• Architecture: LlamaForCausalLM
• Base model family: Meta-Llama-3.1-8B
• Adaptation strategy: 2x2LS continued pretraining
• 2x2LS update set: embeddings, LM head, first 2 transformer layers, and last 2 transformer layers
• Vocabulary size: 128360
• Model dtype in config: bfloat16
• Max position embeddings in config: 131072

Training Data

This checkpoint was trained from local materialized splits in data/splits.

Run-specific data configuration:

• Train split selection: train
• Synthetic mode: mixed
• Synthetic proportion: automatic in mixed mode
• Held-out evaluation: enabled during training

The project also includes a Hugging Face dataset repo used for loading these splits: agustin-lucas/guarani-llm-splits.

Known split files in that dataset repo:

• train
• synthetic
• val_id
• val_ood
• test_id
• test_ood

In this mixed run, training used real training data together with synthetic data according to the mixed-mode training pipeline.

Training Details

This checkpoint was produced by SLURM job 3143012.

• Starting checkpoint: models/meta-llama_Meta-Llama-3.1-8B_260502-1736-guarani-initialized
• Output checkpoint: models/meta-llama_Meta-Llama-3.1-8B_260502-1736-guarani-initialized-mixed-260512-1937
• Training objective: standard causal language modeling loss
• Number of epochs: 2
• Per-device batch size: 4
• Gradient accumulation steps: 2
• Effective batch size: 8 sequences per optimizer step
• Learning rate: 1e-4
• Warmup ratio: 0.05
• Weight decay: 0.01
• LR scheduler: cosine
• Max sequence length: 512
• Mixed precision: bf16
• Gradient checkpointing: enabled
• Random seed: 42
• Hardware used for the recorded run: 1x NVIDIA L40S
• Final reported train loss: 1.3725
• Final held-out eval loss: 1.7734
• Final saved checkpoint step: 52740

Evaluation During Training

This run used periodic held-out evaluation during training.

Recorded validation losses in the training log decreased from approximately 2.4009 early in training to 1.7734 near the end of the run.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "agustin-lucas/llama-3.1-8b-guarani-initialized-mixed"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)

Basic Generation Example

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "agustin-lucas/llama-3.1-8b-guarani-initialized-mixed"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)

prompt = "Paraguai retame"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Use

This model is intended for research on:

• Guarani language model adaptation
• Continued pretraining and mid-training experiments
• Extended-vocabulary language adaptation
• Low-resource language modeling
• QA-style downstream evaluation in Guarani

Limitations

This is an experimental research model and should not be used for high-stakes decisions.

It may:

• generate incorrect or ungrammatical Guarani
• hallucinate facts
• behave inconsistently across prompts or domains
• reflect artifacts or biases from the base model and training data
• perform differently depending on evaluation prompt formatting and token limits

Downstream Evaluation

The project contains at least two recorded MultiWikiQA Guarani evaluations for this exact checkpoint.

MultiWikiQA Guarani, 3-shot, token limit 15

• EM: 17.97
• F1: 38.93
• Examples: 1525
• Skipped: 2

MultiWikiQA Guarani, 3-shot, token limit 150

• EM: 18.89
• F1: 40.30
• Examples: 1525
• Skipped: 2

Citation

If you use this model, cite the associated paper, thesis, or project report when available. If there is not yet a formal citation, cite the repository and model identifier directly.