shehryars715/pythia-410m-step5000-alpaca

Pythia-410M step5000 Alpaca full fine-tune

An intentionally small, inspectable before/after experiment: can one epoch of Alpaca SFT turn a very early Pythia checkpoint from prompt-template repetition into a recognizable instruction follower?

This is a full-parameter supervised fine-tune of the early step5000 revision of EleutherAI/pythia-410m. It was trained for one epoch on 10,000 examples from yahma/alpaca-cleaned, using a separate fixed set of 256 examples for validation.

This repository contains a complete model checkpoint. It is not a LoRA or PEFT adapter and does not require the original checkpoint to be loaded separately.

Download the complete checkpoint · See all raw before/after generations

Why this checkpoint is interesting

• A genuine early-checkpoint transformation: the base step5000 model mostly loops over prompt markers; after SFT it emits answer-shaped responses.
• A full model, not an adapter: useful for direct loading and for studying how every parameter changes during instruction tuning.
• Small enough to explore: 410M parameters is approachable for educational inference and analysis.
• Reproducible: the training split, validation split, prompt format, hyperparameters, deterministic test prompts, and decoding settings are documented.
• Transparent about failure: the included comparisons show improved instruction behavior alongside major factual weaknesses.

At a glance

Property	Value
Starting point	Pythia-410M at pretraining revision step5000
Fine-tuning data	10,000 cleaned Alpaca examples
Validation data	256 fixed held-out examples
Update method	Full-parameter SFT, completion-only loss
Training duration	1 epoch
Context length	384 tokens
Best use	Education, checkpoint analysis, SFT demonstrations
Production ready	No

Model details

• Architecture: GPT-NeoX causal language model
• Base model: EleutherAI/pythia-410m
• Base revision: step5000
• Language: primarily English
• Task: instruction-conditioned text generation
• Fine-tuning method: full-parameter supervised fine-tuning
• Quantization: none
• Adapters: none

The starting revision is only partially pretrained. Alpaca fine-tuning teaches instruction/response structure but does not replace the language pretraining missing from this early checkpoint.

Training data

The source dataset was filtered to remove examples with empty outputs, shuffled with seed 42, and split as follows:

• First 256 shuffled examples: held-out validation set
• Next 10,000 examples: training set

Examples were converted to the original Alpaca prompt format. The target response was kept as a separate completion, with EOS appended. Loss was calculated only on completion and EOS tokens; prompt tokens supplied context but were masked from the loss.

Training procedure

Setting	Value
Epochs	1
Training examples	10,000
Validation examples	256
Maximum sequence length	384
Per-device batch size	2
Gradient accumulation	8
Learning rate	5e-5
Scheduler	Cosine
Warmup ratio	0.03
Weight decay	0.01
Optimizer	AdamW
Gradient clipping	1.0
Packing	Disabled
Gradient checkpointing	Enabled during training
Seed	42

BF16 was selected only on GPUs with compute capability 8.0 or newer; otherwise FP16 mixed precision was used. All model parameters were enabled for training.

Prompt format

Without an additional input:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:

With an additional input:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "shehryars715/pythia-410m-step5000-alpaca"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=dtype,
).to(device)
model.eval()

instruction = "Explain photosynthesis in exactly two sentences."
prompt = (
    "Below is an instruction that describes a task. Write a response that "
    "appropriately completes the request.\n\n"
    f"### Instruction:\n{instruction}\n\n"
    "### Response:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=128,
        repetition_penalty=1.05,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

Evaluation

The training notebook evaluates the untouched checkpoint and the fine-tuned model on the same 256 held-out examples. It also compares deterministic generations on six fixed prompts and includes limited rule-based checks plus a blank human-review table.

What changed in the six-prompt comparison

Observation	Before SFT	After SFT
Avoided the prompt-template repetition seen in the base response	0/6	6/6
Passed the notebook's four transparent surface-rule checks	0/4	1/4

The useful signal is behavioral, not capability-level: the model learned to produce response-shaped text, but most tested answers remain incorrect. The rule checks cover only translation keywords, speed, prime-number logic, and three-item formatting.

Side-by-side results

Prompt	step5000 before SFT	After full fine-tuning
Photosynthesis in exactly two sentences	Repeats “Write a response…” and ### Response:	Produces two complete sentences about photosynthesis
Translate weather sentence into French	Repeats instruction/response markers	Produces a direct sentence response
Average speed: 180 km in 3 hours	Repeats the prompt and markers	Produces a concise numerical response
Python prime-checking function	Repeats the prompt and markers	Produces a Python function and explanation
Exactly three interview tips	Repeats instruction/response markers	Produces three numbered tips
Why leaves appear green	Repeats instruction/response markers	Produces answer-like paragraphs

Example 1: visible instruction-following improvement

Prompt

Explain photosynthesis in exactly two sentences.

Before SFT

Write a response that accurately completes the request.
### Response:
(the same template continues repeatedly)

After SFT

Photosynthesis is a process by which plants and animals convert sunlight, carbon dioxide, and other organic compounds into glucose, oxygen, and other chemical compounds. It is a complex process that involves the conversion of carbon dioxide into glucose, oxygen, and other chemical compounds.

The response now follows the requested two-sentence shape, although its claim about animals is inaccurate.

Example 2: fluent form does not guarantee correctness

Prompt

Calculate the average speed for 180 km in 3 hours.

Before SFT

Calculate the average speed for 180 km in 3 hours.
### Instruction:
(the prompt and markers continue repeatedly)

After SFT

The average speed for 180 km in 3 hours is approximately 0.8 km/h.

The expected answer is 60 km/h. This is a compact example of why the checkpoint should be studied, not trusted.

Example 3: formatting learned better than content

Prompt

Give exactly three concise job interview tips.

After SFT

1. Know Your Job: A good job interview is a great way to learn about your skills, interests, and abilities.
2. Be Prepared: Prepare your resume and cover letter to ensure that you are prepared for the interview.
3. Be Prepared for the Interview: Prepare your resume and cover letter to ensure that you are prepared for the interview.

The response has three distinguishable items, but it is redundant and not especially concise.

The complete, unedited generations are provided in comparison_results.csv. Measured baseline and final validation losses were not present in that CSV, so no loss values are invented here.

Intended uses

• Studying the effect of instruction tuning on an early language-model checkpoint
• Demonstrating full-parameter SFT and completion-only loss
• Small-scale educational experiments with Pythia checkpoints

Limitations and risks

• The step5000 checkpoint is only partially pretrained.
• The model is expected to be weak in factual knowledge, reasoning, safety, formatting reliability, and multi-turn conversation.
• Outputs may be incorrect, biased, incoherent, repetitive, or unsafe.
• The model has not been evaluated sufficiently for consequential decisions or production deployment.
• The simple rule-based checks in the training notebook are not substitutes for broad human evaluation.

Users should validate outputs for their domain and add appropriate safeguards before any interactive use.

License and attribution

The Pythia base model is published under Apache-2.0. At the time this card was written, the Hugging Face metadata for yahma/alpaca-cleaned displayed CC BY 4.0, while text inside its dataset card stated CC BY-NC 4.0. Review the current upstream model and dataset terms before choosing a license for or redistributing this fine-tuned checkpoint.

Citation

@article{biderman2023pythia,
  title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
  author={Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin and others},
  journal={Proceedings of the 40th International Conference on Machine Learning},
  year={2023}
}

@misc{alpaca,
  author={Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and others},
  title={Stanford Alpaca: An Instruction-following LLaMA Model},
  year={2023},
  howpublished={\url{https://github.com/tatsu-lab/stanford_alpaca}}
}