qwen2.5-0.5b-kd-merged-cnndm-50k

Task: abstractive text summarization — given a news article, generate a short summary (2–3 sentences).

Merged full model on the Hub: Qwen/Qwen2.5-0.5B fine-tuned with LoRA on teacher distillation targets, then fused with merge_and_unload(). Load directly via AutoModelForCausalLM.from_pretrained(this_repo) — no PEFT or separate base model.

Training objective: sequence-level knowledge distillation — summaries match Qwen/Qwen2.5-7B-Instruct teacher outputs.

Paired SFT merged model: Harsha901/qwen2.5-0.5b-sft-merged-cnndm-50k

---

Training (notebook 03 — 50k run)

Item	Value
Task	Abstractive summarization (article → summary)
Base model	Qwen/Qwen2.5-0.5B
Teacher (label source)	Qwen/Qwen2.5-7B-Instruct
Training file	teacher_generations.jsonl (50,000 records)
Label field	teacher_summary
Train / val split	49,500 train · 500 val (eval_split_size, seed 42)
Source domain	CNN/DailyMail (abisee/cnn_dailymail, config 3.0.0)
Published artifact	LoRA trained → merged into base → pushed as this repo

Hyperparameters (matched with SFT run)

Setting	Value
Method	LoRA via SFTTrainer (TRL)
LoRA rank / alpha / dropout	r=32, alpha=64, dropout=0.1
LoRA targets	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Epochs	3 max
Per-device batch × grad accum	4 × 4 → effective batch 16
Learning rate	2e-4, cosine schedule, 3% warmup
Optimizer	adamw_torch_fused
Weight decay	0.01
Precision	bf16
Max sequence length	1536
Article pre-truncation	6000 characters, left-truncated
Loss	Completion-only (DataCollatorForCompletionOnlyLM on <|im_start|>assistant\n)
Early stopping	patience=3 on eval_loss
Eval / save	every 200 steps · load_best_model_at_end=True

Summarization prompt (required at inference)

Role	Content
System	You are a concise news summarizer. Write a short summary of the article in 2-3 sentences. Output only the summary itself, with no preamble, headers, or commentary.
User	Article:\n{article}\n\nSummary:

---

Evaluation (notebook 06 — merged model)

Protocol: vLLM greedy decoding (temperature=0), preamble-stripped ROUGE, BERTScore F1 with baseline rescaling, 1000 test examples per dataset (819 for SAMSum). Cross-domain: XSum, SAMSum, DialogSum.

ROUGE-1 / ROUGE-2 (clean)

Benchmark	Type	Teacher 7B	Base 0.5B	KD merged (this)	SFT merged
CNN/DailyMail	in-domain	33.41 / 10.54	8.62 / 1.88	31.36 / 8.99	35.83 / 12.64
XSum	cross	27.22 / 7.16	11.43 / 1.07	21.46 / 3.92	20.02 / 2.86
SAMSum	cross	38.11 / 12.76	11.66 / 1.78	26.12 / 5.06	24.31 / 4.77
DialogSum	cross	30.45 / 8.86	15.00 / 3.07	21.50 / 4.18	19.76 / 3.82

BERTScore F1 (rescaled) · preamble rate (CNN/DM)

	KD merged (this)	SFT merged
CNN/DM BERTScore	19.82	24.07
XSum	21.37	9.50
SAMSum	27.08	16.31
DialogSum	16.78	6.91
CNN/DM preamble rate	3.7%	66.7%

Summarization takeaways

• +22.7 ROUGE-1 over untrained base on in-domain CNN/DM.
• Best cross-domain summarization among 50k students (XSum, SAMSum, DialogSum).
• SFT merged wins only on in-domain CNN/DM ROUGE-1 (+4.5).

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "Harsha901/qwen2.5-0.5b-kd-merged-cnndm-50k"

SYSTEM = (
    "You are a concise news summarizer. Write a short summary of the article in 2-3 sentences. "
    "Output only the summary itself, with no preamble, headers, or commentary."
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

article = "..."
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Article:\n{article}\n\nSummary:"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=160, do_sample=False)

print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

---

When to use

Scenario	Recommendation
Cross-domain summarization	This model
CNN/DM in-domain only	SFT merged 50k

---

Reproducibility

Step	Notebook
LoRA KD training	kd_vs_sft_50k_cnn_dailymail/notebooks/03_kd_lora_train_Qwen2.5-0.5B-student.ipynb
Merge + eval	kd_vs_sft_50k_cnn_dailymail/notebooks/06_eval_with_bertscore_and_genfix_50k.ipynb

Metrics: kd_vs_sft_50k_cnn_dailymail/kd_vs_sft_eval_results/all_results.csv