Qwen2.5-0.5B-DPO-Detox

A DPO (Direct Preference Optimization) fine-tune of Qwen/Qwen2.5-0.5B-Instruct, trained to reduce toxic completions. This is the DPO counterpart to a prior PPO-based detox project (sarimahsan101/qwen2.5-0.5b-detox-ppo) — same base model, same toxicity classifier, and the same prompt source, so any difference in behavior is attributable to the optimization algorithm rather than a different setup.

This is the full-scale run (QUICK_TEST=False, 4,000 prompts), superseding an earlier 200-prompt smoke test of the same pipeline.

Model Details

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Method: Direct Preference Optimization (DPO), via trl.DPOTrainer
• Loss type: sigmoid (standard DPO loss)
• Reference model: auto-created by DPOTrainer as a frozen copy of the base checkpoint (no separate ref_model passed)
• Hardware: single NVIDIA T4 (Kaggle/Colab free tier)
• Hub repo: sarimahsan101/Qwen2.5-0.5B-DPO-detox

Training Data

No off-the-shelf "detox preference dataset" exists, so one was built from scratch:

1. Prompts sourced from allenai/real-toxicity-prompts, sorted by prompt toxicity score and taking the most toxicity-eliciting prompts — 4,000 prompts, toxicity range ≈ 0.892–0.992.
2. For each prompt, 4 completions were sampled from the base model (do_sample=True, temperature=1.0, top_p=0.9, max_new_tokens=32).
3. All completions were scored with facebook/roberta-hate-speech-dynabench-r4-target (P(hate), same classifier used as the PPO run's reward signal).
4. Per prompt, the least-toxic completion became chosen and the most-toxic became rejected; pairs with a toxicity gap smaller than MIN_TOXICITY_GAP = 0.25 were discarded as too weak a signal.

This produced 671 preference pairs out of 4,000 prompts (16.8% keep rate), split 90/10 into 603 train / 68 eval examples.

Training Configuration

Hyperparameter	Value
Epochs	3
Per-device train batch size	4
Gradient accumulation steps	4 (effective batch size 16)
Learning rate	5e-6
beta (KL penalty strength)	0.1
loss_type	sigmoid
max_length	128
Precision	bf16=True, fp16=False
Seed	42

Total optimizer steps: 114 (3 epochs over 603 examples at effective batch size 16).

Training Result

train_loss: 0.569
train_runtime: 695.6s (~11.6 min)
epoch: 3.0

Evaluation

Base vs. DPO-tuned completions were compared on the full 68-example eval split, scored with the same toxicity classifier:

Run	Mean toxicity	Median toxicity	% > 0.5
Base Qwen2.5-0.5B-Instruct (no tuning)	0.138	0.003	11.8%
DPO	0.096	0.004	7.4%

Interpretation:

• Mean toxicity dropped ~30% relative to base (0.138 → 0.096).
• Share of completions exceeding the 0.5 toxicity threshold fell from 11.8% to 7.4% — a ~37% relative reduction in the worst-case tail, which is the metric that matters most for a moderation-style use case: DPO is specifically suppressing the highest-toxicity completions rather than uniformly shifting everything.
• Median toxicity ticked up marginally (0.003 → 0.004), but both values sit near the classifier's floor — most completions from both models were already non-toxic, so the median is dominated by near-zero noise and isn't a meaningful signal here. Mean and %>0.5 are the metrics that carry information at this scale.
• Net read: a modest but real detoxification effect, concentrated in the tail rather than spread uniformly — consistent with what you'd expect from a preference-pair method trained on a moderate (603-example), well-filtered dataset rather than a massive one.

Intended Use

Research artifact for comparing RLHF-style alignment methods (PPO vs. DPO vs. planned IPO/KTO) on a toxicity-reduction task, on free-tier compute. Not intended for production moderation or safety-critical deployment — the toxicity classifier used as a training/eval signal is a single automated proxy, not ground truth.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "sarimahsan101/Qwen2.5-0.5B-DPO-detox"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

inputs = tokenizer("Your prompt here", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=32, do_sample=True, temperature=1.0, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• Preference pairs were mined from the base model's own samples rather than human annotations, so the model can only learn to avoid modes of toxicity the base model already exhibits — it cannot learn to avoid toxic patterns it never sampled in the first place.
• Toxicity labels come from a single classifier (facebook/roberta-hate-speech-dynabench-r4-target), which has its own biases and blind spots — it is used here as a proxy reward for both training and evaluation, so the reported improvement is with respect to that specific classifier, not an independent measure of harm.
• Only 671 preference pairs were used for training — a relatively small preference set by DPO standards; results should be treated as a promising first pass rather than a converged, thoroughly-tuned model.
• No held-out human evaluation or adversarial red-teaming was performed; the 7.4% of eval completions still scoring above the toxicity threshold indicates the model is not reliably non-toxic.