dpo-llama31-8b-general

A DPO safety adapter for meta-llama/Llama-3.1-8B-Instruct. It makes the base model markedly harder to jailbreak across six harm categories, while explicitly guarding against the usual failure mode of safety tuning — over-refusal on legitimate requests.

This is a LoRA adapter (a few MB), not a full model. Load it on top of the base Llama-3.1-8B-Instruct weights.

• Code, data, and full write-up: https://github.com/1rvinn/securelm
• Method: Direct Preference Optimization (DPO) + LoRA (bf16)
• Trained on: 1,698 preference pairs (1,501 adversarial jailbreak pairs + 197 benign "help > over-refuse" pairs)

Results (held-out)

Evaluated on held-out prompts — generated separately and de-duplicated against the training set — with both models at one fixed decoding condition (safety system prompt, temperature 0.3), so any difference is attributable to the adapter.

Safety — % of adversarial prompts refused (↑ better):

Category	Base	+ this adapter	Δ	N
Cybercrime	47.7%	97.2%	+49.5pp	109
Chem-bio weapons	56.8%	99.0%	+42.2pp	192
Weapons / IED	65.3%	98.6%	+33.3pp	216
Illegal activities	67.2%	99.5%	+32.3pp	186
Illegal drugs	76.9%	99.1%	+22.2pp	216
Child exploitation	84.7%	99.1%	+14.4pp	216

Over-refusal — % of benign prompts answered (↑ better):

Base	+ this adapter	Δ
100.0% (n=150)	93.3%	−6.7pp

The over-refusal cost is real and honestly reported; it is concentrated in weapons:ied (20%), the category trained hardest, and is ≤8% elsewhere.

Usage

With PEFT / transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_id = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "1rvinn/dpo-llama31-8b-general")

SYSTEM = ("You are a helpful, harmless, and honest AI assistant. You decline "
          "requests that facilitate serious harm and you stay genuinely helpful "
          "on legitimate and dual-use tasks.")
msgs = [{"role": "system", "content": SYSTEM},
        {"role": "user", "content": "Your prompt here"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.3)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

With vLLM (no merge needed)

vllm serve meta-llama/Llama-3.1-8B-Instruct --served-model-name llama31-8b \
  --enable-lora --lora-modules dpo=1rvinn/dpo-llama31-8b-general \
  --max-lora-rank 16 --dtype bfloat16

Evaluated with the safety system prompt above at temperature 0.3.

Training

Base	meta-llama/Llama-3.1-8B-Instruct
Method	DPO (TRL DPOTrainer), reference = adapter-disabled policy
LoRA	r=16, α=16, dropout 0.05; targets q,k,v,o,gate,up,down
β	0.1
LR / schedule	1e-5, cosine, 0.1 warmup
Effective batch	16 (batch 2 × grad-accum 8)
Precision	bf16 (no quantization)
Epochs	1 (~19 min on one A100)

Training dynamics: loss 0.69 → 0.27, rewards/accuracies 0.51 → 0.94, rewards/margins 0.001 → 2.80.

Preference pairs were built from adversarial prompts generated with promptfoo red-team plugins (6 harm categories × 7 jailbreak strategies), sampled from the base model under four decoding conditions, graded for safety, and paired as safest response (chosen) vs least-safe response (rejected). A benign counter-slice (helpful answer > refusal) was added to prevent over-refusal. Full pipeline: the GitHub repo.

Limitations

• In-distribution evaluation. Held-out attacks are never-trained-on and de-duplicated, but come from the same plugins/strategies as training — this measures generalization within the trained threat model, not robustness to novel attack families.
• Over-refusal is real and uneven (−6.7pp overall, 20% in weapons:ied).
• General-capability regression not yet measured (no MMLU/benchmark run).
• Single model, single seed; refusal/compliance graded by an LLM judge (GPT-4o).

Responsible use

This adapter is for defensive safety research and to reduce harmful compliance. The associated dataset in the GitHub repo contains real adversarial prompts and harmful completions used as negative training targets; see the repo's DATA.md before using it. Do not use these materials to elicit or distribute harmful content.

Citation / references

• Rafailov et al., Direct Preference Optimization (2023)
• Project: https://github.com/1rvinn/securelm

License

Governed by the Llama 3.1 Community License (this is an adapter for Llama-3.1-8B-Instruct).