fbarbosa1/pi05-driving-dpo-cosmos

pi0.5 Driving — Cosmos-3-judged DPO

A π₀.5 driving policy fine-tuned with Direct Preference Optimization (DPO) where the preferences come from NVIDIA Cosmos-3 (the Cosmos3-Nano reasoner) acting as a vision-language judge of candidate trajectories. Post-trained from markmusic/pi05-driving-bc-v2-checkpoint on NVIDIA PhysicalAI-AV (Alpamayo) clips.

TL;DR — honest result. Training is stable (after the two fixes below), but on a 200-clip held-out eval the Cosmos-DPO policy is worse than BC on ADE-to-ground-truth. This is coherent: the VLM judges scene-appropriate safety, not proximity to the logged human path, so optimizing its preference moves the policy away from GT. GRPO remains the better post-training method here. This card is published as a negative/methods result.

Results (200 held-out clips, ADE / FDE in meters, lower is better)

Model	mean ADE	median ADE	mean FDE
Alpamayo-R1 (10B VLA, 4-cam — reference only)¹	2.09	1.53	6.44
pi0.5 BC (pretrained)	4.37	3.63	11.56
pi0.5 GRPO	3.79 (−13%)	2.96	10.63
pi0.5 Cosmos-DPO (this)	5.36 (+23%)	4.64	14.17

¹ Alpamayo-R1 is a 10B vision-language-action model using 4 cameras + ego history + reasoning; pi0.5-driving uses a single front camera + a 2-D state vector. Different harness — shown for scale. BC/GRPO/Cosmos-DPO are all measured with the same FrameSource decode (GRPO's −13% reproduces the model-card claim, so the scale is consistent).

Method

1. Candidates: sample 2 action chunks per clip from BC.
2. Judge: render each pair (front camera + top-down BEV) and ask the Cosmos-3 reasoner (Cosmos3ReasonerForConditionalGeneration, served via vLLM) which path is safer / more scene-appropriate. Its chain-of-thought is genuinely grounded ("avoids the parked cars on the right", "follows the lane and avoids oncoming traffic").
3. Debias: the raw judge has a strong 88% position bias (prefers "path 1"). A swap-consistency filter (judge each pair in both orderings, keep only verdicts that pick the same physical path regardless of position) leaves 246 / 1000 (25%) clean, balanced pairs.
4. DPO + NLL anchor: offline Diffusion-DPO on a flow-matching policy collapses with any β — because log π ≈ −flow_MSE is unnormalized, the margin objective grows by inflating the rejected action's flow-MSE, which degrades the shared flow field (flow_chosen blew up 20–1700×; ADE → 31 m). Fixed with a +λ·flow_chosen anchor (RPO / DPOP-style) that pins the chosen action's likelihood at BC level. With β=10, λ=10 the run is stable end-to-end (flow_chosen ≤ 0.11, ended below BC).

What this is good for

• A worked, reproducible example of VLM-as-judge → offline DPO for a diffusion/flow policy, including the failure mode (shared-flow-field collapse) and the fix (NLL anchor).
• Evidence that ADE-to-GT is the wrong metric for a safety-judged policy — the judge and the metric optimize different things.

Caveats

• Underperforms BC on ADE-to-GT (see above). Not recommended as a driving policy.
• Only 246 clean preference pairs; the judge's signal is mostly position bias before debiasing.
• Weights are not published here (card / methods only).