canyuchen/fedagent-webshop-hardness-std1-ppo-qwen2.5-1.5b

FedAgent — Federated PPO on WebShop (Hardness Partition, std=1) — Qwen2.5-1.5B-Instruct

Milestone checkpoints and full run artifacts from a federated PPO training run of an LLM web-shopping agent, produced with FedAgent ("Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale"), using the accelerated paper recipe on 4× H100-80GB.

Experiment configuration

Item	Value
Benchmark	WebShop (vendored engine, per-client HTTP env service)
Base model	Qwen/Qwen2.5-1.5B-Instruct
Algorithm	PPO (adv_estimator=gae) — actor and critic federated each round
Aggregation	FedAvg over FSDP-sharded checkpoints (server-side, equal weights)
Heterogeneity	Task-level Hardness partition, success_std = 1 (the most skewed setting)
Federation protocol	100 clients total, 2 clients/round, 70 rounds, 3 local epochs/round
Tasks per client	≥100 goals (min_goals_per_client=100)
Seed	base_seed = 42 (client sampling, data partition)
Rollout	windowed multi-turn agent loop (history 2), prompt 4096 / response 512
Optimizers	actor lr 1e-6, critic lr 1e-5, KL loss coef 0.01 (low_var_kl)
Hardware / stack	4× H100-80GB, verl 0.8 + vLLM 0.11 (gpu_memory_utilization=0.45), FSDP
Acceleration	cross-round persistent trainer+vLLM, hot-engine eval, warm env services (paper-equivalent: docs/acceleration.md)
Config file	runs/webshop_hardness-std1_ppo_qwen1.5b_accel/config.yaml (service ports moved to 21000/21200)

Repository layout

round_10/  round_20/  round_30/ ... round_70/     ← milestone checkpoints, every 10 rounds
  ├── actor/    aggregated policy after that round (HF format, safetensors)
  └── critic/   aggregated value model (needed to RESUME federated PPO; not needed for inference)
runs/webshop_hardness-std1_ppo_qwen1.5b_accel/    ← full run artifacts
  ├── config.yaml                 exact experiment config
  ├── federated_summary.json      per-round validation metrics (val_curve + client_curve)
  ├── plots/                      training-dynamics figures (success_rate / task_score)
  ├── driver.log.gz               complete federated-driver log
  ├── launch.sh, *.py, *.sh       orchestration scripts (launcher, janitor, archiver, uploader)
  └── archive_run1_*              an earlier partial run (rounds 0–9) kept for reproducibility reference

Results (unperturbed validation, 64 tasks, temperature 0.4)

The aggregated global model is evaluated every round on a shared, unperturbed WebShop validation service. success_rate = fraction of episodes with full task success; task_score = WebShop partial-credit goal-match score in [0, 1].

Round	success_rate	task_score	reward_mean
0 (base model)	7.8%	0.160	0.78
10	9.4%	0.327	0.94
20	12.5%	0.264	1.25
30	34.4%	0.655	3.44
35 (latest at README time)	34.4%	0.637	3.44
31 (best so far)	40.6%	0.742	4.06

Training was still in progress when this README was written (target: 70 rounds); later milestones are pushed as they complete, and runs/.../federated_summary.json holds the complete curve. Numbers are single 64-task evals — expect a few points of round-to-round noise, amplified by the extreme hardness skew (std=1) of the per-round client draws.

Usage

Inference with the aggregated policy (any milestone):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "canyuchen/fedagent-webshop-hardness-std1-ppo-qwen2.5-1.5b"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="round_30/actor",
                                             torch_dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained(repo, subfolder="round_30/actor")

The model is a WebShop agent: it expects the FedAgent/verl-agent WebShop observation format (multi-turn, windowed history of 2) and emits search[...] / click[...] actions. To evaluate or continue training, use the FedAgent runner:

# evaluate / resume inside the FedAgent repo (see its docs/running.md)
python -m fedagent.fed.run_fed \
  --config runs/webshop_hardness-std1_ppo_qwen1.5b_accel/config.yaml \
  --model-path <local download of round_X/actor>

round_X/critic is the matching aggregated value model: resuming federated PPO from a milestone requires both (actor + critic); plain inference or SFT-style reuse needs only the actor.

Provenance & notes

• Code: https://github.com/sunblaze-ucb/FedAgent (Apache-2.0), which builds on verl 0.8 and the verl-agent WebShop integration.
• This run used the shipped accelerated config fedagent/config/paper_accelerated/task_heterogeneity/ppo/webshop/fed_webshop_ppo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-hardness_success_std-1.yaml with two operational (science-neutral) tweaks: WebShop service ports moved out of the ephemeral range (21000/21200) and vLLM gpu_memory_utilization 0.5 → 0.45.
• Base model license: Qwen2.5 is Apache-2.0; this fine-tune inherits Apache-2.0.

Citation

If you use these checkpoints, please cite FedAgent:

@article{fedagent2026,
  title   = {Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale},
  author  = {Chen, Canyu and Zhu, Kangyu and Chen, Zhaorun and Zhou, Zhanhui and Diao, Shizhe and Lu, Yiping and Li, Tian and Li, Manling and Song, Dawn},
  journal = {arXiv preprint},
  year    = {2026}
}