Harrysunshine/act-libero-goal

ACT — LIBERO-Goal "put the bowl on the plate"

An ACT (Action Chunking Transformer) policy trained with LeRobot on a single LIBERO-Goal task: put the bowl on the plate (task suite libero_goal, task_id = 8).

It reaches 100% success rate (10/10 episodes) on that task in the LIBERO simulator.

This checkpoint was produced as a teaching demo for a robot-learning course: it is small (51.7M parameters), trains in minutes on one GPU, and is meant as a clean, reproducible reference point for "what a working single-task imitation policy looks like".

Model details

Policy	ACT (CVAE + action chunking), lerobot type: act
Parameters	51,671,687 (fp32, 234 tensors)
Vision backbone	ResNet-18, initialised from ResNet18_Weights.IMAGENET1K_V1
Transformer	dim_model=512, 8 heads, 4 encoder layers, 1 decoder layer, FFN 3200, dropout 0.1, post-norm
CVAE	use_vae=true, latent dim 32, 4 VAE-encoder layers, kl_weight=10.0
Action chunking	chunk_size=100, n_action_steps=100, no temporal ensembling
Observation steps	1
Normalisation	mean/std for images, state and actions (use_imagenet_stats=false)

Inputs / outputs

Key	Shape	Meaning
observation.images.image	3 × 256 × 256	agentview (third-person) RGB
observation.images.image2	3 × 256 × 256	robot0_eye_in_hand (wrist) RGB
observation.state	8	proprioceptive state
action (output)	7	6-DoF end-effector delta + gripper, relative control

Training

• Data: lerobot/libero, restricted to the 49 episodes belonging to put the bowl on the plate in the libero_goal suite.
• Optimiser: AdamW, lr 1e-5 (same lr for the backbone), weight decay 1e-4.
• Batch size 256, seed 1000, video_backend=pyav.
• Released checkpoint: the best-scoring evaluation checkpoint, at step ≈ 950 (≈ 53 epochs over the 49-episode subset, 243,200 samples seen).

Evaluation

Evaluated in the LIBERO simulator on the same task it was trained on (libero_goal, task_id = 8, obs_type=pixels_agent_pos, 256×256 observations):

Metric	Value
Success rate (pc_success)	100 (10 / 10 episodes)
Average summed reward	1.0
Step	950

The raw evaluation record is included as eval_summary.json.

Usage

Requires LeRobot with the LIBERO environment (lerobot[libero]). Closed-loop rollout in simulation:

import torch
from lerobot.configs.policies import PreTrainedConfig
from lerobot.envs.configs import LiberoEnv as LiberoEnvConfig
from lerobot.envs.factory import make_env, make_env_pre_post_processors
from lerobot.envs.utils import add_envs_task, preprocess_observation
from lerobot.policies.factory import get_policy_class, make_pre_post_processors
import lerobot.policies  # register policy types

policy_path = "Harrysunshine/act-libero-goal"
task_suite = "libero_goal"
task_id = 8  # "put the bowl on the plate"
device = "cuda"

policy_cfg = PreTrainedConfig.from_pretrained(policy_path)
policy_cfg.device = device
policy = get_policy_class(policy_cfg.type).from_pretrained(
    policy_path, config=policy_cfg, strict=False
)
preprocessor, postprocessor = make_pre_post_processors(
    policy_cfg, pretrained_path=policy_path
)

env_cfg = LiberoEnvConfig(
    task=task_suite,
    task_ids=[task_id],
    obs_type="pixels_agent_pos",
    observation_height=256,
    observation_width=256,
    episode_length=300,
)
env = make_env(env_cfg, n_envs=1)[task_suite][task_id]
env_preprocessor, env_postprocessor = make_env_pre_post_processors(env_cfg, policy_cfg)

policy.reset()
observation, _ = env.reset(seed=[7])
for _ in range(300):
    batch = preprocess_observation(observation)
    batch = add_envs_task(env, batch)
    batch = env_preprocessor(batch)
    batch = preprocessor(batch)
    with torch.inference_mode():
        action = policy.select_action(batch)
    action = postprocessor(action)
    action = env_postprocessor(action)
    observation, _, terminated, truncated, info = env.step(action.cpu().numpy())
    if terminated[0] or truncated[0]:
        break

Limitations

• Single task, no language conditioning. ACT regresses actions from pixels and state only; this checkpoint has only ever seen put the bowl on the plate and will attempt that behaviour regardless of what else is in the scene.
• Simulation only. Trained and evaluated entirely in LIBERO; it has not been deployed on physical hardware and the action space (relative EE control, LIBERO conventions) is simulator-specific.
• Observation format is fixed. Two 256×256 cameras (agentview + wrist) and an 8-d state vector, in exactly that layout.
• Small evaluation set. 100% is over 10 evaluation episodes from LIBERO's initial-state set, not a large-scale benchmark run.

Citation

ACT / Action Chunking with Transformers:

@inproceedings{zhao2023act,
  title     = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
  author    = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
  booktitle = {Robotics: Science and Systems (RSS)},
  year      = {2023}
}

LIBERO benchmark:

@inproceedings{liu2023libero,
  title     = {LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning},
  author    = {Liu, Bo and Zhu, Yifeng and Gao, Chongkai and Feng, Yihao and Liu, Qiang and Zhu, Yuke and Stone, Peter},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2023}
}

---

中文说明

这是一个用 LeRobot 训练的 ACT（动作分块 Transformer） 策略，任务是 LIBERO-Goal 套件中的第 8 个任务 “把碗放到盘子上”（put the bowl on the plate）。

• 数据：lerobot/libero 中属于该任务的 49 条示范轨迹。
• 模型：ResNet-18 视觉主干 + 4 层编码器 / 1 层解码器 Transformer + CVAE（隐变量 32 维）， 一次预测 100 步动作块，51.7M 参数。
• 输入：第三人称相机 + 腕部相机各一路 256×256 RGB，8 维本体状态；输出 7 维动作。
• 评测：在 LIBERO 仿真同任务上 10 条 episode **成功率 100%**。
• 定位：机器人学习课程的教学 demo——足够小、训练几分钟就能复现，用来演示“一个能跑通的 单任务模仿学习策略长什么样”。

限制：只会这一个任务、不吃语言指令、只在仿真里验证过、观测格式固定（两路 256×256 相机

• 8 维状态），100% 是 10 条 episode 的结果而非大规模 benchmark。