File size: 4,320 Bytes
5a4b372 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | ---
library_name: lerobot
license: apache-2.0
tags:
- robotics
- act
- so101
- imitation-learning
- lerobot
datasets:
- BrutalCaesar/phi_so101_8bin_v1
pipeline_tag: robotics
---
# ACT · SO-101 · 8-bin pick-and-place · wrist + front · chunk 50
Action Chunking Transformer trained with [LeRobot](https://github.com/huggingface/lerobot) 0.6.0
on [`BrutalCaesar/phi_so101_8bin_v1`](https://huggingface.co/datasets/BrutalCaesar/phi_so101_8bin_v1).
Two cameras: **wrist** (gripper module) + **front** (desk-level, Logitech Brio 101).
One of six runs in a chunk-size × camera-pair sweep (`{50, 75, 100}` × `{wrist+top, wrist+front}`).
## 🚨 The camera keys are swapped — read this before inference
The dataset was recorded with the wrist and top camera keys transposed. This model therefore
expects the **physical wrist camera** under the key **`observation.images.top`**:
| Feed this physical camera | Under this observation key |
|---|---|
| wrist (gripper module) | `observation.images.top` |
| front (desk-level) | `observation.images.front` |
Wire the overhead camera to `observation.images.top` and the policy gets a close-up it has never
seen in that slot. It will not error — it will just behave badly. The key name is wrong;
the model is not.
Camera framing matters too: the mounts must sit where they sat during recording. A camera that
drifted between recording and evaluation silently invalidates the policy.
## Inputs / outputs
| | Shape | Notes |
|---|---|---|
| `observation.images.top` | `(3, 480, 640)` | physical **wrist** camera, no resize anywhere in the pipeline |
| `observation.images.front` | `(3, 480, 640)` | physical front camera |
| `observation.state` | `(6,)` | SO-101 joint positions |
| **action** | `(50, 6)` | chunk of 50 future actions, all executed (`n_action_steps=50`) |
Normalization ships with the checkpoint as LeRobot 0.6.0 processor files
(`policy_preprocessor*`, `policy_postprocessor*`) — **not** baked into the policy. Load through
`ACTPolicy.from_pretrained` so they are picked up; hand-rolling a forward pass without them
produces actions in the wrong units.
## Training
| | |
|---|---|
| Steps | 100,000 (= 800K samples, ≈14.6 epochs over 54,800 frames) |
| Batch size | 8 |
| Final train loss | **0.055** |
| Optimizer | AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4 |
| Backbone | ResNet-18, ImageNet-pretrained, **not** frozen |
| `dim_model` / `chunk_size` / `kl_weight` | 512 / 50 / 10.0 |
| `n_obs_steps` | 1 (single frame, no history) |
| Seed | 1000 |
| Hardware | 1× A100/H200, Northeastern Explorer cluster |
Everything except `chunk_size` and the camera pair is **LeRobot default**, deliberately — the
sweep is meant to be comparable across people training the same task on the same defaults.
## Train / held-out split
Trained on **89 episodes** covering 6 of the 8 bins. **Left bin 3 and right bin 2 are held out
entirely** — no episode touching them appears in training. Those two bins are the generalization
test; the other six measure fit.
## Evaluation
**Not yet evaluated on hardware.** No success rate is reported here because none has been
measured. The planned protocol is 20 scored rollouts per bin, reported **per bin and never
averaged** — an average over 8 bins hides exactly the failure this split is designed to expose.
Train loss of 0.055 says the model fits the demonstrations. It says nothing about whether it
completes the task.
## Usage
```python
from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("BrutalCaesar/act_so101_8bin_wrist_front_chunk50")
```
Or point a LeRobot eval script at it with `--policy.path=BrutalCaesar/act_so101_8bin_wrist_front_chunk50`,
remembering the camera-key mapping above when you write `--robot.cameras`.
## Known limitations
- Single observation frame, so no velocity information; the policy infers motion from wrist-view context alone.
- The CVAE latent is effectively collapsed at `kl_weight=10.0` on this dataset (KL ≈ 0 throughout training), so this behaves close to a deterministic chunk regressor. Not a defect — expected for ACT — but do not read the latent as a source of behavioral diversity.
- Trained on one lighting setup in one room. No domain randomization beyond LeRobot's default image augmentation.
|