BrutalCaesar/act_so101_cubcyl_poshold_chunk50_bc_3cam

ACT without CVAE (behavioural cloning) - SO-101 - 3 cameras - chunk 50

Action Chunking Transformer with the CVAE objective disabled (use_vae=false) — no latent, no KL term, plain L1 regression onto action chunks — trained on all three camera feeds. 34.20M parameters against 51.57M for the CVAE variant.

Baseline for act_so101_cubcyl_poshold_chunk50_cvae_3cam, identical in every other respect.

Final eval_loss on the held-out 30 episodes: 0.2067 — the highest of six variants, though the whole spread is within noise.

One observation worth recording

This is the only variant whose eval curve sat clearly above the others early on: 0.2330 at 10k and 0.2326 at 20k, while every other run was already near 0.21. It converged by 100k, but it is the sole configuration with no regularization of any kind — no latent, no KL, no noise token — and it is also the one given the most capacity, three ResNet-18 backbones and 903 encoder tokens.

A plausible reading is that the CVAE's contribution here is capacity control rather than mode-encoding (the β sweep having ruled out the latter), and that it only becomes visible once the model has enough capacity to misuse. Two runs is not evidence. Offered as a hypothesis for the rollouts to test, not a conclusion.

✅ Camera keys are correct — and all THREE are required

Physical camera	Observation key
wrist (gripper module)	observation.images.wrist
front (desk-level Brio 101)	observation.images.front
top (overhead, boom arm)	observation.images.top

This policy will not run without the overhead camera connected. Its two-camera sibling (..._chunk50_bc) takes wrist+front only.

Keys verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the transposed mapping from the 8bin models.

Held-out episodes — evaluate on these

[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]

5 episodes from each of the 6 object x container blocks. By episode_index: red cube (25 mm) 0-39, yellow cylinder 40-79, white cube (45 mm) 80-119; cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin is always the target.

Inputs / outputs

	Shape
observation.images.wrist	(3, 480, 640)
observation.images.front	(3, 480, 640)
observation.images.top	(3, 480, 640)
observation.state	(6,)
action	(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed

903 encoder tokens (301 per camera at 640x480, plus latent and state), against 602 for the two-camera variants. Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights — load via ACTPolicy.from_pretrained.

🚨 All six variants are indistinguishable on held-out loss

Variant	cameras	latent	eval_loss @100k
cvae	2	CVAE β=10	0.2041
bc	2	none	0.1988
cvae_kl1	2	CVAE β=1	0.2014
cvae_kl0p1	2	CVAE β=0.1	0.2018
cvae_3cam	3	CVAE β=10	0.2006
bc_3cam	3	none	0.2067

The full spread is 0.0079. A single run's eval_loss bounces by ~0.008 between consecutive checkpoints. The between-model differences do not exceed the within-model noise.

And the metric is biased for this data: the demonstrations contain roughly three distinct grasp approaches per object, so held-out L1 rewards a policy that averages them over one that commits to a single valid approach. A worse number here may be a better policy on the robot.

Only scored rollouts can rank these.

Training

Dataset	phi_so101_cubes_cylinder_v1, 90 of 120 episodes, 49,969 frames
Steps / batch / seed	100,000 / 8 / 1000 (16.0 epochs)
Optimizer	AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone	ResNet-18 per camera, ImageNet-pretrained, not frozen
n_obs_steps	1 — single frame, no history
Hardware	1x H200, 1 h 48 m wall, 5.7 GB VRAM

eval_loss is flat from step 60,000 to 100,000, so no overfitting penalty; this is the 100,000-step checkpoint.

Known limitations

• n_obs_steps=1, so no velocity information.
• Scene-dependent: the cardboard box and white bin as used during recording, and the camera poses as mounted. A moved camera or a different container is a domain shift.
• One room, one lighting setup, one operator.
• ⚠️ A camera frame-rate warning was observed on battery power during roughly the first 50 episodes of collection. Reported but unconfirmed — a pixel-level check found 4-9% stalled frames uniformly with no early-vs-late pattern.
• ⚠️ The white cube is low contrast against the light table, most visibly in the overhead view — which this policy uses. Episodes 80-119 are the likeliest to underperform.