RHYu2233/latent-ecot-coconut-libero90

Latent-ECoT (Coconut) — MiniVLA-1B on LIBERO-90

Latent embodied chain-of-thought VLA policy: instead of generating 250 tokens of text reasoning per decision (explicit ECoT, ~3 s), this policy reasons through 14 continuous latent "thought" vectors (Coconut-style hidden-state feedback) and decodes actions directly (**0.32 s/decision**).

• Base: MiniVLA-1B (Qwen2.5-0.5B + DINOv2/SigLIP 224px), VQ action chunking (10 steps -> 7 codes)
• Trained 140k steps (batch 128) on LIBERO-90 with the Embodied-CoT reasoning annotations, via an 8-stage latentization curriculum (full-text ECoT -> all-latent)
• LIBERO-90: 89.1% (802/900), vs. explicit ECoT 85.2--87.0% and the ECoT-Lite paper's no-CoT Standard-VLA baseline of 82.0% (arXiv:2505.08243, Table 1)

Usage

Requires the latent-ecot codebase. Place the vq/ folder from this repo at the working directory root (the action tokenizer resolves it relatively), then:

from prismatic.models import load_vla
vla = load_vla("<repo_dir>/checkpoints/step-140000-epoch-31-loss=0.4093.pt",
               hf_token="", load_for_training=False)   # auto-detects Coconut from config.json
action_chunk, info = vla.predict_action(
    image, instruction, unnorm_key="libero_lm_90",
    use_latent_cot=True, return_action_chunk=True, return_info=True)

Checkpoint is model-only (optimizer state stripped). See the GitHub repo for training/eval scripts, per-task results, and the supplementary write-up.

Citation

Please cite Coconut (arXiv:2412.06769), ECoT-Lite (arXiv:2505.08243), ECoT (arXiv:2407.08693), MiniVLA, and LIBERO alongside this artifact.