PointmapVLA-LIBERO

PointmapVLA fine-tuned on the four LIBERO suites (spatial, object, goal, 10). This is the checkpoint behind the LIBERO result of the paper and, evaluated zero-shot, the LIBERO-Plus result. Code, recipes and the evaluation harness: PointmapVLA repository.

What it is

PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language backbone with a pointmap encoder (a copy of the vision tower fed raw metric XYZ pointmaps in the robot-base frame, features added patch-wise to the RGB tokens) and a 32-layer layer-wise flow-matching DiT action expert (4 Euler steps at inference). 6.08 B parameters, bf16.

This checkpoint predicts chunks of 8 LIBERO actions (10-D) from two camera views (agentview, wrist), the instruction and an 11-D end-effector state.

Provenance

• Initialised from DAVIAN-Robotics/PointmapVLA-Pretrained (Stage 1 + Stage 2 pretraining on 76.3 M robot frames and 10.7 M 3D spatial VQA samples).
• Fine-tuned for 30,000 optimizer steps, global batch 128 (4 GPUs x 32), AdamW, cosine schedule with 1,500 warm-up steps, learning rates 1e-5 (backbone, pointmap encoder) and 1e-4 (action expert, projections), DeepSpeed ZeRO-2, bf16. One policy for all four suites.
• Data: DAVIAN-Robotics/libero-r6
  • DAVIAN-Robotics/libero-r6-pointmap (1,693 episodes over 40 tasks, 2 cameras).
• On load from the pretrained checkpoint the four action-head tensors whose shape depends on the layout were remapped (shared end-effector/gripper slots copied, new slots fresh); all other weights transferred unchanged.
• Prompt: Task: {instruction} Mode: <end_effector> Pointmap_origin: <robot>, chat template default for thinking (no enable_thinking key; differs from the RoboCasa checkpoint and is recorded in config.yaml).
• Exact config: examples/simBenchmarks/LIBERO/train_files/pointmapvla_libero.yaml in the repository (_80gb.yaml is the same recipe with gradient accumulation).

Files

config.yaml                        architecture and the data recipe (release schema)
dataset_statistics.json            action/state mean/std/min/max/quantiles used for (un)normalisation
base_vlm/                          Qwen3.5-4B config, tokenizer and processor only; no weights
checkpoints/
  model-00001-of-00003.safetensors
  model-00002-of-00003.safetensors
  model-00003-of-00003.safetensors   ~11 GB in total, bf16
  model.safetensors.index.json

State-dict key groups: qwen_vl_interface.model.* (backbone), pointmap_encoder.*, action_model.* (DiT expert), project_layers.* (per-layer projections). The language model head is tied to the token embedding and stored once; the index records it under metadata.tied_weights and the loader re-ties it.

How to load

from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA

model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-LIBERO")
out = model.predict_action(examples)                    # {"normalized_actions": (B, 8, 10)}

examples[i] is a dict with image (2 HWC uint8 arrays, 224 x 224, agentview and wrist, rotated 180° as in the LIBERO datasets), lang, state ((1, 11) float32, normalised), pointmap ((2, 256, 256, 3) float16 metres, robot-base frame) and pointmap_valid ((2, 256, 256) bool). Un-normalise with dataset_statistics.json. In practice, run the policy server from the repository, which does the un-normalisation and the r6 -> axis-angle conversion:

python deployment/model_server/server_policy.py --ckpt DAVIAN-Robotics/PointmapVLA-LIBERO --port 21000 --use_bf16 --seed 7

Evaluation

Protocol: action chunk 8 executed in full, 4 Euler steps, K = 12 parallel environments, server + environment seeds 7, 8, 9 (one run each); numbers are the mean (± sd) over the three seeds. LIBERO: 4 suites x 10 tasks x 50 trials = 2,000 episodes. LIBERO-Plus (zero-shot, same checkpoint): 10,030 perturbed tasks x 1 trial (2402 / 2518 / 2591 / 2519 per suite). One seed alone typically lands within about ±0.2 points of the LIBERO mean.

for s in 7 8 9; do CKPT=DAVIAN-Robotics/PointmapVLA-LIBERO BENCH=libero      SEED=$s scripts/eval/run_eval_libero.sh; done
for s in 7 8 9; do CKPT=DAVIAN-Robotics/PointmapVLA-LIBERO BENCH=libero_plus SEED=$s GPUS=0,1 scripts/eval/run_eval_libero.sh; done

Benchmark	Paper	Our reproduction
LIBERO (4 suites), success rate (%)	99.0	99.1 ± 0.1 (seeds 7, 8, 9)
LIBERO-Plus, success rate (%)	92.4	92.4 ± 0.3 (seeds 7, 8, 9)

Intended use and limitations

• Simulation benchmark checkpoint for LIBERO and LIBERO-Plus (Franka Panda). Not validated on physical robots.
• Layouts are fixed: action 10-D (EE position 3 | EE rotation r6 6 | gripper 1), horizon 8; state 11-D (EE position 3 | EE rotation r6 6 | gripper 2). For another robot or layout, fine-tune PointmapVLA-Pretrained instead.
• Requires metric depth with known intrinsics and camera-to-world extrinsics for the pointmap input; the model was trained with pointmaps present. LIBERO images are rotated by 180°, encoded in the datasets as a negative fx.

License

MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed under the Apache License 2.0. See NOTICE in the repository.

Citation

TBD. The paper is under double-blind review; the BibTeX entry and author list will be added at camera-ready.