SO-101 with LeRobot

Adapt the prepared SO-101 checkpoint with LeRobot. LoRA updates the trunk while the action and conditioning heads train in full. Keep the base checkpoint and its saved processors available for training, resume, and inference. The clips below show real robot demonstrations. Playback is accelerated and pauses between plans are removed; they are not latency benchmarks or validation of every later integration revision.

”Put the blue box into the container.” Trained with about 200 teleoperated demonstrations. Top camera, 4× speed, pauses between plans cut.

”Put the white box from the green cup to the container."

"Put the screwdriver into the container."

"Put the tissues into the container.”

These objects were not included in the task-specific pick demonstrations.

Requirements

Linux, Python 3.12, and an NVIDIA GPU with a CUDA 12.8-compatible driver.
Access to the model weights and the LeRobot integration repository below.
A calibrated SO-101 and two cameras for live rollout.
Your own LeRobot v3 dataset at 30 Hz, with six absolute commanded actions, six measured-state values, and task instructions.

Joint order is shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, and gripper. The delivered standalone SO-101 profile uses degrees for the five arm joints and percentage points for the gripper. Match the LeRobot package’s saved processors, joint order, units, and calibration; do not normalize twice. The SO-101 recipe uses 30 Hz; DROID and the games use 15 Hz. The pinned checkpoint predicts 42 actions and executes the first 32, about 1.07 seconds at 30 Hz, before replanning. LeRobot executes the number of actions specified by the downloaded package’s n_action_steps.

Install the pinned integration

Use a separate environment from standalone flux-action. The model repository pins this LeRobot integration revision:

git clone https://github.com/huggingface/collab-lerobot-nv-bfl-flux3.git
cd collab-lerobot-nv-bfl-flux3
git checkout c027ee363363f8d86767c66786c6891478564d6c
uv venv --python 3.12
source .venv/bin/activate
uv pip install 'torch==2.11.0' 'torchvision==0.26.0' \
  --index-url https://download.pytorch.org/whl/cu128
uv pip install -e '.[training,flux3,peft,diffusion]'
uv pip install 'natten==0.21.6+torch2110cu128' --find-links https://whl.natten.org/
ffmpeg -version

Install FFmpeg with video decoding support if the last command is unavailable. The NATTEN wheel above is for PyTorch 2.11/CUDA 12.8; choose a matching wheel if you change that pair. Run subsequent commands from this checkout with its virtual environment active.

Download the base

Use the released packages from the FLUX 3 Action collection rather than the integration’s older temporary download helper. Download the SO-101 policy, then the shared encoders at the base revision its config references:

hf auth login
hf download black-forest-labs/flux-3-action-so101 \
  --revision c9e13b2aca6a0a472b3ea03fd90cff32d3e85849 \
  --local-dir models/flux-3-action-so101
hf download black-forest-labs/flux-3-action-base \
  --revision 62878e2925e59b7a89ec14463ce89932624c490d \
  --include 'video_vae.safetensors' --include 'text_encoder/*' \
  --local-dir models/flux-3-action-base
policy_dir="$(pwd)/models/flux-3-action-so101"
encoders_dir="$(pwd)/models/flux-3-action-base"

The SO-101 directory contains the policy, saved processors, and normalization statistics. Keep both directories available for training, resume, and adapter inference. Supply your own demonstrations; these downloads do not include them. Inspect the package’s settings:

import json
from pathlib import Path

root = Path("models/flux-3-action-so101")
config = json.loads((root / "config.json").read_text())
for key in ("camera_keys", "camera_layout", "fps", "chunk_size", "n_action_steps"):
    print(key, config.get(key))
assert config["fps"] == 30

Train with the integration’s examples/flux3/lora.json. Keep the package’s normalization statistics fixed for the adapter. Different statistics require a separately prepared base and matching processors, not a substitution during rollout.

Camera keys

Physical camera	Checkpoint key	Position in the model image
Fixed scene camera	`observation.images.scene`	Left
Wrist-mounted camera	`observation.images.wrist`	Right

Each view is resized to 256 × 256 and joined into a 512 × 256 image (width × height). Supply separate streams. Old instructions using top and front as checkpoint keys do not describe this package. If your dataset calls the scene camera top and the wrist camera gripper_cam, append this argument to the training command:

--rename_map='{"observation.images.top":"observation.images.scene","observation.images.gripper_cam":"observation.images.wrist"}'

Use the same physical mapping at rollout. A rename map changes keys; it cannot identify a swapped camera. Inspect the composed image before VAE encoding: scene on the left, wrist on the right.

Train

The shared preset trains on one GPU with batch 2 and accumulation 4:

python -m lerobot.scripts.lerobot_train \
  --config_path=examples/flux3/lora.json \
  --policy.path="$policy_dir" --policy.device=cuda \
  --policy.video_vae_id="$encoders_dir/video_vae.safetensors" \
  --policy.text_encoder_id="$encoders_dir/text_encoder" \
  --dataset.repo_id=YOUR_ORG/YOUR_SO101_DATASET \
  --output_dir=outputs/so101_lora

Replace the dataset ID and add the camera rename map only if needed. Confirm the loaded settings before starting: rank/alpha 32, adapter/head learning rates 1e-4/5e-4, BF16, gradient checkpointing, and EMA decay 0.999. New adapters are created through the top-level peft config. The flag --policy.use_peft=true is for loading an existing adapter. The preset uses effective batch 8 and steps=10000. LeRobot counts steps, save_freq, and eval_steps in microsteps: with accumulation 4, that is 2,500 optimizer updates, and 500 microsteps equal 125 updates. Increase the budget or GPU count explicitly for your dataset, and check peak GPU memory on your hardware.

Resume and select weights

lerobot-train \
  --config_path=outputs/so101_lora/checkpoints/last/pretrained_model/train_config.json \
  --resume=true

Resume from the raw checkpoint to restore optimizer and EMA state. For inference, compare pretrained_model with its pretrained_model_ema sibling using the same tasks, starts, and control settings. EMA helps some runs but is not automatically the better SO-101 checkpoint. Retain the adapter’s original base and encoder files.

Roll out

Use the integration’s synchronous runner for a trained adapter. Set the port, calibrated robot ID, camera devices, and instruction for your setup:

lerobot-rollout \
  --strategy.type=base --inference.type=sync \
  --policy.path=outputs/so101_lora/checkpoints/last/pretrained_model \
  --policy.use_peft=true --policy.device=cuda \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101 \
  --robot.use_degrees=true \
  --robot.cameras='{"scene":{"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30},"wrist":{"type":"opencv","index_or_path":1,"width":640,"height":480,"fps":30}}' \
  --fps=30 --task="put the blue box into the container" --duration=30

This executes commands on the arm. Use the robot ID and calibration that match your training data, and confirm the two live camera views before rollout. The base strategy runs without recording or uploading episodes. Swap the checkpoint path to compare raw and EMA adapters. The runner applies the saved processors and resets state between episodes. This policy supports synchronous rollout. For direct Python calls and processor handling, see the integration guide. The clips show earlier real-robot runs. Evaluate task success and latency on your own arm. For full fine-tuning instead of a task LoRA, follow the separate standalone SO-101 workflow.

Get Started

Account Management

FLUX 3

FLUX.2

FLUX Tools

API Integration

Training

Legacy Models

Requirements

Install the pinned integration

Download the base

Camera keys

Train

Resume and select weights

Roll out

​Requirements

​Install the pinned integration

​Download the base

​Camera keys

​Train

​Resume and select weights

​Roll out

Requirements

Install the pinned integration

Download the base

Camera keys

Train

Resume and select weights

Roll out