Example: teach FLUX 3 Action to fly a drone

Use the fine-tuning recipe to teach a simulated drone to follow flight instructions. Each episode has an instruction and each seed a room layout. The model receives a 256×256 onboard image, its last action, and the instruction. The adapter resizes the image to a 512 × 512 model canvas. This page describes the reported Isaac Sim experiment and its model-side data adapter. The release does not include the scene assets, pilot, recordings, or evaluation runner. An exact reproduction needs those components, including the simulator version, physics settings, and success criteria.

”Fly under the coffee table and hover.” Tested on a room layout and instruction wording absent from the fine-tuning data.

Simulation


World	A living room in Isaac Sim: sofa, coffee table, bookcase, window, landing pad, clutter. Sofa and bookcase against a random wall or free-standing, pad and clutter moved, the drone facing a random direction at take-off. Layout changes per seed
Onboard camera	256×256, altitude and heading drawn onto the frame
Action	`[forward, lateral, up, yaw]` in `[-1, 1]`, velocity commands with first-order lag, 15 Hz
Instructions	Instructions naming destinations: “take off, fly under the table, and hover”, “take off, fly to the sofa, and hover”, “take off, fly to the pad, and land”
Expert	Scripted pilot with a map: take off, yaw to bearing, fly waypoints from a visibility graph, hover, land

What changed in the recipe

The rotor dataset adapter reads an instruction per episode and prepends fly the drone: . Use the same prefix at inference. The recipe uses four action dimensions, a 512 × 512 canvas, and 3,000 updates. Start with the local game config, then create a drone-only config:

import json
from pathlib import Path

config = json.loads(Path("configs/games/local.json").read_text())
config["source_root"] = "rotor=outputs/data/rotor"
config["output_dir"] = "outputs/drone"
Path("configs/games/drone-local.json").write_text(json.dumps(config, indent=2))

The remaining game settings, including caption_dropout=0.0, stay unchanged. The directory outputs/data/rotor needs an index such as:

{
  "length": 300,
  "action_dim": 4,
  "episodes": [
    {"file": "ep_0000.npz", "task": "take off, fly under the table, and hover"},
    {"file": "ep_0001.npz", "task": "take off, fly to the pad, and land"}
  ]
}

Each NPZ contains uint8 RGB frames with shape (300, 256, 256, 3) and float32 action with shape (300, 4). The unprefixed task records the instruction followed by the pilot. The adapter prepends fly the drone: once. Two entries illustrate the format; supply enough episodes for the training batch topology. Use the fine-tuning guide’s data checks and short validation run with the new config, then train and export:

uv run torchrun --nproc_per_node 8 -m flux_action.cli train \
  --config configs/games/drone-local.json
uv run flux-action export-checkpoint --checkpoint outputs/drone/step-3000 \
  --output outputs/drone-export --profile ema_0p10 --dtype bfloat16

This launch uses the current eight-GPU configuration. It is not evidence of the original drone experiment’s hardware or wall time, which the notes do not specify. 800 episodes of 20 seconds at 15 Hz, recorded from the scripted pilot. Five scenarios: standard start, random start, already airborne, near an obstacle, and mid-flight disturbances.

Held-out layouts, unseen wording

Evaluation uses ten layouts excluded from fine-tuning and reworded instructions. For example, “take off, fly to the bookcase, and hover” becomes “go over to the bookshelf and wait there”. The simulator waits for each plan across 89 evaluation flights:

Evaluation instruction	Fine-tuning instruction	Completed
fly to the window and hover in front of it	take off, fly to the window, and hover	10 / 10
take off and land on the pad	take off, fly to the pad, and land	9 / 10
fly under the coffee table and hover	take off, fly under the table, and hover	8 / 10
go over to the bookshelf and wait there	take off, fly to the bookcase, and hover	7 / 10
fly to the bookcase and hover in front of it	take off, fly to the bookcase, and hover	6 / 10
fly around the coffee table and hover on the far side	take off, fly behind the table, and hover	5 / 10
go to the couch	take off, fly to the sofa, and hover	5 / 10
fly to the sofa and hover in front of it	take off, fly to the sofa, and hover	3 / 9
fly to the coffee table and hover above it	take off, fly to the table, and hover	2 / 10

The model completed 55 of 89 flights, with 2 crashes. Success was highest for the window, landing pad, and under-table tasks, and lowest for the sofa and above-table tasks. “Bookshelf” and “couch” were absent from the fine-tuning instructions.

”Fly to the window and hover in front of it.” Completed.

”Go over to the bookshelf and wait there.” Completed; “bookshelf” was absent from the fine-tuning instructions.

”Go to the couch.” Completed.

”Fly to the coffee table and hover above it.” One of two successful flights in ten attempts.

”Fly to the bookcase and hover in front of it.” Failed to reach the bookcase.

Run the model

Load the drone export and supply its instruction, image, and previous action:

import numpy as np
import torch
from flux_action.policy import FluxActionPolicy

policy = FluxActionPolicy.from_pretrained("outputs/drone-export", device="cuda")
frame = np.load("drone-frame.npy")  # (256, 256, 3), uint8 RGB with altitude/heading overlay
last_action = torch.zeros(1, 4, device="cuda")  # episode start only
observation = {
    "images.game": torch.from_numpy(frame).permute(2, 0, 1)[None].to("cuda").float() / 255,
    "state": last_action,
    "task": ["fly the drone: go over to the bookshelf and wait there"],
}
with torch.inference_mode():
    plan = policy.predict_action_chunk(observation)  # (1, 32, 4)
forward, lateral, up, yaw = plan[0, 0].clamp(-1, 1).tolist()

images.game is the adapter’s camera key for all three example environments, including the drone. Image resizing occurs in the policy. To close the loop, execute your chosen number of actions, store the last command actually sent as state, capture a new onboard image, and predict again. Reset policy state and the previous-action vector at episode boundaries. Choose the execution horizon explicitly; the source notes do not establish which horizon produced the reported 55/89 result. The four channels are normalized movement commands. The source notes do not specify their physical speed limits, yaw units, or coordinate transforms. Your simulator adapter must apply the same conventions used by its demonstration pilot. Preserve the altitude/heading overlay and action dynamics at evaluation.

Evaluate

The reported simulator paused for inference. Record whether yours does too, along with layout seeds, instructions, execution horizon, and checkpoint. Define completion distances, hover duration, landing conditions, collisions, and time limits in your evaluator. The release does not supply these thresholds or a full breakdown of unsuccessful flights.

Get Started

Account Management

FLUX 3

FLUX.2

FLUX Tools

API Integration

Training

Legacy Models

Example: teach FLUX 3 Action to fly a drone

Simulation

What changed in the recipe

Held-out layouts, unseen wording

Run the model

Evaluate

​Simulation

​What changed in the recipe

​Held-out layouts, unseen wording

​Run the model

​Evaluate

Simulation

What changed in the recipe

Held-out layouts, unseen wording

Run the model

Evaluate