Running the pipeline end to end

Running the pipeline end to end#

The complete procedure across all five stages, from demonstrations to an evaluation report. Each step links to the page documenting it in detail. For a minimal install check, see First steps.

Before you start#

Hardware, software versions and the calibration references are in Requirements. Two things matter for the run itself: the simulator stages have to run one at a time, because concurrent Isaac Sim processes exhaust GPU memory and an aborted one can hold its allocation until it is killed, and a full recording grid is hours of manual work.

1. Environment#

cd so101-camera-multiview-bench
source .venv/bin/activate
unset PYTHONPATH
export PYTHONNOUSERSITE=1
export OMNI_KIT_ACCEPT_EULA=YES

Those three lines belong in every shell that runs a simulator command. They keep the machine’s globally installed Python packages out of this environment. PYTHONPATH is usually set by a sourced ROS setup and would pull ROS packages in, PYTHONNOUSERSITE stops packages under ~/.local from shadowing the ones you installed here, and OMNI_KIT_ACCEPT_EULA answers the licence prompt that would otherwise stall a headless run. Installation explains each one, and covers installing Isaac Sim and PyTorch cu128.

Verify the environment with:

python -m so101_mvbench.tools.list_envs   # prints the LiftCube-* task IDs

2. Get demonstrations#

Record new demonstrations (the path that produces new data):

python -m so101_mvbench.recording.teleop_recorder \
    --task LiftCube-Sim --action_source gamepad \
    --num_episodes 27 \
    --spawn_bins --spawn_bin_col 1 --spawn_bin_row 1 \
    --repo_id local/bin_c1_r1 --repo_root datasets/01_raw_gamepad \
    --task_name 'lift the cube' --seed 42

Guide the arm to the cube, lift it, and press Y to return to the home pose. Repeat per bin (the recorder appends bin_c<col>_r<row> to the root itself). Controls and constraints: Stage 1 — Recording.

Or use an existing dataset. Any LeRobot v3 dataset with the camera streams and meta/*_scene_state.json files works from stage 3 onward. The reference corpus is on the Hugging Face Hub as ill337/so101-bin-lift-6cam-cylroom; how to download a pinned revision is in Data layout.

Check a recorded bin before spending GPU time on it:

# adjust: BIN_ID counts row-major from 1, so column 1 / row 1 on a 5-column
# grid is 1*5 + 1 + 1 = 7. The tool prints which bin it resolved to.
BIN_ID=7

python -m so101_mvbench.tools.validate_bin \
    --bin_id "$BIN_ID" --dataset_root datasets/01_raw_gamepad/

3. Render the camera views and fuse#

# Extract per-episode trajectories (CPU-only)
for bin in datasets/01_raw_gamepad/bin_c*_r*; do
    python -m so101_mvbench.recording.trajectory_extractor \
        --dataset_root "$bin" --output_dir "$bin/trajectories"
done

# Render all cameras of each bin in one replay pass
for bin in datasets/01_raw_gamepad/bin_c*_r*; do
    python -m so101_mvbench.recording.multicam_replay \
        --dataset_root "$bin" --episodes all \
        --scene_config src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json \
        --output_base datasets --mode create --headless
done

# Concatenate the per-bin datasets into one training dataset
export TMPDIR="datasets/04_fused/.tmp"
mkdir -p "$TMPDIR"   # nothing creates it, and tempfile does not either
python -m so101_mvbench.assembly.bin_fusion \
    --datasets_root datasets/lift_cube_6cam_cylroom \
    --target datasets/04_fused/my_corpus \
    --bins bin_c1_r1 bin_c2_r1

The camera set is defined in the scene-config JSON — changing it is an edit there, not a code change. Append mode, the TMPDIR pitfall, and a known encoder deadlock: Stage 2 — Assembly.

4. Train a policy#

python -m so101_mvbench.training.train_act_camera_perm \
    --cameras wrist front \
    --dataset my_corpus \
    --dataset_root datasets/04_fused/my_corpus \
    --steps 65000 \
    --output_dir outputs/2cam_wrist_front

--cameras selects this policy’s subset from the streams the dataset provides — the variable a viewpoint comparison manipulates. Holdout bins are excluded via --episodes. Details: Stage 3 — Training.

5. Evaluate#

One policy, one bin, in-distribution:

python -m so101_mvbench.evaluation.async_eval \
    --mode async --num_envs 10 --num_episodes 27 \
    --policy_path outputs/2cam_wrist_front/checkpoints/065000/pretrained_model \
    --scene_config src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json \
    --dataset_root datasets/01_raw_gamepad/bin_c1_r1 \
    --output_dir outputs/2cam_wrist_front/eval_cylroom_async_bin_c1_r1/run01 \
    --seed 42 --settle_steps 130 --full_report --headless

For a comparison between camera configurations, use the campaign orchestrator instead of repeating the command by hand — it applies one protocol to every policy and is resume-safe:

POLICY_ROOT=outputs DATASETS_ROOT=datasets/01_raw_gamepad OUT_ROOT=outputs/campaign \
POLICIES="2cam_wrist_front 2cam_wrist_left" \
bash scripts/run_campaign.sh

DRY_RUN=1 prints the exact per-policy commands without booting Isaac. Protocol rationale, OOD variants, and what to check after a killed run: Stage 4 — Evaluation.

Warning

Redirect Isaac Sim output to a file — never pipe it through tail or grep. SIGPIPE kills the simulator instantly. tail -F on the log file afterwards is fine.

6. Report#

python -m so101_mvbench.evaluation.report \
    --dir outputs/2cam_wrist_front/eval_cylroom_async_bin_c1_r1/run01

python -m so101_mvbench.evaluation.aggregate_report \
    --run_dirs outputs/2cam_wrist_front/eval_cylroom_async_bin_c1_r1/run* \
    --output_dir outputs/2cam_wrist_front/eval_cylroom_async_bin_c1_r1/aggregate

CPU-only and repeatable. The aggregate gives mean ± std across the repeats. That spread measures floating-point non-determinism rather than seed variation, so report both together. Available plots: Stage 5 — Reporting.

Expected results#

Stage

Artefacts

Runtime

Completed correctly when

1 Recording

01_raw_gamepad/<bin>/ — parquet, MP4s, a scene_state.json per episode plus one look-ahead, binmap.json

manual, 1–2 min per episode

validate_bin passes for that bin’s ID

2 Replay + fusion

lift_cube_6cam_cylroom/<bin>/ with one MP4 stream per camera, then 04_fused/<name>/ with MANIFEST_fusion.json and validation_proof/

about 20 min per bin, fusion minutes on CPU

The episode count matches the source and the pixel-diff evidence in validation_proof/ shows no deviation

3 Training

checkpoints/<step>/pretrained_model/ and training_state/

under 2 h for 65 k steps, two cameras

Loss drops steeply and then flattens, and checkpoints appear at the save frequency

4 Evaluation

results_async.csv and timing_async.json, plus eval_summary.json, the per-episode CSV, quality_gate.json and videos when --full_report is set

minutes per run, hours for a campaign

Episodes reach 2_lift or 3_home rather than staying at 0_miss, and the videos show the gripper closing on the cube

5 Reporting

report/ with plots and report.md, plus aggregate/ across repeats

seconds

The reported success rate matches what the videos show, and the cross-run spread stays within a few percentage points

Two frequent failure modes:

  • Every episode ends 0_miss, the gripper closes on empty space. The scene state was not restored — check that --dataset_root points at the recording bin (the directory with binmap.json and meta/).

  • Training loss decreases but success stays near zero. Usually a camera mismatch: the policy was trained on streams the evaluation scene config does not provide. Compare --cameras against the scene-config camera names.

Success rates are interpretable only with their conditions. Report episode count, evaluated bins, checkpoint step, and number of repeats alongside every number.