Overview

Overview#

A framework for camera-viewpoint experiments in imitation learning with the low-cost SO-ARM101 manipulator in NVIDIA Isaac Lab.

The question it addresses is which viewpoints a manipulation policy requires. Studying this on physical hardware is expensive: every camera configuration requires repositioning and re-calibrating the rig, and re-recording the full set of demonstrations. Each configuration therefore rests on different demonstrations, which confounds the viewpoint effect with variation in the operator’s execution.

This framework separates the two. The task is teleoperated once, and the recorded trajectory is then re-rendered from an arbitrary set of viewpoints. Policies trained on different camera subsets consequently share identical motions, object poses and physics, so the camera configuration remains the only variable.

Method#

        flowchart LR
    A[Gamepad teleop<br/>N episodes per bin] --> B[Trajectory<br/>extraction]
    B --> C[Multi-camera replay<br/>all views in one pass]
    C --> D[Multi-bin fusion<br/>one training dataset]
    D --> E[ACT training<br/>any camera subset]
    E --> F[Sim evaluation<br/>ID + OOD variants]
    F --> G[Reporting<br/>per-run + cross-run plots]

Each stage runs as a Python module (python -m so101_mvbench.<module>, see Command overview) and is documented on its own page, beginning with Stage 1 — Recording.

Two properties establish the internal validity of the comparison:

  • Shared demonstrations. All camera streams are rendered post-hoc from the recorded action trajectory. Policies trained on different camera subsets are therefore exposed to identical motions, object poses and contact dynamics.

  • Fixed evaluation protocol. Episodes are initialised from deterministic scene states, repeated runs use a fixed seed, and a campaign runner applies identical settings across policies. Differences in success rate are consequently attributable to the camera configuration rather than to run conditions.

Applications#

  • Comparison of camera subsets (wrist only, wrist plus external, multi-view) on one task.

  • Generalisation studies: withholding cube-position bins, or evaluating under the provided out-of-distribution variants (position grid, lighting and colour randomisation).

  • Ablation of any parameter exposed by the scene configuration: viewpoint angles, camera count, scene layout.

  • Standalone use of individual stages. Recorder, fusion tool, evaluator and report generator operate independently of one another.

Outside the intended scope: deployment on physical hardware (the pipeline is simulation-only), policy architectures other than ACT, and tasks other than the provided LiftCube. The task is defined as a standard Isaac Lab environment configuration and can be substituted.

Requirements#

A CUDA-capable GPU, Isaac Sim and no robot. The full list, with the versions this was verified against, is in Requirements.

Scope of this documentation#

This site documents the package so101_mvbench. The joystick interface it uses (gamepad_utils) is vendored inside the package under so101_mvbench.vendor — there is no separate dependency to install.

Proceed with Installation, then Running the pipeline end to end for the complete end-to-end procedure.

Developed within an academic research project on viewpoint selection in imitation learning.