SO-101 Multi-View Benchmark

Which camera viewpoints does a manipulation policy actually need, how many of them, and from where?

That question is easy to ask and expensive to answer. This benchmark answers it by construction: teleoperate the task once, re-render that single demonstration from every viewpoint you care about, train one policy per camera subset, and let an automated evaluation decide which subset wins.

Why the question is hard on real hardware

Every new camera setup means repositioning and recalibrating the rig, then recording a fresh set of demonstrations. Each configuration therefore rests on different human executions, so the effect of the camera is tangled up with the effect of the operator having a better or worse day. You end up comparing recording sessions, not viewpoints.

Replaying one demonstration removes the confound. Identical motions, object poses and contacts feed every camera configuration, and the camera is the only thing left that differs.

How it works, stage by stage

1. Record the demonstration once

A human drives the arm with a gamepad. The cube is placed on a deterministic grid, so the demonstrations cover the workspace instead of clustering wherever the operator felt comfortable.

Across a 5 by 4 grid that is 540 demonstrations covering 180 distinct cube positions, each seen at three orientations. Seen from the wrist camera, whose framing is identical in every episode, the coverage is the point:

2. Render it from every viewpoint

The recorded trajectory is replayed in the simulator, and all cameras of a scene configuration render in the same pass. Viewpoints are given as an angle and a distance, so adding one is an edit to a JSON file rather than a change to the code.

Each viewpoint is a point on a hemisphere around the workspace: an azimuth, an elevation, a radius. Blue are the positions the grid offers, red the ones this configuration uses, green the point they all look at.

3. Train one policy per camera subset

The dataset carries every viewpoint. Which of them a policy is allowed to see is one flag. Two runs that differ only in that flag are a controlled comparison.

4. Evaluate every policy under one protocol, automatically

This is the part that makes the question answerable at scale. Evaluation restores the recorded starting conditions, runs ten copies of the scene at once, and repeats each cell with a fixed seed so the spread across repeats tells you the measurement’s own noise. A campaign runner applies the identical protocol to every policy, so a difference in success rate is a difference between cameras.

5. Read the answer, not just the videos

Every run is turned into per-episode outcomes and plots without you touching a spreadsheet. Left, the policy attempting the task and failing, then succeeding. Right, the same bin counted up: how far each of its 27 episodes got.

Where a policy holds, and where it stops

A camera set that wins on the positions it was trained on has not proven much. The workspace is split into bins, and which bins a policy trains on decides what a result means.

Inside the training bins, the cube is placed between the positions the policy saw. This measures precision within the distribution it knows.

Whole bins are withheld from training and evaluated afterwards. Success there measures spatial generalisation rather than recall, and the withheld set is spread across corners and interior so both extrapolation and gaps are covered.

Beyond position, the same policy can be re-run under lighting and colour randomisation, so a viewpoint that only works in perfect conditions shows itself.

What you get

• A complete pipeline from teleoperated recording to a written report, every step documented as a command you can paste.

• Deterministic replay that turns one demonstration into any set of viewpoints.

• Policy training on any subset of them, one flag apart.

• Automated evaluation: recorded starting conditions, ten parallel environments, same-seed repeats, one protocol for every policy.

• In-distribution and out-of-distribution regimes, plus randomisation variants.

• Reports that state the spread across repeats instead of a single flattering number.

• Two experiments beyond the standard comparison: degrading one camera at a time to find which one the policy leans on, and moving a camera off its trained pose to see what that costs.

What awaits you

Requirements	What you need. A GPU, and no robot.
Installation	Isaac Sim, LeRobot, this package.
First steps	A five-minute check that it runs.
Running the pipeline end to end	The whole procedure end to end.
Stage 1 — Recording	The five stages, each as a step-by-step recipe.
Coordinate frames	The ideas behind it: frames, viewpoints, test regimes, data layout.
Which camera is the policy actually using?	Studies this pipeline supports beyond the standard run.
Troubleshooting	When something goes wrong.

Scope

Simulation only, on NVIDIA Isaac Lab with the SO-ARM101, a low-cost open-source arm. Nothing here deploys to physical hardware, and no robot is required to run it. The shipped task is picking up a cube, defined as an ordinary Isaac Lab environment that you can replace.