# Stage 1 — Recording
You drive the simulated SO-ARM101 to a cube with a gamepad, lift the cube, and
send the arm back to its home pose. One such cycle is one *episode*. This is
the only stage with a human in the loop. Everything downstream replays what
you did here. It is also the stage that bounds every result later on:
imprecise teleoperation cannot be repaired by more training steps or more
cameras.
```{raw} html
```
```{mermaid}
flowchart LR
OP[You + gamepad] -->|teleop| SIM[Isaac Sim LiftCube-Sim]
SIM -->|obs| REC[teleop_recorder]
REC -->|parquet + MP4| DS[(datasets/01_raw_gamepad/<bin>/)]
BIN[BinSpawner bin grid] --> SIM
```
## How to use
### 1. Decide which bin to record
The cube is not placed randomly. The workspace is divided into a grid of
*bins*, and within one bin the spawner walks a 3 × 3 sub-grid of placements
combined with 3 yaw angles (0°, 22.5°, 45°), which gives 27 episodes per bin.
The grid is `--spawn_n_cols` × `--spawn_n_rows`, 5 × 5 by default. The
reference corpus used 5 × 4, so 20 bins and 540 episodes.
```{image} ../_static/diagrams/bin_grid_in_scene.png
:alt: The bin grid spawned as coloured markers on the table, seen past the arm, with four corner bins and the centre bin labelled
:width: 55%
:align: center
```
The grid rendered into the scene, with the marker colour distinguishing bins
and nothing else. Every cube in every episode lands on one of these positions.
The labels are the naming you pass on the command line: `c0` is the column
furthest from the operator, `r0` the row furthest from the arm.
One bin is 27 episodes, and they are not 27 random placements:
```{image} ../_static/diagrams/bin_anatomy.png
:alt: One bin drawn as nine spawn positions, each with three yaw arrows, beside a decoder showing the cube footprint at each yaw
:width: 100%
```
Nine positions in a 5 × 5 cm square, three cube orientations at each, drawn
here for one bin of one evaluation run. The right panel decodes the arrows:
each is one episode's cube yaw, and the overlapping squares show how much the
cube footprint actually turns between 0° and 45°.
Episode by episode, over the whole corpus, that looks like this from the wrist
camera:
```{raw} html
```
You record one bin per session and pick it by column and row:
```bash
--spawn_bins --spawn_bin_col 1 --spawn_bin_row 1
```
### 2. Start the recorder
```bash
python -m so101_mvbench.recording.teleop_recorder \
--task LiftCube-Sim \
--action_source gamepad \
--num_episodes 27 \
--spawn_bins --spawn_bin_col 1 --spawn_bin_row 1 \
--repo_id local/bin_c1_r1 \
--repo_root datasets/01_raw_gamepad \
--task_name 'lift the cube' \
--seed 42
```
The recorder appends `bin_c
_r` to `--repo_root` itself, so the
episodes land in `datasets/01_raw_gamepad/bin_c1_r1/`.
No gamepad at hand? `--action_source sine --auto_record` drives a scripted
motion instead, which is enough to check that the stage runs.
:::{warning}
Send the simulator's output to a file, never through a pipe:
`> record.log 2>&1`. A pipe gives Isaac Sim a `SIGPIPE` and kills it
instantly. Reading the file with `tail -F` afterwards is fine.
:::
### 3. Drive the arm
```{image} ../_static/diagrams/gamepad_mapping.png
:alt: Gamepad mapping used during simulated teleoperation
:width: 75%
```
The sticks drive four arm joints, the shoulder buttons roll the wrist, the
triggers open and close the gripper. The mapping lives in
`vendor/gamepad_utils/configs/so101_lerobot_teleop.json`, so you can rebind it
without touching code.
### 4. End every episode with the home pose
Press **Y** when the cube is lifted. This runs a 1.5 s glide back to the
neutral pose while holding the gripper closed, so the cube does not drop.
Do this every single time. The home pose is the trajectory end the policy
learns, and it is what the evaluator's final stage checks. Episodes that stop
at arbitrary poses teach the policy to stop at arbitrary poses, and the
evaluation's success criterion then measures noise.
### 5. Control episodes from the keyboard
| Key | Effect |
|-----|--------|
| `S` | Start, or stop, recording |
| `R` | Reset the world (also stops recording) |
| `C` | Cancel the running recording |
:::{warning}
After `RECORDING STOPPED` appears, let the process finish on its own. Saving
runs on the simulator's main thread and needs up to a minute or so to flush a
long episode. A kill in the middle leaves a parquet file without its footer,
and `pyarrow` cannot reopen it.
:::
### 6. Extract the trajectories
Stage 2 does not read the LeRobot dataset directly. It replays per-episode
action trajectories, so convert them once (CPU only, no simulator):
```bash
python -m so101_mvbench.recording.trajectory_extractor \
--dataset_root datasets/01_raw_gamepad/bin_c1_r1 \
--output_dir datasets/01_raw_gamepad/bin_c1_r1/trajectories
```
Repeating this is harmless: it overwrites the `.npy` files with identical
content. For all bins at once:
```bash
for bin in datasets/01_raw_gamepad/bin_c*_r*; do
python -m so101_mvbench.recording.trajectory_extractor \
--dataset_root "$bin" --output_dir "$bin/trajectories"
done
```
### 7. Check what you recorded
```bash
# adjust: BIN_ID identifies the bin you recorded. It counts row-major from 1,
# so with a 5-column grid, column 1 / row 1 is 1*5 + 1 + 1 = 7
BIN_ID=7
python -m so101_mvbench.tools.validate_bin \
--bin_id "$BIN_ID" --dataset_root datasets/01_raw_gamepad/
```
The tool prints which bin it resolved the ID to, so check that line before
reading the verdict. It reads the parquet files, counts the MP4s, checks that
every episode has a scene state, and verifies that the cube positions really
fall inside that bin. It exits non-zero on failure.
One caveat at small scale: it expects two MP4s per successful episode. Very
short test episodes end up sharing one video file (see below), and it then
reports fewer MP4s than expected even though the recording is fine.
To watch the episodes, open them in the [Rerun](https://rerun.io/) viewer.
It needs `rerun-sdk >= 0.31` for the LeRobot v3 loader. Install that in a
separate environment if your main one pins an older version:
```bash
rerun datasets/01_raw_gamepad/bin_c1_r1/
```
## What you get
```bash
datasets/01_raw_gamepad/bin_c1_r1/
├── meta/
│ ├── info.json # LeRobot v3 dataset metadata
│ ├── episodes/chunk-000/file-000.parquet
│ └── episode_000000_scene_state.json # one per episode, plus one look-ahead
├── data/chunk-000/file-000.parquet # observations + actions
├── videos/observation.images./chunk-000/ # one MP4 per episode
├── trajectories/episode_000.npy # after step 6, plus trajectories_meta.json
└── binmap.json # which bin, and how far you got
```
Three properties of this output matter downstream:
- **The data says which scene it was recorded in.** `binmap.json` and every
`scene_state.json` carry `workspace_origin_m`. Replay and evaluation read
that value instead of assuming a scene, which is what lets a recording be
replayed somewhere else ({doc}`../concepts/coordinate_frames`).
- **`scene_state.json` is not optional.** It holds the cube pose at episode
start. Without it, replay puts the cube somewhere else and the gripper
closes on empty air. Note that a finished recording holds one more of these
files than it has episodes: the recorder writes the state of the *upcoming*
episode at each reset, so the last one describes an episode that was never
recorded. Count episodes from `binmap.json`, not from these files.
- **Videos are capped at 1 MB per file** via
`update_chunk_settings(video_files_size_in_mb=1)`, instead of LeRobot's
default of 200 MB. A full-length episode exceeds that cap on its own, so in
practice each episode gets its own MP4 and stays individually inspectable.
Very short episodes fit together and then share one file.
## Settings worth knowing
| Flag | Default | What it does |
|------|---------|--------------|
| `--task` | — | Gym task ID. `LiftCube-Sim` for recording. |
| `--action_source` | — | `gamepad` for teleoperation, `sine` for a scripted test motion. |
| `--num_episodes` | — | How many episodes to record (27 = one full bin). |
| `--repo_id` / `--repo_root` | — | Dataset ID and local root. The bin directory is appended for you. |
| `--spawn_bins` | off | Use the deterministic bin grid instead of random placement. |
| `--spawn_n_cols`, `--spawn_n_rows`, `--spawn_bin_size` | 5 / 5 / 0.05 | Grid geometry, bin size in metres. |
| `--spawn_ood_bins` | empty | Spawn on the out-of-distribution grid. A test mechanism: for production, record the full grid and decide the split later. |
| `--scene_config` | cylindrical room | Record in a different scene. The scene's workspace origin is written into the dataset. |
`--help` prints the full list without booting the simulator.
## When something goes wrong
See {doc}`../troubleshooting`, which collects the failures of every stage in one place: a parquet that will not open, a silent console, `validate_bin` naming the wrong bin.
## Where to go next
- {doc}`02_assembly` — turn these trajectories into multi-camera datasets
- {doc}`../concepts/data_layout` — the on-disk format in detail
- {doc}`../concepts/coordinate_frames` — how to read the positions