Data layout

Data layout#

Every dataset in this pipeline is a LeRobot v3 dataset: parquet for actions and proprioceptive state, MP4 per camera stream, and a meta/ directory describing both. What differs between the top-level directories is provenance, and the naming encodes it.

Top-level structure#

datasets/
├── 01_raw_gamepad/<bin>/            # stage 1 — human teleoperation, the only non-synthetic data
├── lift_cube_6cam_cylroom/<bin>/    # stage 2b — per-bin multi-camera replay
└── 04_fused/<merged_name>/          # stage 2c — trainer-ready, several bins concatenated

The replay directory is named <scene_config_stem>_cylroom by multicam_replay, so the camera set is readable from the path.

Raw recording — `01_raw_gamepad/<bin>/`#

datasets/01_raw_gamepad/bin_c1_r1/
├── meta/
│   ├── info.json                        # LeRobot v3 dataset metadata
│   ├── episodes/chunk-000/file-000.parquet
│   └── episode_000000_scene_state.json  # cube pose + workspace_origin_m, one per episode
├── data/chunk-000/file-000.parquet      # observations + actions
├── videos/observation.images.<camera>/chunk-000/
├── trajectories/episode_000.npy         # extracted action trajectories (stage 2 input)
└── binmap.json                          # bin progress + workspace_origin_m

This is the only directory a human produced. Everything else is derived from trajectories/ plus meta/*_scene_state.json.

Per-bin replay — `lift_cube_6cam_cylroom/<bin>/`#

datasets/lift_cube_6cam_cylroom/bin_c0_r0/
├── meta/
│   ├── info.json
│   └── scene_config.json               # archived camera set — guards --mode append
├── data/chunk-000/file-000.parquet
├── videos/observation.images.{wrist,front,left,right,top,back}/chunk-000/
└── timing.json                         # per-phase wall clock of the replay

One directory per bin, 27 episodes each in the reference setup.

Fused training dataset — `04_fused/<merged_name>/`#

datasets/04_fused/merged_6cam_cylroom/
├── meta/, data/, videos/
├── MANIFEST_fusion.json                # which bin contributed which episode range
└── validation_proof/                   # pixel-diff evidence from the merge

MANIFEST_fusion.json is the authoritative bin-to-episode map — the file to read when translating “hold out bin bin_c0_r0” into episode indices, and when checking whether an evaluated bin was part of training.

Getting the reference corpus#

The corpus used during development (merged_6cam_cylroom, 540 episodes, 20 bins × 27, six cameras) is published on the Hugging Face Hub as ill337/so101-bin-lift-6cam-cylroom. Its dataset card lists the sampling design, the held-out bins of the thesis split and a repair of six episodes made after the thesis policies were trained.

REPO_ID=ill337/so101-bin-lift-6cam-cylroom
# the commit this page was written against; replace it with a newer commit or tag
# from the repository's history if you want later changes
REVISION=63395a482e1d2436f617083257165ad75ecc346f

huggingface-cli download "$REPO_ID" \
    --repo-type dataset \
    --revision "$REVISION" \
    --local-dir datasets/04_fused/merged_6cam_cylroom

The earlier 538-episode fusion of the same task in a textured room scene is ill337/so101-bin-lift-6cam. It is not the corpus described on this page.

Always pull and cite a pin (--revision with a tag or commit), never the bare repository name. Hub repos can be renamed, extended, or rewritten — a reproduction would then silently run on different data.

What you can rely on#

Videos are capped at 1 MB per file, set by update_chunk_settings(video_files_size_in_mb=1) after dataset creation, against LeRobot’s default of 200 MB. A full-length episode passes that cap on its own, so each episode normally lands in its own MP4 and stays individually inspectable. Episodes short enough to fit together share a file.
Recorded data declares its frame. workspace_origin_m in binmap.json and every scene_state.json names the scene the episodes were captured in. A dataset without the field is read as the retired floor-level table scene. Mixing frames in one batch is refused loudly. See Coordinate frames.
scene_state.json is mandatory for replay. Cube positions vary per episode, so without scene-state restoration the gripper closes on empty air.
The training data is fully synthetic. All camera streams, wrist included, were rendered during replay — only the action trajectory is human. See Stage 2 — Assembly.

Inspecting a dataset#

rerun datasets/04_fused/merged_6cam_cylroom/

Streams the episodes into the Rerun viewer through its native LeRobot v3 loader. This needs rerun-sdk >= 0.31, because older SDKs cannot read v3 datasets.

Data layout

Contents