API reference

Parameters:

available (list[str])
explicit_bins (list[str])
exclude (list[str])

so101_mvbench.assembly.bin_fusion.validate_post_merge(target, expected_episodes, sources)[source]#

Run post-merge checks including video content verification.

Return type:

Parameters:

target (Path)
expected_episodes (int)
sources (list[BinSource])

so101_mvbench.assembly.bin_fusion.validate_pre_merge(sources, strict)[source]#

Run pre-merge checks. Returns list of warnings.

Return type:

Parameters:

sources (list[BinSource])
strict (bool)

so101_mvbench.assembly.bin_fusion.validate_video_content(target, sources)[source]#

Verify merged videos by comparing mid-episode frames against source bins.

LeRobot v3.0 concatenates all episode videos into one MP4 per camera-key per chunk. Episode boundaries are resolved via timestamp indices in the episodes/*.parquet metadata. Ref: https://huggingface.co/docs/lerobot/en/using_dataset_tools

For each source bin, picks 3 sample episodes (first, middle, last) and compares the mid-frame of each episode in the merged video against the same frame in the source video. This catches wrong ordering, corrupted concatenation, or shifted frame indices.

Return type:

Parameters:

target (Path)
sources (list[BinSource])

so101_mvbench.assembly.bin_fusion.write_manifest(target, config, sources, pre_warnings, post_warnings)[source]#

Write MANIFEST_fusion.json with full provenance.

Return type:

Parameters:

target (Path)
config (FusionConfig)
sources (list[BinSource])
pre_warnings (list[str])
post_warnings (list[str])

Training (stage 3)#

ACT trainer wrapper for camera-permutation experiments.

Expands –cameras wrist front into the LeRobot –policy.input_features JSON, then invokes lerobot.scripts.lerobot_train with the rest of the args.

Reduces a typical Phase-1 baseline command from ~10 lines (multi-line bash with embedded JSON) to a single line:

train_act_camera_perm
–cameras wrist front –dataset bin_c1_r1__wrist_front_sideLR_top –steps 30000 –batch_size 8 –output_dir outputs/2cam_wrist_front

Cameras are looked up from _CAMERA_SHAPES (default: 480×640 RGB for all five workspace cameras: wrist, front, side_l, side_r, top).

Output dir contains LeRobot’s standard checkpoints/, train.log, etc.

so101_mvbench.training.train_act_camera_perm.build_input_features(cameras)[source]#

Build the LeRobot policy.input_features dict from a list of cameras.

Parameters:

cameras (list[str]) – list of camera names (e.g. [“wrist”, “front”]). Each must exist as a key in _CAMERA_SHAPES.

Return type:

dict[str, dict]

Returns:

A dict like:

{
  "observation.images.wrist": {"type": "VISUAL", "shape": [3,480,640]},
  "observation.images.front": {"type": "VISUAL", "shape": [3,480,640]},
  "observation.state":         {"type": "STATE",  "shape": [6]},
}

eval_checkpoint_selector.py — Cherry-Shen-Style Best-Checkpoint-Finder.

Evaluates all checkpoints/NNNNNN/pretrained_model/ directories of a training run against a holdout episode set (eval_loss on the holdout subset) and writes the best step + per-step results to <run_dir>/best_checkpoint.json (plus eval_vs_train_loss.png for visual inspection).

Console-script entry: eval_checkpoint_selector (after pip install -e .).

Usage:

eval_checkpoint_selector –run_dir outputs/2cam_wrist_front: –dataset_root datasets/liftcube_binned/bin_c1_r1__wrist_front_sideLR_top

Reads:

<run_dir>/checkpoints/NNNNNN/pretrained_model/ — all checkpoints <run_dir>/holdout_episodes.json — held-out episode indices <run_dir>/train.log (optional) — for train-loss overlay

Writes:

<run_dir>/best_checkpoint.json — {best_step, best_eval_loss, all_results} <run_dir>/eval_vs_train_loss.png — overlay plot (skipped with –no-plot)

Design decisions (verified against Cherry Shen’s so101_bench/scripts/train/train.py and LeRobot 0.5.1 source, 2026-04-19):

policy.train() + deterministic seed: Cherry’s original uses eval(), but LeRobot 0.5.1 modeling_act.py:149 has a bug — the KLD-loss block lacks an and self.training guard. In eval mode mu_hat/log_sigma_x2_hat are None → TypeError in line 155. We use policy.train() (VAE encoder active) combined with torch.manual_seed(seed) before each checkpoint eval to make the VAE reparameterization-sampling deterministic. torch.no_grad() still prevents weight updates.
Preprocessor applied explicitly: LeRobot 0.5.1 moved normalization out of the policy’s forward() into a separate NormalizerProcessorStep (loaded via make_pre_post_processors). Without this, loss values are computed on unnormalized inputs — orders of magnitude larger than train-time loss, and not comparable. See lerobot_train.py:414 batch = preprocessor(batch).
Empty holdout guard: Cherry’s train.py:472 has if eval_dataset is not None. We mirror that: empty holdout list raises ValueError rather than silently producing a zero-frame dataset. LeRobot 0.5.1 LeRobotDataset(episodes=[]) loads ZERO frames (pa_ds.field.isin([]) = empty filter), not “all” — subsequent ds[0] access would raise IndexError. The guard catches the logical error early with an actionable message.

class so101_mvbench.training.checkpoint_selector.RunLayout(layout, holdout_json, ckpt_dir, best_json_out, plot_png_out, train_log)[source]#

Bases: object

Resolved file paths the selector operates on.

For design03: holdout/ckpt/train_log live in the TRAINING capsule; best_json/plot live in the SELECTION capsule (=write target). For legacy/in-place mode: read AND write paths share a common run_dir.

Parameters:

layout (str)
holdout_json (Path)
ckpt_dir (Path)
best_json_out (Path)
plot_png_out (Path)
train_log (Path | None)

__init__(layout, holdout_json, ckpt_dir, best_json_out, plot_png_out, train_log)#

Parameters:

layout (str)
holdout_json (Path)
ckpt_dir (Path)
best_json_out (Path)
plot_png_out (Path)
train_log (Path | None)

Return type:

None

so101_mvbench.training.checkpoint_selector.create_selection_capsule(training_run, dataset_root, selector_args)[source]#

Create a fresh selection capsule under selections/<job>__<train_ts>/<sel_ts>/.

Writes input sidecars (training_run symlink, holdout copy, dataset symlink, selector_args.json) + README.md + processing/ skeleton, then returns the capsule path along with a RunLayout pointing reads at the training capsule and writes at this new selection capsule.

Return type:

tuple[Path, RunLayout]

Parameters:

training_run (Path)
dataset_root (Path)
selector_args (dict)

Reads from training capsule:

input/holdout_episodes.json (copied verbatim — selection capsule must be self-contained per schema)
output/checkpoints/ (read-only, never moved)
processing/train.log (auto-discovered for plot overlay)

so101_mvbench.training.checkpoint_selector.detect_run_layout(run_dir)[source]#

Detect whether run_dir is a Capsule (input/proc/output) or Legacy run.

Return type:: RunLayout
Parameters:: run_dir (Path)

Capsule (run_train.sh v3.0.0+, in-place selector mode):: run_dir/ ├── input/holdout_episodes.json ├── processing/train.log └── output/{checkpoints/, best_checkpoint.json, eval_vs_train_loss.png}
Legacy (run_train.sh v2.0.0):: run_dir/ ├── holdout_episodes.json ├── checkpoints/{NNNNNN/, last/pretrained_model/train.log} ├── best_checkpoint.json └── eval_vs_train_loss.png

Decision rule: if run_dir/input/ AND run_dir/output/ both exist, it’s a Capsule. Otherwise it’s Legacy. Mismatch (e.g. only input/ exists) is an error.

so101_mvbench.training.checkpoint_selector.eval_policy_on_dataset(policy, preprocessor, dataset, device, batch_size=32)[source]#

Compute mean forward-loss on a dataset without gradient computation.

policy.train() keeps VAE encoder active (LeRobot 0.5.1 bug: eval-mode forward() crashes on KLD-block with None mu_hat, see modeling_act.py:149)
preprocessor(batch) applied to normalize inputs (LeRobot 0.5.1 API)
torch.no_grad() prevents weight updates
Seed is set BEFORE this function is called, to make VAE sampling deterministic

Return type:

float

Parameters:

dataset (lerobot.datasets.lerobot_dataset.LeRobotDataset)
device (torch.device)
batch_size (int)

so101_mvbench.training.checkpoint_selector.evaluate_all_checkpoints(layout, dataset_root, batch_size, device, seed=42)[source]#

Evaluate every checkpoint in layout.ckpt_dir against the holdout set.

Return type:

tuple[dict, list[dict]]

Parameters:

layout (RunLayout)
dataset_root (Path)
batch_size (int)
device (str)
seed (int)

so101_mvbench.training.checkpoint_selector.parse_train_loss_from_log(log_path)[source]#

Parse LeRobot training log for per-K-step average train-loss.

LeRobot logs lines like:

INFO 2026-04-22 14:21:49 ot_train.py:439 step:30K smpl:241K ep:490 epch:20.41 loss:0.052 ...

The step is K-rounded (floor division by 1000). Each K-bucket contains ~log_freq log entries (e.g. 10 entries per K at log_freq=100). We aggregate loss per bucket via mean so the returned list has one (step, mean_loss) tuple per 1000-step window.

Returns tuples of (step_absolute=K*1000, mean_loss), sorted by step. Unreadable / non-matching log is tolerated — returns [] in that case.

Return type:: list[tuple[int, float]]
Parameters:: log_path (Path)

so101_mvbench.training.checkpoint_selector.write_plot(results, best, plot_path, title, train_data=None)[source]#

Generate an eval-loss-over-steps plot, optionally with train-loss overlay.

Return type:

Parameters:

results (list[dict])
best (dict)
plot_path (Path)
title (str)
train_data (list[tuple[int, float]] | None)

Evaluation (stages 4 and 5)#

Task progress tracker for LiftCube evaluation.

Tracks 4 monotonic progress stages, grasp attempts, and early termination conditions. Designed to be testable without Isaac Sim — accepts raw floats, no torch/Isaac Lab dependency.

Stages (v2.0.0 — post-HomeMacro recording era):

0_miss — Gripper never reached the cube (score=0.0) 1_reach — Gripper within REACH_DIST of cube (score=0.33) 2_lift — Cube peak z > initial_z + LIFT_DELTA_REL (score=0.66); “cube has left the table”, very permissive, peak-based. 3_home — Arm has reached the home pose AND cube still held, sustained for HOME_HOLD_STEPS (score=1.0, final success).

Breaking-change vs v1.0.0:

Stage “2_pick” merged into “2_lift” (same semantic: cube off the table).
LIFT_HEIGHT (absolute 12cm) removed — replaced by relative LIFT_DELTA_REL. Motivation: after training with the Y-button Home-Macro, the cube ends up at an arbitrary z-height in the home pose, not at a fixed 12cm.
HOLD_STEPS/SMOOTH_WINDOW (2s sustained at 12cm) replaced by composite home-reached criterion: arm_at_home ∧ cube_not_dropped, sustained HOME_HOLD_STEPS. The cube MUST still be in the gripper when the arm reaches home — otherwise it is a failed task, not a success.

Early Termination:

ET-1 no_approach — dist_3d not reduced by 50% after 10s (stage<1 guard) ET-2 cube_lost — cube_z dropped below table level (ET_CUBE_LOST_DELTA)

References

Cherry Shen (so101_bench): 5 progress stages for pick-and-place.
DEXTRAH (NVlabs): 2s sustained success timeout.
home_macro._HOME_REACHED_THRESHOLD_RAD: source of truth for REACH_HOME_RAD (8.6°), kept in sync manually.

class so101_mvbench.evaluation.eval_tracker.LiftCubeEvalTracker(initial_cube_z, initial_dist_3d, home_pose_rad, home_pose_tol_rad=None, plateau_steps=None)[source]#

Bases: object

Per-episode eval tracker. Call update() once per sim step.

Parameters:

initial_cube_z (float)
initial_dist_3d (float)
home_pose_rad (list[float])
home_pose_tol_rad (float | None)
plateau_steps (dict[str, int] | None)

__init__(initial_cube_z, initial_dist_3d, home_pose_rad, home_pose_tol_rad=None, plateau_steps=None)[source]#

Parameters:

initial_cube_z (float) – Cube z-coordinate at episode start (m).
initial_dist_3d (float) – Initial 3D gripper-to-cube distance (m).
home_pose_rad (list[float]) – Target home pose for arm joints 0..4 (rad). Gripper (joint 5) is not checked — Y-macro holds it.
home_pose_tol_rad (float | None) – Per-joint max error for arm_at_home detection (rad). If None, falls back to REACH_HOME_RAD (0.15 = 8.6°). Authoritative source: evaluation/config/eval.json target_end_position_tolerance_deg.
plateau_steps (dict[str, int] | None) – Per-stage max-stage-stuck ET threshold dict (e.g. {“1_reach”: 600, “2_lift”: 600}). Falls back to ET_PLATEAU_STEPS_DEFAULT if None. Authoritative source: evaluation/config/eval.json plateau_steps.

property result: dict#: Episode summary dict — maps directly to CSV row fields.

property should_terminate: bool#

True when the episode should stop — either early failure or success.

Early-termination on success is a speed optimization: once Stage 3 (arm_at_home ∧ cube_held, 0.5s sustained) is reached, the remaining frames carry no signal. Stopping saves ~60-70% of eval wall-time for successful episodes. Fail-episodes still run to EPISODE_LENGTH_S to give ET-1 (no_approach @ step 600) its full detection window.

update(cube_z, dist_3d, step, arm_joint_pos=None)[source]#

Feed one sim step.

Parameters:

cube_z (float) – Current cube z-coordinate (m).
dist_3d (float) – Current 3D gripper-to-cube distance (m).
step (int) – Sim step index (monotonic).
arm_joint_pos (list[float] | None) – Current joint positions in rad (>=5-dim). Required for Stage-3 home-reached detection. If None, Stage 3 cannot be reached.

Return type:

list[LiftCubeEvalTracker]

Loader for the shipped evaluation config (evaluation/config/eval.json).

Lives in its own module, free of any Isaac import, so it can be unit-tested and reused. It used to be a private function inside async_eval, which launches Isaac Sim at import time – the reason its old test had to be parked.

The JSON stores degrees because humans wrote it; consumers work in radians, so the conversion happens here, once.

so101_mvbench.evaluation.eval_config.load_eval_config(path=None)[source]#

Read the eval config and convert its angles to radians.

Parameters:

path (Path | None) – Override for tests. Defaults to the shipped package resource.

Return type:

dict

Returns:

Dict with target_end_position_rad (list, one entry per joint), target_end_position_tolerance_rad and plateau_steps.

Raises:

FileNotFoundError – If the config file does not exist.
KeyError – If a required key is missing.

SceneStateLoader — apply per-episode scene_state JSONs to Isaac Sim (single-env).

Rewrite of design00 scene_loader.py without the except Exception: return {} silent-continue anti-pattern. All failures raise with clear messages.

Responsibility#

Load episode_NNNNNN_scene_state.json files from a meta-dir into memory
For each episode: write USD attributes into Isaac Sim (lights, yaws, colors, cube)

Not our job#

Validate JSON schema / ranges / NaN → startup.py did that already
Own the list of keys → shared.scene_state.SceneState is SOT
Multi-env broadcasting → eval is single-env, prims[0] is the only prim

Version 1.1.0

exception so101_mvbench.evaluation.scene_loader.SceneApplyError[source]#

Bases: RuntimeError

Raised when applying scene_state to Isaac Sim fails. NOT silent.

class so101_mvbench.evaluation.scene_loader.SceneStateLoader(scene_state_dir, binmap_path, device)[source]#

Bases: object

Iterator over scene_state.json files + Isaac Sim USD applicator.

Parameters:

scene_state_dir (Path)
binmap_path (Path)
device (str)

__init__(scene_state_dir, binmap_path, device)[source]#

Parameters:

scene_state_dir (Path)
binmap_path (Path)
device (str)

apply(env, episode_idx, *, cube_noise_m=0.0, cube_rot_noise_deg=0.0, visual_override=None)[source]#

Apply scene_state for episode_idx to Isaac Sim.

Parameters:

visual_override (VisualConfig | None) – If None (IID), visual config is built from scene_state.json (reproduces training scene exactly). If a VisualConfig is passed (OOD), it overrides all visual properties for controlled variation testing.
episode_idx (int)
cube_noise_m (float)
cube_rot_noise_deg (float)

Return type:

dict

Returns dict with "scene_state" (the loaded JSON) and "placed_cube" (actually placed coordinates from scene_builder).

state_for_episode(episode_idx)[source]#

Return scene_state dict for episode index (0-based).

Wraps with modulo when episode_idx >= len(states), so OOD grids with more episodes than scene_states reuse visual properties cyclically.

Return type:: dict
Parameters:: episode_idx (int)

static visual_config_from_scene_state(ss)[source]#

Build a VisualConfig from a scene_state dict (IID reproduction).

Return type:: VisualConfig
Parameters:: ss (dict)

SimLeRobotSO101Interface — Sim-only subclass of upstream LeRobotSO101Interface.

Policy inference for Isaac-Sim eval must use the EXACT SAME chain as on-robot inference to keep Sim/Hardware parity. Upstream already provides that chain in LeRobotSO101Interface at third_party/lerobot_so101_teleop/…/lerobot_interface.py.

Upstream’s init_device() calls make_robot_from_config() which instantiates a SO101FollowerConfig. That drags in scservo_sdk (Feetech Motor SDK) — NOT a dependency of this package. So we override ONLY init_device() to skip the hardware-connection and build a _RobotStub that provides the four downstream attributes make_policy() and predict_action() need:

self.robot.observation_features (used in make_policy) self.robot.action_features (used in make_policy) self.robot.name (used for DummyDatasetMeta) self.robot.robot_type (used in predict_action per-step)

All OTHER methods (make_policy, sim_obs_to_policy_processor, predict_action, prediction_to_sim_processor) are inherited UNCHANGED from upstream.

Rename-map env → policy is auto-detected from policy’s config.json input_features. Supports three layouts:

wrist + N external cameras (e.g. wrist+front, wrist+front+top)

N external cameras without wrist (e.g. front+top, side_l+side_r)

wrist-only (1-cam)

Convention: env ‘ego’ → policy ‘wrist’; env ‘external’, ‘external_2’, … → non-wrist cameras sorted alphabetically. Falls back to legacy POLICY_RENAME_MAPS only if config.json read fails.

Version 1.2.0 (multi-external + no-wrist rename_map auto-detect)

class so101_mvbench.evaluation.policy_wrapper.SimLeRobotSO101Interface(device, port, id, cameras, fps, kind='leader', rename_map=None)[source]#

Bases: LeRobotSO101Interface

Sim-only subclass — skips Feetech hardware init.

Inherits make_policy / predict_action / sim_obs_to_policy_processor / prediction_to_sim_processor from upstream UNCHANGED. Only init_device is overridden to bypass scservo_sdk.

Parameters:

device (str)
port (str)
id (str)
cameras (dict)
fps (int)
kind (str)
rename_map (dict)

init_device(visualize=False)[source]#

Build a _RobotStub with just the 4 attributes downstream methods need.

No self.robot = make_robot_from_config(self.cfg). No scservo_sdk. No rerun init (we run headless or in the eval GUI viewport separately).

Feature-dict format MUST match the real SO101Follower (see lerobot/robots/so_follower/so_follower.py:66-81):

motor features:   dict[str, type]   -> {"joint_name.pos": float, ...}
camera features:  dict[str, tuple]  -> {"cam_name": (H, W, 3), ...}

Parameters:: visualize (bool)

make_policy(name_or_path, *, temporal_ensemble_coeff=None, force_disable_te=False)[source]#

Override upstream make_policy to accept temporal_ensemble_coeff override.

Body mirrors upstream LeRobotSO101Interface.make_policy 1:1 (same calls, same order) with ONE addition: if temporal_ensemble_coeff is set (or force_disable_te=True), the value is written onto policy_config BEFORE lerobot.policies.factory.make_policy builds the model. This is necessary because ACT wires up its TemporalEnsembler inside __init__ — post-hoc config edits do NOT take effect (the ensembler dataclass is bound at init-time).

TE-semantics:

force_disable_te=True → policy_config.temporal_ensemble_coeff = None
temporal_ensemble_coeff=X → policy_config.temporal_ensemble_coeff = X
neither → keep checkpoint’s config default

Parameters:

name_or_path (str)
temporal_ensemble_coeff (float | None)
force_disable_te (bool)

so101_mvbench.evaluation.policy_wrapper.rename_map_for_policy(policy_path)[source]#

Resolve env→policy camera rename-map for the given policy checkpoint.

Return type:: dict[str, str]
Parameters:: policy_path (Path)

Strategy:

Auto-detect from policy config.json input_features (NVIDIA-style layout)
Fallback: lookup hardcoded POLICY_RENAME_MAPS by JOB_NAME (legacy capsule layout: …/train/<JOB_NAME>/<ts>/checkpoints/<step>/pretrained_model)

Multi-env eval helpers — per-env obs slicing + per-env tracker init.

Role in the data flow:

Data flow 3 (per-env inference): slice_visual_obs() fixes Risk K-1 — the upstream sim_obs_to_policy_processor internally indexes visual[...][0], i.e. without slicing ALL envs would see env-0’s image.
Data flow 5 (per-env tracking): init_trackers() instantiates the existing LiftCubeEvalTracker N times (ADR-003 — no new vectorized tracker), each seeded from its own initial pose.

GPU-free: no Isaac Lab imports; unit-testable with CPU tensors / mocks (tests/test_multienv_obs_utils.py).

so101_mvbench.evaluation.multienv_obs_utils.init_trackers(n_envs, cube, robot, jaw_body_idx, home_pose_rad, *, home_pose_tol_rad=None, plateau_steps=None)[source]#

Build one LiftCubeEvalTracker per environment (ADR-003).

Mirrors the single-env construction in eval_session.setup_episode but indexes each Isaac tensor per environment [i] instead of [0].

Parameters:

n_envs (int) – Number of parallel environments.
cube (Any) – Isaac RigidObject (cube.data.root_link_pos_w shape (N, 3+)).
robot (Any) – Isaac Articulation (robot.data.body_link_pos_w shape (N, B, 3)).
jaw_body_idx (int) – Body index of the gripper jaw.
home_pose_rad (list[float]) – Target home pose (>=5 arm joints).
home_pose_tol_rad (float | None) – Per-joint home tolerance (None -> tracker default).
plateau_steps (dict[str, int] | None) – Per-stage plateau early-termination budgets.

Return type:

Returns:

List of n_envs independent trackers, each seeded from its env’s initial cube height + gripper->cube distance.

so101_mvbench.evaluation.multienv_obs_utils.slice_visual_obs(visual_obs, env_idx)[source]#

Per-env view of the visual obs dict (Risk-K-1 fix).

obs["visual"] maps camera-key -> tensor of shape (num_envs, H, W, C). The upstream sim_obs_to_policy_processor internally grabs visual[...][0]; to feed environment env_idx we slice each camera tensor to [env_idx:env_idx+1] (the length-1 batch dim is preserved) — the downstream [0] then selects exactly env_idx’s image instead of always env 0.

Parameters:

visual_obs (dict[str, Any]) – Dict camera-key -> tensor (num_envs, ...).
env_idx (int) – Environment index to extract.

Return type:

dict[str, Any]

Returns:

Dict camera-key -> tensor (1, ...) (view/slice, not a copy).

Image augmentations at the sim->policy boundary (camera-importance + general).

augment_image_frame() is the single dispatch point where the TiledCamera frame handed from Isaac Sim to the ACT policy can be transformed by arbitrary image- processing algorithms. Pure numpy/OpenCV on ONE uint8 HWC RGB frame -> a NEW frame of the same shape; importable WITHOUT an Isaac Sim boot (CPU-testable). A new algo = one function + one _AUGMENTATIONS entry; no eval-loop change.

Families:: degradations (camera-importance study, strength-graded): blackout, gauss, blur transforms (general image processing): invert, contrast, canny … vieles mehr — just register another function.

so101_mvbench.evaluation.augmentation.augment_image_frame(img, kind, strength=0.0, rng=None)[source]#

Apply ONE image augmentation at the sim->policy boundary (registry dispatch).

Parameters:

img (np.ndarray) – uint8 HWC RGB frame (H, W, C) — the native obs frame (CLAUDE.md §4).
kind (str) – registry key (see AUGMENTATION_KINDS); selects the algorithm.
strength (float) – per-kind knob — gauss=sigma(0-1) · blur=odd kernel · contrast=factor · canny=upper threshold; ignored for blackout/invert.
rng (np.random.Generator | None) – np.random.Generator for stochastic kinds (gauss) — reproducibility.

Return type:

np.ndarray

Returns:

A NEW contiguous uint8 HWC RGB array, same shape as img (input untouched, no channel swap — preserves the obs->policy->video format contract).

so101_mvbench.evaluation.augmentation.check_strength(kind, strength)[source]#

Raise if kind was asked for without a strength that means anything.

async_eval calls this before it builds the environment, so a mislabelled sweep dies in about ten seconds rather than after an hour of evaluation. It cannot fire earlier than that: AppLauncher boots Isaac at import time, well before main() sees the parsed flags.

Return type:

Parameters:

kind (str)
strength (float)

CSV + JSON results writer for evaluation runs.

Writes per-episode results to CSV (Pandas-ready) and an aggregated summary to JSON. Handles multi-run statistics (mean ± std).

class so101_mvbench.evaluation.results_writer.ResultsWriter(output_dir, config)[source]#

Bases: object

Writes evaluation results to CSV + JSON.

Parameters:

output_dir (Path | None)
config (dict)

__init__(output_dir, config)[source]#

Parameters:

output_dir (Path | None) – Directory for output files. None = next to checkpoint.
config (dict) – Eval config dict with keys: policy_path, task, randomize, cube_noise, cube_rot_noise, scene_state, num_runs, num_episodes_per_run.

write(results, run_success_rates, video_settings=None)[source]#

Write CSV + JSON + log summary table.

Return type:

Parameters:

results (list[dict])
run_success_rates (list[float])
video_settings (dict | None)

Async MP4 video recorder for evaluation episodes.

Captures camera frames at a fixed interval and encodes them to MP4 in a background thread (ThreadPoolExecutor). Frames are resized before encoding to reduce file size.

class so101_mvbench.evaluation.video_recorder.VideoRecorder(base_dir, policy_cams=None, fps=10, interval=15, resize=(320, 240))[source]#

Bases: object

Async MP4 recorder for camera streams the policy consumes.

By default (policy_cams=None) captures both camera_ego and camera_external — for backwards compatibility with debug workflows. Passing policy_cams=list(rename_map.keys()) filters to only the cameras the policy actually consumes, so 1-cam policies don’t emit a misleading external.mp4 that was never fed to the model.

Parameters:

base_dir (Path | None)
policy_cams (list[str] | None)
fps (int)
interval (int)
resize (tuple[int, int])

__init__(base_dir, policy_cams=None, fps=10, interval=15, resize=(320, 240))[source]#

Parameters:

base_dir (Path | None)
policy_cams (list[str] | None)
fps (int)
interval (int)
resize (tuple[int, int])

capture(obs, step)[source]#

Grab camera frames if step matches interval.

Cameras not in self._policy_cams are skipped — avoids emitting MP4s for cameras the policy never consumed.

Return type:

Parameters:

obs (dict)
step (int)

flush(ep_dir)[source]#

Encode collected frames to MP4 async, then clear buffers.

Return type:: None
Parameters:: ep_dir (Path)

property settings: dict#: Video settings dict for JSON summary.

wait()[source]#

Wait for all pending encoding jobs.

Return type:: None

eval_report: standalone post-analysis (pandas + matplotlib).

Reads eval_results_random_*.csv + eval_summary.json + quality_gate.json from an eval output directory and produces:

stage_outcome.png — the 4 task stages, split by termination reason attempts_histogram.png — histogram of grasp attempts per episode cube_position_analysis.png — cube-xy scatter colored by success/stage cube_rotation_analysis.png — cube-yaw histogram split by success episode_timeline.png — per-episode outcome Gantt chart timing_and_distance.png — time_to_reach/lift/home + min dist scatter report.txt — plain-text summary report.md — markdown-formatted briefing

Standalone: no Jupyter / nbconvert dependency, no simulator needed.

Usage:: eval_report –dir <eval_output> # via pyproject [project.scripts] entry-point

Metrics doc: see METRICS.md next to this file.

Version 1.1.0 — added cube_position/rotation/timeline/timing plots.

eval_aggregate_report — cross-run statistical analysis of multiple eval-runs.

Reads N eval-output directories (each containing eval_summary.json + eval_results_random_*.csv from a single run_eval.sh execution) and produces a cross-run aggregation report:

aggregate_report.md — markdown summary (mean ± std) aggregate_report.txt — plain-text summary plots/cross_run_success_rate.png — boxplot of success_rate per run plots/per_episode_success.png — fraction-success per episode (1-indexed) plots/cross_run_attempts.png — distribution of num_attempts per run plots/cross_run_stage_distribution.png — stacked bar of stages per run

Designed for thesis-grade statistics: mean, std, min/max, per-episode flakiness.

Standalone — no Jupyter / nbconvert dependency, no simulator needed.

Usage:

eval_aggregate_report –run_dirs <dir1> <dir2> … –output_dir <agg_dir>

Each <dir_N> must contain:

eval_summary.json
eval_results_random_*.csv

Version 1.0.0

Shared matplotlib conventions for the plots this package writes.

One rule so far, but it applies to every figure: the legend belongs BELOW the x-axis. An outside-right legend eats roughly a quarter of the width for a few short entries, and an in-plot one covers data – in the episode timeline it literally hid a bar. Vertical space is the cheap direction.

so101_mvbench.evaluation.plot_style.legend_below(ax, handles=None, fontsize=9, max_ncol=4, y=None)[source]#

Place ax’s legend as a horizontal row underneath the x-axis.

handles overrides the artists collected from the axes, for figures that build proxy handles. Does nothing when there is nothing to label.

y is an axes-fraction offset. Left at None it is derived from a constant PHYSICAL gap, because the same fraction means different distances on a tall square plot and on a short wide one: at a fixed -0.09 the legend landed on top of the x-axis label of the wide rotation and timing figures.

Return type:: None

Audit tools#

Read per-episode CSVs from cube_replay, generate 3-panel divergence plot per episode + a global summary.csv.

CPU-only — no simulator needed (numpy + matplotlib + scipy).

so101_mvbench.audit.cube_plot.pairwise_max(pos_rel, quat)[source]#

Per-frame maxima over all env-pairs of position & rotation divergence.

Let P = { (i, j) : 0 ≤ i < j < E } be the set of unordered env-pairs (size n_pairs = E·(E−1)/2). For each frame t we compute:

max_norm[t]     = max_{(i,j)∈P}   ‖ p_i(t) − p_j(t) ‖₂        [meters]
max_dx[t]       = max_{(i,j)∈P}   | x_i(t) − x_j(t) |          [meters]
max_dy[t]       = max_{(i,j)∈P}   | y_i(t) − y_j(t) |
max_dz[t]       = max_{(i,j)∈P}   | z_i(t) − z_j(t) |
max_quat_deg[t] = max_{(i,j)∈P}   2·arccos( |q_i(t) · q_j(t)| ) · 180/π

where:

p_e(t) = (x_e, y_e, z_e)(t) ∈ ℝ³ is the cube position in env e at frame t (in m, env_origin-offset subtracted, common local frame)
q_e(t) ∈ ℝ⁴ is the unit quaternion in XYZW order (Isaac Lab post-PR-4437) of the cube in env e
the absolute value of the dot product handles the q ≡ −q sign ambiguity, making the angular distance sign-invariant
2·arccos(|q·q'|) is the geodesic rotation distance on SO(3) in radians; multiplied by 180/π → degrees.

The per-axis components (dx, dy, dz) are useful for diagnostics: they reveal WHERE the divergence sits (lateral vs vertical), which the Euclidean norm alone hides.

Shapes: pos_rel (E, T, 3), quat (E, T, 4). Returns: (max_norm, max_dx, max_dy, max_dz, max_quat_deg), each shape (T,).

Parameters:

pos_rel (numpy.ndarray)
quat (numpy.ndarray)

so101_mvbench.audit.cube_plot.per_env_euler_to_ref(quat, ref=0)[source]#

Signed Euler decomposition of the relative rotation R_ref⁻¹·R_e per env.

For each env e (e ≠ ref) at each frame t:

R_e(t) = Rotation.from_quat(q_e(t)) # quat already XYZW, scipy convention matches R_ref(t) = Rotation.from_quat(q_ref(t)) R_rel(t) = R_ref(t)⁻¹ · R_e(t) # relative rotation (roll, pitch, yaw)[e, t] = R_rel(t).as_euler(“xyz”, degrees=True)

where:

q_e is the unit quaternion in XYZW order (Isaac Lab convention since PR #4437, matches scipy.spatial.transform.Rotation “scalar-last”)
“xyz” denotes the intrinsic Tait-Bryan convention (X, then Y′, then Z″)
Output values are SIGNED degrees in (−180°, +180°] per axis

CRITICAL: this is NOT euler(R_e) − euler(R_ref), which would be wrong under angle wrapping and gimbal lock. The correct way is to first compose the relative rotation R_rel = R_ref⁻¹·R_e, then decompose that ONE rotation.

Quaternion sign ambiguity (q ≡ −q): scipy.spatial.transform.Rotation handles this internally — Rotation.from_quat(q) and Rotation.from_quat(−q) yield identical rotation objects.

Shapes: quat (E, T, 4). Returns euler (E, T, 3) signed degrees [roll, pitch, yaw]. The slice [ref] stays zero (R_rel is identity for the reference env itself).

Parameters:

quat (numpy.ndarray)
ref (int)

so101_mvbench.audit.cube_plot.per_env_to_ref_metrics(pos_rel, quat, ref=0)[source]#

Per-env distance and rotation-angle relative to a reference env.

For each env e at each frame t:

dist[e, t]    = ‖ p_e(t) − p_ref(t) ‖₂              [meters]
ang_deg[e, t] = 2·arccos( |q_e(t) · q_ref(t)| )      [degrees]

where:

p_e(t) ∈ ℝ³ is the cube position in env e at frame t (env_origin-offset already subtracted so envs are comparable in a common local frame)
q_e(t) ∈ ℝ⁴ is the unit quaternion in XYZW order (Isaac Lab convention since PR #4437)
q_e · q_ref denotes the component-wise dot product
the absolute value handles the quaternion sign ambiguity: q and −q represent the same rotation, so the distance must be sign-invariant

The factor 2 in 2·arccos(|q·q'|) is the geodesic angle on SO(3); for unit quaternions, arccos(|q·q'|) is HALF the rotation angle between R(q) and R(q’).

Shapes: pos_rel (E, T, 3), quat (E, T, 4). Returns (dist, ang_deg) of shape (E, T).

Parameters:

pos_rel (numpy.ndarray)
quat (numpy.ndarray)
ref (int)

Dataset tools#

List all gym-registered Isaac Lab environments from this package.

Usage:: list_envs

Filters to envs whose ID starts with “Lerobot-” or “LiftCube-” (this package’s namespace), so upstream Isaac Lab task IDs are not shown. Output is sorted alphabetically by task ID for stable, reviewer-reproducible diffs.

Adapted from NVIDIA Sim-to-Real-SO-101-Workshop scripts/list_envs.py.

so101_mvbench.tools.list_envs.main()[source]#

Print this package’s gym-registered envs as a plain table.

Uses os._exit(0) instead of simulation_app.close() because Isaac Sim 6 hangs on close() (see .claude/memory/reference_isaac_sim6_shutdown.md). Without the hard exit the printed table never flushes before the hang.

Return type:: None

validate_bin — Objective per-bin validation for AP-7 gate decisions.

Checks a single recorded bin for: - Parquet readability + episode count consistency with binmap.json - Video file count (2 cameras × n_success episodes) - Scene-state JSON presence for each episode - Cube position inside expected bin rectangle (within half_extent_m tolerance)

Exit code: 0 if all checks pass, 1 on first failure.

Usage:

python validate_bin.py --bin_id 1
python validate_bin.py --bin_id 13 --dataset_root /custom/path

class so101_mvbench.tools.validate_bin.BinLayout(col, row, n_cols, n_rows, bin_size_m, origin_x, origin_y, robot_yaw_rad)[source]#

Bases: object

Compute expected bin center + bounds in world frame.

Parameters:

col (int)
row (int)
n_cols (int)
n_rows (int)
bin_size_m (float)
origin_x (float)
origin_y (float)
robot_yaw_rad (float)

__init__(col, row, n_cols, n_rows, bin_size_m, origin_x, origin_y, robot_yaw_rad)#

Parameters:

col (int)
row (int)
n_cols (int)
n_rows (int)
bin_size_m (float)
origin_x (float)
origin_y (float)
robot_yaw_rad (float)

Return type:

None

expected_center()[source]#

World-frame center of this bin.

Return type:: tuple[float, float]

world_bounds(tolerance_m)[source]#

Return (x_min, x_max, y_min, y_max) of the bin in world frame.

Return type:: tuple[float, float, float, float]
Parameters:: tolerance_m (float)

so101_mvbench.tools.validate_bin.check_binmap(binmap_path)[source]#

Load binmap.json and return (n_success, success_episode_indices).

Return type:: tuple[int, list]
Parameters:: binmap_path (Path)

so101_mvbench.tools.validate_bin.check_cube_positions(success_eps, layout, tolerance_m)[source]#

Verify cube (x, y) for each success episode is inside the bin rectangle.

Return type:

Parameters:

success_eps (list)
layout (BinLayout)
tolerance_m (float)

so101_mvbench.tools.validate_bin.check_parquet(parquet_path, expected_count)[source]#

Load parquet, verify distinct episode_index count. Returns actual count.

Return type:

Parameters:

parquet_path (Path)
expected_count (int)

so101_mvbench.tools.validate_bin.check_scene_states(meta_root, n_success)[source]#

Count scene_state JSON files. Should be ≥ n_success.

Return type:

Parameters:

meta_root (Path)
n_success (int)

so101_mvbench.tools.validate_bin.check_videos(videos_root, n_success, min_size_bytes=50000)[source]#

Count .mp4 files under videos/. Should be 2 * n_success (2 cameras).

Return type:

Parameters:

videos_root (Path)
n_success (int)
min_size_bytes (int)

Delete specific episodes from a LeRobot v3 dataset.

Removes episode data from parquet, videos, scene_states, and binmap. Re-indexes remaining episodes so indices are contiguous.

Usage:

# List all episodes python delete_episodes.py –dataset_root /path/to/bin_c0_r0 –list

# Delete episodes 1 and 3 python delete_episodes.py –dataset_root /path/to/bin_c0_r0 –delete 1,3

# Dry run (show what would be deleted) python delete_episodes.py –dataset_root /path/to/bin_c0_r0 –delete 1,3 –dry_run

so101_mvbench.tools.delete_episodes.delete_episodes(root, delete_indices, dry_run=False)[source]#

Delete specified episodes and re-index remaining.

Return type:

Parameters:

root (Path)
delete_indices (list[int])
dry_run (bool)

so101_mvbench.tools.delete_episodes.list_episodes(root)[source]#

Print all episodes with frame counts.

Return type:: None
Parameters:: root (Path)

Pipeline utilities for LeRobot v3.0 dataset lifecycle management.

Provides centralized dataset validation, cleanup, and resume handling used by all pipeline scripts (recorder, visualizer, trainer, inference).

class so101_mvbench.utils.dataset_manager.DatasetManager[source]#

Bases: object

Manages LeRobot v3.0 dataset lifecycle – validation, cleanup, resume.

static cleanup_corrupt_tail(dataset_path)[source]#

Remove corrupt trailing parquet file(s) from aborted recording.

When Ctrl+C kills the recorder mid-write, the async_episode_processor leaves a corrupt .parquet file. This method detects and removes it, then updates info.json to match the remaining valid data.

Parameters:: dataset_path (Path) – Root path of the LeRobot dataset.
Return type:: bool
Returns:: True if cleanup was performed, False if dataset was already clean.

static handle_existing_dataset(dataset_path, auto=False)[source]#

Check for existing dataset and prompt user for action.

Automatically cleans up corrupt trailing files before presenting the menu. In auto mode or non-interactive sessions, makes a safe default choice without prompting.

Parameters:

dataset_path (Path) – Root path of the LeRobot dataset.
auto (bool) – If True, skip menu (delete incomplete, resume valid).

Returns:

“continue”, “fresh”, “quit”.

Return type:

One of

static validate_dataset(dataset_path)[source]#

Check dataset integrity and return a status dict.

Parameters:: dataset_path (Path) – Root path of the LeRobot dataset.
Returns:: exists (bool), valid (bool), total_episodes (int), total_frames (int), valid_files (int), corrupt_files (list[str])
Return type:: Dict with keys

Parse CLI --episodes range-syntax into a sorted list[int].

Supported forms:

'0' single ID
'0,1,5' comma-list
'0-10' inclusive range (lo ≤ hi)
'0,5-10,15' mixed comma + range
'all' auto-discover from <dataset_root>/trajectories/episode_*.npy

The function is intentionally pure (no Isaac Sim deps) so it can be unit-tested in isolation. Shared across console-scripts that operate on a subset of episodes (synthetic_multicam_recorder, future tools).

so101_mvbench.utils.episodes_spec.parse_episodes_spec(spec, dataset_root)[source]#

Parse a --episodes spec string into a sorted list of integer IDs.

Parameters:

spec (str) – One of: 'all', '<n>', '<n>,<m>', '<lo>-<hi>', or any comma-separated mix of singles + inclusive ranges. Whitespace around tokens is ignored. Case-insensitive for 'all'.
dataset_root (Path) – Source LeRobot dataset directory. Used only when spec == 'all' — globs <dataset_root>/trajectories/episode_*.npy.

Return type:

list[int]

Returns:

Sorted list of unique integer episode IDs. Duplicates within the spec are deduplicated.

Raises:

ValueError – If spec is empty/whitespace, malformed, or contains a reversed range (hi < lo). Also if spec == 'all' but the trajectories/ directory has no matching files.

Scene and geometry#

Shared SO-101 joint constants and coordinate transforms.

Single source of truth for the scene geometry the whole pipeline shares: the workspace-frame anchors, the rigid shift between scenes, the joint metadata, and the raw-degrees to sim-radians transform pair.

Deliberately free of any Isaac Sim / AppLauncher import, so that recording, replay and evaluation can read these constants at module level without booting a simulator. Joint metadata mirrors the LeRobot interface (SO101_USD_MAPPING, SO101_JOINT_ORDER).

so101_mvbench.utils.so101_transforms.ARM_BASE_OFFSET: tuple[float, float, float] = (-0.2, 0.0, 0.7463)#: Rigid translation applied to the ENTIRE coupled configuration when moving from the recording scene into the cylindrical room.

so101_mvbench.utils.so101_transforms.BASE_SCENE_ORIGIN_W: tuple[float, float, float] = (0.0, 0.0, 0.8663)#

{W} origin of the scene LiftCubeSceneCfg is authored in.

Since the cylroom migration the base scene IS the cylindrical room, so scene overrides (RIG_SHIFT) are computed against this origin: for the shipped cylroom configs the rig shift is zero and the base scene spawns as authored. Legacy datasets recorded in the retired floor-level table scene still declare (or default to) HEMISPHERE_ORIGIN_W as their SOURCE origin – that constant stays for data interpretation, this one describes the code’s scene.

so101_mvbench.utils.so101_transforms.CUBE_SPAWN_Z_IN_W: float = -0.06999999999999999#: Cube spawn height expressed in the workspace frame {W}. Scene-independent: the spawn height in any scene’s world frame is its {W} origin z plus this.

so101_mvbench.utils.so101_transforms.FLOOR_CUBE_SPAWN_Z: float = 0.05#: Cube spawn height recorded in scene_state.object_init_pos_m.z, expressed in the recording scene’s floor frame.

so101_mvbench.utils.so101_transforms.FPS: int = 30#: Simulation and recording frame rate (Hz).

so101_mvbench.utils.so101_transforms.HEMISPHERE_ORIGIN_W: tuple[float, float, float] = (0.2, 0.0, 0.12)#

{W} origin of the retired floor-level table scene, in world frame {0}.

Cameras, cube and bin grid are all anchored in the workspace frame {W}; this constant is one scene’s {W} origin expressed in {0} (⁰ξ_W per Corke 2023, §2.1.2). External cameras orbit that point and aim at it.

The scene it describes is no longer built: the package’s base scene is the cylindrical room (BASE_SCENE_ORIGIN_W). The constant stays because it is the frame that recorded data without a frame declaration lives in, so it remains the source origin when replaying or evaluating such a dataset.

so101_mvbench.utils.so101_transforms.PHASE3_FLOOR_Z: float = 0.0237#

pad origin 0.0257 minus half-thickness 0.002). Verified empirically in a settled scene: arm-base world z = 0.0, pad z = 0.0257, cube z = 0.0477. The arm ORIGIN sits ~2.4 cm BELOW this surface – its foot plate contacts the floor, not the origin. Do NOT use the table surface as the arm origin; that over-lifts the rig by 2.4 cm and leaves pad and cube floating.

Type:: Floor surface the recorded rig sat on (= action-pad bottom

so101_mvbench.utils.so101_transforms.ROBOT_TABLE_POS: tuple[float, float, float] = (-0.2, 0.0, 0.7463)#: Arm base world position on the pedestal table. Identical to ARM_BASE_OFFSET because the arm base sits at the world origin in the recording scene.

so101_mvbench.utils.so101_transforms.SO101_ACTION_FEATURES: dict[str, type] = {'elbow_flex.pos': <class 'float'>, 'gripper.pos': <class 'float'>, 'shoulder_lift.pos': <class 'float'>, 'shoulder_pan.pos': <class 'float'>, 'wrist_flex.pos': <class 'float'>, 'wrist_roll.pos': <class 'float'>}#: Action feature schema — 6 joint positions in raw degrees.

so101_mvbench.utils.so101_transforms.SO101_JOINT_ORDER: list[str] = ['shoulder_pan.pos', 'shoulder_lift.pos', 'elbow_flex.pos', 'wrist_flex.pos', 'wrist_roll.pos', 'gripper.pos']#: Joint names in LeRobot observation/action order (with .pos suffix).

so101_mvbench.utils.so101_transforms.SO101_OBSERVATION_FEATURES: dict[str, type] = {'elbow_flex.pos': <class 'float'>, 'gripper.pos': <class 'float'>, 'shoulder_lift.pos': <class 'float'>, 'shoulder_pan.pos': <class 'float'>, 'wrist_flex.pos': <class 'float'>, 'wrist_roll.pos': <class 'float'>}#: Static observation feature schema (without cameras).

so101_mvbench.utils.so101_transforms.SO101_USD_MAPPING: dict[str, dict[str, int]] = {'elbow_flex': {'joint_max': 90, 'joint_min': -100}, 'gripper': {'joint_max': 100, 'joint_min': -10}, 'shoulder_lift': {'joint_max': 100, 'joint_min': -100}, 'shoulder_pan': {'joint_max': 110, 'joint_min': -110}, 'wrist_flex': {'joint_max': 95, 'joint_min': -95}, 'wrist_roll': {'joint_max': 160, 'joint_min': -160}}#: USD joint limits in degrees — defines the mapping between normalized [0, 1] space and physical joint angles.

so101_mvbench.utils.so101_transforms.TABLE_HEMISPHERE_ORIGIN: tuple[float, float, float] = (0.0, 0.0, 0.8663)#

the recording-scene origin shifted by the same rigid delta as everything else. This is the workspace frame {W} origin of the cylindrical-room scene, expressed in world frame {0}.

Type:: Camera-hemisphere origin in the cylindrical room

so101_mvbench.utils.so101_transforms.TABLE_TOP_Z: float = 0.77#

Top cylinder centre 0.745 plus half-height 0.025.

Type:: Tabletop SURFACE height in the cylindrical room

so101_mvbench.utils.so101_transforms.WARMUP_STEPS: int = 60#: Physics warmup after env.reset() — 2 seconds at FPS=30.

so101_mvbench.utils.so101_transforms.WORLD_DELTA_Z: float = 0.7463#: The one rigid z-shift that maps the recorded floor onto the table surface.

so101_mvbench.utils.so101_transforms.build_joint_tensors(device)[source]#

Pre-compute joint min/max tensors for coordinate transforms.

Parameters:: device (str | torch.device) – Torch device (e.g. "cuda:0" or torch.device("cpu")).
Return type:: tuple[torch.Tensor, torch.Tensor]
Returns:: Tuple of (joint_mins, joint_maxs) tensors, each shape (6,), containing USD joint limits in degrees.

so101_mvbench.utils.so101_transforms.raw_degrees_to_sim_radians(raw_values, joint_mins, joint_maxs)[source]#

Convert robot raw degrees to Isaac Sim joint position radians.

Transform chain: raw_degrees → normalized [0,1] → mapped degrees → radians.

Parameters:

raw_values (Tensor) – Raw action values, shape (6,). Joints 0-4: range [-100, 100]. Gripper (5): range [0, 100].
joint_mins (Tensor) – USD joint minimums in degrees, shape (6,).
joint_maxs (Tensor) – USD joint maximums in degrees, shape (6,).

Return type:

Tensor

Returns:

Joint positions in radians, shape (6,).

so101_mvbench.utils.so101_transforms.rigid_shift_from_recording(workspace_origin_w, source_origin_w=(0.2, 0.0, 0.12))[source]#

Translation that carries a recorded rig from its source scene into a target scene.

A scene declares its workspace-frame {W} origin (SceneConfig. workspace_origin_m); recorded data declares the origin of the scene it was captured in (binmap.json / scene_state workspace_origin_m, defaulting to the floor-level table scene at HEMISPHERE_ORIGIN_W for datasets that predate the field). The rigid shift for the ENTIRE coupled rig – arm base, cube, action pad, camera hemisphere – is simply the difference between the two origins. This is the pose-graph property that keeps recorded joint trajectories valid across scenes: the rig moves as one, so the arm-to-cube geometry never changes.

Table-scene recording replayed in the cylindrical room: returns exactly ARM_BASE_OFFSET. Same scene on both sides: returns (0, 0, 0).

Return type:

tuple[float, float, float]

Parameters:

workspace_origin_w (tuple[float, float, float])
source_origin_w (tuple[float, float, float])

so101_mvbench.utils.so101_transforms.sim_radians_to_raw_degrees(sim_obs, joint_mins, joint_maxs)[source]#

Convert Isaac Sim joint positions (radians) to robot raw degrees.

Inverse of raw_degrees_to_sim_radians().

Parameters:

sim_obs (Tensor) – Joint positions in radians, shape (6,).
joint_mins (Tensor) – USD joint minimums in degrees, shape (6,).
joint_maxs (Tensor) – USD joint maximums in degrees, shape (6,).

Return type:

Tensor

Returns:

Raw degree values, shape (6,).: Joints 0-4: range [-100, 100]. Gripper (5): range [0, 100].

SceneConfig — unified scene declaration schema.

Single source of truth for scene structure: cameras, objects, robot, visuals, and the scene’s workspace origin (workspace_origin_m). Consumed by teleop_recorder, multicam_replay, and async_eval via SceneConfig.from_json; the shipped configs live in tasks/scene_configs/. - … and 5 more (see plan-file “9 fragmentierte Configs” inventory)

Schema versioning: major version (1.x) must match SCHEMA_VERSION; minor bumps are forward-compatible for new optional fields with defaults, but they may also tighten validation on existing fields (e.g. a previously optional field becoming required for one type variant). The major bump is reserved for changes that rename or remove fields or alter the semantics of existing values.

Version history:

1.0: Initial schema.
1.1: CameraSpec.radius_m is required for external entries and forbidden on ego entries.
1.2: CameraSpec.pose_id is removed. External cameras declare their hemisphere position via three explicit floats (azimuth_deg, elevation_deg, radius_m). Pre-1.2 configs must be migrated by splitting pose_id into the two angle fields.

class so101_mvbench.utils.scene_config.CameraSpec(type, azimuth_deg=None, elevation_deg=None, radius_m=None, offset_pos_m=None, offset_rot_deg=None, resolution_hw=(480, 640), focal_length=5.0, horizontal_aperture=10.0, up_world=None)[source]#

Bases: object

One camera in the scene.

ego — gripper-mounted (use offset_pos_m + offset_rot_deg: relative to the gripper joint).
external — hemisphere-mounted around the workspace origin. Position: is declared by three explicit floats: azimuth_deg, elevation_deg, radius_m. All three are REQUIRED. The world-frame (x, y, z) is computed from them by utils/camera_grid.spherical_to_cartesian.

Schema 1.2 removed the string-form pose_id (e.g. "az000_el45"): carrying both numbers and a string-encoding of two of them invited parsing, key mangling, and registry mutation downstream.

Parameters:

type (Literal['ego', 'external'])
azimuth_deg (float | None)
elevation_deg (float | None)
radius_m (float | None)
offset_pos_m (tuple[float, float, float] | None)
offset_rot_deg (tuple[float, float, float] | None)
resolution_hw (tuple[int, int])
focal_length (float)
horizontal_aperture (float)
up_world (tuple[float, float, float] | None)

__init__(type, azimuth_deg=None, elevation_deg=None, radius_m=None, offset_pos_m=None, offset_rot_deg=None, resolution_hw=(480, 640), focal_length=5.0, horizontal_aperture=10.0, up_world=None)#

Parameters:

type (Literal['ego', 'external'])
azimuth_deg (float | None)
elevation_deg (float | None)
radius_m (float | None)
offset_pos_m (tuple[float, float, float] | None)
offset_rot_deg (tuple[float, float, float] | None)
resolution_hw (tuple[int, int])
focal_length (float)
horizontal_aperture (float)
up_world (tuple[float, float, float] | None)

Return type:

None

class so101_mvbench.utils.scene_config.ObjectSpec(type, init_pos_m=None, init_rpy_rad=(0.0, 0.0, 0.0), size_m=None, color_rgb=(0.8, 0.8, 0.8), mass_kg=0.05, usd_path=None)[source]#

Bases: object

One scene object: rigid cube or static USD asset.

init_pos_m is expressed in the workspace frame {W}, not in world coordinates: add the scene’s workspace_origin_m to get the world pose. Declaring it {W}-relative is what makes an object placement portable between scenes (see the frame contract).

Parameters:

type (Literal['cube', 'usd_asset'])
init_pos_m (tuple[float, float, float] | None)
init_rpy_rad (tuple[float, float, float])
size_m (tuple[float, float, float] | None)
color_rgb (tuple[float, float, float])
mass_kg (float)
usd_path (str | None)

__init__(type, init_pos_m=None, init_rpy_rad=(0.0, 0.0, 0.0), size_m=None, color_rgb=(0.8, 0.8, 0.8), mass_kg=0.05, usd_path=None)#

Parameters:

type (Literal['cube', 'usd_asset'])
init_pos_m (tuple[float, float, float] | None)
init_rpy_rad (tuple[float, float, float])
size_m (tuple[float, float, float] | None)
color_rgb (tuple[float, float, float])
mass_kg (float)
usd_path (str | None)

Return type:

None

class so101_mvbench.utils.scene_config.RobotSpec(type='so101_white', init_pos_m=(0.0, 0.0, 0.0), init_rot_xyzw=(0.0, 0.0, 0.7071, 0.7071), default_joint_pos_rad=<factory>, home_joint_pos_rad=<factory>, color_rgb=(1.0, 1.0, 1.0))[source]#

Bases: object

Robot config: type, initial pose, default + home joint positions.

Joint dicts use Isaac-USD names (Rotation, Pitch, Elbow, Wrist_Pitch, Wrist_Roll, Jaw) for consistency with LiftCubeEnvCfg.actions.joint_positions.joint_names.

Parameters:

type (str)
init_pos_m (tuple[float, float, float])
init_rot_xyzw (tuple[float, float, float, float])
default_joint_pos_rad (dict[str, float])
home_joint_pos_rad (dict[str, float])
color_rgb (tuple[float, float, float])

__init__(type='so101_white', init_pos_m=(0.0, 0.0, 0.0), init_rot_xyzw=(0.0, 0.0, 0.7071, 0.7071), default_joint_pos_rad=<factory>, home_joint_pos_rad=<factory>, color_rgb=(1.0, 1.0, 1.0))#

Parameters:

type (str)
init_pos_m (tuple[float, float, float])
init_rot_xyzw (tuple[float, float, float, float])
default_joint_pos_rad (dict[str, float])
home_joint_pos_rad (dict[str, float])
color_rgb (tuple[float, float, float])

Return type:

None

class so101_mvbench.utils.scene_config.SceneConfig(schema_version='1.2', workspace_origin_m=(0.0, 0.0, 0.8663), cameras=<factory>, objects=<factory>, robot=<factory>, visual=<factory>)[source]#

Bases: object

Single source of truth for scene declaration.

Parameters:

schema_version (str)
workspace_origin_m (tuple[float, float, float])
cameras (dict[str, CameraSpec])
objects (dict[str, ObjectSpec])
robot (RobotSpec)
visual (VisualSpec)

__init__(schema_version='1.2', workspace_origin_m=(0.0, 0.0, 0.8663), cameras=<factory>, objects=<factory>, robot=<factory>, visual=<factory>)#

Parameters:

schema_version (str)
workspace_origin_m (tuple[float, float, float])
cameras (dict[str, CameraSpec])
objects (dict[str, ObjectSpec])
robot (RobotSpec)
visual (VisualSpec)

Return type:

None

classmethod from_json(path)[source]#

Load SceneConfig from a JSON file.

Hard-fails when the JSON’s schema_version major component does not match SCHEMA_VERSION. JSON list-fields are converted back to tuples for type-consistency with downstream Isaac Lab APIs.

Return type:: SceneConfig
Parameters:: path (Path | str)

to_json(path)[source]#

Serialize SceneConfig to JSON (pretty-printed, 2-space indent).

Return type:: None
Parameters:: path (Path | str)

workspace_origin_m: tuple[float, float, float] = (0.0, 0.0, 0.8663)#: {W} origin in world coordinates. Defaults to the scene the package ships (the cylindrical room). A config that omits it therefore declares the current base scene, not a retired one. The legacy fallback for recorded data without a frame declaration is a separate rule and lives in the consumers (see so101_transforms.rigid_shift_from_recording).

class so101_mvbench.utils.scene_config.VisualSpec(room_light_exposure=1.0, env_light_exposure=1.0, sky_dome_yaw_rad=0.0, action_pad_yaw_rad=0.0, dome_light_y_mirror=True)[source]#

Bases: object

Visual defaults: lighting, sky dome orientation, action pad.

Parameters:

room_light_exposure (float)
env_light_exposure (float)
sky_dome_yaw_rad (float)
action_pad_yaw_rad (float)
dome_light_y_mirror (bool)

__init__(room_light_exposure=1.0, env_light_exposure=1.0, sky_dome_yaw_rad=0.0, action_pad_yaw_rad=0.0, dome_light_y_mirror=True)#

Parameters:

room_light_exposure (float)
env_light_exposure (float)
sky_dome_yaw_rad (float)
action_pad_yaw_rad (float)
dome_light_y_mirror (bool)

Return type:

None

Scene State — capture, save, and load exact scene conditions per episode.

Stores all randomizable parameters so that trajectory replays can reproduce identical scenes across camera permutations. Uses self-documenting dict format with named keys (not positional arrays).

Conventions:

Orientation: Euler RPY (rad), not quaternions
Unit suffixes: _m for meters, _rad for radians
Dict keys match joint/axis names for unambiguous access

class so101_mvbench.utils.scene_state.SceneState(episode_index, seed, task, object_key, object_init_pos_m=<factory>, object_init_rpy_rad=<factory>, action_pad_yaw_rad=0.0, room_light_exposure=0.0, env_light_exposure=0.0, sky_dome_yaw_rad=0.0, robot_color_rgb=<factory>, object_color_rgb=<factory>, object_texture='', table_texture='', table_form='', robot_default_joint_pos_rad=<factory>, look_at_m=<factory>, workspace_origin_m=None)[source]#

Bases: object

Exact scene conditions for one episode.

Saved as JSON after env.reset(), before the first action. The replayer loads this to reproduce identical conditions.

All numeric fields use physical unit suffixes (_m, _rad). Vector quantities use dicts with named keys for self-documentation.

Parameters:

episode_index (int) – Episode number within the recording session.
seed (int) – Random seed used for env.reset().
task (str) – Isaac Lab task name (e.g. “LiftCube-Sim”).
object_key (str) – Scene key of the tracked object (e.g. “cube”, “yellow_ring”).
object_init_pos_m (dict[str, float]) – Object position after reset {“x”, “y”, “z”}.
object_init_rpy_rad (dict[str, float]) – Object orientation after reset {“roll”, “pitch”, “yaw”}.
action_pad_yaw_rad (float) – Action pad yaw angle after reset.
room_light_exposure (float) – Room light exposure value after reset.
env_light_exposure (float) – Environment light exposure value after reset.
sky_dome_yaw_rad (float) – Sky dome yaw angle after reset.
robot_color_rgb (dict[str, float]) – Robot diffuse color {“r”, “g”, “b”} in [0, 1].
object_color_rgb (dict[str, float]) – Object (cube) diffuse color {“r”, “g”, “b”} in [0, 1].
object_texture (str) – Object (cube) texture path or “” if none applied.
table_texture (str) – Action pad texture path or “” if none.
table_form (str) – Action pad USD asset name (e.g. “action-pad-round.usda”).
robot_default_joint_pos_rad (dict[str, float]) – Robot default joint positions {“Rotation”, …}.
look_at_m (dict[str, float]) – Camera look-at point {“x”, “y”, “z”}.
workspace_origin_m (list[float] | None)

__init__(episode_index, seed, task, object_key, object_init_pos_m=<factory>, object_init_rpy_rad=<factory>, action_pad_yaw_rad=0.0, room_light_exposure=0.0, env_light_exposure=0.0, sky_dome_yaw_rad=0.0, robot_color_rgb=<factory>, object_color_rgb=<factory>, object_texture='', table_texture='', table_form='', robot_default_joint_pos_rad=<factory>, look_at_m=<factory>, workspace_origin_m=None)#

Parameters:

episode_index (int)
seed (int)
task (str)
object_key (str)
object_init_pos_m (dict[str, float])
object_init_rpy_rad (dict[str, float])
action_pad_yaw_rad (float)
room_light_exposure (float)
env_light_exposure (float)
sky_dome_yaw_rad (float)
robot_color_rgb (dict[str, float])
object_color_rgb (dict[str, float])
object_texture (str)
table_texture (str)
table_form (str)
robot_default_joint_pos_rad (dict[str, float])
look_at_m (dict[str, float])
workspace_origin_m (list[float] | None)

Return type:

None

classmethod from_env(env, episode_index, seed, look_at, object_key='cube', workspace_origin_m=None)[source]#

Capture scene state from a live Isaac Sim environment.

Must be called after env.reset() and before the first env.step().

Parameters:

env (Any) – Gymnasium-wrapped Isaac Lab environment.
episode_index (int) – Current episode number.
seed (int) – Seed used for this reset.
look_at (tuple[float, float, float]) – Camera look-at point (x, y, z) in meters.
object_key (str) – Scene key of the object to track.
workspace_origin_m (tuple[float, float, float] | None)

Return type:

SceneState

Returns:

SceneState with all captured values.

get_joint_pos_list()[source]#

Return joint positions in SO-ARM101 order.

Order: Rotation, Pitch, Elbow, Wrist_Flex, Wrist_Roll, Gripper.

Return type:: list[float]

get_look_at_tuple()[source]#

Return look-at point as (x, y, z) tuple.

Return type:: tuple[float, float, float]

get_object_pos_tuple()[source]#

Return object position as (x, y, z) tuple.

Return type:: tuple[float, float, float]

classmethod load(path)[source]#

Load scene state from JSON file.

Parameters:: path (Path) – Input file path.
Return type:: SceneState
Returns:: SceneState instance.

save(path)[source]#

Save scene state to JSON file.

Parameters:: path (Path) – Output file path (e.g. episode_000_scene_state.json).
Return type:: None

workspace_origin_m: list[float] | None = None#: Workspace-frame {W} origin of the scene this episode was RECORDED in (world coordinates). Declares the frame of object_init_pos_m. None on episodes recorded before this field existed, which means the floor-level table scene (HEMISPHERE_ORIGIN_W).

so101_mvbench.utils.scene_state.rpy_to_quat_xyzw(roll, pitch, yaw)[source]#

Convert RPY Euler angles (ZYX aerospace convention) to XYZW quaternion.

Inverse of _quat_to_rpy. Result matches Isaac Lab’s post-PR-#4437 XYZW convention used by write_root_pose_to_sim and init_state.rot.

Parameters:

roll (float) – Rotation around X-axis in radians.
pitch (float) – Rotation around Y-axis in radians.
yaw (float) – Rotation around Z-axis in radians.

Return type:

tuple[float, float, float, float]

Returns:

(qx, qy, qz, qw) quaternion tuple.

Shared scene builder — single source of truth for scene setup across all stages.

Used by: gamepad_recorder, ego_recorder, external_batch_recorder, eval_policy. Guarantees identical visual scene for the same inputs, regardless of which stage calls it.

class so101_mvbench.utils.scene_builder.VisualConfig(action_pad_yaw_rad=3.141592653589793, room_light_exposure=-1.0, env_light_exposure=1.0, sky_dome_yaw_rad=3.141592653589793, robot_color_rgb=(0.95, 0.95, 0.95), cube_color_rgb=(0.8, 0.2, 0.2), dome_light_y_mirror=True)[source]#

Bases: object

Visual scene properties applied after env.reset().

Default values match _ZERO_SPATIAL / _ZERO_VISUAL from mdp/__init__.py — deterministic ID-replay baseline.

Immutable (frozen). Create OOD variants via dataclasses.replace().

Parameters:

action_pad_yaw_rad (float)
room_light_exposure (float)
env_light_exposure (float)
sky_dome_yaw_rad (float)
robot_color_rgb (tuple[float, float, float])
cube_color_rgb (tuple[float, float, float])
dome_light_y_mirror (bool)

__init__(action_pad_yaw_rad=3.141592653589793, room_light_exposure=-1.0, env_light_exposure=1.0, sky_dome_yaw_rad=3.141592653589793, robot_color_rgb=(0.95, 0.95, 0.95), cube_color_rgb=(0.8, 0.2, 0.2), dome_light_y_mirror=True)#

Parameters:

action_pad_yaw_rad (float)
room_light_exposure (float)
env_light_exposure (float)
sky_dome_yaw_rad (float)
robot_color_rgb (tuple[float, float, float])
cube_color_rgb (tuple[float, float, float])
dome_light_y_mirror (bool)

Return type:

None

so101_mvbench.utils.scene_builder.apply_scene(env, episode_idx, binmap, *, visual=VisualConfig(action_pad_yaw_rad=3.141592653589793, room_light_exposure=-1.0, env_light_exposure=1.0, sky_dome_yaw_rad=3.141592653589793, robot_color_rgb=(0.95, 0.95, 0.95), cube_color_rgb=(0.8, 0.2, 0.2), dome_light_y_mirror=True), cube_noise_m=0.0, cube_rot_noise_deg=0.0)[source]#

Set up the complete scene — robot joints, cube pose, visual properties.

Call AFTER env.reset() and BEFORE warmup steps. Multi-Env safe: applies to ALL environment instances.

Parameters:

env – Gym-wrapped Isaac Lab environment.
episode_idx (int) – Episode index for BinMap lookup.
binmap – BinMap instance with per-episode cube configs.
visual (VisualConfig) – Visual scene configuration. Use ID_VISUAL_CONFIG (default) for in-distribution replay, or a custom VisualConfig for OOD.
cube_noise_m (float) – Additive uniform noise on cube x/y (for OOD eval).
cube_rot_noise_deg (float) – Additive uniform noise on cube yaw (for OOD eval).

Returns:

{"cube_x_m": float, "cube_y_m": float, "cube_yaw_rad": float}.

Return type:

Dict with actually placed cube coordinates

so101_mvbench.utils.scene_builder.load_visual_config(path)[source]#

Load a VisualConfig from a JSON file.

Only fields present in the JSON are overridden — missing fields keep their ID defaults. This allows OOD configs that vary a single parameter.

Example JSON (vary only cube color):

{"cube_color_rgb": [0.2, 0.8, 0.2]}

Return type:: VisualConfig
Parameters:: path (str | Path)

so101_mvbench.utils.scene_builder.resolve_all_prim_paths(env, asset_name)[source]#

Resolve USD prim paths for ALL envs (Multi-Env safe).

Handles AssetBaseCfg/RigidObjectCfg/ArticulationCfg (.cfg.prim_path) and XformPrimView (._prim_path) which has no .cfg attribute. Returns every match for the {ENV_REGEX_NS} pattern.

Return type:: list[str]
Parameters:: asset_name (str)

Deterministic bin-based stratified cube spawner for recording pipeline.

Divides the robot workspace into an n_cols × n_rows grid of bins. Within each bin, a 3×3 sub-grid of deterministic positions is generated (corners, edge midpoints, center). At each position, 3 fixed yaw rotations are applied (0°, 22.5°, 45°). Episodes are scheduled block-wise: all episodes for one bin complete before advancing to the next.

Adapted from leisaac_experiments.scene.sampling with simplifications: - No ObjectRegistry dependency — half_extent_m passed directly. - No load_from_hdf5() — resume exclusively via BinMap JSON sidecar. - Deterministic sub-grid positions instead of least-visited-first random. - Pre-computed schedule with cursor-based traversal.

Designed to be Isaac-Sim-free: no isaaclab imports at module level. Integration with Isaac Lab happens via post-reset teleport in policy_recorder._apply_spawn_config().

BinMap / BinEpisodeRecord provide offline-persistable episode→bin tracking: each recording session saves a JSON sidecar (<dataset>_binmap.json) that maps every episode index to its bin coordinates, pose, and success flag.

class so101_mvbench.utils.bin_spawner.BinEpisodeRecord(episode_idx, bin_col, bin_row, x_m, y_m, yaw_rad, success)[source]#

Bases: object

Single episode entry in the BinMap index.

Parameters:

episode_idx (int)
bin_col (int)
bin_row (int)
x_m (float)
y_m (float)
yaw_rad (float)
success (bool)

episode_idx#: Dataset episode index.

bin_col#: Grid column of the spawned bin.

bin_row#: Grid row of the spawned bin.

x_m#: World-frame spawn x-coordinate (meters).

y_m#: World-frame spawn y-coordinate (meters).

yaw_rad#: Spawn yaw angle (radians).

success#: True if the episode was marked successful (N key).

__init__(episode_idx, bin_col, bin_row, x_m, y_m, yaw_rad, success)#

Parameters:

episode_idx (int)
bin_col (int)
bin_row (int)
x_m (float)
y_m (float)
yaw_rad (float)
success (bool)

Return type:

None

class so101_mvbench.utils.bin_spawner.BinMap(n_cols, n_rows, object_key, episodes=<factory>, ood_bins=<factory>, workspace_origin_m=None)[source]#

Bases: object

Offline-persistable index mapping episode indices to bin coordinates.

Saved as a JSON sidecar file (<dataset>_binmap.json) alongside the dataset. Enables:

Resume after crash: reconstruct BinSpawner cursor from the count of successful episodes.
Offline analysis: inspect per-bin coverage, identify under-sampled bins, filter episodes by bin.

Parameters:

n_cols (int)
n_rows (int)
object_key (str)
episodes (list[BinEpisodeRecord])
ood_bins (frozenset[tuple[int, int]])
workspace_origin_m (tuple[float, float, float] | None)

n_cols#: Number of grid columns.

n_rows#: Number of grid rows.

object_key#: Active object key (matches BinSpawner.object_key).

episodes#: Ordered list of episode records (append-only).

ood_bins#: Set of (col, row) tuples excluded from sampling.

__init__(n_cols, n_rows, object_key, episodes=<factory>, ood_bins=<factory>, workspace_origin_m=None)#

Parameters:

n_cols (int)
n_rows (int)
object_key (str)
episodes (list[BinEpisodeRecord])
ood_bins (frozenset[tuple[int, int]])
workspace_origin_m (tuple[float, float, float] | None)

Return type:

None

get_episode_config(episode_idx)[source]#

Lookup SpawnConfig for a specific dataset episode index.

Parameters:: episode_idx (int) – Dataset episode index (matches BinEpisodeRecord.episode_idx).
Return type:: SpawnConfig | None
Returns:: SpawnConfig reconstructed from the BinMap entry, or None if the episode_idx is not present. If multiple records share the same episode_idx (retries before success), returns the last occurrence.

classmethod load(path)[source]#

Load BinMap from a JSON sidecar file.

Parameters:: path (Path) – Path to an existing _binmap.json file.
Return type:: BinMap
Returns:: Reconstructed BinMap instance.
Raises:: FileNotFoundError – If path does not exist.

log_episode(episode_idx, cfg, success)[source]#

Append an episode record to the index.

Parameters:

episode_idx (int) – Dataset episode index.
cfg (SpawnConfig) – SpawnConfig used for this episode.
success (bool) – True if the episode was accepted (N key).

Return type:

render_ascii(active_col=-1, active_row=-1)[source]#

Render bin coverage as ASCII grid matching the robot’s view.

The grid is rotated 90° CW from local frame to match what the operator sees looking at the robot from behind (robot_yaw=90°): - Columns run left→right = World +Y → -Y - Rows run top→bottom = World +X (far) → -X (near robot)

Cell format:: [ ] — unvisited (no successful episode) [ N] — visited (N = number of successful episodes) [ *] — active (currently selected bin) [OOD] — out-of-distribution (excluded from sampling)

Parameters:

active_col (int) – Column index of the currently active bin (-1 = none).
active_row (int) – Row index of the currently active bin (-1 = none).

Return type:

str

Returns:

Multi-line string ready for print().

save(path)[source]#

Save BinMap to a JSON sidecar file.

Parameters:: path (Path) – Destination file path.
Return type:: None

visit_counts_grid()[source]#

Return n_rows × n_cols grid of successful episode counts.

Return type:: list[list[int]]
Returns:: 2-D list indexed as grid[row][col], counting only episodes where success=True.

workspace_origin_m: tuple[float, float, float] | None = None#: Workspace-frame {W} origin of the scene the episodes were RECORDED in, expressed in that scene’s world frame. Declares which frame the episode x_m/y_m coordinates live in, so a replay into a different scene can derive its rigid shift. None on datasets recorded before this field existed, which means the floor-level table scene (HEMISPHERE_ORIGIN_W).

class so101_mvbench.utils.bin_spawner.BinSpawner(workspace, n_cols, n_rows, half_extent_m, object_key='cube', robot_yaw_rad=0.0, ood_bins=frozenset({}), selected_bin=None)[source]#

Bases: object

Deterministic bin-based stratified cube spawner.

Pre-computes a schedule of all episodes at construction time:

For each trainable bin (block-wise): generate 3×3 sub-grid positions.
For each position: apply the 3 fixed yaw rotations in ROTATIONS_RAD (0°, 22.5°, 45°).
Episodes are traversed via cursor: N advances, R retries.

All returned coordinates are in the world frame: world = workspace.origin + rotate(local, robot_yaw)

Parameters:

workspace (WorkspaceBounds) – Physical bounds of the spawn region.
n_cols (int) – Number of grid columns.
n_rows (int) – Number of grid rows.
half_extent_m (float) – Object half-extent for safe_bounds margin.
object_key (str) – Object key stored in SpawnConfig (default “cube”).
robot_yaw_rad (float) – Robot base yaw angle in radians for rotating local workspace offsets into world frame. Default 0.0.
ood_bins (frozenset[tuple[int, int]]) – Set of (col, row) tuples to exclude from sampling.
selected_bin (tuple[int, int] | None)

__init__(workspace, n_cols, n_rows, half_extent_m, object_key='cube', robot_yaw_rad=0.0, ood_bins=frozenset({}), selected_bin=None)[source]#

Parameters:

workspace (WorkspaceBounds)
n_cols (int)
n_rows (int)
half_extent_m (float)
object_key (str)
robot_yaw_rad (float)
ood_bins (frozenset[tuple[int, int]])
selected_bin (tuple[int, int] | None)

Return type:

None

property current_bin: tuple[int, int] | None#: (col, row) of the currently active bin, or None.

property episodes_per_bin: int#: Number of episodes per bin (positions × rotations).

mark_retry()[source]#

Mark current episode for retry (cursor stays).

Called on R-key (retry) — next env.reset() will repeat the same position and rotation.

Return type:: None

mark_success()[source]#

Mark current episode as success and advance cursor.

Called on N-key (success) — next env.reset() will pick the next episode in the schedule.

Return type:: None

property n_trainable_bins: int#: Number of bins available for sampling (excluding OOD).

next_config()[source]#

Return the current spawn config, advancing if flagged.

Called by _apply_spawn_config() after each env.reset(). Respects the _advance flag set by mark_success() / mark_retry().

Return type:: SpawnConfig
Returns:: SpawnConfig for the current episode.
Raises:: StopIteration – If all episodes have been recorded.

property positions_per_bin: int#: Number of sub-grid positions per bin (always 9).

property remaining: int#: Number of episodes not yet recorded.

restore_from_binmap(bin_map)[source]#

Restore cursor position from a persisted BinMap.

Sets cursor to the number of successful episodes, which corresponds to the next unrecorded position in the schedule.

Parameters:: bin_map (BinMap) – BinMap loaded from a JSON sidecar file.
Raises:: ValueError – If bin_map grid dimensions differ from this spawner.
Return type:: None

property total_episodes: int#: Total number of episodes in the schedule.

class so101_mvbench.utils.bin_spawner.BinState(*values)[source]#

Bases: Enum

Visual state of a single bin cell for display purposes.

class so101_mvbench.utils.bin_spawner.SpawnConfig(object_key, x_m, y_m, yaw_rad, bin_col, bin_row)[source]#

Bases: object

Concrete spawn configuration for a single episode.

All coordinates are in the world frame (origin offset applied).

Parameters:

object_key (str)
x_m (float)
y_m (float)
yaw_rad (float)
bin_col (int)
bin_row (int)

object_key#: Registry key of the spawned object (e.g. “cube”).

x_m#: World-frame x position (meters).

y_m#: World-frame y position (meters).

yaw_rad#: Object yaw rotation (radians).

bin_col#: Grid column index of the containing bin.

bin_row#: Grid row index of the containing bin.

__init__(object_key, x_m, y_m, yaw_rad, bin_col, bin_row)#

Parameters:

object_key (str)
x_m (float)
y_m (float)
yaw_rad (float)
bin_col (int)
bin_row (int)

Return type:

None

classmethod from_dict(d)[source]#

Deserialize from dict.

Return type:: SpawnConfig
Parameters:: d (dict)

to_dict()[source]#

Serialize to dict for JSON storage.

Return type:: dict

class so101_mvbench.utils.bin_spawner.WorkspaceBounds(x_min_m, x_max_m, y_min_m, y_max_m, origin_x=0.0, origin_y=0.0, yaw_min_rad=-3.141592653589793, yaw_max_rad=3.141592653589793)[source]#

Bases: object

Axis-aligned bounding box of the robot’s reachable workspace.

Min/max values are LOCAL — relative to (origin_x, origin_y). BinSpawner converts to world coordinates as: world = origin + local.

This allows the same workspace shape to be re-used across different table/arm positions simply by changing origin_x/origin_y.

Parameters:

x_min_m (float)
x_max_m (float)
y_min_m (float)
y_max_m (float)
origin_x (float)
origin_y (float)
yaw_min_rad (float)
yaw_max_rad (float)

x_min_m#: Minimum x bound relative to origin (meters).

x_max_m#: Maximum x bound relative to origin (meters).

y_min_m#: Minimum y bound relative to origin (meters).

y_max_m#: Maximum y bound relative to origin (meters).

origin_x#: World-frame x coordinate of the workspace origin (meters).

origin_y#: World-frame y coordinate of the workspace origin (meters).

yaw_min_rad#: Minimum yaw angle (radians). Default: -π.

yaw_max_rad#: Maximum yaw angle (radians). Default: +π.

__init__(x_min_m, x_max_m, y_min_m, y_max_m, origin_x=0.0, origin_y=0.0, yaw_min_rad=-3.141592653589793, yaw_max_rad=3.141592653589793)#

Parameters:

x_min_m (float)
x_max_m (float)
y_min_m (float)
y_max_m (float)
origin_x (float)
origin_y (float)
yaw_min_rad (float)
yaw_max_rad (float)

Return type:

None

classmethod from_center(center_x, center_y, n_cols, n_rows, bin_size, **kwargs)[source]#

Create bounds from grid center, dimensions, and bin size.

Computes x/y extents as center ± (n × bin_size) / 2. Any extra keyword arguments (origin_x, origin_y, yaw_*) are forwarded to the constructor.

Parameters:

center_x (float) – Grid center x in local frame (meters).
center_y (float) – Grid center y in local frame (meters).
n_cols (int) – Number of grid columns.
n_rows (int) – Number of grid rows.
bin_size (float) – Size of each bin in meters.
**kwargs – Forwarded to WorkspaceBounds (e.g. origin_x, origin_y).

Return type:

WorkspaceBounds

Returns:

WorkspaceBounds with computed x/y limits.

safe_bounds(half_extent_m)[source]#

Return inset bounds keeping an object fully within workspace.

Shrinks x/y bounds by half_extent_m on each side. Origin and yaw are preserved unchanged.

Parameters:: half_extent_m (float) – Object half-extent in meters.
Return type:: WorkspaceBounds
Returns:: New WorkspaceBounds with inset x/y limits.
Raises:: ValueError – If margin makes bounds degenerate (max <= min).

so101_mvbench.utils.bin_spawner.apply_cube_pose(env, x_m, y_m, yaw_rad, *, env_origins=None, cube_key='cube')[source]#

Teleport cube to (x, y) with yaw rotation + zero velocity.

Single source of truth for cube spawn, shared by the recorder (teleop_recorder), the replay (multicam_replay), and the eval scene loader (scene_loader).

Quaternion is written to root_state[:, 3:7] in XYZW order (Isaac Lab post-PR #4437 convention, confirmed in base_rigid_object.py:173 docstring). Pure Z-rotation by yaw_rad: (qx=0, qy=0, qz=sin(yaw/2), qw=cos(yaw/2)).

Velocity is zeroed explicitly (root_state[:, 7:] = 0.0) — fixes the “cube retains momentum between episodes” bug where write_root_pose_to_sim was used (which leaves velocity untouched).

Z-position stays at default_root_state[:, 2] (cube spawn height from ArticulationCfg.init_state.pos.z). Matches original recording behavior.

Parameters:

env – Gymnasium-wrapped Isaac Lab environment.
x_m (float) – Target cube X position in meters (world-frame or env-local).
y_m (float) – Target cube Y position in meters.
yaw_rad (float) – Cube yaw rotation in radians (around Z-axis).
env_origins – Per-env world-frame origin tensor of shape (num_envs, 3), required for multi-env (cube gets env_origins + x/y as world pos). Pass None for single-env (default).
cube_key (str) – Scene key for the cube asset (default “cube”).

Return type: