Depth instead of colour, and other things this cannot do yet

Depth instead of colour, and other things this cannot do yet#

The two experiments in this chapter work because the pipeline already carries what they need. This page is the other half of the picture: the study that would fit here and does not run, and why.

Depth versus colour#

The natural next question after “which viewpoint” is “which modality”. A wrist-plus-top pair sees the cube from close range and from above, which is exactly the geometry where depth should beat colour, and where the comparison is cheap because both modalities can be rendered from the same demonstration.

The wrist and top views as colour and as depth, the same frame

The same frame as colour and as depth. The wrist view is where the difference bites: in colour the cube is a red patch against a grey table, in depth it is a raised block with an edge, and the gripper jaws stand out as the nearest thing in the frame.

Recorded during the study this repository was cut from, in the research workspace rather than here.

This does not run today. Every camera in the shipped scene is fixed to colour:

# tasks/lift_cube_env_cfg.py
data_types=["rgb"],

The same is true of the scene-configuration schema, which describes a viewpoint’s position, resolution and lens but has no notion of a channel. So a depth study would need a change in four places, not one:

Where

What is missing

utils/scene_config.py

A modality field on CameraSpec, so a configuration can ask for depth

tasks/lift_cube_env_cfg.py

data_types derived from that field instead of hard-coded

recording/multicam_replay.py

Writing a depth stream into the dataset, which is a different dtype and range than an 8-bit image

training/train_act_camera_perm.py

A shape entry for a single-channel input, since _CAMERA_SHAPES assumes three

None of that is deep, but it is a real change through the whole chain, and it touches the dataset format. It is not a flag.

Note

Isaac Sim renders depth perfectly well: the camera type used here supports distance_to_image_plane alongside rgb. The gap is in this pipeline, not in the simulator.

What else is out of reach#

Question

Why it does not fit today

Does this transfer to a real arm?

The pipeline is simulation-only by design. Nothing here deploys, and no calibration path exists (Requirements).

Does another policy architecture rank viewpoints differently?

Training wraps LeRobot’s ACT specifically. A second architecture means a second trainer.

Does the ranking hold on another task?

The task is an ordinary Isaac Lab environment and can be replaced, but the tracker’s success criterion is written for lift-and-return. A new task needs its own criterion.

Why say this at all#

A framework that lists what it cannot do is easier to trust than one that implies everything is possible. If you need depth, you now know it is a four-file change rather than a missing flag, and you know where to start.

Where to go next#