Which camera is the policy actually using?

Which camera is the policy actually using?#

A multi-view policy sees several streams, and its success rate says nothing about which of them it depends on. It may be leaning on one view and treating the rest as decoration. This experiment finds out by degrading one camera at a time, at inference only, and watching what the success rate does.

No retraining is involved. The same checkpoint is evaluated repeatedly, and the only thing that changes is what one of its cameras shows it.

The wrist camera is zeroed here, and the policy keeps running on what is left. Whether it still succeeds is the measurement.

How it works#

Degradation happens at the boundary where a rendered frame is handed to the policy. The same transformed frame goes into the policy, the frame dumps and the recorded videos, so the video is an honest record of what the policy saw.

Three flags control it:

Flag	Meaning
`--augment_cameras`	Comma list of policy camera names to degrade, for example `wrist` or `front,top`. Nothing happens without it.
`--augment`	Which degradation. Default `blackout`.
`--augment_strength`	How strong, interpreted per mode (see below). Required for `gauss` and `blur`, which have no meaningful zero.

Available modes, from evaluation/augmentation.py:

Mode	What it does	Strength means
`blackout`	Zeroes the frame: total sensor outage	ignored
`gauss`	Adds Gaussian pixel noise	sigma on a 0 to 1 scale, so `0.05` is mild and `0.4` is severe
`blur`	Gaussian blur	odd kernel size, at least 3. `5` is mild, `31` is heavy
`invert`	Inverts colours	ignored
`contrast`	Stretches contrast around mid-grey	factor, above 1 is stronger
`canny`	Replaces the image with its edge map	upper Canny threshold

The first three are the degradation family used for camera importance. The others exist to ask a different question, which is what representation the policy needs rather than how much signal.

What the three degradations do to a frame, at the strengths used above:

The same wrist, front and top frames as original, blacked out, with Gaussian noise, and blurred

Blackout removes the camera outright. Gaussian noise at sigma 0.15 leaves the scene readable to a human but adds pixel-level noise the policy never saw in training. Blur at kernel 15 keeps the layout and destroys the fine detail around the gripper and the cube.

Which combinations to run#

Every run is one cell of a matrix, and each block of that matrix answers a different question. For a policy with three cameras it comes to thirty cells:

Block	Cells	Condition	The question it answers
Control	1	nothing degraded	What is this policy worth today, on this bin, with this seed? Everything else is measured against it.
Knockout	3	`blackout`, one camera at a time	Which camera is load-bearing? The bluntest cut there is.
Noise ladder	9	`gauss` at 0.05, 0.15, 0.40, one camera at a time	Does the policy need the camera, or does it merely prefer it? A camera that survives sigma 0.40 was barely being read.
Blur ladder	9	`blur` at kernel 5, 15, 31, one camera at a time	Is it the fine detail or the layout? Blur keeps where things are and removes what they look like.
Leave one out	6	`gauss` 0.40 and `blur` 31 on every camera except one	Can that one camera carry the task alone?
All at once	2	`gauss` 0.40 and `blur` 31 on all three	The floor. Whatever remains is not coming from the cameras.

The count is 3 + 9n for n cameras, so two make twenty-one cells and four make thirty-nine. The ladders are the part worth keeping when the budget is tight, because a knockout on its own cannot separate a camera the policy depends on from one it glances at.

Pick the bins with the same care. A single interior bin flatters the policy, and camera importance measured only there does not have to hold in a corner where the cube is hard to reach.

Running it#

Establish the baseline first, with no degradation at all, because the comparison is against this number and not against an expectation:

# adjust: the policy, the bin it is evaluated on, and where results go
CKPT=outputs/3cam_wrist_front_top/checkpoints/065000/pretrained_model
BIN=datasets/01_raw_gamepad/bin_c2_r2
SCENE=src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json
OUT=outputs/3cam_wrist_front_top/camera_importance

python -m so101_mvbench.evaluation.async_eval \
    --mode async --num_envs 10 --num_episodes 27 \
    --policy_path "$CKPT" --scene_config "$SCENE" --dataset_root "$BIN" \
    --output_dir "$OUT/baseline" \
    --seed 42 --settle_steps 130 --full_report --headless

Then knock out one camera completely. Repeat once per camera the policy has:

# adjust: the policy, the bin it is evaluated on, and where results go
CKPT=outputs/3cam_wrist_front_top/checkpoints/065000/pretrained_model
BIN=datasets/01_raw_gamepad/bin_c2_r2
SCENE=src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json
OUT=outputs/3cam_wrist_front_top/camera_importance

python -m so101_mvbench.evaluation.async_eval \
    --mode async --num_envs 10 --num_episodes 27 \
    --policy_path "$CKPT" --scene_config "$SCENE" --dataset_root "$BIN" \
    --output_dir "$OUT/blackout_wrist" \
    --augment_cameras wrist --augment blackout \
    --seed 42 --settle_steps 130 --full_report --headless

A graded sweep says more than a knockout, because it separates a camera the policy needs from one it merely prefers:

# adjust: the policy, the bin, where results go, and the camera under test
CKPT=outputs/3cam_wrist_front_top/checkpoints/065000/pretrained_model
BIN=datasets/01_raw_gamepad/bin_c2_r2
SCENE=src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json
OUT=outputs/3cam_wrist_front_top/camera_importance
CAM=wrist

for s in 0.05 0.15 0.40; do
    python -m so101_mvbench.evaluation.async_eval \
        --mode async --num_envs 10 --num_episodes 27 \
        --policy_path "$CKPT" --scene_config "$SCENE" --dataset_root "$BIN" \
        --output_dir "$OUT/gauss_${CAM}_${s}" \
        --augment_cameras "$CAM" --augment gauss --augment_strength "$s" \
        --seed 42 --settle_steps 130 --full_report --headless
done

Blur takes an odd kernel instead of a fraction:

# adjust: the policy, the bin, where results go, and the camera under test
CKPT=outputs/3cam_wrist_front_top/checkpoints/065000/pretrained_model
BIN=datasets/01_raw_gamepad/bin_c2_r2
SCENE=src/so101_mvbench/tasks/scene_configs/lift_cube_6cam.json
OUT=outputs/3cam_wrist_front_top/camera_importance
CAM=wrist

for k in 5 15 31; do
    python -m so101_mvbench.evaluation.async_eval \
        --mode async --num_envs 10 --num_episodes 27 \
        --policy_path "$CKPT" --scene_config "$SCENE" --dataset_root "$BIN" \
        --output_dir "$OUT/blur_${CAM}_${k}" \
        --augment_cameras "$CAM" --augment blur --augment_strength "$k" \
        --seed 42 --settle_steps 130 --full_report --headless
done

Reading the result#

Every run writes its own eval_summary.json, so the comparison is a table of success rates against the baseline. Two cautions decide whether the table means anything:

Repeat before concluding. A single run’s success rate carries the measurement’s own noise. Run each cell the way the campaign does, several times with the same seed, and compare the difference against that spread. A drop smaller than the spread is not a finding.
Watch the videos of the degraded run. The degradation reaches the recorded video, so you can see exactly what the policy was working with. A policy that fails under blackout on a camera it never used is a bug somewhere else.

The interesting outcome is asymmetry: if blacking out one camera costs most of the success rate while the others cost little, that camera is carrying the policy, and the others were along for the ride.

What the matrix looked like when it was run#

The numbers below come from the study this repository was cut from, so they are one policy’s answer and not a property of the method. The policy has the three cameras wrist, front and top, each cell is 27 episodes at seed 42, and each cell was run once.

Two bins were swept, because a single cube position cannot show whether camera importance is a property of the policy or of where the cube happens to be: an easy interior position, and the hardest corner.

The interior bin first:

Success rate per degraded camera and the delta against the reference, interior bin

One camera carries this policy. Blacking out the wrist takes it from a clean 100 percent to zero, and noise at sigma 0.40 on the wrist does the same without removing a single pixel of the layout. The other two are close to free under noise: front and top survive every strength the ladder offers, and top under the heaviest blur still scores 100 percent. Blur is where the wrist finally separates from the others, dropping to 14.8 percent at kernel 31 while front stays at 92.6.

Blacking a camera out is a different matter, and the heatmap is where to read it: removing front outright still costs 39 points. So “the policy barely reads front” holds for degraded pixels and not for no pixels at all, which is the distinction the noise ladder exists to draw.

Two reference lines appear in the figure and they are not the same number. The dashed one is the bin’s success rate from the main evaluation campaign, ten runs of the undegraded policy, and the heatmap deltas are measured against it. The dotted one is the control cell of this sweep, a single run. Those two differing by about six points is the measurement noise, drawn to scale.

The corner bin tells a different story:

Success rate per degraded camera and the delta against the reference, corner bin

Here every camera costs something. Blacking out front or top costs about 13 points each, where in the interior top cost 16 and front 39, and the wrist no longer has a monopoly. What changed is not the policy but the difficulty: at 70 percent to start with, there is less slack to absorb a degraded view.

The leave-one-out block asks the sharper question, whether a camera can carry the task by itself. Every camera except the named one is degraded:

Only clean camera	interior, gauss 0.40	interior, blur 31	corner, gauss 0.40	corner, blur 31
`wrist`	88.9	96.3	3.7	33.3
`front`	0.0	37.0	0.0	22.2
`top`	0.0	48.1	0.0	0.0
none, all three degraded	0.0	29.6	0.0	0.0
undegraded control	100.0	100.0	70.4	70.4

In the interior the wrist alone is worth almost the whole policy, 88.9 against a control of 100. In the corner the same camera alone yields 3.7. So “the wrist is the important camera” is true where the cube is easy to reach and false where it is not, which is exactly the kind of claim a single bin would have gotten wrong.

One honest caveat, the same one this page gives above: these cells were run once each. A six-point difference between two of them is noise. A ninety-point one is not, and every conclusion drawn here rests on the large drops.

What this cannot tell you#

It measures dependence of this trained policy on this camera, not the value of the viewpoint in general. A policy trained without a camera might have learned to use another one better. To ask that question, train separate policies per camera subset and compare them (Stage 3 — Training).

Where to go next#

What happens when the external camera moves? — the same idea for a camera that moved rather than degraded
Stage 4 — Evaluation — the protocol these runs inherit
Stage 5 — Reporting — turning the runs into numbers