Literature
Display
← All papers

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

Evans Han · Yunfan Jiang · Yingke Wang · Haoyue Xiao · Huang Huang · Jianwen Xie · Jiajun Wu · Li Fei-Fei · Ruohan Zhang

Preprint posted · arXiv (preprint) ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

StereoPolicy learns task-relevant binocular features directly from rectified image pairs, retaining a conventional diffusion or vision-language-action policy without reconstructing depth.

Refer to caption
Figure 1: Compared with other visual modalities for robot learning, using stereo image pairs directly as policy input provides implicit 3D information without a depth sensor, works with pretrained 2D encoders, and is easy to deploy on real robots.

At a glance

A shared two-dimensional image encoder and alternating self/cross-attention fuse synchronized stereo observations into action-conditioned features. Tests span three simulation benchmarks and seven physical tabletop/mobile tasks. Stereo fusion improves the tested monocular and simple two-image controls, including pretrained VLA backbones. Its strongest claims concern these matched configurations: transparent glass remains difficult, and avoiding explicit depth reconstruction does not remove camera setup requirements.

1 Introduction

Monocular policies can confuse depth and precise alignment, while reconstructed depth/point clouds introduce sensing errors, missing transparent geometry, and processing latency. StereoPolicy trains binocular fusion through the action objective, using existing two-dimensional visual priors without explicit depth supervision. The study examines task success, camera geometry, encoder/fusion design, and adaptation of pretrained VLAs. It does not claim that all point-cloud policies or all pretrained stereo models are intrinsically inferior.

2 Method

The same perception idea is used in two settings: a diffusion-policy backbone trained from scratch on each task, and pretrained VLA backbones finetuned with stereo features. Within each camera rig the left/right pair is fused; embeddings from separate rigs and robot state are then aggregated.

Refer to caption

Figure 2: StereoPolicy Pipeline. Stereo inputs are encoded by a vision backbone, fused with a Stereo Transformer, and applied to both diffusion policy training and fine-tuning VLA baselines.

2.1 Stereo Feature Extraction

Each camera view supplies a synchronized left/right image pair. The encoder produces binocular features for control rather than an explicit disparity map, depth map, or reconstructed three-dimensional scene.

Stereo image encoding

Left and right images share encoder weights. ResNet18/FPN features are augmented with frozen DINOv2 features on external views, then projected to a common dimension. DINOv2 hurts wrist-view performance in this study and is omitted there; the suggested explanation of egocentric domain mismatch is a hypothesis, not an independently isolated mechanism.

Equation 1
f~t,i⋅=ϕ([ft,i⋅∥ft,iDINO]),\tilde{f}_{t,i}^{\cdot}=\phi\big([f_{t,i}^{\cdot}\|f_{t,i}^{\text{DINO}}]\big), (1)

Stereo feature fusion

The Stereo Transformer alternates attention within each image and cross-attention between images. Two-dimensional rotary positional embeddings encode image coordinates in cross-attention queries and keys. The action loss trains these correspondences without a disparity target.

Equation 2
∀i∈{1,…,N}:zt,i=StereoTransi​(f~t,iL,f~t,iR)∈ℝdz,\forall i\in\{1,\dots,N\}:\quad z_{t,i}=\text{StereoTrans}_{i}\big(\tilde{f}_{t,i}^{L},\tilde{f}_{t,i}^{R}\big)\in\mathbb{R}^{d_{z}}, (2)

Multi-view aggregation

Fusion is performed separately within each stereo camera rig. The resulting view embeddings are concatenated with proprioception to condition the action backbone; this step does not explicitly reconstruct a common three-dimensional scene.

Equation 3
zobs=[zt,1​‖zt,2​‖⋯‖​zt,N‖​st].z_{\text{obs}}=[\,z_{t,1}\|z_{t,2}\|\cdots\|z_{t,N}\|s_{t}\,]. (3)
Refer to caption
Figure 3: Real-World Task Visualizations. Top: Five tabletop tasks, with the final state shown for each. Tasks 3–5 vary by cup material. Bottom: Two mobile manipulation tasks.

2.2 StereoPolicy-DP : Diffusion Policy with StereoPolicy

Diffusion Policy denoises a future action sequence conditioned on stereo observations. The encoder, stereo fusion, and action network are optimized jointly with the standard noise-prediction loss; DINOv2 remains frozen. At inference, iterative denoising produces the action sequence from Gaussian noise. The stereo representation changes conditioning while retaining the diffusion policy objective.

Equation 4
ℒDP=𝔼⁡[‖ϵ−ϵθ​(x(τ),τ,zobs)‖22],\mathcal{L}_{\mathrm{DP}}=\mathbb{E}\!\left[\left\|\epsilon-\epsilon_{\theta}(x^{(\tau)},\tau,z_{\text{obs}})\right\|_{2}^{2}\right], (4)

2.3 StereoPolicy-VLA : Adapting Monocular VLA to Stereo Inputs

For a pretrained VLA, shared visual encoding and stereo fusion replace the monocular image embedding while language, state, and the action backbone retain their interfaces. Finetuning adapts the system to downstream tasks. Preserving the backbone architecture does not mean the entire pretrained system is frozen.

Equation 5
𝐚=πθ​(zobs,xlang,s),\mathbf{a}=\pi_{\theta}(z_{\text{obs}},x_{\text{lang}},s), (5)

3 Experiments

Experiments ask whether stereo fusion improves matched RGB/depth/point-cloud controls, transfers to pretrained VLAs, depends on camera geometry, and can be replaced by generic pretrained stereo representations.

Refer to caption
Figure 4: Simulation Task Visualization, from three benchmarks: OmniGibson (4 tasks), RoboCasa (24 tasks), and Robomimic (3 tasks).

3.1 Tasks and Environments

Hardware comprises a seven-joint Franka tabletop arm and a twenty-four-degree-of-freedom Galaxea R1 Pro mobile bimanual robot. The study uses five tabletop and two mobile real tasks, three Robomimic tasks, twenty-four RoboCasa tasks, and four custom OmniGibson tasks.

3.2 Implementation Details

Training and comparisons keep action backbones, data, image resolution, optimization, and budgets matched where applicable. The appendix gives modality-specific preprocessing and additional parameter-matched controls.

Baselines

Controls include monocular RGB, concatenated two-view RGB, RGB-D, PointNet point clouds, 3D Diffuser Actor, and DP3. The monocular control uses the same external DINO features and adjusted attention capacity. The multi-view control receives the same left/right images as StereoPolicy but concatenates features through a capacity-matched projection, helping distinguish stereo fusion from merely adding a second image.

4 Results and Analysis

Results combine task success, VLA transfer, camera sweeps, representation substitutions, and ablations. The appendix specifies that simulated scores select the highest success over periodic checkpoint evaluations; these are not simply final-checkpoint estimates.

4.1 Experimental Results

StereoPolicy improves the main tested baselines across many conditions, with important scope limits in low absolute glass success, modest VLA gains, and configuration-specific geometric studies.

Method Banana PnP Toast Insert Plastic Cup Hang Steel Cup Hang Glass Cup Hang AVG SR (%)
RGB 12/20 8/20 12/20 10/20 1/20 43.0%
RGB-D 14/20 8/20 11/20 8/20 0/20 41.0%
RGBD-3DDA 13/20 9/20 13/20 10/20 0/20 45.0%
PCD-PointNet 7/20 0/20 5/20 2/20 0/20 14.0%
PCD-DP3 11/20 3/20 8/20 5/20 0/20 27.0%
Multi-View 13/20 9/20 13/20 9/20 1/20 45.0%
StereoPolicy-DP 16/20 12/20 15/20 13/20 3/20 59.0%
Table 1: Real-World Tabletop Task Performance. StereoPolicy-DP consistently outperforms other visual modalities. PCD performs worst in all real tasks; both RGB-D and PCD fail on glass cup hang tasks, demonstrating that explicit 3D methods are sensitive to object texture in practice.
Benchmark OmniGibson Robomimic
Task Strawberry PourWater OpenDoor TurnOnRadio ToolHang Square Transport
# Demos 100 200 100 200 200 300 200 300 100 200 100 200 100 200
RGB 623.362_{\scriptscriptstyle 3.3} 861.686_{\scriptscriptstyle 1.6} 154.115_{\scriptscriptstyle 4.1} 502.850_{\scriptscriptstyle 2.8} 303.330_{\scriptscriptstyle 3.3} 801.680_{\scriptscriptstyle 1.6} 462.846_{\scriptscriptstyle 2.8} 742.874_{\scriptscriptstyle 2.8} 533.453_{\scriptscriptstyle 3.4} 901.690_{\scriptscriptstyle 1.6} 761.976_{\scriptscriptstyle 1.9} 980.998_{\scriptscriptstyle 0.9} 921.992_{\scriptscriptstyle 1.9} 941.694_{\scriptscriptstyle 1.6}
RGB-D 633.863_{\scriptscriptstyle 3.8} 854.785_{\scriptscriptstyle 4.7} 161.616_{\scriptscriptstyle 1.6} 520.052_{\scriptscriptstyle 0.0} 312.531_{\scriptscriptstyle 2.5} 804.380_{\scriptscriptstyle 4.3} 470.947_{\scriptscriptstyle 0.9} 733.473_{\scriptscriptstyle 3.4} 560.056_{\scriptscriptstyle 0.0} 882.588_{\scriptscriptstyle 2.5} 791.979_{\scriptscriptstyle 1.9} 920.092_{\scriptscriptstyle 0.0} 𝟗𝟒1.6\mathbf{94}_{\scriptscriptstyle 1.6} 941.694_{\scriptscriptstyle 1.6}
RGBD-3DDA 744.374_{\scriptscriptstyle 4.3} 931.993_{\scriptscriptstyle 1.9} 260.026_{\scriptscriptstyle 0.0} 613.461_{\scriptscriptstyle 3.4} 483.348_{\scriptscriptstyle 3.3} 𝟏𝟎𝟎0.0\mathbf{100}_{\scriptscriptstyle 0.0} 464.346_{\scriptscriptstyle 4.3} 754.175_{\scriptscriptstyle 4.1} 841.684_{\scriptscriptstyle 1.6} 922.892_{\scriptscriptstyle 2.8} 831.983_{\scriptscriptstyle 1.9} 972.597_{\scriptscriptstyle 2.5} 𝟗𝟒1.6\mathbf{94}_{\scriptscriptstyle 1.6} 𝟗𝟔1.6\mathbf{96}_{\scriptscriptstyle 1.6}
PCD-DP3 453.445_{\scriptscriptstyle 3.4} 632.563_{\scriptscriptstyle 2.5} 34.73_{\scriptscriptstyle 4.7} 310.931_{\scriptscriptstyle 0.9} 301.630_{\scriptscriptstyle 1.6} 694.169_{\scriptscriptstyle 4.1} 351.935_{\scriptscriptstyle 1.9} 644.964_{\scriptscriptstyle 4.9} 401.640_{\scriptscriptstyle 1.6} 760.976_{\scriptscriptstyle 0.9} 692.569_{\scriptscriptstyle 2.5} 881.688_{\scriptscriptstyle 1.6} 632.563_{\scriptscriptstyle 2.5} 722.872_{\scriptscriptstyle 2.8}
Multi-View 703.370_{\scriptscriptstyle 3.3} 912.591_{\scriptscriptstyle 2.5} 243.424_{\scriptscriptstyle 3.4} 563.356_{\scriptscriptstyle 3.3} 354.335_{\scriptscriptstyle 4.3} 822.882_{\scriptscriptstyle 2.8} 473.447_{\scriptscriptstyle 3.4} 772.577_{\scriptscriptstyle 2.5} 642.864_{\scriptscriptstyle 2.8} 921.992_{\scriptscriptstyle 1.9} 801.680_{\scriptscriptstyle 1.6} 960.996_{\scriptscriptstyle 0.9} 921.992_{\scriptscriptstyle 1.9} 941.694_{\scriptscriptstyle 1.6}
StereoPolicy-DP 𝟖𝟐2.8\mathbf{82}_{\scriptscriptstyle 2.8} 𝟏𝟎𝟎0.0\mathbf{100}_{\scriptscriptstyle 0.0} 𝟑𝟒1.6\mathbf{34}_{\scriptscriptstyle 1.6} 𝟕𝟎2.5\mathbf{70}_{\scriptscriptstyle 2.5} 𝟓𝟕1.9\mathbf{57}_{\scriptscriptstyle 1.9} 𝟏𝟎𝟎0.0\mathbf{100}_{\scriptscriptstyle 0.0} 𝟓𝟓3.4\mathbf{55}_{\scriptscriptstyle 3.4} 𝟖𝟐4.3\mathbf{82}_{\scriptscriptstyle 4.3} 𝟗𝟒1.9\mathbf{94}_{\scriptscriptstyle 1.9} 𝟗𝟔2.8\mathbf{96}_{\scriptscriptstyle 2.8} 𝟖𝟖0.9\mathbf{88}_{\scriptscriptstyle 0.9} 𝟏𝟎𝟎0.0\mathbf{100}_{\scriptscriptstyle 0.0} 𝟗𝟒0.0\mathbf{94}_{\scriptscriptstyle 0.0} 𝟗𝟔1.6\mathbf{96}_{\scriptscriptstyle 1.6}
Table 2: Simulation Task Performance of Diffusion Policies across Visual Modalities. Stereo input consistently improves performance, especially under low-data regime. We report average scores and standard deviations across three seed runs. All numbers are reported in percentage (%).
Refer to caption
Figure 5: Average StereoPolicy-VLA Performance on RoboCasa-Kitchen 24 Tasks.
Refer to caption
Figure 6: StereoPolicy-VLA (π0.5\pi_{0.5}) Performance on bimanual mobile manipulation tasks in both real-world and simulation.

( 𝒬 ​ 𝟏 ) StereoPolicy-DP consistently outperforms monocular RGB, RGB-D, point cloud, and multi-view-based policies.

Across five physical tabletop tasks, StereoPolicy succeeds in 59% of the pooled equal-sized trials, versus 43% RGB, 41% RGB-D, 45% 3DDA, 14% PointNet, 27% DP3, and 45% matched multi-view RGB. Individual StereoPolicy counts are 16/20 banana, 12/20 toast, 15/20 plastic cup, 13/20 steel cup, and only 3/20 glass cup. In simulation, examples include ToolHang with one hundred demos improving from 53±3.4% RGB to 94±1.9%, and PourWater with two hundred demos improving from 50% to 70%. Some strong depth baselines tie stereo in particular settings; the evidence supports this task suite rather than universal dominance.

Refer to caption
Figure 7: RGB-D and PCD are fragile in real environments. The glass cup is entirely missing.

( 𝒬 ​ 𝟐 ) StereoPolicy enhances pretrained VLA models, although these models are trained on monocular data.

Stereo fusion improves π0.5 and GR00T-N1.5 across thirty, one hundred, and three hundred demonstrations per RoboCasa task. At three hundred demos, π0.5 rises from 70.31% monocular to 74.40%, and GR00T from 64.30% to 67.50%. Native two-image controls recover some gains, so the appendix separately measures fusion’s added benefit. On mobile π0.5 tasks, real toast pick/place improves 70%→85% and radio 40%→60%; simulated door/radio improve 80%→90% and 70%→85%. These finite task results do not establish unrestricted mobile autonomy.

( 𝒬 ​ 𝟑 ) StereoPolicy-DP is most effective when the baseline is approximately 10% of the target object distance.

The ToolHang sweep varies baselines from two to ten centimetres and camera distance roughly 0.6–1.0 metres. The reported best region is a baseline/distance ratio around 0.09–0.13, balancing disparity and overlap; this is an empirical guideline for the tested optics, task, and training configurations rather than a universal camera constant. Stereo improves all ten tested views, with larger gains for frontal views (+18 points, or +14 elevated) and smaller gains for occluded side views. A separate fixed-policy test evaluates actual baseline transfer.

Equation 6
r=stereo baselinecamera-object distancer=\frac{\text{stereo baseline}}{\text{camera-object distance}} (6)

Refer to caption

(a) Camera Angle Visualization. From left to right: sideview left, agentview left, frontview, agentview right, and sideview right.
Refer to caption
(b) SR across 10 Different Camera Angle Views over ToolHang task for RGB policy and StereoPolicy-DP. StereoPolicy-DP is more robust towards changing camera angles than RGB baseline. Appendix B provides full mapping of camera indices.
Figure 8: Performance of StereoPolicy-DP across different camera angles.
Refer to caption
Figure 9: Effect of stereo baseline and camera–object distance on task success rate.
Method SR (%) Latency (ms)
StereoPolicy-DP 941.994_{1.9} 698
A. Pretrained stereo features
Frozen FS features 764.376_{4.3} 740
B. Stereo-derived geometry
FS depth + RGB-D 444.344_{4.3} 995
FastFS depth + RGB-D 383.338_{3.3} 792
FS depth + 3DDA 282.828_{2.8} 1008
FastFS depth + 3DDA 222.822_{2.8} 777

Table 3: Comparison with pre-trained stereo foundation models on ToolHang (100 demo). FS denotes FoundationStereo. Subscripts are std over three seeds.

( 𝒬 ​ 𝟒 ) Pre-trained Stereo features cannot replace task-adapted stereo fusion.

Replacing task-trained features with frozen FoundationStereo features in a StereoVLA-style control reduces ToolHang success from 94% to 76%. Estimated-depth pipelines also perform worse and take longer here: FoundationStereo+RGB-D reaches 44% at 995 ms, and FoundationStereo+3DDA 28% at 1,008 ms, versus StereoPolicy at 698 ms. Fast variants reduce latency but do not close the task gap. These tests concern the frozen representation and complete pipelines used, not every possible finetuned stereo foundation model.

4.2 Ablations

Across three ablation tasks, full StereoPolicy averages 84%. Duplicating the left image lowers this to 62.7%; removing stereo attention gives 70%, removing FPN 74%, and removing external DINO 76%. Adding DINO to wrist views gives 79.3%. The results support both binocular information and the fusion design, while the duplicated-input attention comparison shows some task-specific architecture benefit even without binocular disparity.

Original figure
Figure 10: Component Ablation. ST denotes StereoTransformer.

5 Related Work

Explicit 3D methods reconstruct depth/point clouds or lift image features using calibration. StereoVLA uses pretrained stereo features and auxiliary geometric supervision. StereoPolicy instead learns within-rig binocular features from action supervision on generic image encoders. This differs from cross-camera or multi-agent scene aggregation and could be combined with those methods; rectified synchronized pairs and suitable camera geometry are still part of its input setup.

6 Conclusion

Direct action-trained binocular fusion is effective across the evaluated diffusion and VLA policies. Remaining limits include poor transparent-object performance, sensitivity to camera geometry, small physical test counts, checkpoint-selected simulation scores, and substantial full-policy denoising latency. Linear probes demonstrate accessible depth information, not that this representation is a complete geometric reconstruction or the sole causal source of success.

Appendix A Task and Data Collection Details

The task and data appendix specifies benchmark selection, physical demonstration counts, and camera/robot configurations.

Simulation Tasks

Robomimic evaluation uses ToolHang, Square, and Transport, excluding already saturated Lift and Can. RoboCasa contributes twenty-four kitchen tasks for VLA evaluation. Four custom OmniGibson tasks—strawberry manipulation, water pouring, door opening, and radio operation—use cuRobo-assisted demonstration collection.

Real Robot Tasks.

Each of five tabletop tasks has two hundred demonstrations: banana-to-container, toast-to-toaster, and hanging plastic, steel, or glass cups. The mobile robot has seventy-five demonstrations each for picking/operating/replacing a radio and transferring toast between hands before shelving it. These datasets and robot-specific tasks bound the reported transfer evidence.

Refer to caption
Figure 11: Example trajectories of real-world tabletop tasks.

A.1 Real-Robot Hardware Setup

The Franka uses an external ZED Mini and a wrist-mounted ZED Mini. The R1 Pro has stereo cameras at both wrists. These physical rigs supply synchronized binocular observations rather than synthetic depth-free monocular inputs.

Appendix B Stereo Hardware Configurations

Camera studies separate training across configurations from applying a fixed trained policy at a changed baseline.

B.1 Stereo Baselines

Default simulated baselines are six centimetres externally and two at the wrist. Physical ZED Mini rigs have a fixed 6.3-centimetre baseline.

B.2 Test-Time Baseline Transfer

A policy trained at a six-centimetre baseline reaches 94% at that configuration, 86% at four centimetres, and 88% at eight, with camera-object distance fixed. This demonstrates moderate baseline tolerance, with best results at the training geometry; it does not establish arbitrary uncalibrated-camera transfer.

Test baseline 4 cm 6 cm 8 cm
Success rate (%) 861.686_{1.6} 941.994_{1.9} 881.688_{1.6}
Table 4: ToolHang SR after training at a 6 cm baseline and changing only the baseline at test time. Subscripts are standard deviations over three seeds.

B.3 Stereo Views across Different Baselines and Distances

The retained image grid shows all tested baselines from 0.02 to 0.10 metres and distances from 0.59 to 0.99 metres. It makes visible the changes in disparity, overlap, and apparent object scale underlying the configuration sweep.

Refer to caption
Figure 12: Stereo camera views across different baselines and distances. Baseline indicates the distance between stereo cameras; distance indicates the camera-object distance.

B.4 Camera Angle Mappings

Ten named views cover frontal, oblique agent, and side positions at ordinary and elevated heights. The original view mapping is retained so the reported success differences can be tied to actual visibility and occlusion.

Refer to caption
Figure 13: Visualization of the camera views. Experimental results are shown in 8(b).

Appendix C Implementation Details

Implementation details specify stereo encoders, diffusion/VLA training, baseline preprocessing, and evaluation selection.

C.1 StereoPolicy-DP Training

Left/right images share ResNet18 and FPN weights. Frozen external DINO features are downsampled by a stride-four convolution, concatenated with CNN features, and projected into tokens. Two stereo layers with eight heads produce a 128-dimensional embedding per view; wrist encoders omit DINO. A GroupNorm UNet is trained from scratch with batch sixty-four and learning rate 10⁻⁴, two observation steps and sixteen predicted actions. Training runs five hundred Robomimic epochs or one thousand OmniGibson epochs.

C.2 StereoPolicy-VLA Training

GR00T-N1.5 finetuning runs sixty thousand steps at 10⁻⁴ learning rate. π0.5 runs eighty thousand steps with global batch 128, one thousand warmup steps, and cosine decay from 2.5×10⁻⁵ to 2.5×10⁻⁶. Both use eight H100 GPUs. At inference, all sixteen predicted actions execute before replanning, which is relevant to responsiveness.

C.3 Baseline Implementations

The matched monocular policy receives left-camera images, the same external DINO features, and capacity-adjusted attention. The matched multi-view policy separately encodes the identical stereo pair and concatenates features without token-level left/right attention. RGB-D repeats depth across three channels for a separate encoder. 3DDA lifts multi-scale RGB features into reconstructed geometry.

PointNet and DP3 fuse calibrated RGB-D point clouds in a common frame, crop the workspace, and farthest-point sample 4,096 points. FoundationStereo-feature controls replace trainable fusion with frozen pretrained features; estimated-depth controls replace the depth source while retaining their downstream RGB-D/3DDA pipelines. These are different interventions and should not be interpreted as one identical depth experiment.

C.4 Evaluation

Simulation evaluates one hundred rollouts every fifty Robomimic epochs or 250 OmniGibson epochs, reporting the highest success achieved over training across three seeds. Physical tasks use twenty trials with randomized starts. Selecting the best evaluated checkpoint can yield more optimistic estimates than a separately chosen final model evaluated once.

Appendix D Additional Ablations

Further ablations examine encoders, attention, foundation-model deployment, geometric probes, point-cloud quality, VLA two-image controls, latency, and wrist-camera removal.

D.1 Vision Encoder

Without DINO, ToolHang with one hundred demonstrations reaches 64% using ResNet alone, 78% CLIP, 80% SigLIP, 82% EdgeNext, and 84% ResNet/FPN. This favors multiscale FPN features in the tested setting, not a universal ranking of pretrained encoders.

Benchmark Robomimic OmniGibson
Task ToolHang-100 Square-100 PourWater-200 Avg. SR
StereoPolicy-DP 941.994_{1.9} 880.988_{0.9} 702.570_{2.5} 84.0
with (L,L) input 621.662_{1.6} 761.676_{1.6} 503.350_{3.3} 62.7
w/o StereoTransformer 822.982_{2.9} 762.876_{2.8} 521.652_{1.6} 70.0
w/o FPN 781.678_{1.6} 801.680_{1.6} 641.664_{1.6} 74.0
w/o DINO 841.684_{1.6} 822.882_{2.8} 622.862_{2.8} 76.0
with DINO on both view 881.688_{1.6} 840.084_{0.0} 663.366_{3.3} 79.3
Table 5: Component Ablation of StereoPolicy-DP. ToolHang and Square use 100 demonstrations; PourWater uses 200. Subscripts are standard deviations over three seeds.
Encoder SR (%)
ResNet18 642.864_{2.8}
OpenCLIP-B/16 782.878_{2.8}
SigLIP-SO400M/14 804.380_{4.3}
EdgeNeXt-S 821.682_{1.6}
ResNet18+FPN 841.684_{1.6}
Table 6: Vision-encoder comparison (w/o DINO) on ToolHang with 100 demonstrations.

D.2 Attention Design

With duplicated left images, joint and stereo attention give similar ToolHang/Square results (64/78 versus 62/76%), while PourWater already favors stereo attention 50% versus 42%. With genuine left/right images, stereo attention reaches 94/88/70%, versus joint attention 82/82/60%. Thus binocular inputs matter, but some PourWater benefit is architectural even without a second viewpoint.

Fusion Views ToolHang-100 Square-100 PourWater-200
Joint attention L,LL,L 641.664_{1.6} 781.678_{1.6} 422.842_{2.8}
Joint attention L,RL,R 821.682_{1.6} 821.682_{1.6} 600.060_{0.0}
Stereo Transformer L,LL,L 621.662_{1.6} 761.676_{1.6} 503.350_{3.3}
Stereo Transformer L,RL,R 941.994_{1.9} 880.988_{0.9} 702.570_{2.5}
Table 7: Matched fusion success rates (%). Task suffixes give demonstration counts. The Stereo Transformer rows correspond to StereoPolicy-DP and its (L,L)(L,L) variant in Table 6. Subscripts are standard deviations over three seeds.

D.3 Pre-trained Stereo Foundation Models on Real Robot

The physical foundation-model comparison measures the entire perception/control pipeline, including its timing.

Real robot with FoundationStereo

FoundationStereo+RGB-D succeeds on banana pick/place in only 1/20 physical trials, versus 16/20 for StereoPolicy. Observations become stale and actions delayed in the depth pipeline. The result combines estimation errors and execution latency; it does not isolate the causal contribution of either.

D.4 Geometric Information

Equal-capacity linear probes predict gripper-to-target camera-frame relative depth from frozen visual features, holding out complete trajectories and layouts. Action-trained stereo features improve errors from 3.1→1.1 cm on ToolHang and 4.0→1.5 on PourWater. Joint-attention stereo errors are 1.9/2.5 cm, and duplicated-input stereo errors 2.7/3.5. These probes show accessible geometric information under the tested probe, without proving depth information alone causes control gains.

ToolHang PourWater
Representation MAE (cm) MAE (cm)
Joint attention, L,LL,L 2.8 3.6
Joint attention, L,RL,R 1.9 2.5
Stereo Transformer, L,LL,L 2.7 3.5
Stereo Transformer, L,RL,R, before training 3.1 4.0
Stereo Transformer, L,RL,R, after training 1.1 1.5
Table 8: Relative-depth mean absolute error from frozen features.

D.5 Discussion on Point Clouds Policy Performance

Point clouds use the same views, calibrated common-frame fusion, workspace cropping, and 4,096-point sampling. Increasing to ten thousand points does not improve the tested result and increases cost. Missing/deformed geometry, especially around reflective/transparent cups and narrow toast slots, is a plausible bottleneck. This finding does not rule out better sensors, reconstruction, or other point-cloud policies.

D.6 StereoPolicy-VLA Performance

The VLA comparison averages all twenty-four RoboCasa tasks equally, with three seeds and one hundred episodes per task/seed. At thirty/one hundred/three hundred demos, π0.5 native two-image scores are 49.01/68.24/70.81%, versus StereoPolicy 51.72/71.54/74.40. GR00T native scores are 45.67/65.24/65.31%, versus stereo 47.12/66.17/67.50. Fusion therefore adds modest gains beyond merely supplying the second image.

# Demos
Backbone Input 30 100 300
π0.5\pi_{0.5} Monocular 48.710.748.71_{0.7} 67.641.067.64_{1.0} 70.310.670.31_{0.6}
π0.5\pi_{0.5} Two-image 49.010.149.01_{0.1} 68.240.768.24_{0.7} 70.811.870.81_{1.8}
π0.5\pi_{0.5} StereoPolicy-VLA 51.720.5\mathbf{51.72}_{0.5} 71.541.0\mathbf{71.54}_{1.0} 74.400.5\mathbf{74.40}_{0.5}
GR00T-N1.5 Monocular 44.980.444.98_{0.4} 63.580.663.58_{0.6} 64.301.164.30_{1.1}
GR00T-N1.5 Two-image 45.671.645.67_{1.6} 65.240.765.24_{0.7} 65.310.765.31_{0.7}
GR00T-N1.5 StereoPolicy-VLA 47.120.9\mathbf{47.12}_{0.9} 66.171.2\mathbf{66.17}_{1.2} 67.501.0\mathbf{67.50}_{1.0}
Table 9: Matched VLA macro-average success (%) on 24 RoboCasa tasks. Two-image: the backbone’s native multi-view input with the left and right images. Subscripts are standard deviations over three seeds.
Latency (ms)
Method p50 / p95
RGB-DP 632 / 674
Joint attention 695 / 731
StereoPolicy 698 / 736
FS + 3DDA 1008 / 1060
FastFS + 3DDA 777 / 819
Table 10: End-to-end inference latency (median / 95th percentile) on one H100 GPU.

D.7 Overhead, computational cost

StereoPolicy has 112 million parameters, including ten million in fusion, with reported 128 GFLOPs and 1.4G peak memory. H100 profiling uses batch one, two observations, sixteen actions, seven action dimensions, and one hundred denoising steps, after one hundred warmups and across one thousand synchronized calls. Median/95th-percentile latency is 698/736 ms, versus 695/731 for joint attention and 632/674 for RGB. The quoted three-millisecond overhead is relative to joint attention; the full pipeline adds sixty-six milliseconds over RGB and is not a three-millisecond controller.

D.8 Removing Wrist Observations

On ToolHang with two hundred demonstrations, removing wrist views changes stereo success 96→94%, RGB 90→86%, RGB-D 88→84%, and DP3 76→62%. External stereo is useful in this one-task test; this is not evidence that wrist cameras can always be removed without cost.

Setting RGB RGB-D PCD RGB-D Multi-View StereoPolicy-DP
PointNet PointNet++ DP3 3DDA
Default (w/ wrist camera) 901.690_{1.6} 882.588_{2.5} 322.832_{2.8} 284.928_{4.9} 760.976_{0.9} 922.892_{2.8} 921.992_{1.9} 𝟗𝟔2.8\mathbf{96}_{2.8}
w/o wrist camera 861.686_{1.6} 841.684_{1.6} 304.930_{4.9} 283.328_{3.3} 622.862_{2.8} 881.688_{1.6} 881.688_{1.6} 𝟗𝟒3.3\mathbf{94}_{3.3}
Table 11: Effect of removing wrist-camera observations on ToolHang with 200 demos. Success rate (%); subscripts are standard deviations over three seeds.
Refer to caption
Figure 14: Failure cases of baseline visual modalities on real-world tabletop tasks. RGB, RGB-D, and point-cloud inputs can be inaccurate for fine-grained manipulation: gripper-target alignment can drift, reflective objects can obscure object boundaries, and transparent objects often produce missing or noisy depth and point-cloud geometry.
Refer to caption
Figure 15: Monocular RGB failures in bimanual mobile manipulation. In the Turn on Radio task, RGB policies often fail at precise handle insertion and button pressing due to insufficient depth cues. In the Toast PnP task, failures commonly occur during grasp alignment and placement due to ambiguous monocular depth cues.

Appendix E Qualitative Failure Cases

Qualitative examples connect failures to geometry and highlight remaining stereo limitations.

Baseline modality failures in tabletop manipulation.

Weak cup boundaries lead to localization/grasp errors, noisy depth misaligns handles with pegs, and transparent glass can disappear from depth/point clouds. Toast failures include vertical or lateral slot misalignment. These examples illustrate plausible failure mechanisms rather than separately controlled causal measurements.

Remaining difficulty on transparent objects.

Glass success remains only 3/20. Refraction, thin rims/handles, and weak boundaries also make RGB correspondence and localization ambiguous. Avoiding depth reconstruction therefore does not solve transparent-object manipulation.

Monocular RGB failures in mobile manipulation.

Monocular mobile policies can approach a radio yet miss its handle or button, and can locate toast yet fail grasp or placement alignment. Stereo improves these tested alignment behaviors, with the original trajectories retained to show both scene context and the specific failure stage.

Concept demonstration

Open image ↗