Literature
Display
← All papers

Augmented Reality for RObots (ARRO): Pointing Visuomotor Policies Towards Visual Robustness

Reihaneh Mirjalili · Tobias Jülg · Florian Walter · Wolfram Burgard

· IEEE Robotics and Automation Letters ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

ARRO isolates task objects and gripper fingers with pretrained detection, segmentation and vision-language models, then composites their original pixels onto a fixed virtual background. Applying the same representation to training demonstrations and deployment images reduces background and robot-appearance shifts. Tests cover four real manipulation tasks, three generalist policies and simulated transfer. Improvements are substantial in several settings, but the grid is not uniformly better than black masking; real-to-sim Diffusion Policy still fails, and segmentation errors and lighting changes remain limitations.

I INTRODUCTION

Visuomotor policies can overfit scene appearance even when their actions depend on only a few objects. ARRO removes background and robot-body pixels instead of trying to collect demonstrations for every visual condition. Its grid supplies a consistent visual reference around retained object and gripper pixels.

The preprocessing models operate without task-specific training or camera calibration. This does not mean a raw-image policy is proven to accept ARRO inputs unchanged: the experiments train or fine-tune policy variants on consistently transformed demonstrations. The claim of avoiding retraining concerns reuse under subsequent scene changes, rather than eliminating initial policy training.

Original PDF figure S1.F1
Fig. 1: Visualization of input formats (top) and performance comparison across four manipulation tasks (bottom). (a) shows the training scene, while (b) depicts an altered scene with visual domain shifts used during inference. (c) and (d) illustrate the corresponding inputs for the masked diffusion policy and ARRO, respectively.

II Related Works

Related approaches use domain randomization, diverse robot datasets, human video, wrist cameras, object-centric features and keypoints. ARRO performs object selection directly in image space and can use third-person demonstrations. Unlike background synthesis, it removes visual variation; unlike Mirage Chen et al. (2024) and SHADOW Lepert et al. (2025), it avoids calibrated robot rendering or geometric alignment. It still requires valid task-object selection and reliable segmentation. The contribution is this preprocessing arrangement, rather than a new policy architecture or a learned universal scene representation.

III Approach

The policy maps an image history to a horizon of task-space actions. ARRO transforms each image while retaining the selected objects’ original locations, colors and appearance. Training and inference use the same transform. It does not reconstruct hidden geometry or translate an incompatible action space between robots.

Original PDF figure S3.F2
Fig. 2: ARRO in a nutshell: Our pipeline segments the robot gripper and task-related using open-vocabulary vision models and overlays them onto a virtual background. We consistently use this process across training and inference to enhance robustness against visual domain shifts.

III-A Open-Vocabulary Segmentation

Initialization has two paths. Grounding DINO Liu et al. (2025) detects named objects and produces bounding boxes. Because robot fingers are poorly represented in ordinary detector training, unprompted segmentation first proposes regions; numeric labels are overlaid on the image, and a vision-language model selects task-relevant regions and gripper-finger keypoints. These boxes and points initialize SAM 2 Ravi et al. (2025).

SAM 2 propagates object and gripper masks through subsequent frames using temporal memory, avoiding a vision-language query on every frame. Algorithm 1 preserves the full initialization sequence. The method depends on correct initial selection and continued tracking; “zero-shot” applies to use of these pretrained vision components, not absence of pretrained data or computational cost.

Equations 1–4
Bi=Detect​(I0,pio)B_{i}=\texttt{Detect}(I_{0},p^{o}_{i}) (1)
{K0,…,Kl}=Segment​(I0)\{K_{0},\ldots,K_{l}\}=\texttt{Segment}(I_{0}) (2)
{K0∗,…,Km∗}=VLM​(I0∗,pt)\{K^{*}_{0},\ldots,K^{*}_{m}\}=\texttt{VLM}(I_{0}^{*},p^{t}) (3)
S0obj,S0gripper=Segment​(I0∣ℬ,𝒦∗)S_{0}^{\mathrm{obj}},\;S_{0}^{\mathrm{gripper}}=\texttt{Segment}(I_{0}\mid\mathcal{B},\mathcal{K}^{*}) (4)

III-B Virtual Scene Recomposition

At each timestep, the union of object and gripper masks selects original RGB pixels; the complementary pixels come from a background image. Equation 5 and Algorithm 2 retain this exact compositing rule. The comparison uses either plain black or a fixed handcrafted colored grid, reused across sequences. The grid offers a consistent image-space reference, not a calibrated estimate of metric depth. The paper calls inference real time but gives no measured frame rate, latency breakdown or initialization-cost benchmark.

Equation 5
I~t=St⊙It+(1−St)⊙IB,\tilde{I}_{t}=S_{t}\odot I_{t}+(1-S_{t})\odot I_{B}, (5)
Algorithm 1 ARRO Initialization

Input: An RGB frame II; object prompts p1o,…,pnop^{o}_{1},\dots,p^{o}_{n}; task prompt ptp^{t}; open-vocabulary object detector Detect (e.g., GroundingDINO); uninitialized segmentation model Segment (e.g., SAM2); and a vision-language model VLM (e.g., GPT-4o).

Output: The initialized segmentation model Segment0\texttt{Segment}_{0}

// Get object bounding boxes for complex objects:
for piop^{o}_{i} in {p1o,…,pno}\{p^{o}_{1},\ldots,p^{o}_{n}\} do
  Bi←Detect​(I,pio)B_{i}\leftarrow\texttt{Detect}(I,p^{o}_{i})
end for
// Run unprompted segmentation on II to get region masks to
// segment simple objects and gripper fingers:
{K0,…,Kl}←Segment​(I)\{K_{0},\ldots,K_{l}\}\leftarrow\texttt{Segment}(I)
// Annotate keypoints in II with numeric labels to retrieve I∗I^{*} and
// identify task-relevant keypoints in I∗I^{*} from {K0,…,Kl}\{K_{0},\ldots,K_{l}\}:
{K0∗,…,Km∗}←VLM​(I∗,pt)\{K^{*}_{0},\ldots,K^{*}_{m}\}\leftarrow\texttt{VLM}(I^{*},p^{t})
// Initialize and return the segmentation model:
return Initialize​(Segment,I,B0,…,Bn,K0∗,…,Km∗)\texttt{Initialize}(\texttt{Segment},I,B_{0},\ldots,B_{n},K^{*}_{0},\ldots,K^{*}_{m})
Algorithm 2 ARRO Masking

Input: RGB frame ItI_{t}; RGB background image IBI_{B}; initialized segmentation model Segmentt\texttt{Segment}_{t} (e.g., SAM2).

Output: A masked RGB frame I~t\tilde{I}_{t} and the updated segmentation model Segmentt+1\texttt{Segment}_{t+1}

// Track object and gripper masks:
Stobj,Stgripper,Segmentt+1​(I)←Segmentt​(I)S^{\text{obj}}_{t},\;S^{\text{gripper}}_{t},\;\texttt{Segment}_{t+1}(I)\leftarrow\texttt{Segment}_{t}(I)
// Combine masks:
St←Stobj∪StgripperS_{t}\leftarrow S^{\text{obj}}_{t}\cup S^{\text{gripper}}_{t}
// Overlay SS on virtual background IBI_{B} retrieve I~\tilde{I}:
I~t←St⊙It+(1−St)⊙IB\tilde{I}_{t}\leftarrow S_{t}\odot I_{t}+(1-S_{t})\odot I_{B}
return I~t,Segmentt+1\tilde{I}_{t},\;\texttt{Segment}_{t+1}

IV Experimental Results

Experiments use an FR3 arm with custom printed fingers, teleoperation or scripted demonstrations, robot states and a third-person camera. Diffusion Policy Chi et al. (2023) is trained per task; Octo Ghosh et al. (2024), OpenVLA Kim et al. (2025) and π₀ Black et al. (2024) are fine-tuned with their released code. π₀ additionally receives a wrist camera. These differing inputs, action representations and pretraining histories matter when comparing architectures.

IV-A Real-World Experiments for Diffusion Policy

Four real tasks cover blue-cuboid pickup, pushing a cube to a red cross, placing an octopus plush toy beside a doll, and dropping a cube into a box and closing its lid. Each task has 90 demonstrations. The varied objects test texture, deformability and changing geometry, but remain tabletop manipulation with known task-relevant objects.

IV-A 1 Performance Across Domain Shifts

Vanilla, black-masked and ARRO policies are each trained for 1,000 epochs and evaluated in ten altered-scene trials per task. Figure 1 reports vanilla/masked/ARRO success of 0/50/100% for pickup, 0/60/90% for pushing, 0/90/100% for the doll task and 30/90/100% for the box task. Background, table texture, robot appearance and distractors change together; the experiment does not isolate each shift’s causal contribution.

Masking explains much of the gain, and the grid improves these four point estimates further. With ten trials, a ten-percentage-point difference is one episode; seed variation and confidence intervals are not reported. Figure 5 preserves the full box and doll sequences, including temporary occlusion and shape changes.

Original PDF figure S4.F3
Fig. 3: Real-world experiment setups for the (a) pick-v1, (b) push-v1, (c) doll-v1 and (d) box-v1 tasks.
Original PDF figure S4.F4
Fig. 4: Handling distractor objects. Top: example input image with distractors (a) and the ARRO-segmented version (b) where the task is to “Push the blue cube that is farther from the yellow cube to the red cross”. Bottom: Success rates for the spatial reasoning tasks.
Original PDF figure S4.F5
Fig. 5: Execution sequences using ARRO on the box-v1 and doll-v1 tasks. By overlaying the segmented task-relevant regions on a virtual background, ARRO neutralizes the effect of visual domain shifts. ARRO’s segmentation remains robust to transient occlusions caused by the robot arm or other objects. The doll is temporarily occluded by the octopus plush toy and the gripper in frame (3), but its segmentation accurately reappears once the occlusion clears, without manual correction.

IV-A 2 Handling Distractor Objects

A vision-language model chooses which blue cube to push from “left/right” or “closer to/farther from the yellow cube.” ARRO then hides distractors, making the selected-object image resemble the original pushing task without collecting new demonstrations. Success is 100% versus 30% for left/right and 90% versus 20% for closer/farther against vanilla. This is task selection by the preprocessing system; it does not establish that the underlying policy acquired general spatial-language reasoning or obstacle avoidance for hidden distractors.

IV-A 3 Handling Occlusions in ARRO

The shown doll sequence recovers its segmentation after occlusion by the arm or plush toy without manual correction. It demonstrates recovery in the pictured execution, rather than a quantified bound on occlusion duration or a comprehensive robustness test. Later limitations explicitly report occasional tracking failures.

IV-B Real-World Experiments for Generalist Policies

Octo and OpenVLA use the blue-cuboid pickup task; π₀ uses a green-cuboid task with side and wrist views because a side camera alone performed poorly. Wrist views use black backgrounds even in the ARRO condition; only the side camera receives the grid. Tracking initializes from a prerecorded image containing all relevant objects, and fingers are selected directly from their fixed initial wrist-view position. These practical adaptations qualify the broader claim of deployment without setup-specific adjustments.

IV-B 1 Performance Across Domain Shifts

Under altered visuals, Figure 6 gives vanilla/masked/ARRO success of 0/30/40% for Octo, 0/40/40% for OpenVLA and 0/60/70% for π₀. Thus all masked inputs help here, but ARRO ties black masking for OpenVLA. π₀ receives extra clutter because it was initially more robust; this is not a controlled common-shift ranking of the three architectures.

IV-B 2 Language Guidance

When a same-shaped red cuboid distracts π₀ from the instructed green cuboid, success is 30% for vanilla, 70% with black masking and 90% for ARRO. Filtering the unwanted cube supplies an effective language-guided target constraint. It does not demonstrate that the policy itself learns to obey color language more reliably.

IV-C ARRO in Simulation

Simulation separates transfer from real FR3 demonstrations into a MuJoCo FR3 replica Todorov et al. (2012) from transfer between simulated FR3 and UR5e arms. The latter keeps the environment and Cartesian action convention fixed. Table I reports performance relative to each condition’s source-domain success, rather than absolute target-domain success.

IV-C 1 Real-to-Sim

Real-to-sim uses ten source-domain real episodes and 100 target-domain simulation episodes. Vanilla real success is 90% for Diffusion Policy, 50% for Octo, 40% for OpenVLA and 100% for π₀. Relative transfer for vanilla/masked/ARRO is 0/0/0% for Diffusion Policy, 0/3/5% for Octo, 0/53/55% for OpenVLA and 79/94/77% for π₀. These ratios should not be read as target success percentages or multiplied by a different variant’s baseline.

ARRO therefore does not solve real-to-sim transfer for Diffusion Policy and provides little transfer for Octo. π₀’s ARRO ratio is lower than both alternatives; the authors attribute the difference partly to its higher ARRO real-domain denominator (90%) and state that masked and ARRO target success are comparable, without tabulating all absolute denominators. The normalized per-step distance rewards in Figure 9 show partial behavioral progress, not successful task completion. Shading denotes episode standard deviation, not uncertainty across independently trained policies.

Original PDF figure S4.F6
Fig. 6: Performance comparison across three generalist models, Octo and OpenVLA and π0\pi_{0}, evaluated under altered visual conditions.
Original PDF figure S4.F7
Fig. 7: Wrist camera view of ARRO for the π0\pi_{0} experiment “pick up the green cube”. Top: Example input image of a cluttered scene and the masked version. Bottom: Scene with a red cube that is masked by ARRO.
Original PDF figure S4.F8
Fig. 8: Experimental setup for the pick task in two environments and one cross-embodiment scenario. Left to right: The real-world FR3 setup, the model of the setup in MuJoCo used for collecting sim-pick-v1, and the UR5e setup for cross-embodiment experiments in MuJoCo.
TABLE I: Relative policy performance of real-to-sim and cross-embodiment experiments after domain transfer.
Real-to-Sim Cross-Embodiment
Policy Camera Control Dataset Vanilla Masked ARRO Dataset Vanilla Masked ARRO
Diffusion Policy Side Cartesian pick-v2 0% 0% 0% sim-pick-v1 0% 80% 99%
Octo Side Cartesian pick-v2 0% 3% 5% sim-pick-v1 71% 87% 87%
OpenVLA Side Cartesian pick-v2 0% 53% 55% sim-pick-v1 29% 72% 80%
π0\pi_{0} Side + Wrist Joints pick-wrist-v1 79% 94% 77% — — — —
Original PDF figure S4.F9
Fig. 9: Normalized mean per step reward over 100 episodes on the sim-pick-v1 task in simulation on the FR3 (same embodiment). All models were trained only on real-world datasets. Shaded areas indicate standard deviations.

IV-C 2 Cross-Embodiment

Cross-embodiment tests use 100 source and 100 target simulation episodes. Relative vanilla/masked/ARRO performance is 0/80/99% for Diffusion Policy, 71/87/87% for Octo and 29/72/80% for OpenVLA. Both masks hide most robot-body differences; absolute Cartesian commands remain compatible. Octo and OpenVLA also encountered UR5e in pretraining, so their result is not transfer to a wholly unseen embodiment.

π₀ is excluded because its joint-space action output is incompatible with the UR5e kinematics. Visual preprocessing alone cannot resolve that mismatch. The controlled simulated transfer does not establish arbitrary cross-robot or real-world transfer without a compatible controller.

V Limitations

Errors in initial segmentation or temporal tracking directly affect policy input, especially when wrist-view objects enter or leave the image. Periodic reinitialization or a detector-triggered reset is proposed, not evaluated as part of the results. Minor mask errors may be tolerated, but no systematic noise or failure-rate curve is reported. Retained object pixels still carry reflections and lighting changes, which ARRO cannot remove; depth or inpainting are future directions.

The evaluation is limited by small real-world samples, a narrow set of tasks and embodiments, unreported training-seed uncertainty, and absent timing measurements. Hiding all nonselected pixels can also remove context that another task needs. Claims of compatibility with any visuomotor policy should be read as an interface possibility, rather than validated universal performance.

VI Conclusion

ARRO is a practical way to canonicalize policy inputs when the task can be expressed through selected object and gripper pixels. Its strongest evidence is resistance to background clutter and appearance changes under consistent training/inference preprocessing. Benefits depend on tracking, input configuration and action compatibility; black masking is competitive in several settings. Dynamic view-dependent backgrounds for wrist cameras remain an untested extension.

ACKNOWLEDGMENT

The authors acknowledge Erlangen National HPC Center resources under the BayernKI project.

Concept demonstration

Open image ↗