At a glance
ARRO isolates task objects and gripper fingers with pretrained detection, segmentation and vision-language models, then composites their original pixels onto a fixed virtual background. Applying the same representation to training demonstrations and deployment images reduces background and robot-appearance shifts. Tests cover four real manipulation tasks, three generalist policies and simulated transfer. Improvements are substantial in several settings, but the grid is not uniformly better than black masking; real-to-sim Diffusion Policy still fails, and segmentation errors and lighting changes remain limitations.
I INTRODUCTION
Visuomotor policies can overfit scene appearance even when their actions depend on only a few objects. ARRO removes background and robot-body pixels instead of trying to collect demonstrations for every visual condition. Its grid supplies a consistent visual reference around retained object and gripper pixels.
The preprocessing models operate without task-specific training or camera calibration. This does not mean a raw-image policy is proven to accept ARRO inputs unchanged: the experiments train or fine-tune policy variants on consistently transformed demonstrations. The claim of avoiding retraining concerns reuse under subsequent scene changes, rather than eliminating initial policy training.
II Related Works
Related approaches use domain randomization, diverse robot datasets, human video, wrist cameras, object-centric features and keypoints. ARRO performs object selection directly in image space and can use third-person demonstrations. Unlike background synthesis, it removes visual variation; unlike Mirage Chen et al. (2024) and SHADOW Lepert et al. (2025), it avoids calibrated robot rendering or geometric alignment. It still requires valid task-object selection and reliable segmentation. The contribution is this preprocessing arrangement, rather than a new policy architecture or a learned universal scene representation.
III Approach
The policy maps an image history to a horizon of task-space actions. ARRO transforms each image while retaining the selected objects’ original locations, colors and appearance. Training and inference use the same transform. It does not reconstruct hidden geometry or translate an incompatible action space between robots.
III-A Open-Vocabulary Segmentation
Initialization has two paths. Grounding DINO Liu et al. (2025) detects named objects and produces bounding boxes. Because robot fingers are poorly represented in ordinary detector training, unprompted segmentation first proposes regions; numeric labels are overlaid on the image, and a vision-language model selects task-relevant regions and gripper-finger keypoints. These boxes and points initialize SAM 2 Ravi et al. (2025).
SAM 2 propagates object and gripper masks through subsequent frames using temporal memory, avoiding a vision-language query on every frame. Algorithm 1 preserves the full initialization sequence. The method depends on correct initial selection and continued tracking; “zero-shot” applies to use of these pretrained vision components, not absence of pretrained data or computational cost.
Equations 1–4
| (1) |
| (2) |
| (3) |
| (4) |
III-B Virtual Scene Recomposition
At each timestep, the union of object and gripper masks selects original RGB pixels; the complementary pixels come from a background image. Equation 5 and Algorithm 2 retain this exact compositing rule. The comparison uses either plain black or a fixed handcrafted colored grid, reused across sequences. The grid offers a consistent image-space reference, not a calibrated estimate of metric depth. The paper calls inference real time but gives no measured frame rate, latency breakdown or initialization-cost benchmark.
Equation 5
| (5) |
Input: An RGB frame ; object prompts ; task prompt ; open-vocabulary object detector Detect (e.g., GroundingDINO); uninitialized segmentation model Segment (e.g., SAM2); and a vision-language model VLM (e.g., GPT-4o).
Output: The initialized segmentation model
// segment simple objects and gripper fingers:
// identify task-relevant keypoints in from :
IV Experimental Results
Experiments use an FR3 arm with custom printed fingers, teleoperation or scripted demonstrations, robot states and a third-person camera. Diffusion Policy Chi et al. (2023) is trained per task; Octo Ghosh et al. (2024), OpenVLA Kim et al. (2025) and π₀ Black et al. (2024) are fine-tuned with their released code. π₀ additionally receives a wrist camera. These differing inputs, action representations and pretraining histories matter when comparing architectures.
IV-A Real-World Experiments for Diffusion Policy
Four real tasks cover blue-cuboid pickup, pushing a cube to a red cross, placing an octopus plush toy beside a doll, and dropping a cube into a box and closing its lid. Each task has 90 demonstrations. The varied objects test texture, deformability and changing geometry, but remain tabletop manipulation with known task-relevant objects.
IV-A 1 Performance Across Domain Shifts
Vanilla, black-masked and ARRO policies are each trained for 1,000 epochs and evaluated in ten altered-scene trials per task. Figure 1 reports vanilla/masked/ARRO success of 0/50/100% for pickup, 0/60/90% for pushing, 0/90/100% for the doll task and 30/90/100% for the box task. Background, table texture, robot appearance and distractors change together; the experiment does not isolate each shift’s causal contribution.
Masking explains much of the gain, and the grid improves these four point estimates further. With ten trials, a ten-percentage-point difference is one episode; seed variation and confidence intervals are not reported. Figure 5 preserves the full box and doll sequences, including temporary occlusion and shape changes.
IV-A 2 Handling Distractor Objects
A vision-language model chooses which blue cube to push from “left/right” or “closer to/farther from the yellow cube.” ARRO then hides distractors, making the selected-object image resemble the original pushing task without collecting new demonstrations. Success is 100% versus 30% for left/right and 90% versus 20% for closer/farther against vanilla. This is task selection by the preprocessing system; it does not establish that the underlying policy acquired general spatial-language reasoning or obstacle avoidance for hidden distractors.
IV-A 3 Handling Occlusions in ARRO
The shown doll sequence recovers its segmentation after occlusion by the arm or plush toy without manual correction. It demonstrates recovery in the pictured execution, rather than a quantified bound on occlusion duration or a comprehensive robustness test. Later limitations explicitly report occasional tracking failures.
IV-B Real-World Experiments for Generalist Policies
Octo and OpenVLA use the blue-cuboid pickup task; π₀ uses a green-cuboid task with side and wrist views because a side camera alone performed poorly. Wrist views use black backgrounds even in the ARRO condition; only the side camera receives the grid. Tracking initializes from a prerecorded image containing all relevant objects, and fingers are selected directly from their fixed initial wrist-view position. These practical adaptations qualify the broader claim of deployment without setup-specific adjustments.
IV-B 1 Performance Across Domain Shifts
Under altered visuals, Figure 6 gives vanilla/masked/ARRO success of 0/30/40% for Octo, 0/40/40% for OpenVLA and 0/60/70% for π₀. Thus all masked inputs help here, but ARRO ties black masking for OpenVLA. π₀ receives extra clutter because it was initially more robust; this is not a controlled common-shift ranking of the three architectures.
IV-B 2 Language Guidance
When a same-shaped red cuboid distracts π₀ from the instructed green cuboid, success is 30% for vanilla, 70% with black masking and 90% for ARRO. Filtering the unwanted cube supplies an effective language-guided target constraint. It does not demonstrate that the policy itself learns to obey color language more reliably.
IV-C ARRO in Simulation
Simulation separates transfer from real FR3 demonstrations into a MuJoCo FR3 replica Todorov et al. (2012) from transfer between simulated FR3 and UR5e arms. The latter keeps the environment and Cartesian action convention fixed. Table I reports performance relative to each condition’s source-domain success, rather than absolute target-domain success.
IV-C 1 Real-to-Sim
Real-to-sim uses ten source-domain real episodes and 100 target-domain simulation episodes. Vanilla real success is 90% for Diffusion Policy, 50% for Octo, 40% for OpenVLA and 100% for π₀. Relative transfer for vanilla/masked/ARRO is 0/0/0% for Diffusion Policy, 0/3/5% for Octo, 0/53/55% for OpenVLA and 79/94/77% for π₀. These ratios should not be read as target success percentages or multiplied by a different variant’s baseline.
ARRO therefore does not solve real-to-sim transfer for Diffusion Policy and provides little transfer for Octo. π₀’s ARRO ratio is lower than both alternatives; the authors attribute the difference partly to its higher ARRO real-domain denominator (90%) and state that masked and ARRO target success are comparable, without tabulating all absolute denominators. The normalized per-step distance rewards in Figure 9 show partial behavioral progress, not successful task completion. Shading denotes episode standard deviation, not uncertainty across independently trained policies.
| Real-to-Sim | Cross-Embodiment | |||||||||
| Policy | Camera | Control | Dataset | Vanilla | Masked | ARRO | Dataset | Vanilla | Masked | ARRO |
| Diffusion Policy | Side | Cartesian | pick-v2 | 0% | 0% | 0% | sim-pick-v1 | 0% | 80% | 99% |
| Octo | Side | Cartesian | pick-v2 | 0% | 3% | 5% | sim-pick-v1 | 71% | 87% | 87% |
| OpenVLA | Side | Cartesian | pick-v2 | 0% | 53% | 55% | sim-pick-v1 | 29% | 72% | 80% |
| Side + Wrist | Joints | pick-wrist-v1 | 79% | 94% | 77% | — | — | — | — | |
IV-C 2 Cross-Embodiment
Cross-embodiment tests use 100 source and 100 target simulation episodes. Relative vanilla/masked/ARRO performance is 0/80/99% for Diffusion Policy, 71/87/87% for Octo and 29/72/80% for OpenVLA. Both masks hide most robot-body differences; absolute Cartesian commands remain compatible. Octo and OpenVLA also encountered UR5e in pretraining, so their result is not transfer to a wholly unseen embodiment.
π₀ is excluded because its joint-space action output is incompatible with the UR5e kinematics. Visual preprocessing alone cannot resolve that mismatch. The controlled simulated transfer does not establish arbitrary cross-robot or real-world transfer without a compatible controller.
V Limitations
Errors in initial segmentation or temporal tracking directly affect policy input, especially when wrist-view objects enter or leave the image. Periodic reinitialization or a detector-triggered reset is proposed, not evaluated as part of the results. Minor mask errors may be tolerated, but no systematic noise or failure-rate curve is reported. Retained object pixels still carry reflections and lighting changes, which ARRO cannot remove; depth or inpainting are future directions.
The evaluation is limited by small real-world samples, a narrow set of tasks and embodiments, unreported training-seed uncertainty, and absent timing measurements. Hiding all nonselected pixels can also remove context that another task needs. Claims of compatibility with any visuomotor policy should be read as an interface possibility, rather than validated universal performance.
VI Conclusion
ARRO is a practical way to canonicalize policy inputs when the task can be expressed through selected object and gripper pixels. Its strongest evidence is resistance to background clutter and appearance changes under consistent training/inference preprocessing. Benefits depend on tracking, input configuration and action compatibility; black masking is competitive in several settings. Dynamic view-dependent backgrounds for wrist cameras remain an untested extension.
ACKNOWLEDGMENT
The authors acknowledge Erlangen National HPC Center resources under the BayernKI project.