Overview
The overview figure retains the segmentation dataset, learned robot mask, augmentation pipeline, and transfer from one collection scene to new evaluation backgrounds.
At a glance
RoboEngine changes the backgrounds of robot demonstrations while preserving the robot, task objects, actions, and other labels. It combines a robot segmentation model trained on RoboSeg with a foreground-conditioned diffusion background generator. Diffusion Policies trained on demonstrations from one scene improve on towel folding across four new scenes and mouse placement across two others. The reported average normalized behavior score rises from 0.20 without generative augmentation to 0.62, while average stage success rises from 15.6% to 60.9%. These are two-task visual-transfer results, with no explicit temporal or multiview consistency guarantee.
I INTRODUCTION
Visual imitation policies can fail when backgrounds, illumination, or table appearance change. Collecting demonstrations in many environments is expensive, while green-screen and calibrated augmentation pipelines impose setup work. RoboEngine targets that setup bottleneck: a learned mask preserves relevant foreground pixels, and a diffusion model supplies varied backgrounds. RoboSeg provides 3,800 annotated images spanning many robot datasets; Robo-SAM fine-tunes a language-conditioned segmenter on these examples.
The “plug-and-play” claim concerns avoiding camera calibration and controlled collection backgrounds. The method still depends on pretrained models, adequate segmentation, task/object descriptions, and compute. Its “physics-aware” backgrounds are learned visual generations conditioned on the foreground, rather than outputs verified by a simulator or physical constraint solver. The real-robot experiments test whether this appearance augmentation helps the downstream policy.
II RELATED WORKS
The approach contributes training-image diversity rather than a new control architecture. Its policies use a modified Diffusion Policy with the same settings across augmentation methods.
II-A Visual Generalizable Imitation Learning
Imitation learning maps visual observations to actions but is sensitive to visual shifts. Alternative remedies include architecture changes, representation pretraining, and broader datasets. RoboEngine focuses on augmentation while adopting Diffusion Policy [3] and a DINOv2 visual encoder for control.
II-B Robot Data Augmentation
Conventional color/crop changes mainly cover modest appearance variation. CACTI [9] and related inpainting approaches alter selected objects with calibration or mask requirements, while GreenAug [5] uses a green-screen foreground. Random images/textures offer large changes but may look implausible. RoboEngine aims to combine foreground preservation, background diversity, and reduced setup; its comparisons use adapted implementations sharing the proposed robot segmenter.
III METHOD
Each demonstration image is separated into robot pixels, task-object pixels, and background. A new background is generated around the retained foreground, and all associated proprioception, instruction, and action data remain attached to the image.
III-A Task Definition
Replacing only the background is intended to preserve the action label: the robot and task objects stay in the same pixels while visual context changes. This assumes the generated scene does not introduce a conflicting task interpretation or relevant physical interaction. It changes appearance and layout cues outside the masks, not the demonstrated robot motion, object geometry, or physical scene state.
III-B RoboSeg Dataset and Robot Segmentation Model
RoboSeg samples 3,800 images from more than 35 robot datasets, covering multiple embodiments, viewpoints, and backgrounds. Masks distinguish the gripper-connected robot component, other visible robot parts, and task-related objects. Fine annotations include thin wires. Each image also has a task instruction and ten GPT-4o-generated scene descriptions, forming a prompt pool.
Robo-SAM fine-tunes the semantic “multitask” version of EVF-SAM [14], a language-conditioned segmenter. The authors report that this version works better than instance segmentation, but give no controlled quantitative ablation for that choice. Robot-mask transfer is evaluated on small held-out and internet-image sets; “open-world” in the prose should not be read as coverage of every robot, camera, or occlusion condition.
III-C Plug-and-Play RoboEngine Augmentation
Robo-SAM identifies robot pixels; EVF-SAM, prompted with task object names, identifies relevant objects. A fine-tuned BackGround-Diffusion model [15] receives these masks and a randomly selected scene description to generate the surrounding background. The toolkit exposes alternative augmentation methods through a configuration flag. The complete original Python example is retained below, including its inconsistency: it creates engine but later calls aug_engine. That variable name must be reconciled before using the listing as runnable code.
The displayed video interface does not imply temporal coherence: the paper explicitly identifies cross-frame consistency as an unsolved limitation. The experimental policy uses an observation horizon of one, which reduces dependence on consistent visual history.
from robo_engine import RoboEngine
# aug_method: engine, texture, imagenet......
aug_method = 'robo_engine'
engine = RoboEngine(
robo_seg_method=['robo_sam_video'],
obj_seg_method=['evf_sam_video'],
aug_method=aug_method,
batch_size=32
)
aug_video = aug_engine.gen_video(obs_video)IV EXPERIMENTS
Evaluation first measures robot segmentation, then compares augmentation methods using physical robot rollouts. Policy experiments cover two tasks and six distinct task-scene combinations, not both tasks in all six environments.
IV-A Segmentation Results
The segmentation test set contains 97 novel-scene images from the same source datasets; a zero-shot set contains 45 robot images downloaded from the internet. All compared language-conditioned models receive the prompt “robot.” Robo-SAM scores 0.8620/0.9037 on the reported GIoU metric, versus EVF-SAM’s 0.6290/0.7777. Figure 3 preserves all qualitative masks from Robo-SAM, EVF-SAM, CLIPSeg, and LISA.
The paper calls this Generalized IoU [37] but does not specify its mask-based implementation in detail. No confidence intervals or separate thin-wire, task-object, or failure-category metrics are given. The illustrative claim that only Robo-SAM produces usable masks is not a measured universal usability threshold.
| Method | Test Set | Zero-shot Set |
|---|---|---|
| CLIPSeg[36] | 0.2810 | 0.4049 |
| LISA[31] | 0.6040 | 0.7571 |
| EVF-SAM[14] | 0.6290 | 0.7777 |
| Robo-SAM (Ours) | 0.8620 | 0.9037 |
| Fold Towel (Good Grasp) | Fold Towel (Finish) | Put Mouse (Good Grasp) | Put Mouse (Finish) | Average | |
|---|---|---|---|---|---|
| No aug | 0.36 / 40.6% | 0.29 / 15.6% | 0.15 / 6.0% | 0.07 / 0.0% | 0.20 / 15.6% |
| Inpainting | 0.36 / 40.6% | 0.34 / 34.4% | 0.63 / 12.5% | 0.10 / 0.0% | 0.24 / 21.8% |
| Background | 0.50 / 53.1% | 0.54 / 62.5% | 0.46 / 37.5% | 0.32 / 18.8% | 0.45 / 43.0% |
| ImageNet | 0.50 / 53.1% | 0.52 / 62.5% | 0.56 / 43.7% | 0.39 / 18.8% | 0.48 / 44.5% |
| Texture | 0.50 / 50.0% | 0.54 / 62.5% | 0.63 / 56.2% | 0.44 / 25.0% | 0.51 / 48.4% |
| RoboEngine | 0.56 / 56.2% | 0.59 / 68.7% | 0.79 / 75.0% | 0.58 / 43.7% | 0.62 / 60.9% |
IV-B Augmentation Baseline and Results
All generative baselines use the proposed Robo-SAM to remove calibration or green-screen prerequisites. Background uses Stable Diffusion 2.1 with a random scene prompt; Inpainting replaces five task-irrelevant objects masked by SAM-V2; ImageNet and Texture paste an image or MIL-dataset texture behind the foreground. No aug omits these generative/background substitutions. ColorJitter and RandomCrop remain active during policy training for every method, including No aug. Thus the comparison is between background-generation strategies with shared masking, rather than between unmodified published systems.
IV-C Real Robot Setting
A Franka Panda with Robotiq gripper uses absolute-pose control and one external RGB view, following the DROID hardware setup [7]. Diffusion Policy replaces its image encoder with DINOv2-Base, uses one observation frame, sets action horizon equal to prediction horizon, and trains for 1,000 epochs. Other hyperparameters follow the cited data-scaling study [8].
Towel folding uses 50 demonstrations with placements in a 35 × 35 cm area; mouse-to-pad placement uses 100 demonstrations in a 15 × 15 cm area. By default every demonstration receives one augmentation, so dataset counts remain 50 and 100. Training data come from one scene. Towel policies are tested in four new scenes and mouse policies in two; each scene has eight evaluations. Tables stay within 73–76 cm height, explicitly limiting the experiment to visual rather than broad spatial generalization.
| Inpainting | Background | Imagenet | Texture | RoboEngine |
| 3.90 sec | 1.91 sec | 0.97 sec | 0.97 sec | 2.17 sec |
IV-D Policy Evaluation Results
RoboEngine’s reported average normalized behavior score is 0.62, versus 0.20 for No aug and 0.51 for Texture. The headline 210% gain corresponds to the relative increase in the rounded behavior score, not a 210-percentage-point success-rate gain. Average stage success is 60.9% versus 15.6% and 48.4%—absolute gains of 45.3 and 12.5 points. For full task completion, towel success is 68.7% versus Texture’s 62.5%, and mouse success is 43.7% versus 25.0%. The average mixes grasp and completion metrics rather than measuring a single complete-task success probability.
At batch size one, reported augmentation time per frame is 2.17 s for RoboEngine, 1.91 s for Background, 3.90 s for Inpainting, and 0.97 s for ImageNet/Texture. These are offline augmentation costs, not policy inference latency; the timing table does not identify its GPU. Inpainting is described as sensitive to object-mask and hyperparameter choices. No separate ablation isolates robot-mask fine-tuning, background-model fine-tuning, or foreground conditioning.
Several reported score aggregates need care. The Inpainting mouse-grasp entry is 0.63 in Table II, while its Table IV raw scene means are 1.25 and 0, inconsistent with that normalization; the four displayed Table II behavior scores also do not average to its stated 0.24. These source values are preserved rather than silently corrected. Raw “Finish” means exceed the stated 0–3 stage scale and appear to represent a combined 0–6 grasp-plus-completion score, as implied by the main normalized values, but this convention is not explained explicitly.
IV-E Performance Scaling Trend
Scaling uses only towel completion in three new scenes, with 50, 100, 200, or 300 augmented demonstrations. A 2× mix condition combines 50 originals with 50 augmentations. Mean raw scores rise from about 1.63 without augmentation to 3.63 at 1×, 4.17 at 2×, 4.21 at 4×, and 4.38 at 6×; the mixed condition scores 3.96. More variants improve this aggregate with diminishing gains, while individual scenes are non-monotonic. Mixing original data helps relative to 1× but does not beat an equally sized all-augmented set.
V CONCLUSION
The evidence supports background augmentation as a useful way to improve visual transfer for the two tested policies, together with robot-mask improvements on the sampled image sets. The method does not enforce temporal coherence, multiview consistency, or 3D geometry. Suggested remedies using video diffusion or depth-based reconstruction are future work. Foreground-mask failure, task-description errors, limited manipulation coverage, score-reporting ambiguities, and small rollout counts remain practical limits; no confidence bands or multi-seed policy-training results quantify uncertainty.
APPENDIX
The appendix records segmentation/background fine-tuning, exact rollout score definitions, all scene-level means, and the scaling experiment’s training schedule.
V-A Details of Model Training
Robo-SAM fine-tuning uses 3,562 training images, 92 validation images, 30 epochs, learning rate 10⁻⁵, and batch size 32. The text says remaining RoboSeg images are evaluated, but these numbers plus the reported 97-image test set do not exhaust the stated total of 3,800; the remaining split is not explained. BackGround-Diffusion fine-tunes on all RoboSeg images for 100 epochs, learning rate 5 × 10⁻³, batch 32, with GPT-4o scene descriptions. Because all images train the background generator, the robot-segmentation holdout should not be treated as an independent holdout for every component.
V-B Evaluation Criteria
Each stage is graded 0–3, and a stage succeeds at score ≥2. For towel grasping: 0 means no contact; 1 means a grasp away from the left edge; 2 means a left-edge grasp outside the central 7.5 cm area; 3 means a grasp inside that area. Folding scores 0 for no fold, 1 for overlap below one third of maximum, 2 for overlap between one and two thirds, and 3 for overlap above two thirds.
For mouse grasping: 0 means no contact; 1 means contact without a stable lift or a drop; 2 means a stable grasp after sliding more than 3 cm; 3 means a stable grasp with little movement. Placement scores 0 for dropping off the pad, 1 for dropping onto it, 2 for stable placement outside the central 10 × 10 cm area, and 3 for stable placement inside it. Exact boundary cases at the folding fractions are not specified. These stage rubrics must be distinguished from the apparently cumulative “Finish” scores in the raw tables.
| Fold Towel (Grasp) | Fold Towel (Finish) | Put Mouse (Grasp) | Put Mouse (Finish) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Student Desk | Pink Sheet | Meeting Room | Scholar Room | Student Desk | Pink Sheet | Meeting Room | Scholar Room | Black Boam | White Sheet | Black Boam | White Sheet | |
| No aug | 0.875 | 1.125 | 1.25 | 1.125 | 1.625 | 1.625 | 1.625 | 2 | 0.875 | 0 | 0.875 | 0 |
| Inpainting | 0.625 | 1.625 | 0.875 | 1.25 | 1.375 | 3 | 1.75 | 2.125 | 1.25 | 0 | 1.25 | 0 |
| Background | 1.875 | 1.25 | 1.5 | 1.375 | 4.125 | 2.875 | 3 | 2.875 | 1.625 | 1.125 | 2.375 | 1.5 |
| ImageNet | 1.75 | 1.5 | 1.25 | 1.5 | 3.75 | 2.75 | 2.875 | 3.125 | 1 | 2.375 | 1.25 | 3.375 |
| Texture | 1.875 | 1.5 | 1.5 | 1.125 | 4.125 | 3.5 | 3 | 2.25 | 1.375 | 2.375 | 2.125 | 3.125 |
| RoboEngine | 2 | 1.75 | 1.5 | 1.5 | 4.5 | 3.25 | 3.125 | 3.375 | 2.5 | 2.25 | 3.875 | 3.125 |
V-C Raw Evaluation Results
Table IV retains all scene-level mean scores from eight evaluations per task/scene. The four towel environments and two mouse environments are shown separately. Means alone cannot reconstruct the number of rollouts exceeding a success threshold, despite the caption’s suggestion that success rates can be calculated from this table; per-rollout scores would be needed. The table is useful for inspecting scene variation but does not provide dispersion or trial-level data.
| Pink Table | Meeting Room | Student Table | |
|---|---|---|---|
| 4.375 | 2.875 | 4.625 | |
| No aug | 1.625 | 1.625 | 1.625 |
| 3.25 | 3.125 | 4.5 | |
| 4.5 | 2.875 | 5.125 | |
| 5 | 3 | 4.625 | |
| 4.625 | 3.375 | 5.125 |
V-D Details of Scaling Trend Experiment
Training uses 1,000 epochs at 1×, 700 at 2× and 2× mix, 400 at 4×, and 300 at 6×. The paper describes the resulting steps as similar, although dataset-size × epoch products scale as 1.0, 1.4, 1.6, and 1.8 relative to 1×, so compute is not exactly matched. Each variant receives eight rollouts in each of three scenes. Table V preserves all raw means; Figure 5 presents their average. The experiment establishes this limited scaling trend without proving saturation or disentangling extra data variety from additional training updates.
ACKNOWLEDGMENT
The authors acknowledge support from Chinese national research programs, Shanghai Qi Zhi Institute, and Tsinghua University.