At a glance
OopsieRecover turns a few human corrective trajectories and language explanations into reusable, context-gated spatial costs. At deployment, it grounds the relevant entities in a new scene, predicts candidate action consequences, and steers a frozen VLA toward lower-cost behavior. Experiments cover three simulated and three physical tasks with pose and scene variations. Improvements depend on reliable geometry, motion prediction, and available corrective candidates; empirical consistency checks are not formal guarantees of recovery.
I Introduction
A corrective action sequence is tied to one scene, while the reason for intervention—such as keeping a mug upright near an obstacle—may transfer. The method uses language, pre-intervention behavior, and the preferred human continuation to infer that reason. A gate limits when the spatial objective applies, and candidate ranking changes execution without retraining the base policy. The paper reports average physical task progress increasing from 35.1% to 61.0%; this is stage completion, not full-task success.
II Related Works
HG-DAgger [6] and related interactive imitation approaches update policies from interventions. Semantic reward and constraint methods instead infer objectives, while inference-time steering ranks or guides action candidates. OopsieRecover combines these ideas by synthesizing correction-specific gated programs from a small set of annotated interventions and applying them to a frozen π0.5 policy [1]. Its comparison is therefore between different ways of using corrective evidence, not between otherwise identical training procedures.
III OopsieRecover
A human supplies a corrective trajectory, a free-form explanation, and a correction-type label. The system groups matching types, grounds relevant entities, synthesizes and checks a spatial program, then applies that program when its trigger is detected. Additional interventions refine an existing correction or create a new type. Adaptation occurs in the correction program and its grounding rather than in VLA weights.
III-A Collecting Semantic Corrections
Each intervention record contains a pre-takeover window, the human-controlled segment, and—when available—a following base-policy segment. The pre-window serves as negative evidence and the continuation as preferred behavior. Human-provided type labels group episodes that should share a corrective principle. This supervision includes semantic annotation and grouping, not only passive collection of robot trajectories.
Equation 1
| (1) |
III-B Corrections as Semantically Grounded Gated Costs
A correction consists of semantic entity labels, a logical gate, a continuous correction cost, and a task-progress term. Entities can be objects, parts, spatial regions, or singleton locations represented by 3D point sets. Costs then express relationships such as clearance, alignment, or orientation using spatial extent. The gate localizes the correction to a relevant context, while a weight trades correction against ongoing task progress.
Equation 2
| (2) |
III-C Corrective Program Synthesis and Refinement
Program synthesis proceeds through entity grounding, cost proposal, threshold calibration, and evidence-based verification/refinement. The VLM supplies hypotheses; the recorded interventions constrain which hypotheses are consistent with observed corrections.
III-C 1 Grounding relevant scene entities
Given task language, correction semantics, and an initial image, the VLM proposes entity labels and seed pixels. Segmentation expands region seeds into masks, and depth lifts sampled pixels into 3D point sets. Singleton entities are lifted directly. Accurate segmentation, depth, and semantic correspondence are prerequisites for meaningful geometric costs.
III-C 2 Proposing a gated cost
Regions are tracked across frames and resampled into point sets rather than tracking every 3D point individually. Their centroid trajectories compactly describe pre-intervention and human-correction motion to the VLM, which generates executable gate, correction, and task terms. This compressed evidence may leave multiple plausible explanations, so the result is an abductive hypothesis rather than uniquely recovered human intent.
III-C 3 Intervention-based calibration
The gate score combines contextual activation with correction cost. Its threshold is the minimum score observed in the pre-intervention calibration window, and multiple episodes use the minimum across episodes. This makes the threshold sensitive to the recorded takeover context and aims to cover all demonstrated failures. It does not calibrate false-positive rates or reliability over unseen states.
Equations 3–4
| (3) |
| (4) | ||||
III-C 4 Gated cost verification and refinement
The ranking check requires the pre-intervention mean score to exceed that of the human continuation and, when available, the resumed base-policy window. The trigger check requires a threshold crossing before takeover. All episodes of a correction type must pass jointly. The task-progress term is excluded from these checks.
When checks fail, the VLM revises gate/correction parameters or logic using the failure evidence. Crucially, if the refinement budget is exhausted, the implementation retains the final program even if it has not passed both checks. “Verification” therefore means empirical consistency testing on recorded episodes, not a formal safety certificate or guaranteed correct program.
Equations 5–6
| (5) |
| (6) |
III-D Correction-Guided Policy Steering
At test time, reference images and seed annotations guide re-identification of semantic entities in the new scene. A threshold crossing activates steering. The base VLA samples finite action chunks, optionally augmented with cardinal actions, and their predicted point-set trajectories are ranked by discounted gated costs. Simultaneous corrections contribute a weighted sum. Simulation uses simulator physics for prediction; physical experiments use a grasp-based dynamics model. The best-ranked candidate cannot guarantee recovery if predictions are inaccurate or no candidate contains an effective correction.
Equations 7–8
| (7) |
| (8) |
IV Experiments
Transfer across task variations. Simulation uses CoffeeMugSetup, BlenderLid, and StartKettle with a RoboCasa-fine-tuned π0.5. Five to eight sequential corrections per task are evaluated at the original reset, new object poses in the same scene, and new poses in unseen scenes. Each method is tested over 50 episodes. Baselines include the frozen policy, intervention fine-tuning, an OLAF reimplementation, and task-description-based VLS steering. OopsieRecover leads or ties in the original setting and separates from the other methods on the two transfer levels under the reported statistical test. VLS receives different supervision, while all correction-based baselines use the same collected intervention data.
Physical experiments. DROID hardware and its π0.5 checkpoint are tested on croffle placement, coffee-pod loading, and shelving. Each task uses three to six semantic corrections and 20-rollout evaluations at each variation level. Outcomes are fractions of task stages completed. The reported analysis separates OopsieRecover from competitors at the original and unseen-scene levels, but it shares a statistical group with the base policy at the intermediate pose-variation level. This is more qualified than a universal significant improvement at every level.
Correction efficiency. On CoffeeMug, OopsieRecover reaches 84% success with eight corrections on 25 held-out randomized scenes. Figure 5 places the HG-DAgger result of 28% at thirty corrections; the prose’s claim of the “same number” conflicts with the plot and caption. The plot also shows non-monotonic performance after the eight-correction peak, so additional data do not establish strictly monotonic improvement.
Targeted failure prevention. A single learned steam-handle collision correction is tested in ten novel scenes. A human deliberately drives the robot toward the failure, and the detector and steering prevent it for the next 100 environment steps in all ten trials. This supports transfer for that controlled failure mode, not a general 100% recovery guarantee across spontaneous deployment failures.
V Limitations
The representation handles spatial relations but does not explicitly model contact force or hidden physical properties. Grounding, tracking, and dynamics errors can affect both triggering and ranking. Finite candidate sets constrain recoverable behavior, and consistency with a few recorded interventions does not establish correctness in all future situations. Distilling successful corrective executions into a policy is proposed as a future extension.
VI Conclusion
Semantic explanations can make a small corrective dataset useful beyond its original scene by expressing when and how behavior should change. The evidence supports gated spatial steering of a frozen VLA on the tested tasks. Its practical limits are the quality of inferred programs, geometry and dynamics, and the corrective behavior present in the candidate set.
VII Acknowledgments
The work acknowledges the UT Austin RobIn Lab and support from Toyota Research Institute, the National Science Foundation, NVIDIA, Amazon, and DARPA TIAMAT. The paper states that its conclusions do not imply sponsor endorsement.