Overview
At a glance
Data Inoculation (DI) changes how robot demonstrations are collected. A human remains in closed-loop control while a residual policy adds bounded, state-dependent perturbations. Online RL trains this policy on the growing dataset; the human's corrections supply examples of recovering from deviations resembling those a learned policy might make.
On three precision-manipulation tasks, the resulting data improves or matches downstream BC and offline RL performance. Gains are largest for cable insertion and cube balancing; keycap removal changes less. DI requires continuous human participation and initially injects poorly informed perturbations.
1 Introduction
Clean demonstrations concentrate on states that a skilled operator visits. A deployed policy makes errors, reaches unfamiliar states, and may lack examples of recovery. More capable policies therefore still depend on demonstration coverage [7] [28].
DI collects corrections throughout shared control, rather than waiting to switch from an autonomous policy to a human after a mistake. Its residual policy is trained to maximize task reward, not to sabotage the operator. Early critic errors nevertheless cause policy-like deviations that the human can correct; later residuals can become more useful.
2 Related Work
Human-in-the-loop learning
DAgger-style methods collect expert labels or interventions where a learner visits [21] [9]. After control switches to a human, however, the trajectory can quickly leave the learner's state distribution. DI combines human commands and learned residuals continuously. Earlier residual shared-autonomy methods learn in simulation or from fixed demonstration data; DI updates its residual policy online during real data collection [23] [22].
Data perturbation
Randomization and augmentation broaden training data, often without changing how operators collect it. Noise-injection methods such as DART deliberately elicit human recovery behavior [10] [19] [24]. DI replaces undirected disturbances with task- and state-dependent residuals learned from reward, and evaluates the collected data with both imitation learning and offline RL.
3 Preliminaries
The robot acts in an MDP with image, proprioceptive, and recent-action observations. A sparse reward of −1 until success favors reaching the goal quickly. A critic estimates action values through TD learning, and a policy maximizes those values. Standard teleoperation records synchronized observations and human commands, but consistent expert execution provides few examples of correcting the learner's eventual errors.
Equation 1
| (1) |
4 Data Inoculation
Shared control: the residual policy observes the scene and current human command, predicts a bounded perturbation, and adds a scaled version to the executed command. The operator sees the result and continually adjusts. Records retain both the clean human command and the perturbed action actually executed.
Online learning: a critic evaluates executed actions, allowing updates from perturbed and unperturbed transitions. Because the next human command is unavailable for a general TD backup, a BC flow model approximates it; the residual policy then perturbs that sampled command. The residual objective maximizes the critic's value with an additional magnitude penalty. Optimizing the same task reward used for downstream extraction is intended to expose policy-relevant errors, although it does not guarantee exact matching of a future policy's mistakes.
Timing: a fixed action delay is used during learning and evaluation, and observations include the action already being executed. To give the operator time to react, the residual remains constant for five control steps: 0.5 seconds at 10 Hz. Downstream extraction follows Q-chunking's BFN variant with five-action chunks and four candidate chunks per decision [12]. The original five equations and complete asynchronous collection/training algorithm follow.
Equations 2–5
| (2) |
| (3) |
| (4) |
| (5) |
Equation
5 Experimental Setup
Robot setup and tasks.
A Franka arm, Robotiq gripper, two wrist cameras, and SpaceMouse teleoperation support three tasks: balance two cubes on a third cube in a T shape; insert a USB-C cable through a narrow case opening; and use a puller to detach a keyboard keycap. Cube balancing is especially sensitive to millimeter-scale placement errors.
Data collection procedure.
Three practiced operators collect data across multiple runs. The study uses nine collection seeds for t-cube and five each for cable and keycap. These are separate collection and policy-training runs, not nine or five different operators.
Policy learning and evaluation.
Policies receive two RGB views plus end-effector pose and gripper state. Each action contains three position changes, yaw change, and absolute gripper position. Policies predict chunks of five actions. Each collection seed produces a separately trained policy evaluated over 20 rollouts.
Comparisons.
DI is compared with clean teleoperation and uniform random perturbations held constant over the same five-step interval. Neither approach perturbs the gripper command. Random amplitudes are 0.05 for t-cube/cable and 0.1 for keycap; an action magnitude of 0.1 corresponds to 3 mm of translation and 0.01 radians of rotation.
BC trains a flow-matching policy on the collected data. Offline RL uses BFN: the BC policy proposes four candidate chunks and a learned critic selects the highest-valued one. The comparisons assess downstream data utility, not just the operator's performance during collection.
6 Results
Downstream success: DI raises cable BC/RL success from 37%/35% with clean data to 66%/67%, and t-cube from 38%/42% to 50%/59%. Keycap improves more modestly, from 63%/81% to 72%/84%. Random perturbations help cable but provide little t-cube benefit and reduce keycap RL success. Figure 2 and the appendix tables preserve all means, standard errors, and per-seed counts.
Collection behavior: Figure 3 shows t-cube progress: DI starts poorly, then improves as residuals become smaller and more informed. This late-training trend differs from aggregate episode success in Table 5: DI's overall collection success is below clean teleoperation on all three tasks. The evidence supports better training data without establishing a uniform improvement in collection efficiency.
Where corrections occur: the t-cube analysis splits trajectories into approach, grasp, transport, placement/release, and retreat. It measures changes in human commands opposing residuals, clips negative correction strength to zero, and weights residual magnitude by that strength. DI concentrates corrections during the difficult placement/release phase; random noise induces more during retreat. This task-specific diagnostic supports the proposed mechanism, rather than proving it across all tasks.
7 Discussion
DI uses shared control to collect demonstrations covering recoverable learner-like deviations. Its strongest result is improved downstream policy success on the two more demanding precision tasks. The study covers one robot setup, three tasks, and a small operator group. Residual learning starts from scratch, so early perturbations are poorly informed, and the operator must remain continuously engaged. Bootstrapping from existing data and progressively reducing human involvement are proposed future directions.
8 Acknowledgments
The work acknowledges support from the Office of Naval Research and credits Seohong Park for the method's name.
Appendix A Compute Resource and Robot Hardware
A workstation uses two RTX-4090 GPUs: one trains the critic, BC model, and residual policy, while the other periodically loads their weights for inference. A Franka Emika Panda uses the SERL impedance controller [14]; a 3Dconnexion SpaceMouse supplies human commands.
Appendix B Task Details
T-cube: grasp the red and pink cubes together and balance both on the yellow cube; success requires both to remain in place. Cable: feed the USB-C connector through the narrow case opening and leave it there after moving the arm away. Keycap: seat the puller's two hooks, loosen and detach the Q keycap, then place it on the table. Figure 5 retains every task trajectory and success-state image.
Task
Task Trajectory
Success State
t-cube
cable
keycap
Appendix C Full Results
Tables 1–4 retain summary success rates and every per-seed count out of 20 evaluation trials. Table 5 separately reports success during data collection: clean/random/DI are 96.9%/95.5%/94.2% for cable, 57.7%/45.3%/54.4% for t-cube, and 95.8%/99.2%/94.0% for keycap. These aggregate collection rates must not be substituted for the downstream learned-policy rates.
| Clean | Random | Ours | ||||
|---|---|---|---|---|---|---|
| Seed | BC | RL | BC | RL | BC | RL |
| 0 | ||||||
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| Mean | ||||||
| Clean | Random | Ours | ||||
|---|---|---|---|---|---|---|
| Seed | BC | RL | BC | RL | BC | RL |
| 0 | ||||||
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 5 | ||||||
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| Mean | ||||||
| Clean | Random | Ours | ||||
|---|---|---|---|---|---|---|
| Seed | BC | RL | BC | RL | BC | RL |
| 0 | ||||||
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| Mean | ||||||
| cable | t-cube | keycap | |
|---|---|---|---|
| Clean | (96.9%) | (57.7%) | (95.8%) |
| Random | (95.5%) | (45.3%) | (99.2%) |
| Ours | (94.2%) | (54.4%) | (94.0%) |
Appendix D Hyperparameters
Shared settings include batch size 256, discount 0.97, five-action chunks, four-way BFN selection, Adam at 0.0003, a ten-critic ensemble, and ten flow steps. The control frequency is 10 Hz with a 0.05-second delay. Task-specific training uses 100,000 steps for t-cube and 50,000 for cable/keycap, with reported data sizes of 10,000/4,000/3,000.
Notation note: Table 7 labels the task-specific DI parameter β, while the method section uses λ for the residual penalty; Table 6 also uses λ for the target-network update rate. The tables retain the source's notation and values rather than silently resolving this inconsistency.
| Parameter | Value |
| Batch size | |
| Discount factor () | |
| Action chunk size () | |
| Best-of-N sampling () | |
| Optimizer | Adam |
| Learning rate | |
| Target network update rate () | |
| Target entropy for residual Gaussian policy | |
| Zero residual action probability | |
| Maximum residual action magnitude | |
| Network width | |
| Network depth | hidden layers |
| Critic pessimistic backup () | |
| Critic ensemble size | |
| Policy flow steps () | |
| Action delay () | |
| Control frequency () |
| Tasks | Random Noise | Ours | Training Steps | Data Size |
|---|---|---|---|---|
| t-cube | ||||
| cable | ||||
| keycap |