Literature
Display
← All papers

Robotic Data Inoculation

Qiyang Li · Zhiyuan Zhou · Charles Xu · Sergey Levine

Preprint posted · arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

Refer to caption
Figure 1: Data inoculation (DI). Left: During data collection, we train a residual inoculative policy with online RL to perturb the human’s closed-loop teleoperation commands. The perturbations steer data collection towards the mistakes the policy trained on the data would make, and elicit human’s corrective feedback, “inoculating” the dataset against such mistakes. Right: We test DI on three challenging real-world robotic tasks that require precise manipulation, and DI consistently improve success rate of policies trained with behavioral cloning and offline RL.

At a glance

Data Inoculation (DI) changes how robot demonstrations are collected. A human remains in closed-loop control while a residual policy adds bounded, state-dependent perturbations. Online RL trains this policy on the growing dataset; the human's corrections supply examples of recovering from deviations resembling those a learned policy might make.

On three precision-manipulation tasks, the resulting data improves or matches downstream BC and offline RL performance. Gains are largest for cable insertion and cube balancing; keycap removal changes less. DI requires continuous human participation and initially injects poorly informed perturbations.

1 Introduction

Clean demonstrations concentrate on states that a skilled operator visits. A deployed policy makes errors, reaches unfamiliar states, and may lack examples of recovery. More capable policies therefore still depend on demonstration coverage [7] [28].

DI collects corrections throughout shared control, rather than waiting to switch from an autonomous policy to a human after a mistake. Its residual policy is trained to maximize task reward, not to sabotage the operator. Early critic errors nevertheless cause policy-like deviations that the human can correct; later residuals can become more useful.

2 Related Work

Human-in-the-loop learning

DAgger-style methods collect expert labels or interventions where a learner visits [21] [9]. After control switches to a human, however, the trajectory can quickly leave the learner's state distribution. DI combines human commands and learned residuals continuously. Earlier residual shared-autonomy methods learn in simulation or from fixed demonstration data; DI updates its residual policy online during real data collection [23] [22].

Data perturbation

Randomization and augmentation broaden training data, often without changing how operators collect it. Noise-injection methods such as DART deliberately elicit human recovery behavior [10] [19] [24]. DI replaces undirected disturbances with task- and state-dependent residuals learned from reward, and evaluates the collected data with both imitation learning and offline RL.

3 Preliminaries

The robot acts in an MDP with image, proprioceptive, and recent-action observations. A sparse reward of −1 until success favors reaching the goal quickly. A critic estimates action values through TD learning, and a policy maximizes those values. Standard teleoperation records synchronized observations and human commands, but consistent expert execution provides few examples of correcting the learner's eventual errors.

Equation 1
L(ϕ)=𝔼𝐚′∼π(⋅∣𝐨′)[(Qϕ(𝐨,𝐚)−r−γQϕ¯(𝐨′,𝐚′))2].\displaystyle L(\phi)=\mathbb{E}_{\mathbf{a}^{\prime}\sim\pi(\cdot\mid\mathbf{o}^{\prime})}\left[\left(Q_{\phi}(\mathbf{o},\mathbf{a})-r-\gamma Q_{\bar{\phi}}(\mathbf{o}^{\prime},\mathbf{a}^{\prime})\right)^{2}\right]. (1)

4 Data Inoculation

Shared control: the residual policy observes the scene and current human command, predicts a bounded perturbation, and adds a scaled version to the executed command. The operator sees the result and continually adjusts. Records retain both the clean human command and the perturbed action actually executed.

Online learning: a critic evaluates executed actions, allowing updates from perturbed and unperturbed transitions. Because the next human command is unavailable for a general TD backup, a BC flow model approximates it; the residual policy then perturbs that sampled command. The residual objective maximizes the critic's value with an additional magnitude penalty. Optimizing the same task reward used for downstream extraction is intended to expose policy-relevant errors, although it does not guarantee exact matching of a future policy's mistakes.

Timing: a fixed action delay is used during learning and evaluation, and observations include the action already being executed. To give the operator time to react, the residual remains constant for five control steps: 0.5 seconds at 10 Hz. Downstream extraction follows Q-chunking's BFN variant with five-action chunks and four candidate chunks per decision [12]. The original five equations and complete asynchronous collection/training algorithm follow.

Equations 2–5
L⁡(ϕ)=(Qϕ​(𝐨,𝐚~)−r−γ​Qϕ¯​(𝐨′,𝐚~′))2.\displaystyle L(\phi)=\big(Q_{\phi}(\mathbf{o},\tilde{\mathbf{a}})-r-\gamma Q_{\bar{\phi}}(\mathbf{o}^{\prime},\tilde{\mathbf{a}}^{\prime})\big)^{2}. (2)
L(ϕ)=(Qϕ(𝐨,𝐚~)−r−γQϕ¯(𝐨′,𝐚′+αΔ′))2,𝐚′∼πβ(⋅∣𝐨′)Δ′∼πθ(⋅∣𝐨′,𝐚′).\displaystyle L(\phi)=\big(Q_{\phi}(\mathbf{o},\tilde{\mathbf{a}})-r-\gamma Q_{\bar{\phi}}(\mathbf{o}^{\prime},\mathbf{a}^{\prime}+\alpha\Delta^{\prime})\big)^{2},\quad\quad\mathbf{a}^{\prime}\sim\pi_{\beta}(\cdot\mid\mathbf{o}^{\prime})\quad\Delta^{\prime}\sim\pi_{\theta}(\cdot\mid\mathbf{o}^{\prime},\mathbf{a}^{\prime}). (3)
L(θ)=−𝔼Δ∼πθ(⋅∣𝐨,𝐚)[Qϕ(𝐨,𝐚+αΔ)−λ∥Δ∥22],\displaystyle L(\theta)=-\mathbb{E}_{\Delta\sim\pi_{\theta}(\cdot\mid\mathbf{o},\mathbf{a})}\big[Q_{\phi}(\mathbf{o},\mathbf{a}+\alpha\Delta)-\lambda\lVert\Delta\rVert_{2}^{2}\big], (4)
𝐨t=(It,pt,𝐚t−Δ​t),\displaystyle\mathbf{o}_{t}=(I_{t},p_{t},\mathbf{a}_{t-\Delta t}), (5)
Algorithm 1 Data Inoculation
Input: Residual scale α\alpha; residual penalty λ\lambda
Output: Inoculated demonstration dataset DD
D←∅D\leftarrow\emptyset; Initialize residual policy πθ\pi_{\theta}, critic QϕQ_{\phi}, and BC flow policy πβ\pi_{\beta}
Async Data Collection Loop:
   𝐨0←\mathbf{o}_{0}\leftarrow initial observation after reset
   while t∈{0,1,⋯}t\in\{0,1,\cdots\} do
      Observe 𝐚t\mathbf{a}_{t} from human teleop
      Δt∼πθ(⋅∣𝐨t,𝐚t)\Delta_{t}\sim\pi_{\theta}(\cdot\mid\mathbf{o}_{t},\mathbf{a}_{t})
      Execute 𝐚~t=𝐚t+α​Δt\tilde{\mathbf{a}}_{t}=\mathbf{a}_{t}+\alpha\Delta_{t}
      Observe 𝐨t+1\mathbf{o}_{t+1} and rtr_{t}
      D←D∪{(𝐨t,𝐚t,𝐚~t,𝐨t+1,rt)}D\leftarrow D\cup\{(\mathbf{o}_{t},\mathbf{a}_{t},\tilde{\mathbf{a}}_{t},\mathbf{o}_{t+1},r_{t})\}
Async Training Loop:
   Sample a training batch (𝐨,𝐚,𝐚~,𝐨′,r)∼D(\mathbf{o},\mathbf{a},\tilde{\mathbf{a}},\mathbf{o}^{\prime},r)\sim D
   Update QϕQ_{\phi} with the following TD loss:
Equation
L⁡(ϕ)=(Qϕ​(𝐨,𝐚~)−r−γ​Qϕ¯​(𝐨′,𝐚′+α​Δ′))2,\displaystyle L(\phi)=\big(Q_{\phi}(\mathbf{o},\tilde{\mathbf{a}})-r-\gamma Q_{\bar{\phi}}(\mathbf{o}^{\prime},\mathbf{a}^{\prime}+\alpha\Delta^{\prime})\big)^{2},
where 𝐚′∼πβ(⋅∣𝐨′)\mathbf{a}^{\prime}\sim\pi_{\beta}(\cdot\mid\mathbf{o}^{\prime}), Δ′∼πθ(⋅∣𝐨′,𝐚′)\Delta^{\prime}\sim\pi_{\theta}(\cdot\mid\mathbf{o}^{\prime},\mathbf{a}^{\prime})
   Update πθ\pi_{\theta} with the following policy loss:
Equation
L(θ)=−𝔼Δ∼πθ(⋅∣𝐨,𝐚)[Qϕ(𝐨,𝐚+αΔ)−λ∥Δ∥22],\displaystyle L(\theta)=-\mathbb{E}_{\Delta\sim\pi_{\theta}(\cdot\mid\mathbf{o},\mathbf{a})}\big[Q_{\phi}(\mathbf{o},\mathbf{a}+\alpha\Delta)-\lambda\lVert\Delta\rVert_{2}^{2}\big],
Update πβ\pi_{\beta} with flow-matching loss using (𝐨,𝐚)(\mathbf{o},\mathbf{a})

5 Experimental Setup

Robot setup and tasks.

A Franka arm, Robotiq gripper, two wrist cameras, and SpaceMouse teleoperation support three tasks: balance two cubes on a third cube in a T shape; insert a USB-C cable through a narrow case opening; and use a puller to detach a keyboard keycap. Cube balancing is especially sensitive to millimeter-scale placement errors.

Data collection procedure.

Three practiced operators collect data across multiple runs. The study uses nine collection seeds for t-cube and five each for cable and keycap. These are separate collection and policy-training runs, not nine or five different operators.

Policy learning and evaluation.

Policies receive two RGB views plus end-effector pose and gripper state. Each action contains three position changes, yaw change, and absolute gripper position. Policies predict chunks of five actions. Each collection seed produces a separately trained policy evaluated over 20 rollouts.

Comparisons.

DI is compared with clean teleoperation and uniform random perturbations held constant over the same five-step interval. Neither approach perturbs the gripper command. Random amplitudes are 0.05 for t-cube/cable and 0.1 for keycap; an action magnitude of 0.1 corresponds to 3 mm of translation and 0.01 radians of rotation.

BC trains a flow-matching policy on the collected data. Offline RL uses BFN: the BC policy proposes four candidate chunks and a learned critic selects the highest-valued one. The comparisons assess downstream data utility, not just the operator's performance during collection.

6 Results

Downstream success: DI raises cable BC/RL success from 37%/35% with clean data to 66%/67%, and t-cube from 38%/42% to 50%/59%. Keycap improves more modestly, from 63%/81% to 72%/84%. Random perturbations help cable but provide little t-cube benefit and reduce keycap RL success. Figure 2 and the appendix tables preserve all means, standard errors, and per-seed counts.

Collection behavior: Figure 3 shows t-cube progress: DI starts poorly, then improves as residuals become smaller and more informed. This late-training trend differs from aggregate episode success in Table 5: DI's overall collection success is below clean teleoperation on all three tasks. The evidence supports better training data without establishing a uniform improvement in collection efficiency.

Where corrections occur: the t-cube analysis splits trajectories into approach, grasp, transport, placement/release, and retreat. It measures changes in human commands opposing residuals, clips negative correction strength to zero, and weights residual magnitude by that strength. DI concentrates corrections during the difficult placement/release phase; random noise induces more during retreat. This task-specific diagnostic supports the proposed mechanism, rather than proving it across all tasks.

Original figure
Figure 2: Task success rates. The results for {t-cube/cable/keycap} are averaged over {9/5/5} seeds with 20 evaluation trials each. Error bars indicate the standard errors. Full table in Tables 1, 3, 2 and 4.
Original figure
Figure 3: Success rate over the course of data collection. DI starts at a lower success rate when the residual policy is untrained, but eventually reaches a higher success rate than clean and random collection.
Refer to caption
Figure 4: Amount of human corrective demonstrations over different phases. DI elicits the most corrections at the most critical phase of the t-cube task.

7 Discussion

DI uses shared control to collect demonstrations covering recoverable learner-like deviations. Its strongest result is improved downstream policy success on the two more demanding precision tasks. The study covers one robot setup, three tasks, and a small operator group. Residual learning starts from scratch, so early perturbations are poorly informed, and the operator must remain continuously engaged. Bootstrapping from existing data and progressively reducing human involvement are proposed future directions.

8 Acknowledgments

The work acknowledges support from the Office of Naval Research and credits Seohong Park for the method's name.

Appendix A Compute Resource and Robot Hardware

A workstation uses two RTX-4090 GPUs: one trains the critic, BC model, and residual policy, while the other periodically loads their weights for inference. A Franka Emika Panda uses the SERL impedance controller [14]; a 3Dconnexion SpaceMouse supplies human commands.

Appendix B Task Details

T-cube: grasp the red and pink cubes together and balance both on the yellow cube; success requires both to remain in place. Cable: feed the USB-C connector through the narrow case opening and leave it there after moving the arm away. Keycap: seat the puller's two hooks, loosen and detach the Q keycap, then place it on the table. Figure 5 retains every task trajectory and success-state image.

Task

Task Trajectory

Success State

t-cube

Refer to caption
Refer to caption

cable

Refer to caption
Refer to caption

keycap

Refer to caption
Refer to caption
Figure 5: Three real-robot tasks. Top: t-cube. The robot needs to move two colored cubes and carefully balance them on top of the yellow cube. Middle: cable. The robot needs to feed a USB-C cable through a desktop machine case port hole that is barely bigger than the cable itself. Bottom: keycap. The robot needs to use a plastic tool to extract a transparent keycap from a keyboard.

Appendix C Full Results

Tables 1–4 retain summary success rates and every per-seed count out of 20 evaluation trials. Table 5 separately reports success during data collection: clean/random/DI are 96.9%/95.5%/94.2% for cable, 57.7%/45.3%/54.4% for t-cube, and 95.8%/99.2%/94.0% for keycap. These aggregate collection rates must not be substituted for the downstream learned-policy rates.

Table 1: Task success rates (mean ±\pm standard error). The results for {t-cube/cable/keycap} are averaged over {9/5/5} seeds with 20 evaluation trials each. Best result per column in bold.
cable t-cube keycap
BC RL BC RL BC RL
Clean 0.37±0.060.37\pm 0.06 0.35±0.070.35\pm 0.07 0.38±0.040.38\pm 0.04 0.42±0.060.42\pm 0.06 0.63±0.09\mathbf{0.63\pm 0.09} 0.81±0.06\mathbf{0.81\pm 0.06}
Random 0.48±0.090.48\pm 0.09 0.56±0.040.56\pm 0.04 0.40±0.050.40\pm 0.05 0.42±0.030.42\pm 0.03 0.69±0.06\mathbf{0.69\pm 0.06} 0.67±0.090.67\pm 0.09
Ours 0.66±0.04\mathbf{0.66\pm 0.04} 0.67±0.06\mathbf{0.67\pm 0.06} 0.50±0.03\mathbf{0.50\pm 0.03} 0.59±0.02\mathbf{0.59\pm 0.02} 0.72±0.05\mathbf{0.72\pm 0.05} 0.84±0.06\mathbf{0.84\pm 0.06}
Table 2: Per-seed successes on cable (successes out of 20 trials per entry). The last row reports the mean success rate ±\pm standard error over the five seeds.
Clean Random Ours
Seed BC RL BC RL BC RL
0 4/20\hphantom{0}4/20 2/20\hphantom{0}2/20 6/20\hphantom{0}6/20 9/20\hphantom{0}9/20 14/2014/20 14/2014/20
1 11/2011/20 9/20\hphantom{0}9/20 13/2013/20 13/2013/20 12/2012/20 17/2017/20
2 7/20\hphantom{0}7/20 7/20\hphantom{0}7/20 7/20\hphantom{0}7/20 12/2012/20 12/2012/20 10/2010/20
3 9/20\hphantom{0}9/20 9/20\hphantom{0}9/20 7/20\hphantom{0}7/20 10/2010/20 12/2012/20 13/2013/20
4 6/20\hphantom{0}6/20 8/20\hphantom{0}8/20 15/2015/20 12/2012/20 16/2016/20 13/2013/20
Mean 0.37±0.060.37\pm 0.06 0.35±0.070.35\pm 0.07 0.48±0.090.48\pm 0.09 0.56±0.040.56\pm 0.04 0.66±0.04\mathbf{0.66\pm 0.04} 0.67±0.06\mathbf{0.67\pm 0.06}
Table 3: Per-seed successes on t-cube (successes out of 20 trials per entry). The last row reports the mean success rate ±\pm standard error over the nine seeds.
Clean Random Ours
Seed BC RL BC RL BC RL
0 9/20\hphantom{0}9/20 10/2010/20 4/20\hphantom{0}4/20 9/20\hphantom{0}9/20 9/20\hphantom{0}9/20 12/2012/20
1 3/20\hphantom{0}3/20 0/20\hphantom{0}0/20 8/20\hphantom{0}8/20 7/20\hphantom{0}7/20 8/20\hphantom{0}8/20 9/20\hphantom{0}9/20
2 10/2010/20 12/2012/20 11/2011/20 12/2012/20 11/2011/20 12/2012/20
3 11/2011/20 9/20\hphantom{0}9/20 6/20\hphantom{0}6/20 7/20\hphantom{0}7/20 11/2011/20 12/2012/20
4 8/20\hphantom{0}8/20 9/20\hphantom{0}9/20 7/20\hphantom{0}7/20 7/20\hphantom{0}7/20 10/2010/20 11/2011/20
5 8/20\hphantom{0}8/20 8/20\hphantom{0}8/20 4/20\hphantom{0}4/20 8/20\hphantom{0}8/20 10/2010/20 12/2012/20
6 6/20\hphantom{0}6/20 9/20\hphantom{0}9/20 9/20\hphantom{0}9/20 8/20\hphantom{0}8/20 10/2010/20 13/2013/20
7 5/20\hphantom{0}5/20 6/20\hphantom{0}6/20 14/2014/20 9/20\hphantom{0}9/20 8/20\hphantom{0}8/20 12/2012/20
8 8/20\hphantom{0}8/20 12/2012/20 9/20\hphantom{0}9/20 8/20\hphantom{0}8/20 13/2013/20 14/2014/20
Mean 0.38±0.040.38\pm 0.04 0.42±0.060.42\pm 0.06 0.40±0.050.40\pm 0.05 0.42±0.030.42\pm 0.03 0.50±0.03\mathbf{0.50\pm 0.03} 0.59±0.02\mathbf{0.59\pm 0.02}
Table 4: Per-seed successes on keycap (successes out of 20 trials per entry). The last row reports the mean success rate ±\pm standard error over the five seeds.
Clean Random Ours
Seed BC RL BC RL BC RL
0 14/2014/20 19/2019/20 17/2017/20 19/2019/20 17/2017/20 20/2020/20
1 14/2014/20 15/2015/20 15/2015/20 13/2013/20 17/2017/20 18/2018/20
2 6/20\hphantom{0}6/20 13/2013/20 15/2015/20 15/2015/20 13/2013/20 17/2017/20
3 13/2013/20 15/2015/20 11/2011/20 9/20\hphantom{0}9/20 12/2012/20 16/2016/20
4 16/2016/20 19/2019/20 11/2011/20 11/2011/20 13/2013/20 13/2013/20
Mean 0.63±0.090.63\pm 0.09 0.81±0.060.81\pm 0.06 0.69±0.060.69\pm 0.06 0.67±0.090.67\pm 0.09 0.72±0.05\mathbf{0.72\pm 0.05} 0.84±0.06\mathbf{0.84\pm 0.06}
Table 5: Success rates during data collection, computed from the recorded episodes. Each entry reports successful episodes over all collected episodes and the corresponding percentage.
cable t-cube keycap
Clean 252/260252/260 (96.9%) 432/749432/749 (57.7%) 113/118113/118 (95.8%)
Random 233/244233/244 (95.5%) 311/686311/686 (45.3%) 122/123122/123 (99.2%)
Ours 178/189178/189 (94.2%) 361/663361/663 (54.4%) 110/117110/117 (94.0%)

Appendix D Hyperparameters

Shared settings include batch size 256, discount 0.97, five-action chunks, four-way BFN selection, Adam at 0.0003, a ten-critic ensemble, and ten flow steps. The control frequency is 10 Hz with a 0.05-second delay. Task-specific training uses 100,000 steps for t-cube and 50,000 for cable/keycap, with reported data sizes of 10,000/4,000/3,000.

Notation note: Table 7 labels the task-specific DI parameter β, while the method section uses λ for the residual penalty; Table 6 also uses λ for the target-network update rate. The tables retain the source's notation and values rather than silently resolving this inconsistency.

Parameter Value
Batch size 256256
Discount factor (γ\gamma) 0.970.97
Action chunk size (hh) 55
Best-of-N sampling (NN) 44
Optimizer Adam
Learning rate 3×10−43\times 10^{-4}
Target network update rate (λ\lambda) 5×10−35\times 10^{-3}
Target entropy for residual Gaussian policy −5×|𝒜|-5\times|\mathcal{A}|
Zero residual action probability 0.10.1
Maximum residual action magnitude ‖Δ​a‖∞≤0.1\|\Delta a\|_{\infty}\leq 0.1
Network width 512512
Network depth 44 hidden layers
Critic pessimistic backup (ρ\rho) 22
Critic ensemble size 1010
Policy flow steps (TT) 1010
Action delay (δ\delta) 0.05 s0.05\text{\,}\mathrm{s}
Control frequency (FF) 10 Hz10\text{\,}\mathrm{Hz}
Table 6: Shared hyperparameters.
Tasks Random Noise Ours Training Steps Data Size
cc β\beta
t-cube 0.050.05 1010 1×1051\times 10^{5} 1000010000
cable 0.050.05 0.030.03 0.5×1050.5\times 10^{5} 40004000
keycap 0.10.1 0.010.01 0.5×1050.5\times 10^{5} 30003000
Table 7: Task-specific hyperparameters.
Concept demonstration

Open image ↗