Literature
Display
← All papers

Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy

Shuo Sha · Yixuan Wang · Binghao Huang · Antonio Loquercio · Yunzhu Li

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

Appendix

Contents

At a glance

A residual copilot corrects human teleoperation commands on three contact-rich assembly tasks. Less than five minutes of real interaction per task supplies a nearest-neighbor human surrogate and simulator calibration; reinforcement learning trains assistance in simulation before deployment with real users. The 16-person study reports higher success, lower reported workload and better downstream imitation learning. Results support this particular assistance pipeline, with important limits in participant coverage, control mappings and evaluation statistics.

I Introduction

Fine alignment and force regulation are difficult through a teleoperation interface even when the operator knows the intended task. This work divides responsibility: humans supply coarse commands, while a learned residual supplies local corrections. It avoids requiring an autonomous expert policy as the main training prior, using a small real dataset to approximate the human command distribution in simulation.

The contribution is a practical real-to-sim-to-real pipeline, not a guarantee that small residuals preserve intent or that a simulated surrogate captures every human response. The task still requires object models/poses, simulator calibration and designed rewards. This edition covers all 17 pages of v2, dated 20 March 2026, including seven complete figures, seven tables, six displayed formula containers, 81 references and the complete participant instruction sheet.

Complete original PDF S1.F1
Fig. 1: Overview of our real-to-sim-to-real shared autonomy framework. A small amount of real teleoperation data (< 5 minutes) is used to construct a lightweight human surrogate, which drives simulation-based training of a residual copilot policy. At deployment, the copilot provides low-level corrective actions that combine with human commands to produce reliable and efficient shared autonomy for fine-grained, contact-rich manipulation tasks, where direct teleoperation struggles with precise alignment, axis-constrained rotation, and contact regulation, including nut threading, gear meshing, and peg insertion.

II Related Works

II-A Shared Autonomy

Shared autonomy combines human control with machine assistance. Expert-prior methods guide commands toward a planner or autonomous policy; copilot methods instead learn assistance conditioned on the command and environment. The latter can avoid first solving full autonomy, but depend on a useful model of the human during training. The paper builds on deep-RL shared autonomy and residual policy learning for shared autonomy, replacing a learned behavioral surrogate with nearest-neighbor retrieval.

Complete original PDF S2.F2
Fig. 2: Method overview of residual copilot learning with a real-to-sim-to-real pipeline. Left: less than five minutes of real-world teleoperation interaction data is collected under admittance control. Center: in simulation, a lightweight kNN-based human surrogate constructed from real-world data generates base actions conditioned on the environment state, while a residual copilot policy is trained with reinforcement learning to produce corrective residual actions. Right: at deployment, the residual copilot assists the human operator in zero-shot, conditioning on the estimated environment state and human base actions. The residual formulation preserves human intent while improving alignment, contact regulation, and execution robustness.

II-B Residual Policy Learning

A residual policy adds a correction to a nominal command, reducing the burden of learning an entire controller. Here the nominal command comes from the human. Penalizing correction magnitude and change encourages restrained assistance, but neither penalty proves preservation of latent intent or safety. Training uses dense regularization/alignment terms as well as sparse terminal rewards, despite the introduction’s emphasis on sparse reward.

II-C Real-to-Sim-to-Real Learning

Real-to-sim-to-real learning first calibrates a task simulator from real interaction, trains at scale in simulation, then deploys physically. Shared autonomy adds a second modeling problem: the simulated human must remain useful under the states induced by assistance. This work uses real teleoperation both for the surrogate and for control/contact calibration, followed by domain randomization.

III Method

III-A Problem Definition

The formulation is a partially observable decision process with environment state and a latent human goal. The copilot observes robot/object state and the human command; it does not explicitly infer the goal. A Gaussian policy outputs a normalized residual, scaled and composed with the human’s pose/gripper target. Translation and gripper corrections add, while rotation corrections compose as quaternions.

PPO maximizes a reward combining general penalties/shaping and task success. The main text calls the backbone an MLP, while the implementation table specifies a shared two-layer LSTM followed by an MLP; the appendix is the more detailed architecture description. The retained formulas define the process, command composition and retrieval metric without changing their notation.

Equations
ℳ=⟨𝒳,𝒜,𝒯,ℛ,Ω,O,γ⟩.\mathcal{M}=\langle\mathcal{X},\,\mathcal{A},\,\mathcal{T},\,\mathcal{R},\,\Omega,\,O,\,\gamma\rangle.
at=ath⊕α​atres,a_{t}=a_{t}^{h}\oplus\alpha a_{t}^{\mathrm{res}},

III-B Human Surrogate Model

The human surrogate retrieves demonstrated commands with a weighted distance over end-effector translation, quaternion geodesic rotation and gripper opening. Metric weights are tuned by nearest-neighbor prediction error. At runtime, it samples among the nearest neighbors with a softmax over negative distance and emits a short consecutive chunk from the selected demonstration, reducing rapid switches between unrelated neighbors.

Chunks span 5–15 steps. Uniform local perturbations are multiplied by a gate updated as 0.8 times its previous value plus 0.2 times a Bernoulli(0.5) sample. The gate changes smoothly, but the perturbation vector is sampled independently each step; this is not a guarantee of temporally smooth noise directions. Retrieval stays on recorded actions before perturbation, whereas perturbed commands and visited states can leave empirical support.

The approach assumes broadly consistent human strategies with and without assistance. It avoids training a parametric behavior model on very little data, but the paper does not fully specify the neighbor count, temperature, metric weights or residual scale needed to reproduce every choice. Human adaptation remains a distribution-shift risk.

Equation 1
d⁡(a,a′)=α1​‖p−p′‖2+α2​dSO⁡(3)​(q,q′)+α3​|u−u′|,d(a,a^{\prime})=\alpha_{1}\|p-p^{\prime}\|_{2}+\alpha_{2}d_{\mathrm{SO}(3)}(q,q^{\prime})+\alpha_{3}|u-u^{\prime}|, (1)

III-C Admittance Control

Task-space admittance lets the robot yield to external wrench while tracking a target. A virtual mass–damper–spring relationship links pose error, velocity, acceleration and measured force/torque. Compliance supports contact-rich assembly, but the experiment’s force cutoff and reported success are not a formal safety certification. Real and simulated controller parameters differ and are retained in Table III.

Equation
f=M​ξ¨+D​ξ˙+K⁡(ξ−ξref),f\;=\;M\ddot{\xi}\;+\;D\dot{\xi}\;+\;K(\xi-\xi_{\mathrm{ref}}),

III-D Real-to-Sim-to-Real

Real teleoperation fits the surrogate and tunes arm/gripper PID, task-space stiffness/damping/inertia, friction and center of mass. Training randomizes controller parameters and perceived object poses to absorb residual simulation mismatch. “Zero-shot” deployment refers to transferring the trained residual without subsequent real-world RL fine-tuning; it does not mean no real data or hardware calibration.

IV Experiments

The experiment asks whether assistance helps humans, whether the nearest-neighbor surrogate generalizes better than behavioral cloning, and whether assisted demonstrations improve a later imitation policy. Tasks use the NIST assembly board with submillimeter radial clearance: mesh a medium gear while retaining the grasp, tighten an M32 nut by at least 180° using three 60° turns, or insert an 8 mm-long peg to the receptacle bottom. Dropping the object before completion or exceeding the robot’s force threshold is failure.

Twelve novice operators perform gear or peg tasks, while four experienced operators perform nut threading. Each participant performs one task; method use alternates after familiarization. Experience and task are therefore confounded, and the study cannot isolate an independent novice-versus-expert effect. Exact participant allocation between gear and peg is not given.

Baselines include a diffusion-policy surrogate trained from augmented small-data demonstrations, an “Expert Pilot” fitted to successful rollouts of the authors’ own residual policy, and delayed/noisy versions of that pilot. Guided-diffusion assistance uses either small-data or expert-rollout training; Residual BC uses the same RL setup with the behavioral-cloning surrogate. The expert is not an independently sourced human expert or oracle. Prior diffusion shared autonomy motivates guidance; the main text’s specific GD-baseline citation [74] instead resolves to DexterityGen in this version.

Complete original PDF S4.F3
Fig. 3: Human experiment results across three contact-rich assembly tasks. Rows correspond to Gear Meshing, Nut Threading, and Peg Insertion. Left: success rate and mean completion time across successful episodes (solid line; shaded region denotes standard deviation; dashed line indicates per-method mean). Right: subjective workload (NASA-TLX subscales; lower is better) and user satisfaction (higher is better). Across tasks, the Residual Copilot improves success rate and can effectively reduce completion time, while lowering reported workload and increasing user satisfaction compared to direct teleoperation and a residual baseline.
Complete original PDF S4.F4
Fig. 4: User study qualitative results. Representative trajectory temporal overlays comparing unassisted teleoperation and our Residual Copilot on three high-precision tasks. Top row (effectiveness): our copilot enables reliable task completion in cases where teleoperation fails due to force, rotation, or grasp errors. Bottom row (efficiency): when both succeed, our copilot consistently reduces completion time and exemplifies smooth motions. Together, these results demonstrate that our method provides both reliable and efficient assistance in challenging manipulation scenarios.
TABLE I: Cross-pilot generalization and robustness in simulation. Rows denote copilots trained with different training pilots; columns evaluate each trained copilot under distinct evaluation pilots across three tasks. Each cell reports task progression (over 1000 simulation trials); see Appendix for metric details. Gray and blue diagonal entries indicate in-distribution evaluation, i.e., a copilot evaluated with the pilot it was trained on; off-diagonal entries measure generalization to unseen pilots. The Residual Copilot maintains consistently high performance across evaluation pilots, demonstrating robustness to pilot mismatch and perturbations such as lag and action noise.
Complete original PDF S4.T1
Original selectable table
Copilot Eval. Pilot
Laggy Pilot Noisy Pilot Expert Pilot BC Pilot kNN Pilot
Gear Peg Nut Gear Peg Nut Gear Peg Nut Gear Peg Nut Gear Peg Nut
No Copilot 0.97 0.94 0.49 0.92 0.91 0.24 0.99 0.99 0.61 0.84 0.71 0.03 0.87 0.85 0.16
GD Expert 0.96 0.89 0.24 0.95 0.91 0.25 0.94 0.90 0.28 0.94 0.89 0.34 0.95 0.90 0.27
GD BC 0.63 0.56 0.00 0.65 0.56 0.00 0.62 0.53 0.01 0.63 0.52 0.01 0.66 0.55 0.00
Residual BC 0.96 0.44 0.00 0.91 0.44 0.00 0.98 0.44 0.00 0.70 0.40 0.00 0.86 0.46 0.02
Residual Copilot (Ours) 0.99 0.92 0.65 0.96 0.92 0.69 0.98 0.93 0.74 0.97 0.83 0.21 1.00 0.99 0.81

IV-A Copilot Performance

Figure 3 reports gear success of 69% for direct teleoperation, 68% for Residual BC and 93% for the proposed copilot; nut success increases from 50% to 80%, and peg success from 72% to 80%. The nut gain is 30 percentage points. Qualitative sequences show corrections to alignment, rotation and grasp approach, but selected examples are not independent statistical evidence.

Completion-time curves include only successful episodes. Their shaded regions are standard deviations, not confidence intervals, and the dashed lines are per-method means. The plots favor assistance overall, but conditional timing can change when assistance also changes which trials succeed. No significance tests, participant-level uncertainty or independent-training variability are reported.

Operators report lower demand and higher satisfaction on seven-point scales, including perceived intent correctness. These are subjective, unblinded responses; they do not measure latent-intent preservation directly. Appendix instructions explicitly teach users to rely on the copilot for insertion and amplified rotation, so improved performance is evidence for the complete instructed system, not proof that human strategy remains unchanged.

IV-B Human Surrogate Analysis

The human gear comparison favors the nearest-neighbor surrogate over Residual BC. In simulation, Table I evaluates each copilot against five pilot types with 1,000 trials per cell. Entries are mean task progression, not binary success rates. With its own nearest-neighbor pilot, the method reaches 1.00/0.99/0.81 for gear/peg/nut, versus 0.87/0.85/0.16 without assistance.

Cross-pilot robustness is substantial but not universal. With the Expert Pilot, unaided gear/peg progression is 0.99/0.99 versus 0.98/0.93 with assistance; with the Laggy Pilot, peg progression is 0.94 unaided versus 0.92 assisted. Under the BC Pilot, GD Expert reaches 0.34 nut progression versus the proposed method’s 0.21. Its nut score also falls from 0.81 with its training pilot to 0.21 with BC. Thus the table does not establish uniformly beneficial assistance or negligible distribution shift.

Residual BC often stalls at low progress, while guided diffusion can pull even a competent pilot away from useful behavior. The claimed exploration/local-minimum explanation is plausible but not directly isolated by the final scores. All original cells and training/evaluation diagonal markings remain available.

IV-C Data Quality Comparison

Five gear-study participants supply demonstrations for two comparisons: equal teleoperation attempts, using all successes, and equal numbers of successful demonstrations. Figure 7 shows 36 direct versus 43 assisted successful episodes under matched attempts, and 34 versus 34 under matched successes. These episode counts are training data, distinct from the 20 downstream evaluation rollouts.

With matched attempts, the assisted-data policy grasps in 18/20 and inserts in 11/20 rollouts, versus 7/20 and 1/20 from direct data. With matched successes, it achieves 19/20 grasps and 9/20 insertions, versus 6/20 and 0/20. The equal-success comparison supports a benefit beyond simply collecting more successful trajectories.

The authors attribute this to more consistent contact timing and approach behavior. However, equal counts and model capacity do not make the datasets otherwise identical: Figure 7 also shows different initial-state coverage. No confidence intervals or independent training seeds are reported. The results demonstrate better downstream performance in this collection setup, rather than isolating trajectory consistency as the sole cause.

TABLE II: Downstream policy progression from copilot vs. teleoperation data (Gear Meshing). Diffusion policies are trained on demonstrations collected under Residual Copilot assistance or direct teleoperation, using matched attempts and matched successes to control for data budget. Results show copilot assistance improves learnability and consistency of the collected demonstrations.
Complete original PDF S4.T2
Original selectable table
Matched attempts Matched successes
Method Grasp Insert Grasp Insert
Teleop 7/20 1/20 6/20 0/20
Residual (Ours) 18/20 11/20 19/20 9/20

V Conclusion

A small-data nearest-neighbor surrogate can support simulation training of useful local teleoperation assistance on these three tasks. The strongest evidence combines human outcomes, surrogate comparisons and downstream policies. Scope remains limited by the small, homogeneous study population, task/experience confounding, unblinded instruction, conditional timing and missing uncertainty estimates. Users may co-adapt to assistance; modeling that adaptation or updating the surrogate online is future work.

Acknowledgment

This work was partially supported by the DARPA TIAMAT program (HR0011-24-9-0430), the Toyota Research Institute, NSF Award #2409661, Samsung Research America, and an Amazon Research Award (Fall 2024). This article solely reflects the opinions and conclusions of its authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.

Appendix A Additional Technical Details

A-A Hardware System

A-A 1 Robot Setup

The platform is a tabletop UFactory xArm 7 with seven joints, parallel gripper and wrist force–torque sensor. A front-mounted Intel RealSense D455 provides RGB-D observations. Figure 5 retains the complete camera, sensor and robot layout.

Complete original PDF A1.F5
Fig. 5: xArm 7 Setup.

A-A 2 Residual Implementation

GELLO supplies commands at 15 Hz. Direct teleoperation sends teacher-arm joint angles as robot joint targets. Assisted teleoperation converts them through forward kinematics to a Cartesian target, adds the residual, then uses inverse kinematics for robot joint targets. Both clip the joint-step ℓ₂ norm for velocity limits. The comparison therefore changes the command-mapping path as well as adding assistance; it is not a perfectly matched Cartesian controller with its residual simply disabled.

A-A 3 Admittance Control

The real robot enables built-in admittance on all six translational/rotational axes. Explicit damping is zero, relying on internal servo behavior for stability. Real/sim translational stiffness is 1,000/200 N/m, rotational stiffness 10/100 Nm/rad, translational virtual mass 0.1/0.125 kg and rotational inertia 0.001/0.015 kg·m². Table III preserves the remaining damping, axis and simulated joint-PID settings.

TABLE III: Controller parameters. Real-world and simulation settings for the task-space admittance and joint-space PID controllers.
Complete original PDF A1.T3
Original selectable table
Controller Parameter Real Sim Unit
Task-space admittance controller KxK_{x} 10001000 200200 N/m
KrK_{r} 1010 100100 Nm/rad
MxM_{x} 0.10.1 0.1250.125 kg
MrM_{r} 1×10−31\times 10^{-3} 0.0150.015 kg⋅\cdotm2
DxD_{x} 00 55 N⋅\cdots/m
DrD_{r} 00 1.21.2 N⋅\cdotm⋅\cdots/rad
adm. axes 66 66 DoF
Joint-space PID controller KpK_{p} – 200200 N⋅\cdotm/rad
KiK_{i} – 2020 N⋅\cdotm/(rad⋅\cdots)
KdK_{d} – 00 N⋅\cdotm⋅\cdots/rad

A-A 4 State Estimation

FoundationPose estimates object poses on a separate 15 Hz thread; the environment retrieves the latest available estimate asynchronously. Robot linear/angular velocities use finite differences followed by low-pass filtering in both domains. This reduces one representation mismatch but leaves pose/latency errors to calibration and randomization.

TABLE IV: Domain randomization (DMR). Per-episode sampling unless noted; base-action DMR is sampled per timestep. For base-action randomization, LL denotes the length of kNN-retrieved action chunks, εt\varepsilon_{t} is i.i.d. residual noise, and (β,pon)(\beta,p_{\mathrm{on}}) parameterize a smooth stochastic gate controlling the activation frequency and temporal correlation of the noise.
Complete original PDF A1.T4
Original selectable table
Group Symbol(s) Distribution / value
Ctrl. Kx,Kr,Mx,MrK_{x},K_{r},M_{x},M_{r} 𝒰⁡(0.95, 1.05)⊙(⋅)\mathcal{U}(0.95,\,1.05)\odot(\cdot)
Pose Δ​pfix,Δ​pheld\Delta p_{\mathrm{fix}},\Delta p_{\mathrm{held}} 𝒩⁡(0, 0.0022​I)\mathcal{N}(0,\,0.002^{2}I)
Init. Δ​p0,Δ​θ0\Delta p_{0},\Delta\theta_{0} 𝒩⁡(0, 0.022​I)\mathcal{N}(0,\,0.02^{2}I),  𝒩⁡(0, 0.0352​I)\mathcal{N}(0,\,0.035^{2}I)
Base LL L∼𝒰​{5,…,15}L\sim\mathcal{U}\{5,\dots,15\}
εt\varepsilon_{t} 𝒰⁡([−0.6,0.6]d)\mathcal{U}([-0.6,0.6]^{d})
β,pon\beta,p_{\mathrm{on}} β=0.8,pon=0.5\beta=0.8,\;p_{\mathrm{on}}=0.5

A-B Residual Policy Training

A-B 1 Representation Spaces

The base command has eight components: three position coordinates, four quaternion entries and gripper opening. The residual has seven: translation, axis–angle rotation and gripper increments. The 35-dimensional observation combines fingertip position/quaternion/gripper state, two object-relative positions, linear/angular velocities, the base command and previous residual.

A rotational residual becomes δq and left-multiplies the base orientation as δq⊗q. Other components add. Running mean/standard-deviation statistics update during training and freeze for inference. The source also says normalization maps values to [−1,1], but mean/standard-deviation normalization alone does not enforce that bound; clipping details are not specified.

A-B 2 Reward Terms

Table V combines residual-magnitude penalty (0.1), tilt (1.0), clipped contact force (0.2), an axis-alignment indicator (0.05), residual-change penalty (0.1), failure/timeout penalty (50) and first-success reward (30). Gear/peg success uses a relative-position tolerance below 2.5 mm; nut success additionally requires cumulative 180° tightening while maintaining the prescribed pose. This is shaped RL with a sparse success bonus, not a purely sparse objective.

TABLE V: Reward terms. aresa_{\text{res}} and aresp​r​e​va_{\text{res}}^{prev} denote current and previous normalized residual actions, θtilt\theta_{\text{tilt}} the end-effector tilt angle from upright, ϕ⁡(F)\phi(F) a clipped contact-force penalty, and px​yp_{xy} the held-object planar position. The success reward is issued once per episode upon first satisfying the task-specific success condition. The termination condition is dropping the held object early on.
Complete original PDF A1.T5
Original selectable table
Category Name Term Scale
ℛgeneral\mathcal{R}_{\text{general}} Regularization −∥ares∥-\lVert a_{\text{res}}\rVert 0.10.1
Tilt Penalty −θtilt-\theta_{\text{tilt}} 1.01.0
Force Penalty −ϕ⁡(F)-\phi(F) 0.20.2
Axis Align 𝟏[∥px​y−px​y∗∥<ϵ]\mathbf{1}[\lVert p_{xy}-p^{*}_{xy}\rVert<\epsilon] 0.050.05
Smoothness −∥aresp​r​e​v−ares∥-\lVert a_{\text{res}}^{prev}-a_{\text{res}}\rVert 0.1
Termination −𝟏​[fail / timeout]-\mathbf{1}[\text{fail / timeout}] 50.050.0
ℛsuccess\mathcal{R}_{\text{success}} Success 𝟏​[first success]\mathbf{1}[\text{first success}] 30.030.0

A-B 3 Domain Randomization

Controller stiffness/inertia is perturbed by ±5%; observed object positions receive 2 mm Gaussian noise, and initial pose perturbations use 2 cm translation and 0.035 rad rotation scales. Base-command randomization includes 5–15-step chunks, bounded uniform residual noise and the smooth gate. Table IV specifies per-episode versus per-step sampling. These ranges describe the evaluated robustness envelope, not arbitrary real-world changes.

TABLE VI: RL hyperparameters. Actor–critic architecture and PPO settings used for training.
Complete original PDF A1.T6
Original selectable table
Hyperparameter Value
Architecture
Backbone LSTM (2 layers, 1024 units)
MLP [512, 128, 64], ELU
Actor–critic shared trunk
Policy head Gaussian (learnable log⁡σ\log\sigma)
PPO / Optimization
γ\gamma / λ\lambda 0.995 / 0.95
LR schedule adaptive (base lr 10−410^{-4})
KL threshold 0.008
Clip ϵ\epsilon 0.2
Entropy coef. 0.0
Critic coef. 2.0
Grad clip 1.0
Update epochs 4
Rollout horizon / actors 128 / 128
Minibatch size 512
TABLE VII: Diffusion policy hyperparameters. State- and vision-based variants share the diffusion process and training setup, differing only in input modality, encoder, and training scale.
Complete original PDF A1.T7
Original selectable table
Category State DP Vision DP
Inputs / Outputs
Observation st∈ℝ20s_{t}\in\mathbb{R}^{20} RGB (200×200)(200{\times}200) + st∈ℝ8s_{t}\in\mathbb{R}^{8}
Action at∈ℝ8a_{t}\in\mathbb{R}^{8} at∈ℝ8a_{t}\in\mathbb{R}^{8}
Horizon (HH) 16 16
Action steps 8 8
Diffusion model
Noise scheduler DDPM DDPM
Train timesteps 100 100
Inference steps 10 10
β\beta schedule Squared cosine Squared cosine
Prediction target ϵ\epsilon (noise) ϵ\epsilon (noise)
FiLM modulation Enabled Enabled
Architecture
Encoder None ResNet-18
U-Net dims [512, 1024, 2048] [512, 1024, 2048]
Group norm groups 8 8
Optimization
Batch size 256 512
Optimizer Adam (lr=10−4\mathrm{lr}=10^{-4}) Adam (lr=10−4\mathrm{lr}=10^{-4})
LR schedule Cosine Cosine
Weight decay 10−610^{-6} 10−610^{-6}
Training steps 40k 10k
Complete original PDF A1.F6
Fig. 6: State-based DP training data distributions. Spatial coverage of 3D fingertip positions, downsampled to 100k points for visualization. The top row shows training data for BC Pilot, obtained by augmenting real-world teleoperation demonstrations; the bottom row shows training data for Expert Pilot, collected from successful rollouts of the residual copilot policies. Columns correspond to the three tasks: Gear Meshing, Nut Threading, and Peg Insertion. The top row (left to right) contains 442k, 786k, and 427k transitions, while the bottom row contains 220k, 758k, and 195k transitions.

A-B 4 RL Hyperparameters

The shared actor–critic trunk has two 1,024-unit LSTM layers followed by an ELU MLP of widths 512, 128 and 64; the Gaussian head learns log standard deviation. PPO uses γ=0.995, λ=0.95, adaptive learning rate starting at 10⁻⁴, KL threshold 0.008, clip 0.2, zero entropy coefficient, critic coefficient 2 and gradient clipping at 1. Four update epochs use 128-step rollouts from 128 actors and minibatches of 512. Observations, values and advantages are normalized.

A-C Simulation Setup

A-C 1 Admittance Control

Simulation uses the same virtual admittance model. Spring/damping wrench is subtracted from measured wrench, inverse virtual inertia gives acceleration, and discrete updates advance velocity and pose. Angular increments are converted to quaternion rotations. The source calls the displayed update forward Euler; if the assignments are executed sequentially as printed, position uses the updated velocity. Table III supplies the actual simulated PID parameters, despite prose referring more generally to PD control.

Equations
fsd=D​ξ˙+K​e.f_{\mathrm{sd}}=D\dot{\xi}+Ke.
ξ¨=M−1​(f−fsd),ξ˙←ξ˙+Δ​t​ξ¨,ξ←ξ+Δ​t​ξ˙.\ddot{\xi}=M^{-1}\!\left(f-f_{\mathrm{sd}}\right),\quad\dot{\xi}\leftarrow\dot{\xi}+\Delta t\,\ddot{\xi},\quad\xi\leftarrow\xi+\Delta t\,\dot{\xi}.

A-C 2 Task Progression Metric

Progression is 1−e/eₘₐₓ with error clipped to the interval, so values lie between zero and one. Gear/peg error is end-effector distance to a predefined success pose, normalized by 15 cm. Nut error is remaining yaw after insertion, with eₘₐₓ=90°. This continuous simulation score differs from the real binary success rule and the appendix’s 180° cumulative tightening condition; Table I’s decimals should not be relabeled success percentages.

A-C 3 Physics Engine

Isaac Lab/Factory supplies gear and peg environments, while the nut uses signed-distance-field collision geometry. PhysX runs the Temporal Gauss–Seidel solver with 192 position iterations and one velocity iteration, conservative friction settings, bounce threshold 0.2, large contact buffers and one partition. These are implementation choices intended to stabilize contact, rather than a proof of physical accuracy or determinism.

A-D Behavior Cloning Policy Training

A-D 1 State-based Diffusion Policy

The state-based Diffusion Policy sees a 20-dimensional state that omits base and previous residual actions. BC data are expanded to 2,000 episodes with trajectory-wide translation/rotation augmentation; the Expert Pilot uses 2,000 successful rollouts of the residual copilot. Figure 6 retains all six spatial distributions.

The figure gives BC transition counts of 442k/786k/427k and expert counts of 220k/758k/195k for gear/nut/peg, respectively, downsampled to 100k points for display. The main text’s approximate “200–400k” expert-transition range therefore does not cover the reported nut dataset. Both variants predict a 16-step action horizon and execute eight steps.

Complete original PDF A1.F7
Fig. 7: Vision-based DP training data distributions. Spatial coverage visualization of initial states in the training set used to train vision-based diffusion policies. Columns show episodes collected via direct vs. copilot-assisted teleoperation. Rows correspond to the two controlled  IV-C settings (same number of successful episodes and same number of teleoperation attempts), isolating how data quality and collection efficiency affect downstream policy learning.

A-D 2 Vision-based Diffusion Policy

The vision policy combines eight proprioceptive components with a 200×200 RGB image encoded by ResNet-18 and a spatial-softmax keypoint layer. Both policy types use a FiLM-conditioned U-Net with widths 512/1,024/2,048, eight groups, 100 DDPM training timesteps, ten inference steps, squared-cosine noise schedule and noise prediction.

Table VII specifies Adam at 10⁻⁴ with cosine learning-rate decay and 10⁻⁶ weight decay. Although prose calls training identical, the table gives state-policy batch size 256 and 40k updates versus vision-policy batch size 512 and 10k updates. The edition retains those concrete differences instead of repeating the broader equivalence claim.

A-E User Study Details

A-E 1 Participant Information

All 16 non-author participants are male students aged 20–30. Novices have fewer than 20 prior teleoperation trajectories, often none; experienced users have at least 50. This narrowly sampled population and different task assignments limit claims about operator demographics or experience-independent assistance.

A-E 2 Participant Questionnaire

All questionnaire items use a seven-point Likert scale from least to most of the named criterion. Lower User Demand and higher User Satisfaction are favorable. The workload items are described as NASA-TLX subscales, but this is an adapted seven-point questionnaire, not a reported conventional weighted 0–100 NASA-TLX aggregate.

A-E 3 Participant Instructions

The complete instruction sheet is preserved below. It matters to interpretation because participants are told what assistance does before evaluation, including when to delegate fine insertion and rotation. Responses and behavior should be understood in that informed, unblinded context.

Appendix B Participant Instruction Sheet

Participants are instructed to prioritize contact safety, task success and smooth input. Direct operation assigns them the full task; assisted operation asks them to focus on coarse motions while the copilot corrects alignment, completes nearby insertions and amplifies screw rotation. The sheet specifies five minutes of practice with each method, alternating conditions and a common teacher-arm starting mount.

Each participant performs only one task: ten trials per method for gear, eight for peg or five for nut. Success/failure definitions match the task description, and questionnaires follow all trials. The sheet describes two methods, while the gear results also include Residual BC; its additional allocation/order is not explained here. The exact original wording and layout remain available as a native PDF block plus selectable text.

Complete original PDF A2.instructions
Complete original participant instruction sheet (Appendix B, PDF page 17).
Original selectable instruction sheet

Method Overview. In this study, you will perform teleoperation using two different methods:

  1. Direct Teleoperation. The robot directly mirrors the motion of the teacher arm by matching its end-effector position and orientation. You have full control of the robot and are responsible for completing the entire task from start to finish.

  2. Copilot-Assisted Teleoperation. In this mode, a copilot agent assists your teleoperation. Your commands from the teacher arm are combined with corrective actions from the copilot before being executed by the robot. You share control with the copilot: you should focus on high-level, coarse motions, while the copilot corrects local position and orientation errors. For insertion tasks, the copilot will complete the insertion once the robot is near the insertion region. For screwing tasks, the copilot will amplify your rotations and regulate contact force.

Teleoperation Objective. Your teleoperation objective has three priorities:

  1. Safety and contact regulation. Always prioritize safe interaction. Noticeable shakiness during contact indicates excessive force and should be avoided.

  2. Task success. Try to successfully complete the task to the best of your ability.

  3. Input smoothness. Aim for smooth motions. Avoid sudden accelerations and excessively fast movements.

Task Descriptions. You will perform one of the following tasks:

  • Gear Meshing: Pick up the medium gear, insert it onto its shaft, and mesh it with the other gears. A trial is successful only if the gear remains grasped until insertion is fully completed.

  • Nut Threading: Pick up an M32 nut and tighten it onto the bolt by at least 180∘180^{\circ}. Due to hardware limits, this is performed as three consecutive 60∘60^{\circ} turns.

  • Peg Insertion: Pick up an 8​mm8\,\mathrm{mm}-long peg and insert it into the base. A trial is successful only if the peg reaches the bottom of the receptacle.

For all tasks, failure is defined as either dropping the object before success or exceeding the xArm safety force threshold.

Experiment Procedure. Before the experiment begins, you will have 5 minutes to familiarize yourself with each teleoperation method. During the experiment, you will alternate between direct teleoperation and copilot-assisted teleoperation. At the start of each trial, you will be told which method to use.

At the beginning of every trial, place the teacher arm on the mount to ensure a consistent initial pose. After the experimenter says “start,” you may move the arm off the mount and begin teleoperating. The experimenter will say “stop” when the trial ends due to either success or failure, and then the next trial will begin.

You will perform only one task in the study. The number of trials per method is:

  • Gear meshing: 10 trials per method.

  • Peg insertion: 8 trials per method.

  • Nut threading: 5 trials per method.

After completing all trials, you will fill out a questionnaire about your experience with each method, including workload and satisfaction.

Concept demonstration

Open image ↗