Overview
Appendix
Contents
At a glance
A residual copilot corrects human teleoperation commands on three contact-rich assembly tasks. Less than five minutes of real interaction per task supplies a nearest-neighbor human surrogate and simulator calibration; reinforcement learning trains assistance in simulation before deployment with real users. The 16-person study reports higher success, lower reported workload and better downstream imitation learning. Results support this particular assistance pipeline, with important limits in participant coverage, control mappings and evaluation statistics.
I Introduction
Fine alignment and force regulation are difficult through a teleoperation interface even when the operator knows the intended task. This work divides responsibility: humans supply coarse commands, while a learned residual supplies local corrections. It avoids requiring an autonomous expert policy as the main training prior, using a small real dataset to approximate the human command distribution in simulation.
The contribution is a practical real-to-sim-to-real pipeline, not a guarantee that small residuals preserve intent or that a simulated surrogate captures every human response. The task still requires object models/poses, simulator calibration and designed rewards. This edition covers all 17 pages of v2, dated 20 March 2026, including seven complete figures, seven tables, six displayed formula containers, 81 references and the complete participant instruction sheet.
II Related Works
II-A Shared Autonomy
Shared autonomy combines human control with machine assistance. Expert-prior methods guide commands toward a planner or autonomous policy; copilot methods instead learn assistance conditioned on the command and environment. The latter can avoid first solving full autonomy, but depend on a useful model of the human during training. The paper builds on deep-RL shared autonomy and residual policy learning for shared autonomy, replacing a learned behavioral surrogate with nearest-neighbor retrieval.
II-B Residual Policy Learning
A residual policy adds a correction to a nominal command, reducing the burden of learning an entire controller. Here the nominal command comes from the human. Penalizing correction magnitude and change encourages restrained assistance, but neither penalty proves preservation of latent intent or safety. Training uses dense regularization/alignment terms as well as sparse terminal rewards, despite the introduction’s emphasis on sparse reward.
II-C Real-to-Sim-to-Real Learning
Real-to-sim-to-real learning first calibrates a task simulator from real interaction, trains at scale in simulation, then deploys physically. Shared autonomy adds a second modeling problem: the simulated human must remain useful under the states induced by assistance. This work uses real teleoperation both for the surrogate and for control/contact calibration, followed by domain randomization.
III Method
III-A Problem Definition
The formulation is a partially observable decision process with environment state and a latent human goal. The copilot observes robot/object state and the human command; it does not explicitly infer the goal. A Gaussian policy outputs a normalized residual, scaled and composed with the human’s pose/gripper target. Translation and gripper corrections add, while rotation corrections compose as quaternions.
PPO maximizes a reward combining general penalties/shaping and task success. The main text calls the backbone an MLP, while the implementation table specifies a shared two-layer LSTM followed by an MLP; the appendix is the more detailed architecture description. The retained formulas define the process, command composition and retrieval metric without changing their notation.
Equations
III-B Human Surrogate Model
The human surrogate retrieves demonstrated commands with a weighted distance over end-effector translation, quaternion geodesic rotation and gripper opening. Metric weights are tuned by nearest-neighbor prediction error. At runtime, it samples among the nearest neighbors with a softmax over negative distance and emits a short consecutive chunk from the selected demonstration, reducing rapid switches between unrelated neighbors.
Chunks span 5–15 steps. Uniform local perturbations are multiplied by a gate updated as 0.8 times its previous value plus 0.2 times a Bernoulli(0.5) sample. The gate changes smoothly, but the perturbation vector is sampled independently each step; this is not a guarantee of temporally smooth noise directions. Retrieval stays on recorded actions before perturbation, whereas perturbed commands and visited states can leave empirical support.
The approach assumes broadly consistent human strategies with and without assistance. It avoids training a parametric behavior model on very little data, but the paper does not fully specify the neighbor count, temperature, metric weights or residual scale needed to reproduce every choice. Human adaptation remains a distribution-shift risk.
Equation 1
| (1) |
III-C Admittance Control
Task-space admittance lets the robot yield to external wrench while tracking a target. A virtual mass–damper–spring relationship links pose error, velocity, acceleration and measured force/torque. Compliance supports contact-rich assembly, but the experiment’s force cutoff and reported success are not a formal safety certification. Real and simulated controller parameters differ and are retained in Table III.
Equation
III-D Real-to-Sim-to-Real
Real teleoperation fits the surrogate and tunes arm/gripper PID, task-space stiffness/damping/inertia, friction and center of mass. Training randomizes controller parameters and perceived object poses to absorb residual simulation mismatch. “Zero-shot” deployment refers to transferring the trained residual without subsequent real-world RL fine-tuning; it does not mean no real data or hardware calibration.
IV Experiments
The experiment asks whether assistance helps humans, whether the nearest-neighbor surrogate generalizes better than behavioral cloning, and whether assisted demonstrations improve a later imitation policy. Tasks use the NIST assembly board with submillimeter radial clearance: mesh a medium gear while retaining the grasp, tighten an M32 nut by at least 180° using three 60° turns, or insert an 8 mm-long peg to the receptacle bottom. Dropping the object before completion or exceeding the robot’s force threshold is failure.
Twelve novice operators perform gear or peg tasks, while four experienced operators perform nut threading. Each participant performs one task; method use alternates after familiarization. Experience and task are therefore confounded, and the study cannot isolate an independent novice-versus-expert effect. Exact participant allocation between gear and peg is not given.
Baselines include a diffusion-policy surrogate trained from augmented small-data demonstrations, an “Expert Pilot” fitted to successful rollouts of the authors’ own residual policy, and delayed/noisy versions of that pilot. Guided-diffusion assistance uses either small-data or expert-rollout training; Residual BC uses the same RL setup with the behavioral-cloning surrogate. The expert is not an independently sourced human expert or oracle. Prior diffusion shared autonomy motivates guidance; the main text’s specific GD-baseline citation [74] instead resolves to DexterityGen in this version.
Original selectable table
| Copilot | Eval. Pilot | ||||||||||||||
| Laggy Pilot | Noisy Pilot | Expert Pilot | BC Pilot | kNN Pilot | |||||||||||
| Gear | Peg | Nut | Gear | Peg | Nut | Gear | Peg | Nut | Gear | Peg | Nut | Gear | Peg | Nut | |
| No Copilot | 0.97 | 0.94 | 0.49 | 0.92 | 0.91 | 0.24 | 0.99 | 0.99 | 0.61 | 0.84 | 0.71 | 0.03 | 0.87 | 0.85 | 0.16 |
| GD Expert | 0.96 | 0.89 | 0.24 | 0.95 | 0.91 | 0.25 | 0.94 | 0.90 | 0.28 | 0.94 | 0.89 | 0.34 | 0.95 | 0.90 | 0.27 |
| GD BC | 0.63 | 0.56 | 0.00 | 0.65 | 0.56 | 0.00 | 0.62 | 0.53 | 0.01 | 0.63 | 0.52 | 0.01 | 0.66 | 0.55 | 0.00 |
| Residual BC | 0.96 | 0.44 | 0.00 | 0.91 | 0.44 | 0.00 | 0.98 | 0.44 | 0.00 | 0.70 | 0.40 | 0.00 | 0.86 | 0.46 | 0.02 |
| Residual Copilot (Ours) | 0.99 | 0.92 | 0.65 | 0.96 | 0.92 | 0.69 | 0.98 | 0.93 | 0.74 | 0.97 | 0.83 | 0.21 | 1.00 | 0.99 | 0.81 |
IV-A Copilot Performance
Figure 3 reports gear success of 69% for direct teleoperation, 68% for Residual BC and 93% for the proposed copilot; nut success increases from 50% to 80%, and peg success from 72% to 80%. The nut gain is 30 percentage points. Qualitative sequences show corrections to alignment, rotation and grasp approach, but selected examples are not independent statistical evidence.
Completion-time curves include only successful episodes. Their shaded regions are standard deviations, not confidence intervals, and the dashed lines are per-method means. The plots favor assistance overall, but conditional timing can change when assistance also changes which trials succeed. No significance tests, participant-level uncertainty or independent-training variability are reported.
Operators report lower demand and higher satisfaction on seven-point scales, including perceived intent correctness. These are subjective, unblinded responses; they do not measure latent-intent preservation directly. Appendix instructions explicitly teach users to rely on the copilot for insertion and amplified rotation, so improved performance is evidence for the complete instructed system, not proof that human strategy remains unchanged.
IV-B Human Surrogate Analysis
The human gear comparison favors the nearest-neighbor surrogate over Residual BC. In simulation, Table I evaluates each copilot against five pilot types with 1,000 trials per cell. Entries are mean task progression, not binary success rates. With its own nearest-neighbor pilot, the method reaches 1.00/0.99/0.81 for gear/peg/nut, versus 0.87/0.85/0.16 without assistance.
Cross-pilot robustness is substantial but not universal. With the Expert Pilot, unaided gear/peg progression is 0.99/0.99 versus 0.98/0.93 with assistance; with the Laggy Pilot, peg progression is 0.94 unaided versus 0.92 assisted. Under the BC Pilot, GD Expert reaches 0.34 nut progression versus the proposed method’s 0.21. Its nut score also falls from 0.81 with its training pilot to 0.21 with BC. Thus the table does not establish uniformly beneficial assistance or negligible distribution shift.
Residual BC often stalls at low progress, while guided diffusion can pull even a competent pilot away from useful behavior. The claimed exploration/local-minimum explanation is plausible but not directly isolated by the final scores. All original cells and training/evaluation diagonal markings remain available.
IV-C Data Quality Comparison
Five gear-study participants supply demonstrations for two comparisons: equal teleoperation attempts, using all successes, and equal numbers of successful demonstrations. Figure 7 shows 36 direct versus 43 assisted successful episodes under matched attempts, and 34 versus 34 under matched successes. These episode counts are training data, distinct from the 20 downstream evaluation rollouts.
With matched attempts, the assisted-data policy grasps in 18/20 and inserts in 11/20 rollouts, versus 7/20 and 1/20 from direct data. With matched successes, it achieves 19/20 grasps and 9/20 insertions, versus 6/20 and 0/20. The equal-success comparison supports a benefit beyond simply collecting more successful trajectories.
The authors attribute this to more consistent contact timing and approach behavior. However, equal counts and model capacity do not make the datasets otherwise identical: Figure 7 also shows different initial-state coverage. No confidence intervals or independent training seeds are reported. The results demonstrate better downstream performance in this collection setup, rather than isolating trajectory consistency as the sole cause.
Original selectable table
| Matched attempts | Matched successes | |||
| Method | Grasp | Insert | Grasp | Insert |
| Teleop | 7/20 | 1/20 | 6/20 | 0/20 |
| Residual (Ours) | 18/20 | 11/20 | 19/20 | 9/20 |
V Conclusion
A small-data nearest-neighbor surrogate can support simulation training of useful local teleoperation assistance on these three tasks. The strongest evidence combines human outcomes, surrogate comparisons and downstream policies. Scope remains limited by the small, homogeneous study population, task/experience confounding, unblinded instruction, conditional timing and missing uncertainty estimates. Users may co-adapt to assistance; modeling that adaptation or updating the surrogate online is future work.
Acknowledgment
This work was partially supported by the DARPA TIAMAT program (HR0011-24-9-0430), the Toyota Research Institute, NSF Award #2409661, Samsung Research America, and an Amazon Research Award (Fall 2024). This article solely reflects the opinions and conclusions of its authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
Appendix A Additional Technical Details
A-A Hardware System
A-A 1 Robot Setup
The platform is a tabletop UFactory xArm 7 with seven joints, parallel gripper and wrist force–torque sensor. A front-mounted Intel RealSense D455 provides RGB-D observations. Figure 5 retains the complete camera, sensor and robot layout.
A-A 2 Residual Implementation
GELLO supplies commands at 15 Hz. Direct teleoperation sends teacher-arm joint angles as robot joint targets. Assisted teleoperation converts them through forward kinematics to a Cartesian target, adds the residual, then uses inverse kinematics for robot joint targets. Both clip the joint-step ℓ₂ norm for velocity limits. The comparison therefore changes the command-mapping path as well as adding assistance; it is not a perfectly matched Cartesian controller with its residual simply disabled.
A-A 3 Admittance Control
The real robot enables built-in admittance on all six translational/rotational axes. Explicit damping is zero, relying on internal servo behavior for stability. Real/sim translational stiffness is 1,000/200 N/m, rotational stiffness 10/100 Nm/rad, translational virtual mass 0.1/0.125 kg and rotational inertia 0.001/0.015 kg·m². Table III preserves the remaining damping, axis and simulated joint-PID settings.
A-A 4 State Estimation
FoundationPose estimates object poses on a separate 15 Hz thread; the environment retrieves the latest available estimate asynchronously. Robot linear/angular velocities use finite differences followed by low-pass filtering in both domains. This reduces one representation mismatch but leaves pose/latency errors to calibration and randomization.
Original selectable table
| Group | Symbol(s) | Distribution / value |
| Ctrl. | ||
| Pose | ||
| Init. | , | |
| Base | ||
A-B Residual Policy Training
A-B 1 Representation Spaces
The base command has eight components: three position coordinates, four quaternion entries and gripper opening. The residual has seven: translation, axis–angle rotation and gripper increments. The 35-dimensional observation combines fingertip position/quaternion/gripper state, two object-relative positions, linear/angular velocities, the base command and previous residual.
A rotational residual becomes δq and left-multiplies the base orientation as δq⊗q. Other components add. Running mean/standard-deviation statistics update during training and freeze for inference. The source also says normalization maps values to [−1,1], but mean/standard-deviation normalization alone does not enforce that bound; clipping details are not specified.
A-B 2 Reward Terms
Table V combines residual-magnitude penalty (0.1), tilt (1.0), clipped contact force (0.2), an axis-alignment indicator (0.05), residual-change penalty (0.1), failure/timeout penalty (50) and first-success reward (30). Gear/peg success uses a relative-position tolerance below 2.5 mm; nut success additionally requires cumulative 180° tightening while maintaining the prescribed pose. This is shaped RL with a sparse success bonus, not a purely sparse objective.
Original selectable table
| Category | Name | Term | Scale |
| Regularization | |||
| Tilt Penalty | |||
| Force Penalty | |||
| Axis Align | |||
| Smoothness | 0.1 | ||
| Termination | |||
| Success |
A-B 3 Domain Randomization
Controller stiffness/inertia is perturbed by ±5%; observed object positions receive 2 mm Gaussian noise, and initial pose perturbations use 2 cm translation and 0.035 rad rotation scales. Base-command randomization includes 5–15-step chunks, bounded uniform residual noise and the smooth gate. Table IV specifies per-episode versus per-step sampling. These ranges describe the evaluated robustness envelope, not arbitrary real-world changes.
Original selectable table
| Hyperparameter | Value |
| Architecture | |
| Backbone | LSTM (2 layers, 1024 units) |
| MLP | [512, 128, 64], ELU |
| Actor–critic | shared trunk |
| Policy head | Gaussian (learnable ) |
| PPO / Optimization | |
| / | 0.995 / 0.95 |
| LR schedule | adaptive (base lr ) |
| KL threshold | 0.008 |
| Clip | 0.2 |
| Entropy coef. | 0.0 |
| Critic coef. | 2.0 |
| Grad clip | 1.0 |
| Update epochs | 4 |
| Rollout horizon / actors | 128 / 128 |
| Minibatch size | 512 |
Original selectable table
| Category | State DP | Vision DP |
| Inputs / Outputs | ||
| Observation | RGB + | |
| Action | ||
| Horizon () | 16 | 16 |
| Action steps | 8 | 8 |
| Diffusion model | ||
| Noise scheduler | DDPM | DDPM |
| Train timesteps | 100 | 100 |
| Inference steps | 10 | 10 |
| schedule | Squared cosine | Squared cosine |
| Prediction target | (noise) | (noise) |
| FiLM modulation | Enabled | Enabled |
| Architecture | ||
| Encoder | None | ResNet-18 |
| U-Net dims | [512, 1024, 2048] | [512, 1024, 2048] |
| Group norm groups | 8 | 8 |
| Optimization | ||
| Batch size | 256 | 512 |
| Optimizer | Adam () | Adam () |
| LR schedule | Cosine | Cosine |
| Weight decay | ||
| Training steps | 40k | 10k |
A-B 4 RL Hyperparameters
The shared actor–critic trunk has two 1,024-unit LSTM layers followed by an ELU MLP of widths 512, 128 and 64; the Gaussian head learns log standard deviation. PPO uses γ=0.995, λ=0.95, adaptive learning rate starting at 10⁻⁴, KL threshold 0.008, clip 0.2, zero entropy coefficient, critic coefficient 2 and gradient clipping at 1. Four update epochs use 128-step rollouts from 128 actors and minibatches of 512. Observations, values and advantages are normalized.
A-C Simulation Setup
A-C 1 Admittance Control
Simulation uses the same virtual admittance model. Spring/damping wrench is subtracted from measured wrench, inverse virtual inertia gives acceleration, and discrete updates advance velocity and pose. Angular increments are converted to quaternion rotations. The source calls the displayed update forward Euler; if the assignments are executed sequentially as printed, position uses the updated velocity. Table III supplies the actual simulated PID parameters, despite prose referring more generally to PD control.
Equations
A-C 2 Task Progression Metric
Progression is 1−e/eₘₐₓ with error clipped to the interval, so values lie between zero and one. Gear/peg error is end-effector distance to a predefined success pose, normalized by 15 cm. Nut error is remaining yaw after insertion, with eₘₐₓ=90°. This continuous simulation score differs from the real binary success rule and the appendix’s 180° cumulative tightening condition; Table I’s decimals should not be relabeled success percentages.
A-C 3 Physics Engine
Isaac Lab/Factory supplies gear and peg environments, while the nut uses signed-distance-field collision geometry. PhysX runs the Temporal Gauss–Seidel solver with 192 position iterations and one velocity iteration, conservative friction settings, bounce threshold 0.2, large contact buffers and one partition. These are implementation choices intended to stabilize contact, rather than a proof of physical accuracy or determinism.
A-D Behavior Cloning Policy Training
A-D 1 State-based Diffusion Policy
The state-based Diffusion Policy sees a 20-dimensional state that omits base and previous residual actions. BC data are expanded to 2,000 episodes with trajectory-wide translation/rotation augmentation; the Expert Pilot uses 2,000 successful rollouts of the residual copilot. Figure 6 retains all six spatial distributions.
The figure gives BC transition counts of 442k/786k/427k and expert counts of 220k/758k/195k for gear/nut/peg, respectively, downsampled to 100k points for display. The main text’s approximate “200–400k” expert-transition range therefore does not cover the reported nut dataset. Both variants predict a 16-step action horizon and execute eight steps.
A-D 2 Vision-based Diffusion Policy
The vision policy combines eight proprioceptive components with a 200×200 RGB image encoded by ResNet-18 and a spatial-softmax keypoint layer. Both policy types use a FiLM-conditioned U-Net with widths 512/1,024/2,048, eight groups, 100 DDPM training timesteps, ten inference steps, squared-cosine noise schedule and noise prediction.
Table VII specifies Adam at 10⁻⁴ with cosine learning-rate decay and 10⁻⁶ weight decay. Although prose calls training identical, the table gives state-policy batch size 256 and 40k updates versus vision-policy batch size 512 and 10k updates. The edition retains those concrete differences instead of repeating the broader equivalence claim.
A-E User Study Details
A-E 1 Participant Information
All 16 non-author participants are male students aged 20–30. Novices have fewer than 20 prior teleoperation trajectories, often none; experienced users have at least 50. This narrowly sampled population and different task assignments limit claims about operator demographics or experience-independent assistance.
A-E 2 Participant Questionnaire
All questionnaire items use a seven-point Likert scale from least to most of the named criterion. Lower User Demand and higher User Satisfaction are favorable. The workload items are described as NASA-TLX subscales, but this is an adapted seven-point questionnaire, not a reported conventional weighted 0–100 NASA-TLX aggregate.
A-E 3 Participant Instructions
The complete instruction sheet is preserved below. It matters to interpretation because participants are told what assistance does before evaluation, including when to delegate fine insertion and rotation. Responses and behavior should be understood in that informed, unblinded context.
Appendix B Participant Instruction Sheet
Participants are instructed to prioritize contact safety, task success and smooth input. Direct operation assigns them the full task; assisted operation asks them to focus on coarse motions while the copilot corrects alignment, completes nearby insertions and amplifies screw rotation. The sheet specifies five minutes of practice with each method, alternating conditions and a common teacher-arm starting mount.
Each participant performs only one task: ten trials per method for gear, eight for peg or five for nut. Success/failure definitions match the task description, and questionnaires follow all trials. The sheet describes two methods, while the gear results also include Residual BC; its additional allocation/order is not explained here. The exact original wording and layout remain available as a native PDF block plus selectable text.
Original selectable instruction sheet
Method Overview. In this study, you will perform teleoperation using two different methods:
-
Direct Teleoperation. The robot directly mirrors the motion of the teacher arm by matching its end-effector position and orientation. You have full control of the robot and are responsible for completing the entire task from start to finish.
-
Copilot-Assisted Teleoperation. In this mode, a copilot agent assists your teleoperation. Your commands from the teacher arm are combined with corrective actions from the copilot before being executed by the robot. You share control with the copilot: you should focus on high-level, coarse motions, while the copilot corrects local position and orientation errors. For insertion tasks, the copilot will complete the insertion once the robot is near the insertion region. For screwing tasks, the copilot will amplify your rotations and regulate contact force.
Teleoperation Objective. Your teleoperation objective has three priorities:
-
Safety and contact regulation. Always prioritize safe interaction. Noticeable shakiness during contact indicates excessive force and should be avoided.
-
Task success. Try to successfully complete the task to the best of your ability.
-
Input smoothness. Aim for smooth motions. Avoid sudden accelerations and excessively fast movements.
Task Descriptions. You will perform one of the following tasks:
-
Gear Meshing: Pick up the medium gear, insert it onto its shaft, and mesh it with the other gears. A trial is successful only if the gear remains grasped until insertion is fully completed.
-
Nut Threading: Pick up an M32 nut and tighten it onto the bolt by at least . Due to hardware limits, this is performed as three consecutive turns.
-
Peg Insertion: Pick up an -long peg and insert it into the base. A trial is successful only if the peg reaches the bottom of the receptacle.
For all tasks, failure is defined as either dropping the object before success or exceeding the xArm safety force threshold.
Experiment Procedure. Before the experiment begins, you will have 5 minutes to familiarize yourself with each teleoperation method. During the experiment, you will alternate between direct teleoperation and copilot-assisted teleoperation. At the start of each trial, you will be told which method to use.
At the beginning of every trial, place the teacher arm on the mount to ensure a consistent initial pose. After the experimenter says “start,” you may move the arm off the mount and begin teleoperating. The experimenter will say “stop” when the trial ends due to either success or failure, and then the next trial will begin.
You will perform only one task in the study. The number of trials per method is:
-
Gear meshing: 10 trials per method.
-
Peg insertion: 8 trials per method.
-
Nut threading: 5 trials per method.
After completing all trials, you will fill out a questionnaire about your experience with each method, including workload and satisfaction.