Literature
Display
← All papers

SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation

Arpit Bahety · Arnav Balaji · Ben Abbatematteo · Roberto Martín-Martín

· Robotics: Science and Systems 2025 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

SafeMimic turns a single third-person RGB-D demonstration into semantic task segments and relative robot motions. It adapts these motions through physical exploration, screening actions with simulation-trained safety classifiers and backtracking when progress fails. Alternative grasps address differences between human and robot morphology; successful attempts train a memory policy.

On seven mobile-manipulation tasks, final success ranges from 40% to 100% over five trials per task. Unsafe actions account for 0.6% of executed actions, versus 14.2% for exploration without the safety filters. The trials are human-monitored: the method reduces measured risk but does not establish risk-free operation.

I Introduction

Copying a human trajectory rarely suffices for a robot with different geometry, reachability and grasping capabilities. Learning from one video therefore requires both understanding the task’s semantic changes and adapting the low-level motions. Real-world trial and error also needs failure prediction and a way to undo unsuccessful attempts.

SafeMimic combines motion tracking and vision-language interpretation with an ensemble of task-agnostic safety Q-functions. These classifiers predict generic failures rather than task reward. The central empirical premise is that such failure predictors can transfer from simulation more readily than an entire manipulation policy. Backtracking and grasp changes extend adaptation beyond small trajectory perturbations; a memory policy reduces repeated exploration.

Complete original Figure 1.
Fig. 1: Robot imitating a single video of a human-demonstrated mobile manipulation task safely and autonomously with SafeMimic. From a video of a multi-step mobile manipulation task (top), SafeMimic extracts an initial motion strategy and adapts it to its own embodiment by exploring new actions in a safe manner. It combines an ensemble of safety Q-functions that predict future unsafe motions (red arrows, motion will collide) and a backtracking mechanism (orange arrows) that enables autonomous exploration until a successful action sequence is found (green arrows).

II Related Work

Prior human-to-robot imitation and ScrewMimic demonstrate action transfer and adaptation, but morphology differences, long task sequences and unsafe exploration remain difficult. Large video corpora offer another route, with substantial data requirements. Autonomous-reset methods address repeated trials without necessarily preventing unsafe actions.

Safe imitation, safety-critic reinforcement learning, shielding and deployment-time failure prediction motivate SafeMimic’s filters. Analytic constraints and collision planners can miss contact-force, grasp-loss and joint-limit failures. Here separate classifiers are pretrained on task-agnostic simulated transitions and applied during adaptation from one human demonstration; they do not provide a formal safety certificate.

Complete original Figure 2.
Fig. 2: Overview of SafeMimic. From an RGB-D video of a human performing a multi-step mobile manipulation task acquired by the robot, SafeMimic uses a combination of human pose tracking models [4, 5] and VLM prompting to perform coarse-to-fine segmentation obtaining semantic changes –“what?”– and human action trajectories –“how?”–, and translating them to the robot’s point of view (left, Sec. III-A). SafeMimic then refines and adapts each task segment safely by sampling and verifying actions before executing them thanks to an ensemble of safety Q-functions pretrained in simulation, QsafeQ_{\textrm{safe}} (top right, Sec. III-B). If forward progress is not possible, SafeMimic autonomously backtracks and tries different actions (orange arrows). When required to overcome large differences in morphology, SafeMimic explores alternative grasp modes (second row of samples), adapting the grasp to enable successful execution. Successful attempts are detected by a VLM that verifies when the parsed semantic change of the segment is achieved. Successes are stored and used to train a policy memory module with geometric augmentations (bottom right, Sec. III-C) that predicts actions (grasps or action trajectories) in subsequent attempts, given a pointcloud and language task description, in order to reduce the need for exploration.

III SafeMimic Framework

The framework first decomposes a demonstration, then explores robot-compatible motions for each semantic goal. A final memory stage learns from successful real executions. Figure 2 preserves the complete information flow, including grasp changes and backtracking.

III-A Factorizing, Parsing and Translating Human Videos

Body/hand tracking, a contact detector and a vision-language model identify what changed and how the person moved. Hip translation separates navigation from stationary manipulation. Navigation receives an object/location goal; manipulation is split further at hand–object contact changes, so each segment has a relatively simple semantic completion condition.

Navigation becomes relative base-pose changes. For manipulation, hand poses are expressed relative to the human body and then converted to successive motion deltas. The robot initially selects the available generated grasp closest to the demonstrated grasp. This is a starting proposal, not a guarantee of morphological compatibility.

The procedure assumes the robot starts near the person’s initial pose. The limitations section explains that this correspondence is obtained by estimating that pose in the map and navigating the robot there. “No calibration needed” therefore should not be read as absence of initial spatial alignment.

III-B Safe and Autonomous Real-World Adaptation

Each segment is treated as a Markov decision process with sparse semantic-goal reward. Failure indicators cover several unsafe conditions. The displayed constrained objective asks for zero expected unsafe-state occupancy, but the implemented learned predictors only approximate this aim.

The stated safety Q-function combines the current failure indicator with its expected value after one transition. Separate classifiers represent individual failure modes; their maximum scores an action. This maximum is not generally the probability of the union of failures, and the expression is not a discounted long-horizon risk value.

Training uses domain-randomized OmniGibson scenes, random or noisy motion-planner actions, point clouds and robot proprioception. Articulated interactions, pick-and-place and navigation supply labeled transitions without requiring a successful task policy in simulation.

During execution, continuous candidates are sampled around the demonstrated motion with Gaussian perturbations; grasping uses three discrete grasp candidates. Candidates are filtered by predicted failure and ordered by their worst classifier score. A vision-language query tests the semantic goal after actions. The threshold is a classifier setting, not a calibrated upper bound on real failure probability.

When no candidate passes, the robot executes the inverse of its previous action. After 50 unproductive action attempts, it backtracks to a grasp segment and tries another grasp mode. This can reverse a door motion or reposition a held object, but cannot undo every physical event. Algorithm 1 retains the recursive exploration and ordering procedure in full.

Equations
max⁡∑t=0Tπ⁡𝔼(st,at)∼ρπ​[R⁡(st,at)]​s.t.​𝔼st∼ρπ​[ℐ⁡(st)]=0,\max_{\pi}\sum_{t=0}^{T}\mathbb{E}_{\left(s_{t},a_{t}\right)\sim\rho_{\pi}}\left[R\left(s_{t},a_{t}\right)\right]\ \text{s.t.}\ \mathbb{E}_{s_{t}\sim\rho_{\pi}}\left[\mathcal{I}\left(s_{t}\right)\right]=0,
Qsafe(st,at)=ℐ(st)+(1−ℐ(st))[𝔼st′∼T(⋅∣st,at)ℐ(st′)]Q_{\mathrm{safe}}\left(s_{t},a_{t}\right)=\mathcal{I}\left(s_{t}\right)+\left(1-\mathcal{I}\left(s_{t}\right)\right)\left[\mathbb{E}_{s_{t^{\prime}}\sim T\left(\cdot\mid s_{t},a_{t}\right)}\mathcal{I}\left(s_{t^{\prime}}\right)\right]
Complete original Figure 3.
Fig. 3: Multi-Step Mobile Manipulation Evaluation Tasks. SafeMimic is evaluated on seven complex multi-step tasks combining navigation and manipulation. From left to right: boxing an item, shelving an item, store_in_drawer, erase_whiteboard, refrigerating an item, fill_pot, load_oven. The tasks involve risky contact-rich phases, grasping steps and the manipulation of constrained mechanisms. SafeMimic is able to adapt actions obtained from one single human demonstration safely and autonomously.

III-C Learning from Previous Successful Exploration

Successful grasp poses and post-grasp action trajectories train an action predictor from point clouds and a language description of the task. A PointNet visual encoder, SentenceTransformer text embedding and MLP head are trained with translation/rotation augmentations.

For later attempts, the memory biases exploration toward robot-successful actions instead of always returning to the human-derived proposal. It supports grasp prediction and, for drawer opening, full trajectory prediction. Segments without a trained memory continue to use the parsed human actions.

IV Experimental Evaluation

Seven tasks combine navigation, grasping and manipulation: boxing, shelving, storing in a drawer, erasing a whiteboard, refrigerating an item, filling a pot and loading an oven. They expose top-down versus side-grasp constraints, heavy articulated doors, limited reach and force-sensitive contact.

Experiments use a Tiago++ mobile manipulator, end-effector inverse-kinematics control, base translation/yaw, a head-mounted Astra S RGB-D camera and wrist force/torque sensing. The enumerated failure types are arm collision, base collision, joint-limit violation, excessive force/torque, grasp loss and dropping objects.

Baselines directly execute parsed motions with or without safety filters, or explore without filters. Two behavior-cloning policies use either all safe simulated actions or successful simulated episodes. These imitation baselines receive SafeMimic’s successful navigation and picking segments and are evaluated only on place/open/close skills, so their setup is not an otherwise identical end-to-end comparison.

Complete original Figure 4.
Fig. 4: Accumulated Success on Multi-Step Tasks. Accumulated success rate at each stage of each of the seven evaluated multi-step mobile manipulation tasks, indicating the percentage of the five trials each method completed up to and including that segment. We compare SafeMimic’s task performance to five baselines: direct execution without safety Q-functions (SQFs), which requires human supervision, direct execution with SQFs, exploration without SQF, Imitation Learning (IL) with all safe actions in simulation and IL only with the successful episodes. Note that the IL baselines were evaluated only on “Place”, “Open”, and “Close” segments and were provided successful solutions for navigation and pick segments. Note as well that some lines overlap at the same rate. Across all tasks, SafeMimic significantly outperforms all baselines and achieves up to 100% success in exploratory adaptation, indicating a superior ability to safely and autonomously refine the demonstrated behaviors.

Experiments and Results

Each task has five attempts. Figure 4 reports accumulated completion through successive segments, rather than independent success for each segment. SafeMimic’s final rates are 100% for boxing, 80% for shelving, drawer storage and whiteboard erasing, 60% for refrigerating and filling a pot, and 40% for loading an oven. Both direct-execution baselines finish none of the seven tasks. Adaptation averages five minutes per navigation segment and fifteen per manipulation segment.

Grasp-mode changes are needed in six tasks; whiteboard erasing is the exception. A human-like pot grasp collides with the sink edge, whereas an explored top-down grasp succeeds. A drawer grasp must change to avoid joint limits. False-positive safety predictions sometimes reduce intermediate success for filtered direct execution.

A human flags unsafe executed actions across all trials. The unsafe-action rates are 13.4% for direct execution without filters, 14.2% for exploration without filters, 10.8% and 9.5% for the two imitation policies, 0.5% for filtered direct execution, and 0.6% for SafeMimic. The change from 14.2% to 0.6% is 13.6 percentage points, not a 13.6% relative reduction or elimination of unsafe events.

Memory evaluation randomizes object poses and encounters varying camera viewpoints from navigation. After initial successful trials, three evaluation trials per task show fewer exploratory actions, with reductions up to 67% in Figure 7. The experiment trains selected pick/open grasps and a drawer-opening trajectory; it does not train a complete reusable policy for every segment.

All three human shelving demonstrations are parsed successfully, and each requires grasp adaptation. One demonstrator in three environments yields task success of 100%, 100% and 66%, with three trials per environment. These small tests support transfer across the sampled people and scenes, rather than a broad robustness guarantee.

Complete original Figure 5.
Fig. 5: Grasping mode adaptation. Two examples (top and bottom rows) of SafeMimic’s grasping mode adaptation. Left column: human demonstrated grasp. Middle column: robot failing when attempting the task by matching the human grasp. Right column: robot succeeding in the task through SafeMimic’s grasp adaptation. In 6 out of 7 tasks, SafeMimic’s grasp adaptation is critical to overcome human-robot embodiment differences and successfully imitate the demonstration.
Complete original Figure 6.
Fig. 6: Safe exploration with Safety Q-function predictions. Examples of predictions of the Safety Q-function (SQF) for two tasks: the opening drawer segment in store_in_drawer (top-row) and placing a bottle in the fridge segment in refrigerating (bottom-row). Red arrows indicate predicted unsafe actions (Qsafe>ϵQ_{\mathrm{safe}}>\epsilon) and green arrows show predicted safe actions. We observe that the simulation-trained SQFs reliably predict unsafe actions such as trying to open the drawer by moving in the wrong direction or trying to place an object in the fridge by colliding with the orange bottle. This enables SafeMimic to safely explore and adapt demonstrations in the real world.
Complete original Figure 7.
Fig. 7: Exploration Reduction with Policy Memory. Number of actions explored by SafeMimic with (right) and without policy memory (left). Successful attempts from an initial exploration are recorded and use to train the policy memory. Object poses are varied relative to the robot before re-evaluating SafeMimic with the learned policy memory on several task segments. The number of exploratory actions significantly reduces in all tasks, up to 67%, demonstrating that learning from prior successes is critical to increase efficiency in mobile manipulation tasks.

V Limitations and Future Work

Simulation must cover conditions resembling deployment, and constructing those scenarios still requires engineering. The classifiers address a predefined subset of failure types; unseen hazards and task failures need additional modeling. Initial human–robot pose correspondence and a separate robot grasp generator are also required.

Backtracking assumes sufficient reversibility and cannot recover a dropped object merely by reversing the last action. Autonomous resetting and learning from observed real-world failures remain future work. The low measured unsafe-action rate is encouraging, but is neither zero nor evidence that monitoring can safely be removed in arbitrary environments.

VI Conclusion

SafeMimic demonstrates a practical combination of semantic video parsing, learned failure screening, trajectory/grasp adaptation and memory. In the tested settings, simulation-trained safety predictors support more successful adaptation and fewer unsafe actions than the compared policies.

Its contribution is an empirical method for reducing exploration risk while transferring one demonstration. Finite human-monitored trials, enumerated failure classes and reversible-reset assumptions delimit the stronger language about safe autonomous learning.

Acknowledgements

The work was carried out in UT Austin’s Robot Interactive Intelligence Lab, supported by the College of Natural Sciences Catalyst Grant CAT-24-MartinMartin.

-A Model Details

The appendix describes a PointNet++ safety encoder, four-layer MLPs for actions and proprioception, and an MLP classification head. End-effector pose and force/torque enter the proprioceptive branch; force/torque is discretized into six bins to reduce the simulation gap. Binary safety labels train the classifiers with cross-entropy.

Table I preserves all settings: threshold 0.7, exploration standard deviation 0.05, batch size 16 and learning rate 10⁻³. It lists 9,000 SQF examples, while Figure 10 reports 43,200 collected state–action pairs; the filtering/split relationship is not explained. The action-delta range is printed as [0.0, 0.15], without clarifying its relation to negative reverse actions.

Algorithm 1 is retained as source pseudocode, including its Require line and 50 numbered steps. Its printed control flow does not spell out every initialization or exhausted-grasp termination case; the edition preserves these details rather than silently inventing an executable implementation.

TABLE I: Hyperparameters for safe exploration, SQF training, baselines, and video parsing.
Hyperparameter Q-function
SQF threshold ϵ\epsilon 0.7
Exploration Standard Deviation σ\sigma 0.05
Grasp Mode Clustering? K-Mediods
Batch Size 16
Optimizer Adam)
Learning Rate 1e-3
SQF Dataset Size 9000
IL (all safe actions) Dataset Size 23471
IL (successful episodes) Dataset Size 4305
Nav/Stationary Pos Threshold 0.07
Nav/Stationary Ori Threshold 0.2
Algorithm 1 SafeMimic Execution Pseudocode
Require: Segment videos V1:NV_{1:N}, video parser fparsef_{\mathrm{parse}}, standard deviation σ\sigma, num action samples mm, safety Q-functions Qsafe1:KQ_{\mathrm{safe}}^{1:K}, policy memory πmem\pi_{\mathrm{mem}}
1: for segment i=1​…​Ni=1...N do
2:     τh​u​m​a​n,\tau_{human}, segment_type, fsuccess←fp​a​r​s​e​(Vi)f_{\mathrm{success}}\leftarrow\ f_{parse}(V_{i})
3:     if segment_type is grasping then
4:         GG ←\leftarrow sample_grasp_modes()
5:         if πmem\pi_{\mathrm{mem}} exists for this segment then
6:             grobot←πmem​(st)g_{\mathrm{robot}}\leftarrow\pi_{\mathrm{mem}}(s_{t})
7:         else
8:             ghuman←τhuman​[−1]g_{\mathrm{human}}\leftarrow\tau_{\mathrm{human}}\left[-1\right]
9:             grobot←g_{\mathrm{robot}}\leftarrow closest(G,ghumanG,g_{\mathrm{human}})
10:         end if
11:         execute(grobotg_{\mathrm{robot}})
12:         GG.pop(grobotg_{\mathrm{robot}})
13:     else⊳\triangleright Segment is nav or manipulation
14:         if πmem\pi_{\mathrm{mem}} exists for this segment then
15:             Sample τ1:m∼πmem(st)\tau_{1:m}\sim\pi_{\mathrm{mem}}(s_{t})
16:         else
17:             Sample τ1:m∼𝒩(τhuman,σ)\tau_{1:m}\sim\mathcal{N}(\tau_{\mathrm{human}},\sigma)
18:         end if
19:         segment_success ←\leftarrow False
20:         while True do
21:             segment_success ←\leftarrow safe_explore(τ1:m\tau_{1:m})
22:             if segment_success then
23:                 break⊳\triangleright Go to next segment
24:             end if
25:             if GG is not empty then
26:                 gr​o​b​o​t∼Gg_{robot}\sim G ⊳\triangleright Explore new grasp
27:                 execute(gr​o​b​o​tg_{robot})
28:                 GG.pop(gr​o​b​o​tg_{robot})
29:             end if
30:         end while
31:     end if
32:     if not segment_success then return False
33:     end if
34: end for
35: return segment_success
36: procedure safe_explore(τ1:m\tau_{1:m})
37:     if len(τ1:m\tau_{1:m}) ==0==0 then ⊳\triangleright End of trajs
38:         return fsuccess​()f_{\mathrm{success}}()
39:     end if
40:     Filter τ1:m\tau_{1:m} if first action Qsafek​(st,a0)<ϵ​∀kQ_{\mathrm{safe}}^{k}(s_{t},a_{0})<\epsilon\ \forall\ k
41:     Sort τ1:m\tau_{1:m} by first action maxkQsafe1:K(st,a0)\max_{k}Q_{\mathrm{safe}}^{1:K}(s_{t},a_{0}) ascending
42:     for each a0a_{0} in τ1:m\tau_{1:m} do
43:         execute(a0a_{0})
44:         success ←\leftarrow safe_explore(next(τ1:m\tau_{1:m}))
45:         if success then return True
46:         end if
47:         execute(−a0-a_{0}) ⊳\triangleright Backtrack one step
48:     end for
49:     return False
50: end procedure

-B Human Video Parsing

The detailed parser first thresholds hip motion, then queries GPT-4o at three frames per second to identify contact changes. A visual contact detector refines transition timing around those coarse detections. Semantic labeling chooses from the eight action forms in Table II, spanning navigation, grasping, picking, placing, opening, closing and wiping.

A backward correction pass receives all ordered labels and frames from the video. It resolves inconsistent object references, such as changing “navigate to sink” into “navigate to drawer” when the subsequent interaction opens the adjacent drawer. The resulting motion segments and semantic subgoals initialize robot exploration.

Figures 8–12 retain the parser diagram, different people/environments, simulated failure-training domains, a backtracking example and robustness-test snapshots. The demonstrations and images illustrate the method’s components; their captions’ broad robustness wording should be read alongside the limited quantitative protocols.

Complete original Figure 8.
Fig. 8: Human video parsing by SafeMimic. An initial RGB-D video demonstration is processed by SafeMimic using a body tracking solution to obtain segments where the human is either navigating or performing stationary manipulation. Initial stationary manipulation segments are further segmented based on changes in the hand-object contact state. For all segments, the navigation and the manipulation ones, SafeMimic prompts a VLM that provide a semantic description of the actions. SafeMimic refines the semantic labeling results by reprompting the VLM for backward consistency between labels. The segmentation (motion and semantic labels) are used by SafeMimic to refine and adapt the actions in a safe and autonomous manner.
TABLE II: Segment Types and Possible Actions
Segment Type Possible Actions
Navigation
Navigating to [object]
Navigating with [object1] to [object2]
Grasping
Reach for and grasp [object]
Manipulation
Pick [object]
Place [object1] in [object2]
Open [object]
Close [object]
Wipe [object1] with [object2]
Complete original Figure 9.
Fig. 9: Snapshots of three different human demonstrations of the same task (top), and demonstrations of the task in three different environments (bottom). We evaluate SafeMimic’s abilities to parse initial demonstrations of different humans, and in different environments. The performance of SafeMimic is not affected (Sec. IV), indicating that our method is robust to variations in these dimensions and is able to explore and adapt actions in different conditions.
Complete original Figure 10.
Fig. 10: Simulation training domains for the safety Q-function functions of SafeMimic. We generate random and targeted actions in multiple simulated scenes in OmniGibson [65]. To increase generalization, the scenes are domain randomized using different objects and locations. We collect 43200 state-action pairs. Point clouds and robot proprioceptive signals are collected during data generation, as well as ground truth for different failure types, and used to train the ensemble of safety Q-function functions that SafeMimic uses for real-world exploration. The process is cheap and scalable and does not require solving the tasks in simulation.
Complete original Figure 11.
Fig. 11: Backtracking actions in SafeMimic. Example sequence of backtracking behavior in Shelving task. When no safe samples are available, or when a segment exploration ends without success, SafeMimic backtracks to previous states by undoing the last actions, possibly stepping back to explore a different grasping mode. This simple but effective mechanism enables autonomous exploration and adaptation of human motion for multi-step mobile manipulation tasks with SafeMimic.
Complete original Figure 12.
Fig. 12: Snapshots of the videos we evaluate SafeMimic’s video parsing module on in -C. We test the module on videos with different camera angles (left column), different levels of clutter (middle column), and different lighting conditions (right column). Our results indicate that SafeMimic is robust to videos in varying and challenging environments.

-C Robustness of the Human Video Parsing Module

Additional shelving videos vary camera angle, lighting and clutter. Reported accuracy is 86%, 100% and 88%, respectively, measured as the fraction of correctly identified video segments. These are parsing scores, not end-to-end robot-task success rates.

Clutter categories are described as few (1–3), several (3–5) and many (>5) distracting objects. Counts of videos/segments underlying each percentage are not stated here, and the first two categories share the boundary value three. The experiment demonstrates tolerance to the tested variations without establishing reliability outside them.

Concept demonstration

Open image ↗