At a glance
SafeMimic turns a single third-person RGB-D demonstration into semantic task segments and relative robot motions. It adapts these motions through physical exploration, screening actions with simulation-trained safety classifiers and backtracking when progress fails. Alternative grasps address differences between human and robot morphology; successful attempts train a memory policy.
On seven mobile-manipulation tasks, final success ranges from 40% to 100% over five trials per task. Unsafe actions account for 0.6% of executed actions, versus 14.2% for exploration without the safety filters. The trials are human-monitored: the method reduces measured risk but does not establish risk-free operation.
I Introduction
Copying a human trajectory rarely suffices for a robot with different geometry, reachability and grasping capabilities. Learning from one video therefore requires both understanding the task’s semantic changes and adapting the low-level motions. Real-world trial and error also needs failure prediction and a way to undo unsuccessful attempts.
SafeMimic combines motion tracking and vision-language interpretation with an ensemble of task-agnostic safety Q-functions. These classifiers predict generic failures rather than task reward. The central empirical premise is that such failure predictors can transfer from simulation more readily than an entire manipulation policy. Backtracking and grasp changes extend adaptation beyond small trajectory perturbations; a memory policy reduces repeated exploration.
II Related Work
Prior human-to-robot imitation and ScrewMimic demonstrate action transfer and adaptation, but morphology differences, long task sequences and unsafe exploration remain difficult. Large video corpora offer another route, with substantial data requirements. Autonomous-reset methods address repeated trials without necessarily preventing unsafe actions.
Safe imitation, safety-critic reinforcement learning, shielding and deployment-time failure prediction motivate SafeMimic’s filters. Analytic constraints and collision planners can miss contact-force, grasp-loss and joint-limit failures. Here separate classifiers are pretrained on task-agnostic simulated transitions and applied during adaptation from one human demonstration; they do not provide a formal safety certificate.
III SafeMimic Framework
The framework first decomposes a demonstration, then explores robot-compatible motions for each semantic goal. A final memory stage learns from successful real executions. Figure 2 preserves the complete information flow, including grasp changes and backtracking.
III-A Factorizing, Parsing and Translating Human Videos
Body/hand tracking, a contact detector and a vision-language model identify what changed and how the person moved. Hip translation separates navigation from stationary manipulation. Navigation receives an object/location goal; manipulation is split further at hand–object contact changes, so each segment has a relatively simple semantic completion condition.
Navigation becomes relative base-pose changes. For manipulation, hand poses are expressed relative to the human body and then converted to successive motion deltas. The robot initially selects the available generated grasp closest to the demonstrated grasp. This is a starting proposal, not a guarantee of morphological compatibility.
The procedure assumes the robot starts near the person’s initial pose. The limitations section explains that this correspondence is obtained by estimating that pose in the map and navigating the robot there. “No calibration needed” therefore should not be read as absence of initial spatial alignment.
III-B Safe and Autonomous Real-World Adaptation
Each segment is treated as a Markov decision process with sparse semantic-goal reward. Failure indicators cover several unsafe conditions. The displayed constrained objective asks for zero expected unsafe-state occupancy, but the implemented learned predictors only approximate this aim.
The stated safety Q-function combines the current failure indicator with its expected value after one transition. Separate classifiers represent individual failure modes; their maximum scores an action. This maximum is not generally the probability of the union of failures, and the expression is not a discounted long-horizon risk value.
Training uses domain-randomized OmniGibson scenes, random or noisy motion-planner actions, point clouds and robot proprioception. Articulated interactions, pick-and-place and navigation supply labeled transitions without requiring a successful task policy in simulation.
During execution, continuous candidates are sampled around the demonstrated motion with Gaussian perturbations; grasping uses three discrete grasp candidates. Candidates are filtered by predicted failure and ordered by their worst classifier score. A vision-language query tests the semantic goal after actions. The threshold is a classifier setting, not a calibrated upper bound on real failure probability.
When no candidate passes, the robot executes the inverse of its previous action. After 50 unproductive action attempts, it backtracks to a grasp segment and tries another grasp mode. This can reverse a door motion or reposition a held object, but cannot undo every physical event. Algorithm 1 retains the recursive exploration and ordering procedure in full.
Equations
III-C Learning from Previous Successful Exploration
Successful grasp poses and post-grasp action trajectories train an action predictor from point clouds and a language description of the task. A PointNet visual encoder, SentenceTransformer text embedding and MLP head are trained with translation/rotation augmentations.
For later attempts, the memory biases exploration toward robot-successful actions instead of always returning to the human-derived proposal. It supports grasp prediction and, for drawer opening, full trajectory prediction. Segments without a trained memory continue to use the parsed human actions.
IV Experimental Evaluation
Seven tasks combine navigation, grasping and manipulation: boxing, shelving, storing in a drawer, erasing a whiteboard, refrigerating an item, filling a pot and loading an oven. They expose top-down versus side-grasp constraints, heavy articulated doors, limited reach and force-sensitive contact.
Experiments use a Tiago++ mobile manipulator, end-effector inverse-kinematics control, base translation/yaw, a head-mounted Astra S RGB-D camera and wrist force/torque sensing. The enumerated failure types are arm collision, base collision, joint-limit violation, excessive force/torque, grasp loss and dropping objects.
Baselines directly execute parsed motions with or without safety filters, or explore without filters. Two behavior-cloning policies use either all safe simulated actions or successful simulated episodes. These imitation baselines receive SafeMimic’s successful navigation and picking segments and are evaluated only on place/open/close skills, so their setup is not an otherwise identical end-to-end comparison.
Experiments and Results
Each task has five attempts. Figure 4 reports accumulated completion through successive segments, rather than independent success for each segment. SafeMimic’s final rates are 100% for boxing, 80% for shelving, drawer storage and whiteboard erasing, 60% for refrigerating and filling a pot, and 40% for loading an oven. Both direct-execution baselines finish none of the seven tasks. Adaptation averages five minutes per navigation segment and fifteen per manipulation segment.
Grasp-mode changes are needed in six tasks; whiteboard erasing is the exception. A human-like pot grasp collides with the sink edge, whereas an explored top-down grasp succeeds. A drawer grasp must change to avoid joint limits. False-positive safety predictions sometimes reduce intermediate success for filtered direct execution.
A human flags unsafe executed actions across all trials. The unsafe-action rates are 13.4% for direct execution without filters, 14.2% for exploration without filters, 10.8% and 9.5% for the two imitation policies, 0.5% for filtered direct execution, and 0.6% for SafeMimic. The change from 14.2% to 0.6% is 13.6 percentage points, not a 13.6% relative reduction or elimination of unsafe events.
Memory evaluation randomizes object poses and encounters varying camera viewpoints from navigation. After initial successful trials, three evaluation trials per task show fewer exploratory actions, with reductions up to 67% in Figure 7. The experiment trains selected pick/open grasps and a drawer-opening trajectory; it does not train a complete reusable policy for every segment.
All three human shelving demonstrations are parsed successfully, and each requires grasp adaptation. One demonstrator in three environments yields task success of 100%, 100% and 66%, with three trials per environment. These small tests support transfer across the sampled people and scenes, rather than a broad robustness guarantee.
V Limitations and Future Work
Simulation must cover conditions resembling deployment, and constructing those scenarios still requires engineering. The classifiers address a predefined subset of failure types; unseen hazards and task failures need additional modeling. Initial human–robot pose correspondence and a separate robot grasp generator are also required.
Backtracking assumes sufficient reversibility and cannot recover a dropped object merely by reversing the last action. Autonomous resetting and learning from observed real-world failures remain future work. The low measured unsafe-action rate is encouraging, but is neither zero nor evidence that monitoring can safely be removed in arbitrary environments.
VI Conclusion
SafeMimic demonstrates a practical combination of semantic video parsing, learned failure screening, trajectory/grasp adaptation and memory. In the tested settings, simulation-trained safety predictors support more successful adaptation and fewer unsafe actions than the compared policies.
Its contribution is an empirical method for reducing exploration risk while transferring one demonstration. Finite human-monitored trials, enumerated failure classes and reversible-reset assumptions delimit the stronger language about safe autonomous learning.
Acknowledgements
The work was carried out in UT Austin’s Robot Interactive Intelligence Lab, supported by the College of Natural Sciences Catalyst Grant CAT-24-MartinMartin.
-A Model Details
The appendix describes a PointNet++ safety encoder, four-layer MLPs for actions and proprioception, and an MLP classification head. End-effector pose and force/torque enter the proprioceptive branch; force/torque is discretized into six bins to reduce the simulation gap. Binary safety labels train the classifiers with cross-entropy.
Table I preserves all settings: threshold 0.7, exploration standard deviation 0.05, batch size 16 and learning rate 10⁻³. It lists 9,000 SQF examples, while Figure 10 reports 43,200 collected state–action pairs; the filtering/split relationship is not explained. The action-delta range is printed as [0.0, 0.15], without clarifying its relation to negative reverse actions.
Algorithm 1 is retained as source pseudocode, including its Require line and 50 numbered steps. Its printed control flow does not spell out every initialization or exhausted-grasp termination case; the edition preserves these details rather than silently inventing an executable implementation.
| Hyperparameter | Q-function |
| SQF threshold | 0.7 |
| Exploration Standard Deviation | 0.05 |
| Grasp Mode Clustering? | K-Mediods |
| Batch Size | 16 |
| Optimizer | Adam) |
| Learning Rate | 1e-3 |
| SQF Dataset Size | 9000 |
| IL (all safe actions) Dataset Size | 23471 |
| IL (successful episodes) Dataset Size | 4305 |
| Nav/Stationary Pos Threshold | 0.07 |
| Nav/Stationary Ori Threshold | 0.2 |
-B Human Video Parsing
The detailed parser first thresholds hip motion, then queries GPT-4o at three frames per second to identify contact changes. A visual contact detector refines transition timing around those coarse detections. Semantic labeling chooses from the eight action forms in Table II, spanning navigation, grasping, picking, placing, opening, closing and wiping.
A backward correction pass receives all ordered labels and frames from the video. It resolves inconsistent object references, such as changing “navigate to sink” into “navigate to drawer” when the subsequent interaction opens the adjacent drawer. The resulting motion segments and semantic subgoals initialize robot exploration.
Figures 8–12 retain the parser diagram, different people/environments, simulated failure-training domains, a backtracking example and robustness-test snapshots. The demonstrations and images illustrate the method’s components; their captions’ broad robustness wording should be read alongside the limited quantitative protocols.
| Segment Type | Possible Actions | |||||
| Navigation |
| |||||
| Grasping |
| |||||
| Manipulation |
|
-C Robustness of the Human Video Parsing Module
Additional shelving videos vary camera angle, lighting and clutter. Reported accuracy is 86%, 100% and 88%, respectively, measured as the fraction of correctly identified video segments. These are parsing scores, not end-to-end robot-task success rates.
Clutter categories are described as few (1–3), several (3–5) and many (>5) distracting objects. Counts of videos/segments underlying each percentage are not stated here, and the first two categories share the boundary value three. The experiment demonstrates tolerance to the tested variations without establishing reliability outside them.