Literature
Display
← All papers

MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

Rutav Shah · Shuijing Liu · Qi Wang · Zhenyu Jiang · Sateesh Kumar · Mingyo Seo · Roberto Martín-Martín · Yuke Zhu

· ICRA 2026 ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 11-page v1 manuscript dated September 11, 2025. It preserves all nine numbered figures, five tables, three displayed equations and 59 references. The teaser is reunited with its caption, and composite figures are restored from the original PDF, retaining native imagery and vector labels. The later publication metadata is preserved separately from this source version.

Complete original PDF figure S0.F1
Fig. 1: Overview. MimicDroid enables few-shot learning for humanoid manipulation by training solely on human play videos—a scalable and diverse data source. At test time, it observes human videos of novel tasks and uses in-context learning to perform the same tasks under different object placements.

At a glance

MimicDroid learns to use a few human demonstration videos as context for humanoid manipulation, without updating policy parameters at deployment. It extracts hand poses from play videos, retrieves segments with similar observation–action patterns and trains a transformer to imitate a target segment conditioned on those examples. Wrist retargeting and random image masking support transfer to the GR1 humanoid. Real-world success is 53%, 23% and 8% across increasingly difficult generalization levels: promising adaptation, with substantial remaining failures on unseen environments.

I Introduction

The problem is rapid adaptation to new objects, layouts and manipulation tasks from one to three videos. Play provides varied, repeated behaviors without task labels or task-specific resets. MimicDroid uses those repetitions to construct training examples for learning how to learn in context. The source’s approximately 18× collection-speed claim comes from prior MimicPlay work [20], not a matched timing study of this dataset.

The policy learns observation–action associations rather than only recognizing a desired goal. Training predicts one retrieved behavior from others; deployment substitutes human examples of the requested task. “Only human videos” describes the real-world task-training data: the method also uses pretrained visual/hand models, and its simulation study uses teleoperated free-floating hands with recorded action labels.

Complete original PDF figure S1.F2
Fig. 2: Method Overview. MimicDroid performs meta-training for in-context learning (Meta-ICL) by constructing context-target pairs from human play videos. For a target segment, we retrieve the top-kk most similar trajectory segments (bottom-left) based on observation-action similarity (bottom-right) to serve as context. These context-target pairs are used to teach the policy in-context learning (top-left). To overcome the human-robot visual gap and avoid overfitting to human-specific visual cues, we apply visual masking to input images (top-right), improving transferability.

II Related Work

Meta-learning methods such as MAML adapt parameters from a few examples [1], while MetaICL trains models to solve new tasks through conditioning [22]. ICRT and Instant Policy demonstrate related in-context robot learning [11][12]. MimicDroid’s contribution is constructing useful context–target pairs from continuous play, reducing dependence on labeled robot demonstrations.

Human-video learning can supply representations, rewards or motion supervision. MimicPlay, OKAMI and EgoMimic provide relevant precedents [20][23][47]. MimicDroid combines RGB-based hand estimates with humanoid wrist retargeting and image masking; it does not solve arbitrary human–robot correspondence or learn robot kinematics from video.

III Methodology

The method constructs action-labeled segments from play, retrieves similar segments for context, and optimizes a causal action-prediction policy. The following subsections retain the complete training objectives and the distinctions between data retrieval, embodiment transfer and deployment.

III-A Overview

Training receives long play videos without discrete task labels. For real videos, inferred hand state augments RGB observations and supplies future action targets. A randomly sampled segment represents an implicit task; related segments become demonstrations. Equation 1 expresses expected behavior-cloning loss over these context–target combinations.

At test time, the frozen policy conditions on the supplied demonstrations and its own observation/action history. Generalization is measured separately for new object positions, new objects and new environments. No gradient update is needed for this adaptation, but hand-pose estimation, retargeting, inverse kinematics and action execution remain parts of the system.

Equation 1
𝔼𝒯∼p⁡(𝒯train)​𝔼σ𝒯ctx,σ𝒯tgt∼𝒯​[ℒ𝒯​(πθ​(σ𝒯tgt∣σ𝒯ctx))],\mathbb{E}_{\mathcal{T}\sim p(\mathcal{T}_{\text{train}})}\;\mathbb{E}_{\sigma_{\mathcal{T}}^{\text{ctx}},\,\sigma_{\mathcal{T}}^{\text{tgt}}\sim\mathcal{T}}\left[\mathcal{L}_{\mathcal{T}}\big(\pi_{\theta}(\sigma_{\mathcal{T}}^{\text{tgt}}\mid\sigma_{\mathcal{T}}^{\text{ctx}})\big)\right], (1)

III-B Constructing Training Samples

WiLoR estimates the wrist pose and finger configuration from RGB [48]. The current hand pose acts as proprioception, while a future pose becomes the action label; the real-world implementation uses the next frame and derives a binary gripper command from finger angles. These are inferred labels and can be wrong under occlusion.

For retrieval, DINOv2 supplies frame features [52]. Equation 2 concatenates their temporal mean with the ordered action sequence, then uses cosine similarity to find related trajectory segments. Visual pooling reduces incidental frame variation; actions retain motion information. The resulting similarity is a proxy for behavioral correspondence, not an annotated guarantee that two clips express the same semantic task. Figure 3 shows retrieval across objects and environments.

Complete original PDF figure S3.F3
Fig. 3: Examples of target and retrieved context examples.
Equation 2
ϕ⁡(σ)=[1T​∑t=1Tg⁡(st),a1,…,aT]\displaystyle\phi(\sigma)=\Big[\frac{1}{T}\sum_{t=1}^{T}g(s_{t}),\ a_{1},\ \ldots,\ a_{T}\Big] (2)

III-C Overcoming the Embodiment Gap

Predicted human wrist poses are mapped to humanoid wrists and converted to joints using inverse kinematics. Operating in Cartesian task space exploits morphological similarity but still depends on reachability and accurate pose estimates. The real experiments use one human hand; simulation uses two free-floating hands.

Random patch masking reduces reliance on embodiment-specific appearances. The main implementation applies it with probability 0.8, sampling one to sixteen erased patches; Table IV summarizes the effective range as zero to sixteen. Unlike explicit hand removal, this augmentation does not require a segmentation module. It improves transfer empirically without guaranteeing invariance to body shape, occlusions or camera changes.

Complete original PDF figure S3.F4
Fig. 4: Overview of our simulation benchmark. We introduce a simulation benchmark to evaluate few-shot learning for humanoid manipulation. It contains 88 hours of play data collected using free-floating hands (Left) across 3030 objects and 88 kitchen environments. Evaluation is structured into three levels: L1, L2, and L3 with increasing difficulty via novel objects and environments, enabling systematic assessment of generalization to humanoids (target) (Right). The bottom-right of each image indicates the source and target embodiments. Language descriptions are included for clarity.

III-D Meta-training for In-Context Learning

Targets are sampled uniformly from play rather than from a hand-enumerated task distribution, and retrieved contexts approximate related examples. Equation 3 minimizes an L1 loss on predicted action chunks conditioned on previous observations/actions and context trajectories. The model predicts 32 future actions; rollout executes the first 16 before querying it again.

Action chunking follows prior imitation-learning practice [53]. It smooths sequential prediction and provides multiple future controls, but a deterministic L1 objective is not itself a full probabilistic model of every valid behavior. Training contains three context segments plus one target; the retrieval-pool size is a separate quantity.

Equation 3
ℒBC=∑t=1T−1‖π(at:t+lpred|s1:t′,a1:t−1,{σc,i}i=1k)−at:t+l‖1\mathcal{L}_{\text{BC}}=\sum_{t=1}^{T-1}\left\|\pi\Big(a_{t:t+l}^{\text{pred}}\ \Big|\ s^{\prime}_{1:t},a_{1:t-1},\ \{\sigma_{c,i}\}_{i=1}^{k}\Big)-a_{t:t+l}\right\|_{1} (3)

IV Experiment Setup

The RoboCasa/robosuite benchmark [31][32] uses eight training kitchens and thirty objects. L1 changes positions with seen objects/environments; L2 introduces objects or object instances in familiar environments; L3 changes both objects and environment, sometimes requiring different motions. Models trained on free-floating-hand play are evaluated on that source embodiment and on GR1.

Simulation supplies eight hours / 320,000 steps from approximately twenty-minute sessions. Real collection uses a static RGB camera and approximately ten-minute single-hand play sessions, totaling 80,000 frames. The main text gives 45 minutes, whereas Table II gives one hour; the source does not reconcile this discrepancy.

WiLoR generates real action labels, DINOv2 is used for retrieval, and a separate frozen Cross-MAE encoder provides policy visual features. The main text specifies ten retrieved candidates; the retrieval ablation prefers fifty. The architecture consumes three context trajectories per training example, so candidate-pool size and actual context count should not be conflated.

Comparisons include goal-image-conditioned H2R [29], video-conditioned Vid2Robot [28] and test-time parameter-efficient fine-tuning [30]. The authors report common play data, augmentations and training pipelines. These compare distinct conditioning/adaptation designs; the results do not isolate a single architectural change or establish superiority over every robot-learning system.


Method Abstract GR1
L1 L2 L3 L1 L2 L3
H2R [29] 0.030.03 0.050.05 0.030.03 0.030.03 0.000.00 0.000.00
Vid2Robot [28] 0.440.44 0.400.40 0.120.12 0.410.41 0.230.23 0.110.11
PEFT [30] 0.470.47 0.350.35 0.000.00 0.290.29 0.210.21 0.010.01
MimicDroid w/o Visual Masking 0.590.59 0.51\mathbf{0.51} 0.220.22 0.370.37 0.350.35 0.090.09
MimicDroid 0.73\mathbf{0.73} 0.390.39 0.27\mathbf{0.27} 0.59\mathbf{0.59} 0.44\mathbf{0.44} 0.26\mathbf{0.26}
TABLE I: Success rates in Abstract and GR1 embodiments in the simulation benchmark.
Original figure
Fig. 5: MimicDroid reduces task misidentification and grasping errors using ICL compared to the video-conditioned baseline.

V Results

Table I reports exact simulation success fractions. MimicDroid scores 0.73/0.39/0.27 on Abstract L1/L2/L3 and 0.59/0.44/0.26 on GR1. Vid2Robot scores 0.44/0.40/0.12 and 0.41/0.23/0.11. Thus the equal-level averages improve by 14.3 and 18 percentage points, respectively. Abstract L2 is slightly worse than Vid2Robot, so the gains are not uniform.

Against PEFT, the table averages imply +19 points for Abstract and +26 for GR1. The narrative’s +29-point Abstract claim is inconsistent with its displayed numbers. PEFT has near-zero L3 success and a larger average source-to-GR1 drop (about 10 points versus 3.3). Forgetting and overfitting are the authors’ explanations; the paper does not directly measure those mechanisms.

Real GR1 success is 0.53/0.23/0.08 versus Vid2Robot’s 0.28/0.08/0.00. Appendix E specifies four tasks per level and ten rollouts per task. The “nearly twofold” description fits L1 approximately; it is not a constant ratio across all levels. The most difficult real setting remains only 8% successful.

Without masking, Abstract success is 0.59/0.51/0.22 and GR1 is 0.37/0.35/0.09. The average embodiment drop is 17 points; full MimicDroid reduces it to about 3.3. Removing masking improves Abstract L2 but hurts all GR1 levels. The EgoMimic comparison prints “0.53%” and “0.58%”; these appear to be fractions consistent with 53% and 58%, but the percent notation is internally inconsistent and the comparison is not broken down by level.

Figure 6’s GR1-L2 scores rise from 0.34 with one context example to 0.39 with three, then plateau around 0.38 before falling to 0.33 at six. Figure 7’s retrieval-pool ablation scores 0.34/0.44/0.49/0.41 for k=3/10/50/100. These support a useful diversity/noise tradeoff; more context or more retrieved data does not always help.

Figure 8 shows improved endpoints with more data, although L1 dips at 256,000 frames before recovering. Between 128,000 and 320,000 frames, L1 rises 0.35→0.59, L2 0.21→0.45 and L3 0.16→0.27. Its final L2/L3 values differ slightly from Table I’s 0.44/0.26; ablation values should be read from their own plots rather than silently merged into one configuration.

Figure 5 partitions evaluated outcomes into 50% success, 26% task misidentification, 16% grasping failure and 8% other errors for MimicDroid, compared with 30/41/21/8 for Vid2Robot. The error percentages refer to the full plotted outcome distribution, not percentages conditional on failure. The figure’s 50% success is not Table I’s 43% GR1 level-average; the aggregation is not fully specified.

Novel motions remain difficult, such as switching from a sideways faucet action to a vertical one. Human hand-size overfitting can also cause collisions in clutter. Confidence intervals, repeated-training variability and detailed simulation rollout counts are not reported, limiting the precision of small differences and broad generalization claims.

Original figure
Fig. 6: Performance rises with more in-context examples but plateaus beyond 3 due to training-time context length.
Original figure
Fig. 7: Performance benefits from more retrieved context segments (kk) per target, but high values introduce noise.
Original figure
Fig. 8: Performance scales consistently with training data across from L1-L3, showing promise for learning from play.
Complete original PDF figure S5.F9
Fig. 9: Examples of real-world evaluations (L1-L3). MimicDroid generalizes to both seen (e.g., chips, bread, and apples) and unseen objects (e.g., potatoes, garlic, and cloth), as well as novel environments. Evaluation tasks include pick-and-place to different receptacles and articulated object manipulation (e.g., opening or closing an oven tray).

VI Limitations and Future Work

The method is trained on deliberately recorded play, not arbitrary internet videos. Web-scale augmentation is proposed rather than evaluated. Hand-pose models fail when hands disappear inside cupboards or behind furniture; combining hand and whole-body tracking is future work [59].

The policy primarily matches low-level state/action patterns and lacks an explicit representation of why a motion matters. Semantically equivalent tasks with different movements remain challenging. Language–trajectory meta-training, more varied operators and more efficient long-context models are suggested extensions, not established results.

VII Conclusion

MimicDroid shows that retrieved play segments can train useful in-context manipulation behavior and transfer it to a humanoid without robot demonstrations in the real training set. The strongest evidence is the controlled baseline and masking comparison. Low real L3 success, inferred-action errors, limited context length and inconsistent reporting constrain claims of broad household adaptability.

Acknowledgments

The authors acknowledge colleagues at UT Austin’s Robot Perception and Learning and Robot Interactive Intelligence labs. Support includes NSF, ONR, DARPA TIAMAT, Army Research Lab and Korea’s IITP National AI Research Lab program.

Appendix A Appendix

The appendix specifies collection, action preprocessing, architecture, training hyperparameters and the exact real/simulated evaluation tasks. These details qualify the headline description of learning only from videos.

A-A Data Collection

A single static camera records real single-hand interactions; simulated data uses two free-floating hands controlled by a six-DoF spacemouse and three cameras. Table II records 80,000 real frames and 320,000 simulated frames. Its one-hour real total differs from the main text’s 45-minute figure; both are retained without inventing a reconciliation.

TABLE II: Human Play Data Collection Summary
Modality Duration per Session Total Hours Total Frames
Real world 1010 minutes 11 hour 80,00080,000
Simulation 2020 minutes 88 hours 320,000320,000

A-B Data Preprocessing

Real labels are next-frame six-DoF wrist poses in the workspace-camera frame plus a binary gripper state. Simulation uses recorded teleoperation actions in the robot-base frame. Observations combine object-centric image crops with current hand state. The text does not fully specify object-crop generation or the complete calibration/retargeting procedure, which matter for reproduction.

A-C Model Architecture

The architecture builds on ICRT [11]. A training example contains three context trajectories and one target, each 256 steps in simulation or 128 in reality. Input subsampling by four yields 512 or 256 interleaved observation/action tokens; this reduces input cost without reducing the predicted action frequency.

Two-layer MLPs encode 16-/8-dimensional proprioception and 14-/7-dimensional actions. Frozen Cross-MAE features [27] are fused with proprioception; a four-layer, 768-dimensional LLaMA-style transformer uses RMSNorm and rotary positional embeddings. An MLP predicts 32 action vectors and the controller executes sixteen before replanning. Table III is retained verbatim, including its attention-pooling dimensions; the source does not completely specify every shape/interface transformation.

TABLE III: Model Summary
Component Details
Vision Encoder Pre-trained CrossMAE [27]
Proprio Encoder MLP: 16/8→76816/8\rightarrow 768
Visual Attention Pooling 33 layers, 768→256768\rightarrow 256
Action Encoder MLP: 14/7→76814/7\rightarrow 768
Transformer 44 layers, LLaMA-style, RMSNorm, 768768-dim
Action Decoder MLP head, 768→32×14/32×7768\rightarrow 32\times 14/32\times 7

A-D Training Details

Training uses AdamW, learning rate 5×10⁻⁴, effective batch size 256, weight decay 0.01, two warmup epochs and 200 total epochs. Eight A5000 GPUs take approximately two days. Table IV preserves all sequence lengths, proprioceptive noise and augmentation ranges. Erased patches each cover roughly 1–4% of the image, with up to sixteen patches, plus brightness and contrast augmentation.

TABLE IV: Training Hyperparameters
Config Value
Optimizer AdamW
Base Learning Rate 55e-44
Effective Batch Size 256
Weight Decay 0.010.01
Warmup Epochs 22
Total Epochs 200200
Sequence Length 512512 (Simulation), 256256 (Real world)
Num Action Prediction Steps 3232
Proprioception Noise 0.0050.005
Brightness Augmentation Uniform(−0.1-0.1, 0.10.1)
Contrast Augmentation Uniform(0.80.8, 1.21.2)
Random Patch Masking 00–1616 patches, Area ∼\sim Uniform(1%1\%, 4%4\%)
Num Cameras 33 (Simulation), 11 (Real world)

A-E Real-world Evaluation

Each of four tasks receives ten real GR1 rollouts at each difficulty level. L1 includes oven-top→plate, rotating-disk→plate, plate→oven-tray and pulling out the tray. L2 uses novel categories such as potato, garlic and cloth and new instances such as oven/bowl; the first task changes destination to a bowl. L3 repeats the bowl transfer, uses drawers→plate, and retains plate→tray and tray pulling in a new environment.

The held-out objects and layouts extend beyond training, but the evaluation stays within a small set of transfer and articulated-object tasks. It does not test unrestricted language commands or every kitchen behavior.

A-F Simulation Benchmarking

Simulation prompts come from held-out free-floating-hand play depicting the target task. Table V retains all twelve tasks and both embodiment images for each: four per difficulty level. L1 includes sink transfers, faucet activation and left-cabinet closure; L2 adds unseen objects and right/left cabinet variants; L3 includes cabinet closure, sink→counter plate, sink→microwave top and a new faucet mechanism.

L3 changes motion requirements as well as appearance: sink heights, faucet direction and receptacle heights may differ. The benchmark therefore measures several shifts together. Simulation action labels are available from teleoperation, unlike the inferred WiLoR labels used for the real video dataset.

TABLE V: Evaluation is structured into three levels with increasing difficulty and 4 tasks in each level.
Level Task Name Abstract Embodiment Humanoid Embodiment
L1 (Seen Objects, Seen Environment) PnPSinkToRightCounterPlate [Uncaptioned image] [Uncaptioned image]
PnPSinkToCabinet [Uncaptioned image] [Uncaptioned image]
TurnOnFaucet [Uncaptioned image] [Uncaptioned image]
CloseLeftCabinetDoor [Uncaptioned image] [Uncaptioned image]
L2 (Unseen Objects, Seen Environment) PnPSinkToRightCounterPlateL2 [Uncaptioned image] [Uncaptioned image]
PnPSinkToCabinetL2 [Uncaptioned image] [Uncaptioned image]
CloseRightCabinetDoorL2 [Uncaptioned image] [Uncaptioned image]
CloseLeftCabinetDoorL2 [Uncaptioned image] [Uncaptioned image]
L3 (Unseen Objects, Unseen Environment) CloseLeftCabinetDoorL3 [Uncaptioned image] [Uncaptioned image]
PnPSinkToRightCounterPlateL3 [Uncaptioned image] [Uncaptioned image]
PnPSinkToMicrowaveTopL3 [Uncaptioned image] [Uncaptioned image]
TurnOnFaucetL3 [Uncaptioned image] [Uncaptioned image]
Concept demonstration

Open image ↗