Literature
Display
← All papers

One-Shot Manipulation Strategy Learning by Making Contact Analogies

Yuyao Liu · Jiayuan Mao · Joshua Tenenbaum · Tomás Lozano-Pérez · Leslie Pack Kaelbling

· ICRA 2025 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

MAGIC transfers a demonstrated manipulation strategy by matching contact geometry on new objects. Pretrained visual features identify plausible regions, then local curvature refines the contact point and normal. Retargeting or motion planning converts the correspondence into actions, which are usually checked in simulation. Scooping, hanging, and hooking experiments show substantial cross-category gains, with additional physical hanging/hooking tests. The demonstrated scope assumes a supplied canonical view and one dominant contact patch, rather than arbitrary multi-contact tool use.

I Introduction

A useful analogy must match both where an object part lies in the overall shape and how its local surface supports contact. Whole-shape alignment can miss a mug handle’s functional similarity to a hanger, while local curvature alone produces many irrelevant candidates. MAGIC combines both cues to reuse one reference strategy efficiently. The method transfers contact geometry and motion structure; it does not train a new end-to-end policy from the single example.

Refer to caption
Fig. 1: We introduce magic (Manipulation Analogies for Generalizable Intelligent Contacts), a pipeline that is capable of learning manipulation strategies from single demonstrations and applying them to novel objects.

II Related Work

Contact-mode planning searches possible interactions, and Learning Reusable Manipulation Strategies [4] learns a contact-sequence macro with specialized sampling. MAGIC instead focuses on finding functional contact correspondences quickly. Affordance/keypoint methods such as KETO [9] and GIFT [10] motivate reusable tool features, but generally use task-specific datasets. MAGIC relies on broadly pretrained features plus geometric analysis and physical verification, so its one-shot claim still depends on substantial pretrained and analytic priors.

III One-Shot Manipulation Strategy Learning

The pipeline extracts reference contacts, proposes globally similar regions, refines them geometrically, generates motion, and checks feasibility. The same correspondence score can also rank alternative tools.

Refer to caption
Fig. 2: The overall pipeline of magic. (a) We first extract contact points from the reference trajectory. (b) Then, we compute a global and local contact point matching score to select candidate contact points on novel objects. (c) The generated contact points will be used for motion retargeting or motion planning, and the final motion will be simulated and verified by a physical simulator.

III-A Problem Definition

The input provides a reference object trajectory, identified manipulanda and strategy, a canonical object view, and reference contact points. In the main tasks, one held object interacts through a single important contact waypoint in a two-dimensional cross-section. Translation, rotation, reflection, and scale matching in that view are lifted back into three-dimensional object motion using the canonical poses. Reference contacts come from simulation or manual annotation for real objects.

The score combines a visual match on the tool with a local contact-geometry match for the interacting pair. Candidate trajectories are simulated and the first successful one is selected. Main evaluations use stable quasi-static motions with simplified density/friction assumptions. When simulation is unavailable for a dynamic case such as the additional hammering comparison, the highest-scoring match is used instead. These assumptions limit the generality of the physical-feasibility claim.

Equation
score​(X,X′)=sdino​(IT,IT′,pT,pT′)+λ⋅scurv​(X,X′),\textit{score}\left(X,X^{\prime}\right)=s_{\textit{dino}}(I_{T},I_{T^{\prime}},p_{T},p_{T^{\prime}})+\lambda\cdot s_{\textit{curv}}(X,X^{\prime}),

III-B Global Contact Point Matching with Pretrained Features

DINOv2 [44] features are computed for the reference and twenty-four target variants: twelve rotations with and without reflection. Joint principal-component reduction suppresses feature noise, and patch-aggregated cosine similarity favors regions rather than single pixels. The three highest-scoring correspondences seed local refinement. Global matching is applied to the tool objects; the subsequent pairwise curvature comparison accounts for how the chosen tool meets the manipulated object.

Refer to caption
Fig. 3: Global and Local Contact Point Matching. The contact point matching process consists of two phases: (a) global matching using DINOv2 [44] features; (b) local matching involving multi-scale curvature estimation and refinements using irrelevant point suppression and convexity matching.
Equation
sdino​(IT,IT′,pT,pT′)=∑p∈patch​(pT),p′∈patch​(pT′)⟨F~T​(p),F~T′i​(p′)⟩.s_{\textit{dino}}\left(I_{T},I_{T^{\prime}},p_{T},p_{T^{\prime}}\right)=\sum_{\genfrac{}{}{0.0pt}{1}{p \in\textit{patch}(p_{T}),}{p' \in\textit{patch}(p_{T'})}}\left\langle\widetilde{F}_{T}(p),\widetilde{F}_{T^{\prime}}^{i}(p^{\prime})\right\rangle.

III-C Local Contact Point Matching with Curvature Estimation

Segmented object edges are examined over multiple neighborhood scales. At each candidate, a local coordinate system aligns with edge variance and a fitted parabola supplies curvature magnitude, radius, and surface normal. The scale-selection rule uses a shared constant of 3.5. The object mask determines convex versus concave contact, and pairwise curvature scoring compares the relevant radii and contact structure.

Thin shapes and distant edges can corrupt the fit. A first estimate places a nearby ray origin; keeping only the first edge intersected along each ray removes irrelevant edges before refitting. A local search also enforces the reference contact’s convexity sign. This produces a more plausible geometric contact but still requires reachability, collision, and physical-stability checks; curvature alone is not a complete force or friction model.

Equations
a=arg⁡min⁡∑i=1na⁡(yi′−a​xi′2)2,a=\arg\min_{a}\sum_{i=1}^{n}\left(y_{i}^{\prime}-ax_{i}^{\prime 2}\right)^{2},
scurv​(⟨IT,IO,pT,pO⟩,⟨IT′,IO′,pT′,pO′⟩)=|r⁡(pT)r⁡(pO)−r⁡(pT′)r⁡(pO′)|.s_{\textit{curv}}\left(\langle I_{T},I_{O},p_{T},p_{O}\rangle,\langle I_{T^{\prime}},I_{O^{\prime}},p_{T^{\prime}},p_{O^{\prime}}\rangle\right)=\left|\frac{r(p_{T})}{r(p_{O})}-\frac{r(p_{T^{\prime}})}{r(p_{O^{\prime}})}\right|.
Original figure
Fig. 4: (a) The observing scale at which we perform estimation has effect on both the magnitude and the sign of the estimated curvature. (b) Filtering out the irrelevant points can give a more accurate curvature estimation. (c) We perform local search within the observation region to find the contact point with the correct convexity.

III-D Generating Object Motions by Making Contact Analogies

Three uses follow from the matched point and normal. Local contact frames can directly retarget a reference object trajectory; a desired hanging pose or force direction can become a waypoint for motion planning; and the joint match score can select among available tools. Scooping uses retargeting, while hanging and hooking use motion planning. Selecting a contact does not by itself solve grasp choice or arm kinematics.

IV Experiments

Simulation uses SAPIEN 2 and a Franka Panda. The floating-gripper variant directly controls tool motion and excludes arm constraints; the harder arm variant adds antipodal grasp sampling and RRT-Connect planning. Scooping covers six spoon-like tools and three ball sizes, hanging covers 134 suitable mugs, and hooking covers four tools and five target shapes/sizes. Baselines include whole-shape PCA/alignment, DINO-only matching, and random curvature search capped at 180 seconds. Most pipelines are nearly deterministic; this explains the omitted run variance but does not remove uncertainty from the finite object/task selection.

IV-A Experiment Results in Simulation

Main transfer results. Floating-gripper success is 94.4% for scooping, 92.9% for hanging, and 100% for hooking with tool selection. With the full arm, these fall to 65.4%, 84.3%, and 63.3%, exposing grasping and motion-planning constraints. DINO-only scores are 55.7/23.9/66.7% in the floating setting. For hooking without selecting the tool, MAGIC reaches 65%, so the 100% result includes tool choice rather than showing every provided tool succeeds.

Shape and representation comparisons. Within-category mug transfer reaches 94.0% for MAGIC versus 41.1% for global feature-based point-cloud alignment; cross-category hanger-to-mug transfer is 92.9% versus zero. DINOv2 plus curvature takes 34.0 seconds on average, versus 50.7 for the Stable-Diffusion/DINO feature alternative and 60.8 for DIFT. DINO matching alone is faster at 12.4 seconds but less successful. The method is therefore efficient relative to the studied geometric-search alternatives, not faster than every baseline or suitable for instantaneous replanning.

Additional tasks and ablations. On KETO’s tasks, MAGIC reaches 64.4% average hammering success versus 56.4%, and 76.3% pushing success versus 71.7%; these use a different task/dataset comparison from the main experiments. Removing feature reduction and patch aggregation especially hurts scooping/hooking, although hanging rises slightly to 93.3%. Removing geometric refinement lowers all main scores. Skipping simulation verification lowers floating-gripper success to 77.8/71.6/50.0%, showing that matching alone does not establish execution feasibility.

Method Scooping Hanging Hooking
FG Arm FG Arm No Tool Sel. FG Arm
PCA 27.6 16.7 0.0 0.0 5.0 0.0 0.0
FPFH+ICP 32.9 27.2 0.0 0.0 6.7 0.0 0.0
DINO Matching 55.7 41.4 23.9 21.6 49.2 66.7 24.0
Curvature-Only 13.9 5.6 68.7 38.1 60.0 76.7 3.3
magic 94.4 65.4 92.9 84.3 65.0 100 63.3
TABLE I: Average success rates (%) of 3 tasks. ‘floating gripper (FG)’ indicates manipulation by directly setting positions and velocities of objects, focusing only on object-object interactions, excluding the robot. All methods except for Curvature-Only run almost deterministically. Curvature-Only has an average std. of 6.6%.
Refer to caption
Fig. 5: (a) Global shape matching fails to provide effective functional alignment due to the lack of awareness of local contact points. (b) Local curvature helps identify more accurate contact points and contact normals. (c) Pretrained DINOv2 features enhance data efficiency by selecting the optimal contact (the red star) and grasping points from those that curvature alone cannot distinguish (the green region). We also visualize the DINOv2 matching heatmaps.
PCA FPFH+ICP DINO Match Curvature-Only magic
Hanging (intra-cat.) 5.2 41.1 64.9 67.9 94.0
Hanging (inter-cat.) 0.0 0.0 23.9 68.7 92.9
TABLE II: Average success rates (%) of Hanging (intra-category transfer vs. inter-category transfer). While PCA and FPFH+ICP can occasionally achieve intra-category transfer, they fail to perform inter-category transfer. The experiments are conducted on the FG variant.
Method Scooping Hanging Hooking Avg. Time
magic (DIFT + Curvature) 62.2 81.3 100 60.8s
magic (SD-DINO + Curvature) 73.8 91.8 96.0 50.7s
magic (DINOv2 + Curvature) 94.4 92.9 100 34.0s
TABLE III: Additional studies on the choice of pretrained image features. We substitute DINOv2 with SD-DINO [46] and DIFT [57]. We present the average success rates (%) and average time consumed of the methods. The experiments are conducted on the FG variant.
Method Hammering Pushing
Hammers →\rightarrow
Hammers
Hammers →\rightarrow
Non-hammers
Average Average
KETO 66.4 46.3 56.4 71.7
MAGIC 75.6 53.7 64.4 76.3
TABLE IV: Comparison between MAGIC and KETO [9] (Learning Keypoint Representations for Tool Manipulation). We present the average success rates (%). We also investigate the intra-category and inter-category generalizablity of magic and KETO, indicated by Hammers →\to Hammers and Hammers →\to Non-hammers, respectively.
Method Scooping Hanging Hooking Avg. Time
DINO Matching (Best Baseline) 55.7 23.9 66.7 12.4s
magic (DINOv2 + Curvature) 94.4 92.9 100 34.0s
(w/o. Feature PCA and Patch) 71.7 93.3 62.0 33.8s
(w/o. Two-Step Curvature) 83.3 53.7 75.6 14.1s
(w/o. Simulation Verification) 77.8 71.6 50.0 19.7s
TABLE V: Ablation studies. We present the average success rates (%) and average runtime of all methods, on the FG variant.

IV-B Real-World Experiments

Two calibrated RealSense cameras supply RGB and depth. Segment Anything masks, point-cloud filtering, downward completion to the tabletop, and alpha-shape reconstruction create meshes for simulation-based planning and checking. On hardware, ten trials are run for each of three cups and four tools. Figure 7 reports rounded aggregate success of 77% for hanging and 65% for hooking, versus 37/50% for DINO matching and 27/48% for curvature-only. The complete figure is retained. Perception and completed geometry remain part of the real pipeline, and the paper directs detailed failure examples to its project website rather than reproducing them in the article.

Refer to caption
Fig. 6: Real-world experiment setup.
Original figure
Fig. 7: Average success rates of real-world experiments.
Refer to caption
Fig. 8: Contact point matching results on real-world objects.

V Conclusion

Combining visual correspondence with local curvature is effective for the evaluated contact analogies, especially across object categories. The current method assumes a supplied canonical two-dimensional view, one dominant contact patch, and known task objects/strategy. Multi-object or multi-contact interaction, richer force reasoning, automatic view selection, and full three-dimensional feature/curvature matching remain future work. The paper acknowledges research-agency and institutional funding for this work.

Refer to caption
Fig. 9: Visualization of demonstrations and novel scenes/objects. We evaluate the ability of one-shot manipulation strategy learning by providing a single demonstration for three tasks: Scooping, Hanging, and Hooking (shown in the first column), and then testing on various unseen instances.
Refer to caption
Refer to caption
Fig. 10: Example episodes of Hanging and Hooking in the real world.

Appendix

The appendix preserves the reference demonstrations and evaluation objects, physical hanging/hooking trajectories, and two simulated examples for each main task. The 134 mugs are filtered to remove shapes unsuitable for hanging, such as mugs without handles. Scooping tools include four spoons, a soup ladle, and a measuring cup; hooking uses scissors, a hanger, a caliper, and a watch against five cylindrical targets.

Additional Alphabet Toolkit examples demonstrate hanging and hooking with unusual shapes in simulation and on hardware. Figures 14–15 show those executions, but the appendix supplies no aggregate alphabet-tool success rates. All trajectory panels are retained rather than replacing these demonstrations with a brief verbal claim.

Refer to caption
Refer to caption
Fig. 11: Two example episodes of Scooping in simulation.
Refer to caption
Refer to caption
Fig. 12: Two example episodes of Hanging in simulation.
Refer to caption
Refer to caption
Fig. 13: Two example episodes of Hooking in simulation.
Refer to caption
Refer to caption
Fig. 14: Visualization of Hanging alphabet objects in simulation (top) and real-world (bottom) settings.
Refer to caption
Refer to caption
Fig. 15: Visualization of Hooking with alphabet objects in simulation (top) and real-world (bottom) settings.
Concept demonstration

Open image ↗