At a glance
MAGIC transfers a demonstrated manipulation strategy by matching contact geometry on new objects. Pretrained visual features identify plausible regions, then local curvature refines the contact point and normal. Retargeting or motion planning converts the correspondence into actions, which are usually checked in simulation. Scooping, hanging, and hooking experiments show substantial cross-category gains, with additional physical hanging/hooking tests. The demonstrated scope assumes a supplied canonical view and one dominant contact patch, rather than arbitrary multi-contact tool use.
I Introduction
A useful analogy must match both where an object part lies in the overall shape and how its local surface supports contact. Whole-shape alignment can miss a mug handle’s functional similarity to a hanger, while local curvature alone produces many irrelevant candidates. MAGIC combines both cues to reuse one reference strategy efficiently. The method transfers contact geometry and motion structure; it does not train a new end-to-end policy from the single example.
II Related Work
Contact-mode planning searches possible interactions, and Learning Reusable Manipulation Strategies [4] learns a contact-sequence macro with specialized sampling. MAGIC instead focuses on finding functional contact correspondences quickly. Affordance/keypoint methods such as KETO [9] and GIFT [10] motivate reusable tool features, but generally use task-specific datasets. MAGIC relies on broadly pretrained features plus geometric analysis and physical verification, so its one-shot claim still depends on substantial pretrained and analytic priors.
III One-Shot Manipulation Strategy Learning
The pipeline extracts reference contacts, proposes globally similar regions, refines them geometrically, generates motion, and checks feasibility. The same correspondence score can also rank alternative tools.
III-A Problem Definition
The input provides a reference object trajectory, identified manipulanda and strategy, a canonical object view, and reference contact points. In the main tasks, one held object interacts through a single important contact waypoint in a two-dimensional cross-section. Translation, rotation, reflection, and scale matching in that view are lifted back into three-dimensional object motion using the canonical poses. Reference contacts come from simulation or manual annotation for real objects.
The score combines a visual match on the tool with a local contact-geometry match for the interacting pair. Candidate trajectories are simulated and the first successful one is selected. Main evaluations use stable quasi-static motions with simplified density/friction assumptions. When simulation is unavailable for a dynamic case such as the additional hammering comparison, the highest-scoring match is used instead. These assumptions limit the generality of the physical-feasibility claim.
Equation
III-B Global Contact Point Matching with Pretrained Features
DINOv2 [44] features are computed for the reference and twenty-four target variants: twelve rotations with and without reflection. Joint principal-component reduction suppresses feature noise, and patch-aggregated cosine similarity favors regions rather than single pixels. The three highest-scoring correspondences seed local refinement. Global matching is applied to the tool objects; the subsequent pairwise curvature comparison accounts for how the chosen tool meets the manipulated object.
Equation
III-C Local Contact Point Matching with Curvature Estimation
Segmented object edges are examined over multiple neighborhood scales. At each candidate, a local coordinate system aligns with edge variance and a fitted parabola supplies curvature magnitude, radius, and surface normal. The scale-selection rule uses a shared constant of 3.5. The object mask determines convex versus concave contact, and pairwise curvature scoring compares the relevant radii and contact structure.
Thin shapes and distant edges can corrupt the fit. A first estimate places a nearby ray origin; keeping only the first edge intersected along each ray removes irrelevant edges before refitting. A local search also enforces the reference contact’s convexity sign. This produces a more plausible geometric contact but still requires reachability, collision, and physical-stability checks; curvature alone is not a complete force or friction model.
Equations
III-D Generating Object Motions by Making Contact Analogies
Three uses follow from the matched point and normal. Local contact frames can directly retarget a reference object trajectory; a desired hanging pose or force direction can become a waypoint for motion planning; and the joint match score can select among available tools. Scooping uses retargeting, while hanging and hooking use motion planning. Selecting a contact does not by itself solve grasp choice or arm kinematics.
IV Experiments
Simulation uses SAPIEN 2 and a Franka Panda. The floating-gripper variant directly controls tool motion and excludes arm constraints; the harder arm variant adds antipodal grasp sampling and RRT-Connect planning. Scooping covers six spoon-like tools and three ball sizes, hanging covers 134 suitable mugs, and hooking covers four tools and five target shapes/sizes. Baselines include whole-shape PCA/alignment, DINO-only matching, and random curvature search capped at 180 seconds. Most pipelines are nearly deterministic; this explains the omitted run variance but does not remove uncertainty from the finite object/task selection.
IV-A Experiment Results in Simulation
Main transfer results. Floating-gripper success is 94.4% for scooping, 92.9% for hanging, and 100% for hooking with tool selection. With the full arm, these fall to 65.4%, 84.3%, and 63.3%, exposing grasping and motion-planning constraints. DINO-only scores are 55.7/23.9/66.7% in the floating setting. For hooking without selecting the tool, MAGIC reaches 65%, so the 100% result includes tool choice rather than showing every provided tool succeeds.
Shape and representation comparisons. Within-category mug transfer reaches 94.0% for MAGIC versus 41.1% for global feature-based point-cloud alignment; cross-category hanger-to-mug transfer is 92.9% versus zero. DINOv2 plus curvature takes 34.0 seconds on average, versus 50.7 for the Stable-Diffusion/DINO feature alternative and 60.8 for DIFT. DINO matching alone is faster at 12.4 seconds but less successful. The method is therefore efficient relative to the studied geometric-search alternatives, not faster than every baseline or suitable for instantaneous replanning.
Additional tasks and ablations. On KETO’s tasks, MAGIC reaches 64.4% average hammering success versus 56.4%, and 76.3% pushing success versus 71.7%; these use a different task/dataset comparison from the main experiments. Removing feature reduction and patch aggregation especially hurts scooping/hooking, although hanging rises slightly to 93.3%. Removing geometric refinement lowers all main scores. Skipping simulation verification lowers floating-gripper success to 77.8/71.6/50.0%, showing that matching alone does not establish execution feasibility.
| Method | Scooping | Hanging | Hooking | ||||
| FG | Arm | FG | Arm | No Tool Sel. | FG | Arm | |
| PCA | 27.6 | 16.7 | 0.0 | 0.0 | 5.0 | 0.0 | 0.0 |
| FPFH+ICP | 32.9 | 27.2 | 0.0 | 0.0 | 6.7 | 0.0 | 0.0 |
| DINO Matching | 55.7 | 41.4 | 23.9 | 21.6 | 49.2 | 66.7 | 24.0 |
| Curvature-Only | 13.9 | 5.6 | 68.7 | 38.1 | 60.0 | 76.7 | 3.3 |
| magic | 94.4 | 65.4 | 92.9 | 84.3 | 65.0 | 100 | 63.3 |
| PCA | FPFH+ICP | DINO Match | Curvature-Only | magic | |
| Hanging (intra-cat.) | 5.2 | 41.1 | 64.9 | 67.9 | 94.0 |
| Hanging (inter-cat.) | 0.0 | 0.0 | 23.9 | 68.7 | 92.9 |
| Method | Scooping | Hanging | Hooking | Avg. Time |
| magic (DIFT + Curvature) | 62.2 | 81.3 | 100 | 60.8s |
| magic (SD-DINO + Curvature) | 73.8 | 91.8 | 96.0 | 50.7s |
| magic (DINOv2 + Curvature) | 94.4 | 92.9 | 100 | 34.0s |
| Method | Hammering | Pushing | ||
|
Hammers Hammers |
Hammers Non-hammers |
Average | Average | |
| KETO | 66.4 | 46.3 | 56.4 | 71.7 |
| MAGIC | 75.6 | 53.7 | 64.4 | 76.3 |
| Method | Scooping | Hanging | Hooking | Avg. Time |
| DINO Matching (Best Baseline) | 55.7 | 23.9 | 66.7 | 12.4s |
| magic (DINOv2 + Curvature) | 94.4 | 92.9 | 100 | 34.0s |
| (w/o. Feature PCA and Patch) | 71.7 | 93.3 | 62.0 | 33.8s |
| (w/o. Two-Step Curvature) | 83.3 | 53.7 | 75.6 | 14.1s |
| (w/o. Simulation Verification) | 77.8 | 71.6 | 50.0 | 19.7s |
IV-B Real-World Experiments
Two calibrated RealSense cameras supply RGB and depth. Segment Anything masks, point-cloud filtering, downward completion to the tabletop, and alpha-shape reconstruction create meshes for simulation-based planning and checking. On hardware, ten trials are run for each of three cups and four tools. Figure 7 reports rounded aggregate success of 77% for hanging and 65% for hooking, versus 37/50% for DINO matching and 27/48% for curvature-only. The complete figure is retained. Perception and completed geometry remain part of the real pipeline, and the paper directs detailed failure examples to its project website rather than reproducing them in the article.
V Conclusion
Combining visual correspondence with local curvature is effective for the evaluated contact analogies, especially across object categories. The current method assumes a supplied canonical two-dimensional view, one dominant contact patch, and known task objects/strategy. Multi-object or multi-contact interaction, richer force reasoning, automatic view selection, and full three-dimensional feature/curvature matching remain future work. The paper acknowledges research-agency and institutional funding for this work.
Appendix
The appendix preserves the reference demonstrations and evaluation objects, physical hanging/hooking trajectories, and two simulated examples for each main task. The 134 mugs are filtered to remove shapes unsuitable for hanging, such as mugs without handles. Scooping tools include four spoons, a soup ladle, and a measuring cup; hooking uses scissors, a hanger, a caliper, and a watch against five cylindrical targets.
Additional Alphabet Toolkit examples demonstrate hanging and hooking with unusual shapes in simulation and on hardware. Figures 14–15 show those executions, but the appendix supplies no aggregate alphabet-tool success rates. All trajectory panels are retained rather than replacing these demonstrations with a brief verbal claim.









