Overview
A manipulation strategy can be reused by preserving its contact sequence while relearning how to choose grasps, placements, and motions for new objects.
At a glance
The system turns one contact-rich demonstration into a symbolic macro, then uses simulated self-play to learn continuous-parameter rankings for that macro. A task-and-motion planner combines these learned mechanisms with ordinary manipulation operations. Six simulated mechanisms and two compositions show large search-efficiency gains; limited physical tests demonstrate edge pushing and tool-assisted reaching. One demonstration supplies the abstract sequence, while simulator trials, object models, and generic planners supply substantial additional knowledge.
1 Introduction
The useful part of a tool-use demonstration may be its sequence of contacts rather than its exact trajectory or object category. A ladle-hooking demonstration can specify grasping a tool, contacting a target, putting the tool down, and grasping the recovered target. The paper follows explanation-based manipulation learning [2]: extract that structure, specialize continuous sampling through self-play, and expose the result as a reusable planning operator.
2 Planning with Contacts and Mechanisms
Generic contact operations can in principle express these strategies, but searching their discrete sequences and continuous parameters is expensive. Learned mechanisms shorten the effective search horizon and focus parameter generation while retaining compatibility with the underlying planner.
2.1 Basic Domain Representation
World state includes known object shapes and poses, robot state, and holding/support relationships. A parameterized operator specifies arguments, preconditions, effects, and a sampler. Grounding its object arguments and continuous values yields an executable action. This representation follows task-and-motion planning [5]; its symbolic relations organize the search but do not replace geometric or physical feasibility checks.
| Name | Args. | Description |
| transit | empty | Move the robot without object collisions. |
| transit-cont | ?x, ?s | Move without holding anything, but have a contact with ?x. ?x is supported by ?s. |
| grasp | ?x, ?s | Grasp the object ?x that is currently supported by ?s. |
| place | ?x, ?s | Place object ?x onto ?s. ?x should be currently held by the robot. |
| move | ?x | Move while holding ?x. |
| move-cont | ?x, ?y, ?s | Move while holding ?x and ?x have contact with ?y. ?y is supported by ?s. |
2.2 Mechanisms
A mechanism contains arguments, an initial contact pattern, a desired certified effect, an ordered sequence of primitive operations, and specialized samplers. For hook use, the macro grasps the tool, moves it against the target, places it down, then grasps the target. Object names become variables, enabling reuse with suitable new shapes rather than replaying fixed demonstration coordinates.
2.3 Planning with Basis Operators and Mechanisms
A bilevel planner, inspired by PDDLStream [6], searches discrete plans with a Fast Forward heuristic, then samples grasps, contacts, and trajectories. Simulation checks grounded action effects; unsuccessful continuous search causes backtracking. Learned macros enter the same operator set. Position-controlled actions may trigger transient object motion before settling, allowing sliding and lever behavior beyond a strictly quasi-static model.
3 Learning New Mechanisms
Learning requires a demonstration containing actions and contact information, a human-specified goal, and a simulator that samples initial configurations where the mechanism is applicable. The system extracts a macro, searches within that sequence on new configurations, trains parameter-scoring models from successful and unsuccessful samples, and adds the resulting mechanism to the planner.
3.1 Extraction of Preconditions and Operation Sequence
Demonstrations are segmented into free, holding, and pushing contact regimes. Each segment records hand–object, indirect object–object, and supporting contacts. The initial graph becomes the mechanism precondition; replacing concrete object identities with variables yields its abstract schema. This abstraction depends on detailed contact labels, which are assumed available rather than inferred robustly from ordinary video.
3.2 Sampler Learning
Self-play fixes the demonstrated discrete sequence but explores object assignments and continuous choices. Samples appearing in a successful plan are labeled positive; other sampled choices are labeled negative. Binary classifiers rank batches from generic samplers, conditioned on previously chosen parameters, allowing early backtracking when a step fails. These scores are learned search heuristics, not certified probabilities that an isolated parameter is always feasible. The paper mentions binary cross-entropy but supplies no displayed loss formula in either the HTML or PDF.
3.3 Implementation
Geometry-based samplers propose grasps, placements, contact surfaces, and arm trajectories. Contact sampling aligns points and opposing normals while avoiding pushes into the supporting surface. Specialized scorers encode shapes with PointNet++ [8] and encode poses/contact quantities with multilayer networks, then predict a score. Each classifier uses one hundred training samples. Learned trajectory scoring covers contact direction and distance, not complete grasp/place arm paths, which remain the motion planner’s responsibility.
4 Experiments
The main experiments use PyBullet to compare individual mechanisms and their composition. Physical deployment is reported separately in Appendix B.3, including perception, replanning, and failure modes.
4.1 Learning Mechanisms from Single Demonstrations
Six tasks cover edge pushing, hooking, lever use, poking through a tunnel, center-of-mass placement, and blocking a slope. Each demonstration is a manually scripted execution. Training and testing sample object configurations selected to make the mechanism feasible, including novel instances and categories.
Over ten trials with a ten-minute timeout, macros plus learned samplers reduce average hook solution time to 3.84 seconds versus 30.82 for macros alone and over 600 for basis operators. Center-of-mass placement falls to 0.97 seconds versus 17.58 and 19.30. Lever use is already easy with the macro: 1.55 seconds with learned sampling versus 1.38 without. Poking remains difficult at 97.76 seconds because its collision-free arm-motion region is narrow. The gain is task-dependent search efficiency, not uniformly faster learned sampling.
| Method | Edge | Hook | Lever | Poking | CoM | Slope&Blocker | ||
| Basis Ops Only |
|
600 | 523.18 | 600 | 19.30 | 600 | ||
|
|
30.82 | 1.38 | 494.30 | 17.58 | 148.57 | ||
| Ours (Macro+Sampler) | 0.57 | 3.84 | 1.55 | 97.76 | 0.97 | 4.11 |
4.2 Planning with Learned Mechanisms
Two new compositions require pushing a thin caliper to an edge before grasping and hooking, or hooking a distant object before placing it on a blocked slope. Previously learned mechanisms are reused without additional training. With ten trials per task and a ten-minute timeout, complete success is 100% and 90% for macros plus samplers, 20% and 0% for macros alone, and 0% for basis operations. Table 3 reports success rates, despite the surrounding source prose calling them planning times. These are deliberately feasible composed tasks rather than arbitrary long-horizon environments.
| Method | Task 1 | Task 2 |
| Basis Ops Only | 0% | 0% |
| Ours (Macro) | 20% | 0% |
| Ours (Macro + Sampler) | 100% | 90% |
5 Related Work
The approach extends sampling-based task-and-motion planning with learned contact-sequence macros. Contact-mode planners supply a related representation, and tool-affordance learning supplies related geometric generalization. The distinguishing combination is extracting a reusable sequence from one labeled demonstration, learning its parameter preferences through simulation, and composing it within a general planner.
6 Limitations and Conclusions
Detailed demonstration contacts and reliable simulation are the main limitations. Inaccurate geometry or physics can defeat transferred mechanisms, and real-world fine-tuning is proposed. The representation supports new instances and compositions within its available operators, but broader contact sensing, compliance, and forceful actions are extensions rather than demonstrated capabilities.
Acknowledgement.
The authors acknowledge reviewer feedback and support from US research agencies, MIT research programs, the MIT–IBM Watson AI Lab, and the Boston Dynamics Artificial Intelligence Institute.
Supplementary material expands the planning implementation, generic geometric samplers, evaluation distributions, and physical robot experiments.
Appendix A Method Detail
The method appendix retains the complete bilevel planning algorithm and explains how contact schemas and generic samplers handle transient physical interactions.
A.1 Planning Algorithm
Discrete search enumerates both primitive and mechanism operators. Continuous search samples each operation’s parameters, checks preconditions, simulates effects, and accepts a sequence only when the goal is achieved. Operations instantiated from a mechanism use that mechanism’s specialized sampler. Algorithm 2 preserves the exact search structure.
A.2 Briefly Dynamic Manipulation
Briefly dynamic manipulation allows objects to accelerate and settle while the robot executes position-controlled motions. Examples include nonrigid contact during transport, sliding after placement, and a support object tipping when another object is added. This is broader than static contact reasoning, but it is not unrestricted dynamic trajectory optimization.
A.3 Basis Operations
Basis schemas distinguish free motion, contact motion, grasping, and placing. The move-with-contact example specifies a held tool, target, support, contact parameters, and joint-space trajectory, then updates robot and object poses through execution. Figure 8 retains the original symbolic definition alongside its contact graph.
A.4 Samplers for Generic Operations
Grasp proposals sample opposing object surfaces, check gripper width, and test inverse-kinematics reachability. Placement proposals align a held surface with an upward-facing support surface. Contact proposals align opposing surface normals while respecting the support direction. A bidirectional rapidly exploring tree connects arm configurations; pushing chooses a distance uniformly between 5 and 25 cm and follows corresponding Cartesian waypoints. These generic geometric primitives supply the candidate set that the learned classifiers reorder.
Appendix B Experiment Detail
The experiment appendix specifies object choices and perturbation ranges, diagnoses sampling effort, and reports a complete physical perception-and-replanning pipeline.
B.1 Mechanism Learning Setup
The six mechanisms use a small set of object models selected for their roles: thin plates/documents for edge or balance tasks, ladles/spoons/calipers as hooks, heavy objects for levers, slender tools for poking, and elongated blockers. Every object starts near a manually defined feasible canonical pose, perturbed by at most 5 cm and 15 degrees. Consequently, the randomized evaluation explores local configuration changes around feasible setups rather than unrestricted clutter or placements.
B.2 Sampler Learning
The hook-use breakdown shows that generic search spends most proposals on tool grasps and tool–target contacts. Learned rankings reduce these bottlenecks substantially, while other stages already require few samples. This explains why mechanism-specific geometric preferences can matter even after the discrete macro is known.
B.3 Physical Robot Deployment
A Franka Panda and calibrated RealSense D435 reconstruct tabletop objects from depth. The pipeline removes background/table points, detects or clusters objects, uses segmentation, and completes partial shapes by extending their bottoms to the table. Reconstructed meshes enter PyBullet for planning. Execution replans after pushing or placement and after detectable failed grasps, so the physical result includes closed-loop state updates rather than one open-loop simulation plan.
Each of two tasks is tested ten times with the same object set in different configurations. Edge pushing succeeds in 80%, with both failures caused by pushing the plate off the table; banana-hook use succeeds in 70%. Tip grasps can be unstable, and completed geometry can create nonexistent contact surfaces. The system cannot recover reliably from such false geometric contacts; replanning helps mainly when a failed grasp is detectable.
| Task | push-to-edge(plate) | hook-use(banana, {ball, apple}) |
| Succ. Rate | 80% | 70% |