Literature
Display
← All papers

Learning Reusable Manipulation Strategies

Jiayuan Mao · Joshua B. Tenenbaum · Tomás Lozano-Pérez · Leslie Pack Kaelbling

· CoRL 2023 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

A manipulation strategy can be reused by preserving its contact sequence while relearning how to choose grasps, placements, and motions for new objects.

At a glance

The system turns one contact-rich demonstration into a symbolic macro, then uses simulated self-play to learn continuous-parameter rankings for that macro. A task-and-motion planner combines these learned mechanisms with ordinary manipulation operations. Six simulated mechanisms and two compositions show large search-efficiency gains; limited physical tests demonstrate edge pushing and tool-assisted reaching. One demonstration supplies the abstract sequence, while simulator trials, object models, and generic planners supply substantial additional knowledge.

1 Introduction

The useful part of a tool-use demonstration may be its sequence of contacts rather than its exact trajectory or object category. A ladle-hooking demonstration can specify grasping a tool, contacting a target, putting the tool down, and grasping the recovered target. The paper follows explanation-based manipulation learning [2]: extract that structure, specialize continuous sampling through self-play, and expose the result as a reusable planning operator.

Refer to caption
Figure 1: A single demonstration can teach a reusable multi-step manipulation strategy.

2 Planning with Contacts and Mechanisms

Generic contact operations can in principle express these strategies, but searching their discrete sequences and continuous parameters is expensive. Learned mechanisms shorten the effective search horizon and focus parameter generation while retaining compatibility with the underlying planner.

Original figure
Figure 2: (a) Planning with basis manipulation operations usually involves a search in the hybrid space of discrete contact mode families (free, holding, pushing, etc.) and continuous parameters (grasping pose, moving trajectories, etc.). (b) Learning mechanisms can help reduce the search space by specifying a sequence of discrete transitions between contact mode families.

2.1 Basic Domain Representation

World state includes known object shapes and poses, robot state, and holding/support relationships. A parameterized operator specifies arguments, preconditions, effects, and a sampler. Grounding its object arguments and continuous values yields an executable action. This representation follows task-and-motion planning [5]; its symbolic relations organize the search but do not replace geometric or physical feasibility checks.

Name Args. Description
transit ⟨\langleempty⟩\rangle Move the robot without object collisions.
transit-cont ?x, ?s Move without holding anything, but have a contact with ?x. ?x is supported by ?s.
grasp ?x, ?s Grasp the object ?x that is currently supported by ?s.
place ?x, ?s Place object ?x onto ?s. ?x should be currently held by the robot.
move ?x Move while holding ?x.
move-cont ?x, ?y, ?s Move while holding ?x and ?x have contact with ?y. ?y is supported by ?s.
Table 1: The list of basis operations. Continuous parameters are omitted. “cont” refers to “contact.”
Original figure
Figure 3: The syntax used to model basis operations and mechanisms.

2.2 Mechanisms

A mechanism contains arguments, an initial contact pattern, a desired certified effect, an ordered sequence of primitive operations, and specialized samplers. For hook use, the macro grasps the tool, moves it against the target, places it down, then grasps the target. Object names become variables, enabling reuse with suitable new shapes rather than replaying fixed demonstration coordinates.

2.3 Planning with Basis Operators and Mechanisms

A bilevel planner, inspired by PDDLStream [6], searches discrete plans with a Fast Forward heuristic, then samples grasps, contacts, and trajectories. Simulation checks grounded action effects; unsuccessful continuous search causes backtracking. Learned macros enter the same operator set. Position-controlled actions may trigger transient object motion before settling, allowing sliding and lever behavior beyond a strictly quasi-static model.

3 Learning New Mechanisms

Learning requires a demonstration containing actions and contact information, a human-specified goal, and a simulator that samples initial configurations where the mechanism is applicable. The system extracts a macro, searches within that sequence on new configurations, trains parameter-scoring models from successful and unsuccessful samples, and adds the resulting mechanism to the planner.

Refer to caption
Figure 4: Learning process: extract contact modes and goals from demonstration; train specialized sampler via trial-and-error in new situations; add new mechanism to planner.

3.1 Extraction of Preconditions and Operation Sequence

Demonstrations are segmented into free, holding, and pushing contact regimes. Each segment records hand–object, indirect object–object, and supporting contacts. The initial graph becomes the mechanism precondition; replacing concrete object identities with variables yields its abstract schema. This abstraction depends on detailed contact labels, which are assumed available rather than inferred robustly from ordinary video.

3.2 Sampler Learning

Self-play fixes the demonstrated discrete sequence but explores object assignments and continuous choices. Samples appearing in a successful plan are labeled positive; other sampled choices are labeled negative. Binary classifiers rank batches from generic samplers, conditioned on previously chosen parameters, allowing early backtracking when a step fails. These scores are learned search heuristics, not certified probabilities that an isolated parameter is always feasible. The paper mentions binary cross-entropy but supplies no displayed loss formula in either the HTML or PDF.

Algorithm 1 Sampler Learning Algorithm
1: procedure SamplerLearning(env,m\textit{env},\textit{m})
2:   Initialize classifiers fif_{i} and dataset 𝒟i←∅\mathcal{D}_{i}\leftarrow\emptyset for each continuous parameters ii in m.
3:   for each iteration do
4:    s0∼env.reset​()s_{0}\sim\textit{env}.\textit{reset}() ⊳\triangleright Sample an initial configuration
5:    gm←RandomGrounding​(m)\textit{gm}\leftarrow\text{RandomGrounding}(m) ⊳\triangleright Apply the mechanism on a random set of objects
6:    plan←\textit{plan}\leftarrow ContinuousSearch(s0,gm.certified,gm.actionss_{0},\textit{gm}.\textit{certified},\textit{gm}.\textit{actions})
7:    for each continuous parameter ii do
8:      for all θ\theta sampled in ContinuousSearch for parameter ii do
9:       if θ\theta is in plan then Add (θ,1)(\theta,1) to 𝒟i\mathcal{D}_{i} ⊳\triangleright Successful samples
10:       else Add (θ,0)(\theta,0) to 𝒟i\mathcal{D}_{i} ⊳\triangleright Failed samples            
11:      Update classifier fif_{i} using LL and 𝒟i\mathcal{D}_{i}      

3.3 Implementation

Geometry-based samplers propose grasps, placements, contact surfaces, and arm trajectories. Contact sampling aligns points and opposing normals while avoiding pushes into the supporting surface. Specialized scorers encode shapes with PointNet++ [8] and encode poses/contact quantities with multilayer networks, then predict a score. Each classifier uses one hundred training samples. Learned trajectory scoring covers contact direction and distance, not complete grasp/place arm paths, which remain the motion planner’s responsibility.

4 Experiments

The main experiments use PyBullet to compare individual mechanisms and their composition. Physical deployment is reported separately in Appendix B.3, including perception, replanning, and failure modes.

4.1 Learning Mechanisms from Single Demonstrations

Six tasks cover edge pushing, hooking, lever use, poking through a tunnel, center-of-mass placement, and blocking a slope. Each demonstration is a manually scripted execution. Training and testing sample object configurations selected to make the mechanism feasible, including novel instances and categories.

Over ten trials with a ten-minute timeout, macros plus learned samplers reduce average hook solution time to 3.84 seconds versus 30.82 for macros alone and over 600 for basis operators. Center-of-mass placement falls to 0.97 seconds versus 17.58 and 19.30. Lever use is already easy with the macro: 1.55 seconds with learned sampling versus 1.38 without. Poking remains difficult at 97.76 seconds because its collision-free arm-motion region is narrow. The gain is task-dependent search efficiency, not uniformly faster learned sampling.

Refer to caption
Figure 5: Illustration of six mechanisms learned by our algorithm. Top row: initial configuration. Bottom row: solution found by our sampling-based planner. P: Plate. B: Block. S: Slope.
Method Edge Hook Lever Poking CoM Slope&Blocker
Basis Ops Only
89.45±5.53\pm 5.53
>>600 523.18±9.22\pm 9.22 >>600 19.30±2.82\pm 2.82 >>600
Ours (Macro)
8.34±2.57\pm 2.57
30.82±5.78\pm 5.78 1.38±0.31\pm 0.31 494.30±50.01\pm 50.01 17.58±1.27\pm 1.27 148.57±10.30\pm 10.30
Ours (Macro+Sampler) 0.57±0.05\pm 0.05 3.84±1.56\pm 1.56 1.55±0.29\pm 0.29 97.76±10.67\pm 10.67 0.97±0.09\pm 0.09 4.11±0.94\pm 0.94
Table 2: Average solution time and stderr of 10 trials, All methods have a 10 mins timeout.

4.2 Planning with Learned Mechanisms

Two new compositions require pushing a thin caliper to an edge before grasping and hooking, or hooking a distant object before placing it on a blocked slope. Previously learned mechanisms are reused without additional training. With ten trials per task and a ten-minute timeout, complete success is 100% and 90% for macros plus samplers, 20% and 0% for macros alone, and 0% for basis operations. Table 3 reports success rates, despite the surrounding source prose calling them planning times. These are deliberately feasible composed tasks rather than arbitrary long-horizon environments.

Refer to caption
Figure 6: Illustration of compositional tasks.
Method Task 1 Task 2
Basis Ops Only 0% 0%
Ours (Macro) 20% 0%
Ours (Macro + Sampler) 100% 90%

Table 3: Average success rate on novel compositional tasks in 10 runs. Timeout is 10 minutes.

5 Related Work

The approach extends sampling-based task-and-motion planning with learned contact-sequence macros. Contact-mode planners supply a related representation, and tool-affordance learning supplies related geometric generalization. The distinguishing combination is extracting a reusable sequence from one labeled demonstration, learning its parameter preferences through simulation, and composing it within a general planner.

6 Limitations and Conclusions

Detailed demonstration contacts and reliable simulation are the main limitations. Inaccurate geometry or physics can defeat transferred mechanisms, and real-world fine-tuning is proposed. The representation supports new instances and compositions within its available operators, but broader contact sensing, compliance, and forceful actions are extensions rather than demonstrated capabilities.

Acknowledgement.

The authors acknowledge reviewer feedback and support from US research agencies, MIT research programs, the MIT–IBM Watson AI Lab, and the Boston Dynamics Artificial Intelligence Institute.

Supplementary material expands the planning implementation, generic geometric samplers, evaluation distributions, and physical robot experiments.

Appendix A Method Detail

The method appendix retains the complete bilevel planning algorithm and explains how contact schemas and generic samplers handle transient physical interactions.

A.1 Planning Algorithm

Discrete search enumerates both primitive and mechanism operators. Continuous search samples each operation’s parameters, checks preconditions, simulates effects, and accepts a sequence only when the goal is achieved. Operations instantiated from a mechanism use that mechanism’s specialized sampler. Algorithm 2 preserves the exact search structure.

Algorithm 2 Bilevel Search Algorithm
1: procedure BilevelSearch(s0,g,operator_schemass_{0},g,\text{operator\_schemas})
2:   plan_gen←SymbolicSearch​(s0,g,operator_schemas)\textit{plan\_gen}\leftarrow\text{SymbolicSearch}(s_{0},g,\text{operator\_schemas}) ⊳\triangleright Explore discrete plans
3:   for all plan∈plan_gen\textit{plan}\in\textit{plan\_gen} do ⊳\triangleright For all candidate sequences of basis operations
4:    ContinuousSearch(s0,g,plans_{0},g,\textit{plan})   
5: procedure ContinuousSearch(s0,g,plans_{0},g,\textit{plan})
6:   grounded_plan←∅\textit{grounded\_plan}\leftarrow\emptyset;  s←s0s\leftarrow s_{0}
7:   for all op∈p​l​a​n\textit{op}\in plan do
8:    for all arg∈op.args\textit{arg}\in\textit{op}.\textit{args} do
9:      arg←InvokeSampler(op.sampler)\textit{arg}\leftarrow\text{InvokeSampler}(\textit{op}.\textit{sampler}) ⊳\triangleright Generate continuous parameters    
10:    if CheckPrecondition​(op,s)\text{CheckPrecondition}(\textit{op},s) then
11:      s←𝒯⁡(s,op)s\leftarrow{\mathcal{T}}(s,\textit{op})⊳\triangleright Simulate the operator with sampled parameters.
12:      grounded_plan←grounded_plan∪{op}\textit{grounded\_plan}\leftarrow\textit{grounded\_plan}\cup\{\textit{op}\}
13:    else break      
14:   if IsGoalAchieved​(grounded_plan,g)\text{IsGoalAchieved}(\textit{grounded\_plan},g) then
15:    return g​r​o​u​n​d​e​d​_​p​l​a​ngrounded\_plan ⊳\triangleright Return the first plan that achieves the goal   
16:   return empty

A.2 Briefly Dynamic Manipulation

Briefly dynamic manipulation allows objects to accelerate and settle while the robot executes position-controlled motions. Examples include nonrigid contact during transport, sliding after placement, and a support object tipping when another object is added. This is broader than static contact reasoning, but it is not unrestricted dynamic trajectory optimization.

Refer to caption
Figure 7: Illustration of three briefly-dynamic manipulation scenarios in the paper.

A.3 Basis Operations

Basis schemas distinguish free motion, contact motion, grasping, and placing. The move-with-contact example specifies a held tool, target, support, contact parameters, and joint-space trajectory, then updates robot and object poses through execution. Figure 8 retains the original symbolic definition alongside its contact graph.

Refer to caption
Figure 8: The modeling of the robot basis operation move-with-contact using a STRIPS-like syntax.

A.4 Samplers for Generic Operations

Grasp proposals sample opposing object surfaces, check gripper width, and test inverse-kinematics reachability. Placement proposals align a held surface with an upward-facing support surface. Contact proposals align opposing surface normals while respecting the support direction. A bidirectional rapidly exploring tree connects arm configurations; pushing chooses a distance uniformly between 5 and 25 cm and follows corresponding Cartesian waypoints. These generic geometric primitives supply the candidate set that the learned classifiers reorder.

Appendix B Experiment Detail

The experiment appendix specifies object choices and perturbation ranges, diagnoses sampling effort, and reports a complete physical perception-and-replanning pipeline.

B.1 Mechanism Learning Setup

The six mechanisms use a small set of object models selected for their roles: thin plates/documents for edge or balance tasks, ladles/spoons/calipers as hooks, heavy objects for levers, slender tools for poking, and elongated blockers. Every object starts near a manually defined feasible canonical pose, perturbed by at most 5 cm and 15 degrees. Consequently, the randomized evaluation explores local configuration changes around feasible setups rather than unrestricted clutter or placements.

B.2 Sampler Learning

The hook-use breakdown shows that generic search spends most proposals on tool grasps and tool–target contacts. Learned rankings reduce these bottlenecks substantially, while other stages already require few samples. This explains why mechanism-specific geometric preferences can matter even after the discrete macro is known.

Refer to caption
Figure 9: Breakdown of samples produced by different samplers for the hook-using task.

B.3 Physical Robot Deployment

A Franka Panda and calibrated RealSense D435 reconstruct tabletop objects from depth. The pipeline removes background/table points, detects or clusters objects, uses segmentation, and completes partial shapes by extending their bottoms to the table. Reconstructed meshes enter PyBullet for planning. Execution replans after pushing or placement and after detectable failed grasps, so the physical result includes closed-loop state updates rather than one open-loop simulation plan.

Each of two tasks is tested ten times with the same object set in different configurations. Edge pushing succeeds in 80%, with both failures caused by pushing the plate off the table; banana-hook use succeeds in 70%. Tip grasps can be unstable, and completed geometry can create nonexistent contact surfaces. The system cannot recover reliably from such false geometric contacts; replanning helps mainly when a failed grasp is detectable.

Refer to caption
Figure 10: Our real-world setup.
Task push-to-edge(plate) hook-use(banana, {ball, apple})
Succ. Rate 80% 70%
Table 4: Success rate of our physical robot setup on two representative mechanisms.
Concept demonstration

Open image ↗