Literature
Display
← All papers

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation

Renhao Wang · Jiayuan Mao · Joy Hsu · Hang Zhao · Jiajun Wu · Yang Gao

· ICLR 2023 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

ProgramPort parses manipulation instructions into executable programs whose visual-grounding modules are separated from action modules. Frozen pretrained visual-language features identify concepts, while demonstration learning fits manipulation behavior. Simulation tests show improved generalization to unseen concepts and combinations, with a small real-robot validation. The method relies on a manually specified grammar and primitive library; it does not learn unrestricted language or arbitrary actions.

1 Introduction

Fine-tuning a monolithic vision-language manipulation model can entangle task-specific actions with general visual concepts. ProgramPort instead grounds individual words and composes their masks through a structured program, aiming to preserve pretrained recognition while learning control from fewer demonstrations.

Refer to caption
Figure 1: Zero-Shot and Compositional Generalization: Our framework, programport, is capable of generalizing to combinations of unseen objects and manipulation behaviors at test time.
Refer to caption
Figure 2: (a) Existing manipulation models entangle visual grounding from pretrained VL models with task-specific policy learning. (b) Our disentangled approach leverages program-based representation of language semantics to yield faithful VL-grounding and improved genearlization.

2 Background

Observations contain a top-down RGB-D image and a language goal. A policy predicts discretized planar end-effector poses for primitives such as picking, placing, and pushing. Transporter-style action maps select locations and rotations. The setting assumes predefined primitives and expert observation–action demonstrations. [23], [2].

Equation 1
𝒯pick=arg​max(u,v)∈H×W⁡𝒬pick​((u,v)∣𝐱t,𝐥t),𝒯place=arg​maxτi∈{τ}⁡𝒬place​(τi|𝐱t,𝐥t,𝒯pick)\mathcal{T}_{\textit{pick}}=\argmax_{(u,v)\in H\times W}\mathcal{Q}_{\textit{pick}}\left((u,v)\mid{\mathbf{x}}_{t},{\mathbf{l}}_{t}\right),\quad\mathcal{T}_{\textit{place}}=\argmax_{\tau_{i}\in\{\tau\}}\mathcal{Q}_{\textit{place}}\left(\tau_{i}|{\mathbf{x}}_{t},{\mathbf{l}}_{t},\mathcal{T}_{\textit{pick}}\right) (1)

3 ProgramPort

A semantic parser builds a hierarchy of visual and action operators. Visual modules are reused without fine-tuning; trainable action modules consume their grounding maps. Program execution converts the instruction into parameters for the available robot primitives. [33], [20].

Refer to caption
Figure 3: The execution of a manipulation program. A CCG parser first generates a manipulation program. The task-agnostic visual grounding modules then identify visual semantics associated with the desired objects and targets. Finally, the task-specific action modules leverage these grounding results to output the manipulation control parameters.

3.1 Programmatic Representations of Manipulation Instructions

A predefined CCG lexicon maps words to syntactic types and semantic functions. Filter operators identify object properties; goal and do operators specify relations and actions. Unknown words receive a syntactic type that permits the sentence to parse, then a likely semantic form. This supports composition within the grammar, but contextual syntax may not uniquely determine a new word’s meaning.

3.2 ProgramPort Architecture

MaskCLIP-style processing converts frozen CLIP features into spatial concept masks. Successive property filters intersect masks using elementwise minima. Action modules multiply grounding masks into intermediate decoder features and final action maps so that the network cannot simply ignore the indicated objects. Placement is conditioned on the chosen pick location and target relation.

Equations 2–3
ℱgrounding​(𝐱t,𝐜)=∑[tile​(ℋ⁡(𝐜))⊙𝒞l​(𝒞v​(𝒢⁡(𝐱t)))]∈ℝH′×W′,\mathcal{F}_{\textit{grounding}}({\mathbf{x}}_{t},{\mathbf{c}})=\sum\left[{\textsc{{tile}}}\left(\mathcal{H}({\mathbf{c}})\right)\odot\mathcal{C}_{l}\left(\mathcal{C}_{v}\left(\mathcal{G}({\mathbf{x}}_{t})\right)\right)\right]\in\mathbb{R}^{H^{\prime}\times W^{\prime}}, (2)
𝒯place=arg​maxτi∈{τ}⁡Up​(ℱgrounding)⊙𝒬place​(τi|𝐱t,𝐥t,𝒯pick,ℱgrounding),\begin{split}\mathcal{T}_{\textit{place}}=\argmax_{\tau_{i}\in\{\tau\}}\text{Up}\left(\mathcal{F}_{\textit{grounding}}\right)\,\,\odot\,\,\mathcal{Q}_{\textit{place}}\left(\tau_{i}|{\mathbf{x}}_{t},{\mathbf{l}}_{t},\mathcal{T}_{\textit{pick}},\mathcal{F}_{\textit{grounding}}\right),\end{split} (3)

Example.

The example program independently grounds a blue hexagon and an orange box, intersects their noun and color masks, then predicts a placement consistent with the relation in. Training minimizes cross-entropy for demonstrated pick and place choices while preserving the modular execution structure.

Equation 4
ℒ=−𝔼𝒯pick[logsoftmax(𝒬pick((u,v)|𝐱t,𝐥t)]−𝔼𝒯place[logsoftmax(𝒬place(τi|𝐱t,𝐥t,𝒯pick)]\mathcal{L}=-\mathbb{E}_{\mathcal{T}_{\textit{pick}}}\left[\log\mathrm{softmax}(\mathcal{Q}_{\textit{pick}}((u,v)|{\mathbf{x}}_{t},{\mathbf{l}}_{t})\right]-\mathbb{E}_{\mathcal{T}_{\textit{place}}}\left[\log\mathrm{softmax}(\mathcal{Q}_{\textit{place}}(\tau_{i}|{\mathbf{x}}_{t},{\mathbf{l}}_{t},\mathcal{T}_{\textit{pick}})\right] (4)

4 Experiments

Ten Ravens/CLIPort manipulation tasks use a simulated suction-gripper robot. Training uses 100 or 1,000 demonstrations, 200,000 iterations, and AdamW; evaluation uses 100 test episodes. Scores range from zero to 100 and generally measure partial episode completion, with task-specific definitions in the appendix. Single-task and joint multitask models are compared.

4.1 Zero-Shot Generalization

Held-out object categories, colors, and descriptors test zero-shot grounding; missing test vocabulary is also removed from the parser lexicon. ProgramPort improves many tasks and often benefits from multitask training. Rope alignment remains difficult because pretrained grounding struggles with abstract spatial descriptors. The table contains exceptions, so the improvements are not uniform across every task and training size.

packing-
box-pairs
packing-google-
objects-seq
packing-google-
objects-group
separating-
small-piles
separating-
large-piles
# demonstrations 100 1000 100 1000 100 1000      100 1000 100 1000
cliport (single) 67.1 72.0 64.1 70.8 79.3 82.1      75.9 74.3 62.4 66.9
cliport (multi) 77.1 72.3 66.9 71.6 77.0 81.7      65.9 63.7 54.2 59.1
programport (single) 72.6 78.0 78.4 79.7 79.0 81.2      76.8 77.9 63.8 69.5
programport (multi) 82.3 85.7 73.5 84.0 80.7 83.9      77.0 78.8 63.0 68.2
align-rope packing-shapes assembling-kits put-blocks-in-bowls towers-of-hanoi
# demonstrations 100 1000 100 1000 100 1000      100 1000 100 1000
cliport (single) 84.8 94.1 42.0 38.0 32.2 34.7      36.8 29.9 94.6 99.0
cliport (multi) 81.2 70.3 38.0 32.0 30.2 26.9      55.9 46.7 75.2 69.1
programport (single) 77.5 86.9 52.0 54.0 40.6 42.4      38.2 38.7 97.6 98.4
programport (multi) 78.4 89.1 57.0 63.0 39.2 46.8      54.2 57.7 97.0 96.3
Table 1: Zero-Shot Generalization. Training and validation splits for all tasks involve different objects, object properties, or other spatial descriptors.
Refer to caption
Figure 4: Qualitative pick and place affordances for programport and cliport. programport compositionally reasons over independent visual semantics in a zero-shot fashion, correctly picking and placing the target object. cliport identifies both pick object and place location incorrectly.

4.2 Compositional Generalization of Visual Grounding and Behaviors

Four additional tasks combine familiar shapes, colors, spatial terms, and manipulation behaviors without direct training on those combinations. At 1,000 demonstrations, multitask ProgramPort scores 76.3, 67.1, 70.5, and 40.2, compared with multitask CLIPort’s 18.2, 8.3, 41.0, and 7.6. The hardest case pushes irregular shapes using a pushing skill trained on blocks; the results are completion scores, not uniformly binary success rates.

packing-color-box packing-location-box separating-location-piles pushing-shapes
100 1000 100 1000 100 1000 100 1000
cliport (single) 23.0 24.0 11.0 13.7 48.1 43.8 2.4 3.2
cliport (multi) 19.0 18.2 10.0 8.3 39.9 41.0 2.7 7.6
programport (single) 64.0 68.0 56.0 62.5 69.2 72.4 32.3 36.9
programport (multi) 72.0 76.3 62.0 67.1 68.4 70.5 36.6 40.2
Table 2: Compositional Generalization. Models can learn different object shapes, colors, and manipulation properties from different tasks. We show that only programport hierarchically composes these learned concepts and behaviors together to perform well in more complex tasks.

4.3 Ablation: Disentanglement of Action and Perception

Replacing predicted grounding maps with ground-truth masks at test time sharply improves all five evaluated tasks. This shows that the action modules can use accurate perception and identifies grounding as a substantial remaining bottleneck. The intervention uses privileged test information and is an ablation, not the deployed system’s performance.

Method Semantic Supervision
packing-
shapes
packing-google-
objects-seq
assembling-
kits
separating-
small-piles
put-blocks-
in-bowls
100 1000 100 1000 100 1000 100 1000 100 1000
programport (single) MaskCLIP 52.0 54.0 78.4 79.7 40.6 42.4 76.8 77.9 38.2 38.7
GT 97.0 98.0 95.8 95.9 92.0 93.4 80.6 86.4 97.2 96.3
programport (multi) MaskCLIP 57.0 63.0 73.5 84.0 40.6 42.4 77.0 78.8 54.2 57.7
GT 96.0 98.0 96.2 96.4 91.0 94.5 81.2 87.2 99.3 97.7
Table 3: Disentanglement of visual semantics and action. Replacing maskclip grounding maps with ground truth segmentation maps of objects and targets yields near-perfect manipulation.

5 Related Work

The approach combines language-conditioned manipulation with neural-symbolic program execution. Its main distinction is grounding individual visual concepts and action arguments before composition, rather than passing a single whole-instruction embedding into a control network.

6 Conclusion

Program structure improves interpretability, data efficiency, and compositional transfer in the tested primitive-based setting. Open-ended language, ambiguous unknown words, object-specific interaction geometry, and more difficult articulated or deformable manipulation remain limitations.

Reproducibility Statement

The appendix specifies the language, task construction, success measures, and model architecture; the main experiment section supplies principal optimization settings.

Acknowledgments

The authors acknowledge academic, governmental, and industry support from Stanford, Tsinghua, Chinese research programs, Analog Devices, JPMorgan Chase, and Salesforce.

Appendix A Appendix

The appendix details the language, task definitions, additional experiments, and module architectures.

A.1 Domain-Specific Language

The DSL includes scene selection, filtering, relations, goals, actions, object union, and plan combination. Scene returns a uniform attention map; union and action concatenation take elementwise maxima. The preserved tables define every operator signature and type.

Operation Signature Semantics
Scene () ⟶\longrightarrow Object Return all objects in visual scene.
Filter (Object, ObjProp) ⟶\longrightarrow Object Return all objects satisfying a vision-language property.
Relate (Object, Object, ObjRel) ⟶\longrightarrow Object Return all objects satistfying a relation between target and reference objects.
Do ({Goal}, Action) ⟶\longrightarrow Plan Perform an action according to a set of goals.
Goal (Object, Object, ObjRel) ⟶\longrightarrow Goal Identify a manipulation goal for an object and target relation.
ObjUnion (Object, Object) ⟶\longrightarrow Object Return the union of two (sets) of objects.
ActionConcat (Plan, Plan) ⟶\longrightarrow Plan Jointly perform two sets of actions.
Table 4: All operations in our DSL for vision-language manipulation.
Type Example Semantics
Object Block, Rope, etc. Manipulation objects.
ObjectProp Red, Dark, etc. Object properties.
ObjRel In, On, etc. Object relationships.
Goal - Manipulation target such as distributions over SE(2) poses.
Plan - End-effector control parameters in SE(2).
Action Pack, Push, etc. Manipulation primitives.
Table 5: All types in our DSL for vision-language manipulation.

A.2 Dataset Details

The benchmark uses 20 synthetic shapes, 56 scanned household objects, and 11 colors. Object train/test sets are disjoint; only red, green, and blue overlap between color splits. The following task definitions distinguish volume, object-count, pose, and binary success metrics.

A.2.1 Basic Tasks

The basic tasks supply individual object concepts, spatial relations, and manipulation behaviors.

Packing Box Pairs.

Pack all boxes of two requested colors into a larger box; score is the fraction of their total volume packed. One requested color may be absent.

Packing Google Objects Seq.

Pack a named scanned object at each step of a one-to-five-step episode; score is the packed fraction of the requested objects’ volume.

Packing Google Objects Group.

Pack every scanned object in a requested category; score is the fraction of that category’s volume inside the box.

Separating Small Piles.

Push 50 small blocks into the zone of a specified color; score is the fraction of blocks in the correct zone.

Separating Large Piles.

Push 15 larger, heavier, higher-friction blocks into a specified zone; score is the fraction reaching that zone.

Align Rope.

Align a 20-bead articulated rope between requested corners of a three-sided square; score measures agreement between bead poses and the target line segment.

Packing Shapes.

Choose one requested shape among four distractors and pack it into a box; success is binary.

Assembling Kits.

Place five shapes into matching kit holes, using language to disambiguate repeated shapes; score measures the fraction and quality of pose matches.

Put Blocks in Bowls.

Place each block of the requested color into a bowl of the requested color; score is the fraction correctly placed, with at most one block per bowl.

Towers of Hanoi.

Execute a predetermined seven-move solution for three-ring Towers of Hanoi; score is the fraction of correct ring placements. The sequence comes from a solver rather than learned planning.

Packing Prepositions.

Pack a requested shape into the box located relative to a distractor object; success is binary and tests relational grounding.

A.2.2 Compositional Tasks

The compositional tasks combine these ingredients in settings withheld from training.

Packing Color Box.

Pack a specified shape into a box of a specified color. Training supplied shapes and colors separately and used only one brown packing box; success is binary.

Packing Location Box.

Pack a specified shape into the box at a named location. The test combines packing with spatial descriptors learned in other tasks; success is binary.

Separating Location Piles.

Push a pile of 50 small blocks into a zone identified by spatial relation rather than color; score is the fraction of blocks in the target zone.

Pushing Shapes.

Push a shape specified by color and geometry into a zone specified by color and location. This combines longer descriptions with object geometry and friction not encountered in pushing training.

Packing Nested Prepositions.

Pack into a box identified by nested spatial relations between distractor shapes. This tests longer relational composition, with binary success.

A.3 Experiments with Prepositional Phrases

With the relation operator, ProgramPort scores 45 versus CLIPort’s 28 on packing with prepositions, and 31 versus 17 on nested prepositions. Both remain far from perfect, illustrating the difficulty of relational grounding.

packing-prepositions packing-nested-prepositions
CLIPort 28.0 17.0
ProgramPort 45.0 31.0
Table 6: Experiments with relate operator. packing-prepositions evaluates zero-shot generalization whereas packing-nested-prepositions evaluates compositional generalization.

A.4 Real World Experiments

A Franka Panda dataset contains 40 household-object pick-and-place demonstrations: 30 for training and ten for validation. Separate test sets of ten seen-object and ten unseen-object cases yield 70% and 80% binary success. These small samples demonstrate feasibility rather than a precise generalization estimate.

packing-objects-seen packing-objects-unseen
ProgramPort 70.0 80.0
Table 7: Real world packing objects task. The seen suffix involves new instructions containing the same objects seen in the training set, whereas the unseen suffix evaluates zero-shot generalization to new instructions containing novel objects.
Refer to caption
Figure 5: Real world packing-shapes task. Pick locations on objects are highlighted in green, and place locations are highlighted in red.

A.5 Additional Affordance Visualizations

The retained affordance visualizations compare concept-level grounding and final pick/place maps across packing, bowl placement, and pushing. ProgramPort exposes intermediate object/property masks, making several localization errors easier to inspect.

Refer to caption
Figure 6: Packing-shapes.
Refer to caption
Figure 7: Put-blocks-in-bowls.
Refer to caption
Figure 8: Separating-small-piles.
Refer to caption
Figure 9: Qualitative affordance evaluations for different zero-shot and compositional tasks.

A.6 Model Details

The architecture diagrams show how CLIP’s value projection and final linear map become spatial 1×1 convolutions. Action decoders receive resized masks by multiplication at multiple levels. The appendix’s ResNet illustration and the main section’s ViT-B/16 specification describe different encoder details; the preserved diagrams retain the source’s presentation.

Refer to caption
Figure 10: Visual grounding module example: filter.
Equation 5
𝐯=ℱ⁡(∑i,jsoftmax⁡(Embq​(AvgPool​(𝒱))⋅Embk​(𝒱i,j)TC)​Embv​(𝒱i,j)){\mathbf{v}}=\mathcal{F}\left(\sum\limits_{i,j}\mathrm{softmax}\left(\frac{\text{Emb}_{q}\left(\text{AvgPool}\left(\mathcal{V}\right)\right)\cdot\text{Emb}_{k}(\mathcal{V}_{i,j})^{T}}{C}\right)\text{Emb}_{v}(\mathcal{V}_{i,j})\right) (5)
Refer to caption
(a) 𝒬pick\mathcal{Q}_{\textit{pick}} submodule
Refer to caption
(b) 𝒬place\mathcal{Q}_{\textit{place}} submodule
Figure 11: Action module example: goal.
Concept demonstration

Open image ↗