At a glance
ProgramPort parses manipulation instructions into executable programs whose visual-grounding modules are separated from action modules. Frozen pretrained visual-language features identify concepts, while demonstration learning fits manipulation behavior. Simulation tests show improved generalization to unseen concepts and combinations, with a small real-robot validation. The method relies on a manually specified grammar and primitive library; it does not learn unrestricted language or arbitrary actions.
1 Introduction
Fine-tuning a monolithic vision-language manipulation model can entangle task-specific actions with general visual concepts. ProgramPort instead grounds individual words and composes their masks through a structured program, aiming to preserve pretrained recognition while learning control from fewer demonstrations.
2 Background
Observations contain a top-down RGB-D image and a language goal. A policy predicts discretized planar end-effector poses for primitives such as picking, placing, and pushing. Transporter-style action maps select locations and rotations. The setting assumes predefined primitives and expert observation–action demonstrations. [23], [2].
Equation 1
| (1) |
3 ProgramPort
A semantic parser builds a hierarchy of visual and action operators. Visual modules are reused without fine-tuning; trainable action modules consume their grounding maps. Program execution converts the instruction into parameters for the available robot primitives. [33], [20].
3.1 Programmatic Representations of Manipulation Instructions
A predefined CCG lexicon maps words to syntactic types and semantic functions. Filter operators identify object properties; goal and do operators specify relations and actions. Unknown words receive a syntactic type that permits the sentence to parse, then a likely semantic form. This supports composition within the grammar, but contextual syntax may not uniquely determine a new word’s meaning.
3.2 ProgramPort Architecture
MaskCLIP-style processing converts frozen CLIP features into spatial concept masks. Successive property filters intersect masks using elementwise minima. Action modules multiply grounding masks into intermediate decoder features and final action maps so that the network cannot simply ignore the indicated objects. Placement is conditioned on the chosen pick location and target relation.
Equations 2–3
| (2) |
| (3) |
Example.
The example program independently grounds a blue hexagon and an orange box, intersects their noun and color masks, then predicts a placement consistent with the relation in. Training minimizes cross-entropy for demonstrated pick and place choices while preserving the modular execution structure.
Equation 4
| (4) |
4 Experiments
Ten Ravens/CLIPort manipulation tasks use a simulated suction-gripper robot. Training uses 100 or 1,000 demonstrations, 200,000 iterations, and AdamW; evaluation uses 100 test episodes. Scores range from zero to 100 and generally measure partial episode completion, with task-specific definitions in the appendix. Single-task and joint multitask models are compared.
4.1 Zero-Shot Generalization
Held-out object categories, colors, and descriptors test zero-shot grounding; missing test vocabulary is also removed from the parser lexicon. ProgramPort improves many tasks and often benefits from multitask training. Rope alignment remains difficult because pretrained grounding struggles with abstract spatial descriptors. The table contains exceptions, so the improvements are not uniform across every task and training size.
|
|
|
|
| ||||||||||||||||
| # demonstrations | 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | ||||||||||
| cliport (single) | 67.1 | 72.0 | 64.1 | 70.8 | 79.3 | 82.1 | 75.9 | 74.3 | 62.4 | 66.9 | ||||||||||
| cliport (multi) | 77.1 | 72.3 | 66.9 | 71.6 | 77.0 | 81.7 | 65.9 | 63.7 | 54.2 | 59.1 | ||||||||||
| programport (single) | 72.6 | 78.0 | 78.4 | 79.7 | 79.0 | 81.2 | 76.8 | 77.9 | 63.8 | 69.5 | ||||||||||
| programport (multi) | 82.3 | 85.7 | 73.5 | 84.0 | 80.7 | 83.9 | 77.0 | 78.8 | 63.0 | 68.2 | ||||||||||
| align-rope | packing-shapes | assembling-kits | put-blocks-in-bowls | towers-of-hanoi | ||||||||||||||||
| # demonstrations | 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | ||||||||||
| cliport (single) | 84.8 | 94.1 | 42.0 | 38.0 | 32.2 | 34.7 | 36.8 | 29.9 | 94.6 | 99.0 | ||||||||||
| cliport (multi) | 81.2 | 70.3 | 38.0 | 32.0 | 30.2 | 26.9 | 55.9 | 46.7 | 75.2 | 69.1 | ||||||||||
| programport (single) | 77.5 | 86.9 | 52.0 | 54.0 | 40.6 | 42.4 | 38.2 | 38.7 | 97.6 | 98.4 | ||||||||||
| programport (multi) | 78.4 | 89.1 | 57.0 | 63.0 | 39.2 | 46.8 | 54.2 | 57.7 | 97.0 | 96.3 | ||||||||||
4.2 Compositional Generalization of Visual Grounding and Behaviors
Four additional tasks combine familiar shapes, colors, spatial terms, and manipulation behaviors without direct training on those combinations. At 1,000 demonstrations, multitask ProgramPort scores 76.3, 67.1, 70.5, and 40.2, compared with multitask CLIPort’s 18.2, 8.3, 41.0, and 7.6. The hardest case pushes irregular shapes using a pushing skill trained on blocks; the results are completion scores, not uniformly binary success rates.
| packing-color-box | packing-location-box | separating-location-piles | pushing-shapes | |||||
| 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | |
| cliport (single) | 23.0 | 24.0 | 11.0 | 13.7 | 48.1 | 43.8 | 2.4 | 3.2 |
| cliport (multi) | 19.0 | 18.2 | 10.0 | 8.3 | 39.9 | 41.0 | 2.7 | 7.6 |
| programport (single) | 64.0 | 68.0 | 56.0 | 62.5 | 69.2 | 72.4 | 32.3 | 36.9 |
| programport (multi) | 72.0 | 76.3 | 62.0 | 67.1 | 68.4 | 70.5 | 36.6 | 40.2 |
4.3 Ablation: Disentanglement of Action and Perception
Replacing predicted grounding maps with ground-truth masks at test time sharply improves all five evaluated tasks. This shows that the action modules can use accurate perception and identifies grounding as a substantial remaining bottleneck. The intervention uses privileged test information and is an ablation, not the deployed system’s performance.
| Method | Semantic Supervision |
|
|
|
|
| |||||||||||||||
| 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | 100 | 1000 | ||||||||||||
| programport (single) | MaskCLIP | 52.0 | 54.0 | 78.4 | 79.7 | 40.6 | 42.4 | 76.8 | 77.9 | 38.2 | 38.7 | ||||||||||
| GT | 97.0 | 98.0 | 95.8 | 95.9 | 92.0 | 93.4 | 80.6 | 86.4 | 97.2 | 96.3 | |||||||||||
| programport (multi) | MaskCLIP | 57.0 | 63.0 | 73.5 | 84.0 | 40.6 | 42.4 | 77.0 | 78.8 | 54.2 | 57.7 | ||||||||||
| GT | 96.0 | 98.0 | 96.2 | 96.4 | 91.0 | 94.5 | 81.2 | 87.2 | 99.3 | 97.7 | |||||||||||
5 Related Work
The approach combines language-conditioned manipulation with neural-symbolic program execution. Its main distinction is grounding individual visual concepts and action arguments before composition, rather than passing a single whole-instruction embedding into a control network.
6 Conclusion
Program structure improves interpretability, data efficiency, and compositional transfer in the tested primitive-based setting. Open-ended language, ambiguous unknown words, object-specific interaction geometry, and more difficult articulated or deformable manipulation remain limitations.
Reproducibility Statement
The appendix specifies the language, task construction, success measures, and model architecture; the main experiment section supplies principal optimization settings.
Acknowledgments
The authors acknowledge academic, governmental, and industry support from Stanford, Tsinghua, Chinese research programs, Analog Devices, JPMorgan Chase, and Salesforce.
Appendix A Appendix
The appendix details the language, task definitions, additional experiments, and module architectures.
A.1 Domain-Specific Language
The DSL includes scene selection, filtering, relations, goals, actions, object union, and plan combination. Scene returns a uniform attention map; union and action concatenation take elementwise maxima. The preserved tables define every operator signature and type.
| Operation | Signature | Semantics |
| Scene | () Object | Return all objects in visual scene. |
| Filter | (Object, ObjProp) Object | Return all objects satisfying a vision-language property. |
| Relate | (Object, Object, ObjRel) Object | Return all objects satistfying a relation between target and reference objects. |
| Do | ({Goal}, Action) Plan | Perform an action according to a set of goals. |
| Goal | (Object, Object, ObjRel) Goal | Identify a manipulation goal for an object and target relation. |
| ObjUnion | (Object, Object) Object | Return the union of two (sets) of objects. |
| ActionConcat | (Plan, Plan) Plan | Jointly perform two sets of actions. |
| Type | Example | Semantics |
| Object | Block, Rope, etc. | Manipulation objects. |
| ObjectProp | Red, Dark, etc. | Object properties. |
| ObjRel | In, On, etc. | Object relationships. |
| Goal | - | Manipulation target such as distributions over SE(2) poses. |
| Plan | - | End-effector control parameters in SE(2). |
| Action | Pack, Push, etc. | Manipulation primitives. |
A.2 Dataset Details
The benchmark uses 20 synthetic shapes, 56 scanned household objects, and 11 colors. Object train/test sets are disjoint; only red, green, and blue overlap between color splits. The following task definitions distinguish volume, object-count, pose, and binary success metrics.
A.2.1 Basic Tasks
The basic tasks supply individual object concepts, spatial relations, and manipulation behaviors.
Packing Box Pairs.
Pack all boxes of two requested colors into a larger box; score is the fraction of their total volume packed. One requested color may be absent.
Packing Google Objects Seq.
Pack a named scanned object at each step of a one-to-five-step episode; score is the packed fraction of the requested objects’ volume.
Packing Google Objects Group.
Pack every scanned object in a requested category; score is the fraction of that category’s volume inside the box.
Separating Small Piles.
Push 50 small blocks into the zone of a specified color; score is the fraction of blocks in the correct zone.
Separating Large Piles.
Push 15 larger, heavier, higher-friction blocks into a specified zone; score is the fraction reaching that zone.
Align Rope.
Align a 20-bead articulated rope between requested corners of a three-sided square; score measures agreement between bead poses and the target line segment.
Packing Shapes.
Choose one requested shape among four distractors and pack it into a box; success is binary.
Assembling Kits.
Place five shapes into matching kit holes, using language to disambiguate repeated shapes; score measures the fraction and quality of pose matches.
Put Blocks in Bowls.
Place each block of the requested color into a bowl of the requested color; score is the fraction correctly placed, with at most one block per bowl.
Towers of Hanoi.
Execute a predetermined seven-move solution for three-ring Towers of Hanoi; score is the fraction of correct ring placements. The sequence comes from a solver rather than learned planning.
Packing Prepositions.
Pack a requested shape into the box located relative to a distractor object; success is binary and tests relational grounding.
A.2.2 Compositional Tasks
The compositional tasks combine these ingredients in settings withheld from training.
Packing Color Box.
Pack a specified shape into a box of a specified color. Training supplied shapes and colors separately and used only one brown packing box; success is binary.
Packing Location Box.
Pack a specified shape into the box at a named location. The test combines packing with spatial descriptors learned in other tasks; success is binary.
Separating Location Piles.
Push a pile of 50 small blocks into a zone identified by spatial relation rather than color; score is the fraction of blocks in the target zone.
Pushing Shapes.
Push a shape specified by color and geometry into a zone specified by color and location. This combines longer descriptions with object geometry and friction not encountered in pushing training.
Packing Nested Prepositions.
Pack into a box identified by nested spatial relations between distractor shapes. This tests longer relational composition, with binary success.
A.3 Experiments with Prepositional Phrases
With the relation operator, ProgramPort scores 45 versus CLIPort’s 28 on packing with prepositions, and 31 versus 17 on nested prepositions. Both remain far from perfect, illustrating the difficulty of relational grounding.
| packing-prepositions | packing-nested-prepositions | |
| CLIPort | 28.0 | 17.0 |
| ProgramPort | 45.0 | 31.0 |
A.4 Real World Experiments
A Franka Panda dataset contains 40 household-object pick-and-place demonstrations: 30 for training and ten for validation. Separate test sets of ten seen-object and ten unseen-object cases yield 70% and 80% binary success. These small samples demonstrate feasibility rather than a precise generalization estimate.
| packing-objects-seen | packing-objects-unseen | |
| ProgramPort | 70.0 | 80.0 |
A.5 Additional Affordance Visualizations
The retained affordance visualizations compare concept-level grounding and final pick/place maps across packing, bowl placement, and pushing. ProgramPort exposes intermediate object/property masks, making several localization errors easier to inspect.




A.6 Model Details
The architecture diagrams show how CLIP’s value projection and final linear map become spatial 1×1 convolutions. Action decoders receive resized masks by multiplication at multiple levels. The appendix’s ResNet illustration and the main section’s ViT-B/16 specification describe different encoder details; the preserved diagrams retain the source’s presentation.
Equation 5
| (5) |