Literature
Display
← All papers

CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations

Bowen Jiang · William Painter Reger · Roberto Martín-Martín

· ICRA 2026 ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

CoDex learns to grasp, move, and actuate functional objects without task demonstrations. A vision-language model identifies interaction points and a goal pose; constrained optimization proposes functional grasps; reinforcement learning refines a parameterized arm–hand motion primitive in simulation. On six real tasks with a Franka arm and LEAP hand, the paper reports 73% combined movement-and-actuation success over five trials per task. “Zero demonstration” still includes pretrained perception, reconstructed object models, simulation training, and a designed primitive and reward structure.

I Introduction

Functional tool use requires coordinating external object pose with an internal mechanism such as a trigger or button. Semantic knowledge alone does not specify stable contacts, and grasp stability alone does not ensure correct aiming or actuation. CoDex connects these requirements through explicit local and global constraints. The resulting pipeline searches for physically usable behavior rather than copying human hand motions, but it remains bounded by its perception, object reconstruction, and control assumptions.

Refer to caption
Fig. 1: Robot executing a Compositional Dexterous Functional Object Manipulation (spraying water on a plant) using CoDex. The robot autonomously performs a task-aware functional grasp, repositions the spray bottle, and actuates the trigger to spray mist, all without any human demonstration. CoDex guides these tasks by bridging high-level semantic understanding through a VLM with low-level physical dexterity via constrained optimization and reinforcement learning.

II Related Work

The work combines three strands: vision-language guidance, task-oriented grasp synthesis, and coordinated arm–hand manipulation. Earlier constraint or visual-prompt methods supply semantic goals; analytic methods supply stable contacts; demonstration or simulation methods supply dexterity. CoDex’s contribution is using common semantic constraints to connect goal selection, grasp initialization, and simulation refinement for objects with internal mechanisms.

Refer to caption
Fig. 2: Overview of the CoDex pipeline. CoDex bridges high-level VLM understanding and low-level dexterity by translating abstract VLM outputs into concrete semantic constraints that guide a two-stage policy learning process. (1) VLM-Generated Semantic Constraints. First, a VLM interprets the user’s input to generate local constraints (key interaction points like the actuation point and function point) and a global constraint (the final object pose). (2) Constrained Optimization. In the first learning phase, these constraints are enforced through analytic constrained optimization to synthesize a diverse set of motion trajectories, each of which includes a task-aware functional grasp. (3) RL Policy Training. In the second phase, these motion trajectories initialize a constraint-guided RL process, which uses the same semantic constraints as a reward function and learns the complete grasp-move-actuate policy.

III CoDex: Compositional Dexterous Functional Object Manipulation

Inputs are a language task and RGB-D scene. Semantic parsing produces interaction points and a final object pose, then analytic grasp search and reinforcement learning construct the motion.

Refer to caption
Fig. 3: Reconstructed objects with their VLM-identified local semantic constraints. The generation process combines semantic and visual information from VLMs (see Sec. III-A) to infer the actuation point, pactp_{\text{act}}, (blue arrow start) and function point, pfncp_{\text{fnc}}, (orange arrow start). The actuation direction dactd_{\text{act}} and function direction dfncd_{\text{fnc}} are parallel to the surface normal at the actuation and function points, pointing into the object (dotted line indicates inside the object). These semantic constraints guide the grasp and motion optimization and training processes.

III-A VLM-Generated Semantic Constraints

LangSAM segments the functional object, Tripo reconstructs its mesh, and depth-derived height scales the reconstruction. Local constraints specify actuation and function points plus the environmental target. The global constraint is a desired six-dimensional object pose. Errors in reconstruction or coordinate alignment can therefore propagate into both grasping and task execution.

III-A 1 Local Semantic Constraints

A vision-language model names relevant parts, Molmo locates them in the image, and FoundationPose aligns image and mesh to map them into 3D. Actuation direction is assumed to follow the inward surface normal at the chosen point. This is a useful simplification for pressing but does not infer arbitrary mechanisms, alternating contacts, or complex force trajectories.

III-A 2 Global Semantic Constraints

VLM-CEM initializes a pose around the functional and target points, then proposes and scores rendered candidates using the accumulated history. Algorithm 1 uses six rounds with ten candidates per round and optionally keeps elite poses. The highest-scoring pose is selected. Render-based semantic plausibility is not a guarantee of 3D alignment, reachability, or collision-free execution; later failures demonstrate this gap.

Algorithm 1 VLM-CEM
1: Function Point pfncp_{\mathrm{fnc}}, Target Location ptgtp_{\mathrm{tgt}}, iterations N=6N{=}6, candidates K=10K{=}10
2: goal pose T∗T^{*}
3: T0←AnchorInit​(pfnc,ptgt)T_{0}\leftarrow\textsc{AnchorInit}(p_{\mathrm{fnc}},\,p_{\mathrm{tgt}})
4: s0←VLMScore​(Render​(T0))s_{0}\leftarrow\textsc{VLMScore}(\textsc{Render}(T_{0}))
5: ℋ←{(T0,s0)}\mathcal{H}\leftarrow\{(T_{0},s_{0})\}
6:
7: for i=1i=1 to NN do
8:   𝒫←VLMPropose​(ℋ,K)\mathcal{P}\leftarrow\textsc{VLMPropose}(\mathcal{H},\,K)
9:   for each T∈𝒫T\in\mathcal{P} do
10:    s←VLMScore​(Render​(T))s\leftarrow\textsc{VLMScore}(\textsc{Render}(T))
11:    ℋ←ℋ∪{(T,s)}\mathcal{H}\leftarrow\mathcal{H}\cup\{(T,s)\}
12:   end for
13:   ℋ←TopR​(ℋ,ρ)\mathcal{H}\leftarrow\textsc{TopR}(\mathcal{H},\,\rho) ⊳\triangleright keep top-ρ\rho elites (optional)
14: end for
15: return T∗←arg⁡max(T,s)∈ℋ⁡sT^{*}\leftarrow\arg\max_{(T,s)\in\mathcal{H}}\,s
Refer to caption
Fig. 4: Human-like (left) and robot-specific (right) examples of initial functional grasp candidates. Our analytic constrained optimization synthesizes functionally valid human-like and robot-specific grasps allowing CoDex to exploit the hand’s full morphology instead of restricting it to the human grasps that can be obtained with imitation learning.
Refer to caption
Fig. 5: Six functional object manipulation tasks in our experiments. They require combining local manipulation of functional objects with internal DoF (flashlight, board spray, water spray, air blower, hot glue gun, and salt grinder) with their global motion in the scene.

III-B Constraint-Guided Policy Training

Analytic candidates restrict reinforcement learning to plausible functional grasps. Simulation then tests and refines the dynamics of the complete motion rather than treating grasp synthesis as the final controller.

III-B 1 Analytic Constrained Optimization

Inverse kinematics samples hand configurations near the actuation point. Optimization maximizes a force-closure robustness metric while enforcing surface contact, collision clearance, fingertip proximity, and fingerpad alignment. Failed searches restart from another sample. These constraints generate diverse robot-compatible candidates, including grasps unlike human examples, but their static validity does not ensure stable transport or successful actuation.

Equation 1
maxq∈𝒬l∗​(q)s.t.l∗​(q)≥lmin(Min F. Closure)s⁡(F​Ki​(q))=0,∀i∈{1,…,nc}(Surface Contact)σj​(q)≥dj,∀ collision pairs ​j(Coll. Avoidance)∃ia​c​t∈Fa​c​t​ s.t.‖F​Kt​i​p​(q,ia​c​t)−pa​c​t‖2≤δd​i​s​t(Act Pt Proximity)np​a​d​(q,ia​c​t)⋅(−da​c​t)≥cos⁡(δa​n​g​l​e)(Act Alignment){\small\!\!\begin{aligned} \max_{q\in\mathcal{Q}}\hskip 9.24994pt&l^{*}(q)\\ \text{s.t.}\hskip 9.24994pt&l^{*}(q)\geq l_{\min}&&\text{(Min F. Closure)}\\ &s(FK_{i}(q))=0,\ \forall i\in\{1,...,n_{c}\}&&\text{(Surface Contact)}\\ &\sigma_{j}(q)\geq d_{j},\ \forall\text{ collision pairs }j&&\text{(Coll. Avoidance)}\\ &\exists i_{act}\in F_{act}\text{ s.t.}\\ &\hskip 9.24994pt\|FK_{tip}(q,i_{act})-p_{act}\|_{2}\leq\delta_{dist}&&\text{(Act Pt Proximity)}\\ &\hskip 9.24994ptn_{pad}(q,i_{act})\cdot(-d_{act})\geq\cos(\delta_{angle})&&\text{(Act Alignment)}\end{aligned}} (1)

III-B 2 Constraint-Guided RL

The policy outputs 38 parameters: sixteen grasp-joint offsets, sixteen pre-grasp fractions, and a six-dimensional hand-pose offset. A fixed multi-stage primitive approaches, grasps, closes the fingers, and moves toward the goal while actuating. PPO refines these parameters with shaped terms for stability, lifting, alignment, and directional force, plus stage and success bonuses; excessive displacement cancels reward. Training in ManiSkill3 reportedly takes about one hour with 2,048 parallel environments. This is simulation-trained primitive selection and refinement, rather than a demonstrated general reactive contact policy.

Refer to caption
Fig. 6: Key stages of the CoDex’s parameterized motion primitive trained in simulation. The policy action space determines (1) the pre-contact approach, (2) grasp pose, (3) finger closing strategy (internal DoF actuation), and (4) object pose change (external DoF actuation).

IV Experimental Evaluation

Real evaluation uses a seven-degree-of-freedom arm and sixteen-degree-of-freedom hand on six tasks, with five trials per task and randomized initial object poses. Success requires both correct movement and actuation. CoDex reports 73% overall success, versus 33% for analytic grasps with the same goal generator and zero for the same learned controller with the tested PIVOT goal generator. Partial movement or actuation is plotted separately. The comparison supports both stages in this setup, but the trial count is small and PIVOT’s failure is specific to the implemented pose-generation variant.

Failure analysis attributes incorrect movement to 23.3% of trials and actuation failures to 23.3%, with overlap. Plausible 2D goals can be wrong in depth or incompatible with the grasp and joint limits. The flashlight task fails throughout, requiring precise contact with a small deformable button. Friction, reconstructed geometry, and compliance mismatch also cause slipping or missed actuation. Local point detection was reliable in this sample, but that observation does not establish general reliability.

Twenty participants rate multiview goal-pose renderings on a five-point scale. Both VLM-CEM variants score above two PIVOT variants, with reported paired Wilcoxon comparisons below p=0.02. Direction-constrained search is particularly helpful for spraying. These ratings assess depicted goal suitability, not physical task success, and the full pipeline still exhibits pose failures.

The grasp-refinement experiment tests three analytic candidates per object, five trials each, and compares their average and a post-hoc best candidate with the learned result. The paper reports gains exceeding 36 percentage points in stability and 60 points in actuation over the candidate average, and more than 12 and 26 points over the best candidate. The “oracle” is the best among these three tested candidates, not an upper bound on all analytic grasping. Simulation refinement particularly helps the slippery grinder, for which initial candidates fail to hold a stable grasp.

Original figure
Fig. 7: Overall success rate comparison on six CD-FOM tasks, evaluated over five real-world trials per task. Solid segments represent the rate of success, while shaded segments show rates of partial success (either correct object movement only or correct internal DoF actuation only). CoDex achieves a 73% success rate, demonstrating the significant benefit of its policy learning stage and VLM-CEM compared to two baselines, pure analytical grasp synthesis [23] and PIVOT [30].
Original figure
Fig. 8: Human study ratings of generated goal poses. We request human feedback on the goal poses generated by our VLM-CEM procedure and baselines (VLM-CEM without rotation changes, PIVOT with rotation and without rotations). We also report the average and standard deviation error bars of the results across all goals for each respective method. On average, the two VLM-CEM methods (ours) are ranked higher in most tasks.
Refer to caption
Fig. 9: Example visualizations of different goal-pose-generation methods on the task clean keyboard. Both variants of VLM-CEM generate both semantically and physically valid global constraints, while the baseline methods perform poorly on the task.
Original figure
Fig. 10: Performance gains of CoDex constraint-guided policy training compared to the direct execution of the 3 and the best analytical grasps from CoDex’s constrained optimization. Total bar height indicates the success rate of achieving a stable grasp through lifting. The bottom segment (darker shade) represents the success rate of achieving both a stable grasp and successful actuation. By training with constraint-guided RL in simulation for the full task, CoDex significantly improves stability and actuation of the objects.

V Limitations

Pinpoint buttons expose limits in fingertip size, reconstruction, and actuation accuracy. The framework targets a single goal pose and actuation point; sustained coupled motions such as cutting while advancing, alternating contacts, and multiple actuation sites remain unsupported. Trajectory-level constraints, tactile feedback, and reactive contact correction are proposed extensions.

VI Conclusion

Semantic constraints provide a practical interface between pretrained visual-language reasoning, analytic grasping, and physics-based refinement. The demonstrated advantage is composition of these stages on a small set of functional tasks without demonstrations. More complex mechanisms and robust closed-loop execution remain future work.

Concept demonstration

Open image ↗