At a glance
CoDex learns to grasp, move, and actuate functional objects without task demonstrations. A vision-language model identifies interaction points and a goal pose; constrained optimization proposes functional grasps; reinforcement learning refines a parameterized arm–hand motion primitive in simulation. On six real tasks with a Franka arm and LEAP hand, the paper reports 73% combined movement-and-actuation success over five trials per task. “Zero demonstration” still includes pretrained perception, reconstructed object models, simulation training, and a designed primitive and reward structure.
I Introduction
Functional tool use requires coordinating external object pose with an internal mechanism such as a trigger or button. Semantic knowledge alone does not specify stable contacts, and grasp stability alone does not ensure correct aiming or actuation. CoDex connects these requirements through explicit local and global constraints. The resulting pipeline searches for physically usable behavior rather than copying human hand motions, but it remains bounded by its perception, object reconstruction, and control assumptions.
II Related Work
The work combines three strands: vision-language guidance, task-oriented grasp synthesis, and coordinated arm–hand manipulation. Earlier constraint or visual-prompt methods supply semantic goals; analytic methods supply stable contacts; demonstration or simulation methods supply dexterity. CoDex’s contribution is using common semantic constraints to connect goal selection, grasp initialization, and simulation refinement for objects with internal mechanisms.
III CoDex: Compositional Dexterous Functional Object Manipulation
Inputs are a language task and RGB-D scene. Semantic parsing produces interaction points and a final object pose, then analytic grasp search and reinforcement learning construct the motion.
III-A VLM-Generated Semantic Constraints
LangSAM segments the functional object, Tripo reconstructs its mesh, and depth-derived height scales the reconstruction. Local constraints specify actuation and function points plus the environmental target. The global constraint is a desired six-dimensional object pose. Errors in reconstruction or coordinate alignment can therefore propagate into both grasping and task execution.
III-A 1 Local Semantic Constraints
A vision-language model names relevant parts, Molmo locates them in the image, and FoundationPose aligns image and mesh to map them into 3D. Actuation direction is assumed to follow the inward surface normal at the chosen point. This is a useful simplification for pressing but does not infer arbitrary mechanisms, alternating contacts, or complex force trajectories.
III-A 2 Global Semantic Constraints
VLM-CEM initializes a pose around the functional and target points, then proposes and scores rendered candidates using the accumulated history. Algorithm 1 uses six rounds with ten candidates per round and optionally keeps elite poses. The highest-scoring pose is selected. Render-based semantic plausibility is not a guarantee of 3D alignment, reachability, or collision-free execution; later failures demonstrate this gap.
III-B Constraint-Guided Policy Training
Analytic candidates restrict reinforcement learning to plausible functional grasps. Simulation then tests and refines the dynamics of the complete motion rather than treating grasp synthesis as the final controller.
III-B 1 Analytic Constrained Optimization
Inverse kinematics samples hand configurations near the actuation point. Optimization maximizes a force-closure robustness metric while enforcing surface contact, collision clearance, fingertip proximity, and fingerpad alignment. Failed searches restart from another sample. These constraints generate diverse robot-compatible candidates, including grasps unlike human examples, but their static validity does not ensure stable transport or successful actuation.
Equation 1
| (1) |
III-B 2 Constraint-Guided RL
The policy outputs 38 parameters: sixteen grasp-joint offsets, sixteen pre-grasp fractions, and a six-dimensional hand-pose offset. A fixed multi-stage primitive approaches, grasps, closes the fingers, and moves toward the goal while actuating. PPO refines these parameters with shaped terms for stability, lifting, alignment, and directional force, plus stage and success bonuses; excessive displacement cancels reward. Training in ManiSkill3 reportedly takes about one hour with 2,048 parallel environments. This is simulation-trained primitive selection and refinement, rather than a demonstrated general reactive contact policy.
IV Experimental Evaluation
Real evaluation uses a seven-degree-of-freedom arm and sixteen-degree-of-freedom hand on six tasks, with five trials per task and randomized initial object poses. Success requires both correct movement and actuation. CoDex reports 73% overall success, versus 33% for analytic grasps with the same goal generator and zero for the same learned controller with the tested PIVOT goal generator. Partial movement or actuation is plotted separately. The comparison supports both stages in this setup, but the trial count is small and PIVOT’s failure is specific to the implemented pose-generation variant.
Failure analysis attributes incorrect movement to 23.3% of trials and actuation failures to 23.3%, with overlap. Plausible 2D goals can be wrong in depth or incompatible with the grasp and joint limits. The flashlight task fails throughout, requiring precise contact with a small deformable button. Friction, reconstructed geometry, and compliance mismatch also cause slipping or missed actuation. Local point detection was reliable in this sample, but that observation does not establish general reliability.
Twenty participants rate multiview goal-pose renderings on a five-point scale. Both VLM-CEM variants score above two PIVOT variants, with reported paired Wilcoxon comparisons below p=0.02. Direction-constrained search is particularly helpful for spraying. These ratings assess depicted goal suitability, not physical task success, and the full pipeline still exhibits pose failures.
The grasp-refinement experiment tests three analytic candidates per object, five trials each, and compares their average and a post-hoc best candidate with the learned result. The paper reports gains exceeding 36 percentage points in stability and 60 points in actuation over the candidate average, and more than 12 and 26 points over the best candidate. The “oracle” is the best among these three tested candidates, not an upper bound on all analytic grasping. Simulation refinement particularly helps the slippery grinder, for which initial candidates fail to hold a stable grasp.
V Limitations
Pinpoint buttons expose limits in fingertip size, reconstruction, and actuation accuracy. The framework targets a single goal pose and actuation point; sustained coupled motions such as cutting while advancing, alternating contacts, and multiple actuation sites remain unsupported. Trajectory-level constraints, tactile feedback, and reactive contact correction are proposed extensions.
VI Conclusion
Semantic constraints provide a practical interface between pretrained visual-language reasoning, analytic grasping, and physics-based refinement. The demonstrated advantage is composition of these stages on a small set of functional tasks without demonstrations. More complex mechanisms and robust closed-loop execution remain future work.