Overview
This edition follows the 27-page arXiv v4 dated 10 November 2026, retaining all 11 figures, seven tables, seven displayed formulas, the complete annotation prompt and 41 references. OpenDexGrasp generates a task-conditioned hand pose directly from language, images and geometry. Its hardware success means grasp-and-lift after manual candidate filtering and a fixed squeeze adjustment; it does not establish autonomous completion of the instructed object-use task. Seven figures are restored from native PDF artwork to preserve small labels and image layers.
At a glance
OpenDexGrasp learns functional grasping through two complementary data sources: broad automatically synthesized grasps and a smaller set derived from teleoperation. A shared representation supports both a generative grasp head and auxiliary surface-affordance supervision. The affordance prediction is not a required intermediate input at inference. Evaluation reports better grasp stability and semantic ratings than a language-adapted DexGraspNet baseline, within the stated simulation and selected-hardware protocols.
1 Introduction
Stable holding is insufficient when a grasp blocks the part needed to use an object. The method couples free-form task semantics, object geometry and hand pose in one generative representation. Coverage-to-Alignment training first learns broad synthetic support, then adapts to more natural demonstrated contacts. This reduces reliance on an affordance-to-optimization cascade, but does not remove perception errors, execution filtering or the need for a downstream controller.
2 Related Work
Geometry-driven generators emphasize feasible holding; language-conditioned and part-grounded methods add task intent. Explicit affordance cascades offer interpretable regions but can propagate grounding errors or require optimization. OpenDexGrasp keeps an auxiliary affordance map while generating poses directly, complementing work on execution-aware grasping rather than solving closed-loop manipulation itself [11], [10].
3 Problem Formulation
Inputs are an instruction, multiple RGB views and a colored point cloud. The 31-D action contains wrist translation, continuous 6-D rotation and 22 joint values. Optional per-point affordance labels supervise training. The shared-latent factorization treats affordance and action as two outputs of the same conditional representation, rather than feeding a predicted map into the grasp generator.
Equation
4 OpenDexVerse and C2A Recipe
OpenDexVerse separates synthetic coverage from demonstrated alignment while using the same model and objectives in both stages.
4.1 OpenDexVerse Construction
OpenDex-Scale supplies 1.24 million grasps over 105 categories; about 557,180 are labeled functional. OpenDex-Align supplies 27,425 demonstrated or transferred grasps over 95 categories, with about 18,650 functional. The latter is mostly transferred data, not 27,000 independently teleoperated grasps.
| Subset | Source | # Categories | # Instances | # Grasps | # Functional Poses | Functional Ratio |
|---|---|---|---|---|---|---|
| OpenDex-Scale | Automatic synthesis + VLM annotation | 105 | 1.11k | 1.24M | 557.18k | 44.93% |
| OpenDex-Align | Teleoperation + Category transfer | 95 | 2.77k | 27.42k | 18.65k | 68.02% |
Object Source.
Category-aligned real object scans from Omni6DPose provide geometry and a common coordinate structure. Objects are selected for plausible dexterous interaction, limiting the distribution before either synthesis or teleoperation [37].
OpenDex-Scale : Large-Scale Functional Coverage.
BODex sampling and optimization produce feasible candidates. A vision-language annotator examines rendered grasps and object/contact context to assign functional labels and language. These labels expand coverage but are automatically generated judgments, not verified downstream task execution [3].
OpenDex-Align : High-Quality Embodied Alignment.
Teleoperation provides ten poses on a representative object for each selected category-size pair. Dense correspondences transfer contacts to compatible instances, followed by optimization and physical validation. The resulting alignment set preserves demonstrated intent while inheriting transfer and filtering assumptions.
4.2 C2A Details
Pretrain on OpenDex-Scale, then continue on OpenDex-Align with the same action and affordance losses. At inference, integrate a learned velocity field from Gaussian noise for 32 Euler steps. The returned hand pose is generated directly; the affordance branch shapes the representation during learning.
Equation
5 Method
The architecture combines pretrained visual-language features with metric point-cloud geometry and a flow-matching action expert.
5.1 Overview
Selected language-model hidden states provide task and view context. A point-cloud token supplies shape, while the action head learns the velocity from noisy poses toward demonstrated or synthesized targets. Auxiliary per-point labels connect the shared features to functional surface regions.
5.2 Open-Vocabulary Semantic-Geometric Latent
Qwen hidden states retain token-level relationships rather than only a final text answer. PointNet++ encodes RGB points at their original metric scale, preserving object size, and retains local features for affordance decoding. Semantic and geometric representations remain complementary.
5.3 Flow-Matching Dexterous Action Generation
Linearly interpolate between Gaussian noise and a target pose, then regress the target-minus-noise velocity. Self-attention combines geometry, learned queries and action tokens; cross-attention conditions on visual-language states. The model learns a conditional pose distribution rather than selecting from a fixed grasp list [13].
5.4 Auxiliary Affordance Grounding
A text-conditioned geometry feature propagates to point-level affordance scores. Focal and Dice losses supervise this branch, with affordance weight 0.3 in the total objective. These labels need not be available for every sample and are not supplied as an inference-stage action bottleneck.
6 Experiments
Experiments evaluate physical grasp metrics, diversity and semantic ratings separately, then test an adapted model on a selected physical object set.
6.1 Experimental Setup
The baseline augments DexGraspNet 2.0 with CLIP language features and uses the same data/splits. Four simulation splits separate functional versus transport grasps and seen versus unseen categories. Stability, penetration and semantic ratings measure different properties; GPT-5 and human naturalness scores are not task-execution success [36].
6.2 Main Results
Functional simulation success rises from 50.88% to 68.07% on seen categories and from 43.22% to 62.96% on unseen categories. Nonfunctional success rises from 63.27% to 75.46% and from 54.49% to 66.73%. Penetration and successful-grasp displacement improve across these splits. Higher perceptual scores and diversity accompany the stability gains, but do not certify functional execution.
| Seen | Unseen | |||||||||||||
| SIV | PD | SD | SR(%) | Style Div. | Perceptual Score | SIV | PD | SD | SR(%) | Style Div. | Perceptual Score | |||
| GPT-5 | Human | GPT-5 | Human | |||||||||||
| Functional | ||||||||||||||
| DexGraspNet 2.0⋆ | 5.12 | 1.04 | 1.51 | 50.88 | 0.95 | 6.48 | 6.10 | 6.86 | 1.62 | 4.29 | 43.22 | 0.81 | 5.87 | 5.37 |
| OpenDexGrasp (Ours) | 1.28 | 0.39 | 1.37 | 68.07 | 1.39 | 7.43 | 7.85 | 1.39 | 0.48 | 1.80 | 62.96 | 1.41 | 6.98 | 6.62 |
| Non-Functional | ||||||||||||||
| DexGraspNet 2.0⋆ | 4.98 | 0.87 | 2.53 | 63.27 | 1.14 | - | - | 5.16 | 0.99 | 3.87 | 54.49 | 1.03 | - | - |
| OpenDexGrasp (Ours) | 1.33 | 0.42 | 0.94 | 75.46 | 1.55 | - | - | 1.30 | 0.44 | 2.29 | 66.73 | 1.25 | - | - |
6.3 Ablation Study
The full model's reported aggregate success is 71.65%. Removing affordance supervision gives 67.49%; removing alignment gives 69.38%; reducing coverage data gives 60.52%; replacing Qwen with CLIP encoders gives 57.26%. Alignment lowers diversity slightly while improving semantic ratings. Not every metric improves: removing affordance supervision has marginally lower penetration depth (0.39 versus 0.41 cm). The backbone ablation also changes representation and capacity.
| Method | SIV | PD | SD | SR(%) | Style Div. | Perceptual Score | |
| GPT-5 | Human | ||||||
| OpenDexGrasp | 1.31 | 0.41 | 1.20 | 71.65 | 1.47 | 7.37 | 7.68 |
| - w/o affordance grounding | 1.37 | 0.39 | 1.33 | 67.49 | 1.26 | 6.82 | 6.63 |
| - w/o OpenDex-Align alignment | 1.58 | 0.53 | 1.44 | 69.38 | 1.52 | 5.96 | 6.04 |
| - reduced OpenDex-Scale scale | 2.03 | 0.85 | 9.76 | 60.52 | 1.06 | 6.51 | 6.79 |
| - w/o pretrained VLM | 5.15 | 0.98 | 2.32 | 57.26 | 1.05 | 6.22 | 6.16 |
6.4 Real-World Experiments
Across ten selected objects with ten accepted poses each, hardware grasp-and-lift success averages 72% versus 59% for the adapted baseline. GPT-5 ratings average 7.5 versus 6.7 and human ratings 7.7 versus 6.5. Some objects tie or have worse individual ratings; the headline is an aggregate. Manual pose rejection and squeeze refinement are part of deployment.
| Metric | Method | Unseen Category | Seen Category | Avg. | ||||||||
| Rice Paddle | Dustpan | Bouquet | Pitcher | Small Bucket | Bottle | Umbrella | Shampoo | Hammer | Brush | |||
| Success Rate | DexGraspNet 2.0⋆ | 40% | 60% | 40% | 50% | 50% | 70% | 80% | 60% | 80% | 60% | 59.0% |
| Ours | 60% | 70% | 50% | 80% | 50% | 100% | 90% | 60% | 90% | 70% | 72.0% | |
| GPT-5 Score | DexGraspNet 2.0⋆ | 6.2 | 7.1 | 5.5 | 6.7 | 7.4 | 7.6 | 6.8 | 6.4 | 7.0 | 6.1 | 6.7 |
| Ours | 8.1 | 7.2 | 6.3 | 7.5 | 7.3 | 8.6 | 7.2 | 6.7 | 8.3 | 7.4 | 7.5 | |
| Human Score | DexGraspNet 2.0⋆ | 6.5 | 6.9 | 4.8 | 6.1 | 7.9 | 8.0 | 5.9 | 4.8 | 7.6 | 6.8 | 6.5 |
| Ours | 7.7 | 7.9 | 7.1 | 8.0 | 7.1 | 9.3 | 7.6 | 6.8 | 8.1 | 7.6 | 7.7 | |
7 Conclusion
The study supports combining broad grasp coverage, demonstrated alignment and auxiliary functional grounding in a direct pose generator. Its strongest evidence concerns the tested grasp distributions and selected physical executions, with downstream use and closed-loop correction still open.
8 Limitation
Performance depends on visual coverage and the pretrained semantic representation. Generated actions are open-loop and lack tactile correction. Fine-grained contact data remain limited, while small parts and thin geometry can produce errors that aggregate stability metrics miss.
Acknowledgments
The authors acknowledge Zhewei Gui and Zimu Han, plus support from the Beijing Natural Science Foundation and the National Natural Science Foundation of China.
Appendix
The appendix preserves implementation parameters, dataset construction, the full annotation prompt, evaluation definitions, additional comparisons and representative failures.
Appendix A Method Details
Implementation details specify the semantic backbone, point-cloud encoder, flow-matching expert and auxiliary decoder.
A.1 Implementation Details
Qwen3-VL-4B-Instruct conditions an 18-block, width-1024 action transformer. Even blocks cross-attend to Qwen states; odd blocks use causal self-attention over geometry, eight query tokens and the noisy action. PointNet++ processes 2,048 metric-scale points. Training uses AdamW, mixed precision and a two-stage trainability schedule; Table 5 gives two million steps, a 1,000-step initial frozen stage and per-GPU batch sixteen. The action horizon is one pose, not a motion trajectory. All loss formulas and numerical settings remain below [23], [21].
Equations
| Hyperparameter | Value |
| Input views | 3 |
| Point-cloud points | 2048 |
| Action dimension | 31 |
| Wrist translation / rotation / joint dimensions | 3 / 6 / 22 |
| PointNet++ stage-1 samples | 512 |
| PointNet++ stage-1 radii | |
| PointNet++ stage-1 neighborhood sizes | |
| PointNet++ stage-2 samples | 128 |
| PointNet++ stage-2 radii | |
| PointNet++ stage-2 neighborhood sizes | |
| Global point-token dimension | 1024 |
| DiT blocks | 18 |
| DiT hidden size | 1024 |
| DiT attention heads | 16 |
| DiT dropout | 0.1 |
| Future query tokens | 8 |
| Action horizon | 1 |
| Qwen hidden states for DiT conditioning | 18 |
| Flow-matching time distribution | |
| Euler integration steps | 32 |
| Affordance fusion dimension | 1024 |
| Point embedding dimension | 512 |
| Focal loss / | 0.5 / 2.0 |
| Dice loss weight | 1.0 |
| Affordance loss weight | 0.3 |
| AdamW betas | |
| AdamW epsilon | |
| Weight decay | |
| Base learning rate | |
| Qwen-VL interface learning rate | |
| PointNet++ learning rate | |
| Action-head learning rate | |
| Affordance-decoder learning rate | |
| Optimization steps | 2M |
| Warmup steps | 1k |
| Minimum learning rate | |
| Per-GPU batch size | 16 |
| Gradient accumulation | 1 |
| Gradient clipping norm | 1.0 |
| Initial frozen-stage duration | 1k steps |
Appendix B OpenDexVerse Construction
The construction appendix explains which objects are selected, how 1,590 human poses expand into the alignment set, and how language labels are produced.
B.1 Object Asset Selection
Starting from 153 Omni6DPose categories and 4,302 instances, graspability filtering retains 105 categories. Size variation creates 185 category-size pairs: 56 single-size, 18 two-size and 31 three-size categories. Teleoperation selects 95 categories/159 pairs, choosing one representative instance per pair. Transfer stays within compatible category-size groups; categories that are too large, thin or awkward are excluded.
B.2 Relationship Between Columns in Table 1
Ten demonstrations per selected pair yield 1,590 directly teleoperated poses. Category-specific DeepSDF models reconstruct shape spaces, interpolate between source and target objects, and carry hand-link contact regions through those correspondences. Optimization refines the target pose under contact, joint and collision constraints. Because reconstructed and original meshes differ, 300 SAPIEN simulation steps reduce residual penetration on the original mesh. Filtering leaves 25,835 transferred poses; together they form 27,425 alignment poses [35].
B.3 Text Description Generation
For each grasp, the annotator receives three hand-object views, three object-only views and structured hand-link/part contact percentages. It labels functional use versus transport, then returns five grasp descriptions and five distinct task instructions. JSON parsing, list length and distinctness checks trigger retries, but do not independently validate semantic correctness. The complete original prompt is preserved, including its fixed-grasp/no-regrasp constraint, examples and output schema.
Goal: Determine whether the grasp is functional, and generate 5 grasp descriptions and 5 corresponding functional instructions for the given right-hand grasp.
Input: You are given six RGB images related to a robotic hand grasping a {instance_name}. The first three images show the grasp itself: Palm view (first-person perspective toward the palm side), Back view (toward the dorsal side), and Finger view (emphasizing fingertips and contact geometry). The next three images are object-only reference views (Front, Side, Top). Use the object-only views to understand overall geometry; base grasp type, contact regions, and functional judgment primarily on the three grasp images and the contact data.
Along with the images, you are provided with pre-computed contact information between the hand and the object, derived from hand link keypoints and object part meshes. The percentage indicates the proportion of keypoints on that hand link in contact with the object part. Format: ‘‘Grasp on <object>: <object_part>: contacted by <hand_link>(<percentage>%), ...’’
Interpretation: The image shows one fixed grasp pose performed by the right hand. All outputs must be consistent with the visible hand-object contact, wrist orientation, and object geometry.
Constraints: Do not require regrasping or changing contact regions. Do not invent unseen parts or functions. All outputs must remain physically plausible. Descriptions answer “How is the object held?” Instructions answer “What task can be done with this hold?”
Diversity requirement: The 5 descriptions must differ significantly in wording and focus. The 5 instructions must also differ in functional intent. Avoid repeated sentence structures or near-duplicate meanings.
Functional Grasp Classification: Determine whether this grasp is functional or non-functional (transport). A grasp is functional if the hand-object configuration enables using the object for its intended purpose (e.g., pouring from a bottle, cutting with a knife), not just holding or transporting it. A grasp is non-functional if it merely secures the object for picking up, moving, or handing off. Base your judgment on finger placement relative to functional parts, wrist orientation, and whether the grasp affords the object’s typical usage motion. Return true for functional, false for non-functional. If non-functional, instructions should describe transport or repositioning tasks rather than object-use tasks.
Descriptions (grasp-focused): Each description MUST cover both: (1) how the hand grasps—grasp type, finger configuration, force pattern; AND (2) which part of the object is contacted. Think of it as answering “How is which part of the object being held?” Do NOT describe task goals or intended use. Bad example: “The hand firmly holds the object.” (missing part + configuration detail). Good example: “A power grasp wraps all four fingers around the cylindrical handle, with the thumb opposing across the shaft.”
Instructions (intent-focused): Describe the real-world task this grasp is suited for—what the person is trying to accomplish. Focus on task-level intent (e.g., driving a screw, pouring liquid, opening a door), not low-level motion. Must be achievable with the exact shown grasp without regrasping. Do NOT repeat or paraphrase content from descriptions. Bad example: “Grip the handle tightly to hold the object.” (describes grasp, not task). Good example: “Use this grip to drive screws into a surface with controlled rotational force.”
Length and style constraint (IMPORTANT): Each item must be ONE short sentence (10–20 words preferred). Be concise and information-dense. Avoid explanations, subordinate clauses, or filler words. Use compact phrasing (like captions, not full explanations). Use varied sentence structures (no fixed template). Each sentence should emphasize one main aspect. English only.
Output format: Return JSON only: {"functional": true, "descriptions": ["...", "...", "...", "...", "..."], "instructions": ["...", "...", "...", "...", "..."]}B.4 Data Visualization
The two full galleries preserve the breadth of synthetic poses and the natural contact arrangements in teleoperated/transferred data. They are representative visual examples, not a separate quantitative comparison of reliability.
Appendix C Detailed Experimental Setup
Evaluation details distinguish physical stability from semantic suitability and document manual selection in hardware deployment.
C.1 Train/Test Split
There are 243 instances across 81 categories: 75 functional-seen, 15 functional-unseen, 138 nonfunctional-seen and 15 nonfunctional-unseen. Each category contributes three instances. Table 6 retains the full category lists; held-out category performance is narrower than unrestricted open-world generalization.
| Test Split | # Cat. | # Inst. | Object Categories |
| Functional - Seen | 25 | 75 | bottle, dish, bowl, facial cream, brush, camera, hair dryer, can, hammer, hand cream, remote control, knife, rubik cube, lipstick, scissor, mouse, shampoo, mug, shoe, table tennis bat, tape measure, umbrella, teapot, toothbrush, toothpaste |
| Functional - Unseen | 5 | 15 | spoon, cup, dumbbell, spanner, whistle |
| Non-functional - Seen | 46 | 138 | banana, bread, ball, cake, belt, calculator, book, candy, box, carrot, corn, flower pot, glasses case, donut, hamburger, egg, handbag, egg tart, hot dog, keyboard, onion, lemon, orange, mango, peach, mangosteen, pear, medicine bottle, pie, pitaya, soap, pizza, sweet potato, pomegranate, timer, power strip, tissue, sausage, tomato, toy boat, toy train, toy bus, toy truck, toy car, toy motorcycle, toy plane |
| Non-functional - Unseen | 5 | 15 | toy animals, chili, chicken leg, wallet, frisbee |
| Total | 81 | 243 |
C.2 Metric Calculation
Penetration metrics, simulated stability, prediction variation and perceptual ratings capture complementary properties. Their conditions and aggregation matter when interpreting the main tables.
Penetration Depth .
Penetration depth is the maximum inward distance of hand vertices relative to the object surface, reported in centimeters. It measures the worst local interpenetration.
Solid Intersection Volume .
Solid intersection volume voxelizes hand and object meshes at one cubic millimeter per voxel and sums overlapping solid voxels. The reported unit is cubic centimeters.
Success Rate .
Hold the predicted hand pose fixed, set object mass to 0.1 kg and friction to 1.0, enable gravity and simulate two seconds. A trial succeeds when displacement stays below 0.2 m. Ten stochastic predictions per test sample contribute to the rate. This tests holding under standardized physics, not whether the instructed object-use action is completed.
Equation
Simulation Displacement .
Simulation displacement is measured only on grasps already counted successful. It therefore excludes failures and must be interpreted alongside success rate, not as unconditional average movement.
Style Diversity .
Diversity averages pairwise differences among ten predictions, equally weighting translation in meters, rotation in radians and the norm of 22 joint-angle differences. Its numerical scale depends on those units and weights. Semantic scores use instruction-plus-multiview renders; twenty human participants rate thirty sampled result groups on a 0–10 scale, alongside GPT-5 ratings. These are judgments of plausibility and alignment, not executed task outcomes.
Equations
C.3 Baseline Details
The adapted baseline concatenates a CLIP instruction embedding with ResUNet14 point features. Language-conditioned graspness selects seed points, a diffusion module predicts wrist poses, and an MLP predicts joints. It uses the same data/splits but a different visual-language representation and two-stage architecture, so the comparison changes multiple modeling choices.
C.4 Real Robot Setup
A Sharpa Wave Hand on a Franka Panda uses a base-mounted camera to estimate object pose, then motion planning executes the generated hand configuration. Operators repeatedly sample and manually reject table-colliding or occluded candidates until ten valid poses remain per object. Every accepted pose executes once, with an additional 0.05 rad closing adjustment per hand joint. Success requires grasping and lifting; hazardous object/motion cases are excluded. The 72% result applies to this selected protocol, not raw model proposals.
Appendix D Additional Experiments
Additional experiments compare direct generation with optimization using the same predicted affordance map and show physical execution sequences.
D.1 Ablation Study: w/o Pose Generation Module
The optimization baseline initializes a 28-D pose toward the highest-affordance region, then runs up to 300 Adam steps. Its five energies penalize object penetration, attract fingers to affordance regions, cover high-score clusters, respect joint limits and reduce self-collision. It shares geometry, hand model and affordance prediction. Direct generation reports 71.65% versus 49.15% success and 0.93 versus 2.84 seconds per grasp, about a threefold speedup in this setup; 32 Euler steps replace the iterative pose optimizer.
| Method | SIV | PD | SD | SR(%) | Style Div. | Perceptual Score | Time (s) | |
| GPT-5 | Human | |||||||
| Affordance + Optimization | 3.22 | 0.73 | 8.87 | 49.15 | 0.83 | 5.57 | 5.10 | 2.84 |
| OpenDexGrasp | 1.31 | 0.41 | 1.20 | 71.65 | 1.47 | 7.37 | 7.68 | 0.93 |
D.2 Affordance Prediction Results
The figure shows predicted affordance, direct grasps and optimization-based grasps for the same instructions. Only the optimization baseline consumes the predicted map at inference. Agreement between the direct pose and the map reflects shared conditioning and auxiliary training, not a hidden affordance-to-pose cascade.
D.3 Real Robot Manipulation Visualization
All ten original four-frame reaching/grasping/lifting sequences and their instruction labels are retained. They demonstrate selected executions rather than complete object-use trajectories.
Appendix E Failure Case Analysis
Keyboard, frisbee and flower-pot examples combine thin geometry with weak part semantics: approximate contact can still penetrate a surface. The shampoo grasp reaches the right semantic neighborhood but misplaces individual fingers on the precise actuation region. The authors propose denser fine-contact demonstrations and stronger geometry/part constraints; these remedies are future work, and failures remain in the edition.