Literature
Display
← All papers

KPGrasp: Scalable Keypoint Flow Matching for Dexterous Grasp Generation

Yuansen Huang · Jiayi Chen · Haoran Liu · Yubin Ke · Bing Han · Jiangran Lyu · Mi Yan · Li Yi · He Wang

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

Project page. This edition follows the complete sixteen-page, 10 October 2026 v2, including nine figures, six numbered tables, both numbered equations and all 54 references.

Complete original PDF block S0.F1
Figure 1: Overview. Unlike prior methods that use pose-joint parameterizations and rely on auxiliary contact losses or contact-based test-time refinement, KPGrasp parametrizes grasps with Euclidean hand keypoints and learns a simple yet scalable Transformer-based flow-matching model. KPGrasp achieves state-of-the-art performance on two simulation benchmarks and generates high-quality, diverse dexterous grasps.

At a glance

KPGrasp generates dexterous grasps as Euclidean hand keypoints, conditioned on object geometry. A Transformer flow model learns from large analytically synthesized datasets with standard flow-matching supervision. Likelihood ranks candidates; rigid registration and inverse kinematics recover executable hand configurations. Strong simulation results and a retrained hardware deployment support this approach, with data quality, partial visibility and kinematic/trajectory failures still limiting performance.

1 Introduction

Large analytic grasp datasets offer a way to amortize expensive physics-based synthesis. KPGrasp asks whether the learned distribution can provide sufficient contact priors without contact-specific training losses or contact-based refinement. Its output places hand keypoints in the same Cartesian frame as the object, making spatial relationships available directly to attention.

The method still relies on analytic data generation, kinematic projection and candidate selection. It removes contact optimization from the generator, not all geometric computation from deployment. The headline Dexonomy gain is 47.4 percentage points, from 28.9% to 76.3%, rather than a 47.4% relative increase.

Complete original PDF block S1.F2
Figure 2: Motivation for Keypoint Output Parameterization. Traditional pose-joint outputs mix non-Euclidean wrist pose with joint angles, leading to SO(3) discontinuities [52], kinematic error accumulation, and loss balancing issues. Our keypoint outputs place the hand in Euclidean space, align naturally with object point clouds, and map cleanly to Transformer tokens.

2 Related Work

2.1 Data-Driven Dexterous Grasp Generation

Generative approaches capture multiple valid grasps better than deterministic regression. DexGrasp Anything, GraspTTA and DexDiffuser use physics/contact guidance, adaptation or auxiliary evaluation. D(R,O) Grasp explicitly models hand-object relationships at quadratic pairwise cost. KPGrasp instead learns a keypoint distribution from high-quality grasps, retaining model-internal likelihood ranking.

2.2 Hand Parameterization

Conventional outputs combine wrist pose and joint angles, requiring treatment of rotations, mixed coordinates and kinematic error propagation. Geometric flow models and equivariant networks address these issues differently. Keypoint manipulation policies and hand-tracking work motivate Cartesian outputs; here they become the distribution modeled by a grasp generator.

3 Preliminary

Flow matching regresses the velocity of a linear path from Gaussian noise to data. Equation 1 retains the squared-error objective; integrating the learned field produces samples. Equation 2 gives the continuous change-of-variables density relation.

Following continuous normalizing flows, a Hutchinson probe estimates divergence using a vector-Jacobian computation. The trace estimator is unbiased at a fixed state, but the deployed five-step integration gives an approximate model score. Learned density is not physical stability probability, and need not equal the true data density written in the ideal formula.

Equations 1–2
ℒFM=𝔼τ,𝐱1∼pdata,𝐱0∼pprior​‖𝐯ϕ​(𝐱τ,τ)−(𝐱1−𝐱0)‖22\mathcal{L}_{\text{FM}}=\mathbb{E}_{\tau,\mathbf{x}_{1}\sim p_{\text{data}},\mathbf{x}_{0}\sim p_{\text{prior}}}\left\|\mathbf{v}_{\phi}(\mathbf{x}_{\tau},\tau)-(\mathbf{x}_{1}-\mathbf{x}_{0})\right\|_{2}^{2} (1)
log⁡pdata​(𝐱1)=log⁡pprior​(𝐱0)−∫01Tr​(∂𝐯ϕ​(𝐱τ,τ)∂𝐱τ)​𝑑τ\log p_{\text{data}}(\mathbf{x}_{1})=\log p_{\text{prior}}(\mathbf{x}_{0})-\int_{0}^{1}\text{Tr}\left(\frac{\partial\mathbf{v}_{\phi}(\mathbf{x}_{\tau},\tau)}{\partial\mathbf{x}_{\tau}}\right)d\tau (2)
𝐱τ=(1−τ)​𝐱0+τ​𝐱1\mathbf{x}_{\tau}=(1-\tau)\mathbf{x}_{0}+\tau\mathbf{x}_{1}
d​𝐱τ/d​τ=𝐱1−𝐱0d\mathbf{x}_{\tau}/d\tau=\mathbf{x}_{1}-\mathbf{x}_{0}
Tr​(∂𝐯ϕ/∂𝐱τ)=𝔼ϵ​[ϵ⊤​(∂𝐯ϕ/∂𝐱τ)​ϵ]\text{Tr}(\partial\mathbf{v}_{\phi}/\partial\mathbf{x}_{\tau})=\mathbb{E}_{\epsilon}[\epsilon^{\top}(\partial\mathbf{v}_{\phi}/\partial\mathbf{x}_{\tau})\epsilon]

4 Method

4.1 Keypoint Grasp Parameterization

Twenty-one Shadow Hand keypoints describe joints, fingertips and rigid-palm anchors in the object point-cloud frame. Forward kinematics converts training grasps into these coordinates. Five palm points support SVD rigid registration; Mink inverse kinematics then recovers joint angles from the remaining points.

This projection enforces hand kinematic compatibility without consulting the object for contact, penetration or force closure. Palm registration error is below 0.4 mm for half the grasps and 0.99 mm for 99%; mean IK reconstruction error is about 0.7 mm. Rare invalid configurations remain.

Equation
𝐊=[𝐤1,𝐤2,…,𝐤J]⊤∈ℝJ×3\mathbf{K}=[\mathbf{k}_{1},\mathbf{k}_{2},\ldots,\mathbf{k}_{J}]^{\top}\in\mathbb{R}^{J\times 3}

4.2 Conditional Keypoint Flow Model

A Minkowski sparse-convolution encoder processes 1,024 object points; farthest-point sampling yields 128 geometry tokens of width 512. Each noisy keypoint becomes a token. Eight DiT blocks alternate hand self-attention and object cross-attention, using eight heads, FFN width 2,048 and time-conditioned adaptive normalization.

The 67.19-million-parameter model trains for 200,000 steps with batch 4,096, Muon at learning rate 2×10⁻⁴, and eight RTX 4090 GPUs, taking about two days. Inference generates one hundred candidates with five Euler steps, estimates scores using one Rademacher probe per trajectory and retains the top ten before rigid registration/IK. The score ranks learned plausibility without an external discriminator.

Complete original PDF block S4.F3
Figure 3: KPGrasp Model. We use a conditional DiT-based network to learn the keypoint flow. The model is trained only with the standard flow-matching objective. During inference, the predicted keypoints are converted into joint angles by inverse kinematics for execution. The right panel shows the adaptive layer normalization that injects the flow time τ\tau into each attention or FFN sublayer.

5 Experiment

Simulation uses complete object point clouds to isolate grasp synthesis. The real system is retrained on simulated single-view observations for a Sharpa hand. Baselines retain their native selection/refinement pipelines; those without rankers are not given an added ranking module. These comparisons assess complete reported methods, not uniformly identical inference budgets.

5.1 Benchmark on Dexonomy

Dexonomy contains 9.5 million grasps over 10.7 thousand objects and 31 grasp types. Evaluation follows its object-level split and six-direction external-force test in MuJoCo. GSR scores retained grasp proposals; OSR requires at least one successful proposal for an object. Contact consistency, penetration and diversity follow the official definitions.

KPGrasp reaches 76.3% GSR, 97.3% OSR, 5.6 mm contact-distance consistency and 2.4 mm penetration. The directly comparable Dexonomy baseline reaches 28.9% GSR; its extra-classifier variant reaches 63.9%, giving a 12.4-point rather than relative-percent gap. KPGrasp ties DexGrasp Anything’s diversity value 23.3 instead of strictly winning that metric.

The qualitative montage illustrates cleaner generated contacts, but selected images do not independently establish every grasp’s stability. Training-data synthesis and simulator force tests carry the physical prior.

Complete original PDF block S5.T1
Table 1: Benchmark on Dexonomy. KPGrasp outperforms baselines on all metrics. The gray row uses a classifier, which needs extra training data and is thus not directly comparable.
Original selectable table
Method GSR(%\%)↑\uparrow OSR(%\%)↑\uparrow CDC(m​mmm)↓\downarrow PD(m​mmm)↓\downarrow Diversity↓\downarrow
UniDexGrasp [47] 24.5 73.2 15.6 11.6 28.0
DexGrasp Anything [51] 21.8 72.2 11.0 14.2 23.3
Dexonomy (w/o classifier) [4] 28.9 78.1 14.8 11.2 24.6
Dexonomy (w/ classifier) 63.9 91.3 13.9 8.6 25.7
KPGrasp (Ours) 76.3 97.3 5.6 2.4 23.3
Complete original figure S5.F4
Figure 4: Qualitative Comparison on Dexonomy. KPGrasp produces stable and diverse grasps on complex objects, while baselines often have severe hand-object penetration.
Complete original PDF block S5.T2
Table 2: Benchmark on DexGrasp Anything. GRAB, RealD, MD, UDG, and DGN denote DexGRAB [51], RealDex [32], MultiDex [26], UniDexGrasp [47], and DexGraspNet [43]. Bold indicates the best performance, and underline indicates the second best. KPGrasp is trained on Dexonomy and evaluated without fine-tuning or contact-based test-time refinement.
Original selectable table
Method Penetration Depth (mm) Success Rate (%)
GRAB RealD MD UDG DGN Avg. GRAB RealD MD UDG DGN Avg.
UniDexGrasp [47] 37.4 39.0 13.5 24.5 31.9 29.3 20.8 27.1 21.6 23.7 33.9 25.4
GraspTTA [23] 51.4 40.1 19.0 21.2 24.5 31.2 14.4 13.3 30.3 21.0 18.6 19.5
SceneDiffuser [22] 41.1 42.0 14.6 25.1 31.0 30.8 39.1 21.7 69.8 28.3 26.6 37.1
UGG [33] 33.2 34.4 10.3 24.5 25.2 25.5 42.7 32.7 55.3 46.0 46.9 44.7
DexGraspAnything [51] 30.4 27.7 11.4 18.8 17.8 21.2 57.9 44.8 79.1 53.1 57.5 58.5
KPGrasp (Ours) 20.8 9.6 6.6 13.0 14.3 12.9 59.8 67.1 62.2 62.3 67.5 63.8

5.2 Benchmark on DexGrasp Anything

The Dexonomy-trained model transfers to five DexGrasp Anything object sets without fine-tuning, with training/evaluation object overlap excluded. This protocol uses IsaacGym and point-cloud-based penetration, so its numbers are not directly interchangeable with Dexonomy metrics.

Mean success is 63.8% versus DGA 58.5%, and mean penetration is 12.9 versus 21.2 mm. KPGrasp wins four of five success columns; on MultiDex it trails DGA 62.2% versus 79.1%, and SceneDiffuser also scores higher at 69.8%.

Retraining DGA on Dexonomy changes its mean only to 59.1%; removing its contact refinement gives 46.6% on original training data and 46.8% on Dexonomy. Data source alone therefore does not explain the comparison, while Appendix E shows that data quality remains essential for KPGrasp itself.

Complete original PDF block S5.T3
Table 3: Ablation Study on Parameterization, Tokenization, and Likelihood Ranking.
Original selectable table
Method GSR(%\%)↑\uparrow OSR(%\%)↑\uparrow CDC(m​mmm)↓\downarrow PD(m​mmm)↓\downarrow Diversity↓\downarrow
Pose-Joint (w/ Euclidean Flow) 62.5 92.0 10.3 3.7 25.7
Pose-Joint (w/ S​E​(3)SE(3) Flow) 55.0 93.9 11.8 5.7 23.8
Pose-Joint (w/ 24 tokens) 63.5 94.6 9.7 3.3 25.8
Pose-Joint (w/ 34 tokens) 66.1 95.1 9.5 3.2 25.7
Keypoint (w/ Single Token) 55.4 93.2 10.1 8.6 23.7
Keypoint (w/o likelihood ranking) 67.0 96.4 7.4 4.4 23.2
Keypoint (Ours) 76.3 97.3 5.6 2.4 23.3
Complete original figure S5.F5
Figure 5: Scaling Behavior. KPGrasp improves as training data, model size, and batch size increase. Keypoint parameterization consistently outperforms the conventional pose-joint variant.

5.3 Ablation Study

Pose-joint outputs with Euclidean rotation projection obtain 62.5% GSR, versus 55.0% with an SE(3) flow. Giving pose-joint models 24/34 tokens yields 63.5%/66.1%; collapsing keypoints to one token gives 55.4%. Representation and spatial tokenization both matter; token count alone does not explain 76.3%.

Without likelihood ranking, success falls to 67.0%, a 9.3-point decrease, while the diversity metric slightly improves from 23.3 to 23.2. Ranking improves selected quality rather than every metric.

Figure 5 varies data, block count and batch size. Larger batches at fixed steps also see more examples. A sample-count-matched batch-512 run takes 1.6 million steps/about thirteen days and reaches 73.5%, versus 76.3% for batch 4,096. Sixteen blocks reach 77.6%; the eight-block default balances capacity and cost. These sweeps do not establish a universal scaling law.

5.4 Real World Experiments

The Galbot platform uses a Sharpa right hand and left-arm RealSense D405 camera. Retargeted analytic synthesis produces 3.3 million tabletop grasps over five thousand objects with simulated partial clouds; KPGrasp is retrained for this setup.

GroundedSAM segments the object, then likelihood-ranked candidates undergo collision filtering and trajectory interpolation. Five trials on each of twenty objects yield 83/100 successful lifts. This is deployability evidence for a retrained, filtered pipeline, not zero-shot transfer of the Shadow Hand model or a broad hardware benchmark. Small pose/calibration errors can invalidate sensitive grasps.

Complete original PDF block S5.F6
Figure 6: Real World Setup. (Left) 20 test objects. (Right) Robot setup.

5.5 Inference Time Analysis

Reported inference includes network sampling, likelihood ranking and IK: four objects per batch, one hundred candidates per object and ten retained per object give 0.032 seconds per returned grasp. This is amortized throughput, not single-grasp request latency or the full perception-to-executed-lift cycle. IK additionally uses 64 CPU workers.

The paper compares roughly 2.8 seconds per DGA grasp with contact refinement and estimates about 24 successful grasps per second by multiplying throughput by 76.3% simulated success. Its comparison with analytic dataset generation uses separately reported hardware/workload statistics; it is not a matched end-to-end experiment.

6 Limitation

Quality depends on the coverage and correctness of analytic training data. Occlusion and sensing errors degrade grasps; likelihood is not a stability certificate, IK can fail, and execution needs collision handling. Real tests cover only twenty objects and one retrained hand/platform. A unified model across different hands remains open.

7 Conclusion

Cartesian keypoint flows plus large grasp datasets provide an effective learned proposal distribution. The strongest evidence is high selected-grasp quality in simulation and successful retrained hardware deployment, with explicit dependence on dataset quality, kinematic recovery and execution filtering. The full gallery below retains all 48 examples.

Complete original PDF block S7.F7
Figure 7: Grasp Gallery. KPGrasp generates similar grasps for objects with the same local geometries. This also shows more examples of high-quality grasps for complex geometries.

Appendix A Inverse-Kinematics Module Analysis

The appendix measures how closely projected robot configurations reproduce generated keypoints and reports the small but nonzero failure tail.

Implementation Details

SVD aligns canonical and predicted palm anchors to recover wrist pose. Position-only keypoint tasks in Mink with the quadprog backend solve the remaining configuration, integrating IK velocity for at most five iterations. This is kinematic optimization rather than object-contact refinement.

Evaluation Settings

The test covers 128,340 attempts: 2,139 objects × six scales × ten proposals. Forward kinematics reconstructs solved keypoints, and Euclidean discrepancies measure agreement with the generated points. These reconstruction errors are distinct from grasp success and hand-object penetration.

Results

Average mean/max errors are 0.669/1.861 mm; 97.37% have maximum error at most 5 mm. Half have mean error below 0.5 mm, and 99% below 2.5 mm. About 0.2% exhibit clear failures such as backward-folded fingers, illustrated in Figure 8.

Adding feasibility constraints to intermediate terminal estimates reduces that failure rate to 0.08% but raises GSR by only 0.4 percentage points. Amortized IK runtime is 0.002 seconds with 64 workers on a 32-core AMD EPYC CPU, not serial per-request latency.

Complete original PDF block A1.F8
Figure 8: Predicted keypoints (red) and FK keypoints reconstructed from the IK solution(blue). The third example shows a rare backward-folding failure (∼0.2%\sim 0.2\% of grasps).
Complete original PDF block A1.T4
Table 4: IK projection accuracy and runtime. Predicted keypoints can be accurately and efficiently projected into configuration space.
Original selectable table
Metric Value
IK max keypoint error ≤\leq 5 mm 97.37%
Avg. mean keypoint error 0.669 mm
Avg. max keypoint error 1.861 mm
64-worker amortized runtime 0.002 s

Appendix B Differentiable FK loss for Pose-Joint variant

A pose-joint model trained only through differentiable forward-kinematics error produces grasp-like configurations but roughly 1% GSR. Adding that loss to standard flow matching gives 66.0%, essentially the 66.1% pose-joint baseline. Cartesian supervision alone does not reproduce the benefit of modeling the flow directly over spatial keypoint tokens.

Appendix C Partial Observation Results

Retraining on single-view point clouds gives 67.4% simulated GSR, 8.9 percentage points below the complete-geometry result 76.3%. Occluded geometry is the authors’ explanation; this is a separate simulated partial-observation comparison, not the 83% physical lift metric.

Appendix D Comparison of Different Sampling Steps

Naive one-step flow sampling nearly fails. A separately distilled FlowMap using Lagrangian Self-Distillation reaches 52.7% at 0.010 seconds per grasp. Three/five/seven Euler steps give 66.5%/76.3%/76.0% at 0.024/0.032/0.040 seconds. Five is the selected quality-throughput trade-off.

Scores use discretized divergence integration; the one-step variant approximates the integral with average velocity. Its distillation and score approximation differ from merely reducing the original model’s solver steps, and finite-step scores are not exact likelihoods of an ideal continuous flow.

Complete original PDF block A4.T5
Table 5: Comparison of different sampling steps. We use 5 Euler steps in the final model because they provide the best trade-off between quality and speed.
Original selectable table
Method Success Rate (%) Inference Time (s)
Ours (1-step FlowMap) 52.7 0.01
Ours (3-step flow matching) 66.5 0.024
Ours (5-step flow matching) 76.3 0.032
Ours (7-step flow matching) 76.0 0.04

Appendix E Additional Results of DexGrasp Anything Benchmark

Training KPGrasp separately on the original five benchmark datasets yields mean success 47.8%, below DGA’s 58.5%; Dexonomy-trained KPGrasp reaches 63.8%. The original training grasps themselves score only 42.8% under the DGA evaluation, suggesting filtering/evaluation mismatch and uneven data quality.

This qualifies the headline: the simple generator benefits substantially from high-quality large-scale data. The comparison does not isolate dataset size, quality and protocol mismatch as independent causes, and DGA’s explicit priors can be more helpful under weaker training data.

Complete original PDF block A5.T6
Table 6: Success rate (%) on the DexGrasp Anything benchmark. Bold indicates the best performance, and underlined indicates the second best. We analyze the results in Sec. E.
Original selectable table
GRAB RealD MD UDG DGN Avg.
Original DGA Training Datasets 62.8 32.7 40.6 22.8 55.2 42.8
DGA Model 57.9 44.8 79.1 53.1 57.5 58.5
KPGrasp (Trained on each dataset) 44.7 45.6 54.7 33.1 61.8 47.8
KPGrasp (Trained on Dexonomy) 59.8 67.1 62.2 62.3 67.5 63.8

Appendix F Additional Details of Real World Experiment

The hardware pipeline changes the hand, retrains on tabletop partial observations and adds collision checks around grasp execution.

F.1 Sharpa Hand Keypoint Annotation

Sharpa receives its own twenty-one-keypoint annotation at mechanical joints, fingertips and rigid-palm anchors. Reusing the representation design does not imply the same trained network transfers unchanged between hand embodiments.

Complete original figure A6.F9
Figure 9: Sharpa Hand Keypoint Annotation. We annotate J=21J=21 keypoints on Sharpa Hand for real world experiment.

F.2 Implementation Details

Shadow Hand templates are retargeted to Sharpa before analytic regeneration. Simulated cameras vary height 30–40 cm and horizontal distance 30–50 cm. At deployment, object segmentation and an estimated table plane align observations with this training distribution.

Candidate interpolation is filtered for collisions. Documented hazards include near-table hand poses, the unused right-wrist camera hitting the table after arm IK, and paths crossing the object when start/target grasps lie on opposite sides. Collision-aware planning and larger synthesis margins are proposed future improvements; grasp quality alone does not ensure a safe approach trajectory.

Concept demonstration

Open image ↗