Literature
Display
← All papers

DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine

Haozhe Lou · Runyi Yang · Wanqi Zhong · Caiyun Liu · Yicheng Liu · Wenhao Liu · Wenlong Ma · Jiawei Xia · Kanzhi Wu · Danda Pani Paudel · Luc Van Gool · Hang Zhao · Yiming Li

Preprint posted · CoRL 2026 (accepted) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete 28-page paper and supplement. The PDF’s main Figures 1–5 and supplementary Figures 1–5 remain distinct, as do the two sets of Tables 1–4. The source’s detached, flattened teaser is joined to its caption and restored from the native PDF. All original figure/table detail, equations, curation algorithm and project presentation/demonstrations remain alongside the notes. Lou and Yang contributed equally.

Complete original Figure 1: Real2Sim data densification pipeline. (a) Real-world demonstration sample via teleoperation. (b) Real-world multi-modal sensing, and register object into a unified coordinate frame. (c) Simulation Data example (d) Differentiable simulator to generate trajectory. (e) Detection and summary agents extract structured object knowledge, which guides generative agents to produce diverse, in-domain instances to get controllable 3D generation.
Figure 1: Real2Sim data densification pipeline. (a) Real-world demonstration sample via teleoperation. (b) Real-world multi-modal sensing, and register object into a unified coordinate frame. (c) Simulation Data example (d) Differentiable simulator to generate trajectory. (e) Detection and summary agents extract structured object knowledge, which guides generative agents to produce diverse, in-domain instances to get controllable 3D generation.

At a glance

DexNinja augments scarce real food-cutting demonstrations with reconstructed/generated assets, deformable simulation and trajectory refinement. Its main four-food experiment reaches 50/100 real cutting successes, versus 31/100 with real data alone, a 19-percentage-point gain. A separate scaling study reaches 63/100 when the reported simulated-episode budget increases to 300. Thus the abstract’s above-60% result belongs to the larger-data setting. Policies remain category-specific, with generalization tested mainly on unseen instances inside trained categories.

1 Introduction

Cutting changes material shape, contact and topology, making both repeated real collection and simulation transfer difficult. The data engine anchors new geometry/material variation to real captured examples, then admits feasible simulated rollouts. Differentiable simulation supports trajectory fitting/refinement; the deployed arm/hand policies are trained by supervised behavior cloning. The paper does not demonstrate end-to-end gradients through physical execution or one general policy for arbitrary food categories.

2 Related Works

The related work connects synthetic demonstrations, differentiable cutting and dexterous/tactile manipulation. The contribution is their combination into an anchored data-generation and deployment pipeline, rather than simulation replacing all real sensing or calibration.

2.1 Data generation and differentiable simulation

Trajectory-generation and Gaussian-splatting methods broaden motion/appearance coverage. RoboNinja, DiSECt and tactile simulators address deformable contact and cutting [8] [43] [44]. DexNinja adds reconstructed category assets, material randomization, trajectory admission and a bridge to articulated scene simulation. The implementation explicitly splits local cutting physics from the full robot engine.

2.2 Dexterous manipulation and Teleoperation

Dexterous grasping and teleoperation provide executable initial grasps and real supervision. Tactile signals help distinguish stable contact, slip and compression when visual geometry alone is insufficient. Frame alignment and sensor mapping support transfer; they do not remove embodiment-specific calibration or establish zero-shot transfer between arbitrary hands.

3 Problem Formulation

Observation combines a scene point cloud, fingertip tactile arrays and proprioception. Food parameters include mass, friction, Young’s modulus, Poisson ratio and changing geometry; the knife is a separate rigid tool. Kinematic and contact-sensitive action components are learned with separate losses. The arm branch also relies on object-centric motion specification and IK, while the hand branch responds to tactile/proprioceptive feedback. All original definitions and the elasticity conversion remain.

Equations 1–4
𝐨t={𝐨tvis,𝐨ttac,𝐨tprop}∈𝒪,\mathbf{o}_{t}=\big\{\mathbf{o}^{\text{vis}}_{t},\ \mathbf{o}^{\text{tac}}_{t},\ \mathbf{o}^{\text{prop}}_{t}\big\}\in\mathcal{O}, (1)
ϑt=(m,μf,E,ν,𝒢⁡(t)),\vartheta_{t}=\big(m,\mu_{f},E,\nu,\mathcal{G}(t)\big), (2)
λ=E​ν(1+ν)​(1−2​ν),G=E2​(1+ν),\lambda=\frac{E\nu}{(1+\nu)(1-2\nu)},\qquad G=\frac{E}{2(1+\nu)}, (3)
𝐨tvis={pk}k=1NP,pk∈ℝ3\mathbf{o}^{\text{vis}}_{t}=\{p_{k}\}_{k=1}^{N_{P}},\qquad p_{k}\in\mathbb{R}^{3} (4)
Complete original Figure 2: Tactile perception visualization of representative food-cutting examples.
Figure 2: Tactile perception visualization of representative food-cutting examples.
Equations 5–6
at=(atkin,attac),atkin∈𝒜kin,attac∈𝒜tac,a_{t}=\big(a_{t}^{\text{kin}},\,a_{t}^{\text{tac}}\big),\qquad a_{t}^{\text{kin}}\in\mathcal{A}^{\text{kin}},\;\;a_{t}^{\text{tac}}\in\mathcal{A}^{\text{tac}}, (5)
a^tkin\displaystyle\hat{a}_{t}^{\text{kin}} =fϕ(𝐨tvis),a^ttac=πθ(𝐨0:ttac,𝐨0:tprop),\displaystyle=f_{\phi}\!\left(\mathbf{o}_{t}^{\text{vis}}\right),\quad\hat{a}_{t}^{\text{tac}}=\pi_{\theta}\!\left(\mathbf{o}_{0:t}^{\text{tac}},\mathbf{o}_{0:t}^{\text{prop}}\right), (6)
ℒ⁡(a^t,at)\displaystyle\mathcal{L}\!\left(\hat{a}_{t},a_{t}\right) =ℒkin​(a^tkin,atkin)+ℒtac​(a^ttac,attac).\displaystyle=\mathcal{L}_{\text{kin}}\!\left(\hat{a}_{t}^{\text{kin}},a_{t}^{\text{kin}}\right)+\mathcal{L}_{\text{tac}}\!\left(\hat{a}_{t}^{\text{tac}},a_{t}^{\text{tac}}\right).

4 Real2Sim2Real pipeline: Dexninja

Stage I reconstructs and registers real food/tool geometry from synchronized teleoperation. Stage II varies assets/materials, refines knife trajectories, solves IK and filters rollouts through collision/cutting checks. Stage III clones the accepted multimodal trajectories with separate arm/hand heads. A stage token controls chunk horizons: nominal 64 for approach/grasp, 128 for knife positioning and 16 for contact-rich cutting. These are step counts; the paper does not supply a common control frequency that would turn them into fixed durations.

Complete original Figure 3: Learning policy to control arm and dexterous hand separately via adaptive action chunking. We give an real-world inference result at bottom.
Figure 3: Learning policy to control arm and dexterous hand separately via adaptive action chunking. We give an real-world inference result at bottom.
Equations 7–8
a^t:t+Ht−1kin=fϕ(𝐨tvis,zt),a^t:t+Ht−1tac=πθ(𝐨ttac,𝐨tprop,zt),\hat{a}_{t:t+H_{t}-1}^{\text{kin}}=f_{\phi}\!\left(\mathbf{o}_{t}^{\text{vis}},z_{t}\right),\quad\hat{a}_{t:t+H_{t}-1}^{\text{tac}}=\pi_{\theta}\!\left(\mathbf{o}_{t}^{\text{tac}},\mathbf{o}_{t}^{\text{prop}},z_{t}\right), (7)
Ht∼p⁡(H∣zt)=λ​Unif​(Hmin​(zt),H¯​(zt))+(1−λ)​Unif​(H¯​(zt),Hmax​(zt)).H_{t}\sim p(H\mid z_{t})=\lambda\,\mathrm{Unif}\!\big(H_{\min}(z_{t}),\bar{H}(z_{t})\big)+(1-\lambda)\,\mathrm{Unif}\!\big(\bar{H}(z_{t}),H_{\max}(z_{t})\big). (8)

5 Continuation on a Shape Manifold for Data Augmentation

The augmentation section describes category-conditioned shape variation and local correction around successful examples. Its mathematical shape-manifold/continuation view is a local modeling argument. Practical generation uses descriptions, image cleaning, image-to-3D and multiple admission checks; semantic plausibility alone is not treated as physical cutting feasibility.

Complete original Figure 4: On-manifold approximation of real-data neighborhood for data augmentation pipeline.
Figure 4: On-manifold approximation of real-data neighborhood for data augmentation pipeline.

Data augmentation pipeline

A detector identifies the food category and a summary extracts shape, surface, color and ripeness attributes. Text/image/3D generation diversifies those attributes while remaining anchored to captured examples. Geometric and simulator checks then decide which assets and trajectories enter training. The pipeline does not claim every language-generated instance is automatically a valid physical material.

Shape generation and local manifold structure

A latent generator G formalizes a local family of category shapes. The stated assumption is local regularity, with a locally Lipschitz generator offered as sufficient. The paper does not measure a certified smooth neighborhood for every generated asset. The implementation’s curated image-to-3D pool is evidence of useful variation, rather than a proof that all assets lie on a known smooth manifold.

Equation 9
𝒢:𝒵→ℳ,z↦𝒢⁡(z),\mathcal{G}:\mathcal{Z}\rightarrow\mathcal{M},\qquad z\mapsto\mathcal{G}(z), (9)

Local regularity and analytical reliability

A first-order expansion of the visual-to-action map motivates small corrections near a successful anchor. Local Lipschitz behavior requires suitable local smoothness/bounded derivative assumptions; differentiability at one point alone is insufficient for the stated neighborhood bound. The argument also does not guarantee task success across contact/topology changes. Empirical simulation admission is still required.

Equation 10
f⁡(o)=f⁡(o⋆)+Jf​(o⋆)​(o−o⋆)+r⁡(o),f(o)\;=\;f(o^{\star})\;+\;J_{f}(o^{\star})(o-o^{\star})\;+\;r(o), (10)

Neighborhood of demonstrations

The formal local corrective policy maximizes simulated feasibility while penalizing departure from the demonstrated action. State, shape and material perturbations define the neighborhood distribution. This expresses how continuation should remain near successful behavior. The supplement concretely describes trajectory optimization and filtering, without a complete optimizer/architecture specification for each per-anchor corrective policy in the formal equation.

Equation 11
θi⋆=arg⁡maxθ​𝔼o∼μi​[S⁡(o,Γθ​(o))−β​‖Γθ​(o)−ai‖2],\theta_{i}^{\star}\;=\;\arg\max_{\theta}\;\mathbb{E}_{o\sim\mu_{i}}\Big[S\!\big(o,\Gamma_{\theta}(o)\big)\;-\;\beta\,\|\Gamma_{\theta}(o)-a_{i}\|^{2}\Big], (11)

Augmented data generation

Augmented pairs are retained only after successful simulated outcomes. This biases the training pool toward demonstrated-neighborhood feasibility rather than arbitrary perturbations. It also means rejected samples and uncovered recovery modes are not automatically learned; later real failures reveal remaining grasp/contact and material-model gaps.

Equation 12
𝒟aug=𝒟0∪⋃i=1N0{(o,a):o∼μi,a=Γθi⋆(o),S(o,a)=1}.\mathcal{D}_{\text{aug}}=\mathcal{D}_{0}\;\cup\;\bigcup_{i=1}^{N_{0}}\Big\{(o,a)\;:\;o\sim\mu_{i},\;a=\Gamma_{\theta_{i}^{\star}}(o),\;S(o,a)=1\Big\}. (12)

Real-world anchoring and admission

Plausibility is operational: semantic/multi-view consistency, scale, valid geometry, simulator compatibility and stable initialization. Rollouts additionally pass IK, collision and cutting-success checks. Randomized mass/friction/elasticity stay category-bounded, although full numerical ranges and every admission threshold are not provided in the manuscript.

6 Experimental Evaluation

Real evaluation distinguishes reaching contact, completing a cut and preserving food integrity. It tests different training sources, selected longer-horizon examples and matched-budget component removals. The main quantitative results use the primary embodiment; the second setup supplies qualitative verification.

6.1 Task setup and evaluation protocol

Four categories—strawberry, apple, banana and potato—have 25 held-out instances each. The stated training protocol uses five real demonstrations and fifty simulated sequences, with a separate scaling study. Progress reaches valid initial cutting contact; Success completes traversal/separation and termination; Integrity is a manual binary damage judgment. The three counts are reported separately, without their joint success-and-integrity rate. Random planar placement remains inside a reachable workspace; numerical placement bounds and rater reliability are not specified.

Complete original Table 1: Sim-to-real gap and the effect of mixing simulation and real demonstrations, with 25 trials per food.
Table 1: Sim-to-real gap and the effect of mixing simulation and real demonstrations, with 25 trials per food.
Accessible table and original citation links
Training
regime
Food Progress ↑\uparrow Success ↑\uparrow Integrity ↑\uparrow
Sim-only Strawberry 10/25 6/25 10/25
Apple 5/25 4/25 5/25
Banana 16/25 12/25 14/25
Potato 15/25 12/25 11/25
Avg. 46/100 34/100 40/100
Real-only Strawberry 12/25 8/25 10/25
Apple 6/25 4/25 4/25
Banana 17/25 12/25 14/25
Potato 9/25 7/25 9/25
Avg. 44/100 31/100 37/100
Sim+Real
(Ours)
Strawberry 19/25 12/25 14/25
Apple 14/25 8/25 7/25
Banana 24/25 18/25 23/25
Potato 19/25 12/25 16/25
Avg. 76/100 50/100 60/100

6.2 Sim-to-real gap and the effect of mixing sim and real data

Across 100 trials per training regime, sim-only gives 46% progress, 34% success and 40% integrity; real-only gives 44/31/37%; mixed data gives 76/50/60%. The mixed success gain over real-only is 19 points, or about 61% relative. Gains are uneven: mixed success is 12/25 strawberry, 8/25 apple, 18/25 banana and 12/25 potato. Apple has eight cutting successes but seven integrity passes, underscoring the separately reported criteria. None of these marginal counts supplies a joint damage-free success rate.

6.3 Complex Motion and Heterogeneous Objects

Lemon trials score 8/10 single-cut, 7/10 two-cut and 3/10 multi-cut. More cuts require repeated adjustment while geometry and contact evolve. The paper says training details follow the main protocol; it does not establish that lemon performance is zero-shot transfer from the four original category policies. The selected videos/filmstrips illustrate behavior and failure, rather than adding to those trial counts.

Complete original Figure 5: Real-world lemon cutting results under different cutting requirements. The success rate is declining as the manipulation complexity increase.
Figure 5: Real-world lemon cutting results under different cutting requirements. The success rate is declining as the manipulation complexity increase.

6.4 Component ablations and failure analysis

Matched-budget removals lower success from 50% to 46% without QC, 44% without material randomization, 39% without trajectory refinement, 43% without tactile observation and 42% without generated assets. Tactile removal has the largest integrity drop, 60→45%. The fifty failed mixed-data trials receive one earliest-cause label: eight registration, nineteen grasp/contact, seventeen material-physics mismatch and six kinematic/collision. These are assigned failure categories, not independent causal effect estimates. Asset curation keeps 1,200 from 1,670 proposals through successive checks.

Complete original Table 2: Component ablations. Δ \Delta is relative to Full DexNinja.
Table 2: Component ablations. Δ\Delta is relative to Full DexNinja.
Accessible table and original citation links
Variant P/S/I Δ\DeltaP Δ\DeltaS Δ\DeltaI
Full DexNinja (Table 1) 76/50/60 – – –
w/o Quality Control 72/46/55 −4-4 −4-4 −5-5
w/o Physical Randomization 72/44/51 −4-4 −6-6 −9-9
w/o Trajectory Refinement 65/39/53 −11-11 −11-11 −7-7
w/o Tactile Observation 73/43/45 −3-3 −7-7 −15-15
w/o Generated Assets (Reconstruction Only) 67/42/50 −9-9 −8-8 −10-10
Complete original Table 3: Failed-trial analysis.
Table 3: Failed-trial analysis.
Accessible table and original citation links
Cause Trials
Perception/registration 8
Grasp/contact control 19
Cutting-physics mismatch 17
Kinematic/safety/collision 6
Complete original Table 4: Quality check of assets.
Table 4: Quality check of assets.
Accessible table and original citation links
Category Proposed 2D QC 3D QC Accepted
Strawberry 397 361 326 300
Apple 376 349 321 300
Banana 453 407 354 300
Potato 444 396 343 300
Clean total 1,670 1,513 1,344 1,200

7 Conclusion

The engine combines real anchoring with simulated geometric/material diversity and accepted trajectory supervision. Evidence supports better within-category cutting transfer than either single data source in the reported setup. It leaves substantial real failures and more difficult multi-cut adjustment unresolved.

8 Limitations

The mounted camera constrains deployment, and current policies are object/category-specific. New categories require new assets and retraining or fine-tuning. Cross-embodiment evidence is qualitative, with calibration/kinematics/action scaling replaced for the new system.

Acknowledgments

The acknowledgment names a vivo–INSAIT collaboration and Bulgarian Ministry of Education and Science support. The full 88-reference bibliography remains, with links from the notes and retained tables.

Original project presentation and demonstrations

The primary DexNinja project supplies an unchanged narrated presentation and three unchanged physical demonstrations, all preserved with original posters. Strawberry/banana clips are labeled 4× speed and the second-setup lemon clip 8×. The original presentation chapter guide is 0:00 Motivation, 0:30 Pipeline, 2:00 Policy learning, 2:30 Experiments and 4:12 Deployment. These are selected demonstrations, rather than extra trials to add to the paper’s success counts.

DexNinja narrated research presentation
Real-world strawberry cutting demonstration, shown at four times speed
Real-world banana cutting demonstration, shown at four times speed
Real-world lemon cutting on a second robot setup, shown at eight times speed

Supplementary Material

The complete supplement covers local cutting simulation, scene registration, policy training, hardware, asset curation, additional quantitative experiments and qualitative executions. Its equation and figure numbering restarts independently of the main text.

1 Simulation Details

Simulation transfers a demonstrated knife motion through explicit world, food, knife and end-effector frames. Local cutting physics supplies contact information, while scene-level kinematics handle the articulated robot. The bridge is important because the chosen robot engine does not natively support the required food cutting dynamics.

1.1 Stage I: Teleoperation and Object Frame

A successful demonstration records synchronized sensor/action streams and estimated food pose. Expressing the knife trajectory in the food frame removes its original world placement. This canonical blade/object relation is retargeted to reconstructed instances; it still requires estimating object pose and the grasp transform accurately.

Equation 1
𝐓KS​(t)=(𝐓SW)−1​𝐓KW​(t),𝐓K,fS=𝐓KS​(tf).{}^{S}\mathbf{T}_{K}(t)=\left({}^{W}\mathbf{T}_{S}\right)^{-1}\,{}^{W}\mathbf{T}_{K}(t),\qquad{}^{S}\mathbf{T}_{K,f}={}^{S}\mathbf{T}_{K}(t_{f}). (1)

1.2 Stage II: Differentiable Cutting Simulator Details

A fixed end-effector/knife transform maps perturbed object-frame knife paths into world-frame robot targets. IK filters joint-space realizations before simulation generates aligned observations/actions. The rigid-grasp transform assumes the knife does not slip in the hand; tool-grasp failures are part of the remaining transfer challenge.

Equations 2–3
𝐓KE​(t)=(𝐓EW​(t))−1​𝐓KW​(t),{}^{E}\mathbf{T}_{K}(t)=\left({}^{W}\mathbf{T}_{E}(t)\right)^{-1}\,{}^{W}\mathbf{T}_{K}(t), (2)
𝐓(i)EW​(t)=𝐓SW​𝐓(i)KS​(t)​(𝐓KE)−1.{}^{W}\mathbf{T}_{E}^{(i)}(t)={}^{W}\mathbf{T}_{S}\;{}^{S}\mathbf{T}_{K}^{(i)}(t)\;\left({}^{E}\mathbf{T}_{K}\right)^{-1}. (3)

Trajectory generation loss.

The differentiable objective samples blade points and uses ground SDF distance, weighted progress, penetration, rotation and motion-consistency terms. Positive SDF is outside the ground. The nonnegative tolerance m permits small numerical penetration; it is not a required positive clearance. The distance-to-ground cutting proxy helps refine paths, while successful material separation is separately checked through the rollout predicate. The complete loss formulas remain, including all averaging and weighting terms.

Equations 4–11
𝐩i,k=𝐱i+αk​wknife​𝐝𝐢𝐫​(θi),αk=kK−1\mathbf{p}_{i,k}=\mathbf{x}_{i}+\alpha_{k}\,w_{\text{knife}}\,\mathbf{dir}(\theta_{i}),\quad\alpha_{k}=\frac{k}{K-1} (4)
𝐝𝐢𝐫⁡(θi)=[−sin⁡θicos⁡θi0]⊤\mathbf{dir}(\theta_{i})=\begin{bmatrix}-\sin\theta_{i}&\cos\theta_{i}&0\end{bmatrix}^{\!\top} (5)
gi=1K​∑k=0K−1srelu⁡(d⁡(𝐩i,k)).g_{i}=\frac{1}{K}\sum_{k=0}^{K-1}\mathrm{srelu}\!\big(d(\mathbf{p}_{i,k})\big). (6)
ℒcut=1T+1​∑i=0Twi​gi2+λprogT​∑i=1T[srelu⁡(gi−(1−ρ)​gi−1)]2,\mathcal{L}_{\text{cut}}=\frac{1}{T+1}\sum_{i=0}^{T}w_{i}\,g_{i}^{2}\;+\;\frac{\lambda_{\text{prog}}}{T}\sum_{i=1}^{T}\Big[\mathrm{srelu}\!\big(g_{i}-(1-\rho)g_{i-1}\big)\Big]^{2},\quad (7)
wi=(i+1T+1)2.w_{i}=\left(\frac{i+1}{T+1}\right)^{2}. (8)
ℒcol=1(T+1)​K​∑i=0T∑k=0K−1[srelu⁡(−(d⁡(𝐩i,k)+m))]4.\mathcal{L}_{\text{col}}=\frac{1}{(T+1)K}\sum_{i=0}^{T}\sum_{k=0}^{K-1}\Big[\mathrm{srelu}\!\big(-(d(\mathbf{p}_{i,k})+m)\big)\Big]^{4}. (9)
ℒrot=1T​∑i=0T−1(θi+1−θi)2,ℒmove=1T+1​∑i=0T(θik−θiv)2\mathcal{L}_{\text{rot}}=\frac{1}{T}\sum_{i=0}^{T-1}(\theta_{i+1}-\theta_{i})^{2},\quad\mathcal{L}_{\text{move}}=\frac{1}{T+1}\sum_{i=0}^{T}(\theta_{i}^{k}-\theta_{i}^{v})^{2} (10)
ℒ=wcut​ℒcut+wcol​ℒcol+wrot​ℒrot+wmove​ℒmove\mathcal{L}=w_{\text{cut}}\mathcal{L}_{\text{cut}}+w_{\text{col}}\mathcal{L}_{\text{col}}+w_{\text{rot}}\mathcal{L}_{\text{rot}}+w_{\text{move}}\mathcal{L}_{\text{move}} (11)

1.3 Tactile Simulation During Cutting

A grasp prior initializes the stabilizing hand. The MPM cutting layer estimates particle/sensor contact forces from penetration and relative tangential motion, bins them spatially and applies small finger corrections when slip is indicated. Those simulated corrections become supervision. Contact positions/directions are expressed in the active tool/end-effector frame, with a scale/offset/noise mapping to real readings. This is model-based synthetic tactile feedback, with residual mismatch reflected in real failures.

1.4 Scene level and hardware registration

Canonical targets are registered to the scene and converted through IK/time parameterization with motion limits. Stabilization precedes the knife motion on a shared timeline. The local deformable simulator passes knife pose, contact points and forces/moments or a net wrench to the articulated scene simulator. This modular bridge supplies consistent interaction information; the paper does not describe a single monolithic robot/food solver or certify every transferred contact outcome.

2 Training Details

The final supervised policy retains multimodal observations and separate kinematic/contact action groups. The curriculum first reaches reliable pre-contact configurations, then uses tactile feedback during cutting. Its training description includes normalized chunk prediction and executed-increment smoothing, rather than a full real-world reinforcement-learning objective.

Equation
𝐨t={𝐨tvis,𝐨ttac,𝐨tprop},at=(atkin,attac).\mathbf{o}_{t}=\left\{\mathbf{o}^{\mathrm{vis}}_{t},\mathbf{o}^{\mathrm{tac}}_{t},\mathbf{o}^{\mathrm{prop}}_{t}\right\},\qquad a_{t}=\left(a^{\mathrm{kin}}_{t},a^{\mathrm{tac}}_{t}\right).

Why a single globally normalized action head is unstable.

Finger and arm actions have different ranges, so one output normalization can make hand motion dominate training or leave it under-actuated. Separate heads and group/stage statistics address that scaling problem. The later ablation measures the concrete head design, without proving every universal-head architecture must fail.

2.1 Stage-conditioned action chunking

Stage labels are approach/grasp, knife positioning and tactile cutting. A two-sided uniform mixture samples horizons around their 64/128/16 nominal values during post-training. At inference the stage is determined by geometry/contact rules. The manuscript leaves exact horizon bounds and mixture weights unspecified, so the complete stochastic execution schedule cannot be reconstructed from the nominal values alone.

Equations 12–13
𝒟aug={(𝐨1:Tn(n),a1:Tn(n),z1:Tn(n))}n=1N\mathcal{D}_{\mathrm{aug}}=\left\{\left(\mathbf{o}^{(n)}_{1:T_{n}},a^{(n)}_{1:T_{n}},z^{(n)}_{1:T_{n}}\right)\right\}_{n=1}^{N}
𝒵={zapp,zknife,zcut},\mathcal{Z}=\left\{z^{\mathrm{app}},z^{\mathrm{knife}},z^{\mathrm{cut}}\right\},
a~^t:t+Ht−1kin=fϕ(𝐨tvis,zt),a~^t:t+Ht−1tac=πθ(𝐨ttac,𝐨tprop,zt),\hat{\tilde{a}}^{\mathrm{kin}}_{t:t+H_{t}-1}=f_{\phi}\left(\mathbf{o}^{\mathrm{vis}}_{t},z_{t}\right),\qquad\hat{\tilde{a}}^{\mathrm{tac}}_{t:t+H_{t}-1}=\pi_{\theta}\left(\mathbf{o}^{\mathrm{tac}}_{t},\mathbf{o}^{\mathrm{prop}}_{t},z_{t}\right), (12)
Ht∼p⁡(H∣zt)=λH​Unif​(Hmin​(zt),H¯​(zt))+(1−λH)​Unif​(H¯​(zt),Hmax​(zt)).H_{t}\sim p(H\mid z_{t})=\lambda_{H}\,\mathrm{Unif}\left(H_{\min}(z_{t}),\bar{H}(z_{t})\right)+(1-\lambda_{H})\,\mathrm{Unif}\left(\bar{H}(z_{t}),H_{\max}(z_{t})\right). (13)
H¯​(zapp)=64,H¯​(zknife)=128,H¯​(zcut)=16.\bar{H}(z^{\mathrm{app}})=64,\qquad\bar{H}(z^{\mathrm{knife}})=128,\qquad\bar{H}(z^{\mathrm{cut}})=16.

2.2 Group-wise action normalization and execution

Targets use group- and stage-specific clipped z-scores. Execution maps normalized outputs to scaled joint increments, then projects into a robot-specific command set. Arm steps are described as smaller than hand steps. Numerical step sizes and the full projection implementation are not provided. This is the stated command mapping rather than simply undoing the target z-score; the original equations preserve that distinction.

Equations 14–16
𝝁g​(z),𝝈g​(z).\boldsymbol{\mu}_{g}(z),\qquad\boldsymbol{\sigma}_{g}(z).
a~τg=clip⁡(aτg−𝝁g​(zt)𝝈g​(zt)+ϵa,−1,1),g∈{kin,tac}.\tilde{a}^{g}_{\tau}=\operatorname{clip}\left(\frac{a^{g}_{\tau}-\boldsymbol{\mu}_{g}(z_{t})}{\boldsymbol{\sigma}_{g}(z_{t})+\epsilon_{a}},-1,1\right),\qquad g\in\{\mathrm{kin},\mathrm{tac}\}. (14)
Δ​𝐪^τg=𝐬g​(zt)⋅clip⁡(a~^τg,−1,1),g∈{kin,tac},\Delta\hat{\mathbf{q}}^{g}_{\tau}=\mathbf{s}^{g}(z_{t})\cdot\operatorname{clip}\left(\hat{\tilde{a}}^{g}_{\tau},-1,1\right),\qquad g\in\{\mathrm{kin},\mathrm{tac}\}, (15)
𝐪τ+1=Π𝒬​(𝐪τ+[Δ​𝐪^τkin,Δ​𝐪^τtac]),\mathbf{q}_{\tau+1}=\Pi_{\mathcal{Q}}\left(\mathbf{q}_{\tau}+\left[\Delta\hat{\mathbf{q}}^{\mathrm{kin}}_{\tau},\Delta\hat{\mathbf{q}}^{\mathrm{tac}}_{\tau}\right]\right), (16)

2.3 Chunked behavior-cloning objective

Chunk behavior cloning sums stage-weighted arm and tactile L1 errors over the sampled horizon. A rate penalty regularizes changes in consecutive executed increments. Arm weighting is larger during approach/positioning; tactile weighting rises during cutting. Complete optimizer, model-width and loss-weight tables are not supplied for exact reproduction.

Equations 17–18
ℒBC=1Ht​∑ℓ=0Ht−1[ωkin​(zt)​‖a~^t+ℓkin−a~t+ℓkin‖1+ωtac​(zt)​‖a~^t+ℓtac−a~t+ℓtac‖1].\mathcal{L}_{\mathrm{BC}}=\frac{1}{H_{t}}\sum_{\ell=0}^{H_{t}-1}\left[\omega_{\mathrm{kin}}(z_{t})\left\|\hat{\tilde{a}}^{\mathrm{kin}}_{t+\ell}-\tilde{a}^{\mathrm{kin}}_{t+\ell}\right\|_{1}+\omega_{\mathrm{tac}}(z_{t})\left\|\hat{\tilde{a}}^{\mathrm{tac}}_{t+\ell}-\tilde{a}^{\mathrm{tac}}_{t+\ell}\right\|_{1}\right]. (17)
ℒrate=1Ht−1​∑ℓ=0Ht−2‖[Δ​𝐪^t+ℓ+1kin,Δ​𝐪^t+ℓ+1tac]−[Δ​𝐪^t+ℓkin,Δ​𝐪^t+ℓtac]‖22.\mathcal{L}_{\mathrm{rate}}=\frac{1}{H_{t}-1}\sum_{\ell=0}^{H_{t}-2}\left\|\left[\Delta\hat{\mathbf{q}}^{\mathrm{kin}}_{t+\ell+1},\Delta\hat{\mathbf{q}}^{\mathrm{tac}}_{t+\ell+1}\right]-\left[\Delta\hat{\mathbf{q}}^{\mathrm{kin}}_{t+\ell},\Delta\hat{\mathbf{q}}^{\mathrm{tac}}_{t+\ell}\right]\right\|_{2}^{2}. (18)

2.4 Sim2Real deployment

Stage switching uses knife proximity and tactile onset. Lost contact for a fixed number of steps returns the system to knife positioning, with lower off-threshold than on-threshold to create hysteresis. All commands pass the stated joint/rate/workspace/collision filter. These mechanisms aid execution but are not a whole-task success or physical damage guarantee.

Equations 19
zt={zapp,dgeo​(t)>ϵapp,zknife,dgeo​(t)≤ϵapp​and​p¯t<pon,zcut,p¯t≥pon.z_{t}=\begin{cases}z^{\mathrm{app}},&d_{\mathrm{geo}}(t)>\epsilon_{\mathrm{app}},\\[5.69054pt] z^{\mathrm{knife}},&d_{\mathrm{geo}}(t)\leq\epsilon_{\mathrm{app}}\;\;\mathrm{and}\;\;\bar{p}_{t}<p_{\mathrm{on}},\\[5.69054pt] z^{\mathrm{cut}},&\bar{p}_{t}\geq p_{\mathrm{on}}.\end{cases} (19)
p¯t<poff,poff<pon,\bar{p}_{t}<p_{\mathrm{off}},\qquad p_{\mathrm{off}}<p_{\mathrm{on}},

3 Implementation Details

The implementation distinguishes algorithmic representation from hardware-specific execution. Both platforms use object-centric blade motion and tactile hand regulation, while transforms, IK, joint limits and action scales adapt to the robot.

3.1 Two-Embodiment Setup Verification

The source identifies the main platform as DR3/DR5 arms with LinkerHand L10 hands and an Orbbec 435Le camera. The second uses dual Franka arms and LinkerBot L20 hands with recalibrated vision. Main quantitative trials belong to the first; the second is a qualitative morphology-shift check. No matched second-platform success table or proof of unchanged learned-weight transfer is supplied.

Complete original Table 1: Two-embodiment hardware verification. Embodiment A is the primary DexNinja platform used for the main experiments. Embodiment B replaces the arms and hands while preserving the object-centric real2sim2real pipeline.
Table 1: Two-embodiment hardware verification. Embodiment A is the primary DexNinja platform used for the main experiments. Embodiment B replaces the arms and hands while preserving the object-centric real2sim2real pipeline.
Accessible table and original citation links
Component Embodiment A: primary setup Embodiment B: transfer setup
Robot arms DR3/DR5 bimanual arm configuration Dual Franka arm configuration
Dexterous hands LinkerHand L10 dexterous hands LinkerBot L20 dexterous hands
Tactile sensing Pressure-based tactile sensors on both hands Pressure-based tactile sensors on both hands
Vision Fixed RGB-D camera (Orbbec 435Le) Fixed RGB-D camera with recalibrated extrinsics
Role in evaluation Main quantitative real-world experiments Cross-embodiment qualitative verification
Complete original Figure 1: Two-embodiment setup verification.
(a) Primary DexNinja setup with DR3/DR5 arms and LinkerHand L10 dexterous hands.
(b) Embodiment-shift setup with Franka arms and LinkerBot L20 dexterous hands.
The real2sim2real pipeline is kept unchanged at the algorithmic level; only robot-specific calibration, kinematic models, IK solvers, joint limits, and safety projections are updated.
Figure 1: Two-embodiment setup verification. (a) Primary DexNinja setup with DR3/DR5 arms and LinkerHand L10 dexterous hands. (b) Embodiment-shift setup with Franka arms and LinkerBot L20 dexterous hands. The real2sim2real pipeline is kept unchanged at the algorithmic level; only robot-specific calibration, kinematic models, IK solvers, joint limits, and safety projections are updated.

3.2 Embodiment-Specific Control and Safety Filters

Normalized outputs are scaled and projected for each robot. Knife targets violating collision/limit/singularity checks are discarded or repaired. Filtered tactile control runs slower than arm pose tracking, but numerical rates are not reported. The separation supports modular deployment while leaving a substantial robot-specific execution layer.

Equations 20–21
at=(atkin,attac).a_{t}=\left(a^{\mathrm{kin}}_{t},a^{\mathrm{tac}}_{t}\right).
Δ​𝐪^t(b)=𝐬(b)⊙clip⁡(a^t,−1,1),b∈{A,B}.\Delta\hat{\mathbf{q}}^{(b)}_{t}=\mathbf{s}^{(b)}\odot\operatorname{clip}\left(\hat{a}_{t},-1,1\right),\qquad b\in\{\mathrm{A},\mathrm{B}\}. (20)
𝐪t+1(b)=Π𝒬(b)​(𝐪t(b)+Δ​𝐪^t(b)),\mathbf{q}^{(b)}_{t+1}=\Pi_{\mathcal{Q}^{(b)}}\left(\mathbf{q}^{(b)}_{t}+\Delta\hat{\mathbf{q}}^{(b)}_{t}\right), (21)

4 Knowledge-Guided Data Augmentation

Knowledge-guided curation turns captured references into clean single-object images and validated 3D assets. Algorithm 1 retains detection, description, reinpainting, multi-view judgment and rejection branches. Visual admission is followed by physical initialization and successful-trajectory checks.

Inputs

Inputs can contain clutter or occlusion. A detector predicts category and a summary records shape/surface/color/ripeness descriptors. The structured attributes condition later generation and judge consistency; they are not direct measurements of mass, friction or elastic constants.

Template-constrained description synthesis

A constrained description template orders the extracted attributes, improving controllability across later generators. The original example is “This is a <adj> <object> with <features>”. The source supplies this example and algorithmic roles, rather than full model-specific prompt transcripts.

Equation
p=Describe⁡(I,y,𝐬),p=\mathrm{Describe}(I,y,\mathbf{s}),

Image cleaning via inpainting and transparency

Inpainting aims for one centered object without hands/tools/clutter and with transparency. Attribute preservation is checked later rather than assumed from generation alone. Clean appearance assets still require scale/geometry and physical admission before simulation use.

Equation
I~=Inpaint⁡(I,p,𝐬),\tilde{I}=\mathrm{Inpaint}(I,p,\mathbf{s}),

Automatic quality control (QC) for 2D

A vision-language judge checks category/attribute match, transparency, single-object composition and artifacts. Failed images may be reinpainted with rejection reasons; persistent failures are discarded. This is the implemented QC route, not a certified error-free semantic verifier.

3D asset generation and multi-view validation

An image-to-3D model such as Trellis produces a Gaussian-splatting or mesh asset [88]. Rendered views are judged for identity, geometry and texture stability. Passing assets then face simulator compatibility and initialization checks; successful cutting/IK/collision checks apply at rollout level.

Complete original Figure 2: Generated instance samples from the knowledge-guided augmentation pipeline.
Figure 2: Generated instance samples from the knowledge-guided augmentation pipeline.

Outputs

Each retained item stores its transparent image, 3D representation, category, attribute record and description. The illustrated pool spans the four training food categories. It broadens an anchored empirical neighborhood without establishing a guaranteed continuous latent manifold of executable behaviors.

Equation
{(I~j,Aj,yj,𝐬j,pj)}j=1Nkeep,\{(\tilde{I}_{j},A_{j},y_{j},\mathbf{s}_{j},p_{j})\}_{j=1}^{N_{\text{keep}}},
Complete original Algorithm 1, including Input, Output, nested quality-control branches and return.
Algorithm 1 Knowledge-guided instance generation with 2D/3D quality control
Original selectable algorithm
Input: Raw real-world captures {In}n=1N\{I_{n}\}_{n=1}^{N}
Output: Curated assets 𝒜={(I~,A,y,𝐬,p)}\mathcal{A}=\{(\tilde{I},A,y,\mathbf{s},p)\}
𝒜←∅\mathcal{A}\leftarrow\emptyset
for n=1n=1 to NN do // Process each capture
   y←Detect⁡(In)y\leftarrow\mathrm{Detect}(I_{n})
   𝐬←Summarize⁡(In,y)\mathbf{s}\leftarrow\mathrm{Summarize}(I_{n},y)
   p←Describe⁡(In,y,𝐬)p\leftarrow\mathrm{Describe}(I_{n},y,\mathbf{s})
   I~←Inpaint⁡(In,p,𝐬)\tilde{I}\leftarrow\mathrm{Inpaint}(I_{n},p,\mathbf{s})
   (q2D,r2D)←Judge2D⁡(I~,y,𝐬,p)(q_{\text{2D}},r_{\text{2D}})\leftarrow\mathrm{Judge2D}(\tilde{I},y,\mathbf{s},p)
   if q2D=0q_{\text{2D}}=0 then
      I~←ReInpaint⁡(I~,In,p,r2D)\tilde{I}\leftarrow\mathrm{ReInpaint}(\tilde{I},I_{n},p,r_{\text{2D}})
      (q2D,r2D)←Judge2D⁡(I~,y,𝐬,p)(q_{\text{2D}},r_{\text{2D}})\leftarrow\mathrm{Judge2D}(\tilde{I},y,\mathbf{s},p)
   if q2D=0q_{\text{2D}}=0 then
      continue
   A←LiftTo3D⁡(I~)A\leftarrow\mathrm{LiftTo3D}(\tilde{I}) // AA is 3DGS or mesh
   {I^k}k=1K←RenderMultiView⁡(A)\{\hat{I}_{k}\}_{k=1}^{K}\leftarrow\mathrm{RenderMultiView}(A)
   (q3D,r3D)←JudgeMultiView⁡({I^k},y,𝐬)(q_{\text{3D}},r_{\text{3D}})\leftarrow\mathrm{JudgeMultiView}(\{\hat{I}_{k}\},y,\mathbf{s})
   if q3D=0q_{\text{3D}}=0 then
      continue
   𝒜←𝒜∪{(I~,A,y,𝐬,p)}\mathcal{A}\leftarrow\mathcal{A}\cup\{(\tilde{I},A,y,\mathbf{s},p)\}
return 𝒜\mathcal{A}

5 Additional Experiments

Additional studies compare a reproduced cutting baseline, scale simulated data, ablate policy organization and inspect pressure regulation. Their protocols and denominators differ from the main hundred-trial study; their success rates should remain attached to their own tables.

5.1 Comparison to baseline

Over forty trials, DexNinja scores 23/40 (57.5%) and the RoboNinja reproduction 12/40 (30.0%), a 27.5-point aggregate gain. It improves strawberry/potato/banana but loses on apple, 3/10 versus 7/10. This comparison is a separate ten-per-category experiment, not the main 25-per-category result.

Complete original Table 2: Baseline comparison between RoboNinja and our method on four objects.
We report successes over 10 trials per object; Avg. aggregates over all 40 trials.
Table 2: Baseline comparison between RoboNinja and our method on four objects. We report successes over 10 trials per object; Avg. aggregates over all 40 trials.
Accessible table and original citation links
Method Strawberry Potato Banana Apple Avg.
RoboNinja (baseline) 1/10 2/10 2/10 7/10 12/40
Ours 6/10 5/10 9/10 3/10 23/40

Baseline implementation.

The baseline keeps RoboNinja’s rigidly attached knife assumption, while DexNinja grasps the tool dexterously. Original force-sensor details could not be recovered, so contact feedback is approximated from tactile readings. The authors explicitly qualify it as a protocol-matched reproduction with an approximate sensing interface. It is not a fully exact recreation of the original hardware/controller.

5.2 Simulation scaling and generation cost

With real data fixed, adding 10/50/100/300 simulated episodes gives 36/50/56/63 successes out of 100. This is an observed data-scaling trend, rather than a fitted universal scaling law. Generation costs are substantial: about ten minutes and 20 GB GPU memory per single-cut episode on an H200, before parallelization. The source estimates N·M/6 GPU-hours and roughly 10+10M GB memory for N episodes of M cuts, excluding other pipeline/training costs.

Complete original Table 3: Scaling law of simulation augmentation for real-world cutting. We train with a fixed real dataset and add N N simulated episodes; values are real-world success (25 trials per food), and Avg. aggregates 100 trials.
Table 3: Scaling law of simulation augmentation for real-world cutting. We train with a fixed real dataset and add NN simulated episodes; values are real-world success (25 trials per food), and Avg. aggregates 100 trials.
Accessible table and original citation links
+Sim episodes NN Strawberry Banana Potato Apple Avg.
10 9/25 14/25 8/25 5/25 36/100
50 12/25 18/25 12/25 8/25 50/100
100 14/25 20/25 13/25 9/25 56/100
300 16/25 22/25 15/25 10/25 63/100

5.3 System-level ablations

The forty-trial head/chunk study scores 11/40 with a universal head, 21/40 with separate heads, 13/40 with fixed horizon 64 and 24/40 for the full system. The full design has the best aggregate count, with task-level ties. Small ten-per-category samples and missing training-seed intervals limit fine-grained ranking; the table supports this implementation’s organization rather than all possible head designs.

Complete original Table 4: System-level ablation study on head design and action chunking across objects. Metric: task success rate (successes / 10 trials).
Table 4: System-level ablation study on head design and action chunking across objects. Metric: task success rate (successes / 10 trials).
Accessible table and original citation links
Object Universal Head Separate Heads Fixed Action Chunk Full System
Apple 2/10 4/10 1/10 5/10
Banana 4/10 7/10 6/10 7/10
Strawberry 2/10 5/10 2/10 7/10
Potato 3/10 5/10 4/10 5/10
Average 11/40 21/40 13/40 24/40

5.4 Tactile regulation versus forceful grasping

A guava example contrasts localized contact with broader activation and visible compression at high force. It illustrates the tradeoff between slip and over-compression. It is qualitative evidence of why regulation matters, not a quantified guarantee that the policy always stays inside a measured safe contact envelope.

Complete original Figure 3: Tactile responses under nominal and high-force contact.
(a,b) Thumb and index tactile maps under nominal contact with an undeformed guava.
(c,d) Thumb and index tactile maps under high-force contact with a compressed guava.
The comparison illustrates why tactile feedback is needed to regulate contact within the slip–over-compression regime.
Figure 3: Tactile responses under nominal and high-force contact. (a,b) Thumb and index tactile maps under nominal contact with an undeformed guava. (c,d) Thumb and index tactile maps under high-force contact with a compressed guava. The comparison illustrates why tactile feedback is needed to regulate contact within the slip–over-compression regime.

5.5 Summary of additional findings

The combined evidence supports useful data densification and separate/stage-conditioned control, with unequal gains by category. Additional simulation raises success but remains costly, and no experiment eliminates the grasp, material-model or long-horizon adjustment failures.

6 Qualitative Results

Full qualitative filmstrips retain primary and second-platform executions, food-specific colors, multiple cuts and adjustment failure. Selected successful sequences complement the complete trial tables without changing their denominators.

Complete original Figure 4: Qualitative real-world deployment results with 2 different embodiment settings. Representative executions across food categories.
Background colors denote the object type: orange (banana), pink (strawberry), red (apple), and yellow (potato).
Figure 4: Qualitative real-world deployment results with 2 different embodiment settings. Representative executions across food categories. Background colors denote the object type: orange (banana), pink (strawberry), red (apple), and yellow (potato).
Complete original Figure 5: Qualitative Multi-cuts Experiments results.
Figure 5: Qualitative Multi-cuts Experiments results.

Knife initialization and handover.

DexGraspNet 2.0 proposes knife-handle grasps and pre-grasp configurations before the policy takes over [87]. This standardizes tool starting pose and clearance. It is an initialization prior, rather than evidence that the learned cutting policy autonomously acquires an arbitrary tool from arbitrary initial configurations.

Multi-cut Experiment Results.

Two cuts largely repeat the single-cut interaction. Three cuts require fruit adjustment and realignment, where accumulated pose/contact errors cause additional failures. The full supplementary Figure 5 includes both successful three-cut execution and a failed adjustment, preserved with their labels.

Phase structure in real rollouts.

Selected rollouts show approach, contact initiation and progressive slicing with small pose corrections. The narrative describes how soft banana, delicate strawberry and resistant potato differ. These observations characterize examples; they do not add a controlled per-material mechanism experiment or a calibrated material estimator.

Common qualitative failure modes.

Observed failures include initial misalignment, premature contact, late jamming/friction and unsuccessful multi-cut realignment. Three-cut adjustment is especially sensitive to accumulated disturbance. These examples are consistent with the main failure counts and show the remaining limits of the learned data distribution.

Concept demonstration

Open image ↗