Literature
Display
← All papers

Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

Xin Liu · Yulin Li · Ziming Li · Pengyu Jing · Zhenhao Huang · Bingyang Zhou · Ziqiu Zeng · Siyuan Luo · Chenkun Qi · Fan Shi

Preprint posted · CoRL 2026 (accepted) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

Project website. This edition follows the complete twenty-seven-page v2 of 24 September 2026. All twelve numbered figures, including six native subplot graphics absent as ordinary images in the HTML, ten numbered tables and fifteen displayed equations are retained. The online controller uses RGB; building each garment’s simulator asset requires an offline RGB-D capture.

At a glance

A learned cloth policy proposes a short action sequence, then a physical simulator evaluates nearby alternatives before the robot executes the first refined action. A single RGB image predicts the current garment mesh, resetting simulation at every control cycle. FLASH supplies parallel contact-rich rollouts; prior-guided model predictive path integral control (MPPI) turns their costs into an action update.

In ten-trial towel experiments, full-system success is 9/10 with one arm and 8/10 with two, compared with 3/10 and 1/10 for the base policy. Additional garment, task and disturbance tests support this approach within calibrated, quasi-static tabletop folding. New garments receive separate assets, estimator training and prior policies; these are not zero-shot garment transfers.

1 Introduction

Cloth state is high-dimensional and partly hidden, while contact, friction and folding make long-horizon prediction sensitive to errors. Learned policies can miss unusual states; uncontrolled simulator rollout can drift away from reality. This system joins learned proposals with short physical look-ahead and repeated visual state correction.

The intended contribution is a practical loop built from FLASH, fixed-correspondence mesh estimation and MPPI. It does not learn a new dynamics model. The experiments examine each component and complete folding performance rather than establish a general robustness guarantee.

2 Related Works

Prior cloth manipulation includes reinforcement/imitation learning, semantic keypoints, learned dynamics and feedback-based trajectory adaptation. MeshGraphNets and AdaptiGraph learn dynamics, whereas this work uses an explicit physical simulator at inference. AdaFold and other feedback approaches motivate correcting rather than blindly replaying a trajectory.

State reconstruction relates to TRTM, Cloth-Splatting and UniClothDiff’s generative state estimation. Here the estimator predicts an asset’s indexed mesh vertices directly from RGB. Sampling-based planning avoids differentiating contact dynamics, while the learned prior narrows the search around plausible folds.

Complete original PDF block S2.F1
Figure 1: Overview of the proposed framework. Offline, we use FLASH to generate synthetic data for real-to-sim training. Online, RGB observations initialize physics rollouts that refine the prior policy through MPPI for closed-loop hardware execution.

3 Method

The method combines a reusable physical rollout engine, garment-specific RGB reconstruction and prior-guided sampling control. The preserved equations distinguish visual mesh prediction from the full position-and-velocity state needed by simulation.

3.1 System Overview

At each cycle, an RGB frame predicts vertex positions, the simulator is synchronized, and the base policy generates a nominal action sequence. MPPI perturbs that sequence, simulates the candidates and averages them using cost-dependent weights. Only the first refined action is executed before observing again.

The simulator state contains positions and velocities, but a single frame estimates only positions. Velocities are reset to zero. That is a quasi-static modeling assumption, not a consequence that zero is the only possible physical velocity consistent with one image; motion would require temporal information or another prior.

Equations 1–2
s^t=fθ​(It),\hat{s}_{t}=f_{\theta}(I_{t}), (1)
Ut⋆=MPPI⁡(s^t,U¯t,ℱ,c),U_{t}^{\star}=\mathrm{MPPI}\left(\hat{s}_{t},\bar{U}_{t},\mathcal{F},c\right), (2)

3.2 Simulator Backbone

FLASH represents cloth as a triangular mesh with finite-element stretching, isometric bending and Signorini–Coulomb contact/friction. Its GPU implementation combines local–global iterations with a nonsmooth Newton contact solve, allowing many short rollouts in parallel.

Physical contact and global deformation structure distinguish it from learned forward prediction, while repeated visual resets limit accumulated state drift. Material parameters and grasp/contact approximations still affect candidate ranking; resetting state does not remove these modeling errors.

Equation 3
xt+1=ℱ⁡(xt,at),x_{t+1}=\mathcal{F}(x_{t},a_{t}), (3)

3.3 RGB-native Real-to-Sim Reconstruction

Each garment has a fixed mesh topology and ordered canonical vertices. The network estimates all corresponding 3D positions, including occluded regions, so its output can directly initialize the simulator. This correspondence is learned for that asset rather than recovered for arbitrary unseen garments.

Equation 4
s^t={𝐩^t,i}i=1N,𝐩^t,i∈ℝ3,\hat{s}_{t}=\left\{\hat{\mathbf{p}}_{t,i}\right\}_{i=1}^{N},\qquad\hat{\mathbf{p}}_{t,i}\in\mathbb{R}^{3}, (4)

Simulator asset construction.

One overhead RGB-D capture supplies the rest mesh: SAM 2 segments the garment, depth is back-projected, ball-pivoting forms a surface, and remeshing/smoothing regularizes it while preserving the boundary. Depth is used in this offline construction, although the deployed estimator consumes RGB alone.

Synthetic supervision.

FLASH generates paired rendered RGB images and ground-truth vertex positions from randomized scripted folds and additional gravity-settled states. Camera calibration aligns these samples with the fixed real viewpoint. Simulation supervision avoids manually labeling full real cloth meshes.

Equation 5
𝒟syn={(I(m),{𝐩i⋆(m)}i=1N)}m=1M.\mathcal{D}_{\mathrm{syn}}=\left\{\left(I^{(m)},\left\{\mathbf{p}_{i}^{\star(m)}\right\}_{i=1}^{N}\right)\right\}_{m=1}^{M}. (5)

Network architecture.

A frozen DINOv2 backbone produces visual patch features. A trainable projection maps them to the canonical-token dimension; four self-attention blocks jointly process image tokens and one learned token per mesh vertex. A shared decoder maps each updated canonical token to its 3D position.

Canonical indices preserve vertex correspondence across deformations. The network uses global image context to infer hidden geometry, but occluded positions remain predictions conditioned on training distribution rather than directly observed ground truth.

Equations 6–8
Vt=P⁡(ΦDINO​(It)),Vt∈ℝMv×d,V_{t}=P\!\left(\Phi_{\mathrm{DINO}}(I_{t})\right),\qquad V_{t}\in\mathbb{R}^{M_{v}\times d}, (6)
Zt=SelfAttn([C;Vt])1:N,Zt∈ℝN×d,Z_{t}=\mathrm{SelfAttn}\left([\,C\ ;\,V_{t}\,]\right)_{1:N},\qquad Z_{t}\in\mathbb{R}^{N\times d}, (7)
𝐩^t,i=Dθ(Zt,i),i=1,…,N.\hat{\mathbf{p}}_{t,i}=D_{\theta}(Z_{t,i}),\qquad i=1,\ldots,N. (8)

Training objective.

Training minimizes smooth-L1 error between predicted and simulated vertex positions. The visual backbone stays frozen; the projection, canonical tokens, attention blocks and decoder are optimized. This supervised objective does not explicitly enforce every physical constraint of the reconstructed mesh.

Equation 9
ℒr2s=1N​∑i=1Nℓsmooth​-​L1​(𝐩^i,𝐩i⋆).\mathcal{L}_{\mathrm{r2s}}=\frac{1}{N}\sum_{i=1}^{N}\ell_{\mathrm{smooth}\text{-}L_{1}}\left(\hat{\mathbf{p}}_{i},\mathbf{p}^{\star}_{i}\right). (9)

Robust augmentation.

Color, illumination, blur, grayscale, erasing and grasp-centered occlusion randomization reduce reliance on simulator appearance and visible grippers. Token masking additionally removes overlapping visual patches during some occlusion augmentations. Appendix D tests the combined augmentation package rather than identifying each component’s separate effect.

3.4 Prior-Guided MPPI Refinement

The nominal sequence comes from a garment-specific MLP prior trained by on-policy imitation of a scripted teacher in FLASH. It supplies a plausible local search center; inference-time refinement adjusts its action choices using the reconstructed state.

Prior-guided online refinement.

Independent Gaussian perturbations produce candidate sequences around the prior. Every candidate starts from the same reconstructed positions and zero velocities, rolls out through the simulator and receives a task/constraint cost. Exponentiated negative costs are normalized, and their weights average the candidate actions.

Deployment uses 256 candidates, a five-action horizon, 5 mm noise scale and temperature 0.08. Costs reward reaching/folding and penalize table/workspace violations, action magnitude and fold-height error. These are soft penalties, not certified hard constraints; the weighted action average is a sampled optimization update rather than proof of optimality or safety.

Equations 10–13
Ut(k)=U¯t+ϵ(k),ϵ(k)∼𝒩(0,Σ),k=1,…,K,U_{t}^{(k)}=\bar{U}_{t}+\epsilon^{(k)},\qquad\epsilon^{(k)}\sim\mathcal{N}(0,\Sigma),\qquad k=1,\ldots,K, (10)
τt(k)=ℱ0:H(s^t,Ut(k)),\tau_{t}^{(k)}=\mathcal{F}_{0:H}\left(\hat{s}_{t},U_{t}^{(k)}\right), (11)
Original PDF equation 12: normalized exponential rollout-cost weights
Original selectable MathML
wk=exp(−S(τt(k))/λ)∑j=1Kexp(−S(τt(j))/λ),w_{k}=\frac{\exp(-S(\tau_{t}^{(k)})/\lambda)}{\sum_{j=1}^{K}\exp(-S(\tau_{t}^{(j)})/\lambda)}, (12)
Ut⋆=∑k=1Kwk​Ut(k).U_{t}^{\star}=\sum_{k=1}^{K}w_{k}U_{t}^{(k)}. (13)

Receding-horizon execution.

Only the first action of the refined sequence is sent to the robot. A new RGB observation then overwrites simulated positions and the process repeats. This closes the loop around model errors and disturbances, within the estimator’s and controller’s operating range.

4 Experiments

The evaluation separates simulator performance, synthetic mesh reconstruction, end-to-end towel folding and selected extensions. The hardware comparisons use ten trials per setting and report descriptive success rates without uncertainty intervals or significance tests.

4.1 Simulator Backbone

Baselines.

The same vanilla MPPI towel-folding protocol compares FLASH with Genesis, Newton and Isaac Sim. Genesis develops unstable dragging/folding behavior and fails all tested settings, so the numerical table covers the other three. These are configured implementations with different models, particle counts and solver parameters, not materially matched engines.

Complete original PDF block S4.T1
Table 1: Simulator backbone evaluation under the same cloth manipulation task and MPPI controller.
Original selectable table
Simulator KK Time ↓\downarrow KP err. ↓\downarrow MSE ↓\downarrow SR ↑\uparrow
Step MPPI (cm) (cm2)
Newton 64 41.0 0.85 8.40±8.728.40\pm 8.72 56.8±73.156.8\pm 73.1 40%
128 55.1 1.14 10.28±11.0810.28\pm 11.08 68.9±93.768.9\pm 93.7 40%
256 104.3 2.15 8.85±10.718.85\pm 10.71 58.1±89.658.1\pm 89.6 60%
512 232.8 4.80 13.02±13.0613.02\pm 13.06 92.3±105.392.3\pm 105.3 35%
Isaac Sim 64 11.7 0.26 9.78±10.749.78\pm 10.74 60.5±104.460.5\pm 104.4 55%
128 22.2 0.50 10.88±11.1310.88\pm 11.13 67.4±108.067.4\pm 108.0 45%
256 41.1 0.92 7.09±5.607.09\pm 5.60 32.4±47.632.4\pm 47.6 55%
512 80.7 1.79 5.84±4.735.84\pm 4.73 24.6±32.824.6\pm 32.8 60%
FLASH 64 19.4 0.40 1.61±1.18\mathbf{1.61\pm 1.18} 3.7±4.3\mathbf{3.7\pm 4.3} 100%
128 36.0 0.75 1.54±0.88\mathbf{1.54\pm 0.88} 3.1±1.9\mathbf{3.1\pm 1.9} 100%
256 68.9 1.42 1.95±1.82\mathbf{1.95\pm 1.82} 5.5±11.0\mathbf{5.5\pm 11.0} 95%
512 137.3 2.84 2.08±1.93\mathbf{2.08\pm 1.93} 3.9±3.2\mathbf{3.9\pm 3.2} 95%

Metrics.

Twenty shared initializations per sampling budget vary towel pose. Success requires keypoint error below 5 cm; final mesh, Chamfer and earth mover distances assess fold shape separately. The mesh MSE uses the moved half’s 231 vertices against mirrored target partners, rather than every mesh vertex against a measured real cloth state. Step time and full MPPI time capture different compute costs.

Results.

Across 64/128/256/512 samples, FLASH succeeds in 100/100/95/95% of trials, with keypoint errors of 1.61/1.54/1.95/2.08 cm. Isaac Sim is about 1.7 times faster per step but reaches only 45–60% success; Newton reaches 35–60%. FLASH’s full MPPI time rises from 0.40 to 2.84 s as samples increase.

These in-simulator folding results favor the selected FLASH configuration, not a universal engine ranking or a direct measurement of physical fidelity. Appendix C separately replays real trajectories to assess agreement with observed cloth geometry.

4.2 State Estimation

Baselines.

DPM adapts UniClothDiff’s point-cloud diffusion estimator to the same garment assets and single overhead view. The DeFM baseline replaces RGB features with a frozen depth backbone and the same canonical reconstruction head. The comparison changes representation/backbone as well as modality, so it is not a pure causal RGB-versus-depth test.

Metrics.

Synthetic held-out episodes provide ground-truth vertex MSE, Chamfer distance and earth mover distance, with inference latency measured on the same GPU. These metrics have different units: MSE is in mm² and the two point-set distances are in mm.

Results.

For towels, the RGB estimator achieves 2.33 mm² MSE, 1.94 mm Chamfer distance and 7.4 ms latency, versus DPM’s 7.17 mm², 3.89 mm and 318.5 ms. For shirts, its corresponding values are 24.23 mm², 3.65 mm and 12.4 ms; DeFM has 43.52 mm², 4.05 mm and 18 ms.

These are synthetic-validation results from the authors’ reproduced baselines. Real observations have larger error in Appendix C, and the adapted DPM protocol differs from UniClothDiff’s original multi-view setting.

Complete original PDF block S4.T2
Table 2: State-estimation comparison. Per-vertex MSE, CD, EMD, and single-frame inference latency on the validation split.
Original selectable table
Asset Real-to-Sim MSE (mm2) ↓\downarrow CD (mm) ↓\downarrow EMD (mm) ↓\downarrow Lat. (ms) ↓\downarrow
Towel Ours 2.33±1.21\mathbf{2.33\pm 1.21} 1.94±0.41\mathbf{1.94\pm 0.41} 1.98±0.45\mathbf{1.98\pm 0.45} 7.4±0.6\mathbf{7.4\pm 0.6}
DPM 7.17±2.227.17\pm 2.22 3.89±0.673.89\pm 0.67 4.01±0.724.01\pm 0.72 318.5±3.7318.5\pm 3.7
DeFM 43.36±34.6643.36\pm 34.66 5.51±0.995.51\pm 0.99 6.53±1.526.53\pm 1.52 12.6±0.512.6\pm 0.5
Long-sleeve Shirt Ours 24.23±11.49\mathbf{24.23\pm 11.49} 3.65±0.78\mathbf{3.65\pm 0.78} 4.67±1.24\mathbf{4.67\pm 1.24} 12.4±0.4\mathbf{12.4\pm 0.4}
DPM 45.74±14.1945.74\pm 14.19 5.93±0.275.93\pm 0.27 8.21±0.658.21\pm 0.65 512.9±6.0512.9\pm 6.0
DeFM 43.52±22.0243.52\pm 22.02 4.05±0.384.05\pm 0.38 5.93±0.915.93\pm 0.91 18.0±0.618.0\pm 0.6
Complete original PDF block S4.F2
Figure 2: Real-world real-to-sim reconstruction. Panel (a) towel and (b) long-sleeve top, each comparing our RGB-native estimation with DPM at different folding stages. From top to bottom, the real observation, predicted vertices overlaid on the image, and the reconstructed cloth.

4.3 Real-World Evaluation

Tasks.

Single- and dual-arm AIRBOT Play systems fold a square towel from randomized initial states using a fixed overhead camera. Each of ten trials includes reaching, grasping and folding. Success requires all necessary grasps and all target corner pairs within 2 cm; failed grasps are not removed or rerun.

Baselines.

Component substitutions replace the RGB estimator with DPM or FLASH with Isaac Sim while retaining the surrounding system. A base-policy condition omits online refinement. The tests show contributions under the selected setup, not that each design is universally necessary.

Results.

Single-arm successes are 9/10 for the full method, 6/10 with DPM, 5/10 with Isaac Sim and 3/10 for the prior alone. Dual-arm results are 8/10, 3/10, 4/10 and 1/10. Full-system grasping succeeds in all single-arm trials and nine dual-arm trials; folding can still fail after a successful grasp.

Towel-cycle compute is about 1.075 s: segmentation 95.8 ms, reconstruction 8.5 ms, state transfer 0.5 ms and MPPI 970 ms. Actuation adds time. The loop therefore supports slow manipulation rather than high-frequency control; MPPI alone takes 2.55 s for the larger shorts mesh.

Complete original PDF block S4.T3
Table 3: Real-world pipeline-variant evaluation. Each row replaces one component of the full pipeline (first row).
Original selectable table
Task Real-to-Sim Backend Controller Real SR (↑\uparrow)
Single-arm diagonal Ours (RGB) FLASH MPPI 9/10
DPM FLASH MPPI 6/10
Ours (RGB) Isaac Sim MPPI 5/10
Ours (RGB) FLASH Base-policy 3/10
Dual-arm symmetric Ours (RGB) FLASH MPPI 8/10
DPM FLASH MPPI 3/10
Ours (RGB) Isaac Sim MPPI 4/10
Ours (RGB) FLASH Base-policy 1/10

4.4 Generalization and Closed-Loop Correction

Tasks.

Extensions fold shirts and shorts, reverse the towel diagonal and recover from a human push during folding. New garments require their own mesh, synthetic data, estimator, material tuning and prior policy. The shared element is the simulator/control framework. Reversing the towel diagonal instead changes goal keypoints without retraining the towel prior.

Shirts fold sleeves inward and then the torso; shorts fold legs toward the waistband and then compact the garment. Trials randomize initial translations by up to 5 cm and yaw by 0.5 rad. Human perturbation amplitude is not quantified by this hardware experiment.

Complete original PDF block S4.T4
Table 4: Generalization and disturbance-recovery evaluation.
Original selectable table
Task Real SR
Long-sleeve 8/10
Shorts 7/10
Towel, reverse diag. 8/10
Towel, mid-fold disturb. 8/10

Results.

Ten trials per condition yield 8/10 shirt folds, 7/10 shorts folds, 8/10 reversed-diagonal folds and 8/10 disturbance recoveries. Successful examples show adaptation beyond the original towel trajectory, but the small samples and per-garment preparation limit claims of generalization.

Complete original PDF block S4.F3
Figure 3: Representative rollouts for the generalization and disturbance-recovery settings. Time progresses from left to right. The four rows visualize long-sleeve shirt folding, shorts folding, recovery from mid-fold human disturbance, and reverse-diagonal towel folding.

5 Conclusion

Repeated RGB reconstruction and physical look-ahead improve the studied cloth-folding policies, with gains from both the simulator and estimator in component comparisons. The practical result is a calibrated simulator-in-the-loop method for selected garments and slow tasks, with explicit preparation and compute costs.

6 Limitations and Future Work

Limits include per-garment training and parameter tuning, a fixed overhead viewpoint, imperfect simulator fidelity and substantial rollout cost. Tested garments are woven textiles of comparable thickness; thick or qualitatively different materials and fast motion are not established capabilities. Broader visual reconstruction and faster rollout engines are future directions.

Acknowledgments

The acknowledgments identify supporting institutions and funding; the technical and empirical claims are those of the paper.

Appendix

A Implementation Details

Appendix A specifies the asset pipeline, network, randomization, controller costs and modeling assumptions needed to reproduce the loop.

A.1 RGB-native Real-to-Sim Reconstruction

A 392×392 image yields 784 frozen DINOv2-S/14 patch features of dimension 384, projected to 256. The towel, shorts and shirt meshes have 464, 1,156 and 1,379 vertices. Four pre-normalized attention blocks use eight heads and dropout 0.1; a shared layer-normalization/two-hidden-layer GELU decoder outputs coordinates.

Canonical tokens start from deterministic Gaussian Fourier features of rest coordinates, with frequency scale 4. The Fourier matrix is fixed while tokens themselves are learnable. Synthetic examples include randomized scripted folds and gravity-settled meshes. Train/validation splitting is 90/10 by episode to avoid nearby frames leaking across the split.

Training uses AdamW, learning rate 3×10⁻⁴, weight decay 10⁻⁴, gradient clipping at 1, batch size 32 and fifty epochs; cosine decay follows three warmup epochs, with EMA 0.999. Grasp occlusion occurs with probability 0.7; token masking probability 0.8 is conditional on that augmentation, not an independent 80% mask of all inputs. Table A1 preserves the complete appearance/occlusion settings.

Preparing a new asset requires roughly one hour of synthetic generation and twelve hours of estimator training on an RTX 4090, in addition to capture/meshing. Thus asset-specific synthetic supervision remains a significant adaptation cost.

Equations 14–15
ci(0)=[sin⁡(2​π​𝐱irest​B);cos⁡(2​π​𝐱irest​B)]∈ℝd,B∈ℝ3×d/2,Bj​k∼𝒩⁡(0,σB2),c_{i}^{(0)}=\big[\,\sin(2\pi\,\mathbf{x}_{i}^{\mathrm{rest}}B);\,\cos(2\pi\,\mathbf{x}_{i}^{\mathrm{rest}}B)\,\big]\in\mathbb{R}^{d},\qquad B\in\mathbb{R}^{3\times d/2},\ B_{jk}\sim\mathcal{N}(0,\sigma_{B}^{2}), (14)
𝐩^i=Wout​GELU​(W2​GELU​(W1​LN​(Zi))),\hat{\mathbf{p}}_{i}=W_{\mathrm{out}}\,\mathrm{GELU}(W_{2}\,\mathrm{GELU}(W_{1}\,\mathrm{LN}(Z_{i}))), (15)
Complete original PDF block S1.T1
Table A.1: Training-time augmentation for the RGB module. Every randomized operation applied during training, with its application probability and parameter range. Color jitter and Gaussian blur run on every frame with parameters sampled from the listed ranges. The grasp-point token mask is propagated only when the grasp-point pixel occlusion is applied, so its probability is conditional on that event.
Original selectable table
Operation Level Prob. Range / strength
Color jitter Image 1.01.0 brightness, contrast, saturation ±0.4\pm 0.4, hue ±0.1\pm 0.1
Random grayscale Image 0.20.2 convert RGB to single channel
Gaussian blur Image 1.01.0 kernel 55, σ∈[0.1,2.0]\sigma\in[0.1,2.0]
Directional lighting Image 0.60.6 gradient depth α∈[0.15,0.40]\alpha\in[0.15,0.40]
Random erasing Image 0.30.3 erased area fraction ∈[0.02,0.15]\in[0.02,0.15]
Grasp-point occlusion Image 0.70.7 half-side ∈[25,55]\in[25,55] px, aspect ∈[0.7,1.5]\in[0.7,1.5], corner jitter ±12\pm 12 px, center jitter ±10\pm 10 px, fill brightness ∈[0,0.05]\in[0,0.05]
Grasp-point token mask Token 0.80.8 DINOv2 patches with >30%>30\% area overlap flagged invalid for fusion

A.2 Prior-Guided MPPI Control

The prior is a 512/256/128 MLP taking sparse mesh features and predicting end-effector actions. MPPI rolls it forward to obtain the nominal horizon before adding noise. Table A2 retains all phase-dependent cost expressions and weights.

Reaching uses 3D pick-point distance; folding uses in-plane corner-to-goal distance, with matching terminal costs. Squared hinge penalties discourage table penetration and workspace violations, and fold-height error is squared. The term described as action smoothness penalizes action magnitude, not differences between consecutive actions. Table/workspace margins are 5 mm and 2 cm.

Complete original PDF block S1.T2
Table A.2: MPPI stage cost components. 𝐱eex​y\mathbf{x}_{\mathrm{ee}}^{xy} denotes the in-plane end-effector position. 𝒲safeinner\mathcal{W}_{\mathrm{safe}}^{\mathrm{inner}} denotes the interior of the safe workspace after shrinking each axis by a soft margin δbnd\delta_{\mathrm{bnd}}. zfold∗z_{\mathrm{fold}}^{*} is the nominal lift height during folding.
Original selectable table
Term Expression Weight Activation
cdistc_{\mathrm{dist}} (reach) ‖𝐱ee−𝐩pick‖2\|\mathbf{x}_{\mathrm{ee}}-\mathbf{p}_{\mathrm{pick}}\|_{2} 1.01.0 reach phase
cdistc_{\mathrm{dist}} (fold) ‖𝐩graspx​y−𝐩targetx​y‖2\|\mathbf{p}_{\mathrm{grasp}}^{xy}-\mathbf{p}_{\mathrm{target}}^{xy}\|_{2} 1.01.0 fold phase
ctablec_{\mathrm{table}} max⁡(0,ztable+δtbl−zee)2\max\!\left(0,\,z_{\mathrm{table}}+\delta_{\mathrm{tbl}}-z_{\mathrm{ee}}\right)^{2} 50.050.0 all phases
climitc_{\mathrm{limit}} axis-wise squared hinge on 𝒲safeinner\mathcal{W}_{\mathrm{safe}}^{\mathrm{inner}} 20.020.0 all phases
cvelc_{\mathrm{vel}} ‖𝐚tc‖22\|\mathbf{a}^{c}_{t}\|_{2}^{2} 10.010.0 all phases
cheightc_{\mathrm{height}} (zee−zfold∗)2(z_{\mathrm{ee}}-z_{\mathrm{fold}}^{*})^{2} 2.02.0 fold phase

A.3 Modeling Assumptions

The gripper is a rigid box that pins selected garment vertices after grasping; finger slip, partial grasp and regrasp dynamics are not modeled. Garment mass, bending and friction are tuned individually. Position resets correct accumulated state drift but cannot make these assumptions exact.

Zeroing velocity restricts the intended regime to quasi-static actions. A proposed RGB-D frontend for broader materials is a future extension, not evidence that the current RGB estimator or tested parameters already handle them.

B Experiment Details

Appendix B fixes the simulator, estimator and robot evaluation protocols and retains the full visual folding sequences.

B.1 Simulator Backbone Comparison

The simulator benchmark uses a 30×30 cm, 464-vertex towel, vanilla MPPI with horizon six, 40 ms action steps and four 10 ms physics steps per action. End-effector increments are clamped to 20 mm. Twenty shared seeds randomize initial translation by up to 8 cm and yaw by 30°. After keypoint error falls below 5 cm, another 150 settling steps precede final shape scoring.

All engines use friction one, but their other parameters differ substantially: Genesis uses PBD and more internal particles; Newton uses one VBD iteration; Isaac uses sixteen XPBD iterations; FLASH uses FEM with eight local–global and ten constraint iterations. Different stiffness, mass/density and solver choices prevent interpreting Table 1 as a matched-material physics experiment. Figure B1 retains all four distribution plots.

Complete original PDF block S2.T1
Table B.1: Simulator backbone configurations under the shared cloth asset, MPPI controller, action clamp (2020 mm/step), and cost. Per-solver iteration counts use different update rules and are not directly comparable as a compute proxy.
Original selectable table
Genesis Newton Isaac Sim FLASH
Dynamic solver PBD VBD XPBD FEM
Solver iters 44 11 1616 88
Time step Δ​t\Delta t 1010 ms 1010 ms 1010 ms 1010 ms
Friction μ\mu 1.01.0 1.01.0 1.01.0 1.01.0
Complete original PDF block S2.F1.sf1
(a) K=64K=64
Complete original PDF block S2.F1.sf2
(b) K=128K=128
Complete original PDF block S2.F1.sf3
(c) K=256K=256
Complete original PDF block S2.F1.sf4
(d) K=512K=512
Figure B.1: Settled-state fold-quality distributions across simulators at four parallel environment counts. Each panel shows the 20-seed distribution of per-vertex MSE (cm2, log left axis), CD (cm), and EMD (cm) for Newton, Isaac Sim, and FLASH under identical seeded initial cloth poses.

B.2 State Estimation

The estimator test shares garment topology, simulated fold distributions and episode-based held-out data. Table B2 specifies the architectures, inputs and training settings. Figure B2 shows synthetic RGB/depth examples of towel and shirt folds and rest states.

DPM consumes 10,000 points from the single overhead depth cloud. DeFM uses a frozen depth encoder with the same canonical head and supervision as the RGB system. Matching these downstream components narrows the comparison while leaving differences in pretraining and representations.

Complete original PDF block S2.F2
Figure B.2: Representative synthetic cloth deformation samples from the state-estimation dataset, covering the towel and the long-sleeve top across the fold and rest-state subsets. Top, RGB views. Bottom, depth maps.
Complete original PDF block S2.T2
Table B.2: Architecture and training configuration of the three state-estimation methods.
Original selectable table
Ours (RGB) DeFM (depth) UniClothDiff DPM (point cloud)
Input RGB 3922392^{2} depth 3922392^{2} 10,00010{,}000 points
Encoder DINOv2 ViT-S/14, frozen DeFM ViT-S/14, frozen KNN-grouper ++ patch MLP
Output head NN canon. tokens, 44L self-attn (d=256d{=}256, h=8h{=}8), MLP same as Ours DDPM denoiser, template-cond.
Loss smooth ℓ1\ell_{1} smooth ℓ1\ell_{1} DDPM noise pred.
Optimizer (AdamW) lr 3​e−43\mathrm{e}{-4}, wd 10−410^{-4} lr 3​e−43\mathrm{e}{-4}, wd 10−410^{-4} lr 10−310^{-3}, wd 10−210^{-2}
Batch size 3232 3232 44 per GPU
Schedule 5050 ep., cosine, 33-ep. warmup, EMA 0.9990.999 same as Ours 900900K steps, const., 500500-step warmup
Gradient clip 1.01.0 1.01.0 1.01.0

B.3 Real-World Evaluation

The hardware uses one or two six-degree-of-freedom arms with binary parallel grippers and synchronized end-effector position increments. The ZED 2i supplies 1280×720 overhead images; the RGB method uses the left color view, while depth supports its baseline and separate evaluation.

The complete trial includes all phases, with the same end-to-end success rule and ten initializations per condition. Runtime measurements report perception, state transfer and rollout separately. Success after grasping remains stricter than successful grasp acquisition alone.

Visualizations.

Figures B3 and B4 preserve full single- and dual-arm towel sequences. They illustrate the observed reach–grasp–fold process; quantitative rates come from the full trial sets rather than these selected examples.

Complete original PDF block S2.F3
Figure B.3: Full-rollout visualization of successful single-arm diagonal folding trials. Time progresses from left to right.
Complete original PDF block S2.F4
Figure B.4: Full-rollout visualization of successful dual-arm symmetric folding trials. Time progresses from left to right.

B.4 Generalization and Closed-Loop Correction

Figures B5–B8 retain shirt folding, shorts folding, reversed-diagonal towel folding and recovery after a mid-fold disturbance. Garment-specific estimator/prior training and tuned physical parameters accompany shirt/shorts transfer. Reversed-diagonal towel folding reuses the learned prior with altered goals.

The disturbance example shows successive visual resets recovering useful actions. Its qualitative human push should not be assigned the precisely controlled 6/9/12 cm amplitudes used in the separate simulation analysis.

Complete original PDF block S2.F5
Figure B.5: Full rollout visualization for long-sleeve shirt folding.
Complete original PDF block S2.F6
Figure B.6: Full rollout visualization for shorts folding.
Complete original PDF block S2.F7
Figure B.7: Full rollout visualization for reverse-diagonal towel folding.
Complete original PDF block S2.F8
Figure B.8: Full rollout visualization for mid-fold disturbance recovery.

C Additional Experimental Analyses

Complete original PDF block S3.F1.sf1
(a) Open-loop simulator replay vs. real cloth
Complete original PDF block S3.F1.sf2
(b) Closed-loop keypoint error
Figure C.1: Simulator fidelity and closed-loop correction. (a) One-sided Chamfer distance between the FLASH mesh and the real cloth when real single-arm end-effector trajectories are replayed open-loop, one line per trajectory. (b) Per-step keypoint error of the closed-loop planner in simulation, one line per trial over 2020 seeds.

C.1 Simulator Fidelity Against Real Cloth Trajectories

Five recorded real trajectories contain 626 frames. The simulator receives the initial mesh once, then replays measured robot motion without visual resets. A one-sided nearest-neighbor Chamfer distance from simulated vertices to recorded depth points has median 13.6 mm, typically 7–9 mm early and larger during dragging.

Flat-cloth registration has a 5.2 mm measurement floor combining camera, segmentation and mesh alignment. This cannot simply be subtracted to isolate physical-model error. The one-sided metric also does not establish material-point correspondence or complete agreement on hidden surfaces. Figure C1 retains both replay and disturbance plots.

C.2 State-Estimation Error on Real Observations

On the same five real trajectories, RGB mesh estimates have 10.2±1.4 mm mean per-trajectory distance to the observed depth cloud. Depth measures error here rather than serving as estimator input. The value exceeds synthetic validation errors and is not directly interchangeable with the simulator replay’s median 13.6 mm: aggregation and evaluated states differ.

C.3 Closed-Loop Error Trajectories and Disturbance Magnitude

Twenty simulated folding seeds yield eighteen successes below the 2 cm criterion, ending at 0.8±0.1 cm after 22–31 steps. The initial 42.2 cm is the approximately full towel diagonal in the task’s keypoint error, not a 42.2 cm reconstruction error.

Controlled 6/9/12 cm pushes eight steps after motion begins require 4±0, 4.1±0.7 and 6±1 steps respectively to return below the pre-disturbance error, over twenty seeds each. This recovery measure is not time to finish folding and does not establish a formal disturbance bound.

Complete original PDF block S3.T1
Table C.1: Mid-fold disturbance sweep in simulation. A push of the stated magnitude is applied eight control steps into the fold, 2020 seeds per magnitude.
Original selectable table
Push (cm) Seeds Recovered within (steps)
66 2020 4.0±0.04.0\pm 0.0
99 2020 4.1±0.74.1\pm 0.7
1212 2020 6.0±1.06.0\pm 1.0

C.4 Reproduction of the DPM Baseline

The DPM reproduction replaces the original multi-view cloud with the available overhead cloud, retrains on the study’s assets/topologies and uses 900,000 steps with batch size four per GPU. AdamW starts at 10⁻³ with weight decay 0.01, 500 warmup steps and clipping at one.

Fifty DDPM denoising steps balance runtime and quality. Raising this to one thousand changes metrics by less than 5% while increasing computation about twentyfold. These settings define the reproduced baseline; its results should not be treated as the original paper’s published benchmark numbers.

D Real-to-Sim Augmentation Ablation

Removing augmentation slightly improves some synthetic metrics but reduces hardware success from 9/10 to 6/10 for single-arm folding and from 8/10 to 5/10 for dual-arm folding. For example, single-arm synthetic MSE improves from 2.33 to 1.85 mm² without augmentation despite worse real outcomes.

The result is consistent with appearance randomization reducing the visual sim-to-real gap, while synthetic error alone is an incomplete deployment predictor. Ten trials per setting and a combined augmentation intervention limit statistical and component-specific conclusions.

Complete original PDF block S4.T1a
Table D.1: Randomization ablation. Validation reconstruction metrics and real-world success rate over 1010 trials.
Original selectable table
Task R2S Aug MSE (mm2) CD (mm) EMD (mm) Real SR (↑\uparrow)
Single-arm diagonal on (Ours) 2.33±\pm1.21 1.94±\pm0.41 1.98±\pm0.45 9/10
off 1.85±\pm1.93 1.72±\pm0.47 1.74±\pm0.51 6/10
Dual-arm symmetric on (Ours) 3.59±\pm2.08 2.42±\pm0.57 2.51±\pm0.68 8/10
off 3.64±\pm2.05 2.37±\pm0.58 2.42±\pm0.60 5/10
Concept demonstration

Open image ↗