Literature
Display
← All papers

Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups

Nicholas Pfaff · Evelyn Fu · Jeremy Binagia · Phillip Isola · Russ Tedrake

· IROS 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

Original research figure 1
Fig. 1: An overview of our system. Objects are placed in the first bin, where the robot picks them up and reconstructs their geometries by moving them in front of a static camera while re-grasping to reduce occlusions (Section III-B & III-C). Next, the robot identifies the object’s physical parameters by following a trajectory designed to be informative for the inertial parameters (Section III-D). Finally, it places the object into the second bin and repeats the process with the next object. The extracted geometric and physical parameters are combined to generate a complete, simulatable object description.

At a glance

Scalable Real2Sim turns a robot's interaction with a rigid object into a simulation asset: a textured visual mesh, convex collision geometry, and estimated mass, center of mass, and rotational inertia. The robot scans objects by rotating and re-grasping them, then follows an informative motion while recording joint torques.

The reconstruction recipe separates background pixels, which should become transparent, from gripper occlusions, which should provide no training signal. The authors release an initial dataset of 20 assets and their observations. Geometry and mass estimates are promising, but rotational inertia can be inaccurate and simulated dynamics are assessed qualitatively. Contact-parameter estimation and grasp slippage remain outside the method.

Project page, code, data, and interactive examples.

I Introduction

Simulation-based learning usually depends on manually prepared geometry and tuned physical parameters. This pipeline automates both for objects passing through an existing pick-and-place setup, using a scanning camera and the robot's joint-torque sensors rather than an additional wrist force-torque sensor. It combines object-centric reconstruction, collision-mesh generation, and physically constrained inertial identification. [3] [4] [5] [6]

The demonstrated workflow deliberately scans and excites each object before depositing it in a second bin. Generating assets passively during ordinary warehouse work is a future possibility, not an evaluated capability. The released 20-object dataset supports component research without requiring a robot.

II Related Work

Prior Real2Sim systems reconstruct geometry or estimate physical parameters, often with manual handling, special hardware, or incomplete asset descriptions. Generative approaches offer scale but may invent shapes and physical properties rather than match a particular real object. Accurate parameters can matter for nonprehensile manipulation. [9] [12]

This work combines established reconstruction and system-identification tools. Its additions adapt photometric reconstruction to moving, occluded objects and provide robot-independent implementations of physical consistency constraints and informative trajectory design. Joint-torque sensing avoids a wrist sensor but measures payload forces indirectly. [17] [18] [19]

III Autonomous Asset Generation Via Robot Interaction

III-A Problem Statement

For each rigid object, the output comprises a textured visual mesh, a union of convex collision meshes, and inertial parameters. The robot picks from one bin, collects scanning views, constructs the geometries, estimates physical properties, and places the object in the other bin. The process repeats without manual intervention during the demonstrated collection run.

III-B Geometric Reconstruction for Visual Geometry

Multi-view observations are collected first; an implicit reconstruction method then produces the textured triangle mesh.

III-B 1 Data Collection and Processing

The robot presents the object along two approximately perpendicular rotation axes, re-grasping to reveal surfaces hidden by the initial grasp. Candidate grasp pairs balance antipodal quality, spatial separation, and reduced mutual occlusion. [24]

Frame selection combines regular subsampling with greedy selection of visually diverse DINO features. SAM2 tracks separate object and gripper masks through the video. Multiple views expose surfaces that a fixed grasp would leave hidden. [25] [26]

Original research figure 2

Fig. 2: Our object scanning method. We use re-grasps to display the object along two perpendicular axes, providing the camera with a complete view of the object.

III-B 2 Geometric Reconstruction

Moving-object observations are expressed in the object's coordinate frame, so a fixed physical camera becomes a virtual moving camera around a static object. Object tracking supplies those camera poses. [27]

Merely excluding the background leaves empty space unconstrained and permits density bleeding. Alpha-transparent training replaces background pixels with iteration-dependent random colors and blends the same colors behind the rendered object. The model can fit those unpredictable colors by becoming transparent outside the object. [14]

Gripper-covered pixels are different: the object may exist behind them. Their loss is ignored rather than driven toward transparency, and gripper masks take precedence over object masks. This distinction prevents holes in occluded surfaces. Optional depth supervision helps photometrically ambiguous, featureless objects. The recipe applies to methods with photometric losses, including the three reconstruction approaches evaluated here.

Figure 3: the full reconstruction comparison, including panels (a) through (i).
Fig. 3: Our object-centric visual reconstruction recipe. From the collected RGB images (a), we obtain the object masks (b) and gripper masks (c). Using only the object masks to ignore background pixels during training (d) results in density bleeding into unoccupied regions (g). Applying alpha-transparent training (e) mitigates density bleeding but incorrectly drives occluded object regions toward transparency (h). Ignoring pixels inside of the gripper mask during training, along with employing alpha transparent training (f), successfully reconstructs an unoccluded object view with no density bleeding (i).

III-C Collision Geometries

Approximate convex decomposition turns the visual mesh into simulator-friendly collision components. [28] [29] The approximation is not universally reliable: gaps can distort hydroelastic pressure fields, while overlapping components can create internal contacts in point-contact models. Primitive shapes offer another speed/accuracy tradeoff. The appropriate representation depends on the simulator and the object; a general solution remains open.

III-D Physical Parameter Identification

Identify the robot once without the payload, identify the grasped composite system, then subtract the robot contribution to recover the object parameters.

III-D 1 Parameters to Identify

The object parameters are mass, center of mass, and a symmetric rotational-inertia tensor. The first moment is mass times center of mass; the pseudo-inertia matrix combines this first moment, mass, and the second moment derived from rotational inertia. Positive definiteness enforces physical consistency. [18]

Robot identification additionally estimates every link's inertia, joint viscous and Coulomb friction, and reflected rotor inertia. Object contact parameters, such as contact friction, are not estimated and remain future work.

Equation 1
𝑱B:=[𝚺B𝐡B𝐡BTmB]≻0,\boldsymbol{J}_{B}:=\begin{bmatrix}{\bf\Sigma}_{B}&{\bf h}_{B}\\ {\bf h}_{B}^{T}&m_{B}\end{bmatrix}\succ 0, (1)

III-D 2 Robot Identification

The manipulator equations are affine in the lumped inertial, friction, and rotor-inertia parameters. Measured joint states and torques therefore form an overdetermined regression. Imposing physical consistency and nonnegative friction/rotor inertia makes identification a convex semidefinite program. The implementation removes unidentifiable directions and adds established regularization. [18] [19] [31]

Convexity concerns this parameter-fitting problem; it does not make the trajectory-design problem below convex or remove sensing and modeling errors.

Equations 2–4
𝐌⁡(𝐪)​𝐪¨+𝐂⁡(𝐪,𝐪˙)​𝐪˙−𝝉𝒈​(𝐪)+𝝉f​(𝐪˙)+𝝉r​(𝐪¨)=𝝉,\mathbf{M}(\mathbf{q})\mathbf{\ddot{q}}+\mathbf{C}(\mathbf{q},\mathbf{\dot{q}})\mathbf{\dot{q}}-\boldsymbol{\tau_{g}}(\mathbf{q})+\boldsymbol{\tau}_{f}(\mathbf{\dot{q}})+\boldsymbol{\tau}_{r}(\mathbf{\ddot{q}})=\boldsymbol{\tau}, (2)
𝐓=𝐖​𝜶+𝐰0.\boldsymbol{\mathrm{T}}=\mathbf{W}\boldsymbol{\alpha}+\mathbf{w}_{0}. (3)
min𝜶⁡‖𝐖​𝜶+𝐰0−𝐓‖22s.t.𝐉n≻0,𝝁v,𝝁c,𝑰r≥𝟎,\min_{\boldsymbol{\alpha}}||\mathbf{W}\boldsymbol{\alpha}+\mathbf{w}_{0}-\boldsymbol{\mathrm{T}}||_{2}^{2}\quad s.t.\quad{\bf J}_{n}\succ 0,\boldsymbol{\mu}_{v},\boldsymbol{\mu}_{c},\boldsymbol{I}_{r}\geq\mathbf{0}, (4)

III-D 3 Optimal Excitation Trajectory Design

Identification needs motion that excites the unknown parameters. The trajectory objective uses ordinary least-squares covariance as a proxy, balancing the regressor's condition number with E-optimality to reduce worst-case uncertainty. A finite Fourier series parameterizes joint motion, constrained by collision avoidance, joint limits, and zero endpoint velocities and accelerations. [21] [22] [23]

A custom augmented Lagrangian solver uses black-box optimization for its subproblems. The authors report better solutions at higher runtime than their gradient-based alternatives; this is acceptable for an offline trajectory designed once per environment. Drake's standard robot descriptions and constraint interface support reuse across robots. This is a practical optimization approach, not a guarantee of globally optimal excitation. [20] [32]

Equation 5
min𝐪⁡(t),t∈[t0,tf]\displaystyle\hskip 12.50002pt\min_{\mathclap{\mathbf{q}(t),t\in[t_{0},t_{f}]}} fc​(𝝀)+γ​fe​(𝝀),\displaystyle f_{c}(\boldsymbol{\lambda})+\gamma f_{e}(\boldsymbol{\lambda}), (5)
s.t.\displaystyle\text{s.t.} collision avoidance,\displaystyle\text{collision avoidance},
joint position/ velocity/ acceleration limits,\displaystyle\text{joint position/ velocity/ acceleration limits},
zero start and end velocities/ accelerations.\displaystyle\text{zero start and end velocities/ accelerations}.

III-D 4 Object Identification

Mass, first moment, and rotational inertia add linearly for composite bodies, so subtracting the unloaded final-link parameters recovers the payload estimate. Estimation error can nevertheless make the difference physically impossible. A second pseudo-inertia constraint is therefore imposed directly on the difference.

The same excitation trajectory is used for robot and object identification to mitigate systematic subtraction errors; a final-link-specific trajectory performed less well. The grasp is assumed rigid. Slippage is not modeled.

Equation 6
m12=m1+m2,𝒉12=𝒉1+𝒉2,𝑰12=𝑰1+𝑰2.m_{12}=m_{1}+m_{2},\quad\boldsymbol{h}_{12}=\boldsymbol{h}_{1}+\boldsymbol{h}_{2},\quad\boldsymbol{I}_{12}=\boldsymbol{I}_{1}+\boldsymbol{I}_{2}. (6)

IV Implementation Details

The experimental setup uses a Kuka iiwa arm, a parallel-jaw gripper with compliant fingers, and several RGBD cameras for workspace observation, bin picking, and scanning. Only one fixed camera supplies the object-scanning views. The claim of camera-based scanning does not imply that the complete demonstrated cell uses only one camera. The released code implements the pipeline.

Original research figure 4

Fig. 4: Our pick-and-place setup. It features a Kuka LBR iiwa 7 arm with a Schunk WSG-50 gripper and Toyota Research Institute Finray fingers. Workspace observations rely on three static RealSense D415 cameras (orange circles), while bin picking uses a RealSense D435 (green circle), and object scanning is performed with another D415 (red circle). All cameras capture 640×480 resolution RGBD images. Objects are picked from the right bin and placed into the left bin. A platform is used in the scanning workspace to enhance the iiwa’s kinematic range during re-grasping.

IV- 1 Grasp and Motion Planning

Antipodal grasp detection selects grasps, and a configuration-space planning method generates collision-aware motions. [24] [33]

IV- 2 Object Tracking

BundleSDF tracks the object in the selected images using object masks, supplying the relative camera poses for reconstruction. Its own mesh is available as a byproduct and serves as one reconstruction option. [27]

IV- 3 Geometric Reconstruction

The object-centric training recipe is applied to Nerfacto, Gaussian Frosting, and Neuralangelo. [34] [35] [13]

IV- 4 Physical Parameter Identification

Robot identification averages ten repeated trajectories; each object uses one trajectory. Velocities and accelerations come from differentiating position measurements, and low-pass filtering is tuned to the identification objective. Robot calibration is repeated at several gripper openings, with the closest opening used for each object to reduce gripper-configuration error.

ICP aligns the reconstructed mesh with the point cloud and known grasp position, expressing the identified inertia in the object's frame. [36]

V Results

V-A Geometric Reconstruction

Chamfer distances to YCB scanner models are 0.93 mm for the mustard bottle, 1.68 mm for the potted-meat can, 5.58 mm for the bleach cleanser, and 0.80 mm for the gelatin box. These are four selected-object comparisons, not a dataset-wide accuracy estimate. Product changes since the original scans can also affect the distances. [39]

The examples show surfaces recovered through re-grasping. Figure 5's scene is manually arranged from individually reconstructed assets, rather than autonomously reconstructed as a whole. BundleSDF can produce poor mesh topology that a strong renderer partly hides; Neuralangelo produces cleaner geometry and more consistent rendering in the illustrated bottle comparison. Visual plausibility alone does not establish watertightness or reliable contact geometry.

Original research figure 5

Fig. 5: Real-world objects (left) and their reconstructed counterparts (right). Each object on the left was individually reconstructed using our pipeline. These assets were then manually arranged in simulation to approximately match their real-world poses and rendered to produce the image on the right. The strong visual similarity is notable, especially given that the reconstructions are rendered triangle meshes rather than neural renders.

Original research figure 6

Fig. 6: A selection of geometric reconstructions. The first two columns show multiple views of the same object (a power inflator in the first column and a Lego block in the second), demonstrating the completeness of our reconstructions. The last two columns highlight close-up views of other objects, illustrating the accuracy of both geometric and visual reconstruction, even for parts that were occluded during scanning. We provide interactive 3D visualizations on our project page.

Original research figure 7

Fig. 7: Comparison of BundleSDF [27] and Neuralangelo [13] reconstructions of a mustard bottle. Blue circles denote Blender [37] renders, while green diamonds represent Meshlab [38] renders. The BundleSDF mesh appears best in Blender but worst in Meshlab due to poor topology (e.g., scattered boundary faces), which requires a powerful renderer to compensate. In contrast, the Neuralangelo mesh maintains consistent quality across both renderers due to its well-structured topology. The effects of poor topology in the BundleSDF mesh appear as black lines, which originate from the mesh itself rather than the texture map. These artifacts are particularly noticeable at the top of the bottle’s body.

V-B Physical Parameter Identification

The identification benchmark uses eight known-inertia configurations with three trials each, averaging 24 measurements. The test object is mounted directly to the last link. Compared with the previously reported PMD-25K force-torque-sensor results, this joint-torque method has lower mean mass and center-of-mass error but higher rotational-inertia error. Different sensing arrangements and the directly mounted benchmark should be kept in view when interpreting the comparison. [17]

In a separate end-to-end test, a 3D-printed mustard bottle has ground-truth mass 427.1 g. Errors are 1.23% in mass, 6.33% in center of mass, and 358.6% in rotational inertia under the cited benchmark metrics. Accurate mass estimates therefore do not establish accurate full dynamics; camera calibration and indirect joint-torque sensing remain material limitations. Table I's error bars are one standard deviation.

TABLE I: Comparison of identification errors between our method and results reported in [17]. ‘FT‘ indicates that the results were obtained with a force-torque sensor, while ‘JT‘ refers to joint-torque sensors. The metrics are the ones proposed in [17]. Error bars represent one standard deviation.
Mass (%) CoM (%) Inertia Tensor (%)
OLS (FT) 3.18 24.9 >>500
PMD-25K (FT) 4.45 4.50 29.11
Ours (JT) 1.34 ±\pm 0.21 2.15 ±1.20\pm 1.20 42.35 ±\pm 15.64

V-C Simulation Performance

The assets run directly in a simulator, with pick-and-place, knock-over, and ramp demonstrations. These are qualitative comparisons. Individual rollouts are sensitive to initial conditions, and the study does not compare trajectory distributions or report quantitative rollout fidelity.

Resetting simulated states to measured real states could support equation-error evaluation, but sufficiently precise state estimation is outside the work's scope. Interactive simulations and demonstration videos remain on the project page.

Original research figure 8

Fig. 8: Our simulation experiments. The left column represents simulations, and the right column represents their real-world counterparts. The first row is pick-and-place, the second is knocking over, and the third is falling down a ramp. Different frames are overlaid transparently to show motion, and videos are available on the project page.

VI Conclusion

The contribution is an integrated automated asset-generation workflow and an object-centric reconstruction recipe that handles gripper occlusion separately from background. Its demonstrated scope is rigid objects in a controlled pick-and-place cell. Scaling beyond the initial dataset still depends on robust grasping and tracking, better rotational-inertia estimates, contact-parameter identification, and stronger validation of simulated dynamics.

Acknowledgement

The work was supported by Amazon.com and the Office of Naval Research. The authors acknowledge Lirui Wang, Andy Lambert, Thomas Cohn, Ge Yang, and Ishaan Chandratreya for discussions.

Concept demonstration

Open image ↗