Literature
Display
← All papers

SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models

Haowen Liu · Shaoxiong Yao · Haonan Chen · Jiawei Gao · Jiayuan Mao · Jia-Bin Huang · Yilun Du

· CVPR 2026 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

SIMPACT builds a task-specific simulator from a calibrated RGB-D observation, then uses simulated outcomes to refine vision–language-model action proposals. The supplementary material details prompts, physical-parameter estimates, timing, failures, and optional replanning.

Original SIMPACT teaser comparing a toppling carton with simulation-enabled action planning.
Figure 1: Simulation-Enabled VLM Action Planning. Given a single RGB–D image and a language task description (left), our method efficiently constructs a physics simulator that enables test-time VLM reasoning with physical grounding. This physically grounded rea- soning allows the robot to succeed in fine-grained manipulation tasks (bottom), outperforming a vanilla VLM planner (top) that lacks awareness of physical dynamics.

At a glance

Without additional model training, SIMPACT combines pretrained perception and vision–language models with rigid or deformable simulation. A model proposes parameterized symbolic actions, reviews simulated trajectories, and refines a plan before physical execution. Seven real manipulation tasks show 40–90% success over ten trials each. Results rely on a calibrated setup, specified action primitives, inferred physical properties, and minutes of planning; main execution is open-loop, with limited recovery experiments reported separately.

1 Introduction

Semantic task descriptions alone can miss consequences of contact, friction, and deformation. SIMPACT adds a constructed physics model so proposed actions have predicted outcomes that a vision–language model can inspect. The contribution combines single-view scene reconstruction with a test-time sampling/refinement loop, using pretrained components rather than new task-specific training. The resulting physical reasoning remains dependent on reconstruction and simulation accuracy.

Refer to caption
Figure 2: Simulation construction from a single RGBD image. Given an RGB-D image and a language task description, our pipeline automatically generates either a mesh-based simulation (top) for rigid objects or a particle-based simulation (bottom) for deformables. After segmenting objects-of-interest via GroundedSAM2 [55], we reconstruct either the 3D shape, scale, and pose of the object for rigid-body simulation, or perform dense sampling of particles within the volumes between the object surface and the table for the particle-based simulation pipeline. In both cases, we prompt the VLM to infer the relevant physical parameters required for simulation.

2 Related Works

Spatially grounded language planners use value maps, keypoints, affordances, or constraints, while model-based controllers evaluate predicted dynamics. SIMPACT combines these ideas with automatic scene construction and multiple physics engines. Unlike the compared Prompting-with-the-Future method [45], it uses the language model for action proposals and refinement as well as outcome evaluation. Comparisons concern these particular implementations and tasks, not a universal ranking of all model-based or learned policies.

3 Method

The input includes a task instruction with requirements and possible success/failure conditions, calibrated RGB-D data, robot state, and reconstructed object properties. Planning constructs a simulator, generates candidates, rolls them out, and asks the model to improve them from visual and numerical context. The final sequence becomes end-effector pose/gripper commands. Simulation-loop feedback occurs before execution; the principal physical policy does not continuously replan during contact.

3.1 Simulation Construction

The simulator state contains rigid poses or deformable particle positions, with geometry and mechanical parameters inferred from the initial observation. GroundedSAM2 [55] segments task-relevant objects. The model selects MuJoCo for rigid objects, a projective-dynamics variant for stiff deformables, or a material-point solver for soft materials. Objects are assumed static before interaction. Material guesses are commonsense priors, not measurements obtained through system identification.

Equation 1
st=Sim​(st−1,at,θ)s_{t}=\textsc{Sim}(s_{t-1},a_{t};\theta) (1)

Mesh-based Rigid Body Simulation.

For rigid objects, Hunyuan3D [57] generates a complete mesh from the segmented image. The mesh is centered and scaled using the observed point-cloud bounding box, then aligned with FoundationPose [63]. Mass, friction, and center of mass enter the physics model. Single-view occlusions and reconstructed unseen geometry can therefore affect contact planning.

Particle-based Deformable Object Simulation.

For deformables, segmented depth yields visible surface points. Particles fill the volume between that surface and the supporting table. Elasticity and plasticity parameters configure deformation. This construction is suited to the tested supported objects, without demonstrating arbitrary hidden geometry, articulated mechanisms, or unrestricted material identification.

Refer to caption
Figure 3: Method overview. Our method first instantiates a physics simulator given the real-world scene. Next, a VLM-based action sampler and optimizer iteratively refine the action sequence towards task success using simulated rollouts as context. The final optimized actions are then executed in the real world.

3.2 Action Planning via Simulation-enabled VLM

The default model supplies ten initial candidates and up to three refinement iterations. Each new candidate is simulated and evaluated using its final image and state; failed attempts are retained as context. The prose describes executing a verified successful plan, but the original pseudocode returns the last candidate when its iteration limit is reached without an explicit failure flag. A simulation verdict therefore should not be interpreted as a formal execution guarantee.

Algorithm 1 Action Planning Algorithm
Original algorithm 1 with complete text, code, and indentation.
Refer to caption
Figure 4: Action optimization process. We show a representative example from the non-toppling push task. The left three images show simulation rollouts from initial VLM-sampled action sequence proposals, all of which fail due to insufficient/overshooting push, or because the bottle topples. From these proposals, the VLM optimizer reasons a non-trivial action update that pushes the bottle for the correct distance without toppling in both simulation and real-world execution.
Table 1: Definition of tasks. For each manipulation task, we list the corresponding instruction and success criteria.
Tasks Instruction Success Condition
Non-toppling push Push the white carton forward to align horizontally with the others. The bottle does not topple and aligns within ±\pm2 cm.
Bowl stacking Grasp the pink bowl at its edge and stack it with the blue bowl. The pink bowl stably lies inside the blue bowl.
Pivoting Make the red pocky box lean vertically against the brown box. The red pocky box reaches vertical pose.
Shape rope Grab the free end of the rope and arrange the rope to a U shape. The deformed rope has an opening ratio in [0.5,2.0][0.5,2.0].
Shape playdoh Squeeze the Play-Doh to a square shape with equal sides. Two sides of the Play-Doh have a ratio within 1.5.
Avoid obstacle Push the orange bottle around the box to the image’s bottom-right. The orange bottle reaches the other side without colliding.
Sweeping Sweep the coffee beans into the purple box. All beans are pushed into the box.

VLMs for Action Proposal Generation.

Action proposals use the initial image, robot pose, object poses and dimensions, task goal, and workspace context. Symbolic primitives such as push, lift, grasp, release, rotate, and move carry continuous offsets or gripper width. A deterministic mapping converts them to pose trajectories. This constrained interface gives the model meaningful choices but bounds the behaviors it can express.

Equation
𝐚i=Action2Pose​(ai=VLM​(I0,ℓtask,s0,ℓsample)).\mathbf{a}^{i}=\textsc{Action2Pose}\big(\textsc{a}^{i}=\textrm{VLM}(I_{0},\ell_{\text{task}},s_{0};\ell_{\text{sample}})\big).

VLMs for Action Optimization.

The optimizer receives candidate actions plus rendered states and numerical object/gripper information at action boundaries. It can change the strategy rather than only perturb nearby numerical parameters. Example updates lower a pushing contact to avoid toppling and adjust travel distance. This is in-context inference over simulated attempts, not parameter learning or an optimization-convergence proof.

Equation 2
𝐚k=VLM⁡(c1,…,cK,ℓopt)\mathbf{a}^{k}=\mathrm{VLM}(c^{1},...,c^{K};\,\ell_{\text{opt}}) (2)

VLMs for Success Evaluation.

A model checks final simulated images and state against task requirements. Negative judgments trigger further refinement within the budget. Errors in the simulator or evaluator can still pass a plan that fails physically; the separate sim–real analysis quantifies such discrepancies for a limited sample.

Equation
TaskSuccess​(𝐬k)=VLM⁡(ITk,sTk,ℓtask,ℓeval).\textsc{TaskSuccess}(\mathbf{s}^{k})=\mathrm{VLM}(I_{T}^{k},s_{T}^{k},\ell_{\text{task}};\ell_{\text{eval}}).
Table 2: Success rates of our method and baselines. For each task, we run 10 trials per method. Our approach consistently achieves a substantially higher success rate than baselines, highlighting the effectiveness of simulation-enabled VLMs for action planning.
Method Non-toppling push Bowl stacking Pivoting Shape rope Shape dough Avoid obstacle Sweeping
π0.5\pi_{0.5}[2] 0% 0% 0% 0% 0% 0% 0%
VoxPoser[26] 0% 20% 0% 0% 0% 0% 20%
MOKA[16] 0% 10% 0% 20% 0% 0% 0%
Ours 80% 60% 40% 90% 80% 80% 70%
Refer to caption
Figure 5: Qualitative results. The figure shows the initial state, execution progress, and final state for six tasks in both the real world (top) and the simulation (bottom). By leveraging VLM’s powerful generalization, rendered simulation images can guide VLM’s test-time reasoning for action planning despite the visual sim2real gap. Fig. 10 shows the results of the remaining task.

4 Experiments

Seven custom physical tasks cover rigid contact, stacking, pivoting, rope shaping, dough shaping, obstacle avoidance, and sweeping. Main comparisons use ten trials per task/method, a small sample with ten-percentage-point resolution.

4.1 Experimental Setup

Hardware is a Franka Research 3 with parallel jaws and one calibrated RealSense D435i camera. Dough manipulation uses a custom printed end effector for broader contact. These embodiment and calibration choices are part of the tested system.

Tasks and Metrics.

Task success uses explicit geometric or completion conditions: carton alignment within two centimeters without toppling, stable nested bowls, a vertically leaning box, rope opening ratio between 0.5 and 2.0, dough side ratio within 1.5, collision-free passage around a box, and every bean swept into its container. The dough criterion permits approximate squareness; it is not exact shape reconstruction.

Table 3: Ablation. Success rates (%) over 10 trials for each task after removing each component of our method. Results demonstrate the importance of VLM-conditioned sampling and the VLM’s simulation-enabled test-time reasoning capabilities.
Method Non-toppling push Bowl stacking Pivoting Shape rope Shape dough Avoid obstacle Sweeping
w/o VLM sampler 0% 10% 0% 0% 0% 0% 20%
w/o simulation rollout 20% 0% 0% 30% 30% 0% 20%
w/o VLM optimizer 30% 50% 30% 40% 80% 20% 70%
Ours 80% 60% 40% 90% 80% 80% 70%
Refer to caption
Figure 6: Qualitative comparison with baseline methods. We show representative failures from baseline methods that lack simulation-enabled reasoning. These methods often choose incorrect action parameters, using improper pushing heights that cause toppling, or attempting to grasp the bowl at its center. Pivoting tasks fail because the baselines do not maintain contact with the box’s side face. For the rope task, baselines place the rope in the wrong direction due to missing deformation reasoning; for the dough-shaping task, baselines fail to plan the perpendicular squeezes needed to form a square; for the sweeping task, baselines fail to push all beans into the box.

Baselines.

Baselines are the pretrained π0.5 policy, VoxPoser’s volumetric affordance planner, and MOKA’s keypoint/affordance planner. MOKA is extended to pushing and squeezing via contact points and directions. These are out-of-domain zero-shot challenges for the baselines, so poor performance does not establish their general inability to perform manipulation.

Implementation details.

Gemini 2.5 Pro supplies proposals, refinement, and evaluation. The stiff-deformable solver is implemented in PyTorch and the material-point solver in Warp. Ten initial proposals plus at most three optimized proposals set the default planning budget.

4.2 Results

Success is 80% non-toppling push, 60% bowl stacking, 40% pivoting, 90% rope shaping, 80% dough shaping, 80% obstacle avoidance, and 70% sweeping. That is fifty successes across seventy task trials. π0.5 records zero in these tests; VoxPoser and MOKA achieve isolated 10–20% results. Simulated and physical rollout panels illustrate the contact strategies, while pivoting remains the weakest SIMPACT task.

4.3 Ablation study

Removing informed sampling is damaging despite five times as many random proposals. Removing simulation context also lowers success. Refinement helps most on pushing, rope shaping, and obstacle avoidance, but dough and sweeping tie the no-optimizer variant at 80% and 70%. More proposals are not always better: increasing ten to twenty improves pushing from 80% to 90%, but reduces stacking from 60% to 20% and rope shaping from 90% to 80%. Context overload is the authors’ explanation, not an independently isolated cause.

Table 4: Sampling length ablation. Success rates (%) over 10 trials varying numbers of in-context examples for tasks non-toppling push, bowl stacking, shape rope.
#Samples Non-toppling push Bowl stacking Shape rope
3 samples 50% 50% 40%
10 samples 80% 60% 90%
20 samples 90% 20% 80%

4.4 Failure Case Analysis

Failures are grouped into perception, planning, and execution. Single-view reconstruction can give wrong geometry; refinement can fail to locate a feasible contact strategy; and simulated success can fail on hardware because of pose or dynamics errors. Planning is the largest reported category, particularly for pivoting.

Refer to caption
Figure 7: Failure case decomposition graph. Failures are categorized as perception, planning, or execution.

4.5 Limitations

The system depends on single-view geometry, commonsense material estimates, and open-loop physical execution. Occlusion and parameter errors create sim–real gaps, while small execution offsets can compound. Proposed improvements include better completion/reconstruction, articulated models, interaction-based parameter identification, and tighter feedback. Optional recovery updates the simulator after failure, but is tested only on two tasks.

5 Conclusion

Simulation-informed proposal refinement improves this set of carefully specified physical tasks without task-specific model training. The evidence supports the tested integration of perception, physics, and language inference; it does not demonstrate universal scene understanding, real-time control, or guaranteed contact accuracy.

Acknowledgements

The acknowledgments credit help with experimental design, hardware, real-to-simulation implementation, and MOKA, and list support from the Kempner Institute, Amazon, and Pickle Robot.

Appendix A Implementation Details

The implementation appendix gives material-parameter questions, the complete symbolic action definitions, and sampling/refinement prompts with an example rollout context. All original prompt text and code formatting are retained.

A.1 Simulation Construction Details

Simulation setup selects a physics model and queries plausible parameters from the scene and object description. The original five-question sequence is preserved as a separate reading block.

Physical Parameters θ phys Prediction

Questions first identify manipulated objects and choose rigid or deformable simulation. Rigid objects receive mass and friction estimates. Stiff deformables use Young’s modulus, Poisson’s ratio, and density; material-point objects additionally select among jelly, metal, sand, foam, and plasticine, with friction angle or yield stress when applicable. These predictions provide simulator defaults rather than experimentally calibrated ground truth.

Q1.

Identify the objects that need to be manipulated from {task instruction}. Determine whether to use a rigid or deformable object simulator based on the object’s material.

Q2.

For rigid {object}, predict its mass and friction parameters.

Q3.

For deformable {object}, decide whether to use Projective Dynamics or the Material Point Method based on the stiffness of the object.

Q4.

For {object} simulated with Projective Dynamics, determine the following physical parameters:

  • Young’s modulus

  • Poisson’s ratio

  • Mass density

Q5.

For {object} simulated with the Material Point Method, first determine the material type of {object} from the set: {jelly, metal, sand, foam, plasticine}. Then output:

  • Young’s modulus

  • Poisson’s ratio

  • Mass density

  • (Optional) Friction angle

  • (Optional) Yield stress

Original physical-parameter questions

A.2 Action Planning Details

Seven primitives define horizontal push, vertical lift/descent, gripper width, release, yaw rotation, and three-dimensional displacement. End-of-primitive snapshots include rigid transforms or downsampled deformable keypoints/bounds. The source prompts and JSON-like examples remain exact, including their differing sampler/optimizer schemas, absolute-coordinate wording alongside delta fields, example object-name inconsistency, and abbreviated comments. These are published prompt outlines, not a fully executable validated interchange specification.

  • PUSH(δx\delta_{x}, δy\delta_{y}): Move the end-effector horizontally from its current position by (δx,δy)(\delta_{x},\delta_{y}) while maintaining its current height.

  • LIFT(δz\delta_{z}): Move the end-effector upwards along the zz-axis by δz\delta_{z}.

  • DESCEND(δz\delta_{z}): Move the end-effector downwards along the zz-axis by δz\delta_{z}.

  • GRASP(dd): Adjust gripper to a target width dd (in meters), where 0.00.0 is fully closed and 0.10.1 is fully open.

  • RELEASE: Fully open the gripper.

  • ROTATE(δyaw\delta_{\text{yaw}}): Adjust end-effector yaw relative to its current orientation by δyaw\delta_{\text{yaw}} (in radians).

  • MOVE(δx\delta_{x}, δy\delta_{y}, δz\delta_{z}): Move the end-effector from current position by (δx\delta_{x}, δy\delta_{y}, δz\delta_{z}).

Original symbolic action definitions
Original figure 8 prompt with complete text, code, and indentation.
Figure 8: Action sampling prompt ℓsample\ell_{\text{sample}} outline. This prompt includes task specifications, input requirements, action primitive definitions, planning guidelines, and output format. It is combined with visual observations and scene context as input to the VLM sampler to generate diverse action sequence proposals. Symbol #\# indicates the number of proposals to generate for each call.
Original figure 9 context with complete text, code, and indentation.
Figure 9: Example rollout context for action optimization in pivoting task. The context contains the action waypoints and the simulated state snapshots at each waypoint, including gripper pose, object poses, and screenshot paths. Only the first entry is shown for repeated fields, with omitted entries summarized using comments.
Refer to caption
Figure 10: Additional qualitative results. Following Fig. 5, this figure shows the initial state, execution progress, and final state for the sweeping tasks.
Original figure 11 prompt with complete text, code, and indentation.
Figure 11: Action optimization prompt ℓopt\ell_{\text{opt}} outline. This prompt includes task, input, and output specifications. It is combined with simulation rollout context as input to the VLM action optimizer to generate optimized action sequences.

Appendix B Supplementary Results

Supplementary tests cover sweeping, sampling-based optimization, sim–real outcome agreement, computation time, scene variation, parameter sensitivity, CALVIN, failure examples, and recovery.

B.1 Additional Qualitative Results

The extra sweeping panel supplies the seventh task’s simulated and physical sequence, complementing the six tasks shown in the main rollout figure.

B.2 Further Ablation Analysis

A random-sampling/CEM variant uses the model only to judge outcomes and reports zero success on these tasks. It follows the referenced method’s released optimizer/hyperparameters. This supports informed proposals and refinement in this action/task setup, without invalidating the original method’s results on its own pick-and-place domain.

B.3 Correlation Between Simulation and Real-World Performance

Five tasks contribute twenty sequences each: ten optimized plans and ten raw proposals. Simulation and reality agree on eighty-nine of these hundred outcomes; eleven are simulated successes that fail physically, and none fail only in simulation. This is outcome agreement on a deliberately mixed sample, not 89% task success or calibrated physical prediction over arbitrary actions. Pivoting and dough have 15% and 20% false-success discrepancy among their twenty samples.

Original figure
Figure 12: Correlation Between Simulation and Real-world Success/Failure. Results from 20 samples per task (100 total). Each rollout is categorized as one of: sim-success/real-success (green), sim-success/real-fail (yellow), sim-fail/real-success (red), and sim-fail/real-fail (blue). Simulation and real outcomes match in 89% of cases (both success or both failure), with 11% showing sim-success/real-fail. We observed no cases where a sequence failed in simulation but succeeded in the real world.

B.4 Computation Time

Mean component times are 1.9 minutes for simulator construction, 2.8 for action sampling, 0.8 for rollouts, and 0.9 for optimization—about 6.4 minutes when summed. Model inference and network calls dominate. Individual simulations take five to eight seconds. The system is a minutes-scale pre-execution planner in this implementation, not a low-latency feedback controller.

Table 5: Computation time. We compute the average computation time over 10 cases from each task.
Component Time (mins)
simulation construction 1.9
action sampling 2.8
simulation rollout 0.8
action optimization 0.9

B.5 Robustness Validation

Illustrations vary distractors, relative poses, object appearance, and materials. They show the kinds of scene changes included in evaluation, but do not provide a large separate robustness benchmark or isolate each variation’s effect.

Refer to caption
Figure 13: Example scene setup variations. Throughout our experiments, we vary the object types, poses, colors and materials to demonstrate the robustness and generalizability of our method.
Refer to caption
Figure 14: Failure cases. Example failure cases in bowl stacking, pivoting, shape rope and shape dough tasks.

B.6 Physics Parameters Estimation Analysis

Ten repeated material predictions are summarized, and a friction sweep probes downstream sensitivity. Estimated consistency does not establish parameter accuracy. Some reported standard deviations are incompatible with their stated ten-sample ranges, so Table 6 should be treated cautiously rather than silently corrected. Friction values 0.01/0.20/0.50/1.00/10.0 produce 40/100/80/80/0% success. This limited sweep supports sensitivity at extreme values, not universal robustness of inferred physics.

Table 6: Robustness analysis of VLM-estimated physics parameters (N=10N=10 samples). The low variance and stable ranges indicate consistent estimation capabilities.
Task Parameter Mean ±\pm Std Range [Min, Max]
Non-toppling Push Mass (kg) 1.033±0.00151.033\pm 0.0015 [1.0,1.05][1.0,1.05]
Friction Coeff. μ\mu 0.36±0.110.36\pm 0.11 [0.3,0.5][0.3,0.5]
Shape Playdoh Poisson’s Ratio 0.43±0.020.43\pm 0.02 [0.40,0.45][0.40,0.45]
Mass Density (kg/m3) 1186±651186\pm 65 [1000,1260][1000,1260]
Refer to caption
Figure 15: Replanning illustration. After initial failed execution, we perform re-planning after simulation update leading to successful completion.
Table 7: Success rate vs friction coefficient μ\mu.
Fric. Coeff. (μ\mu) 0.01 0.20 0.50 1.00 10.0
Success Rate (%) 40 100 80 80 0

B.7 Standardized Benchmark Results

CALVIN evaluation uses forty instruction chains and ground-truth segmentation masks. SIMPACT averages 2.78 completed instructions, above HULC’s 0.67 and RoboFlamingo’s 2.47 but below FLOWER’s 4.53 [56]. Its five-instruction completion is 20.0%, below RoboFlamingo’s 23.5%, despite the higher average length. Privileged masks, the small sample, and differing trained-versus-zero-shot conditions qualify this benchmark comparison.

Table 8: Evaluation results on the CALVIN Long-Horizon Multi-Task Language Control (LH-MTLC) benchmark.
#. Instruct. zero-shot 1 2 3 4 5 Avg. Len.
HULC [40] ✗ 41.8% 16.5% 5.7% 1.9% 1.1% 0.67
RoboFlamingo [35] ✗ 82.4% 61.9% 46.6% 33.1% 23.5% 2.47
FLOWER [56] ✗ 99.4% 95.8% 90.7% 84.9% 77.8% 4.53
Ours ✓ 87.5% 82.5% 47.5% 40.0% 20.0% 2.78

B.8 Failure Cases

Examples show bowl-placement flips and failed dough squeezes after small execution offsets. Pivoting can miss a narrow stable contact range, and rope proposals can lack sufficient diversity. Increasing proposal count is only a possible remedy; the main count ablation also shows degradation when context grows.

B.9 Replanning using Real-world Feedback.

Optional recovery updates the simulator from the new physical state and allows up to three replanning attempts. It recovers three of six initial pivoting failures and both of two obstacle-avoidance failures, with mean attempts 1.67 and 1.0 among the reported recoveries. These small conditional tests support recovery feasibility rather than a general closed-loop success guarantee.

Concept demonstration

Open image ↗