Literature
Display
← All papers

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Nadun Ranawaka · Josiah Wong · Wei-Lin Pai · Wei-Teng Chu · Tianyuan Dai · Masoud Moghani · Hang Yin · Yunfan Jiang · Wesley Durbano · Brandon Huynh · Yu Fang · Danfei Xu · Ruohan Zhang · Li Fei-Fei · Linxi Fan · Bowen Wen · Ajay Mandlekar · Yuke Zhu

Preprint posted · arXiv (preprint) ↗

Robotics17 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete 52-page v4, including every prompt, task card and qualitative reconstruction table. The PDF front matter is dated 25 June 2026; its arXiv version stamp is 5 August 2026. The pipeline supports automatic reconstruction, but the reported robotics scenes were interactively refined after automatic generation.

Complete original figure with every panel, label and legend.
Figure 1: Overview. SimFoundry takes a single real-world input video and automatically reconstructs an interactive, sim-ready digital twin of the scene. Based on the reconstructed digital twin, SimFoundry can further generate an unlimited number of digital cousins — affordance-preserving variants of the original scene, spanning three different axes of variation, which we term object, scene, and task cousins, respectively. These generated simulation environments support both real-to-sim policy evaluation and sim-to-real policy training, enabling policies to be benchmarked and improved at scale before deployment in the real world.

At a glance

SimFoundry reconstructs interactive objects and scenes from a video, then varies object instances, spatial layouts and task goals to generate training environments. Across seven policy-evaluation tasks it reports mean Pearson correlation .911 and mean maximum rank violation .018 between simulated and real performance. Simulation demonstrations can also train policies that transfer to physical robots. Reported gains depend on the particular tasks, training mix, human refinement and checkpoint-selection protocol.

1 Introduction

Real-world demonstrations and systematic robot evaluation are expensive; simulation helps only if scene construction is practical and the resulting geometry, appearance and dynamics support meaningful interaction. SimFoundry combines reconstruction, controlled variation, demonstration generation and evaluation in one modular pipeline. A digital twin approximates the captured scene; digital cousins preserve relevant affordances while changing objects, layouts or tasks. Its studies span articulated objects, multi-step tasks and bimanual manipulation, with separate geometric-fidelity and policy-performance evaluations.

2 Related Work

The system composes image-to-mesh generation, segmentation, depth estimation, pose alignment and articulation tools. Its distinction is the complete evaluation-and-training workflow plus several kinds of scene variation; Table 1 preserves the authors’ comparison with prior systems. Automated digital-cousin generation motivates object diversity, while PolaRiS is the main simulation-evaluation baseline. Detailed related-work distinctions remain in Appendix D.

Complete original figure with every panel, label and legend.
Table 1: System comparison. SimFoundry provides a unified and modular pipeline for real-to-sim scene generation that is more feature complete than prior works.
Selectable original table labels and values
[17]
RialTo
[79]
RoLA
[105]
PolaRiS
[32]
RobotArena-∞\infty
[33]
R2S-Soft
[104]
Re3Sim
[27]
GSWorld
[34]
[96]
MolmoSpaces
[40]
GenieSim
[101]
SimFoundry
(Ours)
Sim-to-real
training
✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✓ ✓
Real-to-sim
policy eval
✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓
Automatic scene
construction
✓ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓
Articulated
objects
✓ ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓
Multi-
embodiment
✗ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓
Asset
generation
✗ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓
Background
reconstruction
✗ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✓
Object
cousins
✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓
Scene
cousins
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓
Task
cousins
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓

3 Preliminaries

Real-to-sim evaluation runs existing policies in reconstructed scenes; sim-to-real training generates demonstrations there before physical deployment. End-to-end success requires every task criterion, while Pearson correlation measures linear agreement across policies and mean maximum rank violation measures the magnitude of ranking errors. MimicGen expands a small set of human demonstrations by transforming and joining subtask trajectories.

Complete original figure with every panel, label and legend.
Figure 2: Method Overview. SimFoundry extracts per-object relevant information (segmentation masks, depth, etc.), generates 3D visual meshes via 2D-to-3D generation models, and compiles the final output scene by annotating relevant physical parameters and sanity checking the overall scene configuration in a physics simulator. SimFoundry additionally supports diverse simulated augmentations along these axes of variation on object, scene, and task: object cousins can be generated by modifying input objects in their image space and re-generating corresponding 3D meshes; scene cousins can augment the configuration of objects; and task cousins can propose viable interactions within the scene.
Equation
πθ:𝒪→𝒜\pi_{\theta}:\mathcal{O}\rightarrow\mathcal{A}

4 SimFoundry: A modular, automated real-to-sim generation pipeline

Extraction chooses a representative RGB frame, estimates depth and camera intrinsics, fits a ground frame, then repeatedly segments and inpaints foreground objects. Generation upsamples object crops, produces meshes, estimates pose/scale, infers articulation, creates collision geometry and assigns physical properties before settling objects in simulation. Augmentation creates affordance-preserving object variants, semantic layout changes and new predicate-defined tasks. These categories may be combined rather than being disjoint. Static backgrounds can use automatically inpainted Gaussian splats, a separate clean capture with manual alignment, or mesh scans; the robotics experiments use mesh backgrounds in some cases.

Complete original figure with every panel, label and legend.
Figure 3: SimFoundry Scene Generation Samples. We show real-world input images (top row), the corresponding reconstructed digital twins generated by SimFoundry (middle), and sampled digital cousin scene variations (bottom). For instance, in the scene that is second from the left, the brown glass bottle becomes narrower for the cousin scene, and in the scene that is third from the right, the digital cousin of the wicker basket has holes near the top that can plausibly be used as handles, while the layout of the scene also changes.

5 Experiments

The evaluation covers DROID’s single-arm platform and a YAM bimanual workcell. It separates prediction of real policy performance, training from simulation demonstrations, and reconstruction quality/throughput.

5.1 Real-to-Sim Policy Evaluation

Existing real-world policies are tested in simulation without adaptation to either simulation framework. This does not mean every policy is unfinetuned: difficult-task policies have already received real task demonstrations.

Setup.

Five pretrained policy types—π0, π0.5, GR00T N1.6/N1.7 and DreamZero—are tested on four easier tasks. Three policy types are separately finetuned on fifty real demonstrations per task for three harder tasks. The different task groups avoid uniformly zero outcomes; they are not a common seven-task evaluation of all five models.

SimFoundry scene evaluations strongly correlate with real-world performance across diverse policies.

SimFoundry reports mean per-task Pearson correlation .911 and mean maximum rank violation .018. Table G.1 retains every real/sim score and unevaluated cell. Correlation indicates useful relative agreement across the evaluated policies; it is not a guarantee of matching absolute success on new policies or tasks.

Complete original figure with every panel, label and legend.
Figure 4: Tasks and Real-to-Sim Policy Evaluation correlations. (Left) We apply SimFoundry to a DROID setup using a single Franka arm (top two rows), and a bimanual setup with two YAM arms (bottom row). Our tasks span multiple types of manipulation, including multi-step, articulated object interaction, and bimanual coordination (Clear Table not shown, more details in Appendix I). (Right) Under zero-shot policy evaluation without simulator-specific finetuning, SimFoundry outperforms the state-of-the-art baseline PolaRiS [32] in simulation-based evaluation correlations. Each marker shape represents a different task from the left. Additional details in Appendix G and Figure G.1.

Sub-task evaluations improve correlations, especially on multi-step tasks.

For the harder finetuned tasks, starting simulations with early subtasks completed raises mean correlation from .902 to .951. These residual-task simulated successes are compared with full real-world task successes, so this is a changed evaluation protocol rather than improved end-to-end scene fidelity under the same initial states.

SimFoundry outperforms state-of-the-art simulation evaluation frameworks and makes fewer assumptions.

Under zero-shot simulation evaluation, SimFoundry’s mean Pearson score exceeds PolaRiS by over .59. PolaRiS benefits from shallow simulation finetuning, but its adapted checkpoint is intentionally excluded because the comparison targets existing policies without simulator-specific adaptation.

5.2 Sim-to-Real Policy Training

Training studies distinguish simulation-only task adaptation, combined real/simulation demonstrations, and multi-task transfer. Pretrained DROID models retain their earlier real-world training even when task-specific finetuning uses only simulation.

Complete original figure with every panel, label and legend.
Figure 5: SimFoundry Data Diversity Improves Policy Performance. (A) Across multiple robot embodiments and multiple tasks, leveraging additional object cousins [17] improves direct Sim-to-Real policy transfer on the original target scene objects and additional held-out unseen objects. (B) Scene cousins improve policy performance on the original scene and allow policy transfer to cousin scenes. (C) Adding task cousins improves performance on related downstream tasks by enabling intra-task transfer. Note: Pot refers to the Pot on Stove task, Trash refers to the Throw Away Trash task and Marker is for the Store Marker task.

Policies trained on SimFoundry-generated data transfer zero-shot to the real world.

Simulation-only training reaches 99% on YAM Pot on Stove and 100% on DROID Stack Dishware in the reported settings. YAM uses a flow-matching policy trained from scratch; DROID finetunes a π0.5 checkpoint already pretrained on real DROID data. Full per-object and held-out-instance results remain in Table G.4.

Co-training with sim and real data further improves performance.

Adding limited real demonstrations improves most DROID settings: π0.5 Store Marker rises from 60% to 92% real success, and π0 Throw Away Trash gains 36 percentage points in simulation. Table G.7 also retains exceptions, including a simulated Stack Dishware decline for π0.

Object cousins improve robustness to unseen objects.

Object cousins vary shape, appearance and topology while aiming to retain function. Gains include fifty percentage points on held-out real Pot on Stove objects and up to twenty points for DROID Throw Away Trash. Held-out object and original-object performance are reported separately.

Scene cousins improve layout generalization.

Scene cousins change semantic spatial relations. They improve simulated Throw Away Trash by 28 points in the twin scene and allow 16% Store Marker success on cousin layouts versus zero without scene variation. These particular scene-cousin results are simulation results.

Multi-Task Sim-to-Real and Task Generalization.

One reconstructed cluttered scene yields thirteen generated-data tasks and seven held-out tasks. A single policy is finetuned on the generated multi-task demonstrations and evaluated in both simulation and reality; Table 2 preserves checkpoint-specific outcomes.

Table 2: Multi-task policy evaluation. Success rates for policies evaluated on seen and held-out tasks in simulation and the real world.
π0.5\pi_{0.5}-DROID π0.5\pi_{0.5}-FT π0.5\pi_{0.5}-DROID-FT
Sim 30 51 61
Sim – held out 37 45 33
Real 28 45 46
Real – held out 26 29 26

SimFoundry can train generalist policies and task cousins improve few-shot downstream learning.

Multi-task training improves seen-task performance over the base DROID checkpoint by up to 31 points in simulation and eighteen in reality. π0.5-FT reaches 29% on held-out real tasks. At fixed total demonstration count, thirteen related task cousins raise simulated Throw Away Trash by sixty points and Store Marker by forty; these gains should not be described as measured real-world improvements.

5.3 System Analysis

Across twelve staged scenes, automatic reconstruction achieves F1 .81–.92 versus SAM3D .66–.71 at a one-centimeter threshold. About three minutes of per-object manual tuning raises F1 to .93–.99. The roughly five-minute-per-object foreground reconstruction average excludes the separate approximately ninety-minute automatic background-inpainting cost.

6 Limitations

The system inherits failures and nondeterminism of its component models, and its settling procedure assumes a single flat tabletop. Appendix C adds depth/scale error, difficult articulation segmentation, background hallucinations and rendering trade-offs.

7 Conclusion

The experiments support reconstructed simulation as a useful environment for ranking existing policies and generating training data. Object, layout and task diversity address different transfer limitations, but the evidence remains tied to the tested robot/task settings, manual scene refinement and selected checkpoints.

8 Acknowledgments

The acknowledgments credit data collection, policy evaluation, hardware support and feedback. Nadun Ranawaka is partially supported by Georgia Tech’s Agricultural Technology Research Program.

Appendix

The complete appendix is retained below, covering reconstruction, augmentation, protocols, policy settings, human intervention and quantitative/qualitative analysis.

Appendix A Overview

Appendices B–L cover FAQs and limits, extended related work, scene generation, digital cousins, detailed results, policy training, robot/task definitions, evaluation, manual editing and system analysis.

Appendix B FAQ

The FAQ confirms that robotics scenes were automatically generated and then manually refined for a few minutes. Physics settling can move estimated objects. Transparency support depends on the mesh generator: Hunyuan 2.1 lacks it, whereas TRELLIS.2 supports it. Base DROID checkpoints score zero on Store Marker and Throw Away Trash but 48–52% on Stack Dishware, so task data adds value beyond pretraining. Main success is binary completion of all criteria, rather than a partial-credit normalized reward.

Appendix C Limitations

Remote VLMs can produce inconsistent inpainting and duplicated or degenerate objects. Monocular depth limits metric scale/shape; articulation depends on reliable segmentation of visible and internal surfaces. Settling assumes a flat support plane. Automatic background inpainting takes about ninety minutes per scene on one GPU and may hallucinate surfaces. Some robotics experiments instead use mesh backgrounds because Gaussian splats suffer near-field clipping and rendering latency.

Appendix D Full Related Work

The expanded discussion distinguishes physically simulated policy learning/evaluation from rendering-only scene reconstruction.

D.1 Real–to–Sim for Simulation Environment Creation and Applications

Prior real-to-sim work is grouped by downstream application and physical modeling requirements.

Real–to–sim–to–real manipulation.

Real-to-sim-to-real manipulation reconstructs scenes for robot training, often with rigid objects and simple pick/place tasks. SimFoundry targets articulated, bimanual and multi-step settings and adds object, scene and task variants to escape a single static twin.

Real–to–sim–to–real for navigation and locomotion.

Navigation and locomotion also use reconstructed environments, but their contact and control demands differ from the manipulation settings evaluated here.

Real–to–sim policy evaluation.

Several simulation-evaluation systems correlate with real performance without demonstrating that their data trains physical agents. The authors position SimFoundry as supporting both uses and general articulated-object tasks; this is their comparison of the cited methods.

Real–to–render.

Rendering-only methods avoid physical simulation, which limits their direct use for contact-sensitive manipulation. SimFoundry models physical interactions and expands each reconstructed scene with controlled variants.

D.2 Imitation Learning from Human Demonstrations and Synthetic Data Generation.

Teleoperation plus behavior cloning scales through large human data-collection efforts. Synthetic generation tools can transform a few demonstrations into many trajectories with broader scene coverage. SimFoundry supplies reconstructed environments and variations for that process, while still requiring initial human demonstrations.

Appendix E Scene Reconstruction

Scene reconstruction separates foreground extraction/generation, articulation, physical settling and background registration.

Complete original figure with every panel, label and legend.
Figure E.1: Articulation and 3DGS Background Pipeline Overview. SimFoundry generates articulated objects by first decomposing a pre-existing mesh into subsequent parts, which are then annotated with relevant joint types, locations, and ranges via a VLM. SimFoundry also can automatically generate a high-fidelity 3DGS background by first generating a synthetic video with removed foreground, extracting extrinsics, and training a 3DDGS to reconstruct the scene geometry.

E.1 Extraction Details

The extraction loop turns the representative image into a metric point cloud, isolates foreground RGB-D crops and masks, and inpaints removed objects before searching the residual scene. Occluded geometry remains inferred rather than directly measured.

Representative Frame Selection

The current default is frame zero. Users should begin filming from a clear, broadly front-facing view of the whole scene with minimal occlusion; there is no automatic best-frame search in this implementation.

E.2 Foundation Model Details

Default modules are DepthAnything3 for monocular/video depth, FoundationStereo for stereo, SAM3 for image masks, Gemini-Pro-3 for reasoning/articulation, Gemini-Pro-3-Image-Preview for editing, PriorDepthAnything for depth inpainting, Hunyuan2.1 or TRELLIS.2 for meshes, FoundationPose for pose refinement, and P3-SAM for mesh segmentation. SAM2 propagates video masks and VOID performs two-pass video inpainting. Segment Any Mesh and Partfield are alternative mesh segmenters.

E.3 Articulated Object Generation

The articulation module builds on Articulate Anymesh and Articulate Anything to turn a static generated mesh into linked parts and joints.

Segmentation

A VLM identifies movable semantic parts and joint types from multiple views. Mesh segmentation labels faces; adjacency majority voting fills unlabeled internal faces. Colored segment renderings let the VLM merge oversegmented pieces into semantic parts. Users can refine segmentation/assignments through the GUI.

Joint Generation

A VLM writes calls to a part-relative joint-placement API, which compiles to URDF. The simulator animates the proposed joints, and a separate critic rates movement and proposes corrections. Iteration stops after the critic exceeds a threshold; plausibility judgments are model-based, not measured physical identification.

Physical Parameters

The VLM estimates link mass, joint friction and damping using part and object volumes as context. These parameters are inferred from visual/semantic information.

E.4 Object Depenetration and Physical Stability

CoACD produces collision meshes. PyBullet steps the assembled scene while resetting velocities to zero after every step to suppress depenetration explosions. Once settled, poses are cached for stable initialization; this may alter alignment with the original observed poses.

E.5 Background Reconstruction and Alignment

Both background paths output a Gaussian splat with a transform into the reconstructed scene. Automatic processing reuses the original capture; manual processing uses a separate foreground-free capture.

E.5.1 Automatic Background Reconstruction and Alignment

Automatic reconstruction performs foreground removal, metric depth/pose recovery, depth-supervised splat training and a derived rigid alignment to the simulator.

Frame preparation and foreground inpainting.

Frames are uniformly subsampled. A VLM enumerates foreground categories, SAM3 refines masks and SAM2 propagates them. VOID first fills removed regions and then re-inpaints residual hallucinations in GPU-sized chunks, producing a synthetic background RGB stream.

Metric depth and pose recovery.

DepthAnything3 runs on original frames for camera poses and inpainted frames for depth. Overlapping chunks are joined by Umeyama similarity fits on shared camera centers. A second fit aligns the inpainted-depth point cloud to the original-stream world and exports seed positions for splat training. Claimed millimeter residuals concern these alignment fits under smooth capture, not ground-truth reconstruction accuracy.

Depth-supervised splat training.

Nerfstudio trains the splat using inpainted RGB plus confidence-masked L1 depth supervision. Original-stream poses place the cameras; per-camera rigid pose optimization absorbs residual mismatch and is reported as essential for sharpness. Depth supervision reduces floaters over textureless surfaces.

Rigid bridge into the simulator world.

The anchor frame’s pose in the original camera world is composed with its ground-plane-aligned extraction pose to map the splat into simulation. This establishes registration through shared estimated poses, rather than a new independent manual alignment.

E.5.2 Manual Background Alignment

The manual path clears foreground objects physically, records another video, reconstructs a clean background and then aligns it in the scene editor.

Clean-stream capture and pose estimation.

A similar camera trajectory captures the emptied scene. Nerfstudio extracts frames and COLMAP estimates intrinsics, camera poses and a sparse initialization cloud.

Splat training on the foreground-free video.

Nerfstudio’s splatfacto-big model trains with photometric supervision on real foreground-free images and exports a PLY. Avoiding inpainting can yield sharper surfaces and silhouettes.

Interactive alignment via the scene editor.

Because the second capture has an independent COLMAP coordinate system and scale, a user aligns translation, rotation and isotropic scale against reconstructed foreground meshes. The saved transform is reused on later loads.

E.5.3 Comparison and Trade-offs

Automatic processing saves capture/alignment effort and better registers rendered viewpoints, but adds approximately ninety minutes of two-pass inpainting on an RTX 3090 and may hallucinate hidden surfaces. A second clean capture improves surface fidelity at the cost of physical scene clearing and manual alignment. Better pixel-alignment metrics do not by themselves establish sharper underlying surface geometry.

Appendix F Digital Cousins Augmentation

Object, scene and task cousins vary different aspects of the reconstructed environment while aiming to preserve executable interactions.

F.1 Object Cousins Augmentation

The object pipeline canonicalizes names, identifies functional parts, proposes geometric/topological/visual changes, edits only the selected component with scene context, and filters outputs for plausibility. Topological variation is suppressed when implausible. Accepted variants retain prompts, descriptions and verification metadata. The complete original prompt templates are restored below.

Complete original figure with every panel, label and legend.
Figure F.1: Object Cousins Prompt Templates. Prompt templates used in the object cousins generation pipeline include functional decomposition, component-wise variation proposal, scene-aware object editing, and real world/scene reasonableness verification. At runtime, the variables in curly brackets are replaced with dynamic values per object and/or component.
Selectable original prompt restored from PDF
VLM Prompts for Object Cousins Generation
Prompt A: Functional Component Decomposition

With the given object image, decompose the object into functional components based on grasp affordance, list the component
     name only in number list, if there is a main body, treat its edge or bottom as a same component inside main body. Make
     sure to only generate the list of component names, don’t generate any other additional text.

Prompt B: Topology Feasibility Check

For the {component} of the object shown, is it reasonable to make variation on topology such that the modified object still
     looks realistic and is reasonable enough that we would see this object in our daily task? Answer with simply yes or no.
      If it is likely not to be normal, make it a no.

Prompt C: Component-wise Variation Proposal

For the component ’{component}’, list {num_geometry} ways to generate variation through Geometry dimension, {num_topology}
     ways through Topological dimension, and {num_visual} ways through Visual dimension:

1. Geometry dimension
2. Topological dimension
3. Visual dimension

Make sure to be determined in the description, prevent using words like or, such as, like, or similar to in the description.
Make sure to still maintain the object to have a realistic and reasonable look like something you would see in the daily
     life.
For example, a red banana and a banana with a hole in it is a bad variation, because we rarely see those in daily life.
But a banana with green skin is a good variation since the unripe banana is usually green.

Format your response as:
Component: {component}
Geometry:
1. [variation description]
...
Topology:
1. [variation description]
...
Visual:
1. [variation description]
...

If topology variation is judged infeasible, the prompt instead requests only Geometry and Visual variations.

Prompt D: Scene-Aware Object Editing

Modify the following component of the object: ’{component}’.
Make sure to generate a realistic modified version, make sure it looks normal and reasonable like something you would
     usually see in the daily life.
Do not generating variations that are weird or unusual, such as a banana with a hole in it or a banana with a red skin.
Apply this {dimension} variation: {variation_description}.
Keep all other components unchanged.
Generate the image with transparent background.
Make sure the modified object style fits the scene context.

Prompt E: Real-World and Scene Reasonableness Verification

You will see a scene image and a candidate object image. Judge whether the candidate object is reasonable in the real world
     and also reasonable in the given scene. Be strict and practical: reject unusual, implausible, or scene-inconsistent
     objects.

Return format:
RealWorldReasonable: <yes/no>; RealWorldReason: <short reason>
SceneReasonable: <yes/no>; SceneReason: <short reason>

F.2 Scene Cousins Augmentation

An anchor is selected by the user or a VLM, then objects receive allowed LeftOf, RightOf, InFrontOf, Behind, OnTopOf or Inside predicates, potentially in combination. The user specifies which predicates are valid for each object/task. BEHAVIOR distractors can be filtered by category, mass, volume and density, then placed without collisions.

F.3 Task Cousins Augmentation

A VLM receives a scene image, interactable-object list, robot constraints and allowable predicates. It proposes tasks requiring changes from the initial state and emits structured semantic groups and goal conditions for simulation/data generation. These proposals remain constrained by inferred affordances and validation, rather than proving arbitrary tasks executable.

Complete original figure with every panel, label and legend.
Figure F.2: Task Cousins Prompt Template. The exact prompt template used to query VLM for proposing tabletop manipulation tasks. Bracketed variables are populated dynamically per scene.
Selectable original prompt restored from PDF
VLM Prompt for Task Proposal
You are a professional robotic expert in sim-to-real, robot learning, and VLAs.

Look at the provided image of a tabletop scene in a simulation (OmniGibson). Using ONLY the objects listed below, propose
     exactly {num_tasks} distinct tasks that a {robot_type} arm can perform in this scene, ensure that each task goal
     requires a change from the current configuration of the scene, for example, do not propose tasks to put a pear in bowl
     if it is already in the bowl. Each task should have clear goal conditions expressible with object states.

Important:
{robot_constraint}
{object_constraints}

Scene object list (id -> category, name; use these names as group identifiers in predicates):
{object_list}

Output format: for each of the {num_tasks} tasks, output a YAML block that can be merged into a task config. Use this exact
     structure for each task.

For each task provide:
1. task_name: short snake_case name (e.g. stack_cup_on_plate)
2. semantic_group_mapping: map each logical group name you use (e.g. cup, plate) to a list containing exactly one scene
     object "name" from the list above (e.g. cup: [blue_cup_syvtml_19], plate: [teal_plate_qdudop_16]).
3. goal_predicates_all: list of predicates that must ALL be true for success. Each predicate has: state, state_kwargs (null
     if not needed), value (true/false), group, other_group (only for binary states like OnTop, Touching).
4. goal_predicates_any: list of predicates where ANY being true yields success (optional; use null if not needed).

Allowed states (from OmniGibson object_states): {states}

For binary relations use group and other_group. For unary states (Open, ToggledOn, etc.) use group and set other_group to
     null or omit.

Example predicate format:
  goal_predicates_all:
    - state: OnTop
      state_kwargs: null
      value: true
      group: cup
      other_group: plate
  goal_predicates_any: null

Output exactly {num_tasks} tasks. Separate each task with "---" and a task number (e.g. "--- Task 2"). Each task block must
     be valid YAML with keys: task_name, semantic_group_mapping, goal_predicates_all, goal_predicates_any.

F.3.1 Task Cousins Example

The example generates thirteen tasks, collects ten human teleoperated demonstrations per task and uses MimicGen to generate one hundred per task before multi-task π0.5 finetuning. All thirteen illustrated instructions/goals remain intact. Two cards say to put an eraser or cup “in” a bowl but specify OnTop, an inconsistency retained from the source.

Complete original figure with every panel, label and legend.
Figure F.3: Task Cousins Generation Example. Real and Sim Scene used in task cousins generation example
Selectable original panel labels

(a) Real Scene

(b) SimFoundry Output

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Baseball in Bowl Instruction: Pick up the baseball, then put it in the bowl. Goal: Baseball Inside of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Black Eraser on Organizer Instruction: Pick up the black eraser, then put it on the organizer. Goal: Black Eraser OnTop of Organizer

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Burger in Bowl Instruction: Pick up the burger, then place the burger in the bowl. Goal: Burger Inside of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Orange Marker on Organizer Instruction: Pick up the orange marker, then put the orange marker on the organizer. Goal: Orange Marker OnTop of Organizer

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Red Bottle in Bowl Instruction: Pick up the red bottle, then put the bottle in the bowl. Goal: Red Bottle Inside of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Red Marker in Cup Instruction: Pick up the red marker, then put the red marker in the cup. Goal: Red Marker Inside of Blue Cup

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Tennis Ball in Bowl Instruction: Pick up the tennis ball, then put the tennis ball in the bowl. Goal: Tennis Ball Inside of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Tennis Ball on Organizer Instruction: Pick up the tennis ball, then put the tennis ball on the organizer. Goal: Tennis Ball OnTop of Organizer

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Black Marker on Plate Instruction: Pick up the black marker, then put the black marker on the plate. Goal: Black Marker OnTop of Plate

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Black Eraser in Bowl Instruction: Pick up the black eraser, then put the black eraser in the bowl. Goal: Black Eraser OnTop of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Orange Cup in Bowl Instruction: Pick up the orange cup, then put the orange cup in the bowl. Goal: Orange Cup OnTop of Bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Orange Cup on Organizer Instruction: Pick up the orange cup, then put the orange cup on the organizer. Goal: Orange Cup OnTop of Organizer

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Place Red Marker on Organizer Instruction: Pick up the red marker, then put the red marker on the organizer. Goal: Red Marker OnTop of Organizer

Appendix G Detailed Experiment Results

The full tables and plots retain individual policy/task outcomes, correlation metrics, object/layout/task ablations and combined real/simulation training results.

G.1 Detailed Results for Real-to-Sim Policy Evaluation

Tables G.1–G.2 report the same real-world scores alongside each simulator’s outcomes. SimFoundry better preserves policy rankings in these settings; many PolaRiS policies perform poorly, especially task-finetuned ones. Unevaluated cells remain distinct from zero.

Table G.1: Real-world vs. simulation success rates (%) for policies evaluated in SimFoundry. The rightmost columns report per-task Pearson rr (higher is better) and MMRV (lower is better) computed between the real-world and simulation success rates across all policies evaluated on that task. Cells marked “–” indicate the policy was not evaluated on that task.
π0\pi_{0} π0.5\pi_{0.5} GR00T N1.6 GR00T N1.7 DreamZero Real↔\leftrightarrowSim agreement
Task Real Sim Real Sim Real Sim Real Sim Real Sim Pearson r↑r\uparrow MMRV ↓\downarrow
Stack Dishware 100 34 100 64 40 0 – – – – 0.883 0.000
Store Marker 48 4 60 20 32 0 – – – – 0.915 0.000
Throw Away Trash 20 0 48 4 0 0 – – – – 0.910 0.067
Serve Fruits 0 4 72 80 4 20 40 32 8 12 0.960 0.016
Cup in Bowl 88 56 100 92 68 40 92 92 100 92 0.907 0.016
Marker in Cup 40 40 92 88 28 28 88 88 88 80 0.995 0.008
Clear Table 0 12 40 36 0 0 8 28 16 28 0.810 0.016
Table G.2: Real-world vs. simulation success rates (%) in PolaRiS.
π0\pi_{0} π0.5\pi_{0.5} GR00T N1.6 GR00T N1.7 DreamZero Real↔\leftrightarrowSim agreement
Task Real Sim Real Sim Real Sim Real Sim Real Sim Pearson r↑r\uparrow MMRV ↓\downarrow
Stack Dishware 100 0 100 8 40 0 – – – – 0.500 0.200
Store Marker 48 0 60 4 32 0 – – – – 0.822 0.053
Throw Away Trash 20 0 48 0 0 0 – – – – – 0.253
Serve Fruits 0 4 72 28 4 24 40 4 8 4 0.480 0.288
Cup in Bowl 88 20 100 36 68 76 92 48 100 68 -0.396 0.280
Marker in Cup 40 0 92 4 28 4 88 12 88 4 0.512 0.176
Clear Table 0 0 40 0 0 4 8 0 16 12 -0.037 0.352
Complete original figure with every panel, label and legend.
Figure G.1: Enlarged Real-to-Sim policy evaluation correlations with task labels. This figure expands the right panel of Figure 4. Each point compares real-world and simulated success rates for a policy-task pair. The dashed diagonal indicates perfect agreement between simulated and real-world success rates. Blue points correspond to SimFoundry evaluations and orange points to PolaRiS evaluations. SimFoundry points lie closer to the diagonal across tasks, matching the higher Pearson correlations and lower MMRV values reported in Table G.1 and Table G.2.

G.1.1 Sub-Task Evaluations improve Real-to-Sim Correlations

A separate residual-task protocol examines behavior after some initial steps have already succeeded.

Sub-task Evaluation Protocol.

Simulation resets to states with early subtasks completed, such as an already-open drawer for Store Marker. The remaining-task success is compared with full end-to-end real success, which changes the simulated initial-state distribution and task burden.

Sub-task evals improve correlations and provide insights into failure modes.

Across three finetuned tasks, this protocol increases mean correlation from .902 to .951. Once the drawer is open, π0.5 usually finishes the remaining Store Marker steps. This diagnoses bottlenecks but does not replace the original end-to-end results.

Table G.3: Real-world vs. simulation success rates (%) correlations improve when evaluating on sub-tasks.
π0\pi_{0} π0.5\pi_{0.5} GR00T N1.6 Real↔\leftrightarrowSim agreement
Task Real Sim Real Sim Real Sim Pearson r↑r\uparrow MMRV ↓\downarrow
Stack Dishware 100 64 100 80 40 24 0.961 0.000
Store Marker 48 52 60 76 32 36 0.981 0.000
Throw Away Trash 20 0 48 8 0 0 0.910 0.067

G.2 Detailed Results for Sim-to-Real Experiments

Training compares the twin alone with the twin plus nine object cousins. Evaluation separates original simulated objects, held-out simulated cousins, original physical objects and held-out physical cousins, so instance generalization is not conflated with transfer on reconstructed objects.

Ablation of Object, Scene and Task Cousins.

The plots report average gains near seventeen points for object cousins and forty for task cousins; scene gains differ for twin and cousin layouts. Their measurement domains differ: the scene/task-cousin ablations in the retained tables are simulated evaluations, whereas object studies include physical transfer. Co-training usually helps but is not uniformly beneficial.

Complete original figure with every panel, label and legend.
Figure G.2: SimFoundry data diversity along different axes scales data generation and policy performance. We ablate how different sources of SimFoundry-generated data improve policy learning and generalization. (A) Object cousins improve robustness across DROID and YAM by training policies on affordance-preserving object variants, yielding an average zero-shot sim-to-real success improvement of 17%17\% and up to a 50%50\% real-world gain on held-out Pot on Stove objects. (B) Scene cousins improve layout generalization by training on semantically modified object arrangements, producing an average improvement of 21%21\%, including a 28%28\% gain on Throw Away Trash in the twin scene and 16%16\% success on held-out Store Marker cousin layouts where the twin-only policy achieves 0%0\%. (C) Task cousins improve downstream task learning by adding demonstrations from related tasks while keeping the total number of demonstrations fixed; 13 task cousins improve Throw Away Trash by 60%60\% and Store Marker by 40%40\% in simulation. (D) Sim-and-real co-training further improves performance by combining scalable SimFoundry demonstrations with limited real data, increasing π0.5\pi_{0.5} real-world Store Marker success from 60%60\% to 92%92\% and improving π0\pi_{0} simulated Throw Away Trash success by 36%36\%. Together, these results show that SimFoundry provides complementary forms of structured data diversity across objects, layouts, tasks, and real/sim data mixtures.

YAM

Twin
++
9 Cousins
Stack
Dishware
Sim Twin 83 92
Sim Cousins 43 66
Real Twin 39 43
Real Cousins 21 42
Pot On
Stove
Sim Twin 85 100
Sim Cousins 17 93
Real Twin 91 99
Real Cousins 14 64
Throw Away
Trash
Sim Twin 97 97
Sim Cousins 97 94
Real Twin 0 28
Real Cousins 2 8

DROID

Twin
++
9 Cousins
Stack
Dishware
Sim Twin 80 88
Sim Cousins 64 92
Real Twin 88 96
Real Cousins 88 100
Store
Marker
Sim Twin 20 60
Sim Cousins 8 28
Real Twin 4 20
Real Cousins 4 4
Throw Away
Trash
Sim Twin 8 48
Sim Cousins 4 48
Real Twin 0 20
Real Cousins 0 8
Table G.4: Policy Robustness Using Object Cousins. Across multiple robot embodiments and multiple tasks, leveraging additional object cousins [17] improves direct sim2real policy transfer on the original target scene objects and additional held-out unseen objects.
Table G.5: Boosting Performance with Scene Cousins (DROID, simulation) Success rates shown below.
twin only + scene cousin
Stack Dishware 80 88
Stack Dishware- cousin 28 64
Store Marker 20 24
Store Marker- cousin 0 16
Throw Away Trash 8 36
Throw Away Trash- cousin 0 36
Table G.6: Boosting Performance with Task Cousins. Adding additional tasks and cousins with the same or similar objects increases performance for the downstream task.
twin only +1 task +7 tasks +13 tasks
Stack Dishware 80 88 100 100
Store Marker 20 36 48 60
Throw Away Trash 8 44 44 68
Table G.7: Co-training with SimFoundry generated data. We compare success rates of models trained with simulation-only data (-S) to those trained with real-world demos (-R) as well as combinations of both simulation and real data (-co-train). Each model type is evaluated in both the real-world scene (-Real) and the SimFoundry reconstruction (-Sim).
π0\pi_{0}-S π0\pi_{0}-R π0\pi_{0}-co-train π0.5\pi_{0.5}-S π0.5\pi_{0.5}-R π0.5\pi_{0.5}-co-train
Stack Dishware- Sim 92 34 76 88 64 100
Stack Dishware- Real 96 100 100 96 100 100
Store Marker- Sim 16 4 40 60 20 60
Store Marker- Real 4 48 80 20 60 92
Throw Away Trash- Sim 0 0 36 48 4 60
Throw Away Trash- Real 0 20 76 20 48 96

G.3 Detailed Object Cousin Ablation

A bimanual ablation fixes the budget at one thousand demonstrations and splits it evenly across the twin plus zero, one, three or nine cousins. Results aggregate twenty-five trials with paired sampled initializations in sim and reality. More cousins generally improve robustness, but Table G.8 includes nonmonotonic task/setting outcomes.

Table G.8: Sim-to-real policy training results across object cousins.
Twin
++
1 Cousin
++
3 Cousins
++
9 Cousins
Stack
Dishware
Sim Twin 83 89 100 92
Sim Cousins 43 44 65 66
Real Twin 39 41 37 43
Real Cousins 21 32 27 42
Pot On
Stove
Sim Twin 85 100 93 100
Sim Cousins 17 27 35 93
Real Twin 91 100 94 99
Real Cousins 14 38 16 64
Throw Away
Trash
Sim Twin 97 98 98 97
Sim Cousins 97 100 97 94
Real Twin 0 9 45 28
Real Cousins 2 17 14 8

Appendix H Policy Details

Policy construction combines human seed demonstrations, trajectory augmentation, domain randomization and simulator-based checkpoint selection.

H.1 Data Generation Details

Approximately ten to fifteen human demonstrations seed MimicGen augmentation. Randomization covers materials and camera poses, plus table height in DROID. The thirteen-task example instead specifies ten seeds and one hundred generated demonstrations per task.

H.2 Policy Training Details

DROID joint-position π0/π0.5 checkpoints are finetuned for 10,000 steps with batch 256 and learning rate 10⁻⁵, evaluated in simulation every thousand steps; the best simulated checkpoint is tested in reality. YAM flow-matching policies train for 40,000 steps using proprioception and top/wrist RGB, with joint-position and gripper actions and analogous checkpoint selection.

Real-to-Sim Policy Details and Selection.

The harder real-to-sim tasks finetune π0, π0.5 and GR00T N1.6; the four simpler tasks also include unadapted GR00T N1.7 and DreamZero. All output joint/gripper positions. Simulation-based checkpoint selection precedes the reported physical evaluation.

Appendix I Robot Platform and Task Details

This appendix specifies the two robot platforms and every original language instruction and completion criterion.

I.1 Robot Embodiments.

DROID combines a Franka Panda, two external ZED-2 views and a wrist ZED-Mini on a portable desk, with Oculus teleoperation. Joint-position control is used in both domains, but simulator gains are increased to reduce tracking error. YAM has two arms, wrist RealSense D405 cameras, an external top-down view, JoyLo teleoperation and joint-position control.

I.2 Task Rubric

The eight original illustrated task rubrics are retained below. Main success requires every listed subtask; only the explicitly separate subtask-evaluation study starts from partially completed states. Tasks cover placement, multi-object cleanup, stacking, opening/filling/closing receptacles and bimanual pot transfer.

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Cup in Bowl Instruction: Pick up the cup, and put the cup inside the bowl. Rubric: 1. Touch cup 2. Pick cup 3. Place in bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Marker in Cup Instruction: Pick up the marker, and put the marker in the cup. Rubric: 1. Touch marker 2. Pick up marker 3. Place marker in cup

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Serve Fruits Instruction: Put both the banana and the apple on the green plate. Rubric: 1. Pick up banana 2. Pick up apple 3. Place banana on green plate 4. Place apple on green plate

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Clear Table Instruction: Put all objects in the brown box Rubric: 1. Place cube in box 2. Place grapes in box 3. Place cup in box 4. Place marker in box 5. Place baseball in box

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Stack Dishware Instruction: Put the bowl on the plate, then put the cup in the bowl. Rubric: 1. Pick up bowl 2. Place bowl on plate 3. Pick up cup 4. Place cup in bowl

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Store Marker Instruction: Open drawer, pick up marker, place marker in drawer, then close drawer. Rubric: 1. Open cabinet drawer 2. Pick up marker 3. Place marker in drawer 4. Close drawer

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Throw Away Trash Instruction: Open the trash can, pick up the cup, put the cup into the trash can, and then close the trash can. Rubric: 1. Open trash can 2. Pick up cup 3. Place cup inside trash can 4. Close trash can

Complete original figure with every panel, label and legend.
Selectable original task instruction and goal

Task: Pot on Stove Instruction: Put the pot on the stove Rubric: 1. Left arm grasps pot handle 2. Right arm grasps pot handle 3. Pot is lifted 4. Pot is placed on Stove

Appendix J Real-to-Sim Policy Evaluations

The evaluation protocol controls reset distributions, action interfaces and checkpoint identity while documenting simulator-specific reconstruction choices.

J.1 Evaluation Protocol

Each task uses twenty-five rollouts. Each object’s reset range is divided into a five-by-five grid; positions are sampled independently without replacement, with sampled rotations held fixed across checkpoints. Real/sim ranges match, but exact positions need not coincide, intentionally testing distributional rather than state-identical correspondence.

Complete original figure with every panel, label and legend.
Figure J.1: Stack Dishware Evaluation Grid. Task grid diagram detailing 25 initial starting positions of each object in Stack Dishware. The outer box represents the tabletop itself. Not to scale.

J.2 Metrics

Main scores are binary complete-task successes. Pearson correlation and mean maximum rank violation summarize their cross-policy relationship between reality and simulation.

Real-to-Sim evaluation metrics.

Pearson measures centered linear score agreement, while MMRV averages the largest real-score gap involved in a simulator-induced ranking inversion for each policy. Both exact formulas remain below. High correlation need not imply calibrated absolute success, and the finite set of evaluated policies defines the reported statistic.

Equations 1–2
ρ⁡(𝐱,𝐲)=∑i=1N(xi−x¯)​(yi−y¯)∑i=1N(xi−x¯)2​∑i=1N(yi−y¯)2,\rho(\mathbf{x},\mathbf{y})=\frac{\sum_{i=1}^{N}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{N}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{N}(y_{i}-\bar{y})^{2}}}, (1)
MMRV(𝐱,𝐲)=1N∑i=1Nmaxj∈{1,…,N}[|xi−xj|⋅𝟙(𝟙[yi<yj]≠𝟙[xi<xj])].\mathrm{MMRV}(\mathbf{x},\mathbf{y})=\frac{1}{N}\sum_{i=1}^{N}\max_{j\in\{1,\ldots,N\}}\left[|x_{i}-x_{j}|\cdot\mathbb{1}\left(\mathbb{1}[y_{i}<y_{j}]\neq\mathbb{1}[x_{i}<x_{j}]\right)\right]. (2)

J.3 PolaRiS Real-to-Sim Experiment Details

PolaRiS supplies a DROID-centered scene composer and Isaac Sim evaluation workflow. The comparison recreates the same tasks and tests existing policies without simulator-specific finetuning.

J.3.1 PolaRiS Custom Environment Creation

Background video is processed with COLMAP and 2D Gaussian Splatting to create a splat/mesh. Some reconstructed scenes reuse SimFoundry object assets, which are manually positioned in PolaRiS to match real initial conditions.

Complete original figure with every panel, label and legend.
Figure J.2: SimFoundry and PoLaRiS Scene Comparison. SimFoundry scene compared with a PolaRiS scene for the task Stack Dishware
Selectable original panel labels

(a) SimFoundry scene

(b) PolaRiS scene

J.3.2 PolaRiS Modification

The implementation is modified to load textures/poses correctly, adds an invisible tabletop collider to prevent objects rolling on an uneven mesh, and manually matches the external policy camera orientation.

J.3.3 Real-to-Sim Policy Evaluations in PolaRiS

PolaRiS runs the same task initializations, checkpoint families and rollout protocol, including difficult-task finetuned models. The comparison therefore evaluates these specific reconstructed/customized environments.

J.3.4 PolaRiS Results Analysis

PolaRiS underpredicts success in the tested scenes. Its released π0.5 checkpoint shallowly adapted with ten-percent simulation/ninety-percent DROID data performs better, but is excluded because the main comparison prohibits simulator-specific policy adaptation.

Appendix K Human Interaction

Optional human tools expose intermediate outputs for targeted corrections and scene refinement.

K.1 Human Intervention Details

The GUI permits object-extraction constraints and quick mesh pose/scale edits. Automation is a supported mode, while these corrections were used in the reported robotics workflow.

K.2 Interactive Pose Refinement

Pose refinement adjusts a similarity transform: three translations, three rotations and one isotropic scale.

Initialization from the automatic estimate.

Editing starts from the automatic estimate with the same depth/inpainted-image point cloud used for initial pose matching. Each object can be accepted unchanged or refined separately.

Overlay visualization.

The mesh overlays the scene cloud; toggling or thinning the displayed cloud exposes alignment errors without changing the estimated geometry.

Manual Adjustment

Keyboard commands change pose/scale with adjustable translation/rotation steps. Saving serializes the replacement transform for future runs.

Iterative refinement across sessions.

Each refinement pass writes a new output, preserving automatic and earlier manual versions for comparisons or rollback. The workflow supports repeated editing sessions rather than destructive replacement.

Use Cases.

Manual refinement is used primarily for real-to-sim evaluation alignment. The authors find it less useful for sim-to-real data generation when randomized object ranges already cover physical test conditions.

Complete original figure with every panel, label and legend.
Figure K.1: Interactive Scene Editor Procedure. The interactive scene editor launches the adjusted object mesh and the (inpainted) dense pointcloud for the scene, allowing the user to adjust object poses to align to the scene pointcloud. This process continues until all object poses have been adjusted. Note that in every step the GUI loads an inpainted pointcloud that erases previous objects to support occluded object pose tuning.
Selectable original table labels and values
Step 1 2 3
Object Adjust Baseball Box Mug
Full Scene Pointcloud
Object Mesh & PC Overlay

Appendix L System Analysis

System analysis separates foreground reconstruction, human refinement, throughput and background registration/fidelity.

L.1 3D Reconstruction Evaluation Details

Twelve YCB tabletop scenes are grouped by occlusion difficulty. Geometry is evaluated against staged quasi-ground truth, with SAM3D transformed into a shared frame and scale.

Complete original figure with every panel, label and legend.
Table L.1: Reconstructed Scenes. Categorized by Easy (No Occlusion), Mid (Slight Occlusion), and Hard (Strong Occlusion).
Selectable original table labels and values
Difficulty Input Scenes
Easy
Med
Hard

L.1.1 Quasi-Ground-Truth Scene Reconstruction

Objects are placed incrementally from back to front. FoundationPose estimates each pose while it is unoccluded using its known CAD mesh; these estimates become quasi-ground truth. Both reconstruction methods later see only the final occluded scene. The benchmark reference is itself estimated rather than independent precision metrology.

L.1.2 SAM3D Reconstruction Pipeline

SAM3D geometry/relative poses are converted to the camera convention and common world transform. A single scene-level scale is chosen by minimizing Chamfer distance against the ground-truth cloud. This resolves scale ambiguity using reference information rather than testing SAM3D’s raw metric-scale prediction.

L.1.3 Quantitative Reconstruction Results

Table L.2 reports Chamfer distance, F1 at a one-centimeter threshold and object bounding-box position error, with means and standard deviations. Automatic SimFoundry generally improves the aligned SAM3D baseline, although hard-scene Chamfer distance is slightly worse (.0091 versus .0088 meters); three-minute-per-object manual refinement improves its geometry further.

Table L.2: Scene Reconstruction Quantitative Results. Entries show Average ±\pm Standard Deviation.
Difficulty Metric SAM3D Zero Shot SimFoundry Zero Shot SimFoundry Tuned (3min/Obj)
Easy Chamfer Dist (m) ↓\downarrow 0.0081±0.00240.0081\pm 0.0024 0.0042±0.00130.0042\pm 0.0013 0.0026±0.00026\mathbf{0.0026\pm 0.00026}
F1 Score ↑\uparrow 0.71±0.150.71\pm 0.15 0.92±0.0710.92\pm 0.071 0.99±0.0069\mathbf{0.99\pm 0.0069}
Pos Error (m) ↓\downarrow 0.016±0.00580.016\pm 0.0058 0.0060±0.00190.0060\pm 0.0019 0.0041±0.00037\mathbf{0.0041\pm 0.00037}
Medium Chamfer Dist (m) ↓\downarrow 0.0087±0.00280.0087\pm 0.0028 0.0047±0.00120.0047\pm 0.0012 0.0033±0.00068\mathbf{0.0033\pm 0.00068}
F1 Score ↑\uparrow 0.66±0.180.66\pm 0.18 0.87±0.0890.87\pm 0.089 0.97±0.026\mathbf{0.97\pm 0.026}
Pos Error (m) ↓\downarrow 0.018±0.00670.018\pm 0.0067 0.0076±0.00380.0076\pm 0.0038 0.0057±0.0030\mathbf{0.0057\pm 0.0030}
Hard Chamfer Dist (m) ↓\downarrow 0.0088±0.00220.0088\pm 0.0022 0.0091±0.00760.0091\pm 0.0076 0.0039±0.0013\mathbf{0.0039\pm 0.0013}
F1 Score ↑\uparrow 0.68±0.140.68\pm 0.14 0.81±0.0710.81\pm 0.071 0.93±0.049\mathbf{0.93\pm 0.049}
Pos Error (m) ↓\downarrow 0.022±0.0100.022\pm 0.010 0.018±0.0180.018\pm 0.018 0.0073±0.0022\mathbf{0.0073\pm 0.0022}

L.1.4 Qualitative Reconstruction Results

Table L.3 preserves all twelve real, SAM3D, automatic SimFoundry and manually refined reconstruction comparisons across easy, medium and hard occlusion.

Complete original figure with every panel, label and legend.
Table L.3: Scene Reconstruction Qualitative Results. We compare SimFoundry against SAM3D across Easy, Medium, and Hard scenes.
Selectable original table labels and values
Scene
Name
Difficulty
Real Scene
Zero Shot
SimFoundry
Zero Shot
SimFoundry
Tuned (3min/obj)
Desk 1 Easy
Desk 2 Easy
Desk 3 Easy
Kitchen 1 Easy
Desk 4 Medium
Desk 6 Medium
Desk 8 Medium
Desk 9 Medium
Desk 5 Hard
Desk 7 Hard
Kitchen 2 Hard
Kitchen 3 Hard
Complete original figure with every panel, label and legend.
Table L.4: Additional real-to-sim reconstruction results. We show additional real-world input images, the corresponding reconstructed digital twins generated by SimFoundry, and sampled digital cousin scene variations. The first two columns report the number of reconstructed objects and the total wallclock time to reconstruct the scene. All reconstructions are run on a machine with an NVIDIA GeForce RTX 3090 GPU with 24GB VRAM.
Selectable original table labels and values
N
Objects
Time
(min)
Original
Recon.
Twin
Cousins
Image
2 19.98
4 34.07
9 41.36
10 42.76
10 49.02
10 51.01
10 51.39
10 58.71
11 67.13

L.2 Comparison between Manual and Automatic Background Pipeline

Five scenes compare backgrounds from automatic inpainting/registration with manually captured and aligned clean backgrounds.

L.2.1 Qualitative Reconstruction Results

The full visual comparison shows occasional automatic-pipeline floaters near support surfaces that occlude foreground objects. Inpainting may hallucinate hidden content even when overall registration metrics improve.

Complete original figure with every panel, label and legend.
Table L.5: Background reconstruction comparison. The results for the real scene alongside results from the manual and automated background pipelines for five dorm scenes.
Selectable original table labels and values
Scene
Real
Scene
Manual
Background Result
Automatic
Background Result
dorm_1
dorm_2
dorm_3
dorm_4
dorm_5

L.2.2 Quantitative Reconstruction Results

Seven image-space metrics are averaged over fifty uniformly sampled frames per scene and then across five scenes. The automatic path improves PSNR, SSIM, MAE, RMSE, normalized cross-correlation, CIE76 color difference and Sobel-edge MAE. Its shared-pose registration can explain much of the advantage over manual alignment; these metrics assess rendered-view agreement and do not isolate geometric surface accuracy from alignment. Every original metric value remains in Table L.6.

Table L.6: Render-vs-real Background Reconstruction Metrics. Render-vs-real agreement averaged over five scenes (dorm_1, dorm_2, dorm_3, dorm_4, dorm_5).
Variant PSNR↑\uparrow SSIM↑\uparrow MAE↓\downarrow RMSE↓\downarrow NCC↑\uparrow Δ​E↓\Delta E\downarrow EdgeMAE↓\downarrow
Manual Pipeline 12.91 0.497 0.1712 0.2294 0.549 20.13 0.0283
Automatic Pipeline 15.29 0.605 0.1275 0.1758 0.749 15.23 0.0248
Concept demonstration

Open image ↗