Literature
Display
← All papers

PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

Wenlong Huang · Yu-Wei Chao · Arsalan Mousavian · Ming-Yu Liu · Dieter Fox · Kaichun Mo · Li Fei-Fei

· CVPR 2026 ↗

Robotics15 min text · CondensedOriginal paper ↗
Contents

Overview

PointWorld predicts how observed scene points move when a robot follows a proposed geometric trajectory. Its shared point-flow representation supports several robot embodiments, while a sampling planner uses the predicted motion to select actions.

Complete original PDF figure S0.F1
Figure 1: PointWorld is a large pre-trained 3D world model that predicts full-scene 3D point flows from a static point cloud and an embodiment-agnostic description of robot actions, represented also as 3D point flows. We curate a large-scale 3D dynamics modeling dataset, spanning single-arm, bimanual, whole-body, mobile manipulation interactions in real and simulated domains. Through careful evaluations, we rigorously study the recipe for scaling up 3D world models. Pretrained on diverse data, a single model enables diverse manipulation behaviors on physical hardware, given only a single RGB-D image captured in the wild, without additional data or finetuning.

At a glance

A large point-cloud dynamics model learns from real DROID demonstrations and simulated BEHAVIOR-1K interactions, totaling roughly two million clips and 500 hours after curation. Modern point-cloud backbones, pretrained image features, movement-weighted robust losses and chunked prediction improve short-horizon flow accuracy. A pretrained model supports eight physical manipulation tasks with manually specified goals; these trials execute a single optimized plan without visual replanning.

1 Introduction

Represent both the environment and the robot’s intended motion in the same 3D space. RGB-D supplies partially observed scene geometry; known robot kinematics supplies fully specified action geometry. Predicting their interaction avoids encoding each embodiment through a different joint/action vector and focuses learning on motion rather than photorealistic appearance.

This edition covers all 38 pages of arXiv v1, dated 7 January 2026, with all seven authors verified. The catalog’s CVPR 2026 publication metadata is retained. The core evidence concerns one-second geometric prediction, transfer between the reported domains, and selected physical tasks, rather than unrestricted physical simulation or general autonomous task specification.

Complete original PDF figure S1.F2
Figure 2: Overview of PointWorld. Given calibrated RGB-D, robot joint-space actions, and a robot description file (URDF), we convert actions to robot flows and concatenate with scene to form a single point cloud serving as an embodiment-agnostic interaction geometry. Scene points are featurized with a frozen DINOv3 encoder, robot points with temporal embeddings, and a point cloud backbone predicts full-scene 3D point flows.
Complete original PDF figure S1.F3
Figure 3: Rich Supervision of 3D World Modeling for Physical Interactions, when conditioned on 3D robot point flows and partial observable RGB-D. The 3D world modeling objective enjoys dense pixel-level supervision while encoding a wide range of capabilities central to robotic manipulation. To predict full-scene evolution, the model needs to implicitly segment objects of interest, identify material property and/or articulation structure, perform implicit shape completion for contact reasoning, propagate robot-object interaction for object-object dynamics, and simultaneously considering the effects of gravity, encapsulated all in a single forward pass of the learned model.

2 Related Work

The representation sits between image-based world models and explicit physics simulators: it learns the motion of observed geometry without requiring a complete object/material decomposition.

World Modeling.

World models may predict pixels, latent features, meshes, radiance fields or particles under various action interfaces. PointWorld uses 3D point flows for both state and action, emphasizing contact geometry and allowing single-pass regression rather than iterative video generation.

Dynamics Models in Robotics.

Analytical simulators encode physical assumptions explicitly; learned dynamics models fit interactions from data and can support planning or policy learning. PointWorld extends particle-style learned dynamics, a family covered by the cited dynamics-model review, toward larger datasets and heterogeneous embodiments, without claiming exact physics laws.

2D and 3D Flows for Manipulation.

Prior flow methods provide representations for policies, goals, rewards or visual servoing. Here, FoundationStereo, VGGT and CoTracker3 construct 3D training supervision, while known robot geometry converts proposed actions to point trajectories.

3 Method

First learn action-conditioned scene motion; then optimize candidate robot trajectories using that learned forward model. The dynamics network itself neither chooses a task nor guarantees an executable robot trajectory.

3.1 3D World Modeling with PointWorld

A single forward pass predicts ten future steps spaced 0.1 seconds apart. This one-second chunk preserves temporal relationships while amortizing the point-cloud backbone across multiple future states.

State Representation.

Remove robot pixels using its URDF and current joints, then back-project the remaining calibrated RGB-D observations into scene points. Point identity is maintained within the imagined rollout. No online point tracker is required at inference, and the initial point set can change between observations. A static initial cloud does not encode initial object velocity.

Action Representation.

Sample surface points on robot links and propagate them through the proposed joint sequence using forward kinematics. Unlike partially visible scene points, the robot action geometry is known even behind an object. Most experiments retain only 300–500 gripper points; assuming the robot actually follows this forecast is distinct from modeling actuation/contact-induced tracking error.

Complete original PDF figure S3.F4
Figure 4: Movement Weighting and Uncertainty Regularization, where the robot releases and drops a yellow cloth. (Bottom Left) The movement weighting, used in the training objective, effectively biases the training towards scene points that are moving at each timestep, computed with the ground-truth flows. (Bottom Right) The uncertainty value, predicted by the model without any ground-truth, regularizes training to prevent overfitting to points that have unreliable ground-truth. Intriguingly, we observe that it also emerges to capture action-conditioned uncertainty arising from the object’s physical properties (e.g., larger variability along the edge of the cloth).

Dynamics Prediction.

Concatenate initial scene points with time-stacked robot points. A Point Transformer V3 backbone processes the cloud; a shared MLP predicts each scene point’s displacement at all ten steps. Frozen DINOv3 image features are projected onto scene points, and robot features carry temporal information. The roughly 0.1-second forward-pass figure excludes the complete iterative planning procedure.

Training Objective.

Most scene points remain still, so uniform regression can neglect contacts. A sigmoid of ground-truth displacement assigns movement weights, normalized across points and timesteps. Huber residuals and predicted log-variance reduce the influence of unreliable real-world tracks (Equation 1). Visibility/depth-invalid labels receive no supervision.

Movement weighting alone can emphasize noisy outliers. The uncertainty and robust-loss terms stabilize it; learned uncertainty is not a demonstrated calibration guarantee or a separation of physical randomness from annotation noise.

Equation 1
12​∑k,iH,NSwk,i⏟movementweight​(ρδ​(𝐏^t+k,i−𝐏t+k,i)⏟Huber losson 3D residual​e−sk,i⏟uncertaintyweight+sk,i⏟uncertaintyreg.),\tfrac{1}{2}\sum_{\begin{subarray}{c}k,i\end{subarray}}^{\begin{subarray}{c}H,N_{S}\end{subarray}}\underbrace{w_{k,i}}_{\begin{subarray}{c}\text{\scriptsize movement}\\ \text{\scriptsize weight}\end{subarray}}\Big(\underbrace{\rho_{\delta}\big(\mathbf{\hat{P}}_{t+k,i}-\mathbf{P}_{t+k,i}\big)}_{\begin{subarray}{c}\text{\scriptsize Huber loss}\\ \text{\scriptsize on 3D residual}\end{subarray}}\underbrace{e^{-s_{k,i}}}_{\begin{subarray}{c}\text{\scriptsize uncertainty}\\ \text{\scriptsize weight}\end{subarray}}+\underbrace{s_{k,i}}_{\begin{subarray}{c}\text{\scriptsize uncertainty}\\ \text{\scriptsize reg.}\end{subarray}}\Big), (1)

3.2 PointWorld for Robotic Manipulation

MPPI samples temporally correlated end-effector trajectories, converts them to robot flows, predicts the resulting scene, and updates a nominal trajectory using exponentiated negative costs. Task cost is mean squared distance between selected scene points and target positions; control penalties limit path length and poor reachability.

The formulation can support receding-horizon control, but Appendix A.6 explicitly says the reported robot trials do not replan. Goals are manually selected through a GUI; language-model goal generation is a proposed extension. No deployment demonstrations or task-specific model fine-tuning are used.

Equation 2
arg​min𝐄0:T\displaystyle\argmin_{\mathbf{E}_{0:T}} ∑k=1T[ctask​(𝐬k)+cctrl​(𝐄k)]\displaystyle\sum_{k=1}^{T}\Big[\,c_{\text{task}}\big(\mathbf{s}_{k}\big)\;+\;c_{\text{ctrl}}\big(\mathbf{E}_{k}\big)\,\Big] (2)
s.t.\displaystyle\text{s.t.} 𝐬1:T=ℱTθ(𝐬0,𝐚1:T),𝐄0=𝐄measured,\displaystyle\mathbf{s}_{1:T}\,=\,\mathcal{F}^{T}_{\theta}\!\big(\,\mathbf{s}_{0},\;\mathbf{a}_{1:T}\,\big),\;\mathbf{E}_{0}\,=\,\mathbf{E}_{\text{measured}},

4 Dataset Curation and Evaluation Protocol

The dataset combines reconstructed real interactions and exact simulator trajectories. Its approximately two million training trajectories are short, overlapping clips rather than two million independent complete demonstrations.

Complete original PDF figure S4.F5
Figure 5: 3D Annotation Quality and Comparisons. FS denotes FoundationStereo [9]; Dataset Extrinsics V1 and V2 are the two DROID extrinsics releases. (Top) Compared to DROID releases, our pipeline yields substantially higher-quality depth and camera pose calibration, resulting in more accurate robot mask overlays and better aligned point clouds (readers are encouraged to zoom in or check out the interactive visualization on the project website for details). (Bottom) We further compute depth reprojection loss (differences between analytical and observed depth of robot surface), and F1 scores of point cloud alignment. We observe purely leveraging existing models (FS, VGGT) are insufficient, and V2 extrinsics improve over V1 by filtering out scenes with poor point cloud alignment but result in significantly lower scene counts. In contrast, our annotation pipeline retains substantially more scenes below 0.100.10 depth-loss criterion and dominates all metrics.
Complete original PDF figure S4.F6
Figure 6: Unseen rollouts from a single pre-trained PointWorld across diverse domains, visualized with Viser [157]. Given RGB-D captures, PointWorld predicts 10-step point flows conditioned on robot flows. We show first prediction, last prediction, and last ground-truth. Green points in GT mark regions occluded during 2D point tracking, for which we observe model predictions are often more accurate because these points are not being supervised in model training. Due to grid downsampling (1.51.5cm) we apply to all point clouds, all 3D visualization is upsampled to image resolution for visual clarity via nearest neighbors. Interactive visualization at project website.
Complete original PDF figure S4.F7
Figure 7: Roadmap for Scaling 3D World Models, measured by ℓ2\ell_{2} error on moving scene points on DROID test set. Starting from an existing baseline [5], we progressively modernize the backbone, stabilize training objectives, leverage pre-trained features, and scale model size, yielding consistent gains in accuracy. Hatched bars indicate settings that are not adopted in the final model.

3D Annotation for Real-World Data.

DROID supplies single-arm teleoperation with two external cameras and a wrist camera. Replace sensor depth with FoundationStereo, initialize extrinsics with VGGT, refine them using known robot geometry, and lift CoTracker3 tracks into 3D. The pipeline recovers about 60% of DROID, nearly 200 hours, with visibly improved alignment.

Reported median extrinsic differences of 1.8 cm and 1.9° use the best 1% of original calibrations as a reference proxy, not independently measured ground truth. Figure 5 compares depth consistency and cross-view alignment; missing inputs can change the number of evaluated scenes.

Simulation (BEHAVIOR-1K).

BEHAVIOR-1K contributes Galaxea R1 Pro mobile bimanual interactions from about 1,100 hours before filtering. Replay simulator state to obtain exact link motion and scene flows. Filtering emphasizes interactions but also retains robot-motion/no-object-motion negatives, a detail more explicit in the appendix than the main text’s contact-only description.

Model Evaluation Protocol.

Report per-point, per-timestep Euclidean displacement error, especially on moving points. Simulation uses exact labels. DROID uses a separate expert trained on the evaluation split only to create shared confidence masks; evaluated dynamics models train on the training split. These are filtered-label errors, not measurements against a fully clean independent ground-truth sensor.

The main text says it keeps the top 80% of points by confidence, whereas Appendix A.5 rejects points below the 0.8 confidence quantile, which would ordinarily keep roughly 20% before voxel aggregation. This unresolved source discrepancy affects the metric’s population.

Interpretation of the Metric.

Errors average over one-second horizons and many points; small changes can alter rollout quality. The claimed standard error ≤10⁻⁵ m reflects this dense evaluation, not an uncertainty bound on physical task success. Points within a clip and overlapping clips are correlated. Mover error, static error and deployment success answer different questions.

5 Experiments

Controlled comparisons examine backbones, objectives, image features, model/data size, action representation, rollout strategy, view count and transfer. Physical planning is evaluated separately from prediction accuracy.

Backbone Params Mem. FLOPs Latency ℓ2\ell_{2} mov. ℓ2\ell_{2} stat.
❍ GBND 1.00​x1.00\mathrm{x} 1.00​x1.00\mathrm{x} 1.00​x1.00\mathrm{x} 13.4613.46 0.0390 0.0066
❍ PointNet 1.03​x1.03\mathrm{x} 0.34​x0.34\mathrm{x} 0.04​x0.04\mathrm{x} 5.935.93 0.0369 0.0084
❍ PointNet++ 1.07​x1.07\mathrm{x} 0.67​x0.67\mathrm{x} 0.06​x0.06\mathrm{x} 327.08327.08 0.0368 0.0073
❍ SparseConv 33.31​x33.31\mathrm{x} 7.18​x7.18\mathrm{x} 1.32​x1.32\mathrm{x} 17.7017.70 0.0396 0.0076
❍ Transformer 41.06​x41.06\mathrm{x} 0.31​x0.31\mathrm{x} 3.38​x3.38\mathrm{x} 30.4330.43 0.0339 0.0071
⚫ PTv3-50M 49.14​x49.14\mathrm{x} 0.30​x0.30\mathrm{x} 0.34​x0.34\mathrm{x} 59.6059.60 0.0331 0.0067
⚫ PTv3-132M 127.22​x127.22\mathrm{x} 0.69​x0.69\mathrm{x} 1.04​x1.04\mathrm{x} 69.6069.60 0.0324 0.0061
⚫ PTv3-411M 398.67​x398.67\mathrm{x} 1.89​x1.89\mathrm{x} 1.90​x1.90\mathrm{x} 102.47102.47 0.0315 0.0059
⚫ PTv3-1B 957.71​x\mathbf{957.71\mathrm{x}} 4.30​x4.30\mathrm{x} 3.57​x3.57\mathrm{x} 123.65123.65 0.0312 0.0056

Table 1: Backbone Comparisons. PTv3 [152] enables massive parameter scaling while retaining similar memory and efficient inference (latency in milliseconds, PTv3-1B ≈0.12\approx 0.12 s).
Complete original PDF figure S5.F8
Figure 8: Real-World Action Inference. PointWorld runs zero-shot with MPC for rigid, deformable, articulated, and tool-use tasks in the wild. Success rates are on top.
Complete original PDF figure S5.F9
Figure 9: Scaling Study. Scaling PointWorld in either data or model size yields roughly log-linear gains in prediction accuracy.

5.1 Scaling 3D World Models: A Roadmap

PTv3 scales to about 957 times the GBND baseline’s parameters with 4.30 times its memory in Table 1. The 1B variant has 123.65 ms latency and mover/static errors of 0.0312/0.0056 m. Larger parameter count alone is not a latency improvement over the smaller baseline.

Figure 7 progressively adds a modern backbone, robust objectives, pretrained image features and capacity. Movement weighting initially worsens error without stabilization. Multi-layer frozen DINOv3 features and larger models produce further gains; the roadmap is sequential, so it does not isolate every possible interaction between changes.

Complete original PDF figure S5.F10
Figure 10: Zero-Shot and Finetuned Generalization to Held-Out Real-World Scenes, where the robot transports a reflective glass bottle. PointWorld pre-trained on DROID or jointly on DROID and BEHAVIOR (D+B) are capable of zero-shot generalizing to unseen environment and motion from a held-out DROID lab’s scene, closing the gap to the specialist variant trained on that lab’s data. PointWorld pre-trained on only simulation data fail to generalize zero-shot. Further finetuning yields more accurate object trajectories of grasped objects.
In-Domain Cross-Domain Held-Out Real From
D→D\text{D}\rightarrow\text{D} B→B\text{B}\rightarrow\text{B} D→B\text{D}\rightarrow\text{B} B→D\text{B}\rightarrow\text{D} D→H\text{D}\rightarrow\text{H} B→H\text{B}\rightarrow\text{H} D+B→H\text{D{+}B}\rightarrow\text{H} Scratch
Zero-Shot ℓ2\ell_{2} mover↓\,\downarrow 0.0315 0.0087 0.1460 0.0558 0.0305 0.0531 0.0300 0.0293
ℓ2\ell_{2} static↓\,\downarrow 0.0059 0.0010 0.0050 0.0058 0.0049 0.0057 0.0063 0.0043
Finetuned ℓ2\ell_{2} mover↓\,\downarrow – – 0.0107 0.0378 0.0271 0.0299 0.0272 0.0293
ℓ2\ell_{2} static↓\,\downarrow – – 0.0003 0.0086 0.0040 0.0046 0.0040 0.0043
Table 2: Generalization of PointWorld across in-domain, cross-domain, held-out real environments under zero-shot and finetuned settings. D denotes DROID, B denotes B1K, H denotes held-out real-world scenes. “From Scratch” denotes specialist trained on the held-out lab’s data. Evaluations are done on unseen samples from the corresponding dataset. PointWorld generalizes within domains, zero-shot transfers to unseen real-world environments, surpasses specialists if finetuned with 20x fewer updates, and benefits from real-sim co-training.
Complete original PDF figure S5.F11
Figure 11: Action representations. Representing actions as point flows on grippers balances effective, efficient contact reasoning and enables positive transfer across heterogeneous embodiments.

5.2 Ablations

Action geometry: gripper-only flows outperform sparse whole-body flows, dense whole-body flows and low-dimensional joint/end-effector inputs across the tested domains. Dense robot clouds add computation and many non-contact points. Prediction: train/test chunks of ten steps drift less than teacher-forced, self-fed or short sliding-window alternatives and need fewer forward passes.

Views: randomizing one to three training views gives the strongest robustness across tested camera counts; added test cameras help. Figure 13 reports sub-centimeter mover errors for that experiment, which should not be conflated with Table 1’s approximately three-centimeter DROID mover errors. Scaling: varying 50M–1B parameters or 5–100% data separately produces approximately log-linear gains within the measured range, not a universal extrapolation law.

The dense whole-body baseline is described as 2,000 points in the prose but labeled 3,000 points in Figure 11; the original figure is retained without silently choosing between them.

5.3 Generalization and Transfer

Table 2 separates in-domain, cross-domain and held-out-lab tests. On the held-out lab, real-only and joint real/simulation pretraining yield zero-shot mover errors of 0.0305 and 0.0300 m, close to the from-scratch specialist’s 0.0293 m. Simulation-only pretraining gives 0.0531 m.

Fine-tuning uses 90% of the held-out lab data and evaluates on the remaining 10%. Its 1/20 training updates is a compute/update reduction, not evidence of twentyfold fewer training examples. Fine-tuned real/joint models reach 0.0271/0.0272 m, beating the specialist in this table; simulation-only reaches 0.0299 m. Real-to-simulation transfer improves more readily than simulation-to-real.

Complete original PDF figure S5.F12
Figure 12: Ablation on Chunked Prediction, where we study different rollout strategies in training and testing. Chunked rollouts at both training and inference time lead to significantly less drift than other baselines while amortizing compute with only a single forward pass of the model.
Complete original PDF figure S5.F13
Figure 13: Ablation on Partial Observability, where we train variants of PointWorld with varying number of cameras and evaluate them on all settings at test time. PointWorld is robust to different levels of partial observability and benefits from additional cameras in both training and inference. Training with randomized camera counts yields the best performance across all test settings.

5.4 Model-Based Planning with PointWorld

A Franka arm with an unseen fin-ray gripper plans from one RealSense capture and manually selected object targets. Figure 8 reports tissue-box push 7/10, book push 2/10, scarf fold 8/10, pillow placement 4/10, microwave opening 3/10, drawer closing 9/10, duster sweeping 6/10 and broom sweeping 6/10.

Those counts sum to 45/80 successes, calculated from the figure. Each task has only ten feasible initial configurations; there is no reported competing physical-controller baseline. Deformable and tool tasks start pre-grasped. The trials demonstrate varied transfer, while manual goals, one-shot execution and limited sample sizes constrain broader autonomy/reliability claims.

6 Conclusion

The contribution is a scalable geometric dynamics representation plus data curation and training practices that support transfer. Broader generalization remains limited by missing initial velocities, noisy perception, robot tracking assumptions and manual objectives. The paper promises code/data/checkpoints; their availability and usability are not independently evaluated here.

Acknowledgments

The authors acknowledge Stanford HAI, Schmidt Sciences and ONR support, plus colleagues who helped with discussion, data, experiments and feedback.

Appendix A Appendix

The appendix supplies limitations, data construction, training and evaluation details, physical-trial protocol and additional visual evidence.

Appendix Contents

All appendix subsections, eight numbered tables, thirteen displayed equation containers and the complete figure series are retained below.

A.1 Extended Discussions on Limitations

These limitations define the scope of the learned forward model and distinguish observed short-horizon transfer from general physical understanding.

Static Initial State.

The initial observation contains no motion history or object velocity. The formulation assumes a static initial scene; fully dynamic starting conditions would need tracked history or recurrent state.

Reward/Cost Specification for Action Inference.

Action optimization needs an explicit reward/cost. Reported tasks use manual point masks and goals. VLM-based goals, learned rewards and policies trained inside the model are future possibilities rather than tested substitutes here.

Fine-Scale Objects and Calibration Noise.

Depth/extrinsic error can be comparable to the width of pens or cables, confusing object and gripper tracks. Better calibration/annotation is needed for fine-scale contact supervision.

Correlation vs. Causation.

Training captures correlations among robot actions, motion and observed change. Exogenous influences are not disentangled as causal mechanisms; good conditional prediction does not prove causal identification.

Lack of Photometric Dynamics.

Predicted outputs move observed 3D points; they do not model lights turning on, screen changes or appearance evolution. A separate appearance/radiance model would be needed.

Rigid-Body Robot Assumption.

Forward kinematics treats the robot as rigid links. Compliance in a fin-ray gripper and other contact-induced deformation are not explicitly represented, even though the deployment gripper is compliant.

Actuation and Tracking Assumptions.

The network predicts scene consequences if the robot follows the supplied path. It does not jointly model whether motor limits, payload, contacts or tracking error will prevent that path from being realized.

Lack of Explicit Physics Priors.

No Newtonian/conservation constraints are imposed. Qualitative gravity, articulation and deformation behavior emerges from data, without a guarantee of physical consistency outside the training distribution.

A.2 DROID 3D Annotation Pipeline

Real-data preprocessing reconstructs geometry and trajectories from synchronized stereo views and recorded joints, regardless of demonstration success. Robot geometry includes the Robotiq 2F-85 gripper and custom camera mount; model inputs ultimately use the external views.

A.2.1 Depth Estimation

FoundationStereo estimates per-view metric depth. Clip to a trusted 0–4 m range and propagate validity masks. This differs from simply accepting the original sensor depth or assuming a near-field stereo cutoff.

A.2.2 Camera Pose Estimation

Estimate one rigid transform per external camera and scene. Use multi-view initialization, then optimize against visible robot geometry whose pose is known from joints.

Camera Pose Initialization.

VGGT estimates time-aligned wrist/external camera relations. Known wrist-to-gripper calibration and forward kinematics tie these to the robot base; averaging over valid wrist frames initializes the external transforms. The displayed formulas retain the original coordinate-frame directions.

Equations
TWt←B=TW←G​TGt←B.T_{W_{t}\leftarrow B}=T_{W\leftarrow G}\,T_{G_{t}\leftarrow B}.
TE0←B(t)=TE0←Wt​TWt←B,T_{E_{0}\leftarrow B}^{(t)}=T_{E_{0}\leftarrow W_{t}}\,T_{W_{t}\leftarrow B},
TCi←B=TCi←E0​TE0←B,TCi←E0=TE0←Ci−1,T_{C_{i}\leftarrow B}=T_{C_{i}\leftarrow E_{0}}\,T_{E_{0}\leftarrow B},\quad T_{C_{i}\leftarrow E_{0}}=T_{E_{0}\leftarrow C_{i}}^{-1},

Camera Pose Refinement.

Refine six-DOF external-camera updates using absolute robot-depth reprojection error. Run 100 steps at learning rate 10⁻³, trust robot depths of 0.3–2 m and require at least 2,000 valid robot pixels per camera/frame; discard frames that fail this requirement.

Equation
Lrobot-depth=1K​∑i,t,k|di,t,kobs−di,t,kpred|,L_{\text{robot-depth}}=\frac{1}{K}\sum_{i,t,k}\bigl|d^{\text{obs}}_{i,t,k}-d^{\text{pred}}_{i,t,k}\bigr|,

A.2.3 Benchmark Metrics for 3D Annotation

Two complementary consistency tests assess calibration/depth rather than full downstream motion-prediction accuracy.

Depth Reprojection.

Render the robot under candidate calibration and compare valid observed/rendered depths with L1 error. Average per frame with point weighting, then aggregate by scene. This uses known robot shape as an alignment reference.

Two-view F1 @ 5/20 mm.

Back-project both external views, remove robot/out-of-workspace points and measure symmetric nearest-neighbor F1 within 5 or 20 mm. Accumulate counts over frames and normalize point counts before scene aggregation. Scenes missing a depth/calibration combination are omitted for that combination.

A.2.4 Occlusion-Aware Tracking in 3D

Construct coherent short trajectories rather than tracking a point across an entire long demonstration.

Clip Filtering.

Use overlapping 16-frame windows with stride one. Retain gripper-state changes or sufficient end-effector motion: open-gripper thresholds are 0.005 m/0.10 rad; closed-gripper thresholds are 0.002 m/0.05 rad. These windows cover approximately one second.

Tracking.

Project workspace/non-robot masks into the images, track permitted pixels with CoTracker3 and back-project each frame through its depth/calibration. Keep visibility and depth-validity flags so unreliable occluded correspondences can be excluded from training.

Postprocessing.

Multi-scale DBSCAN uses radii 0.02/0.05 m and minimum cluster size five; discard points marked outliers in over 20% of frames. Estimate normals with up to 30 neighbors within 0.1 m, orient them toward the camera, then enforce temporal sign consistency.

A.3 BEHAVIOR-1K Data Generation

Approximately 10,000 BEHAVIOR episodes across 50 tasks average 6.6 minutes, giving about 1,100 hours before interaction filtering.

A.3.1 Dataset Replay

Replay recorded simulator states and actions, attaching virtual left/right shoulder and head cameras. Render RGB-D, normals and link segmentation; express extrinsics in the first-step robot-base frame. Reloading stored states avoids accumulating a free-running replay drift.

A.3.2 Clip Filtering

Extract overlapping eleven-frame clips, rejecting empty workspaces, simulator discontinuities and robot trunk/arm collisions. Equation 3 retains object motion with non-base joint motion, object motion with finger contact, or non-moving objects with gripper and joint motion. The last category supplies negative/free-space examples. These selection rules do not by themselves establish a causal model.

Equation 3
¬Ct∧((Mo∧Mj)∨(Mo∧Cf)∨(¬Mo∧Mg∧Mj)).\neg C_{t}\land\bigl((M_{o}\land M_{j})\lor(M_{o}\land C_{f})\lor(\neg M_{o}\land M_{g}\land M_{j})\bigr). (3)

A.3.3 3D Point Flows from Simulation

Back-project first-frame visible object points and attach them to simulator rigid links. Exact link poses propagate those points through time, with local coordinates and trajectories stored compactly. Privileged simulator state constructs labels; it is not directly given to the learned model as an extra state input.

Operation Description
Camera subsampling Sample randomized calibrated RGB-D views per timestep and concatenate their 3D points into a single scene cloud.
Bounds filtering Retain only scene points that stay within a workspace cube (approx. [−3,3]3[-3,3]^{3} m) for the entire clip, dropping particles that ever exit the bounds.
Centering Mean-center first-frame scene and robot points.
Image resize Downscale RGB-D images to 320×180320\times 180.
Voxel downsampling Voxel-grid sampling at 1.51.5 cm; select one point per occupied voxel at t=0t{=}0, and apply the same indices to all timesteps.
Multi-sphere cropping Iteratively remove spheres of points far from the robot (up to three spheres, radii in [0.10,0.80][0.10,0.80] m with buffer 0.250.25 m) until the scene falls below the target budget.
Max scene / robot points Randomly subsample scene points if more than 12 00012\,000 remain after cropping; robot points are capped at 500500 by construction.
Random yaw Uniform rotation about the vertical axis over [−π,π][-\pi,\pi].
Uniform scaling Isotropic scaling with factor sampled uniformly from [0.9,1.1][0.9,1.1].
Random reflection With probability 0.50.5, reflect the scene and robot across either the xx- or yy-axis.
Chromatic auto-contrast Apply chromatic auto-contrast to RGB channels with probability 0.20.2 and blend factor up to 0.20.2.
Chromatic translation Add a global RGB offset with magnitude 2% with probability 0.950.95.
Chromatic jitter Add per-point RGB noise with standard deviation 2% with probability 0.950.95.
Table 3: Data Preprocessing and Augmentations.
Point set Feature Definition
Robot Position pt,jrobotp^{\text{robot}}_{t,j} 3D coordinates of robot points over time.
Robot Color cjrobotc^{\text{robot}}_{j} Constant magenta color (1,0,1)(1,0,1) indicating robot identity, shared across timesteps.
Robot Normal nt,jrobotn^{\text{robot}}_{t,j} Surface normals of robot points from the known robot URDF.
Robot Gripper openness g~t\tilde{g}_{t} Scalar gripper open value per timestep, broadcast to all robot points.
Robot Velocity vt,jrobotv^{\text{robot}}_{t,j} Per-point velocity from mid-point finite differences over pt,jrobotp^{\text{robot}}_{t,j} across time.
Robot Acceleration at,jrobota^{\text{robot}}_{t,j} Per-point acceleration from mid-point finite differences over vt,jrobotv^{\text{robot}}_{t,j} across time.
Scene Position x0,ix_{0,i} 3D coordinates of scene points at the first frame after preprocessing.
Scene Color c0,iscenec^{\text{scene}}_{0,i} RGB color of scene points at the first frame.
Scene Normal n0,iscenen^{\text{scene}}_{0,i} (Estimated) surface normals of scene points at the first frame.
Scene Gripper openness sequence g0:T−1g_{0{:}T-1} Sequence of gripper openness values over the context and prediction horizon, broadcast to every scene point.
Scene Distance-to-robot d0:T−1,id_{0{:}T-1,i} For each timestep, distance from the first-frame position of scene point ii to the closest robot point, stacked across time.
Table 4: Per-Point Input Features.

A.4 Model Training Details

Tables 3–8 preserve complete preprocessing, feature and training/architecture configurations for the 1B and 411M variants.

Data Preprocessing and Augmentations.

Fuse calibrated views, filter to the workspace, mean-center and voxel-sample at 1.5 cm. Multi-sphere cropping caps scene points at 12,000 and robot points at 500; images are 320×180. Random yaw, scaling, reflection and color perturbations augment training. Evaluation disables augmentation and uses deterministic processing.

Per-Point Input Features.

Robot features include position, constant identity color, normals, gripper openness, velocity and acceleration. Finite differences assume zero velocity at both window boundaries. Scene features contain initial position/color/normal, the gripper sequence and distances from each initial scene point to the nearest future robot point.

Equations
ϕt,jrobot=[pt,jrobot,cjrobot,nt,jrobot,g~t,vt,jrobot,at,jrobot],\phi^{\text{robot}}_{t,j}=\bigl[p^{\text{robot}}_{t,j},\,c^{\text{robot}}_{j},\,n^{\text{robot}}_{t,j},\,\tilde{g}_{t},\,v^{\text{robot}}_{t,j},\,a^{\text{robot}}_{t,j}\bigr],
ϕiscene=[x0,i,c0,iscene,n0,iscene,g0:T−1,d0:T−1,i],\phi^{\text{scene}}_{i}=\bigl[x_{0,i},\,c^{\text{scene}}_{0,i},\,n^{\text{scene}}_{0,i},\,g_{0{:}T-1},\,d_{0{:}T-1,i}\bigr],
dt,i=minj∥x0,i−rt,j∥2,d0:T−1,i∈ℝT.d_{t,i}=\min_{j}\,\bigl\|x_{0,i}-r_{t,j}\bigr\|_{2},\qquad d_{0{:}T-1,i}\in\mathbb{R}^{T}.

3D Scene Featurization with DINOv3.

Project initial scene points into each camera, bilinearly sample multi-layer DINOv3 ViT-L/16 patch tokens and average across views passing visibility/depth-consistency checks. Map to 256 channels and fuse with separately projected raw features through normalization and a final linear layer. The image encoder remains frozen.

Equations
uc,i=[u~c,i(1)/u~c,i(3),u~c,i(2)/u~c,i(3)]⊤.u_{c,i}=\bigl[\tilde{u}_{c,i}^{(1)}/\tilde{u}_{c,i}^{(3)},\,\tilde{u}_{c,i}^{(2)}/\tilde{u}_{c,i}^{(3)}\bigr]^{\top}.
fi=1max⁡(1,∑cmc,i)​∑cmc,i​fc,i.f_{i}=\frac{1}{\max\bigl(1,\sum_{c}m_{c,i}\bigr)}\sum_{c}m_{c,i}\,f_{c,i}.

Visibility-Aware Supervision.

Real-data loss uses only visible, depth-valid correspondences. Simulation has exact trajectories, including states behind occlusions, so all scene points contribute. This is a supervision difference between domains, not evidence that real hidden geometry has been observed.

Training Configuration.

The 1B model trains for 300 epochs with batch 1,920 on 128 H100s for 20 days; the 411M experiment model uses 200 epochs, batch 176 and eight H100s for seven days. Both use AdamW at 10⁻⁴, weight decay 10⁻², gradient clipping at five and ten-step prediction. PDF Tables 5/6 describe 1B and 7/8 describe 411M; the HTML’s repeated Table 8 cross-references are conversion errors.

Aleatoric Uncertainty on Simulation Data.

Noise-free simulator residuals can drive predicted variance toward zero, exploding inverse-variance weights and destabilizing mixed training. Replace simulated log-variance with a batch constant matched to real-sample variance, or a fixed small value for simulation-only training. Learn heteroscedastic uncertainty normally on noisy real data.

Setting Value
Optimizer AdamW
Learning rate 1×10−41\times 10^{-4}
Epochs 300
Weight decay 10−210^{-2}
Global batch size 1920 sequences
Gradient clipping Global ℓ2\ell_{2} norm capped at 55
Loss Huber loss with δ=5.0\delta=5.0 with movement weighting and aleatoric uncertainty
Prediction horizon 10 steps
Training GPUs 128 NVIDIA H100 GPUs
Training time 20 days
Table 5: Training Configuration for PointWorld-1B.
Component Values
Grid size 1.51.5 cm
Encoder depth (4, 4, 8, 8, 12, 12, 4)(4,\ 4,\ 8,\ 8,\ 12,\ 12,\ 4)
Encoder channels (256, 384, 384, 512, 512, 768, 1024)(256,\ 384,\ 384,\ 512,\ 512,\ 768,\ 1024)
Encoder heads (8, 12, 12, 16, 16, 24, 32)(8,\ 12,\ 12,\ 16,\ 16,\ 24,\ 32)
Encoder stride (1, 2, 2, 2, 2, 2, 2)(1,\ 2,\ 2,\ 2,\ 2,\ 2,\ 2)
Encoder patch size (256, 256, 256, 256, 256, 256, 256)(256,\ 256,\ 256,\ 256,\ 256,\ 256,\ 256)
Decoder depth (4, 4, 4, 4, 4, 4)(4,\ 4,\ 4,\ 4,\ 4,\ 4)
Decoder channels (256, 384, 384, 512, 512, 768)(256,\ 384,\ 384,\ 512,\ 512,\ 768)
Decoder heads (8, 12, 12, 16, 16, 24)(8,\ 12,\ 12,\ 16,\ 16,\ 24)
Decoder patch size (256, 256, 256, 256, 256, 256)(256,\ 256,\ 256,\ 256,\ 256,\ 256)
Table 6: PointTransformerV3 (PTv3) Architecture for PointWorld-1B. Encoder and decoder configurations are ordered by stage.
Setting Value
Optimizer AdamW
Learning rate 1×10−41\times 10^{-4}
Epochs 200
Weight decay 10−210^{-2}
Global batch size 176 sequences
Gradient clipping Global ℓ2\ell_{2} norm capped at 55
Loss Huber loss with δ=5.0\delta=5.0 with movement weighting and aleatoric uncertainty
Prediction horizon 10 steps
Training GPUs 8 NVIDIA H100 GPUs
Training time 7 days
Table 7: Training Configuration for PointWorld-411M.
Component Values
Grid size 1.51.5 cm
Encoder depth (4, 4, 4, 8, 8, 12, 4)(4,\ 4,\ 4,\ 8,\ 8,\ 12,\ 4)
Encoder channels (256, 256, 256, 384, 384, 512, 768)(256,\ 256,\ 256,\ 384,\ 384,\ 512,\ 768)
Encoder heads (4, 4, 4, 8, 8, 16, 24)(4,\ 4,\ 4,\ 8,\ 8,\ 16,\ 24)
Encoder stride (1, 2, 2, 2, 2, 2, 2)(1,\ 2,\ 2,\ 2,\ 2,\ 2,\ 2)
Encoder patch size (256, 256, 256, 256, 256, 256, 256)(256,\ 256,\ 256,\ 256,\ 256,\ 256,\ 256)
Decoder depth (2, 2, 2, 2, 2, 2)(2,\ 2,\ 2,\ 2,\ 2,\ 2)
Decoder channels (256, 256, 256, 384, 384, 512)(256,\ 256,\ 256,\ 384,\ 384,\ 512)
Decoder heads (4, 4, 4, 8, 8, 16)(4,\ 4,\ 4,\ 8,\ 8,\ 16)
Decoder patch size (256, 256, 256, 256, 256, 256)(256,\ 256,\ 256,\ 256,\ 256,\ 256)
Table 8: PointTransformerV3 (PTv3) Architecture for PointWorld-411M. Encoder and decoder configurations are ordered by stage.

A.5 DROID Evaluation Protocol

Report mean Euclidean errors overall and for moved/static subsets. Real-data confidence filtering makes the denominator part of the evaluation protocol and must accompany the numerical results.

Expert Confidence Filtering.

A separate expert fitted only to the evaluation split predicts confidence used to mark unreliable ground-truth points. Cache low-confidence voxels so every tested model receives the same masks; no expert parameters are shared. The appendix’s 0.8 confidence-quantile cutoff conflicts with the main text’s top-80% wording.

Filtered Evaluation.

Map each tested scene point into world-coordinate voxels and exclude points/timesteps falling in the cached low-confidence regions. Spatial voxel grouping can change retention beyond a simple per-point quantile. Shared masks improve comparability across tested models but remain dependent on the expert’s filtering choices.

Mover/Static Splits.

Average Euclidean errors across valid points at each time, then across time. Mover/static metrics restrict that set using a small ground-truth displacement threshold, whose numerical value is not specified here.

Equation
ℓ2=1T​∑t1|Vt|​∑i∈Vtet,i,\ell_{2}=\frac{1}{T}\sum_{t}\frac{1}{|V_{t}|}\sum_{i\in V_{t}}e_{t,i},

A.6 Real-Robot Experiment Details

The seven-DOF Franka carries an unseen 3D-printed fin-ray gripper on a non-motorized wheeled base. A shoulder-mounted D435 provides RGB and stereo IR for FoundationStereo. Pose interpolation limits steps to 5 mm and 1°, with PyBullet IK and Deoxys joint-impedance tracking at 20 Hz.

A.6.1 Model-Based Planning

Optimize thirty future steps using three model chunks. MPPI uses four-knot cubic splines, 256 candidates per refinement, twenty refinements, noise scales 0.05–0.50 in normalized units, temperature 0.05 and nominal-update EMA 0.9. Planning takes a few seconds, depending on task/model. Warm-started real-time replanning is suggested, but not performed in these experiments.

A.6.2 Task Specification

A GUI uses SAM2 for manually selected object masks and world-frame targets. Add a mild end-effector proximity cost, path-length penalties and IK reachability residuals. Pre-grasp deformable objects and tools before evaluation; the experiment isolates subsequent dynamics rather than grasp acquisition.

A.6.3 Evaluation Protocol

For each of eight tasks, preselect ten random configurations verified as kinematically feasible. A human resets the scene and triggers execution. Meeting the objective counts as success; unsafe optimized plans count as failures. The source does not provide precise numerical tolerance boundaries for each task’s success judgment.

A.6.4 Effect of Training Mixture

Real-only training can predict conservative/static outcomes under contact; simulation-only training can move background points with the target. Joint training appears qualitatively more balanced. No controlled sweep of mixture proportions establishes an optimal real/simulation ratio.

A.7 Additional 3D Annotation Examples

Figures 14–15 retain eight additional side-by-side annotation comparisons, including RGB robot overlays, depth and point clouds. These illustrate improved geometric consistency, without supplying independent per-example calibration ground truth.

Complete original PDF figure Ax1.F14
Figure 14: DROID 3D annotations, including robot-overlaid RGBs, depths, point clouds, and comparisons to original dataset.
Complete original PDF figure Ax1.F15
Figure 15: DROID 3D annotations, including robot-overlaid RGBs, depths, point clouds, and comparisons to original dataset.

A.8 Additional Model Rollouts

Figures 16–21 retain eighteen complete unseen rollout examples with inputs, predictions and ground-truth frames. They cover cloth, grasping, gravity, glass, bimanual lifting, articulation, baskets and whole-body interaction. Green ground-truth points mark occlusion-related tracking gaps; nearest-neighbor upsampling improves display clarity without adding model spatial resolution. Selected visual examples complement, rather than replace, the filtered quantitative evaluation.

Complete original PDF figure Ax1.F16
Figure 16: DROID Unseen Rollouts, including deformable manipulation, robot-object interactions, and object-object interactions.
Complete original PDF figure Ax1.F17
Figure 17: DROID Unseen Rollouts, including deformable manipulation, and grasping behaviors.
Complete original PDF figure Ax1.F18
Figure 18: DROID Unseen Rollouts, including grasping behaviors, gravity effects, and glass objects.
Complete original PDF figure Ax1.F19
Figure 19: BEHAVIOR-1K Unseen Rollouts, including constrained bimanual lifting, gravity effects (dropped laptop), object-object interactions (laptop v.s. table), and articulated manipulation (fridge).
Complete original PDF figure Ax1.F20
Figure 20: BEHAVIOR-1K Unseen Rollouts, including object-object interactions (within basket), gravity effects (within basket), and whole-body behaviors.
Complete original PDF figure Ax1.F21
Figure 21: BEHAVIOR-1K Unseen Rollouts, including bimanual manipulation, whole-body behaviors, and implicit shape completion.
Concept demonstration

Open image ↗