At a glance
DUET-DINO jointly predicts side-camera and wrist-camera features for goal-image planning over translation, rotation, and gripper actions. Cross-view attention combines global context with local gripper geometry. In simulation it reaches 92% reaching success, 72.5% angled-reaching success, and 60% full lift success; the separate physical angled-reaching result is 26.7%. Planning takes roughly 15–17 seconds per step in the reported setup, so the contribution is improved predictive planning rather than real-time reactive manipulation.
I Introduction
A static global view may miss small orientation or contact changes, while a wrist view can lose broader task context. The paper pairs the two views inside the learned dynamics model instead of merely adding independent view costs. It evaluates whether that coupling, combined with DINOv3 representations, improves full end-effector planning. Zero-shot here means planning toward supplied goal images without task-specific policy training, not learning dynamics without robot data.
II Related Work
Related work separates video-generating world models from feature-predicting models and compares their uses in planning.
II-A World Models in Robotics
Pixel-generating models can simulate detailed appearance but add computational cost and may hallucinate physical events. Latent methods instead predict representations. DINO-WM [12] motivates planning with frozen visual features; DUET-DINO extends this direction to coupled external and wrist views. Its DINOv3 encoder [20] is pretrained and frozen: only attention and predictor modules are trained from scratch.
II-B Visual Planning with World Models
Planning can generate target images and infer actions, or directly optimize actions against latent goal distances. The latter commonly uses the cross-entropy method to refine sampled trajectories. DUET-DINO adopts this sampling approach with a cost informed by both cameras, aiming to improve rotational and gripper control as well as translation. Better predicted features help only insofar as their distance supplies a useful task objective.
III Methodology
The model encodes synchronized observations, exchanges information between views, predicts their action-conditioned evolution, and searches for actions that approach paired goal images.
III-A Problem Formulation
Inputs are a static side image, moving wrist image, robot state, and a seven-dimensional end-effector action. The action represents three translational changes, roll/pitch/yaw changes, and gripper-state change. The target is future feature maps in both views. This seven-dimensional action space is distinct from directly predicting the robot’s seven joint commands.
III-B DUET-DINO: Simultaneous Cross-View World Modeling
Each view is independently encoded by frozen DINOv3. Its patches query the other view through cross-attention, after which separate heads predict view-specific next features conditioned on robot state and the shared action. Joint training minimizes absolute feature errors for teacher-forced one-step predictions and a two-step autoregressive rollout. Separate heads preserve camera-specific dynamics, while the cross-attention supplies complementary context. All original model and loss equations are retained.
Equations 1–4
| (1) |
| (2) | ||||
| (3) |
| (4) |
III-C Visual Planning
The planner encodes paired goal images and rolls out sampled action sequences over a receding horizon. It adds the two terminal feature distances and divides by the corresponding zero-action rollout cost. A weighted top-candidate cross-entropy update gives more weight to low-cost sequences before executing and replanning. This normalized feature cost is an optimization proxy; it does not explicitly impose object-contact correctness or collision avoidance.
Equations 5–7
| (5) | ||||
| (6) |
| (7) |
IV Experiments
Experiments vary encoder family, camera combination, joint training, and attention, then test reaching, orientation, sequential lifting, visual changes, and limited hardware transfer.
IV-A Implementation and Training Details
The dynamics training set contains 62,877 filtered DROID [30] trajectories and 5,856 RoboArena [31] trajectories. Eight-frame synchronized clips are sampled at four frames per second. DINOv3 uses 224-pixel inputs and an 840-million-parameter encoder; the V-JEPA 2 comparison uses 256-pixel inputs and a billion-parameter encoder. Predictors train for 120,000 AdamW steps with batch size 256. DUET-DINO training costs approximately 5 × 10²⁰ floating-point operations across four GPUs. Encoder pretraining is outside this reported dynamics-training budget.
IV-B Experimental Setup
Comparisons include single views, independently trained paired views, jointly trained cross-view models, and the official V-JEPA 2-AC checkpoint. Simulation uses 800 candidate sequences, fifty weighted elite candidates, and two optimization iterations, with matched run seeds. Tasks run in RoboLab with a Franka/Robotiq setup and inverse-kinematics execution. Newly trained side-view models use the right camera; the official checkpoint was trained on left-view DROID data with a different training budget, so that comparison changes more than architecture.
IV-C Reach Task
Across one hundred simulated trials over ten targets, reaching succeeds within 5 cm and one hundred planning steps. DUET-DINO reaches 92% versus 79% for the DINO wrist model and 78% for independent dual views; final position error is 5.4 cm versus 13.1 and 9.8 cm. Final error includes all runs and is measured at the last step rather than only at first success. The system handles a corner target initially outside the wrist view. Dual-view planning takes about 15–17 seconds per step and is approximately twice as slow as comparable single-view planning.
| Side | Wrist | Predictor | SR (%) | FPE (cm) |
| V-JEPA 2 Encoder | ||||
| – | V-JEPA 2-AC∗ | |||
| – | Single-view | |||
| – | Single-view | |||
| Independent | ||||
| DUET-VJEPA | ||||
| DINOv3 Encoder | ||||
| – | Single-view | |||
| – | Single-view | |||
| Independent | ||||
| DUET-DINO | ||||
IV-D Angled Reach Task
Angled reaching requires both 5 cm position and 12-degree orientation tolerance within one hundred steps. Across forty simulated runs, DUET-DINO achieves 72.5% success versus 62.5% for wrist-only and 55% for independent dual views, with lower final position and angular errors. Across one hundred runs varying backgrounds and distractors, success is 63%, versus 25% and 45% respectively. The visual-shift result is encouraging but still below the fixed-background score.
Physical evaluation uses three difficult poses and thirty total trials on a Franka Research 3 with side and wrist stereo cameras. DUET-DINO succeeds in 26.7% (eight trials), versus 16.7% for independent dual views and 3.3% for wrist-only. Its average position error is slightly worse than independent dual views, while orientation error improves. Hardware uses only five hundred candidates, one optimization iteration, and fifteen planning steps. A separate filter rejects collision or singularity actions and terminates after three consecutive unsafe proposals. These conditions and the lower success rate qualify the strong simulation headline.
| Side | Wrist | Predictor | SR (%) | FPE (cm) | FAE (∘) |
| V-JEPA 2 Encoder | |||||
| – | V-JEPA 2-AC∗ | ||||
| – | Single-view | ||||
| – | Single-view | ||||
| Independent | |||||
| DUET-VJEPA | |||||
| DINOv3 Encoder | |||||
| – | Single-view | ||||
| – | Single-view | ||||
| Independent | |||||
| DUET-DINO | |||||
IV-E Angled Grasp and Angled Lift-to-Home
Sequential lifting allocates sixty angled-reaching steps, ten grasping steps, and sixty lift-to-home steps. It uses looser reaching tolerances of 15 cm and 20 degrees; lift success requires retaining the object within 20 cm of home. Across forty simulated runs, full lift success is 60% for DUET-DINO, 57.5% for independent dual views, and 20% for wrist-only. The improvement over independent views is one successful trial. Stage-reaching success is 75%, 60%, and 57.5%, respectively. These thresholds differ from the standalone angled-reaching benchmark.
IV-F Ablation
Removing cross-attention reduces reaching success to 81% and angled reaching to 42.5%, supporting learned view exchange. With randomized side-camera shifts on thirty banana-reaching runs, both DUET-DINO and independent DINO dual views achieve 66.7%, while side-only models fail. This ablation supports access to a wrist view but does not show an additional cross-attention advantage for that particular perturbation test.
A DROID-only model trained for 94,500 steps reaches 78% reaching success and the same 72.5% angled-reaching success. The full setup changes both data and training duration, so the reach improvement cannot be attributed solely to RoboArena. Orientation performance does not improve in this comparison.
IV-G Embeddings Analysis
On the analyzed trajectory family, the frozen encoders show comparable magnitudes of temporal feature change. The DINO predictors track the temporal pattern but overestimate change, whereas the V-JEPA wrist predictor underestimates it. Patch-distance visualizations show more coherent object correspondences for DINOv3. These diagnostics point to action-conditioned prediction quality as well as encoder structure; they are evidence from the selected analyses rather than a universal ranking of representation families.
V Conclusion
Coupled side/wrist dynamics improve full end-effector planning in the tested tasks, particularly simulated orientation control. High sampling cost, low hardware success, paired goal-image requirements, and limited real task coverage remain material constraints. Policy-generated action proposals are proposed to reduce the search space, but faster real-time control is future work.
VI Acknowledgements
The paper credits hardware and evaluation assistance, support from the German robotics and research programs, and computing resources at the Erlangen National HPC Center.