Literature
Display
← All papers

DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

Nisarga Nilavadi · Ralf Römer · Moritz Reuss · Michael Krawez · Tobias Jülg · Angela P. Schoellig · Rudolf Lioutikov · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

DUET-DINO jointly predicts side-camera and wrist-camera features for goal-image planning over translation, rotation, and gripper actions. Cross-view attention combines global context with local gripper geometry. In simulation it reaches 92% reaching success, 72.5% angled-reaching success, and 60% full lift success; the separate physical angled-reaching result is 26.7%. Planning takes roughly 15–17 seconds per step in the reported setup, so the contribution is improved predictive planning rather than real-time reactive manipulation.

I Introduction

A static global view may miss small orientation or contact changes, while a wrist view can lose broader task context. The paper pairs the two views inside the learned dynamics model instead of merely adding independent view costs. It evaluates whether that coupling, combined with DINOv3 representations, improves full end-effector planning. Zero-shot here means planning toward supplied goal images without task-specific policy training, not learning dynamics without robot data.

Refer to caption
Fig. 1: Comparison of goal-conditioned 7-DoF action planning toward the target object (banana) between our proposed DUET-DINO world model and V-JEPA 2-AC. DUET-DINO progressively moves the end-effector toward the goal, while V-JEPA 2-AC fails to approach the target.

II Related Work

Related work separates video-generating world models from feature-predicting models and compares their uses in planning.

II-A World Models in Robotics

Pixel-generating models can simulate detailed appearance but add computational cost and may hallucinate physical events. Latent methods instead predict representations. DINO-WM [12] motivates planning with frozen visual features; DUET-DINO extends this direction to coupled external and wrist views. Its DINOv3 encoder [20] is pretrained and frozen: only attention and predictor modules are trained from scratch.

II-B Visual Planning with World Models

Planning can generate target images and infer actions, or directly optimize actions against latent goal distances. The latter commonly uses the cross-entropy method to refine sampled trajectories. DUET-DINO adopts this sampling approach with a cost informed by both cameras, aiming to improve rotational and gripper control as well as translation. Better predicted features help only insofar as their distance supplies a useful task objective.

III Methodology

The model encodes synchronized observations, exchanges information between views, predicts their action-conditioned evolution, and searches for actions that approach paired goal images.

Refer to caption
Fig. 2: Architecture and training of DUET-DINO. The pretrained encoder maps side- and wrist-view observations into latent space. Cross-attention blocks condition each view on complementary information from the other view in the latent space. The view-specific predictor heads then predict future latents conditioned on the action and end-effector state. The dual-view prediction loss ℒpred​(ϕ,ψ)\mathcal{L}_{\mathrm{pred}}(\phi,\psi) combines the teacher-forcing loss computed over all one-step predictions in the clip with an autoregressive loss over the KK-step rollout predictions.

III-A Problem Formulation

Inputs are a static side image, moving wrist image, robot state, and a seven-dimensional end-effector action. The action represents three translational changes, roll/pitch/yaw changes, and gripper-state change. The target is future feature maps in both views. This seven-dimensional action space is distinct from directly predicting the robot’s seven joint commands.

III-B DUET-DINO: Simultaneous Cross-View World Modeling

Each view is independently encoded by frozen DINOv3. Its patches query the other view through cross-attention, after which separate heads predict view-specific next features conditioned on robot state and the shared action. Joint training minimizes absolute feature errors for teacher-forced one-step predictions and a two-step autoregressive rollout. Separate heads preserve camera-specific dynamics, while the cross-attention supplies complementary context. All original model and loss equations are retained.

Equations 1–4
ztv=Eθ​(otv)∈ℝP×Dv∈{side,wrist}.z_{t}^{v}=E_{\theta}(o_{t}^{v})\in\mathbb{R}^{P\times D}\qquad v\in\{\mathrm{side},\mathrm{wrist}\}. (1)
z~tside\displaystyle\tilde{z}_{t}^{\mathrm{side}} =𝒞ϕside​(Q=ztside,K=ztwrist,V=ztwrist),\displaystyle=\mathcal{C}_{\phi_{\mathrm{side}}}\left(Q=z_{t}^{\mathrm{side}},K=z_{t}^{\mathrm{wrist}},V=z_{t}^{\mathrm{wrist}}\right), (2)
z~twrist\displaystyle\tilde{z}_{t}^{\mathrm{wrist}} =𝒞ϕwrist​(Q=ztwrist,K=ztside,V=ztside).\displaystyle=\mathcal{C}_{\phi_{\mathrm{wrist}}}\left(Q=z_{t}^{\mathrm{wrist}},K=z_{t}^{\mathrm{side}},V=z_{t}^{\mathrm{side}}\right).
z^t+1v=Pψv​(z~tv,at,st)v∈{side,wrist}.\hat{z}_{t+1}^{v}=P_{\psi_{v}}\left(\tilde{z}_{t}^{v},a_{t},s_{t}\right)\qquad v\in\{\mathrm{side},\mathrm{wrist}\}. (3)
ℒpred(ϕ,ψ)=𝔼(ot:t+T,at:t+T−1)∼𝒟∑v∈{side,wrist}[1T−1​∑t=2T‖z^tv−ztv‖1⏟ℒTFv+1K​∑k=2K+1‖z^k,ARv−zkv‖1⏟ℒARv].\begin{gathered}\mathcal{L}_{\mathrm{pred}}(\phi,\psi)=\mathbb{E}_{(o_{t:t+T},a_{t:t+T-1})\sim\mathcal{D}}\!\!\!\!\!\!\sum_{v\in\{\mathrm{side},\mathrm{wrist}\}}\\ \Bigg[\underbrace{\frac{1}{T-1}\sum_{t=2}^{T}\left\|\hat{z}_{t}^{v}-z_{t}^{v}\right\|_{1}}_{\mathcal{L}_{\mathrm{TF}}^{v}}+\underbrace{\frac{1}{K}\sum_{k=2}^{K+1}\left\|\hat{z}_{k,\mathrm{AR}}^{v}-z_{k}^{v}\right\|_{1}}_{\mathcal{L}_{\mathrm{AR}}^{v}}\Bigg].\end{gathered} (4)

III-C Visual Planning

The planner encodes paired goal images and rolls out sampled action sequences over a receding horizon. It adds the two terminal feature distances and divides by the corresponding zero-action rollout cost. A weighted top-candidate cross-entropy update gives more weight to low-cost sequences before executing and replanning. This normalized feature cost is an optimization proxy; it does not explicitly impose object-contact correctness or collision avoidance.

Equations 5–7
mina0:H−1J(zg,OPENz^H)s.t.z^τ+1=DUET-DINO(z^τ,aτ),τ=0,…,H−1,z^0=zt,\displaystyle\begin{split}\min_{a_{0:H-1}}\;\,J(z_{\text{g}},\,&\hat{z}_{H})\\ \text{s.t.}\quad\;\hat{z}_{\tau+1}&=\text{DUET-DINO}(\hat{z}_{\tau},a_{\tau}),\;\tau=0,\dots,H-1,\\ \hat{z}_{0}&=z_{t},\end{split} (5)
ℓsidei=∥z^t+Hside,i−zgside∥1,ℓwristi=∥z^t+Hwrist,i−zgwrist∥1.\begin{gathered}\ell^{\mathrm{side}}_{i}\!=\|\hat{z}_{t+H}^{\mathrm{side},i}-z_{\text{g}}^{\mathrm{side}}\|_{1},\;\;\ell^{\mathrm{wrist}}_{i}\!=\|\hat{z}_{t+H}^{\mathrm{wrist},i}-z_{\text{g}}^{\mathrm{wrist}}\|_{1}.\end{gathered} (6)
Ji​(zg,z^t+H)=ℓiside+ℓiwrist(ℓside+ℓwrist)0𝒥={Ji}i=1N.J_{i}(z_{\text{g}},\hat{z}_{t+H})=\frac{\ell_{i}^{\mathrm{side}}+\ell_{i}^{\mathrm{wrist}}}{(\ell^{\mathrm{side}}+\ell^{\mathrm{wrist}})_{0}}\qquad\mathcal{J}=\{J_{i}\}_{i=1}^{N}. (7)

IV Experiments

Experiments vary encoder family, camera combination, joint training, and attention, then test reaching, orientation, sequential lifting, visual changes, and limited hardware transfer.

Refer to caption
Fig. 3: Overview of the proposed simultaneous cross-view world modeling method, showing predicted dual-view latent rollouts, CEM action planning using predicted and goal latents, and the normalized dual-view planning cost J⁡(zg,z^T)J(z_{\mathrm{g}},\hat{z}_{T}).

IV-A Implementation and Training Details

The dynamics training set contains 62,877 filtered DROID [30] trajectories and 5,856 RoboArena [31] trajectories. Eight-frame synchronized clips are sampled at four frames per second. DINOv3 uses 224-pixel inputs and an 840-million-parameter encoder; the V-JEPA 2 comparison uses 256-pixel inputs and a billion-parameter encoder. Predictors train for 120,000 AdamW steps with batch size 256. DUET-DINO training costs approximately 5 × 10²⁰ floating-point operations across four GPUs. Encoder pretraining is outside this reported dynamics-training budget.

IV-B Experimental Setup

Comparisons include single views, independently trained paired views, jointly trained cross-view models, and the official V-JEPA 2-AC checkpoint. Simulation uses 800 candidate sequences, fifty weighted elite candidates, and two optimization iterations, with matched run seeds. Tasks run in RoboLab with a Franka/Robotiq setup and inverse-kinematics execution. Newly trained side-view models use the right camera; the official checkpoint was trained on left-view DROID data with a different training budget, so that comparison changes more than architecture.

Refer to caption
Fig. 4: Static right side-camera view showing the robot in its home pose and the target objects used in the reach tasks: (1) Coffee Pot, (2) Coffee Can, (3) Banana, (4) Juice Carton, (5) Yogurt Cup, (6) Apple, (7) Orange, (8) Bagel, (9) White Pitcher, and (10) Ceramic Mug.

IV-C Reach Task

Across one hundred simulated trials over ten targets, reaching succeeds within 5 cm and one hundred planning steps. DUET-DINO reaches 92% versus 79% for the DINO wrist model and 78% for independent dual views; final position error is 5.4 cm versus 13.1 and 9.8 cm. Final error includes all runs and is measured at the last step rather than only at first success. The system handles a corner target initially outside the wrist view. Dual-view planning takes about 15–17 seconds per step and is approximately twice as slow as comparable single-view planning.

TABLE I: Aggregate reach success rate (SR) and final position error (FPE) over 100100 evaluation runs per model across ten tasks.
Side Wrist Predictor SR (%) ↑\uparrow FPE (cm) ↓\downarrow
V-JEPA 2 Encoder
✓\checkmark – V-JEPA 2-AC∗ 55%55\% 18.618.6±\pm23.223.2
✓\checkmark – Single-view 20%20\% 34.434.4±\pm17.917.9
– ✓\checkmark Single-view 37%37\% 32.532.5±\pm26.626.6
✓\checkmark ✓\checkmark Independent 37%37\% 32.532.5±\pm24.424.4
✓\checkmark ✓\checkmark DUET-VJEPA 41%41\% 39.039.0±\pm38.638.6
DINOv3 Encoder
✓\checkmark – Single-view 18%18\% 28.028.0±\pm15.615.6
– ✓\checkmark Single-view 79%79\% 13.113.1±\pm20.320.3
✓\checkmark ✓\checkmark Independent 78%78\% 9.89.8±\pm13.813.8
✓\mathbf{\checkmark} ✓\mathbf{\checkmark} DUET-DINO 𝟗𝟐%\mathbf{92\%} 5.4\mathbf{5.4}±\pm7.4\mathbf{7.4}

IV-D Angled Reach Task

Angled reaching requires both 5 cm position and 12-degree orientation tolerance within one hundred steps. Across forty simulated runs, DUET-DINO achieves 72.5% success versus 62.5% for wrist-only and 55% for independent dual views, with lower final position and angular errors. Across one hundred runs varying backgrounds and distractors, success is 63%, versus 25% and 45% respectively. The visual-shift result is encouraging but still below the fixed-background score.

Physical evaluation uses three difficult poses and thirty total trials on a Franka Research 3 with side and wrist stereo cameras. DUET-DINO succeeds in 26.7% (eight trials), versus 16.7% for independent dual views and 3.3% for wrist-only. Its average position error is slightly worse than independent dual views, while orientation error improves. Hardware uses only five hundred candidates, one optimization iteration, and fifteen planning steps. A separate filter rejects collision or singularity actions and terminates after three consecutive unsafe proposals. These conditions and the lower success rate qualify the strong simulation headline.

TABLE II: Aggregate angled-reach success rate (SR), final position error (FPE), and final angular error (FAE) over 4040 evaluation runs per model across four tasks.
Side Wrist Predictor SR (%) ↑\uparrow FPE (cm) ↓\downarrow FAE (∘) ↓\downarrow
V-JEPA 2 Encoder
✓\checkmark – V-JEPA 2-AC∗ 2.5%2.5\% 3.63.6±\pm1.51.5 72.672.6±\pm32.232.2
✓\checkmark – Single-view 25.0%25.0\% 3.63.6±\pm1.61.6 51.951.9±\pm31.931.9
– ✓\checkmark Single-view 25.0%25.0\% 36.736.7±\pm22.522.5 67.967.9±\pm37.737.7
✓\checkmark ✓\checkmark Independent 25.0%25.0\% 18.418.4±\pm15.715.7 66.766.7±\pm36.836.8
✓\checkmark ✓\checkmark DUET-VJEPA 25.0%25.0\% 15.715.7±\pm20.620.6 69.769.7±\pm41.741.7
DINOv3 Encoder
✓\checkmark – Single-view 25.0%25.0\% 5.35.3±\pm1.51.5 67.167.1±\pm40.240.2
– ✓\checkmark Single-view 62.5%62.5\% 8.38.3±\pm7.27.2 47.447.4±\pm57.557.5
✓\checkmark ✓\checkmark Independent 55.0%55.0\% 3.23.2±\pm2.52.5 48.848.8±\pm54.454.4
✓\mathbf{\checkmark} ✓\mathbf{\checkmark} DUET-DINO 72.5%\mathbf{72.5\%} 1.9\mathbf{1.9}±\pm1.2\mathbf{1.2} 22.7\mathbf{22.7}±\pm32.2\mathbf{32.2}
TABLE III: Aggregate angled-reach success rate (SR), final position error (FPE), and final angular error (FAE) under visual distribution shifts over 100100 evaluation runs per model.
Side Wrist Predictor SR (%) ↑\uparrow FPE (cm) ↓\downarrow FAE (∘) ↓\downarrow
DINOv3 Encoder
– ✓\checkmark Single-view 25%25\% 15.015.0±\pm10.010.0 54.354.3±\pm42.442.4
✓\checkmark ✓\checkmark Independent 45%45\% 3.33.3±\pm2.42.4 47.347.3±\pm40.840.8
✓\mathbf{\checkmark} ✓\mathbf{\checkmark} DUET-DINO 𝟔𝟑%\mathbf{63\%} 2.6\mathbf{2.6}±\pm2.1\mathbf{2.1} 30.9\mathbf{30.9}±\pm36.5\mathbf{36.5}
TABLE IV: Aggregate angled-reach success rates (SR), final position errors (FPE), and final angular errors (FAE) over 3030 hardware evaluation runs across three tasks.
Side Wrist Predictor SR (%) ↑\uparrow FPE (cm) ↓\downarrow FAE (∘) ↓\downarrow
DINOv3 Encoder
– ✓\checkmark Single-view 3.3%3.3\% 24.924.9±\pm21.021.0 39.939.9±\pm31.531.5
✓\checkmark ✓\checkmark Independent 16.7%16.7\% 14.3\mathbf{14.3}±\pm17.2\mathbf{17.2} 40.040.0±\pm42.042.0
✓\mathbf{\checkmark} ✓\mathbf{\checkmark} DUET-DINO 26.7%\mathbf{26.7\%} 14.714.7±\pm17.117.1 28.0\mathbf{28.0}±\pm17.7\mathbf{17.7}

IV-E Angled Grasp and Angled Lift-to-Home

Sequential lifting allocates sixty angled-reaching steps, ten grasping steps, and sixty lift-to-home steps. It uses looser reaching tolerances of 15 cm and 20 degrees; lift success requires retaining the object within 20 cm of home. Across forty simulated runs, full lift success is 60% for DUET-DINO, 57.5% for independent dual views, and 20% for wrist-only. The improvement over independent views is one successful trial. Stage-reaching success is 75%, 60%, and 57.5%, respectively. These thresholds differ from the standalone angled-reaching benchmark.

TABLE V: Aggregate stage-wise success rates (SR) for angled-lift tasks over 4040 evaluation runs across four tasks.
Side Wrist Predictor Angled-Reach SR (%) ↑\uparrow Lift SR (%) ↑\uparrow
DINOv3 Encoder
– ✓\checkmark Single-view 57.5%57.5\% 20.0%20.0\%
✓\checkmark ✓\checkmark Independent 60.0%60.0\% 57.5%57.5\%
✓\mathbf{\checkmark} ✓\mathbf{\checkmark} DUET-DINO 75.0%\mathbf{75.0\%} 60.0%\mathbf{60.0\%}

IV-F Ablation

Removing cross-attention reduces reaching success to 81% and angled reaching to 42.5%, supporting learned view exchange. With randomized side-camera shifts on thirty banana-reaching runs, both DUET-DINO and independent DINO dual views achieve 66.7%, while side-only models fail. This ablation supports access to a wrist view but does not show an additional cross-attention advantage for that particular perturbation test.

A DROID-only model trained for 94,500 steps reaches 78% reaching success and the same 72.5% angled-reaching success. The full setup changes both data and training duration, so the reach improvement cannot be attributed solely to RoboArena. Orientation performance does not improve in this comparison.

Refer to caption
Fig. 5: Mean latent changes captured by frozen encoders, DINOv3 action-conditioned predictors, and V-JEPA 2 action-conditioned predictors over 30 runs of an angled-reach task.
Refer to caption
Fig. 6: Patch-wise latent distances for DINOv3 and V-JEPA 2 feature maps. Lower ℓ1\ell_{1} distances indicate greater similarity to the selected reference patch. Frames at tt and t+1t+1 are shown after resizing to the respective encoder inputs. Feature maps are visualized at 14×1414\times 14 for DINOv3 and 16×1616\times 16 for V-JEPA 2. DINOv3 produces 196196 patches of 12801280 dimensions, whereas V-JEPA 2 produces 256256 patches of 14081408 dimensions.

IV-G Embeddings Analysis

On the analyzed trajectory family, the frozen encoders show comparable magnitudes of temporal feature change. The DINO predictors track the temporal pattern but overestimate change, whereas the V-JEPA wrist predictor underestimates it. Patch-distance visualizations show more coherent object correspondences for DINOv3. These diagnostics point to action-conditioned prediction quality as well as encoder structure; they are evidence from the selected analyses rather than a universal ranking of representation families.

V Conclusion

Coupled side/wrist dynamics improve full end-effector planning in the tested tasks, particularly simulated orientation control. High sampling cost, low hardware success, paired goal-image requirements, and limited real task coverage remain material constraints. Policy-generated action proposals are proposed to reduce the search space, but faster real-time control is future work.

VI Acknowledgements

The paper credits hardware and evaluation assistance, support from the German robotics and research programs, and computing resources at the Erlangen National HPC Center.

Concept demonstration

Open image ↗