Literature
Display
← All papers

ORV: 4D Occupancy-centric Robot Video Generation

Xiuyu Yang · Bohan Li · Shaocong Xu · Nan Wang · Chongjie Ye · Zhaoxi Chen · Minghan Qin · Yikang Ding · Zheng Zhu · Xin Jin · Hang Zhao · Hao Zhao

· CVPR 2026 ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete revised paper and supplementary material: occupancy-data construction, multiview alignment, simulation-to-real visual transfer, implementation, evaluation, limitations, and licensing. All 28 numbered figures, ten tables, three displayed equations, and three original listings are retained.

[Uncaptioned image]
Figure 1: We condition robot video generation on 4D semantic occupancy sequences and 7-DoF actions collected from real and simulated environments (through methods π\pi and π′\pi^{\prime}). This occupancy-centric conditioning enables faithful, controllable synthesis of single-view, multi-view, and sim-to-real manipulation videos. We also introduce ORV-Data, a curated 4D occupancy dataset for robot manipulation. Across benchmarks and downstream tasks, ORV improves video quality and control alignment, boosting visual planning and policy learning.

At a glance

ORV combines robot action sequences with rendered 4D semantic occupancy to generate manipulation video. A pretrained video diffusion model receives temporally aligned action chunks and coarse geometric guidance; extensions produce multiple views and visually transfer simulated trajectories. Experiments show improved video metrics and downstream planning and policy-learning averages. The full future occupancy sequence is supplied beforehand, so these results concern offline conditioned generation, not autonomous prediction of future geometry.

1 Introduction

Sparse end-effector controls leave ambiguity about how objects should move in image space. ORV adds scene geometry and semantics as visual guidance while retaining a pretrained video model’s appearance synthesis. Its contribution is the combination of action modulation, occupancy rendering, multiview attention, and a curated occupancy dataset. Because occupancy already describes the future scene, visual realism and action alignment should not be confused with independently learning all underlying physical dynamics.

Refer to caption
Figure 2: Establishment of non-interactive methods (π,π′)(\pi,\pi^{\prime}) both in the real-world environment and physical simulator to collect trajectory priors (7-DoF EE Pose) and visual priors (Occupancy).
Refer to caption
Figure 3: Overview of ORV framework. Centered on occupancy representation 𝒞\mathcal{C}, along with actions 𝒜\mathcal{A}, which are extracted from physical simulators (e.g., ManiSkill [27]) or real-world data (e.g., Bridge [93]), we leverage the soft visual priors to enable robot video generation with high visual quality and control alignment. Furthermore, we design a data curation pipeline to construct the robot occupancy data for training purposes. ORV, as a powerful neural simulator, can greatly boost downstream applications (e.g., policy learning, visual planning, etc).

2 Related Work

Related methods condition robot video on actions, language, flow, depth, or segmentation; driving models also use occupancy. ORV adapts the geometric-prior approach to manipulation and combines it with action controls. Compared with purely action-conditioned models, its strongest setting receives richer future information; comparisons therefore separate action-only and occupancy-plus-action variants.

3 ORV: Methodology

The method first specifies its offline conditioning inputs, then describes action and visual injection into a pretrained video transformer. Multiview and simulation-to-real variants reuse the occupancy representation, whose construction is a separate data-processing pipeline.

3.1 Problem Formulation

The generator maps context observations and complete future action/occupancy priors to future RGB video. Actions and occupancies are extracted from recorded trajectories or a physical simulator in one offline pass. The paper’s extraction symbols describe preparation of these priors, not an online policy interacting with generated observations. Occupancy forecasting is a possible future extension rather than a component demonstrated here.

3.2 Occupancy-centric Robot Video Generation

ORV fine-tunes CogVideoX-2B in two stages. Consecutive seven-dimensional actions are grouped to match the video encoder’s temporal compression, embedded, and injected through the existing vision-expert adaptive normalization. Occupancy voxels become nonlearned Gaussian splats; depth-dependent scales reduce rendering artifacts. An encoder and zero-initialized projector add the resulting coarse visual guidance to the initial noise. This avoids a second large expert and the cost or overconstraint of injecting imperfect geometry throughout the denoising layers. Related source: CogVideoX.

3.2.1 ORV-MV: Multiview Robot Video Generation

ORV-MV inserts attention across views before attention across time. Single-view blocks receive action and occupancy guidance; view blocks focus on cross-camera correspondences without action priors. Geometry rendered from calibrated camera positions encourages consistency, but generation still requires first-frame images from all requested views. Reconstructing geometry from one anchor view can leave unseen surfaces incomplete.

Refer to caption
Figure 4: Illustration of three modulations (Expert AdaLN) and injecting actions in our DiT block. And indicates the action paddings serving as the placeholders for reference frames, where ε\varepsilon encodes actions and α,β,γ\alpha,\beta,\gamma are modulation vectors. We use [⋅][\cdot] to indicate the dimensions for simplicity.
Refer to caption
Figure 5: Architecture of ORV-MV, which generates multiview robot manipulation videos with cross-view consistency.
Refer to caption
Figure 6: (a) Overview of Training Dataset Curation Pipeline, which consists of four steps: 1) semantics space construction, 2) occupancy construction, 3) equip occupancy with semantics, and 4) bullet-time occupancy-to-Gaussian renderings in practical usage. (b) Occupancy examples of BridgeData V2 (BridgeV2) [93] and RT-1 [13]. Better to zoom in. Refer to Supplementary Materials for more examples.

3.2.2 ORV-S2R: Bridge Simulation-to-Real Transfer

ORV-S2R renders appearance-agnostic priors from simulated manipulation and combines them with a generated realistic first frame. The intended transfer is visual: a simulator or recorded trajectory supplies the motion. Coarse occupancy reduces sensitivity to the mismatch between reconstructed real geometry and clean parametric meshes, but does not establish that physical robot dynamics transfer automatically.

3.3 4D Occupancy Dataset of Robot Manipulation

The data pipeline captions scenes, clusters roughly 150,000 labels, grounds objects, and tracks their masks. MonST3R reconstructs moving 3D points when depth is unavailable; neural surface reconstruction densifies them. Voxelization and majority voting assign semantic occupancy, and optical-flow consistency filters poor sequences. The resulting supervision is reconstructed rather than ground-truth complete geometry, with errors inherited from the component models. Related source: MonST3R.

Table 1: Evaluation results of Conditional Video Generation on three datasets. Top-1 performance within all variants and each type of model is represented with bold text and underlines.
Method BridgeData V2 [93] DROID [47] RT-1 [13]
PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow
Text-conditioned Generation Models
CogVideoX [119] 19.432 0.752 7.509 83.561 19.238 0.701 6.341 71.536 20.457 0.816 6.243 42.169
Action-conditioned Generation Models
AVID [80] - - - - - - - - 25.600 0.852 2.965* 24.200
HMA [99] 23.636 0.808 8.849 67.096 21.435 0.821 3.108 47.383 25.424 0.840 7.306 84.165
IRASim [136] 25.276 0.833 10.510 20.910 21.632 0.820 5.395 41.031 26.048 0.833 5.600 25.580
ORV (Ours) 25.631 0.873 3.821 17.682 22.034 0.838 4.921 37.094 27.086 0.863 4.210 20.031
Occupancy&Action-conditioned Generation Models
IRASim† [136] 27.352 0.862 9.413 22.503 22.005 0.827 7.892 44.309 27.213 0.847 5.311 42.130
ORV (Ours) 28.258 0.899 3.418 16.525 22.310 0.841 3.222 34.603 28.214 0.878 4.013 19.931
  • *

    FID Scores of AVID [80] have been computed not in evaluation mode according to the official codes and lead to incorrect results. Thus, we ignore it.

  • †\dagger

    We incorporate the same occupancy&action conditions to IRASim.

Table 2: Evaluation results of Visual Planning on VP2 [86] Benchmark. Top-1 performance across 8 tasks and the average success rate are highlighted accordingly. We provide the mean and standard deviation of the success rate (in %) on average over 3 runs. The best and second-best performances are represented with bold text and underlines, respectively.
Method Robosuite Push Flat Block Open Drawer Open Slide Blue Button Green Button Red Button Upright Block Avg. Success
Simulator 93.5±2.2 13.3±0.1 76.7±0.0 71.7±1.2 100.0±0.0 96.7±0.0 90.0±0.0 90.0±0.0 88.4 *
MCVD [92] 77.3±2.6 4.0±1.1 11.7±1.2 18.3±1.0 95.0±3.6 83.3±0.4 73.3±2.6 56.7±2.4 59.4 67.2
FitVid [5] 67.7±5.3 9.2±4.0 25.3±6.9 35.3±4.5 94.0±4.6 84.0±5.3 58.7±5.1 51.3±2.7 59.5 67.3
MaskViT [29] 82.6±2.3 4.0±3.9 4.0±4.5 8.7±5.7 94.7±2.0 64.0±4.3 24.0±7.5 62.2±8.6 48.6 55.0
iVideoGPT[113] 78.3±0.4 3.3±0.7 37.5±1.5 16.1±2.5 95.6±2.9 82.5±3.1 92.2±1.5 44.7±1.7 63.9 72.2
ORV (Ours) 81.4±1.7 6.1±2.0 40.5±1.1 19.9±3.4 96.7±2.5 85.6±3.0 93.2±1.9 44.8±1.4 66.0 74.7
  • *

    Values in this column are normalized by the simulator’s average success rate.

4 Experiments

Experiments examine visual prediction quality, visual planning, and policy learning. Separate ablations test how conditioning is injected, whether geometry and semantics help, how pretraining matters, and how coarse or fine condition maps transfer.

Refer to caption
Figure 7: Qualitative results of versatile Video Generation with full conditions. Given one-frame observation, ORV predict subsequent 15 frames on validation split of Bridge [93], DROID [47], RT-1 [13] datasets. Red boxes denotes the first frame input of the video generation; Orange boxes denotes the ground-truth of the subsequence frames.

4.1 Conditional Video Generation

Video experiments use BridgeData V2, DROID, and RT-1, with about 30,000 action-conditioning updates and another 20,000 for occupancy or multiview training. Table 1 distinguishes action-only baselines from occupancy-conditioned variants. Full ORV improves most reported metrics, including Bridge FVD of 16.525, but does not lead every metric; DROID FID favors HMA. AVID’s reported FID is excluded because its evaluation setting was incorrect. Multiview and visual-transfer demonstrations are qualitative, including a lighting mismatch already present in the source data. Related source: IRASim.

4.2 Visual Planning

Visual planning uses 5,000 Robosuite and 35,000 RoboDesk training trajectories. Although the prose says nine tasks, Table 2 contains eight task columns and averages three runs. ORV reaches 66.0% average success versus iVideoGPT’s 63.9%; simulator-normalized scores are 74.7 versus 72.2. The headline roughly 3.5% gain is relative to that baseline, not 3.5 percentage points of raw success. ORV leads four tasks and is second on four, with a large remaining gap from the physical simulator.

Table 3: Evaluation results on SimplerEnv-WidowX [56] across four manipulation tasks. “+Finetune” indicates the additional finetuning on domain-specific dataset; and “+ORV” indicates that we augment the finetuning dataset with ORV-synthesized data.
Method Spoon on Towel Carrot on Plate Stack Cube Eggplant in Basket Avg. Success
RoboVLM∗† [62] 18.6% 22.9% 8.1% 0.0% 12.4%
+Finetune∗ 27.6% 26.7% 12.1% 52.8% 29.8%
+ORV 32.2% 29.6% 15.7% 57.9% 33.9%
Δ\DeltaImprovement +4.6% +2.9% +3.6% +5.1% +4.1%
SpatialVLA∗† [77] 12.5% 20.8% 20.8% 58.3% 28.1%
+Finetune∗ 12.8% 26.1% 26.5% 79.3% 36.2%
+ORV 14.7% 28.4% 27.8% 83.0% 38.5%
Δ\DeltaImprovement +1.9% +2.3% +1.3% +3.7% +2.3%
  • *

    The results are reproduced locally for fully fair comparisons.

  • †\dagger

    Zero-shot performance (Pretraining).

4.3 Policy Learning

Existing Bridge demonstrations are appearance-randomized using generated initial frames and ORV rollouts. Fine-tuning datasets have equal total size, with synthetic examples comprising roughly one quarter of the augmented mixture. On four SimplerEnv-WidowX tasks, RoboVLM improves from 29.8% to 33.9% and SpatialVLA from 36.2% to 38.5%: gains of 4.1 and 2.3 percentage points, respectively. The larger 13.7% and approximately 6.4–6.5% claims are relative improvements. These are simulated policy evaluations of appearance augmentation, not proof of broad real-robot trajectory generalization. Related source: SIMPLER.

Table 4: Ablation results of Video Generation and Visual Planning on approaches of priors injection.
Variants PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow Success↑\uparrow
CogVideoX 19.432 0.752 7.509 83.561 -
Action Conditions
w/ Text Expert 20.424 0.772 4.104 23.586 52.9
No Chunks 24.813 0.850 3.793 19.944 70.6
Ours (base) 25.631 0.873 3.821 17.682 74.7
Occupancy Map Conditions
ControlNet 26.974 0.865 3.613 20.069 -
Ours (full) 28.258 0.899 3.418 16.525 -
Refer to caption
Figure 8: Illustrations of Appearance Randomization powered by ORV, generating diverse manipulation videos of four tasks. For each manipulation task in Bridge Data [93], we present three examples with distinct visual appearances, demonstrating that ORV generalizes well to varied context inputs and thereby alleviates the challenge of data collection of robot learning.
Table 5: Ablation results of Conditional Video Generation on occupancy conditioning resources and training strategies.
Variants Source PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow
Conditioning Resources
w/o cond. (base) - 25.631 0.873 3.821 17.682
w/ depth Fine 30.288 0.919 3.061 14.321
Coarse 28.031 0.896 4.522 18.548
w/ sem. Fine 28.896 0.901 3.259 16.171
Coarse 27.911 0.896 3.467 17.053
Full cond. Fine 30.431 0.920 2.998 14.301
Coarse 28.258 0.899 3.418 16.525
Training Strategies (w/o Occupancy Conditionings)
From scratch - 23.518 0.811 19.357 84.831
From CogVideoX2B - 25.631 0.873 3.821 17.682

4.4 Ablation Study and Analysis

Action chunks and reuse of the vision expert outperform the tested alternatives. Full coarse guidance improves Bridge PSNR from 25.631 to 28.258, while fully aligned fine guidance reaches 30.431; pretraining strongly improves distributional video metrics. Fine-trained models deteriorate sharply on coarse inputs: FVD rises from 14.321 to 132.685, whereas coarse training tolerates the reverse switch better. The multiview table also reports improvements, though its “with / without” caption appears reversed relative to the numerical order and accompanying interpretation. These tests support robustness to the examined condition mismatch, not arbitrary geometric errors.

Table 6: Ablation results of Multiview Video Generation on occupancy conditionings on BridgeData V2 [93] with 3 views. Numbers are reported as “with / without” visual priors.
Views PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow
View0 (anchor) 25.77 / 28.25 0.87 / 0.89 3.20 / 3.11 14.05 / 12.54
View1 23.04 / 25.87 0.79 / 0.85 3.31 / 3.18 16.36 / 13.67
View2 22.90 / 25.79 0.78 / 0.85 3.32 / 3.19 15.97 / 13.62
Table 7: Ablation results of zero-shot Conditional Video Generation on different occupancy conditioning resources.
Train Val PSNR↑\uparrow SSIM↑\uparrow FID↓\downarrow FVD↓\downarrow
Coarse Coarse 28.031 0.896 4.522 18.548
Coarse Fine 26.608 (-1.423) 0.872 (-0.024) 4.932 (+0.410) 24.134 (+5.586)
Fine Fine 30.288 0.919 3.061 14.321
Fine Coarse 19.048 (-11.240) 0.754 (-0.165) 22.893 (+19.832) 132.685 (+109.792)

5 Conclusion

The evidence supports occupancy-conditioned visual synthesis, multiview coherence, and useful appearance augmentation. Its practical scope remains bounded by the cost and availability of complete future geometric priors, initial view images, and the limited downstream evaluations.

6 Supplementary Video

The supplement refers to muted demonstration videos in an attached anonymous index.html page. The retrieved paper contains static filmstrips but no embedded video files or accessible attachment link; the original reference to that supplementary material is retained without treating unseen videos as reviewed.

7 Datasets Details

BridgeData V2 supplies 60,096 trajectories from 24 environments; raw 480×640 video is used for single-view work and 256×256 data for multiview work. DROID supplies roughly 76,000 trajectories with side and wrist cameras, depth, actions, and language. RT-1 uses Google Robot demonstrations; its count is described as roughly 120,000 in the main setup and over 130,000 here. Dataset versions and sampled clips, rather than those rounded collection totals, determine the actual training input.

8 ORV-MV Details (Section 3.2.1 )

MonST3R supplies dynamic points in an anchor-camera frame, while VGGT estimates camera geometry across views. Their reference depth maps are aligned by least-squares scale and shift fitting; implementation drops the shift and uses scale alone, assuming a common reference frame. Calibrated cameras then render occupancy from side views. This registration is approximate and cannot create geometry never observed by the anchor reconstruction. Related source: VGGT.

Equation 1
Solve:min⁡∑i∈𝒱α,β⁡(α​Di′+β−Di)2,\mathrm{Solve:}\min_{\alpha,\beta}\sum_{i\in\mathcal{V}}\left(\alpha D^{\prime}_{i}+\beta-D_{i}\right)^{2}, (1)
Refer to caption
Figure 9: Illustration of ORV aligning multiview cameras from VGGT [97] under the frame of MonST3R [125] to get the multiview conditioning sequences.

9 ORV-S2R Details (Section 3.2.2 )

ManiSkill scenes combine sampled object layouts with learned grasping policies to produce trajectories and occupancy. A depth-conditioned image generator supplies the realistic initial appearance, and ORV renders the subsequent sequence. This pipeline transfers simulated visual content; the paper explicitly distinguishes it from matching real-world policy dynamics. Different initial-image seeds can yield several appearances for one underlying trajectory.

Refer to caption
Figure 10: Illustration of Simulation-to-Real Generation of ORV.
Refer to caption
Figure 11: Example of transferring multiview poses from VGGT [97] to MonST3R [125]. The comparison of calibrated side views and the side views demonstrates the efficiency.

10 Implementation Details

The implementation appendix specifies data curation, occupancy rendering, model architecture, sequence sampling, augmentation, evaluation, and compute. The retained tables, equations, and listings provide the exact source settings and pseudocode.

10.1 Occupancy Dataset Curation (Section 3.3 )

Occupancy curation has two distinct choices: constructing a compact semantic label space and choosing spatial resolution. Rendering then smooths these discrete voxels into image-space conditions.

10.1.1 Data Construction

Scene captions are embedded with all-MiniLM-L6-v2 and clustered into 51 labels. Occupancy uses the paper’s stated 0.001-cubed voxel units, within a 0.4×0.4×0.4 canonical extent for Bridge and 0.4×0.4×0.6 for DROID and RT-1. These are canonical reconstruction units, not a claimed physical metric calibration. The original captioning prompt is retained.

List the main object classes in the image, with only one word for each class:

10.1.2 Rendering with Adaptive Scaling

Projected Gaussian area scales inversely with squared camera depth. ORV normalizes depth to the interval from one to two and sets Gaussian size to a base scale times normalized depth raised to an empirical exponent. Bridge uses exponent 3.7 and base 0.00023; DROID and RT-1 use 3.2 and 0.00047. Figure 12 shows the parameter sweep. These are dataset-tuned settings, not a learned renderer or universal perspective-correction guarantee.

Refer to caption
Figure 12: Rendering examples of different choices of k1,k2k_{1},k_{2} on BridgeData V2 [93]. The blue box marks the empirically chosen value in our implementation. Better to zoom in.
Table 8: Main hyperparameters of model architecture, where ∗ denotes those that are specialized in our model, while others keep the same as the CogVideoX-2B.
Hyperparameter Value
Model
input channels 32*
attention head dimension 64
number of attention heads 30
number of transformer blocks 30
output channels 16
patch size 2
text embedding dimension 4096
diffusion timestep embedding dimension 512
action embedding dimension 512*
conditioning dimension 1920*
positional encoding sin,cos
VAE
spatial compression ratio 8
temporal compression ratio 4
# self: the instance of the AdaLN method
# self.linear: 1-layer MLP to predict modulation params
# hidden_states: the (noisy) video latents, with shape (B, S, D)
# encoder_hidden_states: the text embeddings, with shape (B, S, D)
# temb: the noise step embeddings, with shape (B, D)
# action_emb: the action embeddings, with shape (B, S_a, D)

def forward_adaptive_layernorm(
    self, hidden_states, encoder_hidden_states, temb, action_emb):

    # Vision Expert AdaLN (timestep + action)
    embedding_dim = hidden_states.shape[-1]
    shift, scale, gate = torch.nn.functional.linear(
        self.silu(temb[:, None, :] + action_emb),
        self.linear.weight[: 3 * embedding_dim],
        self.linear.bias[: 3 * embedding_dim],
    ).chunk(3, dim=-1)

    # Text Expert AdaLN (only timestep)
    enc_shift, enc_scale, enc_gate = torch.nn.functional.linear(
        self.silu(temb),
        self.linear.weight[3 * embedding_dim :],
        self.linear.bias[3 * embedding_dim :],
    ).chunk(3, dim=-1)

    # Modulate Vision Hidden States
    num_patches = hidden_states.size(1) // action_emb.size(1)
    scale = scale.repeat_interleave(repeats=num_patches, dim=1)
    shift = shift.repeat_interleave(repeats=num_patches, dim=1)
    hidden_states = self.norm(hidden_states) * (1 + scale) + shift

    # Modulate Text Hidden States
    encoder_hidden_states = self.norm(encoder_hidden_states) * \
            (1 + enc_scale)[:, None, :] + enc_shift[:, None, :]
    ...
Listing 1: Part illustration of modulation used in ORV (in Python-like codes).

10.2 Video Generation Details (Section 3.2 )

Video generation adapts the pretrained transformer while preserving its temporal encoder conventions. Model details and sampling choices together determine action-to-latent alignment.

10.2.1 Model Details

The vision normalization expert combines timestep and action embeddings; the text expert retains timestep modulation. Depth and semantic latents are concatenated, fused with repeated noise channels, and projected back to the video-latent width. Separate positional encodings distinguish time and view. The encoder uses spatial compression eight and temporal compression four; an extra terminal video frame and padded actions satisfy its sequence-length constraints, and the decoded placeholder frame is discarded. Listing 1 preserves the actual modulation pseudocode and indentation.

Equation 2
zin=MLP⁡(zin+Concat⁡([c1,c2,…]))+zinz_{\text{in}}=\mathrm{MLP}(z_{\text{in}}+\mathrm{Concat}([c_{1},c_{2},\dots]))+z_{\text{in}} (2)
Table 9: Hyperparameters of data preprocessing for training and evaluations, where Δ​f1\Delta f_{1} represents the sample interval of frames within video samples, Δ​f2\Delta f_{2} represents the sample interval among video samples of split data (train, val).
frames raw size sample size latent size Δ​f1\Delta f_{1} Δ​f2\Delta f_{2}
BridgeV2 [93] 16 480×\times640 320×\times480 40×\times60 1 4, 16
DROID [47] 24 180×\times256 256×\times384 32×\times40 3 16, 72
RT-1 [13] 16 256×\times320 320×\times480 40×\times60 2 6, 16
Table 10: Distributions of multiview data of BridgeData V2 [93].
samples proportion(%)
n_view=1 89901 60.79
n_view=2 0 0.00
n_view=3 57978 39.21
total 147879 100.00
Refer to caption
Figure 13: Qualitative Results of ORV with full conditions. Red boxes denote the first frame input of the video generation; Orange boxes denote the ground-truth of the subsequence frames.
Equation 3
PE⁡(t,x,y)\displaystyle\mathrm{PE}(t,x,y) =PEt​(t)⊕PEs​(x,y)\displaystyle=\mathrm{PE}_{t}(t)\oplus\mathrm{PE}_{s}(x,y) →Frame Attn.\displaystyle\rightarrow\quad\text{Frame Attn.} (3)
PE⁡(v,x,y)\displaystyle\mathrm{PE}(v,x,y) =PEv​(v)⊕PEs​(x,y)\displaystyle=\mathrm{PE}_{v}(v)\oplus\mathrm{PE}_{s}(x,y) →View Attn.\displaystyle\rightarrow\quad\text{View Attn.}
Refer to caption
Figure 14: Ablation Results of Depth Condition Map. Without any physical controls, the robot gripper fails to act accurately aligned with the 3D action instructions, due to the accumulation of errors. While ours performs correctly, along with the entire sequence.
Refer to caption
Figure 15: Ablation Results of Semantics Condition Map. Without the guidance of our rendered semantics maps, the model fails to accurately predict the shape deformation of the knife during its motion, whereas ours produce outputs that align well with the real-world appearance.

10.2.2 Training Details

Clips are sampled with dataset-specific resolution, frame spacing, and segment intervals to balance token count and motion coverage. Bridge’s initial multiview pool has 147,879 clips, about 61% single-view and 39% three-view; single-view examples are subsampled to roughly 40% for training. Three-view examples are presented with either two or three views at equal probability. Multiview blocks copy the single-view weights initially. Table 9’s DROID dimensions differ from the dataset prose and do not all match the stated compression factors, so those source values are preserved without silent correction.

10.3 Policy Learning Details (Section 4.3 )

Policy augmentation changes the appearance of existing trajectories, then filters generated examples before fine-tuning pretrained policies. It does not add the trajectory- or viewpoint-diversification experiments proposed later.

10.3.1 Data Augmentation

For sampled Bridge trajectories, the image generator produces three initial appearances and ORV extends each under the original conditions. The appendix reports roughly 40,000 generated videos, with about 10% rejected by automated scoring; the main text describes roughly 30,000 added samples, without fully reconciling these counts. Captioning and quality filtering use the model named Qwen2.5-32B-Instruct and the retained prompts. The quality criteria cover clarity, physical realism, and plausibility; the published prompt is explicitly partial, and automated filtering is not an independent guarantee of physical correctness.

Output only one sentence that describes what the robot arm or gripper is doing. The sentence must strictly start with a verb, not with 'The robot arm' or any subject. Do not use 'is', 'I am', or 'The robot arm'. Only output the instruction in imperative form.
Listing 2: Text instruction used in Qwen2.5-32B-Instruct for video captioning.
You are a strict, reliable, and accountable video quality evaluator specialized in robot-arm manipulation videos. Follow the rules exactly. If you break them, you will be penalized.

ROLE:
- You are an impartial human-like evaluator.
- Behave like a careful human reviewer: inspect frames, identify problems, and assign fair scores.
- Do NOT hallucinate or guess unseen details.

INPUT:
- A video of robot manipulation (assume frames and timestamps are accessible).

TASK:
1. Inspect the entire video. Pay attention to serious defects like geometry collapse, object deformation, or impossible motion.
2. Score the video on each criterion (1-5 scale).
3. For each criterion, only output:
- numeric score (1..5)
- confidence (0.00-1.00)
4. Do NOT output justification for every criterion (to save tokens).
5. Instead, provide ONE short `"summary"` (only 1 sentence) describing the main issues.
6. Compute `"final_score"` as weighted average (weights below).

CRITERIA (score each 1..5):
A. clarity - sharpness, focus, absence of blur, no ghosting_artifacts.
B. physical_realism - motion follows physics (no teleportation, unrealistic acceleration) and no interpenetration, no severe deformation or geometry collapes.
C. overall_plausibility - temporal/spatial consistency, lighting stability, no sudden jumps.
Listing 3: Part of text instruction used in Qwen2.5-32B-Instruct for video evaluation.
Refer to caption
Figure 16: Additional examples of 4D semantic occupancy data (on BridgeData V2 [93]) used in ORV.
Refer to caption
Figure 17: Qualitative comparison of zero-shot conditional video generation under different occupancy conditioning sources (refer to Tab. 7). The label “A+B” denotes training on A and evaluating on B. “Fine” indicates condition maps that are accurately aligned with the ground truth at the pixel level, while “Coarse” refers to those derived from occupancy fields. The PSNR values are calculated against the ground truths.

10.3.2 VLA Post-Finetuning

Both policies start from cross-embodiment pretraining and then adapt to Bridge data. RoboVLM uses full fine-tuning, while SpatialVLA uses low-rank adaptation under their released training procedures. Keeping those model-specific recipes distinguishes this comparison from training new policies from scratch. Related source: SpatialVLA.

Refer to caption
Figure 18: Qualitative results of action-conditioned video generation, where varying input actions enable precise control over the gripper. Better to zoom in.
Refer to caption
Figure 19: Appearance & Trajectory Adaptation Results. Zoom in for better observations.

10.4 Evaluation Details

Video evaluation reports PSNR, SSIM, FID, and FVD on approximately 2,600 generated samples. Planning follows VP2 and policy evaluation follows SIMPLER. Image and video similarity measures quantify appearance or distributional agreement; they do not by themselves certify correct contact physics or downstream safety.

10.5 Computation Resources

Each experiment runs on eight H100 processors. A 30,000-update training stage takes roughly 35 hours, or 11.7 GPU-days, and evaluating roughly 3,000 examples takes about two hours on the same eight-device setup. Bridge occupancy curation occupies about 8 TB. The use of a pretrained backbone reduces training requirements but does not make the overall data pipeline inexpensive.

11 Additional Results

Additional results provide data examples, comparison filmstrips, control ablations, generalization illustrations, and successful policy rollouts. They complement the numerical tables without expanding every qualitative claim into a measured benchmark.

11.1 Curated Occupancy Data

Figure 16 shows additional temporally evolving semantic occupancy examples. These illustrate the reconstruction pipeline’s output rather than a separate quantitative audit of geometric or semantic accuracy.

11.2 Controllable Video Generation (Section 4.1 )

Comparisons show fewer distorted objects and temporal artifacts in selected ORV sequences than in IRASim or Cosmos examples. Depth guidance improves action alignment and semantic maps help preserve manipulated objects. Per-sample plots sort examples by the action-only score before overlaying full-conditioning scores, clarifying that the horizontal axis is an ordered evaluation index. Coarse-versus-fine filmstrips illustrate the distribution-shift result from Table 7.

Refer to caption
Figure 20: Qualitative Comparison Results of Multiview Videos Generation. With our from-reference-view rendered visual conditionings, generated videos under side views achieve better geometric consistency under other side views. Better to zoom in.
Original Figure 21: PSNR and SSIM plots in their original side-by-side arrangement.
Figure 21: Improvement curves of PSNR (left) and SSIM (right) metrics across ordered evaluation samples from BridgeData V2 [93].

11.2.1 Generalizations

The model can vary appearance and respond to modified or random external actions in the shown examples. The authors also state that ORV does not use task text to execute semantically meaningful manipulation: visual synthesis under supplied actions is distinct from choosing actions to accomplish a language goal.

11.2.2 Multiview Videos Generation

Geometry rendered from one anchor reconstruction reduces cross-view object deformation, especially when motion is large or a side camera differs substantially from the anchor. The example still depends on approximate, incomplete geometry and observed first frames. Improved coherence does not establish complete novel-view reconstruction from a single image.

11.2.3 Real-World Validation ( Zero-shot )

A recorded AgileX Piper rollout and its same-action generated counterpart provide a zero-shot hardware illustration. Piper is morphologically similar to the training WidowX platform. Figure 23 is a qualitative video comparison, with no reported multi-task success-rate study or executed closed-loop policy transfer.

11.2.4 Additional Qualitative Results

Figures 26–28 retain the additional uncurated single-view filmstrips. Each example pairs ground truth with generation from the context frame, allowing inspection of temporal appearance and manipulation artifacts beyond the highlighted comparisons.

11.3 Policy Learning (Section 4.3 )

Figure 24 shows four successful SpatialVLA executions after training with augmented data. These are illustrative successes; the full success-rate evidence remains the four-task aggregate comparison in Table 3.

Refer to caption
Figure 22: Comparisons between Cosmos [2] and ORV generated rollouts..
Refer to caption
Figure 23: Real-world rollout and generated rollout (zero-shot).

12 Discussions

The discussion distinguishes supplied geometric priors from future forecasting, visual data synthesis from policy evaluation, and demonstrated appearance variation from proposed trajectory or viewpoint augmentation.

Refer to caption
Figure 24: Successful examples of policy execution on four SimplerEnv-WidowX [56] tasks using our fine-tuned SpatialVLA [77] model.

12.1 Occupancy-centric Framework

Action coordinates constrain motion while coarse occupancy supplies scene-wide geometry without requiring exact pixel alignment. The foundation-model reconstruction pipeline is an important dependency. Unlike an autoregressive interactive model, ORV uses noncausal diffusion and receives all future conditions before generation. Online occupancy forecasting could reduce dependence on a physical simulator, but remains future work. Related source: iVideoGPT.

12.2 World Model for Robot Manipulation

The demonstrated world-model roles are conditioned video simulation and visually diversified training data. Generating new trajectories would still require a policy and dynamics source to supply those trajectories. Using ORV itself as a reliable, reproducible policy evaluator is explicitly outside the paper’s scope.

12.3 Manipulation Data Augmentation

Appearance augmentation is tested; trajectory augmentation and novel-view initial-image generation are proposed directions. New trajectories would require varied policy rollouts in a simulator, while new views need geometrically consistent initial images. These proposed extensions should not be counted as demonstrated improvements to the policy-training data.

Refer to caption
Figure 25: System of long-horizon manipulation data synthesis.

12.4 Limitations and Future Directions.

The main limits are reliance on complete future occupancy, a seven-dimensional end-effector action description that omits detailed joint motion, required initial observations for every view, and short generation horizons. Figure 25 sketches a future long-horizon system rather than evaluating it. The source’s Google-Robot/DROID example is inconsistent with its earlier dataset description: DROID uses Franka, while RT-1 uses the Google platform.

12.5 Social Impact

The authors discuss applications in robotics and media alongside possible misinformation and privacy misuse, and suggest provenance tracking and synthetic-content detection. These are proposed safeguards and broader-impact considerations, not measured properties of the generator.

Refer to caption
Figure 26: Additional Qualitative Results of ORV #1.
Refer to caption
Figure 27: Additional Qualitative Results of ORV #2.
Refer to caption
Figure 28: Additional Qualitative Results of ORV #3.

13 License

The paper lists component licenses separately rather than claiming one uniform license for the pipeline: dataset attribution terms, permissive code licenses, model-specific agreements, and noncommercial terms for some assets or pretrained weights. This section records the source’s license inventory; it does not establish that every combined use has the same permissions.

Concept demonstration

Open image ↗