At a glance
VolumeDP lifts RGB features into a calibrated 3D voxel grid, compresses task-relevant regions into spatial tokens, and conditions a diffusion action decoder on the full token set. It reaches 88.8% average success on the reported LIBERO comparison and improves on tested baselines in ManiSkill, LIBERO-Plus, and four physical tasks. RGB-only refers to the visual input: the system also uses camera calibration, a chosen workspace volume, language, and proprioception. Gains are measured against the paper’s selected baselines, not all contemporary robot policies.
I Introduction
Mapping images directly to spatial actions can make imitation policies sensitive to viewpoint and object layout. VolumeDP introduces an explicit world-coordinate feature volume before action prediction, avoiding a depth sensor while still using geometric calibration. Learned token selection reduces the resulting feature count, and a multi-token decoder preserves interactions among selected regions. The reported improvements motivate this combination, but do not establish that RGB attention recovers uniquely correct geometry or eliminates viewpoint sensitivity.
II Related Work
The design combines diffusion-based imitation with spatial representations developed for robot manipulation and visual perception.
II-A Imitation Learning in Manipulation
Diffusion Policy [2] models multimodal action sequences through conditional denoising. DiT-Block Policy [6] extends transformer diffusion conditioning to richer token inputs. VolumeDP retains diffusion action generation and uses the latter decoder design, while changing how the perceptual conditioning is represented and selected.
II-B 3D Representation in Manipulation
Point-cloud and voxel manipulation methods motivate reasoning in 3D, but commonly require measured depth. VolumeDP instead projects predefined world-grid locations into calibrated RGB images and learns their features. This removes the depth-input requirement while retaining assumptions about camera geometry and workspace extent; its features are learned spatial descriptors rather than a directly measured occupancy map.
III Methodology
The policy has three stages: associate image features with a 3D grid, aggregate selected grid regions into task-conditioned spatial tokens, and denoise an action sequence while attending to those tokens, language, and robot state.
III-A Volumetric Representation
A shared image backbone processes the current and preceding frames. A task- or suite-specific cuboid defines the world-coordinate voxel grid. Camera intrinsics and extrinsics project each voxel into the image; deformable cross-attention samples a few learned offsets around that projection and combines the corresponding features. This limits the cost of global voxel-to-pixel attention. Calibration and workspace bounds are therefore essential inputs to the spatial alignment, even though no depth image is used.
Equations 1–3
| (1) |
| (2) |
| (3) |
III-B Spatial Token Generation
Inspired by TokenLearner [31], convolutional layers process volume features concatenated with the language embedding to predict normalized spatial weights. Weighted sums produce a compact token set. An auxiliary binary-cross-entropy objective encourages attention near the end effector and locations associated with gripper-state changes, using existing proprioception rather than manual segmentation labels. These regions are a task-relevance heuristic, not ground-truth object masks.
| Method | Libero-Spatial | Libero-Object | Libero-Goal | Libero-Long | Average |
| BC-Trans [32] | 48.4 | ||||
| BC-VILT [32] | |||||
| Diffusion Policy [2] | |||||
| Dit-Block Policy* [6] | |||||
| Ours |
| Method | Poke Cube | Place Sphere | Hook Cube | Stack Pyramid | Insert Peg | Average |
| BC | 7.0 | |||||
| Diffusion Policy* [2] | ||||||
| Dit-Block Policy* [6] | ||||||
| Ours |
| Method | Camera | Robot | Language | Light | Background | Noise | Layout | Average |
| Diffusion Policy* [2] | 1.6 | 32.3 | 77.0 | 25.0 | 19.8 | 20.3 | 42.2 | 31.7 |
| Dit-Block Policy* [6] | 1.6 | 18.3 | 58.1 | 47.5 | 29.2 | 26.8 | 41.2 | 31.1 |
| Ours |
III-C Multi-Token Decoder
A conditional diffusion transformer predicts an action sequence from noise, the spatial tokens, a proprioceptive token, and a language token. Timestep and condition features modulate the transformer through the adopted DiT-Block conditioning mechanism. Keeping the token set avoids collapsing all perceptual information into one descriptor. Training uses the standard diffusion noise-prediction objective; other token-injection mechanisms are mentioned but not evaluated here.
Equation 4
| (4) |
IV SIMULATION EXPERIMENT
Simulation evaluates imitation performance, distribution shifts, and architectural ablations. Success rates are averaged over three training seeds; the retained tables show per-task or per-suite results and their reported variability.
IV-A Experimental Setup
LIBERO supplies four ten-task suites with 50 demonstrations per task; failed demonstrations and no-op actions are removed. Five ManiSkill tasks use about 150 scripted demonstrations each. LIBERO-Plus tests policies trained on standard LIBERO under changes in camera, initial robot state, instruction, lighting, background, noise, and object layout.
Inputs are two 256×256 RGB frames, language, and an eight-dimensional proprioceptive state; outputs are seven-dimensional action sequences. A single third-person view is used in LIBERO and two ManiSkill tasks; the other ManiSkill tasks also use a wrist view. A 40×40×40 grid with 32 feature channels is reduced to 200 spatial tokens, then combined with language and state tokens. Workspace bounds are set per suite or task, and all compared methods use 100 denoising steps. The paper does not report an end-to-end inference-latency or calibration-error study.
IV-B Main Results
On LIBERO, VolumeDP averages 88.8% success versus 74.0% for the tested DiT-Block policy and 72.4% for Diffusion Policy: gains of 14.8 and 16.4 percentage points. On ManiSkill, 68.9% exceeds the strongest reported baseline’s 49.6% by 19.3 points. Some behavior-cloning results are taken from prior work, while starred diffusion baselines are the authors’ implementations; the tables preserve that distinction.
LIBERO-Plus success is 53.0%, compared with 31.7% and 31.1% for the two diffusion baselines. Improvement under perturbation is substantial but incomplete: camera-shift success is only 23.1%, although the baselines reach 1.6%. Thus the volume improves robustness within this evaluation rather than making the policy viewpoint invariant. The proposed explanations involving geometric alignment and task-focused selection are supported by ablations but are not separately established for every perturbation.
IV-C Ablation Studies
On LIBERO-Spatial, replacing the volume with 2D features reduces success from 90.7% to 78.2%. Random voxel sampling, a coarser all-voxel grid, and ground-truth-depth occupancy selection reach 62.8%, 68.4%, and 75.8%; removing auxiliary attention supervision yields 82.2%. Collapsing tokens by concatenation or convolutions yields 76.4% and 80.0%. These results support the complete configuration in this suite. The all-voxel comparison also changes grid resolution, and ground-truth depth is used for a particular selection heuristic, so neither isolates a universal disadvantage of full volumes or depth sensing.
| Method |
|
|
|
| |||||||
| (a) | ✓ | ✓ | ✓ | 90.7 | |||||||
| (b) |
|
✓ | ✓ | 78.2 | |||||||
| (c) | ✓ | Random | ✓ | 62.8 | |||||||
| (d) | ✓ | All | ✓ | 68.4 | |||||||
| (e) | ✓ | GT Depth | ✓ | 75.8 | |||||||
| (f) | ✓ | w/o supervision | ✓ | 82.2 | |||||||
| (g) | ✓ | ✓ | Concat | 76.4 | |||||||
| (h) | ✓ | ✓ | Conv | 80.0 |
V REAL WORLD EXPERIMENT
Physical evaluation compares VolumeDP with Diffusion Policy on four manipulation tasks and selected layout, view, and background shifts.
V-A Robot system and Tasks.
A Galaxea R1 Lite provides two six-degree-of-freedom arms, grippers, a head RGB camera, and wrist RGB cameras. Each task uses 100 expert teleoperated demonstrations. Bowl placement and microwave operation use the head view; door opening and nut placement also use a wrist view. The nut task requires placing a nut onto a screw, not demonstrating a complete threaded fastening operation.
V-B Analysis
In-distribution tests use 20 trials per task with object positions sampled within the demonstration region. VolumeDP achieves 85%, 75%, 85%, and 60% success on bowl, microwave, door, and nut tasks, averaging 76.3%, versus 57.5% for Diffusion Policy. The largest gain is on nut placement, but the small trial count limits precision.
Out-of-distribution tests move objects roughly 10 cm beyond the training region, rotate the camera by ±5 degrees horizontally or vertically, or change distractors and background surfaces. Table V reports VolumeDP success of 65%, 40%, and 45% for spatial, view, and environment shifts, versus 35%, zero, and 20% for the baseline. These are useful gains with substantial remaining failures; the paper’s stronger wording about insensitivity should be read against those numbers. The report does not fully specify how calibration is updated for camera perturbations.
| Method | In-distribution | Out-of-distribution | |||||||||||||
|
|
|
|
Avg | Space | View | Env | ||||||||
| Diffusion Policy | 80.0 | 60.0 | 65.0 | 25.0 | 57.5 | 35.0 | 0.0 | 20.0 | |||||||
| Ours | 85.0 | 75.0 | 85.0 | 60.0 | 76.3+18.8 | 65.0 | 40.0 | 45.0 | |||||||
VI CONCLUSIONS
Explicit spatial association and learned token selection improve the tested RGB imitation policies while retaining a multi-token diffusion decoder. Practical dependencies include calibrated cameras, manually chosen workspace bounds, proprioceptive supervision, demonstrations, and a dense intermediate volume. The paper provides no separate limitations section; residual distribution-shift failures, limited physical trials, and unreported runtime and calibration sensitivity remain relevant boundaries of the evidence.