Literature
Display
← All papers

VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning

Tianxing Zhou · Feiyang Xue · Zhangchen Ye · Tianyuan Yuan · Hang Zhao · Tao Jiang

Preprint posted · arXiv (preprint) ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

VolumeDP lifts RGB features into a calibrated 3D voxel grid, compresses task-relevant regions into spatial tokens, and conditions a diffusion action decoder on the full token set. It reaches 88.8% average success on the reported LIBERO comparison and improves on tested baselines in ManiSkill, LIBERO-Plus, and four physical tasks. RGB-only refers to the visual input: the system also uses camera calibration, a chosen workspace volume, language, and proprioception. Gains are measured against the paper’s selected baselines, not all contemporary robot policies.

I Introduction

Mapping images directly to spatial actions can make imitation policies sensitive to viewpoint and object layout. VolumeDP introduces an explicit world-coordinate feature volume before action prediction, avoiding a depth sensor while still using geometric calibration. Learned token selection reduces the resulting feature count, and a multi-token decoder preserves interactions among selected regions. The reported improvements motivate this combination, but do not establish that RGB attention recovers uniquely correct geometry or eliminates viewpoint sensitivity.

Refer to caption
Fig. 1: VolumeDP consistently outperforms Diffusion Policy [2] and DiT-Block Policy [6] by significant margins across LIBERO, ManiSkill, LIBERO-plus simulation benchmarks and real-world manipulation tasks.

II Related Work

The design combines diffusion-based imitation with spatial representations developed for robot manipulation and visual perception.

II-A Imitation Learning in Manipulation

Diffusion Policy [2] models multimodal action sequences through conditional denoising. DiT-Block Policy [6] extends transformer diffusion conditioning to richer token inputs. VolumeDP retains diffusion action generation and uses the latter decoder design, while changing how the perceptual conditioning is represented and selected.

Refer to caption
Fig. 2: Architecture Overview. Our pipeline consists of three core components: (1) Volumetric Representation: Volume-Image Cross-Attention is applied to construct the Volumetric Representation from image. (2) Spatial Token Generation: Important features are extracted to form spatial tokens. (3) Multi-Token Decoder: The spatial tokens are utilized as conditions for the multi-token denoising decoder.

II-B 3D Representation in Manipulation

Point-cloud and voxel manipulation methods motivate reasoning in 3D, but commonly require measured depth. VolumeDP instead projects predefined world-grid locations into calibrated RGB images and learns their features. This removes the depth-input requirement while retaining assumptions about camera geometry and workspace extent; its features are learned spatial descriptors rather than a directly measured occupancy map.

III Methodology

The policy has three stages: associate image features with a 3D grid, aggregate selected grid regions into task-conditioned spatial tokens, and denoise an action sequence while attending to those tokens, language, and robot state.

III-A Volumetric Representation

A shared image backbone processes the current and preceding frames. A task- or suite-specific cuboid defines the world-coordinate voxel grid. Camera intrinsics and extrinsics project each voxel into the image; deformable cross-attention samples a few learned offsets around that projection and combines the corresponding features. This limits the cost of global voxel-to-pixel attention. Calibration and workspace bounds are therefore essential inputs to the spatial alignment, even though no depth image is used.

Equations 1–3
x=i×s+xminy=j×s+yminz=k×s+zminx=i\times s+x_{\text{min}}\quad y=j\times s+y_{\text{min}}\quad z=k\times s+z_{\text{min}} (1)
[u,v,1]⊤=𝐊𝐏​[x,y,z,1]⊤\begin{bmatrix}u,v,1\end{bmatrix}^{\top}=\mathbf{K}\mathbf{P}\begin{bmatrix}x,y,z,1\end{bmatrix}^{\top} (2)
f⁡(𝐐v,𝐅2​d)=DeformAttn​(𝐐v,P⁡(v),𝐅2​d)f(\mathbf{Q}_{v},\mathbf{F}_{2d})=\text{DeformAttn}\!\big(\mathbf{Q}_{v},\;P(v),\;\mathbf{F}_{2d}\big) (3)

III-B Spatial Token Generation

Inspired by TokenLearner [31], convolutional layers process volume features concatenated with the language embedding to predict normalized spatial weights. Weighted sums produce a compact token set. An auxiliary binary-cross-entropy objective encourages attention near the end effector and locations associated with gripper-state changes, using existing proprioception rather than manual segmentation labels. These regions are a task-relevance heuristic, not ground-truth object masks.

TABLE I: LIBERO [7] Results.We report the success rate for the four task suites in the LIBERO benchmark. * means the baseline is achieved by our implementation. The leading performance is highlighted in bold.
Method Libero-Spatial Libero-Object Libero-Goal Libero-Long Average
BC-Trans [32] 68.0±1.068.0\pm 1.0 41.8±1.941.8\pm 1.9 67.8±10.467.8\pm 10.4 15.8±2.515.8\pm 2.5 48.4
BC-VILT [32] 67.2±2.367.2\pm 2.3 43.0±3.943.0\pm 3.9 76.2±3.076.2\pm 3.0 6.5±0.96.5\pm 0.9 48.248.2
Diffusion Policy [2] 78.3±1.178.3\pm 1.1 92.5±0.792.5\pm 0.7 68.3±1.268.3\pm 1.2 50.5±1.350.5\pm 1.3 72.472.4
Dit-Block Policy* [6] 81.1±2.381.1\pm 2.3 89.8±0.589.8\pm 0.5 67.6±0.867.6\pm 0.8 57.6±4.057.6\pm 4.0 74.074.0
Ours 90.7±1.4\mathbf{90.7}\pm 1.4 95.9±0.7\mathbf{95.9}\pm 0.7 89.9±1.2\mathbf{89.9}\pm 1.2 78.7±1.9\mathbf{78.7}\pm 1.9 88.8+14.8\mathbf{88.8}_{{\color[rgb]{1,0,0}+14.8}}
TABLE II: ManiSkill [33] results. We report the success rate for the five tasks from ManiSkill. * means the baseline is achieved by our implementation. The leading performance is highlighted in bold.
Method Poke Cube Place Sphere Hook Cube Stack Pyramid Insert Peg Average
BC 26.5±11.326.5\pm 11.3 6.5±7.96.5\pm 7.9 2.0±2.02.0\pm 2.0 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0 7.0
Diffusion Policy* [2] 64.7±3.164.7\pm 3.1 28.7±6.128.7\pm 6.1 65.3±3.165.3\pm 3.1 47.3±4.647.3\pm 4.6 42.0±7.242.0\pm 7.2 49.649.6
Dit-Block Policy* [6] 66.0±5.366.0\pm 5.3 36.7±1.236.7\pm 1.2 31.0±4.231.0\pm 4.2 24.0±12.024.0\pm 12.0 36.7±5.036.7\pm 5.0 38.938.9
Ours 78.0±8.5\mathbf{78.0}\pm 8.5 54.7±1.2\mathbf{54.7}\pm 1.2 79.0±1.4\mathbf{79.0}\pm 1.4 57.3±8.3\mathbf{57.3}\pm 8.3 75.3±7.0\mathbf{75.3}\pm 7.0 68.9+19.3\mathbf{68.9}_{{\color[rgb]{1,0,0}+19.3}}
TABLE III: LIBERO-Plus [34] results. We report the success rate for the different out-of-distribution(OOD) settings in the LIBERO benchmark to showcase the generalization ability. * means the baseline is achieved by our implementation. The leading performance is highlighted in bold.
Method Camera Robot Language Light Background Noise Layout Average
Diffusion Policy* [2] 1.6 32.3 77.0 25.0 19.8 20.3 42.2 31.7
Dit-Block Policy* [6] 1.6 18.3 58.1 47.5 29.2 26.8 41.2 31.1
Ours 23.1\mathbf{23.1} 50.1\mathbf{50.1} 80.8\mathbf{80.8} 56.5\mathbf{56.5} 51.1\mathbf{51.1} 44.7\mathbf{44.7} 66.6\mathbf{66.6} 53.0+21.3\mathbf{53.0}_{{\color[rgb]{1,0,0}+21.3}}

III-C Multi-Token Decoder

A conditional diffusion transformer predicts an action sequence from noise, the spatial tokens, a proprioceptive token, and a language token. Timestep and condition features modulate the transformer through the adopted DiT-Block conditioning mechanism. Keeping the token set avoids collapsing all perceptual information into one descriptor. Training uses the standard diffusion noise-prediction objective; other token-injection mechanisms are mentioned but not evaluated here.

Equation 4
xk−1=α⁡(xk−γ​ϵθ​(xk,k,𝐅s,z,𝐅text))+𝒩⁡(0,σ2​I)x^{k-1}=\alpha\,\big(x^{k}-\gamma\,\epsilon_{\theta}(x^{k},k,\mathbf{F}_{s},z,\mathbf{F}_{\text{text}})\big)\;+\;\mathcal{N}(0,\sigma^{2}I) (4)

IV SIMULATION EXPERIMENT

Simulation evaluates imitation performance, distribution shifts, and architectural ablations. Success rates are averaged over three training seeds; the retained tables show per-task or per-suite results and their reported variability.

IV-A Experimental Setup

LIBERO supplies four ten-task suites with 50 demonstrations per task; failed demonstrations and no-op actions are removed. Five ManiSkill tasks use about 150 scripted demonstrations each. LIBERO-Plus tests policies trained on standard LIBERO under changes in camera, initial robot state, instruction, lighting, background, noise, and object layout.

Inputs are two 256×256 RGB frames, language, and an eight-dimensional proprioceptive state; outputs are seven-dimensional action sequences. A single third-person view is used in LIBERO and two ManiSkill tasks; the other ManiSkill tasks also use a wrist view. A 40×40×40 grid with 32 feature channels is reduced to 200 spatial tokens, then combined with language and state tokens. Workspace bounds are set per suite or task, and all compared methods use 100 denoising steps. The paper does not report an end-to-end inference-latency or calibration-error study.

IV-B Main Results

On LIBERO, VolumeDP averages 88.8% success versus 74.0% for the tested DiT-Block policy and 72.4% for Diffusion Policy: gains of 14.8 and 16.4 percentage points. On ManiSkill, 68.9% exceeds the strongest reported baseline’s 49.6% by 19.3 points. Some behavior-cloning results are taken from prior work, while starred diffusion baselines are the authors’ implementations; the tables preserve that distinction.

LIBERO-Plus success is 53.0%, compared with 31.7% and 31.1% for the two diffusion baselines. Improvement under perturbation is substantial but incomplete: camera-shift success is only 23.1%, although the baselines reach 1.6%. Thus the volume improves robustness within this evaluation rather than making the policy viewpoint invariant. The proposed explanations involving geometric alignment and task-focused selection are supported by ablations but are not separately established for every perturbation.

Refer to caption
Fig. 3: Visualization of Tasks from ManiSkill. The arrows indicate the sequential steps required to complete each task.
Refer to caption
Fig. 4: Weights Visualization. Active 3D voxels are projected onto the image plane; brighter red indicates higher activation. The learned weights concentrate on the end effector and the target object, indicating task-relevant spatial awareness.

IV-C Ablation Studies

On LIBERO-Spatial, replacing the volume with 2D features reduces success from 90.7% to 78.2%. Random voxel sampling, a coarser all-voxel grid, and ground-truth-depth occupancy selection reach 62.8%, 68.4%, and 75.8%; removing auxiliary attention supervision yields 82.2%. Collapsing tokens by concatenation or convolutions yields 76.4% and 80.0%. These results support the complete configuration in this suite. The all-voxel comparison also changes grid resolution, and ground-truth depth is used for a particular selection heuristic, so neither isolates a universal disadvantage of full volumes or depth sensing.

TABLE IV: Ablation results on Libero-Spatial. Ablating the core components of our pipeline. Replacing each of the parts will cause a performance drop, which demonstrates the efficiency of our model design.
Method
Volumetric
Representation
Spatial Token
Generation
Multi-Token
Decoder
SR
(a) ✓ ✓ ✓ 90.7
(b)
Image
Representation
✓ ✓ 78.2
(c) ✓ Random ✓ 62.8
(d) ✓ All ✓ 68.4
(e) ✓ GT Depth ✓ 75.8
(f) ✓ w/o supervision ✓ 82.2
(g) ✓ ✓ Concat 76.4
(h) ✓ ✓ Conv 80.0

V REAL WORLD EXPERIMENT

Physical evaluation compares VolumeDP with Diffusion Policy on four manipulation tasks and selected layout, view, and background shifts.

Refer to caption
(a) Real World Robot. We highlight the positions of the wrist-mounted and third-person cameras.
Refer to caption
(b) Real World Tasks. To evaluate our model, we run it on four real-world tasks: Placing bowl, Microwave operation, Opening Door and Nut-onto-Screw.
Fig. 5: Real World Setup. Left: the real-world robot. Right: the real-world tasks.

V-A Robot system and Tasks.

A Galaxea R1 Lite provides two six-degree-of-freedom arms, grippers, a head RGB camera, and wrist RGB cameras. Each task uses 100 expert teleoperated demonstrations. Bowl placement and microwave operation use the head view; door opening and nut placement also use a wrist view. The nut task requires placing a nut onto a screw, not demonstrating a complete threaded fastening operation.

V-B Analysis

In-distribution tests use 20 trials per task with object positions sampled within the demonstration region. VolumeDP achieves 85%, 75%, 85%, and 60% success on bowl, microwave, door, and nut tasks, averaging 76.3%, versus 57.5% for Diffusion Policy. The largest gain is on nut placement, but the small trial count limits precision.

Out-of-distribution tests move objects roughly 10 cm beyond the training region, rotate the camera by ±5 degrees horizontally or vertically, or change distractors and background surfaces. Table V reports VolumeDP success of 65%, 40%, and 45% for spatial, view, and environment shifts, versus 35%, zero, and 20% for the baseline. These are useful gains with substantial remaining failures; the paper’s stronger wording about insensitivity should be read against those numbers. The report does not fully specify how calibration is updated for camera perturbations.

TABLE V: Real-World Evaluation. Success rate (SR) over 20 trials on four real-world tasks. Results are reported for both in-distribution (ID) and out-of-distribution (OOD) settings.
Method In-distribution Out-of-distribution
Place
Bowl
Micro-
wave
Open
Door
Place
Nut
Avg Space View Env
Diffusion Policy 80.0 60.0 65.0 25.0 57.5 35.0 0.0 20.0
Ours 85.0 75.0 85.0 60.0 76.3+18.8 65.0 40.0 45.0

VI CONCLUSIONS

Explicit spatial association and learned token selection improve the tested RGB imitation policies while retaining a multi-token diffusion decoder. Practical dependencies include calibrated cameras, manually chosen workspace bounds, proprioceptive supervision, demonstrations, and a dense intermediate volume. The paper provides no separate limitations section; residual distribution-shift failures, limited physical trials, and unreported runtime and calibration sensitivity remain relevant boundaries of the evidence.

Concept demonstration

Open image ↗