Literature
Display
← All papers

Conditioning Matters: Training Diffusion Policies is Faster Than You Think

Zibin Dong · Yicheng Liu · Yinchuan Li · Hang Zhao · Jianye Hao

· NeurIPS 2025 ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

The opening figure links condition-sensitive hidden representations to faster benchmark learning. The complete model diagram, learning curves, rollout examples, five numbered tables, both theoretical results, all equations and original pseudocode are retained.

Refer to caption
Figure 1: Fusing generative condition into the source distribution greatly simplifies diffusion policy training. Diffusion policy trained with our method achieves π0\pi_{0} performance on the LIBERO benchmarks with only 30K gradient steps, which is 2.14x faster than the vanilla model. We also show the cosine similarity and the norm scale change between the policy hidden states before and after injecting condition information, demonstrating that our method fundamentally compels the policy network to utilize condition information, rather than simply embedding conditions into the source distribution.

At a glance

Cocos changes a diffusion policy’s initial noise distribution from a condition-independent Gaussian to one centered on compressed visual-language features. The policy still receives the usual conditioning inputs. On a compact flow-matching Transformer, the change raises LIBERO average success from 86.5% to 94.8% and MetaWorld from 59.5% to 74.8%, with favorable learning curves and real-robot results. The theoretical analysis identifies conditions under which gradients become similar, but does not establish that every conditional prior prevents collapse or guarantee global training convergence.

1 Introduction

A policy can generate plausible actions while using visual or language inputs poorly. The authors observe weaker changes in hidden states when conditions are injected into weaker policies, and call the tendency to fit marginal actions instead of condition-specific behavior “loss collapse.” These representation diagnostics correlate with success; they do not directly measure instruction comprehension or prove a single cause of every failure.

Cocos makes the source action-shaped noise itself depend on the condition, while keeping the existing policy conditioning pathway. Its contribution is an inexpensive change to the generative training setup, rather than a new large vision-language model. Experiments cover 40 LIBERO and 30 MetaWorld tasks plus ten tasks on each of two physical robots. Claims of architecture-wide compatibility extend beyond the one compact RDT recipe tested here.

Refer to caption
Figure 2: Diffusion Policy w/ Cocos. Our approach requires only replacing the standard Gaussian source distribution with a condition-conditioned Gaussian. An autoencoder compresses condition representations to match dimensionality, providing the mean while maintaining a fixed standard deviation.

2 Preliminaries

The policy imitates demonstrated action chunks conditioned on encoded images and language. Flow matching [12] learns a time-dependent velocity that transports an initial distribution into an action distribution. Conditional flow matching trains on tractable source-target interpolation paths; averaging their velocities with the appropriate density weights recovers the marginal transport field. The term “conditional” in this construction refers to the interpolation endpoints and should be distinguished from the robot’s observation/instruction condition.

Equations 1–3
ut​(x):=𝔼q⁡(z)​ut​(x|z)​pt​(x|z)pt​(x)u_{t}(x):=\mathbb{E}_{q(z)}\frac{u_{t}(x|z)p_{t}(x|z)}{p_{t}(x)} (1)
ℒCFM​(θ):=𝔼t,q⁡(z),pt​(x|z)​‖vθ​(t,x)−ut​(x|z)‖2.\mathcal{L}_{\text{CFM}}(\theta):=\mathbb{E}_{t,q(z),p_{t}(x|z)}\left\|v_{\theta}(t,x)-u_{t}(x|z)\right\|^{2}. (2)
pt​(x|z)=pt​(x|x1,x0),ut​(x|z)=ut​(x|x1,x0),p_{t}(x|z)=p_{t}(x|x_{1},x_{0}),~u_{t}(x|z)=u_{t}(x|x_{1},x_{0}), (3)

3 How Generative Conditions Can Lead to Loss Collapse

The analysis asks when a velocity network may fail to distinguish different observation or instruction conditions. Its exact assumptions and proof steps remain available with the original statements.

3.1 Flow with Generative Conditions

Introducing a generative condition yields a condition-specific probability path and target velocity. Lemma 1 relates squared error against the marginal conditional field to squared error against endpoint-conditioned fields, up to an additive parameter-independent constant, assuming positive densities and the regularity conditions supplied in Appendix A. Standard training samples a condition-action pair and independent Gaussian source noise, then predicts the interpolation velocity. Figure 3 illustrates indistinguishable left/right instructions; it is a conceptual mechanism illustration, not a separate quantitative experiment.

Equations 4–7
ut​(x|c):=𝔼q⁡(z)​ut​(x|z,c)​pt​(x|z,c)pt​(x|c).u_{t}(x|c):=\mathbb{E}_{q(z)}\frac{u_{t}(x|z,c)p_{t}(x|z,c)}{p_{t}(x|c)}. (4)
ℒCFMc​(θ):=𝔼t,q⁡(z),pt​(x|z,c)​‖vθ​(t,x,c)−ut​(x|z,c)‖2.\mathcal{L}_{\text{CFMc}}(\theta):=\mathbb{E}_{t,q(z),p_{t}(x|z,c)}\left\|v_{\theta}(t,x,c)-u_{t}(x|z,c)\right\|^{2}. (5)
pt​(x|z,c)={pt​(x|x1,x0),c∈z0,c∉z,and​ut​(x|z,c)={ut​(x|x1,x0),c∈z0,c∉zp_{t}(x|z,c)=\left\{\begin{array}[]{cc}p_{t}(x|x_{1},x_{0})&,~c\in z\\ 0&,~c\notin z\end{array}\right.~,~\text{and}~u_{t}(x|z,c)=\left\{\begin{array}[]{cc}u_{t}(x|x_{1},x_{0})&,~c\in z\\ 0&,~c\notin z\end{array}\right. (6)
ℒCFMc​(θ):=𝔼t,q⁡(x0),q⁡(x1,c),pt​(x|x1,x0)​‖vθ​(t,x,c)−ut​(x|x1,x0)‖2.\mathcal{L}_{\text{CFMc}}(\theta):=\mathbb{E}_{t,q(x_{0}),q(x_{1},c),p_{t}(x|x_{1},x_{0})}\left\|v_{\theta}(t,x,c)-u_{t}(x|x_{1},x_{0})\right\|^{2}. (7)
Refer to caption
Figure 3: Loss collapse causes the policy to degrade to an average. The policy omits language instructions (‘Move it to the left’ or ‘Move it to the right’) and yields actions based on their frequency in the training data. Cocos prevents loss collapse, enabling the policy to produce distinct actions corresponding to ‘Left’ and ‘Right’.

3.2 Loss Collapse

Theorem 1 assumes bounded network Jacobian M, bounded velocity residual D, outputs differing by at most ε across conditions, and an additional output-sensitive Lipschitz bound K on the Jacobian. Under a common sampling measure, the parameter-gradient difference is at most 2(M + KD)ε. This explains how already similar outputs and Jacobians can yield similar updates. It does not establish that all ordinary diffusion policies satisfy these strong assumptions, nor that gradient descent must converge to a condition-independent optimum.

Changing the source distribution adds a difference between sampling measures, so the same O(ε) contraction need not follow. That absence of a small upper bound is not a positive lower bound on useful conditioning: a conditional prior may still be uninformative or nearly identical across conditions. The proof’s “arbitrarily large” claim also needs qualification under its own uniform boundedness assumptions; Appendix A notes the issue explicitly.

Equations
μ⁡(d​y)=q⁡(x1,c)​q​(x0)​pt​(x|x1,x0)​d​x0​d​x1​d​x,\mu({\rm d}y)=q(x_{1},c)\,q(x_{0})\,p_{t}(x|x_{1},x_{0})\,{\rm d}x_{0}{\rm d}x_{1}{\rm d}x,
‖∇θℒCFMc​(θ,c1)−∇θℒCFMc​(θ,c2)‖≤ 2​(M+K​D)​ϵ,\bigl\|\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{1})-\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{2})\bigr\|\;\leq\;2\,(M+KD)\,\epsilon,
v∗​(t,x):=arg⁡minv​𝔼c∈𝒞​𝔼z∼μ​[‖v⁡(t,x)−ut​(x|x1,x0)‖2].v^{*}(t,x):=\arg\min_{v}~\mathbb{E}_{c\in\mathcal{C}}\mathbb{E}_{z\sim\mu}\left[\|v(t,x)-u_{t}(x|x_{1},x_{0})\|^{2}\right].
μc​(d​y)=q⁡(x1,c)​q​(x0|c)​pt​(x|x1,x0)​d​x0​d​x1​d​x,\mu_{c}({\rm d}y)=q(x_{1},c)\,q(x_{0}|c)\,p_{t}(x|x_{1},x_{0})\,{\rm d}x_{0}{\rm d}x_{1}{\rm d}x,

4 Method

Cocos samples source actions from a Gaussian with mean αFϕ(E(c)) and covariance β²I. The feature encoder maps frozen condition embeddings to the action dimensionality; α controls mean strength and β noise scale. Setting α = 0 and β = 1 recovers the standard prior. Both training and inference must sample from the same condition-dependent source.

A small encoder-decoder is trained to reconstruct condition embeddings using negative cosine similarity. The default first trains this autoencoder and then freezes it during policy training. A joint variant instead uses an exponential moving average of the encoder to slow changes to the source distribution. The policy follows a linear interpolation schedule with target velocity x₁ − x₀ and retains its ordinary cross-attention conditioning.

Algorithm 1 is preserved exactly. Its printed update uses θ ← θ + ∇θL without a negative sign or learning rate, despite a squared-error minimization objective; it should be treated as schematic pseudocode rather than executed literally. The theory motivates changed sampling but does not prove that reconstruction-trained features always encode the task distinctions that control requires.

Equations 8–10
ℒCocos​(θ):=𝔼t,q⁡(x0|c),q⁡(x1,c),pt​‖vθ−ut‖2,where​q​(x0|c)=𝒩⁡(x0,α​Fϕ​(ℰ⁡(c)),β2​I),\mathcal{L}_{\text{Cocos}}(\theta):=\mathbb{E}_{t,{\color[rgb]{1,0.23,0.13}q(x_{0}|c)},q(x_{1},c),p_{t}}\left\|v_{\theta}-u_{t}\right\|^{2},~\text{where}~q(x_{0}|c)=\mathcal{N}(x_{0};\alpha F_{\phi}(\mathcal{E}(c)),\beta^{2}I), (8)
ℒ⁡(ϕ)=−𝔼c​Sim​(Gϕ​(Fϕ​(ℰ⁡(c))),ℰ⁡(c)),\mathcal{L}(\phi)=-\mathbb{E}_{c}\text{Sim}\left(G_{\phi}(F_{\phi}(\mathcal{E}(c))),\mathcal{E}(c)\right), (9)
pt​(x|x1,x0)=𝒩⁡(x,t​x1+(1−t)​x0,σ2),ut​(x|x1,x0)=x1−x0.p_{t}(x|x_{1},x_{0})=\mathcal{N}(x;tx_{1}+(1-t)x_{0},\sigma^{2}),~~u_{t}(x|x_{1},x_{0})=x_{1}-x_{0}. (10)
Algorithm 1 Training and Inference Pseudocode for Diffusion Policy with Cocos
1: Policy Network vθv_{\theta}, Condition Network ℰ\mathcal{E}, Pre-trained condition encoder FϕF_{\phi}
2: while not coverge do # Training
3:    (x1,c)∼q⁡(x1,c),t∼𝒰⁡(0,1),(x_{1},c)\sim q(x_{1},c),~t\sim\mathcal{U}(0,1),~x0∼q⁡(x0|c)x_{0}\sim q(x_{0}|c)  Equation 8 # x0∼𝒩⁡(0,I)x_{0}\sim\mathcal{N}(0,I)  (without Cocos)
4:    θ←θ+∇θℒCFMc​(θ)\theta\leftarrow\theta+\nabla_{\theta}\mathcal{L_{\text{CFMc}}}(\theta)  Equation 7
5: end while
6: while not done do # Inference
7:    Observe c,c,~x0∼q⁡(x0|c)x_{0}\sim q(x_{0}|c)  Equation 8 # x0∼𝒩⁡(0,I)x_{0}\sim\mathcal{N}(0,I)  (without Cocos)
8:    Solve ODE d​xt=vθ​(t,xt,c)​d​t{\rm d}x_{t}=v_{\theta}(t,x_{t},c){\rm d}t from t=0t=0 to t=1t=1.
9:    a←x1a\leftarrow x_{1}; Execute action aa
10: end while

5 Experiments

The experiments test learning speed and final success, real-robot behavior, and qualitative conditioning failures. The section labels swap RQ2 and RQ3 relative to the initial question list; the experiment content is unambiguous.

5.1 Experimental Setup

The policy network is a roughly 40-million-parameter compact RDT [15]. Frozen DINOv2-Base [20] and T5-Base [25] encode vision and language; the source autoencoder has one Transformer layer in each encoder/decoder. Comparisons are DP-scratch without pretrained vision, DP-DINOv2 with ordinary source noise, and DP-DINOv2 with Cocos. The 40M count describes the policy backbone, not the total size including frozen encoders and source autoencoder.

LIBERO [13] evaluates four ten-task suites; MetaWorld [34] uses 30 selected tasks. SO100 and xArm each supply ten physical tasks. Simulation reports 150 evaluation trials per task, described as 50 trials at each of three random seeds. The text does not clearly identify independent training runs or define every ± statistic as standard deviation versus standard error.

5.2 Overall Comparison (RQ1)

At the reported endpoint, LIBERO averages are 64.4% for scratch, 86.5% for DINOv2 and 94.8% for default Cocos: an 8.3-percentage-point gain. Goal/Spatial/Object/Long success for Cocos is 93.8/97.5/99.1/88.7%. The learning curves improve early and the paper reports 2.14× faster convergence. This is a gradient-step comparison, not a measured 2.14× reduction in wall time, GPU-hours or total cost including autoencoder training.

MetaWorld averages improve from 59.5% to 74.8%, a 15.3-point or approximately 25.7% relative increase. Gains are not universal: disassemble falls from 72.7% to 39.3%, some button tasks decline slightly, and several tasks tie. The complete task-level table is preserved so the aggregate does not conceal these differences.

Curves compare against reported fine-tuned π₀ [1], OpenVLA-OFT [10] and Seer [29] scores. Cocos reaches similar average performance within 30K policy updates; cited fine-tuning budgets are 50–150K for OpenVLA-OFT and 100–250K for π₀. Different pretrained models, data filtering, batch sizes and total pretraining/auxiliary-training costs prevent interpreting this as a controlled compute-efficiency ranking. Lower hidden-state cosine similarity and larger norm changes support greater conditioning influence, but do not isolate the causal contribution of the source prior from the explicit conditioning path.

Refer to caption

Figure 4: Learning curve on LIBERO benchmark. Dashed line scores are reported by Kim et al. [10].
Table 1: Success rate on LIBERO benchmark. Each of the four task suites includes 10 tasks, and we evaluate 150 trials for each task. The 1st, 2nd highest scores are emphasized with bold and the 3rd with underline. Cocos (0.2 std) is our default configuration, and other Cocos variants serve for an ablation study.
DP-scratch DP-DINOv2 Cocos (0.2 std) Cocos (0.1 std) Cocos (0.4 std) Cocos (VAE) Cocos (EMA)
LIBERO Goal 64.3±1.3 82.9±0.4 93.8±0.3 50.9±1.8 92.3±0.9 92.5±0.3 90.5±0.4
Spatial 76.1±1.6 91.4±0.7 97.5±0.7 97.1±0.4 96.7±0.5 96.1±0.7 97.5±0.2
Object 71.4±0.7 88.9±1.0 99.1±0.6 79.3±0.8 97.6±1.0 98.1±0.7 96.9±0.5
Long 45.9±1.6 82.7±0.2 88.7±0.1 82.7±0.5 89.5±0.4 88.2±0.6 87.3±0.6
Average 64.4 86.5 94.8 77.5 94.0 93.8 93.0
Table 2: Success rate on MetaWorld benchmark. We evaluate 150 trials for each task. The highest scores are emphasized in bold. We take a multi-task setting, using language instructions to differentiate each task.

button-press button-press-td button-press-td-w button-press-w coffee-button door-open door-lock door-unlock drawer-close drawer-open Average
w/ Cocos 100.0±0.0 95.3±0.9 94.7±3.4 97.3±2.5 100.0±0.0 99.3±0.9 44.7±1.9 93.3±4.1 100.0±0.0 100.0±0.0 74.8±0.9
w/o Cocos 97.3±2.5 96.0±1.6 91.3±4.1 99.3±0.9 97.3±2.5 77.3±5.7 3.3±0.9 67.3±5.2 100.0±0.0 100.0±0.0 59.5±3.8
scratch 48.7±8.1 41.3±6.6 32.7±6.6 82.0±5.9 53.3±0.9 50.7±4.7 23.3±2.5 24.7±3.4 88.7±0.9 57.3±5.0 33.6±3.4
faucet-close faucet-open handle-press handle-press-s window-close window-open handle-pull handle-pull-s basketball bin-picking
w/ Cocos 82.0±2.8 100.0±0.0 99.3±0.9 100.0±0.0 100.0±0.0 94.0±3.3 2.7±0.9 9.3±4.7 80.7±8.6 64.0±6.2
w/o Cocos 80.7±3.8 84.0±4.3 98.0±1.6 98.0±1.6 100.0±0.0 89.3±4.1 1.3±0.9 0.0±0.0 0.7±0.9 46.0±5.9
scratch 36.7±3.4 79.3±1.9 76.0±2.8 53.3±3.4 60.7±1.9 26.0±0.0 2.0±1.6 0.0±0.0 0.0±0.0 10.0±4.3
coffee-pull coffee-push dial-turn hammer sweep sweep-into soccer shelf-place disassemble stick-push
w/ Cocos 79.3±6.2 60.7±5.0 60.0±2.8 76.7±6.6 72.0±8.5 66.0±8.6 14.0±3.3 26.7±0.9 39.3±1.9 94.0±4.3
w/o Cocos 72.7±4.7 55.3±5.2 24.0±4.3 72.7±1.9 9.3±2.5 20.0±3.4 14.0±0.9 3.3±2.5 72.7±4.9 12.7±1.9
scratch 21.3±6.8 19.3±5.7 2.7±0.9 25.3±2.5 10.0±4.9 18.0±3.3 6.7±1.9 2.7±0.9 14.7±2.5 40.0±2.8

Refer to caption
Figure 5: Evaluation results on SO100 and xArm platforms. We collect 4 task suites or 10 tasks for each robot platform. Each task provides 20 demonstrations and is tested over 10 trials.

Refer to caption

Figure 6: Rollout samples of DP with Cocos on real-robot manipulation, including MoveTo (Bowl) (Top Left), Wipe (Top Right), Moka (Bottom Left), and Pour (Bottom Right).

5.3 Real-World Experiments (RQ2)

Each physical task supplies 20 demonstrations and ten test trials. Figure 5 gives SO100 suite-average success of 70.0% for Cocos, 47.1% for DINOv2 and 6.3% for scratch; xArm gives 89.2/79.2/43.8%. These bars average the four suites equally, although the suites contain different numbers of tasks. They are not a uniform average over all ten tasks.

SO100 Cocos scores are 75% pickup/place, 85% move, 100% wipe and 20% unfold. xArm scores are 96.7% pickup/place, 90% lid manipulation, 90% pouring and 80% Moka placement. The last ties the DINOv2 baseline, qualifying the claim of improvement in every suite. Rollouts suggest fewer localization and spatial-reference errors, while the small test sets and absent physical-run uncertainty limit statistical conclusions.

5.4 Case Studies (RQ3)

In LIBERO plate movement, selected baseline failures lose the plate outside the wrist view or push it past a stove visible in the third-person view. Cocos succeeds in the displayed comparison. Across sampled image pairs, third-person feature cosine similarity is 0.514 with Cocos versus 0.843 without and 0.971 for scratch; first-person values are 0.206/0.324/0.820. These are similarities of projected features before fusion, distinct from Figure 1’s before/after cross-attention diagnostic.

Real examples show a Cocos policy remaining idle when the requested lid-opening task is already complete, while a baseline executes unnecessary motion. Other baseline examples miss objects or confuse rightward instructions. These selected trajectories are useful failure illustrations, not randomized intervention tests proving that a particular camera or modality caused every error.

Refer to caption
Figure 7: Real robot case study. Top left rollout comes from policy w/ Cocos and the others are from baselines.
Refer to caption
Figure 8: Case study of LIBERO plate movement.

5.5 Ablation Studies

The default source standard deviation β = 0.2 reaches 94.8% LIBERO average. β = 0.4 reaches 94.0%, a learned-variance VAE variant 93.8%, and joint EMA training 93.0%. β = 0.1 drops to 77.5%, driven especially by Goal at 50.9%; narrow priors can be harmful. These few settings support a usable range rather than broad hyperparameter robustness. EMA simplifies staging with a modest lower average, and the paper supplies no significance test establishing equivalence.

6 Conclusion, Limitations, and Future Works

Condition-dependent source noise improves this compact multi-task imitation-learning recipe across several benchmarks and physical tasks. Large-scale pretraining, broad transfer and more expressive priors remain future work. Additional limits include task-level regressions, incomplete compute accounting, missing operational hyperparameters, and a theoretical conclusion stronger than the proved gradient comparison. The evidence supports Cocos as a promising practical modification, not a universal guarantee against condition omission.

Appendix A Proofs of Theorems

The lemma proof expands the two squared errors and matches the parameter-dependent terms using the density-weighted conditional velocity. It requires sufficient tail decay and bounded fields/derivatives to exchange integration and differentiation. Equal gradients imply objectives differing by a constant, as the lemma states; the final proof sentence’s claim that they are simply “equal” omits that qualification.

The theorem writes the residual d = vθ − u and f = (∇θvθ)ᵀd, then uses triangle inequalities and the Jacobian assumptions to obtain 2(M + KD)ε under a common normalized sampling measure. Its notation writes q(x₁,c) inside a measure called independent of the particular c; interpreting c as an integrated training-condition variable rather than the compared query condition is essential. Ordinary data generally has condition-dependent targets, so this notation should be resolved before applying the result as a general training theorem.

Under different measures, the gradient difference splits into a same-measure term bounded by O(ε) and a measure-change term T₁. The proof gives no lower bound on T₁ and no convergence analysis. Moreover, its uniform bounds imply ‖f‖ ≤ MD and therefore ‖T₁‖ ≤ 4MD for probability measures, contradicting literal unbounded growth while fixed M and D remain in force. A dependence not controlled by ε is the defensible conclusion; guaranteed escape from collapse does not follow. Every original equation and theorem statement remains unchanged.

Equations 11–41
∇θ𝔼pt​(x|c)​‖vθ​(t,x,c)−ut​(x|c)‖2=∇θ𝔼pt​(x|c)​(‖vθ​(t,x,c)‖2−2​⟨vθ​(t,x,c),ut​(x|c)⟩+‖ut​(x|c)‖2)=∇θ𝔼pt​(x|c)​(‖vθ​(t,x,c)‖2−2​⟨vθ​(t,x,c),ut​(x|c)⟩)\nabla_{\theta}\mathbb{E}_{p_{t}(x|c)}\|v_{\theta}(t,x,c)-u_{t}(x|c)\|^{2}=\\ \nabla_{\theta}\mathbb{E}_{p_{t}(x|c)}\left(\left\|{v_{\theta}(t,x,c)}\right\|^{2}-2\left\langle{v_{\theta}(t,x,c),u_{t}(x|c)}\right\rangle+\left\|{u_{t}(x|c)}\right\|^{2}\right)\\ =\nabla_{\theta}\mathbb{E}_{p_{t}(x|c)}\left(\left\|{v_{\theta}(t,x,c)}\right\|^{2}-2\left\langle{v_{\theta}(t,x,c),u_{t}(x|c)}\right\rangle\right) (11)
∇θ𝔼q⁡(z),pt​(x|z,c)​‖vθ​(t,x,c)−ut​(x|z,c)‖2=∇θ𝔼q⁡(z),pt​(x|z,c)​(‖vθ​(t,x,c)‖2−2​⟨vθ​(t,x,c),ut​(x|z,c)⟩+‖ut​(x|z,c)‖2)=∇θ𝔼q⁡(z),pt​(x|z,c)​(‖vθ​(t,x,c)‖2−2​⟨vθ​(t,x,c),ut​(x|z,c)⟩)\nabla_{\theta}\mathbb{E}_{q(z),p_{t}(x|z,c)}\|v_{\theta}(t,x,c)-u_{t}(x|z,c)\|^{2}=\\ \nabla_{\theta}\mathbb{E}_{q(z),p_{t}(x|z,c)}\left(\left\|{v_{\theta}(t,x,c)}\right\|^{2}-2\left\langle{v_{\theta}(t,x,c),u_{t}(x|z,c)}\right\rangle+\left\|{u_{t}(x|z,c)}\right\|^{2}\right)\\ =\nabla_{\theta}\mathbb{E}_{q(z),p_{t}(x|z,c)}\left(\left\|{v_{\theta}(t,x,c)}\right\|^{2}-2\left\langle{v_{\theta}(t,x,c),u_{t}(x|z,c)}\right\rangle\right) (12)
𝔼pt​(x|c)​‖vθ​(t,x,c)‖2\displaystyle\mathbb{E}_{p_{t}(x|c)}\left\|{v_{\theta}(t,x,c)}\right\|^{2} =∫‖vθ​(t,x,c)‖2​pt​(x|c)​𝑑x\displaystyle=\int\left\|{v_{\theta}(t,x,c)}\right\|^{2}p_{t}(x|c){\rm d}x (13)
=∬‖vθ​(t,x,c)‖2​pt​(x|z,c)​q​(z)​𝑑z​𝑑x\displaystyle=\iint\left\|{v_{\theta}(t,x,c)}\right\|^{2}p_{t}(x|z,c)q(z){\rm d}z{\rm d}x (14)
=𝔼q⁡(z),pt​(x|z,c)​‖vθ​(t,x,c)‖2\displaystyle=\mathbb{E}_{q(z),p_{t}(x|z,c)}\left\|{v_{\theta}(t,x,c)}\right\|^{2} (15)
𝔼pt​(x|c)​⟨vθ​(t,x,c),ut​(x|c)⟩\displaystyle\mathbb{E}_{p_{t}(x|c)}\left\langle{v_{\theta}(t,x,c),u_{t}(x|c)}\right\rangle =∫⟨vθ​(t,x,c),ut​(x|c)⟩​pt​(x|c)​𝑑x\displaystyle=\int\left\langle{v_{\theta}(t,x,c),u_{t}(x|c)}\right\rangle p_{t}(x|c){\rm d}x (16)
=∫⟨vθ​(t,x,c),𝔼q⁡(z)​ut​(x|z,c)​pt​(x|z,c)pt​(x|c)⟩​pt​(x|c)​𝑑x\displaystyle=\int\left\langle{v_{\theta}(t,x,c),\mathbb{E}_{q(z)}\frac{u_{t}(x|z,c)p_{t}(x|z,c)}{p_{t}(x|c)}}\right\rangle p_{t}(x|c){\rm d}x (17)
=∫⟨vθ​(t,x,c),∫ut​(x|z,c)​pt​(x|z,c)​q​(z)⟩​𝑑z​𝑑x\displaystyle=\int\left\langle{v_{\theta}(t,x,c),\int u_{t}(x|z,c)p_{t}(x|z,c)q(z)}\right\rangle{\rm d}z{\rm d}x (18)
=∬⟨vθ​(t,x,c),ut​(x|z,c)⟩​pt​(x|z,c)​q​(z)​𝑑z​𝑑x\displaystyle=\iint\left\langle{v_{\theta}(t,x,c),u_{t}(x|z,c)}\right\rangle p_{t}(x|z,c)q(z){\rm d}z{\rm d}x (19)
=𝔼q⁡(z),pt​(x|z,c)​⟨vθ​(t,x,c),ut​(x|z,c)⟩\displaystyle=\mathbb{E}_{q(z),p_{t}(x|z,c)}\left\langle{v_{\theta}(t,x,c),u_{t}(x|z,c)}\right\rangle (20)
d⁡(t,x,c)≔vθ​(t,x,c)−ut​(x|x1,x0),f⁡(z,c)≔∇θvθ​(t,x,c)⊤​d​(t,x,c),d(t,x,c)\coloneqq v_{\theta}(t,x,c)-u_{t}\bigl(x|x_{1},x_{0}\bigr),\quad f(z,c)\coloneqq\nabla_{\theta}v_{\theta}(t,x,c)^{\top}\,d(t,x,c), (21)
μ⁡(d​y)=q⁡(x1,c)​q​(x0)​pt​(x|x1,x0)​d​x0​d​x1​d​x,\mu(\mathrm{d}y)=q(x_{1},c)\,q(x_{0})\,p_{t}\bigl(x|x_{1},x_{0}\bigr)\,\mathrm{d}x_{0}\,\mathrm{d}x_{1}\,\mathrm{d}x, (22)
μc​(d​y)=q⁡(x1,c)​q​(x0|c)​pt​(x|x1,x0)​d​x0​d​x1​d​x,\mu_{c}(\mathrm{d}y)=q(x_{1},c)\,q\bigl(x_{0}|c\bigr)\,p_{t}\bigl(x|x_{1},x_{0}\bigr)\,\mathrm{d}x_{0}\,\mathrm{d}x_{1}\,\mathrm{d}x, (23)
‖∇θvθ​(t,x,c)‖≤M,‖d⁡(t,x,c)‖≤D,\bigl\|\nabla_{\theta}v_{\theta}(t,x,c)\bigr\|\leq M,\quad\bigl\|d(t,x,c)\bigr\|\leq D, (24)
‖∇θvθ​(t,x,c1)−∇θvθ​(t,x,c2)‖≤K​‖vθ​(t,x,c1)−vθ​(t,x,c2)‖≤K​ϵ.\bigl\|\nabla_{\theta}v_{\theta}(t,x,c_{1})-\nabla_{\theta}v_{\theta}(t,x,c_{2})\bigr\|\leq K\,\bigl\|v_{\theta}(t,x,c_{1})-v_{\theta}(t,x,c_{2})\bigr\|\leq K\,\epsilon. (25)
∇θℒCFMc​(θ,c)=2​∫f⁡(z,c)​μ​(𝑑y).\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c)=2\int f(z,c)\,\mu(\mathrm{d}y). (26)
‖∇θℒCFMc​(θ,c1)−∇θℒCFMc​(θ,c2)‖\displaystyle\Bigl\|\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{1})-\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{2})\Bigr\| (27)
=2​‖∫f⁡(z,c1)​μ​(𝑑y)−∫f⁡(z,c2)​μ​(𝑑y)‖\displaystyle=2\Bigl\|\int f(z,c_{1})\,\mu(\mathrm{d}y)-\int f(z,c_{2})\,\mu(\mathrm{d}y)\Bigr\| (28)
=2​‖∫[f⁡(z,c1)−f⁡(z,c2)]​μ​(𝑑y)‖.\displaystyle=2\Bigl\|\int\bigl[f(z,c_{1})-f(z,c_{2})\bigr]\mu(\mathrm{d}y)\Bigr\|. (29)
f⁡(z,c1)−f⁡(z,c2)\displaystyle f(z,c_{1})-f(z,c_{2}) =∇θvθ​(t,x,c1)⊤​d​(t,x,c1)−∇θvθ​(t,x,c2)⊤​d​(t,x,c2)\displaystyle=\nabla_{\theta}v_{\theta}(t,x,c_{1})^{\top}\,d(t,x,c_{1})-\nabla_{\theta}v_{\theta}(t,x,c_{2})^{\top}\,d(t,x,c_{2}) (30)
=∇θvθ​(t,x,c1)⊤​[d⁡(t,x,c1)−d⁡(t,x,c2)]\displaystyle=\nabla_{\theta}v_{\theta}(t,x,c_{1})^{\top}\bigl[d(t,x,c_{1})-d(t,x,c_{2})\bigr] (31)
+[∇θvθ​(t,x,c1)−∇θvθ​(t,x,c2)]⊤​d​(t,x,c2).\displaystyle+\bigl[\nabla_{\theta}v_{\theta}(t,x,c_{1})-\nabla_{\theta}v_{\theta}(t,x,c_{2})\bigr]^{\top}d(t,x,c_{2}). (32)
‖∇θℒCFMc​(θ,c1)−∇θℒCFMc​(θ,c2)‖\displaystyle\Bigl\|\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{1})-\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{2})\Bigr\| ≤2​∫(M​ϵ+K​ϵ​D)​μ​(𝑑y)\displaystyle\leq 2\int\Bigl(M\,\epsilon+K\,\epsilon\,D\Bigr)\,\mu(\mathrm{d}y) (33)
=2​(M+K​D)​ϵ.\displaystyle=2\,(M+KD)\,\epsilon. (34)
v∗​(t,x):=arg⁡minv​𝔼c∈𝒞​𝔼z∼μ​[‖v⁡(t,x)−ut​(x|x1,x0)‖2].v^{*}(t,x):=\arg\min_{v}~\mathbb{E}_{c\in\mathcal{C}}\mathbb{E}_{z\sim\mu}\left[\|v(t,x)-u_{t}(x|x_{1},x_{0})\|^{2}\right]. (35)
∇θℒCFMc​(θ,c)=2​∫f⁡(z,c)​μc​(𝑑y).\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c)=2\int f(z,c)\,\mu_{c}(\mathrm{d}y). (36)
∇θℒCFMc​(θ,c1)−∇θℒCFMc​(θ,c2)\displaystyle\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{1})-\nabla_{\theta}\mathcal{L}_{\mathrm{CFMc}}(\theta,c_{2}) (37)
=2​[∫f⁡(z,c1)​μc1​(𝑑z)−∫f⁡(z,c2)​μc2​(𝑑z)]\displaystyle=2\Bigl[\int f(z,c_{1})\,\mu_{c_{1}}(\mathrm{d}z)-\int f(z,c_{2})\,\mu_{c_{2}}(\mathrm{d}z)\Bigr] (38)
=2​[(∫f⁡(z,c1)​μc1​(𝑑z)−∫f⁡(z,c1)​μc2​(𝑑z))+(∫f⁡(z,c1)​μc2​(𝑑z)−∫f⁡(z,c2)​μc2​(𝑑z))]\displaystyle=2\Bigl[\bigl(\int f(z,c_{1})\,\mu_{c_{1}}(\mathrm{d}z)-\int f(z,c_{1})\,\mu_{c_{2}}(\mathrm{d}z)\bigr)+\bigl(\int f(z,c_{1})\,\mu_{c_{2}}(\mathrm{d}z)-\int f(z,c_{2})\,\mu_{c_{2}}(\mathrm{d}z)\bigr)\Bigr] (39)
=T1+T2.\displaystyle\;=\;T_{1}+T_{2}. (40)
T1=2​(∫f⁡(z,c1)​μc1​(𝑑z)−∫f⁡(z,c1)​μc2​(𝑑z))T_{1}=2\Bigl(\int f(z,c_{1})\,\mu_{c_{1}}(\mathrm{d}z)-\int f(z,c_{1})\,\mu_{c_{2}}(\mathrm{d}z)\Bigr) (41)

Appendix B Details of Benchmarks

LIBERO has four ten-task suites with 500 demonstrations per suite; unsuccessful demonstrations are retained, unlike some cited comparison protocols. Policies observe current two-view RGB images, predict 16 actions and execute eight before replanning. MetaWorld uses 20 scripted demonstrations for each of 30 selected tasks, one current RGB view, 16 predicted actions and all 16 executed. Both evaluate 50 trials at three random seeds. Tables 3–4 retain all 70 task descriptions.

SO100 uses two 640×480 cameras at 25 FPS and a 30 Hz controller; xArm uses a third-person L515 at 30 FPS and a 30 Hz controller. Both predict absolute joint-angle chunks of length 64 and execute 40 steps. Observations include current RGB images and current plus ten prior joint states. Although xArm records depth, the stated policy observation is RGB plus joints; depth use is not demonstrated.

Each robot’s ten tasks comprise six object/placement combinations and four other tasks. SO100 adds two move tasks, wiping and unfolding; xArm adds lid opening/closing, pouring and Moka placement. Table 5 retains exact instructions and parameter combinations. SO100 jitter, limited payload and camera lag are acknowledged hardware constraints.

Refer to caption
Figure 9: LIBERO simulation benchmark. We conduct experiments on 40 tasks from four task suites in the LIBERO benchmark. We show two task examples for each suite here.
Refer to caption
Figure 10: MetaWorld simulation benchmark. We conduct experiments on 30 tasks of three difficulty levels in the MetaWorld benchmark. We show all task examples here.
Refer to caption
Figure 11: Real-world experimental setups. We conduct experiments on both SO100 and xArm platform. For each robot, we design a suite of 10 tabletop tasks involving diverse objects.
Table 3: Task description of each task in the LIBERO benchmark.
Task Suite Task Description
LIBERO-Goal open the middle layer of the drawer
put the bowl on the stove
put the wine bottle on the top of the drawer
open the top layer of the drawer and put the bowl inside
put the bowl on the top of the drawer
push the plate to the front of the stove
put the cream cheese on the bowl
turn on the stove
put the bowl on the plate
put the wine bottle on the rack
LIBERO-Spatial pick the akita black bowl between the plate and the ramekin and place it on the plate
pick the akita black bowl next to the ramekin and place it on the plate
pick the akita black bowl from table center and place it on the plate
pick the akita black bowl on the cookies box and place it on the plate
pick the akita black bowl in the top layer of the wooden cabinet and place it on the plate
pick the akita black bowl on the ramekin and place it on the plate
pick the akita black bowl next to the cookies box and place it on the plate
pick the akita black bowl on the stove and place it on the plate
pick the akita black bowl next to the plate and place it on the plate
pick the akita black bowl on the wooden cabinet and place it on the plate
LIBERO-Object pick the alphabet soup and place it in the basket
pick the cream cheese and place it in the basket
pick the salad dressing and place it in the basket
pick the bbq sauce and place it in the basket
pick the ketchup and place it in the basket
pick the tomato sauce and place it in the basket
pick the butter and place it in the basket
pick the milk and place it in the basket
pick the chocolate pudding and place it in the basket
pick the orange juice and place it in the basket
LIBERO-Long put both the alphabet soup and the tomato sauce in the basket
put both the cream cheese box and the butter in the basket
turn on the stove and put the moka pot on it
put the black bowl in the bottom drawer of the cabinet and close it
put the white mug on the left plate and put the yellow and white mug on the right plate
pick up the book and place it in the back compartment of the caddy
put the white mug on the plate and put the chocolate pudding to the right of the plate
put both the alphabet soup and the cream cheese box in the basket
put both moka pots on the stove
put the yellow and white mug in the microwave and close it
Table 4: Task description of each task in the MetaWorld benchmark.
Task Name Task Description
basketball Dunk the basketball into the basket.
bin-picking Grasp the puck from one bin and place it into another bin.
button-press Press a button.
button-press-topdown Press a button from the top.
button-press-topdown-wall Bypass a wall and press a button from the top.
button-press-wall Bypass a wall and press a button.
coffee-button Push a button on the coffee machine.
coffee-pull Pull a mug from a coffee machine.
coffee-push Push a mug under a coffee machine.
dial-turn Rotate a dial 180 degrees.
disassemble Pick a nut out of the peg.
door-lock Lock the door by rotating the lock clockwise.
door-open Open a door with a revolving joint.
door-unlock Unlock the door by rotating the lock counter-clockwise.
drawer-close Push and close a drawer.
drawer-open Open a drawer.
faucet-close Rotate the faucet clockwise.
faucet-open Rotate the faucet counter-clockwise.
hammer Hammer a screw on the wall.
handle-press Press a handle down.
handle-press-side Press a handle down sideways.
handle-pull Pull a handle up.
handle-pull-side Pull a handle up sideways.
shelf-place Pick and place a puck onto a shelf.
soccer Kick a soccer into the goal.
stick-push Grasp a stick and push a box using the stick.
sweep Sweep a puck off the table.
sweep-into Sweep a puck into a hole.
window-close Push and close a window.
window-open Push and open a window.
Table 5: Task description of each task in the SO100 and xArm benchmark. As each parameter combination introduces one task, each task suite has 10 tasks in total. For each task, we test the model for 10 trials.
Task Suite Task Description Parameter
SO100 pick [A] and place it on the [B] side of the table [A]: ["screwdriver", "sponge", "charger"], [B]: ["left", "right"]
move [A] to the center of the table [A]: ["cup", "bowl"]
pick the cloth and wipe the table None
unfold the cloth None
xArm pick [A] and place it on the [B] side of the table [A]: ["scissor", "plier", "tap"], [B]: ["left", "right"]
open the pot lid or put the lid on the pot [open, close]
pour the water from the kettle into the cup None
place the Moka pot on the cooker None

Appendix C Details of Evaluation Metrics

Figure 1 sums layerwise diagnostics with weights 0.5ˡ. The cosine metric takes the minimum across sequence positions after comparing hidden states before and after cross-attention; the norm metric takes the maximum relative norm change. The weights are not normalized and the extrema emphasize the most changed token, so these numbers are not simple average attention strength or direct measures of grounded reasoning. Figure 8 instead averages pairwise similarities of randomly sampled images after projection/position encoding and before policy fusion.

Appendix D Policy Backbone Recipe

The appendix supplies the backbone, conditioning, diffusion schedule and hardware recipe. It does not provide a complete executable training configuration.

D.1 Condition Network

DINOv2-Base and T5-Base remain frozen. Linear projectors and trainable positional embeddings prepare their outputs for the policy; positions start from sine/cosine encodings scaled by 0.2. The policy-side condition representation can change even though the pretrained encoders do not.

D.2 Policy Network

The 12-layer Transformer has width 384 and six heads. Vision and language supply alternating cross-attention keys/values. Diffusion time and robot kinematics enter through AdaLN-Zero [21]. This is the compact RDT backbone used for the controlled experiments, not an ablation across many policy architectures.

D.3 Diffusion Model

Training draws continuous flow times uniformly on [0,1] and embeds them with Fourier scale 0.2, using the linear path in Equation 10. Inference applies equal-step Euler integration. The number of Euler steps is not specified here; optimizer, learning rate, batch size, autoencoder budget, EMA coefficient and a concrete default α are also absent from the supplied recipe, limiting exact reproduction from the paper alone.

D.4 Computation Resources

Training and testing use a server with four RTX 4090 GPUs and an Intel Xeon E5-2678 v3 CPU at 2.50 GHz. No measured per-run duration, total GPU-hours or full project compute is supplied, so the gradient-step claim should remain separate from end-to-end resource savings.

NeurIPS Paper Checklist

The submission checklist asserts reproducibility, error reporting, resource disclosure, ethical compliance and asset attribution; it promises code release shortly after submission. These are author declarations, not independent verification of current availability or complete reporting. Human-subject research, high-risk release safeguards and methodological LLM use are marked not applicable. The detailed scientific limitations above remain relevant despite the checklist’s affirmative answers.

Concept demonstration

Open image ↗