Overview
This edition follows the complete eight-page v3 manuscript dated April 15, 2026. All seven figures, five tables, eight displayed formulas and 43 references are retained. Figure composites are restored from the PDF with native imagery, original labels and all inset panels. Jiaming Xu and Hongyu Chen share the marked equal-contribution designation; Yilun Du and Katherine Driggs-Campbell share advising.
At a glance
Policy Consensus keeps separate diffusion experts for different sensory representations, then combines their denoising predictions using learned weights. The aim is to prevent brief but essential tactile information from being overwhelmed by visually dominant training data. On four RLBench tasks it reaches 66% average success versus 56% for concatenation; real experiments report 65% marker picking and 75% spoon reorientation. The evidence favors policy-level composition in these controlled tasks, while routing dynamics, training details and quantitative robustness evaluation remain incompletely specified.
I Introduction
Vision can guide approach yet become uninformative once a gripper enters an opaque bag. Touch matters mainly after contact, so its useful training signal can be sparse. The introduction’s 90% visual/10% tactile split is an illustrative explanation, not a measured dataset statistic.
The proposed remedy is to learn action distributions separately for each representation and compose their outputs. This differs from concatenating all sensory features before one policy. It supports adding an already trained expert without rebuilding the existing experts, but useful compatibility still depends on their action representation, calibration and learned behavior. The experiments do not prove that every feature-fusion architecture necessarily suppresses touch.
II Related Works
Prior visuotactile learning includes shared feature representations, transformers and stage-guided fusion [2][16][17]. This paper moves composition from the feature representation to the diffusion-policy output. It compares with concatenation and one soft-routing feature mixture, not every advanced fusion alternative.
Energy-based composition combines distributions through products, equivalently adding energies [4][5][7]. Related robot work composes continuous constraints or policies learned from heterogeneous data [8][32]. Here the decomposition is by sensory representation, including touch. Concurrent modality-composable diffusion [41] combines RGB and point-cloud policies with fixed weights.
Diffusion Policy provides a flexible action-distribution model [33]. Its ability to model several possible actions is distinct from the paper’s use of “multi-modal” to mean several sensor representations; the two ideas are related but not interchangeable.
III Approach
The framework separately encodes each sensory representation, maintains two sub-policies per representation, and aggregates their predictions at each denoising step. A router supplies the weights across representations. The source’s energy equations and executable score-averaging rule are both preserved, including their normalization difference.
III-A Problem Formulation
Demonstrations pair robot actions with multiple sensor inputs. Each expert receives its modality embedding together with relevant robot state such as joint angles and gripper status, so a “single-modality” expert is not necessarily deprived of proprioception.
The experiments set Kᵢ=2 sub-policies per modality. Geometry/detail or contact-onset/sustained-force are offered as intended complementary roles, but the manuscript does not specify separate supervision or an ablation demonstrating those particular specializations. A softmax router maps modality embeddings to positive weights summing to one.
Equation
III-B Energy-Based Policy Composition
A base expert is written as a probability density proportional to the exponential of negative energy. Multiplying expert distributions with modality-dependent exponents gives a weighted sum of their energies. This formalizes a preference for actions supported by several experts rather than averaging their final action vectors.
The displayed identity holds for the stated energy functions. It does not guarantee safe actions, calibrated confidence or retention of a rare sensor’s influence: an uninformative expert or a small router weight can still suppress useful behavior. The architecture motivates those properties, while the experiments determine how well they hold.
Equations
III-C Compositional Policy Factorization
Composition occurs within each modality and across modalities. The written modality policy is an unnormalized product of its sub-policy densities; the outer product raises each modality density to its router weight. Separate training protects expert capacity from direct feature competition, but coordination between incompatible or miscalibrated experts remains a modeling issue.
The constraint interpretation follows prior compositional diffusion work [8]. These are learned preferences rather than exact symbolic constraints. The paper provides no hard feasibility test or fallback when experts disagree.
Equations
III-D Score-Based Implementation via Diffusion Models
Each expert uses a denoising diffusion probabilistic model [42]. At every diffusion step the implementation averages the Kᵢ sub-policy noise/score predictions, then forms a router-weighted sum across modalities. The resulting prediction drives the shared denoising process; it does not separately sample several completed actions and average them.
There is a normalization caveat: averaging sub-policy scores corresponds to a geometric-mean/tempered product, whereas the preceding density formulas write an untempered product. With Kᵢ=2 this changes the aggregate energy scale. The manuscript does not reconcile that temperature factor or establish exact sampling from the clean-action product distribution. The retained formulas show both conventions.
Equations
III-E Router Network
The formulation writes weights as wᵢ(sₜ), and Figure 2 feeds sensor embeddings to a router. However, this section says consensus weights are learned during training and fixed at execution; the conclusion likewise says the current router lacks fine-grained or temporally dynamic adjustments. The text does not resolve whether “fixed” refers only to learned parameters or also to output weights.
Accordingly, this edition treats context-dependent action sensitivity as demonstrated, while avoiding a stronger claim that the routing weights themselves are observed to change over time. Softmax weights express relative mixture coefficients; without accounting for expert score magnitudes, they are not direct causal measurements of sensor importance.
III-F Advantages Over Existing Fusion
Separate experts offer a modular way to retain specialized behaviors, and the composition mechanism gives explicit coefficients instead of implicit feature fusion. These are useful design properties, with practical evidence from the following tasks. They do not make robustness, interpretability or arbitrary sensor addition automatic; newly added experts still require training, and missing inputs require a defined composition rule.
IV Experiment
Evaluation asks whether policy composition improves multimodal manipulation, whether it exhibits context-sensitive sensor dependence, and whether already trained policies can be combined under perturbations. Only the main task tables and simulation ablation provide quantitative success rates; robustness and incremental composition are primarily illustrated through rollouts.
IV-A Experimental Setup
The simulation benchmark uses four RLBench tasks [9]: opening a box, opening a drawer, removing an umbrella and lifting a toilet seat. It has 200 demonstration episodes and 200 unseen test configurations in total. Inputs include two RGB views, point clouds and three-dimensional features derived from a pretrained DINO encoder [43]. These representations overlap in information; they are not three independent physical sensors.
Real experiments use a UR5e, two side-view RealSense D415 cameras at 96×128 pixels and FlexiTac tactile arrays [19]. Each finger has a 12×32 pad with 2 mm spatial resolution. The tasks use eighty marker-picking, sixty spoon-reorientation and fifty puzzle-insertion demonstrations.
Simulation baselines include single-representation policies, feature concatenation and a feature-level mixture of experts. Real baselines are RGB-only and RGB-plus-touch concatenation. Real trial counts, uncertainty across training runs, action horizon, diffusion schedule and detailed router/expert training hyperparameters are not specified in this manuscript.
The setup calls completion time a metric for successful trials, but Tables III–IV explicitly include failed-trial timeouts. Their means should therefore be read as timeout-penalized completion times, not success-only execution speeds.
IV-B Main Results
Table I reports 66% average simulation success, versus 56% for concatenation, 54% for RGB-only, 49% for point clouds and 45% for DINO features. The improvement over concatenation is ten percentage points, or about 18% relative. Parameter counts are 263.6 million versus 262.9 million: an increase of 0.7 million, about 0.27%. Per-task simulation results are not given.
Marker picking succeeds at 65%, versus 35% with RGB and 5% with concatenation. Timeout-penalized mean times are 96.5, 107.8 and 117.9 seconds, respectively, with failures assigned 120 seconds. Spoon reorientation scores 75%, 67% and 21%, with means 94.8, 117.1 and 221.5 seconds and a 300-second failure timeout.
Puzzle insertion is decomposed by object. Full goat success is 58% for the method, 35% for concatenation and 25% for RGB. Cow success is 45%, tying concatenation and exceeding RGB’s 30%. The headline 52% is consistent with rounding the equal average of 58% and 45% (51.5%); it is not a reported probability of successfully completing both objects in one run.
Pick/place substeps are substantially easier than full insertion, reaching 95–100% for the method. Their rates should not be multiplied or substituted for full-task success without knowing the evaluation denominators. The method is not strictly superior in every cell, and no insertion completion-time table is supplied.
Figures 4–5 preserve successful rollouts and baseline failure examples. They show that adding touch through this particular concatenation baseline can hurt, but do not establish that tactile sensing or feature-level fusion is inherently detrimental.
| Method | Success Rate | Params (M) |
| Single-Modality | ||
| RGB only | 0.54 | 257.3 |
| Point Cloud only | 0.49 | 251.9 |
| 3D DINO only | 0.45 | 251.9 |
| Multi-Modality | ||
| Concatenation | 0.56 | 262.9 |
| Ours | 0.66 | 263.6 |
| Sub-Task | RGB | RGB+Tac. | Ours |
| Pick up goat | 0.90 | 0.90 | 1.00 |
| Place goat | 0.90 | 0.90 | 0.95 |
| Task success (goat) | 0.25 | 0.35 | 0.58 |
| Pick up cow | 0.85 | 0.80 | 0.95 |
| Place cow | 0.75 | 0.80 | 0.95 |
| Task success (cow) | 0.30 | 0.45 | 0.45 |
| Metric | RGB | RGB+Tac. | Ours |
| Success Rate | 0.35 | 0.05 | 0.65 |
| Avg. Time (s) | 107.8 | 117.9 | 96.5 |
| Metric | RGB | RGB+Tac. | Ours |
| Success Rate | 0.67 | 0.21 | 0.75 |
| Avg. Time (s) | 117.1 | 221.5 | 94.8 |
IV-C Analysis of Learned Modality Dependencies
The importance study adds Gaussian noise separately to each modality and measures normalized L2 change in predicted actions. An exponential moving average with α=0.1 smooths the response. Vision dominates the early approach; after approximately five plotted steps, tactile sensitivity rises during contact while vision still matters.
This is perturbation sensitivity, not a plot of router weights or a direct causal decomposition of task success. Its interpretation depends on noise calibration and action normalization, whose numerical details are not fully reported. A phase-dependent change in expert output can occur even if composition weights remain fixed.
IV-D Robustness and Adaptation Evaluation
Figure 6 shows recovery after a marker is snatched, continued execution with one camera blocked, and successful runs after repositioning task objects. One occluded camera is partial sensor corruption; it does not remove every visual observation. The paper provides illustrative successes but no intervention trial counts or per-condition success rates, so the magnitude and reliability of robustness remain unquantified.
For incremental composition, independently trained RGB and tactile policies are combined with manually fixed weights of 0.5/0.5. Figure 7 shows a successful occluded grasp where RGB-only fails. Existing experts are reused without joint retraining; this example does not use a newly trained adaptive router and does not show arbitrary sensor removal or a broad continual-learning benchmark.
IV-E Ablation Studies
Table V reports 66% with learned routing, 61% with equal weights and 57% with feature-level soft-routing MoE. Learned weighting adds five percentage points over fixed composition; the source’s 7.6% figure is the relative drop from 66 to 61, not a 7.6-point change. Against MoE, the gain is nine points or 15.8% relative.
These comparisons favor the tested router and composition level. They do not isolate the value of having two experts per modality, identify their claimed semantic roles, or resolve the ambiguity about temporal routing behavior. No variance or significance testing is reported.
| Method | Avg. Success Rate |
| Ours (Learned Router) | 0.66 |
| Ablation Variants | |
| Fixed Equal Weights | 0.61 |
| Factorized MoE Fusion | 0.57 |
V Conclusion
The main evidence supports composing modality-specific denoising policies for the tested contact-rich tasks, and shows one useful way to reuse independently trained sensory experts. Sparse touch can matter without forcing all features through a single policy, though substantial real-task failures remain.
Open issues include the router’s execution semantics, the product-versus-average energy scaling, incomplete reproduction details, small controlled task scope and unquantified perturbation robustness. Audio, language, other sensing modalities and broader deployment are proposed extensions. Dynamic routing should be treated as an unresolved or future capability rather than inferred from the sensitivity plot alone.
Acknowledgments
The authors acknowledge assistance with robot setup, experiment design, presentation and manuscript feedback. Support includes National Science Foundation grant 2143435 and PickleRobotics.