Literature
Display
← All papers

Targeting World Models to Compromise Robot Learning Pipelines

Ethan Rathbun · Ahmed Agha · Saaduddin Mahmud · Christopher Amato · Alina Oprea · Eugene Bagdasarian

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete twenty-one-page v2 dated 7 October 2026, including the additional r2Dreamer experiments. All nine figures, five numbered tables, the appendix contents table and seven numbered equations remain.

At a glance

World models can turn externally supplied demonstrations into synthetic training trajectories or learned simulation environments. The paper studies whether apparently safe demonstrations can carry image perturbations whose harmful effects emerge only during this transformation.

Visual Prompt Hijacking changes text-conditioned video generation; Visual Transition Hijacking changes action-conditioned predictions. The strongest downstream result is a backdoored reinforcement-learning policy evaluated in a simulator. The text-conditioned/VLA branch demonstrates corrupted video generation, but does not complete downstream VLA training.

1 Introduction

Robot policies benefit from large demonstration datasets, but collecting new scenes and embodiments is costly. World models promise reusable synthetic data and learned training environments, as in scalable world-model training and VLAW. Their position between third-party data and policy optimization creates a potential trust boundary.

The paper asks whether manipulating only the input images can poison that intermediate process. Its main findings concern a strong white-box threat model, increased prompt-hijacking success under camera/prompt shifts, and a downstream simulated RL backdoor. Claims of priority are the authors’ claims, not an independent survey conclusion.

Complete original figure with every panel, label and legend.
Figure 1: Visual example of our proposed “Visual Prompt Hijacking” attack, which targets text-conditioned world models to overwrite the user’s prompt with a different, malicious prompt. Once fed through a world model, this results in the generation of dangerous learning trajectories which then propagate to downstream robot policies.

2 Problem Setting

Both behavior cloning and reinforcement learning produce a policy from observation histories to action distributions in a partially observed decision process. The source defines a text-conditioned model as mapping a prompt and image to an observation/action trajectory, and an action-conditioned model as predicting the next observation.

In practice, text-to-video generation still needs a separate inverse-dynamics model to supply robot actions. That distinction matters because errors at this conversion stage prevent the paper from validating a complete VLA poisoning pipeline.

Equations 1
θt:𝒯×Ω→Ω∗×A∗θa:A×Ω→Ω\begin{array}[]{cc}\theta_{t}:\mathcal{T}\times\Omega\rightarrow\Omega^{*}\times A^{*}&\theta_{a}:A\times\Omega\rightarrow\Omega\\ \end{array} (1)
π:Ω∗→Δ​A\pi:\Omega^{*}\rightarrow\Delta A
ℳ=(S,A,R,T,Ω,𝒪,γ)\mathcal{M}=(S,A,R,T,\Omega,\mathcal{O},\gamma)

3 Related Work

The related work contrasts synthetic-data generation and learned simulators with earlier direct dataset or reinforcement-learning backdoor attacks.

World Models in Robot Learning

Text-conditioned video models such as Cosmos can propose demonstrations without an expert policy; an inverse-dynamics model then estimates action labels. Action-conditioned models such as DINO-WM support planning or learned simulation.

These routes reduce new data collection but inherit model errors and dependence on the supplied context. Teleoperation datasets, sim-to-real methods and open robot datasets supply the broader pipeline considered here.

Attacks against AI and Robotic

Earlier VLA backdoor attacks poison demonstrations directly, while reinforcement-learning backdoors may manipulate training rewards or transitions. This work instead alters images before a world model transforms them. The source’s broad claim that direct poisoning is trivial to detect is motivation, not a measured comparison in these experiments.

4 Identifying Vulnerabilities in the Robot Learning Supply Chain

A malicious data provider can alter demonstration video frames and has white-box access to the victim world model, including gradients. Actions, downstream prompts and reward functions are outside the attacker’s direct control. Perturbations are bounded in LAB color space; the intended constraint is that the original demonstrations remain safe and useful before synthetic generation.

This is stronger access than a black-box deployment threat. Publicly available model weights motivate the assumption but do not establish transfer to private models or unknown fine-tuning. Perceptual bounds are not a human imperceptibility study; the appendix explicitly calls the chosen budgets relatively large.

Complete original figure with every panel, label and legend.
Figure 2: Visualization of our proposed threat model. By targeting world models within the robot learning pipeline, malicious data providers can effectively implant dangerous behavior or altered transition dynamics into otherwise safe robot teleoperation data. This allows them to poison downstream robot policies while bypassing dataset-level safety checks.

5 Methodology

A backdoor must change behavior when a trigger appears while preserving the intended task without it. Equation 2 formalizes the competing attack and clean-performance objectives. This separates targeted policy poisoning from indiscriminate degradation of the generated training data.

The two mechanisms exploit different model roles: text-conditioned generation can produce the wrong demonstration, while action-conditioned generation can make some actions appear preferable through distorted predicted outcomes.

Equation 2
Attack Success: minπ+[𝔼o∈𝒰⁡(Ω)[ℒadv(π+(δ(o)))]]Attack Stealth: minπ+[𝔼o∈𝒰⁡(Ω)[𝔼π+,π[|Vπ+ℳ(o)−Vπℳ(o)|]]]\begin{array}[]{c}\textbf{Attack Success: }\min_{\pi^{+}}[\mathbb{E}_{o\in\mathcal{U}(\Omega)}[\mathcal{L}_{\text{adv}}(\pi^{+}(\delta(o)))]]\\ \textbf{Attack Stealth: }\min_{\pi^{+}}[\mathbb{E}_{o\in\mathcal{U}(\Omega)}[\mathbb{E}_{\pi^{+},\pi}[|V_{\pi^{+}}^{\mathcal{M}}(o)-V_{\pi}^{\mathcal{M}}(o)|]]]\end{array} (2)

5.1 Visual Prompt Hijacking Attacks

Visual Prompt Hijacking aligns the model’s internal semantic response under a benign prompt with a response associated with an adversarial objective. The paper implements this through the denoising vector field rather than readable instructions overlaid on the image. LAB-bounded perturbations preserve the nominal demonstration input while changing how the generator interprets it.

The prose compares the perturbed benign-prompt encoding with the original-image target-prompt encoding, whereas Equation 3 prints the perturbed image in both terms; the original expression is retained. The proposed link from corrupted videos to a downstream VLA backdoor is not experimentally completed here.

Equation 3
minx→δ𝔼t∼𝒯safe,tδ∼𝒯δ[D(𝒮θ(x→δ,t)||𝒮θ(x→δ,tδ))]s.t.​‖x→−x→δ‖pLAB≤β\begin{array}[]{cc}\min_{\vec{x}_{\delta}}\;\mathbb{E}_{t\sim\mathcal{T}_{\text{safe}},t_{\delta}\sim\mathcal{T}_{\delta}}\;\left[D(\mathcal{S}_{\theta}(\vec{x}_{\delta},t)||\mathcal{S}_{\theta}(\vec{x}_{\delta},t_{\delta}))\right]&\text{s.t.}\;||\vec{x}-\vec{x}_{\delta}||^{\text{LAB}}_{p}\leq\beta\end{array} (3)

5.2 Visual Transition Hijacking Attacks

Unconditional transition hijacking pushes image embeddings away from ordinary model inputs, causing predicted trajectories to diverge from valid dynamics. That alone produces poor training data. The conditional objective instead preserves plausible prediction near a selected action while degrading alternatives, creating an indirect preference during policy learning.

Equations 4–5 retain the exact objectives. The claimed mechanism depends on the world model, reward construction and policy optimizer; prediction degradation alone is not equivalent to demonstrated policy compromise.

Equations 4–5
minx→δ⁡cos⁡(ℰ⁡(x→),ℰ⁡(x→δ))s.t.​‖x→−x→δ‖pLAB≤β.\begin{array}[]{cc}\min_{\vec{x}_{\delta}}\;\cos(\mathcal{E}(\vec{x}),\mathcal{E}(\vec{x}_{\delta}))&\text{s.t.}\;||\vec{x}-\vec{x}_{\delta}||^{\text{LAB}}_{p}\leq\beta.\end{array} (4)
minx→δ𝔼a′,ϵcos(θ(xδ→,a′),θ(x→,a′))+(1−cos(θ(δ(xδ→),a++ϵ),ℰ(x→)))\min_{\vec{x}_{\delta}}\;\mathbb{E}_{a^{\prime},\epsilon}\cos(\theta(\vec{x_{\delta}},a^{\prime}),\theta(\vec{x},a^{\prime}))+(1-\cos(\theta(\delta(\vec{x_{\delta}}),a^{+}+\epsilon),\mathcal{E}(\vec{x}))) (5)

6 Experiments

Experiments evaluate the two branches separately. Text-conditioned Cosmos tests manually scored video behavior under prompt specificity and camera shifts. Action-conditioned DINO-WM supports complete PPO training and simulator evaluation; action-conditioned Cosmos tests only prediction changes.

Greater vulnerability under distribution shift is observed, but the proposed explanation in terms of uncertainty is not directly measured. Specific prompts and familiar cameras reduce many measured attack rates without constituting a certified defense.

6.1 Visual Prompt Hijacking Attacks

Fine-tune the two-billion-parameter Cosmos-Predict 2.5 model on LIBERO, then evaluate nine hazardous-object scenarios in three custom task families. Each object/setting uses five perturbed inputs and four generation seeds per input. Manual scoring follows whether the inferred robot motion would handle the original hazardous object, including cases where the video changes its appearance.

Table 1 shows strong heterogeneity. With the shifted camera and vague prompts, success ranges from 5% for oil to 100% for the gun; specific prompts lower many rates, but the gun remains at 95%. Under the familiar camera, most rates fall, although the vague-prompt gun case reaches 50% and the specific-prompt grater case remains 25%. Clean baselines are mostly zero, with some gun-scene failures.

Object size and learned task priors are proposed explanations, not isolated causal ablations. Collapsed or irrelevant videos count as attack failures under the targeted metric even when they are unusable training data. There are no reported confidence intervals or inter-annotator reliability estimates for this manual evaluation.

The attempted DreamGen-style pipeline fails to yield sufficiently reliable action-labeled demonstrations: generated physics and object identity can be inconsistent, and the fine-tuned inverse-dynamics model generalizes poorly to the new tasks. No downstream VLA poisoning result is established.

Scene Gift Box Stove Microwave
Object Bomb Dynamite Gun Oil Cardboard Cereal Box Fork Knife Grater
Out of Distribution Camera Position, Attack Success Rate
Vague Prompt 20% 50% 100% 5% 10% 90% 25% 20% 55%
Specific Prompt 10% 25% 95% 5% 0% 65% 5% 5% 25%
No Attack 0% 0% 10% 0% 0% 0% 0% 0% 0%
In Distribution Camera Position, Attack Success Rate
Vague Prompt 5% 25% 50% 0% 0% 5% 20% 0% 30%
Specific Prompt 5% 0% 10% 0% 0% 0% 0% 0% 25%
No Attack 0% 0% 5% 0% 0% 0% 0% 0% 0%
Table 1: Performance of Visual Prompt Hijacking attacks against Cosmos-Predict 2.5 evaluated on in distribution and out of distribution tasks along with vague and specific user prompts.

6.2 Transition Hijacking Attacks

Train PPO inside DINO-WM’s Single Wall environment, poisoning 5% of initial states, and evaluate in the actual task simulator. Across five training seeds, poisoned policies move toward the adversary’s target action when triggered while retaining approximately baseline clean task performance. Figure 4 reports action proximity, cosine similarity and clean simulator success with standard-deviation bands.

Attack success is an action-space criterion, not a measured physical damage rate. The full pipeline is demonstrated on this simple simulated navigation task; Push-T training remains difficult.

For the two-billion-parameter action-conditioned Cosmos model, twenty attacks on DROID inputs reduce the difference between clean and perturbed predictions as actions approach the selected target. Figure 5 reports visual and embedding distances with standard deviations. This demonstrates action-dependent prediction distortion on real-world data, without training or deploying a downstream robot policy.

Complete original figure with every panel, label and legend.
Figure 3: Example rollout of one of our backdoor poisoned PPO policies in the Single Wall env. The right column shows the goal state for each rollout while the left and middle columns shows the final state the policy reached given environments either with or without the trigger. We see in the middle column, without the trigger, that the agent successfully solves the task, while in the left “triggered” column the agent consistently takes the “down right” action and fails.
Complete original figure with every panel, label and legend.
Figure 4: Evaluating a VTH backdoor attack against PPO policies trained in the Dino WM on the Single Wall environment. Here, the error bars represent the standard deviation across training seeds.
Complete original figure with every panel, label and legend.
Figure 5: Evaluating VTH attacks against the 2B Cosmos-Predict2.5 action conditioned world model. Here, error bars represent the standard deviation across 20 generated attacks.

7 Conclusion

The results establish world-model inputs as a potential policy-poisoning entry point under white-box assumptions. More realistic environments, unknown models, physical robots, stronger perceptual constraints and complete VLA training remain open.

The authors propose post-generation consistency checks and robustness training of encoders/predictors. Their tested guardrails and automatic video labeling are inadequate in the reported settings, but this does not establish that every guardrail or current world model lacks protection. No evaluated general defense or robustness guarantee is provided.

8 Acknowledgments

The authors acknowledge support from DARPA, NSF, Amazon, Coefficient Giving, Schmidt Sciences and Nvidia. The source funding statements remain attributable to the authors.

Appendix Table of Contents

The original appendix contents table is retained. It omits the later B.1 and B.6 entries; both complete sections are included below.

Section Contents
Appendix A Experimental Details
A.1    VPH Attack Setup
A.2    VTH Attack Setup
A.3    Compute Resources Used
Appendix B Additional Discussion and Experiments
B.2    Further Analysis of Successful VPH Attack Instances
B.3    Analysis of Prediction Sabotage in Text Conditioned World Models
B.4    Preliminary Results Against World Model Guardrails
B.5    Visualization of Visual Trajectory Hijacking Attacks

Appendix A Experimental Details

The appendix specifies model adaptation, perturbation optimization, the learned-simulator reward and computing resources. These implementation assumptions bound the experimental conclusions.

A.1 VPH Attack Setup

LoRA adaptation uses the 256×256 LIBERO rerender associated with Cosmos Policy, while generating 432×432 videos. The authors report better video quality at higher resolution but choose this size for cost; the comparison is qualitative.

Table 2 preserves the exact projected-gradient/momentum parameters and LAB budgets. Optimization uses ten generated frames rather than the normal thirty-two to reduce cost. Step size is interpreted as a fraction of the per-channel bound, so its magnitude cannot be compared directly with an RGB attack budget.

Manual success labels track the original object and implied motion even when the generator changes a grater into food or a knife into another object. This procedure is necessary to understand the metric, but actual action reconstruction is not verified. The authors suspect single-view inverse dynamics contributes to the failed VLA pipeline; multi-view improvement is an untested explanation.

Attack Parameters
Parameter Num Steps β\beta L β\beta AB α\alpha Momentum Frames Gen During Attack
Value 400 5 20 0.025 0.9 10
Table 2: Attack parameters used for our paper’s VPH attack results against Cosmos-Predict 2.5.

A.2 VTH Attack Setup

The action-conditioned experiments use base DINO-WM and Cosmos models. Optimization evaluates short target and non-target action sequences and averages the cosine objective over time, aiming for effects that persist near the chosen action rather than a single exact input.

A CleanRL PPO policy receives concatenated current and goal embeddings from a Gymnasium wrapper. DINO-WM’s frame skip means one model action corresponds to five repeated simulator actions.

Equation 6 combines an embedding-distance term with a goal bonus. The prose calls the first term a bonus for improvement, but the printed minimum with zero makes it nonpositive: it penalizes being farther than the best state and gives zero for improvement. The formula is preserved rather than silently changing its sign.

Equation 6
R(z,zbest,zg)=min(0,||zbest−zg||2−||z−zg||2)+c⋅𝟙[||z−zg||2≤ϵ]R(z,z_{\text{best}},z_{g})=\min(0,||z_{\text{best}}-z_{g}||_{2}-||z-z_{g}||_{2})+c\cdot\mathbbm{1}[||z-z_{g}||_{2}\leq\epsilon] (6)

A.3 Compute Resources Used

A desktop with two RTX 4090 GPUs supports inference and DINO-WM experiments. Cosmos adaptation and perturbation optimization use a server with four H100 GPUs, reported as 96 GB each. Table 3 retains the complete CPU/RAM specification. Total GPU-hours, energy and per-attack runtime are not reported.

Compute Resources Used
Machine GPUs CPU RAM
Desktop 2x Nvidia RTX 4090 AMD Ryzen Threadripper 7980X 192 Gb
Server 4x Nvidia H100 INTEL(R) XEON(R) GOLD 6548Y+ 1 Tb
Table 3: Compute resources used for this paper

Appendix B Additional Discussion and Experiments

Additional analyses distinguish video-labeling errors, object-identity changes, non-targeted prediction collapse, guardrail observations and a modified attack on r2Dreamer.

B.1 Ineffectiveness of VLM Based Labeling

Qwen3-VL labels ten verified positive videos per object. Accuracy spans 10% for forks to 100% for oil and cardboard; it is 50% for graters. This small positive-only test supports concern about missed target actions, but does not estimate false positives or overall accuracy on a balanced success/failure set.

Scene Gift Box Stove Microwave
Object Bomb Dynamite Gun Oil Cardboard Cereal Box Fork Knife Grater
VLM Accuracy 60% 80% 70% 100% 100% 90% 10% 40% 50%
Table 4: QWen3-VL based annotation accuracy on 10 Cosmos Predict 2.5 generated demonstrations per object. All 10 samples per object were verified to demonstrate the target object being picked up.

B.2 Further Analysis of Successful VPH Attack Instances

Successful examples often change the hazardous object into the prompt’s benign object while moving toward its original location. Figure 6 shows grater-to-steak and knife-to-cheese cases. This illustrates a mismatch between generated appearance and the original scene, rather than evidence that safe-looking video implies safe executable robot actions.

Complete original figure with every panel, label and legend.
Figure 6: Interesting VPH attack success cases. In the top row we see a successful attack scenario where the adversarial target object (the cheese grater) gets turned into a steak right before the robot arm picks it up. Similarly, in the bottom row we see an instance where the target object (a knife) turns into a block of cheese immediately after the world model picks it up.

B.3 Analysis of Prediction Sabotage in Text Conditioned World Models

Other perturbations produce newly appearing objects or substitute familiar training objects, then fail both the adversary’s target and the requested task. Figure 7 retains these failure cases. Their potential harm to training is plausible, but a downstream degradation experiment for this subset is not reported.

Complete original figure with every panel, label and legend.
Figure 7: Interesting VPH attack failure cases. In the top row we see a rollout where the world model manifests the user’s prompted object (a chicken breast) in the arm’s gripper. In the bottom row we see a similar case where the model replaces the novel white bowl with one seen more frequently in the Libero dataset. While it tries to pick up the bowl it ultimately fails to place it on the red plate as requested. Both cases show attack failure, as the arm doesn’t pick up the adversary’s target objects, however these also will not be useful training runs for the downstream policy.

B.4 Preliminary Results Against World Model Guardrails

The tested Cosmos guardrails do not block the authors’ hazardous direct prompts or perturbed-image demonstrations. The appendix gives observations rather than a detailed detection benchmark with sample counts, false-positive rates or guardrail-version comparisons. A proposed distinction between visual consistency filtering and robotics safety filtering is interpretation, not verified design documentation.

B.5 Visualization of Visual Trajectory Hijacking Attacks

Figure 8 compares clean and perturbed Cosmos predictions for target versus other actions; the non-target case shows more degradation. Figure 9 displays DINO-WM outcomes across a two-dimensional action grid and a clean reference rollout. These selected visual examples complement the quantitative metrics without measuring a physical attack.

Complete original figure with every panel, label and legend.
Figure 8: Visualization of our VTH attack against Cosmos-Predict 2.5 Action Conditioned. In the right column we show the final frame of generation given a clean initial frame and in the left column we show the final frame of generation given a frame optimized by our VTH attack. In the top row the first action chosen by the agent is the target action while in the bottom row it is another random action. We can see that the bottom row shows increased signs of visual degradation, with significantly more noise in the image along with an increased number of novel objects in the frame.
Complete original figure with every panel, label and legend.
Figure 9: Visualization of our VTH attack against the Dino WM. Here the top right cell represents the target action and all cells moving left or down from there represent steps of size 0.5 away from the target action in each of the agent’s two action dimensions. In the bottom we show the ground truth trajectory given the target action and no attack perturbation.

B.6 Additional Results on r2Dreamer

The original transition-hijacking objective does not reliably collapse r2Dreamer: it often produces a believable but altered state. The authors therefore add a critic-based value term, explicitly manipulating whether alternative actions lead to lower-value predicted states. This requires additional critic access and differs materially from the main method.

Fresh PPO policies trained in this model use 25% poisoned states and are evaluated in Cartpole Swing Up and Reacher Easy simulators across five runs. Triggered action error drops from .49 to .08 and from 1.41 to .51; clean environment returns are 166 versus 256 and 626 versus 629. Table 5 retains the printed 95% intervals, including the unusually large .08±.5 entry.

The high poisoning fraction, two simple environments and modified objective limit generalization. These are supplementary evidence for the model-mediated poisoning route, not a replication of the lower-rate DINO-WM result.

Env Cartpole Swing Up Reacher Easy
Policy Clean Poisoned Clean Poisoned
Triggered Action Error 0.49 ±\pm 0.10 0.08 ±\pm 0.5 1.41 ±\pm 0.22 0.51 ±\pm 0.23
Real Env Return 166 ±\pm 86 256 ±\pm 37 626 ±\pm 12 629 ±\pm 28
Table 5: Performance of a modified VTH attack against r2Dreamer on Cartpole Swing Up and Reacher Easy at poisoning rates of 25%. All values were averaged over 5 runs with a 95% confidence window given.
Equation 7
minx→δ⁡𝔼a′,ϵ​[ℒ⁡(xδ,a)+ψ⁡(θ⁡(ℰ⁡(xδ→),a))−ψ⁡(θ⁡(ℰ⁡(xδ→),a+))]\min_{\vec{x}_{\delta}}\;\mathbb{E}_{a^{\prime},\epsilon}\Big[\mathcal{L}(x_{\delta},a)+\psi(\theta(\mathcal{E}(\vec{x_{\delta}}),a))-\psi(\theta(\mathcal{E}(\vec{x_{\delta}}),a^{+}))\Big] (7)
Concept demonstration

Open image ↗