Literature
Display
← All papers

Refined Policy Distillation: From VLA Generalists to RL Experts

Tobias Jülg · Wolfram Burgard · Florian Walter

· IROS 2025 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

Refined Policy Distillation trains a compact task-specific policy with reinforcement learning while using a vision-language-action teacher to guide its actions. An action-matching term augments PPO, improving exploration while permitting the student to exceed the teacher. Simulation experiments with fine-tuned Octo and OpenVLA show faster learning on several manipulation tasks, including sparse rewards and a changed camera view. Benefits depend on useful teacher actions; the held-out tool-use tasks expose negative transfer when that condition fails.

I Introduction

Large generalist policies offer transferable knowledge but can be slow and brittle under setup changes. Task-specific reinforcement learning can improve performance through interaction, yet exploration is expensive. RPD connects these approaches by querying the teacher at student-visited states and combining its guidance with environment reward. The result is a specialized student, not a refined generalist VLA. All evaluations are simulated; real deployment and sim-to-real transfer remain future work.

Refer to caption
Fig. 1: The architecture of RPD: We distill a VLA teacher into an RL student policy. This has two effects: First, the RL student agent is bootstrapped with guided exploration through the VLA teacher. Second, the RL student policy can interact with the environment and is, thus, able to surpass the VLA’s performance, refining the distilled policy.

II Related Work

Earlier policy distillation transfers behavior between reinforcement-learning agents, while behavioral cloning supplies expert supervision from trajectories. RPD instead queries generalist Octo [4] or OpenVLA [7] during on-policy learning. Compared with Proximal Policy Distillation [23], its mean-squared action loss avoids estimating the full teacher distribution. Compared with collecting expert RL data to train a VLA, the direction of transfer is reversed: the VLA helps train the smaller expert.

III Methodology

The student acts in the environment and receives ordinary reinforcement-learning rewards. At the same visited states, the teacher supplies an action target. Matching that target biases exploration toward plausible behavior; maximizing return allows the student to depart from the teacher where interaction reveals a better solution.

III-A Refined Policy Distillation

RPD augments PPO’s maximized clipped objective with a negative mean-squared error between the student action mean and the expected teacher action. Under the paper’s fixed-variance Gaussian approximation, this corresponds to the action-mean part of behavioral cloning without the variance-shrinking component. In practice, one sampled teacher action estimates the expectation at each step; larger sample counts were not evaluated because of inference cost. The study also tests an absolute-error variant and a likelihood-based behavioral-cloning variant.

Equations 1–6
θ∗=arg⁡maxθ⁡𝔼⁡[∑t=0T−1γt​r​(st,at)]\displaystyle\theta^{*}=\arg\max_{\theta}\mathbb{E}\left[\sum_{t=0}^{T-1}\gamma^{t}r(s_{t},a_{t})\right] (1)
max⁡∑(s,a)∈ρ𝒟θ⁡ln⁡πθ​(a∣s).\displaystyle\max_{\theta}\sum_{(s,a)\in\rho_{\mathcal{D}}}\ln\pi_{\theta}(a\mid s). (2)
ℒRPD​(θ)=ℒRL​(θ)−ℒMSE​(θ)\displaystyle\mathcal{L}_{\text{RPD}}(\theta)=\mathcal{L}_{\text{RL}}(\theta)-\mathcal{L}_{\text{MSE}}(\theta) (3)
ℒRL​(θ)=𝔼t​[min⁡(rt​(θ)​A^t,c⁡(rt​(θ),1−ϵ,1+ϵ)​A^t)]\displaystyle\mathcal{L}_{\text{RL}}(\theta)=\mathbb{E}_{t}\left[\min(r_{t}(\theta)\hat{A}_{t},c(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\right] (4)
ℒMSE​(θ)=𝔼t​[(μ⁡(πθ​(at∣st))−𝔼⁡[atVLA])2]\displaystyle\mathcal{L}_{\text{MSE}}(\theta)=\mathbb{E}_{t}\left[\left(\mu(\pi_{\theta}(a_{t}\mid s_{t}))-\mathbb{E}\left[a^{\text{VLA}}_{t}\right]\right)^{2}\right] (5)
θ∗=arg⁡maxθ​ℒRPD​(θ)\displaystyle\theta^{*}=\arg\max_{\theta}\mathcal{L}_{\text{RPD}}(\theta) (6)

III-B Tasks, Dataset, and VLA Fine-Tuning

Eight ManiSkill3 manipulation tasks use relative task-space commands. Six provide fine-tuning data; PullCubeTool and PokeCube are held out. Evaluation deliberately focuses on tasks intended to be solvable by PPO. Both teacher and student use the simulator’s “human camera,” which includes artificial visual goal markers. Published Octo and OpenVLA checkpoints have zero success on the tested setup, so the authors fine-tune them on simulator-provided expert demonstrations before distillation. Long demonstrations are excluded to match the episode horizon.

The held-out tasks have a decisive reward caveat: sparse success checks only whether the cube reaches its goal, allowing the robot to bypass the requested tool. Dense reward includes tool use. Consequently, high sparse success demonstrates pushing or pulling amid distracting objects, not successful transfer of tool-use knowledge or full language-instruction compliance.

TABLE I: Evaluation rewards (RW) and success rates (SR) of the VLAs on the selected ManiSkill3 tasks.
Task ID Language Instruction Fine-tuned openvla-base openvla-finetuned octo-base octo-finetuned
RW SR RW SR RW SR RW SR
LiftPegUpright-v1 “lift the peg upright” ✓ 0.20 0% 0.46 12% 0.20 0% 0.41 0%
PickCube-v1 “pick up the cube” ✓ 0.06 0% 0.18 6% 0.06 0% 0.17 9%
PullCube-v1 “pull the cube towards the robot base” ✓ 0.07 0% 0.27 92% 0.07 0% 0.24 90%
PullCubeTool-v1 “pull the cube by using the red tool” ✗ 0.10 0% 0.04 12% 0.10 0% 0.03 10%
PushCube-v1 “push the cube away from the robot base” ✓ 0.10 0% 0.12 27% 0.10 0% 0.16 67%
RollBall-v1 “push the ball” ✓ 0.02 0% 0.03 4% 0.02 0% 0.06 10%
StackCube-v1 “stack the red cube on the green cube” ✓ 0.05 0% 0.13 3% 0.05 0% 0.13 1%
PokeCube-v1 “push the cube by using the blue tool” ✗ 0.06 0% 0.09 12% 0.06 0% 0.08 12%
Equation 7
a=(Δ​x,Δ​ω,gripper)∈ℝ7a=(\Delta x,\Delta\omega,\text{gripper})\in\mathbb{R}^{7} (7)
Refer to caption
Fig. 2: Overview of the eight different tasks from ManiSkill3 that were used to distill expert policies with RPD. See Tab. I for the task names (tasks are depicted in row-major order).

III-C VLA Integration Details

A common interface maps image and language batches to relative motion and gripper commands. Separate model environments communicate through an HTTP service, with shared memory used locally to reduce image-serialization cost. The tested OpenVLA implementation lacks effective batched inference and requires substantial preprocessing and parallelization work. Even across four H100 GPUs it remains slower than Octo on one A40, limiting the OpenVLA experiments. Sample efficiency and wall-clock efficiency must therefore be distinguished.

IV Experimental Results

The experiments compare loss variants, six teacher-fine-tuning tasks under dense and sparse rewards, two held-out task variations, and a changed camera view. Most results use Octo; the larger OpenVLA teacher is evaluated only on a subset. Five-seed means and standard deviations are used for the main Octo curves; the OpenVLA PickCube result uses only one run.

IV-A RPD Variants and Baselines

All variants share the student architecture and largely retain ManiSkill3 PPO defaults, with batch size adjusted for teacher memory. On PickCube, mean-squared and absolute-error matching learn faster and more smoothly than PPO, reaching roughly 80% success. The likelihood-based variant remains close to teacher performance, consistent with its variance term restricting exploration. The OpenVLA-guided result is somewhat weaker than Octo-guided training.

The PPD comparison is the authors’ reimplementation, fitting a Gaussian to ten teacher samples to estimate KL divergence. It fails to improve substantially here. The proposed explanations—multimodal teacher actions, teacher quality, and hyperparameter sensitivity—are hypotheses, not isolated causal tests. This result should not be treated as a universal limitation of PPD.

Original figure
Fig. 3: Average success rates during training for vanilla PPO, PPD, and three different RPD variants: RPD-MSE, RPD-L1, and RPD-BC, which all distill from Octo over five runs with different seeds. Shaded areas indicate standard deviations. We also evaluated RPD-MSE on OpenVLA. Due to computing constraints, we could only perform a single training run. The performance of the fine-tuned VLAs is indicated by dashed lines.

IV-B RPD on Different Tasks

Across the six fine-tuning tasks, RPD generally learns faster and with less seed variation than the fixed-hyperparameter PPO baseline. It often reaches higher final success and obtains partial progress on RollBall and StackCube where that baseline struggles. These comparisons establish gains against the chosen common configuration, not against separately optimized PPO for each task.

Original figure
Fig. 4: Success and reward curves for RPD with Octo and for vanilla PPO for both dense and sparse rewards on the six different ManiSkill3 tasks that are part of the fine-tuning dataset. The values are averaged over five runs with different seeds and recorded in an evaluation environment. Standard deviations are indicated by shaded areas. All training runs were performed with the same hyperparameters from the ManiSkill3 PPO baseline. For all tasks, RPD outperforms Octo quickly and converges faster than vanilla PPO. In some cases, it even finds good policies when vanilla PPO fails. This effect increases for sparse rewards.
Original figure
Fig. 5: Success and reward curves for RPD with Octo and for vanilla PPO on tasks that are not part of Octo’s fine-tuning dataset. The curves show average results for five training runs on different seeds and are recorded in evaluation environments. The shaded areas indicate standard deviations. Note that the Octo baseline does not represent the correct reward in the dense reward setting as it solves the tasks without tool use. See Sec. III for details.

IV-C Sparse Rewards and Sample Efficiency

For sparse rewards, the discount factor is increased from 0.8 to 0.99 to retain learning signal from terminal outcomes. Teacher-guided exploration improves learning across the six tasks, while PPO often fails to discover successful behavior. Dense and sparse reward magnitudes are not directly comparable; success rates provide the shared outcome measure. The gains concern environment samples, while teacher inference adds computational work.

IV-D Generalization to New Environments

Generalization is tested by training a new student on task or camera variations using the existing teacher. This is adaptation through further reinforcement learning, not zero-shot transfer of the distilled student. Teacher success alone is an incomplete predictor: nearly useful trajectories can guide exploration even when the teacher rarely finishes.

Hold-Out Tasks

On PullCubeTool and PokeCube, RPD fails under dense rewards that require using the tool, because the teacher supplies inappropriate direct-push or direct-pull behavior. Under sparse rewards, that same behavior solves the relaxed goal-position criterion and RPD outperforms PPO. The positive result therefore concerns task variations with distractors; the negative result shows that irrelevant teacher guidance can hinder learning.

Refer to caption
Refer to caption
Fig. 6: The PushCube task with a variation of the camera perspective. The left side shows the perspective used for fine-tuning Octo. The right side depicts the perspective we used to test RPD’s performance in situations where the VLA needs to generalize from minor setup changes, a task where state-of-the-art VLAs fail.

Change in Camera Perspective

Changing the PushCube camera reduces teacher success from 67% to zero for Octo and from 27% to 4.5% for OpenVLA. Nevertheless, newly trained RPD students outperform PPO in the sparse-reward setting. The interpretation is that some teacher actions remain locally useful despite failure to complete episodes. The student’s visual encoder is trained anew in the changed view, so this does not show direct camera-invariant deployment of an existing student.

Original figure
Fig. 7: Average evaluation success rates of vanilla PPO and of RPD-MSE with both Octo and OpenVLA on a variation of the PushCube task with sparse rewards over five different seeds. Shaded areas indicate standard deviations. The camera perspective was changed as shown in Fig. 6 to investigate the distillation performance of the degraded VLAs.

IV-E Limitations

RPD depends on teacher actions providing a useful bias. Unseen skills or strong domain shift can produce negative transfer, as in dense-reward tool use. Benefits may shrink for easy tasks or well-tuned PPO. Training still requires millions of interactions for harder tasks, making direct physical learning difficult; the paper does not demonstrate real-robot deployment.

V Conclusion and Future Work

Action-mean guidance is a small modification that can turn a weak generalist teacher into a useful exploration aid for a compact task expert. The simulation results support this under the tested rewards and domains, with explicit failures when guidance is misleading. Proposed extensions include transferring students to physical robots and using their generated trajectories to improve future generalist models.

Acknowledgement

The authors acknowledge GeniusRobot support from the German Federal Ministry of Education and Research and computing resources from the Erlangen National HPC Center under the BayernKI project.

Concept demonstration

Open image ↗