At a glance
Refined Policy Distillation trains a compact task-specific policy with reinforcement learning while using a vision-language-action teacher to guide its actions. An action-matching term augments PPO, improving exploration while permitting the student to exceed the teacher. Simulation experiments with fine-tuned Octo and OpenVLA show faster learning on several manipulation tasks, including sparse rewards and a changed camera view. Benefits depend on useful teacher actions; the held-out tool-use tasks expose negative transfer when that condition fails.
I Introduction
Large generalist policies offer transferable knowledge but can be slow and brittle under setup changes. Task-specific reinforcement learning can improve performance through interaction, yet exploration is expensive. RPD connects these approaches by querying the teacher at student-visited states and combining its guidance with environment reward. The result is a specialized student, not a refined generalist VLA. All evaluations are simulated; real deployment and sim-to-real transfer remain future work.
II Related Work
Earlier policy distillation transfers behavior between reinforcement-learning agents, while behavioral cloning supplies expert supervision from trajectories. RPD instead queries generalist Octo [4] or OpenVLA [7] during on-policy learning. Compared with Proximal Policy Distillation [23], its mean-squared action loss avoids estimating the full teacher distribution. Compared with collecting expert RL data to train a VLA, the direction of transfer is reversed: the VLA helps train the smaller expert.
III Methodology
The student acts in the environment and receives ordinary reinforcement-learning rewards. At the same visited states, the teacher supplies an action target. Matching that target biases exploration toward plausible behavior; maximizing return allows the student to depart from the teacher where interaction reveals a better solution.
III-A Refined Policy Distillation
RPD augments PPO’s maximized clipped objective with a negative mean-squared error between the student action mean and the expected teacher action. Under the paper’s fixed-variance Gaussian approximation, this corresponds to the action-mean part of behavioral cloning without the variance-shrinking component. In practice, one sampled teacher action estimates the expectation at each step; larger sample counts were not evaluated because of inference cost. The study also tests an absolute-error variant and a likelihood-based behavioral-cloning variant.
Equations 1–6
| (1) |
| (2) |
| (3) |
| (4) |
| (5) |
| (6) |
III-B Tasks, Dataset, and VLA Fine-Tuning
Eight ManiSkill3 manipulation tasks use relative task-space commands. Six provide fine-tuning data; PullCubeTool and PokeCube are held out. Evaluation deliberately focuses on tasks intended to be solvable by PPO. Both teacher and student use the simulator’s “human camera,” which includes artificial visual goal markers. Published Octo and OpenVLA checkpoints have zero success on the tested setup, so the authors fine-tune them on simulator-provided expert demonstrations before distillation. Long demonstrations are excluded to match the episode horizon.
The held-out tasks have a decisive reward caveat: sparse success checks only whether the cube reaches its goal, allowing the robot to bypass the requested tool. Dense reward includes tool use. Consequently, high sparse success demonstrates pushing or pulling amid distracting objects, not successful transfer of tool-use knowledge or full language-instruction compliance.
| Task ID | Language Instruction | Fine-tuned | openvla-base | openvla-finetuned | octo-base | octo-finetuned | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| RW | SR | RW | SR | RW | SR | RW | SR | |||
| LiftPegUpright-v1 | “lift the peg upright” | ✓ | 0.20 | 0% | 0.46 | 12% | 0.20 | 0% | 0.41 | 0% |
| PickCube-v1 | “pick up the cube” | ✓ | 0.06 | 0% | 0.18 | 6% | 0.06 | 0% | 0.17 | 9% |
| PullCube-v1 | “pull the cube towards the robot base” | ✓ | 0.07 | 0% | 0.27 | 92% | 0.07 | 0% | 0.24 | 90% |
| PullCubeTool-v1 | “pull the cube by using the red tool” | ✗ | 0.10 | 0% | 0.04 | 12% | 0.10 | 0% | 0.03 | 10% |
| PushCube-v1 | “push the cube away from the robot base” | ✓ | 0.10 | 0% | 0.12 | 27% | 0.10 | 0% | 0.16 | 67% |
| RollBall-v1 | “push the ball” | ✓ | 0.02 | 0% | 0.03 | 4% | 0.02 | 0% | 0.06 | 10% |
| StackCube-v1 | “stack the red cube on the green cube” | ✓ | 0.05 | 0% | 0.13 | 3% | 0.05 | 0% | 0.13 | 1% |
| PokeCube-v1 | “push the cube by using the blue tool” | ✗ | 0.06 | 0% | 0.09 | 12% | 0.06 | 0% | 0.08 | 12% |
Equation 7
| (7) |
III-C VLA Integration Details
A common interface maps image and language batches to relative motion and gripper commands. Separate model environments communicate through an HTTP service, with shared memory used locally to reduce image-serialization cost. The tested OpenVLA implementation lacks effective batched inference and requires substantial preprocessing and parallelization work. Even across four H100 GPUs it remains slower than Octo on one A40, limiting the OpenVLA experiments. Sample efficiency and wall-clock efficiency must therefore be distinguished.
IV Experimental Results
The experiments compare loss variants, six teacher-fine-tuning tasks under dense and sparse rewards, two held-out task variations, and a changed camera view. Most results use Octo; the larger OpenVLA teacher is evaluated only on a subset. Five-seed means and standard deviations are used for the main Octo curves; the OpenVLA PickCube result uses only one run.
IV-A RPD Variants and Baselines
All variants share the student architecture and largely retain ManiSkill3 PPO defaults, with batch size adjusted for teacher memory. On PickCube, mean-squared and absolute-error matching learn faster and more smoothly than PPO, reaching roughly 80% success. The likelihood-based variant remains close to teacher performance, consistent with its variance term restricting exploration. The OpenVLA-guided result is somewhat weaker than Octo-guided training.
The PPD comparison is the authors’ reimplementation, fitting a Gaussian to ten teacher samples to estimate KL divergence. It fails to improve substantially here. The proposed explanations—multimodal teacher actions, teacher quality, and hyperparameter sensitivity—are hypotheses, not isolated causal tests. This result should not be treated as a universal limitation of PPD.
IV-B RPD on Different Tasks
Across the six fine-tuning tasks, RPD generally learns faster and with less seed variation than the fixed-hyperparameter PPO baseline. It often reaches higher final success and obtains partial progress on RollBall and StackCube where that baseline struggles. These comparisons establish gains against the chosen common configuration, not against separately optimized PPO for each task.
IV-C Sparse Rewards and Sample Efficiency
For sparse rewards, the discount factor is increased from 0.8 to 0.99 to retain learning signal from terminal outcomes. Teacher-guided exploration improves learning across the six tasks, while PPO often fails to discover successful behavior. Dense and sparse reward magnitudes are not directly comparable; success rates provide the shared outcome measure. The gains concern environment samples, while teacher inference adds computational work.
IV-D Generalization to New Environments
Generalization is tested by training a new student on task or camera variations using the existing teacher. This is adaptation through further reinforcement learning, not zero-shot transfer of the distilled student. Teacher success alone is an incomplete predictor: nearly useful trajectories can guide exploration even when the teacher rarely finishes.
Hold-Out Tasks
On PullCubeTool and PokeCube, RPD fails under dense rewards that require using the tool, because the teacher supplies inappropriate direct-push or direct-pull behavior. Under sparse rewards, that same behavior solves the relaxed goal-position criterion and RPD outperforms PPO. The positive result therefore concerns task variations with distractors; the negative result shows that irrelevant teacher guidance can hinder learning.


Change in Camera Perspective
Changing the PushCube camera reduces teacher success from 67% to zero for Octo and from 27% to 4.5% for OpenVLA. Nevertheless, newly trained RPD students outperform PPO in the sparse-reward setting. The interpretation is that some teacher actions remain locally useful despite failure to complete episodes. The student’s visual encoder is trained anew in the changed view, so this does not show direct camera-invariant deployment of an existing student.
IV-E Limitations
RPD depends on teacher actions providing a useful bias. Unseen skills or strong domain shift can produce negative transfer, as in dense-reward tool use. Benefits may shrink for easy tasks or well-tuned PPO. Training still requires millions of interactions for harder tasks, making direct physical learning difficult; the paper does not demonstrate real-robot deployment.
V Conclusion and Future Work
Action-mean guidance is a small modification that can turn a weak generalist teacher into a useful exploration aid for a compact task expert. The simulation results support this under the tested rewards and domains, with explicit failures when guidance is misleading. Proposed extensions include transferring students to physical robots and using their generated trajectories to improve future generalist models.
Acknowledgement
The authors acknowledge GeniusRobot support from the German Federal Ministry of Education and Research and computing resources from the Erlangen National HPC Center under the BayernKI project.