Literature
Display
← All papers

Flow Policy Gradients for Robot Control

Brent Yi · Hongsuk Choi · Himanshu Gaurav Singh · Xiaoyu Huang · Takara E. Truong · Carmelo Sferrazza · Yi Ma · Rocky Duan · Pieter Abbeel · Guanya Shi · Karen Liu · Angjoo Kanazawa

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

FPO++ adapts flow-matching policy gradients to robot control, with independently clipped noise-sample ratios and a reward-dependent choice of trust-region objective.

Refer to caption
Figure 1: Flow policies for robot control. We show how robust control policies for quadrupeds, humanoids, and manipulators can be trained with reinforcement learning.
Refer to caption
Figure A.1: Legged robots used for experiments. We train policies for the Go2, Spot, H1, and G1 robots in simulation. We deploy policies to physical G1 and Booster T1 robots.
Refer to caption
(a) Can
Refer to caption
(b) Square
Refer to caption
(c) Box Cleanup
Refer to caption
(d) Tray Lift
Refer to caption
(e) Threading

RoboMimic DexMimicGen

Figure A.2: Manipulation tasks. We compare FPO++ to baseline methods on manipulation tasks from RoboMimic [24] and DexMimicGen [43]. We choose tasks that cover diverse embodiments and control regimes: from single-arm to bimanual manipulation, parallel-jaw grippers and dexterous hands, and both short- and long-horizon tasks.

At a glance

FPO++ trains expressive flow policies from rewards without computing action likelihoods. Per-noise-sample ratios improve stability; an asymmetric trust region further helps dense-reward locomotion and motion tracking. Experiments include four simulated legged platforms, physical deployment on two humanoids, and simulated manipulation finetuning. The asymmetric variant is detrimental in the tested manipulation regimes, tuned Gaussian policies remain stronger on some tracking comparisons, and flow training costs more wall-clock time.

1 Introduction

Flow policies can express correlated or multimodal action distributions but make direct likelihood evaluation expensive. This work builds on flow-matching policy gradients to support locomotion, motion tracking, and reward finetuning of demonstration-trained manipulators. Its contribution is an empirically more reliable surrogate update, accompanied by physical transfer and analysis of exploration. It is not a general proof that the surrogate equals an exact action-likelihood policy gradient.

2 Background and Related Work

PPO [2] weights advantages by new/old action-likelihood ratios and clips updates. Flow and diffusion policies are useful in imitation learning but require extra work to compute such likelihoods. DPPO [7] and ReinFlow [8] instead optimize stochastic denoising-trajectory likelihoods.

Flow policy optimization uses changes in conditional flow-matching error as a surrogate ratio. It does not differentiate through environment transitions or unroll the entire sampling process during the policy update. This changes the training objective and its approximations; it does not supply an exact marginalized action density.

Equations 1–2
ψPPO​(ρθ,A^t)=min⁡(ρθ​A^t,clip​(ρθ,1±εclip)​A^t).\psi_{\text{PPO}}\!\left(\rho_{\theta},\hat{A}_{t}\right)=\min\!\left(\rho_{\theta}\hat{A}_{t},\;\text{clip}(\rho_{\theta},1\pm\varepsilon^{\text{clip}})\hat{A}_{t}\right). (1)
maxθ⁡𝔼πθold​[ψPPO​(ρθ,A^t)];ρθ=πθ​(at∣ot)πθold​(at∣ot).\max_{\theta}\;\;\mathbb{E}_{\pi_{\theta_{\text{old}}}}\left[\psi_{\text{PPO}}\!\left(\rho_{\theta},\hat{A}_{t}\right)\right];\quad\rho_{\theta}=\frac{\pi_{\theta}(a_{t}\mid o_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid o_{t})}. (2)

3 FPO++

FPO++ modifies FPO [13] in two ways: it computes and clips a separate surrogate ratio for each sampled noise/time pair, and it can penalize excessive changes on negative-advantage samples with an asymmetric objective. The latter is selected by task/reward regime.

3.1 Preliminaries

FPO estimates conditional flow-matching loss from Gaussian noise and interpolation times for each observed action. The velocity target is action minus noise along the linear interpolation. Exponentiating the old-minus-new loss approximates a likelihood ratio, and an advantage-weighted clipped objective encourages better actions. The Monte Carlo loss samples used for updates are distinct from Euler integration steps used to generate actions.

Equations 3–7
ρ^FPO​(θ)=exp⁡(ℒ^CFM,θold​(at,ot)−ℒ^CFM,θ​(at,ot)),\hat{\rho}_{\text{FPO}}(\theta)=\exp\!\Big(\hat{\mathcal{L}}_{\text{CFM},\theta_{\text{old}}}(a_{t};o_{t})-\hat{\mathcal{L}}_{\text{CFM},\theta}(a_{t};o_{t})\Big), (3)
maxθ⁡𝔼πθold​[ψPPO​(ρ^FPO​(θ),A^t)].\max_{\theta}\;\;\mathbb{E}_{\pi_{\theta_{\text{old}}}}\left[\psi_{\text{PPO}}\!\left(\hat{\rho}_{\text{FPO}}(\theta),\hat{A}_{t}\right)\right]. (4)
atτi\displaystyle a_{t}^{\tau_{i}} =τi​at+(1−τi)​ϵi.\displaystyle=\tau_{i}a_{t}+(1-\tau_{i})\epsilon_{i}. (5)
ℒ^CFM,θ​(at,ot)\displaystyle\hat{\mathcal{L}}_{\text{CFM},\theta}(a_{t};o_{t}) =1Nmc​∑iNmcℓθ(i,t)\displaystyle=\frac{1}{N_{\text{mc}}}\sum_{i}^{N_{\text{mc}}}\ell_{\theta}^{(i,t)} (6)
ℓθ(i,t)=ℓθ​(at,τi,ϵi,ot)\displaystyle\ell_{\theta}^{(i,t)}=\ell_{\theta}(a_{t},\tau_{i},\epsilon_{i};o_{t}) =‖v^θ​(atτi,τi,ot)−(at−ϵi)‖22.\displaystyle=\left\|\hat{v}_{\theta}\!\left(a_{t}^{\tau_{i}},\tau_{i};o_{t}\right)-(a_{t}-\epsilon_{i})\right\|^{2}_{2}. (7)

3.2 FPO++

Original FPO exponentiates the average loss difference, creating one ratio per action and clipping all its noise/time samples together. FPO++ exponentiates each sample’s loss difference separately, sharing the action’s advantage but allowing independent clipping. At the unchanged policy, both ratios are one and their gradients agree; repeated optimization steps produce different updates.

For nonnegative advantages, ASPO uses PPO clipping. For negative advantages, it uses the quadratic penalty from Simple Policy Optimization [33], which pulls ratios back rather than merely zeroing their gradients beyond a boundary. This discourages aggressive increases in flow-matching loss. The trust-region interpretation is based on a surrogate and variational-gap argument, not a measured exact policy KL constraint.

Equations 8–10
ρ^FPO​(θ)=exp⁡(1Nmc​∑i=1Nmc(ℓθold(i,t)−ℓθ(i,t))).\hat{\rho}_{\text{FPO}}(\theta)=\exp\!\left(\tfrac{1}{N_{\text{mc}}}\sum_{i=1}^{N_{\text{mc}}}\bigg(\ell_{\theta_{\text{old}}}^{(i,t)}-\ell_{\theta}^{(i,t)}\bigg)\right). (8)
ρ^FPO++(i)​(θ)=exp⁡(ℓθold(i,t)−ℓθ(i,t)).\hat{\rho}_{\text{FPO++}}^{(i)}(\theta)=\exp\!\left(\ell_{\theta_{\text{old}}}^{(i,t)}-\ell_{\theta}^{(i,t)}\right). (9)
ψSPO​(ρθ,A^t)=ρθ​A^t−|A^t|2​εclip​(ρθ−1)2.\psi_{\text{SPO}}\!\left(\rho_{\theta},\hat{A}_{t}\right)=\rho_{\theta}\,\hat{A}_{t}\;-\;\frac{|\hat{A}_{t}|}{2\,\varepsilon^{\mathrm{clip}}}\,\big(\rho_{\theta}-1\big)^{2}. (10)

3.3 FPO++ Objective

Each action receives several independently sampled noise/time pairs and a common estimated advantage. The full objective sums the per-sample ASPO contributions, then averages over on-policy observations and actions. All original formulas are retained. Manipulation experiments replace ASPO with ordinary PPO clipping while keeping the per-sample ratios, so their reported FPO++ is a task-specific configuration.

Equations 11–12
ψASPO​(ρθ,A^t)={ψPPO​(ρθ,A^t),A^t≥0,ψSPO​(ρθ,A^t),A^t<0.\psi_{\text{ASPO}}\!\left(\rho_{\theta},\hat{A}_{t}\right)=\begin{cases}\psi_{\text{PPO}}\!\big(\rho_{\theta},\hat{A}_{t}\big),&\hat{A}_{t}\geq 0,\\[2.0pt] \psi_{\text{SPO}}\!\big(\rho_{\theta},\hat{A}_{t}\big),&\hat{A}_{t}<0.\end{cases} (11)
maxθ⁡𝔼πθold​[∑i=1NmcψASPO​(ρ^FPO++(i)​(θ),A^t)].\max_{\theta}\;\;\mathbb{E}_{\pi_{\theta_{\text{old}}}}\left[\sum_{i=1}^{N_{\text{mc}}}\;\psi_{\text{ASPO}}\!\left(\hat{\rho}^{(i)}_{\text{FPO++}}(\theta),\hat{A}_{t}\right)\right]. (12)

4 Experiments

The experiments cover learning from scratch, transfer of trained controllers to hardware, finetuning image-conditioned policies, and ablations against flow and Gaussian baselines. Return, task success, and wall-clock cost are separate outcomes, and the appendix documents cases where they disagree.

4.1 Locomotion Benchmarks

IsaacLab velocity-tracking tasks use Go2 and Spot quadrupeds plus H1 and G1 humanoids, with proprioception and commanded linear/angular velocities. Vanilla FPO often collapses or stalls despite sweeps over learning rate, clipping, and Monte Carlo count. FPO++ learns and maintains stronger returns across five seeds. This supports stability in these environments rather than a guarantee across arbitrary high-dimensional control tasks.

Original figure
Figure 2: Improved stability in IsaacLab locomotion environments. We compare episode returns from FPO++ training with FPO; FPO++ curves show the mean over 5 random seeds. For thoroughness and to ensure we are not comparing against a bad set of hyperparameters for the FPO baseline, we include FPO runs for many different hyperparameter choices: all combinations of learning rate ∈{10−5,10−4,3×10−4}\in\{10^{-5},10^{-4},3\times 10^{-4}\}, clip parameter ∈{0.04,0.05,0.06}\in\{0.04,0.05,0.06\}, and Monte Carlo samples ∈{8,16,32}\in\{8,16,32\}. FPO++ addresses stability problems that we were unable to solve by tuning FPO hyperparameters.

Refer to caption

(a) T1 locomotion

Refer to caption

(b) G1 motion tracking

Refer to caption

(c) G1 robustness test
Figure 3: Sim-to-real transfer. We deploy flow policies for locomotion to a Booster T1 and motion tracking to a Unitree G1. Policies are directly deployed to real robots with reduced sampling step counts, demonstrating stable gaits, tracking for long sequences, and robustness to external forces. The arrows in T1 locomotion (Figure 3a) indicate velocity commands, while the arrows in the robustness tests (Figure 3c) highlight external forces.

4.2 Humanoid Sim-to-real

Booster T1 executes a learned locomotion controller, and Unitree G1 executes motion-tracking policies trained with the BeyondMimic protocol [4]. Six reference sequences cover dancing, walking, running, fighting motions, and jumping, with one policy per sequence and roughly two-and-a-half-minute average duration. Hardware sequences and perturbation examples demonstrate transfer using reduced integration steps. The paper does not report a large repeated hardware-trial success-rate study.

Original figure
(a) Per-sample ratios improve locomotion policies.
Original figure
(b) ASPO produces higher, more consistent returns.
Figure 4: FPO++ ablations on locomotion. We plot final training and evaluation returns across many runs; each point is a training run with clipping parameter ∈{0.04,0.05,0.06}\in\{0.04,0.05,0.06\} and random seed ∈{0,1,2,3,4}\in\{0,1,2,3,4\}. (a) compares FPO++ with per-sample ratios (Equation 9) against per-action ratios from prior work (Equation 8); per-sample ratios produce higher and more consistent returns. (b) compares trust region implementations: standard PPO clipping, SPO [33], and our asymmetric ASPO objective.
Original figure
Figure 5: Manipulation finetuning results. We compare the evaluation success rates for FPO++, FPO, and our two DPPO implementations. We show two rows: the first row contains policy success rates with zero-sampling (ϵ=0→\epsilon=\vec{0}), while the second row contains policy success rates using standard random sampling (ϵ∼𝒩⁡(0,I)\epsilon\sim\mathcal{N}(0,I)). For each task, all algorithms are initialized from the same base policy, which is an image-based flow matching policy trained to predict action chunks.

4.3 Manipulation Finetuning

Five RoboMimic/DexMimicGen tasks use image-based demonstration pretraining followed by reward finetuning. Main comparisons initialize methods from the same base flow policy and include fixed-noise and learned-noise DPPO adaptations. Per-sample FPO++ with PPO clipping reaches strong success and generally learns faster. Vanilla FPO can also work well from a pretrained policy. Failures of the adapted DPPO variants depend strongly on exploratory base-policy quality and should be read alongside the original-code comparisons in the appendix.

4.4 Training Ablations

Locomotion ablations combine five seeds with several clip values. Removing per-sample ratios or replacing ASPO reduces average returns and increases variability. Gradient-alignment statistics and flow-field visualizations are consistent with more stable updates and broader action distributions.

ASPO is not uniformly beneficial. With randomly initialized Lift action networks and a frozen pretrained visual encoder, sparse-reward final success is 0.07 for ASPO versus 0.73 for PPO; dense-reward success is 0.28 versus 0.81. Dense ASPO learns to grasp and hover for shaping reward without completing the task. These ten-seed tests implicate reward structure, not simply whether the action policy was pretrained. The paper recommends ASPO for well-aligned dense rewards and PPO clipping for sparse or exploitable shaping regimes.

4.5 Comparison with Gaussian PPO

Across varied parallel-environment counts, FPO++ usually reaches stronger locomotion returns than the tested Gaussian PPO configurations. At a fixed observation, ten thousand sampled actions reveal learned cross-joint correlations, which a diagonal Gaussian cannot represent. Different algorithms also produce different gaits under the same task rewards; Spot’s rewards already include gait-coordination terms, so these are not wholly unshaped gait discoveries.

Tuned Gaussian PPO achieves higher motion-tracking returns when entropy regularization and adaptive learning rates are enabled. On an L40S, G1 locomotion reaches return twenty-five in nineteen minutes for Gaussian PPO versus twenty-three for FPO++; motion tracking can take up to twice as long. Expressiveness and sample efficiency therefore do not imply wall-clock superiority or a universally preferable controller.

Refer to caption
Figure 6: Correlations before and during training. We sample 10,000 actions conditioned on the same observation, and visualize the correlations between action dimensions before and during FPO++ training for an H1 locomotion policy.

5 Conclusion

The work demonstrates practical likelihood-free surrogate updates for flow policies, including humanoid transfer and manipulation finetuning. Per-sample ratios are broadly useful in the tested settings; ASPO requires reward-aware configuration. Future work must reduce computational cost and improve flow-policy regularization/adaptation, especially where a tuned Gaussian baseline already suffices.

Acknowledgments

The acknowledgments thank David McAllister, Arthur Allshire, and Chungmin Kim for discussions.

Appendix A Further related work

Related offline methods use advantage-weighted regression, value-based objectives regularized by generative losses, entropy bonuses, one-step flow policies, score matching, or adjoint matching. These differ from the paper’s online on-policy setting and address distinct challenges in propagating value gradients through generative policies.

Appendix B Experiment Details

Implementation details specify robot embodiments, reward structures, networks, sampling, domain randomization, and manipulation controls. All robot/task figures and hyperparameter tables are retained.

B.1 Locomotion Benchmarking

The locomotion benchmark follows IsaacLab velocity-command tasks, with common training settings and robot-specific rewards. Baselines receive explicit clipping sweeps.

B.1.1 Hyperparameters

Locomotion actor and critic networks have three layers of 256 and 768 hidden units, respectively. Default training uses 4,096 environments and twenty-four steps between updates. Flow actions use sixty-four Euler steps and updates use sixteen Monte Carlo samples per action. Quadrupeds train for 1,500 updates and humanoids for 2,000. Reducing integration steps to eight or sixteen harms training stability; rollout inference is not the dominant training cost in these small-network settings.

B.1.2 Environments and Rewards

Shared rewards track planar velocity and yaw while rewarding foot air time and penalizing excessive motion changes. Robot-specific penalties cover torques, contacts, body motion, and posture. Spot explicitly rewards diagonal-foot coordination and clearance; humanoids include fall penalties and biped air-time terms. Differences in learned gaits therefore arise within specified reward shaping rather than an absence of gait incentives.

B.2 Motion Tracking

The G1 tracking controller has twenty-nine degrees of freedom and runs at fifty hertz, using a five-millisecond simulator timestep with four-step decimation. Rewards, observations, and termination follow the referenced tracking protocol.

Hyperparameter category Used value Sweep range and notes
FPO++
Flow integration steps 6464 {8,16,32,64}\{8,16,32,64\}
Network output uu {u,x0}\{u,x_{0}\}, x0x_{0} denotes data
Samples per action 1616 {8,16,32}\{8,16,32\}
Training
Learning rate 1×10−41\times 10^{-4} 1×10−51\times 10^{-5}, 1×10−41\times 10^{-4}, 3×10−43\times 10^{-4}
Weight decay / Adam betas 1×10−41\times 10^{-4} / (0.9,0.95)(0.9,0.95) AdamW parameters
Clip parameter 0.050.05 {0.03,0.04,0.05,0.06}\{0.03,0.04,0.05,0.06\}
Discount factor (γ\gamma) 0.990.99
GAE lambda (λ\lambda) 0.950.95
Learning epochs 1616 for Go2, others 3232
Minibatches per update 44
Running observation normalization Yes
Table A.1: FPO++ and training hyperparameters used for locomotion. We report both the final values we use for experiments and the values we performed sweeps over.
Hyperparameter category Used value Sweep range and notes
FPO++
Flow integration steps 5050 {10,50}\{10,50\}
Network output uu {u,x0}\{u,x_{0}\}, x0x_{0} denotes data
Samples per action 1616 {8,16,32}\{8,16,32\}
Training
Learning rate 3×10−43\times 10^{-4}
Weight decay / Adam betas 1×10−41\times 10^{-4} / (0.9,0.95)(0.9,0.95) AdamW parameters
Clip parameter 0.010.01 {0.01,0.05,0.1}\{0.01,0.05,0.1\}
Discount factor (γ\gamma) 0.990.99
GAE lambda (λ\lambda) 0.950.95
Learning epochs 55
Minibatches per update 44
Running observation normalization Yes
Table A.2: FPO++ and training hyperparameters used for motion tracking. We report both the final values we use for experiments and the values we performed sweeps over.

B.2.1 Domain Randomization

Randomization varies friction, restitution, joint defaults, torso center of mass, initial pose, and command delay. Perturbations every two to three seconds vary translational and angular velocities. Command delay spans zero to two simulation steps, or zero to ten milliseconds. These specified perturbations support transfer, without establishing robustness to all contacts or hardware variation.

B.2.2 Training Hyperparameters

Tracking training uses 4,096 environments, ninety-six rollout steps per update, and three-layer actor/critic networks with 1,024/512/256 widths. Policies train with fifty flow steps but deploy using five steps and deterministic zero-noise initialization. T1 locomotion uses the locomotion benchmark settings rather than this larger tracking configuration.

B.2.3 CFM loss implementation details

Exponentiating squared-loss differences can be unstable for large noise samples. The final implementation first clamps old and new conditional flow-matching losses, then clamps their difference before exponentiation. Experiments also test Huber losses and gradient-preserving clamping. These numerical safeguards are implementation details needed alongside the mathematical objective.

Original figure
(a) Without CFM loss clamp.
Original figure
(b) With CFM loss clamp.
Figure A.3: Clamping CFM losses improves stability. The x-axis varies the Huber threshold δ\delta; “Off” uses squared CFM loss without Huber. Loss-difference clamping is enabled throughout: orange uses naive clamping, and blue preserves gradients through the clamp. Panels (a) and (b) respectively disable and enable clamping of the old- and new-policy CFM losses before subtraction. Bars show mean final training return with standard error over five seeds.

B.3 Manipulation finetuning

Manipulation uses actor/critic learning rates of 10⁻⁵/10⁻⁴ and advantage-estimation lambda 0.99. Discounts vary from 0.99 to 0.999 by task. Policies use ten flow steps, eight Monte Carlo samples per action chunk, and sixteen-step chunks, summing conditional-flow losses over chunk timesteps. Main DPPO variants adopt velocity prediction to share base policies and are tuned over clipping and exploration settings; they are adaptations, not identical copies of the original implementations.

Refer to caption
Figure A.4: Motion tracking training curves. Vanilla FPO fails in learning to track the motion, while FPO++ trains stably. Hyperparameters are shared between both runs.

Appendix C More experiments

Additional experiments examine environment scaling, emergent gaits, tracking baselines, original-code manipulation comparisons, base-policy sampling, and reward-dependent trust-region behavior.

C.1 Sample efficiency and emergent gaits versus Gaussian PPO

Locomotion comparisons sweep 256 through 4,096 parallel environments at fixed update counts, changing both minibatch size and total environment experience. Curves include five-seed dispersion. Gait examples compare controllers at the same one-meter-per-second command and reward definition.

Original figure
Figure A.5: Sample efficiency for FPO++ and our Gaussian PPO baseline. We compare policy returns between Gaussian PPO and FPO++ with environment counts ∈{28,29,210,211,212}\in\{2^{8},2^{9},2^{10},2^{11},2^{12}\} after 1500 updates for quadrupeds and 2000 updates for humanoids. Minibatch size per policy update increases linearly with the number of parallel environments; other hyperparameters are fixed between runs. Filled regions show standard deviation computed over 5 seeds.
Refer to caption
Refer to caption
Figure A.6: Different gaits from different algorithms. Rollouts using Gaussian PPO (top) and FPO++ (bottom) for 1500 policy updates. Policies are trained with the same rewards, 4096 parallel environments, and given the same forward velocity command (1 m/s). We found differences reproducible across seeds, clip parameters, and learning rates. We use Viser [67] for visualization.

C.2 Comparison with vanilla FPO for motion tracking

A motion-tracking comparison shows vanilla FPO improving initially, then collapsing with spikes in value and surrogate losses. FPO++ remains stable in the displayed training run and produces the deployed policy. This is a concrete instability example, not proof that every original-FPO configuration must fail.

C.3 Comparison with Gaussian PPO for motion tracking

Without entropy bonuses or adaptive learning rates, FPO++ beats the simplified Gaussian tracking baseline. With both additions, Gaussian PPO attains higher return while episode lengths are similar. The experimental flow variant estimates entropy from nearest-neighbor action distances and adjusts learning rate using a noise-prediction-change statistic. That statistic is a heuristic policy-change proxy, not an exact KL divergence, and these additions are separate from the core objective.

Refer to caption
(a) Without entropy regularization or adaptive learning rate.
Refer to caption
(b) With entropy regularization and adaptive learning rate.
Figure A.7: Training curves compared to Gaussian PPO for motion tracking. Blue denotes FPO++ and orange Gaussian PPO. Without entropy regularization or adaptive learning rates, FPO++ achieves higher final returns and longer episodes. With both components enabled, Gaussian PPO attains higher returns, while episode lengths are similar.

C.4 Further comparisons with baselines in manipulation finetuning

Original DPPO and ReinFlow code is compared under four-step chunks, with corresponding FPO variants trained in the adapted pipeline. These methods use different pretrained policies: Can starts near 74–76% success, while Square starts at 28.62% for FPO variants versus 36.50–37.60% for the alternatives. Original baselines use one hundred denoising/flow steps versus ten for FPO. FPO++ reaches the highest reported final success, but the comparison is not identical-base-policy evidence. Each checkpoint evaluates two hundred episodes across fifty environments.

Original figure
Figure A.8: Additional Manipulation Comparison. Finetuning success rates on the RoboMimic Can and Square tasks for flow-based RL methods. All methods start from pretrained behavior-cloning base policies; FPO++ and vanilla FPO share the same base policy, while DPPO and ReinFlow use architectures and implementation from their original papers. Please refer to the base policy setting written in the text for further details. FPO++ learns fastest and attains the highest final success on both tasks, while vanilla FPO, DPPO, and ReinFlow remain stable but underperform relative to FPO++.
Original figure
Figure A.9: Analysis of base policy quality in RoboMimic Can. This figure illustrates how the initial success rate of the base policy impacts different finetuning methods. The top row shows results for a base policy with 71.35% success under zero-sampling versus 10.00% under random sampling, while the bottom row depicts a higher-quality base policy (96.11% zero-sampling / 64.36% random sampling), demonstrating that DPPO variants are significantly more sensitive to the base policy quality.
Original figure
Figure A.10: Ablation study in manipulation tasks. Blue denotes per-sample ratios with PPO, light gray per-action ratios with PPO, and dark gray per-sample ratios with ASPO. Results are averaged over three training seeds per configuration, with 200 evaluation episodes per checkpoint per seed. Per-sample PPO achieves the highest final mean success on both Square and Threading. ASPO underperforms per-sample PPO in both tasks, despite its benefits for locomotion.

C.5 Detailed analysis on the RoboMimic Can experiment

A Can base policy scores 71.35% with deterministic zero-sampling but only 10.00% with random initial noise. Low exploratory success produces few useful terminal rewards, and added denoising noise can worsen this. A stronger base policy scores 96.11%/64.36% under zero/random sampling, and DPPO variants then finetune successfully. This qualifies the main failure examples and separates deterministic evaluation quality from training exploration quality.

C.6 Ablation study for manipulation finetuning

Square and Threading ablations average three training seeds and two hundred evaluation episodes per checkpoint per seed. Per-sample ratios with PPO clipping perform best. ASPO lowers final success despite its locomotion benefits, motivating the additional from-scratch reward tests.

C.7 ASPO under sparse and dense rewards

Lift comparisons use ten seeds, two million steps for sparse rewards or one million for dense rewards, and one hundred evaluation episodes per seed/checkpoint. Only action networks start randomly; the visual encoder remains pretrained and frozen. PPO clipping wins under both reward schemes even after separate tuning. Dense ASPO’s high shaping return alongside low completion demonstrates why return alone cannot establish task success.

Original figure
Figure A.11: ASPO sensitivity to the reward regime in Lift. PPO and ASPO each use their best-tuned configuration for each reward setting, evaluated over ten training seeds. Success uses zero-sampling and the task-completion criterion.

Appendix D Analysis of training dynamics

Training-dynamics analysis examines gradient alignment, conditional action-distribution shape, and deterministic evaluation. These measurements help interpret empirical stability but do not establish convergence guarantees.

D.1 Empirical gradient variance

For each update, the analysis compares individual gradients with their mean using cosine similarity. Per-sample ratios and ASPO yield more aligned gradients, especially later in training. This supports a reduction in directional disagreement; cosine similarity is a proxy rather than a direct complete estimator of gradient variance.

Original figure
Figure A.12: Algorithmic updates in FPO++ reduce gradient variance. We observe that the per-sample ratio and ASPO trust region result in higher cosine similarity between gradients computed within each policy update, especially during the second half of training. Averages and standard deviations are reported over 5 seeds.

D.2 Entropy and full-action-space flow fields

Flow trajectories for all nineteen H1 action dimensions show narrowing distributions during a PPO-clipped policy collapse and broader distributions for ASPO at intermediate and final checkpoints. Visualizations transport ten thousand initial noises with eight Euler steps, distinct from the sixty-four-step training rollouts. The retained single-joint and full-action-space panels show an association between distribution collapse and performance in these runs, not a universal entropy guarantee.

Refer to caption
Figure A.13: Effect of ASPO on entropy. We show a flow field during training of H1 locomotion for a single robot joint, with FPO++ using PPO clipping (left) and ASPO clipping (right). The x-axis is integration step τ\tau.
Refer to caption
Figure A.14: FPO++ with PPO clipping at peak performance. Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured at the point of maximum average return. The flow is still relatively broad, indicating adequate exploration.
Refer to caption
Figure A.15: FPO++ with PPO clipping during policy collapse after convergence. Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured as the average return begins to collapse. The action distribution has narrowed significantly for many joints, demonstrating the entropy collapse that leads to instability and performance degradation.
Refer to caption
Figure A.16: FPO++ with ASPO objective at intermediate training stage. Policy flow field density for the Unitree H1 humanoid trained using the entropy-preserving ASPO objective, captured at an intermediate stage where the average return matches the peak average return of FPO++ with the standard PPO trust region. This distribution is already wider and more exploratory than the PPO-clipped policy at a similar or later time step, contributing to stable training.
Refer to caption
Figure A.17: FPO++ with ASPO objective at converged state. Policy flow field density for the Unitree H1 humanoid trained using the ASPO objective, captured at its final, high-reward converged state. The distribution remains wide and exploratory, confirming that ASPO effectively preserves entropy and prevents the collapse observed with PPO clipping.

D.3 Zero-sampling

Zero-sampling starts flow integration at the zero vector instead of random Gaussian noise. It is a deterministic choice, not generally the mean action of a nonlinear flow distribution. For one perturbed two-and-a-half-minute tracking sequence over one hundred rollouts, zero-sampling returns are 45.1 with five steps and 45.5 with fifty, versus 34.7 and 38.4 for random sampling, with substantial reported dispersion. The small five-versus-fifty deterministic gap enables lower-latency hardware execution, while large random-versus-zero gaps explain why test success can exceed exploratory training success.

Sampling method 5 steps 50 steps
Random sampling 34.7±55.034.7\pm 55.0 38.4±26.638.4\pm 26.6
Zero-sampling 45.1±27.4\mathbf{45.1\pm 27.4} 45.5±23.2\mathbf{45.5\pm 23.2}
Table A.3: Zero-sampling improves motion tracking. Returns are computed over 100 rollouts of a 2.5-minute dancing sequence, with domain randomization and push perturbations. The policy is trained with 50 flow steps; we evaluate using random sampling and zero-sampling with 5 and 50 flow steps.
Concept demonstration

Open image ↗