Literature
Display
← All papers

Proximal Policy Optimization Algorithms

John Schulman · Filip Wolski · Prafulla Dhariwal · Alec Radford · Oleg Klimov

Preprint posted · arXiv (preprint) ↗

Language models6 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition. A controller tries actions and learns which worked better than expected. The advantage A estimates that improvement: positive A encourages the sampled action; negative A discourages it. PPO tries to reuse those observations without rewarding excessively large policy changes.

Reading the math. The ratio r = πnew(a|s)/πold(a|s) compares the probability of action a in state s. A change from 0.2 to 0.3 gives r = 1.5. With clipping width ε = 0.2 and positive A, the clipped objective stops crediting that sample beyond 1.2A. Shared network updates can still change other probabilities, so clipping is not a hard bound on every policy change.

PPO alternates fresh environment rollouts with several minibatch optimization epochs on the same batch. A clipped likelihood-ratio objective discourages excessively beneficial-looking changes without the second-order constrained optimization used by TRPO. In the paper’s 2017 simulated-control and Atari experiments, it offers a favorable simplicity, sample-efficiency and performance tradeoff; the evidence is benchmark-specific.

1 Introduction

The goal is a first-order policy-gradient method that reuses samples while avoiding destructive updates. PPO replaces the usual objective with either clipped ratios or an adaptive KL penalty. Clipping performs best in the tested control ablations. The historical comparison concerns these implementations and architectures, not a general impossibility result for Q-learning or a guarantee that every PPO update improves true return.

2 Background: Policy Optimization

The background distinguishes empirical policy-gradient objectives from trust-region constraints and their theoretical motivation.

2.1 Policy Gradient Methods

The gradient estimator multiplies the score function by an estimated advantage, averaged over sampled timesteps. Differentiating the log-policy objective gives that estimator. Repeatedly optimizing this ordinary loss on a fixed batch can move the policy too far from the one that generated the data; sample reuse needs an additional control mechanism.

Equations
Original eq1.
Original eq2.

2.2 Trust Region Methods

TRPO maximizes an importance-ratio surrogate under an average KL constraint, using local objective/constraint approximations and conjugate gradients. Its theoretical lower-bound motivation involves a maximum-over-states KL quantity, which differs from the practical average penalty. A single fixed penalty coefficient is difficult to tune across tasks and training stages; PPO therefore explores clipping or adaptive penalties.

Equations
Original eq3 4.
Original eq5.

3 Clipped Surrogate Objective

Let r be the new-to-old probability ratio for a sampled action. PPO takes the smaller of rA and the same product with r clipped to [1−ε,1+ε]. For positive advantages, gains stop growing beyond 1+ε; for negative advantages, gains stop growing below 1−ε. Changes that worsen the surrogate remain penalized. The two objectives share the first-order gradient at the old policy.

This is a lower bound on the unclipped sampled surrogate; it does not impose a hard bound on all policy ratios or establish monotonic true-return improvement. Figure 2 illustrates one Hopper update, not a universal KL guarantee.

Equations
Original eq6.
Original eq7.
Figure 1: Plots showing one term (i.e., a single timestep) of the surrogate function LCLIP as a function of the probability ratio r, for positive advantages (left) and negative advantages (right). The red circle on each plot shows the starting point for the optimization, i.e., r = 1. Note that LCLIP sums many of these terms.
Figure 1: Plots showing one term (i.e., a single timestep) of the surrogate function LCLIP as a function of the probability ratio r, for positive advantages (left) and negative advantages (right). The red circle on each plot shows the starting point for the optimization, i.e., r = 1. Note that LCLIP sums many of these terms.
Figure 2: Surrogate objectives, as we interpolate between the initial policy parameter θold, and the updated policy parameter, which we compute after one iteration of PPO. The updated policy has a KL divergence of about 0.02 from the initial policy, and this is the point at which LCLIP is maximal. This plot corresponds to the first policy update on the Hopper-v1 problem, using hyperparameters provided in Section 6.1.
Figure 2: Surrogate objectives, as we interpolate between the initial policy parameter θold, and the updated policy parameter, which we compute after one iteration of PPO. The updated policy has a KL divergence of about 0.02 from the initial policy, and this is the point at which LCLIP is maximal. This plot corresponds to the first policy update on the Hopper-v1 problem, using hyperparameters provided in Section 6.1.

4 Adaptive KL Penalty Coefficient

The alternative optimizes a KL-penalized ratio objective, measures the resulting mean KL, and adjusts β for the next update: halve it below the target divided by 1.5, double it above 1.5 times the target. These are heuristic control rules, and individual updates can overshoot the target. Adaptive KL performs worse than clipping in the reported ablation but remains a useful baseline.

Equations
Original eq8.
Original kl update.

5 Algorithm

Each of N actors collects T timesteps under the old policy. Compute advantages, optimize the surrogate over this NT-sample batch for K minibatch epochs, then refresh the old-policy parameters and collect new data. Shared policy/value networks add a value-error loss and an entropy bonus; the continuous-control experiments instead use separate networks and no entropy bonus. Generalized advantage estimation supplies a truncated advantage estimate; Adam is the usual optimizer.

The native equations preserve the source’s printed finite-horizon exponents and indexing. Equations 10–11 show T−t+1 on the last reward/residual term, an apparent indexing inconsistency for a sequence ending at T−1; this edition does not silently change the original formulas.

Equations
Original eq9.
Original eq10.
Original eq11 12.
Algorithm 1 PPO, Actor-Critic Style
Algorithm 1 PPO, Actor-Critic Style

6 Experiments

The evaluations separately test surrogate choices, continuous-control baselines, a humanoid showcase, and Atari learning speed versus final performance.

6.1 Comparison of Surrogate Objectives

Seven MuJoCo tasks each use one million timesteps and three seeds. Scores average the last 100 episodes, then normalize each environment between a random policy and the best observed result before averaging 21 runs. A two-hidden-layer 64-unit tanh network parameterizes a Gaussian policy, with a separate value network. Clipping ε=.2 scores .82 versus .74 for the best adaptive-KL setting and −.39 without clipping or penalty; one very poor HalfCheetah result drives the negative aggregate.

These are tuned, benchmark-relative aggregates rather than raw cross-task returns or uncertainty estimates. The original ablation display’s clipping-parenthesis typo is retained; Equation 7 states the intended objective.

Equation
Original surrogate variants.
Table 1: Results from continuous control benchmark. Average normalized scores (over 21 runs of the algorithm, on 7 environments) for each algorithm / hyperparameter setting . β was initialized at 1.
Table 1: Results from continuous control benchmark. Average normalized scores (over 21 runs of the algorithm, on 7 environments) for each algorithm / hyperparameter setting . β was initialized at 1.

6.2 Comparison to Other Algorithms in the Continuous Domain

With ε=.2, PPO is compared with tuned TRPO, cross-entropy search, vanilla policy gradients with an adaptive stepsize, A2C and A2C with a trust region. Figure 3 shows learning curves on all seven tasks. PPO generally performs well, but the per-task curves matter more than a universal claim of dominance. A2C is the synchronous counterpart of the asynchronous actor-critic baseline.

Figure 3: Comparison of several algorithms on several MuJoCo environments, training for one million timesteps.
Figure 3: Comparison of several algorithms on several MuJoCo environments, training for one million timesteps.

6.3 Showcase in the Continuous Domain: Humanoid Running and Steering

The Roboschool showcase learns humanoid forward running, changing-target navigation, and a harder task requiring recovery while being struck by cubes. Targets move every 200 timesteps or when reached. Figures 4–5 retain all learning curves and the 12-frame turning sequence. These simulated demonstrations establish the tested behavior; they do not evaluate physical-robot deployment or unseen disturbances.

Figure 4: Learning curves from PPO on 3D humanoid control tasks, using Roboschool.
Figure 4: Learning curves from PPO on 3D humanoid control tasks, using Roboschool.
Figure 5: Still frames of the policy learned from RoboschoolHumanoidFlagrun. In the first six frames, the robot runs towards a target. Then the position is randomly changed, and the robot turns and runs toward the new target.
Figure 5: Still frames of the policy learned from RoboschoolHumanoidFlagrun. In the first six frames, the robot runs towards a target. Then the position is randomly changed, and the robot turns and runs toward the new target.

6.4 Comparison to Other Algorithms on the Atari Domain

On 49 Atari games, the methods share the policy architecture and each setting has three trials. PPO wins 30 games by average reward across training, versus 18 for ACER and 1 for A2C. By final 100-episode reward, ACER wins 28, PPO 19, A2C 1, with 1 tie. Thus PPO’s advantage is stronger in this learning-speed measure; it does not have the highest final score on most games. Appendix B gives all per-game results after 40 million frames.

Table 2: Number of games “won” by each algorithm, where the scoring metric is averaged across three trials.
Table 2: Number of games “won” by each algorithm, where the scoring metric is averaged across three trials.

7 Conclusion

PPO’s contribution is a practical family of first-order policy updates that can reuse on-policy batches for several epochs. The clipped variant is the strongest of the tested objectives, with comparatively simple implementation and competitive benchmark performance. Its empirical stability should be distinguished from a strict trust-region or monotonic-return theorem.

8 Acknowledgements

The authors thank Rocky Duan, Peter Chen and other OpenAI colleagues for comments.

A Hyperparameters

The retained tables give task-specific settings. MuJoCo uses horizon 2048, 10 epochs,minibatches 64 and Adam 3×10⁻⁴. Roboschool uses horizon 512, 15 epochs,minibatches 4096 and 32 or 128 actors, with KL-adjusted stepsize and annealed action log-standard-deviation. Atari uses 8 actors,horizon 128, 3 epochs,minibatches 256; both stepsize and clipping decrease linearly, with value coefficient 1 and entropy coefficient .01. Discount .99 and GAE λ=.95 are shared.

Table 3: PPO hyperparameters used for the Mujoco 1 million timestep benchmark.
Table 3: PPO hyperparameters used for the Mujoco 1 million timestep benchmark.
Table 4: PPO hyperparameters used for the Roboschool experiments. Adam stepsize was adjusted based on the target value of the KL divergence.
Table 4: PPO hyperparameters used for the Roboschool experiments. Adam stepsize was adjusted based on the target value of the KL divergence.
Table 5: PPO hyperparameters used in Atari experiments. α is linearly annealed from 1 to 0 over the course of learning.
Table 5: PPO hyperparameters used in Atari experiments. α is linearly annealed from 1 to 0 over the course of learning.

B Performance on More Atari Games

Figure 6 retains every one of the 49 game panels, all three methods and their seed curves; Table 6 retains every final score. Although their original captions name only PPO and A2C, both also include ACER. Outcomes vary substantially by game, and the three-seed means alone do not establish statistically significant pairwise superiority.

Figure 6: Comparison of PPO and A2C on all 49 ATARI games included in OpenAI Gym at the time of publication.
Figure 6: Comparison of PPO and A2C on all 49 ATARI games included in OpenAI Gym at the time of publication.
Table 6: Mean final scores (last 100 episodes) of PPO and A2C on Atari games after 40M game frames (10M timesteps).
Table 6: Mean final scores (last 100 episodes) of PPO and A2C on Atari games after 40M game frames (10M timesteps).
Concept demonstration

Open image ↗