Proximal Policy Optimization

PPO (Schulman et al. 2017) is a policy-gradient method. It repeats two phases:

1. Collect. Run the current stochastic policy in many copies of the environment to gather a batch of experience, a rollout.
2. Update. Use that batch for several passes of gradient descent, making good actions more likely and bad ones less likely.

The clipped objective

The danger in step 2 is changing the policy too much on one batch of data. The data was collected by the old policy, so it says little about a very different new one, and one big bad update can wreck a good policy. PPO limits the change with a ratio and a clip.

The ratio compares how likely the new policy is to take an action with how likely the old one was:

$$\rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}$$

PPO maximises

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\!\Big(\rho_t(\theta)\,\hat{A}_t,\;\; \operatorname{clip}\big(\rho_t(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\,\hat{A}_t\Big)\right]$$

where $\hat{A}_t$ is the advantage. Once an action’s probability has moved more than $\epsilon$ (usually 0.2) in the helpful direction, the clip removes any further gain, so there is no reason to push it further.

The full loss also trains the critic and usually adds an entropy bonus that keeps some exploration:

$$L = -L^{\text{CLIP}} + c_1 \big(V_\theta(s_t) - \hat{R}_t\big)^2 - c_2\, \mathcal{H}\big[\pi_\theta(\cdot \mid s_t)\big]$$

Why it’s popular

PPO is simple, works on discrete and continuous actions, and is forgiving of settings. It isn’t the most sample-efficient algorithm, but in a fast simulator with many parallel environments that matters less. Most legged-robot locomotion work trained in simulation uses it.

In this project

Every policy on this site is trained with PPO: $\epsilon = 0.2$, 8 passes over each batch in minibatches of 512, the Adam optimiser, and CAPS regularisation added to the loss in place of an entropy bonus. A further term keeps the action means inside $[-1, 1]$ (see action clipping). Collection and updates overlap, using decoupled PPO.