Decoupled PPO

Plain PPO alternates: every worker stops collecting while the learner updates, then waits for the new weights. On a big machine the CPUs sit idle for much of the time.

Overlapping the two is easy to code: workers keep collecting with the weights they have while the learner updates. But then the data comes from a behaviour policy that is a version or two old, and plain PPO’s ratio and clip assume it’s from the current one.

The fix

Hilton, Cobbe and Schulman (2022) separate two roles that plain PPO merges into one “old policy”:

• the behaviour policy $\pi_{\text{behav}}$, which actually collected the data;
• the proximal policy $\pi_{\text{prox}}$, the recent policy that the update shouldn’t stray far from.

The update corrects for the mismatch with an importance weight and clips against the proximal policy:

$$L(\theta) = \mathbb{E}_t\!\left[\frac{\pi_{\text{prox}}(a_t \mid s_t)}{\pi_{\text{behav}}(a_t \mid s_t)}\,\min\!\Big(\rho_t \hat{A}_t,\; \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\,\hat{A}_t\Big)\right], \qquad \rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{prox}}(a_t \mid s_t)}$$

With this, slightly stale data is used correctly, and PPO’s step-size control still works.

In this project

Each of the 52 worker processes runs its own copy of the policy for a whole rollout, so collection overlaps the update. The data is one update old. The clip is centred on the policy at the start of the update, and each sample is weighted by $\pi_{\text{prox}} / \pi_{\text{behav}}$, capped at 2 so that a rare action can’t dominate. On the same 64-vCPU machine this raised throughput from about 14 M to 24 M steps an hour, and CPU use from 28 % to 65 %.