CAPS regularisation

CAPS (Conditioning for Action Policy Smoothness, Mysore et al. 2021) is a regularisation method for smooth control. RL policies often jitter: the network’s output can change sharply between nearby inputs, and on a real motor or a simulated body that becomes visible twitching (see jitter).

CAPS adds two penalties to the training loss. Writing $\mu_\theta(s)$ for the policy’s mean action:

Temporal smoothness: consecutive actions should be close.

$$L_T = \big\lVert \mu_\theta(s_t) - \mu_\theta(s_{t+1}) \big\rVert$$

Spatial smoothness: states that are almost the same should get almost the same action. A nearby state $\bar{s}$ is made by adding a little noise.

$$L_S = \big\lVert \mu_\theta(s) - \mu_\theta(\bar{s}) \big\rVert, \qquad \bar{s} \sim \mathcal{N}(s, \sigma^2)$$

The total loss is

$$L = L_{\text{PPO}} + \lambda_T L_T + \lambda_S L_S$$

Larger weights $\lambda_T, \lambda_S$ give smoother but less responsive control.

Why not just filter the actions?

A low-pass filter on the output also smooths, but it adds lag, and the policy has to learn around the lag. CAPS makes the network itself smooth, so it doesn’t need heavy filtering. In practice the two work well together.

In this project

We use squared distances for both terms, with $\lambda_T = \lambda_S = 0.5$, and make $\bar{s}$ by adding noise with $\sigma = 0.05$ to the normalised observation. CAPS was one of five changes that took calm jitter from 1.7 to 0.01 direction reversals per joint per second. The others were a balance gate, an adaptive action filter, smoothness rewards on the real joint motion, and a foam floor.

In the blocking work, CAPS also smoothed the four block buttons the policy presses itself. A press is exactly a sudden jump in an output, so the fighter never pressed one deliberately until the buttons were exempted (see Blocking).