Regularisation

A model can often fit its training signal in many ways. Regularisation tilts training toward the ways you prefer, usually by adding a penalty to the loss:

$$L_{\text{total}} = L_{\text{task}} + \lambda \, R(\theta)$$

$\lambda$ sets how much the preference matters compared with the task.

Common kinds

• Weight decay (L2). $R = \lVert\theta\rVert^2$. Prefers small weights, which tend to give smoother functions that generalise better.
• Dropout. Randomly switch off units during training, so the network can’t rely on any one of them.
• Entropy bonus. In RL, reward the policy for staying a little random, so it keeps exploring.
• Smoothness penalties. Penalise outputs that change sharply between similar inputs or between steps. CAPS regularisation is one.

In control

For a motor controller, the most useful preference is smoothness. A policy can hold a pose by rapidly correcting back and forth, which scores fine on tracking but looks like trembling (see jitter). Regularising toward smooth actions removes it.

In this project

We regularise with CAPS (temporal and spatial smoothness) and with a penalty that keeps action means inside $[-1, 1]$ (see action clipping).