Action clipping

A Gaussian policy can output any real number, but motors have limits. So the environment clips each action to its range before using it:

$$a^{\text{used}} = \operatorname{clip}(a, -1, 1) = \max\big(-1,\, \min(1, a)\big)$$

The hidden problem

Clipping is flat outside the range. If the policy’s mean $\mu$ drifts to 5, the action used is still 1. Moving $\mu$ from 5 to 4 changes nothing the environment can see, so there is no reward signal to bring it back. The mean can drift further and further, nothing pulls it back, and the joint stays pinned at its limit. Learning on that joint stops.

Fixes

• A bound penalty in the loss on the part of the mean outside the range:

$$L_{\text{bound}} = \sum_i \max\big(0,\, |\mu_i| - 1\big)^2$$

• Squashing: pass the output through $\tanh$, so it can never leave $[-1, 1]$. This changes the log-probability calculation and is common in other algorithms such as SAC.

In this project

In run survive2 the action means drifted to an average size of 17, against a clip at ±1. Every joint stuck at its limit and the policy collapsed (see training divergence). A loss on $|\mu| > 1$ fixed it and has been part of training since.