Stochastic policy

A stochastic policy doesn’t output an action. It outputs a probability distribution, and the action is drawn from it. For continuous actions such as motor targets the usual choice is a Gaussian (normal) distribution: the network outputs a mean $\mu_\theta(s)$ for each action, and there is a standard deviation $\sigma$.

$$a \sim \mathcal{N}\!\left(\mu_\theta(s),\, \sigma^2\right)$$

The log-probability of an action, which algorithms like PPO need, is

$$\log \pi_\theta(a \mid s) = -\sum_{i} \left[\frac{(a_i - \mu_i)^2}{2\sigma_i^2} + \log \sigma_i + \tfrac{1}{2}\log 2\pi\right]$$

Why add randomness

An agent that always does the same thing in the same situation never finds out whether something else would be better. Sampling makes it try small variations, and the ones that turn out well become more likely. This is exploration. As training goes on, $\sigma$ is usually learned and tends to shrink, so the policy explores less once it is good.

Deterministic evaluation

After training you can drop the noise and always take the mean action, $a = \mu_\theta(s)$. This is the deterministic policy, and it is usually what gets deployed and tested.

The two can behave differently. The deterministic policy is smoother and often better, but it hides the mistakes the noisy policy is still making. Those mistakes are what training still has to fix.

In this project

Our standard deviation is a learned parameter per motor, independent of the observation. It starts at $e^{-1} \approx 0.37$.

Training fall rates are measured with sampled actions, and fixed tests with the deterministic policy. That is why survive7 fell in about 10 % of training episodes at the top level but in 0 of 12 fixed tests. It is also the lesson behind “progress videos must sample actions”: a video of the deterministic policy looked fine while training was still having falls.