Policy

A policy is what a reinforcement learning agent learns. It maps each observation to an action. It is usually written $\pi$, and when it has learnable parameters $\theta$ (the weights of a neural network), $\pi_\theta$.

There are two kinds:

• A deterministic policy gives one action for each observation: $a = \pi(s)$.
• A stochastic policy gives a probability distribution over actions, and the action is sampled from it: $a \sim \pi(\cdot \mid s)$. See stochastic policy.

Training changes $\theta$ so that actions that led to more reward become more likely. A trained policy can be frozen: its weights are saved and no longer change, so it behaves the same way every time and can be used as a starting point for new training.

In this project

The policy is a small MLP that runs 50 times a second. It reads 193 numbers about the body and outputs one number for each of its 43 body motors (on the muscular body). Those numbers aren’t joint angles. They are corrections added to the coach’s reference pose, which makes it a residual policy.

The policy trained to hold the stance and rhythm was frozen and became the parent of the shove-recovery policies. Both recovery approaches started from its weights.