Reward shaping

A sparse reward like −1 for a fall is hard to learn from: the penalty may arrive seconds after the mistake. Reward shaping adds terms that give earlier, denser feedback. The risk is that the extra terms change what the best behaviour is, and the agent learns to chase them (see reward hacking).

Potential-based shaping

Ng, Harada and Russell (1999) showed a safe form. Choose any function of the state, a potential $\varphi(s)$, that measures how promising a state is. Add

$$F(s, s') = \gamma\,\varphi(s') - \varphi(s)$$

to the reward on each step, using the same discount $\gamma$ as the task. Then the optimal policy is exactly the same as without shaping.

The reason: over an episode the shaping terms telescope. Each $\varphi$ appears once with a plus and once with a minus, so the total added to the return from a state is just $-\varphi(s_0)$ plus a vanishing tail. It depends on where you start, not on what you do, so it can’t make any behaviour better than another in the long run. What it does change is when the agent finds out: moving toward a better state pays immediately.

In this project

In the survive env, the −1 for a fall came 1–2 s after the mistake that caused it. We added potential-based shaping with

$$\varphi(s) = -\big(\text{capture point outside the feet} + \text{chest tilt past } 10^\circ + \text{pelvis drop past } 3\text{ cm}\big)$$

(each term weighted) and $\gamma = 0.99$. It is about 0 in a calm stance, because all three penalties are zero there. Once it was added, the shove level climbed from 0.2 to 0.7 in 7.4 M steps.