Local optimum

Most learning methods improve a solution by small changes: try something slightly different, and keep it if it scores better. Gradient descent and hill climbing both work this way.

Picture the score as a landscape of hills. A small-step method always walks uphill. It stops on top of whatever hill it is on. That hilltop is a local optimum: every nearby point is lower. There may be a much higher hill elsewhere, the global optimum, but reaching it means first going downhill, which a small-step method never chooses to do.

For a parameter vector $\theta$ and score $J$, $\theta^*$ is a local maximum if

$$J(\theta^*) \ge J(\theta) \quad \text{for every } \theta \text{ close enough to } \theta^*$$

Ways out

• Exploration. A stochastic policy tries random variations, so it can sometimes stumble onto the next hill. If its noise has shrunk too far, it can’t.
• Change the landscape. Rewarding the intermediate steps of the better behaviour (reward shaping) builds a slope from the low hill to the high one.
• Start somewhere else. A different starting point, or a curriculum that teaches the hard part before the easy shortcut is found.

In this project

When the fighter was rewarded for coming back to its stance after a jab, it stopped stepping into the jab. Over 3 s from one press, the policy that barely stepped scored 218, and one that stepped in and stayed forward scored 180. Stepping in and back would have scored more, but it is a long, coordinated sequence, and every small move towards it first scored worse. Its exploration noise had also shrunk to about 0.10. The way out was to change the reward: style and speed up to the impact, and a relaxed return after it.