Glossary · Machine learning
Training divergence
Also called NaN, collapse, policy collapse, instability
When training breaks down instead of improving: the loss blows up, the numbers become NaN, or a good policy suddenly collapses.
Training is said to diverge when it moves away from a solution rather than toward one. In RL it shows up as:
- NaN, “not a number”: a value overflowed or a division by zero happened. Once a single weight is NaN, it spreads to every output in the next calculation and the network is dead.
- Collapse: a policy that was doing well suddenly does very badly, often within a few updates, and doesn’t recover.
- Saturation: outputs pinned at their limits, so gradients carry no information (see action clipping).
Common causes
- A learning rate too large for the current stage of training.
- Rewards or observations with extreme values.
- Feedback loops in RL: a worse policy collects worse data, which teaches it to be worse.
- Reward designs that make strange behaviour pay (see reward hacking).
Defences
- Check every update. If any loss or weight is not finite, throw the update away and restore the weights from just before it.
- Clip gradients so that one unusual batch can’t take a huge step.
- Keep regular checkpoints, so a collapse costs minutes rather than the run.
In this project
The reward-hacking run balance3 diverged to NaN. The trainer now keeps a snapshot before each update and rolls back any update with non-finite numbers. Run survive2 collapsed when its action means drifted to an average size of 17, far outside the clip at ±1, and survive3 to survive6 collapsed when a mature policy was resumed at too high a learning rate.