Early stopping

Training usually improves fast at first and then levels off on a plateau. Running on past the plateau costs time and money and gains little. Early stopping is a rule that decides when to stop.

In supervised learning the classic rule watches the error on held-out data and stops when it starts to rise, which signals the model is memorising its training data. In reinforcement learning there’s no held-out set, so rules usually watch a training metric and stop when it stops improving.

A plateau rule

Training metrics in RL are noisy, so a good rule compares averages over windows rather than single points. Ours is:

1. Average the training fall rate at the top shove level over windows of 10 M steps.
2. Compare the newest window with the window two before it.
3. Stop if the newest isn’t at least 1.5 percentage points better.

Comparing with the window two back, rather than the one just before, means a slow but steady improvement still counts.

In this project

The simple body v2 run was stopped by this rule at 75.6 M steps. On a cloud machine costing about $1 an hour, stopping on the plateau keeps a typical run to $1–2.

The last speed run of 18 October was stopped the same way: its training hit rate sat at about 33 % from 15 M to 31 M steps, so it was stopped at 31.2 M and frozen (see Faster punches).