Learning rate

The learning rate $\eta$ scales each update in gradient descent:

$$\theta \leftarrow \theta - \eta \, \nabla_\theta L$$

It is usually the most important hyperparameter to get right.

• Too high: steps overshoot. The loss jumps around or blows up, and in reinforcement learning a good policy can be wrecked in a few updates.
• Too low: training is stable but slow, and may stall.

Schedules

The rate doesn’t have to be fixed. Common patterns are a warm-up (start small and grow, while the optimiser’s statistics settle) and decay (shrink over time, to make finer adjustments once the model is close).

Why a mature policy needs a smaller rate

A policy early in training is far from any good solution, and big steps help it get somewhere fast. A trained policy sits in a narrow good region. The same step size that was useful early can throw it out of that region. A smaller rate lets it keep improving without forgetting what it learned. This is the usual advice for fine-tuning.

In this project

PPO’s common default is $3 \times 10^{-4}$. Resuming or warm-starting a mature survive policy at that rate wrecked it within about 20 updates (100 % falls), in runs survive3 to survive6. Using $5 \times 10^{-5}$ for any mature policy fixed it, and survive7 went on to hold 860 N shoves.