Gradient descent

The gradient $\nabla_\theta L$ is the vector of how much the loss changes as each parameter changes. It points uphill. Gradient descent takes a step the other way:

$$\theta \leftarrow \theta - \eta \, \nabla_\theta L(\theta)$$

$\eta$ is the learning rate. Picture walking down a foggy hillside by always stepping in the steepest downhill direction.

Stochastic gradient descent

Computing the exact gradient over all the data is slow. Stochastic gradient descent (SGD) estimates it from a small random minibatch and takes many cheap, noisy steps instead of a few exact ones. One full pass over the data is an epoch.

Backpropagation

For a neural network with many layers, the gradient is computed by backpropagation: the chain rule of calculus applied from the output layer back to the input, reusing work along the way. It costs about as much as running the network forward twice.

Adam

Most deep learning, including ours, uses the Adam optimiser (Kingma and Ba 2015). It keeps a running average of each parameter’s gradient $m$ and of its square $v$, and scales each step by them:

$$m \leftarrow \beta_1 m + (1-\beta_1)\,g, \qquad v \leftarrow \beta_2 v + (1-\beta_2)\,g^2, \qquad \theta \leftarrow \theta - \eta\,\frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}$$

($\hat{m}$ and $\hat{v}$ are bias-corrected versions.) Parameters with steady gradients move confidently; noisy ones move cautiously.

In this project

Each PPO iteration makes 8 passes over the batch it collected, in minibatches of 512 samples, using Adam.