Loss function

A loss function turns a model’s mistakes into one number. Lower is better. Gradient descent then adjusts the parameters to bring it down.

Two classic examples:

Mean squared error, for predicting numbers:

$$L = \frac{1}{N}\sum_{i=1}^{N} \big(y_i - \hat{y}_i\big)^2$$

Cross-entropy, for predicting categories:

$$L = -\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(y_i \mid x_i)$$

Losses in reinforcement learning

RL has no correct answers to compare against, so its loss is built from the data the agent collected. In PPO, the loss is the sum of several parts:

• the policy loss, the negative of PPO’s clipped objective, which makes good actions more likely;
• the value loss, a mean squared error that trains the critic to predict returns;
• regularisation terms that prefer some solutions over others, for example smoothness.

The loss isn’t the same as the reward. The reward says what the agent should achieve. The loss is the tool used to update the network toward it.

In this project

Our total loss is the PPO policy loss, plus half the value loss, plus two CAPS regularisation smoothness terms, plus a penalty on action means outside $[-1, 1]$ (see action clipping).