Activation function

Each neuron in a neural network adds up its inputs, each multiplied by a weight, plus a bias, and then passes the sum through a fixed curve: the activation function. For a whole layer:

$$h = f(W x + b)$$

The curve is what makes depth useful. Without it, two layers $W_2 (W_1 x + b_1) + b_2$ are just one layer with the matrix $W_2 W_1$, and no number of layers could represent anything but a straight-line (linear) function. With a bend in every neuron, a network can approximate almost any shape.

Common choices:

• ReLU: $f(z) = \max(0, z)$. Zero for negative inputs, the input itself for positive ones.
• ELU (exponential linear unit): the input itself for positive values, and a smooth curve towards −1 for negative ones:

$$\mathrm{ELU}(z) = \begin{cases} z & z > 0 \\ e^{z} - 1 & z \le 0 \end{cases}$$

Unlike ReLU, it never goes completely flat, so a neuron with negative input still passes a gradient back.

• tanh: an S-curve between −1 and 1, often used on a network’s outputs to keep them in range.

In this project

The fighter’s policy and its critic use ELU in every hidden layer. Two of ELU’s properties made it possible to add a layer without changing the network’s behaviour in the network-size experiment: its output is never below −1, and it is exactly the identity for positive inputs. A new layer with identity weights and a bias of +1 therefore only ever sees positive numbers and passes them through unchanged (see function-preserving growth).