Neural network

A neural network is a flexible function with many adjustable numbers, called weights. Given enough weights and the right values for them, it can approximate almost any mapping from inputs to outputs. Training is the search for those values.

One layer

Each layer takes a vector of inputs $x$, multiplies by a weight matrix $W$, adds a bias vector $b$, and applies a simple nonlinear function $\sigma$ to every element:

$$h = \sigma(Wx + b)$$

Stacking layers gives a deep network. Without the nonlinearity, a stack of layers would collapse into one matrix multiplication and could only learn straight-line (linear) relationships.

Activation functions

Common choices for $\sigma$:

name	formula
ReLU	$\max(0, x)$
tanh	$\tanh(x)$
ELU	$x$ if $x > 0$, else $e^x - 1$

Training

The network’s error is measured by a loss function. Backpropagation uses the chain rule to work out how much each weight contributed to the error, giving the gradient. Gradient descent then nudges every weight to reduce the error. Repeat millions of times.

In this project

Our networks are small by modern standards. Each policy is an MLP with two hidden layers of 256 units and ELU activations, about 125,000 weights. A large language model has hundreds of billions. A small network is enough to map body state to motor corrections, and it runs fast on a CPU 50 times a second.