Observation

An observation is the input to the policy: a list of numbers the environment builds on every step. It is the agent’s only view of the world. Anything that matters for choosing a good action but isn’t in the observation, the agent has to guess.

The full state of a simulation (every position, velocity and contact) is often too large or not all useful. Designers choose what to include. A good observation is:

• sufficient: it holds what’s needed to act well (see the Markov property under markov decision process);
• relative: positions measured from the body rather than from the room, so the policy works wherever the fighter stands;
• scaled: values of similar size, so no input swamps the others.

Normalisation

Raw inputs range from tiny angles to forces of hundreds of newtons. Training is easier if each input has mean 0 and standard deviation 1. A running estimate of each input’s mean $\mu$ and variance $\sigma^2$ is kept during training, and the network sees

$$\hat{o} = \operatorname{clip}\!\left(\frac{o - \mu}{\sqrt{\sigma^2 + \epsilon}},\, -10,\, 10\right)$$

The statistics are saved with the policy, because the trained network only understands normalised inputs.

In this project

The stance policy on the muscular body observes 193 numbers (169 on the simple body, which has fewer joints): the joint errors against the reference and the joint velocities; the pelvis orientation, velocity and offset from its reference; the centre of mass relative to the feet and the force on each foot; the rhythm phase, tempo and amplitude; the previous and filtered actions; and the balance gate.

The statistics drift as training sees new kinds of data. During the walking runs, walking moved some inputs’ running statistics by up to 0.12 standard deviations. A frozen teacher policy was being fed inputs scaled with the student’s drifting statistics instead of its own saved ones, which moved its advice by about 0.007 per motor, enough to make punches miss. Each network has to see its inputs scaled with its own statistics (see Footwork).

The same statistics hid the opponent from the fighter in the blocking work. The running averages keep one sample count for every input, and by then it was 1.2 billion. The opponent’s 208 inputs, added later, barely moved from their starting values (mean 0, spread 1) when their real spread was about 0.29. For five runs the fighter all but ignored the opponent. Re-estimating those inputs’ statistics fixed the scaling (see Blocking).