Reference state initialisation

A policy learns from the situations it actually reaches. If every episode starts at the beginning of a movement, the policy sees the end of the movement only once it is already good at the start. A backflip learnt this way never gets to practise the landing, because it never gets that far.

Reference state initialisation fixes this by starting episodes at a random point $u$ in the reference motion: the body is placed in the reference’s pose at $u$, with the reference’s joint velocities, and the episode carries on from there:

$$u \sim \mathcal{U}(0, 1), \qquad q_0 = q^{\text{ref}}(u), \qquad \dot q_0 = \dot q^{\text{ref}}(u)$$

Every part of the movement then gets practised from the first iteration. The idea was made popular by DeepMimic (Peng et al. 2018), together with early termination when the character falls.

The starts aren’t always physically comfortable: the reference may not be a state a real body could hold, so some of these episodes end in a fall straight away. They are a small cost for the coverage.

In this project

When the fighter learnt to walk, 30 % of the episodes with a walking target started partway through a step ($u$ from 0.05 to 0.95), in the walk reference’s pose and speed. Together with feed-forward on the servos, it took steps from 4–26 % of the planned distance to 56–92 %. Training falls rose from about 0.5 % to about 2 %, mostly from starts in the middle of a pivot (see Footwork).