Reference motion

A reference motion is a description of how a movement should look, frame by frame, without any guarantee that a real body could perform it. It might come from motion capture, from an animator’s keyframes, or from a solver.

The policy’s job is motion tracking: produce a physically real motion that stays close to the reference while keeping its balance. The reward pays for matching:

$$r_{\text{track}} = \sum_i w_i \exp\!\left(-k_i \,\big\lVert x_i - x_i^{\text{ref}}(\phi)\big\rVert^2\right)$$

summed over features such as joint angles, hand and foot positions and pelvis height, at the current phase $\phi$ of the motion.

This approach became popular with DeepMimic (Peng et al. 2018), which trained simulated characters to perform backflips, kicks and dances from motion-capture clips.

Why use a reference

Rewarding only the goal (“don’t fall”, “hit the target”) often produces motion that works but looks strange: flailing arms, odd gaits. A reference adds style. It says not just what to achieve but how it should look.

The trade-off is flexibility. A policy tied closely to a reference can’t do much beyond it. Our survive mode removes the reference off balance for exactly that reason.

In this project

The references are the coach’s solved stance, the rhythm (a 1-beat-per-second weight shift with a heel roll and lead-hand paw) and the keyframed recovery moves. The policy is a residual policy on top of them.

For the strikes, the coach first asked a coding agent for a jab, then corrected it detail by detail (elbow in, hip turn, fist screwed over, chin tucked), and finally explained it with photos and freeze frames of himself. References are built on the muscular body, so they need no retargeting.

The jab showed where an animated reference and physics disagree. The first jab reference slowed the fist to zero on the target and held it there for 0.04 s, which taught a gentle “reach and touch”. The fixed reference goes out in 0.14 s with no hold, and the target sits 7 cm inside the point where the reference’s fist stops, so the punch goes through it. Some details of the reference are then relaxed in training: after the impact, the return to the stance is only loosely scored (see the jab post).