Residual policy

A residual policy doesn’t output what to do. It outputs how to change what a given reference says to do. The final command is

$$q^{\text{target}}_t = q^{\text{ref}}(\phi_t) + \Delta_{\max} \cdot a_t, \qquad a_t \in [-1, 1]$$

where $q^{\text{ref}}(\phi_t)$ is the reference pose at the current phase $\phi_t$ of the motion, $a_t$ is the policy’s output and $\Delta_{\max}$ is the largest correction allowed.

Why it helps

• A good start. A policy that outputs zero follows the reference exactly. Learning starts from a sensible motion instead of random twitching.
• Style is kept. The reference carries the look of the movement, here the coach’s stance. The policy only adds what’s needed to stay balanced.
• Limited damage. $\Delta_{\max}$ caps how far the policy can stray. A small cap on a joint is a hard rule, not a preference.

The cost is that the policy can’t do things far from the reference. That is why the survive mode in our balance post widens the leg range once the body is off balance.

In this project

The residual is up to ±0.4 rad per joint in balance. Off balance in survive mode it widens to ±1.5 rad on the legs and ±0.8 rad on the trunk, but stays at ±0.15 rad on the arms, so the gloves physically can’t leave the face. The coach’s “hands stay up” rule is built into the action space.