Jitter

Reinforcement learning policies often make tiny, rapid corrections that score fine on tracking but look like trembling. On a real robot jitter wastes energy, heats motors and wears gears. On a simulated fighter it simply looks wrong: no coach would accept a stance that shakes.

Causes

• Nothing in the reward against it. If holding a pose by rapid back-and-forth correction scores as well as holding it smoothly, the policy has no reason to prefer smooth.
• Sensitive networks. A small change in the input can produce a large change in the output.
• Exploration noise during training that the deterministic policy partly keeps.
• Rigid contacts. Feet on a perfectly hard floor can chatter between touching and not touching.

Measuring it

We count direction reversals: how often a joint’s velocity changes sign, per joint per second.

$$J = \frac{1}{n_{\text{joints}}\, T}\sum_{j}\sum_{t} \mathbb{1}\!\left[\dot{q}_{j,t}\,\dot{q}_{j,t-1} \lt 0\right]$$

A slow sway reverses once per beat, about once a second. Twitching reverses many times a second.

Fixes

• smoothness terms in the reward on the real joint motion;
• CAPS regularisation on the network;
• a low-pass filter on the actions;
• a softer floor (see soft contact).

In this project

The first stance policy scored 1.7 reversals per joint per second standing calmly, against 0.02 for the reference motion itself. Putting the unchanged policy on a foam gym mat cut that to 0.15. Retraining with a balance gate, an adaptive filter, smoothness rewards and CAPS took it to 0.01.