Gaussian kernel

Many reward terms start as an error: how far a joint is from its target, how many degrees the trunk leans too far. A Gaussian kernel turns an error $e$ into a reward between 0 and 1:

$$r = \exp\!\left(-\frac{e^2}{2\sigma^2}\right)$$

At $e = 0$ the reward is 1. At $e = \sigma$ it is about 0.61, at $2\sigma$ about 0.14, and far away it is nearly 0. The width $\sigma$ says what counts as “close”: errors well under $\sigma$ are hardly punished, errors of a few $\sigma$ get almost nothing.

Compared with a plain penalty like $-e^2$, a Gaussian is bounded: one huge error can’t swamp every other term in the reward, and a policy that is already close gets a smooth slope towards perfect. It is the usual shape for tracking a reference motion (DeepMimic’s terms are Gaussians of pose, velocity and end-effector errors).

A term can also be one-sided: the error counts only in one direction, and the other direction is free.

In this project

The rear knee’s upright term is one-sided: $e$ is how far the chest or pelvis leans back past the reference after the hit (leaning forward of it is free), with $\sigma = 5^\circ$ and weight 1.5, paid every step. It is what took the knee’s lean back from about 10° to about 1° in kneeR4.