Reward function

The reward is a single number the environment returns after each step. It is the only signal of what you want. The agent never sees your intentions, only the score.

Most rewards for movement are a weighted sum of terms, each one a property you care about:

$$r = \sum_i w_i \, r_i$$

A common shape for a tracking term is an exponential of the squared error, which gives 1 for a perfect match and falls smoothly toward 0:

$$r_i = \exp\!\left(-k \,\lVert x - x^{\text{ref}} \rVert^2\right)$$

The scale $k$ sets how forgiving it is. The guard term in our survive env is exactly this, with $k = 50$ on the glove positions.

Sparse and dense rewards

A sparse reward is rare, like −1 only when the body falls. It’s honest but hard to learn from, because most steps give no signal. A dense reward gives something every step, like a score for how close each joint is to the reference. It’s easier to learn from but easier to game. Reward shaping adds dense guidance without changing what the best policy is.

Pitfalls

• Relative sizes. If doing something unintended pays as well as doing the task, the agent may choose it. This is reward hacking.
• Termination. If an episode ends on a fall, the agent loses all its future reward. Positive per-step rewards then make staying alive valuable in itself.

In this project

The stance reward is 85 % tracking (joint angles, the positions of the head, hands and feet, the pelvis, an upright chest, the weight split, the feet) and 15 % smoothness. A fall ends the episode with −1. The survive mode reward is 0.4 per step for staying up plus up to 0.1 for the guard.

The jab reward is split in time. Up to the impact it scores style and speed: the hips’ and foot’s placement, the arms against the reference, the fist at least as fast as the reference’s, and one impact payment of up to 30 per jab, scaled by accuracy, the fist’s speed and how far the hips lag. After the impact the stance reward relaxes 60 % toward “stay up and keep the guard” for half a second, with only the chest’s tilt held to within 4° of the reference.