Auxiliary loss

In reinforcement learning, the only teacher is the reward. It says how well things went, not what should have been done, and it often comes late. Learning which of many choices earned it can take a very long time.

When the right answer is known for some situations, it can be given to the network directly, as an extra loss added to the main one:

$$L = L_{\text{RL}} + w \, L_{\text{aux}}$$

$L_{\text{aux}}$ might be the error in predicting a label, or the distance between the network’s output and a known good action. The weight $w$ sets how much it counts. Unlike a reward, the auxiliary loss pushes the network’s outputs straight towards the answer through gradient descent, so it can teach in thousands of steps what the reward alone might not teach in millions.

It is closely related to policy distillation, where the answers come from a teacher network, and it is a stronger form of reward shaping, which only adds hints to the reward.

In this project

In the blocking post, the opponent is scripted, so the environment knows which punch is coming and which block suits it. A dense reward for pressing the right block button changed nothing in about 32 M steps. An auxiliary loss did: while a punch can be seen, it pulls the suitable button’s output towards +1 and the others towards −1 (weight 2.0). Within 5 M steps the fighter raised a different block for different punches for the first time. The reward still decides whether blocking pays: a block that doesn’t stop the punch still costs damage.