Catastrophic forgetting

A neural network stores everything it knows in one shared set of weights. Training on a new task moves those weights to suit the new task, and nothing protects the old one. If the new data doesn’t exercise the old skill, it can fade quickly, sometimes within a few hundred updates. McCloskey and Cohen named it in 1989.

In reinforcement learning it compounds. A policy that has partly forgotten a skill does it worse, so the data it collects shows that skill doing badly, and training learns from that.

Common protections:

• A small learning rate, so each update moves the weights less.
• Keep practising the old task, by mixing it into the new training.
• Anchor to the old model: penalise the network for drifting from a frozen copy where the old behaviour is wanted (policy distillation), or for moving the weights that mattered most to the old task.

In this project

Adding four punches to the jab policy, at a small learning rate and with the jab still pressed on 20 % of presses, still lost the jab: full-reach hits fell from 28 % to 2–7 % in 2.7 M steps, and its speed from 2.7 to 1.3 m/s. The fighter kept its balance by punching more slowly. A frozen copy of the jab policy as a teacher stopped it. The anchor also kept something unwanted: a head tilt learnt in one earlier run was passed on to every run after it, until the neck and head were freed from the teacher.