Glossary · Machine learning
Fine-tuning
Also called warm start, warm-starting, resume, from scratch
Continuing to train an already-trained model, often on a new or harder task, instead of starting from scratch.
Training from scratch starts with random weights. Fine-tuning (or warm-starting) starts from the weights of a model that has already learned something useful, and trains it further.
Reasons to fine-tune:
- Speed. The model already has useful skills, so it needs much less training.
- Building up. Train a simple skill first, then teach a harder one on top. A fighter who can stand learns to recover from shoves faster than one who can’t stand.
- Resuming. Continue a run that was stopped, from its last checkpoint.
The risk: forgetting
A trained model sits in a narrow region of good weights. If the first updates on the new task are large, they can push it out, and it forgets what it knew. This is catastrophic forgetting. In RL it’s worse, because a policy that has forgotten how to stand collects bad data, and learns from that bad data.
The standard protection is a smaller learning rate.
Resuming isn’t free either
Even resuming the same task can hurt. The optimiser’s running statistics (Adam’s averages) may be reset, and some training state, such as observation normalisation, may be restored differently. The first updates after a restart can be unusually large.
In this project
Both shove-recovery approaches were warm-started from the frozen stance-and-rhythm policy. Later, resuming or warm-starting the mature survive policy at the default learning rate wrecked it within about 20 updates. Dropping to fixed it.
The simple body was trained from scratch instead: no stance parent, starting from level 0 of the curriculum. It reached 860 N shoves in about 40 minutes.
Each jab experiment continues from the best network of the one before, so the jab is a chain of warm starts: for example the most accurate jab so far (85 % hits) came from jabdrive1, warm from jabmidfar1, from jabreach2, from jabposture1.
The punches post goes further. The four new punches were warm-started from the best jab, and without help the jab was forgotten within 2.7 M steps. A frozen copy of the jab policy as a teacher (policy distillation) kept it while the new punches learnt.