Behaviour cloning

Behaviour cloning treats imitation as ordinary supervised learning. Record an expert’s states and actions $(s_i, a_i)$, then train a policy to predict the action from the state, for continuous actions often with a squared error:

$$L(\theta) = \frac{1}{n} \sum_i \left\lVert \mu_\theta(s_i) - a_i \right\rVert^2$$

It is simple and needs no rewards. Its weakness is that it only sees the expert’s states. When the copy drifts, it reaches states nobody showed it (covariate shift). Asking the expert to label the learner’s own states (on-policy imitation, as in DAgger) avoids that.

In this project

The fighter’s teachers are networks, so they can be asked about any state, and the student is never trained on their recordings. It is pulled towards their actions on its own rollouts, with the task rewards still on (policy distillation, see Shadow clones).