Covariate shift

A model is trained on inputs drawn from some distribution. Covariate shift is when it is used on inputs from a different one. It may still be right in theory, but it has no experience there.

For a policy that controls something, covariate shift compounds. A policy copied from an expert makes a small error, which puts it in a state slightly unlike any the expert showed it. There it makes a larger error, and so on. With a per-step error rate $\epsilon$ over $T$ steps, the expected cost of plain imitation can grow like $\epsilon T^2$, against $\epsilon T$ when the learner is trained on its own states (Ross, Gordon and Bagnell, 2011, the DAgger paper).

The fix is to train on the states the learner actually visits (see on-policy).

In this project

A fighter that leans a few degrees too far on a knee is, a moment later, in a state the knee teacher never was in. That is why the student is pulled towards its teachers’ actions on its own rollouts and not on recordings of theirs (see Shadow clones).