Glossary · Reinforcement learning
On-policy
Also called off-policy, on-policy distillation, DAgger
Learning from experience gathered by the policy being trained, rather than from someone else's recordings or from old versions of itself.
A learning method is on-policy when the data it learns from is generated by the current policy. PPO is on-policy: it plays a batch of rollouts with the current network, updates on them a few times, then throws them away and plays again. Off-policy methods can reuse old data or data from another policy, which is more efficient but harder to keep stable.
The distinction also matters for imitation. Copying an expert from the expert’s own recordings is off-policy (behaviour cloning). The copy only ever sees the expert’s states, and once it makes a small mistake it is somewhere the expert never went (covariate shift). DAgger (dataset aggregation) fixes this by letting the learner act and asking the expert what it would have done in the learner’s states. On-policy distillation does the same with a teacher network: the student acts, and the teacher labels the student’s states.
In this project
All training here is PPO, so on-policy. Distillation is on-policy too. Every iteration the student plays 32,768 steps on the GPU, and each step’s teacher, the base model or the one-technique teacher whose technique is running, computes its action for exactly that state. The student is then pulled towards those actions while it is also trained on the task rewards (see Shadow clones).