Glossary · Reinforcement learning
Rollout
Also called batch, horizon, iteration
Running the current policy in the environment for a fixed number of steps to collect experience for one training update.
A rollout is the “collect” half of an on-policy algorithm like PPO. The policy runs in many copies of the environment at once, and every step is recorded: the observation, the action, its probability, the reward, and whether the episode ended.
The number of steps per environment is the horizon. A rollout doesn’t have to line up with episodes. If an episode is still running when the horizon is reached, it carries on in the next rollout, and the critic estimates the value of what’s left (see value function).
One rollout plus the update that follows it is one training iteration.
Larger batches give steadier gradient estimates but fewer updates per hour.
In this project
52 worker processes each run 8 environments, with a horizon of 256 steps. One iteration therefore collects steps, about 35 minutes of simulated time. That data is used for 8 passes of updates in minibatches of 512.