Glossary · Reinforcement learning
Parallel environments
Also called vectorised environments, vectorized environments, copies of the world, worlds, num envs, batch of environments
Many copies of the same training world run side by side, so one policy collects experience from all of them at once.
A reinforcement learning agent learns from experience, and one copy of an environment produces it slowly. So trainers run many copies at once, each with its own random start, and the same policy acts in all of them. Each training iteration (a rollout) steps every copy a fixed number of times, the horizon, and the update uses all of it:
On a CPU the copies are spread over worker processes, a few per core. On a GPU, with a simulator such as MuJoCo Warp, thousands of copies step together.
More copies give more steps a second, but not always faster learning. Each iteration takes longer, and some things in training are counted in iterations, not steps.
In this project
The EC2 trainer runs 72 worker processes with 6 copies each, horizon 64: 27,600 steps an iteration. On the GPU, Training on a rented GPU found that 8,192 copies (horizon 32) gave the most steps a second but hadn’t solved the stance in 12 minutes, because the curriculum waits a fixed number of iterations after every raise. 1,024 copies solved it in 386 s on an RTX 5090.