Environment step

An environment step is one pass of the loop: the policy sees an observation, picks an action, and the environment advances and returns a reward. Each step produces one training sample.

Training length in RL is usually given in steps rather than hours, because steps don’t depend on the computer. “75.6 M steps” means 75.6 million of these samples, gathered across all the parallel environments.

Steps and simulated time

An environment step isn’t the same as a physics timestep. Each environment step may run several physics steps. If the policy acts at $f$ times a second, $N$ steps cover $N / f$ seconds of simulated time:

$$75.6 \times 10^6 \text{ steps} \div 50\ \text{Hz} = 1.5 \times 10^6\ \text{s} \approx 17.5 \text{ days}$$

So the simple body learned to take 860 N shoves in about 17 days of simulated experience, compressed into 1.5 hours on 64 CPU cores.

Sample efficiency

How many steps an algorithm needs to learn a task is its sample efficiency. PPO isn’t very sample-efficient, but each step is cheap in simulation, and it parallelises well. That trade is usually worth it when the simulator is fast.