Timestep

A simulator moves time forward in small jumps of $h$ seconds, the timestep. Smaller steps are more accurate and more stable, especially for fast contacts and stiff joints, but cost more computation per simulated second.

$$\text{physics steps per second} = \frac{1}{h}$$

Control frequency

The policy doesn’t need to act on every physics step. It usually runs at a lower control frequency and holds its command for several physics substeps:

$$\text{substeps} = \frac{1/f_{\text{control}}}{h}$$

Human reaction and muscle response happen over tens of milliseconds, so control at 30–100 Hz is typical for simulated characters and legged robots. Acting less often also makes learning easier: each decision has more visible effect, and episodes have fewer steps to assign credit over.

Hz (hertz) means times per second. 50 Hz is every 20 ms.

In this project

The physics timestep is 2 ms (500 Hz). The policy acts at 50 Hz, so each environment step runs 10 physics substeps. A 10 s episode is 500 policy decisions and 5,000 physics steps.