vCPU

A vCPU is how cloud providers count processor capacity. Depending on the machine, one vCPU is either a whole physical core or one of the two hardware threads that share a core (hyperthreading). On many Intel-based AWS instances it’s a hyperthread. On AWS’s recent AMD-based instances, such as the c7a and c8a families, each vCPU is a full physical core.

Why reinforcement learning wants many

Physics simulation for RL runs mostly on the CPU. Each copy of the environment is independent, so collecting experience splits cleanly across cores: run one worker process per core, each simulating several environments. The more cores, the more steps per hour, almost in proportion, until something else (memory bandwidth, or the learner waiting for data) becomes the limit.

A graphics card (GPU) is ideal for the neural network’s maths but not for a CPU simulator like MuJoCo. Some newer simulators run thousands of environments directly on a GPU.

In this project

Training runs on a 64-vCPU cloud machine: 52 worker processes, each with 8 environments, and the rest for the learner. That collects about 29 M steps an hour. The 2-vCPU server that hosts this project manages about 0.45 M, roughly 60 times slower, which is why training never runs there.

In the blocking work the hyperthreading difference showed up in the numbers. A c8i machine (Intel, two hardware threads per core) simulated about 1.45 times slower than a c8a (AMD, one core per vCPU) with the same vCPU count, so at 192 vCPUs the launch script skips c8i. Doubling the cores doesn’t always pay either: a 192-vCPU c8a ran only 1.35 times faster than a 96-vCPU one, because half its cores waited for the learner, so it cost $0.055 per million steps against $0.037 (see Blocking).