Glossary · Computing
GPU
Also called graphics card, CUDA, graphics processing unit
A graphics processor: a chip with thousands of small cores that do the same sum on lots of numbers at once. Fast for neural networks, less useful for CPU physics.
A GPU (graphics processing unit) was built to draw images: the same small calculation for millions of pixels at once. A CPU has a few powerful cores that are good at varied, branching work. A GPU has thousands of simpler ones that are good at one thing done many times in parallel.
That suits neural networks, which are mostly large multiplications of tables of numbers. Training big models such as chat assistants is done almost entirely on GPUs.
Why it helps less here
In reinforcement learning, time goes on two jobs: simulating the world to collect experience, and updating the network from it. A CPU simulator like MuJoCo steps one copy of the world on one CPU core, with lots of contact checks and branches that a GPU handles poorly. If the network is small, its update is quick on a CPU too. So a GPU only speeds up the smaller job.
Some newer simulators (MuJoCo’s MJX, Isaac Gym) run thousands of worlds on a GPU instead. That changes the picture, but needs a simulator written for it.
In this project
The EC2 trainer runs the physics on CPUs and the small network beside it, so a GPU added little there. Training on a rented GPU moved the stance environment’s physics onto the GPU with MuJoCo Warp. After the code was tuned for it, a GPU rented on vast.ai (an RTX 5080 or L40S, about $0.40 an hour) solved the stance for about $0.065, against $0.30 on a 96-vCPU EC2 spot machine, and in less time. Consumer cards beat data-centre ones on cost: the physics is 32-bit arithmetic in many small steps, and wants clock speed more than memory.