Glossary · Computing
GPU kernel
Also called kernel, kernel launch, launch, CUDA kernel, thread block
One small program sent from the CPU to the GPU, run by thousands of GPU threads at once; every launch has a small fixed start-up cost.
A GPU doesn’t run a whole program by itself. The CPU sends it work one piece at a time. Each piece is a kernel: a short function that thousands of GPU threads run at once, each on its own part of the data (one pixel, one number in a table, one copy of a simulated world). Threads are grouped into blocks that can share memory and wait for each other.
Every launch costs a few microseconds before any work starts, for the CPU to prepare it and for the GPU to start it. With large data that cost is lost in the work. With many small kernels it adds up, and the GPU spends its time waiting. Then speed depends on how fast the CPU can send kernels, which is why the same GPU can run slower in a machine with a slow processor.
The usual fixes: do more in each kernel (fuse several into one), and record a sequence of kernels once and replay it as a CUDA graph.
In this project
In Training on a rented GPU, the muscular body’s mass-matrix factorisation took 24 kernel launches per factorisation, one per level of the body’s tree. A replacement does it in one launch, with a block of 64 threads per copy of the world waiting for each other between levels: 38.6 to 8.9 ms of GPU time per step. A step of the simple body still ran about 1,900 small kernels outside the physics until the reference tables were read in one go.