Profiling

A program is usually slow because of one or two parts, and they are rarely the ones you would guess. A profiler runs the program and records how long each function, or on a GPU each kernel, takes, and how often it runs. The output is a list sorted by time.

Two rules make it useful:

• Speed up the biggest item first. If a part takes 10 % of the time, making it infinitely fast saves only 10 %.
• Measure again after each change. Removing one bottleneck reveals the next, and a change that should help sometimes doesn’t.

GPU work is harder to time because the CPU sends work ahead and carries on. A timer on the CPU measures sending, not doing, unless it waits for the GPU or uses the GPU’s own timers.

In this project

In Training on a rented GPU, profiling on an RTX 5090 found one kernel, the mass-matrix factorisation, taking 53 % of the GPU time for the muscular body. Replacing it made the physics 28 % faster. A later profile of the simple body found 21 ms of each step outside the physics, more than the 11 ms of physics, 8 ms of it in looking up the reference motion. Earlier, profiling a punching run on the CPU found that most of the muscular body’s contacts were its own fingers touching inside the fists (Blocking).

For the blocking environment (Two fighters and vision), profiling found 11,700 of a GPU step’s 13,700 kernels were tiny PyTorch operations, and on the CPU that MuJoCo itself was only about 1.6 ms of a worker’s 4.4 ms a step: the rest was the project’s own Python and numpy around it.