FLOPS

A floating-point operation is one addition or multiplication of two real numbers (floating point). FLOPS counts how many a processor does per second; a teraflop (TFLOPS) is a million million.

A processor’s peak is what it could do if every arithmetic unit worked every cycle:

$\text{peak} = \text{cores} \times \text{clock speed} \times \text{operations per core per cycle}$

Real programs get less. Data has to arrive from memory, small matrices don’t keep every unit busy, and threads wait for each other. Training a neural network is mostly matrix products: multiplying an $m \times k$ matrix by a $k \times n$ one takes $2mkn$ operations. Comparing that count with the time taken shows how much of the processor’s peak a program uses. If it is already a good share, a program can only get much faster by doing fewer operations or cheaper ones, for example in bfloat16.

In this project

In Two fighters and vision, one minibatch of the CPU trainer’s update (2,048 samples, the policy and value networks forwards and backwards) is about 25 billion operations. At 12 threads on an Intel Xeon the update ran at about 1 TFLOPS, about half the 12 cores’ theoretical peak, which is typical for matrices this small. More threads and more learner processes gained little, and only bfloat16 made a large difference.