Glossary · Computing
Half precision
Also called float16, FP16, 16-bit float, reduced precision
Storing a number in 16 bits instead of the usual 32: about three significant digits, half the memory.
A computer stores a real number as a sign, an exponent (how big) and a mantissa (the significant digits), see floating point. The ordinary 32-bit format (float32, single precision) has 23 bits of mantissa: about 7 significant digits.
Half precision (float16, IEEE 754’s 16-bit format) has 5 bits of exponent and 10 of mantissa:
That is about 3 significant digits, and numbers from about 0.00006 up to 65,504 (smaller ones lose digits, bigger ones don’t fit). Turning a 32-bit number into half precision rounds it to the nearest value that fits; turning it back is exact, but the rounding stays.
A neural network’s weights rarely need more than a few digits, so storing them in half precision is a common way to halve a model’s size. It is a simple form of reduced precision. Quantisation usually means going further: mapping the weights onto a small set of integers (8-bit or 4-bit) with a scale factor, for a quarter or an eighth of the size, at more risk to the model’s behaviour. bfloat16 is a different 16-bit format: it keeps float32’s range but has even fewer digits.
In this project
The Try now page runs the student policy in your browser. Its three weight matrices (448 → 512 → 512 → 48, about 530,000 numbers) are stored in half precision, 1.0 MB instead of 2.1 MB; the biases and the observation’s normalisation stay 32-bit. The browser widens the weights back to 32-bit before running, so the maths is as fast as before.
On the same 3,000 control steps, the half-precision network’s 48 joint targets differed from the original’s by 0.014% on average and 0.0008 at most. With 40 cubes and 36 punches each, neither network fell.