Glossary · Computing
All-reduce
Also called allreduce, gradient averaging, learner process, learners
A step where several processes each add in their own numbers and all get back the same total, used to keep copies of a network in sync.
When a network is trained by several processes at once, each one works out the gradient (the direction to nudge every weight, see gradient descent) on its own share of the data. Before anyone updates the weights, the gradients have to be combined, so that every copy of the network takes exactly the same step and they stay identical.
An all-reduce does that: every process puts in its array, the arrays are summed (or averaged), and every process gets the result back.
It is not free. Every process has to wait for the slowest one, and the arrays have to be copied between them, once for every minibatch. Splitting the work over more processes makes each one’s share smaller, but the all-reduce stays the same size, so past a point more processes make training slower, not faster.
In this project
The CPU trainer splits each minibatch over a few learner processes, each with several threads, and all-reduces their gradients (Training on a rented GPU and earlier posts tuned the split). In Two fighters and vision, going from two learner processes to three on a 96-vCPU machine cut the gradient time from 1.27 to 1.00 s an iteration but raised the all-reduce from 0.23 to 0.38 s; the iteration still got 9 % faster. With bfloat16 learners a third process gained nothing: its all-reduce ate the saving.