Glossary · Computing
Embarrassingly parallel
Also called parallel training, trivially parallel, running in parallel
A job that splits into pieces that don't need each other, so each piece can run on its own machine at the same time.
A job is embarrassingly parallel when it splits into tasks that never have to talk to each other while they run. With machines, such tasks take about as long as one. The only shared work is handing out the inputs at the start and collecting the results at the end.
Most jobs are not like that. A single training run, for example, updates one set of weights, and every step depends on the step before. Splitting one run over many machines means keeping their copies of the weights in step (see all-reduce), which costs communication.
The cost of an embarrassingly parallel job is the sum of its pieces, whether they run one after another or all at once. Running them at once saves wall-clock time, not money.
In this project
Each new technique is trained in its own copy of the latest model, a one-technique teacher, on its own rented GPU (see Shadow clones). The teachers share nothing while they train. Only the step that brings them back together, distillation, needs all of them. The two knee teachers trained side by side at every stage, and four evasion teachers started within a minute of each other on four RTX 5090s, each running about 2.1 hours for $1.14–1.56.