Mixture of experts

One way to give a model more room without making every part of it bigger is to split it. A mixture of experts has several networks, the experts, and a small gating network that looks at the input and outputs a weight for each expert. The answer is the experts’ outputs blended by those weights:

$$y(x) = \sum_{k} g_k(x)\, e_k(x), \qquad \sum_k g_k(x) = 1$$

The experts can specialise, each becoming good at the inputs the gate sends it, so adding skills need not crowd out old ones. Large language models use sparse mixtures where only a few experts run for each input, to grow the number of weights without growing the work per step.

For controlling a body, Peng et al. (2019) proposed MCP (multiplicative compositional policies): each expert proposes a distribution over actions and the gate combines them multiplicatively, so several “primitive” skills can act at once, one steering the legs and another the arms.

In this project

It was one of the larger options in the plan if the fighter’s single network had run out of room. The network-size experiment showed that simply making the network wider and deeper gave every skill room to spare, so the fighter is still one network for now (see model capacity).