Function-preserving growth

A trained network sometimes needs to grow: a new input, or more hidden units. If the new parts start with random weights, the network’s outputs change and some of what it learnt is lost before training even resumes. Function-preserving growth adds the new parts so that the network computes exactly the same function as before.

For a new input, the trick is to give its weights into the first layer the value zero. If the first layer computes $h = W x + b$, and a new input $x_{\text{new}}$ is inserted, the new weight matrix is the old one with a column of zeros:

$$W' = [\,W_{\text{before}} \mid \mathbf{0} \mid W_{\text{after}}\,], \qquad W' x' = W x$$

whatever value the new input takes. Gradients still reach the new column, so the network can learn to use the input from the first update.

For wider hidden layers (the Net2Net idea), each new unit gets ordinary random incoming weights but zero outgoing weights. It changes nothing until training gives it a use.

In this project

Every technique has its own input (“this technique is running”), placed just before the last input, the balance gate. A one-technique teacher is the latest model grown by one input this way. The student is grown to all the teachers’ inputs, 448 in all, so on its first step it is the base model exactly (see Shadow clones). One bug in the growing code also rescaled the action outputs for some environments, and that made an untrained, grown copy of the base walk 6/40 instead of 38/40. Once it was fixed, the grown copy scored the same as the base on every row.