How full is a fighter's brain? A network-size experiment

A network that trades skills

The fighter is a policy: fifty times a second it reads 465 numbers about its body, its balance and which technique button is pressed, and outputs 48, a target angle for every joint motor. One network, the student, does everything on the try page. It learns by policy distillation: for each technique there is a teacher, a separate network trained on that one technique, and the student is trained to act like the right teacher whenever that technique runs, and like the base teacher (guard18, the stance, punches, walking and blocks) the rest of the time. In October that meant sixteen teachers poured into one student.

The last few distillations had a pattern. distill12, the student before this experiment, came out with better blocks and kicks than the one before it, and worse knees (their gate score down from 11 to 8) and a worse left elbow (52 down to 49). Earlier students had shown the same kind of trade. A network that can only learn one thing by forgetting another is the textbook sign of a full one (catastrophic forgetting is the extreme case). But there are at least two other explanations: the training budget (each run is a few hours; maybe the student just needed longer), and the teachers themselves, which were trained separately and can ask for incompatible things in similar situations.

Neural networks don’t come with a gauge. There is no reading that says “83 % full”. So the question has to be answered by experiment: give the same student more room, change nothing else, and see whether it does better.

The design

Four networks, one starting point. The obvious experiment, training a bigger network from scratch, would change two things at once: the size and everything the student had already learnt. Instead, every network started as the same function: the kept student distill12-196.6M, grown by function-preserving growth.

• Wider. Each hidden layer gets 512 new neurons. Their incoming weights are random, like any new neuron’s; their outgoing weights are all zero. Whatever the new neurons compute, nothing downstream reads it yet, so the network’s outputs are exactly the old ones. Training then gives the new neurons a use. This is the widening of Net2Net (Chen et al., 2016), with zeros instead of a copy.
• Deeper. A new hidden layer is added just before the output. Its weights are the identity matrix and its bias is +1. The network’s activation function, the ELU, is the identity for positive inputs, and the layer before always outputs more than −1, so after the +1 every input to the new layer is positive and passes through unchanged. The output layer’s bias subtracts the extra 1 again:

$$h_{3} = \mathrm{ELU}(I\,h_{2} + \mathbf{1}) = h_{2} + \mathbf{1} \quad\text{since } h_{2} > -1, \qquad y = W_{\text{out}}\,h_{3} + (b_{\text{out}} - W_{\text{out}}\mathbf{1}) = W_{\text{out}}\,h_{2} + b_{\text{out}}$$

Before launching, I checked this on the real student with 4096 random inputs: the widened policy’s outputs were identical to the original’s (largest difference 0.0), and the deepened one’s differed by 4 × 10⁻⁵, the size of floating-point rounding. Any difference between the four after training is therefore due to their size and nothing else.

A 2 × 2 design. Width and depth were varied separately and together, so their effects can be told apart:

network	hidden layers	policy parameters	relative size
512 × 2 (control, the original)	2 of 512	525,872	1.0
1024 × 2 (wider)	2 of 1024	1,575,984	3.0
512 × 3 (deeper)	3 of 512	788,528	1.5
1024 × 3 (both)	3 of 1024	2,625,584	5.0

(The critic, a second network of the same shape that estimates how well things are going, grows the same way and isn’t counted here.)

The control is the original network trained exactly like the others. It separates “more training helps” from “more room helps”: if the control improves as much as the grown networks, the student simply needed more steps.

Equal steps. Every network trained for exactly 200 million environment steps with distill12’s settings unchanged: the same environments, the same sixteen teachers with the same weight, the same learning rate and the same mix of practice. A bigger network is slower per step, so equal steps means unequal hours; the cost is reported below rather than levelled out.

Two measures. The first is how close the student gets to its teachers, logged throughout training for each teacher separately. The second is the gate, the same fight tests every candidate student goes through, on each network’s final checkpoint.

Distance to the teachers

During training, before every update, the trainer takes the states the student actually visited in its last rollouts and asks the teacher in charge of each state what it would have done. The distance is the mean squared difference between the student’s action and the teacher’s, averaged over the 48 joint commands, and kept separately per teacher:

$$d_g = \frac{1}{|S_g|} \sum_{s \in S_g} \frac{1}{48} \sum_{j=1}^{48} \big(\mu_j(s) - a^{\text{teacher}}_j(s)\big)^2$$

where $S_g$ is the set of visited states in which teacher $g$ is in charge, $\mu(s)$ the student’s action (the mean of its action distribution) and $a^{\text{teacher}}(s)$ the teacher’s. Measuring on the student’s own states matters: a student that drifts into situations its teachers never saw is judged there, not on the teachers’ home ground. Below, distances are multiplied by 1000.

Result 1: bigger networks get closer to their teachers

steps	512 × 2	1024 × 2	512 × 3	1024 × 3
0–25 M	4.60	4.66	4.87	4.91
50–75 M	4.48	4.37	4.42	3.98
100–125 M	4.43	4.03	4.03	3.33
175–200 M	4.28	3.75	3.71	2.88
change	−7 %	−20 %	−24 %	−41 %

Three things stand out.

The control is near its limit. With another 200 M steps of the same training, the original network improved by only 7 %, and its curve is close to flat by the end. More training alone would not have closed the gap.

Width and depth helped about equally, and together most. 1024 × 2 has twice the parameters of 512 × 3 and ended in the same place (3.75 against 3.71). Per parameter, the extra layer was the better buy. The two together did better than either: 1024 × 3’s improvement (41 %) is close to the sum of the other two (20 % + 24 %).

The grown networks start slightly behind. In the first 25 M steps the bigger networks are a little further from their teachers than the control (4.87–4.91 against 4.60), although they started as exactly the same function. The new weights are trained from their first update, and with the Adam optimiser, which scales each weight’s step to its own gradient history, the new ones move as fast as the old. That disturbs the inherited behaviour for a while before it pays off. This is my reading, not something I measured separately; it costs about 10–20 M steps.

The gap is everywhere. If one or two teachers had been in conflict, the extra room would have helped those techniques and left the rest alone. Instead, 1024 × 3 is 30–35 % closer than the control on fifteen of the sixteen teachers, from the kicks, the hardest to imitate (distance 5–7), to the knees and the body straight (2–4). That includes the base, the stance, punches, walking and blocks that were in the network first (0.30 to 0.21). The dash, already the easiest after the base, improves the least (20 %). An even gap across every teacher is what a shortage of room looks like, not a quarrel between two teachers.

Result 2: in the fight tests, no clear difference

Distance to a teacher is a training measure. What matters is whether the fighter fights better, and that is what the gate tests: every technique at aim points in the air and at a heavy bag, with six random seeds; 120 punches; 40 walking goals; 32 blocks; 160 punches from the opponent to defend; punches at 60 set distances; the body straight. Each row counts successes out of a fixed number of tries, played without exploration noise.

For each row I took the difference between a network’s success rate and the control’s, with a 95 % confidence interval (the Agresti–Caffo interval, which behaves sensibly when a row is at 0 or at every try). Rows where all four networks scored the same, such as every kick in the air at 30 of 30, are left out: 21 rows remain.

against 512 × 2	better	worse	tied	clearly different	sign test
1024 × 2	9	10	2	1 (right uppercut at the bag, 3 → 11 of 54)	p = 1.0
512 × 3	8	8	5	0	p = 1.0
1024 × 3	8	10	3	0	p = 0.82

Not one of the 1024 × 3 rows is clearly different from the control. It is ahead on the punches (61 against 56 of 120), the pull counter in the air (6 against 3 of 6) and the mid kicks and left low kick at the bag; it is behind on the blocks (24 against 26 of 32), the knees in the air, the right elbow and the left high kick in the air (23 against 30 of 36). Counting rows instead of looking at each, it is better on 8 and worse on 10: a sign test gives p = 0.82, no evidence either way. The one “clear” row for 1024 × 2 is what chance alone would produce: with 21 rows and 95 % intervals, about one should fall outside by luck.

Why the two measures disagree

A network 41 % closer to its teachers that fights no better needs explaining. Four reasons, from the most to the least certain:

The gate can’t see differences this size. Its rows have 6 to 160 tries. With 30 tries at a success rate near 70 %, a 95 % interval on a difference is about ±23 points wide; with 6 tries, more than ±50. To detect a real 10-point improvement on a single row with the usual 80 % power would take about 330 tries per network. The gate is built to catch a student that has broken something, which it does well, not to rank four students that are all good.

Squared distances exaggerate. The distance is a mean of squares. Its square root, a typical size of the difference in one joint command, falls from 0.065 to 0.054, 18 %, not 41 %. Most of the gain is the bigger network matching the teachers’ larger, rarer actions better, and those may matter less to a hit than to the distance.

The teachers are not perfect either. The fight tests score the student’s outcome: did the knee land within 5 cm at speed? A student that copies its teacher more closely inherits the teacher’s misses too. Once the student is close enough to land what its teacher lands, getting closer still adds little.

200 M steps may be early. 1024 × 3’s distance was still falling steeply when the run stopped. A capacity advantage might show in the fight tests only once the gap in closeness is larger still.

So the honest summary is two-sided. The small network is at its capacity: by the measure it is trained on, more training stopped helping and more room helped on every skill. Whether that capacity is what limits the fighter’s techniques today, the experiment cannot say. Within what the gate can resolve, it isn’t.

What it cost and which machine

Each network trained on its own rented RTX 5090 on vast.ai, all four at once. The bigger networks were slower per step, but much less than their size: the simulation of the body, not the network, takes most of each step.

network	steps per second	hours for 200 M steps
512 × 2	25,400	2.19
1024 × 2	23,900	2.33
512 × 3	24,100	2.30
1024 × 3	21,900 (−14 %)	2.54

Five times the parameters cost 14 % in training speed. The gate is where the size shows: it runs the policy on ordinary CPUs, and 1024 × 3’s gate took 49 minutes on 12 cores, against 30–45 minutes for the others on 16–32. Seven more boxes were rented and never came up (no connection after 4 minutes, or still loading after 10); they were destroyed within minutes and are counted in the cost at the top. Every machine is listed in the work log below.

The decision

I kept 1024 × 3, cap1024x3-200.0M, as the parent of everything that follows. The rule I set before seeing the gate was simple: if nothing in it is clearly against the biggest network, take it. Nothing was. The reason is what comes next. The plan for the coming weeks adds to what this one network has to hold: aiming at a moving opponent, a queue for the next technique so strikes can be joined into combinations, and seeing where the opponent is. Each of those is new behaviour in the same weights. A network that has already shown it has room to spare on every current skill is the better place to put them.

There were bigger changes on the table if size alone hadn’t been enough: a mixture of experts, where a small network learns to blend several specialised ones (Peng et al., 2019), or a learned space of skills that a higher-level controller picks from (Peng et al., 2022). Both are much larger changes than adding neurons. This experiment says they aren’t needed yet.

Limitations

• One run per network. Each size was trained once. The curves are smooth and the gaps large compared to their wobble, so I trust the ordering of the distances; I can’t put an error bar on them.
• Different machines. The four runs were on four hosts (France, Norway, Vietnam, Canada). The steps-per-second figures mix network size with hardware.
• Equal steps, not equal compute. A fair comparison per hour would give the small network about 16 % more steps. Given how flat its curve was, that would not have closed a 33 % gap.
• Growth, not training from scratch. The result says how much extra room helps a student that has already learnt. A network trained at 1024 × 3 from the beginning might end up elsewhere.

What I think, as a coach

This experiment was one of several I asked to run in parallel on 31 October, before the bigger job of the week, the combo-aiming plan: “launch a bunch of steps that can run in parallel, for example … whether we need to increase the network size or more layers”. Before the fight tests came back I had already said “if nothing obviously against it, let’s go 1024x3”, and that is what happened.

When I made the short film at the top of this page, I wanted it to open on the question I care about, “how do we know when the AI’s brain is full”, and to end on what the answer is for: now that the brain is ready for more training, more Muay Thai techniques are coming.

References

• Agresti, A. and Caffo, B. (2000). Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. The American Statistician 54(4).
• Chen, T., Goodfellow, I. and Shlens, J. (2016). Net2Net: accelerating learning via knowledge transfer. ICLR.
• Peng, X. B., Chang, M., Zhang, G., Abbeel, P. and Levine, S. (2019). MCP: learning composable hierarchical control with multiplicative compositional policies. NeurIPS.
• Peng, X. B., Guo, Y., Halper, L., Levine, S. and Fidler, S. (2022). ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics 41(4).
• Rusu, A. A. et al. (2016). Policy distillation. ICLR.

The work logs