Shadow clones: four Muay Thai techniques learnt in parallel

This post is written in the first person. Earlier posts talked about “Tony” and “the coach”. From now on it’s just me.

The Naruto idea

In Naruto Shippuden there is a scene I kept thinking about while planning this. Naruto has to master a technique in far less time than it should take. Kakashi’s solution is the shadow clone. Naruto makes a crowd of copies of himself, they all train at once, and when a clone is dispelled, everything it learnt goes back to the original. A week of practice fits into a day. There is a catch: the clones’ fatigue comes back too.

That is how I wanted to add techniques to the fighter. Until now every new skill was fine-tuned into one model, one after another, with the older skills held in place by a teacher. Each new technique waited for the one before it, and each one risked the others (catastrophic forgetting). Training techniques separately is close to embarrassingly parallel. A knee doesn’t need to wait for a body straight, so they can run on different machines at the same time.

So the plan was:

1. Take the latest model (the base) and make a clone of it for each new technique.
2. Train each clone on its own rented GPU. It only has to learn its technique and keep the stance and balance: “all it needs to know is the stance and the balance and just the specific technique”.
3. Bring the experience back: distil the clones that pass into one student, which keeps everything the base could do.

The rest of this post describes how a clone is made, how the experience is brought back, and what went wrong. Every run is in the work logs.

Making a clone: the one-technique teacher

A teacher starts as a copy of the base: a multi-layer perceptron with two hidden layers of 512 units. It reads 448 inputs and drives 48 motor outputs. Three things turn the copy into a teacher.

A new input, added without changing the network. Each technique gets its own input, “this technique is running”, placed just before the last input (which is the balance gate). The new input’s weights into the first layer start at zero. If $W$ is the first layer’s weight matrix, its columns are split as $W = [\,W_{\text{old}} \mid w_{\text{gate}}\,]$ and the grown network uses

$$W' = [\,W_{\text{old}} \mid \mathbf{0} \mid w_{\text{gate}}\,]$$

so on the first step the grown network behaves exactly like the base (function-preserving growth). It can only start using the input through training. Every teacher reads the same input layout the final student will have, so the student can later learn from all of them on the same observations.

Practice only that technique. Every episode is a punch episode, and every press is the teacher’s technique, with shoves in between so the balance is still tested.

The parent holds everything else. A frozen copy of the base stays on as a teacher everywhere except while the new technique runs (see policy distillation). The teacher is free to change its motion during the technique, and is pulled back to the base’s behaviour the rest of the time. That keeps the stance and the recovery from shoves, which the technique has to start from and end in.

Each technique’s motion comes from my own motion capture, the reference motion the teacher is rewarded for following. For example, the rear straight to the body is a 1.45 s window of one of my takes.

Running the clones side by side

The trainer runs on one GPU: 2,048 copies of the world (parallel environments) for 16 steps each, 32,768 steps per iteration. On a rented RTX 5090 that is about 25,000 to 34,000 steps a second. Each clone gets its own machine. Two knee teachers trained side by side, then their fixes ran side by side. Four evasion teachers started within a minute of each other. Two uppercut teachers ran on two RunPod machines at once.

technique	teacher kept	runs	from the parent	what it took
rear straight to the body	bodycross4-118.0M	4	guard14 (guard18 holding the rest in the last run)	a “no low blows” line, the bag moved to the impact point, then retrained on the base’s slower elbows
rear knee	kneeR3-183.5M	3	guard14	the feet kept where the capture has them, then the steps home at the capture’s pace
switch knee	kneeL3-214.0M	3	guard14	the same
pull counter	pull1-209.7M	1	distill1-282.5M	8/8 in the air and 8/8 at the bag, first try

Training each teacher takes about two hours. Most of the time and money goes into the fixes: several short runs, each warm-started from the last good checkpoint with one change. Because the clones are independent, a failed fix on one technique costs nothing on the others.

Bringing the experience back: distillation

The student starts as the base, grown to read every teacher’s input. It keeps training on the full mix of episodes (punches, blocks, walking, the opponent, shoves and the new techniques). Its loss is PPO’s plus a pull towards the right teacher.

The student’s policy is a Gaussian, $\pi_\theta(a \mid s) = \mathcal{N}(\mu_\theta(s), \sigma^2)$ (see stochastic policy). There are $K$ technique teachers $T_1 \dots T_K$ plus the base $T_0$. Each state is routed to one of them by its inputs:

$$k(s) = \begin{cases} j & \text{if technique } j\text{'s input is on in } s \\ 0 & \text{otherwise (the base)} \end{cases}$$

and every minibatch $B$ of the student’s own experience is trained on

$$\begin{aligned} L(\theta) = {} & L^{\text{clip}}(\theta) + \tfrac{1}{2} L^{V}(\theta) + L^{\text{smooth}}(\theta) \\ & + \lambda \, \frac{1}{|B|} \sum_{s \in B} \left\lVert \mu_\theta(s) - \mu_{T_{k(s)}}(s) \right\rVert^2 \end{aligned}$$

The first three terms are the usual PPO training: the clipped policy objective, the value error and a smoothness penalty on the actions (CAPS). The last is a squared error between the student’s mean action and the routed teacher’s mean action, with weight $\lambda$ (2 in the first runs, 4 in the kept one). A few details matter:

• The states are the student’s own. The teachers label the states the student actually visits (on-policy), not states from their own runs. Copying a teacher from its own recordings (behaviour cloning) breaks down once the student drifts off the teacher’s path, because it has never seen those states (covariate shift). Asking the teacher what it would do in the student’s states is the DAgger fix.
• Each teacher sees what it was trained on. A teacher reads its own slice of the inputs, normalised with its own running statistics. An earlier bug, a teacher seeing inputs scaled by the student’s statistics, had made its advice slightly wrong.
• The rewards stay on. The student still earns every task reward, so it can’t just copy a teacher into a fall.
• One mean, many teachers. Exactly one teacher is in charge of each state, so the targets never conflict. The student has to fit all of them in one network.

The trainer logs one number per teacher to watch: the student’s mean squared error against that teacher’s action means, over the states routed to it, before each update. If the error stalls high on the new techniques while the base’s stays low, the network has run out of room (capacity), and I would widen it. It never stalled. In the first run the base’s error stayed at 0.002 throughout. The body straight fell from 0.028 to 0.005 by 39 M steps, the rear knee from 0.054 to 0.007 and the switch knee from 0.059 to 0.008. They were still falling when the run ended at 282.5 M. Three techniques fit in 512 units with room to spare.

What came back

Every candidate is judged by the gate: deterministic tests of every skill, the old ones and the new ones. Six distillation runs went through it. Figure 2 compares the base with three students, and the table has the full rows.

	punches /120	cross in the air /12	walk /40	blocks /32	opponent /160	distances /60	body straight /24	knees R, L /12	falls
guard18 (base)	59	3	39	27	114	34	–	–	0
distill1 282.5 M	53	0	39	25	109	36	(not in yet)	6/8, 5/8 at 40 M	0
distill2 52.4 M	54	0	38	29	122	38	21	10, 12	0
distill4 131.1 M	60	3	39	24	117	39	20	11, 9	0

The student was picked from snapshots, not from the end of the run. The training averages hardly moved, but the deterministic gate swung from one snapshot to the next (blocks between 21 and 29 in distill3). distill4’s five snapshots were all gated, and the one at 131.1 M was kept. What made distill4 better than distill2 was a stronger pull ($\lambda$ from 2 to 4) and more practice for the cross. That brought the cross in the air back, from 0/12 to the base’s 3/12, and tightened every teacher’s match. The cost is 3 blocks, and putting the block share back (distill5) didn’t recover them.

Not everything comes back

distill6 added the fourth teacher, the pull counter (lean back, then a rear straight to the head). The teacher hits it 8/8 in the air and 8/8 at the bag. The student learnt it at the bag (6/6) but not in the air (1–2/6, about 10 cm short), and none of its snapshots beat distill4 overall. So for now the pull counter lives in its clone, not in the fighter. The next distillation gives it more presses and a stronger pull.

It is a reminder that distillation is not like Naruto dispelling a clone. The experience doesn’t arrive whole. It has to be trained in, alongside everything else the student must keep, and where two skills compete for the same output the result is a trade.

Clones that finished and were still wrong

The worse surprise was clones that finished training, passed or nearly passed the gate, and were wrong when I watched them. The gate says a technique landed. It doesn’t say the technique looked like the one I asked for. These are the main cases. The details, with the numbers, are in the work logs.

The reference. The evasion teachers (slips, rolls, lean back) learnt from my motion capture, but they ran it up to twice as fast. The punch clock, which lets the policy speed up a punch, had been built for punches and was still on for these. A roll squeezed that much becomes a drop. When I asked for a wider, U-shaped head path and the reference itself was widened, the trunk bent sideways to reach it, and the teachers learnt the bend. Every roll and lean-back teacher was judged a failure on video. The slip uppercut is kept only until a better one exists (“looks like a big uppercut [more] than a slip”). They all start again from the original capture.

The training policy. The lead uppercut over distance landed 40/54 in the air, but after it the fighter turned 90° to his right and stayed there. The cause was not the reference, which ends square at every depth. Training rewarded a home heading that was reset at every press and every recovery from a shove, and the opponent turned to face wherever the fighter faced. Nothing paid for facing one direction. Both uppercuts are now retraining with the heading fixed per episode, and the gate now measures the turn after every technique. Smaller cases of the same kind:

• A knee that landed in a southpaw stance 0.6 m forward, fixed with terms for the feet.
• A body straight that went below the belt, fixed with a line it may not cross.
• A body-straight teacher trained on faster elbows than the student has, which missed once distilled and had to be retrained.

Start again, or restart from a checkpoint? Most fixes warm-start from the last checkpoint that was good where it mattered. A teacher whose problem is in its reference has to start again from a checkpoint before it learnt the wrong motion. And the first stepping attempt was given up altogether: the long step never learnt to push off the back leg, and those references will be rebuilt from a better motion capture.

What I think, as a coach

The shadow-clone idea worked the way I hoped where it worked. Three new techniques arrived in one model without breaking any of the old ones, and none of the clones had to wait for another. It also changes how I coach the fighter. I can look at one technique at a time, in a clone that does nothing else, and correct it before it goes into the student.

The failures were mostly the kind a coach knows from people. A student who rushes a slip because nobody told them the timing matters. A fighter who finishes a combination facing the wrong way because their drills never asked them to stay square. My comments on the videos were all about shape and timing, things the numbers didn’t catch:

The roll should not just be a drop, it should roll to the side like a U shape.

Slip and roll are all wrong. It’s going sideways but not by bending the torso.

Keep to my original reference, like the one from motion capture.

After the uppercut the fighter turns 90 degree to the right. It needs to keep facing the opponent.

The lesson I take from these is that the motion capture has to stay mine. Every time the reference was edited to get a cleaner number, the movement got less like Muay Thai.

The work logs

Everything behind this post is in eight work logs, one per area. Each says why every run was started, what changed, what the gate and the videos showed and what it cost. Together they cover every run since the last post, and they add up to the cost at the top of this page and its row on the stats page. The costs are from the vast.ai and RunPod ledgers, each machine’s billed hours times its hourly price. Machines that never came up and RunPod refusals are in them too.

Two of them are not about this post’s techniques but cost money in the same days. The higher guard is the retraining that produced the base every clone started from. Fast stepping is the abandoned attempt mentioned above.

To be continued

This is the first part. The uppercut teachers are retraining with the facing fixed. The evasions start again from my original capture, with the punch clock held to the capture’s pace. The pull counter goes into the next distillation with the uppercuts. Kicks are waiting for better motion capture. Each new technique will be one more clone, and each distillation one more round of bringing the experience back.