Throw a cube at him

What you are seeing

The fighter is a physics simulation in MuJoCo: a 1.75 m, 77 kg body with 83 motors, standing in my Muay Thai stance and keeping its rhythm. Every movement comes from a policy, a neural network that reads 465 numbers about its body 50 times a second (joint angles, speeds, how the weight sits on each foot) and sets the targets for every joint. It is the student, the model that all the techniques are distilled into. Its inputs for blocks and the opponent stay at zero here.

Walk / dash puts a stick on the picture. Push it a little and he walks that way, in short steps that settle back into the stance; push it further and the steps flow into a walk. Either way he keeps facing the way he faced when you pushed, so he can walk sideways or backwards, as in the ring. The stick sets a spot on the floor just ahead of his lead foot, the same goal he was trained with, and the training environment's planner turns it into one step per beat of his rhythm. Out at the edge of the stick, past the inner ring, is the dash: one quick step that way, from my motion capture, the foot on that side first and the other following, back into the stance facing the same way. It goes as far as the dash was trained to go in that direction (about 30 to 65 cm), once each time you push the stick out there; with the keys, hold Shift.

Under Punches, Kicks, and Knees and elbows you can pick the same buttons the network was trained with: the jab, cross, both hooks, both uppercuts, the body straight and the pull counter; the right and left high kicks; the rear and switch knees and both elbows. Each section has a random button that throws a different one of its techniques each time. The button on the picture then throws the one you picked. A press tells it which technique and where the target is; the network then throws it, following a reference motion and timing it itself. In the air the target is the see-through sphere. The heavy bag (35 kg) and the uppercuts' ball (25 kg) are real bodies in the physics, so they swing when he hits them and he has to keep his balance against the impact.

The cube weighs 4 to 6 kg and arrives at 6.5 to 8 m/s, aimed at the chest. Nothing was ever thrown at the fighter in training: to learn his balance he was shoved by invisible pushes from random directions. When a hit sends his capture point outside his feet, he goes into survive mode (the label above the picture turns purple): the rhythm stops, his legs get a much wider range to step with, and the only goal is to stay up. Once he is calm again he settles back into the stance.

It all runs on your device. MuJoCo is compiled to WebAssembly and steps the physics 500 times a second; the network's weights are the ones trained in the cloud, and the simulation code is a line-by-line port of the training environment, checked against it step by step. If he does go down, he is back on his feet two seconds later.

Half-precision weights

To halve the download, the network's weights are stored at half precision: 16-bit numbers with about 3 significant digits instead of the 32-bit numbers (about 7) it was trained with. The 2.1 MB of weights become 1.1 MB, and your browser turns them back into 32-bit numbers before it runs, so the maths is no slower. The rounding is not undone, though: this is a very slightly different network from the one in the cloud.

How different? I ran both networks on the same 4,500 moments of this simulation (every technique in the air and on the bag, a cube, walks, dashes and the rhythm). The 48 joint targets they set differed by 0.019% on average, and by 0.001 at the very most on targets about 0.35 in size. Then I let the half-precision network fight alone through all of it: it didn't fall once. (On the earlier student I also ran both networks alone with the same 40 cubes and 36 punches: neither fell.) A physics simulation magnifies tiny differences over time, so the two networks' movements won't match exactly from one second to the next, but I found nothing that changes how he moves or keeps his balance.