Throw a cube at him

What you are seeing

The fighter is a physics simulation in MuJoCo: a 1.75 m, 77 kg body with 83 motors, standing in my Muay Thai stance and keeping its rhythm. Every movement comes from a policy, a neural network that reads 448 numbers about its body 50 times a second (joint angles, speeds, how the weight sits on each foot) and sets the targets for every joint. It is the student, the model that all the techniques are distilled into. Its inputs for blocks and the opponent stay at zero here.

Walk / dash puts a stick on the picture. Push it a little and he walks that way, in short steps that settle back into the stance; push it further and the steps flow into a walk. Either way he keeps facing the way he faced when you pushed, so he can walk sideways or backwards, as in the ring. The stick sets a spot on the floor just ahead of his lead foot, the same goal he was trained with, and the training environment's planner turns it into one step per beat of his rhythm. Out at the edge of the stick, past the inner ring, is the dash: it lights up, but he still walks, because the dash is still being trained and isn't in this network yet.

Under Punches you can pick the same buttons the network was trained with: jab, cross, both hooks, the lead uppercut and the body straight. The button on the picture then throws the one you picked. A press tells it which punch and where the target is; the network then throws it, following a reference motion and timing it itself. In the air the target is the see-through sphere. The heavy bag (35 kg) and the uppercut's ball (25 kg) are real bodies in the physics, so they swing when he hits them and he has to keep his balance against the impact.

The cube weighs 4 to 6 kg and arrives at 6.5 to 8 m/s, aimed at the chest. Nothing was ever thrown at the fighter in training: to learn his balance he was shoved by invisible pushes from random directions. When a hit sends his capture point outside his feet, he goes into survive mode (the label above the picture turns purple): the rhythm stops, his legs get a much wider range to step with, and the only goal is to stay up. Once he is calm again he settles back into the stance.

It all runs on your device. MuJoCo is compiled to WebAssembly and steps the physics 500 times a second; the network's weights are the ones trained in the cloud, and the simulation code is a line-by-line port of the training environment, checked against it step by step. If he does go down, he is back on his feet two seconds later.

Half-precision weights

To halve the download, the network's weights are stored at half precision: 16-bit numbers with about 3 significant digits instead of the 32-bit numbers (about 7) it was trained with. The 2.1 MB of weights become 1.0 MB, and your browser turns them back into 32-bit numbers before it runs, so the maths is no slower. The rounding is not undone, though: this is a very slightly different network from the one in the cloud.

How different? I ran both networks on the same 3,000 moments of this simulation (cubes, punches and the rhythm). The 48 joint targets they set differed by 0.014% on average, and by 0.0008 at the very most on targets about 0.33 in size. Then I let each one fight alone with the same 40 cubes and the same 36 punches (each punch in the air and on the bag, at three heights): neither fell once. A physics simulation magnifies tiny differences over time, so the two networks' movements won't match exactly from one second to the next, but I found nothing that changes how he moves or keeps his balance.