Coaching an AI fighter

Tony Yum has coached Muay Thai for years. Teaching an AI to throw a punch is a new journey for him. He has been surprised by how familiar it feels. Most of what he knows from teaching people carries straight over.

Real technique, not stock motion

Most papers and videos on simulated martial arts start from stock motion: motion capture clips bought from asset stores, or animations made for game engines such as Unreal. The fighter learns to copy whatever is in the clip, right or wrong.

This project has a different goal: motion as close to real as possible, for the martial arts Tony knows. For now that is Muay Thai. Brazilian Jiu-Jitsu will come later. Grappling means two bodies pressing, gripping and rolling on each other, and the maths behind that is much harder, so striking comes first.

Coaching a technique has two phases. First comes the reference: what the technique should look like. Then comes the training: the fighter learns to do it for real, under physics.

Phase 1: the reference

A reference motion is the target movement that the fighter is later rewarded for following. It is made on the MuJoCo body itself. There are two bodies, a muscular one and a simple one (see the balance post). References are made on the muscular body, so a reference can be used as it is, without a layer of translation from one skeleton to another.

Tony works with a coding agent: an AI that writes and runs the code. He talks to it from his phone. It sends back pictures of the key moments (the start, the impact, the return) or a short low-resolution clip for him to look at. He has tried three ways of getting to a good reference.

Just ask. The first way is to ask the agent to make, say, a jab. What comes back usually looks like a punch, but it isn’t correct. So Tony corrects it a bit at a time: keep the elbow in, step forward, turn the hip, screw the fist over as it goes out, tuck the chin. There are a great many small things to fix.

Describe it all up front. The second way is to describe the whole technique at the start, then try, look and refine until the animation looks right. That is better, but still hard. A description in words leaves too much open.

Show it. The third way is the one a coach uses most: show. Tony takes photos of himself in the key positions, or films himself and picks freeze frames at the critical points. He uses them to explain to the AI where it is going wrong. The recovery steps in the balance post were made this way.

Even with photos, there is a lot left to iron out. The angle of the feet is off. The foot turns more than the hip does, and the whole thing looks awkward. This is the most manual part of the project. Then again, it is no different from teaching a person to throw a punch. A beginner’s first jab also looks like a punch and also isn’t one, and the coach fixes it one detail at a time.

Phase 2: the training

Once the reference looks right, the fighter has to learn to do it with real physics: mass, momentum, feet that slip and a body that can fall. That is reinforcement learning. A neural network, the policy, tries the movement millions of times in simulation. A reward function scores each attempt.

How it runs

The work is split over two machines.

• A small server in Amazon Lightsail runs all the time. It manages the training. Tony’s coding agent runs there, and his phone connects to the same session.
• A large training machine on Amazon EC2 does the heavy work. Tony’s tests showed that a graphics card (GPU) adds very little here. A physics simulator like MuJoCo runs on the processor, and many copies of the fighter train side by side, one per core. So the number of vCPUs is what counts. To save money, the machine is a spot instance, rented cheaply from spare capacity.

The training machine writes its progress to logs in the cloud. The agent on the small server watches them, follows the run and talks to Tony. Together they work out whether the progress is right, or whether the policy needs changing. Sometimes they go back to phase 1 and change the reference.

Every now and then a grid of short videos arrives on Tony’s phone: the fighter jabbing at targets at different distances, or taking shoves from every side. So on a weekend walk with his family, he gets the odd video. If something is obviously wrong, he says so by voice: “I can see this problem.” They come up with ideas, and the next experiment starts. It continues from the network trained so far (fine-tuning), so the fighter’s skill builds on what it has already learnt.

Still coaching, in a new language

This phase is still very much like teaching a person. You watch, you see a problem, you correct it. The difference is that the correction is no longer “turn your hip”. It is a change to what the reward asks for. The questions are new ones.

• What should it follow? Should the hip follow the reference exactly? Or should the reward care about the speed of the fist when it hits the target? Tony found that he cared most about the speed and the style up to and at the impact. The way back can be looser.
• What happens after the punch? In one version the fighter shuffled forward after landing the jab. That’s not good, so the reward asked it to come back to its stance, facing forward with its feet back in place. But forcing it to come back in a set sequence took force out of the punch itself. So the way back was loosened: the stance has to come back, but the punch comes first.
• How hard to push? The fighter also learns to keep its balance while being shoved, as a coach might shove a student. But too much shoving takes away from its ability to punch. Tony’s call: keep the shoves light for now, and get the punch right first.
• How far away is the target? If the target is far away, should it step in further? Lean forward? And if the target is too close, the jab looks bad: a bent arm, no step. Should the jab not be thrown that close at all? Tony’s answer was “forget that”: up close, an elbow or a knee is the right technique, just as a coach would tell a fighter.

Some of these pull against each other. More balance means less punch. A strict return means a weaker impact. Choosing between them is a coaching judgement, not a number. That is why this is a human-in-the-loop project. Tony brings the eye and the martial arts experience. The AI brings ideas on how to turn “his hips are behind” into a change to the policy and its reward. Neither would get far alone. The hybrid is what makes it work.

Learning both ways

While the neural network learns the techniques, Tony is learning too: how to combine his own judgement and martial arts experience with an artificial intelligence. The journey has just started, and he is very excited about it.

Best of all, it doesn’t chain him to a desk late at night. He gets on with his normal life. Now and then he checks a technique, corrects it, thinks it over, and learns as he goes, much as he does when coaching people.