Five punches: the cross, the hooks and the uppercut

The jab took two days. Once it was fast, accurate and stepping in, Tony Yum wanted the rest of the basic Muay Thai punches: the cross, the left and right hooks, and the lead uppercut. All five in one fighter, on command, from the same guard.

This post covers the next 18 hours: ten training runs, from the afternoon of 17 October to 10:00 UTC on 18 October, when the last one finished. The fighter went from punching the air to punching a 35 kg heavy bag, and from one punch to five.

Tony’s verdict at the end, in short: it isn’t perfect, and he is happy with it. A couple of punches still fall a little short. The rest look close to how an ordinary person who trains would throw them. The next step is to make them look like an athlete’s, and that is already under way.

Four new punches

Each new punch started, like the jab, as a reference motion: an animation built pose by pose from Tony’s description and photos, with no physics in it. His specs read like a coach’s notes:

• The cross (the rear straight). The right arm shoots out, the fist turning over so the palm faces the floor. The chest turns left, the lead foot steps in, and the rear foot pivots on its ball with the heel up. The left glove stays on the cheek.
• The hooks. “The power comes from the legs, hips and core, not the arm.” The arm takes the hook shape, elbow bent about 90° at shoulder height, and then stays locked to the chest. The body’s turn carries the fist through the target. A small wind-up the other way comes first.
• The uppercut. The load is a turn, not a squat. The hips and chest turn left over the lead leg, and the fist drops to chest height. Then the turn reverses from the feet up, and the fist drives straight up into the chin.

The fighter then has to do all of this in a physics simulation, with weight, momentum and feet that can slip. One policy runs all five punches. It sees which button was pressed and where the target is, and it gets a reward for following that punch’s reference and hitting the target fast. Figure 1 shows where it ended up.

Air, then a pad, then a heavy bag

Tony’s brief had one more part. Real fighters don’t only shadow box, so the fighter shouldn’t either:

Train from hitting air, then a light target, then a heavy bag, so it learns the impact and the recoil.

So the targets became part of a curriculum (Figure 2). At level 0 the fighter punches a point in the air, and only speed and accuracy count. From level 0 to 1 a pad appears, the size of a focus mitt, on springs, and grows from 0.75 to 3 kg. From level 1 to 2 it becomes a heavy bag on a rope, from 12.5 up to 35 kg. The uppercut gets its own targets, because hitting a hanging bag from below makes no sense: a pad facing down, then a heavy ball hanging at chin height. That was Tony’s idea too.

Lesson 1: don’t forget the jab

The first run started from the best jab and added the four new punches. Within 2.7 M steps, about seven minutes, the jab was falling apart. Its hits dropped from 28 % to 2–7 %, and its speed from 2.7 to 1.3 m/s. The fighter was still on its feet, but it had bought its balance with a slower jab. This is catastrophic forgetting, and an earlier attempt with several buttons had lost the jab in the same way and never got it back.

The fix was to give the old policy a job: teacher. A frozen copy of the best jab policy watches the new one during training. Whenever no new punch is running, in the stance, the rhythm and the jab, the new policy pays a penalty for doing anything different from the teacher. While a cross, hook or uppercut runs, it is free to learn. This is a form of policy distillation, and it worked at once. The jab stayed at 5–6 m/s for the next 36 M steps while everything else was learnt around it.

The hooks and the uppercut were quick. They hit within about 10 M steps. They are pivot punches: the body turns, the arm is carried, and the reference had already worked out the shape. The curriculum then climbed from the air to the 35 kg bag in about 18 M steps. After about an hour and a half of training, four punches were landing on the heavy bag with no falls. The fists arrived a little slower on the bag than in the air, 3–4 m/s against 5–6, as if bracing for the contact.

The fifth punch was the cross, and it was 9–12 cm short.

Lesson 2: the cross that rode the rhythm

The cross was the jab’s old problem again. A straight punch from the rear hand has further to go. To reach, the hips have to drive forward about 19 cm in 0.16 s, at up to 1.75 m/s. The policy’s hips stayed 9–11 cm behind the reference’s, so the arm ran out of reach and the fist stopped short.

The next run gave the cross 40 % of the button presses, a reward for the hips keeping up with the reference, and a target that started 10 cm nearer and stepped back 2 cm whenever the cross hit often enough. The teacher held the other four punches in place. Over 103 M steps the cross went from 17 cm short to reaching its full-reach target.

But the test videos had been flattering it. Each grid of test videos showed three presses per punch, and those three happened to come at good moments. A wider test, six presses per punch at different points in the fighter’s rhythm, found the cross 0 for 12. Tony’s fighter stands in a Muay Thai rhythm, rocking gently between the feet. The cross only worked when the rock was big. From a big rock the fist arrived at 2.6–4.1 m/s. From a near-still stance it stopped 3–7 cm short, at 0 m/s, in 9 presses out of 10. The policy was borrowing the rock’s momentum for its hip drive, not making its own.

Two more runs went at this:

• Press more crosses from a still stance. No better: still 3–5 cm short from still, and worse from a rock. Then Amazon took the machine back (more on that below).
• Pay for the hips’ forward speed and turn directly. The turn got fixed: 5–8° short became spot on. The hips moved about 1 cm nearer. After 61 M steps the cross was no better, and slower from a rock.

Three runs had now failed to make the body drive its hips 19 cm in 0.16 s from a dead stop. That was the point to stop changing the reward, and to ask the coach.

Lesson 3: through the target, not to it

Tony was watching the grids on his phone, and he saw something else first:

That green dot, the target should be inside the heavy bag and inside the small target. Otherwise, you’ll end up hitting the surface. You need to go through. So put the target in the middle.

He was right, and it was a basic mistake. The target’s surface had been placed at the aim point, so every punch was trained to arrive at the surface and stop. The aim point moved inside the target, first at the bag’s centre, then (after “the middle of the bag’s too central”) 7.5 cm in. A hit was now scored where the knuckle meets the surface, and the punch was asked to carry on through it.

That raised a second question: what should a heavy bag reward? The first answer was distance, how far the bag swings. It plateaued at about 18 cm. The fist stayed on the bag for only 0.04–0.08 s, because the reference pulls the arm straight back, so every punch was a single knock. Then Tony ruled distance out entirely:

Never push the bag. It’s all about impact. Impact and snap back. Don’t push into the bag. Impact and pull back.

So the heavy targets now pay for two things. The impact is how fast the bag moves in the first 0.06 s after the fist reaches it, which measures the punch’s momentum. The snap back pays for the fist leaving the bag within 0.2 s. The jab learnt both straight away: 31–34 N s into the bag, and off it in 0.06–0.08 s. The cross and the hooks stayed light.

Lesson 4: make it easier to get right

After the cross had stalled through several runs, Tony looked at it as a coach would look at a student who keeps missing:

The target should be a little bit closer. And also whether it’s considered a hit or not, let’s not make it so binary. You get more reward the closer you are to the target.

Three changes followed in the same run:

• The cross’s target moved 6 cm nearer. A cross that stops short of a target at full stretch is mostly a reach problem, and a real fighter would simply stand a little closer.
• A punch that falls short now scores by its top speed on the way, not zero. Before, the reward jumped from almost nothing to a lot at the surface, and a fist stopping 3 cm short learnt nothing about stopping 2 cm short. Now each centimetre and each m/s counts. This is reward shaping.
• The bars came down. The target for a hit got bigger, 10 cm instead of 5. A test showed the misses were nearly all within 5 cm already, and they missed on speed or impact. So the speed for a hit in the air came down from 3.5 to 3 m/s, and on the heavy bag to about 21 N s from 31. Tony’s answer to that question was two words: “Lower the bars.”

The cross started landing. Full-reach crosses in training went from 0 % to 34–42 % hits, and it hit the bag in three test presses out of three.

Lesson 5: what only a coach sees

With the punches landing, Tony’s notes turned to things no hit rate shows.

A step back before the punch. “Why does it always take a step back?” On the jab, the fighter shifted its weight back, lifted the lead foot and moved it about 5 cm back, then stepped in. On the cross, the rear foot slid back as the heel came up. The reference does neither: the rear foot stays planted and the lead foot steps straight in. A new reward term paid for neither foot going behind its starting spot until the punch returns. The lead foot’s step back went from up to 4 cm to almost none.

The head. “The head’s really tilted to the side.” It was, by about 30–35° in the guard. None of the references tilts it. Tracing back through every saved model found where it came from: a jab run on 17 October that trained against much harder shoves. Before that run the tilt was 5–9°. After it, 20–30°, and the teacher had faithfully passed it on to every run since.

A reward for keeping the head near the reference did nothing in 6 M steps: the head was a small part of a large reward. So the next test was blunt. The neck and head were locked to the reference, and the punches were re-tested. They scored 15 hits out of 40 locked, against 16 out of 40 free, with the same speeds and no falls. The tilt had been doing nothing useful. The neck stays locked, and the head is level (Figure 3).

Where it stands

The final model is from the last run, frozen at 46 M steps. It was tested with no exploration noise (deterministic), six presses per punch at different moments of the rhythm, in the air and against the heavy targets. It never fell. Figure 4 shows the bag row, and Figure 5 the scores.

In words:

• The jab is the best of them: 11 of 12, and 29–33 N s into the 35 kg bag.
• The cross lands on the bag, 5 of 6. In the air it is fast, about 7 m/s, but passes 2–5 cm off or stops short. This is the punch that is still “a little short”.
• The hooks and the uppercut hit every time in the air, 18 of 18, at 5–6 m/s. On the heavy targets they connect, but light: 11–15 N s on the bag, under its bar.

From the coach’s side, that is a fighter with a real combination. The stance holds, the head is up, the feet don’t shuffle back before a punch, and each punch has the right shape: the hips turn, the rear heel comes up on the cross, the hooks are carried by the body. It looks like someone who has trained for a while. Not yet like a fighter.

The path, in numbers

run	what changed	outcome
five buttons	four new punches on top of the best jab	the jab decayed within 2.7 M steps; stopped
teacher	the jab policy anchors everything except the new punches	four punches on the 35 kg bag in 36 M steps; the cross 9–12 cm short
cross	cross 40 % of presses, hip-drive reward, nearer target that steps back	the cross reaches, but only from a big rhythm rock; 0/12 in the wider test
still	most crosses from a near-still stance	no better from still; spot machine reclaimed at 32 M
push	pay for the hips’ forward speed and turn	turn fixed, hips 1 cm nearer; not kept
aim inside	the aim point inside the pad and the bag	punches go through the surface; stopped for the next idea
push the bag	pay for how far the bag swings; uppercut pad and ball	plateaued at ~18 cm swing; Tony: “never push the bag”
snap back	pay for impact in 0.06 s and leaving the bag	the jab learns it; cross and hooks light
nearer, graded	cross 6 cm nearer, graded short punches, bigger target, lower bars	the cross lands: 34–42 % at full reach in training
feet and head	no step back before the punch; head level	the final model: 34/60, no falls

What it cost

The fighter trains in the cloud on Amazon EC2 spot instances, spare capacity sold at a discount that Amazon can take back at two minutes’ notice. Most of this work ran on a c8a.24xlarge: 96 vCPUs, at $1.47–1.80 an hour depending on the zone, against about $4.90 on demand.

The machine changed at the start of this work. On the usual 64-vCPU machine, the simulation wasn’t the bottleneck any more; the learning update was, and the simulator cores sat idle about 40 % of the time. Moving to 96 vCPUs, with bigger batches and fewer passes over each batch, took training from about 22 M to about 40 M steps an hour: 1.8 times the speed for about 1.5 times the price.

Spot had its costs. Amazon reclaimed a machine three times in these 18 hours. Twice the run resumed within minutes from its last checkpoint. Once, late on 17 October, the watcher kept showing the last status line and nobody noticed for an hour, which is the gap in Figure 6.

run	machine-hours	cost
five buttons	0.15	$0.15
teacher	1.5	$1.87
cross	2.8	$4.10
still	0.9	$1.57
push	1.8	$2.91
aim inside	1.0	$1.50
push the bag	1.7	$2.85
snap back	1.3	$2.10
nearer, graded	1.4	$2.15
feet and head	1.7	$2.75
total	about 14	about $22

These are estimates from each run’s start and stop times and the spot price of its zone at the time, not from the bill. About 470 M simulation steps in all, a little under 5 cents per million.

What’s next: from trained to professional

Tony is pleased with this. The fighter throws five recognisable punches from a Muay Thai guard, on command, into the air or a heavy bag, without falling over. But he wants more than a trained amateur.

Measuring the final model against the animations showed why it looks the way it does. Every punch reaches the target on the reference’s own schedule: the jab in 0.22 s, the cross in 0.26 s, the hooks in about half a second. Nothing in the reward paid for getting there sooner. The hooks’ fists peak at 5.6–5.8 m/s, against 8.3–8.8 m/s in the animation, and the cross arrives slowly or short.

A professional’s punch differs in exactly these places. It gets to the target sooner, hits harder at impact, and comes back to the guard faster. So the next runs let the fighter set its own pace, and pay for quickness and for speed at the target. That work started an hour and three quarters after this run finished. It will be the next post, with a video of all five punches.