Faster punches: from trained amateur towards professional

At the end of the five punches post, the fighter could throw a jab, a cross, two hooks and an uppercut, into the air or a 35 kg heavy bag, without falling over. Tony Yum, the coach, was happy with it, but not finished.

His verdict had two parts. First, the punches were okay, but some still fell a little short, above all the cross. Second, the speed. For an average person who trains, it was not bad. At a professional level he expects more: a faster fist at the impact, and the whole punch, from the guard to the target and back, completed faster.

This post covers the afternoon of 18 October: seven training runs, from 11:45 to 16:27 UTC. Some of the fixes were changes to how the fighter is trained and rewarded. One was a change to its body: Tony allowed the elbow a stronger, faster motor.

The video at the top of this post shows the result. The figures below are stills from the MuJoCo simulation.

Measure first

Before changing anything, the first step was to measure. Was the fighter slower than its reference motion, the animation each punch is learnt from? Then something in the training was holding it back. Or was it exactly as fast as the animation? Then the animation itself was the ceiling.

It was the second. Every punch reached the target on the animation’s own schedule, within a few hundredths of a second: the jab in 0.22 s from the button press, the cross in 0.26 s, the hooks in about half a second, the uppercut in 0.36 s. The reason was built into the setup. The policy doesn’t move the joints directly. It adds a small correction to the animation’s pose, and motors pull each joint towards the sum. The animation ran on its own clock, so the fighter could bend the shape of a punch a little, but not its timing.

And the reward didn’t ask for more. Nothing paid for reaching the target sooner, and the pay for fist speed stopped growing once the fist matched the animation’s speed. The fighter was doing exactly what it was paid for.

Lesson 1: let the fighter set the pace

The first change gave the policy control of the clock. It got one more output, the speed of the punch’s clock: anything from the animation’s pace to twice as fast. The animation still shows the shape of the punch, but the policy now decides how fast to play it. A new reward paid, once per punch, for reaching the target sooner than the animation, and the pay for fist speed was allowed to grow to one and a half times its old top.

The first run did the opposite of what was intended. Within 11 M steps, every punch had got slower: the jab took 0.36 s to reach the target instead of 0.22, the left hook 0.92 s instead of 0.52. The clock was being turned down, not up.

The reason was in the reward’s arithmetic. Most of the style terms, how well the hips turn, where the hands are, how upright the body is, are paid on every step, 50 times a second. A punch that takes twice as long collects them twice. That was worth more than the new bonus for being quick, so the policy slowed down to collect. It is a textbook case of reward hacking: the policy did what the numbers said, not what was meant.

The fix was to make the length of a punch irrelevant to its style pay. The clock could now only speed up (one to two times the animation’s pace), and while a punch runs, every per-step reward is multiplied by the clock’s speed. A punch played at double speed collects its style pay at double the rate, for half as long: the same total either way. Only the quickness and the impact can tell a fast punch from a slow one.

The second run worked straight away for the pivot punches. The hooks reached the target in 0.36–0.39 s instead of 0.48–0.52, and the uppercut in 0.24 s instead of 0.36. They were hitting at 5.5–6.3 m/s. Figure 1 shows the difference on the left hook: by the time the old fighter’s fist reaches the target, the new one has hit and is back in its guard.

Two rows of five frames of the MuJoCo fighter throwing a left hook; in the bottom row the fist reaches the green target two frames earlier and returns to the guard — **Figure 1.** The left hook in the air, before (top: the five punches post's final model) and after (bottom: the final model of this post), at the same times after the button press: 0.12, 0.24, 0.34, 0.44 and 0.52 s. The green dot is the target. After: the fist reaches it at 0.34 s and is back in the guard by 0.44 s. Before: it arrives at 0.52 s.

Lesson 2: stopping short shouldn’t pay

The straight punches, the jab and cross, didn’t follow. Both now ran up to about 10 m/s, and then braked and stopped about 2.6 cm short of the target.

Again, the reward explained it. In the five punches work, a punch that fell short was scored by its top speed on the way, so a fist that stopped 3 cm short still learnt something (Tony’s “don’t make it so binary”). With the speed pay raised to one and a half times, stopping 2.6 cm short at 10 m/s now paid more than arriving at 2.4 m/s. A cut fixed it: a punch that stops short is now scored at 0.4 times its top speed, and never above 6 m/s. Stopping short at 10 m/s scores like arriving at 4 m/s, and it earns no quickness pay.

That run’s jab and cross went through the target again, but they arrived slowly, at about 3 m/s. They stayed that way for 50 M steps: the training numbers in Figure 7 below are flat through the whole run.

Lesson 3: an elbow that can straighten

The obvious guess was the hips: perhaps the body wasn’t driving far enough forward for the arm to reach. Measuring showed otherwise. The hips were about where the animation put them. The arm was the problem: the punching elbow never straightened past 37–44°, where the animation reaches 20–25°.

Was that the policy’s choice? A test answered it. The policy’s elbow output was overridden during the punch, and forced to ask for a fully straight arm. The elbow still stopped at 37–40°. So the policy wasn’t choosing a bent arm. The motor couldn’t straighten it in time.

Each joint is driven by a position servo. It pulls towards a target angle like a spring, with a damper that resists motion so the joint doesn’t wobble, and its total torque is capped. On a slow movement the damper hardly matters. On a fast elbow extension it took the whole torque budget, and the animation holds the arm straight for only 0.01 s before it starts back. The fist ran out of time. This was very likely the old “can’t reach” problem of the cross all along.

Tony’s question was a practical one:

If I allow the elbow to straighten faster would that help? I could continue to use the existing neural network right?

Yes on both counts. The network’s inputs and outputs don’t change, so training could carry on from the same model. The elbow motors got a 1.5 times higher torque cap, 90 to 135 N m, and a quarter of the damping, 36 to 13.5 N m s per radian. This is a judgement call about the body, not the training: the coach decided the fighter’s elbow should be able to snap straight like a trained athlete’s, and the simulated arm is now stronger there than before. Figure 2 shows the effect.

Two rows of four side-view frames of the MuJoCo fighter throwing a cross; in the bottom row the arm straightens fully and the fist reaches the green target — **Figure 2.** The cross from the side, before and after the elbow change, at 0.18, 0.20, 0.22 and 0.24 s after the press. Top: the old elbow motor; the arm stays bent and the fist stops short of the target (green). Bottom: the stronger, less damped elbow motor, one training run later; the arm reaches full length at the target.

The elbow straightened at once: 16–17° at full extension, against 37–44° before. The training hit rate jumped from about 18 % to 27 %, and full-reach jabs hit 41–49 % of the time in training, against 10–14 %. It is the largest single step in Figure 7.

Lesson 4: the shoulder has to come through

The jab was now reaching. The cross was still a centimetre or two short. Measuring along the punch line at full extension showed why: the cross’s hips were in the right place, but its chest and rear shoulder were 6–8.5 cm behind the animation’s. The trunk wasn’t turning the rear shoulder through. A cross is thrown from the back foot through the hips and the trunk to the fist, a kinetic chain, and here the chain broke at the shoulder.

Was the trunk too weak? No: the spine and shoulder motors never reached their torque caps during the cross. They simply weren’t being asked. So a new reward term asked them: it pays, on every step of a straight punch, for the punching shoulder being no further behind the animation’s than about 4 cm.

The first version did nothing. The lag opened only in the last one or two steps before the impact, and carried on after it, where the term no longer counted. So it paid almost nothing either way. The second version made it nearly three times stronger (4 per step), narrower (3 cm) and counted it on through the impact. Within 10 M steps the air cross started hitting, for the first time in this whole series. The cross’s shoulder lag at its worst fell from 7–9 cm to 2–6.5 cm, and back to about 0–1 cm by the end of the hit.

After about 15 M steps the training hit rate levelled off at about 33 %. Ten more million steps changed nothing, so the run was stopped at a plateau, at 16:27 UTC.

Where it stands

The final model was tested the same way as in the five punches post: no exploration noise (deterministic), six presses per punch at different moments of the fighter’s rhythm, in the air and against the heavy targets. Figure 3 shows its punches at the moment they reach the target.

Five frames of the MuJoCo fighter at the moment of impact of a jab, cross, left hook, right hook and uppercut — **Figure 3.** The final model's five punches in the air, each at the moment it reaches the target (green): jab at 0.18 s after the press, cross at 0.22 s, left and right hooks at 0.34 s, uppercut at 0.24 s. All five hit.

Every punch now reaches the target sooner than before, from 0.04 s sooner for the jab to 0.19 s for the left hook (Figure 4). Every one of them is also sooner than the animation it learnt from. The cross, the punch Tony singled out, changed most: in the air it used to stop short in four presses out of six, at about 1 m/s. Now it lands in five out of six, at 4.4 m/s. Figure 5 has it frame by frame.

Figure 4. Before (the five punches post's final model) and after (this post's), in the air, deterministic, six presses per punch; medians. Top: time from the button press to the fist reaching the target; the dashed marks either side are the reference animation's time. Bottom: the fist's speed along the punch line as it reaches the target. Before, the cross stopped short in four presses of six.

Two rows of five frames of the MuJoCo fighter throwing a cross; the bottom row reaches the green target sooner and is pulling back by the last frame — **Figure 5.** The cross in the air, before (top) and after (bottom), at 0.06, 0.12, 0.18, 0.22 and 0.26 s after the press, same rhythm state. After: the fist is at the target (green) at 0.22 s, at 5.5 m/s, and already on its way back at 0.26 s. Before: it is still travelling at 0.26 s and arrives at 1.4 m/s, too slow to count as a hit.

In the full test it scored 38 hits out of 60, against 34 before, and it never fell (Figure 6).

Figure 6. The final model's test, six presses per punch, deterministic. Top: hits in the air (within 10 cm at 3 m/s or more). Bottom: hits on the heavy targets (the 35 kg bag; for the uppercut the 25 kg ball). Total: 38 of 60, no falls (before: 34 of 60). The cross went from 0 to 5 in the air; the jab lost one press on the bag.

Not everything got better:

The hooks and the uppercut on the heavy targets still connect light: 10–15 N s of momentum into the bag, under its bar of about 21 N s, as before. The left hook slows from about 4 m/s in the air to about 1 m/s on the bag.
The left hook arrives sooner but a little slower: 4.2 m/s at the target, against 5.3 before. The uppercut is about the same. The quickness came mostly from cutting out wasted time, not from a faster fist.
The straight punches meet the target at the very end of the arm’s reach. They peak at about 8.5 m/s on the way and slow to 4.5–5 m/s by the target. In some rhythm states they still stop just short. A professional punches through the target, with the hips driving or a step in, so the fist is still accelerating at the impact.

The path, in numbers

Every run except the first continued from the model before it (fine-tuning). A frozen copy of an earlier model acted as a teacher for the stance and the rhythm, as in the five punches post (policy distillation), but it no longer had any say over the punches, whose timing was the thing to change.

Figure 7. Training, five runs end to end (the first, the slow-down run, was dropped and isn't shown). Teal: the share of all punches in training that hit. Purple: the share of full-reach crosses that hit. Means per million steps, with exploration noise on. The short-fist run is flat for 50 M steps; the stronger elbow and then the stronger shoulder term are the two jumps.

Figure 8. The seven runs on one clock (UTC, 18 October). Filled bars are the runs whose model was kept (frozen). The 'shoulder' run's first launch crashed at start-up and was relaunched two minutes later. Short gaps between runs were Amazon's spot request for the previous machine still closing.

run	what changed	outcome
own clock	the policy sets the punch’s speed (half to twice the animation’s), a reward for arriving sooner	every punch slower: per-step style pay rewarded long punches; dropped
clock ×1–2	the clock only speeds up; per-step pay scaled by the clock’s speed	hooks and uppercut 25–33 % sooner and hitting; jab and cross stop 2.6 cm short
short fist	a punch that stops short scores at 0.4 times its top speed	jab and cross reach, but slowly; the elbow can’t straighten; flat for 50 M steps
elbows	elbow torque cap ×1.5, damping ×0.25	the elbow straightens; training hit rate 18 % → 27 %
shoulder	reward for the punching shoulder keeping up	too weak: the lag opened after the term stopped counting
shoulder ×4	the shoulder term stronger, narrower, through the impact	the air cross lands; 38/60 in the test; final model

What it cost

All seven runs used the same machine as the end of the five punches work: a c8a.24xlarge on Amazon EC2, with 96 vCPUs, rented as a spot instance in us-east-1d at $1.80 an hour. The same machine on demand costs about $4.90. It ran about 37 M simulation steps an hour; the long “short fist” run managed about 25 M, because the progress videos were being rendered on the same machine.

Every run saved a checkpoint once a minute, and a new run could start from any earlier run’s, so a stopped run’s training was never lost. The one crash (an attribute set too late when a training episode started mid-punch) cost three minutes and 9 cents.

run	machine-hours	cost
own clock	0.38	$0.69
clock ×1–2	0.58	$1.05
short fist	1.6	$2.88
elbows	0.62	$1.11
shoulder (crashed launch)	0.05	$0.09
shoulder	0.4	$0.72
shoulder ×4	0.85	$1.53
total	about 4.5	about $8

These are estimates from each run’s start and stop times and the zone’s spot price at the time, not from the bill. About 160 M simulation steps in all, about 5 cents per million. The same work on demand would have cost about $22.

What’s next

Tony’s two points were the cross falling short and the punches not being fast enough for a professional. The first is mostly fixed: the cross lands in the air and on the bag. The second is half done. Every punch now completes sooner, faster than the animation it learnt from, but the fist isn’t yet hitting harder.

That is the next step. The straight punches need to go through the target at speed rather than reach it at full stretch, which means more drive from the hips, or a step in. The same step-in is needed for targets further away, where a real fighter lunges. And the hooks and the uppercut have to put real weight into the bag, not just touch it.