Blocking: reading a punch you see a tenth of a second late

By the end of the footwork post, the fighter could stand in its stance, throw five punches and walk to a spot. Every target so far had been a point in the air or a bag. None of them hit back.

This post is about defence. Tony Yum, the coach, explains why it is hard:

Blocking is a complicated thing. Firstly, there’s a reaction time. What you see and process is actually behind reality. And you don’t know what your opponent is actually thinking, whether it’s throwing a jab or a hook. All you have is the visual clues, like the turning of the hip, the movement of the shoulder or elbow. That can also become a feint. So it’s a very interesting chess game.

And what this first step leaves out:

In the real combat fight, you’re thinking about distancing. You wouldn’t allow your opponent to stand in striking distance and just stand there. Typically, you would go in and out of the pocket. You might bob your head to avoid jabs. You might be prepared to move, or you might trade. You might think: that quick jab was a bait, and you’re actually thinking to move out of the way and do a big cross. So all that is in consideration. But one baby step at a time.

So: one punch at a time, from an opponent standing in range, and the fighter only allowed to block. No slipping, no stepping back, no countering. It sees the opponent a tenth of a second late, and it has to tell from his first movements which punch is coming.

It covers 20 training runs over about 18 hours, from 23:43 UTC on 19 October to 18:04 UTC on 20 October. The model that was kept finished training at 17:43 UTC on 20 October.

Four blocks

Tony sent photos of the blocks he wanted, and four poses were built from them (Figure 1):

• Left high block, against the opponent’s right hook: the left elbow up in front of the face at forehead height, the forearm back along the temple, chin tucked.
• Right high block, the same on the right, against the left hook.
• Front block, against the jab and cross: both forearms upright and pressed together in front of the face.
• Low front block, against the uppercut: the front block with the elbows down at the chest, so the forearms close the gap under the chin.

The fighter fights bare-fisted. Gloves were tried and dropped the same day: they covered the eyes in the guard, and the fighter will grapple later.

Step 1: blocks on command

Option one was to put an opponent in front of the fighter, let it get hit, and wait for it to discover blocking. With only “getting hit is bad” to go on, a reinforcement learning agent is more likely to invent an odd flinch than a Muay Thai block. So training came in two stages. First, blocks on command: a button says which block, and the policy, the neural network that drives the body, learns to raise it fast and hold it without falling over. Then the button goes, an opponent comes in, and the same policy decides which block and when.

While the button is held, the blocking arm’s motor targets switch to the block’s pose. The rewards pay for speed (a bonus for reaching 90 % of the pose’s cover, in full within 0.15 s), for holding still, for keeping the elbow bent on the way up (Tony’s rule against a wide, telegraphed swing), and for a smooth return to the guard. The legs and trunk keep their stance rewards.

The fighter must not forget what it already knows. Tony’s rule: “make sure that none of the previous techniques are ever forgotten. So it still knows how to balance and it still knows how to do all the punches and walk.” Half the training episodes are blocks; the other half are the walking environment’s exactly, with walking and punches. A frozen copy of the walking model acts as a teacher (policy distillation) whenever no block is held. And no model is kept until it passes a skills gate: at least 58 of 120 test punches, at least 39 of 40 walking targets with no falls, in no more than about 8.3 s each.

Three of the four blocks came at once. The left high block never covered, in 15 M steps. Three runs found out why:

• The poses were worked out on the reference stance, but the trained fighter stands with its chest turned about 8° further and leaning 7° further back. With every joint exactly on the pose’s angle on that trunk, the block covered only 82 % of its design. So at each press, the arm is now fitted to where the head actually is, with inverse kinematics.
• That alone didn’t move it. The policy adds its own small offsets to every motor target, and its guard habit held the left shoulder about 4° low and 8° in: the arm stayed 3.9 cm off the pose. A reward for pulling the elbow and wrist to the pose didn’t break the habit in 10 M steps either.
• What worked was turning the habit down: while a block is up, the policy’s offsets on the blocking arm count only 30 %. In 4.6 M steps all four blocks covered, and every old skill passed the gate.

Then the high blocks were too slow: 0.26 s for the left, 0.21 s for the right. The fighter’s left shoulder swings from 53° to 129° in that block. Each joint is driven by a position servo, and its damping, the part that resists speed, cancelled the pull 20 ms into the swing. No amount of training could fix that. Softening the damping to a quarter while a high block rises did: 0.12 s for the left, 0.08 s for the right, 0.10 s for the front and 0.07 s for the low front block, every press covered, with punches and walking still passing.

Is there time to block?

Before the opponent came in, one question needed numbers: can a block arrive in time at all?

The opponent is the fighter itself. Its own fast punches (jab, cross, both hooks and the uppercut) were recorded once and are replayed, without physics, from in front of it. From those recordings: when is the first visible sign of each punch, and when does the fist land?

The fighter doesn’t see him as an image. It sees 16 keypoints (head, neck, shoulders, elbows, wrists, fists, hips, knees, ankles), each as a 3D vector from its own head, with velocities and a short history of 40 and 80 ms back (frame stacking). It sees them late. Tony chose a vision delay of 100 ms, about a fast professional’s. And it can be blind. Rays go from both eyes to each keypoint, and a point is hidden if the fighter’s own arms are in the way (occlusion).

punch	first moves	lands	warning	cover at landing*
jab	elbow and fist, 0.083 s	0.187 s	0.10 s	0.50 (guard: 0.48)
cross	elbow, 0.079 s	0.220 s	0.14 s	0.68
left hook	fist, 0.110 s	0.342 s	0.23 s	0.82
right hook	elbow, 0.106 s	0.343 s	0.24 s	0.69
uppercut	fist and elbow, 0.081 s	0.235 s	0.15 s	0.79

* The share of the block’s cover in place when the fist lands, reacting at the first movement plus 100 ms. Times are from the opponent’s decision to punch, which the fighter can’t see. With 100 ms of delay, the jab’s 0.10 s of warning is used up before the fighter even knows it has started. Reacting to what it sees, the fighter can’t add anything to its guard against a full-speed jab. Against the hooks there is time for most of a block.

One surprise: in every punch the hips move 20–60 ms after the arm. Tony reads the hip turn; in these punches, the first sign is the elbow or the fist.

From that table Tony set a pass mark per punch, the share of the punch that must be stopped at full speed: jab 0.50 (keep the guard: beating the jab needs anticipation or distance), cross 0.65, hooks 0.70, uppercut 0.75. Damage counts partially: the share of the fist’s path that reaches the head or neck before it meets an arm.

Step 2: the opponent

Now the button went, and the policy got four new outputs of its own: one per block. Punches come one after another, with half a second to two seconds of the opponent moving in his stance in between. Tony calls these slow combos: “it’s one shot after another, so it’s not a real true combo.” The opponent’s speed rises with a curriculum: punches start at 50–60 % speed, and the top speed goes up as the fighter passes.

Getting the fighter to block for itself at all took three runs. Its exploration (random noise on its outputs) flipped a button up and down within a step or two, so a random press dropped the block before the arm moved. Presses now hold their block for at least 0.25 s. Then a smoothness term in training (CAPS) was flattening the buttons, since a press is exactly a sudden jump in an output. The buttons were freed from it. In the third run, the fighter blocked.

It raised the low front block against everything, and it raised it before the opponent punched.

The giveaways

The fighter wasn’t reading punches. It had found a cheaper cue (shortcut learning), and it would do so four times:

1. He came alive before each punch. Between punches the opponent stood frozen, and each recorded punch began 0.18 s before the press with his stance’s movement. So he twitched just before every punch. The opponent was re-recorded with 2 s of stance motion before each punch, so he never stands still.
2. Each punch had its own distance. Tony spotted this one: the hooks stood closer. Head to head, the jab and cross came from 0.78 m, the right hook 0.62 m, the left hook 0.52 m, the uppercut 0.46 m. Telling the distance is easier than telling the punch.
3. The overlap wasn’t enough. Varying each punch’s distance only partly hid it. One distance range (0.57–0.73 m) for every punch fixed it.
4. The uppercut stood him lower. Each recording had its target at its own height on his body, so lining the uppercut up with the fighter’s head stood him 4–5 cm lower. His head is now at one height for every punch.

Two other things went wrong in between, neither of them the fighter’s fault. The network scales its inputs using running averages of what it has seen (observation normalisation). Those averages share one sample count for every input, and by then it was 1.2 billion. The opponent’s inputs, added later, kept their starting values (mean 0, spread 1) when their real spread was 0.29, some 0.016. For five runs the fighter had been all but blind to the opponent. Its buttons pulsed every 0.75 s on its own rhythm, whatever he did. The averages were re-estimated, and still nothing changed: a reward for pressing the suitable button (a dense shaping reward) was too weak a signal.

What worked was telling it directly. The opponent is scripted, so the training knows which punch is coming. An auxiliary loss pulls the right button’s output up while a punch can be seen, and the others down. The damage still decides whether blocking pays. Within 5 M steps, the fighter raised different blocks for different punches, and right away it found giveaways 3 and 4.

With all four gone, in the ninth run against the opponent, the fighter waited. No block before he punched. Then the front block against the jab and cross, the right high block against the left hook, the left high block against the right hook and the low front block against the uppercut, with the high blocks rising 0.24–0.28 s into his punch. The check that proves it reads the punch: move his punch later, and the fighter’s block moves with it.

Full speed

Reading the punch was one thing. Doing it at full speed was another. The curriculum only raised the speed once 80 % of punches at the top speed passed, and it never got there: for 23 M steps nothing faster than 60 % was trained. Lowering the bar to 60 % moved it, and full speed reached 21 of 30 test throws. Then it stuck at 13–19 for the next three runs, including one where a quarter of all punches came at full speed.

What was left was the jab, which the time budget says can’t be read in time, and the right hook. The fighter’s left high block came about 0.1 s after the oracle’s, a script that presses the right block at the opponent’s first movement plus the 100 ms delay. Tony’s first idea was a faster left block. Measured first, it wasn’t slow: softer damping and more torque barely changed the damage. The cause was the timing of its guess. Before the punch can be told apart (about 0.22 s in), the right high block’s button hovered half up, a guess at a left hook. In 3 of 12 right hooks it pressed that guess just before the left button rose, and the 0.25 s hold kept the wrong block up through the hook. So a block may now be swapped for another during its hold; only dropping it waits. Without any training, that alone took the right hooks passed from 3 of 12 to 5.

The last run asked which block actually works at each distance. Holding every block up early against every punch at five distances showed that the rule “one block per punch” was wrong in two places. Against the uppercut, the front block lets only 7–12 % through at 0.59–0.65 m, where the low front block lets 43–68 % through. Against a jab from 0.68 m or more, the left high block stops it better than the front block. The rewards now teach the measured best block at each distance. At 14.5 M steps full speed reached 24 of 30, the best yet, with the jab 6 of 6 and the uppercut 6 of 6.

Where it stands

The kept model, on the skills gate’s tests (8 throws per punch at each speed):

• Against the opponent: 117 of 160 passed, 31 of 40 at full speed, with no falls. The gated model before it scored 78 of 160.
• Blocks on command still work: all 32 presses covered (left high 0.12 s, right high 0.08 s, front 0.11 s, low front 0.07 s), also with shoves.
• Old skills kept: 61 of 120 punches, 40 of 40 walking targets with no falls. But walking got slower: 8.45 s per target, just over the 8.3 s mark. The model before it, which passes every mark, is kept as a fallback.

What it can’t do yet:

• The far right hook. Held up from before the punch, the left high block still lets about 0.44 of a right hook from 0.73 m through. Those hooks start almost straight ahead and slip between the elbow and the fist onto the temple. A search for a better pose within Tony’s shape found almost nothing (0.77 to 0.79 of the paths stopped): the elbow is at the end of its reach. It needs a different shape, which is Tony’s call.
• The mid-range uppercut. The front block at 0.61–0.65 m is only half learnt.
• The jab, mostly. At training distances, 7 of 12 jabs pass.

What Coach Tony thinks

Tony’s most interesting point is about the front block:

One very interesting point is that if the agent always holds a front block, like the forearm blocking the face, he actually gets penalised, not by the reinforcement learning punishment, but rather by the fact that he won’t be able to see if the hands are in front of the face.

That is how the fighter’s vision was built, and it was a deliberate choice. With both forearms in front of its face, the fighter sees none of the 13 points tested on an opponent ahead (Figure 1’s fourth pose): no fists, no elbows, no shoulders. The stance guard sees 10, the low front block all 13 between its fists. There is no reward term for being blind. The cost is meant to be what it costs a real fighter: the next punch comes, and you didn’t see which one. A “can’t see” penalty would first have taught the fighter not to block at all. (There is one small cost for holding any block while no punch is coming, so that it doesn’t keep one up all the time.)

He also sees how far this is from a fight. The punches come one at a time, from an opponent who stands in range and never feints. In his own thinking out loud during the runs: stay out of jab range so the opponent has to step in, and the step is the tell; move the head in the rhythm so a fast jab can miss; learn an opponent’s habits, his rhythm, his favourite combinations. None of that is in this model. But it now sees an opponent, late and partly blind, and picks a block from what he does, not from where he stands.

The path, in numbers

run	what changed	outcome
block1	blocks on command; half the episodes walking and punching; the walker as teacher	three blocks work; the left high block never covers
block2	the arm fitted to the head at each press	left still 5–13 % covered
block3	a reward pulling the elbow and wrist to the pose	the guard habit stays
block4	the policy’s arm offsets at 30 % during a block	all four covered, every skill kept
block5	soft damping while a high block rises	left 0.12 s (from 0.26), right 0.08 s
opp1	the opponent, keypoints, 100 ms delay, real blindness; the fighter’s own buttons	no block ever kept up
opp2	presses hold for 0.25 s	the smoothness term flattens the buttons
opp3	buttons free of the smoothness term	blocks for itself, but from giveaways 1 and 2
opp4	he never stands still; overlapping distances	giveaways gone; guard only
opp5	a reward for the suitable button	no change: found the input-scaling bug
opp6	the opponent inputs rescaled	no change
opp7	an auxiliary loss on the buttons	a block per punch, read from his distance
opp8	one distance range	the right block per punch, the uppercut read from his height
opp9	one head height	reads the punch; never trained past 60 % speed
opp10	the curriculum’s bar 80 % → 60 %	full speed 21/30
opp11	a bigger training batch (a speed test)	the same learning; then a trainer bug, discarded
opp12	opp11 again from its last clean model	full speed flat, 13–15/30
opp13	a quarter of the punches at full speed	full speed flat, 15–17/30
opp14	a block may be swapped during its hold	19/30; kept
opp15	the rewards teach the block that works at each distance	24/30, 93/120: kept

Making training faster and cheaper

The blocking runs were also a day of tuning the training machines. A training run has two sides working at the same time (decoupled PPO). Worker processes simulate the fighter in hundreds of copies of the environment and collect experience. Learner processes use the last batch of experience to update the network. Whichever side is slower sets the pace, and the other side’s processors wait.

The muscular body was wasting time on its own fists. Profiling a punching run counted 38 contacts per physics step, and 31.6 of them were the body touching itself, almost all fingers on fingers inside the fists. The fingers aren’t driven by the policy: motors hold them in a fist, in a pose that overlaps the finger shapes slightly, so the physics resolved about 25 contacts every step for nothing. Turning off contacts between the parts of each hand took that to 12.5 contacts per step and made a physics step 1.8 times faster (742 to 412 µs). The punches still land as before (fist speed 4.06 against 4.12 m/s). This “muscular-fast” body is the default for every run from the fourth opponent run on. It can’t be used for grappling later, when the hand has to open and grip.

A GPU doesn’t help yet. A rented GPU (a GTX 1080 for about $0.06 an hour, on vast.ai) rendered the fighter at 1920×1080 in 0.4 s, so it is useful for videos. But training waits on the physics, which runs on the CPU, not on the small network. Moving the physics to the GPU would mean rewriting every environment, and the muscular body’s 63 egg-shaped muscles are a bad fit for GPU simulators. vast.ai’s large CPU machines were no cheaper per step than the c8a spot machines on EC2.

Bigger machines, then faster code. In the morning the spot quota was raised to 192 vCPUs, and the ninth opponent run moved to a c8a.48xlarge: 192 cores for $3.83 an hour. It ran 1.4 times the steps of the 96-vCPU machine for 2.4 times the price. About 70 % of the time its workers were idle, waiting for the learners. Then the code changed, one thing at a time:

• Shared memory. The network’s weights and the collected experience had been sent between processes through pipes, copied and repacked by one core. Now every process reads them from the same memory. 44 → 50 M steps an hour.
• Four learners instead of two, with 144 workers. 60 M an hour. A test of seven layouts found all of them limited by the learners: more learners do the maths faster, but spend as much extra time agreeing on each update.
• A training batch twice the size. Fewer of those agreements per update. It changes how the network learns, so Tony agreed to test it on its own first: 8 % faster, and the fighter learnt about as well per step.
• Each learner prepares its own part of the batch, instead of one core preparing it all and sending it out. 70 M an hour, 59 % faster than the morning on the same machine. The first version of this had a bug: three learners prepared their part from a buffer the workers were already refilling, so three quarters of each batch was wrong. 9 M steps of training were thrown away, and every change to the trainer is now checked on its first update.

Which machine is cheapest? Faster isn’t cheaper. With the new code, a 96-vCPU c8a.24xlarge ran 50.5 M steps an hour with its workers 85 % busy, a balanced machine. At $1.87 an hour, that is $0.037 per million steps. The 192-vCPU c8a was 1.35 times faster but $0.055 per million, because half its cores still waited for the learners. An m8a with 192 cores did 64 M an hour, nearly as fast as the c8a. The c8i machines split each core into two hardware threads, so a “vCPU” there is half a core: a c8i simulated about 1.45 times slower than a c8a with the same vCPU count. Tony chose cost over speed, and the last two runs went back to 96 vCPUs. Capacity mattered too: the 192-vCPU machines were taken back three times in two hours that afternoon.

What it cost

Most training runs used a c8a.24xlarge on Amazon EC2, a machine with 96 vCPUs, rented as a spot instance in us-east-1 at about $1.4–1.9 an hour. When it had no spare capacity, the launch script fell back to the same size from c8i or c7a. Five runs in the middle used the 192-vCPU machine (above). Amazon took the machine back 13 times. Each run saves a checkpoint often, so it carried on where it stopped on a new machine. In the late afternoon spot capacity ran out altogether, and the last two runs finished on on-demand machines at about $4.15 an hour.

item	machine-hours	cost
20 training runs	about 13	about $30
9 skills gates, punch recordings, the block-by-distance table and a baseline test, on smaller machines	about 3.5	about $2
one speed test of the training layout on 192 vCPUs	about 0.2	about $0.80
total	about 17	about $33

These are estimates from each run’s start and stop times and the price at the time, not from the bill.

What’s next

The fighter can see an opponent and block single punches it can read. The next things it needs are the ones Tony listed. A second, faster punch after the first, so that covering the eyes costs something. Feints. A memory longer than a tenth of a second, to learn an opponent’s rhythm and habits. And above all, distance: stepping out of range, so the opponent has to step in, and the step is the tell.