Correcting the knee

From describing to capturing

When I started, I taught the fighter the way I’d teach a beginner over the phone. I described the technique in words, sent photos of myself from a few angles, and corrected what I saw: tuck the elbow in more, stretch further, turn the hip. The stance and balance and the jab were built that way, as poses designed on the simulated body itself, and it worked well enough for them.

It doesn’t work for bigger movements. A knee or a kick moves the whole body through large angles, quickly, and the order of events matters: which foot leaves the floor first, when the hips turn, where the arms go to balance. Describing all of that in words gets you something that moves, but not something that looks like Muay Thai. So I moved to motion capture.

The pipeline

I don’t use a commercial motion-capture system. Everything is open source, mainly Pose2Sim, which means the heavy parts can run on whatever GPU machine is cheapest. Figure 1 shows the stages. The purple ones end with me looking at the result before the next stage starts.

1. 1
   Filmme, two phones

   A calibration board on the floor, photographed once from each phone's spot. Then the board is taken away and both phones film the technique at 60 fps from 60–90° apart, without moving.

2. 2
   2D keypointsGPU

   A pose estimator (RTMPose, with heels and toes) finds the joints in every frame of both videos: about two minutes for a 30-second take.

3. 3
   Sync and triangulateGPU / CPU

   The two videos are lined up by their sound, each joint is triangulated in 3D from the two views, and the paths are smoothed.

4. 4
   Fit my skeletonCPU

   A skeleton scaled to my measurements is fitted to the 3D points, giving my joint angles frame by frame.

5. 5
   RetargetCPU

   My joint directions and turns are moved onto the fighter's body, and a small inverse-kinematics step keeps planted feet planted.

   I check the sim beside my video, through each phone, frame by frame. I mark what's wrong and choose between copying me exactly and what works on his body.

6. 6
   Table of variationsCPU

   The technique is re-solved for targets a little higher, lower, left and right, so he can strike from wherever he is.

   I check the corners and the in-betweens, where knees twist and feet lift.

7. 7
   Teacherrented GPU

   A copy of the latest model learns only this technique (plus its stance and balance) by reinforcement learning against the table.

   I check short progress clips sent to me through the day.

8. 8
   GateCPU boxes

   Deterministic tests: hits in the air and at the bag, falls, feet, facing, lean.

   I check the final video, slowed down.

9. 9
   Distil into the studentrented GPU

   One student learns every technique from its teacher while keeping all the old skills.

   I check the student's gate and videos. A fault here is fixed in the teacher, then distilled again.

Filming and reconstruction

Two phones see the same moment from two directions, so the depth of every joint is measured by triangulation instead of guessed. My first attempt used one phone, guessed depth from bone lengths, and at one point put the legs the wrong way round in the stance while every picture-based check passed. For triangulation to work, each phone’s lens and position must be known exactly. That is camera calibration, done with a printed board of known size on the floor.

A pose estimator finds my joints in each frame; Pose2Sim lines up the two videos, triangulates the joints, smooths them with a low-pass filter (12 Hz, not the 6 Hz default for walking, which would round off a kick), and fits a skeleton scaled to me using inverse kinematics.

Where it goes wrong

The reconstruction is good, but not good enough to use without checking. The errors I’ve seen fall into three kinds.

• Wrong estimates. A foot judged to be below the floor gets pushed back up, and the body pops up and down for a few frames. A limb pointing straight at a camera collapses onto its joint; a sole facing a camera drags the knee onto the foot; left and right swap when the body turns side-on. In the knee take used below, for 0.06 s after the knee the reconstruction throws my pelvis back 26 cm and my rear foot jumps, so the end of the clip had to be replaced.
• Different bodies. The fighter’s legs are about 25 % longer than mine relative to his trunk. Copying my joint positions folds his elbows into his head and bends his knees oddly, so retargeting copies directions and turns instead, and fixes only what has to touch something. Even then his body has limits mine doesn’t: I plant my rear foot about 31 cm behind my hips after the knee, past his hip’s range, so his lands 8 cm nearer.
• Estimates of what can’t be seen. Forearm roll, wrists and collarbones aren’t visible to the cameras and are held at the stance.

So step 5 is a review. I watch my video beside the simulated fighter rendered through each phone’s view, slowed down, and point out the differences. Each one is a choice: copy me more exactly, or accept a difference that works better on his body. The corrections go into a layer of eased offsets on named joints.

A table of variations

A fighter can’t only strike from one recorded position, because in a fight he and his opponent are both moving. So each technique becomes a table: the same strike re-solved for a grid of targets a little higher, lower, left and right (for the kicks ±10 cm each way). In training the target can be anywhere inside the grid, and the reference motion is interpolated between its nearest entries.

A perfect reference can still produce bad entries in the table. In the low and mid kicks, the foot’s inverse kinematics flicked the knee inwards by about 40° over 4 frames during a switch hop. Nothing measured caught it; I saw it on the preview. So step 6 is another review.

Teacher, gate, student

A teacher is a copy of the latest model that practises only this technique, with every episode a press of it and shoves in between to test balance. It learns by reinforcement learning (PPO), rewarded for following the reference, for landing on the target, and for not falling. It doesn’t copy the reference exactly, and it can’t. The reference has no physics in it: my capture doesn’t know the bag pushes back, or that the fighter’s mass is spread differently from mine. A policy that copied it exactly would fall. During training the GPU box sends me short clips, and I watch them through the day.

The gate then runs deterministic tests: knees in the air and at a heavy bag, six seeds each, with hits, falls, where the feet end, which way he faces and how far he leans. Teachers that pass are distilled into the student (policy distillation, described in the shadow-clones post). The student is checked again, because distillation is a second round of training and things can change in it.

Even after all of that, the result can be wrong.

The case: the right knee

The rear knee to the body comes from my take of 23 October: the lead foot steps in about 28 cm, the rear foot leaves the floor, the hips turn about 70° and the right knee drives up and forward into the opponent’s body, 1.14 m high at impact. The knee comes down, the rear foot lands ahead of where it left, and the fighter steps back into his stance. Its teacher took three runs (work log): the first landed every knee but finished in a southpaw stance 0.6 m forward, the second fixed the feet, and the third, kneeR3, made the steps home. kneeR3 passed the gate: in the air 5/6, at the bag 6/6, feet back in the stance, no falls. It went into the student, distill4.

What the training objective allowed

Leaving out the details, the teacher maximises, at each step $t$,

$$r_t = w_{\text{track}}\, r^{\text{track}}_t + w_{\text{hit}}\, r^{\text{hit}}_t + w_{\text{feet}}\, r^{\text{feet}}_t + \dots - c\,\mathbb{1}[\text{fall}]$$

following the reference, landing the knee (within 5 cm of the target and fast), keeping the feet where the reference has them, and not falling. None of these terms says anything about the shape of the trunk after the knee lands, as long as he doesn’t fall and the pose stays close enough to the reference.

And close enough it was. After the impact, my capture leans back a little: my pelvis 5–9° back and my chest 10–16°, while my stance is near 0° and +1.5°. kneeR3 followed that lean and pushed it further, tipping his pelvis up to about 8° further back than the reference. He bent his lower back about 20° to keep his chest on the reference: hips forward, chest back. That’s an arch. Then he recovered, sometimes with a small hop, and stood in his stance again. Every term was satisfied.

Why that is wrong

I noticed it in the student’s video: “after the knee and the feet land back on the ground the fighter arches its back”. From a reward’s point of view this is a perfectly good knee. From a fighter’s point of view it isn’t. For the half-second he is arched, his hips are pushed forward and his shoulders hang back over them, his feet are busy catching him instead of being set, and he can’t step, block or strike back (Figures 4 and 5). That’s when an opponent hits him. The technique isn’t finished when the knee lands; it’s finished when he’s ready again.

Measuring it

To fix it I first needed a number for it. The trunk lean of a body part is the angle of its “up” axis $\hat z$ along the direction the hips face on the floor, $\hat h$:

$$\theta = \arcsin\!\big(\hat z_{xy} \cdot \hat h\big), \qquad \theta > 0 \text{ forward}$$

computed for the pelvis and the chest. The gate’s new row is the most either one leans back past the reference after the hit:

$$L = \max_{t \,>\, t_{\text{up}}} \; \max_{b \,\in\, \{\text{pelvis},\, \text{chest}\}} \big[\theta^{\text{ref}}_b(t) - \theta_b(t)\big]_+$$

where $[x]_+ = \max(x, 0)$, so leaning forward of the reference counts as zero, and $t_{\text{up}}$ is the start of the upright window below. Against the corrected reference, kneeR3 scores $L$ = 12.4° in the air and 9.7° at the bag.

The fix

Two changes, one to the target and one to the reward.

The reference. From 7.05 s of the capture, just after the hip thrust, to 7.35 s, the pelvis’s lean and the spine’s bend are eased to the stance’s and held there to the end. Before 7.05 s the reference is unchanged, including the impact at 6.933 s. I approved the preview before any training. This is one of those trade-offs between copying me and what works: my own lean back after a knee is small and I recover from it without thinking, but on his body, with the policy free to exaggerate it, it turned into an arch.

A one-sided reward term. After the hit window, every step pays

$$r^{\text{upright}}_t = 1.5\, w_t \exp\!\left(-\frac{\big[\theta^{\text{ref}}_{\text{chest}} - \theta_{\text{chest}}\big]_+^2 + \big[\theta^{\text{ref}}_{\text{pelvis}} - \theta_{\text{pelvis}}\big]_+^2}{2\,(5^\circ)^2}\right)$$

where $w_t$ is the weight the technique’s terms have at that moment. This is a Gaussian kernel on the lean back past the reference: 1.5 when he’s at or forward of the reference, about 0.9 at 5° back, about 0.2 at 10° and almost nothing past 15°. Leaning forward is free, because leaning forward after a knee into the body is what a fighter does anyway.

The new teacher, kneeR4, was warm-started from kneeR3 on one rented RTX 5090 for its 2-hour cap (207.9 M steps, about $1.29). The lean dropped almost at once: by 13 M steps the bag row was down from 9.2° to 1.8°.

Results

Figure 2 shows the lean of the pelvis and the chest through one knee at the bag, step by step, before and after the fix, in the student. Each line is the median of six seeds and the band their range.

The picture is the same in the air and for the teachers. Figure 3 compares the two numbers that matter for all four models, and the table has the rest of the gate.

model	lean past reference, air / bag	hits, air / bag	falls	feet at the end (lead, rear; stance 0.37, 0 m)
kneeR3-183.5M (before)	12.4° / 9.7°	5/6, 6/6	0	0.42–0.44, −0.01–0.02
kneeR4-183.5M (after)	1.3° / 0.9°	0/6, 6/6	0	0.40–0.41, −0.01
distill4-131.1M (student before)	11.4° / 10.7°	in the gate: 11/12 overall	0
distill7-245.8M (student after)	1.4° / 1.2°	2/6, 6/6	0

The arch is the part I pointed at, but what worried me more was the balance. Seen straight from behind (Figure 5), the student before the fix spends the time after the landing catching himself with his feet. After the rear foot comes down it comes up again, 5–11 cm across the six seeds, where after the fix it rises 4–5 cm as he settles it. In seed 1 both feet are off their resting height at once for 0.44 s, a small hop to stay up. After the fix no seed does that. His centre of mass stays between his feet in both models, about two-thirds of the way from the rear foot to the lead foot, so he was never close to falling. He doesn’t fall, but he isn’t set either, and a fighter who isn’t set can’t fight.

What the fix cost, and what it didn’t fix

• Knee hits in the air. kneeR4 hits 0/6 in the air, where the kept kneeR3 snapshot hit 5/6. The fix may play a part, but the air row was unstable before it too: kneeR3’s later snapshots hit 2/6 and 0/6. The air test counts a hit only at 3 m/s within 5 cm of the target, a fist’s speed. The reference knee crosses the target plane at under 1 m/s, so a knee that follows my capture rarely counts. At the bag, which is three quarters of the presses and is scored by the push, every knee lands.
• The hips’ bob. The hips still rise and fall about 4–5 cm as he comes out of the knee, about the same before and after in the student. This was not the complaint, and the fix doesn’t target it.
• The stance mismatch. The upright window brings the pelvis to the reference’s stance, which is about 10° more upright than the guard stance the base model actually stands in. The term only punishes leaning back past the reference, so he ends between the two (Figure 2). A cleaner fix would take the reference’s stance from the base model.
• Not every snapshot. The fix holds across kneeR4’s snapshots (about 1–5° in the air, 0–1.8° at the bag), but the last one, at 206 M steps, leaned 10.8° in the air on one seed. 183.5 M was kept.
• Small samples. Six seeds per row, deterministic. That’s enough to see a 10° effect clearly, and not enough to rank differences of a degree.

Into the student

The fix was made in the teacher, not the student, because the student’s knee is only as good as the teacher it copies. distill7 was warm-started from distill4 and distilled again with kneeR4 as the knee’s teacher (along with two new uppercut teachers), on a RunPod RTX 5090 for 2.5 hours (work log). The distance between the student’s and kneeR4’s actions fell from 0.025 to 0.0027 (mean squared error). The student’s lean past the reference went from 11.4° / 10.7° to 1.4° / 1.2°, the same as the teacher, with no falls, the bag knees 6/6 and the old skills at about distill4’s level. I watched its videos and said: “Looks good.”

What I think, as a coach

On paper, the knee was already good. The policy maximises the knee’s impact on the target, it’s punished if it falls, and it tries to follow my reference, but not exactly, because the physics is different: the bag’s weight, the recoil, his body against mine. If it copied me exactly it would fall over. So when it staggered back after the knee, arched a little and then recovered, that was perfectly fine by everything we had told it.

But when I looked at it, all I could see was this: while he’s off balance, he’s going to get hit. In the gym, a knee you can’t recover from straight away is a bad knee, however hard it lands. Nobody counts that in a score; you just see it, the same way you see a fighter who drops his hands after a punch.

That’s why I don’t trust any stage of this pipeline on its numbers alone. The motion capture can be wrong, the table can twist a knee, the teacher can find a way to satisfy every term and still look wrong, and the student can lose something in distillation. Every time, the fix has been the same: watch it, say what’s wrong in plain words, turn that into something the training can measure, and check again. The knee went from “fine according to all the policy” to something I’d be happy to see in the ring.

The work logs

The runs behind this post are in these work logs: the teacher that fixed the knee, and the evaluation and render boxes that made this post’s figures (EC2 spot, about ten minutes each, because my small development server is throttled on CPU). The earlier knee teachers and the student distill7 are counted in the shadow-clones post and their own work log. The costs are each machine’s billed hours times its hourly price, from the vast.ai ledger and the EC2 spot price, so they are estimates, not the bill.