Keypoint

A keypoint is one named point on a body: the head, a shoulder, an elbow, a wrist, a hip, a knee, an ankle. A list of them, joined up, is a stick-figure skeleton. It says where a body is and how it is posed in a few dozen numbers, where a camera image needs millions of pixels.

Software that finds keypoints in a camera image is called a pose estimator. Phone apps that track a workout or put a filter on a face use one. A robot with a camera can run one too, so a policy that reads keypoints is reading the kind of input a real robot would have.

Positions alone don’t say much about what a body is about to do: a fist at rest and a fist starting a punch look the same in one snapshot. So keypoints are usually given with their velocities, and with a short history of earlier positions (frame stacking).

How to give a point’s position matters for learning. A distance and two angles is natural for people but awkward for a neural network, because an angle jumps from 359° to 0° though the direction barely changed. A 3D vector $(x, y, z)$ in the viewer’s own frame holds the same information without the jump.

In this project

In the blocking post the fighter sees its opponent as 16 keypoints: head, neck, shoulders, elbows, wrists, fists, hips, knees and ankles. Each one is a 3D vector from the fighter’s head, in a frame that turns with the head but stays level, plus its velocity, a flag for whether it can be seen (occlusion), and its positions 40 and 80 ms earlier. Every keypoint arrives 100 ms late (latency) with 1 cm of noise. That is 208 numbers added to the fighter’s observation, 445 in all. Learning from pixels instead would have cost orders of magnitude more training.