Motion retargeting

A clip of a person can’t be played on a different body by copying positions. If the simulated fighter’s legs are longer relative to his trunk, putting his feet exactly where the performer’s were bends his knees oddly, and putting his fists where the performer’s were folds his elbows into his head.

Retargeting chooses what to keep. The common choice, and ours, is to keep directions and turns (which way each bone points, how the pelvis and chest are rotated) and let positions follow from the new body’s lengths. Then a small inverse kinematics step fixes only what must touch something: planted feet flat on the floor and not sliding, and the striking limb reaching its target.

For each frame this is an optimisation over the new body’s joint angles $q$:

$$\min_q \; \sum_{b} w_b \,\big\lVert \hat d_b(q) - \hat d_b^{\,\text{human}} \big\rVert^2 \;+\; \sum_{c} w_c \,\big\lVert p_c(q) - p_c^{\text{target}} \big\rVert^2$$

where $\hat d_b$ is the unit direction of bone $b$ and $p_c$ the position of a contact (a heel, a toe).

In this project

The fighter’s legs are about 25 % longer than the coach’s relative to the trunk. The retarget step (mocap/retarget.py in kakuto-ml) matches bone directions and segment turns, pins the contacts in a second pass, and holds joints the cameras can’t see (forearm roll, wrists, collarbones) at the stance. On the sim’s own motions, rendered by two virtual phones and run through the whole pipeline, the bone directions came back with a median error of 1–6°.