Human-in-the-loop

In human-in-the-loop training, a person is part of the training process. They watch results, decide whether they’re right, and change the task, the reward or the data in response.

It’s used when the goal is hard to write as a formula. “Does this stance look like a fighter’s?” is easy for a coach to answer from a video and very hard to write as a reward function. A reward can measure joint angles and weight splits, but it can’t measure looking right.

Ways to put a human in the loop

• Judging and redesigning. The person watches the policy, rejects what looks wrong, and changes the environment or reward. This is what we do.
• Demonstrations. The person shows the motion, for example with motion capture or photos, and the agent imitates it.
• Preferences. The person compares two clips and says which is better. A model learns to predict their choices and becomes the reward. This is the idea behind reinforcement learning from human feedback (RLHF, Christiano et al. 2017), which is also used to train chat models.

Why it matters

Automatic rewards get gamed (see reward hacking). A person watching the video catches it. Several of our results scored well and looked wrong.

In this project

The coach set every target and judged every result by eye. Decisions that came from watching, not from a metric, include the gym mat, the heel roll in the rhythm, going on past 600 N to 860 N, and rejecting the toeless simple body because its foot looked like a plank.

In the punching work the loop runs through the coach’s phone. Grids of test jabs arrive while he’s out, and he answers by voice. His calls set the direction: speed and style up to the impact matter more than a strict return; keep the shoves low for now; drop the too-close jab (an elbow or knee suits that range). See Coaching an AI fighter.