Reward hacking

An RL agent maximises the reward it is given, not the goal the designer had in mind. When the two differ, the agent will find the difference. This is reward hacking, also called specification gaming.

A well-known example is a boat-racing game where an agent, rewarded for hitting targets, learned to circle in a lagoon hitting the same respawning targets forever instead of finishing the race. Its score beat human players. It never won a race.

It isn’t a bug in the learning algorithm. The algorithm did its job. The reward was wrong.

How to spot it

• The score is high but the behaviour looks wrong when you watch it.
• The agent spends a lot of time in a state you didn’t expect to matter.
• Small reward changes cause large behaviour changes.

Watching videos is the most reliable check, which is one reason human-in-the-loop training works well for motion.

How to fix it

• Make sure no unintended behaviour pays as much as the intended one. Compare the reward per step of each.
• Remove the opportunity in the environment rather than penalising it.
• Use potential-based shaping for guidance terms, which can’t create new optimal behaviours.

In this project

In the push-recovery run balance3, the policy stayed in recovery mode 95 % of the time even with no shoves. A wide, upright recovery pose paid about 0.7 per step, about as much as the stance it could actually hold (about 0.75). So it kept its balance point near the edge of its feet, and recoveries kept re-triggering.

The fix was to cap the recovery reward at about 0.6, below a good stance step, end recovery once the body was calm, and only start a new recovery on a real loss of balance. Afterwards the body spent 0 % of shove-free time in recovery mode.

In the speed work of 18 October, the policy was given control of each punch’s clock and a bonus for arriving sooner. It slowed every punch down instead: the style terms paid on every step, so a longer punch earned more of them than the bonus gave. Scaling the per-step pay by the clock’s rate made a punch’s style pay the same at any speed. Then a punch that stopped 2.6 cm short at 10 m/s scored more than one that arrived at 2.4 m/s, so the jab and cross learnt to stop short until that score was cut (see Faster punches).

In the walking work, a reward paid for the right share of the weight on each foot between steps. The fighter’s way of loading its front foot was to bow over it: a forward lean 8.2° beyond its stance on average, up to 16°. Removing that reward and paying for an upright trunk took the bow away (see Footwork).