Shortcut learning

A learning system takes whatever signal predicts the answer most easily. If the training data contains a cue that wasn’t meant to matter but happens to line up with the answer, it will often learn the cue.

The classic story is Clever Hans, a horse in early-1900s Germany that seemed to do arithmetic, tapping out answers with its hoof. It was reading its questioner’s face, which relaxed when the right number of taps was reached. In machine learning, a famous case is an image classifier for pneumonia that learnt to recognise which hospital’s scanner took the X-ray, because one hospital had more sick patients.

It is a cousin of reward hacking. In reward hacking the agent exploits a flaw in what it is paid for. In shortcut learning it exploits a flaw in what it is shown. Both pass the test and fail the real task.

How to find and fix one

• Change the cue and nothing else, and see whether the behaviour changes. If the model acts the same when the real signal moves, it isn’t using the real signal.
• Remove the cue from the environment or the data. Penalising it rarely works: the model finds the next one.
• Expect several in a row. Fixing one shortcut often reveals the next easiest.

In this project

In the blocking post, the fighter had to read which punch the opponent was throwing. Four times it found a giveaway instead. The opponent stood frozen between punches and came alive 0.18 s before each one. Each punch type was thrown from its own distance (Tony spotted this), and partly overlapping distances still gave it away. With one distance range for all, the uppercut still stood the opponent 4–5 cm lower. Each was removed in turn, and only then did the fighter wait for the punch and read it. The check that showed it: move the opponent’s punch later in time, and see whether the fighter’s blocks move with it.