Glossary · Machine learning
Interpretability
Also called explainability, mechanistic interpretability, explainable AI, XAI, probing, ablation
Looking inside a trained neural network to understand what its parts do and why it acts as it does, instead of judging it only by its behaviour.
A trained neural network works, but nobody wrote down how. Its knowledge is spread over thousands or billions of weights that training set. Interpretability is the work of reading some of that back out: which inputs a decision depended on, what a neuron or group of neurons stands for, and which parts of the network a behaviour needs.
Ways to look
- Recording activity. Run the network on chosen inputs and record what every neuron of a hidden layer does. Neurons that respond to one situation and not others hint at what they represent.
- Comparing patterns. Treat the whole layer’s response to a situation as a vector and compare situations, for example with cosine similarity. Situations the network treats alike have similar vectors.
- Probing. Train a small, simple model to read a quantity (say, which way the body is falling) from a layer’s activity. If it can, the layer holds that information.
- Ablation. Switch neurons off, or hold them at their resting value, and see what breaks. This is the step from “goes together with” to “is needed for”.
Recording and comparing show what goes together; only experiments like ablation show what causes what.
In this project
Torque and neural visualisation of Muay Thai techniques records the second layer of the student distill9-196.6M (512 neurons) through pushes, a knockdown and fourteen techniques. A press lights 108–140 neurons on the next tick; a cross and a rear hook share 28 of their 40 most responsive neurons, a jab and a left high kick 2. No ablation has been run yet.