Statistical power

A test that finds “no clear difference” can mean two things: there is no difference, or the test was too small to see it. Statistical power separates them. It is the probability that a test will declare a difference, if a real difference of a given size exists. The usual target is 80 %.

Power grows with the number of trials and with the size of the difference being looked for, and shrinks with the noise. For comparing two success rates near $p$, the number of tries each side needs to detect a difference $\delta$ with 80 % power at the 95 % level is about

$$n \approx \frac{2\,(1.96 + 0.84)^2\, p (1 - p)}{\delta^2}$$

Halving the difference you want to detect quadruples the trials needed.

An underpowered test is not useless: it can still catch large effects. But its “no difference” is weak evidence that there is none. Its confidence intervals are wide, and that width is the honest summary.

In this project

The fighter’s gate has rows of 6 to 160 tries. To detect a real 10-point improvement on a row with a success rate near 70 %, the formula gives about 330 tries per network. The network-size experiment found no clear gate difference between the four networks, and this is part of why that result says little: the gate is designed to catch a student that has broken something, not to rank students that are all close.