Confidence interval

Measure anything a limited number of times and the result carries some luck. A technique that lands 7 times in 10 tries might land 5 or 9 times in the next ten. A confidence interval puts a range around the measured value that accounts for this. A 95 % interval is built so that, if the experiment were repeated many times, 95 % of the intervals made this way would contain the true value.

For a success rate $\hat p = k / n$ from $n$ tries, the simplest interval is

$$\hat p \pm 1.96 \sqrt{\frac{\hat p (1 - \hat p)}{n}}$$

Its width shrinks only with the square root of $n$: four times the tries for half the width.

To compare two success rates, the interval is put around their difference. When the interval contains zero, the data are consistent with no difference at all, and the comparison is called inconclusive at that level. That is different from showing the two are equal: a wide interval that contains zero also contains large differences either way.

The simple formula breaks down at the edges: at 0 of 6 or 6 of 6 it gives an interval of width zero, as if the result were certain. The Agresti–Caffo interval for a difference fixes this by adding one success and one failure to each group before computing:

$$\tilde p_i = \frac{k_i + 1}{n_i + 2}, \qquad (\tilde p_2 - \tilde p_1) \pm 1.96 \sqrt{\frac{\tilde p_1 (1 - \tilde p_1)}{n_1 + 2} + \frac{\tilde p_2 (1 - \tilde p_2)}{n_2 + 2}}$$

In this project

The network-size experiment put an Agresti–Caffo 95 % interval around each gate row’s difference between a bigger network and the 512 × 2 control. Every one of 1024 × 3’s 21 intervals contained zero. How wide they were depended on the row’s tries: about ±23 points for a 30-try kick row, more than ±50 for a knee row with 6 tries (see statistical power).