Sign test

Sometimes two options are compared on many separate measures at once, and each measure on its own is too noisy to settle anything. The sign test asks a simpler question: across all of them, does one option come out ahead more often than a coin toss would?

It counts only the direction of each comparison, a win or a loss, and drops ties. If the two options were really equal, each comparison would be a fair coin toss, and the number of wins $k$ out of $n$ would follow a binomial distribution. The test’s p-value is the chance of a split at least as lopsided as the one seen, if the options were equal:

$$p = 2 \sum_{i=0}^{\min(k,\,n-k)} \binom{n}{i} \left(\tfrac12\right)^{n}$$

A small p-value (say below 0.05) is evidence that one option really is better. A large one means the split is the kind chance produces all the time.

The test ignores how big each win was, so it is blunt. It also assumes the comparisons are independent, which holds only roughly when the measures share causes.

In this project

In the network-size experiment, 1024 × 3 was better than the 512 × 2 control on 8 gate rows, worse on 10 and tied on 3. The sign test gives p = 0.82: no evidence either way. 1024 × 2 (9 better, 10 worse) and 512 × 3 (8 and 8) give p = 1.0.