Hypothesis Testing and Small Sample Sizes

Abstract

One of the biggest breakthroughs in the past forty years is the derivation of inferential methods that perform well when sample sizes are small Indeed, some practical problems that seemed insurmountable only a few years ago have been solved. But to appreciate this remarkable achievement, we must first describe the shortcomings of conventional techniques developed during the first half of the twentieth century—methods that are routinely used today. At one time it was generally thought that these standard methods are insensitive to violations of assumptions, but a more accurate statement is that they seem to perform reasonably well (in terms of Type I errors) when groups have identical probability curves or when performing regression with variables that are independent. If, for example, we compare groups that happen to have different probability curves, extremely serious problems can arise. Perhaps the most striking problem is described in Chapter 7, but the problems described here are also very serious and are certainly relevant to applied work.