Testing is Confidence Estimation: Partition Multiple Testing

Abstract

Hypothesis testing and confidence set estimation coexist as two related but disparate methods of statistical inference. As is well known, a confidence set is equivalent to a family of hypothesis tests—a relationship often exploited to develop confidence sets. In this paper, an extension of hypothesis testing is considered which often makes testing equivalent to confidence estimation. In the extension, with roots in partition multiple testing, all parameter points are tested—not only those corresponding to the null hypotheses. Consequently, the conclusion of a test is a confidence set, whether or not the null hypothesis is rejected, and the more specific inference of a confidence set is obtained with no loss of power to reject the null hypothesis. The methodology, known to be applicable for individual tests, is extended here to multiple testing, or multiple comparisons.

The literature on confidence estimation is reviewed in view of this extension of testing, to explore the nature and extent of the resulting stronger connections between these two methods of inference. Cases considered include basic t-tests and related extensions, the Tukey-Kramer and Scheffè single-step multiple tests, and certain step-down partition tests. Compelling evidence is provided that the traditional approach to hypothesis testing should in almost all cases be replaced with the partition multiple test extension presented here, so as to yield the same specific conclusions as corresponding confidence sets.