Inference Guided Data Exploration

Abstract

We consider comparing two treatments using a given hypothesis test on the full sample and on all possible subsets, and we separately consider restricting the subsets considered to be those defined based on half-intervals of a covariate. Rather than treating this as a family of hypothesis tests, we instead choose the minimum p-value from the group of hypothesis tests as our test statistic. Simulation is employed to find an approximate critical value to control the type I error for our novel test statistic. These techniques may be used as a rule of thumb for judging the potential significance of a result after a “fishing expedition” has been caried out on a dataset, i.e., a large number of tests of hypothesis were performed on subsets of the data or a subset was selected after inspecting the data. When the technique is restricted to subsets defined based on half-intervals of a covariate, it may be useful as a planned methodology for analyzing an experiment.