Blind Guides for Explorers: P Values, Subgroups and Data Dredging

Abstract

Research investigators are trained to be thorough, to find everything worth findings in their data. Having invested great time and effort in their studies, these scientists want and need to examine the data systematically and completely. In clinical experiments where there are multiple endpoints, investigators will examine the effect of therapy on each one, with the anticipated reward of identifying a therapy effect for the experiment’s primary endpoint. However, investigators are aware that there may be unanticipated findings in other analyses, that, unlike the primary endpoint analysis, are nonprospectively stated evaluations. Investigators believe that, like unsuspected pots of gold, these tantalizing surprises might lie just under the surface, hidden from view, waiting to be found. If the experiment demonstrated that an intervention reduces the incidence of heart attacks, then maybe there is a hidden relationship between marital status and heart attacks. Perhaps there is an unanticipated relationship between the patient’s astrologic sign and the occurrence of a heart attack? Sometimes it is not the investigator who is raising these questions. In the process of publication, reviewers of the manuscript will sometimes ask that additional analyses be carried out. These analyses can include considering the effect of the intervention in subsets of the data. Does the therapy work equally well in women and men? Does it work equally in different racial groups? What about in patients with a previous heart attack? These analyses are demanded by others, but are also not prospectively stated.