“Your Searchlight’s Not On!” Power and P Values

Abstract

Consider an experiment which is coming to an end. At its conclusion, the experiment’s prinicple investigator often feels hemmed in, cornered, and eventually trapped by persistent critics. They left him alone during both the design and the execution of the experiment. However, as soon as he attempts to draw a conclusion from his sample, hesitant and circumspect as these conclusions may be, the specter of sampling error rises up. Even if the results are negative, demonstrating no relationship between the treatment and the disease, the critics close in, asking now about type II error. Isn’t it possible that there was an effect in the population, but, again through the play of chance, the population produced for the investigator a sample that demonstrated no relationship. Critics relentlessly remind him that his findings may be nothing at all, merely the vicissitude of the population, Like a weed, sampling error rises up to strangle the early blossom of his result. Is there no hope?