Statistics in a World Without Science

Abstract

There is an accepted framework for producing scientific evidence (the philosophy of science), and another for using random samples from a population to summarize that evidence (statistical inference). Central to the accepted philosophy of science is the idea of falsifiability, which is generally adhered to in randomized controlled trials (RCTs) using the null hypothesis significance testing approach defined by Neyman and Pearson. Beyond the tight control of RCTs, the flexibility of statistical inference means it is relatively easy to overlook falsifiability, thereby undermining the validity of any scientific conclusion. Perhaps the typical example of this is HARKing (hypothesizing after the results are known), where a statistical analysis of sample data reveals a significant effect that has a very convincing explanation only in retrospect and is reported as if hypothesized a priori. Such ‘discoveries’ may well be valid, but the rate at which they might not be is unknown. This is contributing to a crisis of replication in medical research, where results from published studies are often not reproducible.

Science is considered a difficult subject, and it may not be appropriate to teach the philosophy behind it to medical students who have other time commitments, including also the learning of difficult methods of statistics. Under these circumstances, what can be taught to promote the use of statistics within the scientific framework and mitigate the crisis of replication? roposed suggestions range from producing registered study protocols to the abandonment of statistical significance; importantly, both can be implemented without need for expertise in philosophy. Despite this, the replication crisis persists and the implementation of any proposed solution, and perhaps even awareness of them, is the exception. As (if) we move to better scientific practice, it is likely that those who teach statistics will be required to include the suggestions into their courses.