Improving the Conduct and Reporting of Statistical Analysis in Psychology


We respond to the commentaries Waldman and Lilienfeld (Psychometrika, 2015) and Wigboldus and Dotch (Psychometrika, 2015) provided in response to Sijtsma’s (Sijtsma in Psychometrika, 2015) discussion article on questionable research practices. Specifically, we discuss the fear of an increased dichotomy between substantive and statistical aspects of research that may arise when the latter aspects are laid entirely in the hands of a statistician, remedies for false positives and replication failure, and the status of data exploration, and we provide a re-definition of the concept of questionable research practices.

This is a preview of subscription content, access via your institution.


  1. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.

    PubMed Central  Article  PubMed  Google Scholar 

  2. Bakker, M., & Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in t tests. The power of alternatives and recommendations. Psychological Methods, 19, 409–427.

    Article  PubMed  Google Scholar 

  3. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.

    Article  PubMed  Google Scholar 

  4. Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.

    PubMed Central  Article  PubMed  Google Scholar 

  5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  6. Cronbach, L. J. (1954). Report on a psychometric mission to Clinicia. Psychometrika, 19, 263–270.

    Article  Google Scholar 

  7. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.

    Article  PubMed  Google Scholar 

  8. Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 494, 149.

    Article  PubMed  Google Scholar 

  9. Fisher, R. A. (1938). Presidential Address. Talk given at the 1st Indian Statistical Conference, Calcutta, India.

  10. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.

    PubMed Central  Article  PubMed  Google Scholar 

  11. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.

    Article  PubMed  Google Scholar 

  12. Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). Chicago, IL: The University of Chicago Press.

    Book  Google Scholar 

  13. Martinson, B. C., Anderson, M. S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737–738. doi:10.1038/435737a.

    Article  PubMed  Google Scholar 

  14. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.

    Article  PubMed  Google Scholar 

  15. Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127–134.

    Article  PubMed  Google Scholar 

  16. Sijtsma, K. (2015). Playing with data–Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika. doi:10.1007/s11336-015-9446-0.

  17. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  PubMed  Google Scholar 

  18. Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12, 53–74.

    Article  PubMed  Google Scholar 

  19. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.

    Google Scholar 

  20. Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PloS ONE, 9, e114876.

    PubMed Central  Article  PubMed  Google Scholar 

  21. Waldman, I. D., & Lilienfeld, S. O. (2015). Thinking about data, research methods, and statistical analyses: Commentary on Sijtsma’s (2014) “Playing with data". Psychometrika. doi:10.1007/s11336-015-9447-z.

  22. Wigboldus, D. H. J., & Dotch, R. (2015). Encourage playing with data and discourage questionable reporting practices. Psychometrika. doi:10.1007/s11336-015-9445-1.

Download references

Author information



Corresponding author

Correspondence to Klaas Sijtsma.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sijtsma, K., Veldkamp, C.L.S. & Wicherts, J.M. Improving the Conduct and Reporting of Statistical Analysis in Psychology. Psychometrika 81, 33–38 (2016).

Download citation


  • data exploration
  • questionable research practices
  • remedies for false positives
  • replication failure