Psychometrika

, Volume 81, Issue 1, pp 33–38 | Cite as

Improving the Conduct and Reporting of Statistical Analysis in Psychology

  • Klaas Sijtsma
  • Coosje L. S. Veldkamp
  • Jelte M. Wicherts
Article

Abstract

We respond to the commentaries Waldman and Lilienfeld (Psychometrika, 2015) and Wigboldus and Dotch (Psychometrika, 2015) provided in response to Sijtsma’s (Sijtsma in Psychometrika, 2015) discussion article on questionable research practices. Specifically, we discuss the fear of an increased dichotomy between substantive and statistical aspects of research that may arise when the latter aspects are laid entirely in the hands of a statistician, remedies for false positives and replication failure, and the status of data exploration, and we provide a re-definition of the concept of questionable research practices.

Keywords

data exploration questionable research practices remedies for false positives replication failure 

References

  1. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.PubMedCentralCrossRefPubMedGoogle Scholar
  2. Bakker, M., & Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in t tests. The power of alternatives and recommendations. Psychological Methods, 19, 409–427.CrossRefPubMedGoogle Scholar
  3. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.CrossRefPubMedGoogle Scholar
  4. Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.PubMedCentralCrossRefPubMedGoogle Scholar
  5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  6. Cronbach, L. J. (1954). Report on a psychometric mission to Clinicia. Psychometrika, 19, 263–270.CrossRefGoogle Scholar
  7. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.CrossRefPubMedGoogle Scholar
  8. Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 494, 149.CrossRefPubMedGoogle Scholar
  9. Fisher, R. A. (1938). Presidential Address. Talk given at the 1st Indian Statistical Conference, Calcutta, India.Google Scholar
  10. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.PubMedCentralCrossRefPubMedGoogle Scholar
  11. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.CrossRefPubMedGoogle Scholar
  12. Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). Chicago, IL: The University of Chicago Press.CrossRefGoogle Scholar
  13. Martinson, B. C., Anderson, M. S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737–738. doi:10.1038/435737a.CrossRefPubMedGoogle Scholar
  14. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.CrossRefPubMedGoogle Scholar
  15. Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127–134.CrossRefPubMedGoogle Scholar
  16. Sijtsma, K. (2015). Playing with data–Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika. doi:10.1007/s11336-015-9446-0.
  17. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefPubMedGoogle Scholar
  18. Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12, 53–74.CrossRefPubMedGoogle Scholar
  19. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.Google Scholar
  20. Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PloS ONE, 9, e114876.PubMedCentralCrossRefPubMedGoogle Scholar
  21. Waldman, I. D., & Lilienfeld, S. O. (2015). Thinking about data, research methods, and statistical analyses: Commentary on Sijtsma’s (2014) “Playing with data". Psychometrika. doi:10.1007/s11336-015-9447-z.
  22. Wigboldus, D. H. J., & Dotch, R. (2015). Encourage playing with data and discourage questionable reporting practices. Psychometrika. doi:10.1007/s11336-015-9445-1.

Copyright information

© The Psychometric Society 2015

Authors and Affiliations

  • Klaas Sijtsma
    • 1
  • Coosje L. S. Veldkamp
    • 1
  • Jelte M. Wicherts
    • 1
  1. 1.Department of Methodology and Statistics, Tilburg School of Social and Behavioral SciencesTilburg UniversityTilburgThe Netherlands

Personalised recommendations