Skip to main content
Log in

Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right

Psychometrika Aims and scope Submit manuscript

Abstract

Recent fraud cases in psychological and medical research have emphasized the need to pay attention to Questionable Research Practices (QRPs). Deliberate or not, QRPs usually have a deteriorating effect on the quality and the credibility of research results. QRPs must be revealed but prevention of QRPs is more important than detection. I suggest two policy measures that I expect to be effective in improving the quality of psychological research. First, the research data and the research materials should be made publicly available so as to allow verification. Second, researchers should more readily consider consulting a methodologist or a statistician. These two measures are simple but run against common practice to keep data to oneself and overestimate one’s methodological and statistical skills, thus allowing secrecy and errors to enter research practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abma, R. (2013a). De publicatie fabriek. Over de betekenis van de affaire-Stapel (The publication factory. On the meaning of the Stapel affair). Nijmegen, The Netherlands: Uitgeverij Vantilt.

  • Abma, R. (2013b). Het jaar van Levelt? (The year of Levelt?). De Psycholoog, 48(3), 34–39.

    Google Scholar 

  • Anderson, M. S. (2014). Global research integrity in relation to the United States’ research-integrity infrastructure. Accountability in Research, 21, 1–8.

    Article  PubMed  Google Scholar 

  • Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society: Series A, 132, 235–244.

    Article  Google Scholar 

  • Asendorpf, J. B., et al. (2013a). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.

    Article  Google Scholar 

  • Asendorpf, J. B., et al. (2013b). Authors’ response: Replication is more than hitting the lottery twice. European Journal of Personality, 27, 138–144.

    Article  Google Scholar 

  • Bakker, M., Van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.

    Article  PubMed  Google Scholar 

  • Barnett, V., & Lewis, T. (1994). Outliers in statistical data. Chichester, United Kingdom: Wiley.

    Google Scholar 

  • Bhattacharjee, Y. (2013). The mind of a con man. The New York Times April 28. Retrieved from http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all&_r=0.

  • Berkhout, K. (2012). Het moet ergens in die doos zitten (It has to be somewhere in that box). Nieuwe Rotterdamse Courant NRC, NRC Weekend June 30/July 1.

  • Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.

    Article  PubMed Central  PubMed  Google Scholar 

  • Carey, B. (2011). Fraud case seen as a red flag for psychology research. The New York Times November 2, 2011. Retrieved from http://www.nytimes.com/2011/11/03/health/research/noted-dutch-psychologist-stapel-accused-of-research-fraud.html?_r=0.

  • Carpenter, S. (2012). Psychology’s bold initiative. Science, 335, 1558–1561.

    Article  PubMed  Google Scholar 

  • Cronbach, L. J. (1954). Report on a psychometric mission to Clinicia. Psychometrika, 19, 263–270.

    Article  Google Scholar 

  • De Dreu, C. (2012). Elkaar constructief de maat nemen (Providing constructive criticism). De Psycholoog, 47(10), 34–37.

    Google Scholar 

  • Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(4), e5738. doi:10.1371/journal.pone.0005738.

    Article  PubMed Central  PubMed  Google Scholar 

  • Fanelli, D. (2010). Positive results increase down the hierarchy of the sciences. PLoS ONE, 5(4), e10068. doi:10.1371/journal.pone.0010068.

    Article  PubMed Central  PubMed  Google Scholar 

  • Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 149, 494. Retrieved from http://www.nature.com/news/redefine-misconduct-as-distorted-reporting-1.12411.

  • Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555–561.

    Article  PubMed  Google Scholar 

  • Fischer, G. H. (1974). Einführung in die Theorie psychologischer Tests (Introduction to the theory of psychological tests). Bern, Switzerland: Huber.

    Google Scholar 

  • Fisher, R. A. (1936). Has Mendel’s work been rediscovered? Annals of Science, 1, 115–137.

    Article  Google Scholar 

  • Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7, 639–642.

    Article  PubMed  Google Scholar 

  • Goodstein, D. (2010). On fact and fraud. Cautionary tales from the front lines of science. Princeton: Princeton University Press.

    Google Scholar 

  • Hartshorne, J. K., & Schachner, A. (2012). Tracking replicability as a method of post-publication open evaluation. Frontiers in Computational Neuroscience, 6(3), doi:10.3389/fncom.2012.00008.

  • Hofstee, W. K. B. (2013). Psychologie als wedstrijd. Integer wetenschappelijk onderzoek (Psychology as competition. Honest scientific research). De Psycholoog, 48(4), 28–36.

    Google Scholar 

  • Huff, D. (1954). How to lie with statistics. New York: Norton.

    Google Scholar 

  • Hunt, E. (2013). Calls for replicability must go beyond motherhood and apple pie. European Journal of Personality, 27, 126–127.

    Google Scholar 

  • Hubert, L., & Wainer, H. (2013). A statistical guide for the ethically perplexed. Boca Raton, FL: CRC Press.

    Google Scholar 

  • Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.

    Article  PubMed Central  PubMed  Google Scholar 

  • Ioannidis, J. P. A., & Trakalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253.

    Article  PubMed  Google Scholar 

  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.

    Article  PubMed  Google Scholar 

  • Judson, H. F. (2004). The great betrayal. Fraud in science. Orlando, FL: Harcourt.

    Google Scholar 

  • Kadane, J. B., Schervish, M. J., & Seidenfeld, T. (1996). Reasoning to a foregone conclusion. Journal of the American Statistical Association, 91, 1228–1235.

    Article  Google Scholar 

  • Kahneman, D. (2011). Thinking, fast and slow. London: Penguin Books Ltd.

    Google Scholar 

  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.

    Article  PubMed  Google Scholar 

  • Kevles, D. J. (1998). The Baltimore case: A trial of politics, science, and character. New York: W.W. Norton.

    Google Scholar 

  • KNAW Committee on Research Data. (2012). Zorgvuldig en integer omgaan met wetenschappelijke onderzoeksgegevens (Accurate and correct handling of scientific research data). Royal Netherlands Academy of Arts and Sciences. Amsterdam: The Netherlands. Retrieved from http://www.knaw.nl/Content/Internet_KNAW/publicaties/pdf/20121004.pdf.

  • Kullmann, K. (2012). Zu schön, um wahr zu sein (To good to be true). Der Spiegel, 35, 123–124.

    Google Scholar 

  • Lamont, M. (2009). How professors think. Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press.

    Book  Google Scholar 

  • Lehrer, J. (2010). The truth wears off. Is there something wrong with the scientific method? The New Yorker. Annals of Science.

  • Levelt, W. J. M. (2011). Interim report regarding the breach of scientific integrity committed by Prof. D. A. Stapel. Retrieved from http://www.tilburguniversity.edu/nl/nieuws-en-agenda/commissie-levelt/.

  • Levelt Committee, Noort Committee, Drenth Committee. (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved from http://www.tilburguniversity.edu/nl/nieuws-en-agenda/finalreportLevelt.pdf.

  • Lilienfeld, S. O. (2012a). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. American Psychologist, 67, 111–129.

    Article  PubMed  Google Scholar 

  • Lilienfeld, S. O. (2012b). Scientific Utopia or scientific dystopia? Psychological Inquiry, 23, 277–280.

    Article  Google Scholar 

  • MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490–504.

    Article  PubMed  Google Scholar 

  • Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 161–175.

    Article  Google Scholar 

  • Masicampo, E. J., & Lalande, D. R. (2012). A peculiar preference of p values just below.05. The Quarterly Journal of Experimental Psychology, 65, 2271–2279.

    Article  PubMed  Google Scholar 

  • Nosek, B. A., & Bar-Anan, Y. (2012). Scientific Utopia I: Opening scientific communication. Psychological Inquiry, 23, 217–243.

    Article  Google Scholar 

  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.

    Article  PubMed  Google Scholar 

  • Panter, A. T., & Sterba, S. K. (2011). Handbook of ethics in quantitative methodology. Hove, United Kingdom: Routledge.

    Google Scholar 

  • Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.

    Article  PubMed  Google Scholar 

  • Piegorsch, W. W. (1990). Fisher’s contributions to genetics and heredity, with special emphasis on the Gregor Mendel controversy. Biometrics, 46, 915–924.

    Article  PubMed  Google Scholar 

  • Raftery, A. E. (1995). Bayesian model selection in social research (with Discussion). Sociological Methodology, 25, 111–196.

    Article  Google Scholar 

  • Robbins, H. E. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58, 527–535.

    Article  Google Scholar 

  • Ross, S. M. (1994). A first course in probability (4th ed.). New York: McMillan.

    Google Scholar 

  • Saunders, R., & Savulescu, J. (2008). Research ethics and lessons from Hwanggate: What can we learn from the Korean cloning fraud? Journal of Medical Ethics, 34, 214–221.

    Article  PubMed  Google Scholar 

  • Shavelson, R. J., & Towne, L. (2002). Scientific research in education. Committee on Scientific Principles for Education Research. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

  • Sijtsma, K. (2012). Future of psychometrics: Ask what psychometrics can do for psychology. Psychometrika, 77, 4–20.

    Article  Google Scholar 

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  PubMed  Google Scholar 

  • Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888.

    Article  PubMed  Google Scholar 

  • Simonsohn, U., Nelson, L. D., & Simmons, J. (2014). P-Curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547.

    Article  Google Scholar 

  • Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12, 53–74.

    Article  PubMed  Google Scholar 

  • Tucker, W. H. (1997). Re-reconsidering Burt: Beyond a reasonable doubt. Journal of the History of the Behavioral Sciences, 33, 145–162.

    Article  PubMed  Google Scholar 

  • Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.

  • Van de Poll-Franse, L. V., Horevoorts, N., van Eenbergen, M., Denollet, J., Roukema, J. A., Aaronson, N. K., et al. (2011). The Patient Reported Outcomes Following Initial treatment and Long term Evaluation of Survivorship registry: Scope, rationale and design of an infrastructure for the study of physical and psychosocial outcomes in cancer survivorship cohorts. European Journal of Cancer, 47, 2188–2194.

    Article  PubMed  Google Scholar 

  • Van der Pligt, J. (2013). Het jaar van Stapel (The year of Stapel). De Psycholoog, 48(3), 28–33.

    Google Scholar 

  • Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.

    Article  Google Scholar 

  • Wagenmakers, E. J., Wetzels, R., Borsboom, D., & Van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of Psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432.

    Article  PubMed  Google Scholar 

  • Wagenmakers, E. J., Wetzels, R., Borsboom, D., Van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

    Article  PubMed  Google Scholar 

  • Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6(11), e26828. doi:10.1371/journal.pone.0026828.

    Article  PubMed Central  PubMed  Google Scholar 

  • Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.

    Article  PubMed  Google Scholar 

  • Wilson, M. (2013). Seeking a balance between the statistical and scientific elements in psychometrics. Psychometrika, 78, 211–236.

    Article  PubMed  Google Scholar 

  • Yong, E. (2012). Bad copy. In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature, 485, 298–300.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Jodi Casabianca, Jaap Denissen, Hans Dieteren, Wilco Emons, Ellen Evers, Brian Junker, Jay Kadane, Roger Millsap, Sarah Ryan, Coosje Veldkamp, Jeroen Vermunt, Job van Wolferen, and Jelte Wicherts provided comments on previous drafts of this manuscript. The end result is the author’s responsibility.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Klaas Sijtsma.

Appendix

Appendix

I arrange the birthdays of \(n\) persons in an array. Assuming all dates are equally likely and February 29 is excluded, there are \(T=365^{n}\) different outcomes. Each outcome has the same probability. I ask for the probability that no two persons have their birthdays on the same date or, equally, that \(n\) persons have their birthdays on \(n\) different dates. For the first person, there are 365 different dates available, for the second person 364 dates except the date the first person gave, and so on. Hence, the number of different outcomes containing \(n\) unique dates equals \(U=(365)(364)(363)\ldots (365-n+1)\), and the probability \(P\) that no two persons have their birthday on the same day equals \(U/T\). For \(n=23\), one finds \(P<.5\), so the probability that at least two persons in the group have their birthdays on the same day exceeds .5, and so on for other values of \(n\).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sijtsma, K. Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right. Psychometrika 81, 1–15 (2016). https://doi.org/10.1007/s11336-015-9446-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-015-9446-0

Keywords

Navigation