Psychonomic Bulletin & Review

, Volume 21, Issue 5, pp 1180–1187 | Cite as

The frequency of excess success for articles in Psychological Science

Theoretical Review
First Online:

Abstract

Recent controversies have questioned the quality of scientific practice in the field of psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009 and 2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings have been suppressed, the experiments or analyses were improper, or the theory does not properly account for the data. In total, problems appeared for 82 % (36 out of 44) of the articles in Psychological Science that had four or more experiments and could be analyzed.

Keywords

Statistical inference Statistics Probabilistic reasoning 
This is a preview of subscription content, log in to check access.

Supplementary material

13423_2014_601_MOESM1_ESM.zip (2.1 mb)
ESM 1 (ZIP 2.06 mb)

References

  1. Alter, A. L., & Oppenheimer, D. M. (2009). Suppressing secrecy through metacognitive ease: Cognitive fluency encourages self-disclosure. Psychological Science, 20, 1414–1420.PubMedCrossRefGoogle Scholar
  2. Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764–771.PubMedCrossRefGoogle Scholar
  3. Ashton-James, C. E., Maddux, W. W., Galinsky, A. D., & Chartrand, T. L. (2009). Who I am depends on how I feel: The role of affect in the expression of culture. Psychological Science, 20, 340–346.PubMedCrossRefGoogle Scholar
  4. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi: 10.1177/1745691612459060 CrossRefGoogle Scholar
  5. Balcetis, E., & Dunning, D. (2010). Wishful seeing: More desired objects are seen as closer. Psychological Science, 21, 147–152.PubMedCrossRefGoogle Scholar
  6. Bauer, M. A., Wilkie, J. E. B., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing consumerism: Situational materialism undermines personal and social well-being. Psychological Science, 23, 517–523.PubMedCrossRefGoogle Scholar
  7. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.PubMedCrossRefGoogle Scholar
  8. Birtel, M. D., & Crisp, R. J. (2012). “Treating” prejudice: An exposure-therapy approach to reducing negative reactions toward stigmatized groups. Psychological Science, 23, 1379–1386.PubMedCrossRefGoogle Scholar
  9. Bowles, H. R., & Gelfand, M. (2010). Status and the evaluation of workplace deviance. Psychological Science, 21, 49–54.PubMedCrossRefGoogle Scholar
  10. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.PubMedCrossRefGoogle Scholar
  11. Converse, B. A., & Fishbach, A. (2012). Instrumentality boosts appreciation: Helpers are more appreciated while they are useful. Psychological Science, 23, 560–566.PubMedCrossRefGoogle Scholar
  12. Converse, B. A., Risen, J. L., & Carter, T. J. (2012). Investing in karma: When wanting promotes helping. Psychological Science, 23, 923–930.PubMedCrossRefGoogle Scholar
  13. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29. doi: 10.1177/0956797613504966 PubMedCrossRefGoogle Scholar
  14. Damisch, L., Stoberock, B., & Mussweiler, T. (2010). Keep your fingers crossed! How superstition improves performance. Psychological Science, 21, 1014–1020.PubMedCrossRefGoogle Scholar
  15. de Hevia, M. D., & Spelke, E. S. (2010). Number-space mapping in human infants. Psychological Science, 21, 653–660.PubMedCrossRefPubMedCentralGoogle Scholar
  16. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in a meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 639–634.Google Scholar
  17. Eich, E. (2014). Business not as usual [Editorial]. Psychological Science, 25, 3–6. doi: 10.1177/0956797613512465 PubMedCrossRefGoogle Scholar
  18. Ersner-Hershfield, H., Galinsky, A. D., Kray, L. J., & King, B. G. (2010). Company, country, connections: Counterfactual origins increase organizational commitment, patriotism, and social investment. Psychological Science, 21, 1479–1486.PubMedCrossRefGoogle Scholar
  19. Evans, K. K., Horowitz, T. S., & Wolfe, J. M. (2011). When categories collide: Accumulation of information about multiple categories in rapid scene perception. Psychological Science, 22, 739–746. doi: 10.1177/0956797611407930 PubMedCrossRefPubMedCentralGoogle Scholar
  20. Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5, e10068. doi: 10.1371/journal.pone.0010068 PubMedCrossRefPubMedCentralGoogle Scholar
  21. Fast, N. J., & Chen, S. (2009). When the boss feels inadequate: Power, incompetence, and aggression. Psychological Science, 20, 1406–1413.PubMedCrossRefGoogle Scholar
  22. Fast, N. J., Gruenfeld, D. H., Sivanathan, N., & Galinsky, A. D. (2009). Illusory control: A generative force behind power’s far-reaching effects. Psychological Science, 20, 502–508.PubMedCrossRefGoogle Scholar
  23. Francis, G. (2012a). Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences, 109, E1587.CrossRefGoogle Scholar
  24. Francis, G. (2012b). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi: 10.1177/1745691612459520 CrossRefGoogle Scholar
  25. Francis, G. (2012c). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975–991. doi: 10.3758/s13423-012-0322-y CrossRefGoogle Scholar
  26. Francis, G. (2012d). The same old New Look: Publication bias in a study of wishful seeing. i-Perception, 3, 176–178.PubMedCrossRefPubMedCentralGoogle Scholar
  27. Francis, G. (2012e). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi: 10.3758/s13423-012-0227-9 CrossRefGoogle Scholar
  28. Francis, G. (2013a). Publication bias in “Red, rank, and romance in women viewing men” by Elliot et al. (2010). Journal of Experimental Psychology: General, 142, 292–296. doi: 10.1037/a0027923 CrossRefGoogle Scholar
  29. Francis, G. (2013b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153–169. doi: 10.1016/j.jmp.2013.02.003 CrossRefGoogle Scholar
  30. Francis, G. (2013c). We should focus on the biases that matter: A reply to commentaries. Journal of Mathematical Psychology, 57, 190–195. doi: 10.1016/j.jmp.2013.06.001 CrossRefGoogle Scholar
  31. Gao, T., McCarthy, G., & Scholl, B. J. (2010). The wolfpack effect: Perception of animacy irresistibly influences interactive behavior. Psychological Science, 21, 1845–1853.PubMedCrossRefGoogle Scholar
  32. Garcia, S. M., & Tor, A. (2009). The N-effect: More competitors, less competition. Psychological Science, 20, 871–877.PubMedCrossRefGoogle Scholar
  33. Gelman, A. (2013). Is it possible to be an ethicist without being mean to people? Chance, 26, 51–53.Google Scholar
  34. Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. In Technical report, Department of Statistics, Columbia University. Retrieved January 30, 2014, from www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
  35. González, J., & McLennan, C. T. (2009). Hemispheric differences in the recognition of environmental sounds. Psychological Science, 20, 887–894.PubMedCrossRefGoogle Scholar
  36. Hahn, U., Close, J., & Graf, M. (2009). Transformation direction influences shape-similarity judgments. Psychological Science, 20, 447–454.PubMedCrossRefGoogle Scholar
  37. Hart, W., & Albarracín, D. (2009). What I was doing versus what I did: Verb aspect influences memory and future actions. Psychological Science, 20, 238–244.PubMedCrossRefPubMedCentralGoogle Scholar
  38. Inesi, M. E., Botti, S., Dubois, D., Rucker, D. D., & Galinsky, A. D. (2011). Power and choice: Their dynamic interplay in quenching the thirst for personal control. Psychological Science, 22, 1042–1048.PubMedCrossRefGoogle Scholar
  39. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253.PubMedCrossRefGoogle Scholar
  40. Janssen, N., & Caramazza, A. (2009). Grammatical and phonological influences on word order. Psychological Science, 20, 1262–1268.PubMedCrossRefPubMedCentralGoogle Scholar
  41. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. doi: 10.1177/0956797611430953 PubMedCrossRefGoogle Scholar
  42. Johnson, V. E. (2013). On biases in assessing replicability, statistical consistency and publication bias. Journal of Mathematical Psychology, 57, 177–179. doi: 10.1016/j.jmp.2013.04.003 CrossRefGoogle Scholar
  43. Johnson, V., & Yuan, Y. (2007). Comments on “An exploratory test for an excess of significant findings” by JPA loannidis and TA Trikalinos. Clinical Trials, 4, 254–255. doi: 10.1177/1740774507079437. disc. 256–257.PubMedCrossRefGoogle Scholar
  44. Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. Psychological Science, 20, 1169–1174.PubMedCrossRefGoogle Scholar
  45. Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases. Psychological Science, 23, 661–668.PubMedCrossRefGoogle Scholar
  46. Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614. doi: 10.1177/1745691612462586 CrossRefGoogle Scholar
  47. Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658–676. doi: 10.1002/wcs.72 Google Scholar
  48. Labroo, A. A., Lambotte, S., & Zhang, Y. (2009). The “name-ease” effect and its dual impact on importance judgments. Psychological Science, 20, 1516–1522.PubMedCrossRefGoogle Scholar
  49. Lammers, J., Stapel, D. A., & Galinsky, A. D. (2010). Power increases hypocrisy: Moralizing in reasoning, immorality in behavior. Psychological Science, 21, 737–744.PubMedCrossRefGoogle Scholar
  50. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.CrossRefGoogle Scholar
  51. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432.CrossRefGoogle Scholar
  52. Leung, A. K., Kim, S., Polman, E., Ong, L. S., Qiu, L., Goncalo, J. A., & Sanchez-Burks, J. (2012). Embodied metaphors and creative “acts”. Psychological Science, 23, 502–509.PubMedCrossRefGoogle Scholar
  53. Li, X., Wei, L., & Soman, D. (2010). Sealing the emotions genie: The effects of physical enclosure on psychological closure. Psychological Science, 21, 1047–1050.PubMedCrossRefGoogle Scholar
  54. Maddux, W. W., Yang, H., Falk, C., Adam, H, Adair, W., Endo, Y., … Heine, S. J. (2010). For whom is parting with possessions more painful? Cultural differences in the endowment effect. Psychological Science, 21, 1910–1917.Google Scholar
  55. McGraw, A. P., & Warren, C. (2010). Benign violations: Making immoral behavior funny. Psychological Science, 21, 1141–1149. doi: 10.1177/0956797610376073 PubMedCrossRefGoogle Scholar
  56. Morey, R. D. (2013). The consistency test does not—and cannot—deliver what is advertised: A comment on Francis (2013). Journal of Mathematical Psychology, 57, 180–183. doi: 10.1016/j.jmp.2013.03.004 CrossRefGoogle Scholar
  57. Nordgren, L. F., Morris McDonnell, M.-H., & Loewenstein, G. (2011). What constitutes torture? Psychological impediments to an objective evaluation of enhanced interrogation tactics. Psychological Science, 22, 689–694.PubMedCrossRefGoogle Scholar
  58. Nordgren, L. F., van Harreveld, F., & van der Pligt, J. (2009). The restraint bias: How the illusion of self-restraint promotes impulsive behavior. Psychological Science, 20, 1523–1528.PubMedCrossRefGoogle Scholar
  59. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi: 10.1177/1745691612459058 CrossRefGoogle Scholar
  60. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.Google Scholar
  61. Roediger, H. L., III. (2012, February). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25(2). Retrieved from www.psychologicalscience.org/index.php/publications/observer/2012/february-11-2012-observer-publications/psychology’s-woes-and-a-partial-cure-the-value-of-replication.html
  62. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi: 10.3758/PBR.16.2.225 CrossRefGoogle Scholar
  63. Rounding, K., Lee, A., Jacobson, J. A., & Ji, L.-J. (2012). Religion replenishes self-control. Psychological Science, 23, 635–642.PubMedCrossRefGoogle Scholar
  64. Sackett, A. M., Meyvis, T., Nelson, L. D., Converse, B. A., & Sackett, A. L. (2010). You’re having fun when time flies: The hedonic consequences of subjective time progression. Psychological Science, 21, 111–117.PubMedCrossRefGoogle Scholar
  65. Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? U.S. Americans are more likely than Indians to construe actions as choices. Psychological Science, 21, 391–398.PubMedCrossRefGoogle Scholar
  66. Savani, K., & Rattan, A. (2012). A choice mind-set increases the acceptance and maintenance of wealth inequality. Psychological Science, 23, 796–804.PubMedCrossRefGoogle Scholar
  67. Savani, K., Stephens, N. M., & Markus, H. R. (2011). The unanticipated interpersonal and societal consequences of choice: Victim blaming and reduced support for the public good. Psychological Science, 22, 795–802.PubMedCrossRefGoogle Scholar
  68. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566. doi: 10.1037/a0029487 PubMedCrossRefGoogle Scholar
  69. Senay, I., Albarracín, D., & Noguchi, K. (2010). Motivating goal-directed behavior through introspective self-talk: The role of the interrogative form of simple future tense. Psychological Science, 21, 499–504.PubMedCrossRefPubMedCentralGoogle Scholar
  70. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi: 10.1177/0956797611417632 PubMedCrossRefGoogle Scholar
  71. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597–599. doi: 10.1177/1745691612463399 CrossRefGoogle Scholar
  72. Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57, 174–176. doi: 10.1016/j.jmp.2013.03.006 CrossRefGoogle Scholar
  73. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (in press). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General.Google Scholar
  74. Sterling, T. D. (1959). Publication decisions and the possible effects on inferences drawn from test of significance—or vice versa. Journal of the American Statistical Association, 54, 30–34.Google Scholar
  75. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.Google Scholar
  76. Todd, A. R., Hanko, K., Galinsky, A. D., & Mussweiler, T. (2011). When focusing on differences leads to similar perspectives. Psychological Science, 22, 134–141.PubMedCrossRefGoogle Scholar
  77. Tuk, M. A., Trampe, D., & Warlop, L. (2011). Inhibitory spillover: Increased urination urgency facilitates impulse control in unrelated domains. Psychological Science, 22, 627–633.PubMedCrossRefGoogle Scholar
  78. van Boxtel, J. J. A., & Koch, C. (2012). Visual rivalry without spatial conflict. Psychological Science, 23, 410–418.PubMedCrossRefGoogle Scholar
  79. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi: 10.3758/BF03194105 CrossRefGoogle Scholar
  80. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi: 10.1177/1745691612463078 CrossRefGoogle Scholar
  81. Wakslak, C., & Trope, Y. (2009). The effect of construal level on subjective probability estimates. Psychological Science, 20, 52–58.PubMedCrossRefPubMedCentralGoogle Scholar
  82. West, G. L., Anderson, A. K., Bedwell, J. S., & Pratt, J. (2010). Red diffuse light suppresses the accelerated perception of fear. Psychological Science, 21, 992–999. doi: 10.1177/0956797610371966 PubMedCrossRefGoogle Scholar
  83. Wolfe, J. M. (2013). Registered reports and replications in Attention, Perception, & Psychophysics [Editorial]. Attention, Perception, & Psychophysics, 75, 781–783. doi: 10.3758/s13414-013-0502-5 CrossRefGoogle Scholar
  84. Zhou, X., Vohs, K. D., & Baumeister, R. F. (2009). The symbolic power of money: Reminders of money alter social distress and physical pain. Psychological Science, 20, 700–706. doi: 10.1111/j.1467-9280.2009.02353.x PubMedCrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  1. 1.Department of Psychological SciencesPurdue UniversityWest LafayetteUSA

Personalised recommendations