Skip to main content

The frequency of excess success for articles in Psychological Science

Abstract

Recent controversies have questioned the quality of scientific practice in the field of psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009 and 2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings have been suppressed, the experiments or analyses were improper, or the theory does not properly account for the data. In total, problems appeared for 82 % (36 out of 44) of the articles in Psychological Science that had four or more experiments and could be analyzed.

This is a preview of subscription content, access via your institution.

References

  1. Alter, A. L., & Oppenheimer, D. M. (2009). Suppressing secrecy through metacognitive ease: Cognitive fluency encourages self-disclosure. Psychological Science, 20, 1414–1420.

    PubMed  Article  Google Scholar 

  2. Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764–771.

    PubMed  Article  Google Scholar 

  3. Ashton-James, C. E., Maddux, W. W., Galinsky, A. D., & Chartrand, T. L. (2009). Who I am depends on how I feel: The role of affect in the expression of culture. Psychological Science, 20, 340–346.

    PubMed  Article  Google Scholar 

  4. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060

    Article  Google Scholar 

  5. Balcetis, E., & Dunning, D. (2010). Wishful seeing: More desired objects are seen as closer. Psychological Science, 21, 147–152.

    PubMed  Article  Google Scholar 

  6. Bauer, M. A., Wilkie, J. E. B., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing consumerism: Situational materialism undermines personal and social well-being. Psychological Science, 23, 517–523.

    PubMed  Article  Google Scholar 

  7. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.

    PubMed  Article  Google Scholar 

  8. Birtel, M. D., & Crisp, R. J. (2012). “Treating” prejudice: An exposure-therapy approach to reducing negative reactions toward stigmatized groups. Psychological Science, 23, 1379–1386.

    PubMed  Article  Google Scholar 

  9. Bowles, H. R., & Gelfand, M. (2010). Status and the evaluation of workplace deviance. Psychological Science, 21, 49–54.

    PubMed  Article  Google Scholar 

  10. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.

    PubMed  Article  Google Scholar 

  11. Converse, B. A., & Fishbach, A. (2012). Instrumentality boosts appreciation: Helpers are more appreciated while they are useful. Psychological Science, 23, 560–566.

    PubMed  Article  Google Scholar 

  12. Converse, B. A., Risen, J. L., & Carter, T. J. (2012). Investing in karma: When wanting promotes helping. Psychological Science, 23, 923–930.

    PubMed  Article  Google Scholar 

  13. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29. doi:10.1177/0956797613504966

    PubMed  Article  Google Scholar 

  14. Damisch, L., Stoberock, B., & Mussweiler, T. (2010). Keep your fingers crossed! How superstition improves performance. Psychological Science, 21, 1014–1020.

    PubMed  Article  Google Scholar 

  15. de Hevia, M. D., & Spelke, E. S. (2010). Number-space mapping in human infants. Psychological Science, 21, 653–660.

    PubMed  Article  PubMed Central  Google Scholar 

  16. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in a meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 639–634.

    Google Scholar 

  17. Eich, E. (2014). Business not as usual [Editorial]. Psychological Science, 25, 3–6. doi:10.1177/0956797613512465

    PubMed  Article  Google Scholar 

  18. Ersner-Hershfield, H., Galinsky, A. D., Kray, L. J., & King, B. G. (2010). Company, country, connections: Counterfactual origins increase organizational commitment, patriotism, and social investment. Psychological Science, 21, 1479–1486.

    PubMed  Article  Google Scholar 

  19. Evans, K. K., Horowitz, T. S., & Wolfe, J. M. (2011). When categories collide: Accumulation of information about multiple categories in rapid scene perception. Psychological Science, 22, 739–746. doi:10.1177/0956797611407930

    PubMed  Article  PubMed Central  Google Scholar 

  20. Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5, e10068. doi:10.1371/journal.pone.0010068

    PubMed  Article  PubMed Central  Google Scholar 

  21. Fast, N. J., & Chen, S. (2009). When the boss feels inadequate: Power, incompetence, and aggression. Psychological Science, 20, 1406–1413.

    PubMed  Article  Google Scholar 

  22. Fast, N. J., Gruenfeld, D. H., Sivanathan, N., & Galinsky, A. D. (2009). Illusory control: A generative force behind power’s far-reaching effects. Psychological Science, 20, 502–508.

    PubMed  Article  Google Scholar 

  23. Francis, G. (2012a). Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences, 109, E1587.

    Article  Google Scholar 

  24. Francis, G. (2012b). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi:10.1177/1745691612459520

    Article  Google Scholar 

  25. Francis, G. (2012c). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975–991. doi:10.3758/s13423-012-0322-y

    Article  Google Scholar 

  26. Francis, G. (2012d). The same old New Look: Publication bias in a study of wishful seeing. i-Perception, 3, 176–178.

    PubMed  Article  PubMed Central  Google Scholar 

  27. Francis, G. (2012e). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi:10.3758/s13423-012-0227-9

    Article  Google Scholar 

  28. Francis, G. (2013a). Publication bias in “Red, rank, and romance in women viewing men” by Elliot et al. (2010). Journal of Experimental Psychology: General, 142, 292–296. doi:10.1037/a0027923

    Article  Google Scholar 

  29. Francis, G. (2013b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153–169. doi:10.1016/j.jmp.2013.02.003

    Article  Google Scholar 

  30. Francis, G. (2013c). We should focus on the biases that matter: A reply to commentaries. Journal of Mathematical Psychology, 57, 190–195. doi:10.1016/j.jmp.2013.06.001

    Article  Google Scholar 

  31. Gao, T., McCarthy, G., & Scholl, B. J. (2010). The wolfpack effect: Perception of animacy irresistibly influences interactive behavior. Psychological Science, 21, 1845–1853.

    PubMed  Article  Google Scholar 

  32. Garcia, S. M., & Tor, A. (2009). The N-effect: More competitors, less competition. Psychological Science, 20, 871–877.

    PubMed  Article  Google Scholar 

  33. Gelman, A. (2013). Is it possible to be an ethicist without being mean to people? Chance, 26, 51–53.

    Google Scholar 

  34. Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. In Technical report, Department of Statistics, Columbia University. Retrieved January 30, 2014, from www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

  35. González, J., & McLennan, C. T. (2009). Hemispheric differences in the recognition of environmental sounds. Psychological Science, 20, 887–894.

    PubMed  Article  Google Scholar 

  36. Hahn, U., Close, J., & Graf, M. (2009). Transformation direction influences shape-similarity judgments. Psychological Science, 20, 447–454.

    PubMed  Article  Google Scholar 

  37. Hart, W., & Albarracín, D. (2009). What I was doing versus what I did: Verb aspect influences memory and future actions. Psychological Science, 20, 238–244.

    PubMed  Article  PubMed Central  Google Scholar 

  38. Inesi, M. E., Botti, S., Dubois, D., Rucker, D. D., & Galinsky, A. D. (2011). Power and choice: Their dynamic interplay in quenching the thirst for personal control. Psychological Science, 22, 1042–1048.

    PubMed  Article  Google Scholar 

  39. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253.

    PubMed  Article  Google Scholar 

  40. Janssen, N., & Caramazza, A. (2009). Grammatical and phonological influences on word order. Psychological Science, 20, 1262–1268.

    PubMed  Article  PubMed Central  Google Scholar 

  41. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953

    PubMed  Article  Google Scholar 

  42. Johnson, V. E. (2013). On biases in assessing replicability, statistical consistency and publication bias. Journal of Mathematical Psychology, 57, 177–179. doi:10.1016/j.jmp.2013.04.003

    Article  Google Scholar 

  43. Johnson, V., & Yuan, Y. (2007). Comments on “An exploratory test for an excess of significant findings” by JPA loannidis and TA Trikalinos. Clinical Trials, 4, 254–255. doi:10.1177/1740774507079437. disc. 256–257.

    PubMed  Article  Google Scholar 

  44. Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. Psychological Science, 20, 1169–1174.

    PubMed  Article  Google Scholar 

  45. Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases. Psychological Science, 23, 661–668.

    PubMed  Article  Google Scholar 

  46. Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614. doi:10.1177/1745691612462586

    Article  Google Scholar 

  47. Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658–676. doi:10.1002/wcs.72

    Google Scholar 

  48. Labroo, A. A., Lambotte, S., & Zhang, Y. (2009). The “name-ease” effect and its dual impact on importance judgments. Psychological Science, 20, 1516–1522.

    PubMed  Article  Google Scholar 

  49. Lammers, J., Stapel, D. A., & Galinsky, A. D. (2010). Power increases hypocrisy: Moralizing in reasoning, immorality in behavior. Psychological Science, 21, 737–744.

    PubMed  Article  Google Scholar 

  50. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.

    Article  Google Scholar 

  51. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432.

    Article  Google Scholar 

  52. Leung, A. K., Kim, S., Polman, E., Ong, L. S., Qiu, L., Goncalo, J. A., & Sanchez-Burks, J. (2012). Embodied metaphors and creative “acts”. Psychological Science, 23, 502–509.

    PubMed  Article  Google Scholar 

  53. Li, X., Wei, L., & Soman, D. (2010). Sealing the emotions genie: The effects of physical enclosure on psychological closure. Psychological Science, 21, 1047–1050.

    PubMed  Article  Google Scholar 

  54. Maddux, W. W., Yang, H., Falk, C., Adam, H, Adair, W., Endo, Y., … Heine, S. J. (2010). For whom is parting with possessions more painful? Cultural differences in the endowment effect. Psychological Science, 21, 1910–1917.

  55. McGraw, A. P., & Warren, C. (2010). Benign violations: Making immoral behavior funny. Psychological Science, 21, 1141–1149. doi:10.1177/0956797610376073

    PubMed  Article  Google Scholar 

  56. Morey, R. D. (2013). The consistency test does not—and cannot—deliver what is advertised: A comment on Francis (2013). Journal of Mathematical Psychology, 57, 180–183. doi:10.1016/j.jmp.2013.03.004

    Article  Google Scholar 

  57. Nordgren, L. F., Morris McDonnell, M.-H., & Loewenstein, G. (2011). What constitutes torture? Psychological impediments to an objective evaluation of enhanced interrogation tactics. Psychological Science, 22, 689–694.

    PubMed  Article  Google Scholar 

  58. Nordgren, L. F., van Harreveld, F., & van der Pligt, J. (2009). The restraint bias: How the illusion of self-restraint promotes impulsive behavior. Psychological Science, 20, 1523–1528.

    PubMed  Article  Google Scholar 

  59. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058

    Article  Google Scholar 

  60. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.

    Google Scholar 

  61. Roediger, H. L., III. (2012, February). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25(2). Retrieved from www.psychologicalscience.org/index.php/publications/observer/2012/february-11-2012-observer-publications/psychology’s-woes-and-a-partial-cure-the-value-of-replication.html

  62. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225

    Article  Google Scholar 

  63. Rounding, K., Lee, A., Jacobson, J. A., & Ji, L.-J. (2012). Religion replenishes self-control. Psychological Science, 23, 635–642.

    PubMed  Article  Google Scholar 

  64. Sackett, A. M., Meyvis, T., Nelson, L. D., Converse, B. A., & Sackett, A. L. (2010). You’re having fun when time flies: The hedonic consequences of subjective time progression. Psychological Science, 21, 111–117.

    PubMed  Article  Google Scholar 

  65. Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? U.S. Americans are more likely than Indians to construe actions as choices. Psychological Science, 21, 391–398.

    PubMed  Article  Google Scholar 

  66. Savani, K., & Rattan, A. (2012). A choice mind-set increases the acceptance and maintenance of wealth inequality. Psychological Science, 23, 796–804.

    PubMed  Article  Google Scholar 

  67. Savani, K., Stephens, N. M., & Markus, H. R. (2011). The unanticipated interpersonal and societal consequences of choice: Victim blaming and reduced support for the public good. Psychological Science, 22, 795–802.

    PubMed  Article  Google Scholar 

  68. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566. doi:10.1037/a0029487

    PubMed  Article  Google Scholar 

  69. Senay, I., Albarracín, D., & Noguchi, K. (2010). Motivating goal-directed behavior through introspective self-talk: The role of the interrogative form of simple future tense. Psychological Science, 21, 499–504.

    PubMed  Article  PubMed Central  Google Scholar 

  70. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632

    PubMed  Article  Google Scholar 

  71. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597–599. doi:10.1177/1745691612463399

    Article  Google Scholar 

  72. Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57, 174–176. doi:10.1016/j.jmp.2013.03.006

    Article  Google Scholar 

  73. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (in press). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General.

  74. Sterling, T. D. (1959). Publication decisions and the possible effects on inferences drawn from test of significance—or vice versa. Journal of the American Statistical Association, 54, 30–34.

    Google Scholar 

  75. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.

    Google Scholar 

  76. Todd, A. R., Hanko, K., Galinsky, A. D., & Mussweiler, T. (2011). When focusing on differences leads to similar perspectives. Psychological Science, 22, 134–141.

    PubMed  Article  Google Scholar 

  77. Tuk, M. A., Trampe, D., & Warlop, L. (2011). Inhibitory spillover: Increased urination urgency facilitates impulse control in unrelated domains. Psychological Science, 22, 627–633.

    PubMed  Article  Google Scholar 

  78. van Boxtel, J. J. A., & Koch, C. (2012). Visual rivalry without spatial conflict. Psychological Science, 23, 410–418.

    PubMed  Article  Google Scholar 

  79. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105

    Article  Google Scholar 

  80. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078

    Article  Google Scholar 

  81. Wakslak, C., & Trope, Y. (2009). The effect of construal level on subjective probability estimates. Psychological Science, 20, 52–58.

    PubMed  Article  PubMed Central  Google Scholar 

  82. West, G. L., Anderson, A. K., Bedwell, J. S., & Pratt, J. (2010). Red diffuse light suppresses the accelerated perception of fear. Psychological Science, 21, 992–999. doi:10.1177/0956797610371966

    PubMed  Article  Google Scholar 

  83. Wolfe, J. M. (2013). Registered reports and replications in Attention, Perception, & Psychophysics [Editorial]. Attention, Perception, & Psychophysics, 75, 781–783. doi:10.3758/s13414-013-0502-5

    Article  Google Scholar 

  84. Zhou, X., Vohs, K. D., & Baumeister, R. F. (2009). The symbolic power of money: Reminders of money alter social distress and physical pain. Psychological Science, 20, 700–706. doi:10.1111/j.1467-9280.2009.02353.x

    PubMed  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Gregory Francis.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(ZIP 2.06 mb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Francis, G. The frequency of excess success for articles in Psychological Science . Psychon Bull Rev 21, 1180–1187 (2014). https://doi.org/10.3758/s13423-014-0601-x

Download citation

Keywords

  • Statistical inference
  • Statistics
  • Probabilistic reasoning