Recent controversies have questioned the quality of scientific practice in the field of psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009 and 2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings have been suppressed, the experiments or analyses were improper, or the theory does not properly account for the data. In total, problems appeared for 82 % (36 out of 44) of the articles in Psychological Science that had four or more experiments and could be analyzed.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Alter, A. L., & Oppenheimer, D. M. (2009). Suppressing secrecy through metacognitive ease: Cognitive fluency encourages self-disclosure. Psychological Science, 20, 1414–1420.
Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764–771.
Ashton-James, C. E., Maddux, W. W., Galinsky, A. D., & Chartrand, T. L. (2009). Who I am depends on how I feel: The role of affect in the expression of culture. Psychological Science, 20, 340–346.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
Balcetis, E., & Dunning, D. (2010). Wishful seeing: More desired objects are seen as closer. Psychological Science, 21, 147–152.
Bauer, M. A., Wilkie, J. E. B., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing consumerism: Situational materialism undermines personal and social well-being. Psychological Science, 23, 517–523.
Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.
Birtel, M. D., & Crisp, R. J. (2012). “Treating” prejudice: An exposure-therapy approach to reducing negative reactions toward stigmatized groups. Psychological Science, 23, 1379–1386.
Bowles, H. R., & Gelfand, M. (2010). Status and the evaluation of workplace deviance. Psychological Science, 21, 49–54.
Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.
Converse, B. A., & Fishbach, A. (2012). Instrumentality boosts appreciation: Helpers are more appreciated while they are useful. Psychological Science, 23, 560–566.
Converse, B. A., Risen, J. L., & Carter, T. J. (2012). Investing in karma: When wanting promotes helping. Psychological Science, 23, 923–930.
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29. doi:10.1177/0956797613504966
Damisch, L., Stoberock, B., & Mussweiler, T. (2010). Keep your fingers crossed! How superstition improves performance. Psychological Science, 21, 1014–1020.
de Hevia, M. D., & Spelke, E. S. (2010). Number-space mapping in human infants. Psychological Science, 21, 653–660.
Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in a meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 639–634.
Eich, E. (2014). Business not as usual [Editorial]. Psychological Science, 25, 3–6. doi:10.1177/0956797613512465
Ersner-Hershfield, H., Galinsky, A. D., Kray, L. J., & King, B. G. (2010). Company, country, connections: Counterfactual origins increase organizational commitment, patriotism, and social investment. Psychological Science, 21, 1479–1486.
Evans, K. K., Horowitz, T. S., & Wolfe, J. M. (2011). When categories collide: Accumulation of information about multiple categories in rapid scene perception. Psychological Science, 22, 739–746. doi:10.1177/0956797611407930
Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5, e10068. doi:10.1371/journal.pone.0010068
Fast, N. J., & Chen, S. (2009). When the boss feels inadequate: Power, incompetence, and aggression. Psychological Science, 20, 1406–1413.
Fast, N. J., Gruenfeld, D. H., Sivanathan, N., & Galinsky, A. D. (2009). Illusory control: A generative force behind power’s far-reaching effects. Psychological Science, 20, 502–508.
Francis, G. (2012a). Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences, 109, E1587.
Francis, G. (2012b). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi:10.1177/1745691612459520
Francis, G. (2012c). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975–991. doi:10.3758/s13423-012-0322-y
Francis, G. (2012d). The same old New Look: Publication bias in a study of wishful seeing. i-Perception, 3, 176–178.
Francis, G. (2012e). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi:10.3758/s13423-012-0227-9
Francis, G. (2013a). Publication bias in “Red, rank, and romance in women viewing men” by Elliot et al. (2010). Journal of Experimental Psychology: General, 142, 292–296. doi:10.1037/a0027923
Francis, G. (2013b). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153–169. doi:10.1016/j.jmp.2013.02.003
Francis, G. (2013c). We should focus on the biases that matter: A reply to commentaries. Journal of Mathematical Psychology, 57, 190–195. doi:10.1016/j.jmp.2013.06.001
Gao, T., McCarthy, G., & Scholl, B. J. (2010). The wolfpack effect: Perception of animacy irresistibly influences interactive behavior. Psychological Science, 21, 1845–1853.
Garcia, S. M., & Tor, A. (2009). The N-effect: More competitors, less competition. Psychological Science, 20, 871–877.
Gelman, A. (2013). Is it possible to be an ethicist without being mean to people? Chance, 26, 51–53.
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. In Technical report, Department of Statistics, Columbia University. Retrieved January 30, 2014, from www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
González, J., & McLennan, C. T. (2009). Hemispheric differences in the recognition of environmental sounds. Psychological Science, 20, 887–894.
Hahn, U., Close, J., & Graf, M. (2009). Transformation direction influences shape-similarity judgments. Psychological Science, 20, 447–454.
Hart, W., & Albarracín, D. (2009). What I was doing versus what I did: Verb aspect influences memory and future actions. Psychological Science, 20, 238–244.
Inesi, M. E., Botti, S., Dubois, D., Rucker, D. D., & Galinsky, A. D. (2011). Power and choice: Their dynamic interplay in quenching the thirst for personal control. Psychological Science, 22, 1042–1048.
Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253.
Janssen, N., & Caramazza, A. (2009). Grammatical and phonological influences on word order. Psychological Science, 20, 1262–1268.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
Johnson, V. E. (2013). On biases in assessing replicability, statistical consistency and publication bias. Journal of Mathematical Psychology, 57, 177–179. doi:10.1016/j.jmp.2013.04.003
Johnson, V., & Yuan, Y. (2007). Comments on “An exploratory test for an excess of significant findings” by JPA loannidis and TA Trikalinos. Clinical Trials, 4, 254–255. doi:10.1177/1740774507079437. disc. 256–257.
Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. Psychological Science, 20, 1169–1174.
Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases. Psychological Science, 23, 661–668.
Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614. doi:10.1177/1745691612462586
Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658–676. doi:10.1002/wcs.72
Labroo, A. A., Lambotte, S., & Zhang, Y. (2009). The “name-ease” effect and its dual impact on importance judgments. Psychological Science, 20, 1516–1522.
Lammers, J., Stapel, D. A., & Galinsky, A. D. (2010). Power increases hypocrisy: Moralizing in reasoning, immorality in behavior. Psychological Science, 21, 737–744.
Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432.
Leung, A. K., Kim, S., Polman, E., Ong, L. S., Qiu, L., Goncalo, J. A., & Sanchez-Burks, J. (2012). Embodied metaphors and creative “acts”. Psychological Science, 23, 502–509.
Li, X., Wei, L., & Soman, D. (2010). Sealing the emotions genie: The effects of physical enclosure on psychological closure. Psychological Science, 21, 1047–1050.
Maddux, W. W., Yang, H., Falk, C., Adam, H, Adair, W., Endo, Y., … Heine, S. J. (2010). For whom is parting with possessions more painful? Cultural differences in the endowment effect. Psychological Science, 21, 1910–1917.
McGraw, A. P., & Warren, C. (2010). Benign violations: Making immoral behavior funny. Psychological Science, 21, 1141–1149. doi:10.1177/0956797610376073
Morey, R. D. (2013). The consistency test does not—and cannot—deliver what is advertised: A comment on Francis (2013). Journal of Mathematical Psychology, 57, 180–183. doi:10.1016/j.jmp.2013.03.004
Nordgren, L. F., Morris McDonnell, M.-H., & Loewenstein, G. (2011). What constitutes torture? Psychological impediments to an objective evaluation of enhanced interrogation tactics. Psychological Science, 22, 689–694.
Nordgren, L. F., van Harreveld, F., & van der Pligt, J. (2009). The restraint bias: How the illusion of self-restraint promotes impulsive behavior. Psychological Science, 20, 1523–1528.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058
Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.
Roediger, H. L., III. (2012, February). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25(2). Retrieved from www.psychologicalscience.org/index.php/publications/observer/2012/february-11-2012-observer-publications/psychology’s-woes-and-a-partial-cure-the-value-of-replication.html
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225
Rounding, K., Lee, A., Jacobson, J. A., & Ji, L.-J. (2012). Religion replenishes self-control. Psychological Science, 23, 635–642.
Sackett, A. M., Meyvis, T., Nelson, L. D., Converse, B. A., & Sackett, A. L. (2010). You’re having fun when time flies: The hedonic consequences of subjective time progression. Psychological Science, 21, 111–117.
Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? U.S. Americans are more likely than Indians to construe actions as choices. Psychological Science, 21, 391–398.
Savani, K., & Rattan, A. (2012). A choice mind-set increases the acceptance and maintenance of wealth inequality. Psychological Science, 23, 796–804.
Savani, K., Stephens, N. M., & Markus, H. R. (2011). The unanticipated interpersonal and societal consequences of choice: Victim blaming and reduced support for the public good. Psychological Science, 22, 795–802.
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566. doi:10.1037/a0029487
Senay, I., Albarracín, D., & Noguchi, K. (2010). Motivating goal-directed behavior through introspective self-talk: The role of the interrogative form of simple future tense. Psychological Science, 21, 499–504.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597–599. doi:10.1177/1745691612463399
Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57, 174–176. doi:10.1016/j.jmp.2013.03.006
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (in press). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General.
Sterling, T. D. (1959). Publication decisions and the possible effects on inferences drawn from test of significance—or vice versa. Journal of the American Statistical Association, 54, 30–34.
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.
Todd, A. R., Hanko, K., Galinsky, A. D., & Mussweiler, T. (2011). When focusing on differences leads to similar perspectives. Psychological Science, 22, 134–141.
Tuk, M. A., Trampe, D., & Warlop, L. (2011). Inhibitory spillover: Increased urination urgency facilitates impulse control in unrelated domains. Psychological Science, 22, 627–633.
van Boxtel, J. J. A., & Koch, C. (2012). Visual rivalry without spatial conflict. Psychological Science, 23, 410–418.
Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
Wakslak, C., & Trope, Y. (2009). The effect of construal level on subjective probability estimates. Psychological Science, 20, 52–58.
West, G. L., Anderson, A. K., Bedwell, J. S., & Pratt, J. (2010). Red diffuse light suppresses the accelerated perception of fear. Psychological Science, 21, 992–999. doi:10.1177/0956797610371966
Wolfe, J. M. (2013). Registered reports and replications in Attention, Perception, & Psychophysics [Editorial]. Attention, Perception, & Psychophysics, 75, 781–783. doi:10.3758/s13414-013-0502-5
Zhou, X., Vohs, K. D., & Baumeister, R. F. (2009). The symbolic power of money: Reminders of money alter social distress and physical pain. Psychological Science, 20, 700–706. doi:10.1111/j.1467-9280.2009.02353.x
Electronic supplementary material
Below is the link to the electronic supplementary material.
(ZIP 2.06 mb)
About this article
Cite this article
Francis, G. The frequency of excess success for articles in Psychological Science . Psychon Bull Rev 21, 1180–1187 (2014). https://doi.org/10.3758/s13423-014-0601-x
- Statistical inference
- Probabilistic reasoning