Abstract
During the last decade, confidence in many social sciences, including consumer research, has been undermined by doubts about the replicability of empirical research findings. These doubts have led to increased calls to improve research practices and adopt new measures to increase the replicability of published work from various stakeholders such as funding agencies, journals, and scholars themselves. Despite these demands, it is unclear to which the research published in the leading consumer research journals has adhered to these calls for change. This article provides the first systematic empirical analysis of this question by surveying three crucial statistics of published consumer research over time: sample sizes, effect sizes, and the distribution of published p values. The authors compile a hand-coded sample of N = 258 articles published between 2008 and 2020 in the Journal of Consumer Psychology, the Journal of Consumer Research, and the Journal of Marketing Research. An automated text analysis across all publications in these three journals corroborates the representativeness of the hand-coded sample. Results reveal a substantial increase in sample sizes above and beyond the use of online samples along with a decrease in reported effect sizes. Effect and samples sizes are highly correlated which at least partially explains the reduction in reported effect sizes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
All data and code to reproduce the analyses are available online at https://osf.io/cunfb/.
References
Alfaro, M. E., & Holder, M. (2015). The posterior and the prior in Bayesian phylogenetics. Annual Review of Ecology, Evolution, and Systematics, 37(2006), 19–42. https://doi.org/10.2307/annurev.ecolsys.37.091305.30000003
Anvari, F., & Lakens, D. (2021). Using anchor-based methods to determine the smallest effect size of interest. Journal of Experimental Social Psychology, 96(April). https://doi.org/10.1016/j.jesp.2021.104159.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060
Camerer, C. F., Dreber, A., Forsell, E., Ho, T., Huber, J., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139. https://doi.org/10.1177/0956797615585115
Clinton, J. D. (2012). Using roll call estimates to test models of politics. Annual Review of Political Science, 15(1), 79–99. https://doi.org/10.1146/annurev-polisci-043010-095836Dahl
Dubois, P., Albuquerque, P., Allais, O., Bonnet, C., Bertail, P., Combris, P., Lahlou, S., Rigal, N., Ruffieux, B., & Chandon, P. (2021). Effects of front-of-pack labels on the nutritional quality of supermarket food purchases: Evidence from a large-scale randomized controlled trial. Journal of the Academy of Marketing Science, 49(1), 119–138. https://doi.org/10.1007/s11747-020-00723-5
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341X.2000.00455.x
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H., Capaldi, C. A., Chartier, C. R., Chung, J. M., Cicero, D. C., Coleman, J. A., Conway, J. G., … Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (2015). Bias in meta analysis detected by a simple, graphical test. BMJ, 14(7109), 1–16. https://doi.org/10.1136/bmj.315.7109.629
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5). https://doi.org/10.1371/journal.pone.0005738.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120–128. https://doi.org/10.1037/a0024445
Freese, J., & Peterson, D. (2017). Replication in Social Science. Annual Review of Sociology, 43(1), 147–165. https://doi.org/10.1146/annurev-soc-060116-053450
Gelman, A., & Loken, E. (2014). The statistical Crisis in science. American Scientist, 102(6), 460–465. https://doi.org/10.1511/2014.111.460
Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210. https://doi.org/10.1093/jcr/ucx047
Greenwald, A. G., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996). Effect sizes and p-values: What should be reported and what should be replicated? Psychophysiology, 33, 175–183.
Hoekstra, R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science. Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01203-8
Inman, J. J. (2012). ACR presidential address: The elephant not in the room: The Need for useful, actionable insights in behavioral research. Association of Consumer Research. https://www.acrwebsite.org/assets/pdfs/2012presadd.pdf. Accessed 16 Dec 2022.
Inman, J. J., Campbell, M. C., Kirmani, A., & Price, L. L. (2018). Our vision for the Journal of Consumer Research: It’s all about the consumer. Journal of Consumer Research, 44(5), 955–959. https://doi.org/10.1093/jcr/ucx123
Ioannidis, J. P. A. (2014). How to make more published research true. PLoS Medicine, 11(10), e1001747. https://doi.org/10.1371/journal.pmed.1001747
Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., … Nosek, B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152.https://doi.org/10.1027/1864-9335/a000178.
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., … Nosek, B. A. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225.
Kollat, D. T., Engel, J. F., & Blackwell, R. D. (1970). Current problems in consumer behavior research. Journal of Marketing Research, 7(3), 327–332. https://doi.org/10.2307/3150290
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4(NOV), 1–12. https://doi.org/10.3389/fpsyg.2013.00863.
Lakens, D. (2021). Sample size justification. PsyArXiv. https://doi.org/10.31234/osf.io/9d3yf.
Lakens, D., & Etz, A. J. (2017). Too true to be bad: When sets of studies with significant and nonsignificant findings are probably true. Social Psychological and Personality Science, 8(8), 875–881. https://doi.org/10.1177/1948550617693058
Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. https://doi.org/10.1177/2515245918770963
Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31(2), 107–112. https://doi.org/10.1111/j.2044-8317.1978.tb00578.x
Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9(2), 147–163. https://doi.org/10.1037/1082-989X.9.2.147
Meyvis, T., & van Osselaer, S. M. J. (2018). Increasing the power of your study by increasing the effect size. Journal of Consumer Research, 44(5), 1157–1173. https://doi.org/10.1093/jcr/ucx110
Nosek, B. A. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425.
Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716.
Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530. https://doi.org/10.1177/1745691612465253
Pechmann, C. (2014). Editorial regarding the new submission guidelines at the Journal of Consumer Psychology. Journal of Consumer Psychology, 24(1), 1–3. https://doi.org/10.1016/j.jcps.2013.10.002
Pham, M. T. (2013). The seven sins of consumer psychology. Journal of Consumer Psychology, 23(4), 411–423. https://doi.org/10.1016/j.jcps.2013.07.004
Pham, M. T., & Oh, T. T. (2020). On not confusing the tree of trustworthy statistics with the greater forest of good science: A comment on Simmons et al.’s perspective on preregistration. Journal of Consumer Psychology, jcpy.1213. https://doi.org/10.1002/jcpy.1213.
Popper, K. (2002/1959). The logic of scientific discovery. Routledge.
Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 1–8. https://doi.org/10.1177/2515245919838781.
Scheibehenne, B., Jamil, T., & Wagenmakers, E. J. (2016). Bayesian evidence synthesis can reconcile seemingly inconsistent results: The case of hotel towel reuse. Psychological Science, 27(7), 1043–1046. https://doi.org/10.1177/0956797616644081
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173–1181. https://doi.org/10.1037/0003-066X.47.10.1173
Sheth, J. N. (1982). Consumer behavior: Surpluses & shortages. Advances in Consumer Research, 9(1), 13–16.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Simmons, J., Nelson, L., & Simonsohn, U. (2020). Pre‐registration is a game changer. But, like random assignment, it is neither necessary nor sufficient for credible science. Journal of Consumer Psychology, 53(9), jcpy.1207. https://doi.org/10.1002/jcpy.1207.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5). https://journal.sjdm.org/14/14725/jdm14725.pdf
Stürmer, S., Oeberst, A., Trötschel, R., & Decker, O. (2017). Early-career researchers’ perceptions of the prevalence of questionable research practices, potential causes, and open science. Social Psychology, 48(6), 365–371. https://doi.org/10.1027/1864-9335/a000324
Wells, W. (1993). Discovery-oriented consumer research. Journal of Consumer Researcch, 19(4), 489–504. https://doi.org/10.1007/sl0869-007-9037-x
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6(11), 1–7. https://doi.org/10.1371/journal.pone.0026828
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid P-hacking. Frontiers in Psychology, 7(NOV), 1–12. https://doi.org/10.3389/fpsyg.2016.01832.
Funding
The authors gratefully acknowledge funding from the Swiss National Science Foundation (Project No 172806).
Author information
Authors and Affiliations
Contributions
AKS and BS designed the research and wrote the manuscript, and AKS conducted the analyses. We would like to thank Franziska Wollenscheid, Cecilia Galvan, and Victor Mitchell for their help with coding the data.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Krefeld-Schwalb, A., Scheibehenne, B. Tighter nets for smaller fishes? Mapping the development of statistical practices in consumer research between 2008 and 2020. Mark Lett 34, 351–365 (2023). https://doi.org/10.1007/s11002-022-09662-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11002-022-09662-3