Getting Serious About the Assessment and Promotion of Replicable Sexual Science: A Commentary on Wisman and Shrira (2020) and Lorenz (2020)

To paraphrase a colleague’s observation on the reported effect sizes in Wisman and Shrira (2020): “If the scent is this effective, someone would make a lot of money bottling and selling this stuff!”
One reviewer suggested perhaps these unbelievable effect sizes could be understood as the result of strict laboratory study of the phenomenon, resulting in a real effect with an estimated size that would not be expected to generalize to more “real world” circumstances. Although on its own the Piranha Problem might be resolvable with this kind of explanation, the consistency of outcomes from my remaining evidential value tests lead me to think it is much more likely there isn’t a credible effect to begin with (i.e., a Type I error/false positive).
It should be noted that if much ado is to be made of this test being “nearly significant” this meta-analyzed estimate of d is 1.75x to 2.95x smaller than the d’s reported in Wisman and Shrira’s (2020) Table 1.
The probability (P) that H₀ is true given (|) our data, and the probability that H₁ is true given our data, respectively.
The probability of observing data as—or more—extreme as ours, given that H₀ is true.
Lorenz served as a reviewer for this Commentary, and I sincerely appreciate her constructive engagement despite the critical tone of my commentary.
Archives of Sexual Behavior currently has a “Research Data Policy,” in which authors are encouraged to deposit their data into a public repository. However, there is no meta-data/infrastructure through the submission portal for tracking and sharing this repository (only a drop-down question asking whether it has been deposited somewhere) or other transparency-enhancing documentation (e.g., repository of research materials, preregistration, preprints, etc.,) and required disclosures (e.g., sample size determination, description of measured variables). See OSF for an example of what such a submission portal looks like elsewhere.

References

Aiken, L. S., West, S. G., Sechrest, L., Reno, R. R., Roediger, H. L., III, Scarr, S., … Sherman, S. J. (1990). Graduate training in statistics, methodology, and measurement in psychology: A survey of PhD programs in North America. American Psychologist, 45(6), 721–734. https://doi.org/10.1037/0003-066X.45.6.721.
Article Google Scholar
Albarracín, D., Johnson, B. T., Fishbein, M., & Muellerleile, P. A. (2001). Theories of reasoned action and planned behavior as models of condom use: A meta-analysis. Psychological Bulletin, 127, 142–161. https://doi.org/10.1037/0033-2909.127.1.142.
Article Google Scholar
Bakker, M., Hartgerink, C. H., Wicherts, J. M., & van der Maas, H. L. (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27(8), 1069–1077. https://doi.org/10.1177/0956797616647519.
Article Google Scholar
Bastian, H. (2017a, August 29). Bias in open science advocacy: The case of article badges for data sharing. Retrieved from https://blogs.plos.org/absolutely-maybe.
Bastian, H. (2017b, September 1). What’s open, what’s data? What’s proof, what’s spin? Retrieved from https://blogs.plos.org/absolutely-maybe.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. West Sussex, England: Wiley.
Google Scholar
Brown, N. J., & Heathers, J. A. (2017). The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science, 8(4), 363–369. https://doi.org/10.1177/1948550616673876.
Article Google Scholar
Center for Open Science. (2020, May 5). TOP factor to appear in master journal list. Retrieved from https://www.cos.io/about/news/cos-and-the-web-of-science-collaborate-to-bring-top-factor-to-master-journal-list.
Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered reports: Realigning incentives in scientific publishing. Cortex, 66, A1–A2. https://doi.org/10.1016/j.cortex.2015.03.022.
Article Google Scholar
Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145–153. https://doi.org/10.1037/h0045186.
Article Google Scholar
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. https://doi.org/10.1037/0033-2909.112.1.155.
Article Google Scholar
Conley, T. D., Moors, A. C., Matsick, J. L., Ziegler, A., & Valentine, B. A. (2011). Women, men, and the bedroom: Methodological and conceptual insights that narrow, reframe, and eliminate gender differences in sexuality. Current Directions in Psychological Science, 20, 296–300. https://doi.org/10.1177/0963721411418467.
Article Google Scholar
Cowan, N., Belletier, C., Doherty, J. M., Jaroslawska, A. J., Rhodes, S., Forsberg, A., … Logie, R. H. (2020). How do scientific views change? Perspectives on psychological science. Notes from an Extended Adversarial Collaboration. https://doi.org/10.1177/1745691620906415.
Article Google Scholar
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966.
Article Google Scholar
Devezer, D., Navarro, D., Vandekerckhove, J., & Buzbas, E. O. (2020). The case for formal methodology in scientific reform. https://www.biorxiv.org/content/10.1101/2020.04.26.048306v1.
DeVore, J. (2020, March 9). Men can smell concupiscence in women. Retrieved from https://medium.com/humungus/men-can-smell-horniness-in-women-62dfd038017e.
Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. Basingstoke, England: Palgrave Macmillan.
Google Scholar
Eich, E. (2014). Business not as usual. Psychological Science, 25(1), 3–6. https://doi.org/10.1177/0956797613512465.
Article Google Scholar
Finkel, E. J. (2016). Reflections on the commitment–forgiveness registered replication report. Perspectives on Psychological Science, 11(5), 765–767. https://doi.org/10.1177/1745691616664695.
Article Google Scholar
Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE, 9(10), e109019. https://doi.org/10.1371/journal.pone.0109019.
Article Google Scholar
Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19(2), 151–156. https://doi.org/10.3758/s13423-012-0227-9.
Article Google Scholar
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202.
Article Google Scholar
Gelman, A. (2015, December 15). The piranha problem in social psychology/behavioral economics: The “take a pill” model of science eats itself. Retrieved from https://statmodeling.stat.columbia.edu.
Goh, J. X., Hall, J. A., & Rosenthal, R. (2016). Mini meta-analysis of your own studies: Some arguments on why and a primer on how. Social and Personality Psychology Compass, 10(10), 535–549. https://doi.org/10.1111/spc3.12267.
Article Google Scholar
Haselton, M. G. (2003). The sexual overperception bias: Evidence of a systematic bias in men from a survey of naturally occurring events. Journal of Research in Personality, 37(1), 34–47. https://doi.org/10.1016/S0092-6566(02)00529-9.
Article Google Scholar
Heathers, J. (2017, March 3). Introducing SPRITE (and the case of the carthorse child). Retrieved from https://medium.com/hackernoon/introducing-sprite-and-the-case-of-the-carthorse-child-58683c2bfeb.
Heathers, J. A., Anaya, J., vanderZee, T., & Brown, N. J. (2018). Recovering data from summary statistics: Sample parameter reconstruction via iterative techniques (SPRITE). PeerJ Preprints, 6, e26968v1. https://doi.org/10.7287/peerj.preprints.26968v1.
Article Google Scholar
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124.
Article Google Scholar
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953.
Article Google Scholar
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., & Errington, T. M. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456.
Article Google Scholar
King, K. M., Pullmann, M. D., Lyon, A. R., Dorsey, S., & Lewis, C. C. (2019). Using implementation science to close the gap between the optimal and typical practice of quantitative methods in clinical science. Journal of Abnormal Psychology, 128(6), 547–562. https://doi.org/10.1037/abn0000417.
Article Google Scholar
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., & Batra, R. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225.
Article Google Scholar
Lakens, D. (2015, March 20). How a p value between 0.04-0.05 equals a p value between 0.16-017. Retrieved from http://daniellakens.blogspot.com/2015/03/how-p-value-between-004-005-equals-p.html.
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences, 115(45), E10516–E10517. https://doi.org/10.1073/pnas.1812592115.
Article Google Scholar
Lorenz, T. K. (2020). Reproducibility and registration in sexuality research [Guest Editorial]. Archives of Sexual Behavior, 49(2), 367–372. https://doi.org/10.1007/s10508-020-01650-w.
Article Google Scholar
Ly, A., Raj, A., Etz, A., Marsman, M., Gronau, Q. F., & Wagenmakers, E. J. (2018). Bayesian reanalyses from summary statistics: A guide for academic consumers. Advances in Methods and Practices in Psychological Science, 1(3), 367–374. https://doi.org/10.1177/2515245918779348.
Article Google Scholar
Magnusson, K. (2015). Distribution of p-values when comparing two groups. Retrieved from https://rpsychologist.com/d3/pdist/.
Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9(2), 147–163. https://doi.org/10.1037/1082-989X.9.2.147.
Article Google Scholar
McCoach, D. B., & Adelson, J. L. (2010). Dealing with dependence (Part I): Understanding the effects of clustered data. Gifted Child Quarterly, 54(2), 152–155. https://doi.org/10.1177/0016986210363076.
Article Google Scholar
Miller, O. (2020, March 3). Men can smell when a woman is sexually aroused. Retrieved from https://medium.com/humungus/men-can-smell-horniness-in-women-62dfd038017e.
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018). Psychology’s renaissance. Annual Review of Psychology, 69, 511–534. https://doi.org/10.1146/annurev-psych-122216-011836.
Article Google Scholar
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., & Contestabile, M. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374.
Article Google Scholar
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Article Google Scholar
Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48(4), 1205–1226. https://doi.org/10.3758/s13428-015-0664-2.
Article Google Scholar
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Article Google Scholar
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science IS, 10, 21. https://doi.org/10.1186/s13012-015-0209-1.
Article Google Scholar
R Core Team. (2020). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
Google Scholar
Richard, F. D., Bond, C. F., Jr., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7(4), 331–363. https://doi.org/10.1037/1089-2680.7.4.331.
Article Google Scholar
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638.
Article Google Scholar
Sakaluk, J. K. (2016). Promoting replicable sexual science: A methodological review and call for metascience. Canadian Journal of Human Sexuality, 25(1), 1–8. https://doi.org/10.3138/cjhs.251-CO1.
Article Google Scholar
Sakaluk, J. K. (2019). Expanding statistical frontiers in sexual science: Taxometric, invariance, and equivalence testing. Journal of Sex Research, 56(4–5), 475–510. https://doi.org/10.1080/00224499.2019.1568377.
Article Google Scholar
Sakaluk, J. K., & Graham, C. A. (2018). Promoting transparent reporting of conflicts of interests and statistical analyses at the Journal of Sex Resarch [Editorial]. Journal of Sex Resarch, 55, 1–6. https://doi.org/10.1080/00224499.2017.1395387.
Article Google Scholar
Sakaluk, J. K., & Short, S. D. (2017). A methodological review of exploratory factor analysis in sexuality research: Used practices, best practices, and data analysis resources. Journal of Sex Research, 54(1), 1–9. https://doi.org/10.1080/00224499.2015.1137538.
Article Google Scholar
Sakaluk, J. K., Williams, A. J., Kilshaw, R. E., & Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500–509. https://doi.org/10.1037/abn0000421.
Article Google Scholar
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551–566. https://doi.org/10.1037/a0029487.
Article Google Scholar
Schimmack, U. (2014). Quantifying statistical research integrity: The Replicability-Index. Retrieved from http://www.r-index.org.
Schmitt, D. P., Jonason, P. K., Byerley, G. J., Flores, S. D., Illbeck, B. E., O’Leary, K. N., & Qudrat, A. (2012). A reexamination of sex differences in sexuality: New studies reveal old truths. Current Directions in Psychology, 21, 135–139. https://doi.org/10.1177/0963721412436808.
Article Google Scholar
Schönbrodt, F. D. (2018). p-checker: One-for-all p-value analyzer. Retrieved from http://shinyapps.org/apps/p-checker/.
Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47(5), 609–612. https://doi.org/10.1016/j.jrp.2013.05.009.
Article Google Scholar
Schönbrodt, F. D., Wagenmakers, E. J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods, 22(2), 322–339. https://doi.org/10.1037/met0000061.
Article Google Scholar
Seto, M. (2019). SA editorial about next phase of more open science. Sexual Abuse, 31, 371–373. https://doi.org/10.1177/1079063219838430.
Article Google Scholar
Simmons, J. (2014, April 4). MTurk vs. the lab: Either way we need big samples. Retrieved from http://datacolada.org/18.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632.
Article Google Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2018). False-positive citations. Perspectives on Psychological Science, 13(2), 255–259. https://doi.org/10.1177/1745691617698146.
Article Google Scholar
Simmons, J. P., & Simonsohn, U. (2017). Power posing: P-curving the evidence. Psychological Science, 28, 687–693. https://doi.org/10.1177/0956797616658563.
Article Google Scholar
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630.
Article Google Scholar
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242.
Article Google Scholar
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller. Journal of Experimental Psychology: General, 144(6), 1146–1152. https://doi.org/10.1037/xge0000104.
Article Google Scholar
Srivastava, S. (2012, Sept. 27). A pottery barn rule for scientific journals. Retrieved from https://thehardestscience.com/2012/09/27/a-pottery-barn-rule-for-scientific-journals/.
Stangor, C., & Lemay, E. P. (2016). Introduction to the special issue on methodological rigor and replicability. Journal of Experimental Social Psychology, 66, 1–3. https://doi.org/10.1016/j.jesp.2016.02.006.
Strack, F. (2016). Reflection on the smiling registered replication report. Perspectives on Psychological Science, 11(6), 929–930. https://doi.org/10.1177/1745691616674460.
Strassberg, D. S., & Lowe, K. (1995). Volunteer bias in sexuality research. Archives of Sexual Behavior, 24(4), 369–382.
Article Google Scholar
Szollosi, A., Kellen, D., Navarro, D., Shiffrin, R., van Rooij, I., Van Zandt, T., & Donkin, C. (2019). Is preregistration worthwhile? Trends in Cognitive Sciences. https://doi.org/10.1016/j.tics.2019.11.009.
Article Google Scholar
Tackett, J. L., & Miller, J. D. (2019). Introduction to the special section on increasing replicability, transparency, and openness in clinical psychology. Journal of Abnormal Psychology, 128(6), 487–492. https://doi.org/10.1037/abn0000455.
Article Google Scholar
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., & Nosek, B. A. (2019). Scientific utopia III: Crowdsourcing science. Perspectives on Psychological Science, 14(5), 711–733. https://doi.org/10.1177/1745691619850561.
Article Google Scholar
van Anders, S. M. (2018). Welcome to the 2018 Annual Review of Sex Research; Plus: The search begins! Journal of Sex Research, 55, 407–409. https://doi.org/10.1080/00224499.2018.1441226.
Article Google Scholar
van Rooij, I., & Baggio, G. (2020, February 28). Theory before the test: How to build high-verisimilitude explanatory theories in psychological science. https://doi.org/10.31234/osf.io/7qbpr.
Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884.
Article Google Scholar
Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. https://doi.org/10.3758/BF03194105.
Article Google Scholar
Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., & Bulnes, L. C. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. https://doi.org/10.1177/1745691616674458.
Article Google Scholar
Wagenmakers, E. J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., & Meerhoff, F. (2018). Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review, 25(1), 58–76. https://doi.org/10.3758/s13423-017-1323-7.
Article Google Scholar
Wagenmakers, E. J., Wetzels, R., Borsboom, D., & Van Der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. https://doi.org/10.1037/a0022790.
Article Google Scholar
Wiederman, M. W., & Whitley, B. E. (2002). A preview: The unique nature of sexuality research. In M. W. Wiederman & B. E. Whitley Jr. (Eds.), Handbook for conducting research on human sexuality (pp. 1–6). Mahwah, NJ: Lawrence Erlbaum Associates.
Google Scholar
Williams, A. J., Botanov, Y., Kilshaw, R. E., Wong, R. E., & Sakaluk, J. K. (2020). Potentially harmful therapies: A meta-scientific review of evidential value. Clinical Psychology: Science and Practice. https://doi.org/10.1111/cpsp.12331.
Article Google Scholar
Wisman, A., & Shrira, I. (2020). Sexual chemosignals: Evidence that men process olfactory signals of women’s sexual arousal. Archives of Sexual Behavior, 49, 1505–1516. https://doi.org/10.1007/s10508-019-01588-8.
Article Google Scholar
Yeater, E., Miller, G., Rinehart, J., & Nason, E. (2012). Trauma and sex surveys meet minimal risk standards: Implications for institutional review boards. Psychological Science, 23(7), 780–787. https://doi.org/10.1177/0956797611435131.
Article Google Scholar
Yuan, K. H., & Maxwell, S. (2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30, 141–167. https://doi.org/10.3102/10769986030002141.
Article Google Scholar
Zell, E., Krizan, Z., & Teeter, S. R. (2015). Evaluating gender similarities and differences using metasynthesis. American Psychologist, 70, 10–20. https://doi.org/10.1037/a0038208.
Article Google Scholar

Download references

Acknowledgements

My sincere thanks to Drs. Alexander Williams, Chris Quinn-Nilas, Lorne Campbell, and Neil Lewis Jr. for their constructive feedback on an earlier draft of this Commentary, and to Dr. Jin Goh for consulting on his mini-meta-analysis article and resources.

Author information

Authors and Affiliations

Department of Psychology, University of Victoria, P. O. Box 1700, STN CSC, Victoria, BC, V8W 2Y2, Canada
John K. Sakaluk

Authors

John K. Sakaluk
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John K. Sakaluk.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sakaluk, J.K. Getting Serious About the Assessment and Promotion of Replicable Sexual Science: A Commentary on Wisman and Shrira (2020) and Lorenz (2020). Arch Sex Behav 49, 2743–2754 (2020). https://doi.org/10.1007/s10508-020-01795-8

Download citation

Received: 09 May 2020
Revised: 03 July 2020
Accepted: 06 July 2020
Published: 21 July 2020
Issue Date: November 2020
DOI: https://doi.org/10.1007/s10508-020-01795-8

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions