Abstract
Computer code plays a vital role in modern science, from the conception and design of experiments through to final data analyses. Open sharing of code has been widely discussed as being advantageous to the scientific process, allowing experiments to be more easily replicated, helping with error detection, and reducing wasted effort and resources. In the case of psychology, the code used to present stimuli is a fundamental component of many experiments. It is not known, however, the degree to which researchers are sharing this type of code. To estimate this, we conducted a survey of 400 psychology papers published between 2016 and 2021, identifying those working with the open-source tools Psychtoolbox and PsychoPy that openly share stimulus presentation code. For those that did, we established if it would run following download and also appraised the code’s usability in terms of style and documentation. It was found that only 8.4% of papers shared stimulus code, compared to 17.9% sharing analysis code and 31.7% sharing data. Of shared code, 70% ran directly or after minor corrections. For code that did not run, the main error was missing dependencies (66.7%). The usability of the code was moderate, with low levels of code annotation and minimal documentation provided. These results suggest that stimulus presentation code sharing lags behind other forms of code and data sharing, potentially due to less emphasis on such code in open-science discussions and in journal policies. The results also highlight a need for improved documentation to maximize code utility.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
The data and code used to conduct this analysis are available at https://osf.io/ka9ws/.
Code availability
Not applicable
References
AlNoamany, Y., & Borghi, J. A. (2018). Towards computational reproducibility: Researcher perspectives on the use and sharing of software. PeerJ Computer Science, 4, e163. https://doi.org/10.7717/peerj-cs.163 Retrieved 2022-06-09, from https://peerj.com/articles/cs-163 (Publisher: PeerJ Inc.)
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), https://doi.org/10.1038/533452a. Retrieved 2021-11-22, from https://www.nature.com/articles/533452a
Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2019). Answers to 18 questions about open science practices. Journal of Business and Psychology, 34(3), 257–270. https://doi.org/10.1007/s10869-018-9547-8
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1), 289–300.
Borghi, J. A., & Van Gulick, A. E. (2021). Data management and sharing: Practices and perceptions of psychology researchers. PloS one, 16(5), e0252047.
Brito, J. J., Li, J., Moore, J. H., Greene, C. S., Nogoy, N. A., Garmire, L. X., & Mangul, S. (2020). Recommendations to enhance rigor and reproducibility in biomedical research. GigaScience, 9(6). https://doi.org/10.1093/gigascience/giaa056.
Burgelman, J.-C., Pascu, C., Szkuta, K., Von Schomberg, R., Karalopoulos, A., Repanas, K., & Schouppe, M. (2019). Open Science, Open Data, and Open Scholarship: European Policies to Make Science Fit for the Twenty-First Century. Frontiers in Big Data, 2. Retrieved 2022-06-09, from https://www.frontiersin.org/article/10.3389/fdata.2019.00043
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-9. Retrieved 2021-11-24, from https://www.sciencedirect.com/science/article/pii/S0140673609603299
Christensen, G., Dafoe, A., Miguel, E., Moore, D. A., & Rose, A. K. (2019). A study of the impact of data sharing on article citations using journal policies as a natural experiment. PLOS ONE, 14(12), e0225883. https://doi.org/10.1371/journal.pone.0225883
Crouch, S., Hong, N. C., Hettrick, S., Jackson, M., Pawlik, A., Sufi, S., & Parsons, M. (2013). The software sustainability institute: Changing research software attitudes and practices. Computing in Science Engineering, 15(6), 74–80. https://doi.org/10.1109/MCSE.2013.133 (Conference Name: Computing in Science Engineering)
Eglen, S. J., Marwick, B., Halchenko, Y. O., Hanke, M., Sufi, S., Gleeson, P., & Poline, J-B. (2017). Toward standard practices for sharing computer code and programs in neuroscience. Nature Neuroscience, 20(6), 770–773, https://doi.org/10.1038/nn.4550. Retrieved 2022-06-09, from https://www.nature.com/articles/nn.4550. (Number: 6 Publisher: Nature Publishing Group)
Federer, L. M. (2022). Long-term availability of data associated with articles in PLOS ONE. PLOS ONE, 17(8), e0272845. https://doi.org/10.1371/journal.pone.0272845. Retrieved 2022-08-26, https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0272845. (Publisher: Public Library of Science)
Gabelica, M., Bojčić, R., & Puljak, L. (2022). Many researchers were not compliant with their published data sharing statement: Mixed-methods study. Journal of Clinical Epidemiology, 0(0), https://doi.org/10.1016/j.jclinepi.2022.05.019. Retrieved 2022-06-08, from https://www.jclinepi.com/article/S0895-4356(22)00141-X/abstract. (Publisher: Elsevier)
Garrett-Ruffin, S., Hindash, A. C., Kaczkurkin, A. N., Mears, R. P., Morales, S., Paul, K., & Keil, A. (2021). Open science in psychophysiology: An overview of challenges and emerging solutions. International Journal of Psychophysiology, 162, 69–78. https://doi.org/10.1016/j.ijpsycho.2021.02.005. Retrieved 2021-11-24, from https://www.sciencedirect.com/science/article/pii/S0167876021000441
Giofré, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PLOS ONE, 12(4), e0175583. https://doi.org/10.1371/journal.pone.0175583. Retrieved 2021-11-24, from https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0175583. (Publisher: Public Library of Science)
Grüning, B.A., Lampa, S., Vaudel, M., & Blankenberg, D. (2019). Software engineering for scientific big data analysis. GigaScience, 8(5), https://doi.org/10.1093/gigascience/giz054. Retrieved 2021-11-24, from. https://doi.org/10.1093/gigascience/giz054
Hafer, L., & Kirkpatrick, A. E. (2009). Assessing open source software as a scholarly contribution. Communications of the ACM, 52(12), 126–129. https://doi.org/10.1145/1610252.1610285
Hannay, J.E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software?. 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (pp. 1–8).
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2021). Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017). Perspectives on Psychological Science, 1745691620979806. https://doi.org/10.1177/1745691620979806. Retrieved 2021-11-22, from http://dx.doi.org/10.1177/1745691620979806. Publisher: SAGE Publications Inc.
Howison, J., & Herbsleb, J. D. (2013). Incentives and integration in scientific software production. Proceedings of the 2013 conference on computer supported cooperative work (pp. 459–470). New York, USA: Association for Computing Machinery. Retrieved 2022-06-06, from https://doi.org/10.1145/2441776.2441828.
Ioannidis, J. P. A. (2012). Why Science Is Not Necessarily Self-Correcting. Perspectives on Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056 Retrieved 2021-11-24, from https://doi.org/10.1177/1745691612464056. (Publisher: SAGE Publications Inc)
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166–175. https://doi.org/10.1016/S0140-6736(13)62227-8. Retrieved 2021-10-15, from https://www.sciencedirect.com/science/article/pii/S0140673613622278
Janssen, M. A., Pritchard, C., & Lee, A. (2020). On code sharing and model documentation of published individual and agent-based models. Environmental Modelling & Software, 134, 104873. https://doi.org/10.1016/j.envsoft.2020.104873. Retrieved from https://www.sciencedirect.com/science/article/pii/S1364815220309300
Juavinett, A. L. (2022). The next generation of neuroscientists needs to learn how to code, and we need new ways to teach them. Neuron, 110(4), 576–578. https://doi.org/10.1016/j.neuron.2021.12.001. Retrieved 2022-06-09, from https://www.sciencedirect.com/science/article/pii/S0896627321009946
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456. Retrieved 2022-06-09, from https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456. (Publisher: Public Library of Science)
Kim, J., Kim, S., Cho, H.-M., Chang, J. H., & Kim, S. Y. (2020). Data sharing policies of journals in life, health, and physical sciences indexed in Journal Citation Reports. PeerJ, 8, e9924. https://doi.org/10.7717/peerj.9924. Retrieved 2022-08-23, from https://peerj.com/articles/9924. (Publisher: PeerJ Inc.)
Kim, Y., & Adler, M. (2015). Social scientists’ data sharing behaviors: Investigating the roles of individual motivations, institutional pressures, and data repositories. International Journal of Information Management, 35(4), 408–418. https://doi.org/10.1016/j.ijinfomgt.2015.04.007. Retrieved from https://www.sciencedirect.com/science/article/pii/S0268401215000432
Kleiner, M., Brainard, D., Pelli, D., Ingling, A., Murray, R., & Broussard, C. (2007). What is new in psychtoolbox 3. Perception, 36(14), 1–16.
Kubilius, J. (2014). I-review: Sharing code. i-Perception, 5(1), 75–78.
Laine, R. F., Arganda-Carreras, I., Henriques, R., & Jacquemet, G. (2021). Avoiding a replication crisis in deep-learning-based bioimage analysis. Nature Methods, 18(10), 1136–1144. https://doi.org/10.1038/s41592-021-01284-3. Retrieved 2021- 10-15, from https://www.nature.com/articles/s41592-021-01284-3
Lamprecht, A.-L., Garcia, L., Kuzak, M., Martinez, C., Arcila, R., Martin Del Pico, E., & Capella-Gutierrez, S. (2020). Towards FAIR principles for research software. Data Science, 3(1), 37–59. https://doi.org/10.3233/DS-190026. Retrieved 2022-09-03, from https://content.iospress.com/articles/data-science/ds190026. (Publisher: IOS Press)
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., & Yarkoni, T. (2016). How open science helps researchers succeed. eLife, 5, e16800. https://doi.org/10.7554/eLife.16800. Retrieved 2021-11-24, from https://doi.org/10.7554/eLife.16800. (Publisher: eLife Sciences Publications, Ltd).
Merali, Z. (2010). Computational science: Error. Nature, 467(7317), 775–777. https://doi.org/10.1038/467775a. Retrieved 2021-10-13, from https://www.nature.com/articles/467775a
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., & Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374. Retrieved 25 2021-10-15, from https://www.science.org/lookup/doi/10.1126/science.aab2374. (Publisher: American Association for the Advancement of Science)
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. https://doi.org/10.1177/2515245920918872. Retrieved 2022-06-09, from https://doi.org/10.1177/2515245920918872. (Publisher: SAGE Publications Inc)
Paret, C., Unverhau, N., Feingold, F., Poldrack, R. A., Stirner, M., Schmahl, C., & Sicorello, M. (2021). Survey on Open Science Practices in Functional Neuroimaging. Retrieved 2022-09-01, from https://www.biorxiv.org/content/10.1101/2021.11.26.470115v1. (Pages: 2021.11.26.470115 Section: New Results)
Pashler, H., & Wagenmakers, E. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science. Perspectives on psychological science : a journal of the Association for Psychological Science. https://doi.org/10.1177/1745691612465253
Peirce, J. W. (2007). Psychopy-psychophysics software in python. Journal of neuroscience methods, 162(1–2), 8–13.
Penning de Vries, B. B. L., van Smeden, M., Rosendaal, F. R., & Groenwold, R. H. H. (2020). Title, abstract, and keyword searching resulted in poor recovery of articles in systematic reviews of epidemiologic practice. Journal of Clinical Epidemiology, 121, 55–61. https://doi.org/10.1016/j.jclinepi.2020.01.009. Retrieved from https://www.sciencedirect.com/science/article/pii/S0895435619306018
Pittard, W. S., & Li, S. (2020). The Essential Toolbox of Data Science: Python, R, Git, and Docker. S. Li (Ed.), Computational Methods and Data Analysis for Metabolomics (pp. 265–311). New York, NY: Springer US. Retrieved 2022-06-09, from https://doi.org/10.1007/978-1-0716-0239-3_15
Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007). Sharing detailed research data is associated with increased citation rate. PloS one, 2(3), e308.
Powers, S. M., & Hampton, S. E. (2019). Open science, reproducibility, and transparency in ecology. Ecological Applications, 29(1), e01822. https://doi.org/10.1002/eap.1822. Retrieved 2021-11-27, from https://doi.org/onlinelibrary.wiley.com/doi/abs/10.1002/eap.1822 (_eprint: https://doi.org/onlinelibrary.wiley.com/doi/pdf/10.1002/eap.1822)
Prabhu, P., Kim, H., Oh, T., Jablin, T. B., Johnson, N. P., Zoufaly, M., & Beard, S. (2011). A survey of the practice of computational science. SC ’11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–12). (ISSN: 2167-4337)
Prlić, A., & Procter, J. B. (2012). Ten simple rules for the open development of scientific software. PLOS Computational Biology, 8(12), e1002802. https://doi.org/10.1371/journal.pcbi.1002802. Retrieved 2021-10-13, from https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002802. (Publisher: Public Library of Science)
Resnik, D. B., Morales, M., Landrum, R., Shi, M., Minnier, J., Vasilevsky, N. A., & Champieux, R. E. (2019). Effect of impact factor and discipline on journal data sharing policies. Accountability in Research, 26(3), 139–156. https://doi.org/10.1080/08989621.2019.1591277
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020). Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial. Royal Society Open Science, 7(3), 191818. https://doi.org/10.1098/rsos.191818
Rowhani-Farid, A., & Barnett, A. G. (2018). Badges for sharing data and code at Biostatistics: An observational study (Tech. Rep. No. 7:90). F1000Research. Retrieved 2022-09-01, from https://f1000research.com/articles/7-90. (Type: article)
Samuel, S., & König-Ries, B. (2021). Understanding experiments and research practices for reproducibility: An exploratory study. PeerJ, 9, e11140. https://doi.org/10.7717/peerj.11140. Retrieved 2021-10-15, from https://peerj.com/articles/11140. (Publisher: PeerJ Inc.)
Shamir, L., Wallin, J. F., Allen, A., Berriman, B., Teuben, P., Nemiroff, R. J., & DuPrie, K. (2013). Practices in source code sharing in astrophysics. Astronomy and Computing, 1, 54–58. https://doi.org/10.1016/j.ascom.2013.04.001. Retrieved 2021-11-24, from https://www.sciencedirect.com/science/article/pii/S2213133713000073
Sochat, V. V., Eisenberg, I. W., Enkavi, A. Z., Li, J., Bissett, P. G., & Poldrack, R. A. (2016). The experiment factory: Standardizing behavioral experiments. Frontiers in Psychology, 7. Retrieved 2022-06-09, from https://www.frontiersin.org/article/10.3389/fpsyg.2016.00610
Stodden, V., Guo, P., & Ma, Z. (2013). Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals. PLOS ONE, 8(6), e67111. https://doi.org/10.1371/journal.pone.0067111. Retrieved 2022-06-09, from https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0067111. (Publisher: Public Library of Science)
Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115. Retrieved 2022-06-07, from https://www.pnas.org/doi/full/10.1073/pnas.1708290115. (Publisher: Proceedings of the National Academy of Sciences)
Stoudt, S., Vásquez, V. N., & Martinez, C. C. (2021). Principles for data analysis workflows. PLOS Computational Biology, 17(3), e1008770. https://doi.org/10.1371/journal.pcbi.1008770. Retrieved 2021-11-24, from https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008770. (Publisher: Public Library of Science)
Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., & Frame, M. (2011). Data sharing by scientists: Practices and perceptions. PLOS ONE, 6(6), e21101. https://doi.org/10.1371/journal.pone.0021101. Retrieved 2022-06-07, from https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0021101. (Publisher: Public Library of Science)
Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022). A large-scale study on research code quality and execution. Scientific Data, 9(1), 60. https://doi.org/10.1038/s41597-022-01143-6. Retrieved 2022-03-23, from https://www.nature.com/articles/s41597-022-01143-6. (Number: 1 Publisher: Nature Publishing Group)
Vable, A. M., Diehl, S. F., & Glymour, M. M. (2021). Code review as a simple trick to enhance reproducibility, accelerate learning, and improve the quality of your team’s research. American Journal of Epidemiology, 190(10), 2172–2177. https://doi.org/10.1093/aje/kwab092
Wan, F., Toivo, W., Paterson, H. M., Nordmann, E., McAleer, P., Mavromati, K., & PsyTeachR Team. (2022). Open-source tutorials benefit the field. Nature Reviews Psychology, 1(6), 312–313. https://doi.org/10.1038/s44159-022-00058-8. Retrieved 2022-06-09, from https://www.nature.com/articles/s44159-022-00058-8. (Number: 6 Publisher: Nature Publishing Group)
Wilkinson, M. D., Sansone, S.-A., Schultes, E., Doorn, P., da Silva, Bonino, Santos, L. O., & Dumontier, M. (2018). A design framework and exemplar metrics for FAIRness. Scientific Data, 5(1), 180118. https://doi.org/10.1038/sdata.2018.118. Retrieved 2022-06-09, from https://www.nature.com/articles/sdata2018118. (Number: 1 Publisher: Nature Publishing Group)
Wilson, G., Aruliah, D. A., Brown, C. T., Hong, N. P. C., Davis, M., Guy, R. T., & Wilson, P. (2014). Best practices for scientific computing. PLOS Biology, 12(1), e1001745. https://doi.org/10.1371/journal.pbio.1001745. Retrieved 2021-10-13, from https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745. (Publisher: Public Library of Science)
Zhang, L., & Ma, L. (2021). Does open data boost journal impact: Evidence from Chinese economics. Scientometrics, 126(4), 3393–3419. https://doi.org/10.1007/s11192-021-03897-z
Acknowledgements
The authors thank Dr. Gabriele Bellucci for their constructive remarks on the manuscript.
Funding
This work was supported by grants to NWD from the Taiwan Ministry of Science and Technology (108-2410-H-038-008-MY2; 110-2628-H-038-001-MY4). The work was also supported by the Taiwan Ministry of Education Higher Education Sprout Project.
Author information
Authors and Affiliations
Contributions
AR: Conceptualization, Formal Analysis, Investigation, Methodology, Visualization, Writing - original draft;
VT: Conceptualization, Formal Analysis, Investigation, Methodology, Visualization, Writing - original draft;
YHL: Investigation, Methodology, Writing - review & editing;
LYT: Investigation, Methodology, Writing - review & editing;
NWD: Conceptualization, Methodology, Visualization, Writing - original draft.
Corresponding author
Ethics declarations
Conflict of interest/Competing interests
The authors declare no conflicts of interest.
Ethics approval
Not applicable
Consent to participate
Not applicable
Consent for publication
Not applicable
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rawal, A., Truong, V., Lo, YH. et al. A survey of experimental stimulus presentation code sharing in major areas of psychology. Behav Res (2024). https://doi.org/10.3758/s13428-024-02390-8
Accepted:
Published:
DOI: https://doi.org/10.3758/s13428-024-02390-8