Abstract
Software plays an important role in the advancement of science. Software developers, users, and funding agencies have deep interests in the impact of software on science. This study investigates the use and impact of software by examining how software is mentioned and cited among 9548 articles published in PLOS ONE in 12 defined disciplines. Our results demonstrate that software is widely used in scientific research and a substantial uncitedness of software exists across different disciplines. Findings also show that the practice of software citations varies noticeably at the discipline level and software that is free for academic use is more likely to receive citations than commercial software.
Similar content being viewed by others
References
Candela, L., Castelli, D., Manghi, P., & Tani, A. (2015). Data journals: A survey. Journal of the Association for Information Science and Technology, 66(9), 1747–1762. doi:10.1002/asi.23358.
Chao, T. C. (2011). Disciplinary reach: Investigating the impact of dataset reuse in the earth sciences. Proceedings of the ASIST Annual Meeting,. doi:10.1002/meet.2011.14504801125.
Crowston, K., Howison, J., & Wiggins, A. (2010). Free/Libre open source software development: What we know and what we do not know. ACM Computing Surveys, 40(2), 1–37. doi:10.1145/2089125.2089127.
Duck, G., Nenadic, G., Brass, A., Robertson, D. L., & Stevens, R. (2013). bioNerDS: Exploring bioinformatics’ database and software use through literature mining. BMC Bioinformatics, 14(1), 1.
Hafer, L., & Kirkpatrick, A. E. (2009). Assessing open source software as a scholarly contribution. Communications of the ACM, 52, 126. doi:10.1145/1610252.1610285.
Hann, I.-H., Roberts, J., & Slaughter, S. (2004). Why developers participate in open source software projects: An empirical investigation. In CIS 2004 proceedings (p. 66).
Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software? In Proceedings of the 2009 ICSE workshop on software engineering for computational science and engineering, SECSE 2009 (pp. 1–8). doi:10.1109/SECSE.2009.5069155
Hedley, J. Jsoup: Java HTML Parser. Version 1.7.3 (software). [cited 2015 Oct 16]. Available from https://jsoup.org/
Howison, J., & Bullard, J. (2016). Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology, 67(9), 2137–2155. doi:10.1002/asi.23538.
Howison, J., Deelman, E., Mclennan, M. J., da Silva, R. F., & Herbsleb, J. D. (2015). Understanding the scientific software ecosystem and its impact: Current and future measures. Research Evaluation, 24(4), 454–470.
Howison, J., & Herbsleb, J. (2010). Socio-technical logics of correctness in the scientific software development ecosystem. Workshop on changing dynamics of scientific collaboration workshop at CSCW 2010. Retrieved from http://repository.cmu.edu/isr/496/?utm_source=repository.cmu.edu/isr/496&utm_medium=PDF&utm_campaign=PDFCoverPages
Howison, J., & Herbsleb, J. D. (2011). Scientific software production. In Proceedings of the ACM 2011 conference on computer supported cooperative work—CSCW’11 (pp. 513–522). doi:10.1145/1958824.1958904
Howison, J., & Herbsleb, J. D. (2013). Incentives and integration in scientific software production. in Proceedings of the 2013 conference on computer supported cooperative work—CSCW’13 (p. 459). doi:10.1145/2441776.2441828
Howison, J., & Herbsleb, J. (2014). The sustainability of scientific software: Ecosystem context and science policy. Working paper. University of Texas at Austin. Retrieved from http://james.howison.name/pubs/HowisonHerbsleb-Sustainability.pdf
Huang, X., Ding, X., Lee, C. P., Lu, T., Gu, N., & Hall, S. (2013). Meanings and boundaries of scientific software sharing. In Proceedings of conference on computer supported cooperative work (CSCW) (pp. 423–434). doi:10.1145/2441776.2441825
IBM Corp. SPSS. Version 20 (software). [cited 2015 Oct 16]. Available from http://www-01.ibm.com/software/cn/analytics/spss/downloads.html
Katz, D. S., Choi, S.-C. T., Lapp, H., Maheshwari, K., Löffler, F., Turk, M., et al. (2014). Summary of the first workshop on sustainable software for science: Practice and experiences (WSSSPE1). Journal of Open Research Software, 2(1), 1–21. doi:10.5334/jors.an.
Katz, D. S., Choi, S.-C. T., Wilkins-Diehr, N., Chue Hong, N., Venters, C. C., Howison, J., et al. (2015). Report on the second workshop on sustainable software for science: Practice and experiences (WSSSPE2) (Vol. 1, pp. 1–30). Retrieved from http://arxiv.org/abs/1507.01715
Lakhani, K., & Wolf, R. G. (2003). Why hackers do what they do: Understanding motivation and effort in free/open source software projects. SSRN Electronic Journal,. doi:10.2139/ssrn.443040.
Pan, X., Yan, E., Wang, Q., & Hua, W. (2015). Assessing the impact of software on science: A bootstrapped learning of software entities in full-text papers. Journal of Informetrics, 9(4), 860–871. doi:10.1016/j.joi.2015.07.012.
Piwowar, H. A. (2013). Value all research products. Nature, 493, 159. doi:10.1038/493159a.
Piwowar, H. A., Carlson, J. D., & Vision, T. J. (2011). Beginning to track 1000 datasets from public repositories into the published literature. Proceedings of the ASIST Annual Meeting, 48(1), 1–4. doi:10.1002/meet.2011.14504801337.
Piwowar, H. A., & Chapman, W. W. (2010). Public sharing of research datasets: A pilot study of associations. Journal of Informetrics, 4(2), 148–156. doi:10.1016/j.joi.2009.11.010.
Poisot, T. (2015). Best publishing practices to improve user confidence in scientific software. Ideas in Ecology and Evolution, 8, 50–54. doi:10.4033/iee.2015.8.8.f.
Prabhu, P., Zhang, Y., Ghosh, S., August, D. I., Huang, J., Beard, S., et al. (2011). A survey of the practice of computational science. State of the practice reports on SC’11 (p. 1). doi:10.1145/2063348.2063374
Roberts, J. A., Hann, I.-H., & Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects. Management Science, 52(7), 984–999. doi:10.1287/mnsc.1060.0554.
Robinson-García, N., Jiménez-Contreras, E., & Torres-Salinas, D. (2015). Analyzing data citation practices using the data citation index. Journal of the Association for Information Science and Technology,. doi:10.1002/asi.23529.
Rolland, B., & Lee, C. (2013). Beyond trust and reliability: reusing data in collaborative cancer epidemiology research. In Proceedings of the ACM 2013 conference on computer supported cooperative work (pp. 435–444). doi:10.1145/2441776.2441826
Segal, J., & Morris, C. (2008). Developing scientific software. IEEE Software, 25(4), 18–20.
Stewart, C. A., Almes, G. T., & Wheeler, B. C. (2010). NSF cyberinfrastructure software sustainability and reusability workshop report. http://hdl.handle.net/2022/6701.
Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., et al. (2011). Data sharing by scientists: Practices and perceptions. PLoS ONE, 6(6), e21101.
Trainer, E. H., Chaihirunkarn, C., & Herbsleb, J. D. (2013). The big effects of short-term efforts: A catalyst for community engagement in scientific software. Workshop in sustainable software for science: Practice and experience (WSSSPE) (pp. 1–4). doi:10.6084/m9.figshare.790754
Trainer, E. H., Chaihirunkarn, C., Kalyanasundaram, A., & Herbsleb, J. D. (2015). From personal tool to community resource: What’s the extra work and who will do it? In Proceedings of the 18th ACM conference on computer supported cooperative work and social computing (pp. 417–430). ACM.
Velden, T., Bietz, M. J., Diamant, E. I., Herbsleb, J. D., Howison, J., Ribes, D., et al. (2014). Sharing, re-use and circulation of resources in cooperative scientific work. In Proceedings of the companion publication of the 17th {ACM} conference on computer supported cooperative work & Social computing (pp. 347–350). doi:10.1145/2556420.2558853
Acknowledgments
Xuelian Pan is supported by the Program B for Outstanding PhD candidate of Nanjing University. Erjia Yan is supported by the National Consortium for Data Science (NCDS) Data Fellows program for the project “Assessing the Impact of Data and Software on Science Using Hybrid Metrics”. Also, we are grateful to the reviewers for their valuable comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pan, X., Yan, E. & Hua, W. Disciplinary differences of software use and impact in scientific literature. Scientometrics 109, 1593–1610 (2016). https://doi.org/10.1007/s11192-016-2138-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-016-2138-4