, Volume 109, Issue 3, pp 1593–1610 | Cite as

Disciplinary differences of software use and impact in scientific literature

  • Xuelian Pan
  • Erjia Yan
  • Weina Hua


Software plays an important role in the advancement of science. Software developers, users, and funding agencies have deep interests in the impact of software on science. This study investigates the use and impact of software by examining how software is mentioned and cited among 9548 articles published in PLOS ONE in 12 defined disciplines. Our results demonstrate that software is widely used in scientific research and a substantial uncitedness of software exists across different disciplines. Findings also show that the practice of software citations varies noticeably at the discipline level and software that is free for academic use is more likely to receive citations than commercial software.


Scientific software Software citation Citation analysis Entity citation Digital outputs 



Xuelian Pan is supported by the Program B for Outstanding PhD candidate of Nanjing University. Erjia Yan is supported by the National Consortium for Data Science (NCDS) Data Fellows program for the project “Assessing the Impact of Data and Software on Science Using Hybrid Metrics”. Also, we are grateful to the reviewers for their valuable comments.


  1. Candela, L., Castelli, D., Manghi, P., & Tani, A. (2015). Data journals: A survey. Journal of the Association for Information Science and Technology, 66(9), 1747–1762. doi: 10.1002/asi.23358.CrossRefGoogle Scholar
  2. Chao, T. C. (2011). Disciplinary reach: Investigating the impact of dataset reuse in the earth sciences. Proceedings of the ASIST Annual Meeting,. doi: 10.1002/meet.2011.14504801125.Google Scholar
  3. Crowston, K., Howison, J., & Wiggins, A. (2010). Free/Libre open source software development: What we know and what we do not know. ACM Computing Surveys, 40(2), 1–37. doi: 10.1145/2089125.2089127.Google Scholar
  4. Duck, G., Nenadic, G., Brass, A., Robertson, D. L., & Stevens, R. (2013). bioNerDS: Exploring bioinformatics’ database and software use through literature mining. BMC Bioinformatics, 14(1), 1.CrossRefGoogle Scholar
  5. Hafer, L., & Kirkpatrick, A. E. (2009). Assessing open source software as a scholarly contribution. Communications of the ACM, 52, 126. doi: 10.1145/1610252.1610285.CrossRefGoogle Scholar
  6. Hann, I.-H., Roberts, J., & Slaughter, S. (2004). Why developers participate in open source software projects: An empirical investigation. In CIS 2004 proceedings (p. 66).Google Scholar
  7. Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software? In Proceedings of the 2009 ICSE workshop on software engineering for computational science and engineering, SECSE 2009 (pp. 1–8). doi: 10.1109/SECSE.2009.5069155
  8. Hedley, J. Jsoup: Java HTML Parser. Version 1.7.3 (software). [cited 2015 Oct 16]. Available from
  9. Howison, J., & Bullard, J. (2016). Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology, 67(9), 2137–2155. doi: 10.1002/asi.23538.CrossRefGoogle Scholar
  10. Howison, J., Deelman, E., Mclennan, M. J., da Silva, R. F., & Herbsleb, J. D. (2015). Understanding the scientific software ecosystem and its impact: Current and future measures. Research Evaluation, 24(4), 454–470.CrossRefGoogle Scholar
  11. Howison, J., & Herbsleb, J. (2010). Socio-technical logics of correctness in the scientific software development ecosystem. Workshop on changing dynamics of scientific collaboration workshop at CSCW 2010. Retrieved from
  12. Howison, J., & Herbsleb, J. D. (2011). Scientific software production. In Proceedings of the ACM 2011 conference on computer supported cooperative workCSCW’11 (pp. 513–522). doi: 10.1145/1958824.1958904
  13. Howison, J., & Herbsleb, J. D. (2013). Incentives and integration in scientific software production. in Proceedings of the 2013 conference on computer supported cooperative workCSCW’13 (p. 459). doi: 10.1145/2441776.2441828
  14. Howison, J., & Herbsleb, J. (2014). The sustainability of scientific software: Ecosystem context and science policy. Working paper. University of Texas at Austin. Retrieved from
  15. Huang, X., Ding, X., Lee, C. P., Lu, T., Gu, N., & Hall, S. (2013). Meanings and boundaries of scientific software sharing. In Proceedings of conference on computer supported cooperative work (CSCW) (pp. 423–434). doi: 10.1145/2441776.2441825
  16. IBM Corp. SPSS. Version 20 (software). [cited 2015 Oct 16]. Available from
  17. Katz, D. S., Choi, S.-C. T., Lapp, H., Maheshwari, K., Löffler, F., Turk, M., et al. (2014). Summary of the first workshop on sustainable software for science: Practice and experiences (WSSSPE1). Journal of Open Research Software, 2(1), 1–21. doi: 10.5334/ Scholar
  18. Katz, D. S., Choi, S.-C. T., Wilkins-Diehr, N., Chue Hong, N., Venters, C. C., Howison, J., et al. (2015). Report on the second workshop on sustainable software for science: Practice and experiences (WSSSPE2) (Vol. 1, pp. 1–30). Retrieved from
  19. Lakhani, K., & Wolf, R. G. (2003). Why hackers do what they do: Understanding motivation and effort in free/open source software projects. SSRN Electronic Journal,. doi: 10.2139/ssrn.443040.Google Scholar
  20. Pan, X., Yan, E., Wang, Q., & Hua, W. (2015). Assessing the impact of software on science: A bootstrapped learning of software entities in full-text papers. Journal of Informetrics, 9(4), 860–871. doi: 10.1016/j.joi.2015.07.012.CrossRefGoogle Scholar
  21. Piwowar, H. A. (2013). Value all research products. Nature, 493, 159. doi: 10.1038/493159a.Google Scholar
  22. Piwowar, H. A., Carlson, J. D., & Vision, T. J. (2011). Beginning to track 1000 datasets from public repositories into the published literature. Proceedings of the ASIST Annual Meeting, 48(1), 1–4. doi: 10.1002/meet.2011.14504801337.Google Scholar
  23. Piwowar, H. A., & Chapman, W. W. (2010). Public sharing of research datasets: A pilot study of associations. Journal of Informetrics, 4(2), 148–156. doi: 10.1016/j.joi.2009.11.010.CrossRefGoogle Scholar
  24. Poisot, T. (2015). Best publishing practices to improve user confidence in scientific software. Ideas in Ecology and Evolution, 8, 50–54. doi: 10.4033/iee.2015.8.8.f.CrossRefGoogle Scholar
  25. Prabhu, P., Zhang, Y., Ghosh, S., August, D. I., Huang, J., Beard, S., et al. (2011). A survey of the practice of computational science. State of the practice reports on SC’11 (p. 1). doi: 10.1145/2063348.2063374
  26. Roberts, J. A., Hann, I.-H., & Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects. Management Science, 52(7), 984–999. doi: 10.1287/mnsc.1060.0554.CrossRefGoogle Scholar
  27. Robinson-García, N., Jiménez-Contreras, E., & Torres-Salinas, D. (2015). Analyzing data citation practices using the data citation index. Journal of the Association for Information Science and Technology,. doi: 10.1002/asi.23529.Google Scholar
  28. Rolland, B., & Lee, C. (2013). Beyond trust and reliability: reusing data in collaborative cancer epidemiology research. In Proceedings of the ACM 2013 conference on computer supported cooperative work (pp. 435–444). doi: 10.1145/2441776.2441826
  29. Segal, J., & Morris, C. (2008). Developing scientific software. IEEE Software, 25(4), 18–20.CrossRefGoogle Scholar
  30. Stewart, C. A., Almes, G. T., & Wheeler, B. C. (2010). NSF cyberinfrastructure software sustainability and reusability workshop report.
  31. Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., et al. (2011). Data sharing by scientists: Practices and perceptions. PLoS ONE, 6(6), e21101.CrossRefGoogle Scholar
  32. Trainer, E. H., Chaihirunkarn, C., & Herbsleb, J. D. (2013). The big effects of short-term efforts: A catalyst for community engagement in scientific software. Workshop in sustainable software for science: Practice and experience (WSSSPE) (pp. 1–4). doi: 10.6084/m9.figshare.790754
  33. Trainer, E. H., Chaihirunkarn, C., Kalyanasundaram, A., & Herbsleb, J. D. (2015). From personal tool to community resource: What’s the extra work and who will do it? In Proceedings of the 18th ACM conference on computer supported cooperative work and social computing (pp. 417–430). ACM.Google Scholar
  34. Velden, T., Bietz, M. J., Diamant, E. I., Herbsleb, J. D., Howison, J., Ribes, D., et al. (2014). Sharing, re-use and circulation of resources in cooperative scientific work. In Proceedings of the companion publication of the 17th {ACM} conference on computer supported cooperative work & Social computing (pp. 347–350). doi: 10.1145/2556420.2558853

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.School of Information ManagementNanjing UniversityNanjingChina
  2. 2.Jiangsu Key Laboratory of Data Engineering and Knowledge ServiceNanjingChina
  3. 3.College of Computing and InformaticsDrexel UniversityPhiladelphiaUSA

Personalised recommendations