Skip to main content
SpringerLink
Log in

Disciplinary differences of software use and impact in scientific literature

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Software plays an important role in the advancement of science. Software developers, users, and funding agencies have deep interests in the impact of software on science. This study investigates the use and impact of software by examining how software is mentioned and cited among 9548 articles published in PLOS ONE in 12 defined disciplines. Our results demonstrate that software is widely used in scientific research and a substantial uncitedness of software exists across different disciplines. Findings also show that the practice of software citations varies noticeably at the discipline level and software that is free for academic use is more likely to receive citations than commercial software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Candela, L., Castelli, D., Manghi, P., & Tani, A. (2015). Data journals: A survey. Journal of the Association for Information Science and Technology, 66(9), 1747–1762. doi:10.1002/asi.23358.

    Article  Google Scholar 

  • Chao, T. C. (2011). Disciplinary reach: Investigating the impact of dataset reuse in the earth sciences. Proceedings of the ASIST Annual Meeting,. doi:10.1002/meet.2011.14504801125.

    Article  Google Scholar 

  • Crowston, K., Howison, J., & Wiggins, A. (2010). Free/Libre open source software development: What we know and what we do not know. ACM Computing Surveys, 40(2), 1–37. doi:10.1145/2089125.2089127.

    Article  Google Scholar 

  • Duck, G., Nenadic, G., Brass, A., Robertson, D. L., & Stevens, R. (2013). bioNerDS: Exploring bioinformatics’ database and software use through literature mining. BMC Bioinformatics, 14(1), 1.

    Article  Google Scholar 

  • Hafer, L., & Kirkpatrick, A. E. (2009). Assessing open source software as a scholarly contribution. Communications of the ACM, 52, 126. doi:10.1145/1610252.1610285.

    Article  Google Scholar 

  • Hann, I.-H., Roberts, J., & Slaughter, S. (2004). Why developers participate in open source software projects: An empirical investigation. In CIS 2004 proceedings (p. 66).

  • Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software? In Proceedings of the 2009 ICSE workshop on software engineering for computational science and engineering, SECSE 2009 (pp. 1–8). doi:10.1109/SECSE.2009.5069155

  • Hedley, J. Jsoup: Java HTML Parser. Version 1.7.3 (software). [cited 2015 Oct 16]. Available from https://jsoup.org/

  • Howison, J., & Bullard, J. (2016). Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology, 67(9), 2137–2155. doi:10.1002/asi.23538.

    Article  Google Scholar 

  • Howison, J., Deelman, E., Mclennan, M. J., da Silva, R. F., & Herbsleb, J. D. (2015). Understanding the scientific software ecosystem and its impact: Current and future measures. Research Evaluation, 24(4), 454–470.

    Article  Google Scholar 

  • Howison, J., & Herbsleb, J. (2010). Socio-technical logics of correctness in the scientific software development ecosystem. Workshop on changing dynamics of scientific collaboration workshop at CSCW 2010. Retrieved from http://repository.cmu.edu/isr/496/?utm_source=repository.cmu.edu/isr/496&utm_medium=PDF&utm_campaign=PDFCoverPages

  • Howison, J., & Herbsleb, J. D. (2011). Scientific software production. In Proceedings of the ACM 2011 conference on computer supported cooperative work—CSCW’11 (pp. 513–522). doi:10.1145/1958824.1958904

  • Howison, J., & Herbsleb, J. D. (2013). Incentives and integration in scientific software production. in Proceedings of the 2013 conference on computer supported cooperative work—CSCW’13 (p. 459). doi:10.1145/2441776.2441828

  • Howison, J., & Herbsleb, J. (2014). The sustainability of scientific software: Ecosystem context and science policy. Working paper. University of Texas at Austin. Retrieved from http://james.howison.name/pubs/HowisonHerbsleb-Sustainability.pdf

  • Huang, X., Ding, X., Lee, C. P., Lu, T., Gu, N., & Hall, S. (2013). Meanings and boundaries of scientific software sharing. In Proceedings of conference on computer supported cooperative work (CSCW) (pp. 423–434). doi:10.1145/2441776.2441825

  • IBM Corp. SPSS. Version 20 (software). [cited 2015 Oct 16]. Available from http://www-01.ibm.com/software/cn/analytics/spss/downloads.html

  • Katz, D. S., Choi, S.-C. T., Lapp, H., Maheshwari, K., Löffler, F., Turk, M., et al. (2014). Summary of the first workshop on sustainable software for science: Practice and experiences (WSSSPE1). Journal of Open Research Software, 2(1), 1–21. doi:10.5334/jors.an.

    Article  Google Scholar 

  • Katz, D. S., Choi, S.-C. T., Wilkins-Diehr, N., Chue Hong, N., Venters, C. C., Howison, J., et al. (2015). Report on the second workshop on sustainable software for science: Practice and experiences (WSSSPE2) (Vol. 1, pp. 1–30). Retrieved from http://arxiv.org/abs/1507.01715

  • Lakhani, K., & Wolf, R. G. (2003). Why hackers do what they do: Understanding motivation and effort in free/open source software projects. SSRN Electronic Journal,. doi:10.2139/ssrn.443040.

    Article  Google Scholar 

  • Pan, X., Yan, E., Wang, Q., & Hua, W. (2015). Assessing the impact of software on science: A bootstrapped learning of software entities in full-text papers. Journal of Informetrics, 9(4), 860–871. doi:10.1016/j.joi.2015.07.012.

    Article  Google Scholar 

  • Piwowar, H. A. (2013). Value all research products. Nature, 493, 159. doi:10.1038/493159a.

    Article  Google Scholar 

  • Piwowar, H. A., Carlson, J. D., & Vision, T. J. (2011). Beginning to track 1000 datasets from public repositories into the published literature. Proceedings of the ASIST Annual Meeting, 48(1), 1–4. doi:10.1002/meet.2011.14504801337.

    Article  Google Scholar 

  • Piwowar, H. A., & Chapman, W. W. (2010). Public sharing of research datasets: A pilot study of associations. Journal of Informetrics, 4(2), 148–156. doi:10.1016/j.joi.2009.11.010.

    Article  Google Scholar 

  • Poisot, T. (2015). Best publishing practices to improve user confidence in scientific software. Ideas in Ecology and Evolution, 8, 50–54. doi:10.4033/iee.2015.8.8.f.

    Article  Google Scholar 

  • Prabhu, P., Zhang, Y., Ghosh, S., August, D. I., Huang, J., Beard, S., et al. (2011). A survey of the practice of computational science. State of the practice reports on SC’11 (p. 1). doi:10.1145/2063348.2063374

  • Roberts, J. A., Hann, I.-H., & Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects. Management Science, 52(7), 984–999. doi:10.1287/mnsc.1060.0554.

    Article  Google Scholar 

  • Robinson-García, N., Jiménez-Contreras, E., & Torres-Salinas, D. (2015). Analyzing data citation practices using the data citation index. Journal of the Association for Information Science and Technology,. doi:10.1002/asi.23529.

    Article  Google Scholar 

  • Rolland, B., & Lee, C. (2013). Beyond trust and reliability: reusing data in collaborative cancer epidemiology research. In Proceedings of the ACM 2013 conference on computer supported cooperative work (pp. 435–444). doi:10.1145/2441776.2441826

  • Segal, J., & Morris, C. (2008). Developing scientific software. IEEE Software, 25(4), 18–20.

    Article  Google Scholar 

  • Stewart, C. A., Almes, G. T., & Wheeler, B. C. (2010). NSF cyberinfrastructure software sustainability and reusability workshop report. http://hdl.handle.net/2022/6701.

  • Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., et al. (2011). Data sharing by scientists: Practices and perceptions. PLoS ONE, 6(6), e21101.

    Article  Google Scholar 

  • Trainer, E. H., Chaihirunkarn, C., & Herbsleb, J. D. (2013). The big effects of short-term efforts: A catalyst for community engagement in scientific software. Workshop in sustainable software for science: Practice and experience (WSSSPE) (pp. 1–4). doi:10.6084/m9.figshare.790754

  • Trainer, E. H., Chaihirunkarn, C., Kalyanasundaram, A., & Herbsleb, J. D. (2015). From personal tool to community resource: What’s the extra work and who will do it? In Proceedings of the 18th ACM conference on computer supported cooperative work and social computing (pp. 417–430). ACM.

  • Velden, T., Bietz, M. J., Diamant, E. I., Herbsleb, J. D., Howison, J., Ribes, D., et al. (2014). Sharing, re-use and circulation of resources in cooperative scientific work. In Proceedings of the companion publication of the 17th {ACM} conference on computer supported cooperative work & Social computing (pp. 347–350). doi:10.1145/2556420.2558853

Download references

Acknowledgments

Xuelian Pan is supported by the Program B for Outstanding PhD candidate of Nanjing University. Erjia Yan is supported by the National Consortium for Data Science (NCDS) Data Fellows program for the project “Assessing the Impact of Data and Software on Science Using Hybrid Metrics”. Also, we are grateful to the reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. School of Information Management, Nanjing University, Nanjing, China

    Xuelian Pan & Weina Hua

  2. Jiangsu Key Laboratory of Data Engineering and Knowledge Service, Nanjing, China

    Xuelian Pan

  3. College of Computing and Informatics, Drexel University, 3141 Chestnut Street, Philadelphia, PA, 19104-2875, USA

    Erjia Yan

Authors
  1. Xuelian Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erjia Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weina Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erjia Yan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, X., Yan, E. & Hua, W. Disciplinary differences of software use and impact in scientific literature. Scientometrics 109, 1593–1610 (2016). https://doi.org/10.1007/s11192-016-2138-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-2138-4

Keywords

Search

Navigation