Measuring the scientific impact of e-research infrastructures: a citation based approach?
- 436 Downloads
This micro-level study explores the extent that citation analysis provides an accurate and representative assessment of the use and impact of bioinformatics e-research infrastructure. The bioinformatic e-research infrastructure studied offers common tools used by life scientists to analyse and interpret genetic and protein sequence information. These e-resources therefore provide an interesting example with which to explore how representative citations are as acknowledgements of knowledge in the life sciences. The examples presented here suggest that there is a relation between number of visits to these databases and number of citations; however, a parallel finding shows how citation analysis frequently underestimates acknowledged use of the resources offered on this e-research infrastructure. The paper discusses the implications of the findings for various aspects of impact measurement and also considers how appropriate citation analysis is as a measurement of knowledge claims.
KeywordsCitation analysis Research infrastructure Evaluation Bioinformatics
A shorter version of this paper was presented at the ISSI 2013 conference in Vienna (Jonkers et al. 2013) and at the IWBBIO 2013 conference in Granada. The Spanish Ministry of Economics and Competitiveness funded the project of which this paper forms part through the grant: CSO2011-23508. The first three researchers also received funding from the Ramón y Cajal programme (MINECO) the JAE-DOC programme (CSIC) and the Juan de la Cierva programme (MINECO) of the Spanish Ministry of Economics and Competiveness and the Spanish Research Council (CSIC). The last author acknowledges the EC funded ViBRANT project (Grant RI-261532). SIB Swiss Institute of Bioinformatics allowed for the use of the server web log data used for part of this analysis. We would also like to thank Felix de Moya Anegón for introducing us to the NEXTBIO application “section search” and Isidro Aguillo for advice on the use of Quest’s Funnelweb software. Researchers at the Centre for Science and Technology Studies of Leiden University (NL) provided stimulating ideas in discussions during a research stay of one of the authors. The usual disclaimer applies with respect to those contributions.
The first author worked on this article at the CSIC institute for Public Goods and Policies, but he has currently taken up work at the European Commission. The information and views set out in this chapter do not necessarily reflect the opinion of the first author’s current employer. The European Commission does not guarantee the accuracy of the data included in this study. Neither his current employer nor any person acting on its behalf may be held responsible for the use which may be made of the information contained herein.
Conflict of interest
- Appel, R. D., Bairoch, A., Hochstrasser, D. F. (1994). A new generation of information retrieval tools for biologists: the example of the ExPASy WWW server. Trends in Biochemical Sciences, 19(6), 258–260.Google Scholar
- Ball, R., & Tunger, D. (2006). Science indicators revisited–Science Citation Index versus Scopus: A bibliometric comparison of both citation databases. Journal Information Services and Use, 26, 293–301.Google Scholar
- ExPASy. (2012). http://www.expasy.org/proteomics. Accessed June 2012.
- Fingerman, S. (2006). Web of science and scopus: Current features and capabilities. Issues in Science and Technology Librarianship. doi: 10.5062/F4G44N7B.
- Garfield, E. (1975). The Obliteration Phenomenon. Current Contents, 51(52), 5–7.Google Scholar
- Gorraiz, J., & Schlögl, C. (2007). Comparison of two counting houses in the field of pharmacology and pharmacy. In Proceedings of the international conference of the international society for scientometrics and informetrics, 1 (pp. 854–855).Google Scholar
- Jonkers, K., Derrick, GE, Lopez-Illescas, C., Van den Besselaar, P. (2013) Are citations a complete measure for the usage of e-research infrastructures. In J. Gorraiz, & E. Schiebel et al. (Eds.), Proceedings ISSI 2013 (pp. 136–151). Vienna.Google Scholar
- Lima, T., Auchincloss, A. H., Coudert, E., Keller, G., Michoud, K., Rivoire, C., et al. (2009). HAMAP: a database of completely sequenced microbial proteome sets and manually curated microbial protein families in UniProtKB/Swiss-Prot. Nucleic Acids Research, 37(1), D471–D478. doi: 10.1093/nar/gkn661.CrossRefGoogle Scholar
- Lokker, C., Haynes, R. B., Chu, R., McKibbon, K. A., Wilczynski, N. L., & Walter, S. D. (2012). How well are journal and clinical article characteristics associated with the journal impact factor? A retrospective cohort study. Journal of the Medical Library Association, 100(1), 28–33.CrossRefGoogle Scholar
- Lowry, O. H., Rosebrough, N. J., Farr, A. L., & Randall, R. J. (1951). Protein measurement with the folin phenol reagent. Journal of Biological Chemistry, 193, 265–275.Google Scholar
- Merton, R. K. (1965). On the shoulders of giants: a Shandean postscript (pp. 218–219). New York: Harcourt Brace and World.Google Scholar
- NEXTBIO. (2012). Section search. http://www.applications.sciverse.com/action/appDetail/293416. Accessed June/October 2012.
- Peritz, B. C. (1983). Are methodological papers more cited than theoretical or empirical ones? The case of sociology. Scientometrics, 5(4), 211–218.Google Scholar
- Quest. (2010). Funnel Web Analyzer®—overview. http://www.quest.com/funnel-web-analyzer/index.asp. Accessed June 2012.
- Rehn, C., & Kronman, U. (2008). Bibliometric handbook for Karolinska Institutet V1.05 http://ki.se/content/1/c6/01/79/31/bibliometric_handbook_karolinska_institutet_v_1.05.pdf. Accessed June 2012.
- Reuters, T. (2012). http://thomsonreuters.com. Accessed June/October 2012.
- Scopus. (2012). http://www.scopus.com. Accessed June/October 2012.
- Sigrist, C.J.A., de Castro E, Cerutti, L., Cuche, B.A., Hulo, N., Bridge, A., Bougueleret, L., Xenarios, I. (2012). New and continuing developments at PROSITE, Nucleic Acids Research. doi: 10.1093/nar/gks1067.