A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar
- 1k Downloads
Author-level bibliometric indicators are becoming a standard tool in research assessment. It is important to investigate what these indicators actually measure to assess their appropriateness in scholar ranking and benchmarking average individual levels of performance. 17 author-level indicators were calculated for 512 researchers in Astronomy, Environmental Science, Philosophy and Public Health. Indicator scores and scholar rankings calculated in Web of Science (WoS) and Google Scholar (GS) were analyzed. The indexing policies of WoS and GS were found to have a direct effect on the amount of available bibliometric data, thus indicator scores and rankings in WoS and GS were different, correlations between 0.24 and 0.99. High correlation could be caused by scholars in bottom rank positions with a low number of publications and citations in both databases. The hg indicator produced scholar rankings with the highest level of agreement between WoS and GS and rankings with the least amount of variance. Expected average performance benchmarks were influenced by how the mean indicator value was calculated. Empirical validation of the aggregate mean h-index values compared to previous studies resulted in a very poor fit of predicted average scores. Rankings based on author-level indicators are influenced by (1) the coverage of papers and citations in the database, (2) how the indicators are calculated and, (3) the assessed discipline and seniority. Indicator rankings display the visibility of the scholar in the database not their impact in the academic community compared to their peers. Extreme caution is advised when choosing indicators and benchmarks in scholar rankings.
KeywordsAuthor-level bibliometrics Ranking Bibliometric evaluation Indicator properties Harmonic mean Arithmetic mean
I would like to thank Dr. Jesper W. Schneider and Dr. Kim Wildgaard, M.D. for their feedback on an earlier draft of this paper and the reviewers for their suggestions for revisions. I also acknowledge the ACUMEN collaboration for allowing the continued use of the dataset throughout the final year of my Ph.D.-project. ACUMEN was a European FP7 project completed in Spring 2014, Grant Agreement 266632.
- Archambault, E., & Gagné, E. V. (2004). The Use of Bibliometrics in Social Sciences and Humanities (Social Sciences and Humanities Research Council of Canada, SSHRCC), Montreal: Science Metrix. http://www.science-metrix.com/pdf/SM_2004_008_SSHRC_Bibliometrics_Social_Science.pdf. Accessed 30 March 2015.
- Berk, R. A., & Freedman, D. A. (2003). Statistical assumptions as empirical commitments. In T. G. Blomberg & S. Cohen (Eds.), Punishment and social control (pp. 235–254). New York: Walter de Gruyter.Google Scholar
- Bornmann, L., & Marx, W. (2011). The h-index as a research performance indicator. European Science Editing, 37(3), 77–80.Google Scholar
- Bornmann, L., & Marx, W. (2013). Evaluating individual researchers’ performance. Euorpean Science Editing, 39(2), 30–40.Google Scholar
- Connor, J. (2011). Google scholar citations open to all. Resource document. http://googlescholar.blogspot.dk/2011/11/google-scholar-citations-open-to-all.html. Accessed 30 March 2015.
- Courtault, J. M., & Hayek, N. (2008). On the robustness of the h-index: A mathematical approach. Economics Bulletin, 3(78), 1–9.Google Scholar
- ESO. (2011). Some statistics on recent job hires in Astronomy: Who is getting the jobs and how do you compare? Science News. Resource document. https://www.eso.org/wiki/pub/Science/ScienceNewsLetter/ScienceNews-May-Jun2011.pdf. Accessed 30 March 2015.
- Farhadi, F., Salehi, H., Yunus, M. M., Chadegani, A. A., Farhabi, M., Fooladi, M., et al. (2013). Does it matter which citation tool is used to compare the h-index of a group of highly cited researchers? Australian Journal of Basic and Applied Sciences, 7(4), 198–202.Google Scholar
- Fiorenzo, F., Domenico, M., & Luca, M. (2013). The effect of dirty data on h-index calculation. Scientometrics, 95(3), 1179–1188.Google Scholar
- Freedman, D. A., Pisani, R., & Purves, R. (2007). Statistics (4th ed.). New York, NY: W.W. Norton & Company.Google Scholar
- Gorard, S. (2003). Quantitative methods in social science: the role of numbers made easy. London: Continuum.Google Scholar
- Gratton, R. (2014). Evaluation of Italian atronomical production: 2010–2012. Resource document. arXiv: arXiv:1402.4080v1 [astro-ph.IM]. Accessed 30 March 2015.Google Scholar
- Harzing, A. (2008). Google Scholar—A new data source for citation analysis. Resource document, http://www.harzing.com/pop_gs.htm. Accessed 30 March 2015.
- Harzing, A. (2012). Publish or Perish user’s manual. Resource document, http://www.harzing.com/pop.htm#metrics. Accessed 30 March 2015.
- Haustein, S., & Larivière, V. (2015). The use of bibliometrics for assessing research: Possibilities, limitations and adverse effects. In I. M. Welpe, J. Wollersheim, S. Ringelhan, & M. Osterloh (Eds.), Incentives and performance: Governance of research organizations (pp. 121–139). Switzerland: Springer International Publishing.Google Scholar
- Jacsó, P. (2005). As we may search? Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89(9), 1537–1547.Google Scholar
- Kamphuis, P., & van der Kruit, P. C. (2010). Citations and impact of Dutch astronomy. Resource document arXiv:1011.5311v1 [astro-ph.IM]. Accessed 30 March 2015.Google Scholar
- Kosmoloulos, C., & Pumain, D. (2014). Citation, citation, citation: Bibliometrics, the web and the Social Sciences and Humanities. Resource document, Cybergeo: European Journal of Geography. http://www.cybergeo.eu/index15463.html. Accessed 30 March 2015.
- Kosmulski, M. (2006). A new type Hirsch-index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3), 4–6.Google Scholar
- Meera, B. M., & Manjunath, M., (2012). h-index of astrophysicists at raman research institute: performance of different calculators, Working Group on Libraries, IAU Division XII Commission 5, In L. Utley (Ed.) Special session 15: Data-intensive astronomy, Beijing, China, 23-24 August 2012. http://iau-commission5.wikispaces.com/LibWG+Beijing+2012. Accessed 30 March 2015.
- Miller, C. W., (2006). Superiority of the h-index over the impact factor for physics. Resource document, arXiv: physics/0608183 [physics.soc-ph]. Accessed 30 March 2015.Google Scholar
- Redner, S. (2010). On the meaning of the h-index. Journal of Statistical Mechanics: Theory and Experiment. doi:10.1088/1742-5468/2010/03/L03005.
- Roediger, H. L. (2006). The h index in Science: A new measure of scholarly contribution. Resource document, APS Observer: The Academic Observer. http://www.psychologicalscience.org/index.php/publications/observer/2006/april-06/the-h-index-in-science-a-new-measure-of-scholarly-contribution.html. Accessed 30 March 2015.
- Testa, J. (2012). The Thomson Reuters Journal Selection Process. Resource document, http://wokinfo.com/essays/journal-selection-process/. Accessed 30 March 2015.
- Wilson, V. (2007). A content analysis of Google Scholar: Coverage varies by discipline and by database. Evidence Based Library and Information Practice, 2(1), 134–136.Google Scholar
- Yang, K., & Meho, L. I. (2006). Citation analysis: A comparison of Google Scholar, Scopus, and Web of Science., 2006. In 69th annual meeting of the American Society for Information Science and Technology (ASIST), Austin (US), 3–8 November 2006. http://eprints.rclis.org/8605/1/Yang_citation.pdf. Accessed 9 April 2015.