The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects

Abstract

Researchers are used to being evaluated: publications, hiring, tenure and funding decisions are all based on the evaluation of research. Traditionally, this evaluation relied on judgement of peers but, in the light of limited resources and increased bureaucratization of science, peer review is getting more and more replaced or complemented with bibliometric methods. Central to the introduction of bibliometrics in research evaluation was the creation of the Science Citation Index (SCI) in the 1960s, a citation database initially developed for the retrieval of scientific information. Embedded in this database was the Impact Factor, first used as a tool for the selection of journals to cover in the SCI, which then became a synonym for journal quality and academic prestige. Over the last 10 years, this indicator became powerful enough to influence researchers’ publication patterns in so far as it became one of the most important criteria to select a publication venue. Regardless of its many flaws as a journal metric and its inadequacy as a predictor of citations on the paper level, it became the go-to indicator of research quality and was used and misused by authors, editors, publishers and research policy makers alike. The h-index, introduced as an indicator of both output and impact combined in one simple number, has experienced a similar fate, mainly due to simplicity and availability. Despite their massive use, these measures are too simple to capture the complexity and multiple dimensions of research output and impact. This chapter provides an overview of bibliometric methods, from the development of citation indexing as a tool for information retrieval to its application in research evaluation, and discusses their misuse and effects on researchers’ scholarly communication behavior.

References

  1. Abt HA (1992) Publication practices in various sciences. Scientometrics 24(3):441–447. doi:10.1007/BF02051040 CrossRefGoogle Scholar
  2. Adam D (2002) The counting house. Nature 415(6873):726–729CrossRefGoogle Scholar
  3. Aksnes DW (2003) A macro study of self-citation. Scientometrics 56:235–246CrossRefGoogle Scholar
  4. Archambault E, Larivière V (2009) History of the journal impact factor: contingencies and consequences. Scientometrics 79(3):635–649. doi:10.1007/s11192-007-2036-x CrossRefGoogle Scholar
  5. Bergstrom CT (2007) Eigenfactor: measuring the value and prestige of scholarly journals. Coll Res Libr News 68(5):314–316Google Scholar
  6. Bhattacharjee Y (2011) Saudi universities offer cash in exchange for academic prestige. Science 334(6061):1344–1345CrossRefGoogle Scholar
  7. Biagioli M (2003) Rights or rewards? Changing frameworks of scientific authorship. In: Biagioli M, Galison P (eds) Scientific authorship: credit and intellectual property in science. Routledge, New York, pp 253–279Google Scholar
  8. Birnholtz J (2006) What does it mean to be an author? The intersection of credit, contribution and collaboration in science. J Am Soc Inf Sci Technol 57(13):1758–1770CrossRefGoogle Scholar
  9. Bornmann L, Mutz R (2011) Further steps towards an ideal method of measuring citation performance: the avoidance of citation (ratio) averages in field-normalization. J Informetrics 5(1):228–230. doi:10.1016/j.joi.2010.10.009 CrossRefGoogle Scholar
  10. Bradford SC (1934) Sources of information on specific subjects. Engineering 137:85–86. doi:10.1177/016555158501000407 Google Scholar
  11. Burton RE, Kebler RW (1960) The half-life of some scientific and technical literatures. Am Doc 11(1):18–22. doi:10.1002/asi.5090110105 CrossRefGoogle Scholar
  12. Calver MC, Bradley JS (2009) Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field? Scientometrics 81(3):611–615. doi:10.1007/s11192-008-2229-y CrossRefGoogle Scholar
  13. Cameron BD (2005) Trends in the usage of ISI bibliometric data: uses, abuses, and implications. Portal Libr Acad 5(1):105–125. doi:10.1353/pla.2005.0003 CrossRefGoogle Scholar
  14. Cole JR, Cole S (1967) Scientific output and recognition: a study in the operation of the reward system in science. Am Soc Rev 32(3):377–390CrossRefGoogle Scholar
  15. Cole JR, Cole S (1973) Social stratification in science. University of Chicago Press, ChicagoGoogle Scholar
  16. Cole FJ, Eales NB (1917) The history of comparative anatomy. Part I: a statistical analysis of the literature. Sci Prog 11(43):578–596Google Scholar
  17. Costas R, van Leeuwen TN, Bordóns M (2010) Self-citations at the meso and individual levels: effects of different calculation methods. Scientometrics 82(3):517–537CrossRefGoogle Scholar
  18. De Bellis N (2009) Bibliometrics and citation analysis. From the science citation index to cybermetrics. The Scarecrow Press, LanhamGoogle Scholar
  19. De Lange C, Glänzel W (1997) Modelling and measuring multilateral coauthorship in international scientific collaboration. Part I. Development of a new model using a series expansion approach. Scientometrics 40(3):593–604CrossRefGoogle Scholar
  20. Engels TCE, Ossenblok TLB, Spruyt EHJ (2012) Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics 93(2):373–390. doi:10.1007/s11192-012-0680-2 CrossRefGoogle Scholar
  21. Favaloro EJ (2008) Measuring the quality of journals and journal articles: the impact factor tells but a portion of the story. Semin Thromb Hemost 34(1):7–25. doi:10.1055/s-2008-1066030 CrossRefGoogle Scholar
  22. Flanagin A, Carey LA, Fontanarosa PB, Phillips SG, Pace BP, Lundberg GD, Rennie D (1998) Prevalance of articles with honorary authors and ghost authors in peer-reviewed medical journals. JAMA 280(3):222–224CrossRefGoogle Scholar
  23. Garfield E (1964) “Science citation index”—a new dimension in indexing. Science 144(3619):649–654CrossRefGoogle Scholar
  24. Garfield E (1972) Citation analysis as a tool in journal evaluation. Journals can be ranked by frequency and impact of citations for science policy studies. Science 178(4060):471–479CrossRefGoogle Scholar
  25. Garfield E (1979) Citation indexing - its theory and application in science, technology, and humanities. Wiley, New YorkGoogle Scholar
  26. Garfield E (1983) Citation indexing. Its theory and application in science, technology, and humanities. ISI Press, PhiladelphiaGoogle Scholar
  27. Gingras Y, Larivière V (2011) There are neither “king” nor “crown” in scientometrics: comments on a supposed “alternative” method of normalization. J Informetrics 5(1):226–227CrossRefGoogle Scholar
  28. Glänzel W, Thjis B (2004) The influence of author self-citations on bibliometric macro indicators. Scientometrics 59(3):281–310CrossRefGoogle Scholar
  29. Glänzel W, Debackere K, Thijs B, Schubert A (2006) A concise review on the role of author self-citations in information science, bibliometrics and science policy. Scientometrics 67:263–277CrossRefGoogle Scholar
  30. Glänzel W, Schubert A, Thijs B, Debackere K (2011) A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking. Scientometrics 87(2):415–424. doi:10.1007/s11192-011-0345-6 CrossRefGoogle Scholar
  31. Gonzalez-Pereira B, Guerrero-Bote VP, de Moya-Anegon F (2010) A new approach to the metric of journals’ scientific prestige: the SJR indicator. J Informetrics 4(3):379–391. doi:10.1016/j.joi.2010.03.002 CrossRefGoogle Scholar
  32. Gross PLK, Gross EM (1927) College libraries and chemical education. Science 66(1713):385–389. doi:10.1126/science.66.1713.385 CrossRefGoogle Scholar
  33. Haustein S (2012) Multidimensional journal evaluation. Analyzing scientific periodicals beyond the impact factor. De Gruyter Saur, BerlinCrossRefGoogle Scholar
  34. Hemmingsson A, Mygind T, Skjennald A, Edgren J (2002) Manipulation of impact factors by editors of scientific journals. Am J Roentgenol 178(3):767–767CrossRefGoogle Scholar
  35. Hicks D (2013) One size doesn’t fit all: on the co-evolution of national evaluation systems and social science publishing. Confero 1(1):67–90CrossRefGoogle Scholar
  36. Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 102(46):16569–16572. doi:10.1073/pnas.0507655102 CrossRefGoogle Scholar
  37. Hvistendahl M (2013) China’s publication bazaar. Science 342(6162):1035–1039. doi:10.1126/science.342.6162.1035 CrossRefGoogle Scholar
  38. Larivière V, Gingras Y (2010) On the prevalence and scientific impact of duplicate publications in different scientific fields (1980–2007). J Doc 66(2):179–190CrossRefGoogle Scholar
  39. Larivière V, Gingras Y (2011) Averages of ratios vs. ratios of averages: an empirical analysis of four levels of aggregation. J Informetrics 5(3):392–399. doi:10.1016/j.joi.2011.02.001 CrossRefGoogle Scholar
  40. Larivière V, Gingras Y, Archambault E (2006) Canadian collaboration networks: a comparative analysis of the natural sciences, social sciences and the humanities. Scientometrics 68(3):519–533CrossRefGoogle Scholar
  41. Larivière V, Archambault É, Gingras Y (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900–2004). J Am Soc Inf Sci Technol 59(2):288–296CrossRefGoogle Scholar
  42. Leydesdorff L (2009) How are new citation-based journal indicators adding to the bibliometric toolbox? J Am Soc Inf Sci Technol 60(7):1327–1336. doi:10.1002/asi.21024 CrossRefGoogle Scholar
  43. Leydesdorff L, Opthof T (2010) Normalization at the field level: fractional counting of citations. J Informetrics 4(4):644–646. doi:10.1016/j.joi.2010.05.003 CrossRefGoogle Scholar
  44. Lotka AJ (1926) The frequency distribution of scientific productivity. J Wash Acad Sci 16(12):317–323Google Scholar
  45. Lozano GA, Larivière V, Gingras Y (2012) The weakening relationship between the impact factor and papers’ citations in the digital age. J Am Soc Inf Sci Technol 63(11):2140–2145CrossRefGoogle Scholar
  46. Martin BR (2013) Whither research integrity? Plagiarism, self-plagiarism and coercive citation in an age of research assessment. Res Policy 42:1005–1014. doi:10.1016/j.respol.2013.03.011 CrossRefGoogle Scholar
  47. Martin BR, Irvine J (1983) Assessing basic research. Some partial indicators of scientific progress in radio astronomy. Res Policy 12:61–90CrossRefGoogle Scholar
  48. Martyn J, Gilchrist A (1968) An evaluation of British scientific journals. Aslib, LondonGoogle Scholar
  49. Merton RK (1973) The sociology of science: theoretical and empirical investigations. University of Chicago Press, ChicagoGoogle Scholar
  50. Merton RK (1977) The sociology of science: an episodic memoir. In: Merton RK, Gaston J (eds) The sociology of science in Europe. Southern Illinois University Press, Carbondale, pp 3–141Google Scholar
  51. Merton RK (1988) The Matthew effect in science. II. Cummulative advantage and the symbolism of intellectual property. ISIS 79:606–623CrossRefGoogle Scholar
  52. Moed HF (2002) The impact-factors debate: the ISI’s uses and limits. Nature 415(6873):731–732CrossRefGoogle Scholar
  53. Moed HF (2005) Citation analysis in research evaluation. Springer, DordrechtGoogle Scholar
  54. Moed HF (2010) Measuring contextual citation impact of scientific journals. J Informetrics 4(3):265–277. doi:10.1016/j.joi.2010.01.002 CrossRefGoogle Scholar
  55. Moed HF, van Leeuwen TN (1995) Improving the accuracy of institute for scientific information’s journal impact factors. J Am Soc Inf Sci 46(6):461–467CrossRefGoogle Scholar
  56. Norris M, Oppenheim C (2003) Citation counts and the research assessment exercise V: archaeology and the 2001 research assessment exercise. J Doc 59(6):709–730CrossRefGoogle Scholar
  57. Otlet P (1934) Traité de documentation: le livre sur le livre, théorie et pratique. Editiones Mundaneum, BrusselsGoogle Scholar
  58. Pontille D (2004) La Signature Scientifique: une Sociologie Pragmatique de l’Attribution. CNRS Éditions, ParisGoogle Scholar
  59. Price DJDS (1963) Little science, big science. Columbia University Press, New YorkGoogle Scholar
  60. Price DDS (1981) Letter to the editor: multiple authorship. Science 212(4498):984–1066CrossRefGoogle Scholar
  61. Pritchard A (1969) Statistical bibliography or bibliometrics? J Doc 25(4):348–349Google Scholar
  62. Rinia EJ, van Leeuwen TN, van Vuren HG, van Raan AFJ (1998) Comparative analysis of a set of bibliometric indicators and central peer review criteria. Evaluation of condensed matter physics in the Netherlands. Res Policy 27:95–107CrossRefGoogle Scholar
  63. Rogers LF (2002) Impact factor: the numbers game. Am J Roentgenol 178(3):541–542. doi:10.2214/ajr.178.3.1780541 CrossRefGoogle Scholar
  64. Rowlands I, Nicholas D (2005) New journal publishing models – an international survey of senior researchers. CIBER, University College London, London. Available from: http://www.homepages.ucl.ac.uk/~uczciro/pa_stm_final_report.pdf
  65. Schubert A, Braun T (1996) Cross-field normalization of scientometric indicators. Scientometrics 36(3):311–324CrossRefGoogle Scholar
  66. Seglen PO (1992) The skewness of science. J Am Soc Inf Sci 43(9):628–638CrossRefGoogle Scholar
  67. Seglen PO (1997a) Citations and journal impact factors: questionable indicators of research quality. Allergy 52(11):1050–1056CrossRefGoogle Scholar
  68. Seglen PO (1997b) Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079):498–502. doi:10.1136/bmj.314.7079.497 CrossRefGoogle Scholar
  69. Smalheiser NR, Torvik VI (2009) Author name disambiguation. Annu Rev Inf Sci Technol 43:1–43. doi:10.1002/aris.2009.1440430113 CrossRefGoogle Scholar
  70. Small H (1987) The significance of bibliographic references. Scientometrics 8:321–340CrossRefGoogle Scholar
  71. Snyder H, Bonzi S (1998) Patterns of self-citation across disciplines. J Inf Sci 24(6):431–435CrossRefGoogle Scholar
  72. The PLoS Medicine editors (2006) The impact factor game—it is time to find a better way to assess the scientific literature. PLoS Med 3(6):e291. doi:10.1371/journal.pmed.0030291
  73. Tijssen RJW, Visser MS, van Leeuwen TN (2002) Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics 54(3):381–397CrossRefGoogle Scholar
  74. Todorov R, Glänzel W (1988) Journal citation measures—a concise review. J Inf Sci 14(1):47–56. doi:10.1177/016555158801400106 CrossRefGoogle Scholar
  75. Van Norden R (2013) Brazilian citation scheme outed. Thomson Reuters suspends journals from its rankings for ‘citation stacking’. Nature 500(7464):510–511CrossRefGoogle Scholar
  76. Waltman L, van Eck NJ (2010) The relation between Eigenfactor, audience factor, and influence weight. J Am Soc Inf Sci Technol 61(7):1476–1486. doi:10.1002/asi.21354 CrossRefGoogle Scholar
  77. Waltman L, van Eck NJ (2012) The inconsistency of the h-index. J Am Soc Inf Sci Technol 63(2):406–415. doi:10.1002/asi.21678 CrossRefGoogle Scholar
  78. Weingart P (2005) Impact of bibliometrics upon the science system: inadvertent consequences? Scientometrics 62(1):117–131. doi:10.1007/s11192-005-0007-7 CrossRefGoogle Scholar
  79. Weller K (2015) Social media and altmetrics: an overview of current alternative approaches to measuring scholarly impact. In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M (eds) Incentives and performance - governance of research organizations. Springer, ChamGoogle Scholar
  80. Wislar JS, Flanagin A, Fontanarosa PB, DeAngelis CD (2011) Honorary and ghost authorship in high impact biomedical journals: a cross sectional survey. BMJ 343:6128. doi:10.1136/bmj.d6128 CrossRefGoogle Scholar
  81. Zipf GK (1949) Human behaviour and the principle of least effort. Addison-Wesley, Reading, MAGoogle Scholar
  82. Zitt M (2010) Citing-side normalization of journal impact: a robust variant of the audience factor. J Informetrics 4(3):392–406CrossRefGoogle Scholar
  83. Zuckerman H (1987) Citation analysis and the complex problem of intellectual influence. Scientometrics 12:329–338CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.École de bibliothéconomie et des sciences de l’information (EBSI)Université de MontréalMontréalCanada
  2. 2.École de bibliothéconomie et des sciences de l’information (EBSI)Université de Montréal, Observatoire des sciences et des technologies (OST), Centre interuniversitaire de recherche sur la science et la technologie (CIRST), Université du Québec à MontréalMontréalCanada

Personalised recommendations