Skip to main content
SpringerLink
Log in

De Profundis: A Decade of Bibliometric Services Under Scrutiny

  • Chapter
  • First Online:
Evaluative Informetrics: The Art of Metrics-Based Research Assessment

Abstract

Giving kudos to bibliometric legend Henk Moed is most appropriately achieved by putting his evaluative concepts and approaches into action. The Department for Bibliometrics and Publication Strategies at the University of Vienna can meanwhile draw on rich experience with research evaluation tasks. These activities can certainly be perceived as repercussions of Henk Moed’s work. This is particularly true for his most recent book, Applied Evaluative Informetrics, which strongly supports the deliberate and correct use of bibliometric practices for evaluation purposes. In this paper, we present and discuss the lessons learned after having provided bibliometric services at the University of Vienna for more than a decade. Based on the comparison of our own insights with current practices, declarations and manifestos, we come up with new recommendations and guidelines. This also includes a discussion to what degree alternative metrics are either helpful for research assessment or rather pose a danger to the overall development of research practices. Last but not least we emphasize that bibliometric services should not exclusively be reduced to evaluative tasks. They should rather benefit (particularly young) researchers improving their publication strategies and enhancing their visibility within the research community and beyond.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://bibliothek.univie.ac.at/bibliometrie/en/.

  2. 2.

    https://www.scientometrics-school.eu/.

  3. 3.

    ‘Publish or Perish’ is a software programme that retrieves and analyses academic citations. It relies on GS to obtain the raw citations (see also, http://www.harzing.com/pop.htm).

  4. 4.

    Promotion strategies related to “altmetrics” are considered separately.

  5. 5.

    Of course, other journal impact measures like “Article Influence Score”, “SJR” or “SNIP” can also be used, depending on the data source (Scopus or Web of Science Core Collection).

  6. 6.

    https://www.scopus.com/sources.

  7. 7.

    https://dbh.nsd.uib.no/publiseringskanaler/erihplus/about/index.

  8. 8.

    https://www.leidenranking.com/information/indicators.

  9. 9.

    There is almost no bibliometric topic that has not been addressed by Moed in his monumental work. The dynamical aspects of science maps resulting from combined co‐citation and word analysis also drew his attention in the 1990s (Braam et al., 1991).

  10. 10.

    Report of the Independent Review of the Role of Metrics in Research Assessment and Management, http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/.

References

  • Archambault, É., Beauchesne, O. H., & Caruso, J. (2011). Towards a multilingual, comprehensive and open scientific journal ontology. In Proceedings of the 13th international conference of the international society for scientometrics and informetrics (pp. 66–77). South Africa: Durban.

    Google Scholar 

  • Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271.

    Google Scholar 

  • Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research Libraries News, 68(5), 314–316.

    Google Scholar 

  • Bergstrom, C. T., West, J. D., & Wiseman, M. A. (2008). The eigenfactor™ metrics. Journal of Neuroscience, 28(45), 11433–11434.

    Google Scholar 

  • Bollen, J., Luce, R., Vemulapalli, S. S., & Xu, W. (2003). Usage analysis for the identification of research trends in digital libraries. D-Lib Magazine, 9(5), 1082–9873.

    Google Scholar 

  • Bollen, J., & Van de Sompel, H. (2006). Mapping the structure of science through usage. Scientometrics, 69(2), 227–258.

    Google Scholar 

  • Bollen, J., Rodriguez, M. A., & Van de Sompel, H. (2007). MESUR: Usage-based metrics of scholarly impact (No. LA-UR-07-0663). Los Alamos National Lab. (LANL), Los Alamos, NM (United States).

    Google Scholar 

  • Bollen, J., & Sompel, H. V. D. (2008). Usage impact factor: The effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136–149.

    Google Scholar 

  • Bornmann, L., & Leydesdorff, L. (2014). Scientometrics in a changing research landscape. EMBO Reports, 15(12), 1228–1232.

    Google Scholar 

  • Braam, R. R., Moed, H. F., & Van Raan, A. F. (1991). Mapping of science by combined co‐citation and word analysis. II: Dynamical aspects. Journal of the American Society for Information Science, 42(4), 252–266.

    Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

    Google Scholar 

  • Costas, R., van Leeuwen, T., & Bordons, M. (2010). Self-citations at the meso and individual levels: effects of different calculation methods. Scientometrics, 82(3), 517–537.

    Google Scholar 

  • Costas, R., & van Leeuwen, T. N. (2012). Approaching the “reward triangle”: General analysis of the presence of funding acknowledgments and “peer interactive communication” in scientific publications. Journal of the American Society for Information Science and Technology, 63(8), 1647–1661.

    Google Scholar 

  • De Nooy, W., Mrvar, A., & Batagelj, V. (2018). Exploratory social network analysis with Pajek: Revised and expanded edition for updated software (Vol. 46). Cambridge University Press.

    Google Scholar 

  • Donner, P. (2018). Effect of publication month on citation impact. Journal of Informetrics, 12(1), 330–343.

    Google Scholar 

  • Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. American Documentation, 14(3), 195–201.

    Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Google Scholar 

  • Garfield, E. (1994). The relationship between citing and cited publications: A question of relatedness. Current Contents, 13.

    Google Scholar 

  • Garfield, E. (2004). The agony and the ecstasy—The history and meaning of the journal impact factor. J Biol Chem, 405017(6.355), 6585.

    Google Scholar 

  • Glänzel, W., & Moed, H. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

    Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V., & Moya-Anegon, F. (2009). The SJR indicator: A new indicator of journals’ scientific prestige. arXiv preprint arXiv:0912.4141.

  • Gorraiz, J., & Gumpenberger, C. (2010). Going beyond Citations: SERUM—A new tool provided by a network of libraries. Liber Quarterly, 20(1), 80–93.

    Google Scholar 

  • Gorraiz, J., Purnell, P. J., & Glänzel, W. (2013). Opportunities for and limitations of the B ook C itation I ndex. Journal of the American Society for Information Science and Technology, 64(7), 1388–1398.

    Google Scholar 

  • Gorraiz, J., Gumpenberger, C., & Schlögl, C. (2014). Usage versus citation behaviours in four subject areas. Scientometrics, 101(2), 1077–1095.

    Google Scholar 

  • Gorraiz, J., & Gumpenberger, C. (2015). A flexible bibliometric approach for the assessment of professorial appointments. Scientometrics, 105(3), 1699–1719.

    Google Scholar 

  • Gorraiz, J., Wieland, M., & Gumpenberger, C. (2016). Individual bibliometric assessment@ University of Vienna: From numbers to multidimensional profiles. El Profesional de la Informacion, 25(6), 901–915.

    Google Scholar 

  • Gorraiz, J., Wieland, M., & Gumpenberger, C. (2017). To be visible, or not to be, that is the question. International Journal of Social Science and Humanity, 7(7), 467–471.

    Google Scholar 

  • Gorraiz, J. (2018). A thousand and one reflections of the publications in the mirrors’ labyrinth of the new metrics. El profesional de la información, 27 (2), 231236. http://www.elprofesionaldelainformacion.com/contenidos/2018/mar/01.pdf.

  • Gumpenberger, C., Wieland, M., & Gorraiz, J. (2012). Bibliometric practices and activities at the University of Vienna. Library Management, 33(3), 174–183.

    Google Scholar 

  • Gumpenberger, C., Wieland, M., & Gorraiz, J. (2014). Bibliometrics and Libraries-a promising Liaison. Zeitschrift für Bibliothekswesen und Bibliographie, 61(4–5), 247–250.

    Google Scholar 

  • Gumpenberger, C., Glänzel, W., & Gorraiz, J. (2016). The ecstasy and the agony of the altmetric score. Scientometrics, 108(2), 977–982.

    Google Scholar 

  • Halevi, G., & Moed, H. F. (2014). Usage patterns of scientific journals and their relationship with citations (pp. 241–251). Context Counts: Pathways to Master Big and Little Data.

    Google Scholar 

  • Halevi, G., Moed, H. F., & Bar-Ilan, J. (2016). Does research mobility have an effect on productivity and impact? International higher education, 86, 5–6.

    Google Scholar 

  • Halevi, G., Moed, H., & Bar-Ilan, J. (2017). Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the literature. Journal of informetrics, 11(3), 823–834.

    Google Scholar 

  • Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics, 101(2), 1145–1163.

    Google Scholar 

  • Haustein, S. (2016). Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423.

    Google Scholar 

  • Kurtz, M. J., & Bollen, J. (2011). Usage bibliometrics. arXiv preprint arXiv:1102.2891.

  • Kousha, K., Thelwall, M., & Rezaie, S. (2011). Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus. Journal of the American Society for Information Science and Technology, 62(11), 2147–2164.

    Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Remaining problems with the “New Crown Indicator” (MNCS) of the CWTS. arXiv preprint arXiv:1010.2379.

  • Leydesdorff, L., & Felt, U. (2012). Edited volumes, monographs, and book chapters in the Book Citation Index (BKCI) and Science Citation Index (SCI, SoSCI, A&HCI). arXiv preprint arXiv:1204.3717.

  • Moed, H. F. (1988). The use of online databases for bibliometric analysis. In L. Egghe & R. Rousseau (eds.), Informetrics 87/88 (pp. 15–28). Elsevier Science Publishers, Amsterdam. ISBN 0-444-70425-6.

    Google Scholar 

  • Moed, H. F. (2000). Bibliometric indicators reflect publication and management strategies. Scientometrics, 47(2), 323–346.

    Google Scholar 

  • Moed, H. F. (2005a). Citation analysis of scientific journals and journal impact measures. Current Science, 1990–1996.

    Google Scholar 

  • Moed, H. F. (2005b). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology, 56(10), 1088–1097.

    Google Scholar 

  • Moed, H. F. (2006). Citation analysis in research evaluation (Vol. 9). Springer Science & Business Media.

    Google Scholar 

  • Moed, H. F. (2007a). The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review. Science and Public Policy, 34(8), 575–583.

    Google Scholar 

  • Moed, H. F. (2007b). The effect of “open access” on citation impact: An analysis of ArXiv’s condensed matter section. Journal of the American Society for Information Science and Technology, 58(13), 2047–2054.

    Google Scholar 

  • Moed, H. F. (2008). UK research assessment exercises: Informed judgments on research quality or quantity? Scientometrics, 74(1), 153–161.

    Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of informetrics, 4(3), 265–277.

    Google Scholar 

  • Moed, H. F., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A., et al. (2012). Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics, 92(2), 367–376.

    Google Scholar 

  • Moed, H. F. (2013). New perspectives on the Arts & Humanities. Research Trends, 32, 1.

    Google Scholar 

  • Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science and Technology, 66(10), 1988–2002.

    Google Scholar 

  • Moed, H. F. (2017). Applied evaluative informetrics. Springer International Publishing. ISBN: 978-3-319-60521-0

    Google Scholar 

  • Persson, O., Danell, R., & Schneider, J. W. (2009). How to use Bibexcel for various types of bibliometric analysis. Celebrating scholarly communication studies: A Festschrift for Olle Persson at his 60th Birthday, 5, 9–24.

    Google Scholar 

  • Peters, I., Kraker, P., Lex, E., Gumpenberger, C., & Gorraiz, J. I. (2017). Zenodo in the spotlight of traditional and new metrics. Frontiers in Research Metrics and Analytics, 2, 13.

    Google Scholar 

  • Pudovkin, A. I., & Garfield, E. (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53(13), 1113–1119.

    Google Scholar 

  • Repiso, R., Gumpenberger, C., Wieland, M., & Gorraiz, J. (2019). Impact measures in the humanities: A blessing or a curse? Book of Abstracts QQML 2019. http://qqml.org/wp-content/uploads/2017/09/Book-of-Abstracts_Final_AfterConf_v1.pdf

  • Robinson-Garcia, N., Sugimoto, C. R., Murray, D., Yegros-Yegros, A., Larivière, V., & Costas, R. (2019). The many faces of mobility: Using bibliometric data to measure the movement of scientists. Journal of Informetrics, 13(1), 50–63.

    Google Scholar 

  • Torres-Salinas, D., Robinson-García, N., Cabezas-Clavijo, Á., & Jiménez-Contreras, E. (2014). Analyzing the citation characteristics of books: edited books, book series and publisher types in the book citation index. Scientometrics, 98(3), 2113–2127.

    Google Scholar 

  • Torres-Salinas, D., Gumpenberger, C., & Gorraiz, J. (2017). PlumX as a potential tool to assess the macroscopic multidimensional impact of books. Frontiers in Research Metrics and Analytics, 2, 5.

    Google Scholar 

  • Van Eck, N. J., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523–538.

    Google Scholar 

  • Vinkler, P. (2004). Characterization of the impact of sets of scientific papers: The Garfield (impact) factor. Journal of the American Society for Information Science and Technology, 55(5), 431–435.

    Google Scholar 

  • Vinkler, P. (2010). The evaluation of research by scientometric indicators. Oxford [u.a.]: CP, Chandos Publishing XXI, 313 S. ISBN: 1-84334-572-2.

    Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011). Towards a new crown indicator: An empirical analysis. Scientometrics, 87(3), 467–481.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department for Bibliometrics and Publication Strategies, University of Vienna, Vienna University Library, Boltzmanngasse 5, 1090, Vienna, Austria

    Juan Gorraiz, Martin Wieland, Ursula Ulrych & Christian Gumpenberger

Authors
  1. Juan Gorraiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Wieland
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ursula Ulrych
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christian Gumpenberger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Gorraiz .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria Informatica, Automatica e Gestionale (DIAG), Sapienza University, Rome, Italy

    Cinzia Daraio

  2. Centre for R&D Monitoring (ECOOM) and Dept MSI, KU Leuven, Leuven, Belgium

    Wolfgang Glänzel

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gorraiz, J., Wieland, M., Ulrych, U., Gumpenberger, C. (2020). De Profundis: A Decade of Bibliometric Services Under Scrutiny. In: Daraio, C., Glänzel, W. (eds) Evaluative Informetrics: The Art of Metrics-Based Research Assessment . Springer, Cham. https://doi.org/10.1007/978-3-030-47665-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-47665-6_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-47664-9

  • Online ISBN: 978-3-030-47665-6

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation