Skip to main content
SpringerLink
Log in

Beyond Publication Counts – The Impact of Citations and Combined Metrics on the Performance Measurement of German Business Researchers

  • Chapter
Performance Management im Hochschulbereich

Part of the book series: ZfB-Special Issues ((ZFB,volume 3/2013))

  • 1387 Accesses

Abstract

This paper investigates the effects of going beyond publication counts on the relative performance measurement of German-speaking business administration scholars. Based on data from the Social Science Citation Index, Scopus, and Google Scholar it compares rankings based on publication, citation, and combined measures, such as the h-index. The results from 298 accounting and marketing scholars show that the move from publication to citation counts is a greater step than from citations to the h-indices. A similar observation can be made with respect to refinements of the h-index. We investigate several causes of these effects and show that citation counts and combined measures specify the information content of data sources. The results also suggest that data source coverage is a larger driver of differences than measures. Finally, we find that correlations between rankings based on different data sources can be improved by extending beyond publication measures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Adler NJ, Harzing A-W (2009) When knowledge wins: transcending the sense and nonsense of academic rankings. Acad of Manag Learning & Educ 8(1):72–85

    Article  Google Scholar 

  • Aksnes DW, Taxt RE (2004) Peer reviews and bibliometric indicators: a comparative study at a Norwegian university. Res Eval 13(1):33–41

    Article  Google Scholar 

  • Albers S (2009) Misleading rankings of research in business. German Econ Rev 10(3):352–363

    Article  Google Scholar 

  • Alonso S, Cabrerizo FJ, Herrera-Viedma E, Herrera F (2009) H-Index: a review focused in its variants, computation and standardization for different scientific fields. J Informetrics 3(4):273–289

    Article  Google Scholar 

  • Anonymous (2005) Data point. Science 309(5738):1181

    Google Scholar 

  • ARC (2011) ERA 2010 citation benchmark methodology, http://www.arc.gov.au/pdf/era_2010_citation_benchmark_methods.pdf. Accessed 28 December 2011

  • Backhaus K, Erichson B, Plinke W, Weiber R (2011) Multivariate Analysemethoden, 13th Edition, Berlin: Springer

    Book  Google Scholar 

  • Ball P (2005) Index aims for fair ranking of scientists. Nature 436(7053):900

    Article  Google Scholar 

  • Bar-Ilan J (2008) Which h-index? a comparison of WoS, Scopus and Google Scholar. Scientometrics 74(2):257–271

    Article  Google Scholar 

  • Bartneck C, Kokkelmans S (2011) Detecting h-index manipulation through self-citation analysis. Scientometrics 87(1):85–98

    Article  Google Scholar 

  • Batista PD, Campiteli MG, Kinouchi O (2006) Is it possible to compare researchers with different scientific interests? Scientometrics 68(1):179–189

    Article  Google Scholar 

  • Baum JAC (2011) Free-Riding on power laws: questioning the validity of the impact factor as a measure of research quality in organization studies. Organization 18(4):449–466

    Article  Google Scholar 

  • Beattie V, Goodacre A (2004) Publishing patterns within the UK accounting and finance academic community. Brit Account Rev 36(1):7–44

    Article  Google Scholar 

  • Bornmann L, Daniel HD (2009) The state of h-index research: is the h-index the ideal way to measure research performance? EMBO reports 10(1):2–6

    Article  Google Scholar 

  • Bornmann L, Mutz R, Daniel HD (2008a) Are there better indices for evaluation purposes than the h-index? a comparison of nine different variants of the h-index using data from biomedicine. J Am Soc Inform Sci Tech 59(5):830–837

    Article  Google Scholar 

  • Bornmann L, Mutz R, Hug SE, Daniel HD (2011) A multilevel meta-analysis of studies reporting correlations between the h-index and 37 different h-index variants. J Informetrics 5(3):346–359

    Article  Google Scholar 

  • Bornmann L, Mutz R, Neuhaus C, Daniel HD (2008b) Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Sci Environ Polit 8(1):93–102

    Article  Google Scholar 

  • Bosman J, van Mourik I, Rasch M, Sieverts E, Verhoeff H (2006) Scopus reviewed and compared: the coverage and functionality of the citation database Scopus, including comparisons with Web of Science and Google Scholar, http://igitur-archive.library.uu.nl/DARLIN/2006-1220-200432/Scopus%20doorgelicht%20&%20vergeleken%20-%20translated.pdf

  • Braun T, Glänzel W, Schubert A (2006) A Hirsch-type index for journals. Scientometrics 69(1): 169–173

    Article  Google Scholar 

  • Breuer W (2009) Google Scholar as a means for quantitative evaluation of German research output in business administration: some preliminary results, Working Paper RWTH Aachen, http://ssrn.com/abstract=1280033

  • Brown LD, Gardner JC (1985) Using citation analysis to assess the impact of journals and articles on Contemporary Accounting Research (CAR). J Account Res 23(1):84–109

    Article  Google Scholar 

  • Butler L (2003) Explaining Australia’s increased share of ISI publications: the effects of a funding formula based on publication counts. Res Pol 32(1):143–155

    Article  Google Scholar 

  • Carmona S, Gutierrez I, Camara M (1999) A profile of European accounting research: evidence from leading research journals. Eur Account Rev 8(3):463–480

    Article  Google Scholar 

  • Clermont M (2010) Publish or Perish als Auswertungstool für Google Scholar: Aufbau und Auswertungsmöglichkeiten. Wirtschaftswiss Stud 39(9):418–425

    Google Scholar 

  • Clermont M, Dyckhoff H (2012) Coverage of Business Administration Literature in Google Scholar: Analysis and Comparison with EconBiz, Scopus and Web of Science, Bibliometrie –Praxis und Forschung 1(5):165–5

    Google Scholar 

  • Clermont M, Schmitz C (2008) Erfassung betriebswirtschaftlich relevanter Zeitschriften in der ISI-Datenbaken sowie der Scopus-Datenbank. Z Betriebswirtschaft 78(10):987–1009

    Article  Google Scholar 

  • Cole J, Cole S (1971) Measuring the quality of sociological research: problems in the use of the “Science Citation Index”. Am Sociol 6(1):23–29

    Google Scholar 

  • Conover WJ (1980) Practical non-parametric statistics, 2nd Edition, New York: John Wiley and Sons

    Google Scholar 

  • Costas R, Bordons M (2008) Is g-index better than h-index? an exploratory study at the individual level. Scientometrics 77(2):267–288

    Article  Google Scholar 

  • Cronin B (1981) The need for a theory of citing. J Document 37(1):16–24

    Article  Google Scholar 

  • Cronin B, Atkins HB, Eds. (2000) The web of knowledge: a festschrift in honor of Eugene Garfield, Medford, NJ: American Society for Information Sciences

    Google Scholar 

  • Cronin B, Meho L (2006) Using the h-index to rank influential information scientists. J Am Soc Inform Sci Tech 57(9):1275–1278

    Article  Google Scholar 

  • Demski JS, Feltham GA (1976) Cost determination: a conceptual approach, Ames: Iowa State University Press

    Google Scholar 

  • Dilger A (2010) Rankings von Zeitschriften und Personen in der BWL. Z Manag 5(1):91–102

    Article  Google Scholar 

  • Dorta-González P, Dorta-González M-I (2011) Central indexes to the citation distribution: a complement to the h-index. Scientometrics 88(3):729–745

    Article  Google Scholar 

  • Dyckhoff H, Schmitz C (2007) Forschungsleistungsmessung mittels SSCI oder SCI-X? Internationale Sichtbarkeit und Wahrnehmung der Betriebswirtschaftslehre von 1990 bis 2004. Betriebswirtschaft 67(6):638–662

    Google Scholar 

  • Dyckhoff H, Thieme A, Schmitz C (2005) Die Wahrnehmung deutschsprachiger Hochschullehrer für Betriebswirtschaft in der internationalen Forschung: Eine Pilotstudie zu Zitationsverfahren und möglichen Einflussfaktoren. Betriebswirtschaft 65(4):350–372

    Google Scholar 

  • Egghe L (2006) Theory and practice of the g-index. Scientometrics 69(1):131–152

    Article  Google Scholar 

  • Egghe L, Rousseau R (2006) An informetric model for the Hirsch-index. Scientometrics 69(1): 121–129

    Article  Google Scholar 

  • Franceschet M (2009) A cluster analysis of scholar and journal bibliometric indicators. J Am Soc Inform Sci Tech 60(10):1950–1964

    Article  Google Scholar 

  • Frey BS, Osterloh M (2006) Evaluations: hidden costs, questionable benefits, and superior alternatives. IEW Working Paper 302

    Google Scholar 

  • Frey BS, Rost K (2010) Do rankings reflect research quality? J App Econ 13(1):1–38

    Article  Google Scholar 

  • Fülbier RU, Weller M (2011) A glance at German financial accounting research between 1950 and 2005: a publication and citation analysis. Schmalenbach Bus Rev 63(1):2–33

    Google Scholar 

  • Garfield E (1979) Is citation analysis a legitimate evaluation tool? Scientometrics 1(4):359–375

    Article  Google Scholar 

  • Garvey WD, Griffith BC (1967) Scientific communication as a social system. Science 157(3792):1011–1016

    Article  Google Scholar 

  • Glänzel W (2006) On the opportunities and limitations of the h-index. Science Focus 1(1):10–11

    Google Scholar 

  • Harzing A-W (2011a) Publish or perish, available at www.harzing.com/pop.htm

  • Harzing A-W (2011b) The publish or perish book: your guide to effective and responsible citation analysis, Melbourne: Tarma Software Research Pty Ltd

    Google Scholar 

  • Harzing A-W, van der Wal R (2008) Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit 8(1):62–71

    Google Scholar 

  • Hasselback JR, Reinstein A, Schwan ES (2000) Benchmarks for evaluating the research productivity of accounting faculty. J Account Educ 18(2):79–97

    Article  Google Scholar 

  • Heck JL, Jensen RE, Cooley PL (1990) An analysis of contributors to accounting journals. part I: the aggregate performances. Intern J Account 25(3):202–217

    Google Scholar 

  • HEFCE (2011) Assessment framework and guidance on submissions, http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf

  • Heiber H (1983) Messung von Forschungsleistung der Hochschulen: Ein empirischer Ansatz auf Basis von Zitatenanalysen, Baden-Baden: Nomos

    Google Scholar 

  • Henrekson M, Waldenström D (2011) How should research performance be measured? a study of Swedish econmists. The Manchester School 79(6):1139–1156

    Article  Google Scholar 

  • Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Nat Acad Sci 102(46):16569–16572

    Article  Google Scholar 

  • Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1–2):81–93

    Google Scholar 

  • Kieser A (1998) Going Dutch: Was lehren niederländische Erfahrungen mit der Evaluation universitärer Forschung? Betriebswirtschaft 58(2):208–224

    Google Scholar 

  • Lebo D (1958) Publication versus communication. Science 128(3321):424–425

    Article  Google Scholar 

  • LSE (2011) Maximizing the impacts of your research: a handbook for social scientists (Consultation Draft 3), London

    Google Scholar 

  • MacRoberts MH, MacRoberts BR (1989) Problems of citation analysis: a critical review. J Am Soc Inform Sci Tech 40(5):342–349

    Article  Google Scholar 

  • Martin BR, Irvine J (1983) Assessing basic research: some partial indicators of scientific progress in radio astronomy. Res Pol 12(2):61–90

    Article  Google Scholar 

  • Meho LI, Rogers Y (2008) Citation counting, citation ranking, and h-index of human-computer interaction researchers: a comparison of Scopus and Web of Science. J Am Soc Inform Sci Tech 59(11):1711–1726

    Article  Google Scholar 

  • Meho LI, Yang K (2007) Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J Am Soc Inform Sci Tech 58(13):2105–2125

    Article  Google Scholar 

  • Melin G, Persson O (1998) Hotel cosmopolitan: a bibliometric study of collaboration at some European universities. J Am Soc Inform Sci Tech 49(1):43–48

    Article  Google Scholar 

  • Meyer M, Schäffer U, Gmür M (2008) Transfer und Austausch von Wissen in der Accounting-Forschung: Eine Zitations- und Kozitationsanalyse englischsprachiger Accounting-Journals 1990-2004. Z betriebswirtschaft Forsch 50(2):153–181

    Google Scholar 

  • Meyer M, Waldkirch RW, Zaggl MA (2012) Relative performance measurement of researchers: the impact of data source selction, Schmalenbach Bus Rev 64(3):308–330

    Google Scholar 

  • Moed HF, Visser MS (2008) Appraisal of citation data sources: a report to HEFCE by the Centre for Science and Technology Studies, Leiden University, http://www.hefce.ac.uk/pubs/rdreports/2008/rd17_08/

  • Müller A, Storbeck O (2009) BWL-Ranking: Methodik und Interpretation, Handelsblatt, http://www.handelsblatt.com/politik/bwl-ranking/bwl-ranking-methodik-und-interpretation;2175006. Accessed 21 July 2009

  • Müller H (2010) Wie valide ist das Handesblatt-BWL-Ranking: Zeitschriften und zitationsbasierte Personenrankings im Vergleich. Betriebswirtschaft Forsch Prax 62(2):152–166

    Google Scholar 

  • Noether GE (1981) Why Kendall tau Teaching Statistics 3(2):41–43

    Article  Google Scholar 

  • Panaretos J, Malesios C (2009) Assessing scientific research performance and impact with single indices. Scientometrics 81(3):635–670

    Article  Google Scholar 

  • Persson O, Glänzel W, Danell R (2004) Inflationary bibliometric values: the role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics 60(3): 421–432

    Article  Google Scholar 

  • Prathap G (2010) Is there a place for a mock h-index? Scientometrics 84(1):153–165

    Article  Google Scholar 

  • Rao IKR (2007) Distributions of Hirsch-index and g-index: an empirical study. In: Torres-Salinas, D, Moed, HF (eds) Proceedings of ISSI 2007, Madrid, CSIC, pp 655–658

    Google Scholar 

  • Rassenhövel S (2010) Performancemessung im Hochschulbereich: Theoretische Grundlagen und empirische Befunde, Wiesbaden: Gabler Verlag

    Book  Google Scholar 

  • Rassenhövel S, Dyckhoff H (2007) Die Relevanz von Drittmittelindikatoren bei der Beurteilung der Forschungsleistung im Hochschulbereich. In: Zelewski, S, Akca, N (eds) Fortschritt in den Wirtschaftswissenschaften, Wiesbaden: Gabler, pp 85–112

    Google Scholar 

  • Rosenstreich D, Wooliscroft B (2009) Measuring the impact of accounting journals using Google Scholar and the g-index. Brit Account Rev 41(4):227–239

    Article  Google Scholar 

  • Rost K, Frey BS (2011) Quantative and qualitative rankings of scholars. Schmalenbach Bus Rev 63(1):63–91

    Google Scholar 

  • Saad G (2006) Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics 69(1):117–120

    Article  Google Scholar 

  • Sanderson M (2008) Revisiting h measured on UK LIS and IR academics. J Am Soc Inform Sci Tech 59(7):1184–1190

    Article  Google Scholar 

  • Schreiber M (2008) An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the a-index, and the r-index. J Am Soc Inform Sci Tech 59(9):1513–1522

    Article  Google Scholar 

  • Sidiropoulos A, Katsaros D, Manolopoulos Y (2007) Generalized Hirsch-index for disclosing latent facts in citation networks. Scientometrics 72(2):253–280

    Article  Google Scholar 

  • Smith R (1997) Journal accused of manipulating impact factor. Brit Med J 314(7079):461

    Article  Google Scholar 

  • van Raan AFJ (2004) Measuring science: capita selecta of current main issues. In: Moed, HF, Glänzel, W, Schmoch, U (eds) Handbook of quantitative science and technology research, Dordrecht: Kluwer Academic Publisher, pp 19–50

    Google Scholar 

  • van Raan AFJ (2006a) Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 67(3):491–502

    Google Scholar 

  • van Raan AFJ (2006b) Statistical properties of bibliometric indicators: research group indicator distributions and correlations. J Am Soc Inform Sci Tech 57(3):408–430

    Article  Google Scholar 

  • Vanclay JK (2007) On the robustness of the h-index. J Am Soc Inform Sci Tech 58(10):1547–1550

    Article  Google Scholar 

  • Vieira E, Gomes J (2011) An impact indicator for researchers. Scientometrics 89(2):607–629

    Article  Google Scholar 

  • Voeth M, Herbst U, Loos J (2011) Bibliometrische Analyse der Zeitschriftenrankings VHB-JOURQUAL 2.1 und Handelsblatt-Zeitschriftenranking BWL am Beispiel des Untersuchungsfeldes Marketing. Die Betriebswirtschaft 71(5):439–458

    Google Scholar 

  • Walter A (2011) The effects of coauthorship on the quality of financial research papers. Z Betriebswirtsch 81(2):205–234

    Article  Google Scholar 

  • Weber J, Schäffer U (2011) Einführung in das Controlling, 13th Edition, Stuttgart: Schäffer- Poeschel Verlag

    Google Scholar 

  • Wolf J, Rohn A, Macharzina K (2005) Institution und Forschungsproduktivität: Befunde und Interpretationen aus der deutschsprachigen Betriebswirtschaftslehre. Betriebswirtschaft 65(1):62–77

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. South Westphalia University of Applied Sciences, Lindenstr. 53, 59872, Meschede, Germany

    Prof. Dr. Rüdiger W. Waldkirch

  2. Institute of Management Control and Accounting, Hamburg University of Technology, Schwarzenbergstr. 95, 21073, Hamburg, Germany

    Prof. Dr. Matthias Meyer & Dr. Michael A. Zaggl

Authors
  1. Prof. Dr. Rüdiger W. Waldkirch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prof. Dr. Matthias Meyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dr. Michael A. Zaggl
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institut für Organisationsökonomik, Münster, Westfälische Wilhelms-Universität Münste, Münster

    Alexander Dilger

  2. RWTH Aachen Lehrstuhl Unternehmenstheorie, Aachen

    Harald Dyckhoff

  3. Zentrum für Produktionswirtschaft, FernUniversität Hagen, Hagen

    Günter Fandel

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Gabler | Springer Fachmedien Wiesbaden

About this chapter

Cite this chapter

Waldkirch, R., Meyer, M., Zaggl, M. (2013). Beyond Publication Counts – The Impact of Citations and Combined Metrics on the Performance Measurement of German Business Researchers. In: Dilger, A., Dyckhoff, H., Fandel, G. (eds) Performance Management im Hochschulbereich. ZfB-Special Issues, vol 3/2013. Springer Gabler, Wiesbaden. https://doi.org/10.1007/978-3-658-03348-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-658-03348-4_4

  • Publisher Name: Springer Gabler, Wiesbaden

  • Print ISBN: 978-3-658-03347-7

  • Online ISBN: 978-3-658-03348-4

  • eBook Packages: Business and Economics (German Language)

Publish with us

Policies and ethics

Search

Navigation