Advertisement

Beyond Publication Counts – The Impact of Citations and Combined Metrics on the Performance Measurement of German Business Researchers

  • Rüdiger W. Waldkirch
  • Matthias Meyer
  • Michael A. Zaggl
Chapter
Part of the ZfB-Special Issues book series (ZFB, volume 3/2013)

Abstract

This paper investigates the effects of going beyond publication counts on the relative performance measurement of German-speaking business administration scholars. Based on data from the Social Science Citation Index, Scopus, and Google Scholar it compares rankings based on publication, citation, and combined measures, such as the h-index. The results from 298 accounting and marketing scholars show that the move from publication to citation counts is a greater step than from citations to the h-indices. A similar observation can be made with respect to refinements of the h-index. We investigate several causes of these effects and show that citation counts and combined measures specify the information content of data sources. The results also suggest that data source coverage is a larger driver of differences than measures. Finally, we find that correlations between rankings based on different data sources can be improved by extending beyond publication measures.

Keywords

Citations h-index Performance Measurement Publications Ranking 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adler NJ, Harzing A-W (2009) When knowledge wins: transcending the sense and nonsense of academic rankings. Acad of Manag Learning & Educ 8(1):72–85CrossRefGoogle Scholar
  2. Aksnes DW, Taxt RE (2004) Peer reviews and bibliometric indicators: a comparative study at a Norwegian university. Res Eval 13(1):33–41CrossRefGoogle Scholar
  3. Albers S (2009) Misleading rankings of research in business. German Econ Rev 10(3):352–363CrossRefGoogle Scholar
  4. Alonso S, Cabrerizo FJ, Herrera-Viedma E, Herrera F (2009) H-Index: a review focused in its variants, computation and standardization for different scientific fields. J Informetrics 3(4):273–289CrossRefGoogle Scholar
  5. Anonymous (2005) Data point. Science 309(5738):1181Google Scholar
  6. ARC (2011) ERA 2010 citation benchmark methodology, http://www.arc.gov.au/pdf/era_2010_citation_benchmark_methods.pdf. Accessed 28 December 2011
  7. Backhaus K, Erichson B, Plinke W, Weiber R (2011) Multivariate Analysemethoden, 13th Edition, Berlin: SpringerCrossRefGoogle Scholar
  8. Ball P (2005) Index aims for fair ranking of scientists. Nature 436(7053):900CrossRefGoogle Scholar
  9. Bar-Ilan J (2008) Which h-index? a comparison of WoS, Scopus and Google Scholar. Scientometrics 74(2):257–271CrossRefGoogle Scholar
  10. Bartneck C, Kokkelmans S (2011) Detecting h-index manipulation through self-citation analysis. Scientometrics 87(1):85–98CrossRefGoogle Scholar
  11. Batista PD, Campiteli MG, Kinouchi O (2006) Is it possible to compare researchers with different scientific interests? Scientometrics 68(1):179–189CrossRefGoogle Scholar
  12. Baum JAC (2011) Free-Riding on power laws: questioning the validity of the impact factor as a measure of research quality in organization studies. Organization 18(4):449–466CrossRefGoogle Scholar
  13. Beattie V, Goodacre A (2004) Publishing patterns within the UK accounting and finance academic community. Brit Account Rev 36(1):7–44CrossRefGoogle Scholar
  14. Bornmann L, Daniel HD (2009) The state of h-index research: is the h-index the ideal way to measure research performance? EMBO reports 10(1):2–6CrossRefGoogle Scholar
  15. Bornmann L, Mutz R, Daniel HD (2008a) Are there better indices for evaluation purposes than the h-index? a comparison of nine different variants of the h-index using data from biomedicine. J Am Soc Inform Sci Tech 59(5):830–837CrossRefGoogle Scholar
  16. Bornmann L, Mutz R, Hug SE, Daniel HD (2011) A multilevel meta-analysis of studies reporting correlations between the h-index and 37 different h-index variants. J Informetrics 5(3):346–359CrossRefGoogle Scholar
  17. Bornmann L, Mutz R, Neuhaus C, Daniel HD (2008b) Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Sci Environ Polit 8(1):93–102CrossRefGoogle Scholar
  18. Bosman J, van Mourik I, Rasch M, Sieverts E, Verhoeff H (2006) Scopus reviewed and compared: the coverage and functionality of the citation database Scopus, including comparisons with Web of Science and Google Scholar, http://igitur-archive.library.uu.nl/DARLIN/2006-1220-200432/Scopus%20doorgelicht%20&%20vergeleken%20-%20translated.pdf
  19. Braun T, Glänzel W, Schubert A (2006) A Hirsch-type index for journals. Scientometrics 69(1): 169–173CrossRefGoogle Scholar
  20. Breuer W (2009) Google Scholar as a means for quantitative evaluation of German research output in business administration: some preliminary results, Working Paper RWTH Aachen, http://ssrn.com/abstract=1280033
  21. Brown LD, Gardner JC (1985) Using citation analysis to assess the impact of journals and articles on Contemporary Accounting Research (CAR). J Account Res 23(1):84–109CrossRefGoogle Scholar
  22. Butler L (2003) Explaining Australia’s increased share of ISI publications: the effects of a funding formula based on publication counts. Res Pol 32(1):143–155CrossRefGoogle Scholar
  23. Carmona S, Gutierrez I, Camara M (1999) A profile of European accounting research: evidence from leading research journals. Eur Account Rev 8(3):463–480CrossRefGoogle Scholar
  24. Clermont M (2010) Publish or Perish als Auswertungstool für Google Scholar: Aufbau und Auswertungsmöglichkeiten. Wirtschaftswiss Stud 39(9):418–425Google Scholar
  25. Clermont M, Dyckhoff H (2012) Coverage of Business Administration Literature in Google Scholar: Analysis and Comparison with EconBiz, Scopus and Web of Science, Bibliometrie –Praxis und Forschung 1(5):165–5Google Scholar
  26. Clermont M, Schmitz C (2008) Erfassung betriebswirtschaftlich relevanter Zeitschriften in der ISI-Datenbaken sowie der Scopus-Datenbank. Z Betriebswirtschaft 78(10):987–1009CrossRefGoogle Scholar
  27. Cole J, Cole S (1971) Measuring the quality of sociological research: problems in the use of the “Science Citation Index”. Am Sociol 6(1):23–29Google Scholar
  28. Conover WJ (1980) Practical non-parametric statistics, 2nd Edition, New York: John Wiley and SonsGoogle Scholar
  29. Costas R, Bordons M (2008) Is g-index better than h-index? an exploratory study at the individual level. Scientometrics 77(2):267–288CrossRefGoogle Scholar
  30. Cronin B (1981) The need for a theory of citing. J Document 37(1):16–24CrossRefGoogle Scholar
  31. Cronin B, Atkins HB, Eds. (2000) The web of knowledge: a festschrift in honor of Eugene Garfield, Medford, NJ: American Society for Information SciencesGoogle Scholar
  32. Cronin B, Meho L (2006) Using the h-index to rank influential information scientists. J Am Soc Inform Sci Tech 57(9):1275–1278CrossRefGoogle Scholar
  33. Demski JS, Feltham GA (1976) Cost determination: a conceptual approach, Ames: Iowa State University PressGoogle Scholar
  34. Dilger A (2010) Rankings von Zeitschriften und Personen in der BWL. Z Manag 5(1):91–102CrossRefGoogle Scholar
  35. Dorta-González P, Dorta-González M-I (2011) Central indexes to the citation distribution: a complement to the h-index. Scientometrics 88(3):729–745CrossRefGoogle Scholar
  36. Dyckhoff H, Schmitz C (2007) Forschungsleistungsmessung mittels SSCI oder SCI-X? Internationale Sichtbarkeit und Wahrnehmung der Betriebswirtschaftslehre von 1990 bis 2004. Betriebswirtschaft 67(6):638–662Google Scholar
  37. Dyckhoff H, Thieme A, Schmitz C (2005) Die Wahrnehmung deutschsprachiger Hochschullehrer für Betriebswirtschaft in der internationalen Forschung: Eine Pilotstudie zu Zitationsverfahren und möglichen Einflussfaktoren. Betriebswirtschaft 65(4):350–372Google Scholar
  38. Egghe L (2006) Theory and practice of the g-index. Scientometrics 69(1):131–152CrossRefGoogle Scholar
  39. Egghe L, Rousseau R (2006) An informetric model for the Hirsch-index. Scientometrics 69(1): 121–129CrossRefGoogle Scholar
  40. Franceschet M (2009) A cluster analysis of scholar and journal bibliometric indicators. J Am Soc Inform Sci Tech 60(10):1950–1964CrossRefGoogle Scholar
  41. Frey BS, Osterloh M (2006) Evaluations: hidden costs, questionable benefits, and superior alternatives. IEW Working Paper 302Google Scholar
  42. Frey BS, Rost K (2010) Do rankings reflect research quality? J App Econ 13(1):1–38CrossRefGoogle Scholar
  43. Fülbier RU, Weller M (2011) A glance at German financial accounting research between 1950 and 2005: a publication and citation analysis. Schmalenbach Bus Rev 63(1):2–33Google Scholar
  44. Garfield E (1979) Is citation analysis a legitimate evaluation tool? Scientometrics 1(4):359–375CrossRefGoogle Scholar
  45. Garvey WD, Griffith BC (1967) Scientific communication as a social system. Science 157(3792):1011–1016CrossRefGoogle Scholar
  46. Glänzel W (2006) On the opportunities and limitations of the h-index. Science Focus 1(1):10–11Google Scholar
  47. Harzing A-W (2011a) Publish or perish, available at www.harzing.com/pop.htm
  48. Harzing A-W (2011b) The publish or perish book: your guide to effective and responsible citation analysis, Melbourne: Tarma Software Research Pty LtdGoogle Scholar
  49. Harzing A-W, van der Wal R (2008) Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit 8(1):62–71Google Scholar
  50. Hasselback JR, Reinstein A, Schwan ES (2000) Benchmarks for evaluating the research productivity of accounting faculty. J Account Educ 18(2):79–97CrossRefGoogle Scholar
  51. Heck JL, Jensen RE, Cooley PL (1990) An analysis of contributors to accounting journals. part I: the aggregate performances. Intern J Account 25(3):202–217Google Scholar
  52. HEFCE (2011) Assessment framework and guidance on submissions, http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf
  53. Heiber H (1983) Messung von Forschungsleistung der Hochschulen: Ein empirischer Ansatz auf Basis von Zitatenanalysen, Baden-Baden: NomosGoogle Scholar
  54. Henrekson M, Waldenström D (2011) How should research performance be measured? a study of Swedish econmists. The Manchester School 79(6):1139–1156CrossRefGoogle Scholar
  55. Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Nat Acad Sci 102(46):16569–16572CrossRefGoogle Scholar
  56. Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1–2):81–93Google Scholar
  57. Kieser A (1998) Going Dutch: Was lehren niederländische Erfahrungen mit der Evaluation universitärer Forschung? Betriebswirtschaft 58(2):208–224Google Scholar
  58. Lebo D (1958) Publication versus communication. Science 128(3321):424–425CrossRefGoogle Scholar
  59. LSE (2011) Maximizing the impacts of your research: a handbook for social scientists (Consultation Draft 3), LondonGoogle Scholar
  60. MacRoberts MH, MacRoberts BR (1989) Problems of citation analysis: a critical review. J Am Soc Inform Sci Tech 40(5):342–349CrossRefGoogle Scholar
  61. Martin BR, Irvine J (1983) Assessing basic research: some partial indicators of scientific progress in radio astronomy. Res Pol 12(2):61–90CrossRefGoogle Scholar
  62. Meho LI, Rogers Y (2008) Citation counting, citation ranking, and h-index of human-computer interaction researchers: a comparison of Scopus and Web of Science. J Am Soc Inform Sci Tech 59(11):1711–1726CrossRefGoogle Scholar
  63. Meho LI, Yang K (2007) Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J Am Soc Inform Sci Tech 58(13):2105–2125CrossRefGoogle Scholar
  64. Melin G, Persson O (1998) Hotel cosmopolitan: a bibliometric study of collaboration at some European universities. J Am Soc Inform Sci Tech 49(1):43–48CrossRefGoogle Scholar
  65. Meyer M, Schäffer U, Gmür M (2008) Transfer und Austausch von Wissen in der Accounting-Forschung: Eine Zitations- und Kozitationsanalyse englischsprachiger Accounting-Journals 1990-2004. Z betriebswirtschaft Forsch 50(2):153–181Google Scholar
  66. Meyer M, Waldkirch RW, Zaggl MA (2012) Relative performance measurement of researchers: the impact of data source selction, Schmalenbach Bus Rev 64(3):308–330Google Scholar
  67. Moed HF, Visser MS (2008) Appraisal of citation data sources: a report to HEFCE by the Centre for Science and Technology Studies, Leiden University, http://www.hefce.ac.uk/pubs/rdreports/2008/rd17_08/
  68. Müller A, Storbeck O (2009) BWL-Ranking: Methodik und Interpretation, Handelsblatt, http://www.handelsblatt.com/politik/bwl-ranking/bwl-ranking-methodik-und-interpretation;2175006. Accessed 21 July 2009
  69. Müller H (2010) Wie valide ist das Handesblatt-BWL-Ranking: Zeitschriften und zitationsbasierte Personenrankings im Vergleich. Betriebswirtschaft Forsch Prax 62(2):152–166Google Scholar
  70. Noether GE (1981) Why Kendall tau Teaching Statistics 3(2):41–43CrossRefGoogle Scholar
  71. Panaretos J, Malesios C (2009) Assessing scientific research performance and impact with single indices. Scientometrics 81(3):635–670CrossRefGoogle Scholar
  72. Persson O, Glänzel W, Danell R (2004) Inflationary bibliometric values: the role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics 60(3): 421–432CrossRefGoogle Scholar
  73. Prathap G (2010) Is there a place for a mock h-index? Scientometrics 84(1):153–165CrossRefGoogle Scholar
  74. Rao IKR (2007) Distributions of Hirsch-index and g-index: an empirical study. In: Torres-Salinas, D, Moed, HF (eds) Proceedings of ISSI 2007, Madrid, CSIC, pp 655–658Google Scholar
  75. Rassenhövel S (2010) Performancemessung im Hochschulbereich: Theoretische Grundlagen und empirische Befunde, Wiesbaden: Gabler VerlagCrossRefGoogle Scholar
  76. Rassenhövel S, Dyckhoff H (2007) Die Relevanz von Drittmittelindikatoren bei der Beurteilung der Forschungsleistung im Hochschulbereich. In: Zelewski, S, Akca, N (eds) Fortschritt in den Wirtschaftswissenschaften, Wiesbaden: Gabler, pp 85–112Google Scholar
  77. Rosenstreich D, Wooliscroft B (2009) Measuring the impact of accounting journals using Google Scholar and the g-index. Brit Account Rev 41(4):227–239CrossRefGoogle Scholar
  78. Rost K, Frey BS (2011) Quantative and qualitative rankings of scholars. Schmalenbach Bus Rev 63(1):63–91Google Scholar
  79. Saad G (2006) Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics 69(1):117–120CrossRefGoogle Scholar
  80. Sanderson M (2008) Revisiting h measured on UK LIS and IR academics. J Am Soc Inform Sci Tech 59(7):1184–1190CrossRefGoogle Scholar
  81. Schreiber M (2008) An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the a-index, and the r-index. J Am Soc Inform Sci Tech 59(9):1513–1522CrossRefGoogle Scholar
  82. Sidiropoulos A, Katsaros D, Manolopoulos Y (2007) Generalized Hirsch-index for disclosing latent facts in citation networks. Scientometrics 72(2):253–280CrossRefGoogle Scholar
  83. Smith R (1997) Journal accused of manipulating impact factor. Brit Med J 314(7079):461CrossRefGoogle Scholar
  84. van Raan AFJ (2004) Measuring science: capita selecta of current main issues. In: Moed, HF, Glänzel, W, Schmoch, U (eds) Handbook of quantitative science and technology research, Dordrecht: Kluwer Academic Publisher, pp 19–50Google Scholar
  85. van Raan AFJ (2006a) Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 67(3):491–502Google Scholar
  86. van Raan AFJ (2006b) Statistical properties of bibliometric indicators: research group indicator distributions and correlations. J Am Soc Inform Sci Tech 57(3):408–430CrossRefGoogle Scholar
  87. Vanclay JK (2007) On the robustness of the h-index. J Am Soc Inform Sci Tech 58(10):1547–1550CrossRefGoogle Scholar
  88. Vieira E, Gomes J (2011) An impact indicator for researchers. Scientometrics 89(2):607–629CrossRefGoogle Scholar
  89. Voeth M, Herbst U, Loos J (2011) Bibliometrische Analyse der Zeitschriftenrankings VHB-JOURQUAL 2.1 und Handelsblatt-Zeitschriftenranking BWL am Beispiel des Untersuchungsfeldes Marketing. Die Betriebswirtschaft 71(5):439–458Google Scholar
  90. Walter A (2011) The effects of coauthorship on the quality of financial research papers. Z Betriebswirtsch 81(2):205–234CrossRefGoogle Scholar
  91. Weber J, Schäffer U (2011) Einführung in das Controlling, 13th Edition, Stuttgart: Schäffer- Poeschel VerlagGoogle Scholar
  92. Wolf J, Rohn A, Macharzina K (2005) Institution und Forschungsproduktivität: Befunde und Interpretationen aus der deutschsprachigen Betriebswirtschaftslehre. Betriebswirtschaft 65(1):62–77Google Scholar

Copyright information

© Springer Gabler | Springer Fachmedien Wiesbaden 2013

Authors and Affiliations

  • Rüdiger W. Waldkirch
    • 1
  • Matthias Meyer
    • 2
  • Michael A. Zaggl
    • 2
  1. 1.South Westphalia University of Applied SciencesMeschedeGermany
  2. 2.Institute of Management Control and AccountingHamburg University of TechnologyHamburgGermany

Personalised recommendations