Scientometrics

, Volume 91, Issue 3, pp 751–771 | Cite as

Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

Article

Abstract

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.

Keywords

Successive h-index Percentile h-index Excellence for Research in Australia (ERA) 

References

  1. Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: from informed peer review to Bibliometrics. Scientometrics, 87, 499–514.CrossRefGoogle Scholar
  2. Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206–215.CrossRefGoogle Scholar
  3. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-Index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273–289.CrossRefGoogle Scholar
  4. Anon. (2011). List of forestry universities and colleges. Wikipedia, http://en.wikipedia.org/wiki/List_of_forestry_universities_and_colleges [August 5, 2011].
  5. ANZSRC (2008). Australian and New Zealand Standard Research Classification. http://www.abs.gov.au/Ausstats/abs@.nsf/0/6BB427AB9696C225CA2574180004463E [August 5, 2011].
  6. ARC (2010a). The Excellence in Research for Australia (ERA) Initiative. http://www.arc.gov.au/era/default.htm [August 5, 2011].
  7. ARC (2010b). Ranked Journal List. Australian Research Council. http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls [August 5, 2011].
  8. ARC (2011a). What is the change to the journal indicator? Australian Research Council. http://www.arc.gov.au/era/faq.htm#2012 [August 5, 2011].
  9. ARC (2011b). ERA 2010 National Report. Australian Research Council, p. 320. http://www.arc.gov.au/pdf/ERA_report.pdf [August 5, 2011].
  10. ARC (2011c). ERA 2010 citation benchmark methodology. Australian Research Council, p. 21. http://www.arc.gov.au/pdf/era_2010_citation_benchmark_methods.pdf [August 5, 2011].
  11. ARC (2011d). ERA 2010 results by field of research code. Australian Research Council. http://www.arc.gov.au/era/outcomes_2010/FoRindex [August 5, 2011].
  12. Arencibia-Jorge, R., Barrios-Almaguer, I., Fernández-Hernández, S., & Carvajal-Espino, R. (2008). Successive H indices and its applying in the institutional evaluation: a case study. Journal of the American Society for Information Science and Technology, 59(1), 155–157.CrossRefGoogle Scholar
  13. Atkinson, R. & McLoughlin, C. (2011). Dawn of a new ERA? Australasian Journal of Educational Technology, 27(3), iii–viii. http://ascilite.org.au/ajet/ajet27/editorial27-3.html.
  14. Bornmann, L. (2010). Mimicry in science? Scientometrics, 86(1), 173–177.CrossRefGoogle Scholar
  15. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.Google Scholar
  16. Bornmann, L., de Moya Anegón, F., & Leydesdorff, L. (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5(10), e13327. doi: 10.1371/journal.pone.0013327.CrossRefGoogle Scholar
  17. Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from Biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.CrossRefGoogle Scholar
  18. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011a). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5, 346–359.CrossRefGoogle Scholar
  19. Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Does the h index for assessing single publications really work? A case study on papers published in chemistry. Scientometrics, 89(3), 835–843.Google Scholar
  20. Carr, K. (2009). A message from the Minister. Australian Research Council, Discovery Newsletter, Autumn 2009. http://www.arc.gov.au/rtf/discovery_autumn09.rtf.
  21. Cooper, S., & Poletti, A. (2011). The new ERA of journal ranking. Australian Universities’ Review, 53, 57–65.Google Scholar
  22. Donovan, C., & Butler, L. (2007). Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: An economics pilot study. Research Evaluation, 16(4), 231–242.CrossRefGoogle Scholar
  23. Egghe, L. (2008). Modelling successive h-indices. Scientometrics, 77(3), 377–387.CrossRefGoogle Scholar
  24. Egghe, L. (2011). The single publication H-index and the indirect H-index of a researcher. Scientometrics, 88(3), 1003–1004.CrossRefGoogle Scholar
  25. Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. The FASEB Journal, 22, 338–342.CrossRefGoogle Scholar
  26. Haddow, G., & Genoni, P. (2010). Citation analysis and peer ranking of Australian social science journals. Scientometrics, 85, 471–487.CrossRefGoogle Scholar
  27. Hicks, D. (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57, 393–404.CrossRefGoogle Scholar
  28. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102, 16569–16572.CrossRefGoogle Scholar
  29. Jones, D. (2011). Excellence in Research for Australia—a new ERA. Research Trends 23, 10–11. http://www.researchtrends.com/wp-content/uploads/2011/06/Research_Trends_Issue23.pdf.
  30. Laband, D. N., & Zhang, D. (2006). Citations, publications, and perceptions-based rankings of the research impact of North American forestry programs. Journal of Forestry, 104(5), 162–254.Google Scholar
  31. Labbé, C. (2010). Ike Antkare, one of the great stars in the scientific firmament. ISSI Newsletter, 6(2), 48–52.Google Scholar
  32. Lamp, J. (2011). 2010 finalised journals in an ANZ Field of Research: 0705 Forestry Sciences. http://lamp.infosys.deakin.edu.au/era/?page=jfordet10&selfor=0705 [5 August 2011].
  33. Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis one more time: principles for comparing sets of documents. Journal of the American Society of Information Science and Technology, 62(7), 1370–1381.CrossRefGoogle Scholar
  34. Li, J., Burnham, J. F., Lemley, T., & Britton, R. M. (2010). Citation Analysis: Comparison of Web of Science, Scopus, SciFinder, and Google Scholar. Journal of Electronic Resources in Medical Libraries, 7(3), 196–217.CrossRefGoogle Scholar
  35. Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58, 2105–2125.CrossRefGoogle Scholar
  36. Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et therapiae Experimentalis, 57, 13–18.CrossRefGoogle Scholar
  37. Moosa, I. (2011). The demise of the ARC journal ranking scheme: an ex post analysis of the accounting and finance journals. Accounting & Finance, 51(3), 809–836.CrossRefGoogle Scholar
  38. Moya-Anegón, F. D., Chinchilla-Rodríguez, Z., Vargas-Quesada, B., Corera-Álvarez, E., Muñoz-Fernández, F. J., González-Molina, A., et al. (2007). Coverage analysis of Scopus: A journal metric approach. Scientometrics, 73, 53–78.CrossRefGoogle Scholar
  39. Norris, M., & Oppenheim, C. (2010). The h-index: a broad review of a new bibliometric indicator. Journal of Documentation, 66(5), 681–705.CrossRefGoogle Scholar
  40. Northcott, D., & Linacre, S. (2010). Producing spaces for academic discourse: The impact of research assessment exercises and journal quality rankings. Australian Accounting Review, 20, 38–54.CrossRefGoogle Scholar
  41. Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation, 15, 347–360.CrossRefGoogle Scholar
  42. Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 6. doi: 10.1103/PhysRevE.83.046116.
  43. Rousseau, R., Yang, L., & Yue, T. (2010). A discussion of Prathap’s h2-index for institutional evaluation with an application in the field of HIV infection and therapy. Journal of Informetrics, 4(2), 175–184.CrossRefGoogle Scholar
  44. Ruane, F., & Tol, R. S. J. (2008). Rational (successive) h-indices: An application to economics in the Republic of Ireland. Scientometrics, 75(2), 395–405.CrossRefGoogle Scholar
  45. Runeson, G. (2011). The demise of the journal ranking: a victory for common sense. Australasian Journal of Construction Economics and Building, 11(2), 99–100.Google Scholar
  46. Schreiber, M. (2010). Twenty Hirsch index variants and other indicators giving more or less preference to highly cited papers. Annalen der Physik, 522(8), 536–554.MathSciNetCrossRefGoogle Scholar
  47. Schubert, A. (2007). Successive h-indices. Scientometrics, 70(1), 201–205.CrossRefGoogle Scholar
  48. Schubert, A. (2009). Using the h-index for assessing single publications. Scientometrics, 78(3), 559–565.CrossRefGoogle Scholar
  49. Serenko, A., & Dohan, M. (2011). Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence. Journal of Informetrics, 5(4), 629–648.CrossRefGoogle Scholar
  50. Sinclair, R. (2010). RD&E strategy for the forest and wood products sector. Australia: Forest and Wood Products, p. 57. www.daff.gov.au/__data/assets/word_doc/0010/1770751/forest.doc.
  51. Singh, G., Haddad, K. M., & Chow, C. W. (2007). Are articles in “top” management journals necessarily of higher quality? Journal of Management Inquiry, 16(4), 319–331.CrossRefGoogle Scholar
  52. Slyder, J. B., Stein, B. R., Sams, B. S., Walker, D. M., Beale, B. J., Feldhaus, J. J., et al. (2011). Citation pattern and lifespan: A comparison of discipline, institution and individual. Scientometrics, 89(3), 955–966.CrossRefGoogle Scholar
  53. Svantesson, D. J. B., & White, P. (2009). Entering an era of research ranking—will innovation and diversity survive? Bond Law Review, 21(3), 7.Google Scholar
  54. Taylor, J. (2011). The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22, 202–217.CrossRefGoogle Scholar
  55. Thor, A. & Bornmann, L. (2011). The calculation of the single publication h index and related performance measures: A web application based on Google Scholar data. Online Information Review, 35(2), 291–300. (Utility available here: http://labs.dbs.uni-leipzig.de/gsh/).
  56. Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2, 326–334.CrossRefGoogle Scholar
  57. Vanclay, J. K. (2011). An evaluation of the Australian Research Council’s journal ranking. Journal of Informetrics, 5, 265–274.CrossRefGoogle Scholar
  58. Vanclay, J. K. (2012). What was wrong with Australia’s journal ranking? Journal of Informetrics, 6, 53–54.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  1. 1.Southern Cross UniversityLismoreAustralia
  2. 2.Max Planck SocietyAdministrative HeadquartersMunichGermany

Personalised recommendations