Skip to main content
SpringerLink
Log in

Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: from informed peer review to Bibliometrics. Scientometrics, 87, 499–514.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206–215.

    Article  Google Scholar 

  • Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-Index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273–289.

    Article  Google Scholar 

  • Anon. (2011). List of forestry universities and colleges. Wikipedia, http://en.wikipedia.org/wiki/List_of_forestry_universities_and_colleges [August 5, 2011].

  • ANZSRC (2008). Australian and New Zealand Standard Research Classification. http://www.abs.gov.au/Ausstats/abs@.nsf/0/6BB427AB9696C225CA2574180004463E [August 5, 2011].

  • ARC (2010a). The Excellence in Research for Australia (ERA) Initiative. http://www.arc.gov.au/era/default.htm [August 5, 2011].

  • ARC (2010b). Ranked Journal List. Australian Research Council. http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls [August 5, 2011].

  • ARC (2011a). What is the change to the journal indicator? Australian Research Council. http://www.arc.gov.au/era/faq.htm#2012 [August 5, 2011].

  • ARC (2011b). ERA 2010 National Report. Australian Research Council, p. 320. http://www.arc.gov.au/pdf/ERA_report.pdf [August 5, 2011].

  • ARC (2011c). ERA 2010 citation benchmark methodology. Australian Research Council, p. 21. http://www.arc.gov.au/pdf/era_2010_citation_benchmark_methods.pdf [August 5, 2011].

  • ARC (2011d). ERA 2010 results by field of research code. Australian Research Council. http://www.arc.gov.au/era/outcomes_2010/FoRindex [August 5, 2011].

  • Arencibia-Jorge, R., Barrios-Almaguer, I., Fernández-Hernández, S., & Carvajal-Espino, R. (2008). Successive H indices and its applying in the institutional evaluation: a case study. Journal of the American Society for Information Science and Technology, 59(1), 155–157.

    Article  Google Scholar 

  • Atkinson, R. & McLoughlin, C. (2011). Dawn of a new ERA? Australasian Journal of Educational Technology, 27(3), iii–viii. http://ascilite.org.au/ajet/ajet27/editorial27-3.html.

  • Bornmann, L. (2010). Mimicry in science? Scientometrics, 86(1), 173–177.

    Article  Google Scholar 

  • Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.

    Google Scholar 

  • Bornmann, L., de Moya Anegón, F., & Leydesdorff, L. (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5(10), e13327. doi:10.1371/journal.pone.0013327.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from Biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011a). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5, 346–359.

    Article  Google Scholar 

  • Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Does the h index for assessing single publications really work? A case study on papers published in chemistry. Scientometrics, 89(3), 835–843.

    Google Scholar 

  • Carr, K. (2009). A message from the Minister. Australian Research Council, Discovery Newsletter, Autumn 2009. http://www.arc.gov.au/rtf/discovery_autumn09.rtf.

  • Cooper, S., & Poletti, A. (2011). The new ERA of journal ranking. Australian Universities’ Review, 53, 57–65.

    Google Scholar 

  • Donovan, C., & Butler, L. (2007). Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: An economics pilot study. Research Evaluation, 16(4), 231–242.

    Article  Google Scholar 

  • Egghe, L. (2008). Modelling successive h-indices. Scientometrics, 77(3), 377–387.

    Article  Google Scholar 

  • Egghe, L. (2011). The single publication H-index and the indirect H-index of a researcher. Scientometrics, 88(3), 1003–1004.

    Article  Google Scholar 

  • Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. The FASEB Journal, 22, 338–342.

    Article  Google Scholar 

  • Haddow, G., & Genoni, P. (2010). Citation analysis and peer ranking of Australian social science journals. Scientometrics, 85, 471–487.

    Article  Google Scholar 

  • Hicks, D. (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57, 393–404.

    Article  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102, 16569–16572.

    Article  Google Scholar 

  • Jones, D. (2011). Excellence in Research for Australia—a new ERA. Research Trends 23, 10–11. http://www.researchtrends.com/wp-content/uploads/2011/06/Research_Trends_Issue23.pdf.

  • Laband, D. N., & Zhang, D. (2006). Citations, publications, and perceptions-based rankings of the research impact of North American forestry programs. Journal of Forestry, 104(5), 162–254.

    Google Scholar 

  • Labbé, C. (2010). Ike Antkare, one of the great stars in the scientific firmament. ISSI Newsletter, 6(2), 48–52.

    Google Scholar 

  • Lamp, J. (2011). 2010 finalised journals in an ANZ Field of Research: 0705 Forestry Sciences. http://lamp.infosys.deakin.edu.au/era/?page=jfordet10&selfor=0705 [5 August 2011].

  • Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis one more time: principles for comparing sets of documents. Journal of the American Society of Information Science and Technology, 62(7), 1370–1381.

    Article  Google Scholar 

  • Li, J., Burnham, J. F., Lemley, T., & Britton, R. M. (2010). Citation Analysis: Comparison of Web of Science, Scopus, SciFinder, and Google Scholar. Journal of Electronic Resources in Medical Libraries, 7(3), 196–217.

    Article  Google Scholar 

  • Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58, 2105–2125.

    Article  Google Scholar 

  • Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et therapiae Experimentalis, 57, 13–18.

    Article  Google Scholar 

  • Moosa, I. (2011). The demise of the ARC journal ranking scheme: an ex post analysis of the accounting and finance journals. Accounting & Finance, 51(3), 809–836.

    Article  Google Scholar 

  • Moya-Anegón, F. D., Chinchilla-Rodríguez, Z., Vargas-Quesada, B., Corera-Álvarez, E., Muñoz-Fernández, F. J., González-Molina, A., et al. (2007). Coverage analysis of Scopus: A journal metric approach. Scientometrics, 73, 53–78.

    Article  Google Scholar 

  • Norris, M., & Oppenheim, C. (2010). The h-index: a broad review of a new bibliometric indicator. Journal of Documentation, 66(5), 681–705.

    Article  Google Scholar 

  • Northcott, D., & Linacre, S. (2010). Producing spaces for academic discourse: The impact of research assessment exercises and journal quality rankings. Australian Accounting Review, 20, 38–54.

    Article  Google Scholar 

  • Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation, 15, 347–360.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 6. doi:10.1103/PhysRevE.83.046116.

  • Rousseau, R., Yang, L., & Yue, T. (2010). A discussion of Prathap’s h2-index for institutional evaluation with an application in the field of HIV infection and therapy. Journal of Informetrics, 4(2), 175–184.

    Article  Google Scholar 

  • Ruane, F., & Tol, R. S. J. (2008). Rational (successive) h-indices: An application to economics in the Republic of Ireland. Scientometrics, 75(2), 395–405.

    Article  Google Scholar 

  • Runeson, G. (2011). The demise of the journal ranking: a victory for common sense. Australasian Journal of Construction Economics and Building, 11(2), 99–100.

    Google Scholar 

  • Schreiber, M. (2010). Twenty Hirsch index variants and other indicators giving more or less preference to highly cited papers. Annalen der Physik, 522(8), 536–554.

    Article  MathSciNet  Google Scholar 

  • Schubert, A. (2007). Successive h-indices. Scientometrics, 70(1), 201–205.

    Article  Google Scholar 

  • Schubert, A. (2009). Using the h-index for assessing single publications. Scientometrics, 78(3), 559–565.

    Article  Google Scholar 

  • Serenko, A., & Dohan, M. (2011). Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence. Journal of Informetrics, 5(4), 629–648.

    Article  Google Scholar 

  • Sinclair, R. (2010). RD&E strategy for the forest and wood products sector. Australia: Forest and Wood Products, p. 57. www.daff.gov.au/__data/assets/word_doc/0010/1770751/forest.doc.

  • Singh, G., Haddad, K. M., & Chow, C. W. (2007). Are articles in “top” management journals necessarily of higher quality? Journal of Management Inquiry, 16(4), 319–331.

    Article  Google Scholar 

  • Slyder, J. B., Stein, B. R., Sams, B. S., Walker, D. M., Beale, B. J., Feldhaus, J. J., et al. (2011). Citation pattern and lifespan: A comparison of discipline, institution and individual. Scientometrics, 89(3), 955–966.

    Article  Google Scholar 

  • Svantesson, D. J. B., & White, P. (2009). Entering an era of research ranking—will innovation and diversity survive? Bond Law Review, 21(3), 7.

    Google Scholar 

  • Taylor, J. (2011). The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22, 202–217.

    Article  Google Scholar 

  • Thor, A. & Bornmann, L. (2011). The calculation of the single publication h index and related performance measures: A web application based on Google Scholar data. Online Information Review, 35(2), 291–300. (Utility available here: http://labs.dbs.uni-leipzig.de/gsh/).

  • Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2, 326–334.

    Article  Google Scholar 

  • Vanclay, J. K. (2011). An evaluation of the Australian Research Council’s journal ranking. Journal of Informetrics, 5, 265–274.

    Article  Google Scholar 

  • Vanclay, J. K. (2012). What was wrong with Australia’s journal ranking? Journal of Informetrics, 6, 53–54.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Southern Cross University, PO Box 157, Lismore, NSW, 2480, Australia

    Jerome K. Vanclay

  2. Max Planck Society, Administrative Headquarters, PO Box 101062, Munich, 80084, Germany

    Lutz Bornmann

Authors
  1. Jerome K. Vanclay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jerome K. Vanclay.

Appendix

Appendix

See Tables 8, 9

Table 8 List of journals included within 0705 Forestry Sciences in ERA 2010 (ARC 2010b; Lamp 2011)
Full size table
Table 9  
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Vanclay, J.K., Bornmann, L. Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative. Scientometrics 91, 751–771 (2012). https://doi.org/10.1007/s11192-012-0618-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0618-8

Keywords

Search

Navigation