Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative
- 363 Downloads
Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.
KeywordsSuccessive h-index Percentile h-index Excellence for Research in Australia (ERA)
- Anon. (2011). List of forestry universities and colleges. Wikipedia, http://en.wikipedia.org/wiki/List_of_forestry_universities_and_colleges [August 5, 2011].
- ANZSRC (2008). Australian and New Zealand Standard Research Classification. http://www.abs.gov.au/Ausstats/abs@.nsf/0/6BB427AB9696C225CA2574180004463E [August 5, 2011].
- ARC (2010a). The Excellence in Research for Australia (ERA) Initiative. http://www.arc.gov.au/era/default.htm [August 5, 2011].
- ARC (2010b). Ranked Journal List. Australian Research Council. http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls [August 5, 2011].
- ARC (2011a). What is the change to the journal indicator? Australian Research Council. http://www.arc.gov.au/era/faq.htm#2012 [August 5, 2011].
- ARC (2011b). ERA 2010 National Report. Australian Research Council, p. 320. http://www.arc.gov.au/pdf/ERA_report.pdf [August 5, 2011].
- ARC (2011c). ERA 2010 citation benchmark methodology. Australian Research Council, p. 21. http://www.arc.gov.au/pdf/era_2010_citation_benchmark_methods.pdf [August 5, 2011].
- ARC (2011d). ERA 2010 results by field of research code. Australian Research Council. http://www.arc.gov.au/era/outcomes_2010/FoRindex [August 5, 2011].
- Atkinson, R. & McLoughlin, C. (2011). Dawn of a new ERA? Australasian Journal of Educational Technology, 27(3), iii–viii. http://ascilite.org.au/ajet/ajet27/editorial27-3.html.
- Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.Google Scholar
- Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from Biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.CrossRefGoogle Scholar
- Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Does the h index for assessing single publications really work? A case study on papers published in chemistry. Scientometrics, 89(3), 835–843.Google Scholar
- Carr, K. (2009). A message from the Minister. Australian Research Council, Discovery Newsletter, Autumn 2009. http://www.arc.gov.au/rtf/discovery_autumn09.rtf.
- Cooper, S., & Poletti, A. (2011). The new ERA of journal ranking. Australian Universities’ Review, 53, 57–65.Google Scholar
- Jones, D. (2011). Excellence in Research for Australia—a new ERA. Research Trends 23, 10–11. http://www.researchtrends.com/wp-content/uploads/2011/06/Research_Trends_Issue23.pdf.
- Laband, D. N., & Zhang, D. (2006). Citations, publications, and perceptions-based rankings of the research impact of North American forestry programs. Journal of Forestry, 104(5), 162–254.Google Scholar
- Labbé, C. (2010). Ike Antkare, one of the great stars in the scientific firmament. ISSI Newsletter, 6(2), 48–52.Google Scholar
- Lamp, J. (2011). 2010 finalised journals in an ANZ Field of Research: 0705 Forestry Sciences. http://lamp.infosys.deakin.edu.au/era/?page=jfordet10&selfor=0705 [5 August 2011].
- Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 6. doi: 10.1103/PhysRevE.83.046116.
- Runeson, G. (2011). The demise of the journal ranking: a victory for common sense. Australasian Journal of Construction Economics and Building, 11(2), 99–100.Google Scholar
- Sinclair, R. (2010). RD&E strategy for the forest and wood products sector. Australia: Forest and Wood Products, p. 57. www.daff.gov.au/__data/assets/word_doc/0010/1770751/forest.doc.
- Svantesson, D. J. B., & White, P. (2009). Entering an era of research ranking—will innovation and diversity survive? Bond Law Review, 21(3), 7.Google Scholar
- Thor, A. & Bornmann, L. (2011). The calculation of the single publication h index and related performance measures: A web application based on Google Scholar data. Online Information Review, 35(2), 291–300. (Utility available here: http://labs.dbs.uni-leipzig.de/gsh/).