A ‘perverse incentive’ from bibliometrics: could National Research Assessment Exercises (NRAEs) restrict literature availability for nature conservation?
- 404 Downloads
National Research Assessment Exercises (NRAEs) aim to improve returns from public funding of research. Critics argue that they undervalue publications influencing practice, not citations, implying that journals valued least by NRAEs are disproportionately useful to practitioners. Conservation biology can evaluate this criticism because it uses species recovery plans, which are practitioner-authored blueprints for recovering threatened species. The literature cited in them indicates what is important to practitioners’ work. We profiled journals cited in 50 randomly selected recovery plans from each of the USA, Australia and New Zealand, using ranking criteria from the Australian Research Council and the SCImago Institute. Citations showed no consistent pattern. Sometimes higher ranked publications were represented more frequently, sometimes lower ranked publications. Recovery plans in all countries also contained 37 % or more citations to ‘grey literature’, discounted in NRAEs. If NRAEs discourage peer-reviewed publication at any level they could exacerbate the trend not to publish information useful for applied conservation, possibly harming conservation efforts. While indicating the potential for an impact does not establish that it occurs, it does suggest preventive steps. NRAEs considering the proportion of papers in top journals may discourage publication in lower-ranked journals, because one way to increase the proportion of outputs in top journals is by not publishing in lower ones. Instead, perhaps only a user-nominated subset of publications could be evaluated, a department’s or an individual’s share of the top publications in a field could be noted, or innovative new multivariate assessments of research productivity applied, including social impact.
KeywordsAustralia New Zealand USA UK Nature conservation Threatening process ERA PBRF VTR REF NRAE RAE
We thank, without implication, H. Recher, B. Dell, D. Saunders and an anonymous reviewer for detailed and constructive feedback on earlier versions.
- Adler, R., Ewing, J. & Taylor, P. (2008). Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Instititute of Mathematical Statistics (IMS). http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf. Accessed 28 August 2012.
- ARC (2012). ERA 2012 frequently asked questions. Available from http://www.arc.gov.au/era/faq.htm. Accessed 12 August 2011.
- Box, S. (2010). Performance-based funding for public research in tertiary education institutions: Country experiences. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. Paris: OECD Publishing. doi: 10.1787/9789264094611-en.
- Bryant, K. & Calver, M. (2012). Adaptive radiation in Australian journals in the Arbustocene ERA: an empty niche for JANCO? In P.B.Banks, D. Lunney & C.R. Dickman (Eds.), Science under siege (in press). Sydney: Royal Zoological Society of New South Wales.Google Scholar
- Butler, L. (2010). Impacts of performance-based research funding systems: a review of the concerns and the evidence. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. Paris: OECD Publishing. doi: 10.1787/9789264094611-en.
- Calver, M. C., & King, D. R. (2000). Why publication matters in conservation biology. Pacific Conservation Biology, 6, 2–8.Google Scholar
- Carr, K. (2011). Improvements to Excellence in Research for Australia. Canberra: Australian Government. Available from http://archive.innovation.gov.au/ministersarchive2011/carr/MediaReleases/Pages/IMPROVEMENTSTOEXCELLENCEINRESEARCHFORAUSTRALIA.html. Accessed 9 April 2011.
- Cooper, S., & Poletti, A. (2011). The new ERA of journal ranking: the consequences of Australia’s fraught encounter with ‘quality’. Australian Universities’ Review, 53, 57–65.Google Scholar
- Deville, A., & Harding, R. (1997). Applying the precautionary principle. Sydney: The Federation Press.Google Scholar
- Fairfull, S. J., & Williams, R. J. (2003). Community involvement in natural resource management: Lessons for future water management in catchments of New South Wales. In P. Hutchings & D. Lunney (Eds.), Conserving marine environments: Out of sight, out of mind (pp. 55–61). Sydney: Royal Zoological Society of New South Wales.Google Scholar
- Hicks, D. (2010). Overviews of performance-based research funding systems. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. Paris: OECD Publishing. doi: 10.1787/9789264094611-en.
- Horwitz, P. & Wardell-Johnson, G. (2009). Cultural conservation biology. In M. Calver, A. Lymbery, J. McComb & M. Bamford, M. (Eds.) Environmental biology (pp. 559–578). Melbourne: Cambridge University Press.Google Scholar
- Luwel, M. (2010). Highlights and reflections: rapporteur’s report. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. Paris: OECD Publishing. doi: 10.1787/9789264094611-en.
- Marsh, H., Smith, B., King, M., & Evans, T. (2012). A new era for research education in Australia? Australian Universities’ Review, 54, 83–93.Google Scholar
- McNay, I. (1998). The research assessment exercise (RAE) and after: ‘you never know how it will turn out’. Perspectives: Policy and Practice in Higher Education, 2, 19–22.Google Scholar
- OECD (2010). Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. Paris: OECD Publishing. doi: 10.1787/9789264094611-en.
- SCImago (2007). SJR—SCImago Journal & Country Rank. Consejo Superior de Investigaciones Científicas (CSIC), University of Granada, Extremadura, Carlos III (Madrid) & Alcalá de Henares, Spain. Available from http://www.scimagojr.com. Accessed July–August 2010.
- SCImago (2012). SCImago institutions rankings. SIR World Rankings 2011: Global ranking. Available from http://www.scimagoir.com/ Accessed 9 April 2012.
- Shewan, L. G., & Coats, A. J. S. (2006). The research quality framework and its implications for health and medical research: time to take stock? Medical Journal of Australia, 184, 463–466.Google Scholar
- Stergiou, K. I. S., & Tsikliras, A. C. (2006). Underrepresentation of regional ecological research output by bibliometric indices. Ethics in Science and Environmental Politics, 6, 15–17.Google Scholar
- Stinchcombe, J., & Moyle, L.C. (2002). The influence of the academic conservation biology literature on endangered species recovery planning. Conservation Ecology 6(2), 15. http://www.consecol.org/vol16/iss12/art15/.
- UNESCO (2005). The precautionary principle. World Commission on the Ethics of Scientific Knowledge and Technology (COMEST). United Nations Educational, Scientific and Cultural Organization, Paris.Google Scholar