Abstract
There are increasing moves to deploy quantitative indicators in the assessment of research, particularly in the university sector. In Australia, discussions surrounding their use have long acknowledged the unsuitability of many standard quantitative measures for most humanities, arts, social science, and applied science disciplines. To fill this void, several projects are running concurrently. This paper details the methodology and initial results for one of the projects that aims to rank conferences into prestige tiers, and which is fast gaining a reputation for best practice in such exercises. The study involves a five-stage process: identifying conferences; constructing a preliminary ranking of these; engaging in extensive consultation; testing performance measures based on the rankings on ‘live’ data; and assessing the measures.
In the past, many similar attempts to develop a ranking classification for publication outlets have faltered due to the inability of researchers to agree on a hierarchy. However the Australian experience suggests that when researchers are faced with the imposition of alternative metrics that are far less palatable, consensus is more readily achieved.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arts And Humanities Research Council (AHRC) (2006), The ESF European Reference Index for the Humanities, available at: http://www.ahrc.ac.uk/about/ke/erih_-_responses_to_the_project_from_uk_subject_groups.asp
Australian Bureau of Statistics (ABS) (1998), Australian Standard Research Classification, ABS Catalogue No. 1297.0, Canberra, Available at: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1297.01998?OpenDocument
Butler, L. (2003), Modifying publication practices in response to funding formulas, Research Evaluation, 12(1): 39–46.
Butler, L., Visser, M. S. (2006), Extending citation analysis to non-source item, Scientometrics, 66(2): 327–343.
CiteSeer (2003), CiteSeer.IST Scientific Literature Digital Library, available at: http://citeseer.ist.psu.edu/cs
Department for Education and Skills (DfES) (2006), Reform of Higher Education Research Assessment and Funding, available at: http://www.dfes.gov.uk/consultations/conResults.cfm?consultationId=1404
Department of Education Science and Training (DEST) (2006), Research Quality Framework: Assessing the Quality and Impact of Research in Australia, details available at: http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research_quality_framework/default.htm
Gu, G. (2005), Computer Science Conference Rankings, available at: http://www-static.cc.gatech.edu/:_guofei/CS_ConfRank.htm (accessed 18 October 2006).
HM-Treasurey (2006), Science and Innovation Investment Framework 2004–2014: Next Steps, available at: http://www.hm-treasury.gov.uk/media/1E1/5E/bud06_science_332.pdf
Long, P. M., Lee, T. K., Jaffar, J. (1999), Benchmarking Research Performance in Department of Computer Science, School of Computing, National University of Singapore, available online at: http://www.comp.nus.edu.sg/:_tankl/bench.html (accessed 18 October 2006).
Sastry T., Bekhradnia B. (2006), Using Metrics to Allocate Research Funds: A Short Evaluation of Alternatives to the Research Assessment Exercise, Higher Education Policy Institute, UK, April.
Shanghai Jiao Tong University (SJTU) (2006), Academic Ranking of World Universities — 2006. Shanghai Jiao Tong University, Institute of Higher Education, available at: http://ed.sjtu.edu.cn/ranking.htm
Sidiropoulos, Manolopoulos (2005), A new perspective to automatically rank scientific conferences using digital libraries, Information Processing & Management, 41: 289–312.
Van Leeuwen, T. (2006), The application of bibliometric analyses in the evaluation of social science research. Who benefits from it, and why it is still feasible, Scientometrics, 66(1): 133–154.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Butler, L. ICT assessment: Moving beyond journal outputs. Scientometrics 74, 39–55 (2008). https://doi.org/10.1007/s11192-008-0102-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-008-0102-7