University rankings: What do they really show?
- 905 Downloads
University rankings as developed by the media are used by many stakeholders in higher education: students looking for university places; academics looking for university jobs; university managers who need to maintain standing in the competitive arena of student recruitment; and governments who want to know that public funds spent on universities are delivering a world class higher education system. Media rankings deliberately draw attention to the performance of each university relative to all others, and as such they are undeniably simple to use and interpret. But one danger is that they are potentially open to manipulation and gaming because many of the measures underlying the rankings are under the control of the institutions themselves. This paper examines media rankings (constructed from an amalgamation of variables representing performance across numerous dimensions) to reveal the problems with using a composite index to reflect overall performance. It ends with a proposal for an alternative methodology which leads to groupings rather than point estimates.
KeywordsHigher education Rankings Performance Principal components analysis
I am grateful for comments and suggestions to an anonymous referee, to Geraint Johnes and Swati Virmani, and to the participants at: Efficiency in Education, Politecnico di Milano 20th–21st October 2016; Valuing Higher Education: An appreciation of the work of Gareth Williams, Centre for Higher Education Studies, Institute of Education, University College London, 15th November 2016; the Fourth Lisbon Research Workshop on Economics, Statistics and Econometrics of Education, Lisbon, Portugal, 26th–27th January 2017; the Meeting of the Economics of Education Association, Murcia, 29th–30th June 2017.
- Allcock, D., Johnes, J., & Virmani, S. (2017). Efficiency and VC pay: Exploring the value conundrum. In European workshop on efficiency and productivity analysis, London, 13th–15th June.Google Scholar
- Barr, R. S., Durchholz, M. L., & Seiford, L. (2000). Peeling the DEA onion: Layering and rank-ordering DMUs using tiered DEA. Dallas, TX: Southern Methodist University Technical Report.Google Scholar
- Bekhradnia, B. (2016). International university rankings: For good or ill? HEPI report. http://www.hepi.ac.uk/wp-content/uploads/2016/12/Hepi_International-university-rankings-For-good-or-for-ill-REPORT-89-10_12_16_Screen.pdfHigherEducationPolicyInstitute.
- Boring, A., Ottoboni, K., & Stark, P. B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1.Google Scholar
- Cattell, J. M. (1906). American men of science: A biographical dictionary. New York, NY: Science Press.Google Scholar
- Dill, D. D. (2009). Convergence and diversity: The role and influence of university ranking. In B. M. Kehm & B. Stensaker (Eds.), University rankings, diversity, and the new landscape of higher education (pp. 97–116). Rotterdam: Sense Publishers.Google Scholar
- Dixon, M. (1976). Careers: More means better. London: The Financial Times.Google Scholar
- Dixon, M. (1985). Jobs column: What happened to universities’ graduates?. London: The Financial Times.Google Scholar
- Dixon, M. (1989). Jobs column: Benefits, and risks, of trying for a degree. London: The Financial Times.Google Scholar
- Harman, G. (2011). Competitors of rankings: New directions in quality assurance and accountability. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University rankings: Theoretical basis, methodology and impacts on global higher education (pp. 35–54). Dordrecht: Springer.CrossRefGoogle Scholar
- Hazelkorn, E. (2015). How the geo-politics of rankings is shaping behaviour. Higher Education in Russia and Beyond, 2(4), 6–7.Google Scholar
- HEFCE. (2008). Counting what is measured or measuring what counts? League tables and their impact on higher education institutions in England. Bristol: Higher Education Funding Council for England.Google Scholar
- HEFCE. (2012). Collaborations, alliances and mergers in higher education: Consultation on lessons learned and guidance for institutions. London: HEFCE 2012/06 Higher Education Funding Council for England.Google Scholar
- Hughes, R. M. (1925). A study of the graduate schools of America. Oxford, OH: Miami University Press.Google Scholar
- Johnes, G. (2004). Standards and grade inflation. In G. Johnes & J. Johnes (Eds.), International handbook on the economics of education (pp. 462–483). Cheltenham: Edward Elgar.Google Scholar
- Johnes, J. (2016). Performance indicators and rankings in higher education. In R. Barnett, P. Temple, & P. Scott (Eds.), Valuing higher education: An appreciation of the work of Gareth Williams. London: UCL Institute of Education Press.Google Scholar
- Jump, P. (2014). Cut 50% of universities and bar undergraduates from Oxbridge. Times Higher Education. London, Times Supplements Ltd 25th June 2014.Google Scholar
- Locke, W., Verbik, L., Richardson, J. T. E., & Roger, K. (2008). Counting what is measured or measuring what counts? League tables and their impact on higher education institutions in England. Bristol: Higher Education Funding Council for England.Google Scholar
- Newman, M. (2008). Students urged to inflate national survey marks to improve job options. Times Higher Education London, Times Supplements Ltd. 15th May 2008: 7 15th May 2008.Google Scholar
- Pollard, E., Williams, M., Williams, J., Bertram, C., & Buzzeo, J. (2013). How should we measure higher education? A fundamental review of the Performance Indicators, Part 2: The evidence report. Brighton: Institute for Employment Studies.Google Scholar
- Saltelli, A., Nardo, M., Tarantola, S., Giovannini, E., Hoffman, A., & Saisana, M. (2005). Handbook on constructing composite indicators: Methodology and user guide. Paris: OECD.Google Scholar
- Stark, P. B., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1.Google Scholar
- Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In B. M. Kehm & B. Stensaker (Eds.), University rankings, diversity, and the new landscape of higher education (pp. 3–18). Rotterdam: Sense Publishers.Google Scholar
- Weimer, D. L., & Vining, A. R. (1996). Economics. In D. F. Kettle & H. B. Milward (Eds.), The state of public management (pp. 92–117). Baltimore, MA: Johns Hopkins University.Google Scholar