This paper uses a Bayesian hierarchical latent trait model, and data from eight different university ranking systems, to measure university quality. There are five contributions. First, I find that ratings tap a unidimensional, underlying trait of university quality. Second, by combining information from different systems, I obtain more accurate ratings than are currently available from any single source. And rather than dropping institutions that receive only a few ratings, the model simply uses whatever information is available. Third, while most ratings focus on point estimates and their attendant ranks, I focus on the uncertainty in quality estimates, showing that the difference between universities ranked 50th and 100th, and 100th and 250th, is insignificant. Finally, by measuring the accuracy of each ranking system, as well as the degree of bias toward universities in particular countries, I am able to rank the rankings.
KeywordsLatent trait models Bayesian models University rankings
Many thanks to Isidro F. Aguillo and Robert Morse for kindly supplying the Webometrics and US News & World Report National Universities ratings data respectively. Lutz Bornmann provided helpful comments on a earlier version of this paper.
- Altbach, P.G. (2010). The state of the rankings. Inside Higher Ed (November 11), https://www.insidehighered.com/views/2010/11/11/altbach.
- Hallinger, P. (2014). Riding the tiger of world university rankings in East Asia: Where are we heading? International Journal of Educational Management, 28(2), 230–45.Google Scholar
- International Ranking Expert Group (2011). Ireg ranking audit manual. Tech. report, IREG Observatory on Academic Ranking and Excellence. www.ireg-observatory.org.
- Lee, S. (2009). Reputation without rigor. Inside Higher Ed (August 19), https://www.insidehighered.com/news/2009/08/19/rankings.
- Monks, J., & Ehrenberg, R. G. (1999). The impact of US News & World Report college rankings on admission outcomes and pricing decisions at selective private institutions. NBER Working Paper (7227).Google Scholar
- Rauhvargers, A. (2013). Global university rankings and their impact: Report II. Tech. report. Brussels: European University Association.Google Scholar
- Soh, K. (2011). Don’t read university rankings like reading football league tables: Taking a close look at the indicators. Higher Education Review, 44(1), 15–29.Google Scholar
- Soh, K. (2014). Multicolinearity and indicator redundancy problem in world university rankings: An example using Time Higher Education World University Ranking 2013–2014 data. Higher Education Quarterly, 69(2), 158–174.Google Scholar
- Stan Development Team. (2014). Stan Modeling Language: User’s Guide and Reference Manual. Stan Development Team.Google Scholar
- Usher, A., & Savino, M. (2006). A world of difference: A global survey of university league tables. Tech. report. Toronto, ON: Educational Policy Institute.Google Scholar
- van Vught, F. A., & Ziegele, F. (Eds.). (2012). Multidimensional ranking: The design and development of U-Multirank. New York: Springer.Google Scholar