Scientometrics

, Volume 104, Issue 3, pp 793–807 | Cite as

Measuring university quality

Article

Abstract

This paper uses a Bayesian hierarchical latent trait model, and data from eight different university ranking systems, to measure university quality. There are five contributions. First, I find that ratings tap a unidimensional, underlying trait of university quality. Second, by combining information from different systems, I obtain more accurate ratings than are currently available from any single source. And rather than dropping institutions that receive only a few ratings, the model simply uses whatever information is available. Third, while most ratings focus on point estimates and their attendant ranks, I focus on the uncertainty in quality estimates, showing that the difference between universities ranked 50th and 100th, and 100th and 250th, is insignificant. Finally, by measuring the accuracy of each ranking system, as well as the degree of bias toward universities in particular countries, I am able to rank the rankings.

Keywords

Latent trait models Bayesian models University rankings 

Notes

Acknowledgments

Many thanks to Isidro F. Aguillo and Robert Morse for kindly supplying the Webometrics and US News & World Report National Universities ratings data respectively. Lutz Bornmann provided helpful comments on a earlier version of this paper.

Supplementary material

11192_2015_1584_MOESM1_ESM.pdf (345 kb)
Supplementary material 1 (pdf 344 KB)

References

  1. Aguillo, I. F., Bar-Ilan, J., Levene, M., & Ortega, J. L. (2010). Comparing university rankings. Scientometrics, 85(1), 243–56.CrossRefGoogle Scholar
  2. Altbach, P.G. (2010). The state of the rankings. Inside Higher Ed (November 11), https://www.insidehighered.com/views/2010/11/11/altbach.
  3. Bafumi, J., Gelman, A., Park, D. K., & Kaplan, N. (2005). Practical issues in implementing and understanding bayesian ideal point estimation. Political Analysis, 13(2), 171–87.CrossRefGoogle Scholar
  4. Bornmann, L., Mutz, R., & Daniel, H. D. (2013). Multilevel-statistical reformulation of citation-based university rankings: The leiden ranking 2011/2012. Journal of the American Society for Information Science and Technology, 64(8), 1649–58.CrossRefGoogle Scholar
  5. Bowman, N. A., & Bastedo, M. N. (2009). Getting on the front page: Organizational reputation, status signals, and the impact of U.S. News and World Report on student decisions. Research in Higher Education, 50(5), 415–436.CrossRefGoogle Scholar
  6. Bowman, N. A., & Bastedo, M. N. (2011). Anchoring effects in world university rankings: Exploring biases in reputation scores. Higher Education, 61(4), 431–44.CrossRefGoogle Scholar
  7. Enserink, M. (2007). Who ranks the university rankers? Science, 317(5841), 1026–28.CrossRefGoogle Scholar
  8. Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models. Bayesian Analysis, 1(3), 515–33.MathSciNetGoogle Scholar
  9. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–72.CrossRefGoogle Scholar
  10. Glockner-Rist, A., & Hoijtink, H. (2009). The best of both worlds: Factor analysis of dichotomous data using item response theory and structural equation modeling. Structural Equation Modeling, 10(4), 544–65.MathSciNetCrossRefGoogle Scholar
  11. Goldstein, H., & Spiegelhalter, D. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society Series A (Statistics in Society), 159(3), 385–443.CrossRefGoogle Scholar
  12. Grewala, R., Deardena, J. A., & Lliliena, G. L. (2008). The university rankings game: Modeling the competition among universities for ranking. The American Statistician, 62(3), 232–7.MathSciNetCrossRefGoogle Scholar
  13. Hallinger, P. (2014). Riding the tiger of world university rankings in East Asia: Where are we heading? International Journal of Educational Management, 28(2), 230–45.Google Scholar
  14. Hazelkorn, E. (2007). The impact of league tables and ranking system on higher education decision making. Higher Education Management and Policy, 19(2), 1–24.CrossRefGoogle Scholar
  15. International Ranking Expert Group (2011). Ireg ranking audit manual. Tech. report, IREG Observatory on Academic Ranking and Excellence. www.ireg-observatory.org.
  16. Lee, S. (2009). Reputation without rigor. Inside Higher Ed (August 19), https://www.insidehighered.com/news/2009/08/19/rankings.
  17. Leeuwen, T. N. V., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Raan, A. F. J. V. (2001). Language biases in the coverage of the science citation index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–46.CrossRefGoogle Scholar
  18. Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001.MathSciNetCrossRefMATHGoogle Scholar
  19. Monks, J., & Ehrenberg, R. G. (1999). The impact of US News & World Report college rankings on admission outcomes and pricing decisions at selective private institutions. NBER Working Paper (7227).Google Scholar
  20. Rauhvargers, A. (2013). Global university rankings and their impact: Report II. Tech. report. Brussels: European University Association.Google Scholar
  21. Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.CrossRefGoogle Scholar
  22. Soh, K. (2011). Don’t read university rankings like reading football league tables: Taking a close look at the indicators. Higher Education Review, 44(1), 15–29.Google Scholar
  23. Soh, K. (2014). Multicolinearity and indicator redundancy problem in world university rankings: An example using Time Higher Education World University Ranking 2013–2014 data. Higher Education Quarterly, 69(2), 158–174.Google Scholar
  24. Stan Development Team. (2014). Stan Modeling Language: User’s Guide and Reference Manual. Stan Development Team.Google Scholar
  25. Usher, A., & Savino, M. (2006). A world of difference: A global survey of university league tables. Tech. report. Toronto, ON: Educational Policy Institute.Google Scholar
  26. van Vught, F. A., & Ziegele, F. (Eds.). (2012). Multidimensional ranking: The design and development of U-Multirank. New York: Springer.Google Scholar
  27. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C., Tijssen, R. J., van Eck, N. J., et al. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–32.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2015

Authors and Affiliations

  1. 1.Department of GovernmentUniversity of EssexColchesterEngland

Personalised recommendations