Skip to main content
SpringerLink
Log in

Technical pitfalls in university rankings

  • Published:
Higher Education Aims and scope Submit manuscript

Abstract

Academicians, experts, and other stakeholders have contributed extensively to the literature on university rankings also known as “league tables”. Often the tone is critical usually focused on the subjective aspects of the process; e.g., the list of the universities’ attributes used in the rankings, their respective weights, and the size and composition of the comparison group. These aspects of a ranking are an easy target since, after all, they are based on someone’s opinion even if this person is considered an expert. There are other, purely technical, reasons why ranking schemes are problematic. In this paper we discuss these aspects of rankings by studying the handling of the data, exposing logical mistakes, and raising interpretation issues. We present these as a list of four “pitfalls” invoking in each case an example from an actual rankings. Each case also results in recommendations that address the technical issues involved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Bougnol, M. L., & Dulá, J. H. (2006). Validating DEA as a ranking tool: An application of DEA to assess performance in higher education. Annals of Operations Research, 145, 339–365.

    Article  Google Scholar 

  • Bougnol, M. L., & Dulá, J. H. (2013). A mathematical model to optimize decisions for climbing in multi-attribute rankings. Scientometrics, 95, 785–796.

    Article  Google Scholar 

  • Casper, G., Criticism of College Rankings. (1996). Private letter to the editor of USNWR. http://www.stanford.edu/dept/pres-provost/president/speeches/961206gcfallow.html. Accessed March 2014.

  • Center for World-Class Universities (CWCU), Graduate School of Education of Shanghai Jiao Tong University. (2013). Academic Ranking of World Universities. http://www.shanghairanking.com/ARWU2013.html. Accessed April 2014.

  • CWTS Leiden ranking. (2013). http://www.leidenranking.com/ranking. Accessed April 2014.

  • De Vise, D. (2011). The ups and downs of U.S. news rankings. The Washington Post.

  • Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132, 245–259.

    Article  Google Scholar 

  • Enserink, M. (2007). Who ranks the university rankers? Science, 317, 1026–1028.

    Article  Google Scholar 

  • Filinov, N. B., & Ruchkina, S. (2002). Ranking of higher education institutions in Russia—Some methodological problems. Higher Education in Europe, 27, 407–421.

    Article  Google Scholar 

  • Gladwell, M. (2011). The order of things. The New Yorker.

  • Gnolek, S. L., Falciano, V. T., & Kuncl, R. W. (2014). Modeling change and variation in U.S. News & World Report college rankings: What would it really take to be in the top 20? Research in Higher Education. doi:10.1007/s11162-014-9336-9.

  • Goldstein, H. D., & Spiegelhalter, J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Series, 159, 385–443.

    Article  Google Scholar 

  • Jaschik, S. (2007). Battle lines on ‘U.S. News’. http://www.insidehighered.com/news/2007/05/07/usnews. Accessed March 2014.

  • Joliffe, I. T. (2002). Principal component analysis (2nd ed). Berlin: Springer Series in Statistics.

  • Liu, N.-C., & Cheng, Y. (2005). The academic ranking of world universities. Higher Education in Europe, 30, 127–136.

    Article  Google Scholar 

  • Quacquarelli and Symonds. (2012). World university rankings. http://www.iu.qs.com; Accessed March 2014.

  • Tofallis, C. (2012). A different approach to university rankings. Higher Education, 63, 1–18.

    Article  Google Scholar 

  • US News and World Report. (2013). Best colleges 2014 edition, Published by US News and World Report L.P., Washington, D.C.

  • Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N. J., et al. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the Association for Information Science and Technology, 63, 2419–2432.

    Article  Google Scholar 

  • Webster, T. J. (2001). A principal component analysis of the U.S. News & World Report tier rankings of colleges and universities. Economics of Education Review, 20, 235–244.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to acknowledge that this paper has been inspired and patterned after Dyson et al. (2001).

Author information

Authors and Affiliations

  1. Jacksonville University, Jacksonville, FL, 32211, USA

    Marie-Laure Bougnol

  2. Virginia Commonwealth University, Richmond, VA, 23284, USA

    Jose H. Dulá

Authors
  1. Marie-Laure Bougnol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jose H. Dulá
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marie-Laure Bougnol.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bougnol, ML., Dulá, J.H. Technical pitfalls in university rankings. High Educ 69, 859–866 (2015). https://doi.org/10.1007/s10734-014-9809-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10734-014-9809-y

Keywords

Search

Navigation