Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

What do global university rankings really measure? The search for the X factor and the X entity

Abstract

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.

This is a preview of subscription content, log in to check access.

Notes

  1. 1.

    One exception is the study by Li et al. (2011a), which focuses on the effects of country and language.

  2. 2.

    This principle is based on the functioning of the memory of individuals and the social processes that occur in science.

  3. 3.

    As the sample is reduced, the variance of the variables increases and the t statistics associated with its parameters decrease. This problem occurs in Model 3 (n = 181) and particularly in Model 2 (n = 100).

References

  1. Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206–215.

  2. Adler, N. J., & Harzing, A. W. (2009). When knowledge wins: transcending the sense and nonsense of academic rankings. Academy of Management Learning & Education, 8(1), 72–95.

  3. Altbach, P. (2006). The dilemmas of ranking. International Higher Education, 42, 1–2.

  4. Armstrong, J. S., & Sperry, T. (1994). Business school prestige-research versus teaching. Interfaces, 24(2), 13–43.

  5. Bastedo, M. N., & Bowman, N. A. (2010). The U.S. News and World Report college rankings: Modeling institutional effects on organizational reputation. American Journal of Education, 116, 163–184.

  6. Bentley, R., & Blackburn, R. (1990). Changes in academic research performance over time: A study of institutional accumulative advantage. Research in Higher Education, 31(4), 327–353.

  7. Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237–263.

  8. Bornmann, L. (2011). Peer Review and Bibliometric: Potentials and Problems. In J. C Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  9. Bowman, N. A., & Bastedo, M. N. (2011). Anchoring effects in world university rankings: exploring biases in reputation scores. High Education, 61, 431–444.

  10. Brooks, R. (2005). Measuring University Quality. The Review of Higher Education, 29(1), 1–21.

  11. Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.

  12. Carpenter, G. S., & Nakamoto, K. (1989). Consumer Preference Formation and Pioneering Advantage. Journal of Marketing Research, 26(3), 285–298.

  13. Chen, K., & Liao, P. (2012). A comparative study on world university rankings: a bibliometric survey. Scientometrics, 92, 89–103.

  14. Cole, S., & Cole, J. R. (1967). Scientific output and recognition: A study in the operation of the reward system in science. American Sociological Review, 32(3), 377–390.

  15. Crespi, G. (2007). The UK knowledge production function. In A. Bonaccorsi & C. Daraio (Eds.), Universities and strategic knowledge creation: Specialization and performance in Europe. Cheltenham, UK: Edward Elgar Publishing.

  16. Devinney, T., Dowling, G. R., & Perm-Ajchariyawong, N. (2006). The business schools rankings game. AACSB. Retrieved December 18, 2006, from www.aacsb.edu/resource_centers/Rankings.

  17. Dey, E. L., Milem, J. F., & Berger, J. B. (1997). Changing patterns of publication productivity: Accumulative advantage or institutional isomorphism? Sociology of Education, 70, 308–323.

  18. Dichev, L. D. (1999). How good are business schools rankings? Journal of Business, 72(2), 201–213.

  19. Dill, D. D., & Soo, M. (2005). Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems. Higher Education, 49, 495–533.

  20. Docampo, D. (2011). On using the Shanghai ranking to assess the research performance of university systems. Scientometrics, 86, 77–92.

  21. Dyck, I. J. A. & Zingales, L. (2002). The Corporate Governance Role of the Media. CRSP Working Paper, No. 543. Retrieved October 19, 2005, from http://ssrn.com/abstract=335602.

  22. Fombrun, C., & Shanley, M. (1990). What’s in a name? Reputation building and corporate strategy. Academy of Management Journal, 33(2), 233–258.

  23. Gioia, D. A., & Corley, K. G. (2002). Being good versus looking good: Business schools rankings and the circean transformation from substance to image. Academy of Management Learning and Education, 1(1), 107–120.

  24. Gómez, I., Bordons, M., Fernández, M. T., & Morillo, F. (2009). Structure and research performance of Spanish universities. Scientometrics, 79(1), 131–146.

  25. Gómez-Sancho, J. M. & Pérez-Esparrells, C. (2012). International Higher Education rankings at a glance: How to valorize the research in Social Sciences and Cultural Studies. In A. López-Varela (Ed.), Social Sciences and Humanities. Applications and Theories. Book 1 (pp. 355–374). Rijeka: InTech.

  26. Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis. NJ: Prentice Hall.

  27. Hazelkorn, E. (2009). Impact of Global Rankings on Higher Education Research and the Production of Knowledge, UNESCO Forum on Higher Education, Research and Knowledge, Occasional Paper No. 15.

  28. Huang, M. H. (2012). Opening the black box of QS World University Rankings. Research Evaluation, 21, 71–78.

  29. Jeon, Y., Miller, S. M., & Ray, S. C. (2003). MBA program reputation: Objective rankings for students, employers, and program administrators, Economics Working Paper. Paper 200328. Retrieved October 19, 2012, from http://digitalcommons.uconn.edu/econ_wpapers/200328..

  30. Johnson, D. (1997). Getting noticed in economics: The determinants of academic citations. American Economist, 41(1), 43–52.

  31. Katz, J.S. (1999). The self-similar science system. Research Policy, 28(5), 501–517.

  32. Keith, B. (2001). Organizational contexts and university performance outcomes: The Limited Role of Purposive Action in the Management of Institutional Status. Research in Higher Education, 42(5), 493–516.

  33. Kivinen, O., Hedman, J., & Kaipainen, P. (2013). Productivity analysis of research in natural sciences, technology and clinical medicine: An input–output model applied in comparison of Top 300 ranked universities of 4 North European and 4 East Asian countries. Scientometrics. 94(2), 683–699.

  34. Li, M., Shankar, S., & Tang, K. K. (2011a). Why does the USA dominate university league tables? Studies in Higher Education, 36(8), 923–937.

  35. Li, M., Shankar, S., & Tang, K. K. (2011b). Catching up with Harvard: Results from regression analysis of world universities league tables. Cambridge Journal of Education, 41(2), 121–137.

  36. Lieberman, M. B., & Montgomery, D. B. (1988). First-mover advantages. Strategic Management Journal, 9, 41–58.

  37. Liu, N. C., & Li Liu, Y. C. (2005). Academic ranking of world universities using scientometrics. A comment to the Fatal Attraction. Scientometrics, 64(1), 101–109.

  38. Locke, W., Verbik, L., Richardson, J., & King, R. (2008). Counting what is measured or measuring what counts? League tables and their impact on higher education institutions in England. Bristol, UK: Higher Education Funding Council for England.

  39. Longden, B. (2011). Ranking Indicators and Weights. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  40. Marginson, S. (2007). Global university rankings: implications in general and for Australia. Journal of Higher Education Policy and Management, 29(2), 131–142.

  41. Merton, R. K. (1968). The Matthew effect in science. Science, 159, 56–63.

  42. Morgeson, F. P., & Nahrgang, J. D. (2008). Same as it ever was: Recognizing stability in the Business Week rankings. Academy of Management Learning & Education, 7, 26–41.

  43. O’Connell, C. (2012). Research discourses surrounding global university rankings: exploring the relationship with policy and practice recommendations. Higher Education. doi: 10.1007/s10734-012-9572-x. Published on line: 30 September 2012.

  44. Olivares, M., & Wetzel, H. (2011). Competing in the higher education market: Empirical evidence for economies of scale and scope in German higher education institutions (No. 223). University of Lüneburg Working Paper Series in Economics.

  45. Ordorika, I., & Rodríguez, R. (2010). TIMES ranking in the reputation market/El ranking Times en el mercado del prestigio universitario. Perfiles Educativos, XXXII, 129, 8–29.

  46. Oswald, A. J. (2008). Can we test for bias in scientific peer-review? IZA DP No. 3665, Forschungsinstitut zur Zukunft der Arbeit Institute for the Study of Labor.

  47. Patterson, W. (1993). First-mover advantage: the opportunity curve. Journal of Management Studies, 30(5), 759–777.

  48. Power, M. (1997). The audit society. Oxford: Oxford University Press.

  49. Rauhvargers, A. (2011). Report on rankings 2011: Global university rankings and their impact. Brussels: European University Association.

  50. Rigney, D. (2010). The Matthew effect: How advantage begets further advantage. Columbia University Press.

  51. Safón, V. (2012). Can the reputation of an established business school change? Management in Education, 26(4), 169–180.

  52. Saisana, M., & D’Hombres, B. (2008). Higher Education Rankings: Robustness Issues and Critical Assessment, Report 23487, European Commission, JRC-IPSC, Italy.

  53. Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research Policy, 40(1), 165–177.

  54. Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.

  55. Shin, J. C. (2011). Organizational Effectiveness and University Rankings, In J. C. Shin, R. K. Toutkoushian & U. Teichler, (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  56. Shin, J. C., & Toutkoushian, R. K. (2011). The past, present, and future of University Rankings. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3), Dordrecht: Springer Science.

  57. Smyth, D. J. (1999). The determinants of the reputations of economics departments: Pages published, citations and the Andy Rooney effect. American Economist, 43(2), 49–58.

  58. Stella, A. & Woodhouse, D. (2006). Ranking of Higher Education Institutions. AUQA Occasional Publications Number 6. Australian Universities Quality Agency.

  59. Suarez, F., & Lanzolla, G. (2007). The role of environmental dynamics in building a first mover advantage theory. Academy of Management Review, 32(2), 377–392.

  60. Taylor, P., & Braddock, R. (2007). International university ranking systems and the idea of university excellence. Journal of Higher Education Policy and Management, 29(3), 245–260.

  61. Teichler, U. (2011a). Social contexts and systemic consequence of University Rankings: A meta-analysis of the ranking literature. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  62. Teichler, U. (2011b). The future of university rankings. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  63. Toutkoushian, R. K., & Webber, K. (2011). Measuring the Research Performance of Postsecondary Institutions. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective(Vol. 3). Dordrecht: Springer Science.

  64. Trank, C. Q., & Rynes, S. L. (2003). Who moved our cheese? Reclaiming professionalism in business education. Academy of Management Learning and Education, 2(2), 189–205.

  65. Usher, A., & Savino, M. (2006). A world of difference: A global survey of university league tables. Toronto, ON: Educational Policy Institute.

  66. Usher, A., & Savino, M. (2007). A Global Survey of Rankings and League Tables. In Institute for Higher Education Policy (Ed.), College and University Ranking Systems: Global Perspectives and American Challenges (pp. 23–35). www.ihep.org.

  67. Van Dalen, H. P., & Henkens, K. N. (2005). Signals in science-On the importance of signaling in gaining attention in science. Scientometrics, 64(2), 209–233.

  68. Van Dyke, N. (2005). Twenty years of university reports cards. Higher Education in Europe, 30(2), 103–124.

  69. Van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346.

  70. Van Raan, A. F. J. (2005a). Fatal attraction: Ranking of universities by bibliometric methods. Scientometrics, 62, 133–145.

  71. Van Raan, A. F. J. (2005b). Reply to the comments of Liu et al. Scientometrics, 64(1), 111–112.

  72. Van Raan, A. F. (2006). Statistical properties of bibliometric indicators: Research group indicator distributions and correlations. Journal of the American Society for Information Science and Technology, 57(3), 408–430.

  73. Van Raan, A. F. J. (2007). Challenges in the ranking of universities. In J. Sadlak & L. N. Cai (Eds.), The world-class university and ranking: Aiming beyond status. Shanghai: UNESCO-CEPES, Shanghai Jiao Tong University, Cluj University Press.

  74. Van Vught, F., & Ziegele, F. (Eds). (2011). U-multirank. Design and Testing the Feasibility of a Multidimensional Global University Ranking. Final Report. Consortium for Higher Education and Research Performance assessment. CHERPA-Network.

  75. Von Tunzelmann, N., Ranga, M., Martin, B. & Geuna, AL (2003). The effects of size on research performance: A SPRU review. Science and Technology Policy Research (SPRU), The Freeman Centre, University of Sussex Falmer, Brighton, BN1 9QE, UK.

  76. Williams, R. (2007). Broadening the Criteria: Lessons from the Australian Rankings. In J. Sadlak & L. N. Cai (Eds.), The world-class university and ranking: Aiming beyond status. Shanghai: UNESCO-CEPES, Shanghai Jiao Tong University, Cluj University Press.

  77. Zitt, M., & Filliatreau, G. (2007). Big is (made) beautiful: Some comments about the Shanghai ranking of world-class universities. In J. Sadlak & N.C. Liu (Eds.), The world-class university and ranking: Aiming beyond status (pp. 141–160). Shanghai: UNESCO-CEPES, Institute of Higher Education, Shanghai Jiao Tong University, and Cluj University Press.

Download references

Author information

Correspondence to Vicente Safón.

Appendices

Appendix 1. Variables studied as possible X factors and their measures

University in U.S. 1 if the university is in the U.S., 0 if it is not.

English-speaking country (excluded U.S.). 1 if the official national language is English, 0 if it is not or the university is in the U.S.

Size (students). Variable measured on a 4-point scale, where 1 = small, 2 = medium-sized, 3 = large y 4 = very large. Taken from the Quacquarelli Symonds database. See description at www.topuniversities.com.

Age. Variable measured on a 5-point scale: 1-New (<10 years), 2-Young (<25 years), 3-Established (<50 years), 4-Mature (<100 years), 5-Historic (>100 years). Variable taken from the Quacquarelli Symonds database.

Annual income. Variable taken from the ARWU database. Coded in the form of a ranking, so that a smaller number indicates larger income.

Scope (number of disciplines). Variable measured on a 4-point scale, ranging from 1 = very few subjects offered to 4 = a wide variety of subjects. Variable taken from the Quacquarelli Symonds database.

Activity in hard sciences. This variable measures whether the university carries out activities in hard sciences. It is measured using a dichotomous variable with a value of 1 if the university published papers on Nature and Science between 2007 and 2011, and 0 if not (data taken from the ARWU database).

Orientation towards research. Variable taken from the Quacquarelli Symonds database. Coded in the form of a score from faculty to student ratio. See an explanation at: http://www.iu.qs.com/university-rankings/rankings-indicators/methodology-faculty-student. Student staff ratio is commonly used as a measure of teaching quality (Longden 2011). According to literature, a lower academic staff to student ratio indicates that the university is more teaching oriented than research oriented (Li et al. 2011b).

Reputation. Variable ranging from 0 = no reputation to 100 = the highest reputation among academics. Taken from the “Quacquarelli Symonds Global Academic Survey”, using a sample of 34,000 academics from a broad, balanced variety of subjects and countries. See an explanation at: http://www.iu.qs.com/university-rankings/world-university-rankings/2011-academic-survey-responses/.

Appendix 2. Correlations between the X factors and the affected THE indicators

See Table 5.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Safón, V. What do global university rankings really measure? The search for the X factor and the X entity. Scientometrics 97, 223–244 (2013). https://doi.org/10.1007/s11192-013-0986-8

Download citation

Keywords

  • Rankings
  • Reputation
  • Universities
  • Factor analysis
  • Multivariate regression

Mathematics Subject Classification

  • 62J05
  • 91C15
  • 97B99

JEL Classification

  • L14
  • L25