Abstract
In this paper, we examine whether the quality of academic research can be accurately captured by a single aggregated measure such as a ranking. With Shanghai University’s Academic Ranking of World Universities as the basis for our study, we use robust principal component analysis to uncover the underlying factors measured by this ranking. Based on a sample containing the top 150 ranked universities, we find evidence that, for the majority of these institutions, the Shanghai rankings reflect not one but in fact two different and uncorrelated aspects of academic research: overall research output and top-notch researchers. Consequently, the relative weight placed upon these two factors determines to a large extent the final ranking.
Similar content being viewed by others
Notes
In standardized PCA, the eigenvectors are obtained from the correlation matrix.
Computed after removing the variable “PCP” and transferring its weight equally between the remaining five variables.
Since we are working here with standardized variables, the overall variance here is equal to five, the number of variables.
Obviously, if each university were to contribute an equal share in a factor, that contribution would be 0.67%.
The first principal component is measured on the first principal axis.
Under the normality assumption, the Mahalanobis distances are distributed as the square root of a χ2 distribution with five degrees of freedom.
The STATA code is available from the authors upon request (Verardi and Dehon 2009).
References
Dehon, C., Jacobs, D., & Vermandele, C. (2009). Ranking Universities. Brussels: Editions de l’Université de Bruxelles.
Liu N. C., & Cheng, Y. (2005). Academic ranking of world universities: Methodologies and problems. Higher Education in Europe, 30(2), 127–136.
Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, Calcutta, 2(1), 49–55.
Marcotte, C., Desroches, J., & Poupart I. (2007). Preparing internationally minded business graduates: The role of international mobility programs. International Journal of Intercultural Relations, 31, 655–668.
Razvan, F. (2007). Irreproducibility of the results of the Shanghai academic ranking of world universities. Scientometrics, 72(1), 25–32.
Rousseeuw P. J. (1985). Multivariate estimation with high breakdown point. Mathematical Statistics and Applications, B, 283–297.
van Raan, A. (2005a). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.
van Raan, A. (2005b). Challenges in ranking of Universities, Invited paper for the first international conference on World Class Universities. Shanghai: Shanghai Jaio Tong University, June 16–18, 2005.
Verardi, V., & Dehon C. (2009). Multivariate outlier detection in stata. Working paper.
Vincke, Ph. (2009). University rankings. In C. Dehon, D. Jacobs, & C. Vermandele (Eds.), Ranking Universities (pp. 11–25). Brussels: Editions de l’Université de Bruxelles.
Zitt, M., & Filliatreau, G. (2007). Big is (made) beautiful: Some comments about the Shanghai ranking of world-class universities. In J. Sadlack, & N. C. Liu (Eds.), The world class university and ranking: Aiming beyond status, (pp. 141–160). Romania: UNESCO-CEPES, Cluj University Press.
Acknowledgements
The authors would like to thank Nadine Rons, Ellen Hazelkorn and Marjorie Gassner for their insightful comments. Vincenzo Verardi is Associate Researcher of the FNRS and gratefully acknowledges their financial support. All remaining errors are the authors' responsibility.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Dehon, C., McCathie, A. & Verardi, V. Uncovering excellence in academic rankings: a closer look at the Shanghai ranking. Scientometrics 83, 515–524 (2010). https://doi.org/10.1007/s11192-009-0076-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-009-0076-0