Skip to main content

Reproducibility of the Shanghai academic ranking of world universities results

Abstract

This paper discusses and copes with the difficulties that arise when trying to reproduce the results of the Shanghai academic ranking of world universities. In spite of the ambiguity of the methodology of the ranking with regard to the computation of the scores on its six indicators, the paper presents a set of straightforward procedures to estimate raw results and final relative scores. Discrepancies between estimated scores and the results of the ranking are mostly associated with the difficulties encountered in the identification of institutional affiliations, and are not significant. We can safely state that the results of the Shanghai academic ranking of world universities are in fact reproducible.

This is a preview of subscription content, access via your institution.

References

  1. Australian Government (2011) Australian Government, Department of Education, Employment and Workplace Relations: Staff 2010 Full Time Equivalence. Web address: http://www.deewr.gov.au/HigherEducation/Publications/HEStatistics/Publications/Pages/Staff.aspx.

  2. Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shangai ranking: an MCDM view. Scientometrics, 84(1), 237–263.

    Article  Google Scholar 

  3. Castells, M. (2005). The Network Society: From Knowledge to Policy, chap The Network Society: From Knowledge to Policy (pp. 3–22). Washington: Johns Hopkins Center for Transatlantic Relations.

  4. Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: a closer look at the Shanghai ranking. Scientometrics, 83(2), 515–524.

    Article  Google Scholar 

  5. Docampo, D. (2008). International rankings and quality of the university systems. Revista de Educación, Special Issue, 149–176.

  6. Docampo, D. (2011). On using the Shanghai ranking to assess the research performance of university systems. Scientometrics, 86(1), 77–92.

    Article  Google Scholar 

  7. Docampo, D. (2012). Universities from Spain in the 2011 edition of the Shanghai ranking. Submitted to Revista de Educación. Working paper available at http://www.gts.tsc.uvigo.es/∼ddocampo/ONLINE/docamporevistaeducacion.pdf.

  8. Florian, R. V. (2007). Irreproducibility of the results of the Shangai academic ranking of world universities. Scientometrics, 72(1), 25–32.

    MathSciNet  Article  Google Scholar 

  9. Hazeltorn, E. (2008). Learning to live with league tables and ranking: the experience of institutional leaders. Higher Education Policy, 21, 193–215.

    Article  Google Scholar 

  10. Himanen, P. (2005). The Network Society: From Knowledge to Policy, chap Challenges of the Global Information Society (pp. 337–372). Washington, DC: Johns Hopkins Center for Transatlantic Relations.

  11. Ioannidis, J. P. A., Patsopoulos, N. A., Kavvoura, F. K., Tatsioni, A., Evangelou, E., Kouri, I., Contapoulos Ioannidis, D. G., & Liberopoulos, G. (2007). International ranking systems for universities and institutions: a critical appraisal. BMC Medicine, 5:30.

    Article  Google Scholar 

  12. Kivinen, O., & Hedman, J. (2008). World-wide university rankings: a Scandinavian approach. Scientometrics, 74(3), 391–408.

    Article  Google Scholar 

  13. Leyersdoff, L. (2012). World shares of publications of the USA EU-27 and China compared and predicted using the new Web of Science interface versus Scopus. El profesional de la información, 21(1), 43–49.

    Article  Google Scholar 

  14. Liu, N. C., & Cheng, Y. (2005). Academic ranking of world universities: Methodologies and problems. Higher Education in Europe, 30(2), 127–136.

    Article  Google Scholar 

  15. Liu, N. C., Cheng, Y., & Liu, L. (2005). Academic ranking of world universities using scientometrics: A comment to the “fatal attraction”. Scientometrics, 64(1), 101–109.

    Article  Google Scholar 

  16. Marginson, S. (2005) There must be some way out of here. Tertiary Educ. Management Conference. Perth: Keynote address.

  17. Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics, 5th edn). Boston: Pearson Education, Inc., Allyn and Bacon.

    Google Scholar 

  18. Van Parijs, P. (2009). European higher education under the spell of rankings. Ethical Perspectives, 16(2), 189–206.

    Article  Google Scholar 

  19. Van Raan, A. F. J. (2005a). Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.

    Article  Google Scholar 

  20. Van Raan, A. F. J. (2005b). Reply to the comments of Liu et al. Scientometrics, 64(1), 111–112.

    Article  Google Scholar 

  21. Zitt, M., & Filliatreau, G. (2007). The World Class Universities and Ranking: Siming Beyond Status, chap Big is (made) beautiful: Some comments about the Shangai ranking of world-class universities, Part Two, 141–160. Romania: UNESCO-CEPES, Cluj University Press.

Additional Data sources

  1. ARWU Scores http://www.shanghairanking.com/ARWU2011.html.

  2. Nobel laureates http://nobelprize.org/.

  3. Fields Medals http://www.mathunion.org/index.php?id=prizewinners.

  4. HiCi http://www.isihighlycited.com.

  5. N&S http://www.webofknowledge.com.

  6. PUB http://www.webofknowledge.com.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Domingo Docampo.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Docampo, D. Reproducibility of the Shanghai academic ranking of world universities results. Scientometrics 94, 567–587 (2013). https://doi.org/10.1007/s11192-012-0801-y

Download citation

Keywords

  • Shanghai
  • Ranking
  • Reproducibility
  • Results
  • ARWU