, Volume 98, Issue 3, pp 1955–1970 | Cite as

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

  • Nicolás Robinson-García
  • Clara Calero-Medina


University rankings by fields are usually based on the research output of universities. However, research managers and rankings consumers expect to see in such fields a reflection of the structure of their own organizational institution. In this study we address such misinterpretation by developing the research profile of the organizational units of two Spanish universities: University of Granada and Pompeu Fabra University. We use two classification systems, the subject categories offered by Thomson Scientific which are commonly used on bibliometric studies, and the 37 disciplines displayed by the Spanish I-UGR Rankings which are constructed from an aggregation of the former. We also describe in detail problems encountered when working with address data from a top down approach and we show differences between universities structures derived from the interdisciplinary organizational forms of new managerialism at universities. We conclude by highlighting that rankings by fields should clearly state the methodology for the construction of such fields. We indicate that the construction of research profiles may be a good solution for universities for finding out levels of discrepancy between organizational units and subject fields.


University rankings Fields Address data Institutional structure Subject classification 



Thanks are due to the two anonymous referees for their constructive suggestions. The authors would also like to thank Thed N. van Leeuwen and Daniel Torres-Salinas for their helpful comments on previous versions of this paper. Nicolás Robinson-García is currently supported by a FPU Grant from the Spanish Ministerio de Economía y Competitividad.


  1. Borgatti, S. P., Mehra, A., Brass, D. J., & Labianca, G. (2009). Network analysis in the social sciences. Science, 323, 892–895.CrossRefGoogle Scholar
  2. Bornmann, L., Moya-Anegón, F., & Mutz, R. (2013a). Do universities or research institutions with a specific profile have an advantage or a disadvantage in institutional rankings? A latent class analysis with data from the SCImago Ranking. Journal of the American Society for Information Science and Technology. doi: 10.1002/asi.22923.
  3. Bornmann, L., Mutz, R., & Daniel, H.-D. (2013b). Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012. Journal of the American Society for Information Science and Technology, 64(8), 1649–1658.CrossRefGoogle Scholar
  4. Bornmann, L., Stefaner, M., Moya-Anegón, F., & Mutz, R. (2013c). Ranking and mapping of universities and research-focused institutions worldwide based on highly-cited papers: A visualization of results from multi-level models. Online Information Review. arXiv:1212.0304.Google Scholar
  5. Calero-Medina, C., & van Leeuwen, T. N. (2012). Reorganizing research with the help of bibliometric collaboration networks. Case study in a University Hospital. In C. Calero-Medina (Ed.), Links in science: Linking network and bibliometric analyses in the study of research performance [Doctoral thesis] (pp. 45–66). Leiden: Universiteit Leiden.Google Scholar
  6. Cheng, Y., & Liu, N. C. (2006). A first approach to the classification of the top 500 world universities by their disciplinary characteristics using scientometrics. Scientometrics, 68(1), 135–150.CrossRefGoogle Scholar
  7. Cuxac, P., Lamirel, J.-C., & Bonvallot, V. (2013). Efficient supervised and semi-supervised approaches for affiliations disambiguation. Scientometrics, 97(1), 47–58.Google Scholar
  8. De Bruin, R. E., & Moed, H. F. (1993). Delimitation of scientific subfields using cognitive words from corporate addresses in scientific publications. Scientometrics, 26(1), 65–80.CrossRefGoogle Scholar
  9. Deaton, A. (1997). Analysis of household surveys. Baltimore: Johns Hopkins University Press.CrossRefGoogle Scholar
  10. García, J. A., Rodriguez-Sánchez, R., Fdez-Valdivia, J., Robinson-Garcia, N., & Torres-Salinas, D. (2012). Mapping academic institutions according to their journal publication profile: Spanish universities as a case study. Journal of the American Society for Information Science and Technology, 63(11), 2328–2340.CrossRefGoogle Scholar
  11. Gibbons, M., Limoges, C., Nowotny, H., Schwartzmann, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: Sage.Google Scholar
  12. Glänzel, W., & Schubert, A. (2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56(3), 357–367.CrossRefGoogle Scholar
  13. Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61(11), 2365–2369.CrossRefGoogle Scholar
  14. López-Illescas, C., Moya-Anegón, F., & Moed, H. F. (2011). A ranking of universities should account for differences in their disciplinary specialization. Scientometrics, 88(2), 563–574.CrossRefGoogle Scholar
  15. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 381–422.Google Scholar
  16. Morris, N. (2002). The developing role of departments. Research Policy, 31(5), 817–833.CrossRefGoogle Scholar
  17. Praal, F., Koosten, J., Calero-Medina, C., & Visser, M. S. (2013). Ranking universities: The challenge of affiliated institutes. In 18th International Conference on Science and Technology Indicators.Google Scholar
  18. Pudovkin, A. I., & Garfield, E. (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53(13), 1113–1119.CrossRefGoogle Scholar
  19. Rescher, N. (2006). Epistemetrics. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  20. Robinson-García, N., Moreno-Torres, J. G., Torres-Salinas, D., López-Cózar, E. D., & Herrera, F. (2013a). The role of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities. In Conference Proceedings from the 14th International Society for Scientometrics and Informetrics Conference (Vol. 2, pp. 1550–1565).Google Scholar
  21. Robinson-García, N., Rodriguez-Sanchez, R., García, J. A., Torres-Salinas, D., & Fdez-Valdivia, J. (2013b). Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas. Revista Española de Documentación Científica (in press).Google Scholar
  22. Torres-Salinas, D., Moreno-Torres, J. G., López-Cózar, E. D., & Herrera, F. (2011). A methodology for institution-field ranking based on a bidimensional analysis: The IFQ2A-index. Scientometrics, 88(3), 771–786.CrossRefGoogle Scholar
  23. Van Leeuwen, T. N. (2007). Modeling of bibliometric approaches and importance of output verification in research performance assessment. Research Evaluation, 16(2), 93–105.CrossRefGoogle Scholar
  24. Van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.CrossRefGoogle Scholar
  25. Waltman, L., & van Eck, N. J. (2012). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.Google Scholar
  26. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R. J. W., van Eck, N. J., et al. (2012). The Leiden Ranking 2011/2012: Data collection indicators and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2013

Authors and Affiliations

  • Nicolás Robinson-García
    • 1
  • Clara Calero-Medina
    • 2
  1. 1.EC3: Evaluación de la Ciencia y de la Comunicación Científica, Departamento de Información y ComunicaciónUniversidad de GranadaGranadaSpain
  2. 2.Centre for Science and Technology StudiesLeiden UniversityLeidenThe Netherlands

Personalised recommendations