, Volume 95, Issue 1, pp 435–452 | Cite as

Benchmarking research performance at the university level with information theoretic measures

  • J. A. GarcíaEmail author
  • Rosa Rodriguez-Sánchez
  • J. Fdez-Valdivia
  • Nicolas Robinson-García
  • Daniel Torres-Salinas


This paper presents a new method for comparing universities based on information theoretic measures. The research output of each academic institution is represented statistically by an impact-factor histogram. To this aim, for each academic institution we compute the probability of occurrence of a publication with impact factor in different intervals. Assuming the probabilities associated with a pair of academic institutions our objective is to measure the Information Gain between them. To do so, we develop an axiomatic characterization of relative information for predicting institution-institution dissimilarity. We use the Spanish university system as our scenario to test the proposed methodology for benchmarking three universities with the rest as a case study. For each case we use different scientific fields such as Information and Communication Technologies, Medicine and Pharmacy, and Economics and Business as we believe comparisons must take into account their disciplinary context. Finally we validate the Information Gain values obtained for each case with previous studies.


Information gain Institution-institution similarity Impact-factor histogram Information theoretic measure Information conservation constraint Benchmarking research output 



This research was sponsored by the Spanish Board for Science and Technology (MICINN) under grant TIN2010-15157 cofinanced with European FEDER funds. Nicolás Robinson-García is currently supported by a FPU grant from the Spanish Ministerio de Educación y Ciencia. Thanks are due to the reviewers for their constructive suggestions.


  1. Adams, J., Gurney, K. & Marshall, S. (2007). Profiling citation impact: A new methodology. Scientometrics, 72(2), 325–344.CrossRefGoogle Scholar
  2. Abramo, G., D Angelo, D. A. & Di Costa, F. (2011). National research assessment exercises: A comparison of peer review and bibliometric rankings. Scientometrics, 89(3), 929–941.CrossRefGoogle Scholar
  3. Burbea, J., & Rao, C.R. (1983). On the convexity of divergence measures based on entropy function. IEEE Transactions on Information Theory, 28(3), 489–495.MathSciNetCrossRefGoogle Scholar
  4. Carpenter, M. P., Gibb, F., Harris, M., Irvine, J., Martin, B. R. & Narin, F. (1988). Bibliometric profiles for British academic institutions: An experiment to develop research output indicators. Scientometrics, 14(3–4), 213–233.CrossRefGoogle Scholar
  5. Dolado, J. J., García-Romero, A. & Zamarro, G. (2003). Publishing performance in economics: Spanish rankings (1990–1999). Spanish Economic Review, 5, 85–100.CrossRefGoogle Scholar
  6. Fisher, R. A., (1950). The logic of inductive inference. Journal of the Royal Statistical Society, 98, 39–54; Contributions to Mathematical Statistics. John Wiley and Sons, New York, paper 26.Google Scholar
  7. Garcia, J. A., Fdez-Valdivia, J, Fdez-Vidal, X. R., & Rodriguez-Sanchez, R. (2001). Information theoretic measure for visual target distinctness. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4), 362–383.CrossRefGoogle Scholar
  8. García, J. A., Rodríguez-Sánchez, R., Fdez-Valdivia, J., Robinson-García, N. & Torres-Salinas. (2012a). Mapping academic institutions according to their journal publication profile: Spanish universities as a case study. Journal of the American Society for Information Science and Technology. Accepted for publication.Google Scholar
  9. García, J. A., Rodríguez-Sánchez, R., Fdez-Valdivia, J., Torres-Salinas, D. & Herrera, F. (2012b). Ranking of research output of universities on the basis of the multidimensional prestige of influential fields: Spanish universities as a case of study. Scientometrics. doi: 10.1007/s11192-012-0740-7.
  10. Havrda, J.H., and Charvat, F. (1967). Quantification method of classification processes: Concept of structural α-entropy’. Kybernetika, 3, 30–35.MathSciNetzbMATHGoogle Scholar
  11. Hazelkorn, E. (2011). Rankings and the reshaping of higher education: The battle for world-class excellence. Basingstoke, New York: Palgrave-MacMillan.CrossRefGoogle Scholar
  12. Jeffreys, H. (1946). An invariant form for the prior probability in estimating problems. Proceedings of the Royal Society of London, 186A, 453–461.MathSciNetGoogle Scholar
  13. Kapur, J. N. (1984). A comparative assessment of various measures of directed divergence. Advances in Management Studies, 3, 1–16.CrossRefGoogle Scholar
  14. Kullback, S. (1978). Information theory and statistics. Gloucester, Mass: Peter Smith.Google Scholar
  15. Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79–86.MathSciNetzbMATHCrossRefGoogle Scholar
  16. Lubrano, M., Bauwens, L., Kirman, A. & Protopopescu, C. (2003). Ranking economics departments in Europe: A statistical approach. Journal of the European Economic Association, 1, 1367–1401.CrossRefGoogle Scholar
  17. Lundberg, J. (2006). Bibliometrics as a research assessment tool impact beyond the impact factor [PhD dissertation]. Stockholm: Karolinska Institutet.Google Scholar
  18. Moed, H. F. (2008). UK research assessment exercises: Informed judgements on research quality or quantity? Scientometrics, 74(1), 153–161.CrossRefGoogle Scholar
  19. Ortega, J. L., Lopez-Romero, E. & Fernandez, I. (2011). Multivariate approach to classify research institutes according to their outputs: The case of the CSIC’s institutes. Journal of Informetrics, 5(3), 323–332.Google Scholar
  20. Rao, C. R. (1982). Diversity and dissimilarity coefficients: A unified approach. Theoretic Population Biology, 21(1), 24–43.zbMATHCrossRefGoogle Scholar
  21. Renyi, A. (1961). On measures of entropy and information. Proceedings of the Forth Berkeley Symposium on Mathematical Statistics and Probability, 1, 547–561.MathSciNetGoogle Scholar
  22. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423; 623–656.Google Scholar
  23. Sharma, B.D., and Mittal, D.P. (1977). New non-additive measures of relative information. Journal of Combinatorics and Information System Sciences, 2, 122–132.MathSciNetzbMATHGoogle Scholar
  24. Shin, J. C. (2009). Classifying higher education institutions in Korea: a performance-based approach. Higher Education, 57(2), 247–266.CrossRefGoogle Scholar
  25. Tijssen, R. J. W., van Leeuwen, T. N. & van Wijk, E. (2009). Benchmarking university-industry research cooperation worldwide: performance measurements and indicators based on co-authorship data for the world’s largest universities. Research Evaluation, 18(1), 13–24.Google Scholar
  26. Torres-Salinas, D., Moreno-Torres, J. G., Delgado-Lopez-Cozar, E. & Herrera, F. (2011a). A methodology for Institution-Field ranking based on a bidimensional analysis: The IFQ2A index. Scientometrics, 88(3), 771–786.CrossRefGoogle Scholar
  27. Torres-Salinas, D., Moreno-Torres, J. G., Robinson-García, N., Delgado-López-Cózar, E., Herrera, F. (2011b). Rankings ISI de las Universidades Españolas según campos y disciplinas científicas (Second ed. 2011). El Profesional de la Información, 20(6), 701–709.CrossRefGoogle Scholar
  28. Turner, D. (2007). Benchmarking universities: League tables revisitedOxford Review of Education, 31(3), 353–371.CrossRefGoogle Scholar
  29. Vanclay, J. K. & Bornmann, L. (2012). Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative.Scientometrics, 91(3), 751–771.CrossRefGoogle Scholar
  30. Wiener, N. (1950). The Human use of human beings. Boston: Houghton Mifflin Co.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  • J. A. García
    • 1
    Email author
  • Rosa Rodriguez-Sánchez
    • 1
  • J. Fdez-Valdivia
    • 1
  • Nicolas Robinson-García
    • 2
  • Daniel Torres-Salinas
    • 3
  1. 1.Departamento de Ciencias de la Computación e I. A., CITIC-UGRUniversidad de GranadaGranadaSpain
  2. 2.EC3: Evaluación de la Ciencia y la Comunicación CientíficaUniversidad de GranadaGranadaSpain
  3. 3.EC3: Evaluación de la Ciencia y la Comunicación Científica, Centro de Investigación Médica AplicadaUniversidad de NavarraPamplonaSpain

Personalised recommendations