An Evaluation of Evaluators: Multivariate Statistical Analysis of Journal Evaluation Indicators

Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 241)


Journal evaluation has blossomed over the past four decades. Not only the bibliometric indicators provided by the Journal Citation Reports (JCR), but alternatives such as Google Scholar, Scopus and many other recently-introduced indicators are becoming popular. However, high correlations between journal evaluation indicators indicate that the development of new variants of the indicators has resulted in hardly any additional empirical contribution, so the application of existing indicators is a more promising direction for future research. This paper attempts to show how to evaluate and classify journals using current journal evaluation indicators by various multivariate statistical analysis methods. Data were collected from all journals in the Operations Research and Management Science category of JCR [10] and SCImago Journal and Country Rank ( Analysis in other scholarly fields can be conducted using the same method but with different category data in JCR and SCImago Journal and Country Rank.


Journal evaluation Bibliometric indicators Multivariate statistical analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Garfield E (2006) The history and meaning of the journal impact factor. Journal of the American Medical Association 295:90–93Google Scholar
  2. 2.
    Garfield E, Merton RK (1979) Citation indexing, its theory and application in science, technology, and humanities. Wiley, New YorkGoogle Scholar
  3. 3.
    Pinski G, Narin F (1976) Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing & Management 12:297–312Google Scholar
  4. 4.
    Moussa S, Touzani M (2010) Ranking marketing journals using the Google Scholar-based hg-index. Journal of Informetrics 4:107–117Google Scholar
  5. 5.
    Neuhaus C, Daniel HD (2008) Data sources for performing citation analysis: An overview. Journal of Documentation 64:193–210Google Scholar
  6. 6.
    Braun T, Glanzel W, Schubert A (2006) A Hirsch-type index for journals. Scientometrics 69:169–173Google Scholar
  7. 7.
    Gonzalez-Pereira B, Guerrero-Bote VP,Moya-Anegón F (2010) A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics 4:379–391Google Scholar
  8. 8.
    Braun T, Glanzel W, Schubert A (2006) A Hirsch-type index for journals. Scientometrics 69:169–173Google Scholar
  9. 9.
    Garfield E (2006) Citation indexes to science: A new dimension in documentation through association of ideas. International Journal of Epidemiology 35:1123–1127Google Scholar
  10. 10.
    Journal Citation Reports (2012), Journal Citation Reports Help: Immediacy Index, available at: (accessed April 12, 2013)
  11. 11.
    Lehmann S, Jackson AD, Lautrup BE (2008) A quantitative analysis of indicators of scientific performance. Scientometrics 76:369–390Google Scholar
  12. 12.
    Bordons M, Fernández MT, Gomez I (2002) Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics 53:195–206Google Scholar
  13. 13.
    Bornmann L, Mutz R, Daniel HD (2008) Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology 59:830–837Google Scholar
  14. 14.
    Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 1–38Google Scholar
  15. 15.
    Schafer JL, Olsen MK (1998) Multiple imputation for multivariate missing-data problems: A data analyst’s perspective. Multivariate Behavioral Research 33:545–571Google Scholar
  16. 16.
    Kaiser HF (1974) An index of factorial simplicity. Psychometrika 39:31–36Google Scholar
  17. 17.
    Bergstrom CT, West JD, Wiseman MA (2008) The EigenfactorTMmetrics. The Journal of Neuroscience 28: 11433–11434Google Scholar
  18. 18.
    Xu JP, Li ZM, Shen WJ et al (2013) Multi-attribute comprehensive evaluation of individual research output based on published research papers. Knowledge-Based Systems 43:135–142Google Scholar
  19. 19.
    Liu ZL, Zhang JL, Zou LF (2012) The Evaluation Index System for Enterprise IT Service Quality. In: The Sixth International Conference of Management Science and Engineering Management 219–225Google Scholar
  20. 20.
    Liu ZL, Zhang JL, Zou LF. (2012) A comprehensive assessment of commercial health & fitness clubs’ service quality. International Journal of Management Science and Engineering Management 7:263–267Google Scholar
  21. 21.
    Smith SD (2004) Is an article in a top journal a top article? Financial Management 133–149Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Decision Sciences DepartmentLeBow College of Business, Drexel UniversityPhiladelphiaUSA
  2. 2.Uncertainty Decision-Making LaboratorySichuan UniversityChengduPeople’s Republic of China

Personalised recommendations