Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings
- 1.2k Downloads
We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.
KeywordsBibliometric analysis University rankings Language of publication
The WoS data system covers a number of journals in non-English languages, particularly in German and in French. Publications in these non-English language journals are counted as part of the output of countries, but they generally have a very low impact because only a few scientists outside Germany, Austria and Switzerland are able to read German; we have a similar situation for French. Thus, these non-English publications will ‘dilute’ considerably the average impact of countries such as Germany, Austria and France. This is particularly the case for the more application-oriented fields such as clinical medicine and engineering, and also for the social sciences and humanities. Because clinical medicine represents a considerable part of the entire scientific output of a country (particularly in western countries), this language effect directly influences the overall impact position of universities (van Leeuwen et al. 2001).
This effect has serious consequences for all citation-per-paper based rankings, but also for citations per staff rankings. The most striking result is a too low position of most German and France universities in these rankings, particularly universities with a medical school. As citations play an important role in university rankings such as the Times Higher Education, QS, Shanghai (ARWU) and the Leiden Ranking, all these rankings suffer from this language effect.
Thus, for ranking and comparison with countries such as the US, UK and all countries where almost all WoS-covered publications are written in English, calculation of bibliometric indicators based on only-English publications is a more fair procedure.
In the above context, we focus on the language effect of, particularly European universities in the German and French language areas, but a similar effect will also be present for Italian and Spanish universities. The issue of publication language is related to the problem of ‘local journals’ (but covered in the Web of Science). These journals play only a marginal role in the scientific communication process at an international scale. The publications in these journals mainly transfer knowledge to local or national end-users. For instance, medical knowledge transferred to non-academic hospitals to improve clinical practice. In the technical sciences we find transfer of new knowledge to the engineering practice. As most fields contain local journals, these journals should ideally be excluded from the sets forming the basis for bibliometric studies in international comparison. This is not a simple task and it requires further bibliometric studies.
One can expect that similar language effects will occur when using Scopus data and that this effect will seriously aggravate by the extension of the coverage of the WoS and Scopus with national language journals.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- Leiden Ranking (2010). http://www.socialsciences.leiden.edu/cwts/products-services/leiden-ranking-2010-cwts.html.