Skip to main content

Thomas theorem in research evaluation

Abstract

The well-known “Thomas theorem” in sociology is defined as follows: “if men define situations as real, they are real in their consequences” (Thomas and Thomas, The child in America, Knopf, Oxford, 1928, p. 572). The theorem focuses on “objective consequences of human subjectivity” (Sztompka, Robert K. Merton: An intellectual profile, Macmillan Education, Limited, London, 1986, p. 229). In this Letter to the Editor, we transfer the thought content of the Thomas theorem to university rankings: if rank positions between two universities define performance differences as real, they are real in their consequences (although the university ranking shows only slight differences between the universities’ scores).

International university rankings have become very popular since their introduction at the beginning of the twenty-first century. Currently, six important university rankings exist, including the Academic Ranking of World Universities (ARWU) by Shanghai Ranking Consultancy and the Leiden Ranking by the Centre for Science and Technology Studies (CWTS) (see Huang 2012). In this Letter to the Editor, we would like to elaborate on one of the problems with these rankings: the highlighting of differences between universities by focusing on rank positions, although in fact, only small differences exist in terms of performance. As a rule, the distribution of the institutional performance values is highly skewed: there are far greater differences between the top positions and the rest than there are between the middle and the lower positions. Consequently, small differences in indicator values result in large differences in ranking positions. In many cases, the small differences between universities are located within the error bars of the institutional performance values and hence are meaningless.

References to this problem of university rankings can be found in various papers. Claassen (2015) included several university rankings in a meta-ranking and concluded: “Harvard can be distinguished from institutions ranked outside the top 15 and the top 10 can be said to be of significantly higher quality than those ranked in the 30s and below. But these institutions in the 30s cannot be differentiated from others that are still in the top 100, but lower in ranking” (p. 802). According to Johnes (2018), “the differences between universities … might actually be very slight, yet the rankings suggest to the laypeople who use them that distinctions in performance are potentially large” (p. 603). Piro and Sivertsen (2016) indicate “the huge emphasis on rank position, where the universities’ positions may differ greatly based on statistical insignificant scores” (p. 2264). The authors empirically analyzed differences between Scandinavian universities in two influential university rankings. They concluded that “the rank differences between the highest ranked Nordic universities are mostly not of a substantial kind (i.e. very small differences in indicator scores), but rather based on either size differences or differences in historical or path-dependent features such as Nobel Prizes or on what has been described in several studies as the ‘anchoring effect’, i.e. that the rankings themselves guide future assessments of quality in reputation surveys “ (p. 2275).

In this Letter to the Editor, we propose that this problem of university rankings indicated in several papers (the overemphasis of ranking positions) can be connected to the so-called Thomas theorem in sociology (see Merton 1995): “if men define situations as real, they are real in their consequences” (Thomas and Thomas 1928, p. 572). The theorem focusses on the “objective consequences of human subjectivity” (Sztompka 1986, p. 229). According to Merton (1976) the Thomas theorem is “the single most consequential sentence ever put in print by an American sociologist” (p. 174). We would like to transfer the thought content of the Thomas theorem to university rankings: if ranking positions define performance differences between universities as real, they are real in their consequences (although the university ranking shows only small differences between the universities’ scores). Hammarfelt et al. (2017) formulated a similar statement about university rankings when they pointed out that “rankings as a social technology make highly diverse entities—like universities—measurable through quantitative methods; or in other terms, rankings make universities ‘calculable’” (p. 393). For Bowman and Bastedo (2011), ranking positions function as anchors influencing peer assessments of institutional reputation.

Recently, Waltman et al. (2017) formulated ten principles for the responsible use of university rankings. One of these principles focuses on the handling of ranking positions and indicator values by recommending that “an exclusive focus on the ranks of universities in a university ranking should be avoided; the values of the underlying indicators should be taken into account”. However, university rankings are not the only area in which ranking positions ‘change the world by defining the situation’. Another example are journal indicators, especially the Journal Impact Factor (JIF), annually published by Clarivate Analytics in the Journal Citation Report (JCR). The JIF—roughly defined as the mean citation impact of the papers published in a journal within one year – is published with three decimal places, which allows the ranking of journals despite small citation impact differences between the journals. For example, there are 314 journals in the JCR 2018 journal set ‘mathematics’. The Journal of the Australian Mathematical Society is at ranking position 210 with a JIF of 0.602 (thus, the papers in the journal received about half a citation on average). With nearly the same JIF (0.407), the Journal of Commutative Algebra is at ranking position 284 (thus, 74 positions below the Journal of the Australian Mathematical Society).

There is a strong demand from science policymakers (and the public) for numbers, and particularly for rankings, because rankings clearly reveal the winners and losers among competing units (Marewski et al. 2018). This has made rankings highly fascinating and popular. University rankings are the easiest way of comparing complex research (and teaching) units such as universities (however, without considering their different missions). Against the backdrop of the heterogeneity of such units and the usual small institutional performance differences, large differences of ranking positions in the large middle field become meaningless and should not be cited. Therefore, with this Letter to the Editor, our aim is to point out the danger of mis- or over-interpretation of university rankings by introducing the Thomas Theorem as a label for this general problem with university rankings. In our opinion, the effect of the Thomas theorem in university rankings can be avoided by refraining from institutional ranking positions and by focusing on presenting errors bars (without the point estimate, e.g., the mean).

References

  1. Bowman, N. A., & Bastedo, M. N. (2011). Anchoring effects in world university rankings: Exploring biases in reputation scores. Higher Education,61(4), 431–444. https://doi.org/10.1007/s10734-010-9339-1.

    Article  Google Scholar 

  2. Claassen, C. (2015). Measuring university quality. Scientometrics,104(3), 793–807. https://doi.org/10.1007/s11192-015-1584-8.

    Article  Google Scholar 

  3. Hammarfelt, B., de Rijcke, S., & Wouters, P. (2017). From eminent men to excellent universities: University rankings as calculative devices. Minerva,55, 391–411. https://doi.org/10.1007/s11024-017-9329-x.

    Article  Google Scholar 

  4. Huang, M. H. (2012). Opening the black box of QS World University Rankings. Research Evaluation,21(1), 71–78. https://doi.org/10.1093/reseval/rvr003.

    Article  Google Scholar 

  5. Johnes, J. (2018). University rankings: What do they really show? Scientometrics,115(1), 585–606. https://doi.org/10.1007/s11192-018-2666-1.

    Article  Google Scholar 

  6. Marewski, J.N., & Bornmann, L. (2018). Opium in science and society: Numbers. Retrieved January 14, 2020, from https://arxiv.org/abs/1804.11210

  7. Merton, R. K. (1976). Social knowledge and public policy. Sociological ambivalence (pp. 156–179). New York: Free Press.

    Google Scholar 

  8. Merton, R. K. (1995). The Thomas theorem and the Matthew effect. Social Forces,74(2), 379–422. https://doi.org/10.2307/2580486.

    Article  Google Scholar 

  9. Piro, F. N., & Sivertsen, G. (2016). How can differences in international university rankings be explained? Scientometrics,109, 2263–2278. https://doi.org/10.1007/s11192-016-2056-5.

    Article  Google Scholar 

  10. Sztompka, P. (1986). Robert K. Merton: An intellectual profile. London: Macmillan Education, Limited.

    Book  Google Scholar 

  11. Thomas, W. I., & Thomas, D. S. (1928). The child in America. Oxford: Knopf.

    Google Scholar 

  12. Waltman, L., Wouters, P., & van Eck, N. J. (2017). Ten principles for the responsible use of university rankings. Retrieved June 6, 2017, from https://www.cwts.nl/blog?article=n-r2q274&title=ten-principles-for-the-responsible-use-of-university-rankings

Download references

Acknowledgements

Open Access funding provided by Projekt DEAL.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bornmann, L., Marx, W. Thomas theorem in research evaluation. Scientometrics 123, 553–555 (2020). https://doi.org/10.1007/s11192-020-03389-6

Download citation

Keywords

  • Bibliometrics
  • Thomas theorem