The legacy of quantitative studies of science: increasing complexity of the sciences
Quantitative studies of science have been continuously growing since the end of world war two. The exponential growth in the “golden age of science” (de Solla Price, 1963) was also the beginning of intensified studies about its differentiation, allowing for structures and mechanisms in the evolution of science to be more systematically described. Initially aimed at assisting researchers in finding their ways through an ever-expanding scholarly universe (Garfield, 1968), the role of scientometrics, gradually changed with its academic success and with growing recognition within the policy realm (van Leeuwen 2004, de Rijcke & Rushforth, 2015) such as the emergence of research information systems (RIS) and national science reporting. Today the complexity of the social, economic and epistemic structures of the sciences is still growing, while effects of science policy interventions, changes in scholarly publishing, and trends to open scholarship increasingly demand analyses for processes rather than only focussing on the outputs of knowledge production. Against that background, this special issue aims to present research from the funding line “quantitative science studies” in Germany, leaving room to reflect on the current state of quantitative science studies and scientometrics differentiating between explorative, evaluative and reflexive approaches.
Explorative, evaluative and reflexive bibliometrics
The field of bibliometrics and quantitative science studies is broad, incorporating knowledge and thinking from different fields and disciplines (Moed, 2005; van Raan, 2004). Yet, broadly speaking, three different territories in bibliometrics can be distinguished, explorative, evaluative, and, more recently, reflexive bibliometrics.
Explorative Bibliometrics are understood as bibliometric analyses aiming at identifying structures and actors in science, such as the rise and fall of topical landscapes, the change or emergence of research fields, which can, for instance, be investigated by exploring publishing patterns, citation flows or collaboration structures of research entities at different levels.
Evaluative bibliometrics understood as studies aiming at studying aspects of scientific performance and quality (van Leeuwen 2004, p. 374). Several of these evaluation studies intended to assess research in a given area and subject, but often with the goal of monitoring and steering research. Technical standards for performing evaluation exercise were established, but scientometric and bibliometric methods were sometimes used in a way not intended by scientometricians (Glänzel, 2008). It is the shared concern about the consequences of evaluative bibliometrics that dominated the image of bibliometrics in the eyes of its critics. In particular, the tendency to use bibliometrics for ever more objects of research evaluation resulted in debates about the role of bibliometrics in the governance of science, as more realms of scholarly output can be measured, and ultimately, controlled. Concerns were raised as to whether scientometrics has sufficiently reflected upon and kept track of the consequences and effects of its own developments.
As a consequence, a third stream within bibliometrics is emerging which may be called reflexive bibliometrics, that is, the study of adaptation behaviour in relation to scientometric indicators and metrics. This stream builds closely links up to sociological studies of science by exploring the performative effects of scientometrics on the reward structures of science, as scholarly journals and scholarly communities change their publishing practices in accordance to indicators in a way not intended, leading to sometimes troublesome and visible developments such as impact factor gaming of scholarly journals (Falagas et al. 2008). While evaluative bibliometrics contributed largely to the success of the research field on the policy side, it may also be perceived as a development which allowed for the articulation of interests external to the community (de Rijcke & Rushforth, 2015).
The on-going Institutionalisation of scientometrics in Germany
Against that background, scientometrics and quantitative studies of science are increasingly reflecting its histories, its research practices and its institutions as a community, while keeping track of novel methods and technologies. Yet, while quantitative studies of science are well developed in some Western European countries, such as the Netherlands or Belgium, with close links to science policy studies and studies of research evaluation, this appears to be less so the case in other parts of Europe. Particularly in Germany, though a relatively large community does exist, the interdisciplinary field of science studies appears to be less institutionalized in terms of dedicated research centres and professorships at universities.
According to the German Council for the Sciences and the Humanities (Wissenschaftsrat, 2014), science studies and scientometrics should be supported in strengthening its academic structures, particularly enforcing interdisciplinary collaborations across various research organizations and across disciplines and research fields. The federal ministry of education and research (BMBF) in Germany has therefore launched initiatives to support this community with specific funding lines. One of the means to support the community was to safeguard the provision of bibliometric data and to enforce its use.
For this reason, the competence centre for bibliometrics (KB) has been founded. The goals of the competence center are twofold. First, to provide curated (meta-) data for the scientific community, and, second, to build up methodological competence and expertise for performing large scale scientometric analyses, e.g. through intensified studies of disambiguation techniques, matching algorithms and novel methodological developments, from which the community can benefit. Second, the federal ministry has also established a series of research funding lines in order to leverage the benefits of the KB and support the community. That funding line was published in 2017, with 23 projects lasting from 2018 until 2021 with differences in start and ends. The aim of the funding line is to support structures for research on the science system by broadening the base of researchers using bibliometric data, particularly those curated Meta data provided by the KB. The funding line furthermore aims at supporting a wide range of different research questions about the science system. Three different streams in the funding line were announced, closely related to the different bibliometric research strands–explorative, evaluative, reflexive bibliometrics—mentioned above. The first stream, structures and dynamics, supports analyses of differences in the science and higher education systems, taking institutional conditions into account in order to study differentiation of the science system. This funding line resonates well within the realm of explorative bibliometrics. The second stream, performance measurement and efficiency, addresses research focusing on the measurement of research quality and transfer. It is closely related to studies within evaluative bibliometrics. Finally, the third stream of the funding line, reflexive bibliometrics, however, deals with research on the effects of bibliometric indicators on the behaviour of scientists and research entities.
Topics in this special issue
This special issue brings together research conducted within this funding line, but it also shows currents trends and challenges in scientometrics. The collection of the articles demonstrates that an increased and broadened usage of the KB infrastructure has been achieved.
Enriching and experimenting with new data
Several approaches in the funding line included matching different data sources with data provided by the KB. In addition, the potential of novel data types is explored. For instance, Christian Thiele et al. have explored as to whether clinical trials can be used in order explore dynamics within the medical research community. Clinical trials, it is proposed, may more reliably represent the current state of clinical studies. If they are pre-registered they may reduce publication bias of medical research. The authors explore as to whether registries for clinical trials can be harvested for analyses evaluating clinical trials in Germany. In particular, the authors probe as to whether entries to the German DRKS data base and to ClinicalTrials Gov can be used for comprehensive overview of clinical studies in Germany, whereby duplicates of registries in both data bases are to be identified. The results show that the DRKS can only to a limited extent be used for Germany, as only half of all clinical studies are registered there. Thereby, the authors have shown that it is necessary to combine different registries and a meta register for producing a comprehensive overview of German clinical research, thereby widening the perspective for scientometrics when it comes to sufficiently map biomedicine. This reflects that the funding line has reached out to new actors and communities, coming from different disciplinary backgrounds, such as from medicine, computational linguistics, or computer science; emphasizing novel methods for information retrieval such as machine learning tools.
The openness transformation
Moreover, the collection shows that the community has opened up to new questions. One of the topics that the community is particularly interested in is the transformation towards open scholarship and the consequences this has for scholarly communication, scholarly publishing, and indicator development. Momeni et al. deal with how the flipping of scholarly journals towards Open Access influences their volume of publications and citation impact in order to gain insights as to whether the transition to OA brings positive results. They observed a higher number of publications after flipping but with big disciplinary variances. Hobert et al. analyse the Open Access uptake from an institutional perspective, taking the different organizational frameworks and missions of research institutes into account. Germany is an interesting case for such analyses given its diversity of research organizations and institutional conditions, with a strong and differentiated extra-university research sector. Relying on diverse sources (Web of Science, unpaywall, ISSN-Gold-OA 3.0 list, OpenDOAR), Hobert et al. found that research organization with a stronger orientation towards basic research, such as the Max Planck Society and the university sector show a stronger uptake of Open Access publishing compared to the applied research institutes, such as the Fraunhofer Society. The time frame for the investigation (before agreements with publishers were achieved) was strategically chosen, in order to allow for observing changes before and after the transformation agreements. The analysis shows how quantitative studies of science can be fruitfully combined with issues of governance and science policy in order to strengthen links with sociological and science policy studies.
Structural characteristics at different levels–systems, institutions, communities
While Hobert et al. focus on institutional characteristics of the German science, taking the differentiation of the research sector into account, Pellens et al. focus on another systemic aspect of research and innovation, that is, the participation of firms in scholarly publishing. The research intensity in the corporate sector is an important characteristic of developed research and innovation systems (Edquist, 1997) and the worries about declining engagement in publishing have been repeatedly been interpreted as signs of declining innovative capacities. Against that background, Pellens et al. explored publishing activities of German enterprises between 2008–2016 drawing from the Mannheim Enterprise Panel and Scopus. For Scopus, the institutional disambiguation procedure of the Bielefeld Bibliometric Group has been used in order to reliably identify affiliations of firms. Furthermore, patent data from the German Patent office have been retrieved. The results show that publishing volume of firms stays relatively constant over time, but are strongly concentrating on fewer firms. Yet, different to what could be expected, publications with basic research from firms slightly rise, due to increased collaboration with academia, confirming a larger trend for research and innovation systems.
While Hober et al. and Pellens et al. focus on the institutions in the research system (system level perspective), other contributions in this issue focus on specific fields and disciplines. Wieczorek et al., for instance, map the topical landscape of psychology between 1995 and 2015, applying so called structural topic models (Roberts et al. 2014) to reconstruct themes within a scholarly discipline. Wieczorek et al. hypothesize that particularly the establishment of neuroscience within psychology changed the topical landscape in such a way that the discipline is now stronger oriented towards natural sciences, while topics related to subfields leaning on the humanities lost their influence. Furthermore, the authors claim that associated with these changes comes a trend towards publishing in High Impact (HI) journals. Relying on a database of more than 500.000 abstracts queried from Web of Science, the authors explore the publishing landscape in the field. Furthermore, the authors use the metrics of semantic coherence (internal consistency) and exclusivity (the extent to which the words of a topic are distinct to it) in order to validate the number of topics to which a single publication can be related to. The authors found that the discipline of psychology is indeed changing towards natural science topics, becoming more closely linked to other biomedical fields as a clinical discipline. At the same time, however, the “multiparadigmatic character of the field still appears to be present.
A different approach to investigate structural characteristic of the science system on various level is the analysis of collaboration patterns. Such an approach has been taken to explore German Artificial Intelligence (AI) community by Koopmann et al. The authors aim at providing a novel view on its research dynamics by looking on the field’s collaborative structure depending on different measures of proximity. The authors investigate different dimensions of proximity, that is, cognitive (measured by similarity of textual characteristic), institutional (measured by similarity of sector membership), organizational (measured by similarity of affiliation), social (measured by similarity of academic background, e.g. PhD acquisition at the same organization), geographical (physical co-location) in their effect on collaboration, analysing the DBLP data set (a bibliometric data set of AI researchers), complemented by data from academic websites using German Academic Web (GAW) data set. It is found that social proximity can be perceived as the best indicator for collaboration. The authors experiment with novel approaches to test collaborations, applying the so called HypTrails approach developed for navigation to map collaboration patterns in research and technology (co-inventorship, co-authorship).
Rethinking impacts and performance
While the aforementioned papers focus on structural characteristics of disciplines, research fields, or even research systems, a second goal of the funding line was to fund research that explores, how performance and efficiency in diverse scientific contexts can be understood and measured. The papers collected here witness different viewpoints and perspectives on that issue. Focusing on the discipline of political science, Habicht et al. aimed at investigating productivity and performance patterns of scholars over the entire academic career. They rely on the theory of limited differences, which argues that small differences in researchers’ early productivity accumulate to large differences over long time spans, taking differences of gender, funding, characteristics of education into account. The research draws from a hand-coded data set of political scientists in Germany with at least one publication. The results confirm the thesis of cumulative advantage, with small productivity gains cumulating to larger differences over time. Furthermore, it is found that productivity slightly declines after tenure and that reception of funding leads to higher productivity. The results have interesting implications for policy, as early aspirations for publishing are important.
Another approach for exploring the effects of research are citation analyses, mapping the impact of scholarly output. With their study, Matthias Rüdiger et al. propose a text clustering approach for measuring impact, taking the context of the citation into account in order to measure for what claim a given publication has been cited. Different to established qualitative citation context analysis (Cozzens, 1985), the authors propose an automated method for thematic categories relying on a learned topic modelling algorithm. Such approaches may also give way to more nuanced views on impact as a form of intellectual contribution.
Questioning the impact-metric relationship
Finally, also publications referring to the last stream of the funding line are within this collection, asking questions about as to whether quantitative studies of science themselves have resulted in affecting scholarly behaviour, but also scholarly self-description (Haustein et al., 2016; Blümel and Gauch, 2021). In this way, Max Leckert has aimed at reconstructing patterns of argumentation in the field by comparing two manifestos. He found that these two manifestos have employed different narratives for making sense of quantifications. In dealing with the two manifestos, it becomes visible that the emergence of novel scholarly communication formats and the new unprecedented possibilities for tracing them, has reminded us that it is still difficult to understand what metrics in science may mean. The debate about the emerging field of altmetrics (Haustein et al., 2016; Costas et al., 2014), may be another opportunity to conceptualize what it means to perform a scholarly activity, what it means to measure it, and what can be done with these measurements. The explorations of Leckert reveal that quantitative studies of science is still a contested field, with different identities, particularly challenged by a new digital universe of scholarly information.
Altogether, the different contributions express the diversity of approaches in the field of quantitative science studies. While 70 years of age, bibliometrics is still experienced in integrating new and different approaches. As the example of citation analyses and attempts to theorize citations (Cozzens, 1989; Cronin, 1981; Gauch, 2021) reveal, scientometrics always was multidisciplinary, bringing different disciplinary aspects, sociological, psychological, information science, to the quantitative analysis of scholarly knowledge production. Yet, the example of citation analysis also reveals the conditions that makes such analyses more substantial, that is, if they strongly resonate in and are reflected by a wider community of science studies, studying practices, values and motives of scholarly activities (Blümel and Schniedermann, 2020). In this way, the presented contributions of the funding line show what is possible if links to other research strands are established more sustainably.
Blümel, C., & Gauch, S. (2021). History, Development and Conceptual Predecessors of Altmetrics. In R. Ball (Ed.), Handbook bibliometrics (pp. 191–199). Berlin/Boston: De Gruyter Saur.
Blümel, C., & Schniedermann, A. (2020). Studying review articles in scientometrics and beyond: A research agenda. Scientometrics, 1–18. https://doi.org/10.1007/s11192-020-03431-7
Costas, R., Zahedi, Z., & Wouters, P. (2014). Do “Altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019.
Cozzens, S. (1985). Comparing the sciences: Citation context analysis of papers from neuropharmacology and the sociology of science. Social Studies of Science, 15(1), 127–153.
Cozzens, S. (1989). What do citations count? The Rhetoric First Model, Scientometrics, 15, 437–447.
Cronin, B. (1981). The need for a theory of citation. Journal of Documentation, 37, 16–24.
de Rijcke, S., & Rushforth, A. (2015). To intervene or not to intervene; is that the question? On the role of scientometrics in research evaluation. Journal of the Association for Information Science and Technology, 66(9), 1954–1958.
De Solla Price, D. (1963). Little science, big science. Columbia University Press.
Edquist, R. (1997). Systems of innovation: Technologies, Institutions and Organizations. Routledge.
Falagas, M. E., & Alexiou, V. G. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae Et Therapiae Experimentalis, 56, 223. https://doi.org/10.1007/s00005-008-0024-5
Garfield, E. (1968). World Brain” or “Memex?” Mechanical and Intellectual Requirements for Universal Bibliographic Control. In E. Montgomery (Ed.), The Foundations of Access To Knowledge (pp. 169–196). New York: Syracuse University Press.
Gauch, S. (2021). The ironic becomings of reflexivity–The case of citation theory in bibliometrics. Historical Social Research. https://doi.org/10.12759/HSR.46.2021.2.155-177
Glänzel, W. (2008): Seven Myths in Bibliometrics. About facts and fiction in quantitative science studies. In: Kretschmer, H. & Havemann, F. (eds.): Proceedings of WIS 2008, Fourth International Conference on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting, (pp. 1–9).
Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting “altmetrics”: Viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of Informetrics and Scholarly Communication (pp. 372–405). de Gruyter Mouton.
Moed, H. F. (2005). Citation analysis in research evaluation. Springer.
Narin, F. (1976). Evaluative bibliometrics. The use of publications and citation analysis in the evaluation of scientific activity. Cherry Hill (USA): Computer Horizons.
van Leuuwen, T. (2004). Descriptive versus evaluative bibliometrics: Monitoring and assessing national R&D systems. In Henk F. Moed, Wolfgang Glänzel, & Ulrich Schmoch (Eds.), Handbook of quantitative Science and Technology Research (pp. 373–388). Rotterdam: Kluwer Academic.
Van Raan, A. (2004). Measuring Science. In Henk F. Moed, Wolfgang Glänzel, & Ulrich Schmoch (Eds.), Handbook of quantitative Science and Technology Research (pp. 19–50). Rotterdam: Kluwer Academic.
Wissenschaftsrat (2014). Institutionelle Perspektiven der empirischen Wissenschafts- und Hochschulforschung in Deutschland. Positionspapier. Köln: Wissenschaftsrat.
About this article
Cite this article
Blümel, C., Gauch, S. Introduction to special issue: quantitative studies of science in Germany. Scientometrics 126, 9641–9647 (2021). https://doi.org/10.1007/s11192-021-04203-7