Abstract
What can explain the variation within the disciplinary communities in expressing the support of metrics in the evaluation of research productivity? The aim of this study is to demonstrate that the attitude to the metrics varies between researchers depending on their academic cultural preferences. We conducted a survey of 1850 Russian sociologists with the range of questions regarding the main divisions of sociology and perceived legitimacy of metrics for evaluating academic work. We found that although the majority of sociologists do not support giving more value to the publications that appeared in international databases, 30% of respondent expressed the positive attitude to this policy. Researchers with the orientation to a global science that was measured by their responses and publication behavior pattern tend to support using metrics in research evaluation, while their counterparts committed to a local academic path are more likely to show a negative attitude.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The full version of questionnaire is available in the Supplementary Information.
References
Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Gender differences in research productivity: A bibliometric analysis of the Italian academic system. Scientometrics, 79, 517–539. https://doi.org/10.1007/s11192-007-2046-8
Akbaritabar, A., Casnici, N., & Squazzoni, F. (2018). The conundrum of research productivity: A study on sociologists in Italy. Scientometrics, 114, 859–882. https://doi.org/10.1007/s11192-017-2606-5
Alvesson, M., & Spicer, A. (2016). (Un)Conditional surrender? Why do professionals willingly comply with managerialism. Journal of Organizational Change Management, 29, 29–45. https://doi.org/10.1108/JOCM-11-2015-0221
Andersen, L. B., & Pallesen, T. (2008). Not just for the money? How financial incentives affect the number of publications at Danish research institutions. International Public Management Journal, 11, 28–47. https://doi.org/10.1080/10967490801887889
Auranen, O., & Nieminen, M. (2010). University research funding and publication performance—An international comparison. Research Policy, 39, 822–834. https://doi.org/10.1016/j.respol.2010.03.003
Baccini, A., De Nicolao, G., & Petrovich, E. (2019). Citation gaming induced by bibliometric evaluation: A country-level comparative analysis. PLoS ONE, 14(9), e0221212. https://doi.org/10.1371/journal.pone.0199031
Berman, E. P., & Hirschman, D. (2018). The sociology of quantification: Where are we now? Contemporary Sociology, 47, 257–266. https://doi.org/10.1177/0094306118767649
Biagioli, M., Kenney, M., Martin, B. R., & Walsh, J. P. (2019). Academic misconduct, misrepresentation and gaming: A reassessment. Research Policy, 48(2), 401–413. https://doi.org/10.1016/j.respol.2018.10.025
Biagioli, M., & Lippman, A. (Eds.) (2020). Gaming the metrics: Misconduct and manipulation in academic research. MIT Press.
Broz, L., & Stöckelová, T. (2018). The culture of orphaned texts: Academic books in a performance-based evaluation system. Aslib Journal of Information Management, 70, 623–642. https://doi.org/10.1108/AJIM-03-2018-0063
Buela-Casal, G., & Zych, I. (2012). What do the scientists think about the impact factor? Scientometrics, 92(2), 281–292. https://doi.org/10.1007/s11192-012-0676-y
Butler, L. (2003). Modifying publication practices in response to funding formulas. Research Evaluation, 12, 39–46. https://doi.org/10.3152/147154403781776780
Chun, H., & Sauder, M. (2022). The logic of quantification: Institutionalizing numerical thinking. Theory and Society, 51, 335–370. https://doi.org/10.1007/s11186-021-09453-1
Clark, B. R. (1986). The higher education system: Academic organization in cross-national perspective. University of California Press.
Costas, R., van Leeuwen, T. N., & Bordons, M. (2010). A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact. Journal of the American Society for Information Science and Technology, 61, 1564–1581. https://doi.org/10.1002/asi.21348
Cruz-Castro, L., & Sanz-Menendez, L. (2021). What should be rewarded? Gender and evaluation criteria for tenure and promotion. Journal of Informetrics, 15(3), 101196.
de Rijcke, S., & Stöckelová, T. (2020). Predatory publishing and the imperative of international productivity: Feeding off and feeding up the dominant. In M. Biagioli & A. Lippman (Eds.), Gaming the metrics (pp. 101–111). MIT Press.
de Rijcke, S., Wouters, P., Rushforth, A., Franssen, T., & Hammarfelt, B. (2015). Evaluation practices and effects of indicator use—A literature review. Research Evaluation. https://doi.org/10.1093/reseval/rvv038
Espeland, W. N., Sauder, M., & Espeland, W. (2016). Engines of anxiety: Academic rankings, reputation, and accountability. Russell Sage Foundation.
Etzioni, A. (1975). Comparative analysis of complex organizations, Revised Edition. Simon and Schuster.
Ferlie, E., Musselin, C., & Andresani, G. (2008). The steering of higher education systems: A public management perspective. Higher Education, 56, 325–348. https://doi.org/10.1007/s10734-008-9125-5
Foster, J. G., Rzhetsky, A., & Evans, J. A. (2015). Tradition and innovation in scientists’ research strategies. American Sociological Review, 80(5), 875–908. https://doi.org/10.1177/0003122415601618
Gantman, E. R., & Fernández Rodríguez, C. J. (2016). Literature segmentation in management and organization studies: The case of Spanish-speaking countries (2000–10). Research Evaluation, 25, 461–471. https://doi.org/10.1093/reseval/rvv031
Glänzel, W., & Schoepflin, U. (1999). A bibliometric study of reference literature in the sciences and social sciences. Information Processing and Management, 35, 31–44. https://doi.org/10.1016/S0306-4573(98)00028-4
Gläser, J. (2004). Why are the most influential books in Australian sociology not necessarily the most highly cited ones? Journal of Sociology, 40, 261–282. https://doi.org/10.1177/1440783304046370
Grančay, M., Vveinhardt, J., & Šumilo, Ē. (2017). Publish or perish: How Central and Eastern European economists have dealt with the ever-increasing academic publishing requirements 2000–2015. Scientometrics, 111, 1813–1837. https://doi.org/10.1007/s11192-017-2332-z
Guba, K., & Tsivinskaya, A. (2023). Expert judgments versus publication-based metrics: Do the two methods produce identical results in measuring academic reputation? Journal of Documentation, 79(1), 127–143. https://doi.org/10.1108/JD-02-2022-0039
Hallonsten, O. (2022). Introduction to special section: Causes and consequences of the current evaluation regime in (academic) science. Social Science Information, 61(4), 407–413. https://doi.org/10.1177/05390184231151610
Hammarfelt, B., & Haddow, G. (2018). Conflicting measures and values: How humanities scholars in Australia and Sweden use and react to bibliometric indicators. Journal of the Association for Information Science and Technology, 69, 924–935. https://doi.org/10.1002/asi.24043
Hammarfelt, B., Nelhans, G., Eklund, P., & Åström, F. (2016). The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish universities. Research Evaluation, 25, 292–305. https://doi.org/10.1093/reseval/rvv040
Hammarfelt, B., & Rushforth, A. D. (2017). Indicators as judgment devices: An empirical study of citizen bibliometrics in research evaluation. Research Evaluation, 26, 169–180. https://doi.org/10.1093/reseval/rvx018
Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41, 251–261. https://doi.org/10.1016/j.respol.2011.09.007
Hokka, J. (2019). What counts as ‘good sociology’? Conflicting discourses on legitimate sociology in Finland and Sweden. Acta Sociologica, 62, 357–371. https://doi.org/10.1177/0001699318813422
Honig, B., & Bedi, A. (2012). The fox in the hen house: A critical examination of plagiarism among members of the Academy of Management. Academy of Management Learning and Education, 11, 101–123. https://doi.org/10.5465/amle.2010.0084
Horowitz, M., Haynor, A., & Kickham, K. (2018). Sociology’s sacred victims and the politics of knowledge: Moral foundations theory and disciplinary controversies. The American Sociologist, 49, 459–495. https://doi.org/10.1007/s12108-018-9381-5
Kalfa, S., Wilkinson, A., & Gollan, P. J. (2018). The academic game: Compliance and resistance in universities. Work, Employment and Society, 32, 274–291. https://doi.org/10.1177/0950017017695043
Kanchan, T., & Krishan, K. (2019). The Leiden Manifesto and research assessment. Science and Engineering Ethics, 25(2), 643–644. https://doi.org/10.1007/s11948-017-0012-2
Kassian, A., & Melikhova, L. (2019). Russian Science Citation Index on the WoS platform: A critical assessment. Journal of Documentation, 75(5), 1162–1168. https://doi.org/10.1108/JD-02-2019-0033
Koch, T., & Vanderstraeten, R. (2018). Internationalizing a national scientific community? Changes in publication and citation practices in Chile, 1976–2015. Current Sociology, 67, 723–741. https://doi.org/10.1177/0011392118807514
Krzeski, J., Szadkowski, K., & Kulczycki, E. (2022). Creating evaluative homogeneity: Experience of constructing a national journal ranking. Research Evaluation, 31, 410–422. https://doi.org/10.1093/reseval/rvac011
Kuzʹminov, J. I., & Judkevič, M. M. (2022). Higher education in Russia. Johns Hopkins University Press.
Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Harvard University Press.
Langfeldt, L., Reymert, I., & Aksnes, D. W. (2021). The role of metrics in peer assessments. Research Evaluation, 30, 112–126. https://doi.org/10.1093/reseval/rvaa032
Leathwood, C., & Read, B. (2013). Research policy and academic performativity: Compliance, contestation and complicity. Studies in Higher Education, 38, 1162–1174. https://doi.org/10.1080/03075079.2013.833025
Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators—A state-of-the-art report. Scientometrics, 109, 2129–2150. https://doi.org/10.1007/s11192-016-2150-8
Long, J. S. (1990). The origins of sex differences in science. Social Forces, 68, 1297–1316. https://doi.org/10.1093/sf/68.4.1297
Lovakov, A., Panova, A., & Yudkevich, M. (2022). Global visibility of nationally published research output: The case of the post-Soviet region. Scientometrics, 127, 2643–2659. https://doi.org/10.1007/s11192-022-04326-5
Marina, T., & Sterligov, I. (2021). Prevalence of potentially predatory publishing in Scopus on the country level. Scientometrics, 126, 5019–5077. https://doi.org/10.1007/s11192-021-03899-x
Moskaleva, O., Pislyakov, V., Sterligov, I., & Akoev, M. (2018). Russian Index of Science Citation: Overview and review. Scientometrics, 116(1), 449–462. https://doi.org/10.1007/s11192-018-2758-y
Najman, J. M., & Hewitt, B. (2003). The validity of publication and citation counts for Sociology and other selected disciplines. Journal of Sociology, 39, 62–80. https://doi.org/10.1177/144078330303900106
Önder, Ç., & Erdil, S. E. (2017). Opportunities and opportunism: Publication outlet selection under pressure to increase research productivity. Research Evaluation, 26, 66–77. https://doi.org/10.1093/reseval/rvx006
Pajić, D. (2015). Globalization of the social sciences in Eastern Europe: Genuine breakthrough or a slippery slope of the research evaluation practice? Scientometrics, 102, 2131–2150. https://doi.org/10.1007/s11192-014-1510-5
Pardo-Guerra, J. P. (2022). The quantified scholar: How research evaluations transformed the British social sciences. Columbia University Press.
Park, M., Leahey, E., & Funk, R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613, 138–144. https://doi.org/10.1038/s41586-022-05543-x
Petrovich, E. (2022). Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers. Scientometrics, 127(5), 2195–2233. https://doi.org/10.1007/s11192-022-04341-6
Pölönen, J., Guns, R., Kulczycki, E., Sivertsen, G., & Engels, T. C. (2021). National lists of scholarly publication channels: An overview and recommendations for their construction and maintenance. Journal of Data and Information Science, 6(1), 50–86. https://doi.org/10.2478/jdis2021-0004
Prasad, M. (2021). Pragmatism as problem solving. Socius: Sociological Research for a Dynamic World, 7, 237802312199399. https://doi.org/10.1177/2378023121993991
Reymert, I., Jungblut, J., & Borlaug, S. B. (2021). Are evaluative cultures national or global? A cross-national study on evaluative cultures in academic recruitment processes in Europe. Higher Education, 82, 823–843. https://doi.org/10.1007/s10734-020-00659-3
Scarpa, F., Bianco, V., & Tagliafico, L. A. (2018). The impact of the National Assessment Exercises on self-citation rate and publication venue: An empirical investigation on the engineering academic sector in Italy. Scientometrics, 117(2), 997–1022. https://doi.org/10.1007/s11192-018-2913-5
Seeber, M., Cattaneo, M., Meoli, M., & Malighetti, P. (2019). Self-citations as strategic response to the use of metrics for career decisions. Research Policy, 48(2), 478–491. https://doi.org/10.1016/j.respol.2017.12.004
Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628–638. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9%3c628:AID-ASI5%3e3.0.CO;2-0
Siler, K., & Larivière, V. (2022). Who games metrics and rankings? Institutional niches and journal impact factor inflation. Research Policy, 51, 104608. https://doi.org/10.1016/j.respol.2022.104608
Söderlind, J., & Geschwind, L. (2020). Disciplinary differences in academics’ perceptions of performance measurement at Nordic universities. Higher Education Governance and Policy, 1(1), 18–31.
Sokolov, M. (2019). The sources of academic localism and globalism in Russian sociology: The choice of professional ideologies and occupational niches among social scientists. Current Sociology, 67, 818–837. https://doi.org/10.1177/0011392118811392
Sokolov, M. (2021). Can Russian research policy be called neoliberal? A study in the comparative sociology of quantification. Europe-Asia Studies, 73, 989–1009. https://doi.org/10.1080/09668136.2021.1902945
Tahamtan, I., & Bornmann, L. (2019). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics, 121, 1635–1684. https://doi.org/10.1007/s11192-019-03243-4
Townley, B. (1997). The institutional logic of performance appraisal. Organization Studies, 18, 261–285. https://doi.org/10.1177/017084069701800204
Turner, J. H. (2019). The more American sociology seeks to become a politically-relevant discipline, the more irrelevant it becomes to solving societal Problems. The American Sociologist, 50, 456–487. https://doi.org/10.1007/s12108-019-09420-5
Waltman, L. (2018). Responsible metrics: One size doesn’t fit all. In STI 2018 conference proceedings, 2018 (pp. 526–531).
Wieczorek, O., Münch, R., & Schubert, D. (2022). All power to the reviewers: British sociology under two-level supervision of the Research Excellence Framework. Social Science Information, 61(4), 481–528. https://doi.org/10.1177/05390184231158210
Acknowledgements
This study was funded by the Russian Science Foundation, No. 21-18-00519.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author has no competing interests or conflicts of interests to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Guba, K. Why do sociologists on academic periphery willingly support bibliometric indicators?. Scientometrics 129, 497–518 (2024). https://doi.org/10.1007/s11192-023-04890-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04890-4