Abstract
In this chapter we build upon Moed’s conceptual contributions on the importance of the policy context when using and interpreting scientometric indicators. We focus on the use of indicators in research evaluation regarding internationalisation policies. The globalisation of higher education presents important challenges to institutions worldwide, which are confronted with tensions derived from the need to respond both to their local necessities and demands while participating in global networks. In this context, indicators have served as measures for monitoring and enforcing internationalisation policies, in many cases interpreting them regardless of the policy context in which they are enforced. We will analyse three examples of indicators related to internationalisation. The first one is about international collaborations, under the assumption that a greater number of internationally co-authored publications will benefit a national science system as it will result in higher citation impact. The second one relates to the promotion of English language as the dominant language of science. The third case analyses how different policy contexts shape the selection and construction of indicators, sometimes in a partial way which does not properly reflect the phenomenon under study. The examples illustrate that the interpretation and policy implications of the ‘same’ S&T indicators differ depending on specific contexts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abramo, G., D’Angelo, C. A., & Costa, F. D. (2017). Do interdisciplinary research teams deliver higher gains to science? Scientometrics, 111(1), 317–336. https://doi.org/10.1007/s11192-017-2253-x.
Ackers, L. (2005). Moving people and knowledge: Scientific mobility in the European Union1. International Migration, 43(5), 99–131. https://doi.org/10.1111/j.1468-2435.2005.00343.x.
Adam, D. (2002). The counting house. Nature, 415, 726–729. https://doi.org/10.1038/415726a.
Adams, J. (2012). Collaborations: The rise of research networks. Nature, 490, 335–336. https://doi.org/10.1038/490335a.
Aguillo, I. F., Bar-Ilan, J., Levene, M., & Ortega, J. L. (2010). Comparing university rankings. Scientometrics, 85(1), 243–256.
Altbach, P. G. (2004). Globalisation and the university: Myths and realities in an unequal world. Tertiary Education and Management, 10(1), 3–25. https://doi.org/10.1023/B:TEAM.0000012239.55136.4b.
Aman, V. (2018). A new bibliometric approach to measure knowledge transfer of internationally mobile scientists. Scientometrics, 117(1), 227–247. https://doi.org/10.1007/s11192-018-2864-x.
Andújar, I., Cañibano, C., & Fernandez-Zubieta, A. (2015). International stays abroad, collaborations and the return of spanish researchers. Science Technology & Society, 20(3), 322–348. https://doi.org/10.1177/0971721815597138.
Arrieta, O. A. D., Pammolli, F., & Petersen, A. M. (2017). Quantifying the negative impact of brain drain on the integration of European science. Science Advances, 3(4), e1602232. https://doi.org/10.1126/sciadv.1602232.
Baldridge, D. C., Floyd, S. W., & Markóczy, L. (2004). Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strategic Management Journal, 25(11), 1063–1074. https://doi.org/10.1002/smj.406.
Bassioni, G., Adzaho, G., & Niyukuri, D. (2016). Brain drain: Entice Africa’s scientists to stay. Nature, 535(7611), 231–231. https://doi.org/10.1038/535231c.
Bastedo, M. N., & Bowman, N. A. (2010). U.S. News & world report college rankings: Modeling institutional effects on organizational reputation. American Journal of Education, 116(2), 163–183. https://doi.org/10.1086/649436
Bote, V. P. G., Olmeda-Gómez, C., & de Moya-Anegón, F. (2013). Quantifying the benefits of international scientific collaboration. Journal of the American Society for Information Science and Technology, 64(2), 392–404. https://doi.org/10.1002/asi.22754.
Buela-Casal, G., & Zych, I. (2012). How to measure the internationality of scientific publications. Psicothema, 24(3), 435–441.
Cañibano, C., Fox, M. F., & Otamendi, F. J. (2016). Gender and patterns of temporary mobility among researchers. Science and Public Policy, 43(3), 320–331. https://doi.org/10.1093/scipol/scv042.
Canibano, C., Otamendi, J., & Andujar, I. (2008). Measuring and assessing researcher mobility from CV analysis: The case of the Ramon y Cajal programme in Spain. Research Evaluation, 17(1), 17–31. https://doi.org/10.3152/095820208X292797.
Chavarro, D., Tang, P., & Ràfols, I. (2017). Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling. Research Policy, 46(9), 1666–1680. https://doi.org/10.1016/j.respol.2017.08.002.
Chinchilla-Rodríguez, Z., Bu, Y., Robinson-García, N., Costas, R., & Sugimoto, C. R. (2018). Travel bans and scientific mobility: Utility of asymmetry and affinity indexes to inform science policy. Scientometrics, 116(1), 569–590. https://doi.org/10.1007/s11192-018-2738-2.
Confraria, H., Mira Godinho, M., & Wang, L. (2017). Determinants of citation impact: A comparative analysis of the Global South versus the Global North. Research Policy, 46(1), 265–279. https://doi.org/10.1016/j.respol.2016.11.004.
Costas, R., van Leeuwen, T. N., & Bordons, M. (2010). A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact. Journal of the American Society for Information Science and Technology, 61(8), 1564–1581. https://doi.org/10.1002/asi.21348.
Delanghe, H., Sloan, B., & Muldur, U. (2010). European research policy and bibliometric indicators, 1990–2005. Scientometrics, 87(2), 389–398. https://doi.org/10.1007/s11192-010-0308-3.
Díaz-Faes, A. A., Bowman, T. D., & Costas, R. (2019). Towards a second generation of ‘social media metrics’: Characterizing Twitter communities of attention around science. PLoS ONE, 14(5), e0216408. https://doi.org/10.1371/journal.pone.0216408.
Fang, Z., Lamers, W., Costas, R. (2019). Studying the Scientific Mobility and International Collaboration Funded by the China Scholarship Council. Presented at the ISSI/STI. (2019). Conference. Rome: Italy.
Feld, A., & Kreimer, P. (2019). Scientific co-operation and centre-periphery relations: Attitudes and interests of European and Latin American scientists. Tapuya: Latin American Science, Technology and Society, 0(0), 1–27. https://doi.org/10.1080/25729861.2019.1636620.
Franzoni, C., Scellato, G., & Stephan, P. (2018). Context factors and the performance of mobile individuals in research teams. Journal of Management Studies, 55(1), 27–59. https://doi.org/10.1111/joms.12279.
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: SAGE.
González-Alcaide, G., Valderrama-Zurián, J. C., & Aleixandre-Benavent, R. (2012). The impact factor in non-English-speaking countries. Scientometrics, 92(2), 297–311. https://doi.org/10.1007/s11192-012-0692-y.
Good, B., Vermeulen, N., Tiefenthaler, B., & Arnold, E. (2015). Counting quality? The Czech performance-based research funding system. Research Evaluation, 24(2), 91–105. https://doi.org/10.1093/reseval/rvu035.
Halevi, G., Moed, H. F., & Bar-Ilan, J. (2016a). Does research mobility have an effect on productivity and impact? International Higher Education, 86(86), 5–6. https://doi.org/10.6017/ihe.2016.86.9360.
Halevi, G., Moed, H. F., & Bar-Ilan, J. (2016b). Researchers’ mobility, productivity and impact: Case of top producing authors in seven disciplines. Publishing Research Quarterly, 32(1), 22–37. https://doi.org/10.1007/s12109-015-9437-0.
Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting ‘altmetrics’: Viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of Informetrics and Scholarly Communication (pp. 372–406). Retrieved from http://arxiv.org/abs/1502.05701.
Hazelkorn, E. (2011). Rankings and the reshaping of higher education: The battle for world-class excellence. Palgrave Macmillan. Retrieved from http://eric.ed.gov/?id=ED528844
Henshall, A. C. (2018). English language policies in scientific journals: Signs of change in the field of economics. Journal of English for Academic Purposes, 36, 26–36. https://doi.org/10.1016/j.jeap.2018.08.001.
Hicks, D. (2005). The four literatures of social science. In Handbook of Quantitative Science and Technology Research (pp. 473–496). Retrieved from http://link.springer.com/chapter/10.1007/1-4020-2755-9_22
Hirsch, J. E. (2019). hα: An index to quantify an individual’s scientific leadership. Scientometrics, 118(2), 673–686. https://doi.org/10.1007/s11192-018-2994-1.
Jiménez-Contreras, E., de Moya Anegón, F., & López-Cózar, E. D. (2003). The evolution of research activity in Spain: The impact of the National Commission for the Evaluation of Research Activity (CNEAI). Research Policy, 32(1), 123–142.
Jiménez-Contreras, E., López-Cózar, E. D., Ruiz-Pérez, R., & Fernández, V. M. (2002). Impact-factor rewards affect Spanish research. Nature, 417(6892), 898–898. https://doi.org/10.1038/417898b.
Jonkers, K., & Tijssen, R. (2008). Chinese researchers returning home: Impacts of international mobility on research collaboration and scientific productivity. Scientometrics, 77(2), 309–333. https://doi.org/10.1007/s11192-007-1971-x.
Khan, S. A. (2019). Promoting science in India’s minority languages. Nature, 573, 34–34. https://doi.org/10.1038/d41586-019-02626-0.
Kim, D., Twombly, S., & Wolf-Wendel, L. (2012). International faculty in American Universities: Experiences of academic life, productivity, and career mobility. New Directions for Institutional Research, 155, 27–46. https://doi.org/10.1002/ir.20020.
Larivière, V. (2018). Le français, langue seconde? De l’évolution des lieux et langues de publication des chercheurs au Québec, en France et en Allemagne. Recherches sociographiques, 59(3), 339–363. https://doi.org/10.7202/1058718ar.
Larivière, V., & Gingras, Y. (2010). On the relationship between interdisciplinarity and scientific impact. Journal of the American Society for Information Science and Technology, 61(1), 126–131. https://doi.org/10.1002/asi.21226.
Laudel, G. (2003). Studying the brain drain: Can bibliometric methods help? Scientometrics, 57(2), 215–237. https://doi.org/10.1023/A:1024137718393.
Leeuwen, T. N. V., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Raan, A. F. J. V. (2001). Language biases in the coverage of the science citation index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346. https://doi.org/10.1023/A:1010549719484.
Leta, J., & Chaimovich, H. (2002). Recognition and international collaboration: The Brazilian case. Scientometrics, 53(3), 325–335. https://doi.org/10.1023/A:1014868928349.
Levin, S. G., & Stephan, P. E. (1999). Are the foreign born a source of strength for U.S. science? Science, 285(5431), 1213–1214. https://doi.org/10.1126/science.285.5431.1213
Leydesdorff, L., & Rafols, I. (2011). Indicators of the interdisciplinarity of journals: Diversity, centrality, and citations. Journal of Informetrics, 5(1), 87–100. https://doi.org/10.1016/j.joi.2010.09.002.
Leydesdorff, L., & Wagner, C. S. (2008). International collaboration in science and the formation of a core group. Journal of Informetrics, 2(4), 317–325. https://doi.org/10.1016/j.joi.2008.07.003.
Libaers, D. (2014). Foreign-born academic scientists and their interactions with industry: Implications for university technology commercialization and corporate innovation management. Journal of Product Innovation Management, 31(2), 346–360. https://doi.org/10.1111/jpim.12099.
Lin, Z., Pearce, R., & Wang, W. (2009). Imported talents: Demographic characteristics, achievement and job satisfaction of foreign born full time faculty in four-year American colleges. Higher Education, 57(6), 703–721. https://doi.org/10.1007/s10734-008-9171-z.
Liu, F., Hu, G., Tang, L., & Liu, W. (2018). The penalty of containing more non-English articles. Scientometrics, 114(1), 359–366. https://doi.org/10.1007/s11192-017-2577-6.
Mamiseishvili, K. (2011). Teaching workload and satisfaction of foreign-born and U.S.-born faculty at four-year postsecondary institutions in the United States. Journal of Diversity in Higher Education, 4(3), 163–174. https://doi.org/10.1037/a0022354
Meyer, J.-B. (2001). Network approach versus brain drain: Lessons from the Diaspora. International Migration, 39(5), 91–110. https://doi.org/10.1111/1468-2435.00173.
Moed, H. F., Bruin, R. E. D., & Leeuwen, T. N. V. (1995). New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422. https://doi.org/10.1007/BF02017338.
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131–149. https://doi.org/10.1016/0048-7333(85)90012-5.
Moed, H. F., & van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381(6579), 186.
Moed, H. F., & Van Leeuwen, Th N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. Journal of the American Society for Information Science, 46(6), 461–467. https://doi.org/10.1002/(SICI)1097-4571(199507)46:6%3c461:AID-ASI5%3e3.0.CO;2-G.
Moed, H. F. (2005). Citation analysis in research evaluation (Vol. 9). Retrieved from http://books.google.es/books?hl=en&lr=&id=D9SaJ6awy4gC&oi=fnd&pg=PR9&dq=citation+analysis+in+research+evaluation&ots=FFpZIv-Qg0&sig=w_eOO2xmcRUTReMwdvlVJPo3cno
Moed, H. F. (2017a). A critical comparative analysis of five world university rankings. Scientometrics, 110(2), 967–990. https://doi.org/10.1007/s11192-016-2212-y.
Moed, H. F. (2017b). Applied evaluative informetrics. Cham: Springer.
Moed, H. F., Aisati, M., & Plume, A. (2013). Studying scientific migration in Scopus. Scientometrics, 94(3), 929–942. https://doi.org/10.1007/s11192-012-0783-9.
Moed, Henk F., & Halevi, G. (2014). A bibliometric approach to tracking international scientific migration. Scientometrics, 1–15. https://doi.org/10.1007/s11192-014-1307-6
Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science and Technology, 66(10), 1988–2002. https://doi.org/10.1002/asi.23314.
Nederhof, A. J. (2006). Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review. Scientometrics, 66(1), 81–100. https://doi.org/10.1007/s11192-006-0007-2.
Nerad, M. (2010). Globalization and the internationalization of graduate education: A macro and micro view. Canadian Journal of Higher Education, 40(1), 1–12.
Neylon, C. (2019). Research excellence is a neo-colonial agenda (and what might be done about it). In E. Kraemer-Mbula, R. Tijssen, M. L. Wallace, & R. McLean (Eds.), Transforming research excellence. Retrieved from https://hcommons.org/deposits/item/hc:26133/.
Persson, O., Glänzel, W., & Danell, R. (2004). Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics, 60(3), 421–432. https://doi.org/10.1023/B:SCIE.0000034384.35498.7d.
Piñeiro, C. L., & Hicks, D. (2015). Reception of Spanish sociology by domestic and foreign audiences differs and has consequences for evaluation. Research Evaluation, 24(1), 78–89.
Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish. Aslib Journal of Information Management. https://doi.org/10.1108/AJIM-01-2017-0014.
Quan, W., Mongeon, P., Sainte-Marie, M., Zhao, R., & Larivière, V. (2019). On the development of China’s leadership in international collaborations. Scientometrics, 120(2), 707–721. https://doi.org/10.1007/s11192-019-03111-1.
Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy, 41(7), 1262–1282. https://doi.org/10.1016/j.respol.2012.03.015.
Ràfols, I., Molas-Gallart, J., Chavarro, D. A., & Robinson-Garcia, N. (2016). On the dominance of quantitative evaluation in ‘peripheral’ countries: Auditing research with technologies of distance (SSRN Scholarly Paper No. ID 2818335). Retrieved from Social Science Research Network website, https://papers.ssrn.com/abstract=2818335
Rey-Rocha, J., & Martín-Sempere, M. J. (2012). Generating favourable contexts for translational research through the incorporation of basic researchers into hospitals: The FIS/Miguel Servet research contract programme. Science and Public Policy, 39(6), 787–801.
Robinson, B. J. (2016). Flying in the face of illusion. A comparative study of the variables that interact in English-language scientific journals publishing translations. In L. Ilynska & M. Platanova (Eds.), Meaning in translation. Illusion of precision (pp. 335–351). Newcastle-upon-Tyne: Cambridge Scholar Publishing.
Robinson-Garcia, N., van Holm, E., Melkers, J., & Welch, E. W. (2018). From theory to practice: Operationalization of the GTEC framework. In STI 2018 Conference Proceedings (pp. 1542–1545). Retrieved from https://openaccess.leidenuniv.nl/handle/1887/65243
Robinson-Garcia, N., & Jiménez-Contreras, E. (2017). Analyzing the disciplinary focus of universities: Can rankings be a one-size-fits-all? http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-5225-0819-9.Ch009, pp. 161–185. https://doi.org/10.4018/978-1-5225-0819-9.ch009
Robinson-Garcia, Nicolas, van Leeuwen, T. N., & Rafols, I. (2018b). Using almetrics for contextualised mapping of societal impact: From hits to networks. Science and Public Policy, 45(6), 815–826. https://doi.org/10.1093/scipol/scy024.
Robinson-Garcia, N., Sugimoto, C. R., Murray, D., Yegros-Yegros, A., Larivière, V., & Costas, R. (2019a). The many faces of mobility: Using bibliometric data to measure the movement of scientists. Journal of Informetrics, 13(1), 50–63. https://doi.org/10.1016/j.joi.2018.11.002.
Robinson-Garcia, N., Torres-Salinas, D., Herrera-Viedma, E., & Docampo, D. (2019b). Mining university rankings: Publication output and citation impact as their basis. Research Evaluation, 28(3), 232–240. https://doi.org/10.1093/reseval/rvz014.
Robinson-Garcia, N., Woolley, R., & Costas, R. (2019). Making sense of global collaboration dynamics: Developing a methodological framework to study (dis)similarities between country disciplinary profiles and choice of collaboration partners. ArXiv:1909.04450[Cs]. https://doi.org/10.5281/zenodo.3376411
Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics, 107(2), 357–368. https://doi.org/10.1007/s11192-016-1845-1.
Sivertsen, G. (2018). Balanced multilingualism in science. BiD: Textos Universitaris de Biblioteconomia i Documentació, (40). https://doi.org/10.1344/BiD2018.40.25
Stephan, P. E., & Levin, S. G. (2001). Exceptional contributions to US science by the foreign-born and foreign-educated. Population Research and Policy Review, 20(1–2), 59–79. https://doi.org/10.1023/A:1010682017950.
Sugimoto, C. R., Robinson-Garcia, N., & Costas, R. (2016). Towards a global scientific brain: Indicators of researcher mobility using co-affiliation data. ArXiv:1609.06499[Cs]. Retrieved from http://arxiv.org/abs/1609.06499
Sugimoto, C. R., Robinson-Garcia, N., Murray, D. S., Yegros-Yegros, A., Costas, R., & Larivière, V. (2017). Scientists have most impact when they’re free to move. Nature, 550(7674), 29. https://doi.org/10.1038/550029a.
Tijssen, R. J. W., & van Wijk, E. (1999). In search of the European Paradox: An international comparison of Europe’s scientific performance and knowledge flows in information and communication technologies research. Research Policy, 28(5), 519–543. https://doi.org/10.1016/S0048-7333(99)00011-6.
Torres-Salinas, D., & Jiménez-Contreras, E. (2015). El efecto Cajal: Análisis bibliométrico del Programa Ramón y Cajal en la Universidad de Granada. Revista Española de Documentación Científica, 38(1), e075.
van Holm, E. J., Wu, Y., & Welch, E. W. (2019). Comparing the collaboration networks and productivity of China-born and US-born academic scientists. Science and Public Policy, 46(2), 310–320. https://doi.org/10.1093/scipol/scy060.
Van Raan, A. F. (1997). Science as an international enterprise. Science and Public Policy, 24(5), 290–300. https://doi.org/10.1093/spp/24.5.290.
Vessuri, H., Guédon, J.-C., & Cetto, A. M. (2014). Excellence or quality? Impact of the current competition regime on science and scientific publishing in Latin America and its implications for development. Current Sociology, 62(5), 647–665. https://doi.org/10.1177/0011392113512839.
Wagner, C. S. (2019). Global science for global challenges. In D. Simon, S. Kuhlmann, J. Stamm, & W. Canzler (Eds.), Handbook on science and public policy (pp. 92–103). Edward Elgar Publishing.
Wagner, C. S., & Jonkers, K. (2017). Open countries have strong science. Nature, 550(7674), 32. https://doi.org/10.1038/550032a.
Wagner, C. S., & Leydesdorff, L. (2005). Network structure, self-organization, and the growth of international collaboration in science. Research Policy, 34(10), 1608–1618. https://doi.org/10.1016/j.respol.2005.08.002.
Wagner, C. S., Park, H. W., & Leydesdorff, L. (2015). The continuing growth of global cooperation networks in research: A conundrum for national governments. PLoS ONE, 10(7), e0131816. https://doi.org/10.1371/journal.pone.0131816.
Waltman, L. (2019a). Quantitative literacy for responsible research policy. Presented at the Inaugural lecture by Ludo Waltman as newly appointed Professor of Quantitative Science Studies, Leiden (The Netherlands). Retrieved from https://www.cwts.nl/news?article=n-r2x264&title=inaugural-lectures-by-sarah-de-rijcke-and-ludo-waltman.
Waltman, L. (2019b). Put metrics in context. Retrieved 6 August 2019, from Research Europe website: http://www.researchresearch.com
Welch, E. W., van Holm, E., Jung, H., Melkers, J., Robinson-Garcia, N., Mamiseishvili, K., & Pinheiro, D. (2018). The Global Scientific Workforce (GTEC) Framework. In STI 2018 Conference Proceedings (pp. 868–871). Retrieved from http://hdl.handle.net/1887/65210.
Woolley, R., Robinson-Garcia, N., & Costas, R. (2017). Global research collaboration: Networks and partners in South East Asia. ArXiv:1712.06513[Cs]. Retrieved from http://arxiv.org/abs/1712.06513
Zippel, K. (2017). Women in global science: Advancing academic careers through international collaboration. Stanford University Press.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Robinson-Garcia, N., Ràfols, I. (2020). The Differing Meanings of Indicators Under Different Policy Contexts. The Case of Internationalisation. In: Daraio, C., Glänzel, W. (eds) Evaluative Informetrics: The Art of Metrics-Based Research Assessment . Springer, Cham. https://doi.org/10.1007/978-3-030-47665-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-47665-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-47664-9
Online ISBN: 978-3-030-47665-6
eBook Packages: Social SciencesSocial Sciences (R0)