Abstract
Bibliometric and scientometric analyses are widely in university headquarters, across multiple disciplines, in companies, and governments. Therefore, we need further research and expertise on how this analysis can be used in practice. In this study, we focus on the role of bibliometric analysis in evidence-based policymaking (EBPM). We divide the type of analysis into descriptive, predictive, and explorative analyses, and their different roles in EBPM processes. To discuss the role of scientometrics in EBPM, we illustrate a case of hydrogen energy technologies. We derive four propositions based on arguments on evidence and prerequisites for the analysis, that are necessary for: (1) strict distinction between policy evidence and policy reason, (2) application of relevant type of analysis to each unit process of policymaking, (3) multi-layered expertise including data and algorithms, domain knowledge, and understanding of policy context and social issues, and (4) a knowledge system to archive data, algorithms, and results. This paper contributes broadly to transdisciplinary bibliometric research, and specifically to scientometric research and science-based policymaking.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arimoto, T., & Sato, Y. (2012). Science and society. Rebuilding public trust in science for policy-making. Science, 337(6099), 1176–1177. https://doi.org/10.1126/science.1224004
Asatani, K., Mori, J., Ochi, M., & Sakata, I. (2018). Detecting trends in academic research from a citation network using network representation learning. PLoS ONE, 13(5), e0197260. https://doi.org/10.1371/journal.pone.0197260
Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777. https://doi.org/10.1002/asi.23605
Bloomrosen, M., & Detmer, D. E. (2010). Informatics, evidence-based care, and research; implications for national policy: A report of an American medical informatics association health policy conference. Journal of the American Medical Informatics Association: JAMIA, 17(2), 115–123. https://doi.org/10.1136/jamia.2009.001370
Börner, K., Chen, C. M., & Boyack, K. W. (2003). Visualizing knowledge domains. Annual Review of Information Science and Technology, 37(1), 179–255. https://doi.org/10.1002/aris.1440370106
Bowen, S., & Zwi, A. B. (2005). Pathways to ‘evidence-informed’ policy and practice: A framework for action. PLOS Medicine, 2(7), e166. https://doi.org/10.1371/journal.pmed.0020166
Boyack, K. W., & Klavans, R. (2010). Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately? Journal of the American Society for Information Science and Technology, 61(12), 2389–2404. https://doi.org/10.1002/asi.21419
Boyack, K. W., Klavans, R., & Börner, K. (2005). Mapping the backbone of science. Scientometrics, 64(3), 351–374. https://doi.org/10.1007/s11192-005-0255-6
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., Jäger, J., & Mitchell, R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences of the United States of America, 100(14), 8086–8091. https://doi.org/10.1073/pnas.1231332100
Chen, C. (2013). Mapping scientific frontiers: The quest for knowledge visualization (2nd ed.). Springer.
Clauset, A., Larremore, D. B., & Sinatra, R. (2017). Data-driven predictions in the science of science. Science, 355(6324), 477–480. https://doi.org/10.1126/science.aal4217
Coombs, S. K., & Peters, I. (2017). The Leiden manifesto under review: What libraries can learn from it. Digital Library Perspectives, 33(4), 324–338. https://doi.org/10.1108/DLP-01-2017-0004
Dickinson, H. D. (1998). Evidence-based decision-making: An argumentative approach. International Journal of Medical Informatics, 51(2–3), 71–81. https://doi.org/10.1016/s1386-5056(98)00105-1
Dobrow, M. J., Goel, V., & Upshur, R. E. G. (2004). Evidence-based health policy: Context and utilisation. Social Science and Medicine, 58(1), 207–217. https://doi.org/10.1016/s0277-9536(03)00166-7
Elwyn, G., Frosch, D. L., & Kobrin, S. (2016). Implementing shared decision-making: Consider all the consequences. Implementation Science: IS, 11, 114. https://doi.org/10.1186/s13012-016-0480-9
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Hikima, K., Tsujimoto, M., Takeuchi, M., & Kajikawa, Y. (2020). Transition analysis of budgetary allocation for projects on hydrogen-related technologies in Japan. Sustainability, 12(20), 8546. https://doi.org/10.3390/su12208546
Howlett, M. (2009). Policy analytical capacity and evidence-based policy-making: Lessons from Canada. Canadian Public Administration, 52(2), 153–175. https://doi.org/10.1111/j.1754-7121.2009.00070_1.x
Hsieh, C.-H. (2013). Patent value assessment and commercialization strategy. Technological Forecasting and Social Change, 80(2), 307–319. https://doi.org/10.1016/j.techfore.2012.09.014
Islam, N., & Kajikawa, Y. (2015). Actors’ engagement in sustainable hydrogen energy innovation: A comparative analysis. In Proceedings of Portland international conference on management engineering and technology 2015 (PICMET’15), (pp. 2581–2589).
Ittipanuvat, V., Fujita, K., Sakata, I., & Kajikawa, Y. (2014). Finding linkage between technology and social issue: A literature based discovery approach. Journal of Engineering and Technology Management, 32, 160–184. https://doi.org/10.1016/j.jengtecman.2013.05.006
Kajikawa, Y., Abe, K., & Noda, S. (2006). Filling the gap between researchers studying different materials and different methods: A proposal for structured keywords. Journal of Information Science, 32(6), 511–524. https://doi.org/10.1177/0165551506067125
Kajikawa, Y., Ohno, J., Takeda, Y., Matsushima, K., & Komiyama, H. (2007). Creating an academic landscape of sustainability science: An analysis of the citation network. Sustainability Science, 2(2), 221–231. https://doi.org/10.1007/s11625-007-0027-8
Kajikawa, Y., Yoshikawa, J., Takeda, Y., & Matsushima, K. (2008). Tracking emerging technologies in energy research: Toward a roadmap for sustainable energy. Technological Forecasting and Social Change, 75(6), 771–782. https://doi.org/10.1016/j.techfore.2007.05.005
Kanchan, T., & Krishan, K. (2019). The Leiden manifesto and research assessment. Science and Engineering Ethics, 25(2), 643–644. https://doi.org/10.1007/s11948-017-0012-2
Kay, L., Newman, N., Youtie, J., Porter, A. L., & Rafols, I. (2014). Patent overlay mapping: Visualizing technological distance. Journal of the Association for Information Science and Technology, 65(12), 2432–2443. https://doi.org/10.1002/asi.23146
Klavans, R., & Boyack, K. W. (2006). Identifying a better measure of relatedness for mapping science. Journal of the American Society for Information Science and Technology, 57(2), 251–263. https://doi.org/10.1002/asi.20274
Klavans, R., Boyack, K. W., & Murdick, D. A. (2020). A novel approach to predicting exceptional growth in research. PLoS ONE, 15(9), e0239177. https://doi.org/10.1371/journal.pone.0239177
Lane, J., & Bertuzzi, S. (2011). Measuring the results of science investments. Science, 331(6018), 678–680. https://doi.org/10.1126/science.1201865
Li, Y. T., Huang, M. H., & Chen, D. Z. (2014). Positioning and shifting of technology focus for integrated device manufacturers by patent perspectives. Technological Forecasting and Social Change, 81, 363–375. https://doi.org/10.1016/j.techfore.2013.04.017
Martin, B. R. (2011). The research excellence framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247–254. https://doi.org/10.3152/095820211X13118583635693
Martin, B. R. (2012). The evolution of science policy and innovation studies. Research Policy, 41(7), 1219–1239. https://doi.org/10.1016/j.respol.2012.03.012
Mejia, C., Wu, M., Zhang, Y., & Kajikawa, Y. (2021). Exploring topics in bibliometric research through citation networks and semantic analysis. Frontiers in Research Metrics and Analytics, 6, 742311. https://doi.org/10.3389/frma.2021.742311
Morlacchi, P., & Martin, B. R. (2009). Emerging challenges for science, technology and innovation policy research: A reflexive overview. Research Policy, 38(4), 571–582. https://doi.org/10.1016/j.respol.2009.01.021
Nakamura, H., Suzuki, S., Sakata, I., & Kajikawa, Y. (2015). Knowledge combination modeling: The measurement of knowledge similarity between different technological domains. Technological Forecasting and Social Change, 94, 187–201. https://doi.org/10.1016/j.techfore.2014.09.009
Nakamura, T., Iwabuchi, H., Murata, H., & Sakata, K. (2008). Supply cost and technology issues of hydrogen for fuel cell vehicles. Hydrogen Energy System, 33, 27–34. (in Japanese).
Ogawa, T., & Kajikawa, Y. (2017). Generating novel research ideas using computational intelligence: A case study involving fuel cells and ammonia synthesis. Technological Forecasting and Social Change, 120, 41–47. https://doi.org/10.1016/j.techfore.2017.04.004
Ogawa, T., Takeuchi, M., & Kajikawa, Y. (2018a). Comprehensive analysis of trends and emerging technologies in all types of fuel cells based on a computational method. Sustainability, 10(2), 458. https://doi.org/10.3390/su10020458
Ogawa, T., Takeuchi, M., & Kajikawa, Y. (2018b). Analysis of trends and emerging technologies in water electrolysis research based on a computational method: A comparison with fuel cell research. Sustainability, 10(2), 478. https://doi.org/10.3390/su10020478
Oliver, K., Lorenc, T., & Innvær, S. (2014). New directions in evidence-based policy research: A critical analysis of the literature. Health Research Policy and Systems, 12, 34. https://doi.org/10.1186/1478-4505-12-34
Pawson, R., Wong, G., & Owen, L. (2011). Known knowns, known unknowns, unknown unknowns: The predicament of evidence-based policy. American Journal of Evaluation, 32(4), 518–546. https://doi.org/10.1177/1098214011403831
Pedersen, D. B. (2014). The political epistemology of science-based policy-making. Society, 51(5), 547–551. https://doi.org/10.1007/s12115-014-9820-z
Rafols, I., Porter, A. L., & Leydesdorff, L. (2010). Science overlay maps: A NewTool for research policy and library management. Journal of the American Society for Information Science and Technology, 61(9), 1871–1887. https://doi.org/10.1002/asi.21368
Rodriguez, A., Tosyali, A., Kim, B., Choi, J., Lee, J.-M., Coh, B.-Y., & Jeong, M. K. (2016). Patent clustering and outlier ranking methodologies for attributed patent citation networks for technology opportunity discovery. IEEE Transactions on Engineering Management, 63(4), 426–437. https://doi.org/10.1109/TEM.2016.2580619
Sharman, A., & Holmes, J. (2010). Evidence-based policy or policy-based evidence gathering? Biofuels, the EU and the 10% target. Environmental Policy and Governance, 20(5), 309–321. https://doi.org/10.1002/eet.543
Shibata, N., Kajikawa, Y., & Matsushima, K. (2007). Topological analysis of citation networks to discover the future core articles. Journal of the American Society for Information Science and Technology, 58(6), 872–882. https://doi.org/10.1002/asi.20529
Shibata, N., Kajikawa, Y., & Sakata, I. (2011). Measuring relatedness between communities in a citation network. Journal of the American Society for Information Science and Technology, 62(7), 1360–1369. https://doi.org/10.1002/asi.21477
Shibata, N., Kajikawa, Y., Takeda, Y., & Matsushima, K. (2008). Detecting emerging research fronts based on topological measures in citation networks of scientific publications. Technovation, 28(11), 758–775. https://doi.org/10.1016/j.technovation.2008.03.009
Shibata, N., Kajikawa, Y., Takeda, Y., & Matsushima, K. (2009). Comparative study on methods of detecting research fronts using different types of citation. Journal of the American Society for Information Science and Technology, 60(3), 571–580. https://doi.org/10.1002/asi.20994
Smalheiser, N. R. (2012). Literature-based discovery: Beyond the ABCs. Journal of the American Society for Information Science and Technology, 63(2), 218–224. https://doi.org/10.1002/asi.21599
Small, H., Boyack, K. W., & Klavans, R. (2014). Identifying emerging topics in science and technology. Research Policy, 43(8), 1450–1467. https://doi.org/10.1016/j.respol.2014.02.005
Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. https://doi.org/10.2105/AJPH.2005.066043
Šubelj, L., van Eck, N. J., & Waltman, L. (2016). Clustering scientific publications based on citation relations: A systematic comparison of different methods. PLoS ONE, 11(4), e0154404. https://doi.org/10.1371/journal.pone.0154404
Swanson, D. R. (1986). Undiscovered public knowledge. Library Quarterly, 56(2), 103–118. https://doi.org/10.1086/601720
Takano, Y., & Kajikawa, Y. (2019). Extracting commercialization opportunities of the Internet of Things: Measuring text similarity between papers and patents. Technological Forecasting and Social Change, 138, 45–68. https://doi.org/10.1016/j.techfore.2018.08.008
Takano, Y., Mejia, C., & Kajikawa, Y. (2016). Unconnected component inclusion technique for patent network analysis: Case study of internet of things-related technologies. Journal of Informetrics, 10(4), 967–980. https://doi.org/10.1016/j.joi.2016.05.004
Thomas, D. A., Nedeva, M., Tirado, M. M., & Jacob, M. (2020). Changing research on research evaluation: A critical literature review to revisit the agenda. Research Evaluation, 29(3), 275–288. https://doi.org/10.1093/reseval/rvaa008
Torriti, J. (2010). Impact assessment and the liberalization of the EU energy markets: Evidence-based policy-making or policy-based evidence-making? JCMS: Journal of Common Market Studies, 48(4), 1065–1081. https://doi.org/10.1111/j.1468-5965.2010.02089.x
Tress, G., Tress, B., & Fry, G. (2005). Clarifying integrative research concepts in landscape ecology. Landscape Ecology, 20(4), 479–493. https://doi.org/10.1007/s10980-004-3290-4
Tsay, M. Y. (2008). A bibliometric analysis of hydrogen energy literature, 1965–2005. Scientometrics, 75(3), 421–438. https://doi.org/10.1007/s11192-007-1785-x
Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., Rafols, I., & Börner, K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14–26. https://doi.org/10.1016/j.joi.2010.06.004
Weeber, M., Klein, H., de Jong-van den Berg, L. T. W., & Vos, R. (2001). Using concepts in literature-based discovery: Simulating Swanson’s Raynaud–fish oil and migraine–magnesium discoveries. Journal of the American Society for Information Science and Technology, 52(7), 548–557. https://doi.org/10.1002/asi.1104
Xu, L., & Kajikawa, Y. (2018). An integrated framework for resilience research: A systematic review based on citation network analysis. Sustainability Science, 13(1), 235–254. https://doi.org/10.1007/s11625-017-0487-4
Yang, G.-C., Li, G., Li, C.-Y., Zhao, Y.-H., Zhang, J., Liu, T., Chen, D.-Z., & Huang, M.-H. (2015). Using the comprehensive patent citation network (CPC) to evaluate patent value. Scientometrics, 105(3), 1319–1346. https://doi.org/10.1007/s11192-015-1763-7
Acknowledgements
A part of this research is financially supported by Science for REdesigning Science, Technology and Innovation Policy (SciREX) program, Japan Science and Technology Agency (JST).
Author information
Authors and Affiliations
Corresponding author
Additional information
A part of this paper was presented as a keynote in 10th Annual Global TechMining Conference (11-13th, November, 2020).
Rights and permissions
About this article
Cite this article
Kajikawa, Y. Reframing evidence in evidence-based policy making and role of bibliometrics: toward transdisciplinary scientometric research. Scientometrics 127, 5571–5585 (2022). https://doi.org/10.1007/s11192-022-04325-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-022-04325-6