Skip to main content

Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach


Citation counts have long been considered as the primary bibliographic indicator for evaluating the quality of research—a practice premised on the assumption that citation count is reflective of the impact of a scientific publication. However, identifying several limitations in the use of citation counts alone, scholars have advanced the need for multifaceted quality evaluation methods. In this study, we apply a novelty indicator to quantify the degree of citation similarity between a focal paper and a pre-existing same-domain paper from various fields in the natural sciences by proposing a new way of identifying papers that fall into the same domain of focal papers using bibliometric data only. We also conduct a validation analysis, using Japanese survey data, to confirm its usefulness. Employing ordered logit and ordinary least squares regression models, this study tests the consistency between the novelty scores of 1871 Japanese papers published in the natural sciences between 2001 and 2006 and researchers’ subjective judgments of their novelty. The results show statistically positive correlations between novelty scores and researchers’ assessment of research types reflecting aspects of novelty in various natural science fields. As such, this study demonstrates that the proposed novelty indicator is a suitable means of identifying the novelty of various types of natural scientific research.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2


  1. The journal field refers the 22 scientific fields in the Essential Science Indicators (ESI) of Thomson Reuters.

  2. The reclassification procedures of multidisciplinary field papers were as follows: (i) collecting the references of a focal paper in the multidisciplinary field; (ii) identifying the scientific field of each reference, where a field was identified based on the scientific fields of a journal; (iii) finding the most frequent scientific field in the references of the focal paper, except for multidisciplinary fields; and (iv) using the most frequent scientific field as the scientific field of the focal paper.

  3. These correspond to focal papers without reference papers or having no same-domain papers. For these focal papers, the novelty scores are not calculable or become zero (the latter case is rare in our study; there are only two observations).

  4. As shown in Tables 2 and 3, our novelty scores are close to 1 and their variances are small. Previous research indicators (i.e., those used by Dahlin and Behrens (2005) and Trapido (2015)), which are the basis of our indicators, also have similar features. The small variation in the scores may make it difficult to interpret whether novelty is high or low, especially for the practical use of the indicators. On this point, applying methods such as standardization would help interpret the indicators. Figure 2 is one such example where we adopted percentile representation for the horizontal axis.

  5. This tendency is also confirmed in the other citation windows.

  6. The ordered logit and OLS regression models use the same dependent and independent variables with robust standard errors.


  • Ahmed, T., Johnson, B., Oppenheim, C., & Peck, C. (2004). Highly cited old papers and the reasons why they continue to be cited: Part II. The 1953 Watson and Crick article on the structure of DNA. Scientometrics, 61, 147–156.

    Article  Google Scholar 

  • Baird, L. M., & Oppenheim, C. (1994). Do citations matter? Journal of Information Science, 20(1), 2–15.

    Article  Google Scholar 

  • Bornmann, L., Schier, H., Marx, W., & Daniel, H. D. (2012). What factors determine citation counts of publications in chemistry besides their quality? Journal of Informetrics, 6(1), 11–18.

    Article  Google Scholar 

  • Bornmann, L., Tekles, A., Zhang, H. H., & Fred, Y. Y. (2019). Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data. Journal of Informetrics, 13(4), 100979.

    Article  Google Scholar 

  • Clarivate Analytics. (2020). Web of science core collection help. Accessed 16 October 2020.

  • Dahlin, K. B., & Behrens, D. M. (2005). When is an invention really radical? Defining and measuring technological radicalness. Research Policy, 34(5), 717–737.

    Article  Google Scholar 

  • Fleming, L. (2001). Recombinant uncertainty in technological search. Management Science, 47(1), 117–132.

    Article  Google Scholar 

  • Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431.

    Article  Google Scholar 

  • Igami, M., Nagaoka, S., & Walsh, J. P. (2015). Contribution of postdoctoral fellows to fast-moving and competitive scientific research. The Journal of Technology Transfer, 40(4), 723–741.

    Article  Google Scholar 

  • Kaplan, S., & Vakili, K. (2015). The double-edged sword of recombination in breakthrough innovation. Strategic Management Journal, 36(10), 1435–1457.

    Article  Google Scholar 

  • Lee, Y.-N., Walsh, J. P., & Wang, J. (2015). Creativity in scientific teams: Unpacking novelty and impact. Research Policy, 44(3), 684–697.

    Article  Google Scholar 

  • MacRoberts, M., & MacRoberts, B. (1996). Problems of citation analysis. Scientometrics, 36(3), 435–444.

    Article  Google Scholar 

  • Mednick, S. (1962). The associative basis of the creative process. Psychological Review, 69(3), 220–232.

    Article  Google Scholar 

  • Murayama, K., Nirei, M., & Shimizu, H. (2015). Management of science, serendipity, and research performance: Evidence from a survey of scientists in Japan and the US. Research Policy, 44(4), 862–873.

    Article  Google Scholar 

  • Nagaoka, S., Igami, M., Eto, M., & Ijichi, T. (2010). Knowledge creation process in science: Basic findings from a large-scale survey of researchers in Japan. IIR Working Paper, WP#10–08. Japan: Institute of Innovation Research, Hitotsubashi University.

  • Nelson, R. R., & Winter, S. G. (1982). An evolutionary theory of economic change. Belknap Press of Harvard University Press.

  • Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology.

    Article  Google Scholar 

  • Oppenheim, C., & Renn, S. P. (1978). Highly cited old papers and reasons why they continue to be cited. Journal of the American Society for Information Science, 29, 225–231.

    Article  Google Scholar 

  • Romer, P. M. (1994). The origins of endogenous growth. Journal of Economic Perspectives, 8(1), 3–22.

    Article  Google Scholar 

  • Schumpeter, J. A. (1939). Business cycles: A theoretical, historical and statistical analysis of the capitalist process. McGraw-Hill Book Company.

  • Simonton, D. K. (2003). Scientific creativity as constrained stochastic behavior: The integration of product, person and process perspectives. Psychological Bulletin, 129(4), 475–494.

    Article  Google Scholar 

  • Tahamtan, I., & Bornmann, L. (2018). Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references? Journal of Informetrics, 12(3), 906–930.

    Article  Google Scholar 

  • Thelwall, M. (2017). Web indicators for research evaluation: A practical guide. Synthesis Lectures on Information Concepts, Retrieval and Services, 8(4), i1–i155.

    Google Scholar 

  • Trapido, D. (2015). How novelty in knowledge earns recognition: The role of consistent identities. Research Policy, 44(8), 1488–1500.

    Article  Google Scholar 

  • Uddin, S., Khan, A., & Baur, L. A. (2015). A framework to explore the knowledge structure of multidisciplinary research fields. PLoS ONE, 10(4), e0123537.

    Article  Google Scholar 

  • Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468–472.

    Article  Google Scholar 

  • Verhoeven, D., Bakker, J., & Veugelers, R. (2016). Measuring technological novelty with patent-based indicators. Research Policy, 45(3), 707–723.

    Article  Google Scholar 

  • Walsh, J. P., & Lee, Y. N. (2015). The bureaucratization of science. Research Policy, 44(8), 1584–1600.

    Article  Google Scholar 

  • Wang, J., Lee, Y.-N., & Walsh, J. P. (2018). Funding model and creativity in science: Competitive versus block funding and status contingency effects. Research Policy, 47(6), 1070–1083.

    Article  Google Scholar 

  • Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416–1436.

    Article  Google Scholar 

  • Zdaniuk, B. (2014). Ordinary least-squares (OLS) model. In A. C. Michalos (Ed.), Encyclopedia of quality of life and well-being research.Springer.

Download references


We wish to thank Natsuo Onodera for his invaluable insights regarding the measuring of the novelty score.



Author information

Authors and Affiliations



All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by KM. The first draft of the manuscript was written by KM, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kuniko Matsumoto.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.



See Tables

Table 8 Means of novelty across all the fields by the degree of the relevance of research types


Table 9 Correlation coefficients between novelty across all the fields and the degree of the relevance of research types


Table 10 Means of novelty in each field by the degree of the relevance of research types

10 and

Table 11 Correlation coefficients between novelty in each field and the degree of the relevance of research types


Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Matsumoto, K., Shibayama, S., Kang, B. et al. Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach. Scientometrics 126, 6891–6915 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: