Skip to main content
SpringerLink
Log in

Measuring the innovation of method knowledge elements in scientific literature

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Interest in assessing research impacts is increasing due to its importance for informing actions and funding allocation decisions. The level of innovation (also called “innovation degree” in the following article), one of the most essential factors that affect scientific literature’s impact, has also received increasing attention. However, current studies mainly focus on the overall innovation degree of scientific literature at the macro level, while ignoring the innovation degree of a specific knowledge element (KE), such as the method knowledge element (MKE). A macro level view causes difficulties in identifying which part of the scientific literature contains the innovations. To bridge this gap, a more fine-grained evaluation of academic papers is urgent. The fine-grained evaluation method can ensure the quality of a paper before being published and identify useful knowledge content in a paper for academic users. Different KEs can be used to perform the fine-grained evaluation. However, MKEs are usually considered as one of the most essential knowledge elements among all KEs. Therefore, this study proposes a framework to measure the innovation degree of method knowledge elements (MIDMKE) in scientific literature. In this framework, we first extract the MKEs using a rule-based approach and generate a cloud drop for each MKE using the biterm topic model (BTM). The generated cloud drop is then used to create a method knowledge cloud (MKC) for each MKE. Finally, we calculate the innovation score of a MKE based on the similarity between it and other MKEs of its type. Our empirical study on a China National Knowledge Infrastructure (CNKI) academic literature dataset shows the proposed approach can measure the innovation of MKEs in scientific literature effectively. Our proposed method is useful for both reviewers and funding agencies to assess the quality of academic papers. The dataset, the code for implementation the algorithms, and the complete experiment results will be released at: https://github.com/haihua0913/midmke.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abramo, G., D’Angelo, C. A., & Di Costa, F. (2011). National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics, 89(3), 929–941.

    Article  Google Scholar 

  • Behret, H., & Gumussoy, C. A. (2012). A fuzzy integrated approach for the selection of academic papers to a special issue. International Journal of Applied Management Science, 4(4), 371–384.

    Article  Google Scholar 

  • Bornmann, L., Wallon, G., & Ledin, A. (2008). Is the h index related to (standard) bibliometric measures and to the assessments by peers? an investigation of the h index by using molecular life sciences data. Research Evaluation, 17(2), 149–156.

    Article  Google Scholar 

  • Bryman, A. (2008). Of methods and methodology. Qualitative Research in Organizations and Management, 3(2), 159–168.

  • Cai, L., Tian, J., Liu, J., Bai, X., Lee, I., Kong, X., & Xia, F. (2019). Scholarly impact assessment: A survey of citation weighting solutions. Scientometrics, 118(2), 453–478.

    Article  Google Scholar 

  • Campbell, P. (2008). Escape from the impact factor. Ethics in science and environmental politics, 8(1), 5–7.

    Article  MathSciNet  Google Scholar 

  • Chen, H., & Kanuboddu, B.N. (2021). A fine-grained annotation scheme for research contribution in academic literature. In Proceedings of the 18th International Conference on Scientometrics and Informetrics, pp 241–248.

  • Chen, Y. (2015). Convolutional neural network for sentence classification. Master’s thesis, University of Waterloo.

  • Chu, H., & Ke, Q. (2017). Research methods: What’s in the name? Library & Information Science Research, 39(4), 284–294.

    Article  Google Scholar 

  • Colquhoun, D. (2003). Challenging the tyranny of impact factors. Nature, 423(6939), 479–479.

    Article  Google Scholar 

  • Costanzo, B. P., & Sánchez, L. E. (2019). Innovation in impact assessment theory and practice: How is it captured in the literature? Environmental Impact Assessment Review, 79(106), 289.

    Google Scholar 

  • Dahl, T. (2008). Contributing to the academic conversation: A study of new knowledge claims in economics and linguistics. Journal of Pragmatics, 40(7), 1184–1201.

    Article  Google Scholar 

  • Darling, E. (2015). Use of double-blind peer review to increase author diversity. Conservation Biology, 29(1), 297–299.

    Article  Google Scholar 

  • Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805.

  • Ding, Y., Song, M., Han, J., Yu, Q., Yan, E., Lin, L., & Chambers, T. (2013). Entitymetrics: Measuring the impact of entities. PLoS ONE, 8(8), 71–416.

    Google Scholar 

  • Fan, X., Feng, H., & Yuan, M. (2013). Feature selection algorithm for principal component analysis based on mutual information (in Chinese). Control and Decision, 28(06), 915–919.

    Google Scholar 

  • Frank, M. (2003). Impact factors: Arbiter of excellence? Journal of the Medical Library Association, 91(1), 4–6.

    Google Scholar 

  • Funk, R. J., & Owen-Smith, J. (2017). A dynamic network measure of technological change. Management Science, 63(3), 791–817.

    Article  Google Scholar 

  • Garfield, E. (1999). Journal impact factor: A brief review. CMAJ, 161(8), 979–980.

    Google Scholar 

  • Garfield, E. (2001). Impact factors, and why they won’t go away. Nature, 411(6837), 522–522.

    Article  Google Scholar 

  • Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.

    Article  Google Scholar 

  • Gong, L., & Ji, R. (2018). What does a textcnn learn? arXiv preprint arXiv:180106287.

  • Gupta, V., Giesselbach, S., Rüping, S., & Bauckhage, C. (2019). Improving word embeddings using kernel PCA. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pp 200–208.

  • He, J., & Chen, C. (2018). Predictive effects of novelty measured by temporal embeddings on the growth of scientific literature. Frontiers in Research Metrics and Analytics, 3, 9.

    Article  Google Scholar 

  • He, L., & Du, X. (2020). Research on intelligent evaluation for the content innovation of academic papers (in Chinese). Library and Information Service, 64(1), 93–104.

    Google Scholar 

  • He, Q., Chen, B., Pei, J., Qiu, B., Mitra, P., & Giles, L. (2009). Detecting topic evolution in scientific literature: How can citations help? In Proceedings of the 18th ACM conference on Information and knowledge management, pp 957–966

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National academy of Sciences, 102(46), 16569–16572.

    Article  MATH  Google Scholar 

  • Hua, B. (2016). Types and description rules of knowledge elements about method in academic papers (in Chinese). Journal of Library Science in China, 42(1), 30–40.

    Google Scholar 

  • Hunston, S. (1993). Professional conflict-disagreement in academic discourse. In Text and technology, John Benjamins, p 115.

  • Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The r-and ar-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.

    Article  Google Scholar 

  • Kosmulski, M. (2011). Successful papers: A new idea in evaluation of scientific output. Journal of Informetrics, 5(3), 481–485.

    Article  Google Scholar 

  • Li, D., Liu, C., & Gan, W. (2009). A new cognitive model: Cloud model. International Journal of Intelligent Systems, 24(3), 357–375.

    Article  MATH  Google Scholar 

  • Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P.S., & He, L. (2020). A survey on text classification: From shallow to deep learning. arXiv preprint arXiv:200800364.

  • Lovegrove, B. G., & Johnson, S. D. (2008). Assessment of research performance in biology: How well do peer review and bibliometry correlate? Bioscience, 58(2), 160–164.

    Article  Google Scholar 

  • Lu, Y., Mei, Q., & Zhai, C. (2011). Investigating task performance of probabilistic topic models: An empirical study of PLSA and IDA. Information Retrieval, 14(2), 178–203.

    Article  Google Scholar 

  • Marcondes, C. H., & da Costa, L. C. (2016). A model to represent and process scientific knowledge in biomedical articles with semantic web technologies. KO Knowledge Organization, 43(2), 86–101.

    Article  Google Scholar 

  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:13013781.

  • Mörchen, F., Dejori, M., Fradkin, D., Etienne, J., Wachmann, B., & Bundschus, M. (2008). Anticipating annotations and emerging trends in biomedical literature. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 954–962.

  • Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2013). Absolute and specific measures of research group excellence. Scientometrics, 95(1), 115–127.

    Article  Google Scholar 

  • Myers, G. (1992). ‘in this paper we report...’: Speech acts and scientific facts. Journal of Pragmatics, 17(4), 295–313.

    Article  Google Scholar 

  • Notkins, A. L. (2008). Neutralizing the impact factor culture. Science, 322(5899), 191–191.

    Article  Google Scholar 

  • Onodera, N., & Yoshikane, F. (2015). Factors affecting citation rates of research articles. Journal of the Association for Information Science and Technology, 66(4), 739–764.

    Article  Google Scholar 

  • Packalen, M., & Bhattacharya, J. (2019). Age and the trying out of new ideas. Journal of Human Capital, 13(2), 341–373.

    Article  Google Scholar 

  • Pei, Y., & Liu, X. (2011). Research on improved chi feature selection method in text classification (in Chinese). Computer Engineering and Applications, 47(04), 128–130.

    Google Scholar 

  • Pennington, J., Socher, R., & Manning, C.D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543.

  • Reich, Y. (1995). Measuring the value of knowledge. International Journal of Human-Computer Studies, 42(1), 3–30.

    Article  Google Scholar 

  • Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81(3), 789–809.

    Article  Google Scholar 

  • Ren, Y., Yang, R., & Yin, M. (2010). Text feature selection method based on information gain (in Chinese). Computer Science, 39(11), 127–130.

    Google Scholar 

  • Ronzano, F., & Saggion, H. (2016). Knowledge extraction and modeling from scientific publications. In International workshop on semantic, analytics, visualization, Springer, pp 11–25.

  • Rousseau, R., & Leuven, K. (2008). Reflections on recent developments of the h-index and h-type indices. COLLNET Journal of Scientometrics and Information Management, 2(1), 1–8.

    Article  Google Scholar 

  • Shang, W., Huang, H., & Liu, Y. (2006). Research on feature selection algorithm based on gini index in text classification (in Chinese). Application Research of Computers, 10, 1688–1694.

    Google Scholar 

  • Sombatsompop, N., Kositchaiyong, A., Markpin, T., & Inrit, S. (2006). Scientific evaluations of citation quality of international research articles in the sci database: Thailand case study. Scientometrics, 66(3), 521–535.

    Article  Google Scholar 

  • Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225.

    Article  Google Scholar 

  • Tang, Q, (2014). Short text clustering method based on btm (in Chinese). PhD dissertation, Anhui University.

  • Tkaczyk, D., Szostek, P., Fedoryszak, M., Dendek, P. J., & Bolikowski, Ł. (2015). Cermine: Automatic extraction of structured metadata from scientific literature. International Journal on Document Analysis and Recognition (IJDAR), 18(4), 317–335.

    Article  Google Scholar 

  • Trapido, D. (2015). How novelty in knowledge earns recognition: The role of consistent identities. Research Policy, 44(8), 1488–1500.

    Article  Google Scholar 

  • Uddin, S., & Khan, A. (2016). The impact of author-selected keywords on citation counts. Journal of Informetrics, 10(4), 1166–1177.

    Article  Google Scholar 

  • Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468–472.

    Article  Google Scholar 

  • Walker, R., & Rocha da Silva, P. (2015). Emerging trends in peer review-a survey. Frontiers in Neuroscience, 9, 169.

    Article  Google Scholar 

  • Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416–1436.

    Article  Google Scholar 

  • Wang, J., Lee, Y. N., & Walsh, J. P. (2018). Funding model and creativity in science: Competitive versus block funding and status contingency effects. Research Policy, 47(6), 1070–1083.

    Article  Google Scholar 

  • Wang, M., Zhang, J., Jiao, S., Zhang, X., Zhu, N., & Chen, G. (2020). Important citation identification by exploiting the syntactic and contextual information of citations. Scientometrics, 125(3), 2109–2129.

    Article  Google Scholar 

  • Wang, R., Gao, X., Gao, Z., Li, S., Gao, J., Xu, J., & Deng, W. (2020). Comprehensive reliability evaluation of multistate complex electromechanical systems based on similarity of cloud models. Quality and Reliability Engineering International, 36(3), 1048–1073.

    Article  Google Scholar 

  • Wang, S., Zheng, Z., Sun, Q., Zou, H., & Yang, F. (2011). Cloud model for service selection. In: 2011 IEEE conference on computer communications workshops (INFOCOM WKSHPS), IEEE, pp 666–671.

  • Wang, Z., Shen, X., Huang, R., & Huang, J. (2019). Extracting method knowledge elements from scientific literature: A rule-based approach. Proceedings of the Association for Information Science and Technology, 56(1), 805–807.

    Article  Google Scholar 

  • Wu, L., Wang, D., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature, 566(7744), 378–382.

    Article  Google Scholar 

  • Xie, J., & Xie, W. (2014). Feature selection algorithm based on feature subset discrimination and support vector machine (in Chinese). Chinese Journal of Computers, 37(08), 1704–1718.

    Google Scholar 

  • Yan, E. (2014). Research dynamics: Measuring the continuity and popularity of research topics. Journal of Informetrics, 8(1), 98–110.

    Article  Google Scholar 

  • Yan, X., Guo, J., Lan, Y., & Cheng, X. (2013). A biterm topic model for short texts. In Proceedings of the 22nd international conference on World Wide Web, pp 1445–1456.

  • Yang, K., Zhang, Y., & Li, Y. (2010). Feature selection method based on document frequency (in Chinese). Computer Engineering, 36(7), 33–35.

    Google Scholar 

  • Yao, D., Yang, J., & Zhan, X. (2014). Feature selection algorithm based on random forest (in Chinese). Journal of Jilin University (Engineering and Technology Edition), 44(01), 137–141.

    Google Scholar 

  • Zan, W., Tsim, Y., Yeung, W., Chan, K., & Liu, J. (2007). Probabilistic latent semantic analyses (plsa) in bibliometric analysis for technology forecasting. Journal of Technology Management & Innovation, 2(1), 11–24.

    Google Scholar 

  • Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, H. E. (2017). The science of science: From the perspective of complex systems. Physics Reports, 714, 1–73.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhai, Y., Ding, Y., & Wang, F. (2018). Measuring the diffusion of an innovation: A citation analysis. Journal of the Association for Information Science and Technology, 69(3), 368–379.

    Article  Google Scholar 

  • Hy, Zhang, Ji, P., Jq, Wang, & Chen, Xh. (2016). A neutrosophic normal cloud and its application in decision-making. Cognitive Computation, 8(4), 649–669.

    Article  Google Scholar 

  • Zhang, L., Sun, G., & Guo, J. (2005). Unsupervised feature selection method based on k-means clustering (in Chinese). Application Research of Computers, 42, 23–24.

    Google Scholar 

  • Zhang, Z., Li, S., & Li, Z. (2013). A multi-label feature selection algorithm based on information entropy (in Chinese). Journal of Computer Research and Development, 50(06), 1177–1184.

    Google Scholar 

Download references

Acknowledgements

The authors are grateful to all the anonymous reviewers for their precious comments and suggestions.

Funding

This study was supported by Humanities and Social Science Research Foundation of Ministry of Education of China (Grant Number 21YJA870003) and National Social Science Foundation of China (Grant Number 19ZDA345).

Author information

Authors and Affiliations

  1. School of Information Management, Central China Normal University, Wuhan, 430079, China

    Zhongyi Wang, Keying Wang & Jiyue Liu

  2. Wuhan Polytechnic, Wuhan, 430023, China

    Jing Huang

  3. Department of Information Science, University of North Texas, Denton, TX, 76203, USA

    Haihua Chen

Authors
  1. Zhongyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Keying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiyue Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haihua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by ZW, KW, JL, JH, and HC. The first draft of the manuscript was written by ZW and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Haihua Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Wang, K., Liu, J. et al. Measuring the innovation of method knowledge elements in scientific literature. Scientometrics 127, 2803–2827 (2022). https://doi.org/10.1007/s11192-022-04350-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-022-04350-5

Keywords

Search

Navigation