Skip to main content
SpringerLink
Log in

ResGAT: an improved graph neural network based on multi-head attention mechanism and residual network for paper classification

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Paper classification plays a pivotal role in facilitating precise literature retrieval, recommendations, and bibliometric analyses. However, current text-based methods predominantly emphasize intrinsic features such as titles, abstracts, and keywords, overlooking the valuable insights concealed within reference papers (i.e., cited papers). As a result, this oversight leads to reduced classification accuracy. In contrast, as a practical deep learning approach, graph neural networks incorporate the characteristics of reference papers to enhance paper classification. Nevertheless, traditional graph neural networks encounter limitations when handling intricate multi-level citation relationships in academic papers. To address these challenges, we introduce an enhanced graph neural network model for academic paper classification. This model integrates a multi-head attention mechanism and a residual network structure to dynamically allocate weights to various nodes within the graph, thereby enhancing its ability to handle complex multi-level citation relationships. Our experimental findings on an extensive real-world dataset demonstrate that our model achieves an accuracy of 61%, surpassing traditional graph neural networks by over 4%. Additionally, we have made the relevant datasets and models accessible on our GitHub repository. (https://github.com/xuejianhuang/ResGAT-for-paper-classification).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets analysed during the current study are available in the Github, https://github.com/xuejianhuang/ResGAT-for-paper-classification.

References

  • Asim, M. N., Ghani, M. U., Ibrahim, M. A., Waqar, M., Andreas, D., & Sheraz, A. (2021). Benchmarking performance of machine and deep learning-based methodologies for urdu text document classification. Neural Computing and Applications, 33, 5437–5469.

    Google Scholar 

  • Bafna, P., Pramod, D., & Vaidya, A. (2016). Document clustering: TF-IDF approach. In Proceedings of the 2016 International conference on electrical, electronics, and optimization techniques (ICEEOT), Chennai, India, March 2016 (pp. 61–66).

  • Beel, J., Gipp, B., Langer, S., & Corinna, B. (2016). Paper recommender systems: A literature survey. International Journal on Digital Libraries, 17, 305–338.

    Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    Google Scholar 

  • Boyack, K. W., Newman, D., Duhon, R. J., Richard, K., Michael, P., Joseph, R., & B., Bob, S., André, S., Nianli, M., & Katy, B. (2011). Clustering more than two million biomedical publications: Comparing the accuracies of nine text-based similarity approaches. PLoS ONE,6(3), e18029.

  • Bruna, J., Zaremba, W., Szlam, A., & Yann, L. (2014). Spectral networks and locally connected networks on graphs. In Proceedings of the 2nd international conference on learning representations (ICLR), Banff, Canada, April 2014 (pp. 1–14).

  • Chen, D., Lin, Y., Li, W., Peng, L., Jie, Z., & Xu, S. (2020). Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the 34th AAAI conference on artificial intelligence, New York, February 2020 (pp. 3438–3445).

  • Chen, Q., Du, J., Allot, A., & Zhiyong, L. (2022). LITMC-BERT: Transformer-based multi-label classification of biomedical literature with an application on covid-19 literature curation. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 19(5), 2584–2595.

    PubMed  Google Scholar 

  • Chuanakrud, P., Leelanupab, T., Damrongrat, C., & Nont, K. (2021). Keyword-text graph representation for short text classification. In Proceedings of the 13th international conference on information technology and electrical engineering (ICITEE), Chiang Mai, Thailand, October 2021 (pp. 24–29).

  • Daradkeh, M., Abualigah, L., Atalla, S., & Wathiq, M. (2022). Scientometric analysis and classification of research using convolutional neural networks: A case study in data science and analytics. Electronics, 11(13), 2066.

    Google Scholar 

  • Dong, F., Liu, Y., & Zhou, Y. (2017). Prediction of emerging technologies based on LDA SVM multi class abstract of paper classification. Journal of Intelligence, 36(7), 40–45.

    Google Scholar 

  • Donthu, N., Kumar, S., Mukherjee, D., Nitesh, P., & Marc, L. W. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133, 285–296.

    Google Scholar 

  • Du, J., Vong, C. M., & Chen, C. P. (2020). Novel efficient rnn and LSTM-like architectures: Recurrent and gated broad learning systems and their applications for text classification. IEEE Transactions on Cybernetics, 51(3), 1586–1597.

    Google Scholar 

  • Dzisevič, R., & Šešok, D. (2019). Text classification using different feature extraction approaches. In Proceedings of the 2019 open conference of electrical, electronic and information sciences (eStream), Vilnius, Lithuania, April 2019 (pp. 1–4).

  • Eykens, J., Guns, R., & Engels, T. C. (2021). Fine-grained classification of social science journal articles using textual data: A comparison of supervised machine learning approaches. Quantitative Science Studies, 2(1), 89–110.

    Google Scholar 

  • Eykens, J., Guns, R., Engels, T. C., Catalano, G., Daraio, C., Gregori, M., Moed, H. F., & Ruocco, G. (2019). Article level classification of publications in sociology: An experimental assessment of supervised machine learning approaches. In Proceedings of the 17th international conference on scientometrics & informetrics, Rome, Italy, September 2019 (pp. 738–743).

  • Feng, X., Yue, H., Shuai, X., & Jian, D. X. (2018). Research on short text classification based on paper title and abstract. Journal of Hefei University of Technology: Natural Science, 41(10), 1343–1349.

    Google Scholar 

  • Glänzel, W., & Debackere, K. (2022). Various aspects of interdisciplinarity in research and how to quantify and measure those. Scientometrics, 127, 5551–5569.

    Google Scholar 

  • Glänzel, W., Thijs, B., & Huang, Y. (2021). Improving the precision of subject assignment for disparity measurement in studies of interdisciplinary research. In Proceedings of the 18th international conference of the international society of scientometrics and informetrics, Leuven, Belgium, July 2021 (pp. 453–464).

  • Gong, K. (2023). The influence of discipline consistency between papers and published journals on citations: An analysis of chinese papers in three social science disciplines. Scientometrics, 128, 3129–3146.

    Google Scholar 

  • Gori, M., Monfardini, G., & Scarselli, F. (2005). A new model for learning in graph domains. In Proceedings of the IEEE international joint conference on neural networks, Montreal, Canada, August 2005 (pp. 729–734).

  • Gu, Y., Wang, Y., Zhang, H. R., Jiao, W., & Xingquan, G. (2023). Enhancing text classification by graph neural networks with multi-granular topic-aware graph. IEEE Access, 11, 20169–20183.

    Google Scholar 

  • Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. In Proceedings of the 31th conference on neural information processing systems (NeurIPS), California, USA, December 2017 (pp. 1024–1034).

  • Hao, W., Peng, Y., & Sanhong, D. (2014). The application of machine-learning in the research on automatic categorization of chinese periodical articles. Data Analysis and Knowledge Discovery, 30(3), 80–87.

    Google Scholar 

  • Hartmann, J., Huppertz, J., Schamp, C., & Mark, H. (2019). Comparing automated text classification methods. International Journal of Research in Marketing, 36(1), 20–38.

    Google Scholar 

  • Kandimalla, B., Rohatgi, S., Wu, J., & Lee, G. C. (2021). Large scale subject category classification of scholarly papers with deep attentive neural networks. Frontiers in Research Metrics and Analytics, 5, 600382.

    PubMed  PubMed Central  Google Scholar 

  • Kim, S. W., & Gil, J. M. (2019). Research paper classification systems based on TF-IDF and LDA schemes. Human-Centric Computing and Information Sciences, 9, 1–21.

    Google Scholar 

  • Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th international conference on learning representations (ICLR), Toulon, France, April 2017 (pp. 1–14).

  • Koutsomitropoulos, D. A., & Andriopoulos, A. D. (2022). Thesaurus-based word embeddings for automated biomedical literature classification. Neural Computing and Applications, 34(2), 937–950.

    PubMed  Google Scholar 

  • Liefa, L., & Le Fugang, Z. Y. (2017). The application of LDA model in patent text classification. Journal of Modern Information, 37(3), 35–39.

    Google Scholar 

  • Liu, L., & Dongbo, W. (2018). Identifying interdisciplinary social science research based on article classification. Data Analysis and Knowledge Discovery, 2(3), 30–38.

    Google Scholar 

  • Liu, P., Yuan, W., Fu, J., Zhengbao, J., Hiroaki, H., & Graham, N. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35.

    Google Scholar 

  • Liu, S., Chen, C., Ding, K., Kun, D., Bo, W., Kan, X., & Yuan, L. (2014). Literature retrieval based on citation context. Scientometrics, 101, 1293–1307.

    CAS  Google Scholar 

  • Lucheng, L., Tao, H., Jian, Z., & Zhao, Y. (2020). Research on the method of chinese patent automatic classification based on deep learning. Library and Information Service, 64(10), 75–85.

    Google Scholar 

  • Lv, Y., Xie, Z., Zuo, X., & Yiping, S. (2022). A multi-view method of scientific paper classification via heterogeneous graph embeddings. Scientometrics, 127, 4847–4872.

    Google Scholar 

  • Malekzadeh, M., Hajibabaee, P., Heidari, M., Samira, Z., Ozlem, U., & James, H. J. (2021). Review of graph neural network in text classification. In Proceedings of the IEEE 12th annual ubiquitous computing, electronics, New York, USA, December 2021 (pp. 84–91).

  • Milojević, S. (2020). Practical method to reclassify web of science articles into unique subject categories and broad disciplines. Quantitative Science Studies, 1(1), 183–206.

    Google Scholar 

  • Nam, S., Kim, S. K., Kim, H. G., Victoria, N., & Nansu, Z. (2016). Structuralizing biomedical abstracts with discriminative linguistic features. Computers in Biology and Medicine, 79, 276–285.

    PubMed  Google Scholar 

  • Ni, B., Lu, X., Tong, Y., Tao, M., & Zhixian, Z. (2021). Automated journal text classification based on capsule neural network. Journal of Nanjing University: Natural Science, 57(5), 750–756.

    Google Scholar 

  • Robertson, S. E., & Jones, K. S. (1976). Relevance weighting of search terms. Journal of the American Society for Information science, 27(3), 129–146.

    Google Scholar 

  • Rúbio, T. R., & Gulo, C. A. (2016). Enhancing academic literature review through relevance recommendation: Using bibliometric and text-based features for classification. In Proceedings of the 11th Iberian Conference on Information Systems and Technologies (CISTI), Gran Canaria, Spain, June 2016 (pp. 1–6).

  • Salazar-Reyna, R., Gonzalez-Aleu, F., Granda-Gutierrez, E. M., Jenny, D., Arturo, G. R. J., & Anil, K. (2022). A systematic literature review of data science, data analytics and machine learning applied to healthcare engineering systems. Management Decision, 60(2), 300–319.

    Google Scholar 

  • Scarselli, F., Gori, M., Tsoi, A. C., Markus, H., & Gabriele, M. (2008). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61–80.

    PubMed  Google Scholar 

  • Sethares, W. A., Ingle, A., Krč, T., & Wood, S. (2014). Eigentextures: An SVD approach to automated paper classification. In Proceedings of the 48th Asilomar conference on signals, systems and computers, California, USA, November 2014 (pp. 1109–1113).

  • Shi, Y., Zhang, X., & Yu, N. (2023). Pl-transformer: a pos-aware and layer ensemble transformer for text classification. Neural Computing and Applications, 35(2), 1971–1982.

    Google Scholar 

  • Shu, F., Julien, C. A., Zhang, L., Junping, Q., Jing, Z., & Vincent, L. (2019). Comparing journal and paper level classifications of science. Journal of Informetrics, 13(1), 202–225.

    Google Scholar 

  • Shu, F., Ma, Y., Qiu, J., & Vincent, L. (2020). Classifications of science and their effects on bibliometric evaluations. Scientometrics, 125, 2727–2744.

    Google Scholar 

  • Stewart, G. W. (1993). On the early history of the singular value decomposition. SIAM Review, 35(4), 551–566.

    MathSciNet  Google Scholar 

  • Tran, L., Pham, L., Tran, T., & An, M. (2021). Text classification problems via bert embedding method and graph convolutional neural network. In 2021 International conference on advanced technologies for communications (ATC), Ho Chi Minh, Vietnam, October 2021 (pp. 260–264).

  • Veličković, P., Cucurull, G., Casanova, A., Adriana, R., Pietro, L., & Yoshua, B. (2018). Graph attention networks. In Proceedings of the 6th international conference on learning representations (ICLR), Vancouver, Canada, April 2018 (pp. 1–12).

  • Won, K., Choi, Hd., & Shin, S. (2021). Deep learning-based semantic classification of emf-related scientific literature. ACM SIGAPP Applied Computing Review, 21(2), 48–56.

    Google Scholar 

  • Xinyun, W., Hao, W., Sanhong, D., & Zhang, B. (2020). Classification of academic papers for periodical selection. Data Analysis and Knowledge Discovery, 4(7), 96–109.

    Google Scholar 

  • Yao, L., Mao, C., & Luo, Y. (2019). Graph convolutional networks for text classification. In Proceedings of the 33rd AAAI conference on artificial intelligence, Hawaii, USA, January 2019 (pp. 7370–7377).

  • Yuan, H., Yu, H., Gui, S., & Shuiwang, J. (2023). Explainability in graph neural networks: A taxonomic survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5), 5782–5799.

    PubMed  Google Scholar 

  • Yue, T., Li, Y., Shi, X., Jiedong, Q., Zijiao, F., & Zonghai, H. (2022). Papernet: A dataset and benchmark for fine-grained paper classification. Applied Sciences, 12(9), 4554.

    CAS  Google Scholar 

  • Zhang, C., Li, Z., & Chu, H. (2020). Using full content to automatically classify the research methods of academic articles. Journal of the China Society for Scientific and Technical Information, 39(8), 852–862.

    Google Scholar 

  • Zhang, L., Sun, B., Shu, F., & Ying, H. (2022). Comparing paper level classifications across different methods and systems: An investigation of nature publications. Scientometrics, 127, 7633–7651.

    Google Scholar 

  • Zhang, Z., & Sabuncu, M. (2018). Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd Conference on neural information processing systems (NeurIPS), Montrèal, Canada, December 2018 (pp. 8792–8802).

  • ZhengWei, H., JinTao, M., YanNi, Y., Huang, J., & Tian, Y. (2022). Recommendation method for academic journal submission based on doc2vec and XGBoost. Scientometrics, 127(5), 2381–2394.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of China (No. 72061015), and the Technology Project of Jiangxi Provincial Department of Education (No. GJJ209925).

Author information

Authors and Affiliations

  1. School of VR Modern Industry, Jiangxi University of Finance and Economics, Shuanggang East Street, Nanchang, 330013, Jiangxi, China

    Xuejian Huang

  2. Smart Campus Management Center, Jiangxi University of Finance and Economics, Shuanggang East Street, Nanchang, 330013, Jiangxi, China

    Zhibin Wu, Zhipeng Li & Xiaofang Wu

  3. School of Information Management, Jiangxi University of Finance and Economics, Shuanggang East Street, Nanchang, 330013, Jiangxi, China

    Gensheng Wang

  4. School of Software, Jiangxi University of Finance and Economics, Shuanggang East Street, Nanchang, 330013, Jiangxi, China

    Yuansheng Luo

Authors
  1. Xuejian Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhibin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gensheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhipeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuansheng Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaofang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors have made equal contributions.

Corresponding author

Correspondence to Xuejian Huang.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, X., Wu, Z., Wang, G. et al. ResGAT: an improved graph neural network based on multi-head attention mechanism and residual network for paper classification. Scientometrics 129, 1015–1036 (2024). https://doi.org/10.1007/s11192-023-04898-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-023-04898-w

Keywords

Search

Navigation