Skip to main content
SpringerLink
Log in

Does citation polarity help evaluate the quality of academic papers?

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Citation frequency is an important metric for the evaluation of academic papers, but it assumes that all citations are of equal value. The purpose of this study is to determine the validity of citation polarity, which contains evaluative information such as criticism or praise, in the evaluation of paper quality. In this paper, 3538 citation sentences in papers from ACL conferences were selected and manually annotated for citation polarity. They were divided into best paper group and matching paper group, and tested in heterologous pairs to determine whether there were differences in the positive and negative citations of the two groups, and to further investigate the trend of citation polarity with the increase of citation window. The results of the study showed that the best paper and the matching paper had significant differences in the number of positive and negative citations, and the mean and median values of positive citations in the best group were about 1.5 times higher than those in the matching group. As the citation window increased, the best papers maintained both positive and negative citation dominance over 5 years, and the peak citation in the best group was about three times higher than that in the matching group. Therefore, the metric of citation polarity can help evaluate the quality of papers and provide new ideas for scientific and objective evaluation of academic papers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Abujbara, A., Ezra, J., & Radev, D. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. Conference of the North American chapter of the Association for Computational Linguistics: Human language technologies. Atlanta, Georgia, USA: Association for Computational Linguistics (pp. 596–606).

  • Aljuaid, H., Iftikhar, R., Ahmad, S., et al. (2020). Important citation identification using sentiment analysis of in-text citations. Telematics and Informatics, 56(2), 1–18.

    Google Scholar 

  • Athar, A., & Teufel, S. (2012). Context-enhanced citation sentiment detection. Conference of the North American Chapter of the Association for Computational Linguistics, Montrèal, Canada: Association for Computational Linguistics (pp. 597–601).

  • Bordignon, F. (2020). Self-correction of science: A comparative study of negative citations and post-publication peer review. Scientometrics, 124(2), 1225–1239.

    Article  Google Scholar 

  • Bordignon, F. (2022). Critical citations in knowledge construction and citation analysis: From paradox to definition. Scientometrics, 127(1), 959–972.

    Article  Google Scholar 

  • Bornmann, L. (2015). Inter-rater reliability and convergent validity of F1000Prime peer review. Journal of the Association for Information Science and Technology, 66(12), 2415–2426.

    Article  Google Scholar 

  • Bornmann, L. (2017). Is collaboration among scientists related to the citation impact of papers because their quality increases with collaboration? An analysis based on data from F1000Prime and normalized citation scores. Journal of the Association for Information Science and Technology, 68(4), 1036–1047.

    Article  Google Scholar 

  • Brembs, B. (2018). Prestigious science journals struggle to reach even average reliability. Frontiers in Human Neuroscience, 12(2), 1–2.

    Google Scholar 

  • Bridgstock, M. (1991). The quality of single and multiple authored papers: An unresolved problem. Scientometrics, 21(1), 37–48.

    Article  Google Scholar 

  • Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1), 34–36.

    Article  Google Scholar 

  • Case, D. O., & Higgins, G. M. (2000). How can we investigate citation behavior? A study of reasons for citing literature in communication. Journal of the American Society for Information Science, 51(7), 635–645.

    Article  Google Scholar 

  • Catalini, C., Lacetera, N., & Oettl, A. (2015). The incidence and role of negative citations in science. PNAS, 112(45), 13823–13826.

    Article  Google Scholar 

  • Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting?[J]. Social Studies of Science, 5(4), 423–441.

    Article  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

    Article  Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

    MATH  Google Scholar 

  • Cronin, B., & Meho, L. (2006). Using the h-Index to rank influential information scientists. Journal of the American Society for Information Science and Technology, 57(9), 1275–1278.

    Article  Google Scholar 

  • Dunaiski, M., Visser, W., & Geldenhuys, J. (2016). Evaluating paper and author ranking algorithms using impact and contribution awards. Journal of Informetrics, 10(2), 392–407.

    Article  Google Scholar 

  • Farys, R., & Wolbring, T. (2017). Matched control groups for modeling events in citation data: An illustration of nobel prize effects in citation networks. Journal of the American Society for Information Science and Technology, 68(9), 2201–2210.

    Google Scholar 

  • Garfield, E. (1964). Can citation indexing be automated? Proceedings of the Symposium on Statistical Association. Washington (pp. 84–90).

  • Geng, S., & Yang, J. (2018). A method to evaluate the academic influence of papers based on citation sentiment. Information Studies: Theory & Application, 41(12), 93–98.

    Google Scholar 

  • Geras, A., Siudem, G., & Gagolewski, M. (2020). Should we introduce a dislike button for academic articles? Journal of the Association for Information Science and Technology, 71(2), 221–229.

    Article  Google Scholar 

  • Ghosh, S., & Shah, C. (2020). Identifying citation sentiment and its influence while indexing scientific papers. Proceedings of the 53rd Hawaii International Conference on System Sciences, Hawaii, USA: HICSS2517–2526.

  • Hu, Y. (2015). The relationship between citations of academic papers and academic quality. Library Forum, 35(5), 56–59.

    Google Scholar 

  • Ikram, M. T., & Afzal, M. T. (2019). Aspect based citation sentiment analysis using linguistic patterns for better comprehension of scientific knowledge. Scientometrics, 119(1), 73–95.

    Article  Google Scholar 

  • Jiang, C., & Wei, Q. (2017). Construction of evaluation index system of representative paper in humanities and social sciences and its realization mechanism. Gansu Social Sciences, 44(2), 97–106.

    Google Scholar 

  • Jiang, L., & Zhang, Q. (2020). Research on academic evaluation based on fine-grain citation sentimental quantification. Data Analysis and Knowledge Discovery, 4(6), 129–138.

    MathSciNet  Google Scholar 

  • Kumar, S. (2016). Structure and dynamics of signed citation networks. International Conference Companion on World Wide Web, Montreal, Canada: ACM63–64.

  • Liang, G. Q., Hou, H. Y., Ren, P., Wang, Y. J., Huang, F., Wang, J. X., & Hu, Z. G. (2018). Analysis of correlation between usage count and times cited of high quality literatures. Journal of Intelligence, 37(4), 147–153.

    Google Scholar 

  • Liao, H., Xiao, R., Cimini, G., & Medo, M. (2014). Network-driven reputation in online scientific communities. PLoS ONE, 9(12), 1–18.

    Article  Google Scholar 

  • Liu, Y., Wu, Q., Wu, S., & Gao, Y. (2021). Weighted citation based on ranking-related contribution: A new index for evaluating article impact. Scientometrics, 126(10), 8653–8672.

    Article  Google Scholar 

  • Lu, Y., Luo, J., Xiao, Y., & Zhu, H. (2021). Text representation model of scientific papers based on fusing multi-viewpoint information and its quality evaluation. Scientometrics, 126(8), 6937–6963.

    Article  Google Scholar 

  • Macilwain, C. (2013). Halt the avalanche of performance metrics. Nature, 500(7462), 255.

    Article  Google Scholar 

  • Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61–90.

    Article  Google Scholar 

  • Min, C., Bu, Y., Wu, D., Ding, Y., & Zhang, Y. (2021). Identifying citation patterns of scientific breakthroughs: A perspective of dynamic citation process. Information Processing & Management, 58(1), 1–21.

    Article  Google Scholar 

  • Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science, 5(1), 86–92.

    Article  Google Scholar 

  • Munkhdalai, T., Lalor, J. P., & Yu, H. (2016). Citation analysis with neural attention models. International Workshop on Health Text Mining and Information Analysis, Austin, Texas, USA: Association for Computational Linguistics (pp. 69–77).

  • Muppidi, S., Gorripati, S. K., & Kishore, B. (2020). An approach for bibliographic citation sentiment analysis using deep learning. International Journal of Knowledge-Based and Intelligent Engineering Systems, 24(4), 353–362.

    Article  Google Scholar 

  • Newman, M. E. J. (2009). The first-mover advantage in scientific publication. Europhysics Letters, 86(6), 68001–68006.

    Article  Google Scholar 

  • Newman, M. E. J. (2014). Prediction of highly cited papers. Europhysics Letters, 105(2), 1–6.

    Article  Google Scholar 

  • Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021). scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies, 2(3), 882–898.

    Article  Google Scholar 

  • Niu, Q., Zhou, J., Zeng, A., Fan, Y., & Di, Z. (2016). Which publication is your representative work? Journal of Informetrics, 10(3), 842–853.

    Article  Google Scholar 

  • Salimi, N. (2017). Quality assessment of scientific outputs using the BWM. Scientometrics, 112(1), 195–213.

    Article  Google Scholar 

  • Shadish, W. R., Tolliver, D., Gray, M., & Sen Gupta, S. K. (1995). Author judgements about works they cite: Three studies from psychology journals. Social Studies of Science, 25(3), 477–498.

    Article  Google Scholar 

  • Song, D., Wang, W., Fan, Y., Xing, Y., & Zeng, A. (2022). Quantifying the structural and temporal characteristics of negative links in signed citation networks. Information Processing & Management, 59(4), 102996.

    Article  Google Scholar 

  • Suelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019). Evaluation of citations of the retracted article by Wakefield et al. with fraudulent claims of an association between vaccination and autism. JAMA Network Open, 2(11), 1–10.

    Article  Google Scholar 

  • Suo, C., & Gai, S. (2018). An analysis on the quality and influence of scientific paper based on citations. Information Studies: Theory & Application, 41(5), 11–15.

    Google Scholar 

  • Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. Conference on Empirical Methods in Natural Language Processing, Sydney, Australia: Association for Computational Linguistics (pp. 103–110).

  • Vinkler, P. (1987). Quasi-quantitative citation model. Scientometrics, 12(1–2), 47–72.

    Article  Google Scholar 

  • Wang, D., Song, C., & Barabási, A. (2013). Quantifying long-term scientific impact. Science, 342(6154), 127–132.

    Article  Google Scholar 

  • Wang, H., Tan, Z., & Chen, T. (2016). Research on the factors affecting papers’ citation frequency. Studies in Science of Science, 34(2), 171–177.

    Google Scholar 

  • Xie, Z., Ma, J. X., & Hu, W. (2021). Evaluation of academic representative works: Methods review and research prospects. Information Studies: Theory & Application, 44(12), 190–196.

    Google Scholar 

  • Xu, L., Ding, K., Chen, N., & Li, B. (2020). Corpus construction for citation sentiment in Chinese literature. Journal of the China Society for Scientific and Technical Information, 39(1), 25–37.

    Google Scholar 

  • Xu, L., Ding, K., & Lin, Y. (2022). Do negative citations reduce the impact of cited papers? Scientometrics, 127(2), 1161–1186.

    Article  Google Scholar 

  • Yan, E., Chen, Z., & Li, K. (2020). Authors’ status and the perceived quality of their work: Measuring citation sentiment change in Nobel articles. Journal of the Association for Information Science and Technology, 71(3), 314–324.

    Article  Google Scholar 

  • Yang, P., Sun, X., & Li, W. et al. (2018). Automatic academic paper rating based on modularized hierarchical convolutional neural network. Proceedings of the 56th Annual Meeting of ACL. Melbourne, Australia, Association for Computational Linguistics (pp. 496–502).

  • Yousif, A., Niu, Z., & Nyamawe, A. S. et al. (2018). Improving citation sentiment and purpose classification using hybrid deep neural network model. Proceedings of the International Conference on Advanced Intelligent Systems and Informatics, Cairo, Egypt: Springer (pp. 327–336).

  • Yousif, A., Niu, Z., Tarus, J. K., & Ahmad, A. (2019a). A survey on sentiment analysis of scientific citations. Artificial Intelligence Review, 52, 1805–1838.

    Article  Google Scholar 

  • Yousif, A., Niu, Z., Chambua, J., & Khan, Z. Y. (2019b). Multi-task learning model based on recurrent convolutional neural networks for citation sentiment and purpose classification. Neurocomputing, 335(3), 195–205.

    Article  Google Scholar 

  • Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, H. E. (2017). The science of science: From the perspective of complex systems. Physics Reports, 714(16), 1–73.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, C., & Li, Z. (2019). Extracting sentences of research originality from full text academic articles. Data Analysis and Knowledge Discovery, 3(10), 12–19.

    Google Scholar 

  • Zhang, P., Wang, P., & Wu, Q. (2018). How are the best JASIST papers cited? Journal of the American Society for Information Science and Technology, 69(6), 857–860.

    Google Scholar 

  • Zhou, J., Zeng, A., Fan, Y., & Di, Z. (2018). The representative works of scientists. Scientometrics, 117(3), 1721–1732.

    Article  Google Scholar 

  • Zhou, L., Amadi, U., & Zhang, D. (2020). Is self-citation biased? An investigation via the lens of citation polarity, density, and location. Information Systems Frontiers, 22(1), 77–90.

    Article  Google Scholar 

  • Zhou, Y., Lü, L., & Li, M. (2012). Quantifying the influence of scientists and their publications: Distinguishing between prestige and popularity. New Journal of Physics, 14(3), 1–18.

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by grant from the Natural Science Foundation of China (Nos. 61772103, 61806038, 61976036). We also thank the anonymous reviewers for their constructive comments and suggestions.

Author information

Authors and Affiliations

  1. WISE Lab, Institute of Science of Science and Technology Management, Dalian University of Technology, Dalian, 116024, China

    Linhong Xu, Kun Ding, Yuan Lin & Chunbo Zhang

  2. Software Institute, Dalian University of Foreign Languages, Dalian, 116044, China

    Linhong Xu

Authors
  1. Linhong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuan Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chunbo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Ding.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, L., Ding, K., Lin, Y. et al. Does citation polarity help evaluate the quality of academic papers?. Scientometrics 128, 4065–4087 (2023). https://doi.org/10.1007/s11192-023-04734-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-023-04734-1

Keywords

Search

Navigation