Deep and narrow impact: introducing location filtered citation counting

  • Dangzhi ZhaoEmail author
  • Andreas Strotmann


The present study tests a citation counting method that filters out citations in the introductory and backgrounds sections and then weighs the remaining citations by their in-text frequency. The dataset used comprises articles on bibliometrics available in full text in PubMed Central. This method was inspired by findings from previous studies that in-text frequency indicates importance of citations and citations in Methodology, Results, Discussion, and Conclusions sections tend to be more important to a citing article. We found that this method makes a large difference in author ranking as suggested by a 0.4 correlation between ranking by this method and that by traditional citation counting. Generally, this method has ranked authors concerning biomedical issues higher and those focused on bibliometrics or science communication issues lower compared to traditional citation counting. This rank change pattern suggests that this method appears to have made essential citations stand out more, i.e., citations that studies concerning biomedicine are expected to draw on more heavily. This method has also ranked guidelines or theoretical or methodological frameworks for systematic reviews, meta-analyses, knowledge translation, and scoping studies much higher, indicating that Bibliometrics has been mostly employed in these types of studies in biomedical fields. Unfortunately, citation network analysis doesn’t seem to have been employed much as indicated by key authors representing science mapping being ranked much lower by this method although it has been shown to be informative for these types of studies.


Citation analysis Weighted citation analysis Location-filtered citation counting Research evaluation Bibliometrics Biomedical research fields 


  1. Al Jaber, A.O., & Elayyan, H.O. (2018). Toward quality assurance and excellence in higher education. Gistrup: River Publishers. p. 284.Google Scholar
  2. Bertin, M., & Atanassova, I. (2018). InTeReC: In-text reference corpus for applying natural language processing to bibliometrics. Retrieved April 18, 2019 from
  3. Bertin, M., Atanassova, I., Gingras, Y., & Lariviere, V. (2016). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology,67(1), 164–177.CrossRefGoogle Scholar
  4. Bertram, S. (1972). Citations counts. In A. Pitemick (Ed.), Proceedings of the fourth annual meeting of the American society for information science, Western Canada chapter (pp. 61–67). Vancouver: University of British Columbia.Google Scholar
  5. Bonzi, S. (1982). Characteristics of a literature as predictors of relatedness between cited and citing works. Journal of the American Society for Information Science,33, 208–216.CrossRefGoogle Scholar
  6. Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology,36, 3–72.Google Scholar
  7. Bornmann, L., & Daniel, H.-D. (2008). What co-citation counts measure? A review of studies on citing behavior. Journal of Documentation,64, 45–80.CrossRefGoogle Scholar
  8. Boyack, K. W., Small, H., & Klavans, R. (2013). Improving the accuracy of co-citation clustering using full text. Journal of the American Society for Information Science and Technology,64(9), 1759–1767.CrossRefGoogle Scholar
  9. Boyack, K. W., Van Eck, N. J., Colavizzac, G., & Waltman, L. (2018). Characterizing in-text citations in scientific articles: A large-scale analysis. Journal of Informetrics.,12(1), 59–73.CrossRefGoogle Scholar
  10. Brooks, T. A. (1985). Private acts and public objects: An investigation of citer motivations. Journal of the American Society for Information Science,36(4), 223–229.CrossRefGoogle Scholar
  11. Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science,37(1), 34–36.CrossRefGoogle Scholar
  12. Bu, Y., Waltman, L, and Huang, Y. (2018). A multidimensional perspective on the citation impact of scientific publications. Retrieved April 2018 from.Google Scholar
  13. Cano, V. (1989). Citation behavior—classification, utility, and location. Journal of the American Society for Information Science,40, 284–290.CrossRefGoogle Scholar
  14. Case, D. O., & Higgins, G. M. (2000). How can we investigate citation behavior? A study of reasons for citing literature in communication. Journal of the American Society for Information Science,51(7), 635–645.CrossRefGoogle Scholar
  15. Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science,5(4), 423–441.CrossRefGoogle Scholar
  16. Ding, Y., & Cronin, B. (2011). Popular and/or prestigious? Measures of scholarly esteem. Information Processing and Management,47, 80–96.CrossRefGoogle Scholar
  17. Ding, Y., Liu, X., Guo, C., & Cronin, B. (2013). The distribution of references across texts: Some implications for citation analysis. Journal of Informetrics,7(3), 583–592.CrossRefGoogle Scholar
  18. Doumont, J. (Ed.). (2010). English communication for scientists. Cambridge: NPG Education. Retrieved September 22, 2016, from
  19. Funk, R. J., & Owen-Smith, J. A. (2017). Dynamic network measure of technological change. Management Science,63, 791–817.CrossRefGoogle Scholar
  20. Garfield, E. (1979). Citation indexing—its theory and application in science, technology, and humanities. New York: Wiley.Google Scholar
  21. Hanney, S., Frame, I., Grant, J., Buxton, M., Young, T., & Lewison, G. (2005). Using categorizations of citations when assessing the outcomes of health research. Scientometrics,65, 357–379.CrossRefGoogle Scholar
  22. Herlach, G. (1978). Can retrieval of information from citation indexes be simplified? Multiple mention of a reference as a characteristic of the link between cited and citing article. Journal of the American Society for Information Science,29(6), 308–310.CrossRefGoogle Scholar
  23. Hou, W., Li, M., & Niu, D. (2011). Counting citations in texts rather than reference lists to improve the accuracy of assessing scientific contribution. BioEssays,33, 724–727.CrossRefGoogle Scholar
  24. Hsiao, T.-M., & Chen, K.-H. (2018). How authors cite references? A study of characteristics of in-text citations. ASIST Proceedings,55(1), 179–187.Google Scholar
  25. Hu, Z., Lin, G., Sun, T., & Hou, H. (2017). Understanding multiply mentioned references. Journal of Informetrics,11(4), 948–958.CrossRefGoogle Scholar
  26. Jeong, Y. K., Song, M., & Ding, Y. (2014). Content-based author co-citation analysis. Journal of Informetrics,8(1), 197–211.CrossRefGoogle Scholar
  27. Liu, M. (1993). The complexities of citation practice: A review of citation studies. Journal of Documentation,49, 370–408.CrossRefGoogle Scholar
  28. McCain, K. W., & Turner, K. (1989). Citation context analysis and aging patterns of journal articles in Molecular-Genetics. Scientometrics,17, 127–163.CrossRefGoogle Scholar
  29. Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science,5(1), 86–92.CrossRefGoogle Scholar
  30. Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Washington, DC: Computer Horizons.Google Scholar
  31. Otto, W., Ghavimi, B., Mayr, P., Piryani, R. & Singh, V.K. (2019). Highly cited references in PLOS ONE and their in-text usage over time. Retrieved April 18, 2019 from Scholar
  32. Pak, C., Yu, G., & Wang, W. (2018). A study on the citation situation within the citing paper: Citation distribution of references according to mention frequency. Scientometrics,114(3), 905–918. Scholar
  33. Shadish, W. R., Tolliver, D., Gray, M., & Gupta, S. K. S. (1995). Author judgements about works they cite: Three studies from psychology journals. Social Studies of Science,25(3), 477–498.CrossRefGoogle Scholar
  34. Small, H. (1982). Citation context analysis. In B. J. Dervin & M. J. Voigt (Eds.), Progress in communication sciences, 3 (pp. 287–310). Norwood: Ablex.Google Scholar
  35. Strotmann, A., & Zhao, D. (2015). An 80/20 data quality law for professional scientometrics? In Proceedings of the 15th International Society for Scientometrics and Informetrics Conference, June 30–July 3, 2015, Istanbul, Turkey.Google Scholar
  36. Tabatabaei, N. (2013). Contribution of information science to other disciplines as reflected in citation contexts of highly cited JASIST papers. Montreal. (McGill University P.hD. dissertation).Google Scholar
  37. Tang, R., & Safer, M. A. (2008). Author-rated importance of cited references in biology and psychology publications. Journal of Documentation,64, 246–272.CrossRefGoogle Scholar
  38. Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (pp. 103–110). Stroudsburg.Google Scholar
  39. Thelwall, M. (2019a). Should citations be counted separately from each originating section? Journal of Informetrics,13, 658–678.CrossRefGoogle Scholar
  40. Thelwall, M. (2019b). The rhetorical structure of science? A multidisciplinary analysis of article headings. Journal of Informetrics.,13(2), 555–563.CrossRefGoogle Scholar
  41. Vinkler, P. (1987). A quasi-quantitative citation model. Scientometrics,12(1), 47–72.CrossRefGoogle Scholar
  42. Voos, H., & Dagaev, K. S. (1976). Are all citations equal? Or Did we op. cit. your idem? Journal of Academic Librarianship,1, 20–21.Google Scholar
  43. White, H. D. (1990). Author co-citation analysis: Overview and defense. In C. L. Borgman (Ed.), Scholarly communication and bibliometrics (Vol. 84, p. 106). Newbury Park: Sage.Google Scholar
  44. White, H. D., & McCain, K. W. (1998). Visualizing a discipline: An author co-citation analysis of information science, 1972–1995. Journal of the American Society for Information Science,49, 327–355.Google Scholar
  45. White, H. D., & Wang, P. L. (1997). A qualitative study of citing behavior: contributions, criteria, and meta-level documentation concerns. Library Quarterly,67, 122–154.CrossRefGoogle Scholar
  46. Wu, L. F., Wang, D. S., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature,566, 378–382.CrossRefGoogle Scholar
  47. Zhao, D., Cappello, A., & Johnston, L. (2017). Functions of uni-and multi-citations: Implications for weighted citation analysis. Journal of Data and Information Science,2(1), 51–69.CrossRefGoogle Scholar
  48. Zhao, D., & Strotmann, A. (2008a). Information Science during the first decade of the Web: An enriched author co-citation analysis. Journal of the American Society for Information Science and Technology,59(6), 916–937.CrossRefGoogle Scholar
  49. Zhao, D., & Strotmann, A. (2008b). Evolution of research activities and intellectual influences in Information Science 1996–2005: Introducing author bibliographic coupling analysis. Journal of the American Society for Information Science and Technology,59(13), 2070–2086.CrossRefGoogle Scholar
  50. Zhao, D., & Strotmann, A. (2011). Intellectual structure of Stem Cell research: A comprehensive author co-citation analysis of a highly collaborative and multidisciplinary field. Scientometrics,87(1), 115–131.CrossRefGoogle Scholar
  51. Zhao, D., & Strotmann, A. (2014). The knowledge base and research front of Information science 2006–2010: An author co-citation and bibliographic coupling analysis. Journal of the Association for Information Science and Technology,65(5), 996–1006.Google Scholar
  52. Zhao, D., & Strotmann, A. (2015a). Analysis and visualization of citation networks. In Gary Marchionini (Ed.), Synthesis lectures on information concepts, retrieval, and services. San Rafael: Morgan & Claypool Publishers. Scholar
  53. Zhao, D. & Strotmann, A. (2015b). Re-citation analysis: Promising for research evaluation, knowledge network analysis, knowledge representation and information retrieval? In Proceedings of the 15th International Society for Scientometrics and Informetrics Conference, June 30–July 3, 2015, Istanbul, Turkey.Google Scholar
  54. Zhao, D., & Strotmann, A. (2016). Dimensions and uncertainties of author citation rankings: Lessons learned from frequency-weighted in-text citation counting. Journal of the Association for Information Science and Technology,67(3), 628–671.CrossRefGoogle Scholar
  55. Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology,66(2), 408–427.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.University of AlbertaEdmontonCanada
  2. 2.ScienceXploreBad SchandauGermany

Personalised recommendations