Skip to main content
Log in

What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The purpose of this paper is to update the review of Bornmann and Daniel (J Doc 64(1):45–80, 2008) presenting a narrative review of studies on citations in scientific documents. The current review covers 41 studies published between 2006 and 2018. Bornmann and Daniel (2008) focused on earlier years. The current review describes the (new) studies on citation content and context analyses as well as the studies that explore the citation motivation of scholars through surveys or interviews. One focus in this paper is on the technical developments in the last decade, such as the richer meta-data available and machine-readable formats of scientific papers. These developments have resulted in citation context analyses of large datasets in comprehensive studies (which was not possible previously). Many studies in recent years have used computational and machine learning techniques to determine citation functions and polarities, some of which have attempted to overcome the methodological weaknesses of previous studies. The automated recognition of citation functions seems to have the potential to greatly enhance citation indices and information retrieval capabilities. Our review of the empirical studies demonstrates that a paper may be cited for very different scientific and non-scientific reasons. This result accords with the finding by Bornmann and Daniel (2008). The current review also shows that to better understand the relationship between citing and cited documents, a variety of features should be analyzed, primarily the citation context, the semantics and linguistic patterns in citations, citation locations within the citing document, and citation polarity (negative, neutral, positive).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Source: Tahamtan and Bornmann (2018a, p. 205)

Fig. 2
Fig. 3

Source: adopted and adapted from Hedström and Ylikoski (2010, p. 59)

Similar content being viewed by others

Notes

  1. The figure also includes citer motivation surveys or interviews, which are explained in “Citer motivation surveys or interviews” section.

References

  • Abdullatif, M., Koh, Y. S., Dobbie, G., & Alam, S. (2013). Verb selection using semantic role labeling for citation classification. Paper presented at the 2013 Workshop on Computational Scientometrics: Theory & Applications. https://doi.org/10.1145/2508497.2508502.

  • Abu-Jbara, A., Ezra, J., & Radev, D. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. Paper presented at the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 596–606). Atlanta, Georgia.

  • Aljaber, B., Martinez, D., Stokes, N., & Bailey, J. (2011). Improving MeSH classification of biomedical articles using citation contexts. Journal of Biomedical Informatics,44(5), 881–896.

    Article  Google Scholar 

  • Aljaber, B., Stokes, N., Bailey, J., & Pei, J. (2010). Document clustering of scientific texts using citation contexts. Information Retrieval,13(2), 101–131. https://doi.org/10.1007/s10791-009-9108-x.

    Article  Google Scholar 

  • Anderson, M. H. (2006). How can we know what we think until we see what we said? A citation and citation context analysis of Karl Weick’s the social psychology of organizing. Organization Studies,27(11), 1675–1692.

    Article  Google Scholar 

  • Anderson, M. H., & Sun, P. Y. T. (2010). What have scholars retrieved from Walsh and Ungson (1991)? A citation context study. Management Learning,41(2), 131–145. https://doi.org/10.1177/1350507609341091.

    Article  Google Scholar 

  • Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research,14(3), 396–402.

    Article  Google Scholar 

  • Åström, F. (2014). The context of paratext: A bibliometric study of the citation contexts of Gérard Genette’s Texts. In N. Desrochers, & D. Apollon (Eds.), Examining paratextual theory and its applications in digital culture (pp. 1–23). Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-4666-6002-1.ch001.

    Chapter  Google Scholar 

  • Bakhti, K., Niu, Z., & Nyamawe, A. S. (2018a). A new scheme for citation classification based on convolutional neural networks. Paper presented at the International Conference on Software Engineering and Knowledge Engineering. https://doi.org/10.18293/SEKE2018-14.

  • Bakhti, K., Niu, Z., Yousif, A., & Nyamawe, A. S. (2018b). Citation function classification based on ontologies and convolutional neural networks. In L. Uden, D. Liberona, & J. Ristvej (Eds.), Paper presented at the International Workshop on Learning Technology for Education in Cloud, Communications in Computer and Information Science (pp. 105–115). Cham, Switzerland: Springer.

  • Bennett, B. C., & Prance, G. T. (2000). Introduced plants in the indigenous pharmacopoeia of Northern South America. Economic Botany, 54(1), 90–102.

    Article  Google Scholar 

  • Bertin, M., & Atanassova, I. (2014). A study of lexical distribution in citation contexts through the IMRaD standard. PLOS Neglected Tropical Diseases, 1(200,920), 83–402.

  • Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (2016a). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology,67(1), 164–177. https://doi.org/10.1002/asi.23367.

    Article  Google Scholar 

  • Bertin, M., Atanassova, I., Sugimoto, C. R., & Lariviere, V. (2016b). The linguistic patterns and rhetorical structure of citation context: an approach using n-grams. Scientometrics,109(3), 1417–1434. https://doi.org/10.1007/s11192-016-2134-8.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation,64(1), 45–80.

    Article  Google Scholar 

  • Bornmann, L., Haunschild, R., & Hug, S. E. (2018). Visualizing the context of citations referencing papers published by Eugene Garfield: A new type of keyword co-occurrence analysis. Scientometrics,114(2), 427–437. https://doi.org/10.1007/s11192-017-2591-8.

    Article  Google Scholar 

  • Boyack, K. W., van Eck, N. J., Colavizza, G., & Waltman, L. (2018). Characterizing in-text citations in scientific articles: A large-scale analysis. Journal of Informetrics,12(1), 59–73.

    Article  Google Scholar 

  • Camacho-Miñano, Md M, & Núñez-Nickel, M. (2009). The multilayered nature of reference selection. Journal of the American Society for Information Science and Technology,60(4), 754–777.

    Article  Google Scholar 

  • Chang, Y.-W. (2013). A comparison of citation contexts between natural sciences and social sciences and humanities. Scientometrics,96(2), 535–553. https://doi.org/10.1007/s11192-013-0956-1.

    Article  Google Scholar 

  • Clarke, M. E., & Oppenheim, C. (2006). Citation behaviour of information science students II: Postgraduate students. Education for Information,24(1), 1–30. https://doi.org/10.3233/EFI-2006-24101.

    Article  Google Scholar 

  • Coleman, J. S. (1990). Foundations of social theory. Cambridge, MA: Belknap Press of Harvard University Press.

    Google Scholar 

  • Cristea, I. A., & Naudet, F. (2018). Defending psychiatry or defending the trivial effects of therapeutic interventions? A citation content analysis of an influential paper. Epidemiology and Psychiatric Sciences,27(3), 230–239. https://doi.org/10.1017/S2045796017000750.

    Article  Google Scholar 

  • Dabrowska, A., & Larsen, B. (2015). Exploiting citation contexts for physics retrieval. Paper presented at the proceedings of the Second Workshop on Bibliometric-enhanced Information Retrieval: co-located with the 37th European Conference on Information Retrieval (pp. 14–21). Vienna, Austria.

  • Danell, J.-A. B. (2012). Representation and negotiation of complementary and alternative medicine: A citation context analysis. Science Communication,34(3), 299–333.

    Article  Google Scholar 

  • Di Marco, C., Kroon, F. W., & Mercer, R. E. (2006). Using hedges to classify citations in scientific articles. In J. G. Shanahan, Y. Qu, & J. Wiebe (Eds.), Computing attitude and affect in text: theory and applications (pp. 247–263). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Dong, C., & Schäfer, U. (2011). Ensemble-style self-training on citation classification. Paper presented at 5th International Joint Conference on Natural Language Processing (pp. 623–631). Chiang Mai, Thailand.

  • Doslu, M., & Bingol, H. O. (2016). Context sensitive article ranking with citation context analysis. Scientometrics,108(2), 653–671. https://doi.org/10.1007/s11192-016-1982-6.

    Article  Google Scholar 

  • Erikson, M. G., & Erlandson, P. (2014). A taxonomy of motives to cite. Social Studies of Science,44(4), 625–637. https://doi.org/10.1177/0306312714522871.

    Article  Google Scholar 

  • Galgani, F., Compton, P., & Hoffmann, A. (2015). LEXA: Building knowledge bases for automatic legal citation classification. Expert Systems with Applications,42(17–18), 6391–6407. https://doi.org/10.1016/j.eswa.2015.04.022.

    Article  Google Scholar 

  • González-Teruel, A., & Abad-García, F. (2018). The influence of Elfreda Chatman’s theories: a citation context analysis. Scientometrics,117(3), 1793–1819. https://doi.org/10.1007/s11192-018-2915-3.

    Article  Google Scholar 

  • Halevi, G., & Moed, H. F. (2013). The thematic and conceptual flow of disciplinary research: A citation context analysis of the journal of informetrics, 2007. Journal of the American Society for Information Science and Technology,64(9), 1903–1913. https://doi.org/10.1002/asi.22897.

    Article  Google Scholar 

  • Hammarfelt, B. (2011). Citation analysis on the micro level: The example of Walter Benjamin’s Illuminations. Journal of the American Society for Information Science and Technology,62(5), 819–830. https://doi.org/10.1002/asi.21504.

    Article  Google Scholar 

  • Harwood, N. (2008a). Citers’ use of citees’ names: Findings from a qualitative interview-based study. Journal of the American Society for Information Science and Technology,59(6), 1007–1011. https://doi.org/10.1002/asi.20789.

    Article  Google Scholar 

  • Harwood, N. (2008b). Publication outlets and their effect on academic writers’ citations. Scientometrics,77(2), 253.

    Article  Google Scholar 

  • Harwood, N. (2009). An interview-based study of the functions of citations in academic writing across two disciplines. Journal of Pragmatics,41(3), 497–518. https://doi.org/10.1016/j.pragma.2008.06.001.

    Article  Google Scholar 

  • Hassan, S.-U., Safder, I., Akram, A., & Kamiran, F. (2018). A novel machine-learning approach to measuring scientific knowledge flows using citation context analysis. Scientometrics,116(2), 973–996. https://doi.org/10.1007/s11192-018-2767-x.

    Article  Google Scholar 

  • Hedström, P. (2006). Explaining social change: An analytical approach. Papers: revista de sociologia, 80, 73–95.

    Article  Google Scholar 

  • Hedström, P., & Ylikoski, P. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology,36, 49–67.

    Article  Google Scholar 

  • Hernández-Alvarez, M., & Gomez, J. M. (2015). Survey about citation context analysis: Tasks, techniques, and resources. Natural Language Engineering,22(3), 327–349. https://doi.org/10.1017/s1351324915000388.

    Article  Google Scholar 

  • Hernández-Alvarez, M., Gomez Soriano, J. M., & Martínez-Barco, P. (2017). Citation function, polarity and influence classification. Natural Language Engineering,23(4), 561–588. https://doi.org/10.1017/s1351324916000346.

    Article  Google Scholar 

  • Hu, Z., Chen, C., & Liu, Z. (2015). The recurrence of citations within a scientific article. In A. A. Salah, Y. Tonta, A. A. A. Salah, C. Sugimoto, & U. Al (Eds.), The 15th conference of the International Society for Scientometrics and Informetrics (pp. 221–229). Istanbul: ISSI, Boaziçi University Printhouse.

    Google Scholar 

  • Jha, R., Jbara, A.-A., Qazvinian, V., & Radev, D. R. (2017). NLP-driven citation analysis for scientometrics. Natural Language Engineering,23(1), 93–130.

    Article  Google Scholar 

  • Judge, T., Cable, D., Colbert, A., & Rynes, S. (2007). What causes a management article to be cited—article, author, or journal? The Academy of Management Journal (AMJ),50(3), 491–506.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2018). Can Microsoft Academic help to assess the citation impact of academic books? Journal of Informetrics,12(3), 972–984.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M., & Abdoli, M. (2018). Can Microsoft Academic assess the early citation impact of in-press articles? A multi-discipline exploratory analysis. Journal of Informetrics,12(1), 287–298. https://doi.org/10.1016/j.joi.2018.01.009.

    Article  Google Scholar 

  • Lakshmanan, L., & Ramanathan, L. (2019). Using citation context to improve the retrieval of research article from cancer research journals. Asian Pacific Journal of Cancer Prevention,20(3), 951–960.

    Article  Google Scholar 

  • Leucht, S., Hierl, S., Kissling, W., Dold, M., & Davis, J. M. (2012). Putting the efficacy of psychiatric and general medicine medication into perspective: Review of meta-analyses. The British Journal of Psychiatry,200(2), 97–106.

    Article  Google Scholar 

  • Li, X., He, Y., Meyers, A., & Grishman, R. (2013). Towards fine-grained citation function classification. In R. Mitkov, G. Angelova, & K. Bontcheva (Eds.), Paper presented at the recent advances in natural language processing, Hissar, Bulgaria (pp. 402–407). Bulgaria: INCOMA Ltd.

    Google Scholar 

  • Lin, C.-S. (2018). An analysis of citation functions in the humanities and social sciences research from the perspective of problematic citation analysis assumptions. Scientometrics,116(2), 797–813. https://doi.org/10.1007/s11192-018-2770-2.

    Article  Google Scholar 

  • Lin, C. S., Chen, Y. F., & Chang, C. Y. (2013). Citation functions in social sciences and humanities: Preliminary results from a citation context analysis of Taiwan’s history research journals. Proceedings of the American Society for Information Science and Technology,50(1), 1–5.

    Article  Google Scholar 

  • Liu, M. (1993). Progress in documentation the complexities of citation practice: a review of citation studies. Journal of Documentation,49(4), 370–408.

    Article  Google Scholar 

  • Liu, S., Chen, C., Ding, K., Wang, B., Xu, K., & Lin, Y. (2014). Literature retrieval based on citation context. Scientometrics,101(2), 1293–1307. https://doi.org/10.1007/s11192-014-1233-7.

    Article  Google Scholar 

  • Liu, S., Ding, K., Wang, B., Tang, D., & Qu, Z. (2015). The research of paper influence based on citation context—A case study of the Nobel prize winner’s paper. In A. A. Salah, Y. Tonta, A. A. A. Salah, C. Sugimoto, & U. Al (Eds.), The 15th conference of the international society for scientometrics and informetrics (pp. 241–248). Istanbul: ISSI, Boaziçi University Printhouse.

    Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1984). The negational reference: Or the art of dissembling. Social Studies of Science,14(1), 91–94.

    Article  Google Scholar 

  • McCain, K. W., & Salvucci, L. J. (2016). How influential is Brooks’ Law? A longitudinal citation context analysis of Frederick Brooks’ The Mythical Man-Month. Journal of Information Science,32(3), 277–295. https://doi.org/10.1177/0165551506064397.

    Article  Google Scholar 

  • McCain, K., & Turner, K. (1989). Citation context analysis and aging patterns of journal articles in molecular genetics. Scientometrics,17(1–2), 127–163.

    Article  Google Scholar 

  • Mercer, R. E., Di Marco, C., & Kroon, F. W. (2004). The frequency of hedging cues in citation contexts in scientific writing. Paper presented at the Canadian Society for Computational Studies of Intelligence. Berlin, Heidelberg: Springer.

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago, IL, USA: University of Chicago Press.

    Google Scholar 

  • Milard, B. (2014). The social circles behind scientific references: Relationships between citing and cited authors in chemistry publications. Journal of the Association for Information Science and Technology,65(12), 2459–2468. https://doi.org/10.1002/asi.23149.

    Article  Google Scholar 

  • Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science,5(1), 86–92.

    Article  Google Scholar 

  • Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Technology,41(1), 609–641.

    Article  Google Scholar 

  • Phillips, O., & Gentry, A. H. (1993). The useful plants of Tambopata, Peru: I. Statistical hypotheses tests with a new quantitative technique. Economic Botany, 47(1), 15–32.

    Article  Google Scholar 

  • Pride, D., & Knoth, P. (2017). Incidental or influential?A decade of using text-mining for citation function classification. Paper presented at the 16th International Society of Scientometrics and Informetrics Conference. Wuhan, China.

  • Ramos, M. A., Melo, J. G., & Albuquerque, U. P. (2012). Citation behavior in popular scientific papers: What is behind obscure citations? The case of ethnobotany. Scientometrics,92(3), 711–719. https://doi.org/10.1007/s11192-012-0662-4.

    Article  Google Scholar 

  • Ritchie, A., Robertson, S., & Teufel, S. (2008). Comparing citation contexts for information retrieval. Paper presented at the proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, California (pp. 213–222). New York: ACM.

  • Safer, M. A., & Tang, R. (2009). The psychology of referencing in psychology journal articles. Perspectives on Psychological Science,4(1), 51–53. https://doi.org/10.1111/j.1745-6924.2009.01104.x.

    Article  Google Scholar 

  • Sieweke, J. (2014). Pierre Bourdieu in management and organization studies—A citation context analysis and discussion of contributions. Scandinavian Journal of Management,30(4), 532–543. https://doi.org/10.1016/j.scaman.2014.04.004.

    Article  Google Scholar 

  • Siontis, G. C., Tatsioni, A., Katritsis, D. G., & Ioannidis, J. P. (2009). Persistent reservations against contradicted percutaneous coronary intervention indications: Citation content analysis. American Heart Journal,157(4), 695–701. https://doi.org/10.1016/j.ahj.2008.11.023.

    Article  Google Scholar 

  • Small, H. (2011). Interpreting maps of science using citation context sentiments: A preliminary investigation. Scientometrics,87(2), 373–388. https://doi.org/10.1007/s11192-011-0349-2.

    Article  Google Scholar 

  • Small, H. (2018). Characterizing highly cited method and non-method papers using citation contexts: The role of uncertainty. Journal of Informetrics,12(2), 461–480. https://doi.org/10.1016/j.joi.2018.03.007.

    Article  MathSciNet  Google Scholar 

  • Small, H., Tseng, H., & Patek, M. (2017). Discovering discoveries: Identifying biomedical discoveries using citation contexts. Journal of Informetrics,11(1), 46–62. https://doi.org/10.1016/j.joi.2016.11.001.

    Article  Google Scholar 

  • Sula, C. A., & Miller, M. (2014). Citations, contexts, and humanistic discourse: Toward automatic extraction and classification. Literary and Linguistic Computing,29(3), 452–464. https://doi.org/10.1093/llc/fqu019.

    Article  Google Scholar 

  • Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: a comprehensive review of the literature. Scientometrics,107(3), 1195–1225.

    Article  Google Scholar 

  • Tahamtan, I., & Bornmann, L. (2018a). Core elements in the process of citing publications: Conceptual overview of the literature. Journal of Informetrics,12(1), 203–216. https://doi.org/10.1016/j.joi.2018.01.002.

    Article  Google Scholar 

  • Tahamtan, I., & Bornmann, L. (2018b). Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references? Journal of Informetrics,12(3), 906–930.

    Article  Google Scholar 

  • Tang, R., & Safer, M. A. (2008). Author-rated importance of cited references in biology and psychology publications. Journal of Documentation,64(2), 246–272.

    Article  Google Scholar 

  • Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Paper presented at the EMNLP ’06 proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia (pp. 103–110). Stroudsburg, PA: Association for Computational Linguistics.

  • Thelwall, M. (2018a). Does Microsoft Academic find early citations? Scientometrics,114(1), 325–334.

    Article  Google Scholar 

  • Thelwall, M. (2018b). Microsoft Academic automatic document searches: Accuracy for journal articles and suitability for citation analysis. Journal of Informetrics,12(1), 1–9.

    Article  Google Scholar 

  • Thornley, C., Watkinson, A., Nicholas, D., Volentine, R., Jamali, H. R., Herman, E.,… Tenopir, C. (2015). The role of trust and authority in the citation behaviour of researchers. Information Research, 20(3).

  • Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful citations. Paper presented at the Twenty-Ninth AAAI Conference on Artificial Intelligence. Retrieved from https://www.aaai.org/ocs/index.php/WS/AAAIW15/paper/view/10185/10244.

  • Walsh, J. P., & Ungson, G. R. (1991). Organizational Memory. Academy of Management Review, 16(1), 57–91.

    Article  Google Scholar 

  • Wang, W., Villavicencio, P., & Watanabe, T. (2012). Analysis of reference relationships among research papers, based on citation context. International Journal on Artificial Intelligence Tools, 21(02). https://doi.org/10.1142/s0218213012400040.

    Article  Google Scholar 

  • Willett, P. (2013). Readers’ perceptions of authors’ citation behaviour. Journal of Documentation,69(1), 145–156. https://doi.org/10.1108/00220411311295360.

    Article  Google Scholar 

  • Wright, M., & Armstrong, J. S. (2008). The Ombudsman: Verification of citations: Fawlty Towers of knowledge? Interfaces,38(2), 125–132. https://doi.org/10.1287/inte.1070.0317.

    Article  Google Scholar 

  • Yousif A., Niu Z., & Nyamawe A. S. (2018). Citation classification using multitask convolutional neural network model. In W. Liu, F. Giunchiglia, & B. Yang. (Eds.), Paper presented at the Knowledge Science, Engineering and Management, Changchun, China (pp. 232–243). Cham, Switzerland: Springer.

  • Zahavi, A., & Zahavi, A. (1999). The handicap principle: A missing piece of Darwin’s puzzle. Oxford: Oxford University Press.

    MATH  Google Scholar 

  • Zavrsnik, J., Kokol, P., Del Torso, S., & Blazun Vosner, H. (2016). Citation context and impact of ‘sleeping beauties’ in paediatric research. Journal of International Medical Research,44(6), 1212–1221. https://doi.org/10.1177/0300060516672129.

    Article  Google Scholar 

  • Zhang, G., Ding, Y., & Milojević, S. (2013). Citation content analysis (CCA): A framework for syntactic and semantic analysis of citation content. Journal of the American Society for Information Science and Technology,64(7), 1490–1503. https://doi.org/10.1002/asi.22850.

    Article  Google Scholar 

  • Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology,66(2), 408–427.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Iman Tahamtan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tahamtan, I., Bornmann, L. What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics 121, 1635–1684 (2019). https://doi.org/10.1007/s11192-019-03243-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03243-4

Keywords

Navigation