Evaluating Citation Functions in CiTO: Cognitive Issues

  • Paolo Ciancarini
  • Angelo Di Iorio
  • Andrea Giovanni Nuzzolese
  • Silvio Peroni
  • Fabio Vitali
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8465)


Networks of citations are a key tool for referencing, disseminating and evaluating research results. The task of characterising the functional role of citations in scientific literature is very difficult, not only for software agents but for humans, too. The main problem is that the mental models of different annotators hardly ever converge to a single shared opinion. The goal of this paper is to investigate how an existing reference model for classifying citations, namely CiTO (Citation Typing Ontology), is interpreted and used by annotators of scientific literature. We present an experiment capturing the cognitive processes behind subjects’ decisions in annotating papers with CiTO, and we provide initial ideas to refine future releases of CiTO.


#eswc2014Ciancarini CiTO citation functions human inter-rater agreement mental models usability user testing session 


  1. 1.
    Athar, A.: Sentiment Analysis of Citations using Sentence Structure-Based Features. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 81–87 (2011)Google Scholar
  2. 2.
    Athar, A., Teufel, S.: Context-Enhanced Citation Sentiment Detection. In: Proceedings of Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics 2012, pp. 597–601 (2012)Google Scholar
  3. 3.
    Athar, A., Teufel, S.: Detection of implicit citations for sentiment detection. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pp. 18–26 (2012)Google Scholar
  4. 4.
    Biemann, C.: Chinese whispers: an efficient graph clustering algorithm and its application to natural language processing problems. In: Proceedings of the 1st Workshop on Graph Based Methods for Natural Language Processing (TextGraph 2006), pp. 73–80 (2006)Google Scholar
  5. 5.
    Brooke, J.: SUS: a “quick and dirty” usability scale. In: Usability Evaluation in Industry, pp. 189–194 (1996)Google Scholar
  6. 6.
    Ciancarini, P., Di Iorio, A., Nuzzolese, A.G., Peroni, S., Vitali, F.: Characterising citations in scholarly articles: an experiment. In: Proceedings of 1st International Workshop on Artificial Intelligence and Cognition (AIC 2013), pp. 124–129 (2013), (last visited March 10, 2014) (retrieved)
  7. 7.
    Copestake, A., Corbett, P., Murray-Rust, P., Rupp, C.J., Siddharthan, A., Teufel, S., Waldron, B.: An architecture for language processing for scientific text. In: Proceedings of the UK e-Science All Hands Meeting 2006 (2006)Google Scholar
  8. 8.
    Craik, K.J.W.: The nature of explanation. Cambridge University Press (1967) ISBN: 0521094453Google Scholar
  9. 9.
    Davidson, M.J., Dove, L., Weltz, J.: Mental models and usability. Depaul University, Chicago (1999), (last visited March 10, 2014) (retrieved)
  10. 10.
    Di Iorio, A., Nuzzolese, A.G., Peroni, S.: Towards the automatic identification of the nature of citations. In: Proceedings of 3rd Workshop on Semantic Publishing (SePublica 2013), pp. 63–74 (2013), (last visited March 10, 2014) (retrieved)
  11. 11.
    Gil, Y., Ratnakar, V., Hanson, P.C.: Organic data publishing: a novel approach to scientific data sharing. In: Proceedings of the 2nd International Workshop on Linked Science, LISC 2012 (2012), (last visited March 10, 2014)
  12. 12.
    Hornsey, M.J., Robson, E., Smith, J., Esposo, S., Sutton, R.M.: Sugaring the Pill: Assessing Rhetorical Strategies Designed to Minimize Defensive Reactions to Group Criticism: Strategies to Soften Criticism. Human Communication Research 34(1), 70–98 (2008), doi:10.1111/j.1468-2958.2007.00314.xCrossRefGoogle Scholar
  13. 13.
    Hou, W., Li, M., Niu, D.: Counting citations in texts rather than reference lists to improve the accuracy of assessing scientific contribution. BioEssays 33(10), 724–727 (2011), doi:10.1002/bies.201100067CrossRefGoogle Scholar
  14. 14.
    Jorg, B.: Towards the Nature of Citations. In: Poster Proceedings of the 5th International Conference on Formal Ontology in Information Systems (2008)Google Scholar
  15. 15.
    Kauppinen, T., Baglatzi, A., Keuler, C.: Linked Science: interconnecting scientific assets. In: Data Intensive Science, CRC Press (2013)Google Scholar
  16. 16.
    Lewis, J.R., Sauro, J.: The Factor Structure of the System Usability Scale. In: Kurosu, M. (ed.) HCD 2009. LNCS, vol. 5619, pp. 94–103. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    MacRoberts, M.H., MacRoberts, B.R.: The Negational Reference: Or the Art of Dissembling. Social Studies of Science 14(1), 91–94 (1984)CrossRefGoogle Scholar
  18. 18.
    Manning, C.D., Schütze, H.: Foundations of statistical natural language processing. MIT Press (1999) ISBN: 0262133601Google Scholar
  19. 19.
    Moravcsik, M.J., Murugesan, P.: Some Results on the Function and Quality of Citations. Social Studies of Science 5(1), 86–92 (1975)CrossRefGoogle Scholar
  20. 20.
    Norman, D.A.: The psychology of everyday things. Basic Books (1988) ISBN: 0465067093Google Scholar
  21. 21.
    Osborne, F., Motta, E.: Mining Semantic Relations between Research Areas. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 410–426. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  22. 22.
    Peroni, S., Shotton, D.: FaBiO and CiTO: ontologies for describing bibliographic resources and citations. Journal of Web Semantics 17, 33–43 (2012), doi:10.1016/j.websem.2012.08.001CrossRefGoogle Scholar
  23. 23.
    Qazvinian, V., Radev, D.R.: Identifying non-explicit citing sentences for citation-based summarization. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 555–564 (2010)Google Scholar
  24. 24.
    Sauro, J.: A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. CreateSpace (2011) ISBN: 1461062707Google Scholar
  25. 25.
    Shotton, D.: Semantic publishing: the coming revolution in scientific journal publishing. Learned Publishing 22(2), 85–94 (2009), doi:10.1087/2009202CrossRefGoogle Scholar
  26. 26.
    Shotton, D.: Publishing: Open citations. Nature 502(7471), 295–297 (2013), doi:10.1038/502295aCrossRefGoogle Scholar
  27. 27.
    Strauss, A., Corbin, J.: Basics of Qualitative Research Techniques and Procedures for Developing Grounded Theory, 2nd edn. Sage Publications (1998) ISBN: 0803959408Google Scholar
  28. 28.
    Teufel, S., Carletta, J., Moens, M.: An annotation scheme for discourse-level argumentation in research articles. In: Proceedings of the 9th Conference of the European Chapter of the Association for Computational Linguistics, pp. 110–117 (1999)Google Scholar
  29. 29.
    Teufel, S., Siddharthan, A., Tidhar, D.: Automatic classification of citation function. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 103–110 (2006)Google Scholar
  30. 30.
    Teufel, S., Siddharthan, A., Tidhar, D.: An annotation scheme for citation function. In: Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pp. 80–87 (2009)Google Scholar
  31. 31.
    Zhu, X., Turney, P., Lemire, D., Vellino, A.: Measuring academic influence: Not all citations are equal. To appear in Journal of the American Society for Information Science and Technology (2014), Preprint available at (last visited March 10, 2014)

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Paolo Ciancarini
    • 1
    • 2
  • Angelo Di Iorio
    • 1
  • Andrea Giovanni Nuzzolese
    • 1
    • 2
  • Silvio Peroni
    • 1
    • 2
  • Fabio Vitali
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of BolognaItaly
  2. 2.STLab-ISTCConsiglio Nazionale delle RicercheItaly

Personalised recommendations