Enhancing Opinion Extraction by Automatically Annotated Lexical Resources

(Extended Version)
  • Andrea Esuli
  • Fabrizio Sebastiani
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6562)


In this paper we tackle an opinion extraction (OE) task, i.e., identifying in a text each expression of subjectivity, the subject expressing it, and its possible target. We especially focus on how lexical resources specifically developed for opinion mining could be used to improve the performance of an opinion extraction system. We report results, complete with statistical significance tests and inter-annotator agreement data, on two manually annotated corpora, one of English and one of Italian texts. We evaluate our results using standard evaluation measures and also using a new evaluation measure we have recently proposed.


Opinion Mining Question Answering Opinion Extraction Language Resource Speech Event 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Breck, E., Choi, Y., Cardie, C.: Identifying expressions of opinion in context. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI 2007), Hyderabad, IN, pp. 2683–2688 (2007)Google Scholar
  2. 2.
    Choi, Y., Breck, E., Cardie, C.: Joint extraction of entities and relations for opinion recognition. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), Sydney, AU (2006)Google Scholar
  3. 3.
    Choi, Y., Cardie, C., Riloff, E., Patwardhan, S.: Identifying sources of opinions with conditional random fields and extraction patterns. In: Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, CA, pp. 355–362 (2005)Google Scholar
  4. 4.
    Kim, S.M., Hovy, E.: Identifying opinion holders for question answering in opinion texts. In: Proceedings of the AAAI 2005 Workshop on Question Answering in Restricted Domains, Pittsburgh, US (2005)Google Scholar
  5. 5.
    Kim, S.M., Hovy, E.: Extracting opinions, opinion holders, and topics expressed in online news media text. In: Proceedings of ACL/COLING 2006 Workshop on Sentiment and Subjectivity in Text, Sidney, AUS (2006)Google Scholar
  6. 6.
    Wiebe, J., Breck, E., Buckley, C., Cardie, C., Davis, P., Fraser, B., Litman, D., Pierce, D., Riloff, E., Wilson, T., Day, D., Maybury, M.: Recognizing and organizing opinions expressed in the world press. In: Proceedings of the 2003 AAAI Spring Symposium on New Directions in Question Answering, Stanford, US, pp. 12–19 (2003)Google Scholar
  7. 7.
    Wiebe, J., Wilson, T., Cardie, C.: Annotating expressions of opinions and emotions in language. Language Resources and Evaluation 39(2/3), 165–210 (2005)CrossRefGoogle Scholar
  8. 8.
    Esuli, A., Sebastiani, F.: SentiWordNet: A publicly available lexical resource for opinion mining. In: Proceedings of the 5th Conference on Language Resources and Evaluation (LREC 2006), Genova, IT, pp. 417–422 (2006)Google Scholar
  9. 9.
    Esuli, A., Sebastiani, F., Urciuoli, I.: Annotating expressions of opinion and emotion in the Italian Content Annotation Bank. In: Proceedings of the 6th Conference on Language Resources and Evaluation (LREC 2008), Marrakech, MA (2008)Google Scholar
  10. 10.
    Esuli, A., Sebastiani, F.: Evaluating information extraction. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds.) CLEF 2010. LNCS, vol. 6360, pp. 100–111. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Riloff, E.: Automatically generating extraction patterns from untagged text. In: Proceedings of the 13th Conference of the American Association for Artificial Intelligence (AAAI 1996), Portland, US, pp. 1044–1049 (1996)Google Scholar
  12. 12.
    Kudo, T., Matsumoto, Y.: Fast methods for kernel-based text analysis. In: Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL 2003), Sapporo, JP, pp. 24–31 (2003)Google Scholar
  13. 13.
    Stone, P.J., Dunphy, D.C., Smith, M.S., Ogilvie, D.M.: The General Inquirer: A Computer Approach to Content Analysis. The MIT Press, Cambridge (1966)Google Scholar
  14. 14.
    Hatzivassiloglou, V., McKeown, K.R.: Predicting the semantic orientation of adjectives. In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL 1997), Madrid, ES, pp. 174–181 (1997)Google Scholar
  15. 15.
    Esuli, A., Sebastiani, F.: Random-walk models of term semantics: An application to opinion-related properties. In: Proceedings of the 3rd Language Technology Conference (LTC 2007), Poznań, PL, pp. 221–225 (2007)Google Scholar
  16. 16.
    Magnini, B., Pianta, E., Girardi, C., Negri, M., Romano, L., Speranza, M., Bartalesi-Lenzi, V., Sprugnoli, R.: I-CAB: The Italian content annotation bank. In: Proceedings of the 5th Conference on Language Resources and Evaluation (LREC 2006), Genova, IT, pp. 963–968 (2006)Google Scholar
  17. 17.
    Freitag, D.: Using grammatical inference to improve precision in information extraction. In: Proceedings of the ICML 1997 Workshop on Grammatical Inference, Automata Induction, and Language Acquisition, Nashville, TN (1997)Google Scholar
  18. 18.
    Lavelli, A., Califf, M.E., Ciravegna, F., Freitag, D., Giuliano, C., Kushmerick, N., Romano, L., Ireson, N.: Evaluation of machine learning-based information extraction algorithms: Criticisms and recommendations. Language Resources and Evaluation 42(4), 361–393 (2008)CrossRefGoogle Scholar
  19. 19.
    Pianta, E., Bentivogli, L., Girardi, C.: MultiWordNet: Developing an aligned multilingual database. In: Proceedings of the 1st International Conference on Global WordNet (GWN 2002), Mysore, IN (2002)Google Scholar
  20. 20.
    Yang, Y., Liu, X.: A re-examination of text categorization methods. In: Proceedings of the 22nd ACM International Conference on Research and Development in Information Retrieval (SIGIR 1999), Berkeley, US, pp. 42–49 (1999)Google Scholar
  21. 21.
    Spiegel, M.R., Stephens, L.J.: Statistics, 3rd edn. McGraw-Hill, New York (1999)Google Scholar
  22. 22.
    Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46 (1960)CrossRefGoogle Scholar
  23. 23.
    Eugenio, B.D., Glass, M.: The kappa statistic: A second look. Computational Linguistics 30(1), 95–101 (2004)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Andrea Esuli
    • 1
  • Fabrizio Sebastiani
    • 1
  1. 1.Istituto di Scienza e Tecnologia dell’InformazioneConsiglio Nazionale delle RicerchePisaItaly

Personalised recommendations