Language Resources and Evaluation

, Volume 39, Issue 2–3, pp 165–210 | Cite as

Annotating Expressions of Opinions and Emotions in Language

  • Janyce Wiebe
  • Theresa Wilson
  • Claire Cardie


This paper describes a corpus annotation project to study issues in the manual annotation of opinions, emotions, sentiments, speculations, evaluations and other private states in language. The resulting corpus annotation scheme is described, as well as examples of its use. In addition, the manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.


affect attitudes corpus annotation emotion natural language processing opinions sentiment subjectivity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson, C., McMaster, G. 1982Computer Assisted Modeling of Affective Tone in Written DocumentsComputers and the Humanities1619CrossRefGoogle Scholar
  2. Asher, N. 1986Belief in Discourse Representation TheoryJournal of Philosophical Logic15127189CrossRefGoogle Scholar
  3. Banfield, A. 1982Unspeakable SentencesRoutledge and Kegan PaulBostonGoogle Scholar
  4. Bergler S. (1992) Evidential Analysis of Reported Speech. Ph.D. thesis, Brandeis University.Google Scholar
  5. Biber, D., Finegan, E. 1989Styles of Stance in English: Lexical and Grammatical Marking of Evidentiality and AffectText993124Google Scholar
  6. Breck E., Cardie C. (2004) Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions. In Proceedings of the Twentieth International Conference on Computational Linguistics (COLING 2004). Geneva, Switzerland, pp. 120–126.Google Scholar
  7. Bruce, R., Wiebe, J. 1999Recognizing Subjectivity: A Case Study of Manual TaggingNatural Language Engineering5187205CrossRefGoogle Scholar
  8. Cardie C., Wiebe J., Wilson T., Litman D. (2003) Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering. In Working Notes – New Directions in Question Answering (AAAI Spring Symposium Series).Google Scholar
  9. Castañeda, H.-N. 1977On the Philosophical Foundations of the Theory of Communication: ReferenceMidwest Studies in Philosophy2165186Google Scholar
  10. Chatman, S. 1978Story and Discourse: Narrative Structure in Fiction and FilmCornell University PressIthaca, New YorkGoogle Scholar
  11. Cohn, D. 1978Transparent Minds: Narrative Modes for Representing Consciousness in FictionPrinceton University PressPrinceton, NJGoogle Scholar
  12. Cunningham H., Maynard D., Bontcheva K., Tablan V. (2002) GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (A CL-02). Philadelphia, Pennsylvania, pp. 168–175.Google Scholar
  13. Dave K., Lawrence S., Pennock D.M. (2003) Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. In Proceedings of the 12th International World Wide Web Conference (WWW2003). Budapest, Hungary. Available at
  14. Rivera, J. 1977A Structural Theory of EmotionsInternational Universities PressNew YorkGoogle Scholar
  15. Doležel, L. 1973Narrative Modes in Czech LiteratureUniversity of Toronto PressToronto, CanadaGoogle Scholar
  16. Donnellan, K. 1966Reference and Definite DescriptionsPhilosophical Review60281304Google Scholar
  17. Fauconnier, G. 1985Mental Spaces: Aspects of Meaning Construction in Natural LanguageMIT PressCambridge, MassachusettsGoogle Scholar
  18. Fodor J.D. (1979) The Linguistic Description of Opaque Contexts, Outstanding dissertations in linguistics 13. Garland, New York & London.Google Scholar
  19. Gordon A., Kazemzadeh A., Nair A., Petrova M. (2003) Recognizing Expressions of Commonsense Psychology in English Text. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (A CL-03). Sapporo, Japan, pp. 208–215.Google Scholar
  20. Halliday, M. 1985/1994An Introduction to Functional GrammarEdward ArnoldLondonGoogle Scholar
  21. Hart, R.P.,  et al. 1984Systematic Analysis of Political Discourse: The Development of DICTIONK.S,  eds. Political Communication Yearbook: 1984Southern Illinois University PressCarbondale, Illinois97134Google Scholar
  22. Hatzivassiloglou V., McKeown K. (1997) Predicting the Semantic Orientation of Adjectives. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-97). Madrid, Spain, pp. 174–181.Google Scholar
  23. Heise D. (2001) Project Magellan: Collecting Cross-Cultural Affective Meanings via The Internet. Electronic Journal of Sociology 5(3). Available at
  24. Heise D.R. (1965) Semantic Differential Profiles for 1000 Most Frequent English Words. Psychological Monographs 79(601).Google Scholar
  25. Johnson-Laird, P., Oatley, K. 1989The Language of Emotions: An Analysis of a Semantic FieldCognition and Emotion381123Google Scholar
  26. Kaufer D. (2000) Flaming: A White Paper.
  27. Kaufer, D., Ishizaki, S., Butler, B., Collins, J. 2004The Power of Words; Unveiling the Speaker and Writer’s Hidden CraftLawrence ErlbaumMahwah, New JerseyGoogle Scholar
  28. Krippendorf, K. 1980Content Analysis: An Introduction to its MethodologySage PublicationsBeverly HillsGoogle Scholar
  29. Kuroda, S.-Y. 1973Where Epistemology, Style and Grammar Meet: A Case Study from the JapaneseKiparsky, P.Anderson, S. eds. A Festschrift for Morris HalleHolt Rinehart & WinstonNew York, NY377391Google Scholar
  30. Kuroda, S.-Y. 1976Reflections on the Foundations of Narrative Theory—From a Linguistic Point of ViewDijk, T. eds. Pragmatics of Language and LiteratureNorth-HollandAmsterdam107140Google Scholar
  31. Labov, W. 1984IntensitySchiffrin, D. eds. Meaning, Form, and Use in Context: Linguistic ApplicationsGeorgetown University PressWashington, DC4370Google Scholar
  32. Martin, J. 1992English Text: System and StructureJohn BenjaminsPhiladelphia/AmsterdamGoogle Scholar
  33. Martin, J. 2000Beyond Exchange: APPRAISAL Systems in EnglishHunston, S.Thompson, G. eds. Evaluation in Text: Authorial Stance and the Construction of DiscourseOxford University PressOxford142175Google Scholar
  34. Onega S., Landes J.A.G. (1996) Narratology: An Introduction. Longman.Google Scholar
  35. Ortony, A., Clore, G.L., Foss, M.A. 1987The Referential Structure of the Affective LexiconCognitive Science11341364CrossRefGoogle Scholar
  36. Osgood, C.E., Suci, G., Tannenbaum, P. 1957The Measurement of MeaningUniversity of Illinois PressUrbanaGoogle Scholar
  37. Pang B., Lee L., Vaithyanathan S. (2002) Thumbs up? Sentiment Classification Using Machine Learning Techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2002). Philadelphia, Pennsylvania, pp. 79–86.Google Scholar
  38. Quine, W.V. 1956/1976The Ways of Paradox and Other Essays, Revised and Enlarged EditionHarvard University PressCambridge, MassachusettsGoogle Scholar
  39. Quirk, R., Greenbaum, S., Leech, G., Svartvik, J. 1985A Comprehensive Grammar of the English LanguageLongmanNew YorkGoogle Scholar
  40. Rapaport, W. 1986Logical Foundations for Belief RepresentationCognitive Science10371422CrossRefGoogle Scholar
  41. Riloff E., Wiebe J. (2003) Learning Extraction Patterns for Subjective Expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 105–112.Google Scholar
  42. Riloff E., Wiebe J., Wilson T. (2003) Learning Subjective Nouns Using Extraction Pattern Bootstrapping. In Proceedings of the 7th Conference on Natural Language Learning (CoNLL-2003). Edmonton, Canada, pp. 25–32.Google Scholar
  43. Russell, J. 1980A Circumplex Model of AffectJournal of Personality and Social Psychology3911611178CrossRefGoogle Scholar
  44. Spertus E. (1997) Smokey: Automatic Recognition of Hostile Messages. In Proceedings of the Eighth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-97). Providence, Rhode Island, pp. 1058–1065.Google Scholar
  45. Subasic, P., Huettner, A. 2001Affect Analysis of Text Using Fuzzy Semantic TypingIEEE Transactions on Fuzzy Systems9483496CrossRefGoogle Scholar
  46. Tong R. (2001) An Operational System for Detecting and Tracking Opinions in Online Discussions. In Working Notes of the SIGIR Workshop on Operational Text Classification. New Orleans, Louisianna, pp. 1–6.Google Scholar
  47. Turney P. (2002) Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, Pennsylvania, pp. 417–424.Google Scholar
  48. Turney, P., Littman, M.L. 2003Measuring Praise and Criticism: Inference of Semantic Orientation from AssociationACM Transactions on Information Systems (TOTS)21315346Google Scholar
  49. Uspensky, B. 1973A Poetics of CompositionUniversity of California PressBerkeley, CAGoogle Scholar
  50. Watson, D., Tellegen, A. 1985Toward a Consensual Structure of MoodPsychological Bulletin98219235CrossRefGoogle Scholar
  51. White P. (2002) Appraisal: The Language of Attitudinal Evaluation and Intersubjective Stance. In Verschueren, Ostman, blommaert, Bulcaen (eds.), The Handbook of Pragmatics. Amsterdam/Philadelphia, John Benjamins Publishing Company, pp. 1–27.Google Scholar
  52. Wiebe, J. 1991References in Narrative TextNoûs25457486Google Scholar
  53. Wiebe, J. 1994Tracking Point of View in NarrativeComputational Linguistics20233287Google Scholar
  54. Wiebe J. (2000) Learning Subjective Adjectives from Corpora. In Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI-2000). Austin, Texas, pp. 735–740.Google Scholar
  55. Wiebe J. (2002) Instructions for Annotating Opinions in Newspaper Articles. Department of Computer Science Technical Report TR-02-101, University of Pittsburgh.Google Scholar
  56. Wiebe J., Bruce R., Bell M., Martin M., Wilson T. (2001a) A Corpus Study of Evaluative and Speculative Language. In Proceedings of the 2nd ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-2001). Aalborg, Denmark, pp. 186–195.Google Scholar
  57. Wiebe J., Wilson T., Bell M. (2001b) Identifying Collocations for Recognizing Opinions. In Proceedings of the ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation. Toulouse, France, pp. 24–31.Google Scholar
  58. Wiebe, J., Wilson, T., Bruce, R., Bell, M., Martin, M. 2004Learning Subjective LanguageComputational Linguistics30277308CrossRefGoogle Scholar
  59. Wilks, Y., Bien, J. 1983Beliefs, Points of View and Multiple EnvironmentsCognitive Science795119CrossRefGoogle Scholar
  60. Wilson T., Wiebe J. (2003) Annotating Opinions in the World Press. In Proceedings of the 4th ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-03). Sapporo, Japan, pp. 13–22.Google Scholar
  61. Wilson T., Wiebe J., Hwa R. (2004) Just How Mad Are You? Finding Strong and Weak Opinion Clauses. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI, 2004). San Jose, California, pp. 761–766.Google Scholar
  62. Yi J., Nasukawa T., Bunescu R., Niblack W. (2003) Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques. In Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM-2003). Melbourne, Florida, pp. 427–434.Google Scholar
  63. Yu H., Hatzivassiloglou V. (2003) Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 129–136.Google Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of PittsburghPittsburghUSA
  2. 2.Intelligent Systems ProgramUniversity of PittsburghPittsburghUSA
  3. 3.Department of Computer ScienceCornell UniversityIthacaUSA

Personalised recommendations