Skip to main content

Annotating Expressions of Opinions and Emotions in Language

Abstract

This paper describes a corpus annotation project to study issues in the manual annotation of opinions, emotions, sentiments, speculations, evaluations and other private states in language. The resulting corpus annotation scheme is described, as well as examples of its use. In addition, the manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.

This is a preview of subscription content, access via your institution.

References

  1. C. Anderson G. McMaster (1982) ArticleTitleComputer Assisted Modeling of Affective Tone in Written Documents Computers and the Humanities 16 1–9 Occurrence Handle10.1007/BF02259727

    Article  Google Scholar 

  2. N. Asher (1986) ArticleTitleBelief in Discourse Representation Theory Journal of Philosophical Logic 15 127–189 Occurrence Handle10.1007/BF00305490

    Article  Google Scholar 

  3. A. Banfield (1982) Unspeakable Sentences Routledge and Kegan Paul Boston

    Google Scholar 

  4. Bergler S. (1992) Evidential Analysis of Reported Speech. Ph.D. thesis, Brandeis University.

  5. D. Biber E. Finegan (1989) ArticleTitleStyles of Stance in English: Lexical and Grammatical Marking of Evidentiality and Affect Text 9 IssueID1 93–124

    Google Scholar 

  6. Breck E., Cardie C. (2004) Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions. In Proceedings of the Twentieth International Conference on Computational Linguistics (COLING 2004). Geneva, Switzerland, pp. 120–126.

  7. R. Bruce J. Wiebe (1999) ArticleTitleRecognizing Subjectivity: A Case Study of Manual Tagging Natural Language Engineering 5 IssueID2 187–205 Occurrence Handle10.1017/S1351324999002181

    Article  Google Scholar 

  8. Cardie C., Wiebe J., Wilson T., Litman D. (2003) Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering. In Working Notes – New Directions in Question Answering (AAAI Spring Symposium Series).

  9. H.-N. Castañeda (1977) ArticleTitleOn the Philosophical Foundations of the Theory of Communication: Reference Midwest Studies in Philosophy 2 165–186

    Google Scholar 

  10. S. Chatman (1978) Story and Discourse: Narrative Structure in Fiction and Film Cornell University Press Ithaca, New York

    Google Scholar 

  11. D. Cohn (1978) Transparent Minds: Narrative Modes for Representing Consciousness in Fiction Princeton University Press Princeton, NJ

    Google Scholar 

  12. Cunningham H., Maynard D., Bontcheva K., Tablan V. (2002) GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (A CL-02). Philadelphia, Pennsylvania, pp. 168–175.

  13. Dave K., Lawrence S., Pennock D.M. (2003) Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. In Proceedings of the 12th International World Wide Web Conference (WWW2003). Budapest, Hungary. Available at http://www2003.org.

  14. J. Rivera Particlede (1977) A Structural Theory of Emotions International Universities Press New York

    Google Scholar 

  15. L. Doležel (1973) Narrative Modes in Czech Literature University of Toronto Press Toronto, Canada

    Google Scholar 

  16. K. Donnellan (1966) ArticleTitleReference and Definite Descriptions Philosophical Review 60 281–304

    Google Scholar 

  17. G. Fauconnier (1985) Mental Spaces: Aspects of Meaning Construction in Natural Language MIT Press Cambridge, Massachusetts

    Google Scholar 

  18. Fodor J.D. (1979) The Linguistic Description of Opaque Contexts, Outstanding dissertations in linguistics 13. Garland, New York & London.

  19. Gordon A., Kazemzadeh A., Nair A., Petrova M. (2003) Recognizing Expressions of Commonsense Psychology in English Text. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (A CL-03). Sapporo, Japan, pp. 208–215.

  20. M. Halliday (1985/1994) An Introduction to Functional Grammar Edward Arnold London

    Google Scholar 

  21. R.P. Hart (1984) Systematic Analysis of Political Discourse: The Development of DICTION K.S (Eds) et al. Political Communication Yearbook: 1984 Southern Illinois University Press Carbondale, Illinois 97–134

    Google Scholar 

  22. Hatzivassiloglou V., McKeown K. (1997) Predicting the Semantic Orientation of Adjectives. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-97). Madrid, Spain, pp. 174–181.

  23. Heise D. (2001) Project Magellan: Collecting Cross-Cultural Affective Meanings via The Internet. Electronic Journal of Sociology 5(3). Available at http://www.sociology.org.

  24. Heise D.R. (1965) Semantic Differential Profiles for 1000 Most Frequent English Words. Psychological Monographs 79(601).

  25. P. Johnson-Laird K. Oatley (1989) ArticleTitleThe Language of Emotions: An Analysis of a Semantic Field Cognition and Emotion 3 IssueID2 81–123

    Google Scholar 

  26. Kaufer D. (2000) Flaming: A White Paper. http://www.eudora.com.

  27. D. Kaufer S. Ishizaki B. Butler J. Collins (2004) The Power of Words; Unveiling the Speaker and Writer’s Hidden Craft Lawrence Erlbaum Mahwah, New Jersey

    Google Scholar 

  28. K. Krippendorf (1980) Content Analysis: An Introduction to its Methodology Sage Publications Beverly Hills

    Google Scholar 

  29. S.-Y. Kuroda (1973) Where Epistemology, Style and Grammar Meet: A Case Study from the Japanese P. Kiparsky S. Anderson (Eds) A Festschrift for Morris Halle Holt Rinehart & Winston New York, NY 377–391

    Google Scholar 

  30. S.-Y. Kuroda (1976) Reflections on the Foundations of Narrative Theory—From a Linguistic Point of View T. Dijk Particlevan (Eds) Pragmatics of Language and Literature North-Holland Amsterdam 107–140

    Google Scholar 

  31. W. Labov (1984) Intensity D. Schiffrin (Eds) Meaning, Form, and Use in Context: Linguistic Applications Georgetown University Press Washington, DC 43–70

    Google Scholar 

  32. J. Martin (1992) English Text: System and Structure John Benjamins Philadelphia/Amsterdam

    Google Scholar 

  33. J. Martin (2000) Beyond Exchange: APPRAISAL Systems in English S. Hunston G. Thompson (Eds) Evaluation in Text: Authorial Stance and the Construction of Discourse Oxford University Press Oxford 142–175

    Google Scholar 

  34. Onega S., Landes J.A.G. (1996) Narratology: An Introduction. Longman.

  35. A. Ortony G.L. Clore M.A. Foss (1987) ArticleTitleThe Referential Structure of the Affective Lexicon Cognitive Science 11 341–364 Occurrence Handle10.1016/S0364-0213(87)80010-1

    Article  Google Scholar 

  36. C.E. Osgood G. Suci P. Tannenbaum (1957) The Measurement of Meaning University of Illinois Press Urbana

    Google Scholar 

  37. Pang B., Lee L., Vaithyanathan S. (2002) Thumbs up? Sentiment Classification Using Machine Learning Techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2002). Philadelphia, Pennsylvania, pp. 79–86.

  38. W.V. Quine (1956/1976) The Ways of Paradox and Other Essays, Revised and Enlarged Edition Harvard University Press Cambridge, Massachusetts

    Google Scholar 

  39. R. Quirk S. Greenbaum G. Leech J. Svartvik (1985) A Comprehensive Grammar of the English Language Longman New York

    Google Scholar 

  40. W. Rapaport (1986) ArticleTitleLogical Foundations for Belief Representation Cognitive Science 10 371–422 Occurrence Handle10.1016/S0364-0213(86)80011-8

    Article  Google Scholar 

  41. Riloff E., Wiebe J. (2003) Learning Extraction Patterns for Subjective Expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 105–112.

  42. Riloff E., Wiebe J., Wilson T. (2003) Learning Subjective Nouns Using Extraction Pattern Bootstrapping. In Proceedings of the 7th Conference on Natural Language Learning (CoNLL-2003). Edmonton, Canada, pp. 25–32.

  43. J. Russell (1980) ArticleTitleA Circumplex Model of Affect Journal of Personality and Social Psychology 39 1161–1178 Occurrence Handle10.1037//0022-3514.39.1.29

    Article  Google Scholar 

  44. Spertus E. (1997) Smokey: Automatic Recognition of Hostile Messages. In Proceedings of the Eighth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-97). Providence, Rhode Island, pp. 1058–1065.

  45. P. Subasic A. Huettner (2001) ArticleTitleAffect Analysis of Text Using Fuzzy Semantic Typing IEEE Transactions on Fuzzy Systems 9 483–496 Occurrence Handle10.1109/91.940962

    Article  Google Scholar 

  46. Tong R. (2001) An Operational System for Detecting and Tracking Opinions in Online Discussions. In Working Notes of the SIGIR Workshop on Operational Text Classification. New Orleans, Louisianna, pp. 1–6.

  47. Turney P. (2002) Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, Pennsylvania, pp. 417–424.

  48. P. Turney M.L. Littman (2003) ArticleTitleMeasuring Praise and Criticism: Inference of Semantic Orientation from Association ACM Transactions on Information Systems (TOTS) 21 IssueID4 315–346

    Google Scholar 

  49. B. Uspensky (1973) A Poetics of Composition University of California Press Berkeley, CA

    Google Scholar 

  50. D. Watson A. Tellegen (1985) ArticleTitleToward a Consensual Structure of Mood Psychological Bulletin 98 219–235 Occurrence Handle10.1037/0033-2909.98.2.219

    Article  Google Scholar 

  51. White P. (2002) Appraisal: The Language of Attitudinal Evaluation and Intersubjective Stance. In Verschueren, Ostman, blommaert, Bulcaen (eds.), The Handbook of Pragmatics. Amsterdam/Philadelphia, John Benjamins Publishing Company, pp. 1–27.

  52. J. Wiebe (1991) ArticleTitleReferences in Narrative Text Noûs 25 IssueID4 457–486

    Google Scholar 

  53. J. Wiebe (1994) ArticleTitleTracking Point of View in Narrative Computational Linguistics 20 IssueID2 233–287

    Google Scholar 

  54. Wiebe J. (2000) Learning Subjective Adjectives from Corpora. In Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI-2000). Austin, Texas, pp. 735–740.

  55. Wiebe J. (2002) Instructions for Annotating Opinions in Newspaper Articles. Department of Computer Science Technical Report TR-02-101, University of Pittsburgh.

  56. Wiebe J., Bruce R., Bell M., Martin M., Wilson T. (2001a) A Corpus Study of Evaluative and Speculative Language. In Proceedings of the 2nd ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-2001). Aalborg, Denmark, pp. 186–195.

  57. Wiebe J., Wilson T., Bell M. (2001b) Identifying Collocations for Recognizing Opinions. In Proceedings of the ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation. Toulouse, France, pp. 24–31.

  58. J. Wiebe T. Wilson R. Bruce M. Bell M. Martin (2004) ArticleTitleLearning Subjective Language Computational Linguistics 30 IssueID3 277–308 Occurrence Handle10.1162/0891201041850885

    Article  Google Scholar 

  59. Y. Wilks J. Bien (1983) ArticleTitleBeliefs, Points of View and Multiple Environments Cognitive Science 7 95–119 Occurrence Handle10.1016/S0364-0213(83)80007-X

    Article  Google Scholar 

  60. Wilson T., Wiebe J. (2003) Annotating Opinions in the World Press. In Proceedings of the 4th ACL SIGdial Workshop on Discourse and Dialogue (SlGdial-03). Sapporo, Japan, pp. 13–22.

  61. Wilson T., Wiebe J., Hwa R. (2004) Just How Mad Are You? Finding Strong and Weak Opinion Clauses. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI, 2004). San Jose, California, pp. 761–766.

  62. Yi J., Nasukawa T., Bunescu R., Niblack W. (2003) Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques. In Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM-2003). Melbourne, Florida, pp. 427–434.

  63. Yu H., Hatzivassiloglou V. (2003) Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2003). Sapporo, Japan, pp. 129–136.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Janyce Wiebe.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wiebe, J., Wilson, T. & Cardie, C. Annotating Expressions of Opinions and Emotions in Language. Language Res Eval 39, 165–210 (2005). https://doi.org/10.1007/s10579-005-7880-9

Download citation

Keywords

  • affect
  • attitudes
  • corpus annotation
  • emotion
  • natural language processing
  • opinions
  • sentiment
  • subjectivity