Building Subjectivity Lexicon(s) from Scratch for Essay Data

  • Beata Beigman Klebanov
  • Jill Burstein
  • Nitin Madnani
  • Adam Faulkner
  • Joel Tetreault
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7181)


While there are a number of subjectivity lexicons available for research purposes, none can be used commercially. We describe the process of constructing subjectivity lexicon(s) for recognizing sentiment polarity in essays written by test-takers, to be used within a commercial essay-scoring system. We discuss ways of expanding a manually-built seed lexicon using dictionary-based, distributional in-domain and out-of-domain information, as well as using Amazon Mechanical Turk to help “clean up” the expansions. We show the feasibility of constructing a family of subjectivity lexicons from scratch using a combination of methods to attain competitive performance with state-of-art research-only lexicons. Furthermore, this is the first use, to our knowledge, of a paraphrase generation system for expanding a subjectivity lexicon.


essay writing sentiment analysis sentiment polarity subjectivity lexicon C5.0 lexicon expansion paraphrase generation thesaurus resources 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wiebe, J., Wilson, T., Cardie, C.: Annotating expressions of opinions and emotions in language. Language Resources and Evaluation 39(2-3), 165–210 (2005)CrossRefGoogle Scholar
  2. 2.
    Stone, P., Dunphy, D., Smith, M., Ogilvie, D.: The General Inquirer: A Computer Approach to Content Analysis. MIT (1966)Google Scholar
  3. 3.
    Miller, G.: WordNet: A lexical database. Communications of the ACM 38(11), 39–41 (1995)Google Scholar
  4. 4.
    Lin, D.: Automatic retrieval and clustering of similar word. In: Proceedings of the ACL, Montreal, Canada (1998)Google Scholar
  5. 5.
    Madnani, N., Dorr, B.: Generating Targeted Paraphrases for Improved Translation. ACM Transactions on Intelligent Systems and Technology 3(2) (to appear)Google Scholar
  6. 6.
    Burstein, J., Pedersen, T.: Towards Improving Synonym Options in a Text Modification Application. University of Minnesota Supercomputing Institute Research Report Series, UMSI 2010/165 (2010)Google Scholar
  7. 7.
    Bannard, C., Callison-Burch, C.: Paraphrasing with bilingual parallel corpora. In: Proceedings of the ACL, Ann Arbor, MI, pp. 597–604 (2005)Google Scholar
  8. 8.
    Koehn, P.: EUROPARL: A Parallel corpus for Statistical Machine Translation. In: Proceedings of the Machine Translation Summit (2005)Google Scholar
  9. 9.
    Quinlan, J.R.: C4.5: Programs for machine learning. Morgan Kaufmann Publishers (1993)Google Scholar
  10. 10.
    Popescu, A., Etzioni, O.: Extracting product features and opinions from reviews. In: Proceedings of HLT/EMNLP, pp. 339–346 (2005)Google Scholar
  11. 11.
    Pang, B., Lee, L.: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the ACL, pp. 271–278 (2004)Google Scholar
  12. 12.
    Turney, P.: Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In: Proceedings of the ACL, Philadelphia, pp. 417–424 (2002)Google Scholar
  13. 13.
    Strapparava, C., Mihalcea, R.: SemEval-2007 Task 14: Affective Text. In: Proceedings of SemEval, Prague, Czech Republic (2007)Google Scholar
  14. 14.
    Wilson, T., Wiebe, J., Hoffmann, P.: Recognizing Contextual Polarity in Phrase Level Sentiment Analysis. In: Proceedings of HLT-EMNLP (2005)Google Scholar
  15. 15.
    Godbole, N., Srinivasaiah, M., Skiena, S.: Large Scale Sentiment Analysis for News and Blogs. In: Proceedings of ICWSM (2007)Google Scholar
  16. 16.
    Burstein, J., Marcu, D., Knight, K.: Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays. In: Harabagiu, S., Ciravegna, F. (eds.) Special Issue on Advances in NLP, IEEE Intelligent Systems, vol. 18(1), pp. 32–39 (2003)Google Scholar
  17. 17.
    Choi, Y., Cardie, C.: Learning with Compositional Semantics as Structure Inference for Subsentential Sentiment Analysis. In: Proceedings of EMNLP, pp. 793–801 (2008)Google Scholar
  18. 18.
    Moilanen, K., Pulman, S.: Sentiment Composition. In: Proceedings of Recent Advances in NLP (RANLP), Borovets, Bulgaria, pp. 378–382 (September 2007)Google Scholar
  19. 19.
    Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-Based Method for Sentiment Analysis. Computational Linguistics 37(2), 267–307 (2011)CrossRefGoogle Scholar
  20. 20.
    Polanyi, L., Zaenen, A.: Contextual valence shifters. In: Wiebe (ed.) Computing Attitude and Affect in Text: Theory and Applications, pp. 1–10. Springer, Dordrecht (2006)CrossRefGoogle Scholar
  21. 21.
    Riloff, E., Wiebe, J.: Learning extraction patterns for subjective expressions. In: Proceedings of EMNLP, pp. 105–112 (2003)Google Scholar
  22. 22.
    Wiebe, J., Mihalcea, R.: Word Sense and Subjectivity. In: Proceedings of ACL (2006)Google Scholar
  23. 23.
    Gyamfi, Y., Wiebe, J., Mihalcea, R., Akkaya, C.: Integrating Knowledge for Subjectivity Sense Labeling. In: Proceedings of NAACL (2009)Google Scholar
  24. 24.
    Esuli, A., Sabastiani, F.: Determining term subjectivity and term orientation for opinion mining. In: Proceedings of the EACL (2006)Google Scholar
  25. 25.
    Kim, S., Hovy, E.: Determining the sentiment of opinions. In: Proceedings of COLING 2004 (2004)Google Scholar
  26. 26.
    Andreevskaia, A., Bergler, S.: Mining WordNet for a fuzzy sentiment: Sentiment tag extraction from wordnet glosses. In: Proceedings of the EACL (2006)Google Scholar
  27. 27.
    Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 168–177 (2004)Google Scholar
  28. 28.
    Kanayama, H., Nasukawa, T.: Fully Automatic Lexicon Expansion for Domain-oriented Sentiment Analysis. In: Proceedings of EMNLP (2006)Google Scholar
  29. 29.
    Strapparava, C., Valitutti, A.: WordNet-affect: and affective extension of Word-Net. In: Proceedings of LREC, Lisbon, Portugal (2004)Google Scholar
  30. 30.
    Kamps, J., Marx, M., Mokken, R., de Rijke, M.: Using WordNet to measure semantic orientation of adjectives. In: Proceedings of LREC (2004)Google Scholar
  31. 31.
    Takamura, H., Inui, T., Okumura, M.: Extracting semantic orientation of words using spin model. In: Proceedings of the ACL, pp. 133–140 (2005)Google Scholar
  32. 32.
    Mohammad, S., Dunne, C., Dorr, B.: Generating High-Coverage Semantic Orientation Lexicons from Overtly Marked Words and a Thesaurus. In: Proceedings of EMNLP (2009)Google Scholar
  33. 33.
    Hatzivassiloglou, V., McKeown, K.: Predicting the semantic orientation of adjectives. In: Proceedings of the ACL, pp. 174–181 (1997)Google Scholar
  34. 34.
    Turney, P., Littman, M.: Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems 21(4), 315–346 (2003)CrossRefGoogle Scholar
  35. 35.
    Yu, H., Hatzivassiloglou, V.: Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In: Proceedings of EMNLP, Morristown, NJ, pp. 129–136 (2003)Google Scholar
  36. 36.
    Metzler, D., Dumais, S., Meek, C.: Similarity Measures for Short Segments of Text. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECIR 2007. LNCS, vol. 4425, pp. 16–27. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  37. 37.
    Riezler, S., Vasserman, A., Tsochantaridis, I., Mittal, V., Liu, Y.: Statistical machine translation for query expansion in answer retrieval. In: Proceedings of ACL (2007)Google Scholar
  38. 38.
    Zhou, L., Lin, C., Muntenau, D., Hovy, E.: ParaEval: Using paraphrases to evaluate summaries automatically. In: Proceedings of HLT-NAACL, pp. 447–454 (2006)Google Scholar
  39. 39.
    Owczarzak, K., Groves, D., van Genabith, J., Way, A.: Contextual bitext-derived paraphrases in automatic MT evaluation. In: Proceedings on the Workshop on Statistical Machine Translation, New York, pp. 86–93 (2006)Google Scholar
  40. 40.
    Kauchak, D., Barzilay, R.: Paraphrasing for automatic evaluation. In: Proceedings of HLT-NAACL, New York, pp. 455–462 (2006)Google Scholar
  41. 41.
    Callison-Burch, C., Koehn, P., Osborne, M.: Improved statistical machine translation using paraphrases. In: Proceedings of NAACL, New York, pp. 17–24 (2006)Google Scholar
  42. 42.
    Madnani, N., Resnik, P., Dorr, B., Schwartz, R.: Are multiple reference translations necessary? Investigating the value of paraphrased reference translations in parameter optimization. In: Proceedings of the Eighth Conference of the Association for Machine Translation in the Americas (AMTA), Waikiki, HI, pp. 143–152 (2008)Google Scholar
  43. 43.
    Madnani, N., Dorr, B.: Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods. Computational Linguistics 36(3), 341–387 (2010)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Snow, R., O’Connor, B., Jurafsky, D., Ng, Y.: Cheap and fast – but is it good?: Evaluating non-expert annotations for natural language tasks. In: Proceedings of EMNLP, pp. 254–263. Association for Computational Linguistics, Stroudsburg (2008)Google Scholar
  45. 45.
    Sheng, V., Provost, F., Ipeirotis, P.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceeding of KDD, pp. 614–622 (2008)Google Scholar
  46. 46.
    Callison-Burch, C.: Fast, cheap, and creative: evaluating translation quality using Amazon’s Mechanical Turk. In: Proceedings of EMNLP, pp. 286–295 (2010)Google Scholar
  47. 47.
    Callison-Burch, C., Dredze, M.: Creating speech and language data with Amazon’s Mechanical Turk. In: Proceedings of the NAACL Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp. 1–12 (2010)Google Scholar
  48. 48.
    Akkaya, C., Conrad, A., Wiebe, J., Mihalcea, R.: Amazon Mechanical Turk for subjectivity word sense disambiguation. In: Proceedings of the NAACL Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp. 195–203 (2010)Google Scholar
  49. 49.
    Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costa-juss, M., Banchs, R.: Opinion mining of Spanish customer comments with non-expert annotations on Mechanical Turk. In: Proceedings of the NAACL Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp. 114–121 (2010)Google Scholar
  50. 50.
    Mohammad, S., Turney, P.: Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In: Proceedings of NAACL Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pp. 26–34 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Beata Beigman Klebanov
    • 1
  • Jill Burstein
    • 1
  • Nitin Madnani
    • 1
  • Adam Faulkner
    • 2
  • Joel Tetreault
    • 1
  1. 1.Educational Testing ServiceUSA
  2. 2.Graduate CenterThe City University of New YorkUSA

Personalised recommendations