Behavior Research Methods

, Volume 51, Issue 2, pp 480–492 | Cite as

Measuring the importance of context when modeling language comprehension

  • Justin Garten
  • Brendan Kennedy
  • Kenji Sagae
  • Morteza DehghaniEmail author


It is widely accepted that language requires context in order to function as communication between speakers and listeners. As listeners, we make use of background knowledge — about the speaker, about entities and concepts, about previous utterances — in order to infer the speaker’s intended meaning. But even if there is consensus that these sources of information are a necessary component of linguistic communication, it is another matter entirely to provide a thorough, quantitative accounting for context’s interaction with language. When does context matter? What kinds of context matter in which kinds of domains? The empirical investigation of these questions is inhibited by a number of factors: the challenge of quantifying language, the boundless combinations of domains and types of context to be measured, and the challenge of selecting and applying a given construct to natural language data. In response to these factors, we introduce and demonstrate a methodological framework for testing the importance of contextual information in inferring speaker intentions from text. We apply Long Short-term Memory (LSTM) networks, a standard for representing language in its natural, sequential state, and conduct a set of experiments for predicting the persuasive intentions of speakers in political debates using different combinations of text and background information about the speaker. We show, in our modeling and discussion, that the proposed framework is suitable for empirically evaluating the manner and magnitude of context’s relevance for any number of domains and constructs.


Methodological innovation Text analysis Continuous representations Intent recognition 


Supplementary material

13428_2019_1200_MOESM1_ESM.pdf (71 kb)
(PDF 71.3 KB)


  1. Aizawa, A. (2003). An information-theoretic perspective of tf-idf measures. Information Processing & Management, 39(1), 45–65.CrossRefGoogle Scholar
  2. Austin, J. L. (1975). How to do things with words. Oxford University Press.Google Scholar
  3. Bamman, D., & Smith, N. A. (2015). Contextualized sarcasm detection on twitter. In Icwsm (pp. 574–577).Google Scholar
  4. Bamman, D., Dyer, C., & Smith, N. A. (2014). Distributed representations of geographically situated language. In Proceedings of d annual meeting of the association for computational linguistics (short papers)(Vol. 828, pp. 834).Google Scholar
  5. Capelli, C. A., Nakagawa, N., & Madden, C. M. (1990). How children understand sarcasm: The role of context and intonation. Child Development, 61(6), 1824–1841.CrossRefGoogle Scholar
  6. Carpini, M. X. D., Cook, F. L., & Jacobs, L. R. (2004). Public deliberation, discursive participation, and citizen engagement: A review of the empirical literature. Annual Review of Political Science, 7, 315–344.CrossRefGoogle Scholar
  7. Caruna, R. (1993). Multitask learning: A knowledge-based source of inductive bias. In Machine learning: Proceedings of the tenth international conference (pp. 41–48).Google Scholar
  8. Chong, D., & Druckman, J. N. (2007). Framing public opinion in competitive democracies. American Political Science Review, 101(4), 637–655.CrossRefGoogle Scholar
  9. Christiano, T. (1995). Voting and democracy. Canadian Journal of Philosophy, 25(3), 395–414.CrossRefGoogle Scholar
  10. Cohen, G. L. (2003). Party over policy: The dominating impact of group influence on political beliefs. Journal of Personality and Social Psychology, 85(5), 808.CrossRefGoogle Scholar
  11. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on machine learning (pp. 160–167).Google Scholar
  12. Dawes, R. M., Van de Kragt, A. J., & Orbell, J. M. (1990). Cooperation for the benefit of us?-not me, or my conscience. University of Chicago Press.Google Scholar
  13. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211.CrossRefGoogle Scholar
  14. Fine, G. A. (1983). Sociological approaches to the study of humor. In Handbook of humor research (pp. 159–181). Springer.Google Scholar
  15. Garimella, A., Banea, C., & Mihalcea, R. (2017). Demographic-aware word associations. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2285–2295).Google Scholar
  16. Garten, J., Hoover, J., Johnson, K. M., Boghrati, R., Iskiwitch, C., & Dehghani, M. (2017). Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis. Behavior Research Methods, 1–18.Google Scholar
  17. Garten, J., Kennedy, B., Hoover, J., Sagae, K., & Dehghani, M (2018). Incorporating demographic embeddings into language understanding. Cognitive Science.Google Scholar
  18. Goodwin, C. (1984). Notes on story structure and the organization of participation. Structures of Social Action: Studies in Conversation Analysis, 225–246.Google Scholar
  19. Graham, J., Haidt, J., & Nosek, B. A. (2009). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5), 1029.CrossRefGoogle Scholar
  20. Graves, A., & Schmidhuber, J. (2009). Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in neural information processing systems (pp. 545–552).Google Scholar
  21. Grice, H. P. (1975). . Logic and Conversation, 1975, 41–58.Google Scholar
  22. Heritage, J., & Greatbatch, D. (1986). Generating applause: A study of rhetoric and response at party political conferences. American Journal of Sociology, 92(1), 110–157.CrossRefGoogle Scholar
  23. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.CrossRefGoogle Scholar
  24. Hovy, D. (2015). Demographic factors improve classification performance. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers) (Vol. 1, pp. 752–762).Google Scholar
  25. Hovy, D., & Søgaard, A. (2015). Tagging performance correlates with author age. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers) (Vol. 2, pp. 483–488).Google Scholar
  26. Iyengar, S. (2005). Speaking of values: The framing of american politics. In The forum (Vol. 3, pp. 1–8).Google Scholar
  27. Jerit, J. (2004). Survival of the fittest: Rhetoric during the course of an election campaign. Political Psychology, 25(4), 563–575.CrossRefGoogle Scholar
  28. Ji, Y., Haffari, G., & Eisenstein, J. (2016). A latent variable recurrent neural network for discourse relation language models. arXiv:1603.01913.
  29. Johannsen, A., Hovy, D., & Søgaard, A (2015). Cross-lingual syntactic variation over age and gender. In Proceedings of the nineteenth conference on computational natural language learning (pp. 103–112).Google Scholar
  30. Jost, J. T., Federico, C. M., & Napier, J. L. (2009). Political ideology: Its structure, functions, and elective affinities. Annual Review of Psychology, 60, 307–337.CrossRefGoogle Scholar
  31. Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016). Exploring the limits of language modeling. arXiv:1602.02410.
  32. Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.6980.
  33. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems (pp. 3294–3302).Google Scholar
  34. López, I. H (2015). Dog whistle politics: How coded racial appeals have reinvented racism and wrecked the middle class. Oxford University Press.Google Scholar
  35. Macagno, F., & Walton, D (2014). Emotive language in argumentation. Cambridge University Press.Google Scholar
  36. Mendelberg, T. (2002). The deliberative citizen: Theory and evidence. Political Decision Making, Deliberation and Participation, 6(1), 151–193.Google Scholar
  37. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013a). Efficient estimation of word representations in vector space. arXiv:1301.3781.
  38. Mikolov, T., Yih, W.-t., & Zweig, G. (2013b). Linguistic regularities in continuous space word representations. In Hlt-naacl (pp. 746–751).Google Scholar
  39. Musi, E., Ghosh, D., & Muresan, S. (2016). Towards feasible guidelines for the annotation of argument schemes. ACL, 2016, 82.Google Scholar
  40. Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (icml-10) (pp. 807–814).Google Scholar
  41. Palangi, H, Deng, L, Shen, Y, Gao, J, He, X, Chen, J, & Ward, R (2016). Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 24(4), 694–707.CrossRefGoogle Scholar
  42. Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001), 2001.Google Scholar
  43. Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) (pp. 1532–1543).Google Scholar
  44. Priddy, K. L., & Keller, P. E. (2005). Artificial neural networks: An introduction (Vol. 68). SPIE Press.Google Scholar
  45. Richardson, H. S (2002). Democratic autonomy: Public reasoning about the ends of policy. Oxford University Press on Demand.Google Scholar
  46. Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 626). Cambridge University Press.Google Scholar
  47. Sniderman, P. M., & Theriault, S. M. (2004). The structure of political argument and the logic of issue framing. Studies in public opinion: Attitudes, nonattitudes, measurement error, and change, 133–65.Google Scholar
  48. Sundermeyer, M., Schlüter, R., & Ney, H. (2012). Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.Google Scholar
  49. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems (pp. 3104–3112).Google Scholar
  50. Suykens, J. A., & Vandewalle, J. (1999). Least squares support vector machine classifiers. Neural Processing Letters, 9(3), 293–300.CrossRefGoogle Scholar
  51. Svozil, D., Kvasnicka, V., & Pospichal, J. (1997). Introduction to multi-layer feed-forward neural networks. Chemometrics and Intelligent Laboratory Systems, 39(1), 43–62.CrossRefGoogle Scholar
  52. Swanson, D. L., & Mancini, P (1996). Politics, media, and modern democracy: An international study of innovations in electoral campaigning and their consequences. Greenwood Publishing Group.Google Scholar
  53. Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37(5), 360–363.Google Scholar
  54. Yang, Y., & Eisenstein, J. (2015). Putting things in context: Community-specific embedding projections for sentiment analysis. Arxiv-Social Media Intelligence.Google Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of Southern CaliforniaLos AngelesUSA
  2. 2.Department of LinguisticsUniversity of CaliforniaDavisUSA
  3. 3.Department of Psychology and Computer ScienceUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations