The extended Moral Foundations Dictionary (eMFD): Development and applications of a crowd-sourced approach to extracting moral intuitions from text


Moral intuitions are a central motivator in human behavior. Recent work highlights the importance of moral intuitions for understanding a wide range of issues ranging from online radicalization to vaccine hesitancy. Extracting and analyzing moral content in messages, narratives, and other forms of public discourse is a critical step toward understanding how the psychological influence of moral judgments unfolds at a global scale. Extant approaches for extracting moral content are limited in their ability to capture the intuitive nature of moral sensibilities, constraining their usefulness for understanding and predicting human moral behavior. Here we introduce the extended Moral Foundations Dictionary (eMFD), a dictionary-based tool for extracting moral content from textual corpora. The eMFD, unlike previous methods, is constructed from text annotations generated by a large sample of human coders. We demonstrate that the eMFD outperforms existing approaches in a variety of domains. We anticipate that the eMFD will contribute to advance the study of moral intuitions and their influence on social, psychological, and communicative processes.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

  2. 2.

  3. 3.

  4. 4.

    This set of articles was utilized for separate analyses that are not reported here.

  5. 5.

  6. 6.


  7. 7.

    Full model specifications are provided in SM Section 9.4


  1. Amin, A. B., Bednarczyk, R. A., Ray, C. E., Melchiori, K. J., Graham, J., Huntsinger, J. R., & Omer, S. B. (2017). Association of moral values with vaccine hesitancy. Nature Human Behaviour, 1(12), 873–880.

    PubMed  Google Scholar 

  2. Araque, O., Gatti, L., & Kalimeri, K. (2019). MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction. arXiv preprint arXiv:1904.08314.

  3. Arendt, F., & Karadas, N. (2017). Content analysis of mediated associations: An automated text-analytic approach. Communication Methods and Measures, 11(2), 105-120.

    Google Scholar 

  4. Aroyo, L., & Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1), 15–24.

    Google Scholar 

  5. Bowman, N., Lewis, R. J., & Tamborini, R. (2014). The morality of May 2, 2011: A content analysis of US headlines regarding the death of Osama bin Laden. Mass Communication and Society, 17(5), 639–664.

    Google Scholar 

  6. Brady, W. J., & Crockett, M. J. (2018). How effective is online outrage? Trends in Cognitive Sciences, 23(2), 79–80.

    PubMed  Google Scholar 

  7. Brady, W. J., Wills, J. A., Jost, J. T., Tucker, J. A., & Van Bavel, J. J. (2017). Emotion shapes the diffusion of moralized content in social networks. Proceedings of the National Academy of Sciences, 114(28), 7313–7318.

    Google Scholar 

  8. Brady, W. J., Gantman, A. P., & Van Bavel, J.J. (2019). Attentional capture helps explain why moral and emotional content go viral. Journal of Experimental Psychology: General.

  9. Brady, W. J., Crockett, M., & Van Bavel, J. J. (in press). The MAD model of moral contagion: The role of motivation, attention and design in the spread of moralized content online. Perspectives on Psychological Science

  10. Clifford, S., & Jerit, J. (2013). How words do the work of politics: Moral foundations theory and the debate over stem cell research. The Journal of Politics, 75(3), 659–671.

    Google Scholar 

  11. Crockett, M. J. (2017). Moral outrage in the digital age. Nature Human Behaviour, 1(11), 769–771.

  12. Eden, A., Tamborini, R., Grizzard, M., Lewis, R., Weber, R., & Prabhu, S. (2014). Repeated exposure to narrative entertainment and the salience of moral intuitions. Journal of Communication, 64(3), 501–520.

    Google Scholar 

  13. Feinberg, M., & Willer, R. (2013). The moral roots of environmental attitudes. Psychological Science, 24(1), 56–62.

    PubMed  Google Scholar 

  14. Feinberg, M., & Willer, R. (2015). From gulf to bridge: When do moral arguments facilitate political influence?. Personality and Social Psychology Bulletin, 41(12), 1665–1681.

    PubMed  Google Scholar 

  15. Fisher, J.T., Hopp, F.R., Prabhu, S., Tamborini, R., & Weber, R. (2019). Developing best practices for the implicit measurement of moral foundation salience. Paper accepted at the 105th annual meeting of the National Communication Association (NCA), Baltimore, MD.

  16. Frimer, J., Haidt, J., Graham, J., Dehghani, M., & Boghrati, R. (2017). Moral foundations dictionaries for linguistic analyses, 2.0. Unpublished Manuscript. Retrieved from:

  17. Fulgoni, D., Carpenter, J., Ungar, L. H., & Preotiuc-Pietro, D. (2016). An empirical exploration of moral foundations theory in partisan news sources. Proceedings of the Tenth International Conference on Language Resources and Evaluation.

  18. Gantman, A. P., & Van Bavel, J. J. (2014). The moral pop-out effect: Enhanced perceptual awareness of morally relevant stimuli. Cognition, 132(1), 22–29.

    PubMed  Google Scholar 

  19. Gantman, A. P., & Van Bavel, J. J. (2016). See for yourself: Perception is attuned to morality. Trends in Cognitive Sciences, 20(2), 76–77.

    PubMed  Google Scholar 

  20. Garten, J., Boghrati, R., Hoover, J., Johnson, K. M., & Dehghani, M. (2016). Morality between the lines: Detecting moral sentiment in text. Proceedings of IJCAI 2016 workshop on Computational Modeling of Attitudes, New York, NY. Retrieved from:

  21. Garten, J., Hoover, J., Johnson, K. M., Boghrati, R., Iskiwitch, C., & Dehghani, M. (2018). Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis. Behavior Research Methods, 50(1), 344–361.

    PubMed  Google Scholar 

  22. Gentzkow, M. (2016). Polarization in 2016. Toulouse Network of Information Technology Whitepaper

  23. Graham, J., & Haidt, J. (2012). The moral foundations dictionary. Available at:

  24. Graham, J., Haidt, J., & Nosek, B. A. (2009). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96, 1029–1046.

    PubMed  Google Scholar 

  25. Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & Ditto, P. H. (2011). Mapping the moral domain. Journal of Personality and Social Psychology, 101(2), 366–385.

    PubMed  PubMed Central  Google Scholar 

  26. Graham, J., Haidt, J., Koleva, S., Motyl, M., Iyer, R., Wojcik, S., & Ditto, P. H. (2012). Moral foundations theory: The pragmatic validity of moral pluralism. Advances in Experimental Social Psychology, 47, 55–130.

    Google Scholar 

  27. Gray, K., & Keeney, J. E. (2015). Disconfirming moral foundations theory on its own terms: Reply to Graham (2015). Social Psychological and Personality Science, 6(8), 874–877.

    Google Scholar 

  28. Gray, K., & Wegner, D. M. (2009). Moral typecasting: Divergent perceptions of moral agents and moral patients. Journal of Personality and Social Psychology, 96(3), 505–520.

    PubMed  Google Scholar 

  29. Gray, K., & Wegner, D. M. (2011). Dimensions of moral emotions. Emotion Review, 3(3), 258–260.

    Google Scholar 

  30. Gray, K., Waytz, A., & Young, L. (2012). The moral dyad: A fundamental template unifying moral judgment. Psychological Inquiry, 23(2), 206–215.

    PubMed  PubMed Central  Google Scholar 

  31. Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267–297.

    Google Scholar 

  32. Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4), 814–834.

    PubMed  Google Scholar 

  33. Haidt, J. (2007). The new synthesis in moral psychology. Science, 316(5827), 998–1002.

    PubMed  Google Scholar 

  34. Haidt, J. (2012). The righteous mind: Why good people are divided by politics and religion. New York, NY: Vintage Books

    Google Scholar 

  35. Haselmayer, M., & Jenny, M. (2014). Measuring the tonality of negative campaigning: Combining a dictionary approach with crowd-coding. Paper presented at political context Matters: Content analysis in the social sciences, Mannheim, Germany.

  36. Haselmayer, M., & Jenny, M. (2016). Sentiment analysis of political communication: Combining a dictionary approach with crowdcoding. Quality & Quantity, 1–24.

  37. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.

    PubMed  Google Scholar 

  38. Hoover, J., Johnson, K., Boghrati, R., Graham, J., & Dehghani, M. (2018). Moral framing and charitable donation: Integrating exploratory social media analyses and confirmatory experimentation. Collabra: Psychology, 4(1).

  39. Hopp, F. R., Barel, A., Fisher, J., Cornell, D., Lonergan, C., & Weber, R. (2019a). “I believe that morality is gone”: A large-scale inventory of moral foundations in lyrics of popular songs. Paper submitted to the annual meeting of the International Communication Association (ICA), Washington DC, USA.

  40. Hopp, F. R., Schaffer, J., Fisher, J. T., & Weber, R. (2019b). iCoRe: The GDELT interface for the advancement of communication research. Computational Communication Research, 1(1), 13–44.

    Google Scholar 

  41. Hopp, F. R., Fisher, J., & Weber, R. (2020). A computational approach for learning moral conflicts from movie scripts. Paper submitted to the annual meeting of the International Communication Association (ICA), Goldcoast, Queensland, Australia.

  42. Huskey, R., Bowman, N., Eden, A., Grizzard, M., Hahn, L., Lewis, R., Matthews, N., Tamborini, R., Walther, J.B., Weber, R. (2018). Things we know about media and morality. Nature Human Behavior, 2, 315.

    Google Scholar 

  43. Hutto, C. J., & Gilbert, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth international AAAI conference on weblogs and social media.

  44. Jang, S. M., & Hart, P. S. (2015). Polarized frames on “climate change” and “global warming” across countries and states: Evidence from Twitter big data. Global Environmental Change, 32, 11-17.

    Google Scholar 

  45. Kim, J. S., Greene, M. J., Zlateski, A., Lee, K., Richardson, M., Turaga, S. C., & Seung, H. S. (2014). Space-time wiring specificity supports direction selectivity in the retina. Nature, 509(7500), 331–336.

  46. Koenigs, M., Young, L., Adolphs, R., Tranel, D., Cushman, F., Hauser, M., & Damasio, A. (2007). Damage to the prefrontal cortex increases utilitarian moral judgements. Nature, 446(7138), 908.

    PubMed  PubMed Central  Google Scholar 

  47. Koleva, S., Graham, J., Haidt, J., Iyer, R., & Ditto, P. H. (2012). Tracing the threads: How five moral concerns (especially purity) help explain culture war attitudes. Journal of Research in Personality, 46, 184–194.

    Google Scholar 

  48. Leetaru, K., & Schrodt, P. A. (2013a). Gdelt: Global data on events, location, and tone, 1979–2012. ISA Annual Convention, 2(4), 1–49.

    Google Scholar 

  49. Leetaru, K., & Schrodt, P. A, (2013b). GDELT: Global data on events, location and tone, 1979-2012. Paper presented at the International Studies Association Meeting, San Francisco. CA. Retrieved from

  50. Levy, N. (2006). The wisdom of the pack. Philosophical Explorations 9(1):99–103

  51. Lind, F., Gruber, M., & Boomgaarden, H. G. (2017). Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures, 11(3), 191–209.

    PubMed  PubMed Central  Google Scholar 

  52. Luttrell, A., Philipp-Muller, A., & Petty, R. E. (2019). Challenging moral attitudes with moral messages. Psychological Science, 0956797619854706.

  53. May, J. (2018). Regard for reason in the moral mind. Oxford University Press.

  54. Mooijman, M., Hoover, J., Lin, Y., Ji, H., & Dehghani, M. (2018). Moralization in social networks and the emergence of violence during protests. Nature Human Behaviour, 389–396.

  55. Morgan, G. S., Skitka, L. J., & Wisneski, D. C. (2010). Moral and religious convictions and intentions to vote in the 2008 presidential election. Analyses of Social Issues and Public Policy, 10(1), 307–320.

    Google Scholar 

  56. Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.

    Google Scholar 

  57. Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, 71(2001), 2001.

    Google Scholar 

  58. Rezapour, R., Shah, S. H., & Diesner, J. (2019, June). Enhancing the measurement of social effects by capturing morality. In Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (pp. 35-45).

  59. Sagi, E., & Dehghani, M. (2014). Measuring moral rhetoric in text. Social Science Computer Review, 32(2), 132–144.

    Google Scholar 

  60. Schein, C. (2020). The Importance of Context in Moral Judgments. Perspectives on Psychological Science, 15(2), 207–15.

    PubMed  Google Scholar 

  61. Scheufele, D. A. (2000). Agenda-setting, priming, and framing revisited: Another look at cognitive effects of political communication. Mass Communication & Society, 3, 297–316

    Google Scholar 

  62. Strimling, P., Vartanova, I., Jansson, F., & Eriksson, K. (2019). The connection between moral positions and moral arguments drives opinion change. Nature Human Behaviour, 1.

  63. Tamborini, R. (2011). Moral intuition and media entertainment. Journal of Media Psychology, 23, 39-45.

    Article  Google Scholar 

  64. Tamborini, R., & Weber, R. (2019). Advancing the model of intuitive morality and exemplars. In K. Floyd & R. Weber (Eds.), Communication Science and Biology. New York, NY: Routledge. [page numbers coming soon].

  65. Tamborini, R., Lewis, R. J., Prabhu, S., Grizzard, M., Hahn, L., & Wang, L. (2016a). Media’s influence on the accessibility of altruistic and egoistic motivations. Communication Research Reports, 33(3), 177–187.

    Google Scholar 

  66. Tamborini, R., Prabhu, S., Lewis, R. L., Grizzard, M. & Eden, A. (2016b). The influence of media exposure on the accessibility of moral intuitions. Journal of Media Psychology, 1–12.

  67. Van Leeuwen, F., Park, J. H., Koenig, B. L., & Graham, J. (2012). Regional variation in pathogen prevalence predicts endorsement of group-focused moral concerns. Evolution and Human Behavior, 33(5), 429–437.

    Google Scholar 

  68. Weber, R., Mangus, J. M., Huskey, R., Hopp, F. R., Amir, O., Swanson, R., … Tamborini, R. (2018). Extracting latent moral information from text narratives: Relevance, challenges, and solutions. Communication Methods and Measures, 12(2-3), 119–139.

    Google Scholar 

  69. Wolsko, C., Ariceaga, H., & Seiden, J. (2016). Red, white, and blue enough to be green: Effects of moral framing on climate change attitudes and conservation behaviors. Journal of Experimental Social Psychology, 65, 7–19.

    Google Scholar 

  70. Zhang, Y., Jin, R., & Zhou, Z. H. (2010). Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 1(1–4), 43–52.

Download references

Author information



Corresponding author

Correspondence to René Weber.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hopp, F.R., Fisher, J.T., Cornell, D. et al. The extended Moral Foundations Dictionary (eMFD): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behav Res (2020).

Download citation


  • Crowd-sourced dictionary construction
  • Methodological innovation
  • Moral intuition
  • Computational social science
  • Open data
  • Open materials