Behavior Research Methods

, Volume 50, Issue 1, pp 392–405 | Cite as

Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions

  • David Rodrigues
  • Marília Prada
  • Rui Gaspar
  • Margarida V. Garrido
  • Diniz Lopes


The use of emoticons and emoji is increasingly popular across a variety of new platforms of online communication. They have also become popular as stimulus materials in scientific research. However, the assumption that emoji/emoticon users’ interpretations always correspond to the developers’/researchers’ intended meanings might be misleading. This article presents subjective norms of emoji and emoticons provided by everyday users. The Lisbon Emoji and Emoticon Database (LEED) comprises 238 stimuli: 85 emoticons and 153 emoji (collected from iOS, Android, Facebook, and Emojipedia). The sample included 505 Portuguese participants recruited online. Each participant evaluated a random subset of 20 stimuli for seven dimensions: aesthetic appeal, familiarity, visual complexity, concreteness, valence, arousal, and meaningfulness. Participants were additionally asked to attribute a meaning to each stimulus. The norms obtained include quantitative descriptive results (means, standard deviations, and confidence intervals) and a meaning analysis for each stimulus. We also examined the correlations between the dimensions and tested for differences between emoticons and emoji, as well as between the two major operating systems—Android and iOS. The LEED constitutes a readily available normative database (available at with potential applications to different research domains.


LEED Emoticons Emoji Aesthetic appeal Familiarity Visual complexity Concreteness Valence Arousal Meaningfulness Meaning analysis Normative ratings Android iOS Facebook ICTs 


Author note

Part of this research was funded by grants from the Fundação para a Ciência e Tecnologia awarded to the first (SFRH/BPD/73528/2010), third (UID/PSI/04810/2013), and fourth (PTDC/MHC-PCN/5217/2014) authors, and by a Marie Curie fellowship (FP7-PEOPLE-2013-CIG/631673) awarded to the fourth author. We thank Nuno Porto for his assistance in preparing the figures.

Supplementary material

13428_2017_878_MOESM1_ESM.pdf (309 kb)
ESM 1 (PDF 308 kb)


  1. Antheunis, M. L., Valkenburg, P. M., & Peter, J. (2007). Computer-mediated communication and interpersonal attraction: An experimental test of two explanatory hypotheses. Cyber Psychology and Behavior, 10, 831–836. doi: 10.1089/cpb.2007.9945 CrossRefGoogle Scholar
  2. Antheunis, M. L., Valkenburg, P. M., & Peter, J. (2010). Getting acquainted through social network sites: Testing a model of online uncertainty reduction and social attraction. Computers in Human Behavior, 26, 100–109. doi: 10.1016/j.chb.2009.07.005 CrossRefGoogle Scholar
  3. Beltrone, G. (2015). Everyone is an emoji in this bizarre and terrifying French McDonald’s ad. AdWeek. Retrieved from
  4. Blechert, J., Meule, A., Busch, N. A., & Ohla, K. (2014). Food-pics: An image database for experimental research on eating and appetite. Frontiers in Psychology, 5, 617. doi: 10.3389/fpsyg.2014.00617 CrossRefPubMedPubMedCentralGoogle Scholar
  5. Bonin, P., Peereman, R., Malardier, N., Méot, A., & Chalard, M. (2003). A new set of 299 pictures for psycholinguistic studies: French norms for name agreement, image agreement, conceptual familiarity, visual complexity, image variability, age of acquisition, and naming latencies. Behavior Research Methods, Instruments, & Computers, 35, 158–167. doi: 10.3758/BF03195507 CrossRefGoogle Scholar
  6. Bradley, M. M., & Lang, P. J. (1999a). Affective Norms for English Words (ANEW): stimuli, instruction manual, and affective ratings (Technical Report C-1). Gainesville: University of Florida, Center for Research in Psychophysiology.Google Scholar
  7. Bradley, M. M., & Lang, P. J. (1999b). International Affective Digitized Sounds (IADS): Stimuli, instruction manual and affective ratings (Technical Report B-2). Gainesville: University of Florida, Center for Research in Psychophysiology.Google Scholar
  8. Burnap, P., Gibson, R., Sloan, L., Southern, R., & Williams, M. (2016). 140 characters to victory? using twitter to predict the UK 2015 general election. Electoral Studies, 41, 230–233. doi: 10.1016/j.electstud.2015.11.017 CrossRefGoogle Scholar
  9. Carvalho, P., Sarmento, L., Silva, M. J., & de Oliveira, E. (2009). Clues for detecting irony in user-generated contents: Oh…‼ It’s “so easy”; -). In Proceedings of the 1st international CIKM workshop on topic–sentiment analysis for mass opinion (pp. 53–56). New York: ACM Press. doi: 10.1145/1651461.1651471 CrossRefGoogle Scholar
  10. Chambers, C. T., & Craig, K. D. (1998). An intrusive impact of anchors in children’s faces pain scales. Pain, 78, 27–37. doi: 10.1016/S0304-3959(98)00112-2 CrossRefPubMedGoogle Scholar
  11. Charbonnier, L., van Meer, F., van der Laan, L. N., Viergever, M. A., & Smeets, P. A. M. (2016). Standardized food images: A photographing protocol and image database. Appetite, 96, 166–173. doi: 10.1016/j.appet.2015.08.041 CrossRefPubMedGoogle Scholar
  12. Comesaña, M., Soares, A. P., Perea, M., Piñeiro, A. P., Fraga, I., & Pinheiro, A. (2013). ERP correlates of masked affective priming with emoticons. Computers in Human Behavior, 29, 588–595. doi: 10.1016/j.chb.2012.10.020 CrossRefGoogle Scholar
  13. Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva Affective Picture Database (GAPED): A new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 43, 468. doi: 10.3758/s13428-011-0064-1 CrossRefPubMedGoogle Scholar
  14. Davidov, D., Tsur, O., & Rappoport, A. (2010). Enhanced sentiment learning using Twitter hashtags and smileys. In Proceedings of the 23rd international conference on computational linguistics (pp. 241–249). Stroudsburg: Association for Computational Linguistics. Retrieved from Google Scholar
  15. Derks, D., Bos, A. E. R., & von Grumbkow, J. (2008). Emoticons and online message interpretation. Social Science Computer Review, 26, 379–388. doi: 10.1177/0894439307311611 CrossRefGoogle Scholar
  16. Dresner, E., & Herring, S. C. (2010). Functions of the nonverbal in CMC: Emoticons and illocutionary force. Communication Theory, 20, 249–268. doi: 10.1111/j.1468-2885.2010.01362.x CrossRefGoogle Scholar
  17. Ebner, N. C., Riediger, M., & Lindenberger, U. (2010). FACES—A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior Research Methods, 42, 351–362. doi: 10.3758/BRM.42.1.351 CrossRefPubMedGoogle Scholar
  18. Eysenbach, G. (2011). Infodemiology and infoveillance: Tracking online health information and cyberbehavior for public health. American Journal of Preventive Medicine, 40, S154–S158. doi: 10.1016/j.amepre.2011.02.006 CrossRefPubMedGoogle Scholar
  19. Fane, J., MacDougall, C., Jovanovic, J., Redmond, G., & Gibbs, L. (2016). Exploring the use of emoji as a visual research method for eliciting young children’s voices in childhood research. Early Child Development and Care. doi: 10.1080/03004430.2016.1219730 Google Scholar
  20. Fullwood, C., Orchard, L. J., & Floyd, S. A. (2013). Emoticon convergence in internet chat rooms. Social Semiotics, 23, 648–662. doi: 10.1080/10350330.2012.739000 CrossRefGoogle Scholar
  21. Ganster, T., Eimler, S. C., & Krämer, N. C. (2012). Same same but different!? the differential influence of smilies and emoticons on person perception. Cyber Psychology, Behavior, and Social Networking, 15, 226–230. doi: 10.1089/cyber.2011.0179 CrossRefGoogle Scholar
  22. Garcia-Marques, T., Mackie, D. M., Claypool, H. M., & Garcia-Marques, L. (2004). Positivity can cue familiarity. Personality and Social Psychology Bulletin, 30, 585–593. doi: 10.1177/0146167203262856 CrossRefPubMedGoogle Scholar
  23. Garrido, M. V., Lopes, D., Prada, M., Rodrigues, D., Jerónimo, R., & Mourão, R. P. (2016). The many faces of a face: Comparing stills and videos of facial expressions in eight dimensions (SAVE database). Behavior Research Methods. doi: 10.3758/s13428-016-0790-5 PubMedGoogle Scholar
  24. Gaspar, R., Barnett, J., & Seibt, B. (2015). Crisis as seen by the individual: The norm deviation approach. Psyecology, 6, 103–135. doi: 10.1080/21711976.2014.1002205 CrossRefGoogle Scholar
  25. Gaspar, R., Pedro, C., Panagiotopoulos, P., & Seibt, B. (2016). Beyond positive or negative: Qualitative sentiment analysis of social media reactions to unexpected stressful events. Computers in Human Behavior, 56, 179–191. doi: 10.1016/j.chb.2015.11.040 CrossRefGoogle Scholar
  26. Gülşen, T. T. (2016). You tell me in emojis. In O. Ogata & T. Akimoto (Eds.), Computational and cognitive approaches to narratology (pp. 354–375). Hershey: Information Science Reference.Google Scholar
  27. Han, D. H., Yoo, H. J., Kim, B. N., McMahon, W., & Renshaw, P. F. (2014). Brain activity of adolescents with high functioning autism in response to emotional words and facial emoticons. PLoS ONE, 9, 1–8. doi: 10.1371/journal.pone.0091214 Google Scholar
  28. Hogenboom, A., Bal, D., Frasincar, F., Bal, M., de Jong, F., & Kaymak, U. (2013). Exploiting emoticons in sentiment analysis. In Proceedings of the 28th annual ACM symposium on applied computing (pp. 703–710). New York: ACM Press. doi: 10.1145/2480362.2480498 Google Scholar
  29. Huang, A. H., Yen, D. C., & Zhang, X. (2008). Exploring the potential effects of emoticons. Information Management, 45, 466–473. doi: 10.1016/ CrossRefGoogle Scholar
  30. Jaeger, S. R., Vidal, L., Kam, K., & Ares, G. (2017). Can emoji be used as a direct method to measure emotional associations to food names? Preliminary investigations with consumers in USA and China. Food Quality and Preference, 56, 38–48. doi: 10.1016/j.foodqual.2016.09.005 CrossRefGoogle Scholar
  31. Kaye, L. K., Wall, H. J., & Malone, S. A. (2016). “Turn that frown upside-down”: A contextual account of emoticon usage on different virtual platforms. Computers in Human Behavior, 60, 463–467. doi: 10.1016/j.chb.2016.02.088 CrossRefGoogle Scholar
  32. Kerkhof, I., Goesaert, E., Dirikx, T., Vansteenwegen, D., Baeyens, F., D’Hooge, R., & Hermans, D. (2009). Assessing valence indirectly and online. Cognition and Emotion, 23, 1615–1629. doi: 10.1080/02699930802469239 CrossRefGoogle Scholar
  33. Krohn, F. B. (2004). A generational approach to using emoticons as nonverbal communication. Journal of Technical Writing and Communication, 34, 321–328. doi: 10.2190/9eqh-de81-cwg1-qll9 CrossRefGoogle Scholar
  34. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (2008). International Affective Picture System (IAPS): Affective ratings of pictures and instruction manual (Technical Report A-8). Gainesville: University of Florida.Google Scholar
  35. Lea, M., & Spears, R. (1992). Paralanguage and social perception in computer‐mediated communication. Journal of Organizational Communication, 2, 321–341. doi: 10.1080/10919399209540190 CrossRefGoogle Scholar
  36. Liu, B. (2012). Sentiment analysis and opinion mining. San Rafael: Morgan & Claypool.Google Scholar
  37. Liu, K.-L., Li, W.-J., & Guo, M. (2012). Emoticon smoothed language models for twitter sentiment analysis. In Proceedings of the 26th AAAI conference on artificial intelligence (pp. 1678–1684). New York: AAAI Press.Google Scholar
  38. Ljubešić, N., & Fišer, D. (2016). A global analysis of emoji usage. In proceedings of the 10th Web as corpus workshop (WAC-X) and the EmpiriST shared task (pp. 82–89). Stroudsburg: Association for Computational Linguistics.Google Scholar
  39. Lo, S.-K. (2008). The nonverbal communication functions of emoticons in computer-mediated communication. Cyber Psychology and Behavior, 11, 595–597. doi: 10.1089/cpb.2007.0132 CrossRefGoogle Scholar
  40. McDougall, S. J. P., Curry, M. B., & de Bruijn, O. (1999). Measuring symbol and icon characteristics: Norms for concreteness, complexity, meaningfulness, familiarity, and semantic distance for 239 symbols. Behavior Research Methods, Instruments, & Computers, 31, 487–519. doi: 10.3758/BF03200730 CrossRefGoogle Scholar
  41. Mendonça, R., Garrido, M. V., & Semin, G. R. (2016). A standardized database for Portuguese faces looking towards left, right and the front: The LRF Face Corpus. Manuscript submitted for publication.Google Scholar
  42. Milinovich, G. J., Williams, G. M., Clements, A. C. A., & Hu, W. (2014). Internet-based surveillance systems for monitoring emerging infectious diseases. Lancet Infectious Diseases, 14, 160–168. doi: 10.1016/S1473-3099(13)70244-5 CrossRefPubMedGoogle Scholar
  43. Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., & Hecht, B. (2016). “Blissfully happy” or “ready to fight”: varying interpretations of emoji. In International Conference on Web and Social Media (ICWSM) ’16 (pp. 259–268). New York: AAAI Press.Google Scholar
  44. Moore, A., Steiner, C. M., & Conlan, O. (2013). Design and development of an empirical smiley-based affective instrument. In 1st workshop on emotions and personality in personalized services (pp. 24–30). Rome: UMAP.Google Scholar
  45. Neff, J. (2015). Dove launches curly-haired emojis to end straight-hair dominance. Advertising Age. Retrieved from
  46. Negishi, M. (2014). Meet Shigetaka Kurita, the father of emoji. Wall Street Journal. Retrieved from
  47. Nelson, R. A., Tossell, C. C., & Kortum, P. (2015). Emoticon use in mobile communications :-). In Z. Yan (Ed.), Encyclopedia of mobile phone behavior (pp. 1–11). Hershey: IGI Global.Google Scholar
  48. Novak, P. K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentiment of emojis. PLoS ONE, 10, e0144296. doi: 10.1371/journal.pone.0144296 CrossRefGoogle Scholar
  49. Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2, 1–135. doi: 10.1561/1500000011 CrossRefGoogle Scholar
  50. Park, J., Baek, Y. M., & Cha, M. (2014). Cross-cultural comparison of nonverbal cues in emoticons on twitter: Evidence from big data analysis. Journal of Communication, 64, 333–354. doi: 10.1111/jcom.12086 CrossRefGoogle Scholar
  51. Paul, M. J., & Dredze, M. (2011). You are what you tweet: analyzing twitter for public health. In Fifth international AAAI conference on weblogs and social media (pp. 265–272). New York: AAAI Press.Google Scholar
  52. Pavalanathan, U., & Eisenstein, J. (2015). Emoticons vs. emojis on Twitter: A causal inference approach. arXiv:1510.08480 [cs.CL].Google Scholar
  53. PEW Research Center. (2016). Smartphone ownership and Internet usage continues to climb in emerging economies. Retrieved from
  54. Prada, M., Rodrigues, D., Silva, R. R., & Garrido, M. V. (2015). Lisbon Symbol Database (LSD): Subjective norms for 600 symbols. Behavior Research Methods, 48, 1370–1382. doi: 10.3758/s13428-015-0643-7 CrossRefGoogle Scholar
  55. Proctor, R. W., & Vu, K.-P. L. (1999). Index of norms and ratings published in the psychonomic society journals. Behavior Research Methods, Instruments, & Computers, 31, 659–667. doi: 10.3758/BF03200742 CrossRefGoogle Scholar
  56. Richmond, V., & McCroskey, J. (2009). Human communication theory and research: Traditions and models. In D. Stacks & M. Salwen (Eds.), An integrated approach to communication theory and research (2nd ed., pp. 223–231). New York: Routledge.Google Scholar
  57. Siegel, R. M., Anneken, A., Duffy, C., Simmons, K., Hudgens, M., Lockhart, M. K., & Shelly, J. (2015). Emoticon use increases plain milk and vegetable purchase in a school cafeteria without adversely affecting total milk purchase. Clinical Therapeutics, 37, 1938–1943. doi: 10.1016/j.clinthera.2015.07.016 CrossRefPubMedGoogle Scholar
  58. Skiba, D. J. (2016). Face with tears of joy is word of the year: Are emoji a sign of things to come in health care? Nursing Education Perspectives, 37, 56–57. doi: 10.1097/01.NEP.0000476112.24899.a1 CrossRefPubMedGoogle Scholar
  59. Skovholt, K., Grønning, A., & Kankaanranta, A. (2014). The communicative functions of emoticons in workplace e-mails: :-). Journal of Computer-Mediated Communication, 19, 780–797. doi: 10.1111/jcc4.12063 CrossRefGoogle Scholar
  60. Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning and Memory, 6, 174–215. doi: 10.1037/0278-7393.6.2.174 Google Scholar
  61. Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010). Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61, 2544–2558. doi: 10.1002/asi.21416 CrossRefGoogle Scholar
  62. Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, 63, 163–173. doi: 10.1002/asi.21662 CrossRefGoogle Scholar
  63. Thompson, D., & Filik, R. (2016). Sarcasm in written communication: Emoticons are efficient markers of intention. Journal of Computer-Mediated Communication, 21, 105–120. doi: 10.1111/jcc4.12156 CrossRefGoogle Scholar
  64. Tossell, C. C., Kortum, P., Shepard, C., Barg-Walkow, L. H., Rahmati, A., & Zhong, L. (2012). A longitudinal study of emoticon use in text messaging from smartphones. Computers in Human Behavior, 28, 659–663. doi: 10.1016/j.chb.2011.11.012 CrossRefGoogle Scholar
  65. Tung, F.-W., & Deng, Y.-S. (2007). Increasing social presence of social actors in e-learning environments: Effects of dynamic and static emoticons on children. Displays, 28, 174–180. doi: 10.1016/j.displa.2007.06.005 CrossRefGoogle Scholar
  66. Vaiman, M., Wagner, M. A., Caicedo, E., & Pereno, G. L. (2017). Development and validation of an argentine set of facial expressions of emotion. Cognition and Emotion, 31, 249–260. doi: 10.1080/02699931.2015.1098590 CrossRefPubMedGoogle Scholar
  67. Vashisht, G., & Thakur, S. (2014). Facebook as a corpus for emoticons-based sentiment analysis. International Journal of Emerging Technology and Advanced Engineering, 4, 904–908.Google Scholar
  68. Vidal, L., Ares, G., & Jaeger, S. R. (2016). Use of emoticon and emoji in tweets for food-related emotional expression. Food Quality and Preference, 49, 119–128. doi: 10.1016/j.foodqual.2015.12.002 CrossRefGoogle Scholar
  69. Walther, J. B. (1996). Computer-mediated communication: Impersonal, interpersonal, and hyperpersonal interaction. Communication Research, 23, 3–43. doi: 10.1177/009365096023001001 CrossRefGoogle Scholar
  70. Walther, J. B., & D’Addario, K. P. (2001). The impacts of emoticons on message interpretation in computer-mediated communication. Social Science Computer Review, 19, 324. doi: 10.1177/089443930101900307 CrossRefGoogle Scholar
  71. Wang, H., & Castanon, J. A. (2015). Sentiment expression via emoticons on social media. Retrieved from arXiv:1511.02556Google Scholar
  72. Wang, W., Zhao, Y., Qiu, L., & Zhu, Y. (2014). Effects of emoticons on the acceptance of negative feedback in computer-mediated communication. Journal of the Association for Information Systems, 15(8), 3. Retrieved from CrossRefGoogle Scholar
  73. Wiebe, J., Wilson, T., & Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39, 165–210. doi: 10.1007/s10579-005-7880-9 CrossRefGoogle Scholar
  74. Wohl, J. (2016, August 16). How marketers can win the great emoji arms race. Advertising Age. Retrieved from
  75. Wolff, J. S., & Wogalter, M. S. (1998). Comprehension of pictorial symbols: Effects of context and test method. Human Factors, 40, 173–186. doi: 10.1518/001872098779480433 CrossRefGoogle Scholar
  76. Yuasa, M., Saito, K., & Mukawa, N. (2011). Brain activity associated with graphic emoticons. The effect of abstract faces in communication over a computer network. Electrical Engineering in Japan, 177, 36–45. doi: 10.1002/eej.21162 CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  1. 1.Department of Social and Organizational PsychologyInstituto Universitário de Lisboa (ISCTE-IUL), CIS-IULLisbonPortugal
  2. 2.GoldsmithsUniversity of LondonLondonUK
  3. 3.William James Center for ResearchISPA - Instituto UniversitáriovLisbonPortugal

Personalised recommendations