A comparison of homonym meaning frequency estimates derived from movie and television subtitles, free association, and explicit ratings

  • Caitlin A. Rice
  • Barend Beekhuizen
  • Vladimir Dubrovsky
  • Suzanne Stevenson
  • Blair C. Armstrong


Most words are ambiguous, with interpretation dependent on context. Advancing theories of ambiguity resolution is important for any general theory of language processing, and for resolving inconsistencies in observed ambiguity effects across experimental tasks. Focusing on homonyms (words such as bank with unrelated meanings EDGE OF A RIVER vs. FINANCIAL INSTITUTION), the present work advances theories and methods for estimating the relative frequency of their meanings, a factor that shapes observed ambiguity effects. We develop a new method for estimating meaning frequency based on the meaning of a homonym evoked in lines of movie and television subtitles according to human raters. We also replicate and extend a measure of meaning frequency derived from the classification of free associates. We evaluate the internal consistency of these measures, compare them to published estimates based on explicit ratings of each meaning’s frequency, and compare each set of norms in predicting performance in lexical and semantic decision mega-studies. All measures have high internal consistency and show agreement, but each is also associated with unique variance, which may be explained by integrating cognitive theories of memory with the demands of different experimental methodologies. To derive frequency estimates, we collected manual classifications of 533 homonyms over 50,000 lines of subtitles, and of 357 homonyms across over 5000 homonym–associate pairs. This database—publicly available at:—constitutes a novel resource for computational cognitive modeling and computational linguistics, and we offer suggestions around good practices for its use in training and testing models on labeled data.


Semantic ambiguity Homonyms Meaning frequency Homonym norming methods and data Movie subtitles Free association Homonym meaning annotations 


  1. Agirre, E., & Edmonds, P. (Eds.) (2007). Word sense disambiguation: Algorithms and applications. Dordrecht: Springer Science & Business Media.Google Scholar
  2. Armstrong, B.C., & Plaut, D.C. (2008). Settling dynamics in distributed networks explain task differences in semantic ambiguity effects: Computational and behavioral evidence. In B.C. Love, K. McRae, & V.M. Sloutsky (Eds.) Proceedings of the 30th annual conference of the cognitive science society (pp. 273–278). Austin, TX: Cognitive Science Society.Google Scholar
  3. Armstrong, B.C., & Plaut, D.C. (2016). Disparate semantic ambiguity effects from semantic processing dynamics rather than qualitative task differences. Language, Cognition, and Neuroscience, 31, 940–966. CrossRefGoogle Scholar
  4. Armstrong, B.C., Tokowicz, N., & Plaut, D.C. (2012). eDom: Norming software and relative meaning frequencies for 544 English homonyms. Behavior Research Methods, 44, 1015–1027. CrossRefPubMedGoogle Scholar
  5. Armstrong, B.C., Watson, C.E., & Plaut, D.C. (2012). SOS: An algorithm and software for the stochastic optimization of stimuli. Behavior Research Methods, 44, 675–705. CrossRefPubMedGoogle Scholar
  6. Armstrong, B.C., Zugarramurdi, C., Cabana, A., Valle Lisboa, J., & Plaut, D.C. (2015). Relative meaning frequencies for 578 homonyms in two Spanish dialects: A cross-linguistic extension of the English edom norms. Behavior Research Methods, 1–13.
  7. Balota, D.A., Yap, M.J., Cortese, M.J., Hutchison, K.A., Kessler, B., Loftis, B., ..., Treiman, R. (2007). The English lexicon project. Behavior Research Methods, 39(3), 445–449.CrossRefPubMedGoogle Scholar
  8. Barak, L., Goldberg, A., & Stevenson, S. (2016). Comparing computational cognitive models of generalization in a language acquisition task. In Proceedings of the conference on empirical methods in natural language processing (EMNLP 2016). Austin, TX.Google Scholar
  9. Bartunov, S., Kondrashkin, D., Osokin, A., & Vetrov, D. (2016). Breaking sticks and ambiguities with adaptive skip-gram. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 51, 130–138.Google Scholar
  10. Bennett, A., Baldwin, T., Lau, J.H., McCarthy, D., & Bond, F. (2016). LexSemTm: A semantic dataset based on all-words unsupervised sense distribution learning. In Proceedings of the 54th annual meeting of the association for computational linguistics (ACL) (pp. 1513–1524). Association for Computational Linguistics.
  11. Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4), 977–990. CrossRefPubMedGoogle Scholar
  12. Caramazza, A., & Hillis, A.E. (1991). Lexical organization of nouns and verbs in the brain. Nature, 349 (6312), 788.CrossRefPubMedGoogle Scholar
  13. Col, G., Aptekman, J., Girault, S., & Poibeau, T. (2012). Gestalt compositionality and instruction-based meaning construction. Cognitive Processing, 13(2), 151–170.CrossRefPubMedGoogle Scholar
  14. Coltheart, M., Davelaar, E., Jonasson, J.T., & Besner, D. (1977). Access to the internal lexicon. In S. Dornic (Ed.) Attention and performance VI (pp. 535–555). Hillsdale: Erlbaum.Google Scholar
  15. Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2000). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108(1), 204–256.CrossRefGoogle Scholar
  16. Eisele, A., & Chen, Y. (2010). Multiun: A multilingual corpus from United Nation documents. In The 7th international conference on language resources and evaluation.Google Scholar
  17. Fellbaum, C. (1998) WordNet. New York: Wiley.Google Scholar
  18. Forster, K.I. (2000). The potential for experimenter bias effects in word recognition experiments. Memory and Cognition, 28(7), 1109–1115.CrossRefPubMedGoogle Scholar
  19. Frazier, L., & Rayner, K. (1990). Taking on semantic commitments: Processing multiple meanings vs. multiple senses. Journal of Memory and Language, 29(2), 181–200. CrossRefGoogle Scholar
  20. Frost, R. (2012). A universal approach to modeling visual word recognition and reading: Not only possible, but also inevitable. Behavioral and Brain Sciences, 35(5), 310–329. CrossRefPubMedGoogle Scholar
  21. Frost, R., Armstrong, B.C., Siegelman, N., & Christiansen, M.H. (2015). Domain generality vs. modality specificity: The paradox of statistical learning. Trends in Cognitive Sciences, 19, 117–125.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Gale, W.A., Church, K.W., & Yarowsky, D. (1992). One sense per discourse. In Proceedings of the workshop on speech and natural language (pp. 233–237).Google Scholar
  23. Gernsbacher, M. A. (1984). Resolving 20 years of inconsistent interactions between lexical familiarity and orthography, concreteness, and polysemy. Journal of Experimental Psychology General, 113, 256–281. CrossRefPubMedPubMedCentralGoogle Scholar
  24. Griffiths, T.L., Steyvers, M., & Tenenbaum, J.B. (2007). Topics in semantic representation. Psychological Review, 114(2), 211–244.CrossRefPubMedGoogle Scholar
  25. Hastie, T., Tibshirani, R., & Friedman, J. (2001) The elements of statistical learning. New York: Springer.CrossRefGoogle Scholar
  26. Hino, Y., Kusunose, Y., & Lupker, S.J. (2010). The relatedness-of-meaning effect for ambiguous words in lexical-decision tasks: when does relatedness matter? Canadian Journal of Experimental Psychology, 64(3), 180–196. CrossRefPubMedGoogle Scholar
  27. Hino, Y., Pexman, P.M., & Lupker, S.J. (2006). Ambiguity and relatedness effects in semantic tasks: Are they due to semantic coding? Journal of Memory and Language, 55(2), 247–273. CrossRefGoogle Scholar
  28. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) An introduction to statistical learning: With applications in R. New York: Springer.CrossRefGoogle Scholar
  29. Johns, B.T., & Jones, M.N. (2012). Perceptual inference through global lexical similarity. Topics in Cognitive Science, 4(1), 103–120.CrossRefPubMedGoogle Scholar
  30. Kawamoto, A.H. (1993). Nonlinear dynamics in the resolution of lexical ambiguity: A parallel distributed processing account. Journal of Memory and Language, 32(4), 474–516. CrossRefGoogle Scholar
  31. Kawamoto, A.H., Farrar, W.T., & Kello, C.T. (1994). When two meanings are better than one: Modeling the ambiguity advantage using a recurrent distributed network. Journal of Experimental Psychology: Human Perception and Performance, 20(6), 1233–1247.Google Scholar
  32. Keuleers, E., Lacey, P., Rastle, K., & Brysbaert, M. (2012). The British Lexicon Project: Lexical decision data for 28,730 monosyllabic and disyllabic English words. Behavior Research Methods, 44(1), 287–304.CrossRefPubMedGoogle Scholar
  33. Klein, D.E., & Murphy, G.L. (2001). The representation of polysemous words. Journal of Memory and Language, 45(2), 259–282. CrossRefGoogle Scholar
  34. Klein, D.E., & Murphy, G.L. (2002). Paper has been my ruin: Conceptual relations of polysemous senses. Journal of Memory and Language, 47(4), 548–570.CrossRefGoogle Scholar
  35. Klepousniotou, E. (2002). The processing of lexical ambiguity: Homonymy and polysemy in the mental lexicon. Brain and Language, 81(1-3), 205–223.CrossRefPubMedGoogle Scholar
  36. Klepousniotou, E., & Baum, S.R. (2007). Disambiguating the ambiguity advantage effect in word recognition: An advantage for polysemous but not homonymous words. Journal of Neurolinguistics, 20(1), 1–24. CrossRefGoogle Scholar
  37. Klepousniotou, E., Pike, G. B., Steinhauer, K., & Gracco, V. (2012). Not all ambiguous words are created equal: An EEG investigation of homonymy and polysemy. Brain and Language, 123, 11–21. CrossRefPubMedGoogle Scholar
  38. Klepousniotou, E., Titone, D., & Romero, C. (2008). Making sense of word senses: The comprehension of polysemy depends on sense overlap. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1534–1543. PubMedGoogle Scholar
  39. Koeling, R., McCarthy, D., & Carroll, J. (2005). Domain-specific sense distributions and predominant sense acquisition. In Proceedings of the conference on human language technology and empirical methods in natural language processing (HLT/EMNLP) (pp. 419–426). Stroudsburg, PA, USA: Association for Computational Linguistics.Google Scholar
  40. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th international joint conference on artificial intelligence (IJCAI) (Vol. 2, pp. 1137–1143). San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.Google Scholar
  41. Landauer, T.K., & Dumais, S.T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240.CrossRefGoogle Scholar
  42. Langone, H., Haskell, B.R., & Miller, G.A. (2014). Annotating WordNet. In Workshop on frontiers in corpus annotation (pp. 63–69).Google Scholar
  43. Lau, J.H., Cook, P., McCarthy, D., Gella, S., & Baldwin, T. (2014). Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models. In Proceedings of the 52nd annual meeting of the association for computational linguistics ACL (pp. 259–270). Baltimore, Maryland: Association for Computational Linguistics.Google Scholar
  44. Leacock, C., Miller, G.A., & Chodorow, M. (1998). Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1), 147–165.Google Scholar
  45. Leacock, C., Towell, G., & Voorhees, E. (1993). Corpus-based statistical sense resolution. In Proceedings of the ARPA workshop on human language technology (pp. 260–265).Google Scholar
  46. Lefever, E., & Hoste, V. (2010). Semeval-2010 Task 3: Cross-lingual word sense disambiguation. In Proceedings of the 5th international workshop on semantic evaluation (pp. 15–20).Google Scholar
  47. Li, J., & Jurafsky, D. (2015). Do multi-sense embeddings improve natural language understanding? In L. Marquez, C. Callison-Burch, J. Su, D. Pighin, & Y. Marton (Eds.) Proceedings of the 2015 conference on empirical methods in natural language processing (EMNLP) (pp. 1722–1732). The Association for Computational Linguistics.Google Scholar
  48. Lorge, I. (1937). The english semantic count. Teachers College Record, 39, 65–77.Google Scholar
  49. Luce, R. (2005) Individual choice behavior: A theoretical analysis. Mineola: Dover Publications.CrossRefGoogle Scholar
  50. Maciejewski, G., & Klepousniotou, E. (2016). Relative meaning frequencies for 100 homonyms: British eDom norms. Journal of Open Psychology Data, 4(1), e6. CrossRefGoogle Scholar
  51. McClelland, J.L., & Rumelhart, D.E. (1981). An interactive activation model of the effect of context in perception: Part 1. Psychological Review, 88, 375–407. CrossRefGoogle Scholar
  52. Mirman, D., Strauss, T.J., Dixon, J.A., & Magnuson, J.S. (2010). Effect of representational distance between meanings on recognition of ambiguous spoken words. Cognitive Science, 34(1), 161–173. CrossRefGoogle Scholar
  53. Nelson, D.L., McEvoy, C.L., & Schreiber, T.A. (2004). The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, and Computers, 36(3), 402–407.CrossRefPubMedGoogle Scholar
  54. Nelson, D.L., McEvoy, C.L., Walling, J.R., & Wheeler, J.W. (1980). The University of South Florida homograph norms. Behavior Research Methods, 12(1), 16–37.CrossRefGoogle Scholar
  55. Ng, H. T., & Lee, H. B. (1996). Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In Proceedings of the 34th annual meeting on association for computational linguistics (pp. 40–47).Google Scholar
  56. Palmer, M., Babko-Malaya, O., & Dang, H.T. (2004). Different sense granularities for different applications. In Proceedings of workshop on scalable natural language understanding.Google Scholar
  57. Parks, R., Ray, J., & Bland, S. (1998). Wordsmyth English Dictionary- Thesaurus [Retrieved September 2008 from] (Vol. 1).Google Scholar
  58. Passonneau, R.J., Baker, C., Fellbaum, C., & Ide, N. (2012). The MASC word sense sentence corpus. In Proceedings of the eighth international conference on language resources and evaluation (LREC).Google Scholar
  59. Petrolito, T., & Bond, F. (2014). A survey of WordNet annotated corpora. In Proceedings of the global WordNet conference, GWC-2014 (pp. 236–245).Google Scholar
  60. Pexman, P.M., Heard, A., Lloyd, E., & Yap, M.J. (2017). The Calgary semantic decision project: Concrete/abstract decision data for 10,000 english words. Behavior Research Methods, 49(2), 407–417.CrossRefPubMedGoogle Scholar
  61. Pexman, P.M., Lupker, S.J., & Hino, Y. (2002). The impact of feedback semantics in visual word recognition: Number-of-features effects in lexical decision and naming tasks. Psychonomic Bulletin and Review, 9(3), 542–549. CrossRefPubMedGoogle Scholar
  62. Piercey, C.D., & Joordens, S. (2000). Turning an advantage into a disadvantage: Ambiguity effects in lexical decision versus reading tasks. Memory and Cognition, 28(4), 657–666. CrossRefPubMedGoogle Scholar
  63. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85(2), 59–108.CrossRefGoogle Scholar
  64. Rodd, J.M., Cai, Z.G., Betts, H.N., Hanby, B., Hutchinson, C., & Adler, A. (2016). The impact of recent and long-term experience on access to word meanings: Evidence from large-scale Internet-based experiments. Journal of Memory and Language, 87, 16–37.CrossRefGoogle Scholar
  65. Rodd, J.M., Gaskell, G., & Marslen-Wilson, W. (2002). Making sense of semantic ambiguity: Semantic competition in lexical access. Journal of Memory and Language, 46(2), 245–266. CrossRefGoogle Scholar
  66. Rodd, J.M., Gaskell, M.G., & Marslen-Wilson, W.D. (2004). Modelling the effects of semantic ambiguity in word recognition. Cognitive Science, 28(1), 89–104. CrossRefGoogle Scholar
  67. Rose, T., Stevenson, M., & Whitehead, M. (2002). The Reuters Corpus Volume 1-From yesterday’s news to tomorrow’s language resources. In Proceedings of Language Resources and Evaluation Conference (LREC), (Vol. 2 pp. 827–832).Google Scholar
  68. Seidenberg, M.S., & McClelland, J.L. (1989). A distributed, developmental model of word recognition and naming. Psychological Review, 96(4), 523–568.CrossRefPubMedGoogle Scholar
  69. Swinney, D.A. (1979). Lexical access during sentence comprehension: (Re) consideration of context effects. Journal of Verbal Learning and Verbal Behavior, 18(6), 645–659. CrossRefGoogle Scholar
  70. Swinney, D.A., & Hakes, D.T. (1976). Effects of prior context upon lexical access during sentence comprehension. Journal of Verbal Learning and Verbal Behavior, 15(6), 681–689.CrossRefGoogle Scholar
  71. Tabossi, P. (1988). Accessing lexical ambiguity in different types of sentential contexts. Journal of Memory and Language, 27(3), 324–340. CrossRefGoogle Scholar
  72. Taghipour, K., & Ng, H.T. (2015). One million sense-tagged instances for word sense disambiguation and induction. In Proceedings of the nineteenth conference on computational natural language learning (CoNLL) (pp. 338–344).Google Scholar
  73. Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6(2), 175–184.CrossRefGoogle Scholar
  74. Tversky, A. (1967). Utility theory and additivity analysis of risky choices. Journal of Experimental Psychology, 75(1), 27.CrossRefPubMedGoogle Scholar
  75. Twilley, L.C., Dixon, P., Taylor, D., & Clark, K. (1994). University of Alberta norms of relative meaning frequency for 566 homographs. Memory and Cognition, 22(1), 111–126. CrossRefPubMedGoogle Scholar
  76. Ungemach, C., Chater, N., & Stewart, N. (2009). Are probabilities overweighted or underweighted when rare outcomes are experienced (rarely)? Psychological Science, 20(4), 473–479.CrossRefPubMedGoogle Scholar
  77. Usher, M., & McClelland, J.L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review, 108(3), 550–592. CrossRefPubMedGoogle Scholar
  78. Watson, C. E., Armstrong, B. C., & Plaut, D. C. (2012). Connectionist modeling of neuropsychological deficits in semantics, language and reading. In D. Mostofsky (Ed.) The handbook of the neuropsychology of language.Google Scholar
  79. Williams, J.N. (1992). Processing polysemous words in context: Evidence for interrelated meanings. Journal of Psycholinguistic Research, 21(3), 193–218. CrossRefGoogle Scholar
  80. Yarkoni, T., Balota, D.A., & Yap, M. (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin and Review, 15(5), 971–979. CrossRefPubMedGoogle Scholar
  81. Zlatev, J. (2003). Polysemy or generality? In Mu. In, H. Cuyckens, & B.E. Zawada (Eds.) Polysemy in cognitive linguistics (pp. 447–494). Amsterdam: John Benjamins.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of PittsburghPittsburghUSA
  2. 2.Learning Research and Development CenterUniversity of PittsburghPittsburghUSA
  3. 3.Center for the Neural Basis of CognitionUniversity of PittsburghPittsburghUSA
  4. 4.Department of Computer ScienceUniversity of TorontoTorontoCanada
  5. 5.Department of Psychology and Center for French & Linguistics at ScarboroughUniversity of TorontoTorontoCanada
  6. 6.Basque Center on Cognition, Brain, and LanguageDonostiaSpain

Personalised recommendations