Abstract
The internal validity of several types of experiments in experimental psychology and neuroscience depends in part on the possibility of controlling or manipulating critical lexical variables such as word frequency of occurrence. Two ways of estimating this variable are (1) objective frequency counts and (2) subjective ratings of word frequency. Each method produces estimates that generally agree (i.e., they are highly correlated) but that disagree substantially concerning the relative frequency of a number of words. To investigate this issue more closely, the global and local agreement of subjective frequency estimates was examined in detail for a pool of 6,202 words drawn from the OMNILEX database of French words (Desrochers, 2006; www.omnilex.uottawa .ca). The results indicated that objective and subjective frequencies are strongly correlated, subjective frequencies share a significant amount of bias variance with other lexical characteristics (e.g., imageability), and the codeterminants of subjective frequency are in an antagonistic relationship with one another. The implications of these results for the selection of lexical stimuli are discussed, and multiple variables to aid in item selection are reported. Supplemental materials for this study may be downloaded from brm.psychonomic-journals.org/ content/supplemental.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Adelman, J. S., Brown, G. D. A., & Quesada, J. F. (2006). Contextual diversity, not word frequency, determines word-naming and lexical decision times. Psychological Science, 17, 814–823. doi:10.1037/ a0012887
Allen, P. A., Smith, A. F., Lien, M.-C., Grabbe, J., & Murphy, M. D. (2005). Evidence for an activation locus of the word-frequency effect in lexical decision. Journal of Experimental Psychology: Human Perception & Performance, 31, 713–721. doi:10.1037/0096-1523.31.4.713
Baayen, R. H. (2001). Word frequency distributions. Boston: Kluwer.
Baayen, R. H., Feldman, L. B., & Schreuder, R. (2006). Morphological influences on the recognition of monosyllabic monomorphemic words. Journal of Memory & Language, 55, 290–313. doi:10.1016/j.jml.2006.03.008
Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). The Celex lexical database [CD-ROM-Release 2]. Philadelphia: University of Pennsylvania, Linguistic Data Consortium.
Balota, D. A., & Chumbley, J. (1984). Are lexical decisions a good measure of lexical access? The role of word frequency in the neglected decision stage. Journal of Experimental Psychology: Human Perception & Performance, 10, 340–357. doi:10.1037/0096-1523.10.3.340
Balota, D. A., Cortese, M. J., Sergent-Marshall, S. D., Spieler, D. H., & Yap, M. J. (2004). Visual word recognition of single-syllable words. Journal of Experimental Psychology: General, 133, 283–316. doi:10.1037/0096-3445.133.2.283
Balota, D. A., Ferraro, F. R., & Connor, L. T. (1991). On the early influence of meaning in word recognition: A review of the literature. In P. J. Schwanenflugel (Ed.), The psychology of word meaning (pp. 187–222). Hillsdale, NJ: Erlbaum.
Balota, D. A., Pilotti, M., & Cortese, M. J. (2001). Subjective frequency estimates for 2,938 monosyllabic words. Memory & Cognition, 29, 639–647.
Bauer, D. J., & Curran, P. J. (2005). Probing interactions in fixed and multilevel regression: Inferential and graphical techniques. Multivariate Behavioral Research, 40, 373–400. doi:10.1207/ s15327906mbr4003_5
Benjafield, J., & Muckenheim, R. (1989). Dates of entry and measures of imagery, concreteness, goodness, and familiarity for 1,046 words sampled from the Oxford English Dictionary. Behavior Research Methods, Instruments, & Computers, 21, 31–52.
Bertram, R., & Hyönä, J. (2003). The length of a complex word modifies the role of morphological structure: Evidence from eye movements when reading short and long Finnish compounds. Journal of Memory & Language, 48, 615–634. doi:10.1016/S0749-596X%2802%2900539-9
Bonin, P., Méot, A., Aubert, L., Malardier, N., Niedenthal, P., & Cappell-Toczek, M.-C. (2003). Normes de concrétude, de valeur d’imagerie, de fréquence subjective et de valence émotionnelle pour 866 mots. Année Psychologique, 104, 655–694.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071. doi:10.1037/0033-295X.111.4.1061
Burgess, C., & Livesay, K. (1998). The effect of corpus size in predicting reaction time in a basic word recognition task: Moving on from Kučera and Francis. Behavior Research Methods, Instruments, & Computers, 30, 272–277.
Cohen, P., Cohen, J., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
Coltheart, M., Davelaar, E., Jonasson, J. T., & Besner, D. (1977). Access to the internal lexicon. In S. Dornic (Ed.), Attention and performance VI (pp. 535–555). Hillsdale, NJ: Erlbaum.
Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204–256.
Content, A. (1991). The effect of spelling-to-sound regularity on naming in French. Psychological Research, 53, 3–12. doi:10.1007/ BF00867327
De Groot, A. M. B. (1989). Representational aspects of word imageability and word frequency as assessed through word association. Journal of Experimental Psychology: Learning, Memory, & Cognition, 15, 824–845. doi:10.1037/0278-7393.15.5.824
Desrochers, A. (2006). OMNILEX: Une base de données sur le lexique du français contemporain. Cahiers Linguistiques d’Ottawa, 34, 25–34.
Desrochers, A., & Bergeron, M. (2000). Valeurs de fréquence subjective et d’imagerie pour un échantillon de 1,916 substantifs de la langue française. Canadian Journal of Experimental Psychology, 56, 274–325. doi:10.1037/h0087347
Desrochers, A., Paivio, A., & Desrochers, S. (1989). L’effet de la fréquence d’usage des noms inanimés et de la valeur prédictive de leur terminaison sur l’identification du genre grammatical. Canadian Journal of Psychology, 43, 62–73. doi:10.1037/h0084253
Desrochers, A., & Thompson, G. L. (2009). Subjective frequency and imageability ratings for 3,600 French nouns. Behavior Research Methods, 41, 546–557.
Ellis, N. (2002). Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24, 143–188. doi:10.1017/S0272263102002024
Forget, H. (2005). Valeurs d’imagerie et de fréquence subjective de 354 mots du vocabulaire sexuel de la langue française. Revue canadienne des sciences du comportement, 37, 49–69. doi:10.1037/h0087245
Forster, K. I. (2000). The potential for experimenter bias effects in word recognition experiments. Memory & Cognition, 28, 1109–1115.
Gernsbacher, M. A. (1984). Resolving 20 years of inconsistent interactions between lexical familiarity and orthography, concreteness, and polysemy. Journal of Experimental Psychology: General, 113, 256–281. doi:10.1037/0096-3445.113.2.256
Gilhooly, K. J., & Logie, R. H. (1980). Age of acquisition, imagery, concreteness, familiarity, and ambiguity measures for 1,944 words. Behavior Research Methods & Instrumentation, 12, 395–427.
Glover, S., & Dixon, P. (2004). Likelihood ratios: A simple and flexible statistic for empirical psychologists. Psychonomic Bulletin & Review, 11, 791–806.
Gonthier, I., Desrochers, A., Thompson, G. L., & Landry, D. (in press). Normes d’imagerie et de fréquence subjective pour 1,760 mots monosyllabiques de la langue française. Canadian Journal of Experimental Psychology.
Hino, Y., & Lupker, S. J. (2000). Effects of word frequency and spelling-to-sound regularity in naming with and without preceding lexical decision. Journal of Experimental Psychology: Human Perception & Performance, 26, 166–183. doi:10.1037/0096-1523.26.1.166
Imbs, P. (Ed.) (1971). études statistiques sur le vocabulaire français: Dictionnaire des fréquences. I: Tables alphabétiques. Paris: Didier.
James, C. T. (1975). The role of semantic information in lexical decisions. Journal of Experimental Psychology: Human Perception & Performance, 1, 130–136. doi:10.1037/0096-1523.1.2.130
Kučera, H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.
Leech, G., Rayson, P., & Wilson, A. (2001). Word frequencies in written and spoken English: Based on the British National Corpus. London: Longman.
Lété, B., Sprenger-Charolles, L., & Colé, P. (2004). MANULEX: A grade-level lexical database from French elementary school readers. Behavioral Research Methods, Instruments, & Computers, 36, 156–166.
Lupker, S. J., Brown, P., & Colombo, L. (1997). Strategic control in a naming task: Changing route or changing deadlines? Journal of Experimental Psychology: Learning, Memory, & Cognition, 23, 570–590. doi:10.1037/0278-7393.23.3.570
Lyne, A. A. (1985). The vocabulary of French business correspondence: Word frequencies, collocations and problems of lexicometric method. Geneva: Slatkine-Champion.
Monsell, S. (1991). The nature and locus of word frequency effects in reading. In D. Besner & G. Humphreys (Eds.), Basic processes in reading: Visual word recognition (pp. 148–197). Hillsdale, NJ: Erlbaum.
Monsell, S., Doyle, M., & Haggard, P. (1989). The effect of frequency on visual word recognition: Where are they? Journal of Experimental Psychology: General, 118, 43–71. doi:10.1037/0096-3445.118.1.43
Nelson, D. L., & McEvoy, C. L. (2000). What is this thing called frequency? Memory & Cognition, 28, 509–522.
New, B., Brysbaert, M., Véronis, J., & Pallier, C. (2007). The use of film subtitles to estimate word frequencies. Applied Psycholinguistics, 28, 661–677. doi:10.1017/S014271640707035X
New, B., & Pallier, C. (2005). Manuel de Lexique 3 (Unpublished technical report, version 3.03). Boulogne-Billancourt, France: Université Paris Descartes, Laboratoire de psychologie expérimentale.
New, B., Pallier, C., Brysbaert, M., & Ferrand, L. (2004). Lexique 2: A new French lexical database. Behavior Research Methods, Instruments, & Computers, 36, 516–524.
Norris, D. (2006). The Bayesian reader: Explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113, 327–357. doi:10.1037/0033-295X.113.2.327
Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717–731.
Sailor, K., Brooks, P. J., Bruening, P. R., Seiger-Gardner, L., & Guterman, M. (in press). Exploring the time course of semantic interference and associative priming in the picture—word interference task. Quarterly Journal of Experimental Psychology. doi:10.1080/17470210802254383
Scarborough, D. L., Cortese, C., & Scarborough, H. S. (1977). Frequency and repetition effects in lexical memory. Journal of Experimental Psychology: Human Perception & Performance, 3, 1–17. doi:10.1037/0096-1523.3.1.1
Scharlau, I., & Neumann, O. (2003). Temporal parameters and time course of perceptual latency priming. Acta Psychologica, 113, 185–203. doi:10.1016/S0001-6918%2802%2900157-9
Schilling, H. E. H., Rayner, K., & Chumbley, J. I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory & Cognition, 26, 1270–1281.
Sears, C. R., Siakaluk, P. D., Chow, V. C., & Buchanan, L. (2008). Is there an effect of print exposure on the word frequency effect and the neighborhood size effect? Journal of Psycholinguistic Research, 37, 269–291. doi:10.1007/s10936-008-9071-5
Stadthagen-Gonzalez, H., & Davis, C. J. (2006). The Bristol norms for age of acquisition, imageability, and familiarity. Behavior Research Methods, 38, 598–605.
Strain, E., & Herdman, C. M. (1999). Imageability effects in word naming: An individual differences analysis. Canadian Journal of Experimental Psychology, 53, 347–359. doi:10.1037/h0087322
Taft, M., & Meunier, F. (1998). Lexical representation of gender: A quasiregular domain. Journal of Psycholinguistic Research, 27, 23–45. doi:10.1023/A:1023270723066
Tainturier, M. J., Tremblay, M., & Lecours, A. R. (1992). Educational level and the word frequency effect: A lexical decision investigation. Brain & Language, 43, 460–474. doi:10.1016/0093-934X%2892%2990112-R
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational & Psychological Measurement, 60, 174–195. doi:10.1177/00131640021970448
Thompson, G. L. (2008). Eliminating aggregation bias in experimental research: Random coefficient analysis as an alternative to performing a “by-subjects” and/or “by-items” ANOVA. Tutorials in Quantitative Methods for Psychology, 4, 21–34.
Toglia, M. P., & Batig, W. F. (1978). Handbook of semantic word norms. Hillsdale, NJ: Erlbaum.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was funded in part by a grant from the Canadian Language and Literacy Research Network (to Jean Saint-Aubin, Raymond Klein, and A.D.) and a grant from the Social Sciences and Humanities Research Council of Canada (to A.D. and Stanislaw Szpakowicz).
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Thompson, G.L., Desrochers, A. Corroborating biased indicators: Global and local agreement among objective and subjective estimates of printed word frequency. Behavior Research Methods 41, 452–471 (2009). https://doi.org/10.3758/BRM.41.2.452
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BRM.41.2.452