The (un)reliability of item-level semantic priming effects

  • Tom Heyman
  • Anke Bruninx
  • Keith A. Hutchison
  • Gert Storms
Article

Abstract

Many researchers have tried to predict semantic priming effects using a myriad of variables (e.g., prime–target associative strength or co-occurrence frequency). The idea is that relatedness varies across prime–target pairs, which should be reflected in the size of the priming effect (e.g., cat should prime dog more than animal does). However, it is only insightful to predict item-level priming effects if they can be measured reliably. Thus, in the present study we examined the split-half and test–retest reliabilities of item-level priming effects under conditions that should discourage the use of strategies. The resulting priming effects proved extremely unreliable, and reanalyses of three published priming datasets revealed similar cases of low reliability. These results imply that previous attempts to predict semantic priming were unlikely to be successful. However, one study with an unusually large sample size yielded more favorable reliability estimates, suggesting that big data, in terms of items and participants, should be the future for semantic priming research.

Keywords

Semantic priming Split-half reliability Test–retest reliability Semantic memory 

References

  1. Adelman, J. S., Marquis, S. J., Sabatos-DeVito, M. G., & Estes, Z. (2013). The unexplained nature of reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1037–1053. doi: https://doi.org/10.1037/a0031829 PubMedGoogle Scholar
  2. Balota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., … Treiman, R. (2007). The English Lexicon Project. Behavior Research Methods, 39, 445–459.  https://doi.org/10.3758/BF03193014 CrossRefPubMedGoogle Scholar
  3. Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). The impact of word prevalence on lexical decision times: Evidence from the Dutch Lexicon Project 2. Journal of Experimental Psychology: Human Perception and Performance, 42, 441–458.  https://doi.org/10.1037/xhp0000159 PubMedGoogle Scholar
  4. De Deyne, S., Navarro, D. J., Perfors, A., & Storms, G. (2016). Structure at every scale: A semantic network account of the similarities between unrelated concepts. Journal of Experimental Psychology: General, 145, 1228–1254.  https://doi.org/10.1037/xge0000192 CrossRefGoogle Scholar
  5. de Wit, B., & Kinoshita, S. (2014). Relatedness proportion effects in semantic categorization: Reconsidering the automatic spreading activation process. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 1733–1744.  https://doi.org/10.1037/xlm0000004 PubMedGoogle Scholar
  6. de Wit, B., & Kinoshita, S. (2015). An RT distribution analysis of relatedness proportion effects in lexical decision and semantic categorization reveals different mechanisms. Memory & Cognition, 43, 99–110.  https://doi.org/10.3758/s13421-014-0446-6 CrossRefGoogle Scholar
  7. Durgunoğlu, A. Y. (1988). Repetition, semantic priming, and stimulus quality: Implications for the interactive-compensatory reading model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 590–603.  https://doi.org/10.1037/0278-7393.14.4.590 PubMedGoogle Scholar
  8. Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160.  https://doi.org/10.3758/BRM.41.4.1149 CrossRefPubMedGoogle Scholar
  9. Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York, NY: McGraw-Hill.Google Scholar
  10. Günther, F., Dudschig, C., & Kaup, B. (2016). Latent semantic analysis cosines as a cognitive similarity measure: Evidence from priming studies. Quarterly Journal of Experimental Psychology, 69, 626–653.  https://doi.org/10.1080/17470218.2015.1038280 CrossRefGoogle Scholar
  11. Heyman, T., De Deyne, S., Hutchison, K. A., & Storms, G. (2015). Using the speeded word fragment completion task to examine semantic priming. Behavior Research Methods, 47, 580–606.  https://doi.org/10.3758/s13428-014-0496-5 CrossRefPubMedGoogle Scholar
  12. Heyman, T., Hutchison, K. A., & Storms, G. (2016a). Uncovering underlying processes of semantic priming by correlating item-level effects. Psychonomic Bulletin & Review, 23, 540–547.  https://doi.org/10.3758/s13423-015-0932-2 CrossRefGoogle Scholar
  13. Heyman, T., Van Akeren, L., Hutchison, K. A., & Storms, G. (2016b). Filling the gaps: A speeded word fragment completion megastudy. Behavior Research Methods, 48, 1508–1527.  https://doi.org/10.3758/s13428-015-0663-3 CrossRefPubMedGoogle Scholar
  14. Hutchison, K. A., Balota, D. A., Cortese, M. J., & Watson, J. M. (2008). Predicting semantic priming at the item level. The Quarterly Journal of Experimental Psychology, 61, 1036–1066.  https://doi.org/10.1080/17470210701438111 CrossRefPubMedGoogle Scholar
  15. Hutchison, K. A., Balota, D. A., Neely, J. H., Cortese, M. J., Cohen-Shikora, E. R., Tse, C.-S., … Buchanan, E. (2013). The semantic priming project. Behavior Research Methods, 45, 1099–1114.  https://doi.org/10.3758/s13428-012-0304-z CrossRefPubMedGoogle Scholar
  16. Jones, L. L., & Golonka, S. (2012). Different influences on lexical priming for integrative, thematic, and taxonomic relations. Frontiers in Human Neuroscience, 6, 205.  https://doi.org/10.3389/fnhum.2012.00205 CrossRefPubMedPubMedCentralGoogle Scholar
  17. Jones, M. N., & Mewhort, D. J. K. (2007). Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114, 1–37.  https://doi.org/10.1037/0033-295X.114.1.1 CrossRefPubMedGoogle Scholar
  18. Keuleers, E., Diependaele, K., & Brysbaert, M. (2010). Practice effects in large-scale visual word recognition studies: A lexical decision study on 14,000 Dutch mono- and disyllabic words and nonwords. Frontiers in Psychology, 1, 174.  https://doi.org/10.3389/fpsyg.2010.00174 CrossRefPubMedPubMedCentralGoogle Scholar
  19. Lord, F. M., Novick, M. R., & Birnbaum, A. (1968). Statistical theories of mental test scores. Oxford, UK: Addison-Wesley.Google Scholar
  20. Mandera, P., Keuleers, E., & Brysbaert, M. (2017). Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation. Journal of Memory and Language, 92, 57–78.  https://doi.org/10.1016/j.jml.2016.04.001 CrossRefGoogle Scholar
  21. McNamara, T. P. (2005). Semantic priming: Perspectives from memory and word recognition. New York, NY: Psychology Press.CrossRefGoogle Scholar
  22. Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227–234.  https://doi.org/10.1037/h0031564 CrossRefPubMedGoogle Scholar
  23. Morey, R. D., & Rouder, J. N. (2015). BayesFactor: Computation of Bayes factors for common designs (Version 0.9.12–2). Retrieved from https://CRAN.R-project.org/package=BayesFactor
  24. Peirce, J. W. (2007). PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods, 162, 8–13.  https://doi.org/10.1016/j.jneumeth.2006.11.017 CrossRefPubMedPubMedCentralGoogle Scholar
  25. R Development Core Team. (2016). R: A language and environment for statistical computing (Version 3.3.1). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org Google Scholar
  26. Rastle, K., Davis, M. H., Marslen-Wilson, W. D., & Tyler, L. K. (2000). Morphological and semantic effects in visual word recognition: A time-course study. Language and Cognitive Processes, 15, 507–537.  https://doi.org/10.1080/01690960050119689 CrossRefGoogle Scholar
  27. Shelton, J. R., & Martin, R. C. (1992). How semantic is automatic semantic priming? Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 1191–1210.  https://doi.org/10.1037/0278-7393.18.6.1191 PubMedGoogle Scholar
  28. Stevens, M., & Brysbaert, M. (2016). When do we have enough power in language research? Evidence from priming studies. Unpublished manuscript. Retrieved from http://crr.ugent.be/papers/When%20do%20we%20have%20enough%20power%20in%20language%20research.pdf
  29. Stolz, J. A., Besner, D., & Carr, T. H. (2005). Implications of measures of reliability for theories of priming: Activity in semantic memory is inherently noisy and uncoordinated. Visual Cognition, 12, 284–336.  https://doi.org/10.1080/13506280444000030 CrossRefGoogle Scholar
  30. Tan, L. C., & Yap, M. J. (2016). Are individual differences in masked repetition and semantic priming reliable? Visual Cognition, 24, 182–200.  https://doi.org/10.1080/13506285.2016.1214201 CrossRefGoogle Scholar
  31. Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PLOS ONE, 11, e0152719.  https://doi.org/10.1371/journal.pone.0152719 CrossRefPubMedPubMedCentralGoogle Scholar
  32. Wetzels, R., & Wagenmakers, E.-J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychonomic Bulletin & Review, 19, 1057–1064.  https://doi.org/10.3758/s13423-012-0295-x CrossRefGoogle Scholar
  33. Yap, M. J., Hutchison, K. A., & Tan, L. C. (2017). Individual differences in semantic priming performance: Insights from the Semantic Priming Project. In M. N. Jones (Ed.), Big data in cognitive science (pp. 203–226). New York, NY: Psychology Press.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Tom Heyman
    • 1
  • Anke Bruninx
    • 1
  • Keith A. Hutchison
    • 2
  • Gert Storms
    • 1
  1. 1.Department of Experimental PsychologyUniversity of LeuvenLeuvenBelgium
  2. 2.Montana State UniversityBozemanUSA

Personalised recommendations