Abstract
When processing spoken language sentences, listeners continuously make and revise predictions about the upcoming linguistic signal. In contrast, during comprehension of American Sign Language (ASL), signers must simultaneously attend to the unfolding linguistic signal and the surrounding scene via the visual modality. This may affect how signers activate potential lexical candidates and allocate visual attention as a sentence unfolds. To determine how signers resolve referential ambiguity during real-time comprehension of ASL adjectives and nouns, we presented deaf adults (n = 18, 19–61 years) and deaf children (n = 20, 4–8 years) with videos of ASL sentences in a visual world paradigm. Sentences had either an adjective-noun (e.g., “SEE YELLOW WHAT? FLOWER”) or a noun-adjective (e.g., “SEE FLOWER WHICH? YELLOW”) structure. The degree of ambiguity in the visual scene was manipulated at the adjective and noun levels (e.g., including one or more yellow items and one or more flowers in the visual array). We investigated effects of ambiguity and word order on target looking at early and late points in the sentence. Analysis revealed that adults and children made anticipatory looks to a target when it could be identified early in the sentence. Further, signers looked more to potential lexical candidates than to unrelated competitors in the early window, and more to matched than unrelated competitors in the late window. Children’s gaze patterns largely aligned with those of adults, although they made fewer anticipatory fixations to the target in the early window and were more susceptible to competitors in the late window. Together, these findings suggest that signers allocate referential attention strategically based on the amount and type of ambiguity at different points in the sentence when processing adjectives and nouns in ASL.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability statement
The datasets during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Notes
Capital letters are used to denote ASL sign glosses.
We used a more inclusive criteria for trackloss for children compared to adults to allow for more variable gaze patterns typical of child participants during eye-tracking. The 20% threshold for children is similar to that used in previous studies with young children (e.g., Borovsky et al. 2016; Nordmeyer and Frank 2014).
References
Altmann, G. T., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language,57(4), 502–518. https://doi.org/10.1016/j.jml.2006.12.004.
Anderson, D., & Reilly, J. (2002). The MacArthur communicative development inventory: normative data for American Sign Language. Journal of Deaf Studies and Deaf Education,7(2), 83–106. https://doi.org/10.1093/deafed/7.2.83.
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language,59(4), 390–412. https://doi.org/10.1016/j.jml.2007.12.005.
Barr, D. J. (2008). Analyzing ‘visual world’eyetracking data using multilevel logistic regression. Journal of Memory and Language,59(4), 457–474. https://doi.org/10.1016/j.jml.2007.09.002.
Bavelier, D., Dye, M. W., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences,10(11), 512–518. https://doi.org/10.1016/j.tics.2006.09.006.
Borovsky, A., Ellis, E. M., Evans, J. L., & Elman, J. L. (2016). Lexical leverage: Category knowledge boosts real-time novel word recognition in 2-year-olds. Developmental Science,19(6), 918–932. https://doi.org/10.1111/desc.12343.
Borovsky, A., Elman, J. L., & Fernald, A. (2012). Knowing a lot for one’s age: Vocabulary skill and not age is associated with anticipatory incremental sentence interpretation in children and adults. Journal of Experimental Child Psychology,112(4), 417–436. https://doi.org/10.1016/j.jecp.2012.01.005.
Borovsky, A., Sweeney, K., Elman, J. L., & Fernald, A. (2014). Real-time interpretation of novel events across childhood. Journal of Memory and Language,73, 1–14. https://doi.org/10.1016/j.jml.2014.02.001.
Chen Pichler, D. (2011). Using early ASL word order to shed light on word order variability in sign language. In M. Anderssen, K. Bentzen, & M. Westergaard (Eds.), Variation in the input (pp. 157–177). Dordrecht: Springer.
Coates, J., & Sutton-Spence, R. (2001). Turn-taking patterns in deaf conversation. Journal of Sociolinguistics,5(4), 507–529. https://doi.org/10.1111/1467-9481.00162.
Eberhard, K. M., Spivey-Knowlton, M. J., Sedivy, J. C., & Tanenhaus, M. K. (1995). Eye movements as a window into real-time spoken language comprehension in natural contexts. Journal of Psycholinguistic Research,24(6), 409–436. https://doi.org/10.1007/bf02143160.
Fernald, A., Thorpe, K., & Marchman, V. A. (2010). Blue car, red car: Developing efficiency in online interpretation of adjective–noun phrases. Cognitive Psychology,60(3), 190–217.
Gasser, M., & Smith, L. B. (1998). Learning nouns and adjectives: A connectionist account. Language and Cognitive Processes,13(2–3), 269–306. https://doi.org/10.1080/016909698386537.
Hanna, J. E., Tanenhaus, M. K., & Trueswell, J. C. (2003). The effects of common ground and perspective on domains of referential interpretation. Journal of Memory and Language,49(1), 43–61. https://doi.org/10.1016/s0749-596x(03)00022-6.
Harris, M., Clibbens, J., Chasin, J., & Tibbitts, R. (1989). The social context of early sign language development. First Language,9(25), 81–97. https://doi.org/10.1177/014272378900902507.
Huang, Y. T., & Snedeker, J. (2013). The use of lexical and referential cues in children’s online interpretation of adjectives. Developmental Psychology,49(6), 1090–1102. https://doi.org/10.1037/a0029477.
Hurewitz, F., Brown-Schmidt, S., Thorpe, K., Gleitman, L. R., & Trueswell, J. C. (2000). One frog, two frog, red frog, blue frog: factors affecting children’s syntactic choices in production and comprehension. Journal of Psycholinguistic Research,29(6), 597–626. https://doi.org/10.1023/A:1026468209238.
Kamide, Y., Altmann, G. T. M., & Haywood, S. L. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language,49(1), 133–156. https://doi.org/10.1016/S0749-596X(03)00023-8.
Kimmelman, V. (2012). Word order in Russian sign language. Sign Language Studies,12(3), 414–445. https://doi.org/10.1353/sls.2012.0001.
Knoeferle, P., Carminati, M. N., Abashidze, D., & Essig, K. (2011). Preferential inspection of recent real-world events over future events: evidence from eye tracking during spoken sentence comprehension. Frontiers in Psychology,2, 376. https://doi.org/10.3389/fpsyg.2011.00376.
Knoeferle, P., & Crocker, M. W. (2009). Constituent order and semantic parallelism in online comprehension: Eye-tracking evidence from German. The Quarterly Journal of Experimental Psychology,62(12), 2338–2371. https://doi.org/10.1080/17470210902790070.
Lieberman, A. M., Borovsky, A., Hatrak, M., & Mayberry, R. I. (2015). Real-time processing of ASL signs: delayed first language acquisition affects organization of the mental Lexicon. Journal of Experimental Psychology. Learning, Memory, and Cognition,41(4), 1130–1139. https://doi.org/10.1037/xlm0000088.
Lieberman, A. M., Borovsky, A., & Mayberry, R. I. (2018). Prediction in a visual language: Real-time sentence processing in American Sign Language across development. Language, Cognition and Neuroscience, 33(4), 387–401. https://doi.org/10.1080/23273798.2017.1411961.
Lieberman, A. M., Hatrak, M., & Mayberry, R. I. (2014). Learning to look for language: Development of joint attention in young deaf children. Language Learning and Development,10(1), 19–35. https://doi.org/10.1080/15475441.2012.760381.
MacDonald, K., LaMarr, T., Corina, D., Marchman, V. A., & Fernald, A. (2018). Real-time lexical comprehension in young children learning American Sign Language. Developmental science,21(6), e12672. https://doi.org/10.31234/osf.io/zht6g.
Mani, N., & Huettig, F. (2012). Prediction during language processing is a piece of cake—But only for skilled producers. Journal of Experimental Psychology: Human Perception and Performance,38(4), 843–847. https://doi.org/10.1037/a0029284.
Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG-and MEG-data. Journal of Neuroscience Methods,164(1), 177–190. https://doi.org/10.1016/j.jneumeth.2007.03.024.
Mayberry, R. I., & Squires, B. (2006). Sign language: acquisition. In K. Brown (Ed.), Encyclopedia of language and linguistics (pp. 291–296). Boston: Elsevier.
Melkman, R., Tversky, B., & Baratz, D. (1981). Developmental trends in the use of perceptual and conceptual attributes in grouping, clustering, and retrieval. Journal of Experimental Child Psychology,31(3), 470–486. https://doi.org/10.1016/0022-0965(81)90031-x.
Nation, K., Marshall, C. M., & Altmann, G. T. (2003). Investigating individual differences in children’s real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology,86(4), 314–329. https://doi.org/10.1016/j.jecp.2003.09.001.
Neidle, C., & Nash, J. (2012). The noun phrase. In R. Pfau, M. Steinbach, & B. Woll (Eds.), Sign language: An international handbook (pp. 265–292). Berlin, Boston: De Gruyter Mouton.
Neville, H. J., & Lawson, D. (1987). Attention to central and peripheral visual space in a movement detection task. III. Separate effects of auditory deprivation and acquisition of a visual language. Brain Research,405(2), 284–294. https://doi.org/10.1016/0006-8993(87)90297-6.
Ninio, A. (2004). Young children’s difficulty with adjectives modifying nouns. Journal of Child Language,31(2), 255–285. https://doi.org/10.1017/S0305000904006191.
Nordmeyer, A. E., & Frank, M. C. (2014). The role of context in young children’s comprehension of negation. Journal of Memory and Language,77, 25–39. https://doi.org/10.1016/j.jml.2014.08.002.
Odom, R. D., & Guzman, R. D. (1972). Development of hierarchies of dimensional salience. Developmental Psychology,6(2), 271–287. https://doi.org/10.1037/h0032096.
Pavani, F., & Bottari, D. (2012). Visual abilities in individuals with profound deafness a critical review. In M. M. Murray & M. T. Wallace (Eds.), The neural bases of multisensory processes (pp. 423–448). Boca Raton: CRC Press/Taylor & Francis.
Rubio-Fernandez, P., Mollica, F., & Jara-Ettinger, J. (2018). English and Spanish speakers exploit language structure to increase communicative efficiency. https://psyarxiv.com/gf8qx.
Rubio-Fernández, P., Wienholz, A., Kirby, S., & Lieberman, A. M. (2019). ASL signers vary adjective position to maximize efficiency: A reference production study. Poster presented at the 32nd Annual CUNY Conference on Human Sentence Processing, Boulder, CO.
Sandler, W., & Lillo-Martin, D. (2006). Sign language and linguistic universals. Cambrigde: Cambridge University Press.
Sedivy, J. C. (2003). Pragmatic versus form-based accounts of referential contrast: Evidence for effects of informativity expectations. Journal of Psycholinguistic Research,32(1), 3–23. https://doi.org/10.1023/A:1021928914454.
Sedivy, J. C., Tanenhaus, M. K., Chambers, C. G., & Carlson, G. N. (1999). Achieving incremental semantic interpretation through contextual representation. Cognition,71(2), 109–147. https://doi.org/10.1016/s0010-0277(99)00025-6.
Snedeker, J., & Trueswell, J. C. (2004). The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adult sentence processing. Cognitive Psychology,49(3), 238–299. https://doi.org/10.1016/j.cogpsych.2004.03.001.
Spivey, M. J., Tanenhaus, M. K., Eberhard, K. M., & Sedivy, J. C. (2002). Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology,45(4), 447–481. https://doi.org/10.1016/s0010-0285(02)00503-0.
Spivey, M. J., Tyler, M. J., Eberhard, K. M., & Tanenhaus, M. K. (2001). Linguistically mediated visual search. Psychological Science,12(4), 282–286. https://doi.org/10.1111/1467-9280.00352.
Sutton-Spence, R., & Woll, B. (1999). The linguistics of British Sign Language: An introduction. Cambrigde: Cambridge University Press.
Swisher, M. V. (2000). Learning to converse: How deaf mothers support the development of attention and conversational skills in their young deaf children. In P. Spencer, C. J. Erting, & M. Marschark (Eds.), The deaf child in the family and at school (pp. 21–40). Mahwah: Erlbaum.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science,268(5217), 1632–1634. https://doi.org/10.1126/science.7777863.
Tribushinina, E., & Mak, W. M. (2016). Three-year-olds can predict a noun based on an attributive adjective: evidence from eye-tracking. Journal of Child Language,43(2), 425–441. https://doi.org/10.1017/s0305000915000173.
Trueswell, J. C., Sekerina, I., Hill, N. M., & Logrip, M. L. (1999). The kindergarten-path effect: studying on-line sentence processing in young children. Cognition,73(2), 89–134. https://doi.org/10.1016/S0010-0277(99)00032-3.
Waxman, R. P., & Spencer, P. E. (1997). What mothers do to support infant visual attention: sensitivities to age and hearing status. Journal of Deaf Studies and Deaf Education,2(2), 104–114. https://doi.org/10.1093/oxfordjournals.deafed.a014311.
Weisleder, A., & Fernald, A. (2009). Real-time processing of postnominal adjectives by Latino children learning Spanish as a first language. In Proceedings of the 24th Annual Boston University Conference on Language Development (pp. 611–621).
Acknowledgements
This work was funded by the National Institute on Deafness and Communication Disorders (NIDCD), grant number R01DC015272. We thank Rachel Mayberry and Arielle Borovsky for valuable input, and Marla Hatrak, Michael Higgins, and Valerie Sharer for help with data collection. We are grateful to all of the individuals who participated in this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Wienholz, A., Lieberman, A.M. Semantic processing of adjectives and nouns in American Sign Language: effects of reference ambiguity and word order across development. J Cult Cogn Sci 3, 217–234 (2019). https://doi.org/10.1007/s41809-019-00024-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41809-019-00024-6