Skip to main content
Download PDF

Speech perception without hearing

Perception & Psychophysics volume 62, pages 233–252 (2000)Cite this article

Abstract

In this study of visual phonetic speech perception without accompanying auditory speech stimuli, adults with normal hearing (NH;n=96) and with severely to profoundly impaired hearing (IH;n=72) identified consonant-vowel (CV) nonsense syllables and words in isolation and in sentences. The measures of phonetic perception were the proportion of phonemes correct and the proportion of transmitted feature information for CVs, the proportion of phonemes correct for words, and the proportion of phonemes correct and the amount of phoneme substitution entropy for sentences. The results demonstrated greater sensitivity to phonetic information in the IH group. Transmitted feature information was related to isolated word scores for the IH group, but not for the NH group. Phoneme errors in sentences were more systematic in the IH than in the NH group. Individual differences in phonetic perception for CVs were more highly associated with word and sentence performance for the IH than for the NH group. The results suggest that the necessity to perceive speech without hearing can be associated with enhanced visual phonetic perception in some individuals.

References

  • Auer, E. T., Jr., &Bernstein, L. E. (1997). Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness.Journal of the Acoustical Society of America,102, 3704–3710.

    Article  PubMed  Google Scholar 

  • Auer, E. T.. Jr., Bernstein, L. E., Waldstein, R. S., & Tucker, P. E. (1997). Effects of phonetic variation and the structure of the lexicon on the uniqueness of words. In C. Benoît & R. Campbell (Eds.),Proceedings of the ESCA/ESCOP Workshop on Audio-Visual Speech Processing (pp. 21–24). Rhodes, Greece, September 26–27.

  • Auer, E. T.. Jr.,Waldstein, R. S., Tucker, P. E., &Bernstein.L. E. (1996). Relationships between word knowledge and visual speech perception: I. Subjective estimates of word age-of-acquisition.Journal of the Acoustical Society of America,100, 2569.

    Article  Google Scholar 

  • Benguerel, A. P., &Pichora-Fuller, M. K. (1982). Coarticulation effects in lipreading.Journal of Speech & Hearing Research,25, 600–607.

    Google Scholar 

  • Bernstein, L. E., Coulter, D. C, O’Connell, M. P., Eberhardt, S. P., & Demorest, M. E. (1993). Vibrotactile and haptic speech codes. In A. Risberg, S. Felicetti, G. Plant, & K.-E. Spens (Eds.),Proceedings of the Second International Conference on Tactile Aids, Hearing Aids, and Cochlear Implants (pp. 57–70). Stockholm, June 7–11, 1992.

  • Bernstein, L. E., Demorest, M. E., Coulter, D. C, &O’Connell, M. P. (1991). Lipreading sentences with vibrotactile vocoders: Performance of normal-hearing and hearing-impaired subjects.Journal of the Acoustical Society of America,90, 2971 -2984.

    Article  PubMed  Google Scholar 

  • Bernstein, L. E., Demorest, M. E., &Eberhardt, S. P. (1994). A computational approach to analyzing sentential speech perception: Phoneme-to-phoneme stimulus-response alignment.Journal of the Acoustical Society of America,95, 3617–3622.

    Article  PubMed  Google Scholar 

  • Bernstein, L. E., Demorest, M. E., &Tucker, P. E. (1998). What makes a good speechreader? First you have to find one. In R. Campbell, B. Dodd, & D. Burnham (Eds.),Hearing by eye: II. The psychology of speechreading and auditory-visual speech (pp. 211–228). East Sussex, U.K.: Psychology Press.

    Google Scholar 

  • Bernstein, L. E., &Eberhardt, S. P. (1986a). Johns Hopkins Lipreading Corpus I–II: Disc 1 [videodisc]. Baltimore: Johns Hopkins University.

    Google Scholar 

  • Bernstein, L. E., &Eberhardt, S. P. (1986b). Johns Hopkins Lipreading Corpus III–IV: Disc 2 [videodisc], Baltimore: Johns Hopkins University.

    Google Scholar 

  • Bernstein. L. E., Iverson, P., & Auer, E. T., Jr. (1997). Elucidating the complex relationships between phonetic perception and word recognition in audiovisual speech perception. In C. Benoît & R. Campbell (Eds.),Proceedings of the ESCA/ESCOP Workshop on Audio-Visual Speech Processing (pp. 89–92). Rhodes, Greece, September 26–27.

  • Boothroyd, A., &Cawkwell, S. (1970). Vibrotactile thresholds in pure tone audiometry.Acta Otolaryngologic,69, 381–387.

    Article  Google Scholar 

  • Breeuwer, M., &Plomp, R. (1986). Speechreading supplemented with auditorily presented speech parameters.Journal of the Acoustical Society of America,79, 481–499.

    Article  PubMed  Google Scholar 

  • Catford, J. C. (1977).Fundamental problems in phonetics. Bloomington: Indiana University Press.

    Google Scholar 

  • Chomsky, N., &Halle, M. (1968).The sound pattern of English. New York: Harper & Row.

    Google Scholar 

  • Clouser, R. A. (1977). Relative phoneme visibility and lipreading performance.Volta Review,79, 27–34.

    Google Scholar 

  • Conklin, E. S. (1917). A method for the determination of relative skill in lip-reading.Volta Review,19, 216–219.

    Google Scholar 

  • Conrad, R. (1977). Lipreading by deaf and hearing children.British Journal of Educational Psychology,47, 60–65.

    PubMed  Google Scholar 

  • Davis, H., &Silverman, S. R. (Eds.) (1970).Hearing and deafness. New York: Holt, Rinehart & Winston.

    Google Scholar 

  • Dekle, D. J., Fowler, C. A., &Funnell, M. G. (1992). Audiovisual integration in perception of real words.Perception & Psychophysics,51, 355–362.

    Article  Google Scholar 

  • Demorest, M. E., &Bernstein, L. E. (1992). Sources of variability in speechreading sentences: A generalizability analysis.Journal of Speech & Hearing Research,35, 876–891.

    Google Scholar 

  • Demorest, M. E., &Bernstein, L. E. (1997). Relationships between subjective ratings and objective measures of performance in speechreading sentences.Journal of Speech, Language, & Hearing Research,40, 900–911.

    Google Scholar 

  • Demorest, M. E., Bernstein, L. E., &Dehaven, G. P. (1996). Generalizablity of speechreading performance on nonsense syllables, words, and sentences: Subjects with normal hearing.Journal of Speech & Hearing Research,39, 697–713.

    Google Scholar 

  • Duffy, S. A., &Pisoni, D. B. (1992). Comprehension of synthetic speech produced by rule: A review and theoretical interpretation.Language & Speech,35, 351 -389.

    Google Scholar 

  • Dunn, L. M., &Dunn, L. M. (1981).Peabody picture vocabulary test-revised. Circle Pines, MN: American Guidance Service.

    Google Scholar 

  • Feingold, A. (1995). The additive effects of differences in central tendency and variability are important in comparisons between groups.American Psychologist,50, 5–13.

    Article  Google Scholar 

  • Fisher, C. G. (1968). Confusions among visually perceived consonants.Journal of Speech & Hearing Research,11, 796–804.

    Google Scholar 

  • Geers, A., &Moog, J. (1989). Factors predictive of the development of literacy in profoundly hearing-impaired adolescents.Volta Review,91, 69–86.

    Google Scholar 

  • Green, K. [P.] (1998). The use of auditory and visual information during phonetic processing: Implications for theories of speech perception. In R. Campbell, B. Dodd, & D. Burnham (Eds.),Hearing by eye: II. Advances in the psychology of speechreading and auditory-visual speech (pp. 3–25). East Sussex. U.K.: Psychology Press.

    Google Scholar 

  • Green, K. P., &Kuhl, P. K. (1989). The role of visual information in the processing of place and manner features in speech perception.Perception & Psychophysics,45, 34–42.

    Article  Google Scholar 

  • Heider, F., &Heider, G. M. (1940). An experimental investigation of lipreading.Psychological Monographs,52 (Whole No. 232), 124–153.

    Google Scholar 

  • Hotchkiss, D. (1989).Demographic aspects of hearing impairment: Questions and answers. (Available from Center for Assessment and Demographic Studies, Gallaudet Research Institute, 800 Florida Avenue, N.E., Washington, DC 20002)

  • House, A. S., Williams, C. E., Hecker, M. H., &Kryter, K. D. (1965). Articulation-testing methods: Consonantal differentiation with a closed-response set.Journal of the Acoustical Society of America,37, 158–166.

    Article  PubMed  Google Scholar 

  • Jeffers, J., &Barley, M. (1971).Speechreading (lipreading). Springfield, IL: Charles C. Thomas.

    Google Scholar 

  • Keating, P. A. (1988).A survey of phonological features. Bloomington: Indiana University Linguistics Club.

    Google Scholar 

  • Kreul, E. J., Nixon, J. C, Kryter, K. D., Bell, D. W., &Lamb, J. S. (1968). A proposed clinical test of speech discrimination.Journal of Speech & Hearing Research,11, 536–552.

    Google Scholar 

  • Kuhl, P. K., &Meltzoff, A. N. (1988). Speech as an intermodal object of perception. In A. Yonas (Ed.),Perceptual development in infancy (Vol. 20, pp. 235–266). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Lesner, S. A., &Kricos, P. B. (1981). Visual vowel and diphthong perception across speakers.Journal of the Academy of Rehabilitative Audiology,14, 252–258.

    Google Scholar 

  • Lewellen, M. J., Goldinger, S. D., Pisoni, D. B., &Greene, B. G. (1993). Lexical familiarity and processing efficiency: Individual differences in naming, lexical decision, and semantic categorization.Journal of Experimental Psychology: General,122, 316–330.

    Article  Google Scholar 

  • Lisker, L., &Abramson, A. S. (1964). A cross-language study of voicing in initial stops: Acoustical measurements.Word,20, 384–422.

    Google Scholar 

  • Luce, P. A., &Pisoni, D. B. (1998). Recognizing spoken words: The neighborhood activation model.Ear & Hearing,19, 1–36.

    Article  Google Scholar 

  • Lyxell, B., &Rönnberg, J. (1987). Guessing and speechreading.British Journal of Audiology,21, 13–20.

    Article  PubMed  Google Scholar 

  • Lyxell, B., &Rönnberg, J. (1989). Information-processing skill and speech-reading.British Journal of Audiology,23, 339–347.

    Article  PubMed  Google Scholar 

  • Lyxell, B., &Rönnberg, J. (1991a). Visual speech processing: Word-decoding and word-discrimination related to sentence-based speech-reading and hearing-impairment.Scandinavian Journal of Psychology,32, 9–17.

    Article  PubMed  Google Scholar 

  • Lyxell, B., &Rönnberg, J. (1991b). Word discrimination and chronological age related to sentence-based speechreading skill.British Journal of Audiology,25, 3–10.

    Article  PubMed  Google Scholar 

  • Lyxell, B., &Rönnberg, J. (1992). The relationship between verbal ability and sentence-based speechreading.Scandinavian Audiology,21, 67–72.

    Article  PubMed  Google Scholar 

  • Lyxell, B., Rönnberg, J., Andersson, J., &Linderoth, E. (1993). Vibrotactile support: Initial effects on visual speech perception.Scandinavian Audiology,22, 179–183.

    Article  PubMed  Google Scholar 

  • Macleod, A., &Summerfield, Q. (1990). A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use.British Journal of Audiology,24, 29–43.

    Article  PubMed  Google Scholar 

  • Marslen-Wilson, W. (1990). Activation, competition, and frequency in lexical access. In G. T. M. Altmann (Ed.),Cognitive models of speech processing: Psycholinguistic and computational perspectives (pp. 148–172). Cambridge, MA: MIT Press.

    Google Scholar 

  • Massaro, D. W. (1987).Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Massaro, D. W. (1998).Perceiving talkingfaces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press, Bradford Books.

    Google Scholar 

  • Massaro.D. W., Cohen, M. M., &Gesi, A. T. (1993). Long-term training, transfer, and retention in learning to lipread.Perception & Psychophysics,53, 549–562.

    Article  Google Scholar 

  • McClelland, J. L., &Elman, J. (1986). The TRACE model of speech perception.Cognitive Psychology,18. 1–86.

    Article  PubMed  Google Scholar 

  • McGurk, H., &Macdonald, J. (1976). Hearing lips and seeing voices: A new illusion.Nature,264, 746–748.

    Article  PubMed  Google Scholar 

  • Middleweerd, M. J., &Plomp, R. (1987). The effect of speechreading on the speech-reception threshold of sentences in noise.Journal of the Acoustical Society of America,82, 2145–2147.

    Article  Google Scholar 

  • Miller, G. A., Heise, G. A., &Lichten, W. (1951). The intelligibility of speech as a function of the context of the test materials.Journal of Experimental Psychology,41, 329–335.

    Article  PubMed  Google Scholar 

  • Miller, J. L. (1991). Comment: Bimodal speech perception and the motor theory. In I. G. Mattingly & M. Studdert-Kennedy (Eds.),Modularity and the motor theory of speech perception (pp. 139–143). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Mogford, K. (1987). Lip-reading in the prelingually deaf. In B. Dodd & R. Campbell (Eds.),Hearing by eve: The psychology of tip-reading (pp. 191–211). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Montgomery, A. A., &Jackson, P. L. (1983). Physical characteristics of the lips underlying vowel lipreading performance.Journal of the Acoustical Society of America,73, 2134–2144.

    Article  PubMed  Google Scholar 

  • Montgomery, A. A., Walden, B. E., &Prosek, R. A. (1987). Effects of consonantal context on vowel lipreading.Journal of Speech & Hearing Research.30, 50–59.

    Google Scholar 

  • Moores, D. F., &Sweet, C. (1991). Factors predictive of school achievement. In D. F. Moores & K. P. Meadow-Orlans (Eds.),Educational and developmental aspects of deafness (pp. 154–201). Washington, DC: Gallaudet University Press.

    Google Scholar 

  • Neville, H. J. (1995). Developmental specificity in neurocognitive development in humans. In M. Gazzaniga (Ed.),The cognitive neurosciences (pp. 219–231). Cambridge, MA: MIT Press.

    Google Scholar 

  • Newman, R. S., Sawusch, J. R., &Luce, P. A. (1997). Lexical neighborhood effects in phonetic processing.Journal of Experimental Psychology: Human Perception & Performance,23, 873–889.

    Article  Google Scholar 

  • Nober, E. H. (1967). Vibrotactile sensitivity of deaf children to high intensity sound.Laryngoscope,78, 2128–2146.

    Article  Google Scholar 

  • Owens, E., &Blazek, B. (1985). Visemes observed by hearing impaired and normal hearing adult viewers.Journal of Speech & Hearing Research,28, 381–393.

    Google Scholar 

  • Pelson, R. O., &Prather, W. F. (1974). Effects of visual messagerelated cues, age, and hearing impairment on speechreading performance.Journal of Speech & Hearing Research,17, 518–525.

    Google Scholar 

  • Perfetti, C. A. (1994). Psycholinguistics and reading ability. In M. A. Gernsbacher (Ed.),Handbook of psycholinguistics (pp. 849–894). New York: Academic Press.

    Google Scholar 

  • Rabinowitz, W. M., Eddington.D. K., Delhorne, L. A., &Cuneo, P. A. (1992). Relations among different measures of speech reception in subjects using a cochlear implant.Journal of the Acoustical Society of America,92, 1869–1881.

    Article  PubMed  Google Scholar 

  • Raphael, L. J. (1972). Preceding vowel duration as a cue to the perception of the voicing characteristic of word final consonants in American English.Journal of the Acoustical Society of America,51, 1296–1303.

    Article  PubMed  Google Scholar 

  • Rönnberg, J. (1995). Perceptual compensation in the deaf and blind: Myth or reality? In R. A. Dixon & L. Bäckman (Eds.),Compensating for psychological deficits and declines (pp. 251–274). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Rönnberg, J., Öhngren.G., &Nilsson, L.-G. (1982). Hearing deficiency, speechreading and memory functions.Scandinavian Audiology,11, 261–268.

    Article  PubMed  Google Scholar 

  • Rönnberg, J., Samuelsson, S., &Lyxell, B. (1998). Conceptual constraints in sentence-based lipreading in the hearing-impaired. In R. Campbell, B. Dodd, & D. Burnham (Eds.),Hearing by eye: IL The psychology of speechreading and auditory-visual speech (pp. 143–153). East Sussex, U.K.: Psychology Press.

    Google Scholar 

  • Rosen, S., &Corcoran, T. (1982). A video-recorded test of lipreading for British English.British Journal of Audiology,16, 245–254.

    Article  PubMed  Google Scholar 

  • Rosen Blum, L. D., &Fowler, C. A. (1991). Audiovisual investigation of the loudness-effort effect for speech and nonspeech events.Journal of Experimental Psychology: Human Perception & Performance,17, 976–985.

    Article  Google Scholar 

  • Sankoff, D., &Kruskal, J. B. (Eds.) (1983).Time warps, string edits. and macromolecules: The theory and practice of sequence comparison. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Scheinberg, J. C. S. (1988). An analysis of /pi, Ibl and Im/in the speechreading signal. Unpublished doctoral dissertation, The City University of New York, New York.

    Google Scholar 

  • Schildroth, A. N., &Karchmer, M. A. (1986).Deaf children in America. San Diego: College-Hill.

    Google Scholar 

  • Schwantes, F. M. (1981). Locus of the context effect in children’s word recognition.Child Development,52, 895–903.

    Article  Google Scholar 

  • Sekiyama, K., &Tohkura, Y. (1991). McGurk effect in non-English listeners: Visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility.Journal of the Acoustical Society of America.90, 1797–1805.

    Article  PubMed  Google Scholar 

  • SPSS for Windows 7.0 [Computer Software]. (1996). Chicago, IL: SPSS Inc.

    Google Scholar 

  • Stanovich, K. E. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency.Reading Research Quarterly,1, 32–71.

    Article  Google Scholar 

  • Sumby, W. H., &Pollack, I. (1954). Visual contribution to speech intelligibility in noise.Journal of the Acoustical Society of America,26, 212–215.

    Article  Google Scholar 

  • Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.),Hearing by eye: The psychology of lipreading (pp. 3–52). London: Erlbaum.

    Google Scholar 

  • Summerfield, Q. (1991). Visual perception of phonetic gestures. In I. G. Mattingly & M. Studdert-Kennedy (Eds.),Modularity and the motor theory of speech perception (pp. 117–137). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Summerfield, Q., &McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels.Quarterly Journal of Experimental Psychology,36A, 51–74.

    Google Scholar 

  • Tillberg, I., Rönnberg, J., Svärd, L., &Ahlner, B. (1996). Audiovisual speechreading in a group of hearing aid users: The effects of onset age, handicap age, and degree of hearing loss.Scandinavian Audiology,25, 267–272.

    Article  PubMed  Google Scholar 

  • Tyler, L. K., Cobb, H., &Graham, N. (Eds.) (1992).Spoken language comprehension: An experimental approach to disordered and normal processing. Cambridge, MA: MIT Press.

    Google Scholar 

  • Utley, J. (1946). A test of lip reading ability.Journal of Speech & Hearing Disorders,11, 109–116.

    Google Scholar 

  • Vatikiotis-Bateson, E., Munhall, K. G., Kasahara, Y., Garcia, F., &Yehia, H. (1996). Characterizing audiovisual information during speech. In H.T. Bunnell & W. Idsardi (Eds.),Proceedings ICSLP 96: Fourth Internationa! Conference on Spoken Language Processing (pp. 1485–1488). New Castle, DE: Citation Delaware.

    Chapter  Google Scholar 

  • Walden, B. E., Erdman, S. A., Montgomery, A. A., Schwartz, D. M., &Prosek, R. A. (1981). Some effects of training on speech recognition by hearing-impaired adults.Journal of Speech & Hearing Research,24, 207–216.

    Google Scholar 

  • Walden, B. E., Montgomery, A. A., Prosek, R. A., &Schwartz, D. M. (1980). Consonant similarity judgments by normal and hearing-impaired listeners.Journal of Speech & Hearing Research,23, 162–184.

    Google Scholar 

  • Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., &Jones, C. J. (1977). Effects of training on the visual recognition of consonants.Journal of Speech & Hearing Research,20, 130–145.

    Google Scholar 

  • Waldstein, R. S., Auer, E. T., Jr.,Tucker, P. E. &Bernstein, L. E. (1996). Relationships between word knowledge and visual speech perception: II. Subjective ratings of word familiarity.Journal of the Acoustical Society of America,100, 2569.

    Article  Google Scholar 

  • Wang, M. D. (1976). SINFA: Multivariate uncertainty analysis for confusion matrices.Behavior Research Methods & Instrumentation,8, 471–472.

    Article  Google Scholar 

  • Wang, M. D., &Bilger, R. C. (1973). Consonant confusions in noise: A study of perceptual features.Journal of the Acoustical Society of America,54, 1248–1266.

    Article  PubMed  Google Scholar 

  • Wozniak, V. D., &Jackson, P. L. (1979). Visual vowel and diphthong perception from two horizontal viewing angles.Journal of Speech & Hearing Research,22, 355–365.

    Google Scholar 

  • Yehia, H., Rubin, P., &Vatikiotis-Bateson, E. (1997). Quantitative association of orofacial and vocal-tract shapes. In C. Benoît & R. Campbell (Eds.),Proceedings of the ESCA/ESCOP Workshop on Audio-Visual Speech Processing (pp. 41–44). Rhodes, Greece, September 26–27.

    Google Scholar 

Download references

Author information

Affiliations

  1. University of Maryland, Baltimore County, Catonsville, Maryland

    Marilyn E. Demorest

  2. Department of Communication Neuroscience, House Ear Institute, 2100 West Third Street, 90057, Los Angeles, California

    Lynne E. Bernstein & Paula E. Tucker

Authors
  1. Lynne E. Bernstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paula E. Tucker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marilyn E. Demorest
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lynne E. Bernstein.

Additional information

This study was funded by a grant from the NIH National Institute on Deafness and Other Communication Disorders (DC02107).

—Accepted by previous editor. Myron L. Braunstein

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bernstein, L.E., Tucker, P.E. & Demorest, M.E. Speech perception without hearing. Perception & Psychophysics 62, 233–252 (2000). https://doi.org/10.3758/BF03205546

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.3758/BF03205546

Keywords

  • Impaired Hearing
  • Speech Perception
  • Normal Hearing
  • Visual Speech
  • Nonsense Syllable