Scribe4Me: Evaluating a Mobile Sound Transcription Tool for the Deaf

  • Tara Matthews
  • Scott Carter
  • Carol Pai
  • Janette Fong
  • Jennifer Mankoff
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4206)


People who are deaf or hard-of-hearing may have challenges communicating with others via spoken words and may have challenges being aware of audio events in their environments. This is especially true in public places, which may not have accessible ways of communicating announcements and other audio events. In this paper, we present the design and evaluation of a mobile sound transcription tool for the deaf and hard-of-hearing. Our tool, Scribe4Me, is designed to improve awareness of sound-based information in any location. When a button is pushed on the tool, a transcription of the last 30 seconds of sound is given to the user in a text message. Transcriptions include dialog and descriptions of environmental sounds. We describe a 2-week field study of an exploratory prototype, which shows that our approach is feasible, highlights particular contexts in which it is useful, and provides information about what should be contained in transcriptions.


Hearing Loss Text Message Cochlear Implant Assistive Technology Audio File 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akyol, S., Alvarado, P.: Finding Relevant Image Content for Mobile Sign Language Recognition. In: Proc. of SPPRA, pp. 48–52 (2001)Google Scholar
  2. 2.
    Auer, E.T., Bernstein, L.E., Coulter, D.C.: Temporal and Spatio-Temporal Vibrotactile Displays for Voice Fundamental Frequency: An Initial Evaluation of a New Vibrotactile Speech Perception Aid with Normal-Hearing and Hearing-Impaired Individuals. J. of the Acoustical Society of America 104(4), 2477–2489 (1998)CrossRefGoogle Scholar
  3. 3.
    Black, A.W., Brown, R.D., Frederking, R., Lenzo, J.K., Moody, A., Rudnicky, Singh, R., Steinbrecher, E.: Rapid Development of Speech-to-Speech Translation Systems. In: Proc. of ICSLP, pp. 1709–1712 (2002)Google Scholar
  4. 4.
    California Foundation for Independent Living Centers, NorthView Wins Exclusive Rights to Market SpeechView’s Solutions for Deaf. Assistive Technology Journal 65 (2003),
  5. 5.
    Carter, S., Mankoff, J.: When Participants Do the Capturing: The Role of Media in Diary Studies. In: Proc. of CHI, pp. 899–908 (2005)Google Scholar
  6. 6.
    Federal Communications Commission, What You Need To Know About TRS (2006),
  7. 7.
    Cook, A.M., Hussey, S.M.: Assistive Technologies: Principles and Practice. Mosby, Inc., St. Louis (2002)Google Scholar
  8. 8.
    Deaf Studies Trust, WISDOM: Wireless Information Services for Deaf people On the Move (2003),
  9. 9.
    Doyle, M., Dye, L.: Mainstreaming the Student Who is Deaf of Hard-of-Hearing. Alexander Graham Bell Association for the Deaf and Hard-of-Hearing (2002)Google Scholar
  10. 10.
    Edwards, A.D.N.: Progress in sign language recognition. In: Wachsmuth, I., Fröhlich, M. (eds.) GW 1997. LNCS, vol. 1371, pp. 13–21. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  11. 11.
    Elssmann, S.F., Maki, J.E.: Speech Spectrographic Display: Use of Visual Feedback by Hearing-Impaired Adults During Independent Articulation Practice. American Annals of the Deaf 132(4), 276–279 (1987)Google Scholar
  12. 12.
    Hudson, S., Mankoff, J.: Rapid Construction of Functioning Physical Interfaces from Cardboard, Thumbtacks and Masking Tape. In: Proc. of UIST (to appear, 2006)Google Scholar
  13. 13.
    Huenerfauth, M.: Survey and Critique of ASL Natural Language Generation and Machine Translation Systems Technical Report MS-CIS-03-32, University of Pennsylvania (2003)Google Scholar
  14. 14.
    Iachello, G., Truong, K., Abowd, G., Hayes, G., Stevens, M.: Event-Contingent Experience Sampling To Evaluate Ubicomp Technology In The Real World. In: Proc. of CHI, pp. 1009–1018 (2006)Google Scholar
  15. 15.
    Liu, F.H., Gu, L., Gao, Y., Picheny, M.: Use of Statistical N-Gram Models in Natural Language Generation for Machine Translation. In: Proc. of ICASSP, pp. 636–639 (2003)Google Scholar
  16. 16.
    Mann, W.C., Lane, J.P.: Assistive Technology for Persons with Disabilities. The American Occupational Therapy Association, Inc., Bethesda (1995)Google Scholar
  17. 17.
    Matthews, T., Fong, J., Ho-Ching, F.W., Mankoff, J.: Non-Speech Sound Visualizations for the Deaf. Behaviour & Information Technology (2006) (in press)Google Scholar
  18. 18.
  19. 19.
    Pastor, M., Sanchis, A., Casacuberta, F., Vidal, E.: EuTrans: a Speech-to-Speech Translator Prototype. In: Proc. of Eurospeech, pp. 2385–2389 (2001)Google Scholar
  20. 20.
    Power, M.R., Power, D.: Everyone here speaks TXT: Deaf People Using SMS in Australia and the Rest of the World. J. of Deaf Studies & Deaf Education 9(3), 350–360 (2004)Google Scholar
  21. 21.
    Schmandt, C., Lee, K., Kim, J., Ackerman, M.: Impromptu: Managing Networked Audio Applications for Mobile Users. In: Proc. of MobiSys., pp. 59–69 (2004)Google Scholar
  22. 22.
    Sonido Incorporated, Auditory Visual Articulation Therapy Software (2003),
  23. 23.
    The Reporters Committee for Freedom of the Press, The First Amendment Handbook (2003),
  24. 24.
    von Ahn, L., Liu, R., Blum, M.: Peekaboom: A Game for Locating Objects in Images. In: Proc. of CHI, pp. 55–64 (2006)Google Scholar
  25. 25.
    Woszczyna, M., Coccaro, N., Eisele, A., Lavie, A., McNair, A., Polzin, T., Rogina, I., Rose, C.P., Sloboda, T., Tomita, M., Tsutsumi, J., Aoki-Waibel, N., Waibel, A., Ward, W.: Recent Advances in JANUS: A Speech Translation System. In: Proc. of Eurospeech, pp. 1295–1298 (1993)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Tara Matthews
    • 1
  • Scott Carter
    • 1
  • Carol Pai
    • 2
  • Janette Fong
    • 2
  • Jennifer Mankoff
    • 2
  1. 1.Berkeley Institute of Design, CS DivisionUniversity of CaliforniaBerkeleyUSA
  2. 2.Human Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations