Emotions in Speech: Juristic Implications

  • Erik J. Eriksson
  • Robert D. Rodman
  • Robert C. Hubal
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4343)


This chapter focuses on the detection of emotion in speech and the impact that using technology to automate emotion detection would have within the legal system. The current states of the art for studies of perception and acoustics are described, and a number of implications for legal contexts are provided. We discuss, inter alia, assessment of emotion in others, witness credibility, forensic investigation, and training of law enforcement officers.


acoustic parameters affect emotion emotional categories forensic juristic speech 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cole, R.A., Mariani, J., Uszkoreit, H., Zaenen, A., Zue, V., Varile, G., Zampolli, A. (eds.): Survey of the State of the Art in Human Language Technology. Cambridge Studies In Natural Language Processing Series. Cambridge University Press, Cambridge (1997)Google Scholar
  2. 2.
    Frank, G., Guinn, C., Hubal, R., Pope, P., Stanford, M., Lamm-Weisel, D.: JUST-TALK: An Application of Responsive Virtual Human Technology. In: Proceedings of the Interservice/Industry Training, Simulation and Education Conference, pp. 773–779. National Training Systems Association, Arlington (2002)Google Scholar
  3. 3.
    Fuller, C., Biros, D.P., Adkins, M., Burgoon, J.K., Nunamaker, J.F., Coulon, S.: Detecting Deception in Person-of-Interest Statements. In: Proceedings of the IEEE International Conference on Intelligence and Security Informatics, May 23-24, 2006, pp. 504–509, San Diego (2006)Google Scholar
  4. 4.
    Susca, M.: Connecting Stuttering Measurement and Management: II. Measures of Cognition and Affect. International Journal of Language & Communication Disorders 41(4), 365–377 (2006)CrossRefGoogle Scholar
  5. 5.
    Yacoub, S., Simske, S., Lin, X., Burns, J.: Recognition of Emotions in Interactive Voice Response Systems. In: Proceedings of the European Conference on Speech Communication and Technology, September 1-4, 2003, pp. 729–732, Geneva, Switzerland (2003)Google Scholar
  6. 6.
    Maroney, T.A.: Law and Emotion: A Proposed Taxonomy of an Emerging Field. Law and Human Behavior 30, 119–142 (2006)CrossRefGoogle Scholar
  7. 7.
    Cowie, R., Douglas-Cowie, E., Cox, C.: Beyond Emotion Archetypes: Databases for Emotion Modelling Using Neural Networks. Neural Networks 18, 371–388 (2005)CrossRefGoogle Scholar
  8. 8.
    Hozjan, V., Kačič, Z.: A Rule-Based Emotion-Dependent Feature Extraction Method for Emotion Analysis from Speech. Journal of Acoustic Society of America 119(5), 3109–3120 (2006)CrossRefGoogle Scholar
  9. 9.
    Burns, H.L., Capps, C.G.: Foundations of Intelligent Tutoring Systems: An Introduction. In: Poison, M.C., Richardson, J.J. (eds.) Foundations of Intelligent Tutoring Systems, pp. 1–19. Lawrence Erlbaum, London (1988)Google Scholar
  10. 10.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)Google Scholar
  11. 11.
    Scherer, K.R.: Vocal Communication of Emotion: A Review of Research Paradigms. Speech Communication 40, 227–256 (2003)zbMATHCrossRefGoogle Scholar
  12. 12.
    Solan, L.M., Tiersma, P.M.: Falling on Deaf Ears. Legal Affairs (November/December 2003), Available from as of (August 30, 2004)
  13. 13.
    Forgas, J.: Handbook of Affect and Social Cognition. Lawrence Erlbaum Publishers, New York (2001)Google Scholar
  14. 14.
    Christianson, S.: Emotional Stress and Eyewitness Memory: A Critical Review. Psychological Bulletin 112, 284–309 (1992)CrossRefGoogle Scholar
  15. 15.
    Taylor, J.G., Fragopanagos, N.: The Interaction of Attention and Emotion. Neural Networks 18(4), 353–369 (2005)CrossRefGoogle Scholar
  16. 16.
    Jackson, B.S.: Making Sense in Law. Deborah Charles Publications, Liverpool (1995)Google Scholar
  17. 17.
    Tulving, E., Thomson, D.M.: Encoding Specificity and Retrieval Processes in Episodic Memory. Psychological Review 80, 352–373 (1973)CrossRefGoogle Scholar
  18. 18.
    Olsson, N,: Realism of Confidence in Witness Identification of Faces and Voices. Unpublished doctoral dissertation, Uppsala University, Uppsala, Sweden (2000)Google Scholar
  19. 19.
    Loftus, E., Ketcham, K.: Witness for the Defense. St. Martin’s Press, New York (1991)Google Scholar
  20. 20.
    Loftus, E.: Our Changeable Memories: Legal and Practical Implications. Nature Reviews: Neuroscience 4, 231–234 (2003)CrossRefGoogle Scholar
  21. 21.
    Deffenbacher, K.A., Bornstein, B.H., Penrod, S.D., McGorty, K.: A Meta-Analytic Review of the Effects of High Stress on Eyewitness Memory. Law and Human Behavior 28(6), 687–706 (2004)CrossRefGoogle Scholar
  22. 22.
    Berk-Seligson, S.: The Miranda Warnings and Linguistic Coercion: The Role of Footing in the Interrogation of a Limited-English-Speaking Murder Suspect. In: Cotterill, J. (ed.) Language in the Legal Process, pp. 127–143. Palgrave Macmillan Ltd, New York (2002)Google Scholar
  23. 23.
    Russell, S.: ‘Three’s a Crowd’: Shifting Dynamics in the Interpreted Interview. In: Cotterill, J. (ed.) Language in the Legal Process, pp. 111–126. Palgrave Macmillan Ltd, New York (2002)Google Scholar
  24. 24.
    Wierzbicka, A.: Emotions across Languages and Cultures. Cambridge University Press, Cambridge (1999)Google Scholar
  25. 25.
    Semin, G.R., Görts, C.A., Nandram, S., Semin-Goossens, A.: Cultural Perspectives on the Linguistic Representation of Emotion and Emotion Events. Cognition & Emotion 16(1), 11–28 (2002)CrossRefGoogle Scholar
  26. 26.
    Tsai, J.L., Chentsova-Dutton, Y.: Variation among European Americans in Emotional Facial Expression. Journal of Cross-Cultural Psychology 34(6), 650–657 (2003)CrossRefGoogle Scholar
  27. 27.
    Ekman, P., Friesen, W.V., O’Sullivan, M., Chan, A., Diacoyanni- Tarlatzis, I., Heider, K., Krause, R., LeCompte, W.A., Pitcairn, T., Ricci-Bitti, P.E., Scherer, K.R., Tomita, M., Tzavaras, A.: Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion. Journal of Personality and Social Psychology 53(4), 712–717 (1987)CrossRefGoogle Scholar
  28. 28.
    Ekman, P., Keltner, D.: Universal Facial Expressions of Emotion: An Old Controversy and New Findings. In: Segerstrale, U., Molnár, P. (eds.) Nonverbal Communication: Where Nature Meets Culture, pp. 27–46. Lawrence Erlbaum Associates, Mahwah (1997)Google Scholar
  29. 29.
    Markus, H., Kitayama, S.: Culture and the Self: Implications for Cognition, Emotion, and Motivation. Psychological Review 98, 224–253 (1991)CrossRefGoogle Scholar
  30. 30.
    Mesquita, B., Markus, H.R.: Culture and Emotion: Models of Agency as Sources of Cultural Variation in Emotion. In: Frijda, N.H., Manstead, A.S.R., Fisher, A. (eds.) Feelings and Emotions: The Amsterdam Symposium, pp. 341–358. Cambridge University Press, Cambridge (2004)Google Scholar
  31. 31.
    Kopelman, S., Rosette, A.S., Thompson, L.: The Three Faces of Eve: Strategic Displays of Positive, Negative, and Neutral Emotions in Negotiations. Organizational Behaviour and Human Decision Processes 99, 81–101 (2006)CrossRefGoogle Scholar
  32. 32.
    Karstedt, S.N.E.: Emotions and Criminal Justice. Theoretical Criminology 6(3), 299–317 (2002)CrossRefGoogle Scholar
  33. 33.
    Vidmar, N.: Case Studies of Pre- and Midtrial Prejudice in Criminal and Civil Litigation. Law and Human Behavior 26(1), 73–105 (2002)CrossRefGoogle Scholar
  34. 34.
    Yang, L., Campbell, N.: Linking Form to Meaning: The Expression and Recognition of Emotions through Prosody. In: Proceedings of the 4th ISCA Workshop on Speech Synthesis. August 29 - September 1, 2001, Perthshire, Scotland (2001)Google Scholar
  35. 35.
    Breitenstein, C., Van Lancker, D., Daum, I.: The Contribution of Speech Rate and Pitch Variation to the Perception of Vocal Emotions in a German and an American Sample. Cognition and Emotion 15, 57–79 (2001)CrossRefGoogle Scholar
  36. 36.
    Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., Schröder, M.: FEELTRACE: An Instrument for Recording Perceived Emotions in Real Time. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 19–24. ISCA, Belfast, Ireland (2000)Google Scholar
  37. 37.
    Douglas-Cowie, E., Cowie, R., Schröder, M.: A New Emotion Database: Considerations, Sources and Scope. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 39–44. ISCA, Belfast, Ireland (2000)Google Scholar
  38. 38.
    Laukka, P., Juslin, P.N., Breslin, R.: A Dimensional Approach to Vocal Expression of Emotion. Cognition and Emotion 19(5), 633–653 (2005)CrossRefGoogle Scholar
  39. 39.
    Mullennix, J.W., Bihon, T., Bricklemyer, J., Gaston, J., Keener, J.M.: Effects of variation in emotional tone of voice on speech perception. Language and Speech 45(3), 255–283 (2002)Google Scholar
  40. 40.
    Scherer, K.R., Banse, R., Wallbott, H.G.: Emotion Inferences from Vocal Expression Correlate Across Languages and Cultures. Journal of Cross-Cultural Psychology 32, 76–92 (2001)CrossRefGoogle Scholar
  41. 41.
    Thompson, W.F., Balkwill, L.L.: Decoding Speech Prosody in Five Languages. Semiotica 158(1/4), 407–424 (2006)CrossRefGoogle Scholar
  42. 42.
    Wurm, L.H., Vakoch, D.A., Strasser, M.R., Calin-Jageman, R., Ross, S.E.: Speech Perception and Vocal Expression of Emotion. Cognition and Emotion 15(6), 831–852 (2001)CrossRefGoogle Scholar
  43. 43.
    Polzin, T., Waibel, A.: Emotion-Sensitive Human-Computer Interface. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 201–206. ISCA, Ireland, Belfast (2000)Google Scholar
  44. 44.
    Schröder, M., Cowie, R., Douglas-Cowie, M., Westerdijk, E., Gielen, S.: Acoustic Correlates of Emotion Dimensions in View of Speech Synthesis. In: Proceedings of Eurospeech, pp. 87–90. ISCA, Geneva, Switzerland (2001)Google Scholar
  45. 45.
    Tickle, A.: English and Japanese Speakers’ Emotion Vocalisation and Recognition: A Comparison Highlighting Vowel Quality. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 104–109. ISCA, Belfast, Ireland (2000)Google Scholar
  46. 46.
    Iida, A., Campbell, N., Higuchi, F., Yasamura, M.: A Corpus-Based Speech Synthesis System with Emotion. Speech Communication 40, 161–187 (2003)zbMATHCrossRefGoogle Scholar
  47. 47.
    Westermann, R., Stahl, G., Hesse, F.W.: Relative Effectiveness and Validity of Mood Induction Procedures: A Meta-analysis. European Journal of Social Psychology 26, 557–580 (1996)CrossRefGoogle Scholar
  48. 48.
    Liscombe, J., Venditti, J., Hirschberg, J.: Classifying Subject Ratings of Emotional Speech using Acoustic Features. In: Proceedings of Eurospeech, pp. 725–728. ISCA, Geneva, Switzerland (2003)Google Scholar
  49. 49.
    Bachorowski, J.A., Owren, M.J.: Production and Perception of Affect-Rated Vocal Acoustics. In: Ekman, P., Campos, J.J., Davidson, R.J., de Waal, F.B.M. (eds.) Emotions Inside Out, pp. 244–265. Annals of the New York Academy of Sciences, New York (2003)Google Scholar
  50. 50.
    Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E.: How to find Trouble in Communication. Speech Communication 40, 117–143 (2003)zbMATHCrossRefGoogle Scholar
  51. 51.
    Cowie, R., Cornelius, R.R.: Describing the Emotional States that are Expressed in Speech. Speech Communication 40, 5–32 (2003)zbMATHCrossRefGoogle Scholar
  52. 52.
    Picard, R.W., Vyzas, E., Healey, J.: Toward Machine Emotional Intelligence: Analysis of Affective Physiological State. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1175–1191 (2001)CrossRefGoogle Scholar
  53. 53.
    Banse, R., Scherer, K.: Acoustic Profiles in Vocal Emotion Expression. Journal of Personality and Social Psychology 70(3), 614–636 (1996)CrossRefGoogle Scholar
  54. 54.
    Mozziconacci, S.: Prosody and Emotions. In: Proceedings of Speech Prosody, pp. 1–9. ISCA, Aix-en-Provence (2002)Google Scholar
  55. 55.
    Oudeyer, P.Y.: The Production and Recognition of Emotions in Speech: Features and Algorithms. International Journal of Human-Computer Studies 59, 157–183 (2003)CrossRefGoogle Scholar
  56. 56.
    Nwe, T.L., Foo, S.W., De Silva, L.C.: Speech Emotion Recognition using Hidden Markov Models. Speech Communication 41, 603–623 (2003)CrossRefGoogle Scholar
  57. 57.
    Bänziger, T., Scherer, K.R.: The Role of Intonation in Emotional Expressions. Speech Communication 46(3-4), 252–267 (2005)CrossRefGoogle Scholar
  58. 58.
    McGilloway, S., Cowie, R., Douglas-Cowie, E., Gielen, S., Westerdijk, M., Stroeve, S.: Approaching Automatic Recognition of Emotion from Voice: A Rough Benchmark. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 207–212. ISCA Belfast, Ireland (2000)Google Scholar
  59. 59.
    Williams, C.E., Stevens, K.N.: Emotions and Speech: Some Acoustical Correlates. Journal of the Acoustical Society of America 52, 1238–1250 (1972)CrossRefGoogle Scholar
  60. 60.
    Toivanen, J., Väyrynen, E., Seppänen, T.: Automatic Discrimination of Emotion from Spoken Finnish. Language and Speech 47(4), 383–412 (2004)Google Scholar
  61. 61.
    Paeschke, A., Kienast, M., Sendlmeier, W.F.: F0-Contours in Emotional Speech. In: Proceedings of ICPhS, pp. 929–931. Linguistics Department, San Francisco, USA, University of California, Berkeley (1999)Google Scholar
  62. 62.
    Mozziconacci, S., Hermes, D.J.: Role of Intonation Patterns in Conveying Emotion in Speech. In: Proceedings of ICPhS, pp. 2001–2004. Linguistics Department, San Francisco, USA, University of California, Berkeley (1999)Google Scholar
  63. 63.
    Burkhardt, F., Sendlmeier, W.F.: Verification of Acoustical Correlates of Emotional Speech Using Formant-Synthesis. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 151–156. ISCA, Belfast, Ireland (2000)Google Scholar
  64. 64.
    Dellaert, F., Polzin, T., Waibel, A.: Recognizing Emotion in Speech. In: Proceedings of the ICSLP, pp. 896–900. ICSA, Philadelphia (1996)Google Scholar
  65. 65.
    Lee, C.M., Narayanan, S.S., Pieraccini, R.: Recognition of Negative Emotions from the Speech Signal. In: Proceedings of Automatic Speech Recognition and Understanding, pp. 240–243 (2001)Google Scholar
  66. 66.
    Roy, D., Pentland, A.: Automatic Spoken Affect Analysis and Classification. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp. 363–367. IEEE Computer Society Press, Los Alamitos (1996)CrossRefGoogle Scholar
  67. 67.
    Hubal, R., Frank, G., Guinn, C., Dupont, R.: Integrating a Crisis Stages Model into a Simulation for Training Law Enforcement Officers to Manage Encounters with the Mentally Ill. In: Proceedings of the Workshop on Architectures for Modeling Emotion: Cross-Disciplinary Foundations, American Association for Artificial Intelligence Spring Symposium Series, pp. 68–69. ACM Press, New York (2004)Google Scholar
  68. 68.
    Link, M.W., Armsby, P.P., Hubal, R.C., Guinn, C.I.: Accessibility and Acceptance of Responsive Virtual Human Technology as a Survey Interviewer Training Tool. Computers in Human Behavior 22(3), 412–426 (2006)CrossRefGoogle Scholar
  69. 69.
    Gobl, C., Chasaide, A.N.: The Role of Voice Quality in Communicating Emotion, Mood and Attitude. Speech Communication 14, 189–212 (2003)CrossRefGoogle Scholar
  70. 70.
    Janniro, M.J., Cestaro, V.L.: Effectiveness of Detection of Deception Examinations using the Computer Voice Stress Analyzer. Report No. DoDPI96-R-0005. Department of Defense Polygraph Institute, Fort McClellan (1996)Google Scholar
  71. 71.
    Oudeyer, P.Y.: Novel Useful Features and Algorithms for the Recognition of Emotions in Human Speech. Sony Computer Science Lab, Paris, France (2002) Available at Internet website: Downloaded on (2004)-03-17,
  72. 72.
    Petrushin, V.A., Makarova, V.: Parameters and Fricatives and Affricates in Russian Emotional Speech. In: Proceedings of Speech and Communciation (SPECOM), pp. 423–428, St. Petersburg (June 25-29, 2006)Google Scholar
  73. 73.
    Schröder, M.: Experimental Study of Affect Bursts. In: Proceedings of the ISCA Workshop on Speech and Emotion, pp. 132–137. ISCA, Belfast, Ireland (2000)Google Scholar
  74. 74.
    Engel, R.S., Silver, E.: Policing Mentally Disordered Suspects: A Re-examination of the Criminalization Hypothesis. Criminology 39, 225–232 (2002)CrossRefGoogle Scholar
  75. 75.
    Anderson, J.R., Reder, L.M., Simon, H.A.: Situated Learning and Education. Educational Researcher 25(4), 5–11 (1996)Google Scholar
  76. 76.
    Paschall, M.J., Fishbein, D.H., Hubal, R.C., Eldreth, D.: Psychometric Properties of Virtual Reality Vignette Performance Measures: A Novel Approach for Assessing Adolescents’ Social Competency Skills. Health Education Research: Theory and Practice 20(1), 61–70 (2005)Google Scholar
  77. 77.
    Lee, C.M., Narayanan, S.S., Pieraccini, R.: Classifying Emotions in Human-Machine Spoken Dialogs. In: Proceedings of IEEE ICME, pp. 737–740 (2002)Google Scholar
  78. 78.
    Lee, C.M., Narayanan, S.S.: Toward Detecting Emotions in Spoken Dialogs. IEEE Transactions on Speech and Audio Processing 13(2), 293–303 (2005)CrossRefGoogle Scholar
  79. 79.
    Potapova, R., Potapova, V.: Temporal Correlates of Emotions as a Speaker-State Specific Parameters for Forensic Speaker Identification. In: Proceedings of SPECOM 2003, Moscow (2003)Google Scholar
  80. 80.
    Plutchik, R.: The Psychology and Biology of Emotion. Harper-Collins, New York (1994)Google Scholar
  81. 81.
    Jovičic, S.T., Rajkovic, M., Dordecic, M., Kašic, Z.: Perceptual and Statistical Analysis of Emotional Speech in Man-Computer Communication. In: Proceedings of Speech and Communciation (SPECOM), June 25-29, 2006, pp. 409–414, St. Petersburg (2006)Google Scholar
  82. 82.
    Rose, P.: Forensic Speaker Identification. Taylor & Francis, London (2002)Google Scholar
  83. 83.
    Kaiser, J.F.: On a Simple Algorithm to Calculate the ‘Energy’ of a Signal. In: Proceedings of ICASSP, pp. 381–384 (1990)Google Scholar
  84. 84.
    Slyh, R.E., Nelson, W.T., Hansen, E.G.: Analysis of Mrate, Shimmer, Jitter, and F0 Contour Features Across Stress and Speaking Style in the SUSAS Database. In: Proceedings of ICASSP, pp. 2091–2094 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Erik J. Eriksson
    • 1
  • Robert D. Rodman
    • 2
  • Robert C. Hubal
    • 3
  1. 1.Dept. Philosophy and Linguistics, Umeå UniversitySweden
  2. 2.Dept. Computer Science, North Carolina State UniversityUSA
  3. 3.Technology Assisted Learning Ctr., RTI InternationalUSA

Personalised recommendations