Automatic, Multimodal Evaluation of Human Interaction

Abstract

This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.

This is a preview of subscription content, access via your institution.

References

  1. Adkins M, Twitchell DP, et al (2004) Advances in automated deception detection in text-based computer-mediated communication. In: Proceedings of the SPIE defense and security symposium. Orlando

  2. Bavelas JB (1983) Situations that lead to disqualification. Hum Commun Res 9(2): 130–145. doi:10.1111/j.1468-2958.1983.tb00688.x

    Article  Google Scholar 

  3. Bavelas JB, Chovil N (2006) Hand gestures and facial displays as part of language use in face-to-face dialogue. In: Manusov V, Patterson ML (eds) Handbook of nonverbal communication. Sage, Thousand Oaks, pp 97–115

    Google Scholar 

  4. Benson PJ, Campbell R et al (1999) Enhancing images of facial expressions. Percept Psychophys 61(2): 259–274

    Google Scholar 

  5. Birdwhistell R (1955) Background to kinesics. Etc 13: 10–18

    Google Scholar 

  6. Birdwhistell RL (1970) Kinesics and context. University of Pennsylvania Press, Philadelphia

    Google Scholar 

  7. Bond CF, DePaulo BM (2006) Accuracy of deception judgments. Pers Soc Psychol Rev 10(3): 214–234. doi:10.1207/s15327957pspr1003_2

    Article  Google Scholar 

  8. Bond CF, Omar A et al (1990) Lie detection across cultures. J Nonverbal Behav 14(3): 189–204. doi:10.1007/BF00996226

    Article  Google Scholar 

  9. Bontcheva K, Cunningham H, et al (2002) Using GATE as an environment for teaching NLP. ACL workshop on effective tools and methodologies in teaching NLP

  10. Buller D, Burgoon J (1996) Interpersonal deception theory. Commun Theory 6: 203–242. doi:10.1111/j.1468-2885.1996.tb00127.x

    Article  Google Scholar 

  11. Burgoon JK (1994) Nonverbal signals. In: Knapp ML, Miller GR (eds) Handbook of interpersonal communication. Sage, Beverly Hills, pp 344–390

    Google Scholar 

  12. Burgoon JK, Birk T et al (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17(1): 140–169. doi:10.1111/j.1468-2958.1990.tb00229.x

    Article  Google Scholar 

  13. Burgoon JK, Blair JP, et al (2003) Effects of communication modality on arousal, cognitive complexity, behavioral control and deception detection during deceptive episodes. Annual Meeting of the National Communication Association. Miami Beach

  14. Burgoon JK, Blair JP, et al (2003) Detecting dDeception through linguistic analysis. NSF/NIJ symposium on intelligence and security informatics

  15. Burgoon JK, Buller DB (1994) Interpersonal deception: III. Effects of deceit on perceived communication and nonverbal behavior dynamics. J Nonverbal Behav 18(2): 155–184. doi:10.1007/BF02170076

    Article  Google Scholar 

  16. Burgoon JK, Buller DB et al (1996) Nonverbal communication: the unspoken dialogue. McGraw-Hill, New York

    Google Scholar 

  17. Burgoon JK, Jensen ML et al (2007) Deception and intention detection. In: Chen H, Raghu TS, Ramesh R, Vinze A, Zeng D (eds) Handbooks in information systems. Elsevier B. V. 2, Amsterdam

    Google Scholar 

  18. Burgoon JK, Johnson ML et al (1998) The nature and measurement of interpersonal dominance. Commun Monogr 65: 308–335

    Article  Google Scholar 

  19. Burgoon JK, Le Poire BA (1999) Nonverbal cues and interpersonal judgments: participant and observer perceptions of intimacy, dominance, composure, and formality. Commun Monogr 66: 105–124

    Article  Google Scholar 

  20. Coker DA, Burgoon JK (1987) The nature of conversational involvement and nonverbal encoding patterns. Hum Commun Res 13(4). doi:10.1111/j.1468-2958.1987.tb00115.x

  21. Cunningham H (2002) GATE, a general architecture for text engineering. Comput Hum 36: 223–254. doi:10.1023/A:1014348124664

    Article  Google Scholar 

  22. DePaulo B, Lindsay J et al (2003) Cues to deception. Psychol Bull 129(1): 74–118. doi:10.1037/0033-2909.129.1.74

    Article  Google Scholar 

  23. Ekman P (1985) Telling lies. W. W. Norton & Company, New York

    Google Scholar 

  24. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32: 88–106

    Google Scholar 

  25. Frank MG, Feeley TH (2003) To catch a liar: challenges for research in lie detection training. J Appl Commun Res 31(1): 58–75. doi:10.1080/00909880305377

    Article  Google Scholar 

  26. Gifford R (1994) A lens-mapping framework for understanding the encoding and decoding of interpersonal dispositions. J Pers Soc Psychol 66: 398–412. doi:10.1037/0022-3514.66.2.398

    Article  Google Scholar 

  27. Hair JF, Black WC et al (2006) Multivariate data analysis. Pearson Prentice Hall, Upper Saddle River

    Google Scholar 

  28. Kemp T, Schmidt M, et al (2000) Strategies for automatic segmentation of audio data. IEEE international conference on acoustics, speech, and signal processing. Istanbul

  29. Levine TR, Feeley TH et al (2005) Testing the effects of nonverbal behavior training on accuracy in deception detection with the inclusion of a bogus training control group. West J Commun 69(3): 203–217. doi:10.1080/10570310500202355

    Article  Google Scholar 

  30. Levine TR, Park HS et al (1999) “Accuracy in detecting truths and lies: documenting the “veracity effect. Commun Monogr 66(2): 125–144

    Article  Google Scholar 

  31. Lu S, Tsechpenakis G, et al (2005) Blob analysis of the head and hands: a method for deception detection. Hawaii international conference on system science (HICSS’05). Hawaii

  32. McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago

    Google Scholar 

  33. Meservy TO (2007) Augmenting human intellect: automatic recognition of nonverbal behavior with application in deception detection. Management information systems. PhD, University of Arizona, Tucson. p 233

  34. Meservy TO, Jensen ML (2005) Automatic extraction of deceptive behavioral cues from video. Intelligence and security informatics conference. Springer, Atlanta

    Google Scholar 

  35. Meservy TO et al (2005) Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intell Syst 20(5). doi:10.1109/MIS.2005.85

  36. Meyers LS, Gamst G et al (2006) Applied multivariate research: design and interpretation. Sage Publications, Inc, Thousand Oaks

    Google Scholar 

  37. Qin T, Burgoon JK, et al (2004) An exploratory study on promising cues in deception detection and application of decision trees. In: Proceedings of the 37th Hawai’i international conference on system sciences. IEEE, Kona, HI, Los Alamitos

  38. Rockwell P, Buller DB et al (1997) The voice of deceit: refining and expanding vocal cues to deception. Commun Res Rep 14(4): 451–459

    Google Scholar 

  39. Scherer KR (1982) Methods of research on vocal communication: paradigms and parameters. In: Ekman KRSaP (eds) Handbook of methods in nonverbal behavior research. Cambridge University Press, Cambridge, pp 136–198

    Google Scholar 

  40. Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4): 350–353. doi:10.1037/1040-3590.8.4.350

    Article  Google Scholar 

  41. Trager GL (1961) The typology of paralanguage. Anthropol Linguist 3: 17–21

    Google Scholar 

  42. Vrij A (2000) Detecting lies and deceit: the psychology of lying and implications for professional practice. John Wiley & Sons, Chichester

    Google Scholar 

  43. Vrij A, Edward K et al (2001) Police officers’ ability to detect deceit: the benefit of indirect deception detection measures. Leg Criminol Psychol 6(2): 185–196. doi:10.1348/135532501168271

    Article  Google Scholar 

  44. Whissell C, Fournier M et al (1986) A dictionary of affect in language: IV. Reliability, validity, and applications. Percept Mot Skills 62: 875–888

    Google Scholar 

  45. Zhou L, Burgoon JK et al (2004) Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communication. Group decision and negotiation

  46. Zhou L, Twitchell DP et al (2003) An exploratory study into deception detection in text-based computer-mediated communication. Thirty-sixth annual Hawaii international conference on system sciences (CD/ROM), Big Island, Hawaii, Computer Society Press

  47. Zhou L, Twitchell DP et al (2004) Toward the automatic prediction of deception—an empirical comparison of classification methods. J Manage Inf Syst 20(4): 139–166

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Thomas O. Meservy.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Jensen, M.L., Meservy, T.O., Burgoon, J.K. et al. Automatic, Multimodal Evaluation of Human Interaction. Group Decis Negot 19, 367–389 (2010). https://doi.org/10.1007/s10726-009-9171-0

Download citation

Keywords

  • Deception
  • Deception detection
  • Credibility
  • Credibility assessment
  • Multimodal behavior analysis