Skip to main content

Automatic, Multimodal Evaluation of Human Interaction

Abstract

This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.

This is a preview of subscription content, access via your institution.

References

  • Adkins M, Twitchell DP, et al (2004) Advances in automated deception detection in text-based computer-mediated communication. In: Proceedings of the SPIE defense and security symposium. Orlando

  • Bavelas JB (1983) Situations that lead to disqualification. Hum Commun Res 9(2): 130–145. doi:10.1111/j.1468-2958.1983.tb00688.x

    Article  Google Scholar 

  • Bavelas JB, Chovil N (2006) Hand gestures and facial displays as part of language use in face-to-face dialogue. In: Manusov V, Patterson ML (eds) Handbook of nonverbal communication. Sage, Thousand Oaks, pp 97–115

    Google Scholar 

  • Benson PJ, Campbell R et al (1999) Enhancing images of facial expressions. Percept Psychophys 61(2): 259–274

    Google Scholar 

  • Birdwhistell R (1955) Background to kinesics. Etc 13: 10–18

    Google Scholar 

  • Birdwhistell RL (1970) Kinesics and context. University of Pennsylvania Press, Philadelphia

    Google Scholar 

  • Bond CF, DePaulo BM (2006) Accuracy of deception judgments. Pers Soc Psychol Rev 10(3): 214–234. doi:10.1207/s15327957pspr1003_2

    Article  Google Scholar 

  • Bond CF, Omar A et al (1990) Lie detection across cultures. J Nonverbal Behav 14(3): 189–204. doi:10.1007/BF00996226

    Article  Google Scholar 

  • Bontcheva K, Cunningham H, et al (2002) Using GATE as an environment for teaching NLP. ACL workshop on effective tools and methodologies in teaching NLP

  • Buller D, Burgoon J (1996) Interpersonal deception theory. Commun Theory 6: 203–242. doi:10.1111/j.1468-2885.1996.tb00127.x

    Article  Google Scholar 

  • Burgoon JK (1994) Nonverbal signals. In: Knapp ML, Miller GR (eds) Handbook of interpersonal communication. Sage, Beverly Hills, pp 344–390

    Google Scholar 

  • Burgoon JK, Birk T et al (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17(1): 140–169. doi:10.1111/j.1468-2958.1990.tb00229.x

    Article  Google Scholar 

  • Burgoon JK, Blair JP, et al (2003) Effects of communication modality on arousal, cognitive complexity, behavioral control and deception detection during deceptive episodes. Annual Meeting of the National Communication Association. Miami Beach

  • Burgoon JK, Blair JP, et al (2003) Detecting dDeception through linguistic analysis. NSF/NIJ symposium on intelligence and security informatics

  • Burgoon JK, Buller DB (1994) Interpersonal deception: III. Effects of deceit on perceived communication and nonverbal behavior dynamics. J Nonverbal Behav 18(2): 155–184. doi:10.1007/BF02170076

    Article  Google Scholar 

  • Burgoon JK, Buller DB et al (1996) Nonverbal communication: the unspoken dialogue. McGraw-Hill, New York

    Google Scholar 

  • Burgoon JK, Jensen ML et al (2007) Deception and intention detection. In: Chen H, Raghu TS, Ramesh R, Vinze A, Zeng D (eds) Handbooks in information systems. Elsevier B. V. 2, Amsterdam

    Google Scholar 

  • Burgoon JK, Johnson ML et al (1998) The nature and measurement of interpersonal dominance. Commun Monogr 65: 308–335

    Article  Google Scholar 

  • Burgoon JK, Le Poire BA (1999) Nonverbal cues and interpersonal judgments: participant and observer perceptions of intimacy, dominance, composure, and formality. Commun Monogr 66: 105–124

    Article  Google Scholar 

  • Coker DA, Burgoon JK (1987) The nature of conversational involvement and nonverbal encoding patterns. Hum Commun Res 13(4). doi:10.1111/j.1468-2958.1987.tb00115.x

  • Cunningham H (2002) GATE, a general architecture for text engineering. Comput Hum 36: 223–254. doi:10.1023/A:1014348124664

    Article  Google Scholar 

  • DePaulo B, Lindsay J et al (2003) Cues to deception. Psychol Bull 129(1): 74–118. doi:10.1037/0033-2909.129.1.74

    Article  Google Scholar 

  • Ekman P (1985) Telling lies. W. W. Norton & Company, New York

    Google Scholar 

  • Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32: 88–106

    Google Scholar 

  • Frank MG, Feeley TH (2003) To catch a liar: challenges for research in lie detection training. J Appl Commun Res 31(1): 58–75. doi:10.1080/00909880305377

    Article  Google Scholar 

  • Gifford R (1994) A lens-mapping framework for understanding the encoding and decoding of interpersonal dispositions. J Pers Soc Psychol 66: 398–412. doi:10.1037/0022-3514.66.2.398

    Article  Google Scholar 

  • Hair JF, Black WC et al (2006) Multivariate data analysis. Pearson Prentice Hall, Upper Saddle River

    Google Scholar 

  • Kemp T, Schmidt M, et al (2000) Strategies for automatic segmentation of audio data. IEEE international conference on acoustics, speech, and signal processing. Istanbul

  • Levine TR, Feeley TH et al (2005) Testing the effects of nonverbal behavior training on accuracy in deception detection with the inclusion of a bogus training control group. West J Commun 69(3): 203–217. doi:10.1080/10570310500202355

    Article  Google Scholar 

  • Levine TR, Park HS et al (1999) “Accuracy in detecting truths and lies: documenting the “veracity effect. Commun Monogr 66(2): 125–144

    Article  Google Scholar 

  • Lu S, Tsechpenakis G, et al (2005) Blob analysis of the head and hands: a method for deception detection. Hawaii international conference on system science (HICSS’05). Hawaii

  • McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago

    Google Scholar 

  • Meservy TO (2007) Augmenting human intellect: automatic recognition of nonverbal behavior with application in deception detection. Management information systems. PhD, University of Arizona, Tucson. p 233

  • Meservy TO, Jensen ML (2005) Automatic extraction of deceptive behavioral cues from video. Intelligence and security informatics conference. Springer, Atlanta

    Google Scholar 

  • Meservy TO et al (2005) Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intell Syst 20(5). doi:10.1109/MIS.2005.85

  • Meyers LS, Gamst G et al (2006) Applied multivariate research: design and interpretation. Sage Publications, Inc, Thousand Oaks

    Google Scholar 

  • Qin T, Burgoon JK, et al (2004) An exploratory study on promising cues in deception detection and application of decision trees. In: Proceedings of the 37th Hawai’i international conference on system sciences. IEEE, Kona, HI, Los Alamitos

  • Rockwell P, Buller DB et al (1997) The voice of deceit: refining and expanding vocal cues to deception. Commun Res Rep 14(4): 451–459

    Google Scholar 

  • Scherer KR (1982) Methods of research on vocal communication: paradigms and parameters. In: Ekman KRSaP (eds) Handbook of methods in nonverbal behavior research. Cambridge University Press, Cambridge, pp 136–198

    Google Scholar 

  • Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4): 350–353. doi:10.1037/1040-3590.8.4.350

    Article  Google Scholar 

  • Trager GL (1961) The typology of paralanguage. Anthropol Linguist 3: 17–21

    Google Scholar 

  • Vrij A (2000) Detecting lies and deceit: the psychology of lying and implications for professional practice. John Wiley & Sons, Chichester

    Google Scholar 

  • Vrij A, Edward K et al (2001) Police officers’ ability to detect deceit: the benefit of indirect deception detection measures. Leg Criminol Psychol 6(2): 185–196. doi:10.1348/135532501168271

    Article  Google Scholar 

  • Whissell C, Fournier M et al (1986) A dictionary of affect in language: IV. Reliability, validity, and applications. Percept Mot Skills 62: 875–888

    Google Scholar 

  • Zhou L, Burgoon JK et al (2004) Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communication. Group decision and negotiation

  • Zhou L, Twitchell DP et al (2003) An exploratory study into deception detection in text-based computer-mediated communication. Thirty-sixth annual Hawaii international conference on system sciences (CD/ROM), Big Island, Hawaii, Computer Society Press

  • Zhou L, Twitchell DP et al (2004) Toward the automatic prediction of deception—an empirical comparison of classification methods. J Manage Inf Syst 20(4): 139–166

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas O. Meservy.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Jensen, M.L., Meservy, T.O., Burgoon, J.K. et al. Automatic, Multimodal Evaluation of Human Interaction. Group Decis Negot 19, 367–389 (2010). https://doi.org/10.1007/s10726-009-9171-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10726-009-9171-0

Keywords

  • Deception
  • Deception detection
  • Credibility
  • Credibility assessment
  • Multimodal behavior analysis