Abstract
This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adkins M, Twitchell DP, et al (2004) Advances in automated deception detection in text-based computer-mediated communication. In: Proceedings of the SPIE defense and security symposium. Orlando
Bavelas JB (1983) Situations that lead to disqualification. Hum Commun Res 9(2): 130–145. doi:10.1111/j.1468-2958.1983.tb00688.x
Bavelas JB, Chovil N (2006) Hand gestures and facial displays as part of language use in face-to-face dialogue. In: Manusov V, Patterson ML (eds) Handbook of nonverbal communication. Sage, Thousand Oaks, pp 97–115
Benson PJ, Campbell R et al (1999) Enhancing images of facial expressions. Percept Psychophys 61(2): 259–274
Birdwhistell R (1955) Background to kinesics. Etc 13: 10–18
Birdwhistell RL (1970) Kinesics and context. University of Pennsylvania Press, Philadelphia
Bond CF, DePaulo BM (2006) Accuracy of deception judgments. Pers Soc Psychol Rev 10(3): 214–234. doi:10.1207/s15327957pspr1003_2
Bond CF, Omar A et al (1990) Lie detection across cultures. J Nonverbal Behav 14(3): 189–204. doi:10.1007/BF00996226
Bontcheva K, Cunningham H, et al (2002) Using GATE as an environment for teaching NLP. ACL workshop on effective tools and methodologies in teaching NLP
Buller D, Burgoon J (1996) Interpersonal deception theory. Commun Theory 6: 203–242. doi:10.1111/j.1468-2885.1996.tb00127.x
Burgoon JK (1994) Nonverbal signals. In: Knapp ML, Miller GR (eds) Handbook of interpersonal communication. Sage, Beverly Hills, pp 344–390
Burgoon JK, Birk T et al (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17(1): 140–169. doi:10.1111/j.1468-2958.1990.tb00229.x
Burgoon JK, Blair JP, et al (2003) Effects of communication modality on arousal, cognitive complexity, behavioral control and deception detection during deceptive episodes. Annual Meeting of the National Communication Association. Miami Beach
Burgoon JK, Blair JP, et al (2003) Detecting dDeception through linguistic analysis. NSF/NIJ symposium on intelligence and security informatics
Burgoon JK, Buller DB (1994) Interpersonal deception: III. Effects of deceit on perceived communication and nonverbal behavior dynamics. J Nonverbal Behav 18(2): 155–184. doi:10.1007/BF02170076
Burgoon JK, Buller DB et al (1996) Nonverbal communication: the unspoken dialogue. McGraw-Hill, New York
Burgoon JK, Jensen ML et al (2007) Deception and intention detection. In: Chen H, Raghu TS, Ramesh R, Vinze A, Zeng D (eds) Handbooks in information systems. Elsevier B. V. 2, Amsterdam
Burgoon JK, Johnson ML et al (1998) The nature and measurement of interpersonal dominance. Commun Monogr 65: 308–335
Burgoon JK, Le Poire BA (1999) Nonverbal cues and interpersonal judgments: participant and observer perceptions of intimacy, dominance, composure, and formality. Commun Monogr 66: 105–124
Coker DA, Burgoon JK (1987) The nature of conversational involvement and nonverbal encoding patterns. Hum Commun Res 13(4). doi:10.1111/j.1468-2958.1987.tb00115.x
Cunningham H (2002) GATE, a general architecture for text engineering. Comput Hum 36: 223–254. doi:10.1023/A:1014348124664
DePaulo B, Lindsay J et al (2003) Cues to deception. Psychol Bull 129(1): 74–118. doi:10.1037/0033-2909.129.1.74
Ekman P (1985) Telling lies. W. W. Norton & Company, New York
Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32: 88–106
Frank MG, Feeley TH (2003) To catch a liar: challenges for research in lie detection training. J Appl Commun Res 31(1): 58–75. doi:10.1080/00909880305377
Gifford R (1994) A lens-mapping framework for understanding the encoding and decoding of interpersonal dispositions. J Pers Soc Psychol 66: 398–412. doi:10.1037/0022-3514.66.2.398
Hair JF, Black WC et al (2006) Multivariate data analysis. Pearson Prentice Hall, Upper Saddle River
Kemp T, Schmidt M, et al (2000) Strategies for automatic segmentation of audio data. IEEE international conference on acoustics, speech, and signal processing. Istanbul
Levine TR, Feeley TH et al (2005) Testing the effects of nonverbal behavior training on accuracy in deception detection with the inclusion of a bogus training control group. West J Commun 69(3): 203–217. doi:10.1080/10570310500202355
Levine TR, Park HS et al (1999) “Accuracy in detecting truths and lies: documenting the “veracity effect. Commun Monogr 66(2): 125–144
Lu S, Tsechpenakis G, et al (2005) Blob analysis of the head and hands: a method for deception detection. Hawaii international conference on system science (HICSS’05). Hawaii
McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago
Meservy TO (2007) Augmenting human intellect: automatic recognition of nonverbal behavior with application in deception detection. Management information systems. PhD, University of Arizona, Tucson. p 233
Meservy TO, Jensen ML (2005) Automatic extraction of deceptive behavioral cues from video. Intelligence and security informatics conference. Springer, Atlanta
Meservy TO et al (2005) Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intell Syst 20(5). doi:10.1109/MIS.2005.85
Meyers LS, Gamst G et al (2006) Applied multivariate research: design and interpretation. Sage Publications, Inc, Thousand Oaks
Qin T, Burgoon JK, et al (2004) An exploratory study on promising cues in deception detection and application of decision trees. In: Proceedings of the 37th Hawai’i international conference on system sciences. IEEE, Kona, HI, Los Alamitos
Rockwell P, Buller DB et al (1997) The voice of deceit: refining and expanding vocal cues to deception. Commun Res Rep 14(4): 451–459
Scherer KR (1982) Methods of research on vocal communication: paradigms and parameters. In: Ekman KRSaP (eds) Handbook of methods in nonverbal behavior research. Cambridge University Press, Cambridge, pp 136–198
Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4): 350–353. doi:10.1037/1040-3590.8.4.350
Trager GL (1961) The typology of paralanguage. Anthropol Linguist 3: 17–21
Vrij A (2000) Detecting lies and deceit: the psychology of lying and implications for professional practice. John Wiley & Sons, Chichester
Vrij A, Edward K et al (2001) Police officers’ ability to detect deceit: the benefit of indirect deception detection measures. Leg Criminol Psychol 6(2): 185–196. doi:10.1348/135532501168271
Whissell C, Fournier M et al (1986) A dictionary of affect in language: IV. Reliability, validity, and applications. Percept Mot Skills 62: 875–888
Zhou L, Burgoon JK et al (2004) Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communication. Group decision and negotiation
Zhou L, Twitchell DP et al (2003) An exploratory study into deception detection in text-based computer-mediated communication. Thirty-sixth annual Hawaii international conference on system sciences (CD/ROM), Big Island, Hawaii, Computer Society Press
Zhou L, Twitchell DP et al (2004) Toward the automatic prediction of deception—an empirical comparison of classification methods. J Manage Inf Syst 20(4): 139–166
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jensen, M.L., Meservy, T.O., Burgoon, J.K. et al. Automatic, Multimodal Evaluation of Human Interaction. Group Decis Negot 19, 367–389 (2010). https://doi.org/10.1007/s10726-009-9171-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10726-009-9171-0