Abstract
In this study, we address a cross-domain problem of applying computer vision approaches to reason about human facial behavior when people play The Resistance game. To capture the facial behaviors, we first collect several hours of video where the participants playing The Resistance game assume the roles of deceivers (spies) vs truth-tellers (villagers). We develop a novel attention-based neural network (NN) that advances the state of the art in understanding how a NN predicts the players’ roles. This is accomplished by discovering through learning those pixels and related frames which are discriminative and contributed the most to the NN’s inference. We demonstrate the effectiveness of our attention-based approach in discovering the frames and facial Action Units (AUs) that contributed to the NN’s class decision. Our results are consistent with the current communication theory on deception.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Burgoon, J. K. (2005). Nonverbal measurement of deceit. In V. Manusov (Ed.), The sourcebook of nonverbal measures: Going beyond words (pp. 237–250). Hillsdale, NJ: Erlbaum.
Burgoon, J. K. (2015). When is deceptive message production more effortful than truth-telling? a bakers dozen of moderators. In Frontiers in Psychology, 6.
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018). Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 839–847). IEEE.
Chaudhry, A., Dokania, P. K., & Torr, P. H. (2017). Discovering class-specific pixels for weakly-supervised semantic segmentation (p. 2017). BMVC.
Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., DePaulo, B. M., & Cooper, H. (2003). Cues to deception. In Cues to deception. Psychological Bulletin, 129(1), 74–118.
Dosovitskiy, A., & Brox, T. (2016). Inverting visual representations with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4829–4837).
Dowdall, J., Shastri, D., Pavlidis, I. T., Frank, M. G., Tsiamyrtzis, P., & Ekman, P. (2007). Imaging facial physiology for the detection of deceit. In International Journal of Computer Vision, pages 71, 2, 197–214.
Elkins, A. Burgoon, J. K., Nunamaker, J. F. Jr., Twyman, N. W. (2014). A rigidity detection system for automated credibility assessment. In Journal of Management Information Systems, 31, 173–201.
Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network. University of Montreal, 1341(3), 1.
Friesen, W. V., Ekman P. (1975). Unmasking the face. a guide to recognizing emotions from facial clues. In Englewood Cli s, NJ: Prentice Hall, 71(2), 197–214.
Guerrero, L., Floyd K., Burgoon, J. K. (2010). Nonverbal communication. In Allyn Bacon.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).
Hu, J., Shen, L., & Sun, G. (2017). Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 7.
Jensen M. L., Kruse J., Meservy T. O., Nunamaker J. F. Jr., Burgoon, J. K. (2007). Deception and intention detection. In Handbooks in Information Systems: National Security, pages Vol 2,193–214.
Jetley, S., Lord, N. A., Lee, N., & Torr, H. P. (2018). Learn to pay attention. arXiv preprint arXiv:1804.02391.
Li, K., Wu, Z., Peng, K., Ernst, J., & Fu, Y. (2018). Tell me where to look: Guided attention inference network. CVPR.
Mahendran, A., & Vedaldi, A. (2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5188–5196).
Metaxas, D., Bourlai T., Elkins A., Burgoon, J. K. (2017). Social signals of deception and dishonesty. In Social signal processing. Cambridge, UK: Cambridge University Press, pages 404–428.
Park, J., Woo, S., Lee, J., & Kweon, I. S. (2018). Bam: bottleneck attention module. arXiv preprint arXiv:1807.06514.
Proudfoot, J. G., Wilson D., Schuetzler R., Burgoon, J. K. (2014). Patterns of nonverbal behavior associated with truth and deception: Illustrations from three experiments. In Journal of Nonverbal Behavior, pages 38, 325–354.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. ICCV, 1, 618–626.
Simonyan, K., Vedaldi, A., & Zisserman A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806.
Sun, M., Farhadi, A., & Seitz, S. (2014). Ranking domain-specific highlights by analyzing edited videos. In European conference on computer vision (pp. 787–802). Springer.
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 4489–4497).
Twyman, N. W., Burgoon, J. K., Nunamaker, J. F., Diller, C. B. R., & Pentland, S. J. (2017). A video-based screening system for automated risk assessment using nuanced facial features. Journal of Management Information Systems, 34(4), 970–993.
Wang, F., Jiang, M, Qian, C., Yang, S., Li, C., Zhang, H., Wang X., & Tang, X. (2017). Residual attention network for image classification. arXiv preprint arXiv:1704.06904.
Wei, Y., Feng, J., Liang, X., Cheng, M., Zhao, Y., & Yan, S. (2017). Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. IEEE CVPR, 1, 3.
Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018). Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV).
Yang, H., Wang, B., Lin, S., Wipf, D., Guo, M., & Guo, B. (2015). Unsupervised extraction of video highlights via robust recurrent auto-encoders. In Proceedings of the IEEE international conference on computer vision (pp. 4633–4641).
Yao, T., Mei, T., & Rui, Y. (2016). Highlight detection with pairwise deep ranking for first-person video summarization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 982–990).
Zafeiriou, S., Pantic, M., Burgoon, J. K., Elkins, A. (2015) Unobtrusive deception detection. In In R. Calvo, S. K. DMello, J. Gratch, A. Kappas (Eds.), The Oxford handbook of affective computing. UK: Oxford University Press.
Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818–833). Springer.
Zhang, D., Sung, Y., Zhou, L. (2013). The e ects of group factors on deception detection performance. In Small Group Research, 44, 272–297.
Zhang, X., Wei, Y., Feng, J., Yang, Y., & Huang, T. (2018). Adversarial complementary learning for weakly supervised object localization. In IEEE CVPR.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2921–2929).
Acknowledgement
We are grateful to the Army Research Office for funding much of the work reported in this book under Grant W911NF-16-1-0342.
Funding Disclosure
This research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-16-1-0342. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Wang, L. et al. (2021). Attention-Based Facial Behavior Analytics in Social Communication. In: Subrahmanian, V.S., Burgoon, J.K., Dunbar, N.E. (eds) Detecting Trust and Deception in Group Interaction. Terrorism, Security, and Computation. Springer, Cham. https://doi.org/10.1007/978-3-030-54383-9_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-54383-9_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-54382-2
Online ISBN: 978-3-030-54383-9
eBook Packages: Computer ScienceComputer Science (R0)