Analysis of Facial Expressions Explain Affective State and Trust-Based Decisions During Interaction with Autonomy

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1131)


Trust is a critical factor in the development and maintenance of effective human-autonomy teams. As such, new processes are needed to classify affective state change that could be related to either an accurate or a misaligned change in trust that occurs during collaboration. The task for the current study was a leader-follower, simulated driving task with two different types of driving autonomy, and two different levels of reliability. Facial expression was evaluated to gauge group differences in affect-based trust. Results indicated that the participant sample was best described by four distinct group clusters who varied in their level of subjective trust and facial expressivity.


Human-autonomy teaming Trust-based decision making Affect-based trust 



This research was supported by the Office of the Secretary of Defense through the Autonomy Research Pilot Initiative under MIPR DWAM31168. The views and conclusions of this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the CCDC Army Research Laboratory or US Government. The US Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.


  1. 1.
    Schaefer, K.E.: Measuring trust in human robot interactions: development of the Trust Perception Scale – HRI. In: Lawless, W., Mittu, R., Wagner, A., Sofge, D. (eds.) The Intersection of Robust Intelligence (RI) and Trust in Autonomous Systems. Springer, pp. 191–218 (2012)Google Scholar
  2. 2.
    Freedy, A., de Visser, E., Weltman, G., Coeyman, N.: Measurement of trust in human-robot collaboration. In: Proceedings of the 2007 International Conference on Collaborative Technologies and Systems, pp. 106–114. IEEE, Orlando (2007)Google Scholar
  3. 3.
    Marathe, A., Metcalfe, J., Lance, B.J., Lukos, J., Jangraw, D.C., Lai, K.T., Touryan, J., Stump, E., Sadler, B.M., Nothwang, W., McDowell, K.: The privileged sensing framework: a principled approach to improved human-autonomy integration. Theor. Issues Ergonomics Sci. 19, 283–320 (2018)CrossRefGoogle Scholar
  4. 4.
    Dong, Y., Hu, Z., Uchimura, K., Murayama, N.: Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transp. Syst. 12, 596–614 (2011)CrossRefGoogle Scholar
  5. 5.
    Nass, C., Jonsson, I-M., Harris, H., Reaves, B., Endo, J., Brace, S., Takayama, L.: Improving automotive safety by pairing driver emotion and car voice emotion. In: International Conference on HCI (2005)Google Scholar
  6. 6.
    Lisetti, C.L., Nasoz, F.: Affective intelligent car interfaces with emotion recognition. In: 11th International Conference on HCI (2005)Google Scholar
  7. 7.
    Neubauer, C., Matthews, G., Saxby, D.J.: The effects of cell phone use and automation on driver performance and subjective state in simulated driving. In: Proceedings of the 56th Annual HFES, Boston, Massachusetts (2012)Google Scholar
  8. 8.
    Neubauer, C., Matthews, G., Saxby, D.J.: Fatigue in the automated vehicle: Do games and conversation distract or energize the driver? In: Proceedings of the 57th Annual HFES, Chicago, Illinois (2014)Google Scholar
  9. 9.
    Sarter, N.B., Woods, D.D., Billings, C.E.: Automation surprises. In: Handbook of HFES, 2 edn., pp. 1926–1943 (1997)Google Scholar
  10. 10.
    Gu, H., Ji, Q.: An automated face reader for fatigue detection. In: FGR, pp. 111–116 (2004)Google Scholar
  11. 11.
    Zhang, Z., Zhang, J.: Driver fatigue detection based intelligent vehicle control. In: ICPR 2006: Proceedings of the 18th International Conference on Pattern Recognition, Washington, DC (2006)Google Scholar
  12. 12.
    Metcalfe, J.S., et al.: Building a framework to manage trust in automation. In: Proceedings of SPIE 10194, Micro- and Nanotechnology Sensors, Systems, and Applications IX (2017)Google Scholar
  13. 13.
    John, O.P., Donahue, E.M., Kentle, R.L.: The Big Five Inventory-Versions 4a and 54. University of California, Berkeley, Institute of Personality and Social Research, Berkeley (1991)Google Scholar
  14. 14.
    Greco, V., Roger, D.: Coping with uncertainty: the construction and validation of a new measure. J. Personality Individ. Differ. 3, 519–534 (2001)CrossRefGoogle Scholar
  15. 15.
    Muir, B.M., Moray, N.: Trust in automation Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics 39, 429–460 (1996)CrossRefGoogle Scholar
  16. 16.
    Jian, J.Y., Bisantz, A.M., Drury, C.G.: Towards an empirically determined scale of trust in computerized systems: distinguishing concepts and types of trust. In: Proceedings of the 42nd Annual Meeting of the HFES, Chicago, IL (1998)Google Scholar
  17. 17.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, Amsterdam, North-Holland, pp. 239–250 (1988)Google Scholar
  18. 18.
    Baltrušaitis, T., Zadeh, A., Chong Lim, Y., Morency, L.P.: OpenFace 2.0: Facial Behavior Analysis Toolkit. In: IEEE International Conference on Automatic Face and Gesture Recognition (2018)Google Scholar
  19. 19.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  20. 20.
    Perelman, B.S., Dorton, S., Harper, S.: Identifying consensus in heterogeneous multidisciplinary professional teams. In: Proceedings of the 2019 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA), Las Vegas, NV (2019)Google Scholar
  21. 21.
    Gruen, B., Leisch, F.: Fitting finite mixture of generalized linear regressions in R. Computat. Stat. Data Anal. 51, 5247–5252 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.CCDC U.S. Army Research Laboratory, Human Research and Engineering DirectorateAberdeen Proving GroundUSA

Personalised recommendations