Multi-model Approach to Human Functional State Estimation

  • Kevin DurkeeEmail author
  • Avinash Hiriyanna
  • Scott Pappada
  • John Feeney
  • Scott Galster
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9743)


With the growth and affordability of the wearable sensors market, there is increasing interest in leveraging physiological signals to measure human functional states. However, the desire to produce a reliable universal classifier of functional state assessment has proved to be elusive. In efforts to improve accuracy, we theorize the fusion of multiple models into a single estimate of human functional state could outperform a single model operating in isolation. In this paper, we explore the feasibility of this concept using a workload model development effort conducted for an Unmanned Aircraft System (UAS) task environment at the Air Force Research Laboratory (AFRL). Real-time workload classifiers were trained with single-model and multi-model approaches using physiological data inputs paired with and without contextual data inputs. Following the evaluation of each classifier using two model evaluation metrics, we conclude that a multi-model approach greatly improved the ability to reliably measure real-time cognitive workload in our UAS operations test case.


Context Human performance Modeling and simulation Physiological measurement Workload UAS Cognitive states 



Distribution A: Approved for public release. 88ABW Cleared 01/25/2016; 88ABW-2016-0243. This material is based on work supported by AFRL under Contract FA8650-11-C-6236. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of AFRL.


  1. 1.
    Blackhurst, J., Gresham, J., Stone, M.: The quantified warrior: how DoD should lead human performance augmentation. Armed Forces J. 4, 11 (2012)Google Scholar
  2. 2.
    Wilson, G.F., Russell, C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45(4), 635–644 (2003)CrossRefGoogle Scholar
  3. 3.
    Wang, Z., Hope, R.M., Wang, Z., Ji, Q., Gray, W.: Cross-subject workload classification with a hierarchical bayes model. NeuroImage 59(1), 64–69 (2012)CrossRefGoogle Scholar
  4. 4.
    Gevins, A., Smith, M.E., Leong, H., McEvoy, L., Whitfield, S., Du, R., Rush, G.: Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Hum. Factors 40, 79–91 (1998)CrossRefGoogle Scholar
  5. 5.
    Hoover, A., Singh, A., Fishel-Brown, S., Muth, E.: Real-time detection of workload changes using heart rate variability. Biomed. Sig. Process. Control 7, 333–341 (2012)CrossRefGoogle Scholar
  6. 6.
    Just, M.A., Carpenter, P.A.: The intensity dimension of thought: Pupillometric indices of sentence processing. Can. J. Exp. Psychol. 47(2), 310–339 (1993)CrossRefGoogle Scholar
  7. 7.
    Setz, C., Arnrich, B., Schumm, J., La Marca, R., Troster, G.: Discriminating stress from cognitive load using a wearable EDA device. Technology 14(2), 410–417 (2010)Google Scholar
  8. 8.
    Durkee, K., Geyer, A., Pappada, S., Ortiz, A., Galster, S.: Real-time workload assessment as a foundation for human performance augmentation. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS, vol. 8027, pp. 279–288. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  9. 9.
    Durkee, K., Pappada, S., Ortiz, A., Feeney, J., Galster, S.: Using context to optimize a functional state estimation engine in unmanned aircraft system operations. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2015. LNCS, vol. 9183, pp. 24–35. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  10. 10.
    Narendra, K.S., Balakrishnan, J.: Adaptive control using multiple models. IEEE Trans. Autom. Control 42(2), 171–187 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Rowe, A.J., Liggett, K.K., Davis, J.E.: Vigilant spirit control station: a research testbed for multi-UAS supervisory control interfaces. In: Proceedings of the 15th International Symposium on Aviation Psychology, Dayton, OH (2009)Google Scholar
  12. 12.
    Finomore, V., Popik, D., Dallman, R., Stewart, J., Satterfield, K., Castle, C.: Demonstration of a network-centric communication management suite: multi-modal communication. In: Proceedings of the 55th Human Factors and Ergonomics Society Annual Meeting, HFES, Las Vegas (2011)Google Scholar
  13. 13.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Peter, A.H., Najmedin, M. (eds.) Advances in Psychology, vol. 52, pp. 139–183. North-Holland, Amsterdam (2006)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Kevin Durkee
    • 1
    Email author
  • Avinash Hiriyanna
    • 1
  • Scott Pappada
    • 1
  • John Feeney
    • 1
  • Scott Galster
    • 2
  1. 1.Aptima, Inc.FairbornUSA
  2. 2.Air Force Research LaboratoryDaytonUSA

Personalised recommendations