UAV Operators Workload Assessment by Optical Brain Imaging Technology (fNIR)

  • Kurtulus Izzetoglu
  • Hasan Ayaz
  • James T. Hing
  • Patricia A. Shewokis
  • Scott C. Bunce
  • Paul Oh
  • Banu Onaral
Reference work entry


The use of unmanned aerial vehicles (UAVs) is expected to increase exponentially over the next few years. UAV ground operators are required to acquire skills quickly and completely, with a level of expertise that builds the operator’s confidence in his/her ability to control the UAV under adverse conditions. As UAVs are held to increasingly higher standards of efficiency and safety, operators are routinely required to perform more informationally dense and cognitively demanding tasks, resulting in increased cognitive workloads during operation. Functional brain monitoring offers the potential to help UAV operators meet these challenges. Recent research has demonstrated the utility of near- infrared-based functional brain imaging systems (fNIRs) for the purpose of monitoring frontal cortical areas that support executive functions (attention, working memory, response monitoring). This technology provides portable, safe, affordable, noninvasive, and minimally intrusive monitoring systems with rapid application times for continuous measures of cortical activity. fNIR technology allows continuous monitoring of operators during training as they develop expertise, as well as the capacity to monitor their cognitive workload under operational conditions while controlling UAVs in critical missions. This chapter discusses the utilization of fNIR in the monitoring of a cognitive workload during UAV operation, and as an objective measure of expertise development, that is, the transition from novice to expert during operator training.


Mental Workload Flight Simulator Brain Energy Metabolism Cognitive Workload Onboard Camera 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The U.S. Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick, MD 21702-5014 is the awarding and administering acquisition office. This investigation was funded under a U.S. Army Medical Research Acquisition Activity, Cooperative Agreement W81XWH-08-2-0573 and in part by W81XWH-09-2-0104. The content of the information herein does not necessarily reflect the position or the policy of the U.S. Government or the U.S. Army, and no official endorsement should be inferred.

The authors would also like to thank Justin Menda and Adrian Curtin for conducting experiments and part of data analyses.


  1. A. Ames III, CNS energy metabolism as related to function. Brain Res. Rev. 34(1–2), 42–68 (2000). doi: 10.1016/s0165-0173(00)00038-2Google Scholar
  2. H. Ayaz, M. Izzetoglu, S.M. Platek, S. Bunce, K. Izzetoglu, K. Pourrezaei, B. Onaral, Registering fNIR data to brain surface image using MRI templates. Conference Proceedings - IEEE Engineering in Medicine and Biology Society 2671–2674 (2006). doi: 10.1109/IEMBS.2006.260835Google Scholar
  3. H. Ayaz, P.A. Shewokis, A. Curtin, M. Izzetoglu, K. Izzetoglu, B. Onaral, Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation. J. Vis. Exp. (56), e3443 (2011). doi:10.3791/3443Google Scholar
  4. H. Ayaz, P.A. Shewokis, S. Bunce, K. Izzetoglu, B. Willems, B. Onaral, Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59(1), 36–47 (2012). doi:10.1016/j. neuroimage.2011.06.023CrossRefGoogle Scholar
  5. M. Beauchamp, A. Dagher, J. Aston, J. Doyon, Dynamic functional changes associated with cognitive skill learning of an adapted version of the tower of London task. Neuroimage 20(3), 1649–1660 (2003)CrossRefGoogle Scholar
  6. Y. Benjamini, Y. Hochberg, Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B (Methodological) 57(1), 289–300 (1995)zbMATHMathSciNetGoogle Scholar
  7. D. Boas, J. Culver, J. Stott, A. Dunn, Three dimensional Monte Carlo code for photon migration through complex heterogeneous media including the adult human head. Opt. Exp. 10(3), 159–170 (2002)CrossRefGoogle Scholar
  8. S.C. Bunce, M. Izzetoglu, K. Izzetoglu, B. Onaral, K. Pourrezaei, Functional near-infrared spectroscopy: an emerging neuroimaging modality. IEEE Eng. Med. Biol. Mag. 25(4), 54–62 (2006)CrossRefGoogle Scholar
  9. R.B. Buxton, K. Uludag, D.J. Dubowitz, T.T. Liu, Modeling the hemodynamic response to brain activation. Neuroimage 23, S220–S233 (2004)CrossRefGoogle Scholar
  10. B. Cain, A review of the mental workload literature (vol. RTO-TR-HFM-121-Part-II): DTIC document (2007)Google Scholar
  11. B. Chance, Z. Zhuang, C. UnAh, C. Alter, L. Lipton, Cognition-activated low-frequency modulation of light absorption in human brain. Proc. Natl. Acad. Sci. USA 90(8), 3770–3774 (1993)CrossRefGoogle Scholar
  12. B. Chance, E. Anday, S. Nioka, S. Zhou, L. Hong, K. Worden, R. Thomas, A novel method for fast imaging of brain function, non-invasively, with light. Opt. Exp. 2(10), 411–423 (1998)CrossRefGoogle Scholar
  13. J. Cooper, M.A. Goodrich, Towards combining uav and sensor operator roles in uav-enabled visual search, in Proceedings of ACM/IEEE International Conference on Human-Robot Interaction, Amsterdam, The Netherlands, 2008Google Scholar
  14. M. Cope, The development of a near infrared spectroscopy system and its application for non invasive monitoring of cerebral blood and tissue oxygenation in the newborn infant. Ph.D. thesis, University College London, London, 1991Google Scholar
  15. M. Cope, D. Delpy, System for long-term measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination. Med. Biol. Eng. Comput. 26(3), 289–294 (1988)CrossRefGoogle Scholar
  16. M. Cope, D.T. Delpy, E.O.R. Reynolds, S. Wray, J. Wyatt, P. Van der Zee, Methods of quantitating cerebral near infrared spectroscopy data. Adv. Exp. Med. Biol. 222, 183–189 (1988)CrossRefGoogle Scholar
  17. J.L. Drury, S.D. Scott, Awareness in unmanned aerial vehicle operations. Int. C2 J. 1, 1–10 (2008)Google Scholar
  18. J.L. Drury, J. Richer, N. Rackliffe, M.A. Goodrich, Comparing situation awareness for two unmanned aerial vehicle human interface approaches, in Proceedings of the Conference on Safety, Security, and Rescue Robotics (SSRR), Gaithersburg, MD, 2006Google Scholar
  19. J.B. Fishkin, E. Gratton, Propagation of photon-density waves in strongly scattering media containing an absorbing semi-infinite plane bounded by a straight edge. JOSA A 10(1), 127–140 (1993)CrossRefGoogle Scholar
  20. H. Garavan, D. Kelley, A. Rosen, S.M. Rao, E.A. Stein, Practice-related functional activation changes in a working memory task. Microsc. Res. Tech. 51(1), 54–63 (2000)CrossRefGoogle Scholar
  21. A. Gevins, M. Smith, Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issue Ergon. Sci. 4(1), 113–131 (2003)CrossRefGoogle Scholar
  22. A. Gevins, M. Smith, L. McEvoy, D. Yu, High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. Cereb. Cortex 7(4), 374 (1997)CrossRefGoogle Scholar
  23. E. Gratton, V. Toronov, U. Wolf, M. Wolf, A. Webb, Measurement of brain activity by near-infrared light. J. Biomed. Opt. 10(1), 11008 (2005)CrossRefGoogle Scholar
  24. P. Hancock, M.H. Chignell, Toward a theory of mental work load: stress and adaptability in human machine systems, in Proceedings of the International IEEE Conference on Systems, Man and Cybernetics, Atlanta, GA, 1986Google Scholar
  25. PA. Hancock, P.A. Desmond, Stress, Workload, and Fatigue (Lawrence Erlbaum, Mahwah, 2001)Google Scholar
  26. A.J. Hansen, M. Nedergaard, Brain ion homeostasis in cerebral ischemia. Mol. Chem. Neuropathol. 9(1), 195–209 (1988)Google Scholar
  27. S. Hart, L. Staveland, Development of NASA-TLX (task load index): results of empirical and theoretical research. Hum. Ment. Workload 1, 139–183 (1988)CrossRefGoogle Scholar
  28. J.T. Hing, P.Y. Oh, Development of an unmanned aerial vehicle piloting system with integrated motion cueing for training and pilot evaluation. J. Intell. Robot. Syst. 54(1), 3–19 (2009)CrossRefGoogle Scholar
  29. J.T. Hing, J. Menda, K. Izzetoglu, P.Y. Oh, An indoor study to evaluate a mixed-reality interface for unmanned aerial vehicle operations in near earth environments. Int. J. Intell. Control Syst. 16(2), 132–141 (2011)Google Scholar
  30. Y. Hoshi, M. Tamura, Near-infrared optical detection of sequential brain activation in the prefrontal cortex during mental tasks. Neuroimage 5(4 Pt 1), 292–297 (1997)CrossRefGoogle Scholar
  31. K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei, B. Chance, Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum. Comput. Interact. 17(2), 211–227 (2004)CrossRefGoogle Scholar
  32. M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A. Devaraj, B. Onaral, K. Pourrezaei, Functional near-infrared neuroimaging. IEEE Trans. Neural. Syst. Rehabil. Eng. 13(2), 153–159 (2005)CrossRefGoogle Scholar
  33. K. Izzetoglu, H. Ayaz, A.C. Merzagora, M. Izzetoglu, P.A. Shewokis, S. Bunce, B. Onaral, The evolution of field deployable fNIR spectroscopy from bench to clinical settings. J. Innov. Opt. Health Sci. 4(3), 239–250 (2011)CrossRefGoogle Scholar
  34. F.F. Jobsis, Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 198(4323), 1264–1267 (1977)CrossRefGoogle Scholar
  35. D.B. Kaber, E. Onal, M.R. Endsley, Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload. Hum. Factor Ergon. Manuf. 10(4), 409–430 (2000)CrossRefGoogle Scholar
  36. A. Kelly, H. Garavan, Human functional neuroimaging of brain changes associated with practice. Cereb. Cortex 15(8), 1089 (2005)CrossRefGoogle Scholar
  37. K.K. Kwong, J.W. Belliveau, D.A. Chesler, I.E. Goldberg, R.M. Weisskoff, B.P. Poncelet, R. Turner, Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc. Natl. Acad. Sci. 89(12), 5675 (1992)CrossRefGoogle Scholar
  38. J. Menda, J.T. Hing, H. Ayaz, P.A. Shewokis, K. Izzetoglu, B. Onaral, P. Oh, Optical brain imaging to enhance UAV operator training, evaluation, and interface development. J. Intell. Robot. Syst. 61(1–4), 423–443 (2010). doi:10.1007/s10846-010-9507-7Google Scholar
  39. N. Meshkati, P.A. Hancock, M. Rahimi, S.M. Dawes, Techniques of mental workload assessment, in Evaluation of Human Work: A Practical Ergonomics Methodology, 2nd edn., ed. by J. Wilson, E.N. Corlett (Taylor and Francis, London, 1995)Google Scholar
  40. J.G. Milton, S.S. Small, A. Solodkin, On the road to automatic: dynamic aspects in the development of expertise. J. Clin. Neurophys. 21(3), 134 (2004)CrossRefGoogle Scholar
  41. S. Ogawa, T. Lee, A. Kay, D. Tank, Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc. Natl. Acad. Sci. 87(24), 9868 (1990)CrossRefGoogle Scholar
  42. R.C. Oldfield, The assessment and analysis of handedness: the Edinburgh inventory. Neuropsy-chologia 9(1), 97–113 (1971)CrossRefGoogle Scholar
  43. R. Parasuraman, M. Rizzo, Neuroergonomics: The Brain at Work (Oxford University Press, Oxford, 2007)Google Scholar
  44. R. Parasuraman, G. Wilson, Putting the brain to work: neuroergonomics past, present, and future. Hum. Factor 50(3), 468 (2008)CrossRefGoogle Scholar
  45. M.S. Patterson, B. Chance, B.C. Wilson, Time resolved reflectance and transmittance for the noninvasive measurement of tissue optical properties. Appl. Opt. 28(12), 2331–2336 (1989)CrossRefGoogle Scholar
  46. R.A. Poldrack, A.D. Wagner, M.W. Prull, J.E. Desmond, G.H. Glover, J.D.E. Gabrieli, Functional specialization for semantic and phonological processing in the left inferior Prefrontal Cortex* 1. Neuroimage 10(1), 15–35 (1999)CrossRefGoogle Scholar
  47. C. E. Rash, P.A. LeDuc, S.D. Manning, Human factors in U.S. military unmanned aerial vehicle accidents, in Human Factors of Remotely Operated Vehicles, vol. 7, ed. by N.J. Cooke, H.L. Pringle, H.K. Pedersen, O. Connor. Advances in Human Performance and Cognitive Engineering Research (Emerald Group Publishing Limited, Boston, 2006), pp. 117–131CrossRefGoogle Scholar
  48. M.W. Scerbo, F.G. Freeman, P.J. Mikulka, A brain-based system for adaptive automation. Theor. Issue Ergon. Sci. 4, 1(2), 200–219 (2003)CrossRefGoogle Scholar
  49. A.K. Singh, I. Dan, Exploring the false discovery rate in multichannel NIRS. Neuroimage 33(2), 542–549 (2006)CrossRefGoogle Scholar
  50. G. Strangman, D.A. Boas, J.P. Sutton, Non-invasive neuroimaging using near-infrared light. Biol. Psychiatry 52(7), 679–693 (2002)CrossRefGoogle Scholar
  51. A. Villringer, B. Chance, Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci. 20(10), 435–442 (1997)CrossRefGoogle Scholar
  52. G.F. Wilson, C.A. Russell, Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factor 45(4), 635 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.School of Biomedical EngineeringScience and Health Systems, Drexel UniversityPhiladelphiaUSA
  2. 2.Department of Mechanical Engineering and MechanicsDrexel Autonomous Systems Lab, Drexel UniversityPhiladelphiaUSA
  3. 3.College of Nursing and Health Professions and Schoolof Biomedical Engineering, Science and Health SystemsDrexel UniversityPhiladelphiaUSA
  4. 4.Department of Psychiatry, M.S. Hershey Medical CenterPenn State University College of MedicineHersheyUSA
  5. 5.Formerly with Drexel UniversityCurrently: University of NevadaLas VegasUSA

Personalised recommendations