Skip to main content
SpringerLink
Log in

Identification of Potential Task Shedding Events Using Brain Activity Data

  • Original Paper
  • Published:
Augmented Human Research Aims and scope Submit manuscript

Abstract

In Human–Machine Teaming environments, it is important to identify potential performance drops due to cognitive overload. If identified correctly, they can help improve the performance of the human–machine system by offloading some tasks to less cognitively overloaded users. This can help prevent user error that can result in critical failures. Also, it can improve productivity by keeping the human operators at an optimal performance state. This paper explores a new method for identifying user cognitive load by a three-class classification using brain activity data and by applying a convolutional neural network and long short-term memory model. The data collected from a set of cognitive benchmark experiments were used to train the model, which was then tested on two separate datasets consisting of more ecologically valid task environments. We experimented with various models built with different benchmark tasks to explore which benchmark tasks were better suited for the prediction of task shedding events in these compound tasks that are more representative of real-world scenarios. We also show that this method can be extended across-tasks and across-subject pools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on eeg using lstm recurrent neural network. Emotion 8(10):355

    Google Scholar 

  2. Ayaz H, Shewokis PA, Curtin A, Izzetoglu M, Izzetoglu K, Onaral B (2011) Using mazesuite and functional near infrared spectroscopy to study learning in spatial navigation. J Vis Exp JoVE 56:e3443

    Google Scholar 

  3. Bandara D, Hirshfield L, Velipasalar S (2019) Classification of affect using deep learning on brain blood flow data. J Near Infra Red Spectrosc 27(3):206–219

    Article  Google Scholar 

  4. Bandara D, Velipasalar S, Bratt S, Hirshfield L (2018) Building predictive models of emotion with functional near-infrared spectroscopy. Int J Hum Comput Stud 110:75–85

    Article  Google Scholar 

  5. Bliss JP, Harden JW, Dischinger Jr HC (2013) Task shedding and control performance as a function of perceived automation reliability and time pressure. In: Proceedings of the human factors and ergonomics society annual meeting, vol 57. SAGE Publications, Los Angeles, CA, pp 635–639

  6. Brumby DP, Salvucci DD, Howes A (2009) Focus on driving: how cognitive constraints shape the adaptation of strategy when dialing while driving. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1629–1638

  7. Chan J, Power S, Chau T (2012) Investigating the need for modelling temporal dependencies in a brain-computer interface with real-time feedback based on near infrared spectra. J Near Infrared Spectrosc 20(1):107–116

    Article  Google Scholar 

  8. Chen JY, Barnes MJ (2014) Human-agent teaming for multirobot control: a review of human factors issues. IEEE Trans Hum Mach Syst 44(1):13–29

    Article  Google Scholar 

  9. Comstock JR Jr, Arnegard RJ (1992) The multi-attribute task battery for human operator workload and strategic behavior research. NASA, Washington

    Google Scholar 

  10. Coyle S, Ward T, Markham C, McDarby G (2004) On the suitability of near-infrared (nir) systems for next-generation brain-computer interfaces. Physiol Meas 25(4):815

    Article  Google Scholar 

  11. Cui X, Bray S, Bryant DM, Glover GH, Reiss AL (2011) A quantitative comparison of nirs and fmri across multiple cognitive tasks. Neuroimage 54(4):2808–2821

    Article  Google Scholar 

  12. Cui X, Bray S, Reiss AL (2010) Speeded near infrared spectroscopy (nirs) response detection. PLoS ONE 5(11):e15474

    Article  Google Scholar 

  13. Davidson PR, Jones RD, Peiris MT (2007) Eeg-based lapse detection with high temporal resolution. IEEE Trans Biomed Eng 54(5):832–839

    Article  Google Scholar 

  14. de Visser E, Parasuraman R (2011) Adaptive aiding of human–robot teaming: effects of imperfect automation on performance, trust, and workload. J Cognit Eng Decis Mak 5(2):209–231

    Article  Google Scholar 

  15. Ferrari M, Quaresima V (2012) A brief review on the history of human functional near-infrared spectroscopy (fnirs) development and fields of application. Neuroimage 63(2):921–935

    Article  Google Scholar 

  16. Greenlee ET, Funke GJ, Warm JS, Sawyer BD, Finomore VS, Mancuso VF, Funke ME, Matthews G (2016) Stress and workload profiles of network analysis: not all tasks are created equal. In: Ahram TZ, Nicholson D (eds) Advances in human factors in cybersecurity. Springer, Berlin, pp 153–166

    Chapter  Google Scholar 

  17. Harvey PO, Fossati P, Pochon JB, Levy R, LeBastard G, Lehéricy S, Allilaire JF, Dubois B (2005) Cognitive control and brain resources in major depression: an fmri study using the n-back task. Neuroimage 26(3):860–869

    Article  Google Scholar 

  18. Hennrich J, Herff C, Heger D, Schultz T (2015) Investigating deep learning for FNIRS based BCI. In: EMBC, pp 2844–2847

  19. Herff C, Heger D, Fortmann O, Hennrich J, Putze F, Schultz T (2014) Mental workload during n-back task-quantified in the prefrontal cortex using fnirs. Front Hum Neurosci 7:935

    Article  Google Scholar 

  20. Hirshfield LM, Solovey ET, Girouard A, Kebinger J, Jacob RJ, Sassaroli A, Fantini S (2009) Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 2185–2194

  21. Hochreiter S, Schmidhuber J (1997) Lstm can solve hard long time lag problems. In: Mozer MC, Jordan MI, Petsche T (eds) Advances in neural information processing systems. MIT Press, Cmbridge, pp 473–479

    Google Scholar 

  22. Huettel SA, Mack PB, McCarthy G (2002) Perceiving patterns in random series: dynamic processing of sequence in prefrontal cortex. Nat Neurosci 5(5):485

    Article  Google Scholar 

  23. Janssen CP, Brumby DP (2010) Strategic adaptation to performance objectives in a dual-task setting. Cognit Sci 34(8):1548–1560

    Article  Google Scholar 

  24. Kontogiannis T, Kossiavelou Z (1999) Stress and team performance: principles and challenges for intelligent decision aids. Saf Sci 33(3):103–128

    Article  Google Scholar 

  25. LeCun Y, Bengio Y et al (1995) Convolutional networks for images, speech, and time series. In: Arbib MA (ed) The handbook of brain theory and neural networks, vol 3361, 10th edn. MIT Press, Cambridge

    Google Scholar 

  26. Liu Y (1996) Queueing network modeling of elementary mental processes. Psychol Rev 103(1):116

    Article  Google Scholar 

  27. Meyer DE, Kieras DE (1997) A computational theory of executive cognitive processes and multiple-task performance: part i. Basic mechanisms. Psychol Rev 104(1):3

    Article  Google Scholar 

  28. Miller WD Jr (2010) The us air force-developed adaptation of the multi-attribute task battery for the assessment of human operator workload and strategic behavior. Technical report, Consortium Research and Fellows Program, Arlington VA

  29. Parasuraman R, Hancock PA (2001) Adaptive control of mental workload. In: Hancock PA, Desmond PA (eds) Human factors in transportation. Stress, workload, and fatigue. Lawrence Erlbaum Associates Publishers, pp 305–320

  30. Pashler HE, Sutherland S (1998) The psychology of attention, vol 15. MIT press, Cambridge

    Google Scholar 

  31. Peck EM, Afergan D, Yuksel BF, Lalooses F, Jacob RJ (2014) Using FNIRS to measure mental workload in the real world. In: Gilleade K (ed) Advances in physiological computing. Springer, Berlin, pp 117–139

    Chapter  Google Scholar 

  32. Reeves B, Lang A, Kim EY, Tatar D (1999) The effects of screen size and message content on attention and arousal. Med Psychol 1(1):49–67

    Article  Google Scholar 

  33. Salvucci DD, Taatgen NA (2010) The multitasking mind. Oxford University Press, Oxford

    Google Scholar 

  34. Sirevaag EJ, Kramer AF, Reisweber M, Wickens CD, Strayer DL, Grenell JF (1993) Assessment of pilot performance and mental workload in rotary wing aircraft. Ergonomics 36(9):1121–1140

    Article  Google Scholar 

  35. Smith ME, Gevins A, Brown H, Karnik A, Du R (2001) Monitoring task loading with multivariate eeg measures during complex forms of human–computer interaction. Hum Factors 43(3):366–380

    Article  Google Scholar 

  36. Solovey ET, Zec M, Garcia Perez EA, Reimer B, Mehler B (2014) Classifying driver workload using physiological and driving performance data: two field studies. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp. 4057–4066

  37. Strenzke R, Schulte A (2011) Modeling the human operator’s cognitive process to enable assistant system decisions. GAPRec 2011:38

    Google Scholar 

  38. Strenzke R, Uhrmann J, Benzler A, Maiwald F, Rauschert A, Schulte A (2011) Managing cockpit crew excess task load in military manned–unmanned teaming missions by dual-mode cognitive automation approaches. In: AIAA guidance, navigation, and control conference, p 6237

  39. Tai K, Chau T (2009) Single-trial classification of nirs signals during emotional induction tasks: towards a corporeal machine interface. J Neuroeng Rehabilit 6(1):39

    Article  Google Scholar 

  40. Tamaki T, Hiwa S, Hachisuka K, Okuno E, Hiroyasu T (2016) Region-of-interest estimation using convolutional neural network and long short-term memory for functional near-infrared spectroscopy data. Front Neuroinform 12:10

    Google Scholar 

  41. Treacy Solovey E, Afergan D, Peck EM, Hincks SW, Jacob RJ (2015) Designing implicit interfaces for physiological computing: guidelines and lessons learned using fnirs. ACM Trans Comput Hum Interaction (TOCHI) 21(6):35

    Article  Google Scholar 

  42. Villringer A, Chance B (1997) Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci 20(10):435–442

    Article  Google Scholar 

  43. Van der Linden D, Frese M, Meijman TF (2003) Mental fatigue and the control of cognitive processes: effects on perseveration and planning. Acta Psychol 113(1):45–65

    Article  Google Scholar 

  44. Wang Q, Cavanagh P, Green M (1994) Familiarity and pop-out in visual search. Percept Psychophys 56(5):495–500

    Article  Google Scholar 

  45. Wickens CD (1991) Processing resources and attention. Mult Task Perform 1991:3–34

    Google Scholar 

  46. Wickens CD, Gutzwiller RS, Santamaria A (2015) Discrete task switching in overload: a meta-analyses and a model. Int J Hum Comput Stud 79:79–84

    Article  Google Scholar 

  47. Wickens CD, McCarley JS (2007) Applied attention theory. CRC Press, London

    Book  Google Scholar 

  48. Wickens CD, Santamaria A, Sebok A (2013) A computational model of task overload management and task switching. In: Proceedings of the human factors and ergonomics society annual meeting, vol 57. SAGE Publications Sage CA, Los Angeles, CA, pp 763–767

  49. Zimmermann R, Marchal-Crespo L, Edelmann J, Lambercy O, Fluet MC, Riener R, Wolf M, Gassert R (2013) Detection of motor execution using a hybrid fnirs-biosignal bci: a feasibility study. J Neuroeng Rehabilit 10(1):4

    Article  Google Scholar 

Download references

Acknowledgements

We thank NSF for supporting this research through NSF Award #1816732.

Author information

Authors and Affiliations

  1. Syracuse University, Syracuse, USA

    Danushka Bandara & Senem Velipasalar

  2. Institute of Cognitive Science, University of Colorado Boulder, Boulder, USA

    Trevor Grant & Leanne Hirshfield

Authors
  1. Danushka Bandara
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Trevor Grant
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leanne Hirshfield
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Senem Velipasalar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danushka Bandara.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bandara, D., Grant, T., Hirshfield, L. et al. Identification of Potential Task Shedding Events Using Brain Activity Data. Augment Hum Res 5, 15 (2020). https://doi.org/10.1007/s41133-020-00034-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41133-020-00034-y

Keywords

Search

Navigation