Brain-Based Indices for User System Symbiosis

  • Jan B. F. van ErpEmail author
  • Hans (J. A. ) Veltman
  • Marc Grootjen
Part of the Human-Computer Interaction Series book series (HCIS)


The future generation user system interfaces need to be user-centric which goes beyond user-friendly and includes understanding and anticipating user intentions. We introduce the concept of operator models, their role in implementing user-system symbiosis, and the usefulness of brain-based indices on for instance effort, vigilance, workload and engagement to continuously update the operator model. Currently, the best understood parameters in the operator model are vigilance and workload. An overview of the currently employed brain-based indices showed that indices for the lower workload levels (often based on power in the alpha and theta band of the EEG) are quite reliable, but good indices for the higher workload spectrum are still missing. We argue that this is due to the complex situation when performance stays optimal despite increasing task demands because the operator invests more effort. We introduce a model based on perceptual control theory that provides insight into what happens in this situations and how this affects physiological and brain-based indices. We argue that a symbiotic system only needs to intervene directly in situations of under and overload, but not in a high workload situation. Here, the system must leave the option to adapt on a short notice exclusively to the operator. The system should lower task demands only in the long run to reduce the risk of fatigue or long recovery times. We end by indicating future operator model parameters that can be reflected by brain-based indices.


Task Demand P300 Amplitude High Workload Task Load Workload Level 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science.


  1. Allison BZ, Polich J (2008) Workload assessment of computer gaming using a single-stimulus event-related potential paradigm. Biol Psychol 77(3):277–283 CrossRefGoogle Scholar
  2. Boff KR (2006) Revolutions and shifting paradigms in human factors & ergonomics. Appl Ergon 37(4 SPEC. ISS.):391–399 CrossRefGoogle Scholar
  3. Besserve M, Philippe M, Florence G, Laurent F, Garnero L, Martinerie J (2008) Prediction of performance level during a cognitive task from ongoing EEG oscillatory activities. Clin Neurophysiol 119(4):897–908 CrossRefGoogle Scholar
  4. Boksem MAS, Meijman TF, Lorist MM (2005) Effects of mental fatigue on attention: An ERP study. Cogn Brain Res 25(1):107–116 CrossRefGoogle Scholar
  5. Brown D, Totterdell P, Nonnan M (eds) (1990) Adaptive User Interfaces. Academic Press, London Google Scholar
  6. Byrne EA, Parasuraman R (1996) Psychophysiology and adaptive automation. Biol Psychol 42(3):249–268 CrossRefGoogle Scholar
  7. Dolan RJ (2002) Neuroscience and psychology: Emotion, cognition, and behavior. Science 298(5596):1191–1194 CrossRefGoogle Scholar
  8. Dongen K, Maanen PP (2005) Design for dynamic task allocation. In: Proceedings of the 7th International Conference on Naturalistic Decision Making Google Scholar
  9. Endsley M (1987) The application of human factors to the development of expert systems for advanced cockpits. In: Proceedings of the Human Factors Society 31st Annual Meeting, pp 1388–1392 Google Scholar
  10. Endsley MR, Kaber DB (1999) Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 462–492 Google Scholar
  11. Fairclough SH, Venables L (2006) Prediction of subjective states from psychophysiology: A multivariate approach. Biol Psychol 71(1):100–110 CrossRefGoogle Scholar
  12. Fitts PM (1951) Human engineering for an effective air navigation and traffic control system. Ohio State University Foundation Report, Columbus, OH Google Scholar
  13. Fournier LR, Wilson GF, Swain CR (1999) Electrophysiological, behavioral, and subjective indexes of workload when performing multiple tasks: Manipulations of task difficulty and training. Int J Psychophysiol 31(2):129–145 CrossRefGoogle Scholar
  14. Fowler B (1994) P300 as a measure of workload during a simulated aircraft landing task. Hum Factors 36(4):670–683 Google Scholar
  15. Freeman FG, Mikulka PJ, Prinzel LJ, Scerbo MW (1999) Evaluation of an adaptive automation system using three EEG indices with a visual tracking task. Biol Psychol 50(1):61–76 CrossRefGoogle Scholar
  16. Gaillard AWK, Kramer AF (2000) Theoretical and methodological issues in psychophysiological research. In: Backs RW, Boucsein W (eds) Engineering Psychophysiology: Issues and Applications. Lawrence Erlbaum Associates, Mahwah (NJ), pp 31–58 Google Scholar
  17. Gevins A, Smith ME, Leong H, McEvoy L, Whitfield S, Du R, Rush G (1998) Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Hum Factors 40(1):79–91 CrossRefGoogle Scholar
  18. Gopher D, Donchin E (1986) Workload—An examination of the concept. In: Boff K, Kaufman L, Thomas J (eds) Handbook of Perception and Human Performance, vol 41(1). Wiley, New York, pp 41–49 Google Scholar
  19. Hancock PA (2009) Mind, Machine and Morality; Toward a philosophy of Human-Technology Symbioses. Ashgate Publishing Limited, Farnham (UK) Google Scholar
  20. Hankins TC, Wilson GF (1998) A comparison of heart rate, eye activity, EEC and subjective measures of pilot mental workload during flight. Aviat Space Environ Med 69(4):360–367 Google Scholar
  21. Hendy KC, East KP, Farrell PSE (2001) An informationprocessing model of operator stress and performance. In: Hancock PA, Desmond PA (eds) Stress, Workload and Fatigue. Lawrence Erlbaum Associates, Mahwah (NJ), pp 34–82 Google Scholar
  22. Hockey GRJ (2003) Operator functional state as a framework for the assessment of performance degradation. In: Hockey GRJ, Gaillard AWK, Burov A (eds) Operator Functional State: The Assessment and Prediction of Human Performance Degradation in Complex Tasks. NATO Science Series. IOS Press, Amsterdam, pp 8–23 Google Scholar
  23. Humphrey DG, Kramer AF (1994) Toward a psychophysiological assessment of dynamic changes in mental workload. Hum Factors 36(1):3–26 Google Scholar
  24. Isreal JB, Chesney GL, Wickens CD, Donchin E (1980a) P300 and tracking difficulty: Evidence for multiple resources in dual-task performance. Psychophysiology 17(3):259–273 CrossRefGoogle Scholar
  25. Isreal JB, Wickens CD, Chesney GL, Donchin E (1980b) Event-related brain potential as an index of display-monitoring workload. Hum Factors 22(2):211–224 Google Scholar
  26. Jung T-P, Makeig S, Stensmo M, Sejnowski TJ (1997) Estimating alertness from the EEG power spectrum. IEEE Trans Biomed Eng 44(1):60–69 CrossRefGoogle Scholar
  27. Kok A (2001) On the utility of P3 amplitude as a measure of processing capacity. Psychophysiology 38(3):557–577 CrossRefGoogle Scholar
  28. Kramer AF, Trejo LJ, Humphrey D (1995) Assessment of mental workload with task-irrelevant auditory probes. Biol Psychol 40(1–2):83–100 CrossRefGoogle Scholar
  29. Kruse AA, Schmorrow DD (2005) Session overview: Foundations of augmented cognition. In: Schmorrow DD (ed) Foundations of Augmented Cognition. Lawrence Erlbaum Associates, Mahwah, NJ, pp 441–445 Google Scholar
  30. Laine TI, Bauer KW Jr, Lanning JW, Russell CA, Wilson GF (2002) Selection of input features across subjects for classifying crewmember workload using artificial neural networks. IEEE Trans Syst Man Cybern Part A, Syst Hum 32(6):691–704 CrossRefGoogle Scholar
  31. Lal SKL, Craig AA (2001) Critical review of the psychophysiology of driver fatigue. Biol Psychol 55(3):173–194 CrossRefGoogle Scholar
  32. Lal SKL, Craig A, Boord P, Kirkup L, Nguyen H (2003) Development of an algorithm for an EEG-based driver fatigue countermeasure. J Saf Res 34(3):321–328 Google Scholar
  33. Makeig S, Jung T-P, Sejnowski TJ (2000) Awareness during drowsiness: Dynamics and electrophysiological correlates. Can J Exp Psychol 54(4):266–273 CrossRefGoogle Scholar
  34. Neerincx MA (2003) Cognitive task load design: Model, methods and examples. In: Hollnagel E (ed) Handbook of Cognitive Task Design. Lawrence Erlbaum Associates, Mahwah, NJ, pp 283–305 CrossRefGoogle Scholar
  35. Parasuraman R, Wilson GF (2008) Putting the brain to work: Neuroergonomics past, present, and future. Hum Factors 50(3):468–474 CrossRefGoogle Scholar
  36. Parasuraman R, Sheridan T, Wickens C (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern 286–297 Google Scholar
  37. Paus T, Zatorre RJ, Hofle N, Caramanos Z, Gotman J, Petrides M, Evans AC (1997) Time-related changes in neural systems underlying attention and arousal during the performance of an auditory vigilance task. J Cogn Neurosci 9(3):392–408 CrossRefGoogle Scholar
  38. Petrantonakis PC, Hadjileontiadis LJ (2009) EEG-based emotion recognition using hybrid filtering and higher order crossings. In: International Conference on Affective Computing & Intelligent Interaction. IEEE, New York Google Scholar
  39. Pope AT, Bogart EH, Bartolome DS (1995) Biocybernetic system evaluates indices of operator engagement in automated task. Biol Psychol 40(1–2):187–195 CrossRefGoogle Scholar
  40. Powers WT (1973) Behavior: The Control of Perception. Aldine, Chicago Google Scholar
  41. Prinzel LJ III, Freeman FG, Scerbo MW, Mikulka PJ, Pope AT (2003) Effects of a psychophysiological system for adaptive automation on performance, workload, and the event-related potential P300 component. Hum Factors 45(4):601–613 CrossRefGoogle Scholar
  42. Rouse WB (1988) Adaptive aiding for human/computer control. Hum Factors 30(4):431–443 Google Scholar
  43. Scerbo M (1996) Theoretical perspectives on adaptive automation. In: Parasuraman R, Mouloua M (eds) Automation and Human Performance: Theory and Applications. Lawrence Erlbaum Associates, Mahwah, pp 37–63 Google Scholar
  44. Scerbo MW, Freeman FG, Mikulka PJ (2000) A biocybernetic system for adaptive automation. In: Backs RW, Boucsein W (eds) Engineering Psychophysiology: Issues and Applications. Lawrence Erlbaum Associates, Mahwah (NJ), pp 241–254 Google Scholar
  45. Schneider-Hufschmidt M, Kühme T, Malinowski U (1993) Adaptive User Interfaces—Principles and Practice. Elsevier, Amsterdam Google Scholar
  46. Schnittger C, Johannes S, Arnavaz A, Munte TF (1997) Relation of cerebral blood flow velocity and level of vigilance in humans. NeuroReport 8(7):1637–1639 CrossRefGoogle Scholar
  47. Shaw TH, Warm JS, Matthews G, Riley M, Weiler EM, Dember WN, Tripp L, Finomore V, Hollander TD (2006) Effects of sensory modality on vigilance performance and cerebral hemovelocity. In: Proceedings of the Human Factors and Ergonomics Society, pp 1619–1623 Google Scholar
  48. Sheridan TB (1997) Function allocation: algorithm, alchemy or apostasy. In: ALLFN’97, pp 307–316 Google Scholar
  49. Sheridan TB, Verplank WL (1978) Human and computer control of undersea teleoperators. MIT Man-Machine Systems Laboratory, Cambridge, MA, Tech. Rep Google Scholar
  50. Smith ME, Gevins A, Brown H, Karnik A, Du R (2001) Monitoring task loading with multivariate EEG measures during complex forms of human-computer interaction. Hum Factors 43(3):366–380 CrossRefGoogle Scholar
  51. Stampi C, Stone P, Michimori A (1995) A new quantitative method for assessing sleepiness: The alpha attenuation test. Work Stress 9(2–3):368–376 CrossRefGoogle Scholar
  52. Torsvall L, Akerstedt T (1987) Sleepiness on the job: Continuously measured EEG changes in train drivers. Electroencephalogr Clin Neurophysiol 66(6):502–511 CrossRefGoogle Scholar
  53. Trejo LJ, Kramer AF, Arnold JA (1995) Event-related potentials as indices of display-monitoring performance. Biol Psychol 40(1–2):33–71 CrossRefGoogle Scholar
  54. Ullsperger P, Freude G, Erdmann U (2001) Auditory probe sensitivity to mental workload changes—An event-related potential study. Int J Psychophysiol 40(3):201–209 CrossRefGoogle Scholar
  55. Van Orden KF, Viirre E, Kobus DA (2007) Augmenting task-centered design with operator state assessment technologies. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), LNAI 4565, pp 212–219 Google Scholar
  56. Veltman JA (2002) A comparative study of psychophysiological reactions during simulator and real flight. Int J Aviat Psychol, pp 33–48 Google Scholar
  57. Veltman JA, Gaillard AWK (1998) Physiological workload reactions to increasing levels of task difficulty. Ergonomics 5:656–669 CrossRefGoogle Scholar
  58. Viirre E, Wing S, Huang R, Strychacz C, Koo C, Stripling R, Cohn J, Chase B, Jung T (2006) EEG markers of spatial disorientation. In: Proc 2nd Int Conf Aug Cog, 1 Google Scholar
  59. Vinge V (2006) 2020 computing: The creativity machine. Nature 440(7083):411 CrossRefGoogle Scholar
  60. Vuckovic A, Radivojevic Chen V, ACN Popovic D (2002) Automatic recognition of alertness and drowsiness from EEG by an artificial neural network. Med Eng Phys 24(5):349–360 CrossRefGoogle Scholar
  61. Warm JS, Matthews G, Parasuraman R (2009) Cerebral hemodynamics and vigilance performance. Mil Psychol 21(SUPPL 1):1 Google Scholar
  62. Watter S, Geffen GM, Geffen LB (2001) The n-back as a dual-task: P300 morphology under divided attention. Psychophysiology 38(6):998–1003 CrossRefGoogle Scholar
  63. Wilson GF, Russell CA (2003) Operator functional state classification using multiple psychophysiological features in an air traffic control task. Hum Factors 45(3):381–389 CrossRefGoogle Scholar
  64. Wilson GF, Purvis B, Skelly J, Fullenkamp P, Davis I (1987) Physiological data used to measure pilot workload in actual flight and simulator conditions. In: Proceedings of the Human Factors Society, 31th Annual Meeting, pp 779–783 Google Scholar
  65. Yamada F (1998) Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game: Useful tools for psychophysiology in ergonomics. Ergonomics 41(5):678–688 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2010

Authors and Affiliations

  • Jan B. F. van Erp
    • 1
    Email author
  • Hans (J. A. ) Veltman
    • 1
  • Marc Grootjen
    • 2
  1. 1.TNO Human FactorsSoesterbergThe Netherlands
  2. 2.EagleScienceHaarlemThe Netherlands

Personalised recommendations