Skip to main content
SpringerLink
Log in

Design and evaluation of an adaptive virtual reality training system

Virtual Reality volume 27, pages 2509–2528 (2023)Cite this article

Abstract

Successful operation of military aviation depends on effective pilot training. The current training capabilities of the United States Air Force might not be sufficient to meet the demand for new pilots. To help resolve this issue, this study focused on developing a prototype of an adaptive virtual reality (VR) training system. The system was built leveraging the three key elements of an adaptive training system including the trainee’s performance measures, adaptive logic, and adaptive variables. The prototype was based on a procedure for an F-16 cockpit and included adaptive feedback, temporal display features, and various difficulty levels to help trainees maintain an optimal level of cognitive workload while completing their training. An experiment with 20 human participants was conducted, and a trend favoring the use of adaptive training was identified. Results suggested that adaptive training could improve performance and reduce workload as compared to the traditional non-adaptive VR-based training. Implementation of adaptive VR training has the potential to reduce training time and cost. The results from this study can assist in developing future adaptive VR-training systems.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  • Abdurrahman UA, Yeh S-C, Wong Y, Wei L (2021) Effects of neuro-cognitive load on learning transfer using a virtual reality-based driving system. Big Data Cogn Comput 5(4):54

    Google Scholar 

  • Abusharha AA (2017) Changes in blink rate and ocular symptoms during different reading tasks. Clin Optom 9:133

    Google Scholar 

  • Ariali S, Zinn B (2021) Adaptive training of the mental rotation ability in an immersive virtual environment. Int J Emerg Technol Learn 16:20

    Google Scholar 

  • Backs RW, Navidzadeh HT, Xu X (2000) Cardiorespiratory indices of mental workload during simulated air traffic control. Proc Human Factors Ergon Soc Annu Meet 44:89–92

    Google Scholar 

  • Beatty J (1982) Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol Bull 91(2):276

    Google Scholar 

  • Belani M (2020) Evaluating virtual reality as a medium for vocational skill training. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems

  • Besson P, Bourdin C, Bringoux L, Dousset E, Maïano C, Marqueste T, Mestre DR, Gaetan S, Baudry J-P, Vercher J-L (2013) Effectiveness of physiological and psychological features to estimate helicopter pilots’ workload: A Bayesian network approach. IEEE Trans Intell Transp Syst 14(4):1872–1881

    Google Scholar 

  • Bhagat KK, Liou W-K, Chang C-Y (2016) A cost-effective interactive 3D virtual reality system applied to military live firing training. Virtual Real 20(2):127–140

    Google Scholar 

  • Bian D, Wade J, Warren Z, Sarkar N (2016) Online engagement detection and task adaptation in a virtual reality based driving simulator for autism intervention

  • Biondi FN, Cacanindin A, Douglas C, Cort J (2021) Overloaded and at work: investigating the effect of cognitive workload on assembly task performance. Hum Factors 63(5):813–820

    Google Scholar 

  • Bloom BS (1984) The 2 sigma problem: the search for methods of group instruction as effective as one-to-one tutoring. Educ Res 13(6):4–16

    Google Scholar 

  • Brookings JB, Wilson GF, Swain CR (1996) Psychophysiological responses to changes in workload during simulated air traffic control. Biol Psychol 42(3):361–377

    Google Scholar 

  • Carretta TR, Dunlap RD (1998) Transfer of training effectiveness in flight simulation: 1986 to 1997

  • Cater JP, Huffman SD (1995) Use of the remote access virtual environment network (RAVEN) for coordinated IVA—EVA astronaut training and evaluation. Presence: Teleoper Virtual Environ 4(2):103–109

    Google Scholar 

  • Causse M, Sénard J-M, Démonet JF, Pastor J (2010) Monitoring cognitive and emotional processes through pupil and cardiac response during dynamic versus logical task. Appl Psychophysiol Biofeedback 35(2):115–123

    Google Scholar 

  • Charles RL, Nixon J (2019) Measuring mental workload using physiological measures: a systematic review. Appl Ergon 74:221–232

    Google Scholar 

  • Chemuturi R, Amirabdollahian F, Dautenhahn K (2013) Adaptive training algorithm for robot-assisted upper-arm rehabilitation, applicable to individualised and therapeutic human-robot interaction. J Neuroeng Rehabil 10(1):1–18

    Google Scholar 

  • Colle HA, Reid GB (1998) Context effects in subjective mental workload ratings. Hum Factors 40(4):591–600

    Google Scholar 

  • Collet C, Salvia E, Petit-Boulanger C (2014) Measuring workload with electrodermal activity during common braking actions. Ergonomics 57(6):886–896

    Google Scholar 

  • Dahlstrom N, Nahlinder S (2009) Mental workload in aircraft and simulator during basic civil aviation training. Int J Aviat Psychol 19(4):309–325

    Google Scholar 

  • Druzdzel MJ (1999) SMILE: structural modeling, inference, and learning engine and GeNIe: a development environment for graphical decision-theoretic models

  • Fairclough SH, Venables L (2006) Prediction of subjective states from psychophysiology: a multivariate approach. Biol Psychol 71(1):100–110

    Google Scholar 

  • Feidakis M (2016) A review of emotion-aware systems for e-learning in virtual environments. Formative assessment, learning data analytics and gamification. Elsevier, New York, pp 217–242

    Google Scholar 

  • Fenton N, Neil M (2018) Risk assessment and decision analysis with Bayesian networks. CRC Press, Boca Raton

    MATH  Google Scholar 

  • Fenton NE, Neil M, Caballero JG (2007) Using ranked nodes to model qualitative judgments in Bayesian networks. IEEE Trans Knowl Data Eng 19(10):1420–1432

    Google Scholar 

  • Fricoteaux L, Thouvenin I, Mestre D (2014) GULLIVER: a decision-making system based on user observation for an adaptive training in informed virtual environments. Eng Appl Artif Intell 33:47–57

    Google Scholar 

  • Garcia AD, Schlueter J, Paddock E (2020) Training astronauts using hardware-in-the-loop simulations and virtual reality

  • Gartner, W. B., & Murphy, M. R. (1979). Concepts of Pattmeo. Survey of methods to assess workload, 3

  • Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ 23(6):778–798

    Google Scholar 

  • Gawron VJ (2008) Human performance, workload, and situational awareness measures handbook. CRC Press, Boca Raton

    Google Scholar 

  • Goettl BP (1993) Analysis of skill on a flight simulator: Implications for training. Proc Hum Factors Ergon Soc Annu Meet 37:1257–1261

    Google Scholar 

  • Grier R, Wickens C, Kaber D, Strayer D, Boehm-Davis D, Trafton JG, John M (2008) The red-line of workload: theory, research, and design

  • Hairston WD, Maldjian JA (2009) An adaptive staircase procedure for the E-Prime programming environment. Comput Methods Programs Biomed 93(1):104–108

    Google Scholar 

  • Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Advances in Psychology, vol 52. Elsevier, New York, pp 139–183

    Google Scholar 

  • Heloir A, Nunnari F, Haudegond S, Havrez C, Lebrun Y, Kolski C (2014) Design and evaluation of a self adaptive architecture for upper-limb rehabilitation. ICTs for Improving patients rehabilitation research techniques. Berlin, pp 196–209

  • Hicks TG, Wierwille WW (1979) Comparison of five mental workload assessment procedures in a moving-base driving simulator. Hum Factors 21(2):129–143

    Google Scholar 

  • Hill SG, Iavecchia HP, Byers JC, Bittner AC Jr, Zaklade AL, Christ RE (1992) Comparison of four subjective workload rating scales. Hum Factors 34(4):429–439

    Google Scholar 

  • Hoepf M, Middendorf M, Epling S, Galster S (2015) Physiological indicators of workload in a remotely piloted aircraft simulation

  • Hunter J (2021) The truth about the air force’s biggest changes to pilot training since the dawn of the jet age. The Drive. Retrieved May 15, 2022 from

  • Jones N, Kiely J, Suraci B, Collins DJ, De Lorenzo D, Pickering C, Grimaldi KA (2016) A genetic-based algorithm for personalized resistance training. Biol Sport 33(2):117

    Google Scholar 

  • Jorna PGAM (1992) Spectral analysis of heart rate and psychological state: a review of its validity as a workload index. Biol Psychol 34(2–3):237–257

    Google Scholar 

  • Kelley CR (1969) What is adaptive training? Hum Factors 11(6):547–556

    Google Scholar 

  • Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG (1993) Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol 3(3):203–220

    Google Scholar 

  • Kramer AF (1991) Physiological metrics of mental workload: a review of recent progress. Multiple-task performance, 279–328

  • Labedan P, Darodes-De-Tailly N, Dehais F, Peysakhovich V (2021) Virtual reality for pilot training: study of cardiac activity

  • Lahiri U, Bekele E, Dohrmann E, Warren Z, Sarkar N (2012) Design of a virtual reality based adaptive response technology for children with autism. IEEE Trans Neural Syst Rehabil Eng 21(1):55–64

    Google Scholar 

  • Landsberg CR, Van Buskirk WL, Astwood Jr RS, Mercado AD, Aakre AJ (2010) Adaptive training considerations for use in simulation-based systems

  • Landsberg CR, Mercado AD, Van Buskirk WL, Lineberry M, Steinhauser N (2012) Evaluation of an adaptive training system for submarine periscope operations

  • Lang Y, Wei L, Xu F, Zhao Y, Yu L-F (2018) Synthesizing personalized training programs for improving driving habits via virtual reality. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR)

  • Losey S (2021) The air force is still short on pilots and hopes tech will help close the gap

  • Luo L, Yin H, Cai W, Lees M, Zhou S (2013) Interactive scenario generation for mission-based virtual training. Comput Anim Virtual Worlds 24(3–4):345–354

    Google Scholar 

  • Maggio MG, Russo M, Cuzzola MF, Destro M, La Rosa G, Molonia F, Bramanti P, Lombardo G, De Luca R, Calabrò RS (2019) Virtual reality in multiple sclerosis rehabilitation: a review on cognitive and motor outcomes. J Clin Neurosci 65:106–111

    Google Scholar 

  • Mariani A, Pellegrini E, Enayati N, Kazanzides P, Vidotto M, De Momi E (2018). Design and evaluation of a performance-based adaptive curriculum for robotic surgical training: A pilot study. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 2162–2165)

  • Marucci M, Di Flumeri G, Borghini G, Sciaraffa N, Scandola M, Pavone EF, Babiloni F, Betti V, Aricò P (2021) The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence. Sci Rep 11(1):1–15

    Google Scholar 

  • May JG, Kennedy RS, Williams MC, Dunlap WP, Brannan JR (1990) Eye movement indices of mental workload. Acta Physiol 75(1):75–89

    Google Scholar 

  • Mazloum A, Kumashiro M, Izumi H, Higuchi Y (2008) Quantitative overload: a source of stress in data-entry VDT work induced by time pressure and work difficulty. Ind Health 46(3):269–280

    Google Scholar 

  • McCarthy C, Pradhan N, Redpath C, Adler A (2016). Validation of the Empatica E4 wristband.

  • Metalis SA (1991) Heart period as a useful index of pilot workload in commercial transport aircraft. Int J Aviat Psychol 1(2):107–116

    Google Scholar 

  • Milstein N, Gordon I (2020) Validating measures of electrodermal activity and heart rate variability derived from the empatica E4 utilized in research settings that involve interactive dyadic states. Front Behav Neurosci. https://doi.org/10.3389/fnbeh.2020.00148

    Article  Google Scholar 

  • Monfort SS, Sibley CM, Coyne JT (2016) Using machine learning and real-time workload assessment in a high-fidelity UAV simulation environment. Next-Generation Analyst IV (Vol. 9851, p. 98510B). International Society for Optics and Photonics

  • Moray N (2013) Mental workload: its theory and measurement, vol 8. Springer Science & Business Media, Cham

    Google Scholar 

  • Moreno A (2019) Combat Jet Cockpit. https://assetstore.unity.com/packages/3d/vehicles/air/combat-jet-cockpit-74709#description

  • Morrison JE, Hammon C (2000) On measuring the effectiveness of large-scale training simulations

  • Mühlberger A, Herrmann MJ, Wiedemann G, Ellgring H, Pauli P (2001) Repeated exposure of flight phobics to flights in virtual reality. Behav Res Ther 39(9):1033–1050

    Google Scholar 

  • Mulder LJM (1992) Measurement and analysis methods of heart rate and respiration for use in applied environments. Biol Psychol 34(2–3):205–236

    Google Scholar 

  • NASA (2010) NASA Human integration design handbook (HIDH)-NASA (Vol. 3407). SP-2010

  • Oberhauser M, Dreyer D (2017) A virtual reality flight simulator for human factors engineering. Cogn Technol Work 19(2):263–277

    Google Scholar 

  • Oberhauser M, Dreyer D, Braunstingl R, Koglbauer I (2018) What’s real about virtual reality flight simulation? Aviat Psychol Appl Hum Factors 8:22–34

    Google Scholar 

  • Orlansky J, Dahlman CJ, Hammon CP, Metzko J, Taylor HL, Youngblut C (1994) The value of simulation for training (IDA Paper P-2982). Institute for Defense Analyses, Alexandria, VA., USA

  • Pallavicini F, Argenton L, Toniazzi N, Aceti L, Mantovani F (2016) Virtual reality applications for stress management training in the military. Aerospace Med Hum Perform 87(12):1021–1030

    Google Scholar 

  • Peretz C, Korczyn AD, Shatil E, Aharonson V, Birnboim S, Giladi N (2011) Computer-based, personalized cognitive training versus classical computer games: a randomized double-blind prospective trial of cognitive stimulation. Neuroepidemiology 36(2):91–99

    Google Scholar 

  • Popovic S, Horvat M, Kukolja D, Dropuljic B, Cosic K (2009) Stress inoculation training supported by physiology-driven adaptive virtual reality stimulation

  • R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing Vienna, Austria. https://www.R-project.org/

  • Radianti J, Majchrzak TA, Fromm J, Wohlgenannt I (2020) A systematic review of immersive virtual reality applications for higher education: design elements, lessons learned, and research agenda. Comput Educ 147:103778

    Google Scholar 

  • Recarte MA, Nunes LM (2000) Effects of verbal and spatial-imagery tasks on eye fixations while driving. J Exp Psychol Appl 6(1):31

    Google Scholar 

  • Rehmann AJ (1995) Handbook of human performance measures and crew requirements for flightdeck research

  • Rogers RO, Boquet A, Howell C, DeJohn CA (2007) An experiment to evaluate transfer of low-cost simulator-based upset-recovery training

  • Roscoe AH (1992) Assessing pilot workload. Why measure heart rate, HRV and respiration ? Biol Psychol 34(23):259–287

    Google Scholar 

  • Roscoe SN, Bergman CA (1980) Flight performance control. Aviation Psychology. Iowa State University Press, Iowa

    Google Scholar 

  • Rossol N, Cheng I, Bischof WF, Basu A (2011) A framework for adaptive training and games in virtual reality rehabilitation environments

  • Saurav K, Dash A, Solanki D, Lahiri U (2018) Design of a VR-based upper limb gross motor and fine motor task platform for post-stroke survivors. In: 17th International conference on computer and information science (ICIS) (pp. 252–257)

  • Schultheis MT, Rizzo AA (2001) The application of virtual reality technology in rehabilitation. Rehabil Psychol 46(3):296

    Google Scholar 

  • Schuurmans AAT, de Looff P, Nijhof KS, Rosada C, Scholte RHJ, Popma A, Otten R (2020) Validity of the Empatica E4 wristband to measure heart rate variability (HRV) parameters: a comparison to electrocardiography (ECG). J Med Syst 44(11):1–11

    Google Scholar 

  • Sirevaag EJ, Kramer AF, Reisweber CDWM, Strayer DL, Grenell JF (1993) Assessment of pilot performance and mental workload in rotary wing aircraft. Ergonomics 36(9):1121–1140

    Google Scholar 

  • Stetz MC, Long CP, Wiederhold BK, Turner DD (2008) Combat scenarios and relaxation training to harden medics against stress. J Cyberther Rehabil 1(3):239–247

    Google Scholar 

  • Steuer J (1992) Defining virtual reality: dimensions determining telepresence. J Commun 42(4):73–93

    Google Scholar 

  • Summa S, Basteris A, Betti E, Sanguineti V (2015) Adaptive training with full-body movements to reduce bradykinesia in persons with Parkinson’s disease: a pilot study. J Neuroeng Rehabil 12(1):1–13

    Google Scholar 

  • Sweller J (1988) Cognitive load during problem solving: Effects on learning. Cogn Sci 12(2):257–285

    Google Scholar 

  • Tattersall AJ, Hockey GRJ (1995) Level of operator control and changes in heart rate variability during simulated flight maintenance. Hum Factors 37(4):682–698

    Google Scholar 

  • Unity (2021) Unity—Manual: Unity User Manual 2020.3 (LTS). Unity Technologies. Retrieved 05/15/2021 from https://docs.unity3d.com/Manual/index.html

  • Veltman JA (2002) A comparative study of psychophysiological reactions during simulator and real flight. Int J Aviat Psychol 12(1):33–48

    Google Scholar 

  • Veltman JA, Gaillard AWK (1996) Physiological indices of workload in a simulated flight task. Biol Psychol 42(3):323–342

    Google Scholar 

  • Wang Z, Zheng L, Lu Y, Fu S (2016) Physiological indices of pilots’ abilities under varying task demands. Aerospace Med Hum Perform 87(4):375–381

    Google Scholar 

  • Wiederhold BK, Wiederhold MD (2008) Virtual reality for posttraumatic stress disorder and stress inoculation training. J Cyberther Rehabil 1(1):23–35

    MathSciNet  Google Scholar 

  • Wilson GF (1992) Applied use of cardiac and respiration measures: practical considerations and precautions. Biol Psychol 34(2–3):163–178

    Google Scholar 

  • Wilson GF, Russell CA (2003) Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum Factors 45(4):635–644

    Google Scholar 

  • Wilson GF, Purvis B, Skelly J, Fullenkamp P, Davis I (1987) Physiological data used to measure pilot workload in actual flight and simulator conditions. Proc Hum Factors Soc Annu Meet 31:779–783

    Google Scholar 

  • Wise JA, Hopkin VD, Garland DJ (2010) In: JA Wise, VD Hopkin, DJ Garland ed Handbook of aviation human factors (2 ed) (2009)

  • Yerkes RM, Dodson JD (1908) The relation of strength of stimulus to rapidity of habit-formation. Punishment: Issues and experiments, 27–41

  • Zadeh LA (1978) Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets Syst 1(1):3–28

    MathSciNet  MATH  Google Scholar 

  • Zahabi M, Abdul Razak AM (2020) Adaptive virtual reality-based training: a systematic literature review and framework. Virtual Real 24(4):725–752

    Google Scholar 

  • Zhang J (2007) Effect of age and sex on heart rate variability in healthy subjects. J Manipulative Physiol Ther 30(5):374–379

    Google Scholar 

  • Zhang Y, Tsai S-B (2021) Application of adaptive virtual reality with ai-enabled techniques in modern sports training. Mobile Inf Syst 2021:1–10

    Google Scholar 

  • Zheng B, Jiang X, Tien G, Meneghetti A, Panton ONM, Atkins MS (2012) Workload assessment of surgeons: correlation between NASA TLX and blinks. Surg Endosc 26(10):2746–2750

    Google Scholar 

  • Zhou J, Asteris PG, Armaghani DJ, Pham BT (2020) Prediction of ground vibration induced by blasting operations through the use of the Bayesian Network and random forest models. Soil Dyn Earthq Eng 139:106390

    Google Scholar 

Download references

Acknowledgements

This study was funded by the Air Force Research Laboratory (AFRL). The opinions and conclusions expressed are solely those of the author(s) and do not represent the opinions or policy of AFRL.

Author information

Authors and Affiliations

  1. Wm. Michael Barnes Department of Industrial and Systems Engineering, Texas A&M University, College Station, TX, 77843-3131, USA

    César Iván Aguilar Reyes, David Wozniak, Angel Ham & Maryam Zahabi

Authors
  1. César Iván Aguilar Reyes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Wozniak
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Angel Ham
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maryam Zahabi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maryam Zahabi.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to report.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aguilar Reyes, C.I., Wozniak, D., Ham, A. et al. Design and evaluation of an adaptive virtual reality training system. Virtual Reality 27, 2509–2528 (2023). https://doi.org/10.1007/s10055-023-00827-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-023-00827-7

Keywords

search

Navigation