An empirical analysis was performed to compare the effectiveness of different approaches to training a set of procedural skills to a sample of novice trainees. Sixty-five participants were randomly assigned to one of the following three training groups: (1) learning-by-doing in a 3D desktop virtual environment, (2) learning-by-observing a video (show-and-tell) explanation of the procedures, and (3) trial-and-error. In each group, participants were trained on two car service procedures. Participants were recalled to perform a procedure either 2 or 4 weeks after the training. The results showed that: (1) participants trained through the virtual approach of learning-by-doing performed both procedures significantly better (i.e. p < .05 in terms of errors and time) than people of non-virtual groups, (2) the virtual training group, after a period of non-use, were more effective than non-virtual training (i.e. p < .05) in their ability to recover their skills, (3) after a (simulated) long period from the training—i.e. up to 12 weeks—people who experienced 3D environments consistently performed better than people who received other kinds of training. The results also suggested that independently from the training group, trainees’ visuospatial abilities were a predictor of performance, at least for the complex service procedure, adj R 2 = .460, and that post-training performances of people trained through virtual learning-by-doing are not affected by learning styles. Finally, a strong relationship (p < .001, R 2 = .441) was identified between usability and trust in the use of the virtual training tool—i.e. the more the system was perceived as usable, the more it was perceived as trustable to acquire the competences.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Ahlberg G et al (2007) Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 193:797–804. doi:10.1016/j.amjsurg.2006.06.050
Ai-Lim Lee E, Wong KW, Fung CC (2010) How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Comput Educ 55:1424–1442. doi:10.1016/j.compedu.2010.06.006
Akins R, Tolson H, Cole B (2005) Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 5:37
Alippi C, de Russis C, Piuri V (2003) A neural-network based control solution to air-fuel ratio control for automotive fuel-injection systems. Syst Man Cybern Part C Appl Rev IEEE Trans 33:259–268. doi:10.1109/TSMCC.2003.814035
Anastassova M, Burkhardt J-M (2009) Automotive technicians’ training as a community-of-practice: implications for the design of an augmented reality teaching aid. Appl Ergon 40:713–721. doi:10.1016/j.apergo.2008.06.008
Anastassova M, Burkhardt J-M, Mégard C, Ehanno P (2005) Results from a user-centred critical incidents study for guiding future implementation of augmented reality in automotive maintenance. Int J Ind Ergon 35:67–77. doi:10.1016/j.ergon.2004.08.005
Anderson JR (1982) Acquisition of cognitive skill. Psychol Rev 89:369–406
Arthur JW, Day EA, Bennett JW (1997) Dyadic versus individual training protocols: loss and reacquisition of a complex skill. J Appl Psychol 82:783–791
Arthur W Jr, Bennett W Jr, Stanush PL, McNelly TL (1998) Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform 11:57–101. doi:10.1207/s15327043hup1101_3
Bandura A (1992) Observational learning. In: Squire L (ed) Encyclopedia of learning and memory. Macmillan, New York
Belardinelli C, Blümel E, Müller G, Schenk M (2008) Making the virtual more real: research at the fraunhofer IFF virtual development and training centre. Cogn Process 9:217–224. doi:10.1007/s10339-008-0216-0
Borsci S, Federici S, Lauriola M (2009) On the dimensionality of the system usability scale (SUS): a test of alternative measurement models. Cogn Process 10:193–197. doi:10.1007/s10339-009-0268-9
Borsci S, Kurosu M, Federici S, Mele ML (2013) Computer systems experiences of users with and without disabilities: an evaluation guide for professionals. CRC Press, Boca Raton. doi:10.1201/b15619-1
Borsci S, Lawson G, Broome S (2015) Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Comput Ind 67:17–26. doi:10.1016/j.compind.2014.12.002
Bowman DA, Gabbard JL, Hix D (2002) A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence Teleoperators Virtual Environ 11:404–424. doi:10.1162/105474602760204309
Brooke J (1996) SUS: A “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL (eds) Usability evaluation in industry. Taylor & Francis, London, pp 189–194
Carlson P, Peters A, Gilbert SB, Vance JM, Luse A (2015) Virtual training: learning transfer of assembly tasks visualization and computer graphics. IEEE Trans 21:770–782. doi:10.1109/TVCG.2015.2393871
Christine Roy M, Dewit O, Aubert BA (2001) The impact of interface usability on trust in web retailers. Internet Res 11:388–398. doi:10.1108/10662240110410165
Fitts PM, Posner MI (1967) Human performance. Brooks/Cole, Monterey
Flavián C, Guinalíu M, Gurrea R (2006) The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inf Manag 43:1–14. doi:10.1016/j.im.2005.01.002
Garg AX, Norman GR, Eva KW, Spero L, Sharan S (2002) Is there any real virtue of virtual reality?: the minor role of multiple orientations in learning anatomy from computers. Acad Med 77:S97–S99
Good P (2000) Permutation tests: a practical guide for resampling methods for testing hypotheses. Springer, New York
Hall CR, Stiles RJ, Horwitz CD (1998) Virtual reality for training: evaluating knowledge retention. In: Virtual reality annual international symposium, 1998. Proceedings, IEEE 1998, pp 18–18, pp 184–189. doi:10.1109/VRAIS.1998.658488
Hamilton EC et al (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 16:406–411. doi:10.1007/s00464-001-8149-z
Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Peter AH, Najmedin M (eds) Advances in Psychology, vol 52. North-Holland Press, Amsterdam, pp 139–183. doi:10.1016/S0166-4115(08)62386-9
Jayaram S, Jayaram U, Kim YJ, DeChenne C, Lyons KW, Palmer C, Mitsui T (2007) Industry case studies in the use of immersive virtual assembly. Virtual Real 11:217–228. doi:10.1007/s10055-007-0070-x
Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG (1993) simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int Journal Aviat Psychol 3:203–220. doi:10.1207/s15327108ijap0303_3
Kolb DA (1984) Experiential Learning as the Science of Learning and Development. Prentice Hall, Englewood Cliffs
Kolb AY, Kolb DA (2005) The Kolb Learning Style Inventory—Version 3.1 2005 Technical Specifications. Hay Group Holdings Inc, Boston
Lewis JR (2014) Usability: lessons learned … and yet to be learned. Int J Hum Comput Interact 30:663–684. doi:10.1080/10447318.2014.930311
Lewis JR, Sauro J (2009) The factor structure of the system usability scale. In: Kurosu M (ed) Human centered design, vol 5619. Lecture notes in computer science. Springer, Berlin Heidelberg, pp 94–103. doi:10.1007/978-3-642-02806-9_12
Li JR, Khoo LP, Tor SB (2003) Desktop virtual reality for maintenance training: an object oriented prototype system (V-REALISM). Comput Ind 52:109–125. doi:10.1016/S0166-3615(03)00103-9
Lippert SK, Michael Swiercz P (2005) Human resource information systems (HRIS) and technology trust. J Inform Sci 31:340–353. doi:10.1177/0165551505055399
Mcknight DH, Carter M, Thatcher JB, Clay PF (2011) Trust in a specific technology: an investigation of its components and measures ACM Trans Manage. Inf Syst 2:1–25. doi:10.1145/1985347.1985353
Michalos G, Makris S, Papakostas N, Mourtzis D, Chryssolouris G (2010) Automotive assembly technologies review: challenges and outlook for a flexible and adaptive approach. CIRP J Manufact Sci Technol 2:81–91. doi:10.1016/j.cirpj.2009.12.001
Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst E77-D:1321–1329
Nagendran M, Gurusamy KS, Aggarwal R, Loizidou M, Davidson BR (2013) Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. doi:10.1002/14651858.CD006575.pub3
Ottosson S (2002) Virtual reality in the product development process. J Eng Des 13:159–172. doi:10.1080/09544820210129823
Parry G, Newnes L, Huang X (2011) Goods, Products and Services. In: Macintyre M, Parry G, Angelis J (eds) Service Design and Delivery. Service Science: Research and Innovations in the Service Economy. Springer, New York, pp 19–29. doi:10.1007/978-1-4419-8321-3_2
Parsons TD, Larson P, Kratz K, Thiebaux M, Bluestein B, Buckwalter JG, Rizzo AA (2004) Sex differences in mental rotation and spatial rotation in a virtual environment. Neuropsychologia 42:555–562. doi:10.1016/j.neuropsychologia.2003.08.014
Peters M, Laeng B, Latham K, Jackson M, Zaiyouna R, Richardson C (1995) A redrawn Vandenberg and Kuse mental rotations test—different versions and factors that affect performance. Brain Cogn 28:39–58. doi:10.1006/brcg.1995.1032
Sasse M (2005) Usability and Trust in Information Systems. In: Mansell R, Collins B (eds) Trust and Crime in Information Societies. Edward Elgar, Cheltenham, pp 319–348
Seth A, Vance J, Oliver J (2011) Virtual reality for assembly methods prototyping: a review. Virtual Real 15:5–20. doi:10.1007/s10055-009-0153-y
Sharples S, Cobb S, Moody A, Wilson JR (2008) Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29:58–69. doi:10.1016/j.displa.2007.09.005
Stefanidis D, Korndorffer JR Jr, Sierra R, Touchard C, Dunne JB, Scott DJ (2005) Skill retention following proficiency-based laparoscopic simulator training. Surgery 138:165–170. doi:10.1016/j.surg.2005.06.002
Stefanidis D et al (2007) Construct and face validity and task workload for laparoscopic camera navigation: virtual reality versus videotrainer systems at the SAGES learning center. Surg Endosc 21:1158–1164. doi:10.1007/s00464-006-9112-9
Stone RT, Watts KP, Zhong P, Wei C-S (2011) Physical and cognitive effects of virtual reality integrated training. Hum Fact J Hum Fact Ergon Soc 53:558–572. doi:10.1177/0018720811413389
Stork A, Sevilmis N, Weber D, Gorecky D, Stahl C, Loskyll M, Michel F (2012) Enabling virtual assembly training in and beyond the automotive industry. In: Virtual Systems and Multimedia (VSMM), 18th international conference on, 2–5 Sept 2012, pp 347–352. doi:10.1109/VSMM.2012.6365944
Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems, Ft. Lauderdale, Florida, USA. ACM, 642626, pp 73–80. doi:10.1145/642611.642626
Wayne DB, Siddall VJ, Butter J, Fudala MJ, Wade LD, Feinglass J, McGaghie WC (2006) A longitudinal study of internal medicine residents’ retention of advanced cardiac life support skills. Acad Med 81:S9–S12
Webel S, Bockholt U, Engelke T, Gavish N, Olbrich M, Preusche C (2013) An augmented reality training platform for assembly and maintenance skills. Robot Auton Syst 61:398–403. doi:10.1016/j.robot.2012.09.013
Xu W, Gao F (2011) Research of virtual reality technology in automotive engine assembly teaching. In: Information technology and artificial intelligence conference (ITAIC), 2011 6th IEEE joint international, 20–22 Aug 2011, pp 167–169. doi:10.1109/itaic.2011.6030177
Yuviler-Gavish N, Gutierrez T, Webel S, Rodriguez J, Tecchia F (2011a) Design guidelines for the development of virtual reality and augmented reality training systems for maintenance and assembly tasks BIO web of conferences 1, pp 29–21, pp 29–24 doi:10.1051/bioconf/20110100029
Yuviler-Gavish N, Yechiam E, Kallai A (2011b) Learning in multimodal training: visual guidance can be both appealing and disadvantageous in spatial tasks. Int J Hum Comput Stud 69:113–122. doi:10.1016/j.ijhcs.2010.11.005
Yuviler-Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ 23:778–798. doi:10.1080/10494820.2013.815221
This paper was completed as part of Live Augmented Reality Training Environments (LARTE)—101509 project. The authors would like to acknowledge the Technology Strategy Board for funding the work.
Appendix 1: Demographic profile of participants
Participants were recruited in the area of Nottinghamshire in UK. The overall cohorts were composed for the 69.7 % of European citizen, and 47.7 % of the participants were English native speakers. Professionals, and students from high schools and universities of the city of Nottingham were involved in the study. Demographics data (see Appendix 2, Q from 4 to 9a) showed 35 % of participants had a level of education from high school to college, whereas 43 % have achieved or are studying for a bachelor degree, and 22 % of participants have achieved (or are studying) for a master or higher levels of education (Q5). 55.4 % of participants reported that their experience with LEGO® Technic was quite low—from never to sometimes (Q6). 41.7 % of the participants have previous experience of VE systems (Q7); 63.1 % of participants reported a low level of experience with video games (Q8). 78.5 % of the cohort had never used the Lego® Digital Designer (LDD) system (Q9), and 21.5 % have from rare to fairly often experience in use of this tool. Among the participants, only 13.8 % had experienced sickness in the past using immersive or desktop virtual systems (Q7a–Q9a).
Participants were randomly distributed in three training groups of respectively: 22 (male 13, age M 32.09, SD 9.02), 21 (male 12, age M 28.52, SD 7.04), and 22 (male 14, age M 28.72, SD 7.31) people.
Appendix 2: Scenarios of first and second (target) procedures
Scenario of first procedure: The 4×4 Crawler car of a client does not work at all. Your manager promised to the client that you can conduct this procedure immediately, and the client is waiting for you. After a diagnostic test, your colleagues suggest to you to change the entire engine. Therefore, you have to remove the old engine from the car and place the new one as you learned during the training. After the procedure, please use the controller to test the car functioning.
Scenario of target procedure: A client reports to your head that his new car, a 4×4 Crawler, has some problems when he turns to the left. After a diagnostic, test your colleagues suggested to you that there is a malfunctioning of the left front damper of the car. Therefore, you have to remove the broken damper and place the new one, as you learned during the training. After the procedure please use the controller to test the car functioning.
Appendix 3: Demographic survey
Appendix 4: Selection of instructions kit
As a support to operate on the real car, you can use only one of the two types of instruction kit trainer showed to you. Please select the type you prefer to help you during the procedures. Once you put your choice you will be allowed to use only the type of kit you have selected, therefore take the most useful for you:
About this article
Cite this article
Borsci, S., Lawson, G., Salanitri, D. et al. When simulated environments make the difference: the effectiveness of different types of training of car service procedures. Virtual Reality 20, 83–99 (2016). https://doi.org/10.1007/s10055-016-0286-8
- Car service maintenance
- Training effectiveness
- Training evaluation
- Virtual reality