When simulated environments make the difference: the effectiveness of different types of training of car service procedures

Abstract

An empirical analysis was performed to compare the effectiveness of different approaches to training a set of procedural skills to a sample of novice trainees. Sixty-five participants were randomly assigned to one of the following three training groups: (1) learning-by-doing in a 3D desktop virtual environment, (2) learning-by-observing a video (show-and-tell) explanation of the procedures, and (3) trial-and-error. In each group, participants were trained on two car service procedures. Participants were recalled to perform a procedure either 2 or 4 weeks after the training. The results showed that: (1) participants trained through the virtual approach of learning-by-doing performed both procedures significantly better (i.e. p < .05 in terms of errors and time) than people of non-virtual groups, (2) the virtual training group, after a period of non-use, were more effective than non-virtual training (i.e. p < .05) in their ability to recover their skills, (3) after a (simulated) long period from the training—i.e. up to 12 weeks—people who experienced 3D environments consistently performed better than people who received other kinds of training. The results also suggested that independently from the training group, trainees’ visuospatial abilities were a predictor of performance, at least for the complex service procedure, adj R 2 = .460, and that post-training performances of people trained through virtual learning-by-doing are not affected by learning styles. Finally, a strong relationship (p < .001, R 2 = .441) was identified between usability and trust in the use of the virtual training tool—i.e. the more the system was perceived as usable, the more it was perceived as trustable to acquire the competences.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Ahlberg G et al (2007) Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 193:797–804. doi:10.1016/j.amjsurg.2006.06.050

    Article  Google Scholar 

  2. Ai-Lim Lee E, Wong KW, Fung CC (2010) How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Comput Educ 55:1424–1442. doi:10.1016/j.compedu.2010.06.006

    Article  Google Scholar 

  3. Akins R, Tolson H, Cole B (2005) Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 5:37

    Article  Google Scholar 

  4. Alippi C, de Russis C, Piuri V (2003) A neural-network based control solution to air-fuel ratio control for automotive fuel-injection systems. Syst Man Cybern Part C Appl Rev IEEE Trans 33:259–268. doi:10.1109/TSMCC.2003.814035

    Article  Google Scholar 

  5. Anastassova M, Burkhardt J-M (2009) Automotive technicians’ training as a community-of-practice: implications for the design of an augmented reality teaching aid. Appl Ergon 40:713–721. doi:10.1016/j.apergo.2008.06.008

    Article  Google Scholar 

  6. Anastassova M, Burkhardt J-M, Mégard C, Ehanno P (2005) Results from a user-centred critical incidents study for guiding future implementation of augmented reality in automotive maintenance. Int J Ind Ergon 35:67–77. doi:10.1016/j.ergon.2004.08.005

    Article  Google Scholar 

  7. Anderson JR (1982) Acquisition of cognitive skill. Psychol Rev 89:369–406

    Article  Google Scholar 

  8. Arthur JW, Day EA, Bennett JW (1997) Dyadic versus individual training protocols: loss and reacquisition of a complex skill. J Appl Psychol 82:783–791

    Article  Google Scholar 

  9. Arthur W Jr, Bennett W Jr, Stanush PL, McNelly TL (1998) Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform 11:57–101. doi:10.1207/s15327043hup1101_3

    Article  Google Scholar 

  10. Bandura A (1992) Observational learning. In: Squire L (ed) Encyclopedia of learning and memory. Macmillan, New York

    Google Scholar 

  11. Belardinelli C, Blümel E, Müller G, Schenk M (2008) Making the virtual more real: research at the fraunhofer IFF virtual development and training centre. Cogn Process 9:217–224. doi:10.1007/s10339-008-0216-0

    Article  Google Scholar 

  12. Borsci S, Federici S, Lauriola M (2009) On the dimensionality of the system usability scale (SUS): a test of alternative measurement models. Cogn Process 10:193–197. doi:10.1007/s10339-009-0268-9

    Article  Google Scholar 

  13. Borsci S, Kurosu M, Federici S, Mele ML (2013) Computer systems experiences of users with and without disabilities: an evaluation guide for professionals. CRC Press, Boca Raton. doi:10.1201/b15619-1

    Google Scholar 

  14. Borsci S, Lawson G, Broome S (2015) Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Comput Ind 67:17–26. doi:10.1016/j.compind.2014.12.002

    Article  Google Scholar 

  15. Bowman DA, Gabbard JL, Hix D (2002) A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence Teleoperators Virtual Environ 11:404–424. doi:10.1162/105474602760204309

    Article  Google Scholar 

  16. Brooke J (1996) SUS: A “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL (eds) Usability evaluation in industry. Taylor & Francis, London, pp 189–194

    Google Scholar 

  17. Carlson P, Peters A, Gilbert SB, Vance JM, Luse A (2015) Virtual training: learning transfer of assembly tasks visualization and computer graphics. IEEE Trans 21:770–782. doi:10.1109/TVCG.2015.2393871

    Google Scholar 

  18. Christine Roy M, Dewit O, Aubert BA (2001) The impact of interface usability on trust in web retailers. Internet Res 11:388–398. doi:10.1108/10662240110410165

    Article  Google Scholar 

  19. Fitts PM, Posner MI (1967) Human performance. Brooks/Cole, Monterey

    Google Scholar 

  20. Flavián C, Guinalíu M, Gurrea R (2006) The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inf Manag 43:1–14. doi:10.1016/j.im.2005.01.002

    Article  Google Scholar 

  21. Garg AX, Norman GR, Eva KW, Spero L, Sharan S (2002) Is there any real virtue of virtual reality?: the minor role of multiple orientations in learning anatomy from computers. Acad Med 77:S97–S99

    Article  Google Scholar 

  22. Good P (2000) Permutation tests: a practical guide for resampling methods for testing hypotheses. Springer, New York

    Book  MATH  Google Scholar 

  23. Hall CR, Stiles RJ, Horwitz CD (1998) Virtual reality for training: evaluating knowledge retention. In: Virtual reality annual international symposium, 1998. Proceedings, IEEE 1998, pp 18–18, pp 184–189. doi:10.1109/VRAIS.1998.658488

  24. Hamilton EC et al (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 16:406–411. doi:10.1007/s00464-001-8149-z

    Article  Google Scholar 

  25. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Peter AH, Najmedin M (eds) Advances in Psychology, vol 52. North-Holland Press, Amsterdam, pp 139–183. doi:10.1016/S0166-4115(08)62386-9

    Google Scholar 

  26. Jayaram S, Jayaram U, Kim YJ, DeChenne C, Lyons KW, Palmer C, Mitsui T (2007) Industry case studies in the use of immersive virtual assembly. Virtual Real 11:217–228. doi:10.1007/s10055-007-0070-x

    Article  Google Scholar 

  27. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG (1993) simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int Journal Aviat Psychol 3:203–220. doi:10.1207/s15327108ijap0303_3

    Article  Google Scholar 

  28. Kolb DA (1984) Experiential Learning as the Science of Learning and Development. Prentice Hall, Englewood Cliffs

    Google Scholar 

  29. Kolb AY, Kolb DA (2005) The Kolb Learning Style Inventory—Version 3.1 2005 Technical Specifications. Hay Group Holdings Inc, Boston

    Google Scholar 

  30. Lewis JR (2014) Usability: lessons learned … and yet to be learned. Int J Hum Comput Interact 30:663–684. doi:10.1080/10447318.2014.930311

    Article  Google Scholar 

  31. Lewis JR, Sauro J (2009) The factor structure of the system usability scale. In: Kurosu M (ed) Human centered design, vol 5619. Lecture notes in computer science. Springer, Berlin Heidelberg, pp 94–103. doi:10.1007/978-3-642-02806-9_12

  32. Li JR, Khoo LP, Tor SB (2003) Desktop virtual reality for maintenance training: an object oriented prototype system (V-REALISM). Comput Ind 52:109–125. doi:10.1016/S0166-3615(03)00103-9

    Article  Google Scholar 

  33. Lippert SK, Michael Swiercz P (2005) Human resource information systems (HRIS) and technology trust. J Inform Sci 31:340–353. doi:10.1177/0165551505055399

    Article  Google Scholar 

  34. Mcknight DH, Carter M, Thatcher JB, Clay PF (2011) Trust in a specific technology: an investigation of its components and measures ACM Trans Manage. Inf Syst 2:1–25. doi:10.1145/1985347.1985353

    Google Scholar 

  35. Michalos G, Makris S, Papakostas N, Mourtzis D, Chryssolouris G (2010) Automotive assembly technologies review: challenges and outlook for a flexible and adaptive approach. CIRP J Manufact Sci Technol 2:81–91. doi:10.1016/j.cirpj.2009.12.001

    Article  Google Scholar 

  36. Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst E77-D:1321–1329

    Google Scholar 

  37. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou M, Davidson BR (2013) Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. doi:10.1002/14651858.CD006575.pub3

    Google Scholar 

  38. Ottosson S (2002) Virtual reality in the product development process. J Eng Des 13:159–172. doi:10.1080/09544820210129823

    Article  Google Scholar 

  39. Parry G, Newnes L, Huang X (2011) Goods, Products and Services. In: Macintyre M, Parry G, Angelis J (eds) Service Design and Delivery. Service Science: Research and Innovations in the Service Economy. Springer, New York, pp 19–29. doi:10.1007/978-1-4419-8321-3_2

    Chapter  Google Scholar 

  40. Parsons TD, Larson P, Kratz K, Thiebaux M, Bluestein B, Buckwalter JG, Rizzo AA (2004) Sex differences in mental rotation and spatial rotation in a virtual environment. Neuropsychologia 42:555–562. doi:10.1016/j.neuropsychologia.2003.08.014

    Article  Google Scholar 

  41. Peters M, Laeng B, Latham K, Jackson M, Zaiyouna R, Richardson C (1995) A redrawn Vandenberg and Kuse mental rotations test—different versions and factors that affect performance. Brain Cogn 28:39–58. doi:10.1006/brcg.1995.1032

    Article  Google Scholar 

  42. Sasse M (2005) Usability and Trust in Information Systems. In: Mansell R, Collins B (eds) Trust and Crime in Information Societies. Edward Elgar, Cheltenham, pp 319–348

    Google Scholar 

  43. Seth A, Vance J, Oliver J (2011) Virtual reality for assembly methods prototyping: a review. Virtual Real 15:5–20. doi:10.1007/s10055-009-0153-y

    Article  Google Scholar 

  44. Sharples S, Cobb S, Moody A, Wilson JR (2008) Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29:58–69. doi:10.1016/j.displa.2007.09.005

    Article  Google Scholar 

  45. Stefanidis D, Korndorffer JR Jr, Sierra R, Touchard C, Dunne JB, Scott DJ (2005) Skill retention following proficiency-based laparoscopic simulator training. Surgery 138:165–170. doi:10.1016/j.surg.2005.06.002

    Article  Google Scholar 

  46. Stefanidis D et al (2007) Construct and face validity and task workload for laparoscopic camera navigation: virtual reality versus videotrainer systems at the SAGES learning center. Surg Endosc 21:1158–1164. doi:10.1007/s00464-006-9112-9

    Article  Google Scholar 

  47. Stone RT, Watts KP, Zhong P, Wei C-S (2011) Physical and cognitive effects of virtual reality integrated training. Hum Fact J Hum Fact Ergon Soc 53:558–572. doi:10.1177/0018720811413389

    Article  Google Scholar 

  48. Stork A, Sevilmis N, Weber D, Gorecky D, Stahl C, Loskyll M, Michel F (2012) Enabling virtual assembly training in and beyond the automotive industry. In: Virtual Systems and Multimedia (VSMM), 18th international conference on, 2–5 Sept 2012, pp 347–352. doi:10.1109/VSMM.2012.6365944

  49. Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems, Ft. Lauderdale, Florida, USA. ACM, 642626, pp 73–80. doi:10.1145/642611.642626

  50. Wayne DB, Siddall VJ, Butter J, Fudala MJ, Wade LD, Feinglass J, McGaghie WC (2006) A longitudinal study of internal medicine residents’ retention of advanced cardiac life support skills. Acad Med 81:S9–S12

    Article  Google Scholar 

  51. Webel S, Bockholt U, Engelke T, Gavish N, Olbrich M, Preusche C (2013) An augmented reality training platform for assembly and maintenance skills. Robot Auton Syst 61:398–403. doi:10.1016/j.robot.2012.09.013

    Article  Google Scholar 

  52. Xu W, Gao F (2011) Research of virtual reality technology in automotive engine assembly teaching. In: Information technology and artificial intelligence conference (ITAIC), 2011 6th IEEE joint international, 20–22 Aug 2011, pp 167–169. doi:10.1109/itaic.2011.6030177

  53. Yuviler-Gavish N, Gutierrez T, Webel S, Rodriguez J, Tecchia F (2011a) Design guidelines for the development of virtual reality and augmented reality training systems for maintenance and assembly tasks BIO web of conferences 1, pp 29–21, pp 29–24 doi:10.1051/bioconf/20110100029

  54. Yuviler-Gavish N, Yechiam E, Kallai A (2011b) Learning in multimodal training: visual guidance can be both appealing and disadvantageous in spatial tasks. Int J Hum Comput Stud 69:113–122. doi:10.1016/j.ijhcs.2010.11.005

    Article  Google Scholar 

  55. Yuviler-Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ 23:778–798. doi:10.1080/10494820.2013.815221

    Article  Google Scholar 

Download references

Acknowledgments

This paper was completed as part of Live Augmented Reality Training Environments (LARTE)—101509 project. The authors would like to acknowledge the Technology Strategy Board for funding the work.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Simone Borsci.

Appendices

Appendix 1: Demographic profile of participants

Participants were recruited in the area of Nottinghamshire in UK. The overall cohorts were composed for the 69.7 % of European citizen, and 47.7 % of the participants were English native speakers. Professionals, and students from high schools and universities of the city of Nottingham were involved in the study. Demographics data (see Appendix 2, Q from 4 to 9a) showed 35 % of participants had a level of education from high school to college, whereas 43 % have achieved or are studying for a bachelor degree, and 22 % of participants have achieved (or are studying) for a master or higher levels of education (Q5). 55.4 % of participants reported that their experience with LEGO® Technic was quite low—from never to sometimes (Q6). 41.7 % of the participants have previous experience of VE systems (Q7); 63.1 % of participants reported a low level of experience with video games (Q8). 78.5 % of the cohort had never used the Lego® Digital Designer (LDD) system (Q9), and 21.5 % have from rare to fairly often experience in use of this tool. Among the participants, only 13.8 % had experienced sickness in the past using immersive or desktop virtual systems (Q7a–Q9a).

Participants were randomly distributed in three training groups of respectively: 22 (male 13, age M 32.09, SD 9.02), 21 (male 12, age M 28.52, SD 7.04), and 22 (male 14, age M 28.72, SD 7.31) people.

Appendix 2: Scenarios of first and second (target) procedures

  • Scenario of first procedure: The 4×4 Crawler car of a client does not work at all. Your manager promised to the client that you can conduct this procedure immediately, and the client is waiting for you. After a diagnostic test, your colleagues suggest to you to change the entire engine. Therefore, you have to remove the old engine from the car and place the new one as you learned during the training. After the procedure, please use the controller to test the car functioning.

  • Scenario of target procedure: A client reports to your head that his new car, a 4×4 Crawler, has some problems when he turns to the left. After a diagnostic, test your colleagues suggested to you that there is a malfunctioning of the left front damper of the car. Therefore, you have to remove the broken damper and place the new one, as you learned during the training. After the procedure please use the controller to test the car functioning.

Appendix 3: Demographic survey

figurea

Appendix 4: Selection of instructions kit

As a support to operate on the real car, you can use only one of the two types of instruction kit trainer showed to you. Please select the type you prefer to help you during the procedures. Once you put your choice you will be allowed to use only the type of kit you have selected, therefore take the most useful for you:

  • Video manual

  • Paper Manual

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Borsci, S., Lawson, G., Salanitri, D. et al. When simulated environments make the difference: the effectiveness of different types of training of car service procedures. Virtual Reality 20, 83–99 (2016). https://doi.org/10.1007/s10055-016-0286-8

Download citation

Keywords

  • Automotive
  • Car service maintenance
  • Training effectiveness
  • Training evaluation
  • Virtual reality