The merger of game-based approaches and Virtual Reality (VR) environments that can enhance learning and training methodologies have a very promising future, reinforced by the widespread market-availability of affordable software and hardware tools for VR-environments. Rather than passive observers, users engage in those learning environments as active participants, permitting the development of exploration-based learning paradigms. There are separate reviews of VR technologies and serious games for educational and training purposes with a focus on only one knowledge area. However, this review covers 135 proposals for serious games in immersive VR-environments that are combinations of both VR and serious games and that offer end-user validation. First, an analysis of the forum, nationality, and date of publication of the articles is conducted. Then, the application domains, the target audience, the design of the game and its technological implementation, the performance evaluation procedure, and the results are analyzed. The aim here is to identify the factual standards of the proposed solutions and the differences between training and learning applications. Finally, the study lays the basis for future research lines that will develop serious games in immersive VR-environments, providing recommendations for the improvement of these tools and their successful application for the enhancement of both learning and training tasks.
Sutherland described “The Ultimate Display”  as “a room within which the computer can control the existence of matter”, clearly underlining the immense potential of technological innovation to enhance the learning rates of almost any professional skills training. Teaching has therefore to adapt itself to this new technology, quite unlike traditional oral-based education that is mainly focused on abstract rather than practical learning skills, resulting in a weaker and less robust understanding of the topic . However, Virtual Reality (VR) environments have been excluded from educational settings, due to the high cost of VR equipment. Their usage over the past 50 years has been restricted to military applications and research institutes . Throughout that time, research objectives have been focused on technological issues: the development of VR-environments and both hardware and software [13, 162]. In parallel, educational researchers have described any educational experience that introduces the user to visual and auditory experiences as a “virtual world”. The reviews on these topics have underlined both learning [72, 120] methods that employ conventional computer graphics on a monitor or other 2D displays. This concept of virtual worlds is nowadays categorized as low-immersive VR.
Some 15 years ago, high-immersive VR emerged with the development of devices that surround the user in large 3D viewing areas, such as the Head-Mounted Display (HMD) and the Cave Automatic Virtual Environment (CAVE) . The development of those devices was accompanied by the first VR-environments applied to educational tasks in specific knowledge areas: mathematics, language, business, health, computer science, and project management [9, 37, 62]. The main reviews of these initial educational VR experiences outlined their two guiding principles: 1) the fascination among young people with new technologies, including the clear example of VR, suggests greater interest in learning in those environments ; and, 2) VR could facilitate a visual understanding of complex concepts  for students and reduce misconceptions .
This first generation of immersive VR devices was also applied to training. The high cost of VR equipment was no obstacle to the military that exploited the effectiveness of simulation exercises. VR-based simulations offered a secure space to conduct exercises that would otherwise be risky and costly in real life. [79, 109]. These devices were also tested in training for sports [69, 99] and especially in industry, where new employees receive ‘risk-free’ training in a virtual manufacturing scenario . Finally, medicine and especially surgery are also considered promising fields for VR training .
At this stage in the incorporation of the VR learning environment into traditional learning methods, a debate emerged over which procedures could best achieve the perception of a user presence in the VR-environment. This feeling of immersion and presence is identified as a key factor for the enhancement of learning rates . Presence might be defined as the immediate perception of the user of “being there” and a feeling of existing inside the virtual environment . Presence is therefore a very subjective experience. Immersion can be defined as the technological fidelity of VR that the hardware and software can evoke  and it can be objectively evaluated. Immersion is therefore considered in this review as a better key objective for VR experiences than presence.
However, immersion and presence have only been key objectives of VR experiences nowadays, because of the improvements, over the last five years, in the quality of HMDs and their significant reduction in cost (e.g. the launch in 2013 of Oculus Rift™ dk1). Moreover, the second bottleneck for the large-scale development of VR-environments, the software tools, was eased with the launch of the free versions of two powerful motor engines: Unreal Engine™ and Unity™. These new software programs have permitted the rapid development of user interactions with the VR-environment, opening the way towards the design of serious games in VR immersive environments.
However, although the VR-environment will produce the effect of immersion, a second element is required to achieve high learning rates: user interactivity with the VR-environment. The use of games is the natural way to achieve high levels of interactivity. Serious Games (SGs) are activities designed to entertain users in an environment from which they can also learn and be educated and trained in well-defined areas and tasks. Unlike traditional teaching environments, where the teacher controls the learning (e.g., teacher-centered), SGs present a learner-centered approach to education. The trainee feels in control of an interactive learning process in an SG, thereby facilitating active and critical learning . Different reviews have described the use of SGs in education and training. Malegiannaki  analyzed the use of spatial games in formal education related to Cultural Heritage issues, concluding that there were still many challenges relating to effective storytelling and the evaluation of the effect on student learning performance. Ibrahim  reviewed serious games in programming education, seeking to summarize findings on initial user perceptions towards the use of games in terms of motivation and learning. In the case of training, some researchers  have pointed to the most-effective final use of these experiences, which relates to the recreation of situations that could not otherwise be done in real life, including ethical dilemmas, and dangerous and even impossible situations, in terms of time and space. But all those reviews analyzed serious games which do not use immersive VR-environments, mainly because they have only very recently been launched.
While Virtual Reality Serious Games (VR-SGs) should improve user experiences and, therefore, knowledge acquisition, it is also clear that immersive VR-environments pose new questions on the best way to design efficient serious games for such environments. The main questions that present and future research will have to answer can be directly linked with the different stages of the definition of immersive VR-SGs shown in Fig. 1.
In the first stage, two key items should be clearly defined before creating immersive VR-SGs: the target audience and the application domain. There are four key objectives for a VR-SG: interaction, immersion, user involvement and, to a lesser extent, photorealism . Each objective will play a different role depending on the target public and the application domain.
In the second stage of VR-SG design, the materials necessary for the immersive VR-SGs are created and included in the VR-environment. Different questions can be addressed: which are the best technologies to be used for the construction of the VR-environment? Which is the best game design for a certain application? If a game experience is to be a meaningful experience for players, it needs to have certain basic elements. Interactivity should therefore be designed with clarity: the required inputs and outputs, the short- and long-term goals that shape the player’s experience, a well-designed ramp for beginners to learn the ropes; and a game structure that offers genuine play, rather than quiz-style questions and answers.
Finally, the third stage consists of the evaluation of the VR-SG performance. The evaluation should take four different elements into account: 1) the key factors to be evaluated; 2) the way they are evaluated; 3) the number of individuals testing the serious game; and, 4) the existence or otherwise of a reference group. There is no clear consensus on how to evaluate serious games for educational and training tasks. For example in the case of computing education , this fact has been clearly remarked: “As a result, we can confirm that most evaluations use a simple research design in which, typically, the game is used and afterwards subjective feedback is collected via questionnaires from the learners”. The findings of Egenfeldt-Nielsen also showed that most educational games are evaluated in an ad-hoc manner. An evaluation mode that involves the administration of the game to very small validation groups of end users and then data collection, typically through the administration of a questionnaire .
Two final remarks should be added before finishing this Introduction. First, this review refers to Virtual Reality immersive serious games. Therefore, immersion should be a key factor in the research under analysis. Following this approach, many of the articles identified in a first stage of the survey were excluded from subsequent analysis, because they referred to 2D virtual reality, far removed from the concept of immersiveness that is relevant to the development of 3D HMDs.
Second, this review considers two different approaches to the learning process: the acquisition of new knowledge and the development of new skills. While the first has traditionally been seen as a combination of theory and problem-solving capability, the second has been directly related to practical skills and decision-making ability. However, there is no clear difference in the nature of the final process: learning. Therefore, this review considers both educative and training approaches to the learning process, even though they are analyzed separately, because the VR-SGs listed in the bibliography are carefully thought out, designed and evaluated from different perspectives.
The methodology followed in the literature review was composed of four stages, as shown in Fig. 2 (educational results in bold and training results in italics). First, a search in the databases was performed with the keywords (“virtual reality” OR “head-mounted display”) AND (education OR learning) for educational papers, and (“virtual reality” OR “head-mounted display”) AND (training) for training papers. Two interdisciplinary research databases were used, to ensure an exhaustive search: SCOPUS and Web of Science, both identified as suitable databases for serious games searches . The search was conducted in July 2019. Secondly, some additional references cited in the selected literature were considered, in an example of a snowball effect, as their titles clearly reflected their suitability for inclusion in the survey. Finally, the survey was extended to industrial magazines, VR/AR associations and technical congresses closer to the industry (e.g. the IEEE International Symposium on Mixed and Augmented Reality), to identify industrial efforts to recreate VR simulators for training tasks. But most of the research from those sources contained no quantitative evaluations and was not, therefore, considered in this survey. So only 3 papers, from among the total of 52 articles identified from these sources, could be added to the final survey.
Having filtered out all duplicated papers, 6751 and 4432 articles were considered for the educational and training categories, respectively. Then, their abstracts were read, and irrelevant papers were removed considering the objective of this review. Most of the articles were excluded from the survey, because the core of their work referred to 2D virtual reality, far apart from the concept of 3D immersivity in relation the development of 3D HMDs. In any case, the search was not restricted to new 3D HMDs, so some articles on CAVEs and first-generation 3D HMDs were considered. Then, those articles that focused on VR solutions designed to enhance the recovery of patients from different illnesses and post-operative complications were filtered out, because their evaluation was focused on health indicators, rather than on learning and skills improvement. A total of 171 and 235 relevant articles were left, following that filtering process, under the two categories of education and the training, respectively.
In the fourth and final stage, the full text of each remaining article was analyzed and the articles with no final-user performance evaluation of the virtual environment were filtered out. In all, 68 [1, 3,4,5,6,7,8, 10, 11, 18, 26, 28,29,30, 33,34,35,36, 40, 45, 46, 49, 51, 60, 63,64,65,66,67,68, 76, 80, 81, 83, 86,86,87,89, 94, 97, 101, 102, 104,104,106, 110,110,111,113, 117, 118, 121,120,123, 127,126,129, 132,131,134, 138, 142, 144, 149, 150, 153, 156, 160, 164] and 67 [2, 10, 14, 16, 17, 19, 21,22,23, 25, 27, 31, 38, 39, 41,42,43, 47, 50, 52,53,54,55,56,57,58,59, 61, 70,71,72,74, 77, 78, 82, 85, 91,91,93, 95, 96, 100, 103, 107, 108, 114, 116, 119, 125, 126, 131, 135,134,137, 139, 141, 145, 147, 148, 151, 152, 154, 155, 157,156,159, 161, 163] articles were considered for both surveys, representing a good balance between education and training. This balance was unexpected, because training is only one sector of education as a whole and no immediate explanation was found. Interestingly, other authors have also found similar balances between training and learning, for instance in application to project management software . Although there was an important overlap between the articles of both categories in previous stages of the survey process, no manuscript can be considered in both categories at this final stage. The complete list of these manuscripts with their different classifications is provided in the supplementary material. The sample size in this review is comparable to reviews on similar topics, such as the 102-paper review of serious games for software project management  and the 129-paper review of empirical evidence on computer games . It is also larger than other studies that analyzed virtual educational environments (53 papers)  and the effect of spatial games for cultural heritage (34 papers) .
2.2 Data distribution
Some general ideas on VR-SGs can be directly extracted from the data on year of publication and the main congresses and journals in which the work was published.
Figure 3 shows the temporal evolution of the selected references. As expected, the launches of both VR hardware and software have, since 2015, boosted the number of publications on these topics, while a progressive short-term increase in such publications is still to be expected, although 2018 was an exception in this trend. The low number of articles in 2019 is directly related to the date of survey: before the annual conferences on these topics and after the publication in 2018 of only the first issues of the relevant journals. Although the growing trend is more stable in the training field, this result could change in the short term and further analysis of its evolution over coming years will contribute to a coherent conclusion.
Finally, Fig. 4 shows the distribution of the articles between journals and scientific conferences. The information leads to the direct conclusion that there is a preference for publishing training applications in journals, while educational applications are mainly presented at conferences. If a deeper analysis is done to identify the preferred journals and conferences, the result shows the absence of any established publication forums for VR-SGs. The main congresses detected in the survey for educational applications were: AHFE -Conference on Applied Human Factors and Ergonomics- (3 articles), CHI PLAY -Play, Games and Human-Computer Interaction- (2 articles), AVR -Conference on Augmented Reality, Virtual Reality and Computer Graphics- (2 articles) and EDUCON -IEEE Global Engineering Education Conference Engineering Education Through Student Engagement- (2 articles). The main congresses for training applications were: VAMR -International Conference on Virtual, Augmented and Mixed Reality- (3 articles) and MELECON -Mediterranean Electrotechnical Conference- (2 articles). Likewise, the preferred journals for educational applications in the survey were: Behavior & Information Technology (3 articles) and Virtual Reality (2 articles). The preferred journals for training applications were: IEEE Transactions on Visualization and Computer Graphics (3 articles) and Mathematics, Science and Technology Education (2 articles). The major conferences and journals on these topics therefore included only 29% and 26% of the articles in the survey, respectively. The main reason for this result is the novelty of the topics, which fall outside the scope of established journals with high-impact scores in the Journal of Citation Reports, added to which the conferences on these topics are very recent.
3 Analysis of the article
The results of both surveys are arranged in this section under application domains and target public, technological implementation, game design, performance evaluation procedures and results. The aim of this analysis is the identification of factual standards or differences between the proposed solutions in both fields.
3.1 Application domain and target public
The target audience of the studies was classified into three classes: general public, students and professionals. Figure 5 presents the respective percentages of the articles in the survey that belong to those three classes. For a deeper analysis, the professionals were classified into four subclasses in the training case: teachers, health services, industry, and sports professionals.
Three conclusions may at first sight be extracted from this figure. Firstly, around one fourth of the studies (22% for educational games and 25% for training applications) belong to the class “general public”. Most papers related to VR-SGs for museums and other types of exhibitions belong to that class, where the final user is unrestricted; the papers that study the technological issues of VR and SGs also belong to this class. Secondly, more than two thirds of the educational applications are focused on students at different levels, as there is a natural correlation between students and education. There are studies for all the learning stages, from kindergarten to university, with a higher proportion of studies focused on undergraduate students. A clearly lower proportion of students is found in the training survey; most of them refer to medical applications and focus on training students in different hospital operations, see Fig. 6. Thirdly, almost half of the SG-VRs for training are specifically designed for professionals, mainly in industry and medicine, and less so in educational institutions and sports. It is interesting to note the small niche for VR-SGs to train teachers (e.g. related to the development of skills to detect bullying and to improve presentation skills).
Surprisingly, only medicine presents a significant quantity of articles in both categories (training and education). Medicine therefore appears to be a more mature domain for VR-SGs, because a broader range of final applications has been studied in that area. Unlike medicine, sports and industry only present training applications. As regards education, consideration is mainly given to either students or the general public, with undergraduate students playing a central role. Much remains to be done to find the best orientation of VR-SGs in the various final applications, as the immediate solutions of the pairs ‘education-learning’ and ‘skills-training’ have only recently been extensively applied.
3.2 Technological implementation and game design
Different technical solutions can be selected for the same application, all the more so given the diversity of VR-SGs applications and with such different target publics, as observed in previous subsections. Usually the technical solutions should be based on three choices: the visualization display, the game engine, and game typology. Figures 6, 7 and 8 show the selected HMDs, the game engine and the serious game typology presented in the survey for training and educational applications, respectively.
Figure 6 shows the selected HMDs for training and educational applications. The two branded HMDs presented in the survey -Oculus Rift (in its three versions) and HTC Vive- are the most widely used, as well as cardboards connected to smartphones. The least recent articles of the survey used Sony HMZ-T1, Nvis nvisir sx111, and Emagin z800 HMDs; these HMDs are clustered in the graph, in Fig. 6, under the class “First generation of HMDs”.
Figure 6 shows that Oculus Rift is the most common HMD (>40% of the cases), while HTC Vive is used in around 25% of the applications. The other 35% of applications in use are: 1) low-immersion solutions such as cardboards or Gear VR; 2) very expensive solutions (i.e. CAVEs); and, 3) self-designed or not stated in the article.
Figure 7 shows the selected game engine for both training and educational applications. The game engines presented in the survey were the most widely used in the gaming industry at the time of this research: Unity 3D and Unreal Engine over the last 3 years. XVRtechnology, Worldviz and Ogre3D were mainly used in older works and are clustered in the class “Old game engines”. Figure 7 shows that Unity 3D is the preferred solution, while no other motor engine exceeds 15% of mentions in the references. The most likely reasons for the widespread use of Unity 3D are its low cost and its ease of implementation with HMDs. Besides, a quarter of all the studies (25%) contain no statement of which game engines were used. They usually omit any reference to the development of the VR-SG, limiting themselves to its applications. These VR-SGs were developed by an external provider, so it may be assumed that the researchers were only interested in the application of the VR-SG to certain well-defined tasks and its effects. Finally, although the difference between educational and training solutions was not significant, the educational applications presented a higher use of Unity 3D than the training applications. The articles that describe the use of Unreal Engine were presented over the past three years, a period that coincides with its conversion to free software, which may point to stronger growth in the future for this software that stands out for its photorealistic capabilities, a key factor for training purposes for certain SGs .
Figure 8 shows the game typologies, both for the training and the educational applications, divided into four classes: explorative interaction, explorative, interactive experience and passive experience. Explorative interactions are those games that allow the user to explore and to interact freely with the virtual environment. A more restricted solution is the explorative experience, which allows free exploration of the virtual environment, although no direct interaction. The interactive experiences permit user interaction with the environment, but no free movement through it. Finally, the most restricted solution is the passive experience, in which user interactivity and movement are very limited.
The most common solution, especially for training, is the interactive experience, as shown in Fig. 8. This solution is more affordable than explorative experiences that require the complete development of the VR-environment. In the case of interactive experiences, the VR-environment will only have to be developed in high resolution in the areas where the user is permitted, while any secondary area can be roughly modelled, saving costly human and computational resources . Along the same lines, the number of explorative experiences is very limited, due to their high cost. Besides, no clear use of explorative experiences for both learning and training is evident, because the user has no clear objective in the VR-environment. They are therefore mainly used as complements rather than core educational resources in the educational process. There are very few passive experiences and they are clearly connected to the use of cardboards (see Fig. 9), in view of the useful interactive and explorative experiences provided by those devices, despite their technical limitations. Although, these solutions are not very common, they are presented here because of their very low economic cost for both creation and implementation in the classroom.
The analysis of Fig. 8 leads to the conclusion that the interactive experience is the preferred VR-SG for training and education, due to its balance between costs, technological development, immersive feeling, and potential to stimulate learning and skills improvement. Explorative experiences might be more suitable for research tasks and, although still too expensive for mass use, show a promising potential for future growth.
Figure 9 presents a detailed analysis of the correlation between the different HMDs and the VR-SGs typologies. It compares the use of each kind of 3D Display in the different typologies of VR experiences. This figure shows that explorative and explorative-interaction VR-experiences are only developed for CAVEs and high-quality HMDs such as Oculus Rift and HTC Vive, because of the higher computational capabilities of the workstations that control these devices. In contrast, passive experiences, as mentioned, are clearly connected to the useful interactive and explorative experiences achieved with cardboards, despite their technological limitations.
3.3 Performance evaluation
As previously outlined in the Introduction, one of the most conflicting issues in the use of serious games and VR-environments for education and training is the evaluation of the learning experience. Four different elements should be considered for this evaluation: 1) the key factors that should be evaluated; 2) the way they are evaluated; 3) the number of subjects that test the serious game; 4) and, the existence or otherwise of a reference group.
Regarding the first point, five different key factors were identified from the surveys: user satisfaction, learning rate, skills improvement, immersion and usability. Figure 10 shows the proportion of studies that evaluate these key factors. User satisfaction is not included in this figure, because all the selected articles in the survey evaluated it besides other key factors. As with the target audience, a significant difference between training and educational applications was noted: the educational applications were mainly focused on knowledge acquisition, while the training applications were designed for skills improvement. Despite this clear trend, some educational applications were also focused on skills improvement and some training applications were for knowledge acquisition. In any case, the evaluation of both skills improvement and knowledge acquisition is balanced in the survey, leading to a new question: are VR-SGs equally good for both tasks or is it just a consequence of a balanced survey between training and educational applications? Finally, studies focused on immersion and usability were very rare, although both factors could play a main role in the learning rate, as previous studies have stated . It may therefore be concluded that the researchers considered two key factors -user satisfaction and a key factor directly related to the objective of the experience (whether learning rate or skills improvement)-. However, other key factors such as immersion and usability, which have a direct correlation with a successful experience, were not considered.
In addition, the type of evaluation can generate different results, if it is not performed in a standard way. Figure 11 shows the different methods used to measure the key factors: questionnaires, interviews with users, data recordings, and direct user observation. Figure 11 shows that the questionnaire is the most common solution to evaluate knowledge acquisition in educational applications. The training applications showed a balance between the use of questionnaires and metrics on user experiences directly extracted from the recorded data. The use of the other two types of evaluation -interview with users and direct observation of the user- was very rare, as was the simultaneous use of more than one type of evaluation. In the case of the recorded data, the most common indicators were: 1) physiological data directly correlated with the proposed task, mainly in relation to medical applications; and, 2) the game score in educational applications. This group of metrics appears to be a more objective source of information than questionnaires.
Finally, the number of subjects that test the serious game will add weight to the statistical significance of the conclusions of each study. Figure 12 shows the size of the target group that tested the VR-SGs. There is a trend in the educational studies to use larger target groups than in the training studies, perhaps because the number of students available during the evaluation stage of the study was higher than the number of professionals (e.g. a degree module can have more than a hundred students in a small-medium university, while a medium-sized hospital may have fewer than 20 cardiovascular surgeons). In any case, the size of the target group was very limited compared with other educational applications, as in the case of SGs for teaching computing , where the mean average size was around 50 students. One reason might be due to the high average cost of hardware for VR-environments compared with more traditional learning methodologies.
3.4 Results of the performance evaluation
There is one common conclusion presented in all the articles under analysis: user satisfaction is higher with the VR-SG experience than with other learning methodologies. This conclusion justifies the guiding principle that higher learning rates and skills improvement can be expected from VR-SGs (implying greater engagement, interest and motivation), in comparison with traditional learning and training methods. However, this line of reasoning may only be true in some cases and all possible scenarios should be scientifically validated.
Following this first general conclusion, in each article the pros and cons of the selected technology and methodology are discussed for the corresponding final application. From this discussion, the real value of each article can be understood. Table 1 shows the main conclusions in relation to each of the articles (after removing the conclusion on the increased overall satisfaction with the VR experience). The first three rows refer to positive results: VR-SGs increased the learning rate or improved certain skills compared with other learning or practice techniques. The studies with positive results were classified at three different levels. Item number 1: studies that provided well justified conclusions. Item number 2: studies that showed preliminary results. Finally, item number 3: studies that showed potential results without sufficient justification. Consideration was given to the size of the target audience in this three-point classification and to the existence of a reference group that is taught or trained with a different methodology. These three rows (items number 1 to 3) account for 75% and 86% of the studies on education and training, respectively. Therefore, most of the studies arrived at the following conclusion: VR-SGs are a suitable tool for both educational and training objectives regardless of the technical solution.
Support for the use of VR-SGs in education and training was not forthcoming in all cases: no clear advantage for VR-SGs was observed in 6% and 5% of the studies compared with traditional methodologies. Item numbers 4 to 6 of Table 1 show the percentage of studies that achieved the same performance level for both the reference and VR-SGs group (item number 4), those that achieved worse results with the VR-SGs group (item number 5) and those that arrived at no conclusion (item number 6), mainly because of weaknesses in the experimental design. The proposed tasks for these VR-SGs should be analyzed in detail to understand those negative conclusions. In the educational field, two kinds of VR-SGs showed lower learning rates: those that shared supplementary medical knowledge with undergraduate students and those designed to impart abstract scientific concepts on the curricula of Bachelor degrees. Even though the studies demonstrated lower learning rates than traditional teaching methodologies, they also identified higher levels of motivation, engagement and interest among the students. Lower skills improvements were noted with VR-SGs rather than 2D-screen simulators, in the case of simulators for training, driving, navigation, and pedestrian behavior. Those lower levels of improvement might be due to the low levels of experience with HMD setups among users. Therefore, the use of VR-SGs still has to be optimized in relation to very abstract concepts and skills that require extensive movements within a 3D environment. Finally, around 10% of the studies (shown in Table 1 and Fig. 11) were focused on the evaluation of usability and immersion with no measurement of learning or training goals.
Advancing with this analysis, some conclusions on VR-SG experiences and their impact on training and education can be outlined. Nevertheless, the marked differences between the target audiences and the fields of application of the papers that were surveyed complicate any statistical conclusions on those issues. Regarding their educational impact, most research works pointed (in order of importance) to: 1) the main advantage of these solutions for communicating visually acquired knowledge; 2) greater student motivation when working in a VR-environment rather than in a traditional one; and, 3) the synergies with traditional teaching methodologies, focusing each methodology on different learning topics (e.g. traditional teaching can be used to empower the relationship between different concepts presented in VR-environments with extensive discussions between students moderated by the teacher).
Regarding the impact on training, most studies have (in order of importance) pointed out that: 1) VR-SG solutions have a very interesting cost-effective relation (highly accurate learning, low learning times, high visualization and understanding…); 2) the immediate transfer of behavioral skills in VR-environments to the real world; and, 3) the potential to heighten learning skills in a risk-free environment. Finally, research from both fields has outlined that the impact on training is often measured among final users whose experience of VR-environments and interfaces is very limited. They expect that the impact of VR-SGs will be much higher in the short-term, as those devices permeate daily life and the final users will become familiar with them before any learning/training experiences. The same argument (low user familiarity with VR devices and interfaces) was also mentioned in the studies with negative results for VR-SG solutions as a possible explanation for their poor performance.
4 Future research lines
Different future research lines have been proposed in the articles included in the two surveys: some directly in the present Section and some identified in the discussion of the “Results” Section. Besides, the analysis of the surveys, presented in Sections 3 and 4, raises some open questions.
One of the most demanding improvements proposed in the survey is the use of robust evaluation methods that will increase confidence in the results. This comment has already appeared in the first reviews on Virtual Reality applied to teaching ten years ago . In many cases, the studies used no reference group at all, because they drew no comparison between the performance of their VR-SGs and other learning methodologies. However, most of the study cases with a reference group tested the VR-SGs in target and reference groups of very limited size. Therefore, the enlargement of the size of both groups would be advisable in the future to achieve conclusions with a degree of statistical significance. This lack of comparison or the limited size of testing groups is also mentioned in similar reviews on the analysis of the educational use of video games , SGs for learning software project management , and spatial games for Cultural Heritage topics . Besides, most studies used only one of the following evaluation procedures: questionnaires, user interviews, data recording, and direct user observation. A combination of two of these procedures, especially questionnaires and indicators extracted from data recording, would also increase confidence in the results, especially if standardized questionnaires were created. This strategy would increase the validity and reliability of the conclusions, as others authors have pointed out . The definition of new indicators that are directly connected to learning rates is necessary, in relation to the indicators taken from recorded data. Up until the present, the proposed indicators have only shown a solid relation with the proposed performance of the task in medical applications, while the SG score is the only indicator considered in the educational applications.
Besides, although four different key factors (learning rate, skill improvement, immersion and usability) were identified in this review, only one key factor was measured in the studies under analysis. The development of study cases that evaluate up to three of them would be of great interest, combining learning improvements, immersion and usability. In this way, it will be possible to reach new conclusions on the correlation between the design parameters of the VR-SGs and the learning goals, as other authors have outlined for similar tasks, such as spatial games for cultural heritage  and ball-based sports improvement . Besides, design strategies of VR-SGs may be identified in this way. For instance, VR-SGs have some way to go, before they reach an optimal level of use for teaching very abstract concepts and training skills that require complex movements in a 3D environment. Along those lines, comparative studies of VR-SG efficiency are needed between final users with extensive experience of video-gaming and users whose interests are unrelated to such games.
The two surveys raised some open questions on the best design strategies of the VR-SGs for different learning objectives and final applications. First, are VR-SGs equally efficient at presenting learning tasks and at skills improvement? In those reviews, the VR-SG applications are balanced between skills improvement and knowledge acquisition, although there was no clear evidence that VR-SGs were equally effective at both tasks; a conclusion that arises from the balanced structure of both surveys. Second, has the best design of VR-SGs already been identified for each type of final application? Very few VR-SGs have been designed for skills improvement in education and for knowledge acquisition in industrial tasks (like industry, sports or medicine). In other words, there are very few applications in some fields where VR-SGs might be very effective, but where these applications are not so immediate or expected. Therefore, an effort of imagination and open-thinking will be required to find the best design of VR-SGs in many final applications. Third, should the VR-SGs be embedded in a much lengthier learning process? Nowadays VR-SGs are presented as isolated learning experiences, where previously acquired knowledge can be applied to new problems, exercised in new contexts, thereby motivating students to seek further information. However, no correlation with other learning methodologies exists, nor is there a broader learning process and the main roles to play in this scenario.
There are also strong budget limitations on the VR-SGs analyzed in this study. Up until now, user satisfaction with these experiences has been high, certainly due in part to their novelty. In the near future, the development of a broad offer of VR-commercial games will mean more demanding end-users towards final VR-SG quality. Therefore, the development of low-cost high-visual quality methodologies for the design of VR-environments will be a clear requirement. Along the same lines, VR-SGs based on explorative interaction experiences have, up until the present, been very rare, due to their higher costs. Nevertheless, those experiences might provide higher learning rates than other VR-SG typologies and their use has a strong growth potential that should be studied.
Budget limitations have other consequence for the development of VR-SGs: VR-experiences tend to be very short and short exposure times to knowledge clearly limits the learning rate . Short viewing times were expected in the past, due in part to the immaturity of HMD technology that caused VR sickness syndrome . But those problems now appear to have been resolved with the new generation of HMDs and new strategies for user interaction with the VR-environment . Besides, if longer VR-experiences are developed, the learning time can be considered a key factor and effective time ranges for different learning tasks can be done. However, lengthier VR-SG experiences will depend on two new requirements: 1) a multidisciplinary team with specific skill sets, unlike most of the academic research groups working on these issues; and, 2) the development of rich storytelling VR-SGs with a clear orientation towards the final objective of the learning experience. The absence of oriented storytelling is especially clear in the 10% of studies that concluded that VR provided no improvements, although no clear learning objective was identified in those VR-SGs. The same weakness was also mentioned in the context of spatial games for the teaching of Cultural Heritage .
Finally, Fig. 13 presents a visual summary of the main characteristics of immersive VR-SGs and their application collected in the survey for both education and training tasks. Each of the largest circles is split into four quarters, one for each characteristic of the VR-SGs: target audience, type of game, type of evaluation, and key factors to consider. The surface of each smaller circle is proportional to the number of papers included in each category. The color coding is as follows: red refers to the most common solution nowadays, grey to secondary solutions, and yellow is used for the solutions that appear to be the most promising in the near future.
In the field of education, the majority of the target audience are students, especially university students, perhaps because VR-SGs are easily accessible through university research groups. Interactive experiences evaluated by means of questionnaires, through which knowledge acquisition can be ranked, are perhaps the most balanced means of assessment. However, the development of immersive VR-SGs in the near future will be very different, once they enter into mass production and become affordable products; significant growth is expected for primary school applications and general applications for the public. VR-SGs will be explorative-interactive experiences, due to their greater effectiveness in relation to different audiences and the evaluation will include additional key factors, especially immersion, using various evaluation procedures: from questionnaires to recorded data on personal performance throughout the experience.
With regard to training courses, most target audiences are industrial workers, perhaps due to the high budgets in this sector for training new employees and the imperative need for risk prevention in the workplace. In this field, the interactive experience evaluated by means of recorded data, where skills improvement can be measured, appears to be the most balanced solution. But, significant growth of applications for both students and teachers is likely in the near future; VR-SG will become an explorative-interactive experience and the evaluation will include more key factors, especially complex skills performance and immersion, using different evaluation procedures: from questionnaires to recorded data.
Immersive Virtual Reality Serious Games, if they are not already, will soon be capable of changing the way we perform many learning and training tasks. The technology and therefore the potential of both presence and immersion to boost VR learning processes is advancing at a rapid pace. Nevertheless, a lot of research work remains to be done, before these changes may be introduced at all stages of a learning procedure: from design strategies to the evaluation of key factors. In this review, 86 articles on VR-SGs for education or training have been analyzed. Thousands of papers that might appear to be related to immersive VR-SGs are stored on the main scientific databases. However, the limited size of the sample is because most papers, neither refer to non-immersive solutions, such as 2D virtual reality worlds, nor include a performance evaluation of the VR-environment with final users. Evaluation therefore remains a critical issue to assure reasonable conclusions related to learning rates. The survey analysis has resulted in the following conclusions:
The launch of new high-quality affordable hardware and software media for VR has, since 2015, boosted the number of publications on these topics. A progressive short-term increase in such publications can still be expected. Although there is a lack of well-established publication forums for VR-SGs, there is a preference for training applications to be published in journals, while educational applications are mainly presented in conferences.
VR-SG applications that involve learning and knowledge dissemination have, up until now, been considered for educational purposes, while the applications for industry and sports are still restricted to skills training. Some niches for VR-SGs to be used for training at educational institutions have been identified, such as sensitivity to bullying and motivating presentations for teachers. Medicine seems to be a very mature sector and both kinds of applications (skills improvement and knowledge acquisition) have been developed for hospital staff. Finally, important work remains to be done in the sports and industry sectors to prepare educational VR-SGs of interest that will assist professionals in acquiring the knowledge that they will require.
Oculus Rift was preferred as an HMD rather than HTC Vive, especially in education, perhaps because of its lower price and easier configuration. On the other hand, HTC Vive was slightly preferred for training, certainly because of its better capabilities in video games of the explorative interaction type.
Unity 3D was the preferred game engine, perhaps due to its reliable documentation and easy implementation with HMDs. Use of Unreal Engine in training applications, although in a minority, was of slightly greater significance. One reason might be that Unreal Engine renders more realistic virtual environments than Unity 3D, a key factor for certain VR-SGs that are applied to training.
The interactive experience is the preferred VR-SG for training and education, due to its balance between costs, nowadays-technological development, immersion feeling and the possibilities that users have of learning and improving their skills. Explorative experiences might be more suitable for research tasks. Finally, passive experiences, although very economic, are very limited and rarely achieve significant learning and skill improvements.
Two key factors were usually considered: user satisfaction and an indicator related to the objective of the experience (whether learning rate or skills improvement). Only rarely were other key factors such as immersion and usability considered. Key factors directly related to the user experience should be considered, to assure the success of the VR-experience, and their correlation with the learning rates should be measured.
Explorative and explorative interaction VR-experiences were only developed for CAVEs and high-quality HMDs, because of the higher computational capabilities of the workstations that control these devices. In contrast, passive experiences were clearly connected to the use of cardboards, because of their technological limitations.
Four different types of evaluation systems were found in the survey, although only two played a main role: questionnaires and recorded data. Questionnaires were the most common solution to evaluate knowledge acquisition in educational applications. In training applications, the use of questionnaires was balanced by metrics from the recorded data that were directly related to the user experience. Only very rarely were two types of evaluation procedures used in the same evaluation process.
The target audience was usually of a very limited size, due to the high cost of the hardware compared with the more-conventional teaching solutions. The reference group, if one existed at all, had the same limitation; a fact that limited the emergence of rigorous conclusions from those studies.
A common conclusion in all the articles that were surveyed was the higher user satisfaction with the VR-SG experience than with other learning methodologies. This conclusion was used to justify higher learning rates or skills improvement with VR-SGs rather than with traditional learning and training methodologies.
Only 30% of the studies really demonstrated that VR-SGs enhanced learning and training in their respective domains, while no clear advantage was observed in 10% of the studies with regard to the use of VR-SGs compared with conventional methodologies. This result shows that VR-SGs are still a very open research topic for learning and training.
Nowadays, most of the final users enjoy the experience, but are not sufficiently familiar with the interfaces to benefit from the full potential for learning and training. The design of VR-SGs should therefore include an extensive pre-training stage, in which students gain sufficient skills through their interaction with the VR-environment.
The proposed lines of future research lead us to suggest that immersive VR-SGs will measure many key factors of a different nature within large user groups compared with a significant reference group. These experiences will belong to the explorative interaction experiences category and will be systematically integrated in standard learning programs. Finally, some of the most promising VR-SGs will belong to certain fields of application where potential effectiveness is high, even though they are not frequently employed nowadays.
Abulrub AG, Attridge A, Williams MA (2011) Virtual reality in engineering education: The future of creative learning. Int J Emerg Technol Learn
Adjorlu A, Serafin S (2019) Head-Mounted Display-Based Virtual Reality as a Tool to Teach Money Skills to Adolescents Diagnosed with Autism Spectrum Disorder. In Interactivity, Game Creation, Design, Learning, and Innovation, pp. 450–461
Alaguero M, Checa D, Bustillo A (2017) Measuring the impact of low-cost short-term virtual reality on the user experience, vol. 10324 LNCS
Alhalabi WS (2016) Virtual reality systems enhance students’ achievements in engineering education. Behav. Inf. Technol
Alves Fernandes LM et al (2016) Exploring educational immersive videogames: an empirical study with a 3D multimodal interaction prototype. Behav Inform Technol
Amin A, Gromal D, Tong X, Shaw C (2016) Immersion in cardboard VR compared to a traditional head-mounted display. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Andreoli R et al (2016) Immersivity and Playability Evaluation of a Game Experience in Cultural Heritage. Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection, pp. 814–824
Babu SK, Krishna S, Unnikrishnan R, Bhavani RR (2018) Virtual Reality Learning Environments for Vocational Education: A Comparison Study with Conventional Instructional Media on Knowledge Retention. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 385–389
Backlund P, Hendrix M (2013) Educational Games – Are They Worth The Effort? Games Virtual Worlds Serious Appl
Bailenson JN, Yee N, Blascovich J, Beall AC, Lundblad N, Jin M (2008) The Use of Immersive Virtual Reality in the Learning Sciences: Digital Transformations of Teachers, Students, and Social Context. J Learn Sci 17:102–141
Ball C, Johnsen K (2017) An accessible platform for everyday educational virtual reality. In: 2016 IEEE 2nd Workshop on Everyday Virtual Reality, WEVR 2016
Bell JT, Fogler HS (1995) The Investigation and Application of Virtual Reality as an Educational Tool. Proc Am Soc Eng Educ
Bell JT, Fogler HS (1995) Low Cost Virtual Reality and its Application to Chemical Engineering. Comput Syst Technol Div Commun (Am Inst Chem Eng) 18
Bhargava A, Bertrand JW, Gramopadhye AK, Madathil KC, Babu SV (2018) Evaluating Multiple Levels of an Interaction Fidelity Continuum on Performance and Learning in Near-Field Training Simulations. IEEE Trans Vis Comput Graph 24(4)
Bowman DA, McMahan RP (2007) Virtual reality: How much immersion is enough? Computer (Long Beach Calif)
Bozgeyikli L, Raij A, Katkoori S, Alqasemi R (2017) Effects of instruction methods on user experience in virtual reality serious games. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Bracq M-S et al (2019) Learning procedural skills with a virtual reality simulator: An acceptability study. Nurse Educ Today 79:153–160
Bruno F et al (2018) Virtual dives into the underwater archaeological treasures of South Italy. Virtual Reality 22(2):91–102
Bucher K, Blome T, Rudolph S, von Mammen S (2019) VReanimate II: training first aid and reanimation in virtual reality. J Comput Educ 6(1):53–78
Bustillo A, Alaguero M, Miguel I, Saiz JM, Iglesias LS (2015) A flexible platform for the creation of 3D semi-immersive environments to teach Cultural Heritage. Digit Appl Archaeol Cult Herit 2(4):248–259
Butt AL, Kardong-Edgren S, Ellertson A (2018) Using Game-Based Virtual Reality with Haptics for Skill Acquisition. Clin Simul Nurs
Buttussi F, Chittaro L (2017) Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Trans Vis Comput Graph
Çakiroğlu Ü, Gökoğlu S (2019) Development of fire safety behavioral skills via virtual reality. Comput Educ 133:56–68
Calderón A, Ruiz M (2015) A systematic literature review on serious games evaluation: An application to software project management. Comput Educ
Carbonell-Carrera C, Saorin JL (2017) Virtual Learning Environments to Enhance Spatial Orientation. Eurasia J Math Sci Technol Educ
Caro V, Carter B, Dagli S, Schissler M, Millunchick J (2018) Can Virtual Reality Enhance Learning: A Case Study in Materials Science. In: 2018 IEEE Frontiers in Education Conference (FIE), pp. 1–4
Cerezo CE et al (2019) Virtual reality in cardiopulmonary resuscitation training: a randomized trial. Emergencias Rev la Soc Esp Med Emergencias 31(1):43–46
Chang B, Sheldon L, Si M, Hand A (2012) Foreign language learning in immersive virtual environments. In Proceedings of SPIE - The International Society for Optical Engineering
Checa D, Alaguero M, Arnaiz MA, Bustillo A (2016) Briviesca in the 15th c.: A virtual reality environment for teaching purposes. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Checa D, Bustillo A (2019) Advantages and limits of virtual reality in learning processes: Briviesca in the fifteenth century. Virtual Reality
Checa D, Ramon L, Bustillo A (2019) Virtual Reality Travel Training Simulator for People with Intellectual Disabilities. Augmented Reality, Virtual Reality, and Computer Graphics:385–393
Chen S, Pan Z, Zhang M, Shen H (2013) A case study of user immersion-based systematic design for serious heritage games. Multimed Tools Appl
Cheng K-H, Tsai C-C (2019) A case study of immersive virtual field trips in an elementary classroom: Students’ learning experience and teacher-student interaction behaviors. Comput Educ 140:103600
Cheng A, Yang L, Andersen E (2017) Teaching Language and Culture with a Virtual Reality Game. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ‘17
Chittaro L, Buttussi F (2015) Assessing knowledge retention of an immersive serious game vs. A traditional education method in aviation safety. IEEE Trans Vis Comput Graph
Chu PY, Chen LC, Kung HW, Su SJ (2017) A study on the differences among M3D, S3D and HMD for students with different degrees of spatial ability in design education. Communications in Computer and Information Science
Connolly TM, Boyle EA, Macarthur E, Hainey T, Boyle JM (2012) A systematic literature review of empirical evidence on computer games and serious games. Comput Educ 59:661–686
Dean D, Millward J, Mulligan L, Saleh I, Wise C, Higgins G (2018) Evaluating alternative input techniques for building and construction VR training. In 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 1001–1004
Diez HV, García S, Mujika A, Moreno A, Oyarzun D (2016) Virtual training of fire wardens through immersive 3D environments. In Proceedings of the 21st International Conference on Web3D Technology - Web3D ‘16
Dinis F, Sofia Guimaraes A, Rangel B, Martins J (2017) Development of virtual reality game-based interfaces for civil engineering education. pp. 1195–1202
Dobson HD et al (2003) Virtual reality - New method of teaching anorectal and pelvic floor anatomy. Dis Colon Rectum 46:349–352
Dorozhkin D et al (2017) OR fire virtual training simulator: design and face validity. Surg Endosc Other Interv Tech
dos Santos MCC, Sangalli VA, Pinho MS (2017) Evaluating the Use of Virtual Reality on Professional Robotics Education. In: 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC)
Egenfeldt-Nielsen S (2006) Overview of research on the educational use of video games. Digit Kompet
Erolin C, Reid L, McDougall S (2019) Using virtual reality to complement and enhance anatomy education. J Vis Commun Med:1–9
Fang L, Chen SC (2019) Enhancing the learning of history through VR: The thirteen factories icube experience. In: Lecture Notes in Educational Technology, Springer International Publishing, pp. 37–50
Farahani N et al (2016) Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides. J Pathol Inform
Freina L, Ott M (2015) A literature review on immersive virtual reality in education: state of the art and perspectives. eLearning & Software for Education:133–141
Ghani I, Rafi A, Woods P (2016) Sense of place in immersive architectural virtual heritage environment. In Proceedings of the 2016 International Conference on Virtual Systems and Multimedia, VSMM 2016
Gonzalez DC, Garnique LV (2018) Development of a Simulator with HTC Vive Using Gamification to Improve the Learning Experience in Medical Students. In: 2018 Congreso Internacional de Innovación y Tendencias en Ingeniería (CONIITI), pp. 1–6
Gopinath Bharathi AKB, Tucker CS (2015) Investigating the Impact of Interactive Immersive Virtual Reality Environments in Enhancing Task Performance in Online Engineering Design Activities. International Design Engineering Technical Conferences & Computers and Information in Engineering Conference
Grabowski A, Jankowski J (2015) Virtual Reality-based pilot training for underground coal miners. Saf Sci
Gulec U, Yilmaz M, Isler V, O’Connor RV, Clarke PM (2019) A 3D virtual environment for training soccer referees. Comput Stand Interfaces 64:1–10
Gutierrez F et al. (2008) The effect of degree of immersion upon learning performance in virtual reality simulations for medical education
Gutierrez-Maldonado J, Andres-Pueyo A, Jarne A, Talarn A, Ferrer M, Achotegui J (2017) Virtual reality for training diagnostic skills in anorexia nervosa: A usability assessment. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Harrington CM et al (2018) Development and evaluation of a trauma decision-making simulator in Oculus virtual reality. Am. J. Surg.
Hatsushika D, Nagata K, Hashimoto Y (2018) Underwater VR Experience System for Scuba Training Using Underwater Wired HMD. In OCEANS 2018 MTS/IEEE Charleston, pp. 1–7
Hilfert T, Teizer J, König M (2016) First Person Virtual Reality for Evaluation and Learning of Construction Site Safety
Hong J, Hwang M, Tai K, Tsai C (2018) Training Spatial Ability through Virtual Reality. In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 1204–1205
Hsu WC, Lin HCK, Lin YH (2017) The research of applying Mobile Virtual Reality to Martial Arts learning system with flipped classroom. In Proceedings of the 2017 IEEE International Conference on Applied System Innovation: Applied System Innovation for Modern Technology, ICASI 2017
Huang Y, Churches L, Reilly B (2015) A Case Study on Virtual Reality American Football Training. In: Proceedings of the 2015 Virtual Reality International Conference on ZZZ - VRIC ‘15
Ibrahim R, Yusof RC, Mohamed H, Jaafar A (2011) Students Perceptions of Using Educational Games to Learn Introductory Programming. Comput Inf Sci
Innocenti ED et al (2019) Mobile virtual reality for musical genre learning in primary education. Comput Educ 139:102–117
Isabwe GMN, Moxnes M, Ristesund M, Woodgate D (2018) Children’s interactions within a virtual reality environment for learning chemistry. Advances in Intelligent Systems and Computing
Jackson RL, Winn W (1999) Collaboration and learning in immersive virtual environments. In: Proceedings of the 1999 conference on Computer support for collaborative learning
Jacoby D, Ralph R, Preston N, Coady Y (2019) Immersive and Collaborative Classroom Experiences in Virtual Reality. In: Proceedings of the Future Technologies Conference (FTC) 2018, pp. 1062–1078
Janßen D, Tummel C, Richert A, Isenhardt I (2016) Towards Measuring User Experience, Activation and Task Performance in Immersive Virtual Learning Environments for Students
Johnston APR et al (2018) Journey to the centre of the cell: Virtual reality immersion into scientific data. Nano Sci Technol NVIDIA Traffic 19(2):105–110
Justham LM et al (2004) The use of virtual reality and automatic training devices in sport: A review of technology within cricket and related disciplines. In Proceedings of the 7th Biennial Conference on Engineering Systems Design and Analysis, ESDA 2004
Kahlert T, van de Camp F, Stiefelhagen R (2015) Learning to juggle in an interactive virtual reality environment,” in Communications in Computer and Information Science
Khanal P et al (2014) Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles. J Biomed Inform
Kiili K (2005) ‘Digital game-based learning: Towards an experiential gaming model’, Internet High Educ
Kilteni K, Bergstrom I, Slater M (2013) Drumming in immersive virtual reality: The body shapes the way we play. IEEE Trans Vis Comput Graph
Kleven NF et al (2014) Training nurses and educating the public using a virtual operating room with Oculus Rift. In: Proceedings of the 2014 International Conference on Virtual Systems and Multimedia, VSMM 2014
Korakakis G, Pavlatou EAA, Palyvos JAA, Spyrellis N (2009) 3D visualization types in multimedia applications for science learning: A case study for 8th grade students in Greece. Comput Educ 52(2):390–401
Kwon C (2019) Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies. Virtual Reality 23(1):101–118
Leder J, Horlitz T, Puschmann P, Wittstock V, Schütz A (2019) Comparing immersive virtual reality and powerpoint as methods for delivering safety training: Impacts on risk perception, learning, and decision making. Saf Sci 111:271–286
Lee G, Chung K, Youn C (2017) A Study on the Quantitative Design and Documentation for the VR Based Training Contents Realism. Advanced Multimedia and Ubiquitous Engineering:573–579
Lele A (2013) Virtual reality and its military utility. J Ambient Intell Humaniz Comput 4(1):17–26
Li PP, Chang PL (2017) A study of virtual reality experience value and learning efficiency of museum-using shihsanhang museum as an example. In: Proceedings of the 2017 IEEE International Conference on Applied System Innovation: Applied System Innovation for Modern Technology, ICASI 2017
Li Y, Elmaghraby AS, El-Baz A, Sokhadze EM (2016) Using physiological signal analysis to design affective VR games. In: 2015 IEEE International Symposium on Signal Processing and Information Technology, ISSPIT 2015
Liang H, Liang F, Wu F, Wang C, Chang J (2018) Development of a VR Prototype for Enhancing Earthquake Evacuee Safety. In: Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp. 20:1–20:8
Limniou M, Roberts D, Papadopoulos N (2008) Full immersive virtual environment CAVETM in chemistry education. Comput Educ
Lin F, Ye L, Duffy VG, Su CJ (2002) Developing virtual environments for industrial training. Inf Sci (Ny) 140(1–2):153–170
Liu X, Liu Y, Zhu X, An M, Hu F (2016) Virtual reality based navigation training for astronaut moving in a simulated space station. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Loizides F, El Kater A, Terlikas C, Lanitis A, Michael D (2014) Presenting Cypriot Cultural Heritage in Virtual Reality: A User Evaluation. In Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection, pp. 572–579
Lorenzini C, Carrozzino M, Evangelista C, Tecchia F, Bergamasco M, Angeletaki A (2015) A Virtual Laboratory: An immersive VR experience to spread ancient libraries heritage. In 2015 Digital Heritage International Congress, Digital Heritage 2015
Madden JH et al (2018) Virtual reality as a teaching tool for moon phases and beyond. Physics Education Research Conference Proceedings 2018
Makransky G, Terkildsen TS, Mayer RE (2019) Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn Instr 60:225–236
Malegiannaki I, Daradoumis T (2017) Analyzing the educational design, use and effect of spatial games for cultural heritage: A literature review. Comput Educ
Mallam SC, Nazir S, Renganayagalu SK, Ernstsen J, Veie S, Edwinson AE (2019) Design of Experiment Comparing Users of Virtual Reality Head-Mounted Displays and Desktop Computers. In Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018), pp. 240–249.
Mallaro S, Rahimian P, O’neal EE, Plumert JM, Kearney JK (2017) A Comparison of Head-Mounted Displays vs. Large-Screen Displays for an Interactive Pedestrian Simulator. no. 17
Manouchou E, Stavroulia KE, Ruiz-Harisiou A, Georgiou A, Sella F, Lanitis A (2016) A feasibility study on using virtual reality for understanding deficiencies of high school students. In: Proceedings of the 18th Mediterranean Electrotechnical Conference: Intelligent and Efficient Technologies and Services for the Citizen, MELECON 2016
Marks S, White D, Magdics M (2018) Evaluation of a Virtual Reality Nasal Cavity Education Tool. In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 193–198
Mast MS, Kleinlogel EP, Tur B, Bachmann M (2018) The future of interpersonal skills development: Immersive virtual reality training with virtual humans. Hum. Resour. Dev. Quarterly, pp. 1–17. wileyonlinelibrary.com/journal
Matsas E, Vosniakos G-C, Matsas E, Vosniakos G-C (2008) Design of a virtual reality training system for human–robot collaboration in manufacturing tasks. Int J Interact Des Manuf 11:139–153
Meyer OA, Omdahl MK, Makransky G (2019) Investigating the effect of pre-training when learning through immersive virtual reality and video: A media and methods experiment. Comput Educ 140:103603
Mikropoulos TA, Natsis A (2011) Educational virtual environments: A ten-year review of empirical research (1999-2009). Comput Educ 56:769–780
Miles HC, Pop SR, Watt SJ, Lawrence GP, John NW (2012) A review of virtual environments for training in ball sports. Comput Graph 36(6):714–726
Mitsuhara H, Iguchi K, Shishibori M (2017) Using digital game, augmented reality, and head mounted displays for immediate-action commander training. Int J Emerg Technol Learn
Moesgaard T, Witt M, Fiss J, Warming C, Klubien J, Schoenau-Fog H (2015) Implicit and Explicit Information Mediation in a Virtual Reality Museum Installation and its Effects on Retention and Learning Outcomes Implicit and Explicit Information Mediation in a Virtual Reality Museum Installation and its Effects on Retention and Learning Implicit and Explicit Information Mediation in a Virtual Reality Museum Installation and its Effects on Retention and Learning Outcomes
Muller N, Panzoli D, Galaup M, Lagarrigue P, Jessel J (2017) Learning mechanical engineering in a virtual workshop: A preliminary study on utilisability, utility and acceptability. In 2017 9th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games), pp. 55–62
Nedel L et al (2016) Using Immersive Virtual Reality to Reduce Work Accidents in Developing Countries. IEEE Comput Graph Appl 36(2):36–46
Nicola S, Stoicu-Tivadar L, Patrascoiu A (2018) VR for Education in Information and Tehnology: Application for Bubble Sort. In: 2018 13th International Symposium on Electronics and Telecommunications, ISETC 2018 - Conference Proceedings
Oberdörfer S, Heidrich D, Latoschik ME (2019) Usability of Gamified Knowledge Learning in VR and Desktop-3D. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 175:1–175:13
Oberdörfer S, Latoschik ME (2019) Knowledge Encoding in Game Mechanics: Transfer-Oriented Knowledge Learning in Desktop-3D and VR. Int J Comput Games Technol 2019
Ojados Gonzalez D, Martin-Gorriz B, Ibarra Berrocal I, Macian Morales A, Adolfo Salcedo G, Miguel Hernandez B (2017) Development and assessment of a tractor driving simulator with immersive virtual reality for training to avoid occupational hazards. Comput Electron Agric
Ortegon T et al. (2019) Prototyping Interactive Multimodal VR Epidural Administration. In 2019 IEEE International Conference on Consumer Electronics (ICCE), pp. 1–3
Pallavicini F, Argenton L, Toniazzi N, Aceti L, Mantovani F (2016) Virtual Reality Applications for Stress Management Training in the Military. Aerosp Med Hum Perform
Pallavicini F et al (2018) What distinguishes a traditional gaming experience from one in virtual reality? An exploratory study. Advances in Intelligent Systems and Computing
Papachristos NM, Vrellis I, Mikropoulos TA (2017) A Comparison between Oculus Rift and a Low-Cost Smartphone VR Headset: Immersive User Experience and Learning. In: Proceedings - IEEE 17th International Conference on Advanced Learning Technologies, ICALT 2017
Parong J, Mayer RE (2018) Learning Science in Immersive Virtual Reality. J Educ Psychol
Peixoto B, Pinto D, Krassmann A, Melo M, Cabral L, Bessa M (2019) Using Virtual Reality Tools for Teaching Foreign Languages. New Knowledge in Information Systems and Technologies:581–588
Pérez L, Diez E, Usamentiaga R, García DF (2019) Industrial robot control and operator training using virtual reality interfaces. Comput Ind 109:114–120
Petri G, Gresse von Wangenheim C (2017) How games for computing education are evaluated? A systematic literature review. Comput Educ
Petri K et al (2019) Training using virtual reality improves response behavior in karate kumite. Sport Eng 22(1):2
Pinto D, Peixoto B, Krassmann A, Melo M, Cabral L, Bessa M (2019) Virtual Reality in Education: Learning a Foreign Language. New Knowledge in Information Systems and Technologies:589–597
Polcar J, Horejsi P (2015) Knowledge Acquisition and Cyber Sickness: a Comparison of Vr Devices in Virtual Tours. Mod Mach Sci J
Przkora R, Mcgrady W, Vasilopoulos T, Gravenstein N, Solanki D (2015) Evaluation of the Head-Mounted Display for Ultrasound-Guided Peripheral Nerve Blocks in Simulated Regional Anesthesia. Pain Med (United States)
Psotka J (1995) ‘Immersive training systems: Virtual reality and education and training’, Instr. Sci., vol. 23, no. 5–6, pp. 405–431
Rasheed F, Onkar P, Narula M (2015) Immersive virtual reality to enhance the spatial awareness of students,” 7th Int. Conf. HCI, IndiaHCI
Ray AB, Deb S (2017) Smartphone Based Virtual Reality Systems in Classroom Teaching - A Study on the Effects of Learning Outcome. In: Proceedings - IEEE 8th International Conference on Technology for Education, T4E 2016
Reiners T, Wood LC, Gregory S (2014) Experimental study on consumer-technology supported authentic immersion in virtual environments for education and vocational training. In: Proceedings of ASCILITE 2014 - Annual Conference of the Australian Society for Computers in Tertiary Education
Ritterfeld U, Weber R, Fernandes S, Vorderer P (2004) Think science!: entertainment education in interactive theaters. Comput Entertain
Rojo D, Mayor J, Rueda JJG, Raya L (2019) A Virtual Reality Training Application for Adults With Asperger’s Syndrome. IEEE Comput Graph Appl 39(2):104–111
Roldán JJ, Crespo E, Martín-Barrio A, Peña-Tapia E, Barrientos A (2019) A training system for Industry 4.0 operators in complex assemblies based on virtual reality and process mining. Robot Comput Integr Manuf 59:305–316
Roussos M, Johnson A, Moher T, Leigh J, Vasilakis C, Barnes C (1999) Learning and Building Together in an Immersive Virtual World. Presence Teleoperators Virtual Environ
Roussou M (2004) Examining young learners’ activity within interactive virtual environments. Proceeding 2004 Conf. Interact. Des. Child. Build. a community - IDC ‘04
Roussou M, Slater M (2017) Comparison of the Effect of Interactive versus Passive Virtual Reality Learning Activities in Evoking and Sustaining Conceptual Change. IEEE Trans Emerg Top Comput
Ruikar DD, Hegadi RS, Santosh KC (2018) A Systematic Review on Orthopedic Simulators for Psycho-Motor Skill and Surgical Procedure Training. J Med Syst
Sankaranarayanan G et al (2018) Immersive virtual reality-based training improves response in a simulated operating room fire scenario. Surg Endosc
Šašinka Č et al (2018) Collaborative Immersive Virtual Environments for Education in Geography. ISPRS Int J Geo-Information 8(1)
Selzer MN, Gazcon NF, Larrea ML (2019) Effects of virtual presence and learning outcome using low-end virtual reality systems. Displays 59:9–15
Seo JH et al (2018) Anatomy builder vr: Applying a constructive learning method in the virtual reality canine skeletal system. Advances in Intelligent Systems and Computing
Setiawan A, Agiwahyuanto F, Arsiwi P (2019) A Virtual Reality Teaching Simulation for Exercise During Pregnancy. Int J Emerg Technol Learn 14(1)
Shewaga R, Uribe-Quevedo A, Kapralos B, Alam F (2017) A Comparison of Seated and Room-Scale Virtual Reality in a Serious Game for Epidural Preparation. IEEE Trans Emerg Top Comput
Shi Y, Du J, Ahn CR, Ragan E (2019) Impact assessment of reinforced learning methods on construction workers’ fall risk behavior using virtual reality. Autom Constr 104:197–214
Smith JR, Byrum A, McCormick TM, Young NT, Orban C, Porter CD (2018) A Controlled Study of Stereoscopic Virtual Reality in Freshman Electrostatics
Sousa Santos B et al (2009) Head-mounted display versus desktop for 3D navigation in virtual reality: A user study. Multimed Tools Appl
Stapleton AJ (2004) Serious Games: Serious Opportunities. In Australian Game Developers’ Conference
Stavroulia KE, Ruiz-Harisiou A, Manouchou E, Georgiou K, Sella F, Lanitis A (2016) A 3D virtual environment for training teachers to identify bullying. In Proceedings of the 18th Mediterranean Electrotechnical Conference: Intelligent and Efficient Technologies and Services for the Citizen, MELECON 2016
Stepan K et al (2017) Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol
Steuer J (1992) Defining Virtual Reality: Dimensions Determining Telepresence. J Commun
Stratos A, Loukas R, Dimitris M, Konstantinos G, Dimitris M, George C (2016) A Virtual Reality Application to Attract Young Talents to Manufacturing. Procedia CIRP 57:134–139
Su C-H, Xiao P-W, Fan K-K, Xu S (2017) A Preliminary Study on the Learning Satisfaction and Effectiveness of VR Weight Training Assisting Learning System for Beginners. EURASIA J Math Sci Technol Educ 13(9):1305–8223
Sutherland IE (1965) The ultimate display. Proc Congr Int Fed Inf Process
Sveinbjörnsdóttir B, Jóhannsson SH, Oddsdóttir J, Sigurðardóttir TÞ, Valdimarsson GI, Vilhjálmsson HH (2019) Virtual discrete trial training for teacher trainees. J Multimodal User Interfaces 13(1):31–40
Teräs H, Chang V, Wood LC (2014) Authentic, immersive, and emotional experience in virtual learning environments: The fear of dying as an important learning experience in a simulation
Terlikkas C, Poullis C (2014) Towards a more effective way of presenting Virtual Reality museums exhibits. Comput. Vis. Theory Appl. (VISAPP), 2014 Int. Conf., 2014
Tomas B, Maskeliunas R, Reda B, Vitalija K, Indre J, Mindaugas D (2017) Virtual Reality in Education: New Ways to Learn. Inf Softw Technol:457–465
Tzanavari A, Charalambous-Darden N, Herakleous K, Poullis C (2015) Effectiveness of an Immersive Virtual Environment (CAVE) for teaching pedestrian crossing to children with PDD-NOS. In: Proceedings - IEEE 15th International Conference on Advanced Learning Technologies: Advanced Technologies for Supporting Open Access to Formal and Informal Learning, ICALT 2015
Vahdatikhaki F, El Ammari K, Langroodi AK, Miller S, Hammad A, Doree A (2019) Beyond data visualization: A context-realistic construction equipment training simulators. Autom Constr 106:102853
Violante MG, Vezzetti E, Piazzolla P (2019) Interactive virtual technologies in engineering education: Why not 360° videos? Int J Interact Des Manuf 13(2):729–742
A. Vourvopoulos, A. Ferreira, and S. B. i Badia (2016) NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback. In: Proceedings of the 3rd International Conference on Physiological Computing Systems
Walch M et al (2017) Evaluating VR Driving Simulation from a Player Experience Perspective. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ‘17
Wegner K, Seele S, Buhler H, Misztal S, Herpers R, Schild J (2017) Comparison of Two Inventory Design Concepts in a Collaborative Virtual Reality Serious Game. Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play - CHI PLAY ‘17 Extended Abstracts
Werrlich S, Daniel A, Ginger A, Nguyen P, Notni G (2018) Comparing HMD-Based and Paper-Based Training. In: 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 134–142
Wilkerson W, Avstreih D, Gruppen L, Beier KP, Woolliscroft J (2008) Using immersive simulation for training first responders for mass casualty incidents. Acad Emerg Med
Wilson AS, O’Connor J, Taylor L, Carruthers D (2017) A 3D virtual reality ophthalmoscopy trainer. Clin Teach
Xie Y, Ryder L, Chen Y (2019) Using Interactive Virtual Reality Tools in an Advanced Chinese Language Class: a Case Study. TechTrends 63(3):251–259
Xu J et al (2018) A VR-based the emergency rescue training system of railway accident. Entertain Comput
Youngblut C (1998) Educational Uses of Virtual Reality Technology. IDA Doc. D-2128
Zhang H (2017) Head-mounted display-based intuitive virtual reality training system for the mining industry. Int J Min Sci Technol
Zhang X, Jiang S, de Pablos PO, Lytras MD, Sun Y (2017) How virtual reality affects perceived learning effectiveness: a task–technology fit perspective. Behav Inform Technol 36(5):548–556
This investigation was partially supported by the GruaRV project (Reference Number INVESTUN/18/0002) of the Consejeria de Empleo of the Junta de Castilla y León (Spain).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
About this article
Cite this article
Checa, D., Bustillo, A. A review of immersive virtual reality serious games to enhance learning and training. Multimed Tools Appl 79, 5501–5527 (2020). https://doi.org/10.1007/s11042-019-08348-9
- Virtual reality
- Systematic literature review
- Serious game