Simulators in bridge operations training and assessment: a systematic review and qualitative synthesis

This article presents a systematic review and qualitative synthesis of the use of simulators in maritime education and training (MET), with a focus on bridge operations during navigation training and assessment. The review found 34 articles published in a wide range of academic journals, displaying a global field of research consisting of three main disciplines: Maritime professionals (n = 15), Human factors (n = 13) and Education (n = 6). An important conclusion made after synthesising the results of the studies is that while the potential of using simulators in training and assessment are clear, little is known about which instructional practices would ensure valid and reliable results of simulator-based education. Since MET institutions train their students for one of the most safety-critical industries in the world, there is a need for empirical studies that explore the use of simulator-based training and assessment further to lay the foundation for an evidence-based educational practice.


Introduction
Furthermore, the use of simulators in MET is regulated in the Standard of Training, Certification and Watchkeeping for Seafarers (STCW) convention that provides regulations for training in simulators as well as performance-based competency tests (Section A-I/6; Section A-I/12). The latest update of the STCW code, the Manila amendments 2010, has a greater focus on technical proficiency and the non-technical skills of team management and resource management on the bridge than previous conventions. Although the practice of using simulators is well regulated and widespread in MET, it seems few studies address the pedagogical use and benefits of simulator-based training in this domain. The overall aim of the current systematic review is to create an overview of the research field, common themes of interest, conclusions made to date and to synthesise the collective knowledge of the field at large.
While the systematic review methodology has been commonly used in educational practice in healthcare over the last two decades, and had a great impact of what is known as an Bevidence-based practice^, the use outside of medicine has been limited (Bearman et al. 2012). The benefits of doing systematic reviews are several: ideally a well-executed literature review should make studies assessable and guide the reader towards the literature. It should provide trustworthiness and accountability of the review process, opening up for the readers to make their own judgement of the quality and meaning of the evidence. Moreover, a structured review reveals conceptual and value positions that can otherwise remain hidden within the discourse of different disciplines. The qualitative synthesis approach was chosen since research in the field is cross-disciplinary and consists of both quantitative and qualitative research methods. It is regularly used within systematic reviews as a way of pooling different sets of data and thus gathering collective wisdom from a range of different research methodologies (Bearman and Dawson, 2013).

Method of the systematic review
The systematic review methodology draws on the Cochrane handbook, and the key features of conducting a systematic review in an explicit, reproducible and methodological way as presented in Bearman et al. (2012): & A clearly defined set of objectives with pre-defined inclusion criteria for studies & A systematic search that attempts to identify all studies that would meet the eligibility criteria & An assessment of the validity of the findings of the included studies & A systematic presentation and synthesis of the characteristics and finding of the included studies The criteria for inclusion in this review are that articles should study the use of simulators for training and assessing bridge operations in MET. Furthermore, the studies should be peer-reviewed journal articles, searchable in major academic databases, available in English and published between the years 2000 and 2016.
The keywords that were used were grouped into three categories and combined during the search, following the strategy of Bearman et al. (2015). The first category included words in relation to the maritime domain: Bnavigation^, Bbridge^, Bship^, Bmarine^and Bmaritime^. The second category contained words related to the use of simulators, but to cover all possible variations the search used Bsimula*^. The third category of words used in the search is connected to MET: Btraining^, Bassessment^and Beducation^. A first search was conducted by using Google Scholar, resulting in 71 found articles on simulator-based maritime training and assessment. While Google Scholar is known to include searches from student work, organisational and technical reports, journals lacking in academic credibility and so on, it was necessary to go over the results from the Google Scholar search to ensure the quality of the studies found (cf. Gehanno et al. 2013). After an initial screening, based on whether the articles meet the criterion of being published in a peer-reviewed academic journal, 41 articles remained. After reading through the abstracts and the results sections, ten more papers were excluded from the review, mainly because the scope of the paper was outside the use of simulators for training purposes (e.g., addressing simulator design questions on an algorithmic level rather than their usefulness for training). In all, 31 peer-reviewed journal articles were included in the review after the Google Scholar search.
Although Gehanno et al. (2013) concluded that the coverage of a Google Scholar search was 100 % in comparison to searches in so-called Bgold standard^databases in medicine, others state that Google Scholar should never be used in isolation (e.g. Giustini, 2005). In order to ensure that all relevant studies were included in the review, the next step of the literature search involved two different libraries and two different librarians, one with focus on educational literature and the other from a technological university were guiding the search through the chosen databases: ERIC, Education Research Complete, ProQuest, Scopus, Marine Technology Abstracts and IEEE Explore. Six more items were found of which three was included in the review after screening, showing that a Google Scholar Search alone does not suffice. In the last stage of the article search, items found through recommendations were included (n = 1). A flow chart of the systematic search is presented in Appendix 1, and the result of the systematic search is presented with a detailed overview of the articles in Appendix 2. Table 1 is a summary of the field at large, containing information on the research domain, the methods used, geographical location, as well as a list of academic journals where the studies were published.

The qualitative synthesis
As the studies included in the review are both quantitative and qualitative, the systematic review is followed by a qualitative synthesis to draw conclusions on topics and themes that are recurrent in the articles and the conclusions made to date. A narrative summary approach to qualitative synthesis that aims to produce a selective account of the evidence was chosen, since it allows for going beyond mere description towards interpretations and reflections on a higher level of abstraction than for example thematic analyses (Dixon-Woods et al. 2005). The synthesis is organised in accordance to each of the disciplines with a section providing an insight into the nature of the studies, the major themes and foci identified in the articles, as well as reflections on their contributions towards the MET research field at large.

The knowledge and experience of maritime professionals
As can be seen in Tables 1 and 2, a large amount of the found articles come from what was categorised as Bmaritime professionals^. These are most often well-experienced seafarers and maritime instructors at leading positions in maritime universities, and they come from MET institutions from all over the world. It is also notable that these articles are based on knowledge of the field and experiences rather than empirical data. At times, surveys are used, but mainly in order to collect the experience and opinions of maritime professionals in a wider sense. While some of these articles provide expert advice to inform policymakers in the industry (e.g. Barsan, 2004;Barsan et al. 2007), their main contribution is something other than empirical results to inform an Bevidence-based practice^. They offer an insight into MET, its history and challenges, and formulates important questions that are connected both to Human factors and Education research. Maritime professionals are most often positive towards simulators in training, yet remain concerned about the challenge that MET faces when introducing new technologies and replacing old traditions. Some of the main concerns are the upgrades of the IMO and STCW conventions and the practical impact it will have on MET. In order to ensure that future mariners can act properly and safely in practice, the conventions stress that simulators should be used for training and certification of proficiency and non-technical skills. This in turn raises a number of questions amongst maritime professionals, most commonly, if simulator-based training work in terms of improving safety at sea and reduce human error (e.g. Hanzu-Pazara et al., 2008). Another type of questions is more didactic: What would be the optimum training to ensure nontechnical skills transfer? How can non-technical skills be effectively assessed on both individual and group level? What cultural factors need to be identified and addressed in training? (Pekcan et al. 2005). Also, a number of papers call for simulator technologies that are more advanced, as well as better-trained instructors, to meet these new demands on MET institutions (e.g. Ali, 2008;Hanzu-Pazara et al. 2010).

Human factors research and the usefulness of simulators in training
Human factors is a field of research that is well established in the maritime domain according to Grech et al. (2008). They describe the main interest of the field to be the human and their use of the elements in a technical system: the well-being of the human and the overall performance of the human-technology system. Hence, one central interest in human factors is to develop technologies that better fit the cognitive abilities of the user, creating safer systems for different aspects of the maritime domain. In the review, it can be seen as an interest in user testing how simulators can be designed to support different aspects of navigation training, such as ice navigation (Cemal and Burak, 2012) or precision navigation on rivers (Demchenkov, 2011). Of interest is also the relationship between simulator fidelity and the quality and transferability of training. The prevailing idea in most domains that use simulators in training is that if the simulation is close to the real-world task, the more likely it is that skills transfer from one context to another. In a case study on simulators used both in aviation and shipping, Dahlström et al. (2009) concluded that transferability of training should not rely on high-fidelity simulators alone. The primary reason for this is that is economical; it is costly to train in high-fidelity environments, and the training sessions are often designed to follow rehearsed roles, duties and procedures, and supports the development of procedural skills. Low-fidelity simulators on the other hand offer opportunities to train repeatedly for unanticipated and escalating situations, which is highlighted as important for developing resilience. Hence, based on Dahlström et al. (2009), it can be suggested that training programmes should combine the use of high-fidelity and lowfidelity simulators in their curriculum. Another interest in Human factors is if simulator-based training works efficiently; for example, if the development of different so-called non-technical skills such as situation awareness (SA) or decision making can be trained in simulators (Saus et al., 2010(Saus et al., , 2012Chauvin et al. 2009). These studies use experimental and mixed method approaches to isolate and measure SA during different training conditions. The results show that the trainees subjective SA as well as the perceived realism of the training event has a positive effect on the perceived learning outcome of the trainees. Drawing on the results of Saus et al. (2010), it is important to design training procedures and scenarios that enable students to perceive the simulation as a realistic training event, and at the same time design scenarios that are well adjusted to their level of competence in order for them to benefit from the learning experience. Also, Chauvin et al. (2009) found that students that were practising SA and decision making in simulators improved in analysing complex situations. However, it is notable that students that were part of an on-board-training programme showed significantly better results that the simulator-based training group. These results are important because they point to the value of on-board-training, and call for caution before replacing on-board-time with simulator-time in MET.

Educational research on learning and learning activities
Education is a diverse scientific field with a broad interest in learning. It has been described as the Bnurturing if the mental capacities through witch the learners come to know, understand, judge, reflect and behave intelligently^ (Pring, 2005, p. 32). Educational research generally takes an interest in the educational activity, i.e. the transactions between teachers and learners with a focus on how the learners develop their seeing and understanding towards the object of learning. Although the search found few studies in Education (n = 6), the interactional approach to study learning and learning activities is dominant in the included studies that draw on ethnographic approaches to study simulator-based training and assessment in a naturalistic way. These studies highlight the complexities of considering questions of choosing the right level of fidelity for different training purposes or facilitating simulator training. When designing simulatorbased training, the degree of fidelity must meet the requirements of the work tasks and learning objectives, while also attending to the specific nature of the learners' level of knowledge or expertise (Hontvedt 2015). Also, the work practices that take place on the bridge are heavily reliant on time, space and temporality in an intricate way that can hardly be simulated in an educational setting (Hontvent and Arnseth 2013). At the same time, they argue that the practice of simulator training is closely intertwined with the maritime profession's way of organising and perceiving the world. Their results support research on simulations in the healthcare domain viewing realism as an interactional matter as well as an instructional concern. Hence, simulators showed clear potential for learning, but these studies have shown that how the training is organised and carried out far exceeds the simulator. Kobayashi (2005) studied maritime policy and documents to draw conclusions about simulators and their use for training and assessment competences involved in safe navigation and came to the conclusion that simulators are well suited for training and assessing the competencies involved in safe navigation if they are used properly. Studies of the actual application of simulator-based assessments point towards serious problems regarding the use of competency tests in the MET system. Emad and Roth (2008) comes to the conclusion that not only is the learning objectives not fulfilled in the MET system, the assessment system has changed the learning objectives. Instead of striving to learn the skills and knowledge required on board ships, the objective in the current MET system is to pass competency tests. In fact, Gekara et al. (2011) are warning that the haphazard ways in which assessments are currently being implemented pose a possible safety hazard for the shipping industry. Moreover, Sampson et al. (2011) discovered that maritime instructors lack knowledge and are very uncertain of how to make assessments of competency in the simulator. The results of these studies are not surprising. It is far from evident how to conduct assessment based on observation of actions in the simulated environments and the instructors' concern connects to longstanding pedagogical debates about the character of knowledge in action. Consequently, more studies are needed in order to provide guidelines for simulator-based assessments of competency to ensure validity and reliability of the assessment methods or models.

Conclusions and further directions
The aim of the current systematic literature review is to give an overview of the field at large, its main interests and an overview of some important findings. It should be considered a guide to the literature for its reader rather than to provide any deeper analysis on different issues, as for example the questions regarding STCW identified in the synthesis (cf. Bearman et al. 2012). Following this approach, the current systematic literature review found 34 articles published in a wide range of academic journals, displaying a global field of research consisting of three main disciplines: Maritime professionals (n = 15), Human factors (n = 13) and Education (n = 6). Hence, simulator-based maritime training seems to be a rather small and quite diverse field of research, and several of the studies found lack empirical data as a basis for analysis (n = 13). The main result of the systematic literature review is that there are more questions than answers regarding the use of simulators in bridge operation training. While initial results point towards simulators as useful for training both procedural and non-technical skills, results also point towards the need for skilled instruction during training since Bthe simulation far exceeds the simulator^to lend the words of Hontvedt and Arnseth (2013). It is also important to take seriously the empirical results from Emad and Roth (2008) and Gekara et al. (2011) that shows that there is potential advantages of simulator-based training and assessment, but that they are currently being poorly implemented, which poses possible safety hazards for the shipping industry. Since MET institutions train their students for one of the most safety-critical industries in the world, there is a need for empirical studies on these questions to enhance the quality of training and assessment and lay the foundation for an evidence-based practice for simulation-based training of seafarers.
Acknowledgments The research project is financed by several co-parties: Linnaeus Centre for Research on Learning, Interaction and Mediated Communication in Contemporary Society (LinCS), University of Gothenburg Learning and Media Technology Studio (LETStudio), Institution for Shipping and Marine Technology at Chalmers University of Technology and Institution for Education, Communication and Learning at University of Gothenburg. I am also grateful towards Camilla Olsson, librarian at the University of Gothenburg and Yommine Hjalmarsson, librarian at the Chalmers University of Technology for assisting me with their expertise on information retrieval.

Appendix 1
Flow chart over the literature search (adapted from Moher et al. 2009 Barsan et al. (2007) Maritime professionals Intends to underline some of the aspects that must be revised for training in maritime education in regards to STCW updates Analysis of maritime policy Informing curriculum design regarding IMO compulsory courses Benedict et al. (2006) Human factors

Developing a Bsurveillance tool^and
Bevaluation tool^for assessment of student conduct during training sessions Software development and user testing The computerised assessment tools allow for in-depth search within the replay data and facilitates the calculation of Results showed that decision-making exercises did tend to improve the trainees' capacity to analyse a complex situation; however, they did not have a clear impact on the manoeuvre performed.
There was an obvious difference between the trainees who experienced complex avoidance situations during their on-the-job training period and those who did not pointing towards the value of on-board-training in comparison to simulator-based training Dahlström et al. (2009) Human factors Explores theoretical issues related to the relationship between simulator fidelity and the quality and transferability of training Case studies of accidents from aviation and shipping Transferability of training cannot rely only on high-fidelity simulators when work is highly complex and safety-critical. Also, lower-fidelity simulation can provide competence development with pedagogical and economic advantages Software development and expert qualitative assessments of navigational simulators Suggests the SC-method for implementing a way of evaluating the effectiveness of the navigational simulators for training shipmasters to control vessels on inland waterways Emad and Roth (2008) Education Highlight the contradictions in the current MET system and to theorise the failure to make the training useful A qualitative approach including document and policy analysis as well as ethnographic fieldwork There are contradictions in the education and training system that do not allow the targeted objectives to be fulfilled. Fundamentally, the assessment system has changed the objectives of the education and training practises from learning skills and knowledge required on-board ships to passing competency examinations Gekara et al. (2011) Education Examines the use of computer-based assessment, including simulators, for certifi-

cation of seafarers within MET
A cross-cultural qualitative approach including observation, interviews and document analysis Potential advantages of computer-based assessment. However, the design and application can be problematic and must be considered carefully. In shipping, the haphazard ways in which computer-based assessments are currently being implemented pose a possible safety hazard Gerganov (2014) Maritime professionals Highlight some aspects of the use of simulators in training and assessment to improve navigation safety and reduce accidents on the ships Experience based The use of simulators in training will ensure the maintenance of high level of training of crews of vessels, increasing safety of navigation and reduce accidents in the Navy Hanzu-Pazara et al. (2008) Maritime professionals Considers the impact on simulation-based training on maritime human error Experience based Training in simulators shows promising results of reducing human error caused show several weaknesses in terms of cheating. Also, they merely measure root learning. Instructors lack knowledge and are uncertain of how to conduct assessment of competence in simulators for certification Saus et al. (2010) Human factors Tested the effects of experience, perceived realism and SA on the perceived learning outcome of simulator-based navigation training Experimental In order to enhance the learning outcomes, it is necessary to design training procedures and scenarios that enable students to achieve functional fidelity and to generate and maintain SA during training. This can further Human factors Tested the relationship between individual differences and SA during training in a navigation simulator Mixed-methods approach Resilient personality type predicted both subjective and observer-rated SA and managed their stress levels Stan and Buzbuchi (2012) Maritime professionals To consider the vocational training of deck officers, but also the combination of simulator-based training and time at sea in the vocational system

Experience based
The vocational training is in accordance with the IMO, STCW and DNV, for training and bridge, engine and liquid cargo handling simulators Suppiah (2007) Maritime professionals Exploring how simulators can bridge theory and practice Experience based Simulators combined with traditional lectures appear to be an effective way to facilitating learning and allow for competency to be demonstrated and assessed Yousefi and Seyedjavadin (2012) Human factors Investigates the role of human factors and CRM training in minimising ship accidents at sea Case study of maritime accidents Seafarers need to acquire comprehensive understanding of technical facts through active learning processes that enhances deep understanding, but also to develop technical, cognitive and social skills. Skills for group/team work, good communications and resolving issues are as essential to bridge team work Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.