Background

Simulation-based education (SBE) is a popular mode of teaching in undergraduate nursing education [1]. This pedagogical approach provides students with an opportunity to practise non-technical skills (e.g. communication, teamwork, problem solving) and clinical skills (e.g. venepuncture), in a controlled environment that poses no risk of harm to patients and allows the possibility of direct feedback and guidance from teaching staff [2]. Educational outcomes are influenced by SBE quality that is governed by technology, training, resources and SBE design elements [3]. As SBE has evolved, its design elements have been adjusted to overcome learning barriers such as excessive student anxiety [4]. SBE has also been tailored to suit a variety of settings, populations, time frames and session frequency [5].

SBE designs include tag team simulation, simulated virtual and augmented environments and game-based themes within the traditional simulation, to accommodate ever-expanding university enrolments whilst maintaining student experience and satisfaction [6]. In the context of evolving SBE, the level of evidence for SBE effectiveness in nursing education is diverse, and educational outcomes remain ambiguous [7]. SBE can refer to a low-fidelity task trainer that takes 5 min to complete a skill whilst working one-on-one with a facilitator, or a complex high-fidelity immersive scenario involving a group of students performing complex tasks over a prolonged period of time [8]. The pedagogy of SBE generally fails to acknowledge the complex variability in design elements [9] that has implications for learning outcomes. As such, there is a need to evaluate the effect of SBE on learning and to describe SBE design elements to determine whether specific elements have a superior effect on learning outcomes. The aim of this systematic review is to identify, appraise and synthesise the best available evidence regarding the effectiveness of SBE and the impact of design elements on undergraduate nurses’ learning outcomes. To address this aim, there are three review objectives:

  1. (i)

    To determine the effect of SBE on learning outcomes

  2. (ii)

    To describe the variability in SBE elements

  3. (iii)

    To explore the interplay between SBE elements and learning outcomes

Methods

Design

This systematic review protocol was developed in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) [10] statement (Additional file 1) and the accompanying elaboration and explanation guide [11]. The planned systematic review will be reported in accordance with the PRISMA statement [12] and registered on the International Prospective Register of Systematic Reviews (PROSPERO) (CRD42021244530).

Eligibility criteria

Study selection will be guided by the eligibility criteria and the Population, Intervention, Comparison, Outcome (PICO) pneumonic [13].

Types of studies

Primary quantitative research studies published in English from 1 January 1990 will be considered. There has been a sustained increase in the volume of research focused on SBE since the 1990s related to the affordability of high-fidelity simulators [5]. Relevant quantitative designs including randomised controlled trials and quasi-experimental studies with a control group for comparison will be included.

Population

The population of interest is undergraduate nursing students, aged 18 years or over, engaged in SBE in an academic setting, such as a university.

Intervention

SBE is the intervention of interest. In general terms, SBE is a teaching technique used to enhance a learning setting to appear like the real world. In healthcare, there are a variety of design elements that can impact the effectiveness of SBE that in undergraduate nursing education settings is usually conducted in a physical environment with face-to-face teaching. Elements include devices such as manikins or task trainers that are used to allow students to interact in a manner that represents real clinical practice.

Fidelity refers to the realism of the simulation environment [14], and low, medium or high-fidelity environments are elements of simulation that can influence learning outcomes. Simulated patients, including trained actors, and/or integrated simulators that incorporate technology influence fidelity and as such the realism of the interaction between the learner and the manikin. In addition, adjustment of the manikin’s physiological data based on learner decision-making or conversing with the learner through built-in speakers to simulate a patient conversation [15] is a common element. Task trainers are another element of simulation designed to mimic a part of the patient’s body such as a manikin arm for intravenous insertion or something as simple as an orange, to practise injections.

Comparator

Conventional education modalities such as didactic lectures, passive (one-way) classroom teaching or small group seminars will provide a comparative control cohort.

Outcome

The primary outcome will be the SBE effect. This effect will be measured by assessing at least one measure of knowledge, skills or attitude as an endpoint. For the purposes of this review, knowledge is defined as learnt information (e.g. theoretical knowledge relating to the intended learning outcome of the simulation activity) acquired within the simulation activity. The measurement of knowledge will be determined by an academic outcome assessment. Skills are defined as the ability to develop psychomotor function aligned with the successful performance of a particular procedure (e.g. change a wound dressing), and attitude is defined as how worthwhile the learner believes the activity is in relation to their learning [16].

Exclusion criteria

Given the primary aim of the review is to assess the effect of SBE, which necessitates the need for a comparator group, case-control, cohort, cross-sectional and single-group observational studies will be excluded. Literature, narrative, integrative mixed methods and systematic reviews, observational cohort studies, abstracts, letters, commentary, editorials, opinion pieces and grey literature will be excluded. SBE can use modified realities such as augmented reality and virtual reality. These emerging digital modalities have a limited body of evidence and are a developing area of practice [17] so beyond the scope of this review. The use of pre-simulation interventions such as online learning modules or smart device technologies may influence the relationship between SBE and traditional learning so will be excluded. Similarly, studies that combine low-fidelity SBE elements with traditional education approaches and compare these to medium- or high-fidelity SBE will be excluded. Postgraduate nursing students, midwifery students or students receiving SBE in a clinical setting (e.g. a hospital) are excluded. Manuscripts published in languages other than English will be excluded. This is an unfunded study, so translation costs are beyond the investigator’s capacity.

Information sources and search strategy

Databases to be searched from 1 January 1990 include the Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Medical Literature Analysis and Retrieval System Online (MEDLINE), American Psychological Association (APA) PsycInfo and the Education Resources Information Centre (ERIC) via the EBSCO host platform. The Excerpta Medica database (EMBASE) will be searched via the OVID platform. The MEDLINE search strategy is included in Table 1.

Table 1 EBSCO MEDLINE search strategy
Full size table

Primary quantitative studies included in systematic reviews captured by the search strategy and studies identified through secondary searching relevant study reference lists will also be eligible for inclusion. Specific search terms will be developed in MEDLINE with assistance from a senior librarian experienced in the conduct of systematic reviews, using text words and subject headings. Primary search terms include ‘nursing students’, ‘simulation training, ‘patient simulation’ and ‘immersive simulation’ with common Boolean operators and symbols for exploding terms (*, +) as illustrated in Table 2. Each database will be searched using these broad terms with either MeSH or Emtree terms with appropriate permutations.

Table 2 Search concepts
Full size table

Data management, selection and screening

Database search results will be imported into the reference management software Endnote© VersionX9 and uploaded into Covidence. Covidence is a tool for effective collaborative title and abstract screening, full-text screening, data abstraction and quality assessment [18]. Covidence automatically identifies, sorts and removes duplicate studies. Two reviewers (MJ and RW) will independently screen the title and abstract of citations with a third reviewer (LM) available to moderate disagreements with a view to reaching consensus. Articles that meet the eligibility criteria will be sourced for full-text review. Two reviewers (MJ and RW) will complete the full-text screening with a third reviewer (LM) available to moderate disagreements to achieve consensus. Screening outcomes will be documented and reported in a PRISMA flow diagram [12].

Data extraction

Full-text data extraction will be undertaken in duplicate by two reviewers (MJ and LB) and conflicts resolved with arbitrary discussion. Data extraction will take place using a modified version of the existing Covidence data extraction template. Study characteristics to be recorded include publication details (authors, publication year), demographic characteristics, methodology, intervention and comparator group details and reported outcomes. To ensure consistency between reviewers, periodic meetings will be concurrently held during the screening process to resolve disagreements by discussion. A third reviewer (LM) will act as an adjudicator in the event of an unresolved agreement.

Data items

The following data items will be extracted from the selected studies: study setting, sample, inclusion and exclusion criteria, aim, design element/s employed, unit of allocation, start and end dates of the study, duration of participation, baseline group differences, frequency and duration of the intervention, outcome measured (e.g. knowledge acquisition; skill improvement, attitude and satisfaction), tool used to measure outcome, validity of the tool/s, comparator group education method and sustainability of outcome/s.

Outcomes and prioritisation

The primary outcome is the SBE effect measured by assessing at least one measure of knowledge, skills or attitude as an endpoint. Outcomes will be compared between the groups who do and do not participate in a SBE intervention. Secondary outcomes include describing variability in SBE design elements and sub-group analyses to explore the interplay between SBE elements and learning outcomes. In the case of discrepancies in the outcomes reported, contact with corresponding authors will be attempted by email to obtain relevant data.

Risk of bias in individual studies

Each randomised trial will be assessed for possible risk of bias using the revised Cochrane Collaboration tool for assessing the risk of bias (RoB 2). This tool focuses on trial design, conduct and reporting to obtain the information relevant to the risk of bias [19]. Based on the answers provided within the tool, trials will be categorised as ‘low’ or ‘high’ risk of bias. Where a lack of detail impacts the capacity to assess the risk bias, this will be noted as ‘unclear’ and authors contacted for more information. If there is disagreement, a third author will be consulted to act as an arbitrator. The Risk Of Bias In Non-Randomized Studies – of Interventions (ROBINS-I) tool will be used for bias assessment in studies with this design. The ROBINS-I tool shares many features with the RoB 2 tool as it focuses on a specific result, is structured into a fixed set of domains of bias, includes signalling questions that inform risk of bias judgements and leads to an overall risk-of-bias judgement [20].

Quality appraisal

Two authors (MJ and LB) will independently assess each article for quality using the Mixed Methods Appraisal Tool (MMAT) [21]. This critical appraisal tool allows the appraisal of five categories of studies including qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies and mixed methods studies. As recommended, each study will be reviewed by two authors by completing the appropriate study categories identified within the MMAT. An overall score will not be used to rate the quality of study, but instead, sensitivity analyses will be completed to determine the impact of study quality on outcomes by contrasting components as recommended [11].

Data synthesis

The primary unit of analysis will reflect each endpoint measure; knowledge acquisition, skill improvement or attitude. Dichotomous outcomes will be extracted and analysed using odds ratio (OR) with 95% confidence interval (CI), and continuous outcomes will be represented using the mean difference (MD) or the standardised mean difference (SMD) when outcomes are measured using different scales, with 95% CI. In the event of missing data, we will attempt to contact the primary authors to obtain relevant information. Meta-analysis will be undertaken if two or more studies with comparable design and outcome measures meet the eligibility criteria. Pooled data will be analysed using the DerSimonian and Laird method for random-effects models in RevMan [22]. This model is used when reported outcome effects differ amongst studies but follow some similar distribution [23]. Findings from meta-analysis will be illustrated using forest plots.

The I2 test of heterogeneity will be used to determine the level of variation related to diversity rather than chance. Rather than simply stating whether heterogeneity is present or not, it will be measured using the I2 test [24]. Low heterogeneity will be reflected as an I2 result between 0 and 40%, moderate heterogeneity between 30 and 60%, substantial heterogeneity between 50 and 90% and considerable heterogeneity between 75 and 100% [25]. If a high heterogeneity level (I2 > 50%) amongst trials exists, study design and characteristics will be reported and sensitivity analyses conducted to reduce variability with a view to being able to undertake meta-analysis. It is assumed that specific design elements will underpin the need for subgroup analyses. If data are not suitable for meta-analysis, findings will be presented descriptively in the form of a narrative synthesis according to categories outlined in the SWiM guideline [26]. Due to the potential of a large amount of data that could be conveyed textually, the content will be sequenced to follow the same structure to ensure consistency across results [27].

Meta-bias

Study selection for this review will be guided by the PICO pneumonic and the framework outlined in this protocol to reduce the risk of bias. Hand searching relevant systematic reviews will contribute to a reduction in publication bias [28]. To reduce bias associated with selecting studies, two independent reviewers will be used throughout the screening and data collection process [29]. To address bias when synthesising studies, this protocol has been registered prospectively to promote transparency and replicability [10]. Two reviewers will appraise the quality of studies, and low-quality studies will be excluded to avoid inappropriate influence on results [10].

Confidence in cumulative evidence

The overall effectiveness of simulation and the impact of design elements on undergraduate nurses will be assessed with the Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE) approach. The GRADE approach allows an assessment of the certainty of the body of evidence for each individual outcome [30]. Quality of evidence is dependent on the within-study risk of bias, directness of evidence, heterogeneity, precision of effect estimates and risk of publication bias [23]. Two authors will independently assess the quality of evidence regarding the effectiveness of simulation on each outcome. All disagreements will be resolved through discussion and consensus. The certainty of evidence from pooled analyses will be categorised as high, moderate, low or very low [30]. If data are not suitable for pooled meta-analysis and studies are grouped for synthesis, the SWiM guideline [26] will provide a framework for narrative analysis and GRADE criteria will be used to examine the certainty of the evidence for each analytical grouping.

Discussion

SBE is seen as a panacea to counteract the potential for inadequate exposure to real-life clinical scenarios. It has become an increasingly popular mode of teaching yet reports of its effectiveness remain diverse. There is a variety of simulation designs, and embedded elements have implications for learning outcomes that are not well described or scrutinised in the literature. This systematic review will provide a detailed summary of evidence for SBE and SBE design elements in the context of undergraduate nursing education. Findings will contribute to the body of literature examining specific design elements by establishing the effect of SBE on learning outcomes and demonstrating the degree of variability in these elements.