Introduction

Research–practice partnerships have recently become more widespread in education (Coburn and Penuel 2016). Such partnerships frequently strive to close the gap between a steadily growing body of research and a lack of improvement in educational practice. Specifically, research–practice partnerships address this gap by developing and disseminating solutions to highly-relevant educational problems and creating practical knowledge applicable to real-life contexts (Butler 2008; Ormel et al. 2012). Educational design research (EDR) focusses on the creation and evaluation of specific interventions as well as the scientific exploration of the design and production process itself (Design-Based Research Collective 2003). EDR and similar approaches may thus benefit greatly from investigating close collaborations between researchers and practitioners and could potentially optimize these collaborations.

One domain in which EDR on collaborations between researchers and practitioners can be applied is research on simulation-based learning. In this research area, intriguing research questions on learning and assessment remain unresolved (Heitzmann et al. 2017), and research programs can develop valuable educational products. Research programs on simulation-based learning possess three crucial features that could also characterize other interesting research–practice partnerships. First, research is carried out in (interdisciplinary) teams. Second, research programs collaborate on a management level as well as on the level of individual projects. Third, research and design coincide within at least a single phase of these programs, due to the custom development of complex products (such as simulation components) over a long period.

A framework for research and design in research programs on simulation-based learning is lacking. Such a framework is an essential complement to the scarce literature on the interplay between research and design in the field of EDR, which has not yet sufficiently delineated the phases, activities, and products of research programs that create complex scientific artifacts (McKenney and Reeves 2012). In particular, the current literature does not include a dual-level, multi-project framework for collaborative research and design (i.e., a model that delineates the role of a research committee and various subordinate research projects). Such a framework could support researchers and designers in creating effective research materials and simulations. Moreover, it could aid team leaders in systematically monitoring and guiding personnel and their collaboration.

In this article, we will first outline the relevant theoretical background and then propose a dual-level framework for multi-project research programs on simulation-based learning. This framework outlines phases, activities, roles, and products of simulation research and design. Notably, it applies to research programs with a research committee and multiple subordinate research projects focusing on the assessment and facilitation of knowledge and competences. We offer recommendations for each level and phase of the framework and illustrate the proposed framework with examples from the actual research and design process of the research program COSIMA. This research program investigates the use of simulations to facilitate diagnostic competences in the domains of medical education and teacher education. In the discussion, we will summarize our findings, address the implications for theory and practice, discuss the limitations of the framework, and provide directions for future research.

Theoretical background

The theoretical background of our framework is grouped into two parts. In the first part, we review the literature on research and design in educational research programs. Due to its focus on all types of educational research programs, this section is relatively broad in scope and more general. In the second part, we summarize simulation research and design in medical education and teacher education. Thus, the scope of the second part is more narrow due to its domain-specific focus. Together, the two parts complement one another, summarizing both the general and domain-specific theoretical background of our framework.

Research and design in educational research programs

We first outline the characteristics and stages of research programs, before elucidating relevant frameworks from educational design research. Finally, we describe important roles in educational research and design.

Characteristics and stages of research programs

According to the team science literature, research programs can be characterized as multi-team systems in which research projects collaborate and work autonomously to achieve common objectives (Shuffler et al. 2015). Similar to units in other contexts, these multi-team systems can be organized in a rather flat or strongly hierarchical structure and can include units of different sizes stemming from the same or different institutions (DeCostanza et al. 2014). Research programs often have a relatively strong hierarchical structure and consist of a research committee and subordinate research projects (Bozeman and Boardman 2014). While the research committee performs leadership tasks such as creating a common vision (Gray 2008) and answering overarching research questions, the subordinate research projects investigate more specific research questions. Next, we present the stages that typically take place within research projects before discussing this point specifically with respect to research programs.

Traditional models delineated research, regardless of the specific context, as a linear process comprised of clearly separable stages (e.g., Finley and Pocoví 2000). Such stages include a literature review, formulation of hypotheses, conducting empirical investigations, analyzing data and communicating findings. Research process models like this can be used to describe individual research projects and have been applied in multiple disciplines to investigations with various methodologies. Unfortunately, these models barely resemble scientists’ actual, rather complex, research process. Contemporary models depict the research process more accurately as the adaptive and iterative application of activities (e.g., evidence generation, drawing conclusions, communicating and scrutinizing findings) to the current status of an operation (Fischer et al. 2014; Reiff et al. 2002). These activities are relatively specific and can occur in various research contexts, including individual research projects as well as research programs. In a model of collaborative research, Sonnenwald (2007) described that research programs typically traverse through the stages of foundation, formulation, sustainment, and conclusion. The foundation stage designates conditions before and during the start of a collaboration and includes but is not limited to resource availability and building professional networks. In the formulation stage, collaborators devise a research program and apply for a grant. During the sustainment stage, the research program that has received funding and started its activities is evaluated continuously and kept on track. In the conclusion stage, outcomes of the research program are published, and new frameworks and proposals are created.

In summary, it can be said that research programs often consist of a research committee and multiple individual, subordinate research projects. Likewise, the research process in this specific context can be described quite well by the stages mentioned by Sonnenwald (2007). The new framework proposed in this article will thus characterize research programs as dual-level, multi-team systems and contain phases based on the stages delineated by Sonnenwald (2007).

Educational design research

The term EDR subsumes multiple approaches, such as design-based research, that simultaneously pursue the objectives of creating scientific knowledge and designing educational artifacts (McKenney and Reeves 2012). However, in the context of EDR, there are only a few models and frameworks that adequately represent research and design in research programs. Next, we will provide a brief overview of these models and frameworks.

Bannan-Ritland (2003) proposed one of the first EDR models to address research and design on a research program level, the integrative learning design framework (ILDF). The ILDF comprises the phases of exploration, enactment, and evaluation. This framework encompasses a rapid prototyping approach and aims for the adoption of the designed product. Middleton et al. (2008) presented an innovative EDR model for research projects with the “compleat” design experiment. According to the “compleat” design experiment, an empirical investigation (e.g., a randomized controlled trial) is designed and developed as part of an educational research process. A feasibility study, rapid prototyping, and pilot study are significant steps within the design and validation of the empirical investigation. The osmotic model (Ejersbo et al. 2008) combines a research process model for research projects with a product design model. This model depicts the interaction between research and design as ideal if both processes take place simultaneously and provide necessary input for one another. According to another EDR model by Akkerman et al. (2013), the three epistemic practices of research, design, and educational change take place concurrently in single research project EDR. The model highlights that each of these epistemic practices involves different subjects (e.g., researcher, designer, and change agent), tools (e.g., specific underlying models) and outcomes (e.g., a publication, a developed application or a developed application integrated into a curriculum). Synthesizing different EDR models, the generic model for EDR (McKenney and Reeves 2012) delineates the prototypical phases in which scientific knowledge and an educational intervention are produced and refined. The model depicts research and design as of equal status and posits that EDR processes in single research projects follow the iterative phases of (1) analysis and exploration, (2) design and construction, and (3) evaluation and reflection. During these phases, an educational artifact and scientific knowledge are developed and translated to practice.

In short, the described models and frameworks highlight essential phases and products of EDR. The new framework proposed in this paper will describe an EDR process in a research program. EDR processes in research programs have already been described by the ILDF (Bannan-Ritland 2003). However, our framework goes beyond this and other EDR frameworks by offering a separate description for the research committee and research project level as well as a context-specific focus on simulation research and design. Moreover, our framework will embed elements of the generic model for EDR (McKenney and Reeves 2012), which emphasizes an equal status of research and design. An equal status of research and design is crucial in one phase of our framework when research materials and simulation components are developed and evaluated simultaneously. We will, thus, incorporate the design and construction, and evaluation and reflection phases of the generic model for EDR in this phase of our framework. In the next section, we will discuss major roles that play a part in educational research and design.

Roles in educational research and design

Throughout the educational research and design process, team members can fulfill multiple roles, including consultant, researcher, and designer (McKenney and Brand-Gruwel 2018). In the consultant role, team members mainly provide training, access to professional networks, and (scientific) advice to team members. In the researcher role, team members conduct (empirical) studies or carry out literature research to gain knowledge or develop theory. In the designer role, team members specify and develop artifacts. The developed artifacts are evaluated and improved until they meet the specified requirements. Apart from these roles, content experts fulfill an important role in educational research and design processes (Lee 1994) because they choose, create and evaluate the content of educational artifacts based on their extensive professional knowledge. The new framework presented in this paper will include all of these described roles, not least because these roles seem to be highly applicable for the context of research on simulation-based learning.

Simulation research and design in medical education and teacher education

An integrative perspective on simulation research and design in medical education and teacher education may enable generalization to other contexts and is warranted for three key reasons. First, simulations act as important “approximations of practice” (Grossman et al. 2009, p. 2058) within college courses in both domains and can be used for similar training and assessment purposes (Kaufman and Ireland 2016; Ryall et al. 2016). Second, some joint conceptualizations of knowledge and competences already exist. This is particularly the case for diagnostic competences (Förtsch et al. 2018; Heitzmann et al. 2019). Third, we can assume that the instructional design of simulations in both domains is rather similar. This should be the case because similar design processes emerge for creating interactive simulations in medicine and teacher education (Dotger et al. 2010), and comparable design features (e.g., feedback and clear educational aims) must be considered in both domains (Badiee and Kaufman 2015). Furthermore, we believe that the instructional design process should be particularly similar in both domains if we focus on simulations that assess and train knowledge and competences and exclude simulations that emphasize assessing and training other types of skills, such as motor skills. In the following section, we outline important features and terms in simulation-based learning.

Features and terms in simulation-based learning

Simulations can be defined as authentic models of professional situations that can be manipulated by participants (Jones et al. 2015; Kaufman and Ireland 2016). In contrast, simulators refer to (technical) devices and environments deployed to conduct simulations (Khan et al. 2011). This distinction of terms is meaningful for the design process because it demonstrates that simulations and simulators are distinct artifacts that have to be designed separately. Above all, simulations can be characterized based on their modality (Ziv et al. 2000). Live simulations involve a professional standardized person such as a standardized patient or student who has received training and prepared diligently for the case (Barrows and Abrahamson 1964). In role-play simulations, participants are usually only prepared through a short training phase and receive a supporting script before they interact with each other in different roles (e.g., physician and patient; Gartmeier et al. 2015; Simpson 1985). Digital simulations are conducted using a computer and a virtual person or environment to interact with (de Jong 1991). Alternatively, simulations can be characterized by their purpose, either assessment, facilitation, or research (Crawford 1966). As targeted research and design processes should consider the critical features of modality and purpose, the new framework proposed in this article will provide specific recommendations for these two features. Moreover, our framework will address the assessment and conceptualization of knowledge and competences in medical education and teacher education.

Assessment and conceptualization of knowledge and competences

Knowledge has long been assessed in medical education and teacher education through traditional paper–pencil tests using constructed response (e.g., short answer questions) or closed response formats (e.g., multiple-choice questions; Kastner and Stangla 2011). Nowadays, however, knowledge is increasingly assessed using such formats as part of simulation-based learning. Knowledge tests are therefore frequently integrated into testing environments before, during, or after simulation-based assessment. The most common knowledge classification in teacher education (Shulman 1987) categorizes knowledge based on content, while a popular knowledge classification in medical education also makes categorizations based on structure (Paris et al. 1983). Based on knowledge classifications from both fields, Förtsch et al. (2018) proposed an interdisciplinary framework for medical education and teacher education in which knowledge can be differentiated according to types of knowledge (i.e., the structure of knowledge) as well as content-related facets of knowledge. In contrast to knowledge, competences can be assessed rather well directly within simulations by observing participants’ demonstrated performance (Blömeke et al. 2015; Miller 1990). Assessment can also take the form of multiple situation evaluations in which participants move from one evaluation station to the next (Harden et al. 1975). The result and process of participants’ demonstrated performance are often evaluated in such assessments by raters applying rating scales (Rothman et al. 1996) and evaluated in comparison to solutions by one or even multiple experts (Charlin et al. 2010). The new framework presented in this article will offer specific recommendations for assessing knowledge and competences based on the presented literature from both domains. We discuss different frameworks for research with simulations below.

Research with simulations

To our knowledge, no comprehensive framework for multi-project research programs on simulation-based learning is available. However, an existing framework for multi-center simulation research (Cheng et al. 2017) provides some insights that are also applicable to research programs. According to this framework, multi-center simulation research is conducted in four separate but overlapping phases, in which the project is first planned and developed, before a study is conducted and findings are communicated. More literature is available on specific topics within simulation research, such as creating valid and reliable simulation scenarios and instruments, standardizing studies, recording, evaluating and reporting data and adherence to research ethics (Cheng et al. 2014a, 2016; Lamé and Dixon-Woods 2018). In addition, simulations have been used in a large number of studies to conduct basic research (Cook et al. 2013; Kaufman and Ireland 2016) as well as EDR (e.g., de Coninck et al. 2019; Hirumi et al. 2016a; Koivisto et al. 2018). These studies demonstrate that materials developed for research purposes include instruments that assess knowledge and competences, an educational intervention, and an experimental procedure (see Table 1 for our own definitions of these terms). The phases of research with simulations described by Cheng et al. (2017) and the classification of research materials in Table 1 will be included in the new framework presented in this article.

Table 1 Overview of research materials
Full size table

Design of simulations

Currently, only limited literature is available regarding the design of simulations. Cheng et al. (2014a) provided an overview of simulation research and design in pediatrics, including design recommendations for selecting an appropriate simulator, designing simulation scenarios (see Table 2 for the definition), and establishing sufficient authenticity. In the framework for multi-center simulation research mentioned above, Cheng et al. (2017) point out that researchers should scrutinize log data and evaluate the simulation scenarios created through pilot studies and with institutional review committees. Apart from these recommendations, there are only frameworks and recommendations for the design of specific types of simulations, such as live simulations (Khan et al. 2013a, b; Sturpe and Schaivone 2014) and digital simulations (Posel et al. 2015; Zary et al. 2006), as well as specific design activities such as scenario design (Benishek et al. 2015). Regarding the simulation design process in research programs, a number of instructional design studies from teacher education (e.g., Christensen et al. 2011; Ferry and Kervin 2007) and medical education (e.g., Hirumi et al. 2016b; Jensen et al. 2015) illustrate that various simulation components have been developed. These simulation components typically include learning and testing environments, simulation scenarios, case vignettes, as well as briefings and debriefings (see Table 2 for definitions of these terms). The classification of simulation components in Table 2 will be incorporated into the new framework proposed in the next section.

Table 2 Overview of simulation components
Full size table

A dual-level framework for simulation research and design in multi-project research programs

Overview of the framework

In line with the notion of hierarchical, multi-team systems (Bozeman and Boardman 2014; DeCostanza et al. 2014; Shuffler et al. 2015), we propose a framework that describes a research program as a dual-level, multi-project system consisting of one research committee and two or more research projects. Based on the stages of research programs (Sonnenwald 2007), and the phases of the multi-center model for conducting simulation-based research programs (Cheng et al. 2017), we posit that the research committee and research projects pass through the following program/project phases: 1. Creating a foundation for a research program or project; 2. Constructing and adapting simulations and research materials; 3. Conducting empirical investigations; and 4. Analyzing data and communicating results. Decisions in the first program/project phase affect all subsequent process phases. During the second project phase, the research materials and simulation components listed in Tables 1 and 2 are constructed and adapted, following the design and construction as well as evaluation and reflection phases of the generic model for EDR (McKenney and Reeves 2012). Products developed within the research program include scientific knowledge as well as practical applications and tools. Even though earlier program/project phases create research materials and simulation components as well as products for subsequent program/project phases, they are not necessarily executed as separate phases in a strictly linear format. Rather, program/project phases can overlap with each other and go through multiple iterations, drawing on previously developed research materials, simulation components, and products. For instance, one program/project phase can start before another has been completed. Members of the research committee and the research projects take on the roles of consultant, researcher, designer (McKenney and Brand-Gruwel 2018), and content expert (Lee 1994) throughout the program/project phases. While team members on both levels can fill multiple roles at the same time, certain roles can also be stressed for specific members in particular program/project phases. The described framework is illustrated for one research committee and multiple fictional research projects in Fig. 1.

Fig. 1
figure 1

A dual-level framework for simulation research and design in multi-project research programs

Full size image

Below, the four separate program/project phases will be explained in terms of the two levels of the framework. Relevant research materials and simulation components were defined in Tables 1 and 2. Recommendations for and an overview of the activities in the different phases will be presented for the research committee level in Table 3 and for the research project level in Table 4. Concurrently to our explanation of the program/project phases, we will illustrate the proposed framework by reporting on the research process that took place in the research program COSIMA. This research program investigates facilitating diagnostic competences with scaffolding in simulation-based learning environments in higher education and consists of a research committee and eight research projects. Seven empirical research projects gather data through experiments in the domains of medical education and teacher education; one meta-analysis project synthesizes data. A research committee leads and coordinates activities between the research projects. As an example research project, we will report on research project COSIMA 4 that investigates facilitating interactive diagnostic competences in simulated clinical history-taking.

Table 3 Checklist for conducting research and design on the research committee level
Full size table
Table 4 Checklist for conducting research and design on the research project level
Full size table

Phase 1: creating a foundation for a research program or research project

Research committee level

At the beginning of this phase, suitable collaborators with common interests are identified, form a team, and apply for a grant posing an overarching research question for the prospective research program (Cheng et al. 2017; Sonnenwald 2007). The research committee’s proposal may include a preliminary version of a conceptual framework, a common methodology, and a strategy to synthesize data from different research projects. After program start, the research committee states common objectives for all projects, coordinates activities between projects, assigns tasks, provides background knowledge on the topic and establishes a common terminology (Beck et al. 2017; Bennett and Gadlin 2012; Hall et al. 2012).

Research project level

In this phase, each research project creates a research proposal. In order to do so, the project first poses a research question and develops a research design. Then, the project determines a professional situation to simulate with high practical relevance, operationalizes relevant knowledge and competences, and chooses a suitable simulator and instructional support (see details and recommendations on these activities in Table 4). Similar to the analysis and exploration phase of the generic model for EDR (McKenney and Reeves 2012), researchers concurrently gain a theoretical understanding of the problem and determine possible solutions. In the current context, expert interviews and literature reviews are carried out to gain first insights into relevant variables and effective educational interventions for the selected simulated scenario and topic. Researchers, designers, and content experts also evaluate similar studies and commercial products to decide, with input from consultants, whether it is more sensible to purchase and customize or to newly develop simulators and simulations.

Illustration

The research committee for COSIMA consists of 12 professors from the fields of medical education, teacher education, and educational psychology. Several postdocs and Ph.D. students also carry out tasks for the research committee. The collaborators who formed the research committee for the research program COSIMA drafted a proposal concerning the use of simulations to facilitate diagnostic competences, including a literature review, a preliminary conceptual model (see Fig. 2), and a mentoring concept for research projects. After the official start of the research program, the research committee stated common objectives at a kick-off retreat. It also provided background knowledge on conceptualizing and assessing diagnostic competences in a monthly colloquium.

Fig. 2
figure 2

A framework for facilitating diagnostic competences with simulations (Heitzmann et al. 2019)

Full size image

The principal investigators of the research project COSIMA 4 are two professors of medical education and one professor of educational psychology, who are also part of the research committee. Before the official project start, the principal investigators selected conducting a medical interview in an emergency room as a relevant professional situation for the proposal and specified four studies with a cross-sectional experimental design. Moreover, they decided to simulate the medical interview in the four studies with live simulations with professional actors, digital simulations, and role-play simulations. They also selected different types of reflection phases and roles as the form of instructional support to investigate. After the official project start, a Ph.D. student in learning sciences and a board-certified general practitioner joined the team. An external IT company was contracted to produce the learning and testing environments. Next, the Ph.D. student and general practitioner acquainted themselves with the literature on instructional support, simulations, and history-taking and conducted a site visit to a simulation center.

At both levels, this phase began about one year before the submission of the grant proposals. It was put on hold during the evaluation of the proposals and was completed approximately half a year after the official start of the research program.

Phase 2: constructing and adapting simulations and research materials

Research committee level

In this phase, the research committee creates research and design requirements for the research projects, including operationalizations and specifications for the research materials and simulation components (see Table 3). The fulfillment of these requirements in the projects is monitored (Cheng et al. 2017) throughout this phase, and researchers and designers are guided in fulfilling the requirements. In addition, the research committee conveys a common knowledge base (Sonnenwald 2007) on the content and the design of research materials and simulation components. Apart from these tasks, the research committee may support the research projects as needed with tasks such as ethical approval, legal issues with external contractors, and conflict resolution (Cheng et al. 2017; Hall et al. 2012).

Research project level

Simulations and research materials are constructed and adapted in the research projects following two phases of the generic model for EDR (McKenney and Reeves 2012): 1) design and construction, and 2) evaluation and reflection.

Design and construction

In this subordinate phase, team members generate and assess possible resolutions, specify requirements, and develop products (McKenney and Reeves 2012). The activities performed by each team member depend considerably on their role and the developed research materials and simulation components (see Tables 1 and 2 for definitions and Table 4 for specific recommendations). However, we can summarize that consultants and researchers mainly decide on a learning and testing environment, design instruments that assess knowledge and competences, develop a measurement procedure, and create an experimenter’s guide. Designers primarily focus on building a simulator and a learning and testing environment. Content experts focus on creating simulation scenarios, case vignettes as well as briefings and debriefings. As one of the last steps, the developed research materials and simulation components are embedded in the learning and testing environment according to the experimental procedure.

Evaluation and reflection

This subordinate phase consists of a formative evaluation of prototypes of the developed research materials and simulation components through expert workshops and pilot studies, and a deliberate reflection on the research and design process (McKenney and Reeves 2012). Expert workshops typically evaluate the authenticity and difficulty of the developed simulation scenarios and case vignettes. Moreover, expert workshops can scrutinize the content and accuracy of expert solutions presented to participants in debriefings. Pilot studies mainly provide an assessment of the measurement properties of the instruments assessing knowledge and competences and evaluate the usability and fidelity of the simulation. Reflecting on the research and development process, team members evaluate individual work and collaboration, as well as compliance with the milestones of the research project and research program.

Interaction between the subordinate phases

As various research materials and simulation components are created during the second phase, the two subordinate phases may be iteratively repeated several times. Each of these subordinate phases may focus on one piece of research material or simulation component or several simultaneously. Therefore, multiple cycles may need to be completed before all research materials and simulation components are fully developed and positively evaluated. If the evaluation of all required research materials and simulation components is successful, an empirical investigation can be conducted.

Illustration

In this phase, the research committee of COSIMA guided the development of a common test battery for all research projects. This test battery included motivational and cognitive scales and tests. Moreover, the research committee created a measurement procedure for all research projects that specified the number and timing of measurements for common variables. Throughout this phase, the research committee conveyed knowledge on simulation components and research materials (e.g., on the structure and timing of reflection phases) and offered guidance. In addition to these tasks, the research committee filed for ethical approval in cooperation with the projects.

The research project COSIMA 4 repeated this phase twice. In the first design and construction cycle, prototypes of the digital simulation and live simulation were created. Both prototypes involved only two case vignettes and were created with student assistants as actors. Moreover, knowledge tests on dyspnea were created. These materials were integrated into the selected learning and testing environment CASUS (Instruct 2018). Afterwards, a pilot study primarily investigated the usability of the simulation prototypes. Based on the results of the pilot study, a second design and construction cycle took place. In this cycle, a professional programmer was contracted with improving the created digital simulation. The professional programmer developed a simplified version of the digital simulation with fast-streaming video clips. To improve the live simulations, an acting coach was hired. The acting coach supported the project in preparing and recruiting professional standardized patients. Subsequently, nine case vignettes for the two types of simulations were created (see the blueprint in Fig. 3), and an expert workshop evaluated their accuracy and difficulty. After some revisions, the quality of the research materials and simulation components was evaluated positively, and the next phase began.

Fig. 3
figure 3

Blueprint for the case content in the simulations

Full size image

This phase was completed by the research committee one year after the official program start. Research project COSIMA 4 finished this phase approximately one and a half years after the official program start.

Phase 3: conducting empirical investigations

Research committee level

In this phase, the research committee specifies final targets for the empirical investigations within the research projects (e.g., regarding sample size and composition) in alignment with the research program’s initial goals and current status. Also, the research committee monitors the standardization across research projects (Cheng et al. 2017). Moreover, the research committee monitors whether the empirical investigations are proceeding without issue and whether issues co-occur in research projects. Some co-occurring issues can be resolved more easily by a superordinate project, including senior members, than by individual research projects. Besides, the research committee provides guidance for the research projects regarding individual problems (Cheng et al. 2017). Knowledge is primarily conveyed on conducting empirical investigations with simulations, recruiting participants, and recording data.

Research project level

In this phase, researchers prepare the developed materials, recruit the sample of the target population, and carry out and document the empirical study. Regarding the preparation of the developed materials, facilities must be organized in advance, and involved personal must be trained. Regarding recruitment, participants can be recruited on or off campus as well as online. Boosting recruitment, the importance of incentives should also be stressed. Concerning carrying out the study, there are two important types of data collection: lab-based data collection and web-based data collection. Both types of data collection come with particular advantages and disadvantages (see Reips 2000) and we offer recommendations for both in Table 4. Moreover, we believe there are also peculiarities in simulation research that depend on the simulation modality. We provide recommendations for carrying out live simulations with professional actors, digital simulations, and role-play simulations in Table 4. With regards to documentation, funding agencies typically recommend the meticulous documentation of research materials and the empirical investigation as well as open science practices such as sharing data (Earle et al. 2013). Lastly, complex investigations involving multiple simulators, instruments, or many participants have to be organized efficiently, saving costs and minimizing the use of facilities. For example, complex investigations may require protocols that organize the flow of participants. Such protocols can guarantee that participants complete different parts of the investigation punctually and in the correct sequence (for more recommendations on this topic, see Khan et al. 2013a).

Illustration

During the first phase of empirical investigations, the research committee of COSIMA adhered to the initially proposed sample size in all research projects. It conveyed knowledge on organizing simulation studies (e.g., training standardized patients) and recording log data and videos. In addition, the research committee supported research projects when issues with recruitment or facilities (e.g., simulation centers) occurred.

In this phase, the research project COSIMA 4 successfully conducted a full empirical study with N = 86 medical students (see Fig. 4 for a screenshot of live and digital simulations). Behavior was captured with a camera, and data was recorded with log files recording all activities of participants.

Fig. 4
figure 4

Screenshots from the empirical study

Full size image

This phase was completed by the research committee for the first time one and a half years after the official program start. The research project COSIMA 4 completed this phase for the first time one year and eight months after the official program start.

Phase 4: analyzing data and communicating results

Research committee level

In this phase, the research committee provides data reporting specifications to all empirical research projects (Tobi and Kampen 2018) and gathers data systematically. When data from several projects has been merged, the research committee can use special methods to analyze large data sets (see recommendations in Table 3). Also, the research committee can convey knowledge on data analysis to the research projects by organizing methods workshops. Above all, the research committee can create in this phase publications based on project data as well as publications that do not rely on project data (see recommendations in Table 3). Moreover, the research committee can develop a strategy that helps the research projects to increase the visibility of findings. Such a strategy should address how publications and key findings are advertised and reported in a comprehensible manner, also understandable to a non-scientific audience. Finally, the research committee also starts to generate first ideas for writing extension grants.

Research project level

This phase frequently starts with coding and preparing data. For digital simulations, log-file data must be transformed, cleaned, and scored. For role-play and live simulations, video recordings of behaviour must be coded with coding schemes. Then, suitable statistical analyses are applied. Results are communicated in such research projects mainly through theoretical and empirical articles and talks at conferences. Moreover, the research projects also make their research more visible to a broader audience by reporting key findings also in popular media formats (e.g., on Twitter or ResearchGate).

Illustration

The research committee of COSIMA created detailed specifications for reporting results in the research projects and compiled data from all seven empirical projects into one file. The research committee has published two conceptual articles so far that do not rely on project data (Förtsch et al. 2018; Heitzmann et al. 2019). In addition, a registered report is in preparation that will evaluate aggregated data from multiple projects using structural equation modeling techniques.

After completing the first study for the research project COSIMA 4, data from the digital simulations’ log files was transformed using scripts in the statistical software R. The behavior of learners in live simulations was coded by student assistants using a coding scheme. First analyses of the data have been presented in conference talks, and one empirical article on the study has been written and submitted for peer-review.

This phase was completed by the research committee for the first time two years after the official program start. For the research project, this phase ended for the first time two years and three months after the official program start.

Interaction between the phases and subsequent research process

Within the research committee and research projects, the first and second phases usually have to be completed only once and are followed by iterations of the third and fourth phases. If a study requires additional development, the second phase must be repeated. Analyzing data and communicating results in phase 4 for the research committee may begin earlier than for the subordinate research projects (e.g., via the creation of theoretical articles for the projects) and may continue even after all projects have ended. When the research program comes to an end, the research committee and research projects have three options. They can write another proposal for a grant with a new topic, submit a request for an extension, or finish research on this topic.

Illustration

The research committee successfully applied for an extension grant for a second phase of the program together with the research projects after two and a half years. This application process led to a short additional cycle of the first phase. Overlapping with this phase, the research committee worked on publishing findings and was thus in the fourth phase. The research project COSIMA 4 was during that time simultaneously in the third and fourth phases, conducting empirical studies and publishing findings from its first empirical study.

Illustration of roles taken and products created

In the research committee, all professors served as consultants, as they made strategic decisions concerning the research program. All professors also acted as researchers, developing a common theoretical framework and working on publications. Certain professors engaged on this level as content experts, choosing specific topics and contexts for the research program. In the described research project COSIMA 4 , the Ph.D. students mainly served as researcher and instructional designer; the general practitioner acted primarily as content expert. The professors primarily took on the roles of consultants in the research project, guiding the Ph.D. student and general practitioner. The contracted IT companies served as instructional designers by providing and developing the learning and testing environments.

Over the course of the phases, the research committee and research projects created scientific knowledge as well as practical applications and tools. Scientific knowledge includes codified knowledge (such as theoretical and empirical articles) and uncodified knowledge (such as employees’ implicit knowledge about creating simulations) within a team (Nonaka 1994). Codified scientific knowledge is mainly created in the fourth phase, while uncodified knowledge develops throughout all phases. Practical applications and tools were primarily developed in the second phase. Particularly intricate applications and tools that were not strictly necessary for the research program were developed in subsidiary projects or will be refined after the end of the research program.

Reasons for the framework’s suitability for research programs

Finally, we provide reasons for why the presented framework is particularly suitable to describe research and design in dual-level, multi-project research programs. The main argument for our framework’s suitability in this context is its successful modelling of the aforementioned dual-level structure. As we have seen, team members’ activities on both levels are rather heterogeneous (see also Tables 3 and 4). Moreover, roles cannot be strictly assigned to certain team members on each level. In contrast, team members took on multiple roles that changed to some extent over the course of the research program. Also, the schedules of the research committee and research projects varied, leading to a situation in which the two levels were in different program/project phases at a given point in time.

Discussion

Summary

The proposed framework elucidates research and design in multi-project research programs empirically investigating simulation-based learning. The framework was derived on the basis of theory and specifies the activities of a research committee and subordinate research projects in four program/project phases in order to create scientific knowledge as well as practical applications and tools. Illustrations from the research program COSIMA exemplified that the basic premises of this framework are applicable. As expected, the program/project phases were completed at different times on each level, and iterated and overlapped considerably. The illustration of research project COSIMA 4 demonstrated that the second phase of the proposed framework represents the simultaneous research and design of simulation components and research materials well.

Implications for theory

Contributing to the EDR literature, the proposed framework complements other descriptive models of simultaneous research and design processes. The proposed framework is briefly compared to the most similar frameworks from this literature that served as models. At the same time, however, the ways in which our framework goes beyond existing models are also clarified. The ILDF (Bannan-Ritland 2003) is a general research and design framework directed at the program level comprising the phases of exploration, enactment, and evaluation. The ILDF applies a rapid prototyping approach and addresses the adoption of created artifacts. In comparison to the proposed framework, the ILDF is more general, does not delineate leadership tasks and seems more geared towards developing artifacts for applied research. The generic model for EDR (McKenney and Reeves 2012) and the osmotic model by Ejersbo et al. (2008) address research and design on a project level and posit the equal status of research and design throughout the project. Due to the equal status of research and design in all phases of these frameworks, the generic model for EDR and the osmotic model may be most suitable to characterize and guide applied research in regular EDR contexts. We believe, however, that our framework is more suitable for research programs that focus on basic research and thus often contain only one (iterative) phase in which research and design are equally important. The “compleat” design experiment model (Middleton et al. 2008) integrates the design of an empirical investigation into an educational research process. This model is to some extent similar to our proposed framework because both locate research and design simultaneously in the middle phase of an extensive research process. Contrary to the proposed framework, the “compleat” design experiment focusses on a single research project instead of a research program. Moreover, it describes the development of regular randomized controlled trials instead of research materials and simulations for an empirical investigation of simulation-based learning. In the model from Akkerman et al. (2013), the three epistemic practices of research, design, and educational change are on an equal level over the entire course of a single research project. Consequently, this model seems more suitable than our framework for contexts like translational science in which educational change (i.e., adoption of a created intervention) is crucial. Concluding this analysis of similar models and the described theoretical background, the proposed framework is the only dual-level, multi-project EDR framework. It places a stronger emphasis on research than the other described frameworks, with the exception of the “compleat” design experiment (Middleton et al. 2008). Moreover, the proposed framework is more context- and content-specific than the other EDR models. Concerning the context-specificity, the proposed framework demonstrates that the organizational structure (e.g., team level) can significantly affect EDR and that theories from other fields, such as team science or organizational psychology, can offer valuable insights for representing the context in EDR frameworks. With regard to the content-specificity, our proposed framework is more specific than the other EDR frameworks due to its focus on simulations that assess and facilitate knowledge and competences in medical education and teacher education. In the proposed framework, the challenging need to develop complex content for simulation components and research materials probably led to the equal status of research and design in the second program/project phase. These conclusions on context- and content- specificity show that educational design researchers who create novel EDR frameworks should not hesitate to explore specific contexts and contents that pose unique demands on research and design.

The presented framework also adds to the literature on simulation research and design in medical education and teacher education. In contrast to our framework, the framework by Cheng et al. (2017) focusses on carrying out one empirical study across different simulation centers. Similarly to our framework, Cheng et al.’s (2017) framework includes recommendations for different phases that can serve as a valuable resource for research and design. Other frameworks and models are more specific than our framework, and focus on specific fields such as pediatrics (Cheng et al. 2014a), live simulations (Khan et al. 2013a; Sturpe and Schaivone 2014), and digital simulations (Posel et al. 2015; Zary et al. 2006). Our framework extends this literature by providing an overview of all typical research materials and simulation components that have to be designed and developed and recommendations for this process. By focusing on simulations assessing knowledge and competences, our framework remains sufficiently specific, while being still applicable to many other contexts.

Implications for practice

Leaders of research programs, as well as entire research committees, may gain from the presented framework for conducting such programs. The overview of and recommendations for activities outlined for the research committee level (see Table 3) provide valuable ideas that can be followed and included in grant proposals, which increasingly comprise a section on project coordination. Leaders of research programs and entire research committees can also use the presented framework to track in what phases research programs and projects currently are, what specific research materials and simulation components have been produced, what activities are taking place and which role different team members are playing. These points may improve the management and coordination of research projects, help to conduct state-of-the-art research, and facilitate standardization.

Members of research projects can follow the recommendations for activities in the described project phases and read up on the literature resources provided for certain aspects (see Table 4). They can also use the presented framework to monitor in what phases their research projects currently are, what specific research materials and simulation components have been developed, what activities are taking place, and what role they are playing. These points should support members of research projects in developing high-quality simulation components and research materials, help to conduct state-of-the-art research, and keep the project within its timeline and budget.

Limitations of the framework and directions for future research

Of course, the presented framework is not without limitations that require consideration. One limitation of the presented framework results from the employed methodology. The framework was derived from theory, but illustrated through a research program in which the authors are active members. At first glance, this methodology may seem less scientific to educational researchers, who primarily conduct experiments in highly-standardized laboratory settings (Collins et al. 2004). However, as EDR “focuses on understanding the messiness of real-world practice” (Barab and Squire 2004, p. 3), such a methodology is required for the development of new context-specific frameworks and has been employed in the development of other EDR frameworks (e.g., Bannan-Ritland 2003). Another limitation of the presented framework is that only one illustration of a research committee and one illustration of a research project from the domain of medical education were provided in the text. Even though this small number of illustrations was necessary to convey our framework in a comprehensible way, it also impairs to some extent the framework’s generalizability to other domains, notably teacher education. In response to these two points of criticism, we have provided reasons for why our framework depicts this type of research program well and specified commonalities between research and design in medical education and teacher education.

The described framework touches upon two topics that seem especially promising for future research independent of the proposed framework. One interesting topic for future EDR frameworks is team science. The organizational structures and collaborative processes in multi-team research and design programs seem to have not yet been sufficiently addressed in EDR frameworks (compare the frameworks presented in McKenney and Reeves 2012). For instance, new EDR frameworks could put a stronger emphasis on the EDR process and integrate models of collaborative problem-solving (see Graesser et al. 2018). Another exciting avenue of research concerns EDR methodology. With the rise of new technologies such as online collaboration software, research and design processes and the created products are increasingly stored electronically. This development will allow educational design researchers to validate EDR frameworks and investigate research and design processes using electronic data without noticeable intrusions. This validation of EDR frameworks with electronic data would respond to a long-standing request for the EDR literature (Kelly 2006).

With regard to the proposed framework, it would be interesting to find out whether the described phases and activities also occur in research programs from other domains. There are other domains in which training knowledge and competences with simulations could be investigated fruitfully within a research program. For instance, simulations are already used frequently for the purposes described above in nursing education (Cant and Cooper 2017) and should become more and more popular in foreign language learning in the future (Blyth 2018). Moreover, it could be investigated whether the presented framework can be applied successfully in research programs that develop complex artifacts other than simulations, such as serious games. Serious games refer to the creation of complex and immersive virtual worlds that resemble digital simulations. Thus, their design process could follow rather similar principles to that of simulations (Kirkley et al. 2007). Research programs on serious games could take away from our model recommendations for implementing assessment and training, for example. Apart from this, it remains an open question whether the presented framework is applicable in different organizational structures. The framework has been illustrated with one interdisciplinary research program that was conducted at multiple research groups at several universities. Larger research centers, for instance, also have a relatively strong hierarchical structure but nevertheless coordinate several full research programs and their associated research projects on a superordinate level.

Conclusions

We have presented a framework for research and design in research programs consisting of one research committee and multiple research projects. This framework is particularly suitable for research programs that focus on simulation-based learning but could potentially also be used in other research and design settings that comprise the design of complex scientific artifacts. On a theoretical level, the framework contributes primarily to the literature on EDR by offering a unique dual-level perspective that delineates relevant phases, activities, roles, and products. Moreover, on a practical level, the framework may guide the research and design process of leaders and members of research programs. We hope that the checklists provided in this article are a valuable resource for this aim.