The learning effects of United Nations simulations in political science classrooms

Abstract

How do active learning environments—by means of simulations—enhance political science students’ learning outcomes regarding different levels of knowledge? This paper examines different UN simulations in political science courses to demonstrate their pedagogical value and provide empirical evidence for their effectiveness regarding three levels of knowledge (factual, procedural and soft skills). Despite comprehensive theoretical claims about the positive effects of active learning environments on learning outcomes, substantial empirical evidence is limited. Here, we focus on simulations to systematically test previous claims and demonstrate their pedagogical value. Model United Nations (MUNs) have been a popular teaching device in political science. To gain comprehensive data about the active learning effects of MUNs, we collect data and evaluate three simulations covering the whole range of simulation characteristics: a short in-class simulation of the UN Security Council, a regional MUN with different committees being simulated, and two delegations to the National Model United Nations, for which the students prepare for 1 year. Comparative results prove that simulations need to address certain characteristics in order to produce extensive learning outcomes. Only comprehensive simulations are able to achieve all envisioned learning outcomes regarding factual and procedural knowledge about the UN and soft skills.

Introduction

There is considerable support for the argument that active learning techniques promote students learning outcomes better than traditional “passive” approaches. In particular, simulation games are described as being useful pedagogical tools for teaching political science. Simulation games regarding the working procedures of the United Nations (UN) are called Model United Nations (MUNs) and claim to simulate real processes as realistically as possible, to educate the participants about the structure and functionality of the UN, and to raise awareness about International Relations (IR) (Wesel 2000: 575). This paper seeks to add to the debate on the validity of UN simulations as active learning techniques by testing whether or not they enhance students’ learning outcomes, measured as a perceived increase in factual and procedural knowledge and soft skills. We define learning outcomes broadly as “what a learner knows or can do as a result of learning” (Otter 1992: i) and follow Allan (1996) in differentiating generic academic outcomes such as factual and procedural knowledge about the UN and key transferrable skills. They may have been taught directly or they may have been learned as a consequence of the student’s involvement in the MUN learning experience. We test the hypothesis that simulations need to include a certain range of characteristics in order to increase students’ learning effects regarding factual or procedural knowledge or soft skills. Therefore, the following research question will guide our discussion: Can active learning environments – by means of simulations – enhance political science students’ learning outcomes on different levels of knowledge?

To answer this research question, we discuss the potentials and benefits of simulation games in teaching about the UN and IR. Therefore, we operationalize characteristics of UN simulations before evaluating simulations that cover the whole range of these characteristics in order to gain comprehensive data and provide empirical evidence of their effectiveness. For systematizing our findings, we apply three levels of knowledge and develop a corresponding framework for UN simulations.

UN simulations and learning outcomes

Academic literature shows a growing consensus on the benefits of using active learning methods in political science classrooms (Andonova and Mendoza-Castro 2008; Asal and Blake 2006). In this paper, we focus specifically on simulations, which are “interactive events” in which “it is the environment that is simulated […] but the behaviour is real” (Jones 1995: 7). They are especially beneficial in political science, because “like card games and sports, politics is something that makes the most sense if it is actually played” (Grant 2004: viii). Simulations do not only help to gather factual knowledge, but also to apply this knowledge in a specific situation. Most notably, simulating negotiations gives students the opportunity to experience the dynamics of international bargaining procedures (Chasek 2005; McIntosh 2001) and to develop soft skills (Asal and Blake 2006).

Arguably, there is not enough empirical evidence to support the claim that simulations produce meaningful learning outcomes (Chin et al. 2009). Several studies tried to fill this gap by applying systematic measurements such as pre- and post-test questionnaires (Krain and Lantis 2006) or introducing a control group (Lay and Smarick 2006; Powner and Allendoerfer 2008). However, these studies were only able to present mixed results about learning outcomes and effectiveness. This paper aims to provide empirical evidence for the claim that simulations indeed enhance students’ learning outcomes and convey substantive knowledge about political processes. In contrast to previous studies, we comparatively evaluate three different types of simulations with the same methodological approach, therefore providing new insights. Additionally, we focus on students’ perceptions regarding their learning outcomes and shift the analytical focus to a new perspective when analysing the effects of simulations. This paper adds to the current debate by integrating the participants’ views into the assessment of the learning effects of UN simulations.

We focus on UN simulations as those that imitate a political process as close as possible to the real-life process. Empirical studies were able to support the claim that students who participate in UN simulation have a higher learning success (Powner and Allendoerfer 2008: 86). Hazelton and Jacob (1983: 96) were able to show that 98.7% of the participants prefer the learning outcomes they gather through UN simulations to those of a classic seminar. Engel et al. (2017) offer some interesting insights into the effects of UN simulations on four different levels of knowledge: factual, conceptual, procedural and metacognitive knowledge. Their results prove that deep learning had occurred across all four levels, but that most progress was achieved with regard to conceptual and metacognitive knowledge. Other scholars have used similar frameworks to study the effects of UN simulations. Calossi and Coticchia (2017: 17) are able to prove that factual knowledge about IR increases after engaging in UN simulations. By participating in UN simulations, students can also improve their soft skills in areas of “creative thinking, critical thinking, problem solving, foundations, inquiry/analysis, information literacy” (Fortin 2002: 20). We adjust this framework to three levels of knowledge (factual, procedural, and soft skills) in order to determine and evaluate the learning outcomes of UN simulations. We argue that, in order to increase a student`s learning outcomes on three levels of knowledge, a broad range of characteristics should be addressed by simulations.

Research design: assessing the learning effects of UN simulations

In general, a proper assessment of simulations can help to demonstrate that they are productive tools for learning and that their prevalence in classrooms is justified (Raymond and Usherwood 2013). Our empirical analysis is based on the hypothesis that simulations—as active learning techniques—need to include a certain range of characteristics in order to achieve learning outcomes regarding factual or procedural knowledge or soft skills. We expect that more comprehensive simulations, including more profound preparation and longer simulation stages, are better suited to teach academic outcomes and lead to students’ improvement of key soft skills. We operationalize learning outcomes indirectly by measuring the students’ individual levels of achievement.

Case selection

In order to empirically test our hypothesis, we selected three UN simulations that cover the whole range of simulation characteristics. The first empirical example deals with a simulation of the UN Security Council (SC) embedded in a conflict resolution course taught at Goethe-University Frankfurt during the winter term 2017/2018. It was designed for graduate students with preliminary knowledge in IR and included 45 participants. The simulation was designed to give students a practical example of conflict resolution (Knittel and Neukirchen 1999). The students were given role cards and simplified Rules of Procedure (RoP) and prepared individually. Overall, 18 students actively participated in the simulation as delegates or journalists; the remaining students acted as observers. Evaluation of the learning outcomes was based on a pre- and post-survey conducted in the week before and after the simulation. We were able to assign individual identification numbers to each student who remained constant over the survey dates. In total 75% of the participants completed both surveys.

MainMUN, the second empirical example, is a simulation for students, which has been taking place at Goethe University since 2004.¹ Nearly half of the participants study in Frankfurt; the rest comes from other (international) universities.² The delegates prepare themselves individually for the role of representing “their” country with the help of background guides. The actual simulation lasts three and a half days. MainMUN represents a relatively realistic and negotiation-oriented simulation that requires a lot of preparation from the participants. MainMUN 2018 attracted 196 participants to simulate the work of six committees. Again, evaluation of the learning outcomes is based on pre- and post-surveys, which were conducted in the first and last sessions. Due to the size of the group and limitations in the evaluation software, it was not possible to assign individual identification. Thus, we will only be able to make general claims about students’ learning outcomes. In total, 66% of the participants completed the pre-survey and 52% completed the post-survey.

The third empirical example deals with the world’s biggest UN simulation, NMUN, which takes place every year in New York. The dataset used for our study consists of the NMUN classes of 2016/2017 and 2017/2018 and includes twenty-eight students from Goethe-University. In order to participate in NMUN, the students undergo a 1-year preparation course. During NMUN itself, the students work every day from dusk till dawn to enforce their country’s position. The evaluation of the learning outcomes is based on pre- and post-surveys. Additionally, the students of the NMUN class of 2016/2017 had the opportunity to formulate learning objectives, which formed the basis of our questionnaires. In total, 94% of the students completed the pre-survey. The post-survey was conducted at the end of NMUN and was completed by 87%.

All three UN simulations follow the traditional stages of simulation games, which consists of preparation, simulation, and reflection (Massing 2010: 171). But the simulations differ in the extent and intensity in which they deal with all three stages. We operationalized the three stages by employing twelve characteristics (Table 1). Along those lines, our three cases can be assigned to three different categories of UN simulations that differ with regard to their intensity and comprehensiveness.

Table 1 Three categories of UN-simulation games.

Method and data collection

The assessment³ of the learning outcomes in the three simulations is based on pre- and post-surveys with closed and open questions. The complete sample consists of 159 students. The written surveys took place shortly before and after the simulation games, in order to measure the effects of the simulations as directly as possible. We expect all three simulations to produce positive changes with regard to two types of learning outcomes: generic academic outcomes (measured as factual and procedural knowledge) and soft skills. Both types of outcomes are subject-specific and individualised to a certain extent (Allan 1996). In order to acknowledge the individualised learning process and to include students in the research process, we developed concrete learning outcomes in an inclusive process with the students of the NMUN class of 2016/2017. This way, our analysis will produce data on students’ perceptions regarding their learning outcomes, which have not been seriously considered so far.

The final questionnaires were designed in two steps. Questions regarding the participant’s disposition towards and their knowledge about the UN (factual knowledge) were based on the measurement model by Oberle and Leunig (2017) and had thus been validated in previous studies. Questions regarding personal learning objectives (procedural knowledge and soft skills) were developed in cooperation with the students and represent a new measurement for the analysis of learning effects. The closed questions on UN-related dispositions and assessments of the students’ learning outcomes were built around a five-level Likert scale, which allowed an approval or rejection of statements. The results are thus not hard learning outcomes but the perceptions of students. The post-surveys additionally included questions on the students’ satisfaction with the simulation and the evaluation of its effects, and thus had a feedback function.

Results

The following section presents the results of a descriptive statistical analysis of three UN simulations with regard to the achievement of pre-defined learning outcomes. They present a first insight into the different learning effects of simulations and aim to substantiate previous theoretical claims (Raymond and Usherwood 2013). Since we are limited in the presentation of our results, we will focus on the most relevant for each simulation category.

Basic UN simulation: emergency session of the UN Security Council

In the basic category we evaluate a 3-h simulation of a UNSC meeting. In order to examine a potential increase in factual knowledge of the participants before and after the simulation, we asked them how resolutions are passed in the Security Council and gave the students three different answer options. Even in this short simulation, students should learn about this procedural rule by following the negotiations. Initially, 52% picked the correct answer “with simple majority without veto”; after the simulation game, this number increased to 59% (Fig. 1). The simulation could indeed improve the factual knowledge of some students. However, the proportion of wrong answers remains quite high as well, pointing to the limits of this short simulation game.

Fig. 1

How are resolutions passed in the UN Security Council? (Pre- and post-survey)

Most respondents express a constant level of knowledge, while 67% of the delegates of permanent SC members indicate a knowledge increase by one level after the simulation. For the non-permanent members and conflict parties, the results are mixed, with half of them stating an unchanged level of knowledge about the UN and half showing an increase by one level. Most of the observers indicate an unchanged level of knowledge. But four of them report an increase by one level and one student stated to have gained a knowledge increase by two levels after the simulation.⁴

In the post-survey, the students expressed a significant increase in the understanding of political processes and the functioning of the SC (Fig. 2, n = 34). According to their statements, the insight into how difficult it is to find a common solution or compromise was particularly impressive. Since reservations about the UN often relate to the duration of decision-making processes, this learning effect appears particularly relevant. In contrast, for most students, the simulation game was not able to depict the everyday relevance of the UN.

Fig. 2

Simulation game effects from students' point of view (in per cent)

Apart from factual knowledge, the survey also included questions regarding procedural learning outcomes and soft skills. The results of these questions are based on self-assessment. In summary, the simulation enabled the students to deepen their knowledge of international politics and to become acquainted with the working methods and rules of the UN (Fig. 3, n = 34). In comparison, the participants were less successful in achieving soft skills like leading debates or giving speeches. Those goals are more intricate, and it is thus not surprising that they were only fully achieved by around 10%. It seems that, while basic simulations are able to increase factual and simplified procedural knowledge, they are less suitable to convey soft skills.

Fig. 3

Individually achieved learning objectives of UN simulation participants (in per cent)

In summary, it can be stated that participation in the UN simulation led to a significant increase of interest in the topic “UN”. After the simulation, the participants have a considerably higher level of factual knowledge, which they also perceive themselves. Overall, the results confirm that simulations are a sensible method for communicating the challenges of international conflict resolution.

Moderate UN simulation: MainMUN

Compared to the previous example, MainMUN has a longer preparation phase and the simulation itself is based on more complex RoP. In order to examine the factual knowledge of the participants before and after MainMUN, the question “how many permanent members are in the SC?” was asked with different answer options. Initially, 74% picked the correct answer “5”; after the simulation game, this number increased to 83% (Fig. 4).⁵ Over the course of the survey period, participants were able to find out how many permanent members are in the SC and how they influence the decision-making process within the UN.

Fig. 4

How many permanent members are in the UN Security Council? (MainMUN pre- and post-survey)

When asked about their assessment of MainMUN, participants expressed a significant increase in the understanding of political processes and the functioning of the UN (Fig. 5, n = 101). According to their statements, the insight into how difficult it is to find a common solution was particularly impressive. Furthermore, the simulation was able to impart a better understanding of the working procedures and negotiation structures within the UN. MainMUN was less successful in disclosing the ways in which the UN influences everyday life.

Fig. 5

MainMUN effects from students' point of view (in per cent)

With regard to learning outcomes, all of them have been fully or partially achieved by half of the participants, which substantiates the positive effects of simulations (Fig. 6, n = 101). Not surprisingly, the best results were obtained for the goals “exchange and dialogue with other students” and “to have fun”, which were fully achieved by nearly 50% and at least partly achieved by nearly all respondents. Due to the enforcement of the strict RoP, 74% of the participants at least rather achieved (one-third fully achieved) the goal of becoming acquainted with them. MainMUN was able to increase the participants’ proficiency in English (67% partially or fully achieved) and successfully conveyed diplomatic behaviour (69% partially or fully achieved). With regard to factual knowledge, 74% of the respondents partially or fully achieved deepening their knowledge of international politics and 73% at least partially acquired increased knowledge about UN working methods. Learning objectives that were less successfully achieved include the leading of debates (13% fully achieved), writing speeches and presenting them confidently (22% fully achieved), successfully implementing a country’s position (18% fully achieved), and developing problem-solving skills (22% fully achieved). Those five items represent rather intricate learning outcomes that we would expect to take longer to fully achieve. It is thus not surprising that fewer participants were able to fully achieve them in a 3-day simulation without intensive preparation.

Fig. 6

Individually achieved learning objectives of MainMUN participants (in per cent)

In summary, it can be stated that participation in MainMUN led to a significant increase in the respondents’ factual knowledge. After the simulation, the participants had a better understanding of political processes and the functioning of the UN. They gained insights into the difficulties of international negotiations and of finding a common solution. A closer look at learning objectives revealed that UN simulations are indeed able to convey the working methods within the UN, to increase proficiency in English and to offer an introduction to diplomatic behaviour. MainMUN did less well with those learning objectives that are more protracted and that apparently cannot be achieved in the short amount of simulation time.

Comprehensive UN simulation: NMUN

NMUN represents perhaps the most comprehensive UN simulation that addresses all three stages of simulation games. Here, students had the opportunity to indicate the importance of several learning outcomes during their preparation for NMUN (Fig. 7, n = 28). In summary, all learning outcomes are very important or rather important for more than 70% of the participants. The outcomes “to successfully implement the country’s position” (85% very important) and “to learn to lead debates” (71% very important) are most important. In contrast, only 32% of the students chose “to develop problem-solving skills”, while 36% chose “to become acquainted with the UN rules of procedure”, as being very important. Those results show that even if participating in a NMUN class with the same preparation and the same learning materials, the learning outcomes of the students will still be individual.

Fig. 7

Importance of learning objectives for NMUN preparation from students’ point of view (in per cent)

In the post-survey, the students were asked to indicate which learning outcomes they had achieved (Fig. 8, n = 26). The outcomes “to become acquainted with the UN rules of procedure”, “to successfully implement the country’s position”, “to have fun” and “exchange dialogue with other students” show the best results and were fully achieved by 40–50% of all participants and at least partly achieved by 80% of the respondents. This range of achieved learning outcomes shows that participating in a simulation gives students the chance to improve their knowledge on all three levels. Only two outcomes were less successfully achieved. A closer look should be taken at the category “to develop problem-solving skills”. Here, only one quarter opted for “fully achieved” and three-quarters for “partially achieved”. The question is how the participants defined “problem solving”—which could be either finding a solution for the problems discussed at the simulation or developing skills to solve the problems that arise in discussions.

Fig. 8

Individually achieved learning objectives of NMUN participants (in per cent)

Comparing the important learning outcomes during the preparation with the achieved learning outcomes shows that the outcome “acquisition of knowledge about UN working methods” was only fully reached by 35% but was important for the participants during the preparation (68% very important). This is surprising, as UN simulations should increase procedural knowledge and the results of MainMUN showed that this is possible. A total of three categories performed much higher in the aspect of “fully achieved” compared to the importance during the preparation. These are “to become acquainted with the UN rules of procedure” (58% fully achieved, compared to 36% very important), “to have fun” (65% fully achieved, compared to 36% very important) and “exchange and dialogue with other students” (65% fully achieved, compared to 39% very important). The results support the findings on these categories from the MainMUN simulation. They indicate that a longer preparation period helps the students to reflect on their learning outcomes and to gain higher outcomes in those three categories.

In the end, the participants were asked to evaluate their learning effects of NMUN (Fig. 9, n = 24). A comparison with MainMUN results shows that the effects are almost the same, but performance is higher for NMUN. There is a significant increase in the understanding of political processes and the functioning of the UN. At the same time, both simulations have not been able to convey ways in which the UN influences the students’ everyday life. It is important to note that it became very clear for the students how difficult it is to find a common solution. This outcome provides an answer to the results of the learning outcomes and the category “to develop problem-solving skills” that was only fully achieved by one-fifth of the participants.

Fig. 9

NMUN effects from students' point of view (in per cent)

In summary, the participation in NMUN led to a significant increase in the respondents’ factual and procedural knowledge about the UN and in their soft skills. Comparing the NMUN results to the other two simulations, it becomes clear, that the achievement of rather comprehensive learning outcomes takes a longer time and requires extensive training and preparation.

Conclusion

Believing in the potentials and benefits of simulation games, this paper aimed to contribute to filling the research gap in quantitative studies regarding the learning outcomes of students participating in UN simulations. With empirical data from UN simulations including different characteristics, we showed differences in the perceived achievement of students’ learning outcomes regarding factual and procedural knowledge and soft skills.

Our analysis was guided by the hypothesis that simulations need to address a certain range of characteristics in order to achieve learning outcomes regarding factual or procedural knowledge and soft skills. Results show that all simulations enabled students to achieve their learning outcomes. It is notable that factual knowledge improves even with a basic simulation. These results indicate that knowledge of the UN can be gathered quickly with the help of an active learning approach. With regard to the effect of simulation characteristics on the achievement of these learning outcomes, our results show that, in a comprehensive simulation the students can gain a better understanding of difficult topics and display a greater success in achieving learning outcomes in all three types of knowledge. Going back to Table 1, we can thus conclude that instructors should put specific emphasis on the preparation phase of a simulation, which should also be done in a course setting and not just alone at home. The active support through a faculty advisor also intensifies the preparation, leading to better outcomes. If possible, international experience should be included in the simulation design, either through international experts or a change in location. In the end, debriefing sessions and reflections are helpful in allowing students to acknowledge their achievements (Table 2).

Table 2 Learning outcomes of UN-simulations according to different simulation categories.

Still, the results indicate (as shown in the chapter on comprehensive UN simulations) that learning outcomes are an individual process. A possible explanation could be that the results are based on self-evaluation and every student might evaluate him or herself in a different way, as we did not set a baseline for levels of achievement. Therefore, it would be interesting for further research to answer the question of why and how these learning outcomes are achieved. It would be compelling to focus more on the process of how these learning outcomes relate to differences in the preparation and reflection stages. These remaining questions also illustrate the limitations of this study. For further research it would thus be interesting to have a control group that is not taking part in the preparation or reflection stage to differentiate to what extent the positive effects on the learning outcomes are really a result of the simulation. It would be interesting to test for example the problem-solving skills before and after the simulation, as a questionnaire has its limitations, such as a personal bias or different understandings of skills.

Based on empirical evidence, our paper was able to substantiate the theoretical claim that active learning techniques have a positive effect on students’ learning outcomes. The more characteristics are addressed in UN simulations, the more learning outcomes regarding all levels of knowledge are achieved. A short in-class simulation is able to convey factual knowledge and simplified RoP but the improvement of soft skills is not within its reach. As the simulation covers more characteristics, more learning outcomes can be achieved by the participants. In contrast to the in-class simulation, MainMUN was able to improve basic soft skills like giving speeches. Only a comprehensive simulation, including a preparation seminar applying active learning techniques, was able to achieve learning outcomes on all three levels. We can thus conclude that it is meaningful to invest in active learning techniques in order to improve political science students’ learning outcomes. If the learning outcomes include procedural knowledge and/or soft skills, instructors need to design simulations that address the whole set of characteristics.