1 Introduction

The role of science in policy making has been a subject of a lively debate, especially regarding the effective communication of scientific information about climate change to policy makers and public audiences. Climate change generates particular knowledge-related challenges for policy makers, especially the need to deal with uncertainty (Marx et al. 2007), understand complex-system dynamics, imagine possible long-term futures (Milkoreit 2015, 2019) and assess different solutions and their long-term impacts (Tàbara et al. 2017). Communicating these characteristics of climate change is challenging, which can lead to misinterpretations and limit the uptake and use of available information (Enserink et al. 2013; Wardekker et al. 2008). Hence, the complex, uncertain and long-term nature of climate change calls for science-policy communication tools which involve multiple forms of experience and representation (Milkoreit 2015).

Serious games, in particular role-playing simulation (RPS) games, offer a promising avenue for effective science-policy engagement regarding climate change (Vervoort 2019). In RPS games, participants take on particular roles and interact with others according to set rules that simulate the outcomes of their interactions. That is, RPS games allow for a ‘safe space’ to experiment with policy responses to complex problems; to learn interactively about complex system dynamics; to simulate a ‘lived experience’ of future realities, including the impacts of present decisions and policies; and to explore different perspectives of stakeholders (e.g. Fleming et al. 2020; Rumore et al. 2016; Sterman et al. 2015). Through this experiential engagement, RPS games can affect different cognitive and emotional processes of participants, including their knowledge, beliefs and risk perceptions (Meya and Eisenack 2018; Van Pelt et al. 2015).

A number of serious games regarding climate change have been developed over the last decade (Galeote et al. 2021; Flood et al. 2018; Reckien and Eisenack 2013; Wu and Lee 2015), but empirical research on game effects has remained limited. With few exceptions (e.g. Parker et al. 2016; Onencan et al. 2016), most climate games target lay audiences or students (Galeote et al. 2021; Rumore et al. 2016) rather than policy makers. Correspondingly, research on game effects among policy audiences, i.e. the games’ effectiveness as science-policy engagement processes, remains particularly thin. Beginning to fill this gap, we investigated the effects of a RPS game tailored specifically to the needs and interests of the international climate negotiation community on players’ risk perceptions regarding climate tipping points.

We developed a RPS game with the purpose to provide a multi-modal, experiential learning environment regarding climate tipping points and their relevance for global climate governance. We conducted game workshops with observers of the international climate negotiations (primarily representatives of non-governmental organizations) and scientists and analysed the effects of gameplay on their risk perceptions. While we explore a range of risk dimensions, we focus, in particular, on psychological distancing, the “subjective experience that something is close or far away from the self, here, and now” (Trope and Liberman 2010, p. 440).

Climate tipping points present an interesting case to study science-policy engagement, because it encapsulates all the challenging characteristics of climate change (uncertainty, complexity, long time horizons), but it has significant additional risk implications due to their large-scale nature, potential irreversibility and implications for human wellbeing. Moreover, climate tipping points are still largely ignored by international policy makers (Milkoreit 2015, 2019) and preliminary evidence suggests that risk perceptions regarding tipping points might not match risk realities (Bellamy and Hulme 2011).

We discuss key concepts and theories that inform our research design in Sect. 2, concluding with the literature-informed hypotheses. Section 3 (and the supplementary materials) details our methodological approach and challenges, including a description of the RPS game. The results of our quantitative and qualitative analyses are summarized in Sects. 4 and 5, respectively. In Sect. 6, we explore the reasons for our findings and seek to explain the differences between the quantitative and qualitative analyses. We conclude with implications for future research on serious games as science-policy engagement tools.

2 Perceptions of climate change risks and role-play simulation games

Below, we review relevant insights of two interdisciplinary research fields that inform our research design: work on factors that influence climate-related risk perceptions among different audiences and research on innovative approaches, including serious gaming, to overcome communication challenges at the science-policy interface.

2.1 Perceptions of risks associated with climate change

Climate change risk perceptions have been studied extensively, including their influence on people’s willingness to engage in pro-environmental behaviours or to support specific policies. This large body of work points to a broad set of interrelated variables affecting climate-related risk perceptions, including knowledge, personal experience of climate impacts, emotions, cultural worldviews and psychological distancing. Many of these variables can potentially be influenced through serious gameplay. Most relevant for our study are knowledge, emotions and distancing.

While several studies have indicated that climate change knowledge is positively related to risk perceptions (Milfont 2012; Xie et al. 2019), others point to a more complicated relationship as knowledgeable individuals exist at both ends of the risk perception spectrum (Capstick and Pidgeon 2014; Kahan et al. 2012). Pre-existing beliefs and cultural worldviews may, in fact, be stronger predictors of climate change risk perceptions than knowledge (Kahan et al. 2012; Libarkin et al. 2018).

Emotions play an important role in moral judgements of acceptability of risks (Roeser 2006, 2012) and have been found to influence moral reasoning of climate negotiation participants (Milkoreit 2015). However, the role of emotions is contentious in the risk perception literature. On the one hand, limited emotional involvement has been offered as an explanation for the lack of urgency related to climate change. Hence, scholars have argued for stronger emotional appeals in climate risk communication (Leiserowitz 2006; Weber 2010; Feldman and Hart 2016) and explored the emotional engagement potential of different forms of communication (Gustafson et al. 2020). On the other hand, research has demonstrated that messages focusing on specific emotions, especially fear, can be ineffective at best and counterproductive at worst (Ettinger et al. 2021; O’Neill and Nicholson-Cole 2009). Others point to a complex, sometimes counterintuitive, and nuanced role of emotions in climate communication, emphasizing the importance of message tailoring to specific target populations (Chapman et al. 2017).

Similar to other environmental risks, perceiving climate change risks involves psychological distancing mechanisms. Psychological distancing in the context of climate change refers to the tendency to believe that climate change affects people in geographically distant locations and people in the distant future more seriously than those who live close to us in space and time, including ourselves (Leiserowitz 2006; Spence et al. 2012). In addition, climate change is more often perceived as a collective-societal rather than an individual-personal risk (Van der Linden 2015). Uncertainty regarding the types and timing of climate change impacts invites distancing. The idea that we do not know exactly when certain effects will set in allows us to doubt their seriousness and potential impact on our own lives (Jones et al. 2017; Spence et al. 2012).

These four dimensions of distancing (geographical, temporal, social, likelihood) build on construal level theory (CLT; Trope and Liberman 2010), which posits that perceptually distant phenomena are construed on a higher, more abstract level and closer phenomena on a more concrete, contextualized level. These findings are insightful for climate communication: framing climate change as a local rather than global issue, or as an individual rather than societal threat, can reduce the psychological distance (“proximising”; Jones et al. 2017; Loy and Spence 2020). However, CLT has been criticized for having a limited explanatory value of climate (in)action, especially because its use neglects the complex cognitive processes of risk processing as well as changes in belief systems (Brügger et al. 2016; Brügger 2020).

2.2 Perceptions of risks associated with climate tipping points

As defined by Lenton (2011, p. 201), “A climate ‘tipping point’ occurs when a small change in forcing triggers a strongly non-linear response in the internal dynamics of part of the climate system, qualitatively changing its future state”. Examples of climate tipping processes include the bleaching and die-off of coral reefs and the melting of the Greenland Ice Sheet. Key characteristics of tipping-point dynamics include the fundamental reorganization of a system (the tipping element) during its movement from one stable state to another, the abruptness (i.e. non-linearity) of the change process and the potential irreversibility of these changes on time scales relevant for human decision-making. There is a high degree of uncertainty regarding their timing and impacts as well as social responses to tipping points (Dessai and Van der Sluijs 2007). Recent evidence suggests that tipping points may be closer than previously thought (Lenton et al. 2019). Hence, climate scientists repeatedly stress the urgency of stricter climate policies and stronger mitigation efforts to prevent future tipping processes in the climate system (Lontzek et al. 2015).

These characteristics, especially non-linearity and uncertainty regarding the time of their occurrence, pose a number of risk perception challenges (Sterman 2011). Humans usually perceive time and causation in a linear fashion and have difficulty perceiving non-linear temporal and causal dynamics (Dessai and Van der Sluijs 2007; Chi and Roscoe 2002), often lack cognitive skills to understand the behaviour of complex systems (Plate 2010; Milkoreit 2015) and face difficulties making sense of sudden, rapid and exponential changes (Pereira and Viola 2018). The relevant time horizons of climate tipping—ranging from years to millennia—present particular challenges for the human mind and for environmental governance (Galaz 2019). There is also emotional resistance to dealing with tipping points because it might cause anxiety, fear and feelings of powerlessness (Milkoreit 2015). Further, the availability heuristic limits the imagination of possible futures resulting from triggering climate tipping points: such counter-intuitive futures require information that is not readily available in people’s memories (Pahl et al. 2014). Making sense of such counter-intuitive future states should therefore involve the active and systematic ‘imagining the unimaginable’ (Dessai and Van der Sluijs 2007).

The literature specific to risk perceptions regarding tipping points is very limited and based exclusively on empirical research with lay audiences. Bellamy and Hulme (2011) found that their study participants were less concerned about tipping points compared to climate change in general. While tipping points were perceived as dangerous, study participants considered them unlikelyFootnote 1 and expected them to pose risks only to distant, developing countries and in the long-term future. A second study (Lowe et al. 2006) investigated changes in risk perceptions after seeing the movie ‘The Day After Tomorrow’, which used a narrative of extremely rapid climate change similar to a tipping-point scenario. After watching the movie, participants were more concerned about climate change but perceived abrupt climate changes as less likely and more temporally distant. The researchers explain this result by the fact that the movie is science fiction, creating disbelief among participants. The findings from these limited studies imply that distancing is a crucial mechanism in forming risk perceptions of climate tipping points. Based on these somewhat dated but highly relevant findings, we expected that psychological distancing mechanisms would play a particularly strong role in risk perceptions associated with climate tipping points in our study.

2.3 Role-play simulation games for science-policy engagement

Serious games, i.e. games with an educational purpose, have a long history in policy settings and decision support, because of their ability to simulate the implementation of policies and regulations, which enables the anticipation of their (unintended) effects (Mayer 2009). Simulation games are a particular type of serious game that have been used since the 1960s when researchers and policy makers started to realize the possibilities of combining computer simulations with human agency. They can support science-policy engagement regarding complex systems, as they allow players to experience the dynamics of social-environmental systems, and the impacts of their decisions over various time horizons (Alessi and Kopainsky 2015).

Recognizing these capacities, simulation games have been applied in various environmental policy domains, such as water management (Zhou and Mayer 2018) and urban sustainability (Mangnus et al. 2019). A large and growing number of games are aimed specifically at climate change engagement (Flood et al. 2018; Galeote et al. 2021; Reckien and Eisenack 2013; Wu and Lee 2015), potentially offering a science communication approach that is complementary to and supportive of scientific assessment reports. Most prominent among these simulation games is the World Climate Simulation (Sterman et al. 2015). Other examples of (role-play) simulation games include Sustainable Delta (Van Pelt et al. 2015), WeShareIt (Onencan et al. 2016) and CAULDRON (Parker et al. 2016). Rather than presenting scientific information to policy makers in written, visual and verbal forms, serious gameplay arguably involves richer, multi-modal engagement with the scientific information on possible futures, including more emotional, tactile, interactive and experiential forms of engagement (Candy and Dunagan 2017).

A vast body of literature on the effects of serious climate games suggests an array of potential cognitive, affective and behavioural effects, including learning about complex systems and increased concern about climate change (Galeote et al. 2021). For example, the World Climate Simulation of the international climate negotiations has been shown to increase knowledge and interest in learning about climate change as well as affective engagement with the topic (Rooney-Varga et al. 2018). A handful of studies on simulation games specifically demonstrated influence on perceptions of environmental risks (Rebolledo-Mendez et al. 2009), including increases in concern (Rumore et al. 2016) and improved understanding of uncertainty (Van Pelt et al. 2015). Apart from these studies, the empirical evidence of the effectiveness of RPS games remains scarce.

Given this growing evidence on the specific strengths of RPS games, they provide a promising toolset for science-policy engagement regarding climate tipping points. They can immerse players in creating and experiencing possible pathways into the future, bringing to life scenarios that are challenging to imagine (Vervoort 2019). Games can provide a safe space for experimenting with decision-making strategies regarding tipping points (Flood et al. 2018; Meya and Eisenack 2018). The iterative feedbacks between player decisions and game behaviour combined with the game’s ability to compress time enable participants to better understand the complex, non-linear and abrupt dynamics of the climate system over time (Meya and Eisenack 2018; Plate 2010). Moreover, players can gain insights into other players’ beliefs and perceptions regarding tipping-point risks through their role-play interactions, which can improve their understanding of the governance challenges involved (Rumore et al. 2016; Valkering et al. 2013).

Our study draws on the literatures reviewed above to study the impacts of a RPS game designed to engage members of the international climate governance community in a learning process about climate tipping points. Drawing on prior findings in this literature, especially regarding knowledge, multi-modal engagement and psychological distancing, we hypothesized that:

  • H1: Gameplay increases concern about climate tipping points

  • H2: Gameplay increases perceived seriousness of climate tipping points

  • H3: Gameplay decreases psychological distance of climate tipping points

    • H3a: Gameplay decreases perceived geographical distance of climate tipping points.

    • H3b: Gameplay decreases perceived social distance of climate tipping points.

    • H3c: Gameplay decreases perceived temporal distance of climate tipping points

  • H4: Gameplay increases the perceived likelihood of climate tipping points today.

3 Research design

As described in more detail below, we designed a RPS game with the specific purpose of providing a learning environment about climate tipping points for the international climate policy community, including negotiators, non-state actors and scientists. Aiming to make the RPS game maximally valuable and relevant for this community (see Chapman et al. 2017 on message tailoring), we had conducted a series of surveys and interviews to gauge the existing information and learning needs regarding climate tipping points. The results indicated that a high share of climate negotiators (61%), but also members of the NGO community (53%), was unfamiliar with the specific concept of climate tipping points (Milkoreit 2019). Expecting relatively high baseline levels of concern about climate change in general, but nevertheless low levels of knowledge and concern about climate tipping points among our game audience, our game design aimed to introduce players to the key characteristics of tipping points (esp. their temporal features and their expected impacts over time), to indicate links to climate governance (esp. to global temperature and mitigation) and to make the concept more relatable and relevant.

3.1 The tipping point negotiations: a role-play simulation game

We developed a novel RPS game consisting of two parts: (1) a role-play simulation of the international climate negotiations between 2018 and 2043 and (2) a climate fiction storytelling exercise for the year 2118 (see Supplementary Material I for more details). The second part builds on the results of the first and provides a very different interaction and learning context for participants—formal diplomacy role play vs. informal story development in a small group.

In part 1, participants role-played a simplified version of the United Nations Framework Convention on Climate Change (UNFCCC) negotiations with the added negotiation goal of preventing future climate tipping points. Players adopted the roles of diplomats, official delegates of UNFCCC member states, who were provided information about the country they represented, including a national budget and a negotiation mandate. Delegates ‘negotiated’ in face-to-face interactions before they made climate-related spending decisions for the country they represented, allocating national funds to four categories of action: mitigation, adaptation, negative-emission technology development and international climate finance. Their decisions were entered into a computer interface, and a simple integrated climate-economy model calculated their cumulative effects on the global climate, esp. temperature change and changes in the probability of passing climate tipping points, and the global economy over the coming 5-year time period. This information was presented to participants on a large screen after each negotiation round, providing immediate feedback to players regarding the response of the climate-economy system to their aggregate decisions, which players could use to inform their negotiation behaviour and action pledges in the next round.

Part 1 bears some similarities to the World Climate Simulation (Sterman et al. 2015), i.e. the combination of facilitated international negotiation role-play, entry of player decisions into a computer interface and information feedbacks provided by a scientifically rigorous (i.e. realistic) climate-economy model, but differs significantly in its implementation of role-play rules, the use of computer simulations and the important role of tipping points. Most importantly, it reflects the bottom-up pledge-and-review logic of the Paris Agreement (Falkner 2016), asking players who represent individual countries to submit action pledges in 5-year intervals. Each round of gameplay represented 5 years in real time and led towards the submission of nationally determined contributions (NDCs) by each player/country.

Five tipping points were included in the RPS game: coral reef dieback, the collapse of West Antarctic Ice Sheet (WAIS), the collapse of Arctic Summer Sea Ice (ASSI), boreal forest biome shift and changes in the Indian summer monsoon. At the beginning of the RPS game, the coral-reef tipping point has already been passed, two tipping points (WAIS and ASSI) have small (less than 10%) probabilities of occurring and the probability of triggering the two remaining tipping points is zero at the current global temperature level. The probabilities of passing these four tipping points increase throughout the game, depending on players’ decisions and the effects of mitigation spending on global temperature change over time. In each round, the computer model calculates the current probability of passing a specific tipping point and conducts a digital ‘dice roll’ to determine whether or not a tipping point is triggered. Given their trigger temperature ranges, WAIS and ASSI are often triggered during gameplay, but the boreal forest biome shift and the Indian summer monsoon are not. If a tipping point is triggered in a negotiation round (including the coral reef dieback), participants representing affected countries receive an offline ‘event card’ for this tipping point in each following round, detailing the impacts and corresponding costs of the ongoing tipping process on the country.

After five rounds, a workshop facilitator guides participants through a reflection on the outcomes of their negotiations, and the likely implications of the trends they created for the rest of the century. Then, the game moves to part 2, which consists of a climate fiction storytelling exercise that involves the imagination of possible futures (the year 2118) that could have been created by their collective decisions during their earlier negotiations. Participants step out of their roles as diplomats and are organized into small groups. Groups develop stories of characters experiencing the effects of tipping points in several locations around the world. They are provided a story preamble, introducing the location, character and context, as well as some scientific information about the location’s climate in 2118. The groups share their stories with all workshop participants and engage in a final reflection on the imagined world they created.

The game is preceded by an introduction to the science of climate tipping points and an explanation of the game rules. The workshop concludes with a debriefing session facilitated by a research team member.

Supplementary Materials I, II and III contain more details on game design and characteristics, countries and negotiation alliances and the expected effects of specific game features on participants’ risk perceptions.

3.2 Game workshops and participant recruitment

Following a well-established approach to studying game effects (van Pelt et al. 2015; Rooney-Varga et al. 2018), we conducted a number of serious game workshops and conducted pre- and post-game surveys with workshop participants to investigate changes in various dimensions of participants’ risk perceptions. We conducted seven game workshop sessions between May 2018 and December 2019, during which 84 study participants played the game (see Supplementary Material VI). Four workshops took place at the sidelines of UNFCCC negotiation sessions in Bonn (SB48) and Katowice (COP 24), three workshops took place during the 2018 and 2019 fall meetings of the American Geophysical Union (AGU) in Washington, D.C. and San Francisco and one was hosted by a university in the fall semester of 2018.

Recruitment efforts for these workshops varied depending on context, but usually involved personal invitation letters emailed to one or two representatives of an organization or country. For the game sessions during SB48, we focused on diplomats and a diverse set of observer organizations (UNFCCC 2017). For the workshops in Washington, D.C. and San Francisco, we approached non-governmental organizations registered as UNFCCC observers, Congressional staffers and scientists attending the AGU conferences. Recruitment among climate diplomats is generally challenging due to their geographic distribution, time constraints and confidentiality concerns, which provide significant constraints for data collection. Only a small share of workshop participants represents this group (8%). The largest group of participants comprised of scientists (37%), followed by NGO representatives (27%). This sectoral make-up of participants is important for the interpretation of our results. Our participants had 21 different nationalities; a majority of them came from the USA (46%) and the European Union (23%). Details regarding recruitment and additional demographic information are included in Supplementary Material VI.

Targeting a policy community rather than a lay audience presents significant methodological challenges. Given our significant recruitment challenges regarding climate negotiators, our study population (UNFCCC observer groups and scientists) only partly overlaps with the target audience of the game (state delegates, observers and scientists). We interpret our findings with this difference in mind.

3.3 Data collection and analysis

Game effects were investigated by means of pre- and post-game workshop surveys and observations during the debriefing of game workshops. In particular, we analysed to what extent our game was able to affect concern, perceived seriousness, perceived distance and perceived likelihood of tipping points. Our survey data was supplemented with game observations and notes from post-game debriefing sessions.

Participants could complete the pre- and post-workshop surveys either online ahead of/after the event or paper-based at the workshop location. Half of the workshop participants completed both surveys and could be included in our analysis (n = 42). Among these, scientists were in the large majority (57%), followed by NGO representatives (19%). Our study combined qualitative and quantitative analyses to maximize insights from this relatively small data set.

Survey questions (see Supplementary Material IV) included ten risk perception items regarding concern of climate tipping points (H1), perceived seriousness of risks associated with climate tipping points in general (H2), perceived geographical distance of tipping points (H3) (perceived risk to participants’ own vs. other countries (H3a), social distance (perceived risks of tipping points to seriously affect their personal life) (H3b), perceived temporal distance, i.e. the time scale on which participants expected the impacts of four tipping points (coral reef dieback, ASSI, WAIS and boreal forests) to unfold (H3c)) and perceived current likelihood of two tipping points: coral reef dieback and ASSI (H4). The latter two are arguably tipping points with a high likelihood in the present. Participants responded to concern, perceived seriousness and social and geographical distance items on 5-point Likert scales (not serious/concerned–extremely serious/concerned). Perceived temporal distance of expected impacts was measured using a 3-point scale (immediately or after a few years/after a few decades/after a century), and likelihood was indicated on 7-point Likert scales (exceptionally unlikely/ < 5%–virtually certain/ < 99%). The latter corresponded to the likelihood assessment scale of the Intergovernmental Panel on Climate Change (IPCC).

We analysed survey responses with paired sample t tests. We qualitatively analysed answers to the open question ‘What did you learn from the game workshop?’ in order to identify potential effects on risk perceptions.

We elicited comments and feedback from participants regarding their workshop experience and their risk perceptions during a debriefing session at the end of each event. The debriefing protocol contained a number of set questions, including closed questions (e.g. “Are you now more or less concerned about climate tipping points?”) and open questions (“What did you learn?”, “How did the game make you feel?”). Debriefings were semi-structured, i.e. they permitted a departure from the protocol to allow for the pursuit of participants’ comments and ideas, with the goal to establish a conversation. During the debriefing, multiple (at least two) team members took observational notes, which later provided the basis for a qualitative analysis. Due to the nature of this process, no clear statements can be made about the representative nature of the statements made during debriefing or their relative importance.

The qualitative analysis of responses to open survey questions and debriefing notes took place in four steps: (1) data organization in Excel; (2) development of a simple codebook containing for concern/worry, perceived distance, perceived seriousness and perceived likelihood (see Supplementary Material V); (3) coding of the full data set by two researchers (followed by discussion and resolution of all mismatches between the resulting codes to establish 100% alignment); and (4) synthesis of (a) multiple coded items from each event in the same coding category into general statements for each session and (b) statements across multiple sessions into general statements on game effects on risk perceptions regarding tipping points.

4 Quantitative analysis of game effects

4.1 Concern and perceived seriousness

While participants showed a slight increase in their level of concern about climate tipping points between pre- and post-workshop surveys (Mpost 4.62 > Mpre 4.46), this difference was not statistically significant on the α = 0.05 level (p = 0.135). Comparing game effects for non-scientists only revealed similar results (p = 0.167) (Supplementary Material VII). Therefore, the quantitative analysis does not support H1. Similarly, perceived seriousness of climate tipping points was scored slightly higher in post-workshop surveys compared to pre-workshop surveys (Mpost 4.55 > Mpre 4.50), but again, this difference was not statistically significant on the α = 0.05 level (p = 0.599). Comparing game effects for non-scientists only revealed similar results (p = 0.341) (Supplementary Material VII). We find no support for H2 based on our survey data.

One possible explanation for this null result is the occurrence of a ceiling effect. Ceiling effects can explain the absence of learning or other intervention in highly knowledgeable—or highly concerned—populations. If the surveyed population scores very high prior to the intervention, the results are positively skewed, and the intervention itself does not have a measurable effect (Koedel and Betts 2010). In other words, individuals who believe to be at the upper end of the available spectrum of knowledge or concern either do not experience an increase in knowledge and concern or they are not able to report such an increase because the Likert scales do not provide them higher scale values. Statistically, the chance of finding intervention effects is reduced under these circumstances (e.g. Judson 2012).

Our data indicates the presence of a ceiling effect (see Fig. 1): as opposed to our expectations, the majority of our study population was already highly concerned about climate tipping points before joining a game workshop. Prior to gameplay, 95% of participants perceived climate tipping points already as very serious (54% extremely serious and 41% rather serious), and 85% of participants were already very concerned (58% extremely concerned and 28% rather concerned). A self-selection bias in the recruitment process might have played a significant role, i.e. individuals who accepted our invitation tended to do so because of their knowledge and concern about tipping points.

Fig. 1
figure 1

The number of responses per answer category for two variables (perceived seriousness and concern) which indicate a ceiling effect: responses are on the upper end of the Likert scale, which may explain the absence of a statistically significant difference

Full size image

This has two implications. First, given the high level of baseline concern, the game experience was less likely to further increase concern among these players: based on their self-assessment, they were already as concerned as possible. Second, if players’ concern actually increased due to game experience, a detection of this change was impossible because they had to use the same Likert scale values in the post-game survey to express their level of concern. The ceiling effect is presumably not associated with the level of expertise, as no significant differences in concern, perceived seriousness and perceived distance were found between scientists and non-scientists (Supplementary Material VII). Also, no significant differences in risk perceptions were found between participants from developing vs. developed countries (Supplementary Material VII). Rather than the level of knowledge or nationalities, a more likely explanation for the ceiling effect therefore seems the self-selection bias (highly concerned individuals may have been more likely to participate in the game). A second explanatory factor could be the relatively small study population (42 completed pre- and post-game surveys), which also decreases the likelihood of statistically significant effects.

4.2 Perceived distance

We expected gameplay to result in reduced perceived distance of climate tipping points (H3) along three dimensions: geographical, social and temporal (see Table 1 for a summary of results). Geographical distance was measured by comparing two items: perceived risk of tipping points affecting participants’ own country and other countries. Our results generally confirm the main finding in the literature that people perceive climate impact risks as geographically distant, i.e. affecting people in other countries more than their own country. Among our participants, prior to the game, the risk of climate tipping points affecting other countries was perceived as significantly higher than risks to the participants’ own country (Mother 4.7 > Mown 3.95, p < 0.01).

Table 1 Summary of the quantitative effects of the game on four dimensions of psychological distance of climate tipping points
Full size table

After gameplay, we expected an increase in perceived risks for participants’ own country and stable perceptions regarding risks to other countries. Our analysis did not detect any statistically significant changes to the pre-game perceptions of geographic distance (see Table 1). The absence of a significant game effect may be explained by the limited diversity in nationalities of participants, or that during the negotiation, role-play participants represented countries other than their own, disabling transferring their game-based insights to their own country.

Second, we expected that participants’ perceived risk of tipping points affecting them personally to increase after gameplay, indicating a decrease in perceived social distance. As shown in Table 1, the slight increase in perceived personal risk in absolute terms (Mpost 3.61 < Mpre 3.51) was not statistically significant on the α = 0.05 level. Our results therefore do not support H3b. One explanation for the lack of an observable game effect might be the disconnection between the roles participants played during the game—representatives of countries about which they generally knew very little—and their personal lives.

Third, we expected participants to perceive the impacts of tipping points in the game to be temporally closer after gameplay. Our findings of four tipping points were mixed.

In line with expectations, the coral reef tipping point was indeed perceived as temporally closer after gameplay (Mpost 1.15 < Mpre 1.28); however, this difference was not statistically significant. No significant differences were found for the ASSI and WAIS tipping points, the former indicating no change at all and the latter a non-significant increase in perceived temporal distance (see Table 1). A potential explanation is that the simulation game only covered a 25-year time span, therefore lacking information on impacts that may unfold over centuries (which was the highest end of the Likert scale).

We found a significant effect of gameplay on the perceived temporal distance of only one tipping point: the biome shift of the boreal forests (p < 0.01). Opposite to our expectations, this tipping point was perceived as slightly more distant after playing the game (Mpre (1.55) < Mpost (1.87)). A reason for this effect can be explained by the game’s simulated time period (2018–2043): global temperature increase was very unlikely to reach a point where the probability of passing this tipping point would rise above zero. Consequently, the tipping point was not triggered during any of the game workshops, which likely decreased participants’ perceptions of its temporal distance. When comparing pre- and post-game surveys of non-scientists only, we observed a significant decrease in perceived temporal distance of the coral reef tipping point after gameplay (Mpost (1.00) < Mpre (1.36)) (Supplementary Material VII).

4.3 Perceived likelihood

Finally, we expected participants to perceive the current likelihood of triggering the tipping points included in the game to be higher after the game. We only measured perceptions regarding coral reef dieback and ASSI. As shown in Table 1, the perceived likelihood of triggering the coral reef tipping point was indeed higher in post-workshop surveys compared to pre-workshop surveys (Mpost 6.41 > Mpre 5.50) and this difference was significant on the α = 0.05 as well as α = 0.01 level. The percentage of participants that perceived the likelihood of passing the coral reef tipping point as virtually certain increased from 13% prior the game to 53% after gameplay. This increase is likely linked to a combination of two game features: (1) in a pre-game presentation, the facilitators presented information indicating that scientists believe that this tipping point has already been passed and (2) the game treats the passing of the coral reef tipping point as a reality from the very beginning. In other words, participants could not influence its passing or prevention, and they were repeatedly faced with the continuing, and increasingly costly consequences of this tipping point for the country they represented.

For the ASSI tipping point, however, perceived likelihood was lower after gameplay, as indicated by a statistically significant decrease between pre- and post-game surveys (Mpost 4.91 < Mpre 5.47, p = 0.01). While this effect is surprising and was not intended by our game, it could be explained by the results of actual gameplay. Despite the increasing probability of the ASSI tipping point over the course of the simulation in all game sessions, this tipping point was only triggered in three of the eight workshops. Participants might have been more familiar with—and possibly more concerned about—the ASSI based on ample reporting on this tipping point in climate research and media in the past.

5 Qualitative analysis of game effects

Our qualitative analysis identified effects in all four dimensions of risk perceptions. Table 2 provides a summary of the analysis, including select quotes of participants’ comments and debriefing notes by observers. Some of these findings confirm the results of our quantitative analysis (re H1, H2 and H4), but others contradict and challenge these results, especially regarding the perceived distance of climate tipping points.

Table 2 Summary of the results of the qualitative analysis, thematically clustering participants’ comments during the post-game debriefing and the corresponding observer notes based on the four dimensions of risk (concern, seriousness, distance, likelihood)
Full size table

5.1 Perceived seriousness and concern

Participants’ survey responses and comments during the debriefing indicated that most of them were more concerned about climate tipping points after the game than they had been before. Most speakers in the debriefing even used the phrase “much more concerned”. In two different sessions, one participant indicated that they had been very concerned before and that their level of concern had remained unchanged. These observations support our argument that our quantitative analysis might have revealed ceiling effects (see Sect. 4).

Second, in contrast to the quantitative results, participants’ comments indicated that the game, in particular the storytelling component, increased their perceptions of seriousness, especially regarding the severity of tipping-related threats to humans and societies, including participants’ families. The following statements illustrate this effect: “I learned how traumatic climate change will be for coastal dwellers” and “I started feeling a little bit of panic for them [my children]”. The mention of potential human suffering suggests that the game was able to generate understanding for the potential human significance of tipping points and might have generated or increased a sense of urgency: “We truly only have just one shot to get it right”. Again, the survey design might have been unable to pick up these changes due to the fixed Likert scale, ceiling effects among the study population (largely representing scientists and NGO representatives with high levels of concern prior to gameplay), and the limited statistical power due to a small number of observations.

5.2 Perceived distance

Possibly the most interesting game effects concern perceived psychological distance in all four dimensions. Post-game comments and observations indicate that several participants perceived tipping points to be closer—geographically, socially and temporally—after the game, aligning their perceptions better with the state of scientific knowledge. The game triggered reflections on potential tipping-point effects on their home country and hometown rather than other countries (i.e. decreased geographical distance), and on their families and their own lives (decreased social distance). For example, one participant stated “Knowing what implications it has, made me think about how they will impact me, my family, my town. How substantial it is”. One of the main mechanisms through which the game, especially the storytelling component, countered the psychological process of distancing was rendering abstract scientific ideas tangible and concrete. This could have happened, for example, in specific moments when the whole group paid attention to the passing of a tipping point or the information on event cards, but, in particular, during the effort of imagining the experience of individuals living in the year 2118 in the storytelling exercise. Illustrative statements included “The concrete narratives force you to make it real” and “The real-life story helps it feel personal”. These game features guided players’ focus towards the human rather than environmental impacts created by tipping points, and they rendered abstract knowledge more tangible, even personal.

These comments regarding participants’ perceptions of the distance of climate tipping points contradict our quantitative analysis, highlighting the methodological challenges associated with studying serious game effects. We suspect that this contradiction in our findings is due—at least to some extent—to the fact that the group of participants that completed two surveys and the group that actively participated in the post-game debriefing did not fully overlap.

5.3 Perceived likelihood

Regarding perceived likelihood, a number of participants indicated that based on the game, they had a better understanding of the fact that the likelihood of passing different tipping points is tied to temperature change, and that different tipping points have different probabilities over time. For example: “I learned more specifics about the likely timing/probability of crossing various tipping points”. Importantly, many participants recognized that some tipping points may already have been passed, especially the coral reef dieback tipping point. These game effects confirm our quantitative findings. They could have been caused by the pre-game presentation on the science of climate tipping points and by the way different tipping points were integrated into the game, especially the fact that different tipping points affected the game experience in different ways (e.g. one tipping point had been triggered before game start, two tipping points were never triggered during gameplay).

6 Discussion

The insights created by the two distinct analyses above raise a number of important questions regarding methodological approaches to studying effects of RPS games on risk perceptions and point to challenges for future research.

First, our findings suggest that our study population—mainly scientists and NGO representatives engaged in the international climate negotiations—tend to be very concerned about climate tipping points and might have been even more so after participating in one of our game workshops. Unlike Bellamy and Hulme (2011), we did not compare concern about tipping points with climate change more generally. But given their findings—tipping-related concern was lower among their participants than concern about climate change in general—there appears to be a difference in levels of concern regarding tipping points between the general public and those professionally engaged in addressing climate change. This could be studied further with due attention to the problem of self-selection bias during participant recruitment.

Further, we started this investigation with the suspicion that the lack of engagement with tipping points in the international climate negotiations so far might be the result of a lack of concern, possibly linked to a poor understanding of the science of tipping points. Hence, our game was designed for a particular target audience, with climate change negotiators in the UNFCCC as the most important subgroup. Due to recruitment and data collection barriers with this subgroup, we ended up collecting data primarily from scientists and NGO representatives, who tended to be highly concerned to begin with. This circumstance has two important implications for the interpretation of our results. First, given that we found some game effects despite high levels of concern prior to gameplay among our study participants, the game might be even more effective among less well-informed climate negotiators. Second, risk perceptions might not offer a strong explanation for the lack of political attention to the issue, at least not for certain actor groups such as NGOs. Future research could seek to confirm this finding for national and international policy makers, and investigate a wider range of relevant perceptions, such as efficacy (the perceived individual or collective ability to influence climate change) (Feldman and Hart 2016; Milfont 2012).

Our findings related to concern and perceived seriousness point to a couple of important biases at work in our research. First, participants who were already highly concerned about climate tipping points may have been more likely to respond to invitations to participate in our study (self-selection bias) than those less concerned. Future research could attempt to counter this effect with different recruitment techniques, i.e. avoiding an emphasis on tipping points in recruitment efforts, and by targeting other study populations, i.e. those not professionally engaged in climate change. More generally, these findings point to the importance of audience-tailoring in game design and research. Future research could explore the effects of the Tipping Point Negotiations game on risk perceptions and knowledge of tipping points among more diverse audiences.

Self-selection bias also heavily influenced the comments we solicited from participants in our debriefing sessions. Not all workshop participants spoke up equally often, and some might not have provided any comments at all. The notes that form the majority of the qualitative analysis of game effects reflect only the thoughts expressed by active debriefing participants and might present an incomplete picture of game effects among the larger group of participants.

Second, although our study population covered 21 nationalities, these predominantly represented the Global North; only 14% of participants were from the Global South. Previous research revealed large differences between countries in risk perceptions depending on their local climate characteristics (Lee et al. 2015) as well as significant differences in their national interests and corresponding climate negotiation positions (Blaxekjær and Nielsen 2015). Negotiation participants from the Global South may perceive risks differently based on their national context. Future studies could further investigate differences in game effects across different nationalities or geographies.

Third, for some participants, the context of the ongoing climate negotiation sessions, and the presence of other workshop participants when completing surveys may have influenced survey responses, e.g. leading to reports of heightened concern (social desirability). Future studies could avoid this influence of context with different choices for workshop timing and locations as well as avoiding on-site surveys.

Fourth, our findings confirm Bellamy and Hulme’s (2011) arguments regarding the important role played by psychological distancing mechanisms in risk perceptions related to climate tipping points. Our results suggest that simulation games, in particular RPS games, can have effects on psychological distance, such as making climate risks more tangible, as suggested in previous research (Meya and Eisenack 2018). However, our results also revealed the challenges of intentional and effective game design, pointing to the need for careful attention to the link between psychological mechanism and game design choices. For example, we did not consider the possibility that not triggering a tipping point (e.g. boreal forest dieback) during the game could increase perceived temporal distance.

More generally, our study highlights the methodological challenges associated with studying the effects and effectiveness of RPS games as science-policy engagement tools in public policy and global governance (Mayer et al. 2014). On top of the challenges related to game design, e.g. devising game-based learning and engagement processes that have intended effects, our project grappled with standard recruitment problems when targeting an elite population—participants in global climate change negotiations. Limited recruitment success coupled with inconsistent data, e.g. participants who completed only one of two surveys or provided incomplete survey responses—resulted in significant data limitations. The small data set undermined the reliability of our quantitative analysis and limited the kinds of analysis we were able to complete. For example, the small number of observations did not allow us to explore demographic differences in risk perceptions. More generally, relying more heavily on qualitative analyses and using more than one analytic approach might be important guidance for future research.

Finally, no control group was included in this analysis and therefore no cause-effect claims can be made regarding the influence of the game on risk perceptions. Future work could include a group that only completes the pre-game survey and participates in a standard exercise of science communication, e.g. a panel discussion of the findings of an IPCC report.

Despite these often unavoidable problems and limitations, our findings strongly suggest that serious games, in particular RPS games, have large potential as an engagement tool at the science-policy interface. Although RPS games in themselves may be powerful in creating a ‘safe space’ to experiment with and learn about the effects of collective decision-making (Rumore et al. 2016), our research reveals that to bring the impacts of collective decisions ‘home’ to participants, it is important to immerse them in the lived experience of people facing such impacts in the future. Multiple strategies for such imaginative future immersion exist, including the use of visualizations and virtual and/or augmented reality experiences (Vervoort 2019). In our case, we deployed a storytelling exercise, in which small groups of participants explored this human dimension starting with a simple story preamble. The exercise generated emotions and reflections on the impacts of climate change on participants’ hometowns and families. This is in line with recent work on storytelling, revealing that personal stories can influence risk perceptions through eliciting affective responses, most notably due to relatable characters and salient impacts (Gustafson et al. 2020). Given that the role of emotions remains debated in risk perception and climate communication literature, future research should further investigate whether and how games could emotionally engage participants to influence risk perceptions. RPS games thus offer powerful science-policy engagement tools as they allow for a combination of multiple modes of engagement—role-playing, decision modelling, visualization and storytelling—which may influence understanding, beliefs and perceptions in different ways. This is arguably an important capacity given the long-term and intangible character of climate change, which complicates science communication. This unique capacity to ‘experientially’ engage people in possible futures (Candy and Dunagan 2017) could be further explored in future research. A key point of focus should be the careful and reflexive integration of gameplaying processes in processes of governance (Vervoort and Gupta 2018).

7 Conclusion

In this study, we examined the effects of a RPS game with a focus on climate tipping points on players’ risk perceptions associated with climate tipping points. The RPS game immersed participants—largely scientists and representatives of NGOs engaged in international climate negotiations—in possible futures by simulating impacts on collective decision-making to avoid tipping points and through the imagination of future realities their collective decisions could create. We investigated multiple aspects of risk perceptions: concern, perceived seriousness, perceived distance and perceived likelihood.

Our results indicate that gameplay increased concern and perceived seriousness among participants, although our quantitative analysis was unable to capture this effect due to ceiling effects (high levels of concern prior to gameplay), most likely caused by a self-selection bias as well as a small study population as a result of recruitment challenges Our qualitative analysis provided evidence that gameplay increased a sense of urgency among many participants triggering reflections on the severity of the diverse threats related to climate tipping points. Observing these effects among a study population with relatively high levels of initial concern implies that our game might be even more effective among a less knowledgeable and less concerned population. In other words, we consider ceiling effects and self-selection bias as (research design-related) factors that potentially reduced the game’s effectiveness. Furthermore, our findings indicate that the game reduced psychological distance of climate tipping points. Reconciling contradicting results of our two analytic approaches in favour of the qualitative findings, we argue that the simulation game, especially its storytelling component, affected how participants perceived the geographical, social and temporal distance of tipping points. The engagement in a storytelling exercise, imagining the lives of specific characters in specific locations in the year 2118, rendered the abstract scientific notion of tipping points more tangible and concrete by illuminating the human dimension of climate tipping points, i.e. the personal, physical and emotional experience of tipping point impacts.

Our analysis also revealed the significant methodological challenges involved in studying the effects of serious games to decision-making and policy support. Depending on the target population—in our case participants in the international climate change negotiations—recruitment and data gathering are time consuming and often have limited success. Designing studies that are flexible, use multiple analytic approaches and can handle small data sets might be important for future work in this area.

Our findings advance the current understanding of the potential power of serious gaming as a science-policy engagement tool. Games could offer a fruitful alternative to the currently dominant ‘cognitive’ model focusing on information presentation in assessment reports of the IPCC. Regarding the urgency of accelerating the political response to climate change, we therefore encourage the use of alternative, more experiential science communication and engagement tools at the climate science-policy interface.