1 Introduction

Climate change and measures taken against it are complex and their effects are often associated with a considerable time delay (IPCC, 2023). These complex and delayed processes cannot be evaluated based on personal experience alone - scientific evidence needs to be considered (Avagyan, 2021; Swim et al., 2009). To help policymakers and the public estimate the development of climate change as well as the effectiveness of specific measures taken against it, scholars developed tools that simulate climate change based on scientific evidence (Rooney-Varga et al., 2018, 2020; Hensel et al., 2023). The simulations provide graphical output that directly changes as the user shifts levers symbolizing different means of climate change mitigation. This leads to a hands-on experience in searching for means with a strong impact and allows to test and potentially correct one’s knowledge. Surveys reveal an increasing desire for such interactive solutions that are ideally accessible on the Internet at any time (Pearce at al., 2015; Taddicken, 2013). However, the effectiveness of such web-based climate change simulations is not sufficiently clear. So far, their effectiveness has only been tested in combination with face-to-face role-play. To our knowledge, no study so far tested the isolated effect of the simulations. For large scale interventions, it would be highly relevant to know whether the simulations alone affect the evaluation of climate protection measures. The present research set out to investigate whether using the climate change simulation tool En-roads improves laypersons’ understanding of climate change and influences their evaluations of measures taken against it.

En-roads is a climate simulation program that simulates the influences of different parameters on global climate change (Climate Interactive & MIT Sloan Sustainability Initiative, 2019). Based on historical climate data, climate change can be simulated up to the year 2100. Users can change almost 30 parameters in the sectors energy, economy and the public sector (including traffic, buildings, etc.) and get feedback on the effects of these parameters on greenhouse gas emissions and the global climate. Users can thus explore the influence of different political strategies (e.g. electrification, CO2 prices) on energy prices, sea levels, and temperature. All underlying calculations and models are publicly accessible, which makes En-roads a transparent program.

Simulations have been shown to increase understanding of complex systems (see Plass & Schwartz, 2014, for a review). However, evidence for the effectiveness of simulations in the context of climate change is still scarce. Rooney-Varga and colleagues (2018) investigated the effects of interactive role-playing games in small groups in combination with the use of the simulation software C-roads. Dealing with the simulation program in this setting led to an increase in knowledge, to higher motivation to learn more about climate change and actively do something about it. Role-playing games in small groups combined with the simulation program En-roads revealed similar effects (Rooney-Varga et al., 2020). In addition, participants in these climate simulation games report that the game increased their climate change risk perception and their experiences of collective efficacy (Hensel et al., 2023). These studies show that climate change simulations in combination with role-play can be effective. However, from the extant evidence it is not clear whether the effects of the interventions are driven by participating in the simulation or by enacting the face-to-face role-play (or the combination of both). Furthermore, interventions that involve face-to-face role-play are highly time consuming. They cannot be provided in a flexible way where and when they are needed in educational settings. Also, after being introduced to this variant of employing simulation, people cannot flexibly revisit the simulation when prompted by questions in daily life. For instance, people cannot get back to the source of information of the role-play whenever they want to check and explore the validity of claims on the effectiveness of climate measures that are brought into play by politicians via the media. Thus, the demand for exclusively web-based solutions is high (Pearce at al., 2015; Taddicken, 2013). To address this issue, we investigated whether using En-roads without face-to-face role-play reveals similar learning effects on climate change knowledge and behavioral intentions.

In addition to these learning processes, climate change simulations may influence our evaluations of measures taken against it. Research on environmental politics has documented that climate change mitigation can gain high salience before elections (cf. Carter & Pearson, 2020). If citizens can easily distinguish between effective and symbolic mitigation measures, this might influence votes, which can yield substantive impact in climate politics.

1.1 The present research

We tested whether using En-roads online influences knowledge of climate change and motivation to act against it. As reactions to climate change as well as motivations for climate protection are closely associated with emotions (see Brosch, 2021; Landmann, 2020, for reviews), we investigated whether the climate simulation changes emotional reactions towards climate change. Furthermore, we investigated the estimated effectiveness of specific measures taken against it and the perception of one’s own competence to evaluate political measures and participate in political processes. We explored whether different instructions for using En-roads have different effects. We report all manipulations, all data exclusions and measures used in this study. Data and syntaxes are available at https://osf.io/cwks5/. Data were analyzed using R, version 2022.12.0 (RStudio Team, 2020) and the packages ggplot2, version 3.4.1 (Wickham, 2016), GGally, version 2.1.2 (Schloerke et al., 2022) and SPSS (IBM Corp., 2021).

In the current online study, we randomly assigned participants to one of four conditions. There were three groups that all involved En-roads usage, but differed in the amount of structure and instruction provided. Given the lack of prior studies with lay persons using En-roads, we included this variation to explore what level of guidance might be necessary and beneficial. On the one hand, from a constructivist perspective it might be good to have users find out themselves how the app works. It is conceivable that most users use the simulation intuitively. Trying to operate the app and finding a path to the 1.5°goal, users construe knowledge by exploring and discovering what can be done with the app and how (cf. Kirschner et al., 2006).

On the other hand, past work on web-based simulations in teaching statistics has shown that many persons attentively study the current state of the simulation while neglecting to actually manipulate it – unless explicitly guided (Zhao et al., 2023). Furthermore, complex tasks can lead to over-taxing working memory, hence providing guidance can help secure that working memory resources are used for the most relevant content as aspects of how to work on the app are outsourced rather than demanding working memory capacity. Therefore, Kirschner et al. (2006) suggest guiding users when simulations are complex. Yet, simple step-by-step instructions (similar to cooking recipes) can reduce cognitive engagement too much, suggesting that a point of good balance between structure and options for free exploration has to be targeted empirically (cf. Rutten et al., 2012).

2 Method

A total of (N = 271) students of the University of Hagen were recruited via the psychology department’s virtual laboratory, gave their informed consent and participated online in exchange for course credit. The University of Hagen is a distance learning university characterized by high diversity with respect to students’ age, political attitudes, family status, and occupation (about 80% of the students are working or self-employed during their studies; Stürmer et al., 2018). A total of 17 participants indicated that they had not participated seriously and 14 participants did not agree to the use of their data after finishing the study. The remaining 249 participants (60 male, 181 female, 2 diverse, 6 persons did not indicate their gender) were between 19 and 63 years old (Mage = 31.8, SD = 9.7).

Participants were randomly assigned to one of four conditions: (1) freely exploring En-roads (exploration), (2) exploring En-roads with additional information about setting options (informed exploration), (3) using En-roads with advice for each step to reach the 1.5° goal (step-by-step guide), and (4) receiving En-roads screen shots with the 1.5° solution without interacting with the tool (control condition).

Participants were instructed that En-roads is a simulation that can help explore the effectiveness of different measures to mitigate climate change. In the experimental conditions, their task was to use eight minutes to find a set of measures that would, according to En-roads, limit global temperature increase to 1.5° by 2100. After setting a timer, they were provided with the link to the German language version of En-roads showing the default settings at the beginning. Participants in the control condition were provided with two screen shots from En-roads: the default setting and a screenshot fulfilling the 1.5° goal. They also had eight minutes to study how the 1.5° goal can be reached with these settings. Note that we deleted hints to the source of the screenshots so that participants would not search the Internet to start the simulation when being presented with the screen shots.

After working on En-roads (or the screen shots), participants evaluated the efficacy of specific factors to reduce climate change, their personal (general and task-specific) political efficacy, and they indicated their emotional reactions to climate change. The scales were presented in the following order.

Efficacy of specific factors to reduce climate change was assessed in two ways: Nine factors (i.e., subsidy for renewable energy, energy-efficient housing and industry, CO2pricing, growth of humanity and economy, taxing fossil fuels, energy efficient transport, deforestation, afforestation) were evaluated with regard to their efficacy to reduce climate change on a scale ranging from 1 (no influence) to 4 (strong influence). In addition, participants selected one of six factors they predicted would have the strongest influence till the end of this century (see Fig. 2).

Political efficacy was assessed with the Political Efficacy Short Scale (PEKS, 4 items, e.g., I am well placed to assess important political issues and decisions regarding measures to combat climate change, Beierlein et al., 2012, α = 0.51). Participants indicated to which extent they agree with the statements on scales ranging from 1 (not at all) to 5 (completely).

Task efficacy was assessed with three items targeted at the simulation tool (e.g., With the help of publicly available tools such as En-roads, I can distinguish effective from non-effective climate policy decisions, 3 items, α = 0.90). Participants indicated to what extent they agree with the statements on scales ranging from 1 (not at all) to 5 (completely).

The evaluation of political proposals was assessed for nine proposals that consisted of excerpts from the final declaration of the UN Climate Change Conference in Glasgow 2021 and the coalition agreement 2021 of the coalition currently in power in Germany. Each proposal was evaluated with four items (i.e., If the adopted measure is consistently pursued, this will make a significant contribution to stopping climate change; Even if the adopted measure is consistently pursued, it has no influence on global warming (reverse coded); The adopted measure is clearly formulated and provides a clear plan; The measure is not formulated specifically enough to assess whether the measure could make a difference (reverse coded)). As the results did not differ between the specific proposals, we collapsed the evaluation of all political proposals to one scale (36 items, α = 0.89).

Emotional concern about climate change were assessed with 12 items adapted from the Positive and Negative Affect Schedule (PANAS-X; Watson & Clark, 1994; German translation by Röcke & Grühn, 2003; e.g., afraid, angry, sad, α = 0.91). Participants indicated to what extent they experience the respective negative emotion when thinking about climate change on scales ranging from 1 (not at all) to 5 (very much).

Socio-political control was assessed with a subscale of the Spheres of Control Scale (SOC-3) adapted for perceived influence on climate policy (10 items, e.g., By taking an active role as citizens in climate policy and social affairs, we can influence events around the world, Paulhurst & van Selst, 1990, α = 0.76). Participants indicated how they evaluate the statements on scales ranging from 1 (completely wrong) to 7 (completely correct).

Beliefs about climate change impacts were assessed with six items from Rooney-Varga et al. (2018). Participants evaluated how likely a set of impacts are for the next decades (i.e., Increased temperatures globally, Increased incidence and intensity of heat waves, Increased rates of extinction of plant and animal species, Increased global sea level, Increased intensity of storms across many regions, An overall decrease in clean, potable water globally, α = 0.93) on scales ranging from 1 (very likely) to 4 (very unlikely). The scales were recoded such that higher values indicate stronger agreement.

Beliefs about climate change mitigation were assessed with eight items adapted from Rooney-Varga et al. (2020). Factor analysis with varimax rotation revealed three factors with eigenvalue > 1, which explained 67.78% of variance. These factors represented urgency for action (4 items: The cost of cutting emissions is so high that it makes more sense to wait and see how bad climate impacts are before taking action (reverse coded); We could effectively address climate change if there were strong political and social will to do so; At the current level of greenhouse gas emissions, carbon dioxide levels will continue to rise as emission rates are higher than the rate at which carbon dioxide is removed from the atmosphere; Changes in policy and energy supply are needed to effectively tackle climate change; α = 0.79), belief in technical solutions (2 items: New technologies will be developed that will solve the problems of climate change; Using current technologies, it is physically possible to dramatically reduce emissions; α = 0.35) and benefits of climate protection (2 items: Successfully combating climate change will have positive effects on public health; Successfully dealing with climate change will have a positive impact on social justice; α = 0.67). Participants indicated to what extent they agree with the statements on scales ranging from 1 (completely) to 5 (not at all). The scales were recoded such that higher values indicate stronger agreement.

Motivation for pro-environmental behavior was assessed with four items from Rooney-Varga et al. (2018). Participants indicated how likely it is that they engage in specific behaviors (i.e., Take action to reduce your personal carbon footprint; Discuss climate change with your family and friends; Discuss climate change with your peers; Take some form of political action in support of climate change policy, α = 0.77) on scales ranging from 1 (very likely) to 4 (very unlikely). The scale was recoded such that higher values indicate higher motivation.

Usability was assessed with the System Usability Scale (SUS, 10 items, e.g., I find En-roads easy to use, Brooke, 1996; German translation by Gao et al., 2020, α = 0.90). Participants indicated to what extent they agree with these statements on scales ranging from 1 (not at all) to 5 (completely).

3 Results

3.1 Non-parametric tests

To test whether using En-roads affected how influential participants perceived different factors to be, we used non-parametric tests to account for scale level and non-normal distribution. We used Kruskal-Wallis-Tests to investigate whether participants’ evaluation of the effectiveness of specific factors on the reduction of climate temperature differ between the conditions. Sensitivity power analysis for testing differences between groups with non-parametric tests (two-tailed, Laplace distribution, α = .05, 1-β = .80, group size = 58) with gpower (Faul et al., 2009) revealed that these analyses were able to detect medium effects of d = .43 or larger. Results are shown in Fig. 1. Participants who had used En-roads evaluated taxation of fossil fuels, H(3) = 8.86, \(p\) = .031, subsidies for renewable energy, H(3) = 10.22, \(p\) = .017, and energy efficacy (transport and traffic: H(3) = 12.22, \(p\) = .007; building and industry: H(3) = 9.60, \(p\) = .022) as less effective compared to the control condition. Reversely, breakthrough of new zero carbon, H(3) = 11.97, \(p\) = .007, CO2 pricing, H(3) = 7.94, \(p\) = .047, as well as economic and population growth, H(3) = 7.86, \(p\) = .040, were rated as more effective by participants who had used En-roads compared to the control group. The efficacy estimates for deforestation, H(3) = 3.45, \(p\) = .328, and afforestation, H(3) = 5.99, \(p\) = .112, did not differ significantly between the conditions (see Fig. 1).

Fig. 1
figure 1

Estimated efficacy of specific factors to reduce climate change

Note. Efficacy was rated on scales ranging from 1 (no influence) to 4 (strong influence). Numbers indicate frequencies of the selected responses; the size of the circles indicates the proportion of the selected response within the respective condition

Full size image

Participants’ selection of the factor that they think will have the greatest influence on the reduction of climate temperature with regard to the end of this century resulted in a nominal scale. Correspondingly, we used a Chi² Test to investigate whether participants’ responses to that question differed between conditions. Sensitivity power analysis (α = .05, 1 – β = .80, df = 15) with gpower (Faul et al., 2009) revealed that the study was able to detect effects with a critical x2 of 25.00. As shown in Fig. 2, CO2 removal was mentioned more frequently and mobility (transport and traffic) less frequently in the experimental conditions compared to the control condition, x2(15) = 25.9, p = .040.

Fig. 2
figure 2

Selection of the most influential factor (y-axis) dependent on the experimental condition (x-axis)

Note. Numbers indicate frequencies of the selected factors; the size of the circles indicates the proportion of the selection within the respective condition

Full size image

3.1.1 Analyses of variance

All other measures were assessed with multiple items which created reliable scales. Accordingly, we used analyses of variance to test whether participants’ responses to these scales differed between the conditions. We conducted ANOVAs with condition as predictor and political efficacy, task efficacy, evaluations of political proposals, usability, beliefs about climate impact and mitigation, emotional concern and motivation as dependent variables. Sensitivity power analysis (α = .05, 1 – β = .80, N = 245, 4 groups) with gpower (Faul et al., 2009) revealed that the study was able to detect medium effect sizes of f = .21 or larger. Results are shown in Figs. 3 and 4. Correlations between the scales are reported in the Supplemental Material (Figure S1 and Figure S2).

Task efficacy differed significantly between the conditions, F(3, 245) = 20.25, p < .001, ηp2 = .20. Participants who used En-roads more strongly agreed that publicly available tools such as En-roads help them evaluate climate policies compared to participants in the control condition. This effect was particularly strong for participants who engaged in informed exploration (see Fig. 3B). To investigate the stability of this effect, we repeated the analysis while controlling for perceived socio-political control. The effect of condition on task efficacy was stable when controlling for socio-political control, F(3, 244) = 20.54, p < .001, ηp2 = .20.

Political efficacy, F(3, 244) = 1.26, p = .288, ηp2 = .02, the evaluation of political proposals, F(3, 245) = 0.70, p = .555, ηp2 < .01, usability, F(2, 182) = 1.43, p = .242, ηp2 = .02, impact beliefs, F(3, 243) = 0.01, p = .998, ηp2 < .01, urgency beliefs, F(3, 245) = 0.10, p = .961, ηp2 < .01, technology beliefs, F(3, 245) = 0.40, p = .756, ηp2 < .01, benefit beliefs, F(3, 245) = 0.65, p = .584, ηp2 < .01, emotional concern, F(3, 245) = 1.35, p = .261, ηp2 = .02, and motivation for pro-environmental behavior, F(3, 244) = 1.23, p = .301, ηp2 = .02, did not differ significantly between the conditions. Participants evaluated their own competence to participate in political processes similarly regardless of whether they had participated in an En-roads task (Fig. 3A). They also evaluated the described political proposals as similarly effective regardless of whether they had used En-roads (Fig. 3C). Usability ratings for the app were high in all three En-roads conditions and the type of the En-roads task did not affect evaluations of usability (Fig. 3D). Furthermore, participants’ beliefs about climate impact (Fig. 4A), the urgency to act (Fig. 4B), their beliefs in technological solutions (Fig. 4C) and benefits of climate protection measures (Fig. 4D) as well as their emotional concern about climate change (Fig. 4E) and their motivation for pro-environmental behavior (Fig. 4F) did not differ significantly between the conditions.

Fig. 3
figure 3

Political efficacy, task efficacy, the evaluation of political proposals and usability dependent on condition

Note. The box-plots show the minimum, lower quartile, median, upper quartile and maximum of the data points, the dashed line shows the overall mean for each dependent variable, asterisk indicate significant differences from the mean of the En-roads conditions with *** p < .001, **** p < .0001

Full size image
Fig. 4
figure 4

Beliefs, emotional concern and motivation dependent on condition

Note. The box-plots show the minimum, lower quartile, median, upper quartile and maximum of the data points, the dashed line shows the overall mean for each dependent variable, asterisk indicate significant differences from the mean of the En-roads conditions with *** p < .001, **** p < .0001

Full size image

4 Discussion

The present research shows that using the web-based climate change simulation program En-roads affects how lay persons perceive efficacy of measures to reduce climate change and their own ability to evaluate these measures. Previous research had already shown that using En-roads in combination with face-to-face role-play can increase knowledge, motivate learning about climate change and actively doing something about it (Rooney-Varga et al., 2018, 2020). This left open, whether either part of the intervention is sufficient to obtain an effect or whether the combination would be needed. It was not clear whether a pure online variant of using such a simulation could prove effective. The present research sharpens and extends the above findings by revealing the effects of the exclusively web-based use of En-roads without face-to-face role-play.

4.1 Climate change simulations affect evaluations of climate protection measures

Climate simulations like En-roads demonstrate that single measures reveal only small effects on global temperature. This may explain why taxation of fossil fuels, subsidies for renewable energy, and energy efficacy (transport and traffic, building and industry) were evaluated as less effective by participants who were using En-roads compared to the control group in the present study. An important learning outcome of using En-roads may thus be that more general measures are needed. In line with this, participants who had used En-roads evaluated breakthrough of new zero carbon, CO2 pricing, and economic and population growth, as more effective compared to the control group. Hence, the selective effects of using En-roads on evaluations of climate change impacts may be attributed to the generality of the proposed measures – favoring more general measures. These findings are consistent with previous research showing that users construe knowledge by exploring and discovering apps (cf. Kirschner et al., 2006).

Furthermore, participants who had used En-roads, more frequently selected CO2 removal as the most influential factor for the reduction of climate temperature compared to the control group. Experiencing that the simulated measures have only very small effects on climate temperature may be frustrating and elicit the impression that measures different from that - like CO2 removal - are more effective. However, current technology is able to remove only small amounts of CO2 and this is highly energy consuming (German Environment Agency, 2021a). It is therefore not an influential factor on climate temperature so far. While climate simulations may increase the adequacy in evaluating the factors covered by the simulation, they may - as a negative side effect - amplify false beliefs about factors that are not covered.

The changes in evaluations of climate change impacts has a considerable potential to shape attitudes towards climate change policies. Some people like to believe that uncomfortable measures such as saving energy or reducing meat consumption are not necessary and that they can be replaced with other measures. Climate simulations provide the experience that single measures are not sufficient and may thus reduce the avoidance of uncomfortable climate change policies. In the present study, however, evaluations of political proposals were not affected by the simulation. Firm attitudes and behaviors are difficult to change by a single intervention (cf. Landmann, 2020; Nisa et al., 2019). Participants may have already formed firm attitudes about the political proposals that are resistant to influences of a single simulation.

Climate simulations in combination with role-playing can increase knowledge about climate change and pro-environmental behavioral intentions (Rooney-Varga et al., 2018, 2020). In the present study, however, using En-roads did not significantly affect lay-persons’ general beliefs about climate change mitigation and impacts, emotional concern about climate change, or motivation for pro-environmental behavior. Climate simulations may be less intense compared to its combination with role play and thus less effective in changing beliefs, concerns and intentions. Repeatedly using climate change simulations or a better integration of the new knowledge in existing attitudes may be necessary for effects on these dimensions.

4.2 Informed exploration increases feelings of self-efficacy

Participants who used En-roads more strongly agreed that publicly available tools such as En-roads help them evaluate climate policies compared to participants in the control condition. This effect was particularly strong for participants who engaged in informed exploration. In the informed exploration condition, participants freely explored En-roads but received additional information about setting options. They thus received more guidance compared to participants in the free exploration conditions but with more degrees of freedom compared to the step-by-step guide, that advised each step to reach the 1.5° goal. On the one hand, guiding users when simulations are complex can be advantageous (Kirschner et al., 2006). Yet, simple step-by-step instructions (similar to cooking recipes) can reduce cognitive engagement too much (cf. Rutten et al., 2012). The informed exploration condition seems to be a good balance between structure and options for free exploration. This guiding information could be implemented to the En-roads website to advance its usage and increase feelings of self-efficacy (cf. Mostafa, 2015, for the general importance of this factor).

4.3 Limitations and future research

Not all variables differed significantly between the conditions. Using the climate change simulation program En-roads affected lay persons’ perceived efficacy of measures to reduce climate change and their perceived own ability to evaluate these measures. Evaluations of political proposals, general political efficacy, general beliefs about climate change mitigation and impacts, emotional concern about climate change, as well as motivation for pro-environmental behavior were not affected by the simulation. Taking other reports regarding a lack of influence on policy support (cf. Tschötschel et al., 2021) into account, future research should investigate conditions that facilitate effects of simulations on these attitudes and intentions. Increasing the intensity of the experience or using repeated simulations may facilitate the effects (cf. Landmann, 2020).

This study was conducted in Germany, which is a WEIRD (Western, Educated, Industrialized, Rich, Democratic) society and belongs to the temperate climate zone. The consequences of climate change are increasingly evident in Germany with regard to flooding and extreme heat (German Environmental Agency, 2021b). Still, other regions are at higher risk for loosing larger amounts of livable area due to climate change and are less capable to adapt to these changes (IPCC, 2022). Participants in regions, which are more vulnerable to climate change due to their climatic zone or their financial situation, may respond differently to the climate change simulation used in the present study. The political and economic system may also impact the results. In non-democratic or less industrialized societies, evaluations of political measures may systematically differ from evaluations of such measures in democratic and industrialized societies. Testing the generalizability of the present results in different countries of different climatic zones including non-WEIRD societies is thus highly important.

Although the sample was diverse with regard to age and job status, it was not representative of the German population with regard to gender and education. This limits the generalizability of the results. Especially the optimal degree of guidance in climate simulations may depend on a person’s education and their previous experiences with simulation platforms. Furthermore, whether attitudes, concerns and intentions with regard to climate change are influenced by simulations may depend on pre-existing (polarized) attitudes towards climate change mitigation. Future studies should investigate these potential moderators.

5 Conclusion

The present research shows that the web-based climate change simulation tool En-roads can help the public estimate the progression of climate change as well as the effectiveness of specific measures taken against it. In the present study, a single online simulation changed the evaluation of measures against climate change. Hence, the tool can be used online to educate lay-persons about climate change mitigation. Practitioners can use the present results to estimate the expected effects of using En-roads in online interventions. The methods of the present study can be used to evaluate these interventions. Furthermore, the present research provides information about how the instructions for using the simulation program should be formulated. Providing information about setting options in combination with free exploration led to the highest perceptions of self-efficacy. Taken together, using En-roads online can help inform and empower citizens.