Introduction

Energy policy issues in Japan are summarized as “3E + S,” which stands for energy security, economic efficiency, environmental protection, and safety (METI 2018). Complex trade-offs often exist among these issues. For example, regarding energy security and measures against climate change, Japan needs a reliable and diverse energy production framework driven by renewable energy technologies to phase out nuclear power and avoid severe accidents (Nesheiwat and Cross 2013). Constraints on the usage of nuclear power increase the cost of achieving carbon emission reduction targets because of additional investments in low-carbon or highly efficient technologies (Politt et al. 2014; Oshiro et al. 2017). Under such trade-offs, the aforementioned policy issues are prioritized differently by different stakeholders. For example, the Japanese government plans to continue utilizing nuclear power as a low carbon, quasi-domestic power source (METI 2018), whereas the Japanese public appear to distrust the government and industry regarding the safe management of nuclear power plants (Poortinga et al. 2013); furthermore, they strongly support a decrease in the usage of nuclear power (Chapman and Itaoka 2018). As another example, the Japanese government has set a target of reducing national greenhouse gas emissions by 26% by 2030, compared to the emission levels in 2013. Japanese industries regard this target as extremely ambitious, whereas others suggest that the target is insufficient with regard to limiting global warming in 2100 to 2 °C above pre-industrial levels (Keidanren 2017; Kuramochi et al. 2017; Kuriyama et al. 2019). Such differences in priorities often prevent consensus on the long-term targets of energy policies.

To understand these trade-offs among policy issues and conflicts among stakeholders, professionals in the energy sector require an interdisciplinary perspective focusing on the interactions among technology, society, and the environment. Such a perspective is the essence of sustainability science, which aims to analyze the factors threatening the sustainability of natural and human systems and to transform these systems in a sustainable manner (Kates et al. 2001; Komiyama and Takeuchi 2006; Uwasu et al. 2009; Yarime et al. 2012; Barth and Michelsen 2013). A typical example showing the interdisciplinarity of sustainability science is the climate change scenarios that are used to explore possible futures, the assumptions they depend upon, and the courses of action that could bring them about (SENSES 2020). These scenarios express multiple future paths considering cyclic interdependency among socioeconomic development, energy and land use, greenhouse gas emissions, climate change, and impacts of climate change on economy and society. The comparative analyses of these paths are utilized to induce policy implications for transforming energy systems to avoid catastrophic climate changes (Shiraki et al. 2020; Sugiyama et al. 2020).

The training of professionals with interdisciplinary perspectives is imperative for the further development of sustainability science including scenario analysis. Particularly, sustainable development requires the involvement of the engineering sector (Nicolaou and Conlon 2012); engineers currently in training will be future leaders and professionals who hold crucial positions in the economic and political spheres (Pérez-Foguet and Lazzarin 2019). They will be required to understand the broader social, economic, and environmental implications of technological artifacts they design, manufacture, and operate (Allenby et al. 2009). Therefore, engineering courses in universities should deliver interdisciplinary education to train new critically thinking engineers (Tejedor et al. 2018). There are many macroscopic studies focusing on curricula and course design for sustainability science and engineering education (Mintz and Tal 2014; Kishita et al. 2018; Tamura et al. 2018; Franco et al. 2019). However, there are few microscopic studies focusing on the types of teaching methods suitable to cultivate such an interdisciplinary perspective. Game-based learning is recognized as a suitable method for teaching complex and parallel processes in technology, economics, and politics (Abt 1970). The games designed for education represent the structures and inherent interactions in target technological, social, and environmental systems. Such games are simulations of the real world driven by the decisions of participants who play the role of stakeholders in a target system (Greenblat 1988). By making decisions as virtual stakeholders, participants experience systems in contexts that are unavailable to them in the real world (Duke 1974). A well-designed game can simulate both structural elements and differential perceptions of system participants (Greenblat 1974). After the gameplay, in-game experiences and the real-world situation are linked through an interactive discussion called debriefing (Garris et al. 2002). By utilizing the characteristics of games, game-based learning provides an experiential environment wherein participants can learn about a target system (Mayer et al. 2004); moreover, it encourages students to think holistically and reflect critically on the learning material (Whalen et al. 2018). Game-based learning has been utilized for the education and scientific communication of various sustainability-related issues, such as energy issues (Suzuki 2016; Ando et al. 2019), water resource management (Hertzog et al. 2014; Douven et al. 2014; Craven et al. 2017), farming system management (Speelman et al. 2014; Farrié et al. 2015; Salvini et al. 2016), circular economy (Whalen et al. 2018), and climate change (Juhola et al. 2013; van Pelt et al. 2015; Matzner and Herrenbrück 2016). Previous studies have used observations of gameplay, records of debriefings, questionnaire surveys, or ex-post interviews to show that, in the process of game-based learning, participants become aware of complex interactions inherent to target systems and are encouraged to openly discuss and share ideas with other participants. However, only a limited number of studies have quantitatively evaluated the percentage of participants that learn something relevant to the predetermined learning targets through game-based learning.

Suzuki (2020) performed content analysis to evaluate the learning effect of a game-based class designed to teach energy policy issues to university students in an engineering course. The learning targets of the game-based class were twofold: to cultivate a perspective to overcome the trade-offs among policy issues; and to cultivate students’ abilities and attitudes to build a consensus by overcoming conflicts of opinions. An objective perspective overlooking the whole system was assumed with regard to the former target, whereas the subjective perspective of stakeholders participating in the system was assumed regarding the latter. The percentage of participants that learned something relevant to these learning targets was estimated through the content analyses of free-form reflection reports submitted by 90 students. The results showed that both targets were achieved to some extent. However, the study had two limitations: first, the content analysis was performed by a single researcher; and second, the participants belonged to different universities and grades.

Herein, we evaluate the learning effect of the same game-based class as Suzuki (2020). The aim of this study is to obtain a holistic view of learning in classrooms by investigating which topics students learn, and whether these topics are relevant to predetermined learning targets. The methodological limitations of the earlier study are solved as follows. First, content analysis was performed by three researchers to improve the objectivity and transparency of the evaluation, in addition to enabling the discovery of information that may have been missed by a single researcher. Second, we gathered 128 reports submitted in classes held as part of a single course at 1 university to ensure that the students were equivalent, in that students from the same course and grade are expected to have similar levels of prior knowledge. Through these quantitative evaluations, this study clarifies the effect of game-based learning on the interdisciplinary learning process of energy policy issues; furthermore, it contributes to the promotion of this teaching method in engineering courses.

Materials and methods

This section describes the board game, the game-based class, and the method for analyzing reflection reports. The design process and the rules of the board game, which is known as Power Generation Mix, are explained in Suzuki (2016). Detailed information on the game can also be obtained by contacting the corresponding author.

“Power generation mix” board game

Considering the two learning targets explained in the introduction section, the fundamental requirement for the game is to represent the trade-offs among policy issues and conflicts among stakeholders in an easily understandable manner. Therefore, the theme of the game is to select an appropriate mix of energy sources through discussions among stakeholders who have different opinions about power supply systems. In the game, the trade-offs among policy issues are represented by the differences in the advantages and disadvantages of various types of power plants, and the conflicts among stakeholders are represented as the differences in the target energy mixes among the decision makers. As such, the game is designed as an allegory to intuitively represent the essence of energy policy issues rather than as a simulation to precisely reproduce an actual energy system.

Power Generation Mix is an analog board game that does not require the support of electronic devices. The game is played by five or six players, and it generally takes 1 hour, including the explanation of rules. The game is set in a small fictional country where one electric power company exclusively supplies electricity. At the beginning of the game, all the electricity demand is covered by five oil-fired power plants owned by the company. However, the company needs to decrease its dependence on oil to address resource depletion and climate change. The players play the roles of board members of the electric power company, and their task is to decide, through discussions and voting, the types of power plants to be built to replace the aged oil-fired power plants. All players share the same purpose, which is to satisfy the electricity demand. However, the board members are sent from various organizations that have different opinions about the future energy mix. Therefore, the players also have individual objectives to ensure that the decisions of the company are desirable for their respective organizations.

Figure 1 is a conceptual figure of Power Generation Mix. The characteristics of power plants and scoring policies for board members are exogenously provided. The energy mix of the company is decided by the votes of players. The criteria of the entire power supply system, such as power generation costs and carbon dioxide emissions, are derived from the characteristics of the power plants and the energy mix. The score of each player is derived from the condition of the power supply system at the end of the game and the scoring policies of the board members. The player with the highest score wins the game.

The electric power company can build five types of power plants: coal-fired, gas-fired, nuclear, wind, and solar. The characteristics of these power plants are described by their power generation costs, carbon dioxide emissions, and the probabilities of three kinds of risk events. Details regarding the risk events are provided in subsequent sections of this manuscript. These parameters are set focusing on the relative relationships among power plants so that players can understand the trade-offs among them in a limited period of time. For example, the carbon dioxide emissions from coal-fired, gas-fired, and other types of power plants are set to 5, 3, and 0, based on the ratios among the typical carbon intensities of their respective fuels.

There are four criteria for the power supply system: power generation costs, carbon dioxide emissions, reserve margins, and the number of nuclear power plants. The power generation cost and carbon dioxide emissions are calculated from the values of the individual power plants included in the system. The reserve margin is calculated from the demand–supply balance of electricity during a game.

The board members belong to one of six organizations: a Council of Advisers, an Environmental Non-Profit Organization (NPO), a Commercial Company Association, a Manufacturing Company Association, a Government, and a Consumer Association. Each board member has their own scoring policy that defines the relationship between the four criteria of the power supply system and their game score. That is, the scoring policies show the subjective interpretations of the objective criteria, which are different among the board members. For example, the Environmental NPO gains a high score by reducing carbon dioxide emissions and the number of nuclear power plants. In contrast, the Manufacturing Company Association gains a high score by reducing power generation costs and keeping the reserve margin high. The former can win the game by successfully promoting wind and solar power plants while the latter can win by successfully promoting coal-fired and nuclear power plants.

The status of the power supply system and the stakeholders are expressed using paper boards, cards, and colored tokens. Figure 2 shows the components of Power Generation Mix.

Figure 3 shows the flow of the gameplay. Before starting the game, players choose a role from the six available roles. In the game, a time unit known as a “term” recurs five times. Each term consists of five phases: aging, strategy meeting, voting, events, and updating. In the aging phase, one oil-fired power plant is removed from the game because it reaches the end of its life span. In the strategy meeting phase, the players discuss which power plant should be built instead. The discussion is cut off after 4 minutes. In the voting phase, each player casts one vote for a power plant that they would like to build in that term. If a player chooses not to build a power plant, they can vote for “None.” If only one type of power plant is at the top of the list, this power plant is immediately built. However, if power plants of more than one type are at the top with the same number of votes, neither of them are built. No power plant is built if the highest number of votes goes to “None.” In the events phase, the occurrence of risk events is probabilistically determined by drawing event cards. Inherent risk events are different among power plants. Fuel price escalation, inherent to coal- and gas-fired power plants, leads to additional power generation costs. Decrease in output, inherent to wind and solar power plants, temporally drops the output of these types of power plants to zero. Severe accidents, inherent to nuclear power plants, result in an increase in power generation costs and the shutdown of all nuclear power plants. In the updating phase, the balance between demand and supply is checked. The value of electricity demand is fixed to three throughout the game, and the value of electricity supply is equal to the number of power plants that have not suffered from risk events. If the electricity supply is lower than the demand, a blackout occurs, and the game is immediately terminated. In this case, all players lose the game. If the electricity supply is equal to or higher than the demand, the values of the four criteria are updated depending on the type of the newly built power plant.

After the fifth term, each player calculates their score to determine the winner. To win the game, players need to build power plants that match their scoring policies. However, if new power plants are not built because of split votes, all players lose the game via a blackout. Therefore, players are required to balance their individual benefit and public benefit through discussions and votes.

Fig. 1
figure 1

Conceptual figure of the power generation mix board game

Full size image
Fig. 2
figure 2

Components of power generation mix

Full size image
Fig. 3
figure 3

Gameplay flow of power generation mix

Full size image

Overview of the game-based class

All game-based classes reported herein took place at the College of Engineering Systems of the University of Tsukuba, Japan. The college has a 10-week class called “Introduction to Energy Studies,” which aims to comprehensively teach the subjects of energy and environmental issues from the viewpoints of recourse, economy, and environment. The game-based class was taught in the second and third weeks of this class to broadly introduce students to the energy issues in Japan. The numbers of students who participated in this class were 30, 33, and 65 in 2017, 2018, and 2019, respectively. The number of students was higher in 2019 due to a change in the category of this class in our target college; until 2018, this class was a specialized subject only for students in energy engineering courses, while from 2019 onwards, it became a specialized subject for students in both energy engineering and environmental development courses. This change likely had little effect on the level of students’ prior knowledge, because these two courses had the same curricula in the first and second years, and this class was held at the beginning of the third year. They had comprehensively learned the basics of system, mechanical, electrical, civil, and architectural engineering.

The game-based class was taught for 2 weeks; the first week was for gameplay, and the second week was for debriefing. The overall process of this class is shown in Fig. 4. At the beginning of class, an opening lecture was presented to briefly explain the energy policy issues in Japan. Then, the students were divided into groups of 5–6 members to play Power Generation Mix. Subsequently, the students were assigned an intermediate report comprising three tasks: reporting the progress of the game from their subjective viewpoints; listing at least three criteria to evaluate the power supply system other than the four criteria that appeared in the game; and listing at least two advantages and two disadvantages of any power generation technology. The first task aimed to get students to reflect on the game and express their experiences in their own words, while the second and third tasks aimed to get students to survey the technological and social aspects of actual power supply systems. The students were required to submit their reports within 3–4 days, because the results were needed for the lecturer to prepare slide materials aggregating the student’s reports for the next class.

The debriefing in the second week aimed to combine the knowledge and experience obtained during the gameplay with actual policy issues based on the three tasks. Further, this process was designed so that students learned from each other’s experiences, which were affected by the difference in the results of games and personal characteristics, by encouraging students to share their own experiences with group members as well as with the whole classroom. First, students were divided into the same groups as those during the gameplay, after which they shared their subjective viewpoints and discussed how their decision-making process could be improved. Following this, a representative from each group presented a summary of their discussions to all students. The students in other groups freely asked questions and commented on the presentations. In 2017 and 2018, 10 min were reserved for the presentation and discussion of each group while it decreased to 5 min per group in 2019 because of increase in participants.

For the second and third tasks, a lecturer prepared slides that aggregated the answers to these tasks. Following the discussion of the first task, the slide materials with aggregated results were presented by the lecturer to the students. Some of the criteria and characteristics of power generation technologies were explained by the students themselves, whereas others were explained by a lecturer. Tables 1 and 2 provide summaries of the slides used in the 2019 class: Table 1 shows the criteria of power supply systems, and Table 2 shows the characteristics of gas-fired power plants. These materials indicate that the students’ reports were considerably in line with the criteria and characteristics of the real world. Therefore, the role of the lecture here is to improve students’ knowledge and to support the sharing of knowledge.

A concluding lecture was presented at the end of the debriefing. The lecturer explained how the game simplified an actual power supply system such that students were not exposed to inaccurate information and stereotypes. For example, the game omitted certain significant aspects of policy issues, such as the impact of international conflicts on the supply chain of fossil fuels and the accumulation of spent nuclear fuel caused by the lack of an efficient nuclear fuel cycle. The game also does not consider recent changes in the values of stakeholders; for example, that some private companies are currently willing to invest in low-carbon energy sources. Then, the themes of the game-based class, the trade-offs among policy issues and the conflicts among stakeholders, were explained.

Finally, the students were assigned free-form reflection reports whose theme was “report what you learned through this class.” These reports were required to comprise 2000 to 3000 Japanese characters, which corresponds to approximately 1000 to 1500 words in English. Students were given 1 week before submitting reports. Some reflected on the knowledge and experiences they obtained through the class, while others further surveyed the actual energy system to enhance their discussions.

Fig. 4
figure 4

Overall process of the game-based class

Full size image
Table 1 Criteria of power supply systems submitted by the students and categorized by the authors
Full size table
Table 2 Advantages and disadvantages of gas-fired power plants, as submitted by the students
Full size table

Analytical methods

Content analysis is used to develop a qualitative coding rule through the careful reading of texts by researchers; it is often adopted to extract key themes and topics from a set of free-form texts (Asayama et al. 2017; Whalen et al. 2018). The procedure of content analysis in this study was as follows. First, all paragraphs in all reports were given temporal labels that briefly explained the kinds of learnings reported in each paragraph. Multiple temporal labels could be given to any paragraph. Second, the temporal labels with similar meanings were combined into singular labels, which were defined as a set of homogeneous contents. The role of this process was to reduce ambiguities in the definitions of the assigned labels and enhance the objectivity and transparency of the categorization of the learning effects. Hereafter, the singular labels finally determined through this process are called “topics.” Most reports included more than one topic. Third, the number of reports including each topic was counted. In this process, each report was counted only once even if a certain topic was included in more than one paragraph. The careful reading and in-depth discussions by the three authors were repeated throughout this process.

This heuristic approach was adopted to infer what the students learned and how they learned it, without being constrained by the preliminary hypotheses of the authors. Being different from experiments in laboratories, the game-based classes in classrooms have many non-controllable conditions such as the number of students, composition of group members, results of games, and discussions in debriefings. This inability to control conditions which are inherent to game-based classes makes it difficult to form preliminary hypotheses about the topics students will learn in the classes. Therefore, this study put emphasis on obtaining a holistic view of the learnings observed in the classrooms, and compared these learnings with the predetermined learning targets, rather than rigorously verifying the achievement of the learning targets themselves.

Results

Identified topics

From the content analysis, most of the descriptions in the students’ reports were categorized into six major topics: (1) complexity of systems; (2) improvement of systems; (3) usage of nuclear power; (4) social dilemma; (5) subjective reality; and (6) consensus building. The definitions of these topics are summarized in Table 3. Hereafter, the detail of these topics are explained based on citations from the students’ reports; the authors chose examples that well represented the characteristics of each topic. Note that these citations were freely translated from Japanese to English, and some redundant expressions were omitted while ensuring that the essence of the content remained unchanged.

Table 3 Definitions of six main topics identified by the content analysis
Full size table

Complexity of systems

This topic comprises descriptions related to the complexity of energy systems, such as a variety of criteria to evaluate a power supply system or differences in the advantages and disadvantages of power plants. Many students expressed their surprise owing to the numerosity of the criteria, in addition to providing brief explanations of the criteria. Here is an example of such descriptions:

“I was surprised by the numerosity of the criteria and the advantages and disadvantages of various types of power plants because I found only two or three of them by myself. … The ‘environmental problem’ covers a broad range of topics to be focused on, such as carbon dioxide emissions, air pollution, and deforestation owing to land development.”

There were also many descriptions related to the advantages and disadvantages of power sources. Here is an example focusing on a biomass-fired power plant:

“The primary advantage of a biomass-fired power plant is its carbon neutrality: the carbon dioxide emissions from biomass fuels were originally captured from the atmosphere during the growth of the biomass. … However, biomass-fired power plants also have some disadvantages, such as a higher power generation cost and lower thermal efficiency compared with coal-fired or gas-fired power plants. The higher cost is caused by fuel supply processes including drying, chipping, shaping, and transportation.”

These descriptions show that the students learned various criteria and characteristics of power generation technologies that are necessary to understand the trade-offs among policy issues and to evaluate the performance of whole energy systems.

Improvement of systems

This topic comprises descriptions of methods to improve whole energy systems, such as the optimization of the energy mix and the research and development of new technologies. Several descriptions in this topic pointed out that the combined use of different types of power sources is important to overcome their mutual limitations and to avoid the risks inherent to each technology.

“If there are limited types of power plants in a power supply system, a fatal incident affecting one type of power source, such as a sharp increase in fuel prices or a severe accident may be fatal to the whole power system. In contrast, if various types of power plants are included in the system, the shutdown of one type of power plant can easily be compensated by other types of power plants.”

“I learned that power sources are categorized as base load, middle load, and peak load power sources: base load power sources have relatively low power generation costs and a constant output; peak load power sources have relatively high costs and a flexible output; and middle load power sources have intermediate characteristics. I also learned how to effectively use different types of power sources depending on the present situation.”

Apart from energy mix optimization, some students pointed out that the research and development of novel energy production technologies can mitigate the trade-offs among policy issues by improving the economic viability, environmental performance, and safety of the technologies.

“If the initial and operational costs for solar power systems are reduced through technological development, solar power systems are expected to be acceptable to those who desire low power generation costs.… Similarly, coal-based power may be acceptable to those who emphasize environmental protection if carbon dioxide emissions arising from the burning of coal are decreased.”

These descriptions indicate that the students not only recognized the trade-offs among policy issues but also considered the actions necessary to overcome these trade-offs.

Usage of nuclear power

The public acceptance of nuclear power has decreased since the Great East Japan Earthquake in 2011, whereas the Japanese Government still regards nuclear power as a major power source. Considering this, the students expressed various opinions about the usage of nuclear power. Many students objectively discussed the conditions for accepting the use of nuclear power rather than subjectively stating whether they were for or against the usage of nuclear power. Further, some of the students reported changes in their opinions as well as what they had newly discovered through the learning experience in the class. Two examples are presented below.

“I thought that nuclear power plants should be stopped after the Fukushima accident. However, now I think that nuclear technology is a stable power source and that it is safe as long as accidents are prevented. … Of course, we need to contemplate how to prevent severe accidents and how to mitigate the damage of accidents that unfortunately may occur.”

“Nuclear power should be promoted because of its clean and efficient characteristics. However, I heard that one classmate from Fukushima is against nuclear power. Actually, I did not experience power shortages or blackouts after the Fukushima accident, because my hometown is in the west of Japan. I might have a different opinion if I was in the same situation as this member.”

These descriptions indicate that at least some of the participants reconsidered their opinions regarding the importance of nuclear power owing to the game-based class.

Social dilemma

This topic comprises descriptions related to the social dilemma that is defined as a conflict between individual benefit in the present and social welfare in the future. Under this dilemma, each individual receives a higher payoff for a socially defective choice than for a socially cooperative choice, but all individuals are better off if they all cooperate rather than defect (Dawes 1980). Two well-known models of this type of social dilemma are the Prisoner’s Dilemma (Rapoport and Orwant 1964) and the Tragedy of the Commons (Hardin 1968). Moreover, the depletion of fossil fuels, air pollution, and anthropogenic climate change, which are strongly related to the sustainability of energy systems, are also often regarded as a social dilemma (Lloyd 2007; Wood 2011). Some students reported that they discovered dilemmas in the actual world through the experiences in the game-based class; most of them mentioned global warming as an example of such a dilemma.

“Most people do not accept clean energy if it requires higher cost compared with conventional energy sources, despite recognizing the significance of environmental conservation. Moreover, energy suppliers focus too much on increasing their profits.”

“Probably, most people tend to neglect environmental issues because it takes a long time before the consequences can be felt. They put too much emphasis on the relatively short-term challenges, such as stable supply or economic viability, whose profits are easily recognized.”

Subjective reality

This topic comprises descriptions about the plurality and instability of subjective realities. Some students mentioned that the subjective evaluations of an energy system are not necessarily shared by its stakeholders, such as private companies, governments, or consumers, because of the differences in their social statuses and personal values, even if they share the objective information about the current status of the system, such as the power source mix, costs of generation, and environmental load.

“In the game, I played the role of a representative of an environmental NPO and experienced difficulty in persuading other players not to build carbon-intensive power plants. They did not ignore the fact that carbon dioxide emissions have negative impacts; however, they still did not curb the emissions. Similarly, the advocates of nuclear power in the real world appear to know the disadvantages of the technology, such as the risk of severe accidents and high-level radioactive wastes, but they assign a higher priority to its advantages such as a stable power supply and low carbon dioxide emissions.”

Aside from the differences in the interpretation of the same fact, a few students pointed out that such interpretations by a person or an organization may change upon the occurrence of particular incidents. They reported that they changed the perception of risk related to a certain technology after the occurrence of the risk event.

“All members of our group lost the game because we built three nuclear power plants and a severe accident took place. … I could not imagine that such a thing would really happen. Then, I recalled the press conferences by professionals and politicians immediately after the Fukushima accident. They said exactly the same thing that I felt.”

These reports suggest that students had a virtual experience of the difficulties the policy makers face when they plan the future energy mix and found how the plurality and instability of subjective realities affect the future plans for energy systems.

Consensus building

This topic comprises descriptions of the actions to be taken to build a consensus on future energy systems among various stakeholders. Some students mentioned desirable attitudes for consensus building, such as mutual understanding, trust, and compromise. A typical description is as follows.

“It is impossible to perfectly understand each other because we have different views and ideas. However, it is important to try to understand our differences, and making efforts to advance our mutual understanding, such as presenting and discussing our opinions with others, is a meaningful way to build a consensus.”

Other students mentioned that objective knowledge, information, and data should be shared as the premises of discussions, and such premises should be shared not only among professionals but also between professionals and non-professionals.

“It is essential to survey the specialisms of other stakeholders before discussions to eliminate what we have not heard, and to reach a mutually agreeable compromise. Thus, merely having information regarding the themes and participants of meetings is insufficient; rather, daily efforts to read economic newspapers and professional journals in various fields are required.”

“I revised my idea so that we could explain to the common citizens why something should be advanced, win the understanding of people who are anxious about this, and decrease the gap in the understanding between professionals and citizens.”

Furthermore, some students described the necessity of political support, such as setting up a structure to reflect minority opinions or subsidize new technologies. These reports suggest that the game-based class helped the students to overcome the plurality and instability of subjective realities and build a consensus among the stakeholders.

Relevance with learning targets

The first four topics are relevant to the first learning target: the cultivation of the perspectives necessary to overcome the trade-offs among policy issues. The improvement of the system (topic 2) is highly relevant to the first learning target because it includes proposals to tackle the trade-offs such as the combined use of different types of power plants and the promotion of novel technologies. The complexity of the system (topic 1), that is, understanding the causes of trade-offs, includes a variety of criteria and technological limitations also relevant to the first learning target, even though this does not include proposals to overcome the trade-offs. The usage of nuclear power (topic 3) and the social dilemma (topic 4) also involve an objective understanding of social structures behind specific policy issues; thus, these topics are also relevant to the first learning target. The fifth and sixth topics are relevant to the second learning target: cultivation of the students’ abilities and attitudes to build a consensus. Consensus building (topic 6) is highly relevant to the second learning target because it includes a wide range of proposals on the attitudes of stakeholders, communication among stakeholders, and political support. In contrast, subjective reality (topic 5) remains in explanations regarding barriers to making desirable decisions and reaching a consensus, for example, the diversity in subjective interpretations of objective information and changes in the perception of risk from severe incidents; this is therefore only moderately relevant to the second learning target. In summary, the improvements of the system and consensus building are highly relevant to the first and second learning targets because they include proposals related to energy policy issues. The complexity of the system, usage of nuclear power, and social dilemma are all moderately relevant to the first learning target because the trade-offs between these issues need to be understood. Subjective reality is moderately relevant to the second learning target because it includes the understanding of the barriers to consensus building.

Holistic view of learnings in classroom

Figure 5 shows the distribution of students who learned the six main topics based on the number of reflection reports that included descriptions that were categorized into these topics. The blue, green, and yellow bars show the distribution of students in 2017, 2018, and 2019, respectively, who learned each of the main topics, and the red points show the total distribution across the 3 years. Focusing on the total distribution across the 3 years, the improvement of the system and consensus building, which are highly relevant to the first and the second learning targets, respectively, were learned by 44% and 86% of the students, respectively. The complexity of the system, usage of nuclear power, and social dilemma, which are moderately relevant to the first learning target, were learned by 65%, 25%, and 30% of the students, respectively. The subjective reality, which is relevant to the second learning target, was learned by 66% of the students. From these results, we estimated the proportion of the students who learned something relevant to the two learning targets. First, the relevance of learning was evaluated under strict conditions: the number of students who reported the topics that were highly relevant to the targets were counted. Under this condition, only “improvement of the system” (topic 2) was considered to be relevant to the first learning target, and only “consensus building” (topic 6) was considered to be relevant to the second learning target. Figure 6 shows the cross-tabulation of the proportion of students who learned something relevant to the two learning targets. The share of students who simultaneously learned something relevant to both targets is 37% (the purple bar Fig. 6). The shares of students who learned something relevant to either of the targets are 7% and 49%, respectively (blue and red bars in Fig. 6, respectively). The share of students who did not learn anything relevant to the targets is 7% (white bar in Fig. 6). These results suggest that more than one-third of the students learned something relevant to both learning targets.

Fig. 5
figure 5

Share of students who learned the main six topics. The blue, green, and yellow bars show the share in 2017, 2018, 2019, respectively, and the red plots show the total share across all 3 years

Full size image
Fig. 6
figure 6

Share of students who learned topics highly relevant to the two learning targets. Only topics (2) and (6) were considered to be relevant to the first and second targets, respectively

Full size image

Next, the relevance of learning was evaluated under more relaxed conditions: the number of students who reported the topics that are highly or moderately relevant to the targets were counted. Here, the first four topics were considered to be relevant to the first learning target, and the fifth and sixth topics were considered to be relevant to the second learning target. Figure 7 shows the result of cross-tabulation under these relaxed conditions: 83% of students learned something relevant to both targets. These results suggest that a relatively high number of students learned something relevant to the predetermined learning targets.

Fig. 7
figure 7

Share of students who learned topics relevant to the two learning targets under relaxed conditions compared with those of Fig. 6. Topics (1–4) were considered to be relevant to the first learning target, and (5) and (6) were considered to be relevant to the second target

Full size image

Although prior knowledge of students appears to be equivalent among the 3 years, Fig. 5 suggests that there might be a difference in the learnings of students among these years, in that the share of students who learned each topic (%) varied from year to year. Therefore, the significance of these differences was investigated using the chi-square test. For each topic, the null hypothesis was “The year in which the game-based class was held and the share of students who learned a topic were independent of each other.” Table 4 shows the results of tests: Chi-square value and p value for each topic. The p values show the probability that the null hypothesis is correct. For two of the four topics relevant to the first learning target (topics 2 and 4), the p values were below 5%; the share of students who learned these topics appeared not to be independent from the year. In common with both topics, the share in 2019 was the lowest for these years. For the two topics relevant to the second learning target (topics 5 and 6), the p values were much higher than 5%; the share of students appeared to be independent from the year. The higher p values were caused by relatively small differences in the shares among years. In conclusion, the learning effects relevant to the first learning target were partially different among years while those relevant to the second learning target were robust throughout the 3 years.

Table 4 Results of chi-square tests. Null hypothesis: “The year in which the game-based class was held and the share of students who learned a topic were independent of each other”
Full size table

How students learned

Some of students described “how they learned” and “what they learned.” Through careful reading and in-depth discussions by the authors, these descriptions were divided into three categories: voluntary survey, other participants, and differences between game and reality. The voluntary survey comprised self-motivated investigations whose topics were not picked up in the class. For example, there was a student who surveyed and reported on the decision-making process to abandon nuclear power in Germany.

“I learned that the German government decided to abandon all nuclear power plants from 2022 responding to the Great Earthquake and the Fukushima accident in Japan. The Ethics Commission responsible for this decision comprises not only nuclear power professionals but also various stakeholders such as Christian scholars. This framework is useful for picking up various opinions.”

Apart from this example, there were a broad range of topics, including a psychological experiment regarding the short-sighted tendency of humans, energy mix of European countries, and current status of domestic fossil fuel mining in Japan. The number of reports under this category was 16 (13%) for the 3 years. The “other participants” category comprises descriptions regarding the presentations or surveys performed by other students during the debriefing. The following is an example of a student learning about consensus building from the group presentations.

“There were two impressive groups in my view. The first group shared a common sense to assign the highest priority to the benefit of citizens prior to strategy meetings. Such a principle appears to make consensus building easier. The second group largely changed their decision-making process from the first to the second term. In the first term, they failed to build a power plant due to the conflict of opinions. However, they built a plant by a unanimous vote in the second term. This change appeared to be caused by the experiences gained in the first term.”

Many other reports mentioned the learnings from other students in the same class, which mainly comprised various criteria and processes for decision making. The number of reports under this category was 55 (43%) for the 3 years. The “differences between the game and reality” category comprised discussions regarding how the real world was different from the game, where it was pointed out that an actual energy system is much more complex than a game, and that additional factors need to be considered in the real world. The following is an example of such discussions:

“The greatest difference between the real word and a game is responsibility. If we actually participate in a meeting simulated in the game, we need to make decisions through more careful discussions representing the voices of people who have similar opinions to ours and by thinking futuristically.”

The number of reports under this category was 22 (17%) for the 3 years.

Discussion

Effectiveness of the game-based class

The first theme of the game-based class, i.e., the development of an energy system by overcoming trade-offs among policy issues, is a significant research theme in the field of energy systems engineering. The studies in this field aim to identify an optimal future scenario that satisfies the energy demand and achieves environmental targets with minimum social costs (Cheng et al 2015; Shiraki et al. 2016, 2020; Komiyama and Fujii 2017; Santen et al. 2017; Sugiyama et al. 2020). In contrast, the second theme, consensus building among stakeholders with different values, has been studied in the field of social psychology in the context of growing sustainability-related issues, including resource depletion and climate change. There have been many experimental studies regarding enabling cooperation among stakeholders under the social dilemma (Edney and Harper 1978; Dawes 1980; Yamagishi 1988; Mulder et al. 2006; Kitakaji and Ohnuma 2019). The results of our study suggest that quite a few students learned these two themes in an integrated interdisciplinary manner. Further, students’ descriptions of how they learned suggest that at least some of them actively tried to deepen their understanding through voluntary surveys, the presentations and surveys of other students, and the considerations of differences between a game and reality. Some earlier studies determined that game-based learning enhances the motivation of students (Abt 1970; Garris et al. 2002; Madani et al. 2017). The active learnings observed in this study may also be induced by the characteristics of game-based learning. Such motivation to study is especially important in interdisciplinary energy systems education because it is impossible to teach all the aspects of energy policy issues in the limited time available.

These findings enhance the conclusion of an earlier study (Suzuki 2020): the game-based class is an effective method for interdisciplinary energy systems education, and it supports the active learning of students. The content analysis in this study was performed by three researchers, whereas that in the earlier study was performed by only one researcher. Consequently, the temporal labels denoting the content learning types were combined during the process, so that all three authors agreed upon the definition for each singular label (topic) and were able to clarify the differences among them; the final number of topics was 6 in this study, whereas it was 16 in the earlier study. The discussions among the authors during content analysis appear to have enhanced the objectivity and transparency in defining the topics. Despite this improvement in the analytical method, the topics identified in this study were closely relevant to the learning targets identified in the earlier study. Consequently, this study is an improvement of the earlier study; it supports the earlier findings through a more objective and transparent method.

Role of debriefing

As shown in Fig. 5 and Table 4, the learning effects relevant to the second learning target were robust throughout the 3 years, while the learnings relevant to the first learning target were different among the years; they appear to be lowest in 2019. Although the learning effects in the game-based classes were affected by various conditions, the large increase in the number of students in 2019 may have affected the learnings in the classroom. Assuming that the increase in the number of students is the main cause of the decrease in learning effects, the reason why only the learning effects relevant to the first learning target decreased can be explained as follows. A game is a medium that puts a stronger emphasis on the subjective aspects of reality than its objective aspects (Crawford 1984). Therefore, the subjective perspective of stakeholders related with the energy system, relevant to the second learning target, can be directly learned from the gameplay. However, the objective perspective overlooking the whole system, relevant to the first learning target, cannot be learned without an ex-post process connecting the game and real world, because the energy system is largely simplified in the game. The gameplay in the first week was not affected by the increase in the number of students, because a larger number of students could be accepted by simply increasing the number of groups. However, the debriefing in the second week was substantially affected by this change. The total time for the presentations from each group was extended, and the time for other contents, including questions and comments among students and supplemental explanation from a lecturer, were shortened. For example, the time for presentation and discussion per group is reduced from 10 min in 2017 and 2018 to 5 min in 2019. Since the topics of presentations were the same among the 3 years, the decrease in the learning effect appear to be caused by the lack of time for communication among students. Although the relevance of this inference needs to be verified in future work, it does appear to be a useful suggestion related to the role of debriefing and the ideal number of students in a game-based class.

Limitations

This study evaluated learning effects via the content analysis of free-form reflection reports. This method is suitable to heuristically capture the type and range of learning, which cannot be known initially. However, this method has some limitations. First, the subjective interpretation of researchers cannot be completely excluded. Second, the achievements of students, i.e., how deeply they might have learned the topics, cannot be examined. Third, the writing skill of students affects the interpretations of researchers. One solution that addresses all three problems is to design closed questions relevant to the topics identified by this study, such as the questions “point out and explain the issues occurring when the share of photovoltaics in the power generation mix is raised to 50%” (relevant to topic 1), or “briefly explain an example of social dilemma related to the energy system” (relevant to topic 4). In this way, the achievements of students could be objectively examined by correctness of answers to these questions. Further, students were not required to possess high writing skills to complete the free form report with 2,000 characters.

As mentioned above, in this study, a large number of participants had prior knowledge of the basics of energy and system engineering, which helped them to understand the theme of the class. Therefore, the results of this study do not guarantee that this game-based class is also effective for students in lower grades or for students at different colleges. To expand the target audience of this game-based class to students in lower grades or in non-engineering courses, a lecture on energy technologies and energy systems may need to be given in advance. Furthermore, Power Generation Mix may be too simple for professional education because this game is designed for introductory education. For professional use, another game should be designed to simulate the real energy supply system more precisely. Power Generation Mix reflects natural and social constraints inherent to Japan, such as poor fossil fuel resources, high population density, and a power grid that is isolated from other countries. Therefore, this game cannot be directly applied to learn energy policy issues in foreign countries. However, social structures represented by the game, such as trade-offs among 3E + S and conflicts of values among stakeholders, are universal energy policy issues. Therefore, the game can be adapted to teach the issues of other countries by modifying its components, such as the available technologies, stakeholders, and risk events.

Conclusions

This study evaluated the effectiveness of a game-based class for teaching energy policy issues in engineering courses. The results of content analysis suggest that a significant number of students learned the trade-offs among policy issues and conflicts among stakeholders in an integrated manner. These findings are consistent with the conclusions of earlier studies: game-based learning is effective for interdisciplinary education on the sustainability of society. The unique contribution of this study is the quantitative evaluation of relevance between the predetermined learning targets and the topics actually learned by students through content analysis by multiple authors. The results of this study also suggest a difference between direct learning through gameplay and ex-post learning through debriefing; the subjective aspects of the energy system were directly learned via the gameplay, whereas the objective aspects were learned with the help of the debriefing. It is important to identify the student perceptions that are stimulated by the game itself and those that may be attributable to the secondary course contents and learning objectives (Whalen et al. 2018). Such identification is beneficial for designing and developing game-based classes and for appraising the minimum learning effects when there is insufficient time for debriefing. Furthermore, knowledge about the learning mechanism can be useful when adopting other methods for facilitating learning, such as a case method and a role play in which participants virtually experience the roles of real stakeholders.

The current version of the game-based class is focused on energy policy issues in Japan, and it assumes university students in engineering courses as participants. Further, this study does not consider the relationship between the game-based class and other educational contents in an engineering course. These limitations suggest three potential themes for future studies. The first theme is to revise Power Generation Mix to target foreign countries and regions. The understanding of energy policy issues in foreign countries is important not only for foreign students but also for Japanese students. The second theme is to develop another game-based class that can be used beyond higher education in engineering courses. A game-based class for high school students enables the interdisciplinary sustainability education of younger generations and increase their motivation to pursue higher education. In contrast, the inclusion of more real games for professional education is expected to promote bi-directional discussions regarding policy issues between participants and lecturers. The third theme is the macroscopic evaluation of game-based learning, such as the synergy between game-based learning and lectures in a 10- to 15-week course and the role of game-based learning in engineering curricula. These themes will further contribute to the promotion of game-based learning and are expected to support teachers in exploring teaching methods suitable for interdisciplinary sustainability education. Furthermore, knowledge about game-based learning is expected to be useful to involve non-professional stakeholders in the modeling and scenario planning process of sustainability-related issues (D’Aquino and Bah 2013; Voinov 2016) and to lead their perception and behavior in a sustainable direction.