1 Introduction

“Serious games” have been used to help students connect educational content with real-world situations. Bergeron’s well-cited definition of a serious game is an “interactive computer application, with or without significant hardware component, that has a challenging goal, is fun to play and engaging, incorporates some scoring mechanism, and supplies the user with skills, knowledge, or attitudes useful in reality” [1]. Although serious games are now primarily digital, there is a long history in using non-digital serious games for the purpose of learning before the invention of the computer [2]. By combining elements of education and entertainment, serious games fall between simulations (which focus on realistic skill development) and general games (which focus on entertainment) on the spectrum of different types of computer games [3]. Research studies on the impact of serious games have emerged in several fields, especially in psychology, health, education, energy, business, and information systems [4].

Researchers have applied different theories to explain how serious games can impact players. Although a wide range of theories have been used in serious games (e.g., Self-Determination Theory, Theory of Planned Behavior, Social Cognitive Theory, Flow Theory, Motivational Theory, Theory of Fun, Cognitive Evaluation Theory) [4], perhaps the most overarching explanation is the Persuasive Game Design (PGD) model [5,6,7]. According to the model, designers create game elements that can transport players from the “real world” towards a separate “game world.” In the game world, the players have immersive and enjoyable experiences with game elements (e.g., challenges, exploration) that intentionally overlap with the real world. In turn, the players’ knowledge, attitudes, and behaviors that were impacted in the game world can then be applied back to the real word, known as the “transfer effect” [5,6,7].

Serious games have been used as a method for training students in a wide variety of health-related professions, with some success. Developers have created serious games regarding specific topics for those studying medicine, nursing, dentistry, physical and occupational therapy, and psychology [8, 9]. Several systematic reviews of the literature have been conducted on serious games across health-related disciplines, finding that games ranged from being modest to strong in effect for improving specific areas of knowledge, attitudes, and skills [3, 9,10,11,12,13,14,15].

Currently, three serious games have been developed and tested for students and healthcare professionals within the area of infectious epidemiology. Two of those games were created in response to the COVID-19 pandemic. Suppan and colleagues created the “Escape COVID-19” game specifically for hospital staff [16], which was tested using a randomized control trial, showing an improved willingness of employees at long-term care facilities in Switzerland to adopt safe infection prevention and control procedures [17]. In addition, Hu and colleagues developed “Covidgame” to provide medical students in China with information about COVID-19. Their case–control study found that students who played the game maintained a higher level of knowledge retention of COVID-19 information compared to students who attended an online lecture that included the same content [18].

The other serious game, known as “Solve the Outbreak” (StO) was created by the Centers for Disease Control and Prevention (CDC), with the objective of teaching players about infectious diseases, outbreaks, and how epidemiologists use their knowledge to save lives [19]. In this online game, players assume the role of a field epidemiologist and progress through several case studies by answering multiple choice questions regarding logical steps in conducting an outbreak investigation in response to case scenarios, clues, and data. The game provides feedback to players for correct and incorrect answers. There are two levels in the game, with 12 outbreak case studies in the first level, and eight outbreaks in the second level. Players must earn a perfect score in the first level in order to advance to the next level. In all, there are 13 badges that players can earn in order to be promoted from “Trainee” to “Disease Detective,” with the potential to earn nine ribbons based on various skills (e.g., surveillance, data analysis, statistics), all of which can be shared on social media. The game has a “Help” tab that provides gameplay instructions, and a “Learn” tab that provides information about epidemiology, epi curves, terms, and resources for teachers to implement the game into curriculums. StO is available online [19] and through mobile app stores.

Although StO was developed as a serious game, and created by a respected authority on infectious outbreaks, there remains a lack of research on the game as a learning tool. Several experts have suggested using StO as a method to teach epidemiological concepts for college students [20, 21]; however, to the authors’ knowledge, only one article has mentioned the game’s usefulness. Professor Fred Krebs asked medical students enrolled in his course on infectious diseases at Drexel University to play StO. He then invited students to critique the game’s scientific content, design, and relevance. Most of his students recommended that Professor Krebs continue to assign the game in future sections of the course [22].

Given the recommendations to use StO as a teaching tool, and given the limited amount of research on the game, the purpose of this study was to receive undergraduate college students' feedback regarding StO as a serious game for learning about infectious epidemiology. The creators of the PGD model recommend that games be evaluated, and that although randomized control trials are considered the most rigorous approach in testing serious games, another option is qualitative testimonials of players to provide information about their game experience [7]. By conducting this research as a qualitative debriefing study, the authors sought to explore aspects of the PGD model with StO: the students’ experience with the game’s elements, enjoyment in the game world, and any “transfer effects” from the game [5,6,7].

Debriefing is a valuable method for the learning process in an educational context. Lederman defines debriefing as “the process in which people who have had an experience are led through a purposive discussion of that experience” [23]. Although debriefing has its roots in the military (for coping with war trauma and to develop new tactics based on decisions and outcomes on the battlefield) and psychology (for explaining to participants the true purpose of a study that involved an element of deception), it also became an important tool for education (for teachers to help students process what was learned from an educational experience) [23]. As such, the educational focus of serious games makes debriefing a credible, important methodology for researching what players experience and learn from a game [23,24,25,26].

2 Methods

During the spring of 2022, undergraduate students enrolled in two sections of a junior-level epidemiology course (located at a public university in the southeast region of the US) were assigned to individually complete the CDC’s StO game. The game was used to complement a chapter from the course’s textbook, which was published by the CDC [27], that focised on the 13 steps of conducting an outbreak investigation. Of the StO’s 20 total outbreak case studies, several overlaped with steps of conducting an outbreak investigation as covered in the textbook chapter, with the exception of steps 1, 9, and 13 (Table 1).

Table 1 CDC textbook steps of investigating infectious outbreaks that are covered in StO case studies
Full size table

The assignment was asynchronous, with it being introduced on the first day of class, and due sometime before the week of final exams. Students were instructed to earn a perfect score in both levels, and then submit a screenshot of their score as proof of completing the game. Most students completed the assignment the week of the due date. Students reported that the game required between two to five hours to complete, due to variation in how fast students read the material, and variation among students in mastering the game’s first level. It is worth noting that the study was conducted during the COVID-19 pandemic, but that face-to-face learning had resumed at the university.

After obtaining IRB approval, all 30 students enrolled in the course were invited to participate in the study. In all, five focus group interviews were conducted with 16 students (response rate = 51%) who gave their informed consent to participate. The participants were all undergraduate juniors or seniors, 15 (93%) were traditional-aged college students, and 13 (81%) were female. Demographic information on race and ethnicity was not collected.

Focus group interviews were conducted during finals week, and included two to five students per focus group. Debreifings were conducted as focus groups to be consistent with previous debriefing literature [28,29,30,31], and also to align best with the students’ busy schedules. Participants were offered an incentive of pizza and soft drinks. The professor of the course and two research assistants conducted the interviews in two classrooms on campus. The research assistants were trained by the professor in conducting focus group interviews before data collection. During participant recruitment, and also immediately before data collection, the professor clarified that deciding to participate in the study, or deciding not to, would not help nor hurt students’ standing or grades in the course. The focus groups were digitally recorded and then transcribed verbatim. All identifying information was removed during the transcription process.

During the interviews, students were asked questions that were adapted from Thiagarajan’s [32] and Lennon’s [33] game debriefing interview guides, given the guides’ previous use in infectious disease games in varying contexts, and the guides’ applicability to the components of the PGD model. Lennon’s interview guide [33], which was adapted from Theigarajan’s guide [32], has been successfully employed in multiple geographic settings (Asia and the U.S.), various age groups (e.g., elementary, high school, college), and various types of games (e.g., board games, digital games) [29,30,31, 33,34,35]. In regards to the PGD model, the following debriefing questions reflected students’ immersive and enjoyable experiences in the game world: “How did you feel after you finished the game? Did you like the game?” To explore the transfer effect, students were asked the following questions: “What was the game about? Did the game cover anything that you could practically apply in the future? Did you learn anything new from playing the game? In the future, is there anything that you would suggest to a team of coworkers during an outbreak investigation based from what you learned in the game?” Finally, the last interview question focused on retrieving student feedback about the game’s elements (e.g., challengs, awards): “Is there anything that you can suggest to improve the game?”.

It is important to note here that although debriefing is primarily used to enhance the learning process of students when playing serious games [23,24,25,26], debriefing has also been used by studies as a qualitative research method [29,30,31, 33,34,35]. Given that debriefing is, in essence, a form of qualitative interviewing, the debriefing questions that were asked to participants were also viewed as a qualitative interview guide. Based on the purpose of this study, and the recommendation that games be evaluated by obtaining qualitative feedback from players [7], the authors determined that debriefing was the most suitable methodology for studying players’ experiences with StO.

Transcripts were then analyzed using common methods of qualitative research. First, the authors were “immersed” in the data by reading the transcripts several times [36]. The transcripts were read in order to understand both the “manifest content” (i.e., the text itself) and the “latent content” (i.e., the underlying meaning of the text) [37]. Second, the authors worked independently and also as a team [38] to identify, define, and refine major themes that emerged from the data (i.e., “crystallization”) [36]. Key statements from transcripts were then clustered under those common themes. During data analysis, it was estimated that “saturation” (i.e., when distinct themes no longer emerge from the data) [39] took place during the third focus group interview. The data was analyzed in Microsoft Word.

3 Results

A total of four themes emerged from the data. The first theme (Enjoying the Game World) reflected the PGD model of having a positive experience in the game world that players were transported to. The next two themes (Cheating the Game’s Elements, Suggesting Improvements to the Game’s Elements) fell within the PGD model of game elements, which students contstructively criticized. In the fourth theme (Inverted Transfer Effect) students report that the game was an effective tool for applying course content that was already learned during the course, which illustrates a type of inverse to the PGD model’s transfer effect.

3.1 Theme #1: enjoying the game world

Students reported that they enjoyed playing in the game world. The students liked assuming the role of an expert epidemiologist who traveled to new field locations. One student stated, “I did like how it [StO] was an assignment because it was fun… It challenges the brain a little bit a lot of new scenarios, you got to be the expert, it was fun” [Participant #5]. Another student said, “You were basically like the epidemiologist, and you got to decide what was the next step to take in order to figure out what caused the outbreak, and so that was cool. How they do it, the process of it” [Participant #8]. In addition, another student specified enjoying the aspect of travel in regards to being a field epidemiologist in the game, “I liked the geography. It [StO] was taking you to different places in the world. Like, ‘Let’s see where it’s taking me today’ … I thought it was, for me anyways, was the most fun parts of epidemiology and public health by trying to get to the bottom of it. I thought it was fun” [Participant #14].

Students also described enjoying the realistic aspects of the game in regards to outbreaks. One student said, “I really enjoyed it [StO]. I would say probably the most fun part about it for me was again just learning about the different diseases and how they are spread. I just thought it was really interesting. I felt like the examples that they gave and the real-world stories, they were actually realistic. You could tell it could be real. I know it was a game and a lot of it was written for the game, but it was very real life and I appreciated it” [Participant #13].

3.2 Theme #2: cheating the game’s elements

Students also reported that certain aspects of StO’s design elements made it easy to cheat the game’s questions. During the first attempt of answering questions, the game informs players of incorrect and correct answers. Students reported quickly answering the game’s questions without actually reading the prompts, and taking notes which answers were correct. Then, on the students’ second attempt, they reported already knowing the correct answers and could quickly finish each of the game’s levels. For example, one student said, “… I would do it [answer StO’s questions] and it would tell me what I got wrong and what I got right. So, it makes it easy just to try and find the right answer, rather than actually read it and understand it. You know. I just didn’t really take anything away. It was just kind of busy work for me” [Participant #6].

In addition, students found that the CDC openly provided the game’s answer key online. Those students reported that they were able to complete the entire game without solving any problems on their own. One student confessed, “I found the PDF with the answers [to the StO questions]. It wasn’t really cheating, it was like, using my resources… I kept getting one wrong, and I was like, ‘I don't know what's going on.’ I got so fed up, I was just like, ‘Let’s just look on the CDC’s website.’ Then, I was like, ‘Just let me use this [answer key]’” [Participant #4]. Another student recommended, “I think they [the CDC] should take the [StO] answer key offline, because they had the whole answer key just sitting on the website. Or, maybe just send it [the answer key] to instructors” [Participant #8].

3.3 Theme #4: suggesting improvements to the game’s elements

Several students recommended that StO could be improved by the CDC by reducing the number of case studies and questions. Students viewed each of the game’s two levels as being similar in structure and difficulty, and they recommended that the CDC either make game’s two levels more distinct, or remove one of the levels. One student said, “I was expecting level two to be different in some way. And even if it wasn’t necessarily in terms of difficulty, I was just expecting something a little bit different. And in my opinion, it was very, very similar. Part of me was thinking, ‘Why even separate it into two different levels if it's all essentially the same thing? You know, why not just play the whole game?’ So, I guess I was just expecting it to be different in some way, and it wasn’t to me” [Participant #13]. Another exclaimed, “It [StO] was too long. Like, maybe if we could just do one level that would be a lot better, because I think it was just kind of repetitive. But I think it would be better at just one level. [If a game is] longer than 30 min, you’re going to start losing people’s attention, I think” [Participant #4].

The study’s participants also believed that the game’s design elements contained too much text to read. One student stated, “I don’t know if I’m the only one to experience this, but I have trouble with reading comprehension, so it was not easy to comprehend… And then actually making the words and the paragraphs on the outbreaks shorter. Not so word heavy, because then you just get lost” [Participant #4]. The students also thought that the game’s readings and multiple choice questions were repetitive, and did not feature enough multimedia. One student suggested, “I would also suggest they add different kinds of media to make it more interactive. Like videos, pictures, or actual interactive games. I mean, it is a game after all. I feel like it was just a lot of reading” [Participant #16]. Similarly, another student said, “I did not like how it was all reading and not much interactive material. Like videos or something would make it more entertaining” [Participant #9].

Finally, several students reported feeling anxious throughout the game because there was not an option to create a login in order to save the players’ progress throughout the game. One student reported, “I personally did not like how there was not an autosave function. I was scared to close my laptop in case my tab closed. I did that when I first started, and lost my progress, and had to start over” [Participant #9]. Other students felt the same way. A student stated, “They [the CDC] really need to find a way to let us save our progress. I think that is a big thing. You get kind of stressed out about it. Like, if my computer just restarts it would just go away. All that work and time” [Participant #11]. Another shared: “I did not like that the game did not have an autosave or account function. My computer likes to restart itself a lot and I was scared to let it do so because I did not want to lose the progress I had spent so much time to make… I would suggest they develop a save-as-you-go function. I think it would increase use” [Participant #16].

3.4 Theme #4: inverted transfer effect

Participants reported that they appreciated StO in terms of applying outbreak investigation concepts that were already covered in the course. This represents an inverse to the PGD model, which posits that players apply knowledge, attitudes, and behaviors gained in the game world back to the real world. In this case, students used their knowledge that was gained in the real world towards the game world. Specifically, students stated that they enjoyed applying their knowledge in the form of a game, compared to other forms of assessment, such as exams, papers, or presentations. For instance, one student stated, “…it [StO] definitely reiterated what we were doing in here [the course], so it was like hands-on learning. It wasn’t just that I read it from the book, I took a quiz, like we actually used what we did in class and used it towards this game” [Participant #7]. Another student said, “…I did learn a lot and preferred [StO] compared to a paper or a test. I feel like it gave me a better opportunity to use my knowledge I’ve learned from all of my classes… I’m interested in epidemiology as a future career. I liked learning about how outbreaks can range from targeting specific populations to anyone and everyone. The outbreaks we solved reflected that” [Participant #16]. Another student discussed applying course content regarding the chronological steps of outbreak investigation: “I learned more about diseases and how they’re spread, and what that looks like, and just in general the steps that are taken to investigate and figure out what could be causing disease. We have talked about that in a lot of different classes that we've taken, but actually applying it was a little bit different. I definitely learned a lot of new things [from StO]” [Participant #13].

4 Discussion

This preliminary study saught to explore undergraduate students’ perspectives towards the CDC’s StO game as an assignment for a college-level epidemiology course. Focus group debriefing interviews discovered that students experienced key aspects of the PGD model, including that they enjoyed their time in the game world, had contstructive criticism about the game’s elements, and appreciated applying knowledge from the course in the game’s case studies. Although the study had several limitations, the findings provided valuable implications for instructors, and also for the game’s designers.

The study’s findings suggest that StO’s case studies served as an enjoyable outlet for students to apply their knowledge from course content. The PGD model, as well as a body of research, indicates that game enjoyment is an important factor for enhancing learning when playing serious games [5,6,7, 26, 40]. A specific aspect of what students enjoyed was the realistic nature of the case studies that were featured in the game. Making case studies that are realistic is a recommended best practice for case studies [41]. As such, since several experts have recommended the use of case studies when teaching epidemiology [42,43,44,45], professors may want to consider using StO as a supplemental activity when teaching the topic of investigating infectious outbreaks, depending on the steps of outbreak investigation that they are covering in their courses.

Although students enjoyed the case studies, they also provided suggestions to improve several aspects of the game’s elements, which could potentially enhance enjoyment of the game world. Based on students’recommendations, StO game designers may want to incorporate other types of media (i.e., videos) in place of readings, in order reduce repetition. In addition, game designers may want to consider making the second level of the game more distinctive compared to the first level, or perhaps only having a single level in the game. If desigenrs are not able to do so, instructors who choose to use StO may want to only assign the first level of the game, due to students’ perception of the cases’ repetitiveness, and also the game’s requirement to make a perfect score in order to advance to the second level.

The findings from this study also indicate that cheating StO may disrupt the game world and limit potential transfer effects. An inherent part of a game world is known as the “magic circle,” which includes the physical or non-physical space that a game takes place (e.g., the board in chess, the digital space in computer games) and also the rules that govern how the game should be played [46, 47]. According to the PGD model, players need to be immersed in the game and have an enjoyable experience to help achieve desired effects (e.g., changes in knowledge, attitudes, behaviors), and then hopefully transfer those effects to the real world [5,6,7]. In this study, students reported cheating by finding the StO answers online, and also by using “degenerate strategies” (i.e., finding weakeness in the game elements to exploit a short cut in winning a game) [47] when StO provided students with the correct answers to incorrectly answered multiple choice questions. By doing so, students indicated a disruption to the magic circle of the game world [48], thus limiting their opportunity to apply their knowledge of investigating infectious outbreaks. There are practical implications of this finding to preserve the game’s magic circle, including that game administrators remove the answer sheet from online, and perhaps allow players earn less than a perfect score in order to access the second level of the game.

An interesting finding from the study was the inverse transfer effect that happened. The PGD model states that the knowledge, attitudes, and behaviors that players’ gain in the game world can then be transferred to the real world [5,6,7]. Although StO was assigned at the beginning of the semester, most students completed it just before the due date, which was after the topic of outbreak investigation was covered in the course. Thus, the findings suggest that the game world could also be a space for players to reinforce what was originally gained in real world. Based on this study, professors who decide to assign StO as a course assignment may want to consider if they want students to use the game as a space to practice the knowledge gained in the course, or as a tool for students to gain that knowledge upfront as a way to prepare for outbreak investigation information that will be covered in class. A recommendation future studies that are based on the PGD model would be to measure potential inverse transfer effects that may be happening when playing StO or any other serious game.

The study’s limitations should be considered when interpreting the findings. Given the study’s qualitative nature and its convenience sample, the findings cannot be generalized beyond the participants. However, it is important to note that although qualitative research may not be generalizable, the findings may be “transferable” to classrooms with similar contexts [49]. In addition, the study was conducted during the spring semester of 2022, during the COVID-19 pandemic, which was discussed frequently and in-depth throughtout the course. The students’ experiences with COVID-19 could have influenced their engagement with the course material and StO. As such, their views of the game may be different than students who play the game outside of a global outbreak. Finally, there may have been bias in student responses, as their professor helped to conduct focus group interviews. It is worth noting, however, that students’ honesty in how they cheated the game may indicate that students were transparent about their experiences with the game.

5 Conclusions

Students reported that they enjoyed playing StO, and that it was effective for applying their knowledge of epidemiology course content on the process of investigating outbreaks. However, students found ways to cheat the game’s answers, and they recommended ways that the game’s elements could be improved. Instructors should consider using StO as a tool for teaching their students about investigating infectious outbreaks. Designers of the game should consider adapting the game based on students’ constructive feedback.