Keywords

1 Introduction

Nowadays, the whole world is facing severe challenges posed by hackers, spammers and a large pool of attackers who are motivated by a variety of reasons. To prevent attacks from high-tech criminals, some organizations use a cybersecurity training tool to enhance its information assurance posture. However, creating an engaging training tool, which holds trainees’ attention sufficiently long to impart awareness, is a considerable challenge. Moreover, many forms of training fail the studied concepts because the training is by rote learning and does not require users to think about and apply. Generally, to increase motivation and engagement of users, many trainers has designed their training tools as a serious game. A serious game is the combination between game learning sciences and digital entertainment. Similar to simulations, serious games present a virtual reality of varying fidelity that allows learners to explore, experiment, or simply engage in learning. Therefore, our research aims at designing a training tool, which helps users enhance their cybersecurity knowledge, by using a serious game design model. Moreover, we also try to find a suitable gameplay which helps users understand the knowledge obviously. In this study, we use the Activity Theory-based Model of Serious Games (ATMSG) [1] and a story gameplay to design the training game. The details are shown in the next sections.

2 Theoretical Background

The learning effects of serious games in studies across educational contexts are inconclusive [10]. One of the recommendations is to ensure that game objectives and learning objectives correspond. It is really difficult to determine whether or not a game supports the learning of students because a learning objective and outcomes are unclear. There exist several models such as RETAIN [4], DODDEL [7], and the 5/10 method [6] which support to design serious games. However, in this study, we use ATMSG [1] to implement our game design idea. The ATMSG provides a comprehensive way to investigate, in detail, how a serious game is structured, and uses activity theory as the theoretical background.

In ATMSG, educational serious games are seen as used in the context of four activities: the gaming activity, the learning activity, the intrinsic instructional activity and the extrinsic instructional activity. There is a four-steps-approach that progressively guides the user in applying the ATMSG to the design or analysis of educational serious games. These steps take the user from a high-level understanding of the activities to the concrete components which implement those activities. The user identifies game components with the help of the taxonomy of serious game components. The details of the four-steps-approach are shown in [1].

In comparison to other models, ATMSG offers a more precise model for the analysis of the educational and gaming aspects of a game, allowing users to perform a more exhaustive decomposition of components as the game unfolds, and to link these components to the overall learning objectives [1]. Moreover, the design using ATMSG is expressed as tables and charts. That facilitates game developers to understand the designer’s idea. That is a reason why we choose ATMSG. In our design, we use a gameplay in story games as our main gameplay. We call it a “story gameplay”. The story gameplay uses storytelling technique to lead players to experience game events. Hence, we apply this gameplay into our game to support users in clearly understanding the given problem and its solution.

3 Design of Cybersecurity Awareness Training Game

Firstly, we introduce our game design idea. Our game design idea focuses on making a cybersecurity training game for users to learn and understand obviously cybersecurity concepts. We aim at giving them some reality situations of security attack. For example, “How to recognize phishing emails?”, “How to protect your information at public locations?”. In those situations, users will control their character and find the solution for given problems. We will lead users through a story step by step and they must make choices, which reflect their character’s behavior. After each situation, a system will evaluate users’ answer and gives them the right answer with its explanation. Following this idea, we design a game and describe it in detail by using the four-steps-approach of ATMSG. Critical points of each step are shown in the next discussion.

Step 1 - Describe the activities: In this step, we highlight the main aspects of activities to understand this game easily. Moreover, to make it convenient to the implementation, test and evaluation, we choose the topic “Cybersecurity awareness training in a university”. It not only makes convenient for us but also players (university students) to be familiar with given situations in the game and to keep cybersecurity concepts in mind easily. As a result, the main subject of gaming and learning activities is university students who will gain knowledge by experiencing directly real cybersecurity problems and learning to avoid and solve them. Our game demo aims at providing the training tool for students in our university (JAIST). Therefore, game resources such as images, learning content or story reflect the daily life at JAIST.

Step 2 - Represent the game sequence: In this step, we provide a diagram to present our game sequence by using UML. The game includes two main activities: “Problem Solving” and “Practice”. Both game activities have different gameplay. The Problem Solving will give players a new situation (story) to experience. Players must understand given situations and choose their behavior carefully to get a perfect score. If players choose a wrong answer, they cannot get a score in this event. An explanation and a right answer will be given after evaluation. While Practice gives players a quiz game, which contains questions about cybersecurity awareness. Players must answer them repeatedly and they will be punished if they choose a wrong answer. Whereas Problem Solving is commonly used to support players in understanding situations, Practice helps players remember cybersecurity concepts. Practice’s gameplay is easy to understand, so in our demo, we only give an example. On the other hand, Problem Solving is more complicated than Practice, and our research tries to examine the effect of story gameplay at understanding of users. Thus, we only focus on designing Problem Solving for our game demo.

Step 3 - Identify actions, tools and goals: In order to understand a game sequence easily, we are going to identify components that are related to each node in the game sequence. Those nodes are composed from their actions, tools and goals. We first choose the relevant components directly from the taxonomy of serious games (Table 11 in [1]), and fill them into the three layers of each activity involved (gaming, learning, intrinsic and extrinsic instruction). The extrinsic instructional activity is performed outside by the teacher or instructor in the context of the overall learning setting. Our game purpose is not to create an open learning environment for a teacher to teach anything they want. We provide a training tool which helps players in enhancing their awareness about cybersecurity problems in a specific organization (e.g. university, company, etc.). Therefore, in our case, we do not consider an extrinsic instruction, but we simply fill nine layers in total. Almost all components were selected based on main gameplay in order to clearly show designer’s ideas by the game actions, tools and goals. Therefore, this step can help game developers understand the designer’s idea thoroughly.

Step 4 - Description of the implementation: In this step, we provide a more detailed description of our implementation. We explain what is being done, using which tool, and with what purpose in each block of a table. We also explain how the use of such components and characteristics support the achievement of the entertainment and/or pedagogical goals of the game.

The combination of the four steps described above provides a comprehensive view of the structure of our game, from its high-level purposes and general characteristics to its concrete implementation. In this study, we used Novelty [8] to implement our design. Novelty provides simple methods to create our own visual novel game. A visual novel game [9], also known as a story game, is an interactive game. Typically, the majority of players’ interaction is limited to clicking to keep the text, graphics and sound moving on while making narrative choices along the way. Our detailed design and demo is available at [3].

4 Evaluation and Discussion

To collect players’ feedback on the demo, we created a survey which is available at [2]. The survey requires players to evaluate the demo in multiple aspects of the gameplay and educational value. They must rate on a scale from 1 (worst) to 10 (best). We had 10 participants who filled out the questionnaire. Six of them are at the beginner level in cybersecurity knowledge and other participants are at intermediate level. The details of the survey question and its results are shown in Tables 1 and 2. The survey questions are categorized into three groups. The first group (Q1–Q6) is used to test abilities of the game demo such as clarity, game length, content, and enjoyment aspect. It helps us figure out a comfortable game setting in official implementation. The second group (Q7–Q9) aims at evaluating learning purposes and the effect of story gameplay on helping players understand cybersecurity concept. The last question is used to estimate how much this game improves players’ motivation.

Table 1. Survey questions.
Full size table
Table 2. Survey results: average score per question.
Full size table

The responses to the survey show some advantages of the game. The first advantage is that the game is very easy to play for both groups of participants (Q4, the average in total is 9.6). Besides, the game is rated highly on two aspects that are what to do (Q1, the average in total is 8.7) and the story of the game (Q2, the average in total is 7.5). However, the interest of this game is rated not so high by beginners (Q3, the average is 6.0), but it is interesting for learners who are at intermediate level (Q3, the average is 7.3). The reason is that the game provides new materials to beginners more than intermediate learners (Q6, the averages are 7.8 and 7.0). That makes the beginners feel that the game length is quite long (Q5, the average is 6.5), so they become boring when playing this game. On the contrary, the game length is suitable for intermediate learners (Q5, the average is 5.5) so they feel more comfortable than the others. According to the results of questions from 7 to 9, we have high ratings on the aspect of understanding a problem situation, how to solve the problem and improving students’ understanding (Q7–Q9, the averages in total are 8.4, 7.8 and 7.6). However, those aspects are rated by beginner higher than by intermediate learners. The reason is that the game was designed to be a training tool for awareness raising for a beginner to obtain new knowledge. Therefore, this game is more helpful for beginner than intermediate learners. It seems that the game is not good at making enjoyment, so it improves the motivation of users not so well (Q10, the average in total is 6.7). In conclusion, the advantage is that the game can help the students improve their understanding of cybersecurity awareness. By giving a story, the game aims to lead players to understand cybersecurity problems and resolutions, even though it teaches an intermediate student nothing new in the material.

5 Conclusion

In this study, we presented the design of a security awareness training tool. By using the ATMSG paradigm, which offers a more precise model for the design of the educational and gaming aspects of a game, we present our game idea in detail. Moreover, the model ensures that game objectives and learning objectives correspond. The four-steps-approach helps game developers follow the idea of designer easily. To evaluate the suitability of the game structure and a story gameplay, we have built a game demo which is implemented by using the Novelty software. After the game was played and evaluated by university students, the result indicated that the game was rated well on the aspect of understanding cybersecurity problems and solutions. Therefore, the game which we developed by following this design help us avoid the rote learning of users when using the training tool. However, to improve the enjoyment of this game and the learners’ motivation, we should add more game actions and game elements.

Although the game can help users in understanding the cybersecurity concept, it is not good at making enjoyment for players because it is simple to play by clicking and reading. Therefore, our future works aim at fairly changing in a gameplay which increases interaction of users such as control character, click on items, etc. to make the game more interesting. Moreover, the balance between tasks and rewards in a serious game is very important for its entertaining aspect, so we aim to find a suitable learning content structure in order to improve enjoyment in the designed game by applying some game theories on measuring the game enjoyment such as game refinement theory [5].