Learning in Doing: A Model of Design and Assessment for Using New Interaction in Educational Game

Abstract

To put into practice what has been learned is considered as one of the most important education objectives. In the traditional class, it is difficult for learners to engage in practices since teachers usually convey knowledge and experience by speaking or simple demonstrations, like using slides and videos. This brings obstacles for learners to apply what they have learned to solve real problems. Educational game with the novel input and out technologies is one of the solutions for learners to engage in the learning activities. It can not only effectively encourage learners to learn positively and vividly, promote learning interests and motivation, and enhance the engagement, but also improve their imagination, learning performance and other learning behaviors. This paper first discusses the learning theories related to educational game. Then a literature review is conducted by collecting data on the topics of assessment of educational game with new interaction. Next, we propose a model to guide the design and the evaluation of educational game due to the missing studies. This model underlies an educational game to foster the garbage classification learning with virtual 3D output and natural gesture input using Leap Motion, which is a tracking device of hands and objects. To analyze users’ learning behaviors and evaluate their performance and experience, we conduct an evaluation with 22 college learners. Results showed that the use of the natural interaction not only made the learning interesting and fostered the engagement, but also improved the absorption of knowledge in practices. Finally, a discussion of the challenges and future directions is presented.

1 Introduction

Educational game is a game designed for a primary purpose of pedagogy rather than pure entertainment, which could provide learners plenty of opportunities to put into practice what they have learned and to solve real problems. Educational game using novel interfaces and technologies offers rich user experience for learners to engage in the learning activities, encouraging learners to react positively and vividly, promoting learning interests and motivation, and improving their imagination, learning performance as well as other learning behaviors. Educational game has gained many interests from researchers in the field of educational technology and other fields related to this topic. Although educational game supported by new interfaces and technologies provides rich experience and practice opportunities, the design and assessment of the game is lack of guidance. Going a step further, the perspectives of design and assessment of educational game are more based on pedagogical points. This paper revolves around the topic of educational game, the terminology of which varies among researches. To make it clear the base of this paper, we list and describe these terminologies commonly used in the topic of educational game:

• (a) Educational Game

  Educational game has been widely used when mentioning a game with the purpose of enhancement of learning, like in [12, 15]. The work [12] derived the pedagogical requirements into three aspects, that is, integration with online education, adaption and assessment. It also discussed general design principles for educational games, including (1) choosing an appropriate genre, (2) adding assessment and adaption to the design, and (3) integration with an online environment.

• (b) Game-based Learning

  Game-based Learning also has been widely used in educational research on game, like in [7, 9, 14]. In the work [7, 14], game-based learning refers to the game enhancing knowledge and skills acquisition and involving the activities of problem solving and competition. Kirriemuir and Mcfarlane [7] proposed and emphasized two key themes on the development of games for education, including “Make Learning Fun and From Fun to Flow” and “Learning through Doing”.

• (c) Serious Game

  Serious game with educational objectives is studied by many researchers. Serious game, or called applied game, is a game designed for a primary purpose other than just entertainment [2, 10]. The “serious” defines the purposes of the game for education [3], scientific exploration, health care, emergency management, engineering, etc. The point of serious games explicitly emphasizes the additional pedagogical value of fun and competition.

• (d) Other Terminologies

  We also survey and examine the recent literature with regard to educational learning. The following terminologies have been used: design-based learning [6], educational computer games [6], digital game based learning [1], educational gaming [16], e-learning games [5], and educational digital games [12].

This paper focuses on proposing a model taking consideration of both pedagogical and technical perspectives to design and assess educational game. It surveys design guidance, and assessment aspects, and discusses the current issues. To solve the problem as we stated above, we propose a model considering both pedagogy and technical affordance of educational game. Based on the guidance of this model, we implement a game using gestures based interaction of Leap Motion with virtual 3D output and conduct an evaluation to assess the learning from aspects of performance, usability and user experience. Results showed that: (1) it was easy and interesting to learn, perform tasks and play in this educational game. (2) This game provided a good experience for learners and promoted positive emotions, and the game had an appropriate level on challenge. (3) Although there was no difference in performance between pre-test and post-test, the scores were higher in the post-test and it showed that participants experienced a positive emotional state when learning in playing. The use of the natural interaction not only makes the learning interesting and fosters the engagement, but also improves the absorption of knowledge in practices. A final discussion on the challenges and future directions is presented.

2 Design Principles and Assessment of Educational Game

2.1 Design Principles

To inform the design and evaluation of educational games that best meets learner’s needs, we propose a model considering both pedagogical context and technologies to achieve the learning benefits. This model provides guidance for the design and the assessment based on our previous studies and the survey. It describes the design guidance, game process and assessment dimensions.

As shown in Fig. 1, the design guidance includes three elements: pedagogical context, features of technologies and the game, roles of the game in learning activities. With regard to pedagogical context, constructivism underlies this model, which proposes that the contexts, activities, and social interactions in the learning environment promote the construction of new knowledge. In the pedagogical context, four key requirements are figured out when the model is employed to guide the design of the application, which are learning objectives, learning styles, learning activities and tasks, as well as motivated learning outcomes. The learning objectives are based on Bloom’s taxonomy [4]. The levels of immersion, interactivity and playfulness vary among the various technologies and the game. We also take into account the roles of the game, namely, the game as an application for education purpose, as an instructing tool, and as a learning environment.

Besides, we identify four elements contained in the educational game process: learn, practice (or doing), competition and collaboration.

Finally, in the module of assessment dimensions, we identify three types of measures: performance and usability, cognitive and affective states, as well as the social interaction. The rationale behind the division is based on that the measures are used more commonly in which field. In the cycle of design and assessment, a good performance, usability, positive cognitive states and learning emotions, as well as social interaction are the motivated outputs in the design guidance.

Fig. 1.

The model supporting the process from the design to evaluation of educational game.

2.2 Assessment

Assessment of educational game determines how to improve the game and measures whether a game achieves initial objectives. With regard to assessment, various measurement, instruments and approaches are applied to evaluate the game from aspects of performance, cognitive states, affective states and social interaction. As shown in Table 1, we list several researches and the assessment of educational games, including research paper and survey paper. A number of methods have been used to evaluate the educational game. We first categorize the dimensions that are taken into consideration to measure. Then, as shown in Table 2, we classify and list the measures corresponding to the dimension.

Table 1. Current assessment of educational game.

Table 2. The methods for measuring four dimensions of learning outcomes and their objectivity.

Assessment dimensions:

• (a) Performance

  Performance is commonly used to judge the learning outcomes directly, including measures like task completion time, test scores, reaction time, interaction time, and accuracy of interaction.

• (b) Usability

  Usability is the ease of use and learnability of a tool, device or an application. To measure usability, plenty of instruments and scales with different emphasis according to the actual application have been designed and developed. The work [11] proposed an extended TAM model to develop the measurement, including the scales of perceived playfulness, perceived ease of use, perceived usefulness, attitude toward using, behavioral intentions to use and actual use. What are required to emphasize is the satisfaction, playfulness and flow. These two items are tested commonly when measuring the usability of an application. Satisfaction means that the user feels satisfied with the interaction and the use of the application. Playfulness represents the state that the user perceives the interaction with the game and finds it enjoyable and interesting. Therefore, the satisfaction and playfulness under the dimension of usability merely refers to the items measuring the interaction other than emotion states. Besides, the flow represents the attention on the interaction with the game, which is placed in the category of usability. However, measuring the user’s attention on the learning contents in the game is classified in the category of cognitive states.

• (c) Cognitive states

  Cognitive state is one of the crucial factors determining whether the learning is successful. Cognitive load, engagement, attention are widely used cognitive states in learning.

• (d) Affective states

  In learning, affective processes are intertwined with cognitive processes. When playing games, the intense and diverse of emotions of the user are induced. In the field of Human-Computer Interaction, the user experience is valued increasingly, which focuses on measuring the user’s emotions when interacting with an application. In learning, affective dimensions of the learner’s experience have increasingly drawn the attentions of researchers and been considered as an essential factor fostering learning. In the study of educational game, affective states are studied including positive emotions (e.g., pleasure and satisfaction), negative emotions (e.g., confusion and frustration), and neutral emotions (e.g., boredom).

• (e) Social interaction

  Social interaction refers to the learner’s interactions with peers and teachers. When taking in consideration the social interaction, educational game should provide tasks as a bridge to link socially the learner and other persons involved.

3 From Design to Practice: A Study on Gesture Based Interaction of Leap Motion

In this section, we present a model supporting the process from the design to the evaluation. Based on the guidance of this model, we implement a game using gestures based interaction of Leap Motion with virtual 3D output and conduct an evaluation to assess the learning from aspects of performance, usability and user experience.

3.1 Design and Application

We propose an educational game with virtual reality output and natural gesture input using Leap Motion, which is a tracking device of hands and objects. Leap Motion could be used to assist students to explore in the virtual situations using hands. Based on this output and input technique, we design and develop an interactive educational game to foster the garbage classification learning. This game is set up with two sessions, namely the learning and the garbage sorting task. The learning session includes interactive gesture learning and garbage classification knowledge learning. In this scenario, students learn and practice the interactive gestures through sorting the given garbage objects. When the learner is satisfied with mastering the input technique, the interface of knowledge explanation will be presented. Students learn garbage classification knowledge via reading the contents on this interface. The garbage sorting task is designed to evaluate the performance and experience of students as well as make them engaged in this practice.

As shown in Fig. 2, this game process contains three steps: learn, practice and competition, which are realized into knowledge learning module, practice module, and garbage sorting competition module. In the garbage sorting competition module, the participants were asked to sort the garbage as accurately as possible, as well as considering the time consumption (Fig. 3).

Fig. 2.

The design and assessment of Game Garbage Classification based on proposed model.

Fig. 3.

The configuration, the implementation and the game process of Game Garbage Classification. (a) Leap Motion (b) Using hand to interact (c) Gesture to grasp (d) Learning contents (e) Competition

3.2 Evaluation and Results

To assess users’ learning, we conduct an evaluation with 22 college students as participants. We design a questionnaire including usability tests and emotion state measures, the usability questions are based on Technology Acceptance Model (TAM) [11], including perceived usefulness, perceived of ease of use, and perceived playfulness. We employ and adjust the scales to make the questionnaire appropriate for our game application. Our questionnaire is composed of perceived usefulness, ease of learning and perceived playfulness. The emotion state scales are proposed and created based on learning process emotional cycle model [8] by Kort et al. Three main questions have been studies:

• (a) In how far do users perceive a good usability when playing this game using gestures? In this question, we explore the users’ perceived of ease of use, perceived usefulness and perceive playfulness.

• (b) What are their affective states? In this part, we investigate their positive and negative states induced by the game.

• (c) Whether there exists significant difference between pre-test performance and post-test performance?

We recruited 22 student participants, including 7 males and 15 females, aged between 19 and 24. The evaluation began with an explanation of the user study. The questionnaire attached to the explanation contained three parts: the background information, the usability test in Likert scale form and the emotion scales. Then all the participants were instructed to learn and use this educational game. A system time logging tool was integrated to record the competition completion time of the game. Finally, participants finished the questionnaire. The following results are presented:

• (a) usability

  In this part, we investigate how users interact with virtual objects in the game using proposed gestures. The gestures include moving, grasping, and selecting objects. We asked participants to respond to the Likert questionnaire items with regard to perceived usefulness. Overall, the median scores were all above 4 with regards to text explanation, images used and navigation, except for the score of evaluating the pointing interaction at the beginning (lower than 3). With regard to ease of learning, we found that the median of all questions were all 4. Playfulness represents a relatively enduring tendency, three dimensions of which were defined by [11], including concentration, curiosity and enjoyment. A perceived playfulness was measured and the median were all 4. Results showed that all participants thought it was not difficult to perform, interact and learn in this game, and the interaction and contents were interesting.

• (b) Affective states

  To measure the emotions induced by the game learning, we employed the emotion sets created by Kort et al. and added one option of “don’t have such emotion” for each emotion set. The axis of emotions include “anxiety-confidence”, “boredom-fascination”, “frustration-euphoria”, “dispirited-encouraged”, and “terror-enchantment”. Results showed that positive emotions like hopeful, curiosity, interest were generated mostly for participants. A few of participants had negative emotions like indifference, dissatisfied. This indicated that this game provided a good experience for learners and promoted positive emotions, and the game had an appropriate level on challenge.

• (c) Performance

  Shapiro-Wilk tests of observed values (correct rate of performance) showed that data were normally distributed. Therefore, we employed t-test to evaluate if there was a significant difference in performance between the pre-test and the post-test. In the pre-test, participants were asked to sort the garbage in the questionnaire. In the post-test, participants played the game in the competition module. There was no statistically significant difference (p > 0.05) between the pre-test and the post-test on the correct rate of performance. This result indicated that the performance of playing garbage sorting when learning in this game and when learning without this game showed no statistically significant difference. The scores were higher in the post-test and it showed that participants experienced a positive emotional state when learning in playing.

3.3 Discussion

• (a) Question 1: In how far do users perceive a good usability when playing this game using gestures? In this question, we found that Game Garbage Classification based on proposed model provided a good usability and learning experience for users. It was easy and interesting to learn, perform tasks and play in this educational game.

• (b) Question 2: What are their user experience and emotion states? In this part, we investigate their positive affect and negative affect. Results indicated that this game provided a good experience for learners and promoted positive emotions, and the game had an appropriate level on challenge.

• (c) Question 3: Whether there exists significant difference between the pre-test performance and the post-test performance? In this part, we found that there was no difference in performance between the pre-test and the post-test. The result indicated that the performance of playing garbage sorting when learning in this game and when learning without this game showed no statistically significant difference. The scores were higher in the post-test and it showed that participants experienced a positive emotional state when learning in playing.

4 Challenges and Future Directions

Game with new input and output technologies are promising in education to meet the pedagogical objectives. In the last decade, educational games have been proved to be successful for fostering learning effectively. Researchers have done plenty of attempts to investigate aspects of educational games. However, challenges still exist when games are applied in education. We concluded four main challenges that the teachers may encounter when generating educational game to foster learning:

• (a) Challenges from the perspective of learning models

  Currently, few studies provide a practical and clear guideline or learning model to inform the design of educational game with new input and output technologies. The levels that the related theories explain how to design an effective game are vague and not clear. With regard to various game applications, the pedagogical basement behind the game and objectives are commonly described insufficiently. Besides, learning models should indicate the integration of educational game and the learning environment, that is, how to integrate the game into conventional class and smart classroom. The roles of the game can be defined either as a supported tool or extended as the learning environment. The organization of learning varies along the variation of the roles of the game. Building an appropriate learning model to inform the design challenges the teachers and instructors, and expects them to well employ learning theories.

• (b) Challenges from assessment of learning process and outcomes

  One of the benefits from educational games is improving the user experience in learning. In contrast to raising the performance of learners, the goal of the game is more on the enhancement of the learning process, and making the learners engage in learning at a high level. In some studies, researchers found that new technologies introduced in learning did not make the learning outcomes significantly different from the performance of learning in a conventional way. Therefore, design and develop effective assessment tools to measure affective states in the learning process in real time is efficient for stakeholders involved in educational game applications. The current assessment technologies supporting real time measures are mainly classified into two ways: physical measures and physiological measures. The former measures include physical features as detection input like facial expression and body gestures. The latter ones include brain activity measures like EEG, NIR and fMRI, and body syndrome detections. These measures are challenging due to the high cost and high complexity to build the related recognition system and to explain the results.

• (c) Challenges from device cost

  Make devices universal in the conventional class or smart classroom would lead to additional cost burden. New technologies and devices introduced in learning like Leap Motion are specific and additional for a class of learners. The Virtual Reality headsets and controllers also cost more than the pads which are used commonly in some schools deployed in smart rooms. If these interactive devices are expected to be widely used in learning, how to reduce the cost is required to be considered profoundly.

• (d) Challenges of reducing difficulties of teachers in using new technologies to create games

  Interactive educational games are being used at K-12 and higher education institutions. One of challenges that the teachers or instructors may encounter is that the difficulties of using new technologies to create games. To design an appropriate game requires mastering the technology or at least having the ability to be involved in the development team and exacting clear users’ needs. To ensure this, the teachers or instructors should be well trained to use the technologies and to solve the technical problems that they may encounter. To go a step further, teachers or instructors should have the ability to adjust the contents in educational game in an actual usage, which may increase the burden of theirs. To balance the cost of time and energy of teachers and their outputs challenges and it requires a practical solution for teachers to follow.