1 Introduction

In recent years, art education has been widely accepted by school educators as well as administrators, indicating its importance in providing useful guidelines for all aspects of the modern school curricula on developing students’ creativity, imagination, and aesthetics. Nevertheless, many educators also agree that art education allows students to develop the ability to express themselves by covering wide range of subjects and languages that hardly could be offered by other educational means [1].

Creativity is defined as the production of original, unexpected and useful work [2]. Many studies has reported that creative individuals were not only more productive and satisfied with their occupations, they were more flexible and enterprising within their groups [3, 4]. Following these studies of creativity, it seems that creativity and creative skills have now been regarded as highly important in almost every field of work and education in our competitive modern society [5]. In addition, The Organization for Economic Co-operation and Development (OECD) termed modern societies as problem-solving society and to cope with the advancement of technology along with the change in society, the necessity of creative problem-solving ability can be a key to become successful in this unpredictable world [6]. STEM education, which aims to boosting students’ interest and competitiveness in science, technology, engineering and mathematics, is a vital part of education policy of many countries. Many educators and researchers, however, argue that STEM is overlooking creativity-related components. They argue that STEM should be amended to STEAM where the letter “A” refers to the field of Art which will contribute in developing students’ creativity and imagination.

The learning through digital games is a wave of the future [7]. The curiosity, joy, and pride brought by game have been considered to be the key point of a successful education system and many studies have agreed on that and have reported the positive effect of game-based learning on promoting long-term user engagement and motivation. Actually, the idea of embedding education into entertainment was started in 1954 by Walt Disney. From then onwards, serious games have become more popular in a wide range of educational and training applications and their effectiveness have also been acknowledged in different fields, such as education, health, business, welfare and safety. Previous studies have also reported students who had gameplay in the classroom scored significantly higher citing its ability to engage and motivate learners in the educational process [8]. Therefore, it would certainly have positive benefits and also could impact the player in a real life context.

Microsoft Kinect is motion sensing vision-based sensor. It allows players to interact with their application using a natural interface that employs gestures thus eliminating the game controller. The invention of highly reliable and inexpensive sensors such as Kinect and Wii had boost up the development of serious games including dynamics activities, various sports, and dancing that required gesture movement. Hue reported the potential of Kinect as interactive technology and have discussed how it can facilitate and enhance teaching and learning [7].]. For an instance, players will be able to perform required dance tasks in intuitive manner which is better representative of the real world [8]. Hence, learning to create and appreciate kinetic aesthetics may be more important than ever for the development of the students, especially to those who focus more on studying in STEAM education fields.

This paper reports on the developing a serious game for art education, especially, opera the Mozart’s Magic Flute. The primary purpose of this game is educational and complementary for art class rather than just entertainment. Students play this game to learn and practice the characters of opera by following gesture recognition-based performance guidance for Mozart’s Magic Flute. The game is directed by student’s movements and gestures which are recognized the RGB-D (Kinect) camera. The context of the game consisted of acting, role playing, singing songs, dancing, doing art, and speaking. The player will be able to learn different activities and enjoy by following Mozart’s Magic Flute’s guidance in the game.

2 STEAM Education

STEAM is an acronym referring to the academic fields of science, technology, engineering, art, and mathematics. Up until now, STEM disciplines have widely been recognized as very important fields with regard to technology and workforce development. However, nowadays STEAM fields, is becoming more essential as arts-based education offers opportunities for students to develop their creativity and imagination.

Today we require professionals and creative thinkers who go beyond disciplines for solving multifaceted issue and complex problems [9]. However in order to do so, teaching and learning through the connecting the arts and sciences is essential. In addition, many historical cases shows that these connections are already innate for the most effective and innovative STEM fielders [10, 11]. In this view point, STEAM education has become an essential paradigm for creative and aesthetic converged teaching and learning in STEM and Arts disciplines [12]. Since 2011, Education Ministry of The Republic of Korea has accepted STEAM education for national policy. In addition, Korea Foundation for the Advancement of Science and Creativity (KOFAC) have been promoting the STEAM campaign and are operating the teacher training for STEAM education nationally. Also, different national organizations and professional communities of science and technology have agreed on the integrative approach of STEAM disciplines as a critical element for restructuring school education which will initially be able to stimulate a domain of the conscience to the students in the field of science and will draw induction for the field of science [13]. Hence, to catch up with increasing the importance of creativity, educators and researchers are accenting the role of Arts disciplines in STEAM education.

3 Mozart’s Opera Magic Flute and STEAM Education

The Magic Flute is an opera in two acts by Wolfgang Amadeus Mozart. The opera consists of many dimensions which makes it unique as a whole such as the human voice, the orchestral music, the visual arts (scenery, costumes, and special effects), the drama, and the dance. Hence all, the components of opera theater are included all STEAM fields.

Designing and setting-up stage of scenery is necessary to understanding technology, engineering, and mathematics. Characters, properties, and background (e.g. animals, flute, darkness and lightning,) are good topics of science classes. In addition, learning song and dancing are included not only fine art class but also good themes of science and mathematics class. (e.g. harmonics and newton’s laws of motion).

Magic Flute’s story is Sarastro who the wise priest of Isis and Osiris, has taken Pamina to the temple for the purpose of releasing her from the influence of her mother, the Queen of the Night. The queen induces the young Prince Tamino to go with Papageno in search of her daughter and free her from the power of Sarastro; Tamino accomplishes his end, but becomes the disciple of Sarastro, whose mild-ness and wisdom he has learned to admire. The prince and the princess are united [15].

4 Development of Game

4.1 System Architecture

Our system consists of a Kinect sensor, a PC, a TV Kiosk, a backlit projection screen, and a projector. In order to develop the game and implement the logics, we have utilized Zigfu, a programming development toolkit and Unity, a cross-platform game creation system [16]. Figure 1 shows the system architecture and user interface of this system (Fig. 2).

Fig. 1.
figure 1figure 1

STEAM: A framework for teaching across the disciplines [14]

Full size image
Fig. 2.
figure 2figure 2

System architecture and User Interface

Full size image

4.2 Game System Setting

The purpose of this research was to develop the educational and essential game by following gesture recognition-based performance guidance for Mozart’s Magic Flute for art education. The game was directed by student’s movements and gestures by recognizing the RGB-D (Kinect) camera. The context of the game consisted of acting, role playing, singing songs, dancing, doing art, and speaking. The player would learn everything by enjoying and by following Mozart’s Magic Flute’s guidance in the game. By participating in the game, it would enable young students in the range of 10 ~ 18 years old to learn and practice performing skills such as dance, acting, speaking progressively and repeatedly. In addition, it is possible that not only playing the game for students but also showing performance for audience at one time. The audience and their parents can watch the students’ performance on the simple stage when students, who looking at TV kiosk, are playing at sixth stage of rehearsal mode. At that time, projection screen is a stage background. Teaching artist guides and teaches the students playing the game (Fig. 3).

Fig. 3.
figure 3figure 3

Game system set-up, teaching artists, and students participating in the gesture-recognition-based game in exhibition (at 2014 Education Donation Fair, Il-san, Korea).

Full size image

4.3 Activity-Based Scenarios in Serious Game

The game was designed to be suitable for after school musical theatre classes in public schools, in which students learned Mozart’s Magic Flute. Generally, these education programs were designed for preparing the musical style performing art theatre. Students learn naturally by completing every level of the game through acting and dancing as characters of the Magic Flute. The prototype game has six levels. Each level was developed and designed to be suitable for many different classes (Fig. 4).

Fig. 4.
figure 4figure 4

The scheme of the art educational game, gesture recognition-based performance guidance

Full size image

The first stage. The first is introductory and includes a video and briefing about Mozart’s Magic Flute. In this mode, players can understand how to play this game and the full story of Magic Flute while watching the video lectures.

The second stage. The second is an acting tutorial which involves gestures by recognition-based game with the Kinect. The player assumes the role of Tamino who is a prince in the story. The story line of the game is that Tamino is running away from a huge serpent. The player acts as Tamino in the game.

The third stage. The third is designed for acting and learning science through the adopted STEAM Educational concepts. The scene of this level is a flock of birds flying in the sky. These birds are dancing and controlled by the sounds of Tamino’s magic flute in the story. The player also plays a bird in the game. The player’s bird-like gesture is recognized by the Kinect. Plus, the game teaches students how birds fly by showing the four forces of flight – weight, lift, drag, and thrust – affecting the flight of birds. This game concept is designed for STEAM education. Students learn the scientific knowledge while they are playing and acting like the bird in this game (Fig. 5).

Fig. 5.
figure 5figure 5

(Left) The scene in the second level, acting guidance. (Right) The scene in the third level, acting and science knowledge guidance.

Full size image

The fourth stage. The forth of this game is created for reading and speaking the character’s lines in the script. The character in the game and the player are interacting, and speaking following the instructions using arm and hand gestures. The fifth level is a dancing practice game using the Kinect. The player will learn dance moves through watching dancing videos. The arrangement of dance was used in the musical theatre (Fig. 6).

Fig. 6.
figure 6figure 6

(Left) The scene in the fourth level, script guidance. (Right) The scene in the fifth level, dance guidance.

Full size image

The final stage. The final mode is designed for preparation for a performance, Mozart’s Magic Flute. This final level is cumulative, and require knowledge and practice in the other four levels, which proceeded to successively. By playing the game, students will prepare for the performance of Mozart’s Magic Flute (Fig. 7).

Fig. 7.
figure 7figure 7

(Left up) A player participating in an acting tutorial. (Right up) A player participating in gesture-based script selection. (Left down) A player participating in acting practice and STEAM education mode. (Right down) A player participating in gesture-recognition-based dance practice game (at 2014 Education Donation Fair, Il-san, Korea).

Full size image

4.4 User Experience

We tested the gesture-recognition-based art education game with numerous participants at exhibition. In this part, we report the feedback from a wide range of participants who experienced the game.

The art education game was exhibited for four days at the 2014 Education Donation Fair hosted by the Ministry of Education and the Korea Foundation for the Advancement of Science & Creativity (KOFAC), held at Korea International Exhibition Center (KINTEX, Il-san, Korea) from September 18-21, 2014.

Many students visited our booth and experienced and played the art education game. The participants were mainly students, teachers, and university professors. The students’ age was from 7 to 18. During playing the game, students were passionate to approach the game-based learning and students were thrilled by following most of the components of an art game. Additionally, the students are actively participating in the game-based leaning and the outcome was remarkable. A few students initially looked abashed when they were performed acting and dancing but absorbed in it once they tried it.

5 Discussions

In our game, scenario of this game is based on Mozart’s Magic Flute in order to use it for opera performance class. In exhibition, we succeed in operation of the booth including this game with teaching artist’s teaching. This game doesn’t have the automatic evaluation system for students’ performance. So, we needed teaching artists for introduce this game-based learning class for opera to students. Their brief guidance and teaching is a component of game. Teaching artist’s teaching and helping are necessary to progress learning session. It is not enough for student by oneself to start and learn this educational game for Magic Flute. This is because we couldn’t make it using all story yet. Now, the game has six stages for learning and practicing the opera, Magic Flute. In the future, we will include the additional stage which has same format, and different story.

6 Conclusion

In this paper, we proposed the educational game by following gesture recognition-based performance guidance for Mozart’s Magic Flute for art education. The game was directed by student’s movements and gestures by recognizing the RGB-D (Kinect) camera. The game taught students acting, dancing, and STEAM education contents. This game is designed for learning and preparing the musical style performing art theatre. Students learn naturally by completing every stage of the game through acting and dancing as characters of the Magic Flute. The player would learn everything by enjoying and by following Mozart’s Magic Flute’s guidance in the game. This game could use not only standing alone a game for student but also performing a one-act play for opera theater class. We designed and implemented example system based on the concept of Opera theater class game, and tested with a number of participants at 2014 Education Donation Fair, Il-san, Korea. The feedbacks from the participants confirmed that this game motivated and increased students’ activities and learning. We believe that this game is suitable for after school musical theatre classes in public schools, in which students learned Mozart’s Magic Flute. Gesture recognition-based performance guidance for Mozart’s Magic Flute game by Kinect would be a great answer for the next generation’s performing art education. This effort and approach of this project would greatly help the students to understand the importance of art education.