Keywords

1 Introduction

Our understanding of the effectiveness of using games enabled by a portable marker less finger and hand position capture device (e.g. the Leap Motion Controller) to train children with Autism Spectrum Disorder (ASD) fine motor and motor gesture skills is very limited due to few empirical studies. The study described in this paper is believed to be one of the first to shed light on the suitability and degree of effectiveness of such a training approach to improve children’s motor impairments.

ASD is a neurodevelopmental disorder affecting 1 in 68 children in the US [1]; it is characterized by such core impairments as repetitive and constrained behaviors and lack of social communication knowledge and skills [1]. Of note, many children with autism demonstrate severe impairment with reciprocal interaction significantly impacting their joint attention [2] and collaborative interaction and play [3] which in turn decreases their ability to participate in meaningful learning, social and play interactions [4].

In this paper, we report on two pilot studies conducted in the classroom of a large Children’s Educational Development Center in a southern Chinese city for children with ASD. To the best of our knowledge, it is one of the first attempts to study the acceptability, usability and possible benefits of finger- and hand-gesture enabled interactions for special education.

2 Related Work

Few previous empirical studies have provided us with sufficient understanding of both the acceptability and usability of the finger- and hand-gesture based motionless games/applications for individuals with ASD. However, two indirect lines of prior research are relevant to our current study.

Recent studies revealed that the under-explored motor abnormalities are also prevalent in ASD [5] and are believed to interfere with the development of adaptive skills [6, 7]. The motor abnormalities, known as associated symptoms [8] may include problems with gross and fine motor skills [9, 34], skilled motor gestures [6, 10] and motor learning patterns [11]. However, children with autism often demonstrate strong visual perceptual abilities for objects as they have been found to be drawn to consistency of visual stimuli (both static and dynamic), matching and discriminating salient features [12–15, 33], while conversely there have been impairments noted in the visual perception abilities for facial features [14–16].

Although a wide variety of games and applications have been developed to improve individuals’ social communication, eye-hand coordination and cognitive skills etc., fewer games and applications target the individual’s motor impairments.

2.1 Benefits of Motion-Based Games and Applications

Several recent studies on the possible advantages motion-based touchless games provide for children with ASD [17–20] have been found. In particular, these studies revealed that the whole-body interaction enabled by Kinect games can enhance children’s motor skills and social behaviors [19, 20, 35], reduce their distress and increase emotion-releasing abilities [17]. Custom-made Kinect games can further be used for attention and cognitive skills training [18]. The field studies conducted in [17] indicate that motion-based gaming can promote cognitive learning and attentional skills, supporting both selective and sustained attention.

2.2 Use of Gesture Games and Applications for Fine Motor Skills Training

In addition to Kinect, Leap Motion, an increasingly popular touchless hand and finger motion tracking device, has been examined to demonstrate its accuracy in measuring hand movement in pointing task among healthy adults [21]. Due to its extreme portability, Leap Motion offers a greater advantage than Kinect for training on fine motor skills, such as that needed by individuals with ASD. A full understanding of the benefits of motion-based games (including both whole-body and hand-finger gesture-based interactions) requires examining that of the latter. However, little previous empirical evidence directly addresses the issue, especially in China where technology-based application use is rarely used in both the school and at home [22].

Following this path of research, we have seen so far one attempt which includes a series of Leap Motion-based games to train children’s fine motor skills empirically studied in a special education center in Beijing [23]. Results showed that the ‘innovative’ interaction poses bigger challenges—children have difficulty adjusting their hand gestures in a 3D space.

3 Experimental Design and Assessment

3.1 Study One: Drawing Game

The Game.

A simple drawing game was implemented using C#. Players can use both hand or finger gestures to draw on the canvas (Fig. 1). In order to assess children’s attention and engagement level and their emotional state, we made the game rules very simple by supporting only two simple actions—drawing through hand-finger air gesture (depicted by the boy in Fig. 2) and cleaning via a circle gesture.

Fig. 1.
figure 1

The game entry user interface of the drawing game

Full size image
Fig. 2.
figure 2

During testing, a grandma helped her grandson with ASD drawing in the canvas

Full size image

Participant.

Five pairs of boys and their family members were recruited by the Center to participate in this game. (In the Center, close to 90 % of the children are boys). The boys participating in this study have comparable (similar) intelligence and abilities (M = 4.8, SD = 1.8), referred to as C1 (the oldest, 8 years old), C2, C3, C4 and C5 respectively.

Experimental Conditions.

One purpose of this experiment was to examine whether finger-gesture based manipulation would bring more fun and usability (in terms of ease of use) to the children. Therefore, we tested the game in the school with which children feel familiar and comfortable. The game was projected onto a wall via a projector; and to avoid children’s curiosity towards the tiny Leap Motion controller, we ‘hid’ the controller into a box leaving the interaction area open (as depicted in Fig. 2). Each family member/child pair was given a maximum 15 min of play in order not to fatigue the children [17, 18] due to their young age.

Measures.

Qualitative data, such as the level of enjoyment and engagement, and the degree of attention on screen objects were collected through observations, commonly seen in the literature [17, 24, 25]. These variables were among the behavioral and emotional signals examined in previous research [17].

Experiment Procedures.

Before the experiment, the head teacher was given the Chinese translation of the game including a short description of how to play it. Parents were informed of the purpose of the experiment, experimental procedures and their rights. Each child entered the testing room with their parents (except for one with his grandmother, C3). The developer first introduced the Leap Motion device, and demonstrated how to draw with two finger-gestures. Then, the parent/grandmother-child pair interact with the game together. After the testing was over, structured interviews based on the one developed in [26] for a Chinese family were conducted at the end of the play session. The aim was to obtain information on the child’s behaviors and preferences, and phone- and computer-game uses at home.

Results and Discussions.

In contrast to previous findings that children were mostly reluctant to touch a singing plant [22], both children and their family members showed high engagement in the drawing game; their attention level was very high during the entire game-playing session. All children showed high levels of sustained attention, and most of the time when they were playing the game with the help of the parents, their eyes remained focused towards the visual stimulus in the game. Both children and parent/grandma’s emotional states were enthusiastic. Structured interview with all family members indicate the high satisfaction of the family with the game, especially regarding the Leap Motion device since it was their initial encounter with such a ‘magic’ controller. Overall, for all of the parents, it was the very first time they were aware of the existence of such a device and they all expressed amazement at such an innovative way of user interaction. While all children had been given opportunities to play with mobile phones at home, their family members claimed that none of the games and applications are specifically designed for their children; in fact, according to the teachers at the Center, this was the first-ever such educational application. An earlier game, the Yuudee (Little Rain Drop), launched in May 2015 [27], lacks the personalization mechanism where teachers can adaptively change the contents based on children’s responses and conditions, therefore, it has not gained popularity among special education schools in China. With this said, the teachers at this Center were very open to our drawing game. However, it was observed that the children had difficulty manipulating and using their fingers; we designed a second more portable web application which utilizes hand-gesture.

3.2 Study Two: Zoo and House Game Study

The Application.

The improved version was implemented using JavaScript and can be accessed at: http://www.tmywk.com/ as a web-enabled application (a Leap Motion gesture controller is required). Currently, there are two available scenes: a zoo and a house. When the child interacts with one screen object (an image), its Chinese name (a word) will be shown accompanied by a sound file to spell out its name (see Fig. 3). This learning process is commonly used in the famous Lovass model of Applied Behavior Analysis (ABA) in which individuals with ASD are required to pair a word with a corresponding image (through the use of familiar materials and intermittent rewards with an aim to enhance children’s vocal language understandings). Since it is well known that children with ASD frequently experience sensory overload and may be averse to excessively loud and high pitched sound [28], the auditory sound was presented at a moderate volume, and the tones fall into the frequency range of human speech. None of the children in our experiment demonstrated any observable startle response; instead, the children’s attention to the screen objects was continuously dedicated. Teachers also voiced no complaints about the sound and the tones.

Fig. 3.
figure 3

The zoo interface in the zoo and home game

Full size image

Participants.

Five boys with the same intelligence and abilities participated in this study (M = 6.3, SD = 2.4), referred to as C1, C2, C3, C4 and C5 respectively; four boys were reported to have interest in playing computer games (C1, C2, C3, and C5). Their head teacher recommended the children because they were informed of the nature of the game beforehand (as required by the Autism Center).

Experimental Conditions.

One of the purposes of this second experiment was to examine whether hand-gesture based manipulation would bring more fun and usability (in terms of ease of use) to the child, therefore, we tested the game in the school again with which children feel familiar and comfortable.

Measures.

Qualitative data included the level of enjoyment and engagement, the degree of attention on screen objects, and additionally, the degree of independent manipulation by children were collected through observations (commonly seen in the literature [12, 17, 24, 25]). These variables were among the behavioral and emotional signals examined in previous research [17]. Although the conclusions derived from these data cannot provide a convincing picture of the design patterns for such a game, they are sufficient to inform the designers of whether the new design managed to serve its intended goals.

Experiment Procedures.

Each child entered the testing room with either the home-room teacher or the head-teacher. The teachers had been informed earlier about the nature of the experiments (with a Chinese translation of our game). The game designer provided brief instructions to the children on how to play the game through a short demonstration (lasting less than one minute). After the instruction, the children were given the time to play the game. Prior to this experiment, and unlike some other testing in which a given time is allocated, Chinese children had seldom been exposed to such gesture-based game and technology-enabled games before [22]. This is only the second time such a gesture-based application has ever tested in the ten-year period that the Center has been educating children. Therefore, free-play was administered in order to assess the usability of the application. After each play round, teachers were interviewed to provide their perspectives on the children’s noticeable behaviors and the overall benefits of the game.

Results and Discussions.

All the children showed high enthusiasm while playing the game, and with very minimum training needed (see Fig. 4 on one testing moment). Among the five boys participating in the game, three older boys C3, C4 and C5, were extremely attentive to the game, played it with ease, and showed high sustained attention [17]. Among the qualitative data collected through the analysis of the videos and observations during the testing, the level of enjoyment and engagement, the degree of attention on screen objects and the degree of independent manipulation were all very high. Compared to the similar observations in Study One, the children needed less assistance to activate the device and play the game. Since the game was also projected into the wall, two younger boys C1 and C2, demonstrated high interest during game-play as they were observed to stare at the wall display instead of the computer screen. Of note, child C4 stopped working in the application when it was time for him to attend a class; however he later went back on his own to the game during the class break. According to the homeroom teacher this was the first time he had independently initiated participation in an activity.

Fig. 4.
figure 4

During testing, a boy was observed interacting with the application at ease

Full size image

In summary, all the children were able to show sustained attention to the on-screen stimuli. They demonstrated high skills in mimicking the developer’s demonstration. The results lead us to believe that the usability of such a hand-motion based touchless application is very satisfactory. To our surprise, the teachers’ feedback and reactions in Study Two were much more positive when compared to those from Study One: they revealed that although it is the first time they had ever seen such a type of game tested in their center, it was beyond their expectations to observe the children’s reactions to the application as well as their quick and deep immersion in it.

Unlike previous studies [17, 18] examining the learning benefits of such motion-based touchless games relative to children’s attentional skills, our work attempts to understand the acceptability of such games (Study One and Two) and the learnability and usability of it (Study Two). It is unclear what design rules would best be formulated in both gesture-based and multi-modal interactive applications since some cultural issues unique to this population have contributed to the overall acceptance of these applications. In addition, children seldom receive strengthened in-home intensive training in parallel to their school-based ABA training programs. Chinese parents largely rely on the Autism Center to undertake all the interventions. However, our two experiments have drawn attention from both the teachers and parents who have encouraged us on in further development of such games.

4 Lessons Learned

Our current understanding of the possible benefits of motion-based application/games for individuals with ASD are extremely limited, partly due to the small number of empirical studies, particularly in China. Hence, overall, the two studies conducted and reported here offer a valuable first glimpse into the acceptability and usability of motion-based touchless games for Chinese children with ASD.

Results of our two pilot studies reveal practices and perspectives on the use of finger- and hand-gesture enabled interactions that add depth to our previous understanding of such use. In particular, while mid-air finger-gesture interaction is challenging for younger children with ASD, hand-gesture interaction offers more fun and playability in the learning process. However, we expect some resistance from parents and special education teachers who may remain suspicious of the use of such games/applications.

5 Conclusion

The research presented here provided an opportunity to learn about how cause and effect contributes to motor skill acquisition and learning. In particular, the researchers observed that when the child is engaged in a motivating task he will practice the required movements, and through this practice develop motor control accompanied by true learning to build a foundation and provide carryover into more complex movement patterns. Additionally, and perhaps more importantly, when the parent/caregiver observes the child’s ability to activate and sustain engagement in a task, the parent is more likely to carry over the practice at home further solidifying the child’s skill development. This training protocol is beneficial for both the teachers and the parent/caregiver, as they recognize the ease of use when provided with the initial instructions, reinforced with observation of the child’s focused and sustained engagement in the activating the game and attending to the visual stimulus that results from the child’s motor movement (either the full hand gesture or the more refined 1 or 2 finger gesture).

Teachers can build on the visual motor abilities the children gain through the use of the games provided here to integrate learning concepts (examples include matching colors/shapes; categorization of images and finally matching words to pictures to promote reading and language acquisition). Finally, children can develop social skills through turn taking and collaborative play with the computer games. Developmentally the child is given the opportunity to simultaneously link the kinesthetic and visual systems together. For example, the child will move his hand or fingers and watch the visual stimulus that results from the movement. This is foundational for numerous necessary eye hand coordination tasks for daily living skills (buttoning a jacket, opening containers/book bags, folding clothes) and school/learning skills (using scissors, arts and crafts, drawing, letter formation and handwriting). In the United States, occupational therapists who work with children with ASD, use play and school based activities to promote the children’s bilateral hand coordination and eye hand coordination (visual motor integration). For example, for handwriting, children must hold the paper with one hand and write with the other. Foundational eye hand coordination activities, such as the finger- and hand-gesture enabled interactions with fun and engaging computer games and programs, provides children the opportunity to develop these abilities. Additionally, multiple studies have reported the positive impact of the use of a multi-sensory approach to teach children with ASD and to promote social interactions and engagement [29–31]. Through the use of motion based visual interactive computer games and programs, the child with ASD benefits from a multisensory approach to learning (the motor act of activating the device and the viewing the visual stimulus).

For a nation where even the prevalence study has only started two years ago [32] and government-funded special education is both expensive and limited, in terms of both quality and quantity, affordable and portable technological solutions such as those outlined here could offer great helps to families living with ASD. We hope our study sheds light on the degree of benefits of such hand and figure motion control games for Chinese children with ASD, and also provides preliminary data to call for more HCI community engagement in such research in China.