Children who have been diagnosed with autism tend to perceive human social behaviours as complex, difficult to interpret, and potentially overwhelming. As such, they might subsequently withdraw from social interaction because they can find it difficult to communicate or socially interact with other people. If they do not understand how to practice proper social interaction and/or communication, these children can face many difficulties later in life . However, because children with autism enjoy playing with mechanical devices, particularly robots [39, 42], one of our laboratory’s projects, AuRoRA, has pioneered the use of robotic toys as therapeutic and educational aides. These robotic toys are designed to teach children with autism basic social skills to help them to communicate and interact with others, such as turn-taking and imitation . Since the project’s beginnings in 1998, many encouraging results have been found [13, 15, 64]. Specifically, discoveries have been made regarding how children with autism interact differently with other people than with robots, and how interactions among children with autism can be successfully mediated by these robots [47–49, 65].
Research has shown that humanoid robots, whether used as toys programmed to dance to specific pieces of music or remotely-operated robotic “puppets”, can promote imitative free-form play among pairs of children with autism . Additionally, such robots can also foster triadic interactions among themselves, a child with autism, and a human experimenter . Such behaviours are necessary in order for children to engage in social play, a form of play in which children with autism have significant difficulty participating due to the social impairments that are characteristic of their disorder . Previous work has shown that children with autism can engage in free-form, unstructured forms of social play known as associative play, and our earlier research has suggested that children with autism can engage in a more organized and complex form of social play, known as cooperative play, with robots in the context of an after-school club . However, it has not yet been shown whether children with autism can participate in cooperative play, specifically when this form of play is implemented as a simple, dyadic, collaborative video game. Furthermore, it has not been shown whether playing cooperatively with humanoid robots has any effect on collaborative play skills among children with autism when compared to playing with a human being. This article presents a novel experimental setup consisting of an autonomous humanoid robot playing a collaborative, dyadic video game with children with autism. Furthermore, the article also describes our evaluations of the childrens’ experiences using our setup, our results from data analysis of the interaction games, and our suggestions for improving the systems which comprise our experimental setup and the ways in which they are used.
The remainder of this article is structured as follows: Sect. 1.1 discusses work related to key topics in this article, and Sect. 1.2 describes the approach taken towards the implementation of the experimental setup, the research question which motivated the design of our experimental setup, and our expectations for how children would respond to our setup as well as the trends we would expect to see in the data gathered. Section 1.3 discusses the participants of our study and our reasons for choosing them, and Sect. 2.1 describes the methodology behind our pilot study. Section 3 explains the logic and design choices of our dyadic video game and autonomous robot, as well as hurdles we had to overcome in their implementation. Section 4 lists the kinds of data gathered in our study and explains why those data were chosen. Section 5 discusses how the data was analyzed as well as which trends were observed in it, and Sect. 6 offers our evaluations of the children’s experiences with our experimental setup, as well as interpretations of the study’s data findings and the implications that this will have on future work. Section 7 summarizes the article and, based on the lessons learnt from the pilot study, outlines a more thorough experiment to be conducted in the future which will use an improved experimental setup. Individuals whose assistance and efforts have helped make this work possible are listed in Acknowledgements.Footnote 1
Autism is a lifelong developmental disability which is characterized by deficits in social interaction, impaired social communication, and restricted interests as well as stereotyped behaviours . Although these impairments can appear in a variety of forms and degrees of severity among the individuals diagnosed, they will generally impair the person’s ability to understand, relate to, and socially interact with other people. Children can manifest these symptoms through specific behaviours, such as displaying positive affect in social settings significantly less often than neurotypical (non-autistic) children , displaying positive affect while looking directly at another person significantly less often than either neurotypical or mentally retarded children , initiating joint attention using pointing (the selection and focus of gaze on the same object as someone else) far less than other children , and having difficulties in initiating and sustaining social play . By observing how often these behaviours occur, one can quantify the quality of specific social interactions among those with autism .
Because children with autism particularly enjoy playing with computers and electronic devices , some researchers have studied how video games can be used to help these children. Because horizontal visual displays promote more group work and cooperation than vertical ones , some researchers have used video games displayed on horizontal interfaces to promote collaboration and social interaction among children with autism. Piper, O’Brien, Morris, et al developed a game called SIDES (Shared Interfaces to Develop Effective Social skills) using a Diamondtouch table display to detect and distinguish the hand-table contact of up to four players. In evaluating the game, the researchers found that while one group of children with autism played more cooperatively when the game enforced its own rules of turn-taking and piece ownership, another group played best when the game’s rules were not enforced at all . Bauminger, Goren-Bar, Gal et al also developed a collaborative electronic interface based on a Diamondtouch display known as StoryTable, in which pairs of children could create different stories by jointly touching and dragging items on the display surface. Three pairs of children diagnosed with high-functioning autism played with this interface multiple times per week over the course of three weeks, and the researchers found that after participating in all of the play sessions, the children displayed more social behaviours such as making eye contact, positive affect while making eye contact, and sharing emotions than they did beforehand. . Additionally, the children displayed fewer stereotypically “autistic” behaviours while playing with the StoryTable than they did while participating in other activities, and also spent more time playing social games, whether simple or complex, as well as less time playing in parallel with another child after participating in the study . In a similar study, children with autism played with digital jigsaw puzzles on a Diamondtouch table which could be programmed to require either cooperative or individual touching and dragging in order for pieces to be moved around on the board. After pairs of children with autism repeatedly participated in each of the game’s play styles, it was found that the children exhibited more coordinative moves, more moves in general, and had greater proportions of simultaneous activity while playing the puzzle game cooperatively than when the children played separately but in parallel .
Since it was first suggested that robots positively affect the social interactions of children with autism , many researchers have studied this phenomenon in more detail. In addition to the above mentioned projects, Fasel and others used simulated systems and robotic ones to study normal and abnormal development of joint attention in infants with and without autism . Later, the small robot Keepon, which was developed by Kozima, Nakagawa, and others, showed that it could establish triadic interactions among itself, a young child with autism, and either another child or the autistic child’s parent/caregiver [33, 34]. In order to improve the diagnostic methods for autism among children, Scassellati worked on open-loop robots and systems for automatically tracking the movement of a child, measuring their direction of gaze, and categorizing the prosody of their voice [54, 55]. Feil-Seifer and Matarić found that children with autism socially interacted more with robots that directly responded to their actions than they did to robots that behaved randomly or were completely unresponsive [19, 21]. Michaud and Théberge-Turmel built robots with many different designs (a ball, an elephant, etc.) and examined how children with autism interacted with them in order to see which one was both played best with and was most helpful for developing social skills . Similarly, Kim, Leyzberg, Short and others had children with autism interact with Pleo, a dinosaur-like robot, as well as interview with an adult human, and found that the children were more socially engaged while interacting with the robot .
Social play is an effective method through which children learn about social interaction. Research suggests that since children with autism find it difficult to play with other children as well as participate in pretend play, it is particularly difficult for them to learn about social interaction . As such, a great deal of work has focused why children with autism show difficulties in engaging in the abovementioned two forms of play. Wolfberg and Schuler discovered that children with autism find it easier to participate in symbolic play with other neurotypical children through assistance from external support structures, such as a helpful teacher . Similarly, it was discovered by Charman and Baron-Cohen that more children with autism could engage in pretend play, in which a simple object is substituted for another more complicated one, when they were assisted with appropriate prompting from the experimenters .
Drawing on studies about how groups of people can constructively work together, researchers of human-robot interaction have examined how heterogeneous groups of robots and other people can best collaborate with each other. Fong, Thorpe, and Baur found that participants in their experiment were able to accomplish more tasks when they collaborated with many different autonomous robots than when they manually controlled the robots’ behaviours . Hinds, Roberts, and Jones studied how different appearances and status roles of robots affected the task-solving performance of different collaborative pairings of humans and robots . Drury, Scholtz, and Yanco described an awareness framework for different collaboration scenarios of humans and robots, and were able to re-examine specific failures among teams of humans and robots in terms of various awareness deficiencies . Sidner, Lee, and Lesh studied how conversational gestures and gaze patterns can be used by robots to better engage people in collaborative, socially assistive interactions .
This present study incorporates ideas from the many different research areas mentioned in this section. Specifically, because research shows that there are specific social behaviours that children with autism will perform less often than non-autistic children due to their impairments in interacting and communicating with others, the experimental setup described in this article utilizes the frequency with which these behaviours are displayed to determine the change in social engagement between a child and their play partner over the course of different play sessions. In our setup, one of these play partners is the autonomous humanoid robot KASPAR which also reacts to specific forms of communication, as research shows that children with autism are particularly socially engaged when interacting with robots that respond to the children’s behaviour. The play sessions focus on social, cooperative play, as studies have shown that the particular difficulty which children with autism have with this style of play may further hinder their development of basic social skills, and the video game used in the play sessions uses a horizontally-oriented screen because it has successfully been shown to foster cooperative play among participants with autism. Furthermore, the robot’s behaviours, its role in the play sessions, and its degrees of expressiveness were designed according to findings from related research on successful collaboration between humans and robots.
Purpose of Experimental Setup, Goal of Study, and Expectations
While our earlier research suggested that children with autism were capable of playing cooperatively in the context of an after-school robotics class, its experimental setup was designed for children who were relatively high-functioning and therefore capable of interacting with others in a group setting . Because this earlier design was limited in terms of the variety of children with autism who could benefit from it, we designed a new experimental setup that would still be engaging for the children while focusing on cooperative play and robots, without requiring the participants to have as developed sets of social skills. Furthermore, we designed this form of cooperative play and the set of robotic behaviours such that the experimental setup could be used in studies that would more easily and readily compare the children’s degrees of social interaction with other children to their social interactions with a robot.
The purpose of designing an experimental setup involving an autonomous, humanoid robot playing a dyadic cooperative video game was to have children with autism become engaged in both the cooperative form of play as well as their social interactions with the other player. This supports the intended usage of our experimental setup in a number of ways.
Firstly, while children with autism have difficulties in participating in social play because of their impairments in socially interacting and communicating with other people , our setup instantiates a social play setting as a cooperative video game in an effort to make children with autism play with others as well as interact with them. Because such a setting uses clearly structured and codified forms of interaction as well as electronic components in the forms of game controllers, the cooperative video game is intended to be both more appealing for children with autism as well as a simpler interactive context in which they can participate. Furthermore, because successfully playing our cooperative video game would require participants to socially interact, we also felt that the children with autism would socially interact with others simply out of a desire to accomplish tasks in the video game.
Secondly, children with autism have shown both increased amounts of social engagement while interacting with robots than they have while interacting with people , as well as a preference for interacting with a humanoid robot than with a zoomorphic robot . As such, we felt that when the children with autism were presented with a humanoid robot as a play partner, they would become more engaged in both their cooperative play activities as well as their social interactions.
We therefore expected that even though children with autism do not often participate in social play because of their social impairments, the children would both participate in and enjoy playing the cooperative video game because of its clearly-defined rules and the simplified nature of social interactions within its context. Furthermore, the nature of the cooperative video game would also help the children to socially interact with other individuals, even if they only wanted to accomplish tasks in the video game. Additionally, we also expected that the humanoid robot would serve as a catalyst to make children with autism become more socially engaged and socially interactive while playing the cooperative video game.
Novel Pilot Study
While the goal of this study was to evaluate and test our experimental setup using children with autism, the additional aim of gathering data in this study’s equipment test was to practice analyzing the kinds of data that we expect to find in future studies, in which we will use objective measurements to determine whether dyadically collaborating with a humanoid robot while playing an explicitly cooperative game would change a child with autism’s collaborative dyadic interactions with a human in the same context. This is a novel and interesting aim for a number of reasons.
Firstly, previous research has shown that when used as social mediators, robots can help children with autism to interact with other people, including other autistic children, in novel ways [20, 33, 34, 47–49, 65]. These earlier studies compared the children’s interactions in the contexts of the experiments with second-hand reports of the children’s earlier interactions in different settings. In addition, such studies have mainly focused either on single autistic children interacting dyadically with a robot or on single children triadically interacting with a robot as well as their parent or carer. However, no earlier studies have used the same experimental setting to compare dyadic interactions of single autistic children and a human adult with the dyadic interactions of the same children and a humanoid robot.
Secondly, the abovementioned earlier studies examined how children with autism interacted and played in open-ended, exploratory settings with robots. According to Parten’s research on play , we can classify some of the forms of play in these studies as parallel (two children with autism play in their own ways with the same robot at the same time, either without acknowledging each other or by acknowledgment without communication [48, 65]), some as associative (a child with autism imitates a robot and communicates with its human adult controller [29, 46–49]), and on a few occasions, cooperative (two high-functioning children with autism spontaneously interact and communicate to organize a game together with a reactive robot , or a child with autism plays a two-player game with an experimenter while interacting minimally with them ). In these studies as well as others, there have been few cases of the children participating in cooperative play. This is not surprising, as that specific form of play requires frequent communication and interaction among its participants, and by definition, children with autism have great difficulty with these social activities. However, this study is novel because it asked autistic children to participate in cooperative play by continually communicating and interacting with both a human and a robot. Additionally, although almost all of the previous studies involved children with autism playing with robots in semi-organized ways without any specific goals, this study asked multiple autistic children to play in an organized, collaborative manner with a robot to achieve a specific, common goal.
Thirdly, few pilot studies on autonomous social robots for children with autism actually evaluate the impact of their complete systems on members of their target audience, in the sense that they might not test real autonomous robots in the presence of children with autism while also observing changes in their behaviour. Instead, pilot studies might utilize virtual robots in place of real, physical robots , they might only ask neurotypical children to participate in their pilot studies instead of children with autism , or they might only test to see whether their robotic system is capable of behaving successfully with children with autism while also observing whether the children can interact with their robot . In contrast, this pilot study observed and tracked social behaviours of children with autism over multiple sessions of interacting with a real, fully autonomous robotic system.
Because children with autism have difficulties with generalizing behaviour and skills between settings , we wanted to design the autistic children’s interactions with both the robot and the human adult to be as similar as possible in order to ensure the highest likelihood of skill transference between the two settings. To this end, we used a humanoid robot known as KASPAR  (see Fig. 1) and programmed it to play with an autistic child using actions, gestures, and spoken phrases similar to those used by the human adult participating in our study. KASPAR is a minimally expressive robot that has a simplified form of human-like features and behaviours, thus allowing children with autism to explore social interaction in a safe and predictable environment in which they feel comfortable (see Sect. 3.2 for details). Drawing upon the deliberately strong similarities between the behaviours of the human and the robot, as well as earlier studies’ claims of autistic children’s increased displays of social engagement with robots, we expected that, in future studies, the autistic children’s social engagement and displays of positive affect during a play session with KASPAR would partially transfer over into a subsequent play session with a human adult. Furthermore, we also expected that such objective measurements during a subsequent play session with a human adult would be greater and more frequent than those during a play session which preceded playing with KASPAR; in short, the children would play more collaboratively with a human partner after having played with the robot than they did beforehand.
Six children with autism participated in this preliminary study from a local school for children with special needs; none of these children had interacted with KASPAR or played our collaborative game before. We specifically did not include a group of neurotypical, or non-autistic, children as a controlling factor in our study. This is because we did not want to distinguish or contrast neurotypical children with autistic children, as our research group is more interested in studying robot-assisted play as a tool for autism therapy than studying the nature of autism as a psychological disorder. We therefore adopted an approach commonly used in the field of assistive technology and focused on our particular user group. Five boys and one girl participated in our study (see Table 1), and while we did not have access to the children’s individual diagnoses for autism, their head teacher confirmed for us that each child had previously been diagnosed with autism by a medical professional. We received permission to report each child’s degree of communicative competency according to the P-scale (performance scale), which is a set of performance criteria used by all British schools for children with special needs working below level 1 of the UK’s national curriculum. The criteria in the P-scales rate the children’s ability to listen properly and speak coherently on a scale from one (being briefly aware of interactions with familiar people) to eight (linking up to four key-words in sentences while demonstrating an understanding of causality, or listening and responding appropriately to questions regarding causality) . The study lasted three weeks, and almost all the participants played one game session per day on four days during this period; one of the children played only three video game sessions. Additionally, because the children themselves were underage and had difficulties in communicating, the parents signed consent forms on behalf of their children before the study began for them to participate in our study and be recorded on video.