Abstract
Promoting the social-emotional development of kindergartners is of special relevance as will lay the foundations for emotion regulation in later childhood and adulthood stages. Considering that tangible storytelling tables are already used for language and literacy skills in kindergarten, we addressed the problem of designing a storytelling intervention aimed at social-emotional development suitable in such a context by using an emotional laden story as content and embedding a guidance method that can be implemented with either a human or robot guide to enhance the learning setting. The study considered two guided storytelling activities (one traditional guided by the teacher, and one in which guidance was provided by a robot) and a control condition without additional guidance. The three conditions were compared in terms of kindergartners’ enactment process, an emotion recognition test and a story recall test. The results show that the guidance method properly supported emotion naming, children involvement and goal completion during the storytelling activity whereas the intervention supported the learning gain on emotion recognition. The study revealed that both robot and human guidance did not differ significantly in the performance tests but did outperform the control. In view of the results, this research is helpful for researchers and teachers to create in an informed way a range of environments in the kindergarten class based on storytelling tables, either with or without guidance, and with or without robot support. Future work may further investigate how specific interaction issues concerning robot embodiment (e.g., voice and behavioral cues to direct children’s attention) might enhance or not the children’s performance.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Research on interactive storytelling typically focusses on language and literacy skills [1, 5, 24] but also has positive effects on kindergartners’ social-emotional development [58]. The social-emotional development of kindergartners lays the foundation for children’s social-emotional development through middle childhood, adolescence, and beyond [12], and providing support for optimal environments at early ages may also contribute to develop both higher language and social-emotional skills [15]. Early social emotional skills are related to how socially, emotionally, academically and professionally skilled we are later in life [13, 18] and having higher social-emotional skills in kindergarten is related to building stronger relationships with peers, teachers, and parents [43] and professional outcomes at age 25 [20]. Fortunately, kindergartners can be taught social-emotional skills. The results from a meta-analysis by Durlak et al. [13] on the effects of social-emotional training programs in schools revealed that these programs not only improved students’ social-emotional skills, but also had a positive effect on students’ attitudes towards school, pro-social behavior and their academic performance.
An important aspect of social-emotional skill development in kindergartners is emotion regulation. Kindergartners need to learn how they can regulate their emotions in order to behave appropriate in different social situations [6, 28]. Regulation of emotions starts with the recognition of emotions. According to McCabe and Marshall [36] adequate emotion recognition skills are essential for optimal learning and development of children between 5 to 9 years old within the general school context.
Picture books can play an important role in the social and emotional development of kindergartners and more specifically in learning how to recognize emotions. There is a large selection of picture books with a social-emotional content that have the potential to introduce kindergartners to a wide range of emotions. Although van der Bolt [58] assumed that picture books could also have a positive impact on kindergartners’ social-emotional development, Kwant [30] was among the first who actually investigated whether this assumption was indeed true. In her quasi-experimental study Kwant [30] investigated the impact of reading and talking about picture books with a social-emotional content on the social-emotional competence of kindergartners. Kwant [30] assessed emotion recognition with a test that measured both the basic emotions (happiness, anger, fear and sadness) and the more complex emotions (shyness and jealously). Result of her study showed that the picture book activity facilitated emotion recognition among these kindergartners. The positive results can be explained by three mechanisms. First, it is likely that kindergartners by reading aloud and talking about picture books, develop insight into their own feelings and emotions, and into those of others, since picture books have the potential to introduce kindergartners to all kinds of emotions and provide information about the situation in which the characters feel and express these emotions [58]. Kindergartners can really enjoy or shiver at the experiences of fictional characters in books. This form of framed experience enables them to observe or, through empathy, undergo the experiences and emotions of fictional others [58]. This is closely related to empathy, the ability to recognize and understand other people’s emotions. Empathy normally emerges at the age of four and develops gradually toward adolescence and can be enhanced and trained by the help of adults who model empathic behavior [38], and it is essential in moral development to deal with more profound complex important issues such as bullying or racism at later developmental stages during the adolescence [39]. Picture books, like all fiction, represent emotions of fictional characters’, as well as their interpretation of each other’s emotions. However, unlike novels, picture books evoke our emotional engagement through images as well as words [38] and therefore have the potential to provide additional cues in the form of facial expressions, showing typical postures. Second, fiction has some advantages which are missing in reality. The introduction of emotions in a fictional world often occurs in a controllable and demarcated situation, still they can feel real for kindergartners. Fiction also offers more room for insight and reflection. Kindergartners get the opportunity to dwell on the emotions, since they are spectators and no actual participants [29]. Last, fiction invites kindergartners to actively engage in all kinds of storytelling activities. Some of these key points and design features can often be found in recent research on storytelling and social-emotional development (e.g., [26, 48]). Enacting the story is a storytelling activity that can foster the social and emotional development of kindergartners and more specifically it provides them the opportunity to actively learn to recognize emotions. As remarked by Bateman [4], storytelling provides opportunities for children to practice suitable emotional responses, and retelling emotional stories have been a technique either proposed and used in learning settings for social-emotional development (e.g., [8, 24]). A picture book introduces kindergartners to all kinds of emotions in a way that is appealing and understandable for them. When they enact the story they explore and play with all these emotions and social situations. What kindergartners actually do when they enact the story is to dramatize the story. The play-like action of dramatizing stories is not only highly motivating for young children, it also allows them to think in more sophisticated ways [40]. Through play kindergartners can easily remember, imagine, and recreate images and ideas from their previous experiences, even though these mental operations might be too difficult if the same children had to simply try to think about or discuss them [60]. Dramatization of children’s stories mimics this function of play and it also provides opportunities to build social skills [40] and early childhood teachers have a relevant role in preparing them to deal with either past and future events with deep emotional impact by using dramatic play and storytelling [14].
Dutch elementary schools often use so-called storytelling tables (in Dutch: de verteltafel) as an interactive storytelling activity in which kindergartners enact the story. The storytelling table is used to play-out or enact a certain picture book in pairs or small groups by making use of finger puppets, props and a decor. At this moment this type of interactive storytelling is typically used to foster language skills and literacy skills and less often to promote kindergartners' social-emotional development. This is remarkable, since the activity certainly invites children to explore all kinds of emotions [30] and therefore seems suited to foster the emotion recognition of kindergartners. A potential problem with interactive storytelling activities, including enactment is that kindergartners lack real-life experience of a full range of emotions and social situation, and have not fully developed their theory of mind, the ability to recognize how other people think and feel. This can be practiced by enactment and pretend play [40]. Teacher modeling and prompting as pointed in [8, 14, 26] can help students recall emotions and enact emotions during play. This suggests that for optimal exploration of emotions using enactment with a storytelling table, sort of teacher guidance is needed, to ensure that pupils focus on the emotional content of the story. Hence, a first important contribution of our research to make the field move forward is to demonstrate how guidance prompts and questions can be embedded into a kindergarten storytelling table setting and investigate to which extent the method for guidance instruction has an effect on the story enactment process and emotion recognition. Prior to establishing our experimental study, we revise the state of the art of technology-enhanced storytelling for emotional development to analyze how different settings can also be supported by technology and the main conclusions reached
1.1 Technology-enhanced storytelling for emotional development
Interactive storytelling, in which users can influence the storyline through actions and interactions with the characters, results in engaging experiences [46]. It also increases a user’s desire to keep interacting with the system [19, 21]. Nowadays, we can use technology to provide children with assistance during an interactive storytelling learning activity as illustrated in, for example to mention a few, Kocaman-Karoglu [23], Kory-Westlund and Breazeal [25], and Fridin [17].
Nevertheless, there are only a few interactive storytelling systems designed to promote young children’s emotional understanding. A seminal example is PUPPET [34], a 3D virtual environment with autonomous characters with different personalities, emotions and conflicting goals, where children (7-9 y.o.) controlled a sheep avatar through the mouse and keyboard peripherals to understand characters’ emotional states [34]. Further investigation is yet needed for younger children. A similar approach is shown in the work in progress by Muravevskaia [37], embedding narrative and dialogue with characters in the virtual environment. It uses a first-person view for the avatar and it is interfaced with a Virtual Reality (VR) headset and handheld VR controllers. Bratitsis [7] used a 2D digital story created in Scratch to explore how typically developing children show empathy towards situations related to autism spectrum disorders. The report emphasizes the potential of pre-scripted digital stories to support emotional understanding as a mean to improve inclusion. The work in [42] presents a picture book app, which based on active-coping and support-seeking strategies, offers pre-scripted emotional situations in the interactive story to be used in a tablet. To strengthen the impact of interactions to be carried out by children in response to character’s emotions, authors designed a range of very specific touch gestures instead of using simple taps (e.g., five deep breaths implemented as drag gestures on the character’s stomach and chest area to cope with anger). Based on similar coping techniques, the research in [45] discussed the design of a platform with two apps for supporting emotional development (one for children, other for parents). Rather than using pre-scripted stories for children, it enables interaction with a chatbot app that maintains a conversation (intended as a personal story) about emotions, supporting in this way its understanding, its coping, and gauging in this way their feelings. Parents can track children progress and be aware of exercises and therapists' support.
When targeting young children, there is a trend to consider more embodied and tangible interactions. For example, StoryFaces is a touch-pad based storytelling system that allows children to watch pre-made stories involving virtual characters with different emotions [44]. emoPuppet [35] combines a digital display with puppet fabrics, which can be operated easily by the child from another touch screen to control the facial expression. It does not suggest a specific working activity. FingerAR [3] combines hand-puppets that are augmented digitally on a mirrored screen in front of children through an Augmented Reality (RA) paradigm. The system allows children to change the facial expressions of puppets dynamically to express emotional state during the pretend play experience. The study concludes that, when allowed, children include wider range of emotional states and more often. Wallbaum et al. [61] propose a tangible interactive diorama which includes a puppet character that can be manipulated to move its arms and legs, and interchange facial expressions. It is also able to record and play audio to be used as part of the reenactment. Silva et al., [48] designed a playbrick, a tangible brick device with embedded sensors and display, used to show pre-made short emotional stories that children can match with 5 basic emotions (Anger, Fear, Happiness, Sadness, Surprise), revealing very high accuracy except for the stories related to Surprise emotion. It can be used with the humanoid robot Zeno to deliver human-like emotion facial expressions in the classroom. The system has been validated using a questionnaire with typically developed children, however authors' future work evaluation is expected to involve children with autism spectrum disorder (ASD). On this strand too, Azevedo et al. [2] reported three game activities aimed at imitating emotions, recognizing emotions, and matching emotions to narratives displayed in a PC, involving a tangible controller with 6 buttons labeled with emojis that match the 6 basic Ekman’s emotions, but the evaluation with children with ASD is limited to test whether the interface with the emojis is properly understood and usable enough to run the activity. Storytelling has also been used as an emerging approach to very complex social issues in adults to promote emotional changes in attitudes [62]. Despite such higher-order emotional skills involved, some tangible storytelling settings are starting to explore how intercultural issues can be treated through narratives in young children [53, 54]. In this case digital manipulatives allow children to combine items from different intercultural backgrounds, trigger story elements and reflect on the behavior displayed by the characters on screen.
The use of robots to embody agents and provide external representations for emotions in storytelling is being investigated intensely. Soute & Nijmejer [49] and Leite et al. [32] use small robots (the IxI Play and the Keepon, respectively) that stand in front of children to play out part of the narrative. While the first work carried out an exploratory study to investigate how to embed the robot and picture books in a regular school context to enable future applications, the second one designed a set of fully pre-scripted episodes which included high-level emotional situations with feelings such as social inclusion, frustration, and social cooperation. The study revealed that the recall when interacting in group (3 children) was lower than when playing individually, and that emotional understanding varies among situations. The ALIZ-E robot [56], embodied in a humanoid Nao robot, aimed to develop adaptive emotional responses and expressions in robot-child interactions. Following a Wizard-of-Oz (WoZ) protocol to enact the behavior, the observational study revealed more expressive behavioral responses by children and more positively reactions to the robot. Conti et al. [11] also used a Nao. Their results concluded that the inclusion of expressive behavior as part of the storytelling performance helped to recall more details. Interestingly, Ligthart et al. [33] went beyond WoZ and demonstrated how design patterns can be successfully embedded into a storytelling Nao robot with autonomous behavior to make the task more active for children. Although this research is not specifically focused on emotional development, it does suggest that behavior realization including emotions could be part of the enactment. The fluffy robot TEGA has also been used in teleoperated and pre-scripted story setups to investigate how entrainment leads to higher word recall [26], and how an expressive emphatic version of the robot contributes to vocabulary recall and robot likeability [24]. Both works stand out by having clear methods, combining active book reading with interactions with the robot and retelling to strengthen recall of key target vocabulary. More on the effect of using contextual prompts during collaborative storytelling is explored in [52], where the child had to retell the created story to a robot, not reporting higher recalls. The work in [55] looked at the narrative structure of the story retold by the child after using a setting to create stories that involved tangibles, an action tablet app and a Cozmo robot playing the role of main character. Results showed that introducing emotional responses by the robot led to differences in the way children incorporated emotions in their stories, which encouraged children to explore and link emotions to possible causes and consequences.
Table 1 summarizes the main aspects of the contributions reviewed. Although it is not aimed to extensively survey existing technology-enhanced storytelling settings for emotional development, it still offers an overview of the key issues and technological materials being developed in such state-of-art settings. As displayed in the table, the range of technology is diverse but the inclusion of robots as agents capable of rendering emotions is an important strand, which may help to mediate interactions and direct the learning activity. The robot settings differ from one to another, but it is common that robots remain in a demarcated location and adopt a companion/peer role using a touch screen for visualizing part of the story/content. None of the settings is shaped as a storytelling table, with tangibles, that would suit materials and approaches currently implemented in kindergarten even better.
It is worth noting we consider the works revised as good examples targeted at elementary/primary school age, which could also be used with older kindergartners. However, most of them involved children from primary school, often not reporting the age frequencies or means. Furthermore, some systems are suitable for exploring emotions, but they might be too open-ended for kindergartners without any further clear guidance or specifically designed instruction method. This entails most settings tested (and therefore the conclusions reached) may have their relevance diminished due to potential biases by older children, who have higher development. Hence, it is highly advisable that any study focuses specifically on kindergarten children, especially in view of the systematic review on physical digital play technology by Torres et al. [57] that concluded that there is a need to carry out more studies of play technologies in relation to social-emotional skills in general, and emotion regulation and emotion understanding/awareness in particular. Thus, besides maintaining a storytelling table approach based on picture books that can be transferable to a classroom in the future, our second important contribution will focus on the integration of robot technology to deliver expressive emotional behavior, using a clear and simple guidance method as for kindergartners to understand robot’s prompts and queries, and then evaluate how the guidance instruction method is characterized when using the robot. In this respect, it is important to compare both human guidance versus robot guidance in order to offer alternatives that can be taken in an informed way to the class. As next section will show, our approach uses a social robot character in the story, rather than a companion, to guide interactions that emphasize emotional states during the storytelling activity. Furthermore, to test whether the guidance instruction method works properly we will consider a control condition without guidance, which would correspond to the most basic storytelling table activity that can be implemented without technology in any kindergarten class.
2 Experimental user study
2.1 Method
Our goal for this experimental study is to evaluate a guidance instruction method suitable for a tangible storytelling table in kindergarten to help emotional development and to assess how well it can be supported by a robot taking part in the story. The key research questions that drive our study can be formulated as follows:
-
RQ1: Does the guidance instruction method work properly as to support emotional development? And how differently does human vs robot guidance function?
-
RQ2: Does the storytelling intervention lead to social-emotional learning?
-
RQ3: Does the guidance instruction lead to higher story recall?
This study used an experimental between-group design with three conditions. Two experimental conditions were developed in which kindergartners were either facilitated by a robot or a human guide as part of their interactive storytelling activity; kindergartners in the control condition worked on the interactive storytelling activity without any support. Full descriptions of the conditions are presented in section 2.3 (Settings). This study has a mixed-method approach. As described in detail in section 2.4 (Instruments), we look at multiple data as response variables about the enactment process, emotion recognition and story recall in order to carry out a comprehensive analysis. Hence, quantitative data was obtained from an emotion recognition test, measuring kindergartners’ recognition of emotional states. This test was administered as a pre- and posttest. Also, a story recall test was administered, measuring kindergartners’ ability to recall the story. Quantitative data regarding reactions, interactions and explorations were observed and coded from the video recordings of the interventions. To gain a deeper understanding of the enactment during the activities, a more qualitative analysis has been carried out based on the video recordings too. Two excerpts for each condition will be presented and described in more detail. These three sets of metrics, regarding enactment process, emotion recognition and story recall respectively, will serve to answer the corresponding research questions above, and the results and discussion sections will be organized accordingly for the sake of clarity.
2.2 Participants
A total of 73 kindergartners between 4 and 6 years old (M = 5.00 SD = 0.56) participated in this study. The ethics committee of the University approved the research study and parental permission was obtained through an informed consent form. The kindergartners (36 boys and 37 girls) were students of two different schools in the Netherlands. Within the Dutch education system elementary schools have eight grades, grade 1 (age 4) through grade 8 (age 12). Most schools combine groups 1 and 2 in the same class. In the first two grades the focus is on learning through play, and preparation for formal math, reading and writing instruction. that will start in grade 3 (at approximately 6 years of age). Kindergartners in each class were randomly grouped in dyads who were then randomly assigned to one of the three conditions. Due to the uneven number of participants one group consisted of three children. This group was excluded from further analysis. Resulting in 22 kindergartners in the control condition, 22 kindergartners in the human guidance condition and 26 kindergartners in the robot condition.
2.3 Settings
The setting is described in terms of the picture book, the storytelling table and the way guidance is provided in the activity for each condition.
2.3.1 Picture book
The picture book that was used in this study is ‘Jacobus’ written by Steven Pont and Mark Janssen [41]. The book was selected because it introduces the kindergartners to all kinds of emotions in a way that is appealing and understandable for them. The story is about a skunk called Jacobus who is, unlike the other animals, not invited to the special animal party. But Jacobus thinks a skunk can also be special, because a skunk can stink. Jacobus wants to show this at the party in front of all the other animals. The emotions that appear in the picture book are sadness, fear, happiness, anger, jealously and shyness.
2.3.2 Storytelling table and conditions
All students worked in dyads on a storytelling table activity that focused on the enactment of the picture book. The physical storytelling table consisted of a tabletop decor and physical puppets for the main character and subcharacters (see Figure 1). During the intervention all kindergartners made use of the storytelling table, but in a different way, depending on which condition they were assigned to. Kindergartners were randomly assigned to either one of the conditions: (1) an interactive storytelling activity with a technology facilitated scripted social robot or (2) with human guide facilitated scripted support or (3) the control condition: an interactive storytelling activity without any support or guidance. In the control condition, kindergartners enacted the story in dyads and made use of the storytelling table and the puppets. In all conditions kindergartners got a short introduction in which they were introduced to the storytelling table, the characters and the goal of the activity. Thereafter, kindergartners in the control condition independently enacted the story without any further support. In all conditions the researcher stayed in the room, but in the robot and control condition the teacher took a retracted attitude. The two experimental conditions are now explained in more detail.
Human guidance condition
Within this condition, kindergartners enacted the story in dyads on the storytelling table and were facilitated by a human guide who gave directions, asked questions about the story and gave prompts to the participants. The questions and prompts were based on a prescribed script, which is the same as in the robot condition. The script consisted of specific questions and prompts the child could respond to. For each category of responses, the script provided a reaction. This could either be silence or a follow-up question or prompt. An important part of the instruction includes to embed questions on emotions systematically to reinforce recognition and reasoning by children during the discourse of emotional events. An example of such a sequence is as follows:
Question: Can you tell me how Jacobus (main character) is feeling?
Response by the child: Jealous.
Response by the system: So you think Jacobus is jealous?
Other prompts require children to move around the table and enact the scene that is located at a specific part of the table. Kindergartners were able to interact with the puppets and move them along the storytelling table. After a short general introduction, which was the same for all participants, the actual activity started. The human guide gave directions and kindergartners received their guidance through the activity. The human guide was an elementary teacher, familiar in working with young children and trained to give guidance following the prescribed story.
Robot guidance condition
Within this condition, participants also enacted the story in dyads on the storytelling table but were facilitated by a robot that gave directions, asked questions about the story and gave prompts to the kindergartners. The questions and prompts were based on a prescribed script as in the human guidance condition. So, kindergartners in the robot guidance condition received the same feedback comments as kindergartners in the human guidance condition. Kindergartners were able to interact with the robot and the puppets, and move them along the storytelling table. After a short general introduction, which was the same for all participants, the robot introduced itself to the kindergartners. Then the actual activity started. The robot gave directions and kindergartners were guided through the activity by the robot.
2.4 Instruments
2.4.1 Robot platform
The robot used is based on an affordable platform developed in [10], which supports the deployment of tele-operated robot behavior (e.g. short utterances, display emotions, etc.) and that has been used as a social robot in learning applications [59]. The platform has been expanded and further developed to fit the purpose of this research. Specifically, a durable and robust robot case has been designed to resist intensive children’s handling (see Figure 1). The motorized wheels were replaced by a set of tire-wheels without motors as we wanted children to move the robot by hand to make it comparable across conditions and prevent any motion effects as a result of possible glitches in robot tracking [9]. The robot comes with designer and controller apps that allow us to both create and control robot behaviors following a Wizard-of-Oz (WoZ) approach [50]. To facilitate reproducibility across trials by a trained experimenter, the user interface is highly optimized to make the enactment/play manageable as large number of assets are available in realistic scenarios as the one in the present study which involves an actual picture book. Appendix D overviews the screens for these apps (Figs. 3 and 4).
2.4.2 Pre/posttest emotion recognition
The quantitative component of the study consists of an emotion recognition pre-test and a post-test. This test is a Dutch translation of the Assessment of Children’s emotional Skills (ACES) which is developed and validated by Schultz and Izard [47]. This Dutch translation of the test was used by Kwant [30]. The assessments measure social-emotional understanding including distinguishing and recognizing of other people’s emotions in social situations. The ability of the kindergartners to correctly recognize and label character’s emotional states was assessed. The test focused on the emotions fear, happiness, anger, and sadness, jealously and shyness. Before the test started, pictures of the emotional states (fear, happiness, anger and sadness) were presented and introduced to the kindergartners. The test started with 15 short social stories about the emotions fear, happiness, anger and sadness. An example of one of the stories could be: “Sam has made a drawing. His friends say he really likes the drawing. How do you think Sam will feel?” The researcher read the stories to the kindergartner. Then the kindergartner was given the opportunity to respond to the question by addressing an emotion and pointing to the picture with the corresponding emotional state. Then the researcher removed the pictures of the emotional states and introduced the participants with the terms shyness and jealously. The test continued with 6 more short social stories. An example of one of the stories could be: “Lars is going to football training. In the dressing room Pim shows his new tracksuit. It is exactly the tracksuit Lars also wants to have! How do you think Lars will feel?” The researcher read the stories to the kindergartner. Then the kindergartner was given the opportunity to respond to the question by addressing an emotion. Kindergartners could score a maximum of 21 points for the test. All emotions within the test also appear in the picture book.
2.4.3 Recall test
Quantitative data was obtained from a story recall test measuring kindergartners’ ability to recall the story. The six most important and meaningful scenes were selected for this recall test and a photocopy of these scenes was made. These photocopies were presented to the kindergartners. They had to individually put theses photocopies of the six scenes into the right order as a line from left to right (see Figure 2). Each well placed photocopy of the scene was scored with 1 point. Kindergartners could score a maximum of 6 points. We chose this accumulative scoring method because one misplaced photocopy would entangle the storyline, even though the next photocopies after the misplaced one might be in the right order.
2.4.4 Coding and analysis
Video-recordings during the interactive storytelling activity were made using a Canon EOS 550D. They were scored and analyzed in order to get insight into the processes during the activity. Kindergartners worked on the interactive storytelling activity in dyads but there may be differences between the two individuals. Therefore, we will look at the individual level as well. In order to analyze the video-recordings two scoring schemes were developed; one focusing on the dyad level and one focusing on the individual level.
The first scoring scheme focused on whether the dyads did complete the goal of each scene within the activity (Appendix A, see Table 7). There were two scoring options: dyads did complete the goal of the scene or dyads did not complete the goal of the scene. For example, the goal of the first scene was that Jacobus approaches the peacock and eventually asks him if he can come to the special animal party. But the peacock says Jacobus is not allowed to come to the party, because he is not special. If kindergartners enacted this scene accordingly, then they completed this goal. Two coders (the second and third author) coded all interaction data. Inter-rater agreement (Cohen’s Kappa) reached .71. The first scoring scheme also focused on whether the dyads did complete the emotional goal of each scene within the activity (Appendix A, see Table 8). For example, the emotional goal of the first scene was that Jacobus feels sad about not being invited to the party. If kindergartners explicitly name the correct emotional state of Jacobus, then they completed this goal. Inter-rater agreement (Cohen’s Kappa) reached .95.
The second coding scheme focused on the level of involvement at the individual level for each scene within the activity (Appendix A, see Table 9). There were four scoring options: the intended goal has been achieved with very low involvement, the intended goal has been achieved with low passive involvement, the intended goal has been achieved with active involvement (type 1), the intended goal has been achieved with active involvement (type 2). The level of involvement is scored based on a scale for scanning the level of involvement by Laevers et al. [31] (Appendix B, see Table 11). Inter-rater agreement (Cohen’s Kappa) reached .84. The second coding scheme also focused on which emotion the individual kindergartners named and whether they did complete the emotional goal of each scene within the activity (Appendix A, see Table 10). The specific emotion the kindergartner named was coded and each well named emotion was scored with 1 point. Inter-rater agreement (Cohen’s Kappa) reached .95.
2.4.5 Procedure
The study consisted of five sessions over a period of three weeks. Each session lasted up to a maximum of twenty minutes. The first week the researcher (who was also trained as a teacher) introduced herself to the children and then she individually conducted the emotion recognition pre-test for all kindergartners. By the end of the first week, the classroom teacher read the book ‘Jacobus’ following the guidelines presented by the research team. In the second week, the researcher visited the school and read the picture book once again on the first day of the week. Then, dyads were escorted to a quiet room to do the activity. Right after the activity, the kindergartners made a short recall test of the story. All activities were video-taped. In the third week the researcher individually conducted the emotion recognition post-test for all kindergartners. In all schools the same researcher administered the pre- and post-test and took care of the implementation of the experiment. Only the first reading session was performed by the regular classroom teacher.
3 Results
This section reports the analysis of indicators for the enactment process coming from the observations of the interactive storytelling activity, then the emotion recognition test, the recall test and, finally, the relation between the enactment process and the performance of the participants. Section 4, Discussion, will interpret the results and answer the research questions.
3.1 Enactment process indicators
As indicators for the enactment processes, we used both dyad and the individual measures. The mean scores for each dimension of the coding scheme are presented in Table 2.
Table 3 summarizes the results of statistical analysis carried out. One-way ANOVAs revealed significant differences among conditions for all the enactment process indicators. Tukey post hoc tests revealed that there were statistically significant differences between the control and human guided condition and between the control and the robot guidance condition, whereas no statistically significant differences between the human guidance condition and the robot guidance condition. In general, differences between grades were not found, except for the level of goal completion among dyads and the level of involvement at the individual level, in both cases kindergartners of grade 2 had a higher score than kindergartners of grade 1.
3.2 Emotion recognition test
The effect of the interactive storytelling activity on kindergartners’ recognition of emotions was examined through the emotion recognition test. The mean number of correctly answered items and the mean gain score on the emotion recognition pre- and posttests is given in Table 4. A paired t-test indicated that overall kindergartners scored significantly higher on the posttest (M = 16.20, SE = .32) than on the pretest (M = 14.33, SE = .34), t(69) = 5.85, p < .001, r = .58).Footnote 1
Paired t-tests indicated that kindergartners had a significant gain score in all conditions: control condition t(21) = −3.10, p = .005, r = .56; in the human guidance condition t(21) = −3.01, p = .007, r = .55; and in the technology enhanced condition t(25) = −4.24, p < .001, r = .65. The control condition yielded lower mean gains, followed by the robot guidance condition, and then by the human guidance condition. However, one-way ANOVA did not reveal statistically significant differences in gain scores between conditions (F(2, 67) = .235, p = .791, ω = −.02). An additional adjustment taking into account children’s school grade, did not result in statistically significant difference in gain score between condition, F(2, 66) = .091, p = .76, partial η2 = .001. An independent-samples t-test indicated that there was no significant difference in mean gain score between grades, t(68) = −.40, p = .197, r = .05.
3.3 Story recall test
The mean score for the recall test is given in Table 5. Recall is on average higher in the robot guidance condition, and lowest for the control condition. However, a one-way ANOVA did not reveal statistically significant differences in story recall scores between the three conditions (F(2, 67) = .819, p = .445, ω = −.01). An independent-samples t-test indicated there was also no statistically significant difference in recall scores between grades, t(68) = .299, p = .766, r = .001).
3.4 Relation between enactment process and kindergartners’ test performance
We report the relation between the emotion recognition gain scores and story recall scores with the level of involvement and the amount of correctly named emotions during the activity.
Pearson correlation was administered to determine the strength and direction of the linear relationship between the gain score on the emotion recognition test and, the level of individual engagement and individual emotion naming. There was a small positive correlation between level of involvement at the individual level and gain score on the emotion recognition test, r = .236, p = .049. Subsequently, a regression analysis including the level of individual involvement, and emotion naming on the individual level was conducted to investigate if these factors predicted individual children’s gain scores on the emotion recognition test. No significant regression equation was found (F(2, 67) = 2.32, p = .106).
With respect to the scores on the story recall test we found a positive correlation between the level of involvement at the individual level and the recall score r = .319, p = .007. A significant regression equation was found (F(2,67) = 3.809, p = .027), with and R2 of .102. The level of involvement on the individual level was a significant predictor of scores on story recall. Emotion naming at the individual level did not add significantly to the equation (see Table 6).
3.5 Case analysis
Finally, to gain a deeper understanding of the enactment processes during the interactive storytelling activity in all three conditions, we present six representative excerpts and a thorough analyses for each excerpt (see Appendix C). The excerpts reveal in more detail the differences between the mutual experimental conditions and the control condition. All excerpts relate to the first scene of the story. The general goal of this scene is that Jacobus approaches the peacock and asks him if he can come to the special animal party. But the peacock says Jacobus is not allowed to come to the party, because he is not special. The emotional goal of the scene is that kindergartners literally name that Jacobus feels sad. The following paragraphs summarize the main conclusions that can be derived from the excerpts regarding kindergartners’ dialogues during the interactive storytelling activity.
In the control condition, most kindergartners completed the general goal of the scene and were actively involved. However, naming of the emotions rarely occurred, see Tables 12 and 13, for examples. In the human guided condition, most kindergartners completed the general goal and the emotional goal of the scene and were actively involved, except one kindergartner who is rather passively involved, see Tables 14 and 15 for examples. In the robot guidance condition, most kindergartners completed the general goal of the scene and were actively involved. However, they did not all complete the emotional goal of the scene, see Tables 16 and 17 for examples.
4 Discussion
4.1 RQ1: Does the guidance instruction method work properly as to support emotional development? And how differently does human vs robot guidance function?
The guidance instruction method worked properly since all enactment process indicators regarding emotion naming, children involvement, and goal completion show that both experimental (guided) conditions outperformed the control condition. Specifically, dyads in both experimental conditions showed significantly higher levels of goal completion. At the individual level, higher levels of involvement were found in both guided conditions. This means that kindergartners in the guided conditions enacted more scenes from the original story and were more individually involved in doing so. Kindergartners in the guided conditions recognized and named significantly more emotions (at both the individual and dyad level) during enactment compared to kindergartners in the control condition. This is in line with our expectations since kindergartners in both guided conditions received questions about the story that literally guided them through the story. Within the guided conditions kindergartners received questions about the emotional state of the characters, which resulted in significant higher levels of emotion naming. This is also consistent with observations in [24], who used a different setting but also considered questions prompted by the robot such as “How do you think the boy feels now?” when facing emotional situations. Kindergartners in the control condition were free to enact the story as they wanted to. If they wanted to change the complete storyline or just skip some scenes, they could. That would be fine in some of the revised systems, which were more open-ended and designed to support creative storytelling [3, 44], but then they would not have a way to consider more complex emotions such as jealously or shyness. Thus, being creative without actually enacting the expected story would result in lower levels of goal completion among dyads, as well as emotion recognition and naming in the experimental conditions. After all, it is important to remark that our setting is designed to use a specific curated story with emotional laden as content.
Both the human guide and the robot shaped the learning environment for the kindergartners through asking systematically specific questions that help them focus on to the main content of the story [22], stimulating the enactment of key scenes from the picture book. In principle, we might expect the human guidance condition to affect kindergartners’ enactment significantly more positively than the robot guidance condition, since kindergartners are more used to human-human interaction than human-robot interaction and humans are better able to communicate both verbal and nonverbal information than robots. However, no differences were found in any of the indicators between both guided conditions, which suggests that the guidance instruction method with the scripted approach is simple enough as to be implemented successfully, and therefore guidance mediated by a robot can be a potentially interesting tool for education that must be further explored and investigated. The designed guidance led to consistent results regarding emotion naming, involvement and goal completion, therefore bringing opportunities for enhanced learning. The qualitative examination of students’ enactment process provides some insight into the way students responded to guidance provided by the robot and the human guide based on the case analysis. In short, kindergartners that were guided by the robot seemed to feel slightly more hesitant or uneasy during the activity in comparison to kindergartners that were guided by the human, perhaps due to the limited cues and nonverbal behavioural feedback available in the robot, which might not be as natural as a human. Children did not always know how or when to intervene. The rather strict and monotone voice of the robot, not adapting to different situations, might also have contributed to this, which according to previous research may even negatively influence how people perceive working with the robot [27] and could even cause that people find the robot less trustworthy [51]. Kory Westlund et al. [24] demonstrated that expressive voices instead of flat voices in automatic speech synthesizers would yield advantages in children emulating emotional phrases by the robot. Our experimental design does not specifically account for investigating these fine-grain interaction aspects, such as voice-naturalness and adaptive/responsive eye-contact to direct children’s attention. Thus, they are worth investigating in specific experiments in the future and explore how the human and the robot guided conditions could become different in this regard.
4.2 RQ2: Does the storytelling intervention lead to social-emotional learning?
Another finding of this study is that, overall, the storytelling intervention as a whole led a significant positive effect on kindergartners’ recognition of emotions (as measured by the emotion recognition test), regardless of the condition kindergartners were assigned to. Children learned to recognize emotions. This is in line with the general idea that by engaging in interactive storytelling activities (such as storytelling tables) kindergartners develop insight into their own feelings and emotions and in those of others [58] and can learn how to recognize emotions [30]. However, no significant differences among conditions were found; therefore, the evidence gathered is not conclusive to establish differences as they all offered similar gains in learning to recognize emotions from short written stories. This leads us to reflect on some points that could be revised when designing experimental conditions for social-emotional learning support in future studies. For example, the interactive storytelling table session was a relative short part in the intervention. Furthermore, the activity was introduced as a play-like activity and kindergartners were not literally told that they were going to learn something from the activity. A more focused introduction of the activity that explained that the activity was intended to practice the recognition of emotions might have led to more conscious engagement in learning how to recognize emotions. In addition, kindergartners were not familiar with guided storytelling table activities. The guided storytelling table activities therefore were less familiar than the unguided activity. Kindergartners in the guided condition might therefore have focused more on the actual guidance (human or robot) than the content of the activity, and therefore might have learned less than we expected. Anyway, a very positive confirmatory point that results reveal is that the reading sessions included in the procedure during the first two weeks together with the exposure to the playing activity led to an overall positive effect in learning.
4.3 RQ3: Does the guidance instruction lead to higher story recall?
With respect to the recall of the story, we found no significant differences between conditions despite the lower numbers for the control condition. Hence, the guidance instruction condition cannot be considered significantly better for story recall compared to the control condition. This is a sensible result as the guidance instruction method is actually focused on emotional reasoning and not on story events, but we still needed to check whether the fact that questions in the guidance method are situated in the story might have an effect or not as reported in [11]. This result opens an opportunity for redesigning the guidance method, incorporating more questions regarding comprehension of the story [24], to make the goal of story comprehension and recall at least as important as emotion recall. Finally, concerning the relation between the enactment process and the story recall test, the results revealed that higher involvement on the individual level is a significant predictor of scores on story recall. Hence, another possible action for improvement on the guidance method could be to consider addressing the questions to each child evenly in order to enhance their involvement.
4.4 Final remarks, limitations and future work
This study means a first step in investigating the effect of a robot-enhanced interactive storytelling activity on kindergartners’ storytelling enactment, recognition of emotions and story recall. Kindergartners enacted a story on emotions using a storytelling table. To answer the research questions, three versions of the same storytelling table activity were compared in terms of students’ score on an emotion recognition test, as well as their enactment and recall of the content of the story. The three conditions differed with respect to the provided guidance. The two experimental conditions offered guidance provided by either a human guide or mediated through a robot, respectively. Kindergarteners in the control condition completed the activity without additional guidance. Before the actual experimental sessions, students got the chance to listen to the book twice. First it was read to them by their regular classroom teachers, and subsequently by the experimenter (also a qualified teacher), who guided the kindergartners during data collection.
In our study, we worked with a large and homogeneous sample of kindergartners, and did not mix kindergarteners with primary school students, as is reported in many related studies. Therefore, we argue that the conclusions reached are generalizable to the group of kindergartners, in contrast to many of the reviewed related work. Furthermore, the results are highly relevant for practice since using a tangible storytelling table is already part of everyday practices in many kindergarten classrooms. In such cases, integrating the instructional method based on tangible storytelling tables with robot guidance support can also be implemented by using any other robot platform capable of expressing emotions. Thus, this research is helpful to create in an informed way a range of environments in the kindergarten class based on storytelling tables, either with or without guidance, and with or without robot support. This makes our study complete and more comprehensive.
There are also some limitations associated to the study as already pointed out in the discussion, concerning aspects that can be examined in alternative experiments. Although the guidance method did lead higher emotion naming during the enactment, the results on story recall and learning emotions in the long term suggest that there are research opportunities for improvements in future work studies. It would be worth investigating a longer intervention by extending with more storytelling table sessions, so that it assures a more profound effect on learning. The guidance method could also be modified and expanded in a future work to check whether more complex guidance, not only with questions on emotion understanding but also about the story, can lead to enhanced story recall. Future work may also address and explore how interaction issues concerning robot embodiment (e.g., voice and behavioral cues to direct children’s attention) might enhance or not the children’s performance. Finally, we are using WoZ, which means that a person needs to take control of the robot behaviour. A future work would consist of bringing an autonomous robot version to support the guidance in the activity.
Notes
The r symbol represents the effect size [16]
References
Aram D (2008) Parent-child interaction and early literacy development. Early Educ Dev 19:1–6. https://doi.org/10.1080/10409280701838421
Azevedo J, Silva V, Soares F, Pereira AP, Esteves JS (2018) An Application to Promote Emotional Skills in Children with Autism Spectrum Disorders. In: Göbel S et al (eds) Serious Games. JCSG 2018. Lecture notes in computer science, vol 11243. Springer, Cham. https://doi.org/10.1007/978-3-030-02762-9_30
Bai Z, Blackwell AF, Coulouris G (2015). Exploring expressive augmented reality: the FingAR puppet system for social pretend play. Proceedings of the 33rd annual ACM conference on human factors in computing systems, 1035-1044. https://doi.org/10.1145/2702123.2702250
Bateman A (2020) Young children’s affective stance through embodied displays of emotion during tellings. Text & Talk 40(5):643–668. https://doi.org/10.1515/text-2020-2077
Berenst J, Kwant A, de Glopper K (2007) Leerzame gesprekken? Interacties met kleuters tijdens voorleessessies (instructive conversations? Interaction with kindergartners in read aloud sessions). Toegepaste Taalwetenschap in Artikelen 78:85–94. https://doi.org/10.1075/ttwia.78.08ber
Blair C, Diamond A (2008) Biological processes in prevention and intervention: the promotion of self-regulation as a means of preventing school failure. Dev Psychopathol 20:899–911. https://doi.org/10.1017/S0954579408000436
Bratitsis T (2016) A digital storytelling approach for fostering empathy towards autistic children: lessons learned. Proceedings of the 7th international conference on software development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, 301-308. ACM. https://doi.org/10.1145/3019943.3019987
Brunetti E, Mugione M, Zampella M (2021) Feeling dice: a game-based methodology to develop emotional competence in preschoolers. CEUR workshop proceedings, 2817, in 1st workshop on technology enhanced learning environments for blended education - the Italian e-learning conference, teleXbe 2021
Catala A, Moreno A (2018) Smart navigation for a storytelling multi-robot setting. In IROS 2018 Second workshop on multi-robot perception-driven control and planning, 3 pages. https://citius.usc.es/p/1237
Catala A, Theune M, Gijlers H, Heylen D (2017) Storytelling as a creative activity in the classroom. Proceedings of the 2017 ACM SIGCHI conference on creativity and cognition, 237-242. https://doi.org/10.1145/3059454.3078857
Conti D, Di Nuovo A, Cirasa C, Di Nuovo S (2017) A comparison of kindergarten storytelling by human and humanoid robot with different social behavior. Proceedings of the companion of the 2017 ACM/IEEE international conference on human-robot interaction, 97–98. https://doi.org/10.1145/3029798.3038359
Denham SA, Brown C (2010) “Plays nice with others”: social-emotional learning and academic success. Early Educ Dev 21:652–680. https://doi.org/10.1080/10409289.2010.497450
Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB (2011) The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev 821:405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x
Ebbeck M, Yim HYB, Wei T (2020) Preparing children for an uncertain future: the role of the early childhood teacher. J Early Child Teach Educ 41(3):223–240. https://doi.org/10.1080/10901027.2019.1617808
Farkas C, Álvarez C, Cuellar M, Avello E, Gómez DM, Pereira P (2020) Mothers' competence profiles and their relation to language and socioemotional development in Chilean children at 12 and 30 months. Infant Behav Dev 59:101443. https://doi.org/10.1016/j.infbeh.2020.101443
Field AP (2017) Discovering statistics using IBM SPSS Statistics, 5th edn, pp1–104 Sage
Fridin M (2014) Storytelling by a kindergarten social assistive robot: a tool for constructive learning in preschool education. Comput Educ 70:53–64. https://doi.org/10.1016/j.compedu.2013.07.043
Heckman JJ, Kautz T (2012) Hard evidence on soft skills. Labour Econ 19(4):451–464. https://doi.org/10.1016/j.labeco.2012.05.014
Homan G, Kubat R, Breazeal C (2008) A hybrid control system for puppeteering a live robotic stage actor. Proceedings of RO-MAN 2008, 54-359. IEEE. https://doi.org/10.1109/ROMAN.2008.4600691
Jones DE, Greenberg M, Crowley M (2015) Early social-emotional functioning and public health: the relationship between kindergarten social competence and future wellness. Am J Public Health 105:2283–2290. https://doi.org/10.2105/AJPH.2015.302630
Kelleher C, Pausch R, Kiesler S (2007) Storytelling Alice motivates middle school girls to learn computer programming. Proceedings of the SIGCHI conference on human factors in computing systems, 1455-1464. ACM. https://doi.org/10.1145/1240624.1240844
Kim Y, Baylor AL (2006) A social-cognitive framework for pedagogical agents as learning companions. Educ Technol Res Dev 54:569–596
Kocaman-Karoglu A (2015) Telling stories digitally: an experiment with preschool children. Educ Media Int 52:340–352. https://doi.org/10.1080/09523987.2015.1100391
Kory Westlund JM, Jeong S, Park HW, Ronfard S, Adhikari A, Harris PL, DeSteno D, Breazeal CL (2017) Flat vs Expressive Storytelling: Young Children's Learning and Retention of a Social Robot's Narrative. Front Hum Neurosci 11:295. https://doi.org/10.3389/fnhum.2017.00295
Kory-Westlund JM, Breazeal C (2015) The interplay of robot language level with children's language learning during storytelling. Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction extended abstracts, 65-66. https://doi.org/10.1145/2701973.2701989
Kory-Westlund JM, Breazeal C (2019) Exploring the effects of a social Robot's speech entrainment and backstory on young Children's emotion, rapport, relationship, and learning. Front Robot AI 6:54. https://doi.org/10.3389/frobt.2019.00054
Kuo IH, Rabindran J, Broadbent E, Lee YI, Kerse N, Stafford R, MacDonald BA (2009) Age and gender factors in user acceptance of healthcare robots. Proceedings of the 18th IEEE international symposium on robot and human interactive communication, Toyama, Japan, 214-219. https://doi.org/10.1109/ROMAN.2009.5326292
Kurki K, Järvenoja H, Järvelä S, Mykkänen A (2017) Young children’s use of emotion and behavior regulation strategies in socio-emotionally challenging day-care situations. Int J Educ Res 41:50–62. https://doi.org/10.1016/j.ecresq.2017.06.002
Kwant A (2007) Met prentenboeken kun je alles. Taal Lezen Primair 22:1–3
Kwant A (2011) Geraakt door prentenboeken. Effecten van het gebruik van prentenboeken op de sociaal-emotionele ontwikkeling van kleuters. Proefschrift. Eburon
Laevers F, Buyse E, Daems M, De Bruyckere G, Declercq B, Silkens K, Snoeck G (2009) Werken aan kwaliteit vanuit het kinderperspectief: Welbevinden en betrokkenheid als richtsnoeren. ZiKo II. Eindverslag Partnerschap Kind & Gezin en Expertisecentrum Ervaringsgericht Onderwijs. Centrum voor Ervaringsgericht Onderwijs
Leite I, McCoy M, Lohani M, Ullman D, Salomons N, Stokes C, Rivers S, Scassellati B (2015) Emotional storytelling in the classroom: individual versus group interaction between children and robots. Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, 75–82. https://doi.org/10.1145/2696454.2696481
Ligthart MEU, Neerincx MA, Hindriks KV (2020) Design patterns for an interactive storytelling robot to support Children's engagement and agency. Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. Association for Computing Machinery, New York, NY, USA, 409–418. https://doi.org/10.1145/3319502.3374826
Marshall P, Rogers Y, Scaife M (2002) The value of a virtual environment for learning about narrative SIGGROUP Bull. 23, 2 (August 2002), 14–15. https://doi.org/10.1145/962185.962190
Martínez JI (2014) EmoPuppet: low-cost interactive digital-physical puppets with emotional expression. In Proceedings of the 11th conference on advances in computer entertainment technology, (ACE '14). Association for Computing Machinery, New York, NY, USA, Article 44, 1–4. https://doi.org/10.1145/2663806.2663873
McCabe PC, Marshall DJ (2006) Measuring the social competence of preschool children with specific language impairment: correspondence among informant ratings and behavioral observations. Top Early Child Spec Educ 26:234–246. https://doi.org/10.1177/02711214060260040401
Muravevskaia E (2017) Empathy development in young children using interactive VR games. Extended abstracts publication of the annual symposium on computer-human interaction in play, 715-718. ACM. https://doi.org/10.1145/3130859.3133229
Nikolajeva M (2013) Picture books and emotional literacy. Read Teach 67:249–254. https://doi.org/10.1002/trtr.1229
Palaigeorgiou G, Vroikou G, Nikoleta C, Bratitsis T (2021) Wearable E-Textile as a Narrative Mediator for Enhancing Empathy in Moral Development. In: Auer ME, Tsiatsos T (eds) Internet of Things, Infrastructures and Mobile Applications. IMCL 2019. Advances in intelligent systems and computing, vol 1192. Springer, Cham. https://doi.org/10.1007/978-3-030-49932-7_44
Paley VG (2004) A child's work: the importance of fantasy play. Chicago University Press
Pont S, Janseen M (Jan 23, 2013) Jacobus. Van Goor, Amsterdam
Ratri D, Choi Y (2017) Children book app as emotional learning media. In: Proceedings of the 11th international conference on ubiquitous information management and communication (IMCOM '17). Association for Computing Machinery, New York, NY, USA, article 72, 1–8. https://doi.org/10.1145/3022227.3022298
Raver CC (2002) Emotions matter: making the case for the role of young children's emotional development for early school readiness. Soc Policy Rep Soc Res Child Dev 16:1–20. https://doi.org/10.1002/j.2379-3988.2002.tb00041.x
Ryokai K, Raffle H, Kowalski R (2012) Storyfaces: pretend-play with ebooks to support social-emotional storytelling. Proceedings of the 11th international conference on interaction design and children, 125-133. ACM. https://doi.org/10.1145/2307096.2307111
Ryu SJ, Tan JM, Wohn DY (2021) Dot's world: an emotional development support platform for children. In Interaction design and children (IDC '21). Association for Computing Machinery, New York, NY, USA, 568–572. https://doi.org/10.1145/3459990.3465198
Schoenau-Fog H (2011) Hooked!–evaluating engagement as continuation desire in interactive narratives. Interact Storytelling:219–230. https://doi.org/10.1007/978-3-642-25289-1_24
Schultz D, Izard CE (1998) Assessment of children’s emotion skills (ACES). University of Delaware, Newark
Silva V, Soares F, Esteves JS, Pereira AP, Matos D (2020) Adequacy of Game Scenarios for an Object with Playware Technology to Promote Emotion Recognition in Children with Autism Spectrum Disorder. In: Ahram T, Karwowski W, Pickl S, Taiar R (eds) Human Systems Engineering and Design II. IHSED 2019. Advances in intelligent systems and computing, vol 1026. Springer, Cham. https://doi.org/10.1007/978-3-030-27928-8_44
Soute I, Nijmeijer H (2014) An OWL in the classroom: Development of an interactive storytelling application for preschoolers. Proceedings of the 2014 Conference on interaction design and children, 261-264. ACM. https://doi.org/10.1145/2593968.2610467
Steinfeld A, Jenkins OC, Scassellati B (2009, March) The oz of wizard: simulating the human for interaction research. Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, 101-108. https://doi.org/10.1145/1514095.1514115
Stern SE (2008) Computer-synthesized speech and perceptions of the social influence of disabled users. J Lang Soc Psychol 27:254–265. https://doi.org/10.1177/0261927X08318035
Sun M, Leite I, Fain Lehman J, Li B (2017). Collaborative Storytelling between Robot and Child: A Feasibility Study. In Proceedings of the 2017 Conference on interaction design and children (IDC '17). Association for Computing Machinery, New York, NY, USA, 205–214. https://doi.org/10.1145/3078072.3079714
Sylla C, Pires Pereira IS, Sá G (2019) Designing manipulative tools for creative multi and cross-cultural storytelling. In: Proceedings of the 2019 on creativity and cognition (C&C '19). Association for Computing Machinery, New York, NY, USA, 396–406. https://doi.org/10.1145/3325480.3325501
Sylla C, Gil M, Pereira ÍSP (2022) Untangling the complexity of designing tools to support tangible and digital intercultural story telling in troubled times: a case in point. Literacy 56:3–17. https://doi.org/10.1111/lit.12263
ter Stal S, Catala A, Theune M, Reidsma D (2020) Designing a Smart Toy Interactive Setting for Creating Stories. In: Brooks A, Brooks E (eds) Interactivity, Game Creation, Design, Learning, and Innovation. ArtsIT DLI 2019 2019. Lecture notes of the Institute for Computer Sciences, social informatics and telecommunications engineering, vol 328. Springer, Cham. https://doi.org/10.1007/978-3-030-53294-9_44
Tielman M, Neerincx M, Meyer J, Looije R (2014) Adaptive emotional expression in robot-child interaction. Proceedings of the 2014 ACM/IEEE international conference on human-robot interaction (HRI '14), 407-414. https://doi.org/10.1145/2559636.2559663
Torres PE, Ulrich PIN, Cucuiat V, Cukurova M, Fercovic De la Presa MC, Luckin R, Carr A, Dylan T, Durrant A, Vines J, Lawson S (2021) A systematic review of physical–digital play technology and developmentally relevant child behaviour. Int J Child-Comput Interact 30:100323, ISSN 2212-8689. https://doi.org/10.1016/j.ijcci.2021.100323
van der Bolt L (2000) Ontroerend goed: Een onderzoek naar affectieve leeservaringen van leerlingen in het basis- en voortgezet onderwijs. Academisch proefschrift. Print Partners Ipskamp. https://hdl.handle.net/11245/1.177195. Accessed 24 Oct 2022
Verhoeven G, Catala A, Theune M (2020) Designing a playful robot application for second language learning. In: Interactivity, game creation, design, learning, and innovation. Springer, pp 385–394. https://doi.org/10.1007/978-3-030-06134-0_42
Vygotsky LS (1967) Play and its role in the mental development of the child. Sov Psychol 12:6–18. https://doi.org/10.2753/RPO1061-040505036
Wallbaum T, Ananthanarayan S, Borojeni SS, Heuten W, Boll S (2017) Towards a tangible storytelling kit for exploring emotions with children. Proceedings of the on Thematic Workshops of ACM Multimedia 2017, 10–16. https://doi.org/10.1145/3126686.3126702
Zancanaro M, Stock O, Schiavo G, Cappelletti A, Gehrmann S, Canetti D, Shaked O, Fachter S, Yifat R, Mimran R, Weiss PL (2020) Evaluating an automated mediator for joint narratives in a conflict situation. Behav Inform Technol 39(9):1022–1037. https://doi.org/10.1080/0144929X.2019.1637940
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has partially been funded by the Spanish Ministry of Science, Innovation and Universities under Juan de la Cierva programme (IJC2018–037522-I). The writing of this work has received financial support from the Consellería de Educación, Universidade e Formación Profesional (accreditation 2019–2022 ED431G-2019/04, reference ED431C2022/19) and the European Regional Development Fund (ERDF).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Author contribution
All three authors have contributed significantly to the authorship of this work. The specific labors are summarized in the following: Alejandro Catala: Conceptualization, Software, Resources, Writing - Review & Editing, Visualization, Validation, Funding acquisition; Hannie Gijlers: Conceptualization, Methodology, Formal analysis, Resources, Writing - Review & Editing, Supervision, Validation; Iris Visser: Conceptualization, Methodology, Software, Data Curation, Formal analysis, Resources, Investigation, Writing - Original Draft, Validation.
Ethics, materials and data
Prior to conducting the study, this research obtained the ethics approval by the ethics committee of the University and parental permission was obtained through an informed consent form. Research was conducted following approved research protocols and procedures. All materials, including software, have been created, produced and adapted as needed by the authors, and the data have been gathered exclusively by means of the research procedures described in this research.
Conflict of interest
The authors have no other relevant financial or non-financial interests to disclose and no further competing interests to declare that are relevant to the content of this article.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A
Appendix B
Appendix C
Analysis excerpt 1. Control condition: Joost and Emma. In the excerpt presented in Table 12, Joost and Emma did complete the general goal of the scene (turns 1–7). Both students show that they are actively involved with the activity, both verbal and auditory. In the excerpts we see that both students enact the story verbally, where they listen to each other and react on each other. However, they did not literally name the emotional state of Jacobus, nor did they speak about how Jacobus felt. Therefore, the emotional goal of the scene was not completed. These findings were scored accordingly in the coding scheme. Furthermore, Emma seems uncertain and asks Joost for help (turn 2).
Analysis excerpt 2. Control condition: Eva and Djay. In the excerpt presented in Table 13, Eva and Djay did complete the general goal of the scene (turns 1–10). Both students are actively involved with the activity, both verbal and auditory. In the excerpts we see that both students enact the story verbally, where they listen to each other and react on each other. Although the emotional state of Jacobus was not explicitly named, Eva makes a sad sound (turn 10), which might indicate that Jacobus feels sad after hearing that he is not allowed to come to the party. However, they did not complete the emotional goal of the scene, since they did not explicitly name the emotional state of Jacobus. These findings were scored accordingly in the coding scheme. Furthermore, Djay seems uncertain (turn 3) and puzzling (turn 7). Eva seems really confident and she even helps Djay (turn 8).
Analysis excerpt 3. Human guided condition: Jelle and Yana. In the excerpt presented in Table 14, Jelle and Yana did complete the general goal of the scene (turns 1–19). Both students are actively involved with the activity, both verbal and auditory. In the excerpts we see that both students enact the story verbally, where they interacted with each other and the human guide. They also explicitly named the emotional state of Jacobus (turns 18–19) after the human guide asked for it, therefor the emotional goal of the scene is also completed. These findings were scored accordingly in the coding scheme. Both students were sometimes distracted by something else and not paying attention to the activity (turn 7 and turn 13). Also, Yana began to talk about her own birthday party and the invitations (turn 8). Furthermore, both students seemed to feel at ease with the human guide. They reacted naturally and quite fast on her questions, where they looked the human guide in the eyes. There was a real conversation between the human guide and both students.
Analysis excerpt 4. Human guided condition: Collin and Caitlynn. In the excerpt presented in Table 15, Collin and Caitlynn did complete the general goal of the scene (turns 1–11). However, Caitlynn was passively involved, since she did not say anything and sometimes looked away from the activity (turn 7) or looked puzzling (turn 10). Collin was actively involved with the activity, both verbal and auditory. Collin also explicitly named the emotional state of Jacobus (turn 9) after the human guide asked for it, therefor the emotional goal of the scene is also completed. These findings were scored accordingly in the coding scheme. Caitlynn seemed to be a bit overwhelmed by the activity and seemed to feel uncomfortable with the human guide. She reacted really shy, did not speak and did not looked the human guide in the eyes. However, she was one of the youngest students. This was her second week at school and her teacher described her as a really shy and quiet child. Collin also looked puzzling at one moment (turn 6). However, he seemed to feel at ease with the human guide. He reacted naturally on her questions, where he looked the human guide in the eyes.
Analysis excerpt 5. Robot guidance condition: Simon and Noa. In the excerpt presented in Table 16, Simon and Noa did complete the general goal of the scene (turns 1–13). Both students were actively involved with the activity, but mostly auditory. They also explicitly named the emotional state of Jacobus (turns 12–13) after the tablet robot asked for it, therefor the emotional goal of the scene is also completed. These findings were scored accordingly in the coding scheme. Both students seemed to feel a bit uncomfortable with the tablet robot and the whole situations. It seemed like they did not know how to behave in front of the tablet robot. However, they both listened really carefully to the tablet robot.
Analysis excerpt 6. Robot guidance condition: Dylano and Amra. In the excerpt presented in Table 17, Dylano and Amra did complete the general goal of the scene (turns 1–20). Both students are actively involved with the activity, both verbal and auditory. In the excerpts we see that both students enact the story verbally, where they interacted with each other and the tablet robot. They named the wrong emotional state of Jacobus (turn 11 and turn 14), therefor the emotional goal of the scene is not completed. These findings were scored accordingly in the coding scheme. Both students really seemed to enjoy the activity and also seemed to feel at ease with the tablet robot. They laughed a lot, were curious about the next reaction of the tablet robot and were amazed about its ability to speak.
Appendix D
Episode list in Edit mode(left), filter view (center), and the control view (right). This is part of the controller app for the robot, which comes with an improved asset management module to make the enactment/play manageable and more user-friendly by reusing, grouping filtering and re-arranging assets in episodes. The enactment can be controlled from the app connected to the robot via Wi-Fi network link
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Catala, A., Gijlers, H. & Visser, I. Guidance in storytelling tables supports emotional development in kindergartners. Multimed Tools Appl 82, 12907–12937 (2023). https://doi.org/10.1007/s11042-022-14049-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-14049-7