1 Introduction

Research on interactive storytelling typically focusses on language and literacy skills [1, 5, 24] but also has positive effects on kindergartners’ social-emotional development [58]. The social-emotional development of kindergartners lays the foundation for children’s social-emotional development through middle childhood, adolescence, and beyond [12], and providing support for optimal environments at early ages may also contribute to develop both higher language and social-emotional skills [15]. Early social emotional skills are related to how socially, emotionally, academically and professionally skilled we are later in life [13, 18] and having higher social-emotional skills in kindergarten is related to building stronger relationships with peers, teachers, and parents [43] and professional outcomes at age 25 [20]. Fortunately, kindergartners can be taught social-emotional skills. The results from a meta-analysis by Durlak et al. [13] on the effects of social-emotional training programs in schools revealed that these programs not only improved students’ social-emotional skills, but also had a positive effect on students’ attitudes towards school, pro-social behavior and their academic performance.

An important aspect of social-emotional skill development in kindergartners is emotion regulation. Kindergartners need to learn how they can regulate their emotions in order to behave appropriate in different social situations [6, 28]. Regulation of emotions starts with the recognition of emotions. According to McCabe and Marshall [36] adequate emotion recognition skills are essential for optimal learning and development of children between 5 to 9 years old within the general school context.

Picture books can play an important role in the social and emotional development of kindergartners and more specifically in learning how to recognize emotions. There is a large selection of picture books with a social-emotional content that have the potential to introduce kindergartners to a wide range of emotions. Although van der Bolt [58] assumed that picture books could also have a positive impact on kindergartners’ social-emotional development, Kwant [30] was among the first who actually investigated whether this assumption was indeed true. In her quasi-experimental study Kwant [30] investigated the impact of reading and talking about picture books with a social-emotional content on the social-emotional competence of kindergartners. Kwant [30] assessed emotion recognition with a test that measured both the basic emotions (happiness, anger, fear and sadness) and the more complex emotions (shyness and jealously). Result of her study showed that the picture book activity facilitated emotion recognition among these kindergartners. The positive results can be explained by three mechanisms. First, it is likely that kindergartners by reading aloud and talking about picture books, develop insight into their own feelings and emotions, and into those of others, since picture books have the potential to introduce kindergartners to all kinds of emotions and provide information about the situation in which the characters feel and express these emotions [58]. Kindergartners can really enjoy or shiver at the experiences of fictional characters in books. This form of framed experience enables them to observe or, through empathy, undergo the experiences and emotions of fictional others [58]. This is closely related to empathy, the ability to recognize and understand other people’s emotions. Empathy normally emerges at the age of four and develops gradually toward adolescence and can be enhanced and trained by the help of adults who model empathic behavior [38], and it is essential in moral development to deal with more profound complex important issues such as bullying or racism at later developmental stages during the adolescence [39]. Picture books, like all fiction, represent emotions of fictional characters’, as well as their interpretation of each other’s emotions. However, unlike novels, picture books evoke our emotional engagement through images as well as words [38] and therefore have the potential to provide additional cues in the form of facial expressions, showing typical postures. Second, fiction has some advantages which are missing in reality. The introduction of emotions in a fictional world often occurs in a controllable and demarcated situation, still they can feel real for kindergartners. Fiction also offers more room for insight and reflection. Kindergartners get the opportunity to dwell on the emotions, since they are spectators and no actual participants [29]. Last, fiction invites kindergartners to actively engage in all kinds of storytelling activities. Some of these key points and design features can often be found in recent research on storytelling and social-emotional development (e.g., [26, 48]). Enacting the story is a storytelling activity that can foster the social and emotional development of kindergartners and more specifically it provides them the opportunity to actively learn to recognize emotions. As remarked by Bateman [4], storytelling provides opportunities for children to practice suitable emotional responses, and retelling emotional stories have been a technique either proposed and used in learning settings for social-emotional development (e.g., [8, 24]). A picture book introduces kindergartners to all kinds of emotions in a way that is appealing and understandable for them. When they enact the story they explore and play with all these emotions and social situations. What kindergartners actually do when they enact the story is to dramatize the story. The play-like action of dramatizing stories is not only highly motivating for young children, it also allows them to think in more sophisticated ways [40]. Through play kindergartners can easily remember, imagine, and recreate images and ideas from their previous experiences, even though these mental operations might be too difficult if the same children had to simply try to think about or discuss them [60]. Dramatization of children’s stories mimics this function of play and it also provides opportunities to build social skills [40] and early childhood teachers have a relevant role in preparing them to deal with either past and future events with deep emotional impact by using dramatic play and storytelling [14].

Dutch elementary schools often use so-called storytelling tables (in Dutch: de verteltafel) as an interactive storytelling activity in which kindergartners enact the story. The storytelling table is used to play-out or enact a certain picture book in pairs or small groups by making use of finger puppets, props and a decor. At this moment this type of interactive storytelling is typically used to foster language skills and literacy skills and less often to promote kindergartners' social-emotional development. This is remarkable, since the activity certainly invites children to explore all kinds of emotions [30] and therefore seems suited to foster the emotion recognition of kindergartners. A potential problem with interactive storytelling activities, including enactment is that kindergartners lack real-life experience of a full range of emotions and social situation, and have not fully developed their theory of mind, the ability to recognize how other people think and feel. This can be practiced by enactment and pretend play [40]. Teacher modeling and prompting as pointed in [8, 14, 26] can help students recall emotions and enact emotions during play. This suggests that for optimal exploration of emotions using enactment with a storytelling table, sort of teacher guidance is needed, to ensure that pupils focus on the emotional content of the story. Hence, a first important contribution of our research to make the field move forward is to demonstrate how guidance prompts and questions can be embedded into a kindergarten storytelling table setting and investigate to which extent the method for guidance instruction has an effect on the story enactment process and emotion recognition. Prior to establishing our experimental study, we revise the state of the art of technology-enhanced storytelling for emotional development to analyze how different settings can also be supported by technology and the main conclusions reached

1.1 Technology-enhanced storytelling for emotional development

Interactive storytelling, in which users can influence the storyline through actions and interactions with the characters, results in engaging experiences [46]. It also increases a user’s desire to keep interacting with the system [19, 21]. Nowadays, we can use technology to provide children with assistance during an interactive storytelling learning activity as illustrated in, for example to mention a few, Kocaman-Karoglu [23], Kory-Westlund and Breazeal [25], and Fridin [17].

Nevertheless, there are only a few interactive storytelling systems designed to promote young children’s emotional understanding. A seminal example is PUPPET [34], a 3D virtual environment with autonomous characters with different personalities, emotions and conflicting goals, where children (7-9 y.o.) controlled a sheep avatar through the mouse and keyboard peripherals to understand characters’ emotional states [34]. Further investigation is yet needed for younger children. A similar approach is shown in the work in progress by Muravevskaia [37], embedding narrative and dialogue with characters in the virtual environment. It uses a first-person view for the avatar and it is interfaced with a Virtual Reality (VR) headset and handheld VR controllers. Bratitsis [7] used a 2D digital story created in Scratch to explore how typically developing children show empathy towards situations related to autism spectrum disorders. The report emphasizes the potential of pre-scripted digital stories to support emotional understanding as a mean to improve inclusion. The work in [42] presents a picture book app, which based on active-coping and support-seeking strategies, offers pre-scripted emotional situations in the interactive story to be used in a tablet. To strengthen the impact of interactions to be carried out by children in response to character’s emotions, authors designed a range of very specific touch gestures instead of using simple taps (e.g., five deep breaths implemented as drag gestures on the character’s stomach and chest area to cope with anger). Based on similar coping techniques, the research in [45] discussed the design of a platform with two apps for supporting emotional development (one for children, other for parents). Rather than using pre-scripted stories for children, it enables interaction with a chatbot app that maintains a conversation (intended as a personal story) about emotions, supporting in this way its understanding, its coping, and gauging in this way their feelings. Parents can track children progress and be aware of exercises and therapists' support.

When targeting young children, there is a trend to consider more embodied and tangible interactions. For example, StoryFaces is a touch-pad based storytelling system that allows children to watch pre-made stories involving virtual characters with different emotions [44]. emoPuppet [35] combines a digital display with puppet fabrics, which can be operated easily by the child from another touch screen to control the facial expression. It does not suggest a specific working activity. FingerAR [3] combines hand-puppets that are augmented digitally on a mirrored screen in front of children through an Augmented Reality (RA) paradigm. The system allows children to change the facial expressions of puppets dynamically to express emotional state during the pretend play experience. The study concludes that, when allowed, children include wider range of emotional states and more often. Wallbaum et al. [61] propose a tangible interactive diorama which includes a puppet character that can be manipulated to move its arms and legs, and interchange facial expressions. It is also able to record and play audio to be used as part of the reenactment. Silva et al., [48] designed a playbrick, a tangible brick device with embedded sensors and display, used to show pre-made short emotional stories that children can match with 5 basic emotions (Anger, Fear, Happiness, Sadness, Surprise), revealing very high accuracy except for the stories related to Surprise emotion. It can be used with the humanoid robot Zeno to deliver human-like emotion facial expressions in the classroom. The system has been validated using a questionnaire with typically developed children, however authors' future work evaluation is expected to involve children with autism spectrum disorder (ASD). On this strand too, Azevedo et al. [2] reported three game activities aimed at imitating emotions, recognizing emotions, and matching emotions to narratives displayed in a PC, involving a tangible controller with 6 buttons labeled with emojis that match the 6 basic Ekman’s emotions, but the evaluation with children with ASD is limited to test whether the interface with the emojis is properly understood and usable enough to run the activity. Storytelling has also been used as an emerging approach to very complex social issues in adults to promote emotional changes in attitudes [62]. Despite such higher-order emotional skills involved, some tangible storytelling settings are starting to explore how intercultural issues can be treated through narratives in young children [53, 54]. In this case digital manipulatives allow children to combine items from different intercultural backgrounds, trigger story elements and reflect on the behavior displayed by the characters on screen.

The use of robots to embody agents and provide external representations for emotions in storytelling is being investigated intensely. Soute & Nijmejer [49] and Leite et al. [32] use small robots (the IxI Play and the Keepon, respectively) that stand in front of children to play out part of the narrative. While the first work carried out an exploratory study to investigate how to embed the robot and picture books in a regular school context to enable future applications, the second one designed a set of fully pre-scripted episodes which included high-level emotional situations with feelings such as social inclusion, frustration, and social cooperation. The study revealed that the recall when interacting in group (3 children) was lower than when playing individually, and that emotional understanding varies among situations. The ALIZ-E robot [56], embodied in a humanoid Nao robot, aimed to develop adaptive emotional responses and expressions in robot-child interactions. Following a Wizard-of-Oz (WoZ) protocol to enact the behavior, the observational study revealed more expressive behavioral responses by children and more positively reactions to the robot. Conti et al. [11] also used a Nao. Their results concluded that the inclusion of expressive behavior as part of the storytelling performance helped to recall more details. Interestingly, Ligthart et al. [33] went beyond WoZ and demonstrated how design patterns can be successfully embedded into a storytelling Nao robot with autonomous behavior to make the task more active for children. Although this research is not specifically focused on emotional development, it does suggest that behavior realization including emotions could be part of the enactment. The fluffy robot TEGA has also been used in teleoperated and pre-scripted story setups to investigate how entrainment leads to higher word recall [26], and how an expressive emphatic version of the robot contributes to vocabulary recall and robot likeability [24]. Both works stand out by having clear methods, combining active book reading with interactions with the robot and retelling to strengthen recall of key target vocabulary. More on the effect of using contextual prompts during collaborative storytelling is explored in [52], where the child had to retell the created story to a robot, not reporting higher recalls. The work in [55] looked at the narrative structure of the story retold by the child after using a setting to create stories that involved tangibles, an action tablet app and a Cozmo robot playing the role of main character. Results showed that introducing emotional responses by the robot led to differences in the way children incorporated emotions in their stories, which encouraged children to explore and link emotions to possible causes and consequences.

Table 1 summarizes the main aspects of the contributions reviewed. Although it is not aimed to extensively survey existing technology-enhanced storytelling settings for emotional development, it still offers an overview of the key issues and technological materials being developed in such state-of-art settings. As displayed in the table, the range of technology is diverse but the inclusion of robots as agents capable of rendering emotions is an important strand, which may help to mediate interactions and direct the learning activity. The robot settings differ from one to another, but it is common that robots remain in a demarcated location and adopt a companion/peer role using a touch screen for visualizing part of the story/content. None of the settings is shaped as a storytelling table, with tangibles, that would suit materials and approaches currently implemented in kindergarten even better.

Table 1 Summary of technology-enhanced systems to support emotional development
Full size table

It is worth noting we consider the works revised as good examples targeted at elementary/primary school age, which could also be used with older kindergartners. However, most of them involved children from primary school, often not reporting the age frequencies or means. Furthermore, some systems are suitable for exploring emotions, but they might be too open-ended for kindergartners without any further clear guidance or specifically designed instruction method. This entails most settings tested (and therefore the conclusions reached) may have their relevance diminished due to potential biases by older children, who have higher development. Hence, it is highly advisable that any study focuses specifically on kindergarten children, especially in view of the systematic review on physical digital play technology by Torres et al. [57] that concluded that there is a need to carry out more studies of play technologies in relation to social-emotional skills in general, and emotion regulation and emotion understanding/awareness in particular. Thus, besides maintaining a storytelling table approach based on picture books that can be transferable to a classroom in the future, our second important contribution will focus on the integration of robot technology to deliver expressive emotional behavior, using a clear and simple guidance method as for kindergartners to understand robot’s prompts and queries, and then evaluate how the guidance instruction method is characterized when using the robot. In this respect, it is important to compare both human guidance versus robot guidance in order to offer alternatives that can be taken in an informed way to the class. As next section will show, our approach uses a social robot character in the story, rather than a companion, to guide interactions that emphasize emotional states during the storytelling activity. Furthermore, to test whether the guidance instruction method works properly we will consider a control condition without guidance, which would correspond to the most basic storytelling table activity that can be implemented without technology in any kindergarten class.

2 Experimental user study

2.1 Method

Our goal for this experimental study is to evaluate a guidance instruction method suitable for a tangible storytelling table in kindergarten to help emotional development and to assess how well it can be supported by a robot taking part in the story. The key research questions that drive our study can be formulated as follows:

  • RQ1: Does the guidance instruction method work properly as to support emotional development? And how differently does human vs robot guidance function?

  • RQ2: Does the storytelling intervention lead to social-emotional learning?

  • RQ3: Does the guidance instruction lead to higher story recall?

This study used an experimental between-group design with three conditions. Two experimental conditions were developed in which kindergartners were either facilitated by a robot or a human guide as part of their interactive storytelling activity; kindergartners in the control condition worked on the interactive storytelling activity without any support. Full descriptions of the conditions are presented in section 2.3 (Settings). This study has a mixed-method approach. As described in detail in section 2.4 (Instruments), we look at multiple data as response variables about the enactment process, emotion recognition and story recall in order to carry out a comprehensive analysis. Hence, quantitative data was obtained from an emotion recognition test, measuring kindergartners’ recognition of emotional states. This test was administered as a pre- and posttest. Also, a story recall test was administered, measuring kindergartners’ ability to recall the story. Quantitative data regarding reactions, interactions and explorations were observed and coded from the video recordings of the interventions. To gain a deeper understanding of the enactment during the activities, a more qualitative analysis has been carried out based on the video recordings too. Two excerpts for each condition will be presented and described in more detail. These three sets of metrics, regarding enactment process, emotion recognition and story recall respectively, will serve to answer the corresponding research questions above, and the results and discussion sections will be organized accordingly for the sake of clarity.

2.2 Participants

A total of 73 kindergartners between 4 and 6 years old (M = 5.00 SD = 0.56) participated in this study. The ethics committee of the University approved the research study and parental permission was obtained through an informed consent form. The kindergartners (36 boys and 37 girls) were students of two different schools in the Netherlands. Within the Dutch education system elementary schools have eight grades, grade 1 (age 4) through grade 8 (age 12). Most schools combine groups 1 and 2 in the same class. In the first two grades the focus is on learning through play, and preparation for formal math, reading and writing instruction. that will start in grade 3 (at approximately 6 years of age). Kindergartners in each class were randomly grouped in dyads who were then randomly assigned to one of the three conditions. Due to the uneven number of participants one group consisted of three children. This group was excluded from further analysis. Resulting in 22 kindergartners in the control condition, 22 kindergartners in the human guidance condition and 26 kindergartners in the robot condition.

2.3 Settings

The setting is described in terms of the picture book, the storytelling table and the way guidance is provided in the activity for each condition.

2.3.1 Picture book

The picture book that was used in this study is ‘Jacobus’ written by Steven Pont and Mark Janssen [41]. The book was selected because it introduces the kindergartners to all kinds of emotions in a way that is appealing and understandable for them. The story is about a skunk called Jacobus who is, unlike the other animals, not invited to the special animal party. But Jacobus thinks a skunk can also be special, because a skunk can stink. Jacobus wants to show this at the party in front of all the other animals. The emotions that appear in the picture book are sadness, fear, happiness, anger, jealously and shyness.

2.3.2 Storytelling table and conditions

All students worked in dyads on a storytelling table activity that focused on the enactment of the picture book. The physical storytelling table consisted of a tabletop decor and physical puppets for the main character and subcharacters (see Figure 1). During the intervention all kindergartners made use of the storytelling table, but in a different way, depending on which condition they were assigned to. Kindergartners were randomly assigned to either one of the conditions: (1) an interactive storytelling activity with a technology facilitated scripted social robot or (2) with human guide facilitated scripted support or (3) the control condition: an interactive storytelling activity without any support or guidance. In the control condition, kindergartners enacted the story in dyads and made use of the storytelling table and the puppets. In all conditions kindergartners got a short introduction in which they were introduced to the storytelling table, the characters and the goal of the activity. Thereafter, kindergartners in the control condition independently enacted the story without any further support. In all conditions the researcher stayed in the room, but in the robot and control condition the teacher took a retracted attitude. The two experimental conditions are now explained in more detail.

Fig. 1
figure 1

Storytelling table with the robot platform

Full size image

Human guidance condition

Within this condition, kindergartners enacted the story in dyads on the storytelling table and were facilitated by a human guide who gave directions, asked questions about the story and gave prompts to the participants. The questions and prompts were based on a prescribed script, which is the same as in the robot condition. The script consisted of specific questions and prompts the child could respond to. For each category of responses, the script provided a reaction. This could either be silence or a follow-up question or prompt. An important part of the instruction includes to embed questions on emotions systematically to reinforce recognition and reasoning by children during the discourse of emotional events. An example of such a sequence is as follows:

Question: Can you tell me how Jacobus (main character) is feeling?

Response by the child: Jealous.

Response by the system: So you think Jacobus is jealous?

Other prompts require children to move around the table and enact the scene that is located at a specific part of the table. Kindergartners were able to interact with the puppets and move them along the storytelling table. After a short general introduction, which was the same for all participants, the actual activity started. The human guide gave directions and kindergartners received their guidance through the activity. The human guide was an elementary teacher, familiar in working with young children and trained to give guidance following the prescribed story.

Robot guidance condition

Within this condition, participants also enacted the story in dyads on the storytelling table but were facilitated by a robot that gave directions, asked questions about the story and gave prompts to the kindergartners. The questions and prompts were based on a prescribed script as in the human guidance condition. So, kindergartners in the robot guidance condition received the same feedback comments as kindergartners in the human guidance condition. Kindergartners were able to interact with the robot and the puppets, and move them along the storytelling table. After a short general introduction, which was the same for all participants, the robot introduced itself to the kindergartners. Then the actual activity started. The robot gave directions and kindergartners were guided through the activity by the robot.

2.4 Instruments

2.4.1 Robot platform

The robot used is based on an affordable platform developed in [10], which supports the deployment of tele-operated robot behavior (e.g. short utterances, display emotions, etc.) and that has been used as a social robot in learning applications [59]. The platform has been expanded and further developed to fit the purpose of this research. Specifically, a durable and robust robot case has been designed to resist intensive children’s handling (see Figure 1). The motorized wheels were replaced by a set of tire-wheels without motors as we wanted children to move the robot by hand to make it comparable across conditions and prevent any motion effects as a result of possible glitches in robot tracking [9]. The robot comes with designer and controller apps that allow us to both create and control robot behaviors following a Wizard-of-Oz (WoZ) approach [50]. To facilitate reproducibility across trials by a trained experimenter, the user interface is highly optimized to make the enactment/play manageable as large number of assets are available in realistic scenarios as the one in the present study which involves an actual picture book. Appendix D overviews the screens for these apps (Figs. 3 and 4).

2.4.2 Pre/posttest emotion recognition

The quantitative component of the study consists of an emotion recognition pre-test and a post-test. This test is a Dutch translation of the Assessment of Children’s emotional Skills (ACES) which is developed and validated by Schultz and Izard [47]. This Dutch translation of the test was used by Kwant [30]. The assessments measure social-emotional understanding including distinguishing and recognizing of other people’s emotions in social situations. The ability of the kindergartners to correctly recognize and label character’s emotional states was assessed. The test focused on the emotions fear, happiness, anger, and sadness, jealously and shyness. Before the test started, pictures of the emotional states (fear, happiness, anger and sadness) were presented and introduced to the kindergartners. The test started with 15 short social stories about the emotions fear, happiness, anger and sadness. An example of one of the stories could be: “Sam has made a drawing. His friends say he really likes the drawing. How do you think Sam will feel?” The researcher read the stories to the kindergartner. Then the kindergartner was given the opportunity to respond to the question by addressing an emotion and pointing to the picture with the corresponding emotional state. Then the researcher removed the pictures of the emotional states and introduced the participants with the terms shyness and jealously. The test continued with 6 more short social stories. An example of one of the stories could be: “Lars is going to football training. In the dressing room Pim shows his new tracksuit. It is exactly the tracksuit Lars also wants to have! How do you think Lars will feel?” The researcher read the stories to the kindergartner. Then the kindergartner was given the opportunity to respond to the question by addressing an emotion. Kindergartners could score a maximum of 21 points for the test. All emotions within the test also appear in the picture book.

2.4.3 Recall test

Quantitative data was obtained from a story recall test measuring kindergartners’ ability to recall the story. The six most important and meaningful scenes were selected for this recall test and a photocopy of these scenes was made. These photocopies were presented to the kindergartners. They had to individually put theses photocopies of the six scenes into the right order as a line from left to right (see Figure 2). Each well placed photocopy of the scene was scored with 1 point. Kindergartners could score a maximum of 6 points. We chose this accumulative scoring method because one misplaced photocopy would entangle the storyline, even though the next photocopies after the misplaced one might be in the right order.

Fig. 2
figure 2

Photocopies of the six scenes into the right order as a line from left to right

Full size image

2.4.4 Coding and analysis

Video-recordings during the interactive storytelling activity were made using a Canon EOS 550D. They were scored and analyzed in order to get insight into the processes during the activity. Kindergartners worked on the interactive storytelling activity in dyads but there may be differences between the two individuals. Therefore, we will look at the individual level as well. In order to analyze the video-recordings two scoring schemes were developed; one focusing on the dyad level and one focusing on the individual level.

The first scoring scheme focused on whether the dyads did complete the goal of each scene within the activity (Appendix A, see Table 7). There were two scoring options: dyads did complete the goal of the scene or dyads did not complete the goal of the scene. For example, the goal of the first scene was that Jacobus approaches the peacock and eventually asks him if he can come to the special animal party. But the peacock says Jacobus is not allowed to come to the party, because he is not special. If kindergartners enacted this scene accordingly, then they completed this goal. Two coders (the second and third author) coded all interaction data. Inter-rater agreement (Cohen’s Kappa) reached .71. The first scoring scheme also focused on whether the dyads did complete the emotional goal of each scene within the activity (Appendix A, see Table 8). For example, the emotional goal of the first scene was that Jacobus feels sad about not being invited to the party. If kindergartners explicitly name the correct emotional state of Jacobus, then they completed this goal. Inter-rater agreement (Cohen’s Kappa) reached .95.

The second coding scheme focused on the level of involvement at the individual level for each scene within the activity (Appendix A, see Table 9). There were four scoring options: the intended goal has been achieved with very low involvement, the intended goal has been achieved with low passive involvement, the intended goal has been achieved with active involvement (type 1), the intended goal has been achieved with active involvement (type 2). The level of involvement is scored based on a scale for scanning the level of involvement by Laevers et al. [31] (Appendix B, see Table 11). Inter-rater agreement (Cohen’s Kappa) reached .84. The second coding scheme also focused on which emotion the individual kindergartners named and whether they did complete the emotional goal of each scene within the activity (Appendix A, see Table 10). The specific emotion the kindergartner named was coded and each well named emotion was scored with 1 point. Inter-rater agreement (Cohen’s Kappa) reached .95.

2.4.5 Procedure

The study consisted of five sessions over a period of three weeks. Each session lasted up to a maximum of twenty minutes. The first week the researcher (who was also trained as a teacher) introduced herself to the children and then she individually conducted the emotion recognition pre-test for all kindergartners. By the end of the first week, the classroom teacher read the book ‘Jacobus’ following the guidelines presented by the research team. In the second week, the researcher visited the school and read the picture book once again on the first day of the week. Then, dyads were escorted to a quiet room to do the activity. Right after the activity, the kindergartners made a short recall test of the story. All activities were video-taped. In the third week the researcher individually conducted the emotion recognition post-test for all kindergartners. In all schools the same researcher administered the pre- and post-test and took care of the implementation of the experiment. Only the first reading session was performed by the regular classroom teacher.

3 Results

This section reports the analysis of indicators for the enactment process coming from the observations of the interactive storytelling activity, then the emotion recognition test, the recall test and, finally, the relation between the enactment process and the performance of the participants. Section 4, Discussion, will interpret the results and answer the research questions.

3.1 Enactment process indicators

As indicators for the enactment processes, we used both dyad and the individual measures. The mean scores for each dimension of the coding scheme are presented in Table 2.

Table 2 Mean scores of the enactment process indicators test for the three conditions
Full size table

Table 3 summarizes the results of statistical analysis carried out. One-way ANOVAs revealed significant differences among conditions for all the enactment process indicators. Tukey post hoc tests revealed that there were statistically significant differences between the control and human guided condition and between the control and the robot guidance condition, whereas no statistically significant differences between the human guidance condition and the robot guidance condition. In general, differences between grades were not found, except for the level of goal completion among dyads and the level of involvement at the individual level, in both cases kindergartners of grade 2 had a higher score than kindergartners of grade 1.

Table 3 Summary of statistics analysis for the enactment process indicators
Full size table

3.2 Emotion recognition test

The effect of the interactive storytelling activity on kindergartners’ recognition of emotions was examined through the emotion recognition test. The mean number of correctly answered items and the mean gain score on the emotion recognition pre- and posttests is given in Table 4. A paired t-test indicated that overall kindergartners scored significantly higher on the posttest (M = 16.20, SE = .32) than on the pretest (M = 14.33, SE = .34), t(69) = 5.85, p < .001, r = .58).Footnote 1

Table 4 Means scores of correctly answered items, mean gain scores and standard deviations for the emotion recognition test for the three conditions
Full size table

Paired t-tests indicated that kindergartners had a significant gain score in all conditions: control condition t(21) = −3.10, p = .005, r = .56; in the human guidance condition t(21) = −3.01, p = .007, r = .55; and in the technology enhanced condition t(25) = −4.24, p < .001, r = .65. The control condition yielded lower mean gains, followed by the robot guidance condition, and then by the human guidance condition. However, one-way ANOVA did not reveal statistically significant differences in gain scores between conditions (F(2, 67) = .235, p = .791, ω = −.02). An additional adjustment taking into account children’s school grade, did not result in statistically significant difference in gain score between condition, F(2, 66) = .091, p = .76, partial η2 = .001. An independent-samples t-test indicated that there was no significant difference in mean gain score between grades, t(68) = −.40, p = .197, r = .05.

3.3 Story recall test

The mean score for the recall test is given in Table 5. Recall is on average higher in the robot guidance condition, and lowest for the control condition. However, a one-way ANOVA did not reveal statistically significant differences in story recall scores between the three conditions (F(2, 67) = .819, p = .445, ω = −.01). An independent-samples t-test indicated there was also no statistically significant difference in recall scores between grades, t(68) = .299, p = .766, r = .001).

Table 5 Mean scores of the story recall test for the three conditions
Full size table

3.4 Relation between enactment process and kindergartners’ test performance

We report the relation between the emotion recognition gain scores and story recall scores with the level of involvement and the amount of correctly named emotions during the activity.

Pearson correlation was administered to determine the strength and direction of the linear relationship between the gain score on the emotion recognition test and, the level of individual engagement and individual emotion naming. There was a small positive correlation between level of involvement at the individual level and gain score on the emotion recognition test, r = .236, p = .049. Subsequently, a regression analysis including the level of individual involvement, and emotion naming on the individual level was conducted to investigate if these factors predicted individual children’s gain scores on the emotion recognition test. No significant regression equation was found (F(2, 67) = 2.32, p = .106).

With respect to the scores on the story recall test we found a positive correlation between the level of involvement at the individual level and the recall score r = .319, p = .007. A significant regression equation was found (F(2,67) = 3.809, p = .027), with and R2 of .102. The level of involvement on the individual level was a significant predictor of scores on story recall. Emotion naming at the individual level did not add significantly to the equation (see Table 6).

Table 6 Regression coefficients for scores on story recall
Full size table

3.5 Case analysis

Finally, to gain a deeper understanding of the enactment processes during the interactive storytelling activity in all three conditions, we present six representative excerpts and a thorough analyses for each excerpt (see Appendix C). The excerpts reveal in more detail the differences between the mutual experimental conditions and the control condition. All excerpts relate to the first scene of the story. The general goal of this scene is that Jacobus approaches the peacock and asks him if he can come to the special animal party. But the peacock says Jacobus is not allowed to come to the party, because he is not special. The emotional goal of the scene is that kindergartners literally name that Jacobus feels sad. The following paragraphs summarize the main conclusions that can be derived from the excerpts regarding kindergartners’ dialogues during the interactive storytelling activity.

In the control condition, most kindergartners completed the general goal of the scene and were actively involved. However, naming of the emotions rarely occurred, see Tables 12 and 13, for examples. In the human guided condition, most kindergartners completed the general goal and the emotional goal of the scene and were actively involved, except one kindergartner who is rather passively involved, see Tables 14 and 15 for examples. In the robot guidance condition, most kindergartners completed the general goal of the scene and were actively involved. However, they did not all complete the emotional goal of the scene, see Tables 16 and 17 for examples.

4 Discussion

4.1 RQ1: Does the guidance instruction method work properly as to support emotional development? And how differently does human vs robot guidance function?

The guidance instruction method worked properly since all enactment process indicators regarding emotion naming, children involvement, and goal completion show that both experimental (guided) conditions outperformed the control condition. Specifically, dyads in both experimental conditions showed significantly higher levels of goal completion. At the individual level, higher levels of involvement were found in both guided conditions. This means that kindergartners in the guided conditions enacted more scenes from the original story and were more individually involved in doing so. Kindergartners in the guided conditions recognized and named significantly more emotions (at both the individual and dyad level) during enactment compared to kindergartners in the control condition. This is in line with our expectations since kindergartners in both guided conditions received questions about the story that literally guided them through the story. Within the guided conditions kindergartners received questions about the emotional state of the characters, which resulted in significant higher levels of emotion naming. This is also consistent with observations in [24], who used a different setting but also considered questions prompted by the robot such as “How do you think the boy feels now?” when facing emotional situations. Kindergartners in the control condition were free to enact the story as they wanted to. If they wanted to change the complete storyline or just skip some scenes, they could. That would be fine in some of the revised systems, which were more open-ended and designed to support creative storytelling [3, 44], but then they would not have a way to consider more complex emotions such as jealously or shyness. Thus, being creative without actually enacting the expected story would result in lower levels of goal completion among dyads, as well as emotion recognition and naming in the experimental conditions. After all, it is important to remark that our setting is designed to use a specific curated story with emotional laden as content.

Both the human guide and the robot shaped the learning environment for the kindergartners through asking systematically specific questions that help them focus on to the main content of the story [22], stimulating the enactment of key scenes from the picture book. In principle, we might expect the human guidance condition to affect kindergartners’ enactment significantly more positively than the robot guidance condition, since kindergartners are more used to human-human interaction than human-robot interaction and humans are better able to communicate both verbal and nonverbal information than robots. However, no differences were found in any of the indicators between both guided conditions, which suggests that the guidance instruction method with the scripted approach is simple enough as to be implemented successfully, and therefore guidance mediated by a robot can be a potentially interesting tool for education that must be further explored and investigated. The designed guidance led to consistent results regarding emotion naming, involvement and goal completion, therefore bringing opportunities for enhanced learning. The qualitative examination of students’ enactment process provides some insight into the way students responded to guidance provided by the robot and the human guide based on the case analysis. In short, kindergartners that were guided by the robot seemed to feel slightly more hesitant or uneasy during the activity in comparison to kindergartners that were guided by the human, perhaps due to the limited cues and nonverbal behavioural feedback available in the robot, which might not be as natural as a human. Children did not always know how or when to intervene. The rather strict and monotone voice of the robot, not adapting to different situations, might also have contributed to this, which according to previous research may even negatively influence how people perceive working with the robot [27] and could even cause that people find the robot less trustworthy [51]. Kory Westlund et al. [24] demonstrated that expressive voices instead of flat voices in automatic speech synthesizers would yield advantages in children emulating emotional phrases by the robot. Our experimental design does not specifically account for investigating these fine-grain interaction aspects, such as voice-naturalness and adaptive/responsive eye-contact to direct children’s attention. Thus, they are worth investigating in specific experiments in the future and explore how the human and the robot guided conditions could become different in this regard.

4.2 RQ2: Does the storytelling intervention lead to social-emotional learning?

Another finding of this study is that, overall, the storytelling intervention as a whole led a significant positive effect on kindergartners’ recognition of emotions (as measured by the emotion recognition test), regardless of the condition kindergartners were assigned to. Children learned to recognize emotions. This is in line with the general idea that by engaging in interactive storytelling activities (such as storytelling tables) kindergartners develop insight into their own feelings and emotions and in those of others [58] and can learn how to recognize emotions [30]. However, no significant differences among conditions were found; therefore, the evidence gathered is not conclusive to establish differences as they all offered similar gains in learning to recognize emotions from short written stories. This leads us to reflect on some points that could be revised when designing experimental conditions for social-emotional learning support in future studies. For example, the interactive storytelling table session was a relative short part in the intervention. Furthermore, the activity was introduced as a play-like activity and kindergartners were not literally told that they were going to learn something from the activity. A more focused introduction of the activity that explained that the activity was intended to practice the recognition of emotions might have led to more conscious engagement in learning how to recognize emotions. In addition, kindergartners were not familiar with guided storytelling table activities. The guided storytelling table activities therefore were less familiar than the unguided activity. Kindergartners in the guided condition might therefore have focused more on the actual guidance (human or robot) than the content of the activity, and therefore might have learned less than we expected. Anyway, a very positive confirmatory point that results reveal is that the reading sessions included in the procedure during the first two weeks together with the exposure to the playing activity led to an overall positive effect in learning.

4.3 RQ3: Does the guidance instruction lead to higher story recall?

With respect to the recall of the story, we found no significant differences between conditions despite the lower numbers for the control condition. Hence, the guidance instruction condition cannot be considered significantly better for story recall compared to the control condition. This is a sensible result as the guidance instruction method is actually focused on emotional reasoning and not on story events, but we still needed to check whether the fact that questions in the guidance method are situated in the story might have an effect or not as reported in [11]. This result opens an opportunity for redesigning the guidance method, incorporating more questions regarding comprehension of the story [24], to make the goal of story comprehension and recall at least as important as emotion recall. Finally, concerning the relation between the enactment process and the story recall test, the results revealed that higher involvement on the individual level is a significant predictor of scores on story recall. Hence, another possible action for improvement on the guidance method could be to consider addressing the questions to each child evenly in order to enhance their involvement.

4.4 Final remarks, limitations and future work

This study means a first step in investigating the effect of a robot-enhanced interactive storytelling activity on kindergartners’ storytelling enactment, recognition of emotions and story recall. Kindergartners enacted a story on emotions using a storytelling table. To answer the research questions, three versions of the same storytelling table activity were compared in terms of students’ score on an emotion recognition test, as well as their enactment and recall of the content of the story. The three conditions differed with respect to the provided guidance. The two experimental conditions offered guidance provided by either a human guide or mediated through a robot, respectively. Kindergarteners in the control condition completed the activity without additional guidance. Before the actual experimental sessions, students got the chance to listen to the book twice. First it was read to them by their regular classroom teachers, and subsequently by the experimenter (also a qualified teacher), who guided the kindergartners during data collection.

In our study, we worked with a large and homogeneous sample of kindergartners, and did not mix kindergarteners with primary school students, as is reported in many related studies. Therefore, we argue that the conclusions reached are generalizable to the group of kindergartners, in contrast to many of the reviewed related work. Furthermore, the results are highly relevant for practice since using a tangible storytelling table is already part of everyday practices in many kindergarten classrooms. In such cases, integrating the instructional method based on tangible storytelling tables with robot guidance support can also be implemented by using any other robot platform capable of expressing emotions. Thus, this research is helpful to create in an informed way a range of environments in the kindergarten class based on storytelling tables, either with or without guidance, and with or without robot support. This makes our study complete and more comprehensive.

There are also some limitations associated to the study as already pointed out in the discussion, concerning aspects that can be examined in alternative experiments. Although the guidance method did lead higher emotion naming during the enactment, the results on story recall and learning emotions in the long term suggest that there are research opportunities for improvements in future work studies. It would be worth investigating a longer intervention by extending with more storytelling table sessions, so that it assures a more profound effect on learning. The guidance method could also be modified and expanded in a future work to check whether more complex guidance, not only with questions on emotion understanding but also about the story, can lead to enhanced story recall. Future work may also address and explore how interaction issues concerning robot embodiment (e.g., voice and behavioral cues to direct children’s attention) might enhance or not the children’s performance. Finally, we are using WoZ, which means that a person needs to take control of the robot behaviour. A future work would consist of bringing an autonomous robot version to support the guidance in the activity.