Keywords

1 Introduction

Humanoid robots offer new opportunities for playful interactions with humans. Playful interaction design in Human Computer Interaction (HCI) is rooted in the perspective of humans playful creatures or as “Homo Ludens” engaging in playful, ludic activities, which take place within fixed limits of time and place, and according to freely accepted but binding rules [1]. Playful activities absorb the players and evoke intense feelings of joy and tension or excitement, bringing the players beyond the experience of the “ordinary life” [1]. The distinction between playful interactions, serious games and gamification has been made in relation to the non-utilitarian character of play [2]. Playful interactions have been associated with such aspects as curiosity, exploration and the experience of wonder [3], as well as with an inherently pleasurable experience [4]. Research studies explored the possibilities and effects of playful interaction in the field on Human Robot Interaction (HRI) including educational robotics. For example, [5] developed an educational robotic system with a driving robot and a programming-board with command-bricks to support tangible, social and playful interactions in context of school education. Other studies with children, focused on the potential of playful human-robot interaction for learning and cognitive development. For example [6] designed playful child-robot interactions for language learning and the results showed promising effects. Studies have shown that playful interactions with technologies can be beneficial not only for children but for human players of all ages. Playful activities develop their potential in physical and social interactions especially through the engagement of the whole-body of all players [7]. These aspects, especially physical and social aspects of playful interactions including whole-body engagement, have played an important role in the design of the application “NAO says” presented in this paper.

The application “NAO says” is based on the popular imitation game “Simon says”, in which three or more players follow the command “Simon says” and act on the follow-up task. If, however, the game leader Simon does not say “Simon says” but a player acts according to the task, this player must quit the game. In “NAO says” the humanoid robot NAO plays the role of Simon and asks the players to engage in playing a series of mini-activities by following the command “NAO says”. NAO is a small humanoid robot which affords multimodal interaction through speech, non-verbal sounds, visual pattern recognition, gestures and touch. NAO can be animated to move its head, arms, legs in space and to manifest emotional reactions through dialogue, sounds, body movements and light effects. The mini-activities include imitating body movements of the robot and solving simple mathematical riddles while playfully interacting though speech and touch. These playful interactions are embedded in a social context and include a group of players following the commands of the robot together at the same time. “NAO says” as a design-led intervention aims to create a pleasurable experience through playful human-robot interaction embedded in less constrained settings, without the pressure to follow strict rules of a game.

This paper reports on the design, programming, implementation, playtesting and evaluation of this multimodal, playful interaction with NAO in two pre-studies and the main pilot study with altogether 209 participants of all age groups. The reminder of this paper is structured as follows. Section 2 outlines the design and the programming of the application “NAO says” with references to the different versions of the game “Simon says”. Section 3 describes the design of the studies themselves, i.e. two pre-studies with university students and the main pilot study, and the research methods applied in these studies. Section 4 presents the results from the two pre-studies and the main pilot study focusing on players’ perceptions of the NAO robot as game leader and the game “NAO says”, and perceived levels of stress before and after playing the game. The paper ends with conclusions and recommendations for further research.

2 Design and Programming

The application “NAO says” was designed to afford playful interactions with the humanoid robot NAO and was inspired by the popular imitation game “Simon says”. The game “Simon says” is structured around playful interactions in a group of players, who follow the command “Simon says”, engaging in a series of playful mini-activities. These mini-activities in the “Simon says” game vary from game to game version, but usually include physical, cognitive and social components which allow to engage gaze, speech, and motion. “Simon says” game has been developed and applied in many different versions. The classic version of the game involves human players playing in the physical room and one of the players taking the role of Simon. Studies on “Simon says” with human players explored its effects for learning outcomes, especially in language learning. The study by [8] showed that playing “Simon says” had a significant effect on listening comprehension of senior high school students. The study by [9] showed that the game improved vocabulary mastery in learning English.

More recently, different versions of the “Simon says” game have been developed using technologies to support the gameplay. For example, [10] developed the “Simon says” game as a mobile application for mobile phones with commands focusing on color identification. A study in the context of long term care (LTC), applied a “Simon says” activity with a robot, in which older adults took turns as leaders [11]. The researchers concluded that robots are promising for social engagement of older adults who suffer from apathy [11]. Another version of the “Simon says” game was developed with a humanoid social robot and included a computational model of turn-taking to support a more natural interaction during the gameplay [12]. Finally, the study by [13] focused on bodily movements and implemented the human pose detection library OpenPose to capture players’ poses [13].

2.1 Design

The design of the “NAO says” game presented in this paper focused on the multimodal and multi-sensory playful interaction of the NAO robot and human players of all ages. The use case scenario for the design of the “NAO says” game was a popular public event “Long Night of Sciences” which takes place every year at research institutions all over Germany on a specific day in June. On this day, scientific and science-related institutions open their doors and invite general public to visit and actively participate in experiments, demonstration, lectures, science shows, and guided tours. Playing “NAO Says” was offered as an interactive game event during “Long Night of Sciences 2022” at Berlin University of Applied Sciences, Germany, on 2nd July 2022. The “NAO says” game was embedded in a social context of this public event with multiple, voluntary participants engaging in playful interactions with the robot. The setting of the scenario was defined to be a university laboratory room, which was open during the event for general public to walk in and participate in the game at defined times. Based on previous experiences from the event “Long Night of Sciences” and the character of the “NAO says” game, the scenario defined families with children and young people as the primary audience and target group.

The “NAO says” game was designed from the human-centered design (HCD) perspective, following the scenario-based design (SBD) approach [14]. Scenarios are task based and descriptive, i.e. events and activities are strung together in purposeful sequences and provide a real-world description of the contents, flow, and dynamics [14]. The design was developed and tested iteratively. The design process included a joint co-creation of a scenario-based script in a project team, joint programming by two authors of this paper, playtesting in two pre-studies with university students, and finally the implementation and evaluation in the main study with 190 participants of all ages during the event “Long Night of Sciences 2022”.

The gameplay in “NAO says” includes a series of playful mini-activities which encompass physical and cognitive tasks. During the gameplay, human players are asked by the NAO robot to follow only when they hear the command “NAO says”. The rule from the classic version of the “Simon says” game, in which a player drops out of the game if he/she follows although there was no command, was not included in the interaction design. The reason not to include this rule was the focus on the playful, more stress-free and less competitive interaction rather than following the strict rules of the game and players having to quit the game.

The “NAO says” gameplay includes a total of ten playful mini-activities. Some of these mini-activities were based on existing animations for NAO, which are available in standard libraries of the Choregraphe software used to program the NAO robot. These included the “Saxophone”, “Elephant”, “Gorilla” and “Take a picture”. These ready-made building blocks for the game were selected as suitable for playful interactions, since they contain both clear body movements and corresponding sounds, which enhance playful engagement. Further existing animations, such as the “Air guitar”, were combined with new sound effects, which were imported as free sound files from the Internet and integrated in Choregraphe. For the purpose of the “NAO says” game, some own animations were programmed in Choregraphe and added to the gameplay. The self-developed animations included: “Stand on one leg”, “Rub tummy, pad head”, “Wave arms above the head”, “Smile” and “Maths”. In total, the following ten mini-activities were used in the “NAO says” gameplay: (1) Gorilla, (2) Elephant, (3) Air guitar, (4) Saxophone, (5) Take a picture, (6) Stand on one leg, (7) Rub tummy, pad head, (8) Wave arms above the head, (9) Smile, and (10) Maths. Figure 1 visualises all mini-activities arranged into categories: animals, music, body and other.

Fig. 1.
figure 1

Playful mini-activities included in the “NAO says” game.

Full size image

2.2 Programming

The programming of the “NAO says” game was done using the Choregraphe software (Version 2.8.6). The game was designed in the English and German language versions which were tested in two pre-studies with students. Figure 2 visualises the programming of the “NAO says” game in Choregraphe with different elements such as animations, animated say, speech recognition and tactile sensors (bumpers).

Fig. 2.
figure 2

Programming of “NAO says” in Choregraphe (English version).

Full size image

When programming the game, four main challenges emerged: (1) How to program new mini-activities and which existing animations can be adapted for the purpose of “NAO says”?; (2) How to make the interaction with NAO possibly seamless with a larger number of humans players and observers present in the room and standing at a relatively large distance from the robot?; (3) How to make the gameplay exciting without repeating the same sequence of mini-activities? These challenges were addressed in the programming phase as described below.

How to program new mini-activities and which existing animations can be adapted to the purpose of the game? In order to create new mini-activities, such as “Stand on one leg” or “Rub tummy, pat head”, the Timeline editor, an integrated tool in Choregraphe, was to store different positions of the robot in a timeline and play one after the other as a fluid sequence of animations. The Choregraphe Timeline editor enables the programmer to adjust every single angle of NAO’s motors using simple sliders. Additionally, this method was combined with the use of the Animation mode. When NAO is in Animation mode, the angles of the joints on the real robot can be adjusted and the joint position saved as a point on the timeline. From there, NAO can be moved to the next position. This procedure results in a fluid animation from the individual positions at the end. When creating animations, special attention must be paid to the length between the individual positions of the animation. If the length is too short, jerky movements can occur. These not only look unnatural, but also cause the motors to heat up more quickly which can even damage the robot. In addition, special care must be taken to ensure that the NAO robot is always in a stable position. The ready-made animation “Gorilla” available in the Choreographe library demonstrates this problem: In “Gorilla”, NAO drops forward onto its hands and sometimes falls over, either because the animation is executed too quickly or because the ground is not ideally flat. Therefore, creating new animations like “Stand on one leg” was particularly difficult to implement, as NAO had to be kept in a stable position during the entire animation.

How to make the interaction with NAO possibly seamless with a larger number of humans players and observers present in the room and standing at a relatively large distance from the robot? The scenario was designed for participation of multiple players and observers, all present in one room with NAO during the public event at university premises. The presence of many participants enhances the risk of a high volume of background noises which may impede speech recognition of the NAO robot. Therefore, the decision was made to limit the number of mini-activities with human-robot interaction via speech. In fact, the only mini-activity, in which speech input from the participant is necessary, is the maths activity. In the maths activity NAO asks “How much is 3 multiplied by 3?” and expects the answer “nine” from the participants. Also during the game “NAO says” NAO also asks a number of times “Did you understand everything?” and waits until the answer “Yes” is said by a participant. However, these interactions via speech are only possible when there are no background noises in the room and possibly only one participant at a time speaks loudly and clearly. In the scenario at the public event with approx. 20 participants in the room at the same time, a loud and clear response was foreseen to not be feasible. Therefore, it was decided to keep the threshold very low so that everything that sounds similar to “nine” could be recognized by the NAO robot as nine. However, this method had the disadvantage as, for example, “nineteen” or other numbers are also recognised as “nine”. Therefore, the final threshold was set to 30%, i.e. any utterances that sounds 30% like “nine” are recognized as a “nine”.

How to make the gameplay exciting without repeating the same sequence of mini-activities? As described above, a pool of mini-activities was created to provide a variety of non-recurring playful mini-activities, and in this way to enhance the user experience. In order to make the gameplay exciting, the randomization principle was applied in the programming of the game. Randomness of game elements is linked to uncertainty, which is a frequently overlooked in game design, but an important element for the overall game experience as it holds players’ interest and enhances engagement [15]. To incorporate the randomization for the path-finding command “NAO says” before movement, a random variable was included which was then queried with an If-condition. The challenge here was that the positive feedback (praise) given by the NAO robot at the end of the mini-activity was not in the same programming level as the dice rolling of the random number. To address this challenge, a separate random number was chosen for each mini-activity and duplicated this mini-activity. In the final version, different responses of the robot were programmed depending on the use or non-use of the command “NAO says”.

The different versions of the “NAO says” game were tested in the two pre-studies and the final version in the main pilot study at the public event as described below.

3 Methods and Studies

Following the design and the programming phase, the game was play-tested with university students in two pre-studies and the final version was implemented and evaluated during the public event with participants of different ages. Playtesting is a popular method in game research used to test perceptions and preferences of players, allowing designers to modify the game before delivering the final version [16]. The key facts about the pre-studies and the pilot study are summarized below and in Table 1.

Pre-study 1.

The first pre-study involved a sample of ten university students, who volunteered to test the English version of the initial version of the “NAO says” game. Participants were asked to fill in an online survey before and after the game. One of the key results from the first pre-study was the wish of students to play the game in the German language version. Therefore the German language version was created in Choregraphe and tested in the second pre-study.

Pre-study 2.

The second pre-study involved a sample of nine students, who volunteered to test the German version of the “NAO says” game and did not participate in the first pre-study. This version of the game also included a slower pace of NAO’s speaking as the result from the first pilot study clearly indicated the need for slower speed to understand better what to do in each mini-activity. Like in the first pre-study, the participants were asked to fill in an online survey before and after the game.

Pilot Study.

The main pilot study took place during the public event “Long Night of Sciences” with participants of different ages. Out of approx. 260–280 participants on that day, 190 persons filled in the evaluation survey which was administered before and after playing the game with NAO. The survey was paper-based to ensure high participation of persons without digital devices and of younger children.

Table 1. Summary of the pre-studies and the main pilot study of the game “NAO says”.
Full size table

4 Results

The key results from the studies related to: (1) perceptions of the robot and the game “NAO says”, and (2) perceived stress level before and after playing the game are described in the sections below.

4.1 Perceptions of the NAO Robot and the Game “NAO Says”

The data about the perceptions of the participants of the NAO robot as game leader and of the game “NAO says” was collected via online surveys in the pre-studies and via a paper-and-pencil survey in the main study. Both online surveys included additional questions which were not asked during the main study due to the specific conditions of the public event. The online surveys ask the question How did you perceive NAO as a game leader? This question was answered by rating five pairs of semantic items from the Likeability Scale of the Godspeed questionnaires rated on a scale from 1 to 6 [17]. Table 2 summarises the results from the Likeability Scale.

Table 2. Perceptions pre-studies and the main pilot study of the game “NAO says”.
Full size table

The results show that in both pre-studies NAO was perceived as a likeable, friendly, kind, pleasant and nice game leader. The data also shows higher values in the pre-study 2 in which the German version of the game was used which may indicate that the use of the local language may have enhanced positive perceptions of the robot.

Next, perceptions of playful interactions with the robot were captured in all three studies using the simple question “How did you like the game?” and asking participants to assess their perception on a scale from 1 = not at all, to 6 = very much. The mean values were as follows: (1) pre-study 1, n = 10: M = 5.10 (SD .738), (2) pre-study 2, n = 9: M = 5.33 (SD .707), (3) pilot-study, n = 190: M = 5.25 (SD = .885). These results indicate that participants in all three studies enjoyed playful interactions with NAO. The foreign language version of the game in English in the first pre-study received the lowest value, which again indicates that the language choice affects user experience. The high average rating of M = 5.25 in the main study with 190 participants show that participants in different age groups liked the game.

4.2 Perceived Level of Stress Before and After the Game

Perceived psychological stress was measured to explore whether there were any changes in how stressed or relaxed participants felt before and after playful interaction with NAO. The data about perceived stress was collected via online surveys in the two pre-studies and via a paper-and-pencil survey in the main study. Psychological stress was reported by the participants before and after playful interactions with NAO using the Perkhofer Stress Scale, which is a validated single item scale [17]. The participants assessed their stress level on the scale from 1 = no stress (“fully relaxed”) to 6 = fully stressed (“anxious”) before and after the game. To explore the differences in perceived stress before and after playing “NAO says”, the dependent samples (paired) t-test was computed at the 95% confidence level and two-tailed p-value using IBM SPSS software. The comparison of means showed that in all three studies the mean values for perceived stress before the game were slightly higher compared to the values after playing the game. In the first pre-study (n = 10) the mean values were M (before) = 2.60 (SD .843) and M (after) = 2.40 (SD 1.174). In the second pre-study (n=9) the mean values were M (before) = 2.56 (SD 1.014) and M (after) = 1.33 (SD 1.000). In the third pre-study the mean values were M (before) = 2.11 (SD .910) and M (after) = 1.74 (SD .917). Table 3 summarises the results for all three t-tests.

Table 3. Paired samples t-test: Perceived psychological stress before and after the “NAO says” game.
Full size table

The results show that the means of perceived stress before and after the game differed significantly only in the main pilot study. It can be concluded that the perceived level of stressed was statistically lower after the game “NAO says” and changed from 2.11 ± 0.91 to 1.74 ± 0.92 (p < 0.001). The results also show that the initial stress level of the 190 participants in the main study during the public event was slightly lower compared to the two pre-studies with students, which may be explained by the leisure character of the event compared to participation in classroom settings. Nevertheless, the participation of volunteers visiting the laboratory during the public event limits the possibilities of generalizing the results of the study. It scan be assumed that the participants in the main study differed from the general population and from populations with special needs in regards to their level of initial motivation to participate as well as their interest in and attitudes towards robots. Therefore it is recommended to conduct a broader follow-up study involving a more diverse sample and including variables such as interest, motivation and attitudes towards robots.

5 Conclusions

This paper reported on the design, programming, implementation and evaluation of playful interactions during the game “NAO says” with the humanoid robot NAO in two pre-studies with students and one pilot study with 190 participants of different ages. The exploratory results in all three studies showed that the players perceived NAO as a likeable, friendly, kind, pleasant and nice game leader, and enjoyed playful interactions with NAO. Additionally, there was a significant difference in the perception of own’s psychological stress before and after the game with NAO in the pilot study with 190 participants. The results also indicate possible effects of different language versions of the game on user experience. The results presented in this paper are to be understood as preliminary, exploratory results and as a starting point for further research. Further studies should be conduced with diverse samples and look closer into possible effects of different versions of the game. The paper also pointed out several challenges in the design and programming of the game “NAO says” and how these were addressed. Further studies could explore in more detail which design strategies of playful interactions in games like “NAO says” and which types of feedback from the robot are most effective for specific target audiences. Furthermore, future studies could explore how different types of playful interactions with robots may affect the perception of mood and stress as well as physical stress measures.