A gamified social robotics platform for intensive therapies in neurorehabilitation

Abstract

The use of social assistive robots for interactive stimulation has strong potential in neurorehabilitation therapies. It is of particular interest in the case of pediatric patients to promote children’s motivation and adherence, specially when those robots are able of guide gamified activities, as it is the case of NAOTherapist. NAOTherapist is a Social Assistive Robotics (SAR) platform for hands-off rehabilitation based on upper-limb activities, that was originally designed for pediatric patients with Cerebral Palsy (CP) or Obstetric Braxial Plexus Palsy (OBPP). Formerly, it endowed the therapists with tools to perform rehabilitation exercises. This paper proposes the gamification of NAOTherapist in order to incorporate additional characteristics which allow its intensive use in new rehabilitation procedures, such as the Hand-Arm Bimanual Intensive Therapy (HABIT). This intensive therapy setting involves daily activities in several consecutive days, which require a strong engagement of the patients with the therapeutic methods and the acceptation of the NAOTherapist as a rehabilitation system. The gamified system shows very accurate results considering the different aspects defined in the USUS methodology; namely Usability, Social acceptance, User experience and Societal impact.

1 Introduction

Child neurorehabilitation new approaches seek to achieve the recovery of damaged neuronal zones and atrophied muscles by the repetition of different therapeutic exercises, both physical and cognitive. There is a special modality of these therapies, which are currently in the ascendant, for children with Unilateral Cerebral Palsy, in the form of Intensive Therapy Camps, such as the Hand-Arm Bimanual Intensive Therapy (HABIT) [1], created at Columbia University. The goal of HABIT is to help children to improve the dexterity and coordination of both arms in daily functions. Given its intensive nature, having fun and the use of game-like activities to keep children motivated is of special relevance in these kind of therapies.

Robots are starting to cover certain social needs (like elderly support and care), and are progressively integrating into new environments and application fields, where the human–robot interaction has prominence [2]. The appearance of new needs around the development of devices to improve the response of patients has opened new lines of research in the field of social robotics. The main routes of research aim to take advantage of the social and emotional attributes of these platforms to maintain patient commitment, as well as to motivate, educate, train, communicate, monitor performance, improve healthy habits and provide companionship and support to people [3]. This kind of social robotics platforms could be easily integrated into medical cyber-physical system (MCPS), storing and sharing digitally the patients’ evolution [4].

Employing robotics for interactive stimulation has strong potential compared to other technologies like video-games, especially in relation to children because they have the presence of a real partner [2]. This is of particular importance when treating children because it can encourage more direct involvement not only in the game but also in the activity. However, there are other issues of these platforms that are still in very early stages and present great challenges to the community [5], such as the autonomous control of human robot interaction in healthcare environments or even the ability to adapt the platform to the patient after several sessions.

In this work, we propose the introduction of gamification strategies in the NAOTherapist platform to make advances in different aspects of human–robot interaction, like usability, social acceptance, user experience, social impact and patient improvement. In addition, this work offers the evaluation results of the proposed social robotics-based therapeutic gamification platform within an intensive therapy environment in a Intensive Therapy Camp. During these Intensive Therapy protocols, patients receive rehabilitation for up to 6 h a day, so the use of this tool was proposed as one of the activities in their therapy plan. The incorporation of the gamified NAOTherapist would mean an improvement in the motivation and adherence of the participants, as well as an enrichment of the therapeutic work context. The evaluation was carried out during 10 working days following an adaptation of the USUS methodology [6], whose criteria measure usability, user experience, social acceptance and social impact. Data collection was exhaustive during those days: questionnaires for both patients and therapists, session logs, sensor data capture, discussion groups and interviews.

To date, ungamified version of NAOTherapist had been evaluated in a first contact with patients and in a long-term for 4 months [7, 8]. However, there was no evaluation in the area of socially assistive robotics in rehabilitation in a context of intensive work with daily sessions, where the level of motivation of therapists and patients is challenged, and leads to the necessity of changing constantly the therapeutic activity in order to maintain attention and effort. In this context, gamification techniques and dynamics that managed to perpetuate motivation and establish emotional bonds between the children and the robot are very relevant.

2 Background

This section introduces the Hand-arm Bimanual Intensive Therapy, which defines the environment where the gamified version of the robotic platform NAOTherapist was incorporated and evaluated. Then, it describes the robotic platform itself, including its main characteristics and previous evaluations.

2.1 Hand-arm bimanual intensive therapy

Hand-Arm Bimanual Intensive Therapy (HABIT) [1] is a type of intervention for children with hemiplegia which involves intensive bimanual training with the objective of improving the ability to perform bimanual activities in daily function. HABIT has been shown to be very effective [9]. The success is due to the application of daily intensive training based on many repetitions with exercise variability [10], progressive increase of complexity, motivation [11], and positive feedback [12]. These concepts also represent the needs of pediatric patients in this rehabilitation process.

In HABIT the children perform a multitude of therapeutic activities, hidden under a relaxed atmosphere of game. These activities, in turn, are designed to respect the individualized treatment, being personalized according to the needs of each patient. This setting is specially interesting in several dimensions for incorporating activities with a social robotics platform as NAOTherapist. From the point of view of the children it is attractive to play with an humanoid robot. From the point of view of the therapist, including additional sessions with the robot increases the variety of the activities the children can perform. From the point of view of social assistive robotics and human–robot interaction, and particularly for the case of NAOTherapist, it constitutes a great challenge since it is a demanding environment where the sessions with the robot should be designed in such a way that they meet requirements as maintaining the engagement and high commitment of the children, dealing with the novelty effect (loss of interest in the robot); autonomous adaptation to their daily needs; and being a useful and configurable tool for the therapists. At the same time, intensive therapy camps provide an unique experimental setting for evaluating the robotic platform with real-world patients.

The specific Intensive Therapy implementation where the activities with the robot were included was accomplished in the form of a camp of 21 days (Monday to Saturday) in the summer of 2017. This Camp was held at the European University of Madrid (UEM) following Charles & Gordon 2006 protocol [1]. To the authors knowledge, it was the first time the HABIT methodology was implemented in Spain.

In the Camp there were 10 patients and 14 volunteer therapists/physiotherapists participated. It was especially aimed at Cerebral Palsy (CP) patients with hemiplegia and ages between 5 and 13 years. During the Camp, the children performed a multitude of therapeutic activities which were designed to respect the individualized treatment, being personalized according to the needs of each patient. Rehabilitation focused on the affected limb/s of the patient and daily training for more than 6 h. In some cases there was more than one therapist per patient.

The robotic platform was deployed for 10 days, treating 10 patients and being used by the 14 therapists. At the end of the Camp, 110 clinical sessions were satisfactorily executed. Figure 1 shows one of these patients playing with NAOTherapist platform during Intensive Therapy Camp.

Fig. 1

Patient interacting with NAOTherapist platform in the Intensive Therapy Camp

2.2 NAOTherapist

Socially Assistive Robotics (SAR) emerges from the intersection of assistive robotics and socially interactive robotics. This category includes robots that provide assistance through social interaction [13,14,15,16]. Current trends of SAR seek to accomplish their goals with no physical interaction with the patient [17]. These robots should be able to move autonomously in human environments, interact and socialize with people.

NAOTherapist is a cognitive robotic architecture whose main goal is to develop hands-off upper-limb rehabilitation sessions autonomously with a social robot for patients with physical impairments [18]. The system incorporates a NAO robot as the social interactive entity and a RGB-D sensor (Microsoft Kinect 2) to monitor the users’ movements. The NAOTherapist system is designed to interact with pediatric patients with Cerebral Palsy (CP) or Obstetric Brachial Plexus Palsy (OBPP). The robot proposes a set of therapeutic activities so that the patient is training within a game context. The autonomous coherent behavior is achieved through an artificial intelligence technique known as Automated Planning [19].

The NAOTherapist prototype was initially focused on upper-limb rehabilitation [7, 8]. The system has five exergames to motivate the patient: mirror, memory (also called Simon), inverse memory, NAO says, teach me and dance with NAO. The therapist starts the robotic architecture and the robot shows upper-limb poses with its own arms that a patient has to imitate in front of the 3D sensor. If he does it wrong, the robot corrects him and shows how to perform the exercise correctly: visually (mirrored correction) and verbally. A typical rehabilitation session lasts from 20 to 30 min.

The capabilities of patients can differ widely, so it is necessary to customize the level of difficulty while training for rehabilitation purposes. This explains how the system behaves by being more permissive or not according to the performance and success of the patient during the session. The 3D sensor captures anthropometric data several times per second (joint mobility range). The pose comparison values and threshold are also used to change the color of the eyes of the robot from white to green according to the correctness of the pose. The system can be adapted to each user with specific poses and it will adjust the required accuracy automatically (adaptive threshold). At the end of every exercise, the robot rewards the user dynamically with dances, stories, etc. depending on the detected interest on therapy.

The platform was involved in three different evaluation episodes: first contact, long-term adherence and intensive therapy. The first contact phase was held from October 2014 to February 2015. During this period, 117 typically developing children interacted with the earliest prototype in an only session [7]. The main objective was to assess the child Robot Interaction (cHRI) provided by the platform. In the same phase, a pilot study was conducted with 3 patients for collecting feedback and new improvement requirements. In the second episode, the platform was deployed in the Virgen del Rocío University Hospital for a long-term adherence study [8]. For 4 months (November 2015 to March 2016), 9 patients with OBPP and CP had weekly rehabilitation sessions, the first two months with traditional therapy and the second two with NAOTherapist.

This paper refers to the Intensive Therapy Camp Evaluation. The system was highly improved since it would be evaluated in an environment of maximum demand [20]. Patients had to be daily engaged with the robot. Game mechanics were included as narrative immersion and new interactive game-like activities. The adaptation mechanism was also improved by making it more specific about the affected part of the patient. Additionally, a configuration interface which offer therapists the possibility to configure and execute the sessions by themselves was incorporated. Also the rewards catalog of the adaptive reward system was expanded. The process to gamify any social assistive robotic platform, as well as the evaluation methodology suggested, are described in the next section.

3 Gamifying socially assistive robotics

This section describes the main elements considered to perform the gamification of a SAR system as NAOTherapist. Firstly, we contribute a framework for Gamified SAR-based therapies, where we define the main elements required to perform such gamification. Secondly, we define a general use case, where we show how a regular therapeutic session is performed with the gamified NAOTherapist.

3.1 A framework for gamified SAR-based therapy

The fundamental requirements necessary to design a gamified SAR-based therapy are: (1) therapeutic goals, (2) patient’s interests, abilities and challenges, and (3) characteristics of the SAR platform. With these three ingredients the therapist would be able to complete the aspects defined in the table of Fig. 2. The first four aspects are related to the diagnosis and interview of the patient to determine the therapeutic goals, as well as the interests and challenges that the patient wants to achieve. The rest of the table contains aspects related to game mechanics divided into three fundamental categories: game-like activities (little games, big games), immersion (narrative, role-play, theatrical prop), and instruments (rewards, challenges, levels, score). The example or use case in Fig. 2 refers to a patient with cerebral palsy (diparesis), low mobility of the upper extremities and difficulties in handling objects (level III of the MACS scale [21]). The interviews determined that his main challenge would be to gain autonomy in his daily routine to be able to dress, eat and bathe himself. At the same time, his interests are explored and a great fondness for science fiction and space travel is detected. In order to improve the patient’s functional capacity, two therapeutic goals are established: the improvement of the upper extremity mobility ranges and ability for objects manipulation.

Fig. 2

Gamified SAR-based Therapy Framework

The therapist must be familiar with the SAR platform: e.g., a 50 cm humanoid robot with 5 degrees of freedom in its upper joints and a RGB-D sensor to track the patient’s movements. From these requirements, the therapist in collaboration with the robotics designer, design the game-like activities. In the first place, two "little games" are proposed: "mirror game" or imitation game in which the robot proposes a sequence of postures that the patient has to imitate. From the clinical point of view, this exercise trains proprioception and range of patient mobility. In the second "little game", called "pick-up", the patient has to pick up and manipulate a set of objects related to his daily life (comb, toothbrush, shower head). These games are subordinated to the main game or "big game" that pursues more functional objectives. This complementary game is the well-known "Simon says", where the robot challenges the patient to use objects of daily life related to dressing, eating or bathing. The therapeutic activities and objectives are implicitly embedded in the expected challenges without the patient being aware of it. Game designers refer as "suspension of disbelief" to describe the state of mind in which the player is aware that it is a game, but is willing to pretend that it is a form of reality [22].

Within the game mechanics, immersion aims to introduce the patient into a fictitious environment aligned with their abilities, interests and preferences, which encourages them to maintain focus and concentration during the sessions. In the example depicted in Fig. 2, a story is told in which the robot is a space explorer and accidentally crashes its spaceship into Earth. The robot will need the help of the child to be able to repair its body damaged components and its spaceship. The robot will not remember anything at first, but the more help it receives from the patient, the more memories it will share with him or her. This perspective in which the patient is committed to help his or her robotic friend fosters the bond between both and therefore improves the quality of the interaction.

Regarding gaming instruments, a system is designed to reward the patient after each activity with animations, dances or storytelling. The challenge aspect would be implicit in the game’s narrative: “helping the robot repair the damaged components so he can return to his planet”. The therapist can design different levels of difficulty for the patient and receive a score after each exercise.

Developing therapy sessions with a social robot may not guarantee an effective commitment to long-term therapy, since overexposure to it may cause the patient to become accustomed and lose the "novelty effect". Maintaining motivation and active engagement is one of the main challenges in child–robot interaction. The gamification aspects defined here are the necessary ingredients to design gamified therapies based on SAR. The proposed framework enhances the current robotic rehabilitation interventions by immersing patients in a game environment that suits their preferences, while meeting their personal goals and challenges.

3.2 Scenery of interaction: use case

The use case of rehabilitation sessions represented in Fig. 3 corresponds to the imitation or mirror game, in which the patient must imitate the different poses performed by the robot. Green boxes represent the training stage in which the robot and patient perform the exercises together and blue boxes refer to the welcome and parting interactive stage. The interaction flow of this and every game in NAOTherapist integrates the three main concepts, “request, return, and reward”. So, in this example, the robot request is defined by asking the patient to imitate the same robot pose, then the return involves those actions related to the pose verification and correction. Finally, the reward element appears when the exercise is finished. Figure 3 represents the integration of these three elements as well as the involvement of gamification elements, such as immersion.

Fig. 3

Execution flow of Mirror Game use case

The use case starts when the patient enters the experimental room and finds the robot placed in the demonstration area. Then, the system tracks the patient and starts capturing his/her body characteristics. The patient is one or two meters away from the robot in the training area. The robot greets and welcomes telling a story. After introducing the first exercise, the training begins. In the mirror game, exercises consist of a sequence of poses. Depending on the exercise configuration, the patient must maintain each pose for a certain amount of time. The robot is in charge of driving the training process giving instructions and feedback on what to do at each time. Each patient’s pose is verified with respect to that shown by the robot. If both poses differ, the system executes a correction mechanism. Patients have two attempts performing a pose correctly: after the first failed attempt, the robot shows the incorrect arm or arms and tells the patient that the pose must be corrected. In the second correction, the robot imitates the detected patient’s posture and shows how to move the arms to achieve the correct pose. This is called “mirrored correction”. These mechanisms provide helpful feedback to users and help them to get closer to the correct pose. If the patient fails after these two tries, the pose is skipped. The system executes the rest of poses that comprises the exercise sequentially until it finishes. A break is programmed between exercises, when the patient is rewarded by the robot. In these pauses, the robot shows animations, choreography or tell stories to increase motivation after each exercise. Once all the exercises are completed, the training is finished. The robot closes the session with a cheerful farewell, inviting him to play with him again the next day.

3.3 SAR-based activities

The NAOTherapist model for game-like activities is based on Automated Planning [19], which constitutes a robust and flexible solution to incorporate different games according to the therapeutic goals. The built-in games are described below.

3.3.1 “Mirror” game

In the Mirror game, the robot shows a set of preset postures by the therapist, which the patient must correctly imitate and maintain for a given period of time. While the patient imitates each of these poses, it is monitored that they are performed correctly, with the help of a 3D motion sensor. A common threshold is used for all patients for checking correctness. In case the patient pose is not considered correct, the system directs the interaction to provide instructions to the patient for correcting the pose. There are two more attempts, with two different types of corrections. First, the robot corrects the patient verbally, indicating which arm should be corrected (or both arms if applicable). In the second correction, the robot imitates the patient’s posture and shows him how to move the arms from that posture to achieve the correct pose. In this way, each exercise of therapy consists only of a set of poses that the robot shows and that the patient should try to imitate.

3.3.2 “Memory” game

The Memory or Simon game is an adaptation of the Electronic Simon, but using poses instead of colors. This activity consists of the following: the robot performs one or several poses in a row, which the patient must memorize and repeat correctly and in the same order. The difficulty of this game increases as rounds are completed, increasing the number of poses to memorize. This activity works to a greater extent the cognitive side of the patients, in addition to physics, being a good type of exercise for therapies.

3.3.3 “NAO says” game

Another game designed specifically for hand-arm bimanual therapies is the "NAO Says" game. This game is very similar to the well-known game of Simon Says, where the robot takes the role of Simon and issues instructions to the child. The kind of instructions given by the robot may consist of touching a part of the body (for example, NAO says touch your shoulder), or adopting a basic stance (NAO says sit down). In the same way as in the Mirror game, if the child does not perform the request correctly, the robot corrects him in different ways until reaching the maximum number of attempts or until he performs it correctly. In the case of touching a part of the body, the child can do it with either hand, since the method for monitoring this exercise checks the distance between the main parts of the body and both hands. This exercise provides a more cognitive aspect to the therapy. It works to a greater extent verbal comprehension, and planning and sequencing of patient movements. In order to perform this activity correctly, the child must have good body awareness and good proprioception.

3.3.4 “Dance with NAO” game

The Dancing with NAO game was included as another novelty. This activity is very similar to the exercise of Mirror, but hidden under a greater and distended atmosphere of game, more specifically of dance.

The execution flow of this game is as follows. The robot first tells the child that he is going to teach him a dance. Then, it reproduces the dance choreography completely. After that, the robot teaches the dance to the child step by step. This part of the game is very similar to Mirror, since here the robot shows different poses that the patient must imitate one by one. When all different poses belonging to the dance choreography have been completed and the child has learned the dance, the robot asks the patient to try to dance together.

For the point of view of the cognitive aspect, it exercises memory and procedural memory, since the robot performs first the sequence of poses one by one with the aim of carrying out all of them in a row afterward, similarly to the Memory game.

3.3.5 “Teach Me” game

The Teaching Me or Teaching NAO game implies a change of roles, in which the patient becomes the therapist showing poses to the robot that it should imitate later. The child is the protagonist of the therapy acting as an active subject and directing the session. In this way, the patient works to a greater extent the executive function of the movements’ planning, not just having to imitate another subject. He has to take the responsibility of being a good teacher. We expect that having such a greater prominence within the therapy, his motivation and involvement in the exercises increase drastically.

The possible poses that the child can teach are defined in a catalogue. This catalogue is available for both, the child and the therapist so that they can select which pose to teach. First, the robot asks the patient to teach it to perform a new pose, which is supposed to be unknown for the robot. Once the child performs the new pose, advised or not by the therapist, and holds it for a few seconds, the robot identifies that pose as a new one. It is considered as a new pose in the sense that though it was within the catalog of possible poses, but it was not being used in the sessions so far. It is at this moment when the robot tries to imitate the same pose shown by the child. In order to give a certain realism to this situation, a random component is introduced to simulate that the robot fails somehow to imitate the pose, being rather different from the one taught by the child. In case the robot pose is wrong, it realizes of its error. Then, it asks the child to remind him the correct pose again, in order to try doing it correctly again. The same can be done several times in a row, with different poses to teach.

4 Evaluation factors

Despite the fact that there are increasingly more robotic approaches integrated in care settings where social interactions occur, evaluating a SAR platform is a complicated task: (1) there are many aspects to consider from the clinical objectives for which it is designed, to the fluency of the patient-robot interaction; (2) there are hardly any evaluation standards or methodologies for SAR platforms in the literature [23]; and (3) most of the works have problems of evaluation continuity and present difficulties to find enough participants for the studies or to prolong them over time [2].

When making assessments, it is important to keep in mind that the perception that humans have of these robotic technologies is different from other computing devices [24]. SAR platforms evoke more anthropomorphic mental models, that is, users seek similarity to human beings in their form and behavior. Therefore, social robots are mostly considered as partners rather than as work tools.

This work considers the USUS methodology [6], a theoretical evaluation framework with a user-centered development from human–robot interaction perspective in work environments. This methodology also applies to the clinical practice since it can help to understand how to improve the design and construction of new platforms, as well as to evaluate the medical utility of the tool.

USUS deals with the assessment in collaborative human–robot situations and tries to answer a general question: “if people experience robots as a support for cooperative work and accept them as part of society”. Thus, it offers a holistic evaluation perspective. The evaluation factors defined in USUS are: usability, user experience, social acceptance and social impact. Usability includes effectiveness, that in our case is not fully evaluated from a clinical point of view (clinical improvement of the patients) that would require the isolation of the evaluated variables, higher number of patients and control groups, but on specific variables about the interaction with Naotherapist, as will be described later.

5 Experimental design

Intensive therapy Camp lasted 21 days and was held between July 13 and August 2, 2017. The training sessions were 5–6 h, and took place every day at the European University of Madrid, from Monday to Saturday. The project was approved by the Ethics Committee of Hospital Niño Jesús, with code R0066/16, in December 2016. A total of 10 children aged 6 to 13 years old affected by unilateral CP attended the camp. Each patient was assigned a personal therapist/s who accompanied them during the therapeutic activities, most of which were transformed into a game. One of these activities consisted of a rehabilitation session with the NAOTherapist platform, lasting approximately 20–30 min, which was carried out once a day for 11 days (10 exercise sessions + 1 calibration). This section describes the entire experimental process that was carried out for the collection of participant data.

5.1 Procedure design

Before beginning the study, clinical professionals were trained to use the platform. In a previous meeting they were introduced to the robot and learned how to use the graphical interface that configures the sessions and executes the system. From the first moment, the idea was that the therapists were able to manage the platform by themselves.

Once the study began, the schedules assigned to each patient were established daily. Therapists accompanied them to the room where they carried out the activity with the robot. Once there, therapists were in charge of setting up the session for their patients. All sessions followed the same procedure. For setting up the session, therapists selected 2 or 3 gamified activities. The available games were: mirror, memory, inverse memory, Nao says, dance with me and teach me. Although the therapists were totally free to choose any of these games, the most common session consisted of: mirror, memory and Nao says, except the last session that was to play dance with me. After this selection, therapists had to adapt the activities and establish progressions that guaranteed the patient’s improvement, that is, the poses, mimics and requests from the robot could be more demanding if they saw a favorable patient’s progress. Finally, this configuration was saved and the session was started.

During the session execution, patients stood about 1.5 ms from the robot which was initially sleeping in different positions. To increase the variety, the patient could find the robot sometimes sitting, sometimes lying down or even squatting. The RGB-D sensor was located just behind the robot. Therapists were located next, to configure and, if necessary, to give indications to the patient. The execution of the robot was completely autonomous, so there was neither teleoperation nor any kind of human intervention. The structure of the use case followed a structure in which every session began and ended telling a story, which helped to improve the patient’s immersion in the activity. The robot told them that he came from another planet. Due to an accident, his spaceship had crashed, and he needed their help to be able to self-repair and reconfigure his circuits. To do this, the exercises proposed by the robot were the key to getting back to his planet. A change of roles was raised in which for the first time the patient was the one who helped the robot. Patients always answered affirmatively to: “do you want to help me?” and they were very committed to this task. Every day the story continued and the robot gave more and more details about his planet and how much they were helping him.

After each gamified activity, the robot rewarded the patient with a personalized reward or paused to rest. The rewards were adapted to patient preferences. This was a key point as a proposal to improve motivation and adherence to the activity. The system considered the number of attempts the patient had needed to complete the exercise multiplied by a random value. This determined the probability that the reward was very good, good or instead, a rest was made. The idea was that patients were aware of an effect-reward paradigm: the more effort during the activity, the better and more related would be the reward received. Only in this way, it could be guaranteed that the patient was motivated to improve their progression throughout the study.

5.2 Materials

For the data collection, quantitative and qualitative methods were used in different phases, considering the perspectives of the patients and the clinicians. The materials were:

• Questionnaires and structured interviews. Three pairs of questionnaires (for clinicians and patients) were designed for each of the evaluation phases: pre-evaluation, post-session and post-evaluation. In total 6 questionnaires with items based on the Likert scale (from 1: do not agree to 5: do fully agree) and open questions. Except for the pre-evaluation questionnaire, the design and purpose of the different questions was aimed at evaluating each of the USUS framework factors: perception of usability, social acceptance, user experience and social impact of the NAOTherapist platform. The pre-evaluation was aimed at collecting demographic data of patients as well as their previous experience with technology.

• Objective data. During the patient-robot sessions, the perception system collected the angles of the patient’s joints and the evolution of the thresholds throughout the rehabilitation activities. Thresholds implicitly determine the patient’s ability to improve and adapt to the platform. The initial threshold of each pose is calibrated initially for each patient Its value is used to determine automatically whether the patient’s pose is correct. A decrease in the threshold of a patient represents he/she has improved in activities. It was also of interest to collect the number of attempts and other logs of the session flow.

• Observations. The observations recorded by the experts who were present throughout the study were also taken into account. Throughout the sessions, their impressions about the robot-patient interaction were collected as potential improvements of the system.

5.3 Hypotheses

According to the USUS evaluation factors and the purpose of this study, the following hypotheses were defined:

• H1. Usability: NAOTherapist is a usable platform.

• H2. Social Acceptance: NAOTherapist is accepted by the participants (patients and healthcare professionals)

• H3. User Experience: Participants are having good experiences when interacting with NAOTherapist platform

• H4. Social Impact: NAOTherapist has a positive impact in society.

• H5. Patient Improvement: Patients improve in NAOtherapist activities throughout the study.

5.4 Study protocol

This section describes the camp evaluation procedure relating the phases with the materials administered to evaluate the USUS factors. Figure 4 describes this relationship between phases, materials and evaluation factors.

As Fig. 4 shows, the study differentiates three chronologically ordered evaluation phases: pre-evaluation (Pre.), post-session (PS.) and post-evaluation (PE.). The pre-evaluation phase aimed at collecting sociodemographic data of patients, as well as their previous experience in technology through interviews and questionnaires (Patients=P. Pre. and Therapist=T. Pre.). After the pre-test, a presentation of the platform was made to the clinical professionals, before starting the Intensive Therapy Camp. The patients had a zero calibration session with the platform to initialize all threshold values.

Fig. 4

Evaluation Procedure based on USUS framework

The second evaluation phase was carried out after each session. Both professionals and patients filled out their corresponding post-session questionnaire (P. PS. And T. PS.). The objective of these questionnaires was to evaluate the factors of utility, social acceptance and user experience, as well as to collect comments and improvement suggestions for the next sessions. Thus, if something in the session was not going well, it could be solved for the following ones. During this phase, the system also collected anthropometric data of the patients (perception logs), that is, the angles of the patients’ joint skeleton and the progression of the thresholds throughout the sessions. These patient data aimed to demonstrate the usefulness of the prototype in terms of how the patient learns and improves in activities with the robot.

The last phase was called post-evaluation. It raised more global questions about the experience of patients and experts, and also about the future potential of the tool and its impact on society and their jobs. Both groups responded questions to evaluate to the four USUS factors (utility, social acceptance, user experience and social impact) through questionnaires, interviews and open questions.

5.5 Patients

The platform was adapted to the pathology treated in the intensive therapy camp, Infantile Cerebral Palsy (ICP) with hemiparesis (only one side was affected). The group of patients was quite homogeneous in terms of clinical condition. The inclusion and exclusion criteria were aligned with those of the Camp, so that all patients were eligible to enjoy the sessions with the robot without exception. The criteria to consider were:

• Inclusion criteria:

  • Patients aged 6–13 years suffering from ICP and hemiparesis.

  • Recruited for the Intensive Therapy Camp at UEM.

  • Clinically stable and capable to start the treatment.

  • Authorization by their parents or guardians with the corresponding signed agreement.

• Exclusion criteria:

  • Visual difficulties.

  • Pain that makes it impossible to perform exercises.

  • Other associated neurological pathologies.

Table 1 Patients that participated in the study

Table 2 Pre-test administered to participating patients to determine their previous experience and perception of technology (Likert scale 1–5)

Table 1 summarizes the information about the 10 patients who were chosen as participants of the first intensive therapy camp in Spain, of which 80% were males. The average age of the patients was 8.6 and the standard deviation is 2.0, with a difference of 6 years between the smallest and the oldest.

A peculiarity of the study was that one of the participants was Italian. The robot language was Spanish. Then, this patient had a therapist who helped him with the translation of the explanations. This occurred especially at the first sessions. Throughout the days the patient perfectly understood what he had to do and the therapist assisted him only in the translation of the storytelling that the robot offered daily.

The last column of Table 1 relates each patient to their therapists. Having more than one therapist depended on the needs of the patient and the workload that could be assumed by them. Importantly, the therapists in charge were those who configured, monitored and evaluated the patients during the study with the NAOTherapist platform. They were also responsible for monitoring and responding to the questionnaires related to their patients.

All participants completed 10 sessions with the robot except P10 who had to interrupt the treatment in the middle of the camp for personal reasons. The sessions of this patient had not been considered in the evaluation results.

With the objective of collecting data about the previous experience in technology of the participants, a questionnaire was made before starting the study. The objective was to determine their degree of acceptance toward technology in therapy and if they had previous experiences that could condition them.

The results of this questionnaire, based on Likert scale (1–5), are shown in Table 2. Regarding the use of technological devices, 70% of the patients had used tablets, 50% smartphones and 40% computers. Everyone used these devices almost daily (4.1 ± 0.99). Their experience in therapy had been positive since everyone recognized that they liked it (4.75 ± 0.79). On the contrary, hardly any of them had used some kind of technological device during their rehabilitation (0.3 ± 0.48) and none had done any kind of robotic therapy. Except for P04 who declared being afraid of robots, all participants said they would like to do their therapy with a robot and even have it at home. A very interesting evolution was that of the P04 patient who, in addition to recognizing his fear in the pre-test, in the first days he felt insecure with the NAO robot, but over time he ended up creating very strong emotional ties with it.

5.6 Clinical professionals

The group of health professionals consisted of 14 volunteers, see Table 3. The average age was 25.6 ± 6.25. 43% of the professionals were students: 4 were physiotherapy students (T01, T06, T09 and T11) and there was also a doctoral student (T05). The rest of volunteers worked as physiotherapists (T02, T07, T08, T10, T14) or as occupational therapists (T03, T04, T12). One of the therapists had also studied psychology and other physical education. As shown in Table 3, the background of the volunteers was quite heterogeneous within the scope. This was considered very positively, since the platform would be evaluated from different perspectives and all of them important in the field of rehabilitation. All professionals were Spanish nationals except T06 who was Italian and responsible for supporting the Italian patient.

Table 3 Healthcare professionals that participated in the study.

A pre-test was also completed by the therapists pursuing the same objective: to determine their previous experience with technology; to know their perception about how a robot can help in therapy and if, in their point of view, it could be difficult to learn to manipulate it. This test was administered the day of the platform presentation to the professionals, but before having any information about the system.

Table 4 Pre-test administered to participating clinicians to determine their previous experience and perception of technology (Likert scale 1–5)

The results are shown in Table 4. 50% of professionals used tablets, 85% had experience with computers and 92% had smartphones. They all used these devices daily. This result is quite consistent considering the average age (25.6) of the experts. Regarding the question of whether they used technology to innovate in their rehabilitation sessions, some of them responded affirmatively (3.27 ± 1.34), although few acknowledged having used it to improve treatment adherence (2.35 ± 1.15). The most used device was the Wii game console with sports games and the balance board pack. As to the use of a robotic platform for therapy, only one therapist says that he has used the Lokomat, a robotic gait orthosis equipped with a modern body weight discharge system [25]. None of them had used previously a social robot or virtual avatar that interacts socially with the patient. Most experts were optimistic as to whether they believed it would be easy to manipulate a robotic platform (3.42 ± 0.64), which is also consistent because it is a sample with considerable experience in the use of technology. Regarding whether they believed that the platform could be useful in pediatric therapy, most thought it was a very good option (3.64 ± 0.63) for both to improve motivation and adherence to children’s treatments. Regarding the fear of being replaced, they all considered that a social robot could never replace the therapist at work (1.42 ± 0.75), although they did recognize that there were mechanical tasks that could ease their work or was useful as an automatic mechanism for patient evaluation.

6 Results

In order to test the hypotheses raised, this section summarizes the main results of the evaluation of the NAOTherapist platform in the intensive therapy camp. The following sections are organized based on each of the target criteria: Usability (Sect. 6.1), Social Acceptance (Sect. 6.2), User Experience (Sect. 6.3), and Societal Impact (Sect. 6.4). Nine patients and 14 clinical professionals were fully involved in this camp. A total of 90 sessions + 10 calibration sessions were executed and evaluated. All of them were carried out without any incident. The results related to the post-session phase are average values of all the questionnaires filled out after each session. In the post-evaluation phase, a single questionnaire was filled out per participant about their general opinion of the experience.

6.1 Usability

Determining whether the platform NAOTherapist is usable or not, responds to the hypothesis: “H1. Usability: NAOtherapist is going to be usable”. Usability is defined as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" [26]. It is probably one of the most important evaluation factors of this study.

The evaluation of usability is subdivided into a set of indicators: effectiveness, efficiency, learnability, flexibility, robustness and utility. Thus, it also relates to the hypothesis: “H5. Patient Improvement: Do patients improve in NAOTherapist activities throughout the study?”.

6.1.1 Effectiveness

Effectiveness is defined as “the accuracy and completeness with which users achieve specified tasks” [26]. In other words, the ability of the system to perform the task for which it was designed. In our case, it is necessary to evaluate that NAOTherapist is capable of providing robotic rehabilitation sessions and that these sessions are carried out effectively, having an impact on the patient. Therefore, the patient’s objective progress is considered a relevant indicator of effectiveness. This progress is determined from the recollected data of the patient through the perception system, which contains information about the range angles of the joints and the evolution of the threshold values of the poses. Similarly, the number of corrections, attempts and improved poses provide very relevant information about the evolution of patients. For all the rehabilitation sessions developed, the distances between the poses and the resulting thresholds after the completion of them were collected, together with a label that indicates whether the corresponding pose was performed correctly or not. Additionally, the threshold information was then organized by patient, session and pose, being able to observe the improvement presented by each patient in each pose.

Table 5 Results related to effectiveness

Table 5 shows the results. A general improvement of participants can be observed. The data is coherent since the percentage of failed attempts is inversely proportional to the average progress. The column Improved/Retrogress poses refer to the number of poses in which the patient has finally improved/worsened.

It is important to note that the results obtained for each child are not entirely comparable to each other. This is because each patient performed sessions adapted to their needs, also designed by different therapists. Then, the progression is dependent, externally to the platform, of the therapist and the specific characteristics of each patient

Before starting the analysis of the patients’ progress, it is important to remember what information is captured by the platform during the rehabilitation sessions and how it is treated. The perception system captures the nearest user in front of the RGB-D sensor, also generating an anthropometric model of joint angles. When the robot indicates a pose to be performed, the target pose is compared with that set by the patient, obtaining a measure of the distance between them. Then, this distance is compared with the threshold corresponding to the pose for that patient. The adaptive threshold is in turn adjusted according to the correctness of the current pose. Thus, the information to observe the patient’s progress is based on this adaptive threshold, which will be reduced throughout the therapy if the patient really shows an improvement in the mobility of the affected area. Figure 5 shows the boxplots of the distribution of the progress for the affected arm and for the overall pose (minimum, first quartile, mean, median, third quartile and maximum). Isolated points are outliers. According to the results in this figure, 90% of the patients improved from 5% to 15% in their affected arm and also in the general average progress. Only one patient did not obtain such improvement due to his cognitive characteristics. Throughout the sessions an emotional bond was created so intense that he preferred to interact verbally with the robot, neglecting his training quality. This fact may seem negative, but the therapist in charge interpreted it as a productive situation because the degree of concentration of the patient was so high. Although the patient did not pay attention to the poses, the intensity of the therapy could be maintained without the need for robot corrections. This greatly increases the potential of the platform. These results could also be affected by the improvement of the motor skills of the patients when receiving intensive therapy treatment.

Fig. 5

Objective effectiveness indicator based on the patient’s progress

Additionally to the objective data, the opinion of the clinical professionals is also very relevant to evaluate effectiveness. Therefore, an open question (T.PE.Q21) was also formulated in the post-evaluation test to determine whether the patient improved in some functional, cognitive or motivational aspect from the point of view of therapists. 70% of therapists detected an improvement in their patients in terms of joint range, gross motor fluidity, motivation, attention and cognitive processing. 18% of the experts did not perceive any apparent improvement that could be attributed to the use of the platform, but to the general methodology of the camp. The remaining 12% did not express their opinion about it.

6.1.2 Efficiency

Efficiency is defined as “the resources expended in relation to the accuracy and completeness with which users achieve goals” [26]. In this study, resources and costs to reach the goals are related to the fluency of the patient-robot interaction. This interaction must be fluent enough to minimize the time/cost of the session. However, a global value for these resources cannot be established, since the interaction is perceived differently depending on the abilities of each specific patient.

Asking children about efficiency or fluency is rather complicated. Then, the questions related to these aspects were formulated to the therapists in each post-session questionnaire (T.PS.Q7) and in the global evaluation of the post-evaluation questionnaire (T.PE.Q2). The answers to these questions are shown in Table 6. According to the 5 point Likert scale (the scale goes from 1, strongly disagree, to 5, strongly agree). In both cases they considered the interaction was very fluent, obtaining results of 4.32 ± 0.69 in the cumulative average of all sessions and 3.88 ± 0.62 in post-evaluation. There is a difference of 0.44. Therefore, it seems that the therapists had a slightly worse perception for the fluency of the interaction after the overall experience. However, the average value for the post-session represents an average value of 90 questionnaires at different times, while the value for post-evaluation was obtained from 14 questionnaires performed the end of the study.

Table 6 Questions related to the factor of efficiency

6.1.3 Learnability

The USUS framework defines learnability as “how easy can a system be learned by novice users” [6]. This indicator is evaluated as the ease that patients have to understand the task with the robot. To evaluate learnability several questions were proposed to patients and therapists in each post-session questionnaire and the in post-evaluation. Table 7 summarizes the mean and standard deviation of the responses of therapists (T.PS.Q4 and T.PE.Q1) and patients (P.PS.Q6 and P.PE.Q1/Q2) at the different evaluation phases. The two groups determined that the patients were able to perfectly understand the task with the robot with values above 4.0. Patients were also asked if the robot’s poses were easy to understand/imitate (P.PE.Q2), the result obtained was 3.56 ± 1.01. They expressed that they had found some difficulties on imitating certain postures suggested by the robot. Given the heterogeneity in age and height, we believe the size of the robot could be small in some cases. Also, the fact that the robot is placed on the ground can make it difficult for some patients to get a good perspective.

Table 7 Questions related to the factor of learnability

In NAOTherapist, learnability is also related to the patients’ improvement. When they learn the activities there is more room for improvements. Therefore, part of the results of the effectiveness indicator (Table 5) are also relevant here. As aforementioned, 90% of the patients had an improvement according to objective data. The number of improved poses was 21.88 ± 7.52, much higher than the number of retrogress poses, 9.11 ± 7.01.

Finally, all patients without exception found the robot’s explanations clear and easy to follow. The feedback offered by the platform was clear and facilitated the postural control of patients. The design of the interaction encouraged patients to learn and improve their activities.

6.1.4 Flexibility

According to the USUS framework, flexibility is defined as “the capability to carry out a variety of tasks in unstructured environments and adapt to situations” [6]. In other words, it refers to the different capabilities and forms of adaptation provided by the system to achieve the objectives. As mentioned above, NAOTherapist has the ability to give a coherent response to the patient’s actions. The two interaction channels used by the system are verbal and visual, both used to achieve a natural course of activities and to provide feedback to the user. Thus, the flexibility of NAOTherapist is determined by evaluating the capabilities of the platform to guide patients through their interaction channels to achieve the objectives.

To evaluate flexibility, two questions were formulated to the therapists. The questions and results are shown in Table 8. The first one (T.PS.Q6), to know if the robot had guided correctly to correct the patient’s postures, was asked after each session. The average accumulated value is 4.06 ± 0.83, so the therapists unanimously considered that the robot’s feedback was useful for the patient. In the post-evaluation phase, it was asked about the ability of the platform to adapt to the patient’s conditions (T.PE.Q6). Therapists responded positively about the platform adaptation capabilities, although the think there is some room for improvement. Adaptive thresholds were individual values of each patient, but therapists felt that the system should consider additional information about the patient’s condition (emotions, previous attempts, objectives) before making the decision of being more demanding.

Patients were also asked about the usefulness of the feedback provided by the platform by a question related to the eye lights. They responded affirmatively (3.89 ± 1.17). They confessed to use visual feedback of eye color in many occasions to determine if they were close to the correct pose in the game of remembering the sequence of poses (Memory).

Table 8 Questions related to the factor of flexibility

6.1.5 Robustness

Robustness is defined by USUS framework as “the level of support provided to the user to enable a successful achievement of tasks and goals” [6]. In this study, this aspect refers to the capabilities of NAOTherapist, prolonged and consistent over time, to correct and prevent patients’ errors. The platform is designed to propose a set of pose-based activities to help patients with those that they perform incorrectly. The key to success is to make the system robust for all poses, all patients and all possible situations.

Table 9 shows a summary of the results obtained to evaluate robustness with two questions included in the post-evaluation questionnaire. One of these questions was for therapists, to know if they think that the corrections made by the robot were accurate enough (T.PE.Q3). The result 3.06 ± 0.77 is positive in average, but some therapists said there were occasions and poses where the corrections were not entirely accurate. The other question (P.PE.Q15) was for the patients. Their response was consistent with the opinion of the therapists, obtaining 2.67 ± 1.22. They think the robot sometimes asked them to repeat a pose that they considered they had done correctly.

The level of demand of the robot has been questioned in previous evaluations [7, 8], considering the platform as too demanding or pick when recognizing the poses. In many cases the errors occurred due to a problem of precision in the 3D-sensor recognition of the user’s skeleton. Other times, the system was too demanding with patients and made them repeat poses that could be correct from the point of view of the therapist. The robustness in the recognition and correction of poses could be the most criticized and with more room for improvement aspect of the system.

Table 9 Questions related to the factor of robustness

6.1.6 Utility

According to the USUS framework, the utility indicator is defined as “the capability of the interface to be used to reach a certain goal or to perform a certain task” [6]. The utility of NAOTherapist has been evaluated in previous studies with great acceptance by experts, family members and patients [7, 8]. For the evaluation of utility in the intensive therapy camp, three questions were asked to therapists in the post-session and post-evaluation phases (see Table 10).

After each session, the therapist in charge was asked whether the session had been useful for the patient (T.PS.Q12). According to the results of Table 10, the averaged value accumulated by all therapists is 4.10 ± 0.92, considering the experience useful for the patient. In the post-evaluation phase, two questions were formulated: whether the robot provided a positive therapeutic experience for the patient (T.PE.Q11), that obtained a 3.56 ± 1.03; and whether the robot was useful for therapeutic treatments (T.PE.Q12) whose average value was 3.31 ± 0.70. Both responses were very aligned, and although the result is lower in post-evaluation than in post-session, most therapists accepted its use in pediatric therapy. Based on the observations, they saw a lot of potential to work cognitive aspects, attention and functional motor activities. Also, some of them believe that the tool has a great diagnostic potential to measure the patient while performing therapy in a uninhibited and active way. The motivational incentive and its impact on patient therapy was an unanimous opinion among all experts.

Table 10 Questions related to the factor of utility

6.2 Social acceptance

Social acceptance of NAOTherapist is related to second hypothesis: “H2: Is NAOTherapist accepted by the participants?”. Although previous studies also took into account family members [7, 8], in this study the participants are the patients and the therapists in charge. The USUS framework defines social acceptance as “an individual’s willingness based on interaction experiences to integrate a robot into an everyday social environment” [6]. The indicators that evaluate this factor applied to this study are: effort expectancy, attitude toward using technology, self-efficacy, attachment and reciprocity. These indicators are derived from the UTAUT (Unified Theory of Acceptance and Use of Technology) model [27].

6.2.1 Effort expectancy

Effort expectancy is defined by the UTAUT model as “the degree of ease associated with the use of the system” [27]. In our study, it refers to the effort and difficulty for using or learning to use the NAOTherapist platform. This indicator was evaluated through two questions to the therapists in the post-evaluation phase (see Table 11).

Regarding the ease of deployment and operation of the robot (T.PE.Q16), most experts responded positively (3.50 ± 0.52). The second question obtained 4.13 ± 0.72, considering the robot configuration task as a very simple task. This configuration is performed through a graphical interface. Based on the observations, therapists generally considered that the NAOTherapist platform was quite easy to deploy, operate and configure. The exercise configuration interface offers a simple and intuitive design. Although the results were very good, it is important to highlight that the average age of the experts is 25.6 years old and they have high experience in technology. It is true that none of them had worked with a robot, however, in the pre-study interviews (Pre-evaluation questionnaires) they recognized to use electronic devices (tablets, smartphones and computers) daily, and even some of them had used video consoles and electronic games in their treatments. This previous experience is consistent with their perception of the effort expectancy.

Table 11 Questions related to the factor of effort expectancy

6.2.2 Attitude toward using technology

The attitude toward using technology is defined by the USUS framework as “sum of all positive or negative feelings and attitudes about solving working tasks supported by a humanoid robot” [6]. During the pre-evaluation phase, patients were asked about their previous experience with technological devices. Most of them used a tablet (70 %), computer (40 %) or smartphone (50 %) almost daily. Everyone knew what a robot was. Although they had never interacted with one, they showed a very positive predisposition to do therapies with robots.

This indicator was evaluated in the post-session and post-evaluation phases. The therapists were asked about the attitude and predisposition of the patients during the sessions and the patients were asked after every session if they had been focused and had struggled to do the exercises.

Table 12 summarizes the responses of both collectives. Both therapists (T.PS.Q9 and T.PS.Q10) and patients (P.PS.Q4 and P.PS.Q5) shared that the latter had been engaged to the sessions and trained hard. The average values of the sessions were above 4 on the 5-point Likert scale. Additionally, the same question was asked to therapists in the post-evaluation phase (T.PE.Q8). The answer obtained was 4.47 ± 0.72, considering that patients had a high commitment and motivation with the robot’s activities.

Table 12 Questions related to the factor of attitude toward using technology

6.2.3 Self-efficacy

The USUS framework defines self-efficacy as “people’s beliefs about their capabilities to produce designated levels of performance” [6]. To determine this indicator, patients evaluated their ability to fulfill the activities. The two fundamental tasks were typically to mimic the poses and to remember sequence of poses.

The self-efficacy indicator was evaluated using the two questions in Table 13. According to the results in this table, the patients were self-confident with the imitation part of the poses (3.11 ± 0.60). However, they acknowledged having more trouble remembering the sequence of poses in the Memory game.

Table 13 Questions related to the factor of self-efficacy

6.2.4 Attachment

Attachment is defined by the USUS framework as “an affection-tie that one person forms between him/herself and another person or object—a tie that binds them together in space and endures over time” [6]. Attachment is one of the most important indicators in NAOTherapist. The bond or emotional ties between the robot and the patient usually emerge naturally after a prolonged exposure to the robot. There is a personification of the robot considering it as a social entity. The patient-robot bond favors adherence and the desire to continue working with it.

This indicator was evaluated in the post-session and post-evaluation questionnaires (see Table 14). After each session, patients were asked if they wanted to play again with the robot tomorrow (P.PS.Q8). The cumulative average of all sessions was 4.89 ± 0.42. This extraordinary result shows that the attachment of all patients was very high and that they always wanted to play with the robot again. In the post-evaluation phase, two questions were asked regarding this indicator: if they would like to continue doing therapy with the robot (P.PE.Q11), that obtained a 4.67 ± 0.71; and if they would like to have the robot at home (P.PE.Q12), whose average value was 4.33 ± 1.00. Both results were very positive in terms of attachment.

Table 14 Questions related to the factor of attachment

An open question was also raised regarding the name of the robot (P.PS.Q9). The objective was to determine the degree of personification perceived by the patient. Most of them chose names of other camp mates, family members or pets, which demonstrates a positive affective bond to the platform.

Table 15 Questions related to the factor of reciprocity

Table 16 Questions related to the factor of embodiment

6.2.5 Reciprocity

Reciprocity is defined in the USUS framework as “the principle of give-and-take in a relationship, but it can also mean the mutual exchange of performance and counter-performance. It is the positive or negative response of individuals toward actions of others” [6]. Reciprocity attempts to determine if the user perceives that the interaction with the robot is real and it is not a simple “machine” that collects the data, that is, there is a reciprocal two-way interaction channel between the patient and the robot.

To evaluate this indicator, patients were asked two questions in the post-evaluation phase: if the robot could see them (P.PE.Q5) and hear them (P.PE.Q6). According to Table 15, patients answered that the robot was able to see them (3.22 ± 1.64), although it was uncertain that it could hear them (2.67 ± 1.32). Unlike past studies, due to the overexposure of the platform in such a short period of time, some patients realized that the robot was actually “deaf”. A fact that had not occurred until the intensive therapy camp, since that question was always scored better [7].

Another perspective of reciprocity is to imagine what else the platform could provide us. They were asked to imagine what other things they would like to play with the robot. In general terms, patients responded: board games, sports, hide and seek, cards or dancing together.

6.3 User experience

Evaluating the user experience with NAOTherapist deals with the third hypothesis: “H3. User Experience: Do participants have good experiences when interacting with NAOTherapist platform?”. The USUS framework proposes a definition of this factor based on the Alben’s general concept of user experience. It refers to “aspects of how people use an interactive product: the way it feels like in their hands, how well they understand how it works, how they feel about it while they are using it, how well it serves their purposes, and how well it fits into the entire context in which they are using it” [28]. The indicators that evaluate the user experience are: embodiment, emotion, human-oriented perception and feeling of security.

6.3.1 Embodiment

As defined in the USUS-framework, embodiment is described as “the relationship between the robot and its environment” [6], the perceived impression not only of the physical aspect but also of the user’s expectations. The evaluation of the embodiment in this study focuses on whether the patient’s expectations in terms of enjoyment have been satisfied by the NAOTherapist platform.

In order to cross the results, the same patient-centered question was asked to therapists (acting as an observer) and patients (T.PS.Q8 and P.PS.Q7) after each session. The results are shown in Table 16. Both perspectives agreed that patients enjoyed playing with the robot with values above 4.5 on average. In the post-evaluation phase, the same question obtains almost the same result (4.89 ± 0.33) as the accumulated post-session value. In view of the results, it can be stated that the system has satisfied the patients’ expectations with a new form of game-based therapy that was fun and enjoyable.

Embodiment is also defined as the user’s perception from a more descriptive point of view. To assess how users saw the robot, in the post-evaluation phase, they were given a list of adjectives and asked to mark the five adjectives that best describe the platform. The results are shown in Fig. 6. There are two categories: human-oriented and object-oriented adjectives. At first glance, it is observed that human-oriented were more frequently selected than object-oriented adjectives. In addition, positive adjectives (happy, beautiful, modern, easy) were more selected in both categories than negatives ones (impatient, clueless, silly, resistant). This trend determines that patients saw the robot more as a human than an artificial entity, attributing it positive characteristics.

6.3.2 Emotion

According to USUS framework, the emotion indicator “implies that people tend to interact with computers and robots socially” [6]. Emotions is a fundamental aspect to evaluate in human–robot interaction processes and even more so when users are children. For this, a cross-assessment scheme was proposed: the patient evaluated his own emotions and the observer (therapist in this case) responded about the patient’s perceived emotions. In this way, one could cross the results and draw interesting conclusions about personal and observed emotional perception. The emotions were evaluating using the SAM (Self-Assessment Manikin) scale on a five-point scale to assess the valence, arousal and dominance [29]. Emotional valence classifies positive and negative emotions—unhappy to happy, arousal assesses the level of excitement and alert—nervous to calm, and dominance determines the control over the situation—submissive to dominant. Figure 7 shows the SAM scale used in our questionnaires.

The evaluation of emotions was done in each post-session questionnaire to patients (P.PS.Q1) and therapists (T.PS.Q1). Table 17 presents a summary of the average accumulated values and the std. deviation for the valence, arousal and dominance categories. Figure 7

More visually, these results are plotted as radar chart in Fig. 8. In this graph both the perception of the patients and that of the therapists are drawn. According to the results, the patients felt very happy, quite calm and with great control of the situation during the sessions. Therapists agreed on the positive valence and disagreed slightly on the arousal and dominance. The differences determine that the patients considered that they had more control and were calmer, and the therapists saw their patients more submissive and nervous. However, these differences are so small and both perceptions were very aligned.

Fig. 6

Frequency of selection of the adjectives proposed in the post-evaluation questionnaire

Fig. 7

SAM (Self-Assessment Manikin) scale on a five-point scale to assess for the A valence (unhappy to happy), B arousal (nervous to calm) and C dominance (submissive to dominant)

Table 17 Questions related to the factor of emotion

Fig. 8

Difference in the perception of emotions between therapists and patients according to the SAM scale

6.3.3 Human-oriented perception

Human-Oriented Perception is defined in the USUS framework as “the capabilities of a social robot to simulate human perception” [6]. In relation to this indicator, the platform was evaluated in terms of how the movements are reproduced by both the patient and the robot. In addition to perceiving the user state, the system must also ensure that patients are able to reproduce the movements of the robot naturally.

The evaluation of this point focuses on how the therapist perceives the movements reproduced by the patients after each session (T.PS.Q5) and the naturalness of the robot movements (T.PE.Q7). Table 18 shows the results of the first question, which obtains a cumulative average value of 4.05 ± 0.67 of all sessions. This means that most therapists considered that, despite doing an imitation exercise with a robot, the patient managed to reproduce it naturally. The second response, related to the movements of the robot, obtained an average rating of 3.31 ± 0.87. Some therapists commented that the NAO platform presented certain physical design restrictions which limited their movements, e.g. elbow flexion less than $90^{\circ }$, moving fingers, etc.

Table 18 Questions related to the factor of human-oriented perception

6.3.4 Feeling of security

Feeling of security is considered a key aspect in human–robot interaction. “As soon as humans collaborate together with robots in the same environment, safety and security issues arise” [30]. One of the keys for patients to feel safe with the robot is to offer an interaction at a social distance that does not threaten their personal space. In all sessions with NAOTherapist, the patient was always at least one and a half meters away from the robot that remained in the same position throughout the session. In addition, the height of an NAO robot is 0.5 ms, so at the end any patient felt insecure when interacting with the platform.

However, there are other aspects regarding the feeling of security that were evaluated. The sessions consisted of a set of physical rehabilitation activities, and the robot offered the necessary feedback so that the patient could perform them correctly. It was interesting to determine if the flow of interaction could overwhelm or stress patients generating some kind of insecurity. Table 19 summarizes the results to the questions proposed to therapists (T.PE.Q5) and patients (P.PE.Q13 / Q19) in the post-evaluation phase. The results suggest that patients had barely felt overwhelmed during the sessions with the robot (3.75 ± 0.71). This statement was also shared by the therapists (3.62 ± 0.50). Nor they perceived that the platform had scolded them in achieving the exercises (3.78 ± 0.67), so that the feeling of security was maintained during the study.

Table 19 Questions related to the factor of feeling of security

A particular case was that of patient P04 who in the first sessions was afraid of the robot. However, after a few sessions this feeling disappeared being one of the cases that more emotional ties forged with it. In the last sessions, this patient was willing to touch him and sit with him. The conclusion reached is that for some participants it is necessary to make a previous introduction to the robot so that they know it before starting to play.

6.4 Societal impact

Societal Impact of NAOTherapist is related to the fourth hypothesis: “H4. Social Impact: Is the impact of NAOTherapist in the society positive?”. The USUS framework defines societal impact as “every effect of an activity on the social life of a community in general and more specific for the proposed framwework” [6]. This factor describes future assumptions about the impact that the robotic platform would have on society and its influence on neurorehabilitation treatments. The indicators that evaluate this factor and that have been applied to this study are: quality of life and working conditions.

6.4.1 Quality of life

Within the USUS framework, Quality of Life indicator is focused on “the integration of intelligent robotic technology into everyday life” [6]. This indicator was evaluated through interviews and open questions to therapists in the post-evaluation phase. In general terms, therapists valued the potential of the tool and the impact on the patient’s quality of life. The motivational incentive of the platform could strengthen adherence to treatment, so that patients arrived more excited at the clinic.

Regarding quality of life, two questions were raised to the therapists: The first, T.PE.Q13: What contribution does the robot make that a human therapist does not get?. Most of the responses recognized the improvement in motivation and concentration of the patient. They all stated that the game-like activities with the platform aroused their imagination and managed to keep their attention for longer. Under the patient’s perspective, the robot was an innovative element adding value to the intensive therapy camp. The second question to therapists was T.PE.Q21: Have you seen improvement in the patient by the use of the platform?. As previously mentioned, 70% of therapists stated that their patients had improved when using the platform. The majority of the responses focused on a functional improvement. In general, most of patients failed less and achieved better results with the robot postures, and recognized an improvement in attention, concentration and motivation. For all of therapists, it was a pleasure to see how patients enjoyed the rewards/dances, their faces of surprise and their conversations toward the robot room about what would be today’s game with NAO.

6.4.2 Working conditions

According to the USUS framework, working conditions indicator “includes all aspects affecting how people carry out their job and how employers take care of their employees, including things like working contracts, wages, working times and work organization”[6]. To assess this indicator, several questions were raised to therapists in the post-evaluation phase. The idea was to determine if they would be interested in having this platform in their clinic and what impact it could have on their work.

Regarding this indicator, therapists were asked if patients work with the robot as in conventional therapy. Table 20 shows the results to this question. Both in the post-sessions phase and in the post-evaluation phase, the responses were very aligned. Therapists believe that in conventional therapy they get the patient to work in similar conditions. They must strive to maintain the motivation and engagement throughout the session. On the other hand, they admitted that the platform could provide them with great help in this regard.

Table 20 Questions related to the factor of working conditions

A key question was T.PE.Q14: Would you like to have this robot in your rehabilitation center? 80% of the therapists were interested in using NAOTherapist in their rehabilitation sessions, the remaining 20% considered that some issues should be improved before involving it in their therapies. They were also asked about potential uses: T.PE.Q15: How would you use the robot in your therapies? Most of the responses proposed its use as a tool to improve adherence to treatment that could diversify the activities of a session. Using it as an incentive or reward after the session was also discussed. Other therapists saw great potential in automatic patient measurement compared to manual measurement methods. In goniometry, the measurements are dependent on the expert who takes them. Among different experts, different results are usually obtained. Therapists considered that capturing the patient’s mobility ranges while interacting with the robot could save time, ensuring the reliability of the data.

The last part of the interview focused on the perception of the tool as a threat in their work. Therapists were asked: T.PE.Q18: Do you think this robot could replace a therapist?. 100% of therapists responded that the tool could not replace them in their workplace since their presence was necessary to configure and monitor the session. NAOTherapist was always perceived by the therapist as a tool to support and monitor the treatment.

Cost-benefit analysis should have been an interesting study also in this topic of the evaluation. However, since it is more related to the business side, it was not analyzed.

6.5 Summary of the evaluation

The NAOTherapist platform was evaluated at the intensive therapy camp for patients with cerebral palsy. Among the participants, there were 10 patients between 6 and 12 years old and 14 clinical professionals (occupational therapists, physiotherapists, psychologists and physical educators). NAOTherapist participated for 11 consecutive daily sessions offering game-like activities to patients. A total of 110 sessions were carried out without any incident. The therapists assigned to each patient were responsible for configuring and adapting the session to the patient. From there, the platform ran autonomously.

The clinical study focused on evaluating the four factors within the USUS framework (utility, social acceptance, user experience and societal impact) through interviews, questionnaires and objective data collected by the system. Three evaluation phases were distinguished: pre-test, post-session and post-evaluation. More than 220 patient and clinical professionals questionnaires were administered and analyzed. More than 3 h of motion perception data per each patient were also collected.

In summary, Fig. 9 shows the average score obtained from 0 to 5 of the USUS factors and the key indicators. The overall results are very promising: 3.8 in usability, 3.7 in social acceptance, 4.1 in user experience and 4.0 in societal impact. In detail, we can see that there are indicators that have room for improvement. For example, the robustness indicator obtained a 2.9 fundamentally due to the lack of precision in the robot corrections and a 2.6 self-efficacy, since it was difficult for patients to remember the poses in the Memory game. In addition to trying to improve these two aspects, therapists expressed the need to continue improving the patient’s adaptive abilities from the feedback offered to the degree of personalization and configuration of the therapy (flexibility 3.7).

According to the observations, NAOTherapist was able to offer a fluent cHRI (efficiency 4.1) with an easy-to-follow methodology for the patient (learnability 4.1). Patients reproduced naturally the robot movements (human-oriented perception 3.7). 90% of the patients improved in the use of the platform by reducing their thresholds in most of the poses of the affected arm (effectiveness 4.5). The tool was seen as an added value in neurorehabilitation sessions as an incentive and improvement in adherence to treatment (utility 3.7 and attitude toward using technology 4.5). Therapists were able to easily operate and configure the tool (effort expectancy 3.8). Patients demonstrated that sessions were very good due to the strong positive affective bond with the robot (attachment 4.7). They had the impression that they were interacting with a real interactive agent (reciprocity 3.0), although some of the patients realized that the robot could not hear them. The emotional valence obtained was very positive, they felt happy, calm and dominant throughout the study (emotion 4.0). Patients did not see the robot as a threat, although the level of demand could frustrate them at some point (feeling of security 3.7). In general terms, therapists saw the potential of the tool and the positive impact on the patient’s quality of life (quality of life 4.1). The motivational incentive of the platform could strengthen adherence to treatment, so that patients arrived more excited at the clinic. 80% of the therapists were interested in using the current NAOTherapist prototype and the remaining 20% after some improvements (working conditions 3.9). The tool would allow them to diversify the activities of a session. Using it as an incentive or reward after the session was also discussed. Other therapists saw great potential in capturing the patient’s mobility ranges while interacting with the robot, saving time while ensuring the reliability of the data.

Fig. 9

Summary of evaluation factors in the study

6.6 Ethical considerations

Ethical issues also arises when using social robots and gamification tools in clinical and other environments. Such issues affect different aspects, from the well-being of the people, practical care and legal or justice related issues [31]. In this work, we have tackled indirectly with many of these aspects. For instance, data protection practices were established. Non-contact rehabilitation was performed to avoid blows between the robotic platform and the patients. Emotional attachment, deception, or trust are only a few of the ethical issues that have been taken into account in this study through different variables analyzed, like effort expectancy, the attitude toward using technology, self-efficacy, attachment, etc, most of them included in the area of Social Acceptance Sect. 6.2. SAR evaluation methodologies should be aware of the evolution of this area and include explicitly the ethical issues in their evaluation factors.

7 Discussion

The autonomous social robotic prototype NAOTherapist was designed to carry out hands-off neurorehabilitation sessions based on upper-limb gamified activities. This tool has an expert-friendly design to provide health professionals the opportunity to adapt each treatment to every patient. The system incorporates positive reinforcements that motivate and guide the patients during their treatment, improving the adherence of these to the therapy. This robotic technology has been validated in the rehabilitation of pediatric patients with motor needs compared to conventional treatments, providing very promising results.

The platform was involved in three different evaluation scenarios: first contact, long-term adherence and intensive therapy. In the first contact phase, 117 typically developing children interacted with the earliest prototype in a unique session. The main objective was to determine whether the cHRI provided by the platform was good enough to carry out the sessions. To date, this is the largest evaluation in the literature of SAR in pediatrics for motor rehabilitation. The lesson learned in this first iteration was that “sometimes less is more”, that is, given the possibility of including a multi-modal perception system to offer a more complex cHRI (voice recognition, emotions, etc.), an more simple approach was chosen guaranteeing a fluid, safe and efficient interaction. In the same phase, a pilot study was conducted with 3 patients of the Virgen del Rocío University Hospital (VRUH) where the platform demonstrated to be very promising and useful in therapy. In these first sessions with patients, the need to integrate mechanisms of adaptation and customization of therapies was detected, with the motto “every patient is a world”.

In the second episode, the platform was deployed in the VRUH for a long-term adherence study. For 4 months, 9 patients with obstetric brachial plexus palsy and cerebral palsy had weekly rehabilitation sessions, the first two months with traditional therapy and the second two with NAOTherapist. According to clinical measures, patients presented a slight improvement in their motor skills after this study. This was especially evident in those patients who attended all scheduled sessions. Relatives in general considered patients to be more motivated to attend the hospital when having sessions with the robot. Although the level of adherence was acceptably good, this evaluation compromised “the novelty effect of the platform”, that is, patients lost interest in the platform as time passed.

In the third episode, which is described in this paper, NAOTherapist participated in an intensive therapy camp with 10 patients with cerebral palsy with daily sessions for 11 days. The system was highly improved since it would be evaluated in an environment of maximum demand. When having daily sessions, patients had to be engaged throughout the study. One hundred ten clinical sessions and more than 220 questionnaires were administered and analyzed. Objective perception data demonstrated that 90% of patients improved in the robotic activities. In summary, the results of the USUS factors and key indicators were very promising (3.8 in usability, 3.7 in social acceptance, 4.1 in user experience and 4.0 in societal impact, in a 1–5 scale).

8 Conclusions

One of the main contributions of this work is the incorporation of gamification into robotic rehabilitation therapies to improve patient adherence. This study has shown against previous versions of NAOTherapist that the system manages to engage patients in a completely gaming environment, where the robot transports them to a different reality through storytelling, challenges and rewards. The lesson learned was that “every effort has its rewards”, since gamification mechanics had managed to maintain patient adherence throughout the study with significant results (4.7 out of 5 in the attachment indicator). However, therapists also expressed the need to continue improving the patient’s adaptive abilities from the feedback offered to the degree of personalization of the therapy. This need was taken into account for the following studies.

Each of those evaluations allowed the platform to evolve, incorporating functionalities and detecting new future needs. In total, 244 different children (21 of them pediatric patients) interacted with NAOTherapist in a total of 429 sessions executed without significant incidences. Of these 429 sessions, 206 were in clinical settings. Regarding to the rest of the stakeholders, 11 relatives and 20 clinical experts were consulted through interviews and questionnaires. Despite these extensive evaluations, there is still much work to do to achieve the ultimate intended goal: “the incorporation of technologies, such as NAOTherapist, in routine therapeutic procedures”. Although these studies offer an initial experience from different scenarios in the search of new requirements, the results presented here help to establish a solid base to extend this line of research aiming at offering novel tools to healthcare professionals. Evaluations of how gamification tools can help in the long-term [32, 33] in the clinical setting should be required.