ENRICHME: Perception and Interaction of an Assistive Robot for the Elderly at Home

Abstract

Recent technological advances enabled modern robots to become part of our daily life. In particular, assistive robotics emerged as an exciting research topic that can provide solutions to improve the quality of life of elderly and vulnerable people. This paper introduces the robotic platform developed in the ENRICHME project, with particular focus on its innovative perception and interaction capabilities. The project’s main goal is to enrich the day-to-day experience of elderly people at home with technologies that enable health monitoring, complementary care, and social support. The paper presents several modules created to provide cognitive stimulation services for elderly users with mild cognitive impairments. The ENRICHME robot was tested in three pilot sites around Europe (Poland, Greece, and UK) and proven to be an effective assistant for the elderly at home.

1 Introduction

Modern assistive technologies help humans to improve their quality of life. In particular, assistive robotics is an important research field that focuses on monitoring and improving the daily life of people. Beyond this, which usually requires direct physical contact for assistance, the field of socially assistive robotics [25, 48] investigates the usage of robots that assist humans through social interactions (e.g. speech for appointment and medicine intake reminders [56]). Considering the ageing population and their increasing life expectancy, the societal and economical impact of such robots for the elderly could be significant.

Many socially assistive robotic systems have been proposed by various research projects [40, 41, 49, 57]. In this paper, we describe a robotic system that enables health monitoring, complementary care and social support for elderly people at home. The system is part of the ENRICHME¹ project [58], which aims at providing physical, social and cognitive assistance to elderly people with mild cognitive impairments (see Fig. 1).

Fig. 1

The ENRICHME system integrates an assistive robot for the elderly, here shown while interacting with a user

The ENRICHME system consists of three main elements [7]: the mobile robot platform, the Ambient Intelligence System (AIS), and the Networked Care Platform (NCP), as shown in Fig. 2. The first two elements are deployed in the elderly home, while the latter is accessible via the internet. The robot platform acts as a companion and a physical system interface for the elderly users. The AIS is a dedicated computer interfaced with the robot, the different smart home sensors, and the NCP. The latter includes a visualization tool accessible by the medical staff to monitor the data made available by the robot and the smart home environment, pre-processed by the ENRICHME federated systems.

Fig. 2

ENRICHME system architecture

A significant part of the ENRICHME’s research focuses on the robotics aspect of the system. The mobile robot is a customized platform modified to provide several important services, based on the needs and technological feasibility of our user requirements. The latter were defined with the active participation of elderly people and formal/informal carers [17], highlighting the need of services such as domestic monitoring, object finding, personalized reminders and cognitive games. In order to provide these services, the robot includes several innovative solutions in the design, multisensor perception of humans, RFID-based object localization, and interaction behaviors.

In this paper, we presents some key technological advances that enable the robotic innovations and the delivery of the above services. It includes in particular the following main contributions:

• 1. an improved multisensor human perception for an assistive robot enhanced by a thermal camera;

• 2. a new RFID-based object mapping and localization for mobile robots;

• 3. a set of use case experiences and qualitative results from real human–robot interactions with elderly users.

The remainder of this paper is organized as follows. Related projects and robot solutions are presented in Sect. 2. Then, Sect. 3 provides hardware and software details of our mobile robot platform. The approaches used to detect, identify, and monitor the user are presented in Sect. 4. The RFID-based object localization is explained in Sect. 5. The human–robot interaction and behavior coordination are explained in Sect. 6. Experiments, results and discussion of various use case scenarios are presented in Sect. 7. Finally, Sect. 8 concludes the paper summarizing achievements and current limitations of the ENRICHME solutions.

2 Related Work

2.1 Human Perception

Human motion analysis has been an important part of robust human–robot interaction (HRI) [8, 13, 29]. The robot’s perception process typically starts with the task of human detection, which can be made very challenging by problems such as occlusion and noisy observations in domestic environments. Hence, various algorithms have been proposed in order to perform robust human detection using 2D laser scanners [8, 42], RGB-D sensors [33, 35], thermal cameras [13, 28].

Human perception often includes re-identification, which aims to establish a consistent labeling of the observed people across multiple cameras or in a single camera in non-contiguous time intervals [6]. There has been many approaches focusing on appearance-based methods using RGB images [22]. However, using color causes problems as it is not discriminative for long-term applications. In [4], human re-identification is performed on soft biometric traits extracted from skeleton data and geodesic distances extracted from the depth data. The authors in [52] tackle the problem by extracting features from the skeleton of the person, such as distances between the joints and their ratios. However, as skeleton data is not robust for body motion and occlusion, these approaches have strong limitations. In addition, point cloud matching has a high computational cost.

The solutions implemented in ENRICHME and described in this paper improve previous human detection approaches by combining 2D laser, depth sensor and thermal camera information in order to achieve robust human perception in cluttered home environments. Also, our human re-identification goes beyond the state-of-the-art by using biometric features based on body part volumes [14]. Thanks to these features, indeed, people can be successfully identified under challenging conditions (e.g. person partially occluded by furniture or not facing the robot).

In assistive technologies, wearable devices [55] have some advantages over contact-free devices [30] regarding accuracy. However, they might be impractical in many cases, especially for elderly people with cognitive impairments, who could forget to wear them. Most of the contact-free approaches utilize color images to measure heart rate [3, 62] and respiration rate [60]. Since these approaches rely on color information, illumination changes can affect the measurements reliability. Thermal images can be used to avoid the effect of illumination on color images. In [30], the heart rate is estimated by applying fast Fourier transform (FFT) on the temperature signal. The authors in [1] monitor the respiration rate by detecting temperature changes on the nose region. These approaches, however, assume a fixed camera setup, requiring the user to stay still in front of the sensor for a considerable amount of time, which is not feasible for monitoring elderly people at home. In our system, instead, temperature, respiration and heart rates are extracted in real-time from the face of the user using a robot thermal camera [15], exploring for the first time the feasibility of fully contact-free physiological monitoring with a mobile assistive robot for the elderly.

2.2 RFID-Based Object Localization

Finding common objects in a house is a challenging task for many elderly people. In terms of object localization techniques, radio frequency-based approaches [9] are among the least intrusive solutions. Also, among them, RFID provides the smallest and cheapest devices [46].

Many RFID-based techniques rely on the Received Signal Strength Indication (RSSI), a propagation metric from the underlying radiowave process. However, a pure physical model is usually too complex to be efficiently modelled. Statistical approaches that model the tag detection event itself or the RSSI value, like in [18], avoid these complexities. A compromise solution is to combine a physical approach with Bayesian models. Joho et al. [37] first proposed to include the RSSI into a sensor model. They extended the model provided by Hähnel et al. [34], based on tag detection probabilities over a grid map. A further extension was proposed by Liu et al. [45]. This model used an optimized particle filter that can be used online.

In most of the previous work though, object localization relies on several fixed, often expensive, RFID antennas sparse in the environment. The solution presented in this paper, instead, relies on a single sensor installed on a mobile robot, making it one of the first robotic systems that use RFID technology for object mapping and localization in domestic environments. In addition, most of the existing systems adopt custom-made or active devices, increasing the size of the tags and the overall price of the installation. The new RFID object localization presented in this paper uses passive UHF tags, which are commercially available, cheap and with smaller footprints.

2.3 Human–Robot Interaction

Multiple research projects have been focused on providing personalized care for the elderly, more specifically mental and physical stimulation. Research has been carried out to provide robotic or computer based solutions to help the elderly to maintain and improve their cognitive level, such as cognitive games specially developed for the elderly. Two computer based solutions are represented by the M3W² and the Sociable³ projects. Both of them contain games that address different cognitive categories: attention, language, memory, executive functions, logical reasoning, orientation.

One of the robotic solutions proposed is a long-term study presented in [61]. A social robot was used for a period of six months to help the elderly improve their level of attention with a game called “Name that Tune”. The authors have found that after 22 trials (out of a total of 80 trials), the number of errors and the reaction time decreased while the difficulty level increased, thus successfully improving the level of attention of the end users. With respect to physical activities, the authors of [23] have proposed an exercise coach for the elderly. The robot proposes multiple exercise game scenarios (i.e., imitation game, memory game, workout game, sequence game) in an one-on-one interaction session. Personalized feedback is provided throughout the interaction session. As later presented in this paper, our system builds upon previous solutions to integrate advanced cognitive games in a mobile assistive robot for elderly people with mild cognitive impairments.

2.4 Elderly-Care Robotics

In recent years, the development of innovative technologies for a sustainable and high quality home care of the elderly has become of primary importance. Extensive research has been carried out in the area of mobile autonomous robot companions operating in domestic environments. In particular, socially assistive robots focus on motivating, coaching, and training users through non-physical interaction [48]. The FP7 CompanionAble project [49] tried to address the issues of social inclusion and homecare of persons with mild cognitive impairments, prevalent among the older population. The project focused on combining the strengths of a mobile robotic companion with the advantages of a stationary smart home. The FP7 Mobiserv project [54] developed an integrated and intelligent home environment for the provision of health, nutrition and well-being services to older adults. The goal has been to develop and use up-to-date technology like a companion robot, smart home, and smart clothes in an intelligent and easy to use way to support independent living of older persons. The FP7 Robot-Era project [19] aims to enhance the quality and the acceptability of current robotic services by using three different platforms for indoor and outdoor application. The project exploits previous knowledge on robotics and ambient intelligence technologies, cognitive-inspired robot learning architectures, elderly user-needs, methodology of design for acceptability, and standards fundamental for the real deployment. Another FP7 project, called GiraffPlus [16], tests how a network of sensors in cooperation with a robot can help older people live safer, more independent lives and enjoy social life from their home. The system includes a telepresence robot that moves around the person’s home and enables them to interact with family, friends and healthcare professionals via video conference. Their system comes with sensors throughout the home and in wearable devices that are designed to detect activities like cooking, sleeping or watching television, but they also provide medical information, like blood pressure and body temperature.

Using the robotic platform Bandit, Fasola et al. [24] presented a robot exercise coach that is designed to motivate and engage elderly users in simple physical exercise. Based on a user study with older adults, the authors validated the robot’s effectiveness at motivating physical exercise and showed that there is a clear preference by older adults for a physically embodied robot coach, rather than a virtual one, in terms of enjoyableness, helpfulness, and social attraction. The HealthBots project [36] examined the role of telepresence and humanoid robots in assisted living communities. The robots were deployed as companions and nursing assistants in a retirement village of around 650 residents. The authors showed that the majority of the residents enjoyed interacting with such robots. The authors in [39] presented a human-like assistive communication robot, Matilda, which was used for improving the emotional well-being of older people in three residential care facilities in Australia, involving 70 participants. They showed that Matilda positively engaged older people in group and one-to-one activities, making them feel more productive and useful through personalized care. Li et al. [43] present a pilot study conducted with Tangy robot, which is designed to facilitate multi-user recreational activities. The authors used Tangy to assist multiple Bingo game sessions with groups of elderly residents in a long-term care facility. They showed that Tangy was able to autonomously and effectively facilitate Bingo games in real interaction settings by determining its appropriate assistive behaviors. A post-interaction questionnaire showed that they enjoyed playing the game with the robot and that they liked its socially interactive attributes. Mucchiani et al. [51] developed design guidelines for an assistive robot based on query of clinicians, caregivers and older adults. In their study, authors found that hydration and walking encouragement were critical daily activities where human–robot interaction can be useful. They used a customized Savioke Relay robot to perform both. Through observation of older adults with the robot and post-interaction surveys, they found that perceived usefulness, perceived ease of use, perceived enjoyment and intention to repeat use were rated highly by the participants. They also observed that the human-human interaction was enhanced by the presence of the robot.

The H2020 Mario project [41] aims to help people with dementia by enabling them to stay socially active using touch, verbal, and visual human–robot interaction tools. Another H2020 project called GrowMeUp [47], focus on allowing the elderly people to live for longer in their own environment without losing contact with their social circles, staying active either via teleconference or other social facilities provided within the system. Additionally, they investigate behavior analysis solutions to detect and report emergencies or respond to the person’s non-verbal cues and commands. The RAMCIP project [40] aims to develop a service robot, capable to assist older persons in a wide range of daily activities, being at the same time an active promoter of the user’s physical and mental health by deciding when and how to assist the user. It is accomplished either by initiating a multi-modal human–robot communication or by fulfilling a robotic manipulation task.

Our ENRICHME system builds on top of these projects by adding robust multi-sensor human perception, customized interaction tools, and an RFID ecosystem to locate personal items in the user’s home environment, which are later presented in this paper. It also incorporates a non-intrusive solution to monitor the user and collect relevant physiological parameters [15]. In addition, it integrates an advanced robot behavior framework to coordinate high-level tasks for assisting user’s daily activities [27].

3 Robot Platform

A customized version of TIAGo Iron, shown in Fig. 3, was designed to fit the requirements of the ENRICHME project. The following sections detail the hardware and software architectures of the robot platform.

Fig. 3

Evolution of TIAGo iron to the ENRICHME version

3.1 Robot Hardware Architecture

In order to get TIAGo to fit the requirements of the ENRICHME project, several modifications on the hardware is done. The base robot, which includes a differential drive base, a lifting torso and a pan/tilt head, is kept. Several sensors such as RFID system, thermal camera are added on top of the base robot.

In an RFID system there is at least one reader and typically multiple tags. Tags are very simple radio devices, capable of transmitting a short signal after a reader request. These tags do not require a battery, as they can be powered by the reader’s radio signal itself. This reader is the only active radio frequency device on the RFID network. Improvements in the design of RFID readers have increased their popularity in the robotics field. Modern readers are compact enough to be embedded on a mobile robot,

The thermal camera, RFID antenna and its reader, the environment sensor and the touchscreen is added to the base design. The following hardware devices were selected:

• Optris PI450 thermal camera

• Mti Wireless Edge LTD. MT-242025/TRH/A antenna

• ThingMagic Mercury6e RFID reader module

• UiEM01 Environmental monitor providing temperature, humidity and light measurements

Figure4 presents the hardware architecture of the robot, specifying the connection buses used to integrate the different devices required.

Fig. 4

Hardware architecture of the robot platform

3.2 Overall Software Architecture

The ENRICHME system consists of four main software modules: (i) navigation, (ii) human perception, (iii) RFID-based object localization, (iv) interaction and behavior coordination. An overall schematic of the system is presented in Fig. 5. Using 2D laser scanner and depth sensor, an advanced navigation software is implemented. The details of the navigation software is described in the next subsection (Sect. 3.3). Human perception performs human detection, tracking, and reidentification using 2D laser scanner, depth sensor, and thermal camera (Sect. 4). Object localization uses RFID antenna and reader to detect RFID tagged objects in the user’s flat (Sect. 5). Finally, audio-visual and spatial interaction with the user is implemented by the interaction and behavior coordination module (Sect. 6).

Fig. 5

Software architecture of the robot platform

Standard software operating systems and middle-wares have been adopted in the ENRICHME system. The robot works with Robot Operating System (ROS)⁴ over Ubuntu and runs all dedicated software for each module (e.g., human perception) on ROS. The implementation details of the modules are presented in the next sections. The robot also communicates with the AIS computer using serialized ROS messages, i.e., via ROS topics and services.

3.3 Advanced Navigation Software

Specific ROS-based navigation software was developed for the ENRICHME system. The purpose of the improved navigation software was twofold: first, to increase the safety of the navigation to prevent collisions with the elderly and the environment; second, to provide an easy-to-use graphical interface to map every home and configure additional meta-data like Virtual Obstacles (VOs), Zones Of Interest (ZOIs) and Points Of Interest (POIs). Figure 6 shows a snapshot of the GUI to define meta-data on map. ZOIs are painted in green, POIs are the blue circles, and VOs are in red.

Fig. 6

Navigation setup GUI for TIAGo ENRICHME

POIs were required in order to define the dock station position within the map and to define pre-defined locations where the robot was meant to wait for instructions. ZOIs were needed in order to have topological information about the objects detected by RFID. Finally, VOs are always useful to prevent the robot entering some specific areas.

Apart from the different meta-data that helps to improve the autonomous navigation of the robot, a key development was to integrate the depth information from the RGB-D camera of the robot’s head to the laser scan of the base. This ensures that obstacles that cannot be detected by the laser are effectively avoided thanks to the camera. A typical example that appears in household environments is a table high enough so that the laser detects only the legs but does not detect the plane of the table. In this case, the camera does detect the table plane and the robot circumvents the table.

4 Human Perception

4.1 Detection and Tracking

This sub-module is responsible for detecting and tracking people around the robot. Two different human detectors are used: (i) 2D laser-based legs detector, and (ii) RGB-D camera-based upper-body detector.

The first detector is a 2D laser-based leg detector [2], which extracts a set of features, including the number of beams, linearity, circularity, radius, mean curvature, and mean speed. The AdaBoost algorithm [59] is then employed to turn a set of supervised weak classifiers into a strong classifier, detecting legs from laser range data. The second detector is a RGB-D based upper body detector [50]. This detector applies template matching on depth images. To reduce the computational load, this detector first runs a ground plane estimation to determine a region of interest (ROI) that is the most suitable to detect the upper body of a standing or walking person.

After the people nearby the robot are detected, a Bayesian tracker is used to track them. It extends and improves the solution proposed by [8], which allows combining multiple sensor data, independently from the particular detection type and frequency. The estimator adopted for our system is the unscented Kalman filter (UKF) [38]. It is also possible to switch between UKF and SIR filters, or choose EKF, since they have all been implemented in ROS (Sect. 4.4). A gating procedure is applied using a validation region around each new predicted observation [5]. New validated detections are associated to the correct target using a simple Nearest Neighbor (NN) data association algorithm.

During testing, we observed that, under particular circumstances, both leg and upper-body detectors were affected by a significant number of false positives in real home environments. Chairs were falsely detected as human legs; in others, bulky shelves were falsely detected as human upper-body. To solve the problem, the human detectors have been improved exploiting the temperature information of the thermal camera, mounted on the top of the robot (Fig. 4).

As humans are typically detectable within a limited temperature range (i.e., 32–40 $^{\circ }$C), they can be effectively segmented in thermal images most of the time. In the adopted solution, the position of human in thermal images is determined by applying thresholding and connected component analysis (Fig. 7). The thermal detection is used to validate the information from the other two detectors. The flow diagram of the improved human detection/tracking system is illustrated in Fig. 8.

Fig. 7

a Image obtained from the thermal camera of the ENRICHME robot; b the person’s position can be found applying thresholding and connected component analysis

Fig. 8

The flow diagram of the human detection/tracking module using thermal information

4.2 Re-identification

Human re-identification is an important field in computer vision and robotics. For long-term applications, such as domestic service robots, many existing approaches have strong limitations. For instance, appearance and color based approaches are not applicable as people change often their clothes. Face recognition requires a clear frontal image of the face, which may not be possible all the time (e.g., a person facing opposite the camera, see Fig. 9a). Skeletal data is not always available because of self-occluding body motion (e.g., turning around) or objects occluding parts of the body (e.g., passing behind a table, see Fig. 9b and [53]).

Fig. 9

In a real-world scenario, re-identification should cope with a different views and b occlusions

In order to deal with the above limitations, we use a volume-based human re-identification algorithm [14] that uses biometric features, including body part volumes and limb lengths. In particular, we extract height, shoulder width, length of face, head volume, upper-torso volume and lower-torso volume. As these features are neither view dependent nor based on skeletal data, they do not require any special pose. In real-world scenarios, most of the time, lower body parts of people are occluded by some object in the environment (e.g., chair). As our features are extracted from upper body parts, they are robust to occlusions by chairs, tables and similar types of furniture, which makes our approach very suitable for applications in domestic environments.

The re-identification module uses the upper body detector in [50] to find humans in the scene, segments the whole body of a person and extracts biometric features. Classification is performed by a support vector machine (SVM). The flow diagram of the respective sub-modules is presented in Fig. 10. In particular, the depth of the body is firstly estimated from the bounding box detected via an upper body detector (Fig. 11a). Body segmentation is performed by thresholding the whole image using the estimated depth level (Fig. 11b). Then, important landmark points, including head point, shoulder points and neck points, are detected. Using these landmark points, height of the person, distance between shoulder points, face’s length, head’s volume, upper-torso’s volume, and lower-torso’s volume are extracted as biometric features (Fig. 11c).

Fig. 10

Flow diagram of the re-identification module

Fig. 11

The result of a the upper body detector, b the body segmentation and landmark point detection: (a–e) head, neck, left, right, and hip points, (c) , and c the extracted biometric features: (1) height of the person, (2) distance between shoulder joints, (3) length of face, (4) head volume, (5) upper-torso volume, and (6) lower-torso volume

4.3 Physiological Monitoring

In assistive technologies, several approaches (from wearable [55] to contact-free [30]) have been proposed for monitoring physiological parameters. Although wearable devices may have advantages over contract-free devices regarding accuracy, they may not be reliable, especially for elderly people with cognitive impairment (e.g., forgetting to wear a heartbeat monitoring device). Since most of the contact-free approaches require to be in front of a fixed camera, it is hard to use such approaches in real-world applications (e.g., at home).

This module is based on a contact-free physiological monitoring system [15]. The system detects the subtle changes in temperature on thermal images acquired from a thermal camera. A dedicated software was developed for the robot to monitor some important physiological parameter levels, including body temperature, respiration rate and heartbeat rate.

The approach was developed to extract human physiological data including face temperature, respiratory rate and heartbeat rate levels, from the thermal camera (Optris PI-450). The challenges include face tracking in thermal images and physiological feature extraction from temperature information. The main steps of the process are shown in Fig. 12.

Fig. 12

Flow diagram of the physiological monitoring

4.4 ROS Implementation

The software system consists of several ROS packages for human perception: (i) 2D laser-based legs detector, (ii) RGB-D camera-based upper-body detector, (iii) Bayesian tracking, (iv) volume based re-identification, and (v) physiological monitoring. The flow diagram of the modules is presented in Fig. 13.

Fig. 13

Flow diagram of the ROS human perception

The ROS package developed for human perception includes a one-click launch file. Parameters for relevant modules can be set using a YAML configuration file. Our tracker’s modular design allows for the easy replacement or addition of detectors. To add a detector to the tracking framework, we provide the information via the YAML based configuration file.

5 RFID-Based Object Localization

Object localization in ENRICHME is based upon a discretized map (a grid map) where tag detections are mapped by the robot. Each tag detection is probabilistically associated to a small area around the robot. After several detections, we characterize the cumulated probability of detection over relevant areas of the map. ENRICHME’s grid-map framework for object localization is hence designed to detect objects on a region-based accuracy level. This assumption allows a more efficient detection in house environments, where accuracy is less relevant. This section describe the components and the implementation of our object localization module.

5.1 System Description

Object localization in ENRICHME is based on the driver described in [10]. In essence, we use a trilateration approach as described by [9]. However, we use robot movements to provide multipath and removing the need of multiple antennas. Instead of computing the intersection areas between different detections, we assign weights to areas after each tag detection based on the number of detections. When the robot detects a tagged object, it updates the corresponding tag grid map using our confidence model. Cells within the measurement area increase its weight while the remaining ones decrease it. As the robot moves around the environment, the cells belonging to the intersection of the detection areas will receive higher weights than the others. This way, high weight areas emerge over time, effectively creating a probabilistic map of possible object’s location(s).

This updated grid map is used to localize a tagged object within a specific room. Our application requires knowing which area is the most probable location of an object, more than a probabilistic map itself. We compute location probabilities by dividing the probability map into relevant regions and comparing relative region cumulated weights. The process is illustrated by the sequence in Fig. 14 for a set of RFID detections of a single object recorded by the moving robot.

Fig. 14

RFID object localization in an elderly flat. As the robot moves and detects a tagged object (on the left), a cumulative probability map is built by intersecting the detection areas. Map color is based on a heat scale (i.e. bright red = high probability; blue = low probability). (Color figure online)

As the robot moves and detects a tagged object (on the left), a cumulative probability map is built by intersecting the detection areas. The map is based on a heat scale (i.e. brighter red = high probability; dark blue = low probability). Next subsections will describe in detail our confidence model, and how grid maps are updated and used to locate objects.

5.2 Confidence Model

We develop a simplified confidence model, describing most likely tag locations around the RFID antenna given a positive detection. This model was built moving a robot equipped with an RFID antenna around a tag in a known position. Reference tag was placed at the same height as the antenna, approximately 0.65 m, and over a plastic panel. The robot started at 4.5 m away from the tag, facing it. Approximately every 30 cm, the robot would perform a complete spin while recording, before taking another step towards the tag. In Fig. 15, we captured tag readings’ information over a robot centered—pose 0,0 facing right- grid map. This antenna centered grid was used to describe tag detection.

Fig. 15

Confidence model from recorded data

We represented detection probability, as number of positive readings compared to the total number of detections. Using captured data, we defined three areas, represented in different colors in Fig. 15a. Black area in front of the robot corresponds to a detection probability above 75%. Gray area around robot comprehends probabilities between 30 and 75%. And white area is regions with probabilities under 30%. These observations match with those made by [34] for their simplified model.

We use a similar simplified confidence model, based on two parameters, a detection radius and a frontal cone. This simplified confidence model is represented in Fig. 15c. It comprises two parts: a 60 cone starting on the robot and a 2 m. diameter circle, centered 1.3 m. ahead of the robot. The cone is the highest detection likelihood area, whereas the circle comprises most of the habitual detection locations.

We also analyzed received signal strength indicator (RSSI) distribution. Fig. 15b shows a gray-scaled grid with RSSI values higher than the average. Unsurprisingly RSSI distribution is a circular shape, as suggested by [32]. Its diameter (around 2 m.) depends on the transmitted power by the reader. A bigger transmitted power is not advisable, as it would increase reflections that could be identified as ’ghost’ detections. RSSI is a relevant information provided by RFID readers. Our model will use RSSI values to compute weight when updating the gridmap.

5.3 Grid-Map Updating

Once we have determined our confidence model, it can be used to update tag grid maps. Each time a tagged object is detected by the RFID reader, region described by our confidence model at current robot position is more likely to contain the tag. A positive detection within a region also means that tag is less likely to be outside it. As a result, cells inside this confidence model at robot position increment its weight, while cells outside it decrement its weight.

Inside the confidence region we make a distinction between the central cone and the surrounding circle. Cells inside this region cone have relatively higher weight than those inside the circle. Received RSSI also is taken into account. The stronger the signal is, the higher the increment inside the confidence area and the decrement outside it. This process is summarized in Algorithm 1.

5.4 Object Localization

ENRICHME robot is conceived as a ’virtual companion’, offering support to people with MCI. General advice in these situations is to provide just enough help so that users remain somehow active. Our RFID-based localization algorithm accuracy was targeted to room level (regions) in order to make users to finally retrieve objects. This granularity may be insufficient for some large rooms, so a smaller level was defined comprising smaller subregions inside a room. User could then ask again to locate an object within the previously reported region.

Using this map partitioning into regions and/or subregions, we compute the overall cumulated cell weight over each region in the detection grid map. This value provides a confidence value on the likelihood of the tag to be located in every region. Global map weight is used to normalize region cumulated weights into a region confidence factor ranging between 0 and 100%.

5.5 ROS Implementation

Each one of the elements in Fig. 16 corresponds to a different ROS node. The RFID reader driver node rfid_node is based upon the ROS driver implementation described in [10]. This node outputs tag reading messages, which contain tag identification number (ID), received signal strength (RSSI), phase and carrier frequency.

Fig. 16

Flow diagram of the ROS modules for RFID object detection

This information is received by the detection grid maps, implemented by the node rfid_grid_map. Each node is configured to monitor one object, assigned to one particular tag ID. Each rfid_grid_map also uses robot localization information to properly update the detected tag using the detection model. Its input is a OccupancyGrid ROS message that shares current map size, resolution and frame.

The third relevant node is rol_server, which implements the region probability estimation service. This service is used by Robot interface to get a list of tracked objects and most probable regions where each object is located. It reads occupancy maps, builds a region probability list and offers it to the robot in an unified interface. The list of implemented actions of this server is summarized in Table 1.

Table 1 Actions implemented in the findObject service

6 Interaction and Behavior Coordination

For an every-day interaction with the elderly user, a web-based graphical user interface (GUI) was developed for the touchscreen mounted on the torso of the robot. The GUI features eight custom developed applications: Cognitive games, Health tips, Physical activities, Find object, Environmental data (i.e., robot sensors; house sensors), Agenda, Call somebody, Weather forecast and News reading. These applications (also shown in Fig. 17) were designed based on the use cases of the ENRICHME project and were customized based on the feedback received from the early test participants of the project. Next, each application is briefly presented.

6.1 GUI Applications

Find Object As presented in Sect. 5, the user can use the ENRICHME robot to localize different objects around the house. Therefore, a visual interface was designed for the RFID-based object localization module. The user selects the object that he/she wishes to find with the help of the robot. The robot will display the rooms/regions in descending order of the probability of finding the object in that room/region. The last location shown is the last place where the object was located. The interface for object localization can use different hints with increasing levels of detail, instead of showing immediately the exact known location on the screen. The aim of this approach is to let the users remember the object locations by themselves, which supports the maintenance of cognitive capabilities.

Cognitive Games For helping the elderly maintain and even improve their cognitive abilities, nine cognitive games were designed and developed for this research project. A review of the existing cognitive games designed for the elderly showed that none of them met the requirements of the current project. Most of the games are black-boxes (e.g., Sociable), therefore, do not provide access to the performance throughout the game, and instead only at the end of the game provide a final score. We want to be able to provide customized feedback at each step of each game; as well as a customized performance score at the end of each game. Other cognitive games platforms are not free to use (e.g., Luminosity) and are not easy to integrate in the current architecture of the system. Taking all these into consideration and based on the input from the experts and the early test participants, we have decided to implement nine cognitive games, each with multiple difficulty levels. Each game is presented in Sect. 6.2.

Fig. 17

Main menu of the GUI on the robot’s monitor

Health Tips In this activity, the user can listen to a series of healthy eating tips presented by the robot. The tips can be customized for each country. The user can select to go to the next/previous tip or to listen to the same tip again. Currently the system provides nine tips related to a healthy diet: eating more fruits and vegetables, eating less salt, getting active and having a healthy weight. The healthy tips are based on the British Nutrition Foundation recommendations.⁵

Physical Activities For maintaining their physical abilities, the GUI features a physical activities application. The user can select the desired workout time (i.e., 5 min or 10 min). Once the time is selected, the user is presented with a series of upper body exercises (i.e., head, shoulders, arms). The number of repetitions for each exercise is customizable. Both auditory and visual feedback is provided by the robot for each exercise. The application uses the skeleton data from the RGB-D camera. Based on the position of the 22 joints, the angles between the the joints of the upper body are computed before providing the feedback for each exercise. Currently, a total of 13 exercises were implemented: two for the head, one for the shoulders, two for the body and eight for the arms. In Fig. 18 the interface for this activity is shown. The physical exercises were decided together with the physical therapists from the care facilities.

Environmental Data In this application, the user can select between visualizing the data from two types of sensors: the Uniscan UiEM01 environmental monitoring sensor mounted on the robot and the Smart Home Sensors mounted around the house.

Agenda The purpose of this application is for the user of the ENRICHME system to be able to visualize the appointments and reminders set up through the Networked Care Platform (NCP).

Call Somebody This application enables the user to perform video calls. The application was custom designed using WebRTC technology. The user has also the option of texting the contacts that are saved in the phone book of the application.

Weather Application In this application the user can see the weather forecast for its location. There are two visualization options: hourly forecast and daily forecast. The application uses the API of openweathermap.org.

News Reading The application uses the RSS 2.0 feed of different newspapers to gather the latest news and display them to the end user. The newspaper and the news category can be selected by the user. The news are presented both written and read out loud by the robot.

Fig. 18

Interface for the physical activities application

6.2 Cognitive Games

As previously mentioned, nine cognitive games were developed. They are shown in Fig. 19.

Fig. 19

ENRICHME cognitive games

Digit and Letter Cancellation In these two games the user has to find all occurrences of a randomly chosen digit (i.e., Digit cancellation, see Fig. 20a) or letter (i.e., Letter cancellation) in a given time. Written and verbal feedback is provided after each item is selected. For correct answers, the digit turns green, while for wrong answers, the digit turns red. The game ends either when the user finds all the occurrences, or when the time is up. The difficulty of the game is based on the total number of digits and the given time. The lowest difficulty is defined by 30 digits and 50 s game time. The game also features two hints: first, the description of the game, and second the number of elements that still need to be found. The final score is given in percentage and it is based on the number of mistakes, the number of correct answers and the number of total digits or letters that needed to be found.

Fig. 20

Some examples of cognitive games

Integer and Decimal Matrix Task In these two games, the user is presented with matrices containing multiple digits (either integers or decimals). The purpose is for the user to solve as many matrices as possible in 5 min time. Solving a matrix means to find the two digits whose sum equals 10. The difficulty level of this game is defined by the size of the matrix: $2\times 2$, $2\times 3$, $2\times 4$, $3\times 3$, $3\times 4$, and $4\times 4$. After each guess the user receives both visual and auditory feedback. A new matrix appears only after a correct answer.

Hangman In this game (Fig. 20b) the user has to use the given alphabet in order to guess a word which is part of one of the following categories: fruits, vegetables, cities, animals, and objects around the house. For each word there is a limited number of hints that can be used (i.e., the number of unique letters in the word—2). The first hint is always the description of the word. The next hints are letters that appear in the word. There is no time limit for guessing the word. For each wrong letter pressed a new piece of the hangman is added.

Puzzle The user has to reconstruct the image by dragging and snapping together the puzzle pieces. The pieces snap towards each other once they are in the right place. There is no time limit. The image can be reconstructed anywhere on the screen. The background image represents the hint of this game. There are 3 difficulty levels: easy (i.e., 12 puzzle pieces), medium (i.e., 30 puzzle pieces), and difficult (i.e., 54 puzzle pieces).

Memory game In this game the user is asked to memorize a set of images. First, the images are shown on the screen for a given time. Then, another set of images are shown on the screen and the user is asked if the images shown on the screen was part of the previous set or not. The game can be customized to have any number of trials. Currently it is implemented to include 3 trials and 3 difficulty levels. The images are selected as part of six categories: armchair, clock, hat, phone, sofa, and suitcase. For the easiest level, all images are part of different categories. For the medium level. 2 images are part of the same category, while for the difficult level all images are part of the same category.

Speed game In this game, the screen is divided into two parts. In each part, there is a combination of a letter and a number. The user has to decide in which of the two parts appears the combination VOWEL + EVEN NUMBER. There is no time limit for this game, but the user should answer as quickly as possible. No difficulty levels are implemented for this game. Both visual and auditory feedback is provided after each trial.

Stroop game In this game the user has to decide the color of the text, displayed on the screen. The text is a word (RED, GREEN, BLUE, YELLOW) shown in different colors (red, green, blue, yellow). The user has to press on the button corresponding to the color of the text, and not the text written on the screen. Half of the trials are congruent (i.e., the color corresponds to the color on the screen, for example, Red written in red color), while the other half are incongruent trials (i.e., the color does not correspond to the text on the screen, for example, Blue written in green color). No hints, and no difficulty levels are implemented for this game.

6.3 Behavior Coordination

To coordinate the behavior of the robot, including interactive applications found on the touchscreen, the Hybrid Behavior Based Architecture (HBBA) [27] is used. Its integration within ENRICHME is shown in Fig. 21. It is a three layer architecture combining two robot control paradigms, hybrid and behavior-based, and can be likened to the Belief-Desire-Intention (BDI) model [31]. In HBBA, high-level Motivation modules generate Desires that represent perceptual and action goals for the robot. The description of Desires are never platform-specific. For instance, they can describe the need to reach a location, but does not specify which exact path to follow or velocity profile to apply.

Fig. 21

The hybrid behavior-based architecture (HBBA) implemented in the ENRICHME project. HMI and RH represent the touchscreen and robot hardware, respectively

A Motivation module is responsible for regulating a single aspect of the overall robot behavior, for instance trying to reach its docking station when the batteries are depleted. In ENRICHME, each interactive application (Sect. 6) is managed by a Motivation module for Cognitive Stimulation (CS), Physical Stimulation (PS) and Social Stimulation (SS). For instance, the physical activities motivation generates Desires to locate the user of the robot, keep the robot in place but allow rotations to always face the user and track its body movements during physical monitoring, and display the instructions on the touch screen.

From the current set of Desires, HBBA first resolves conflicts on a priority-based basis and selects Strategies to fulfill each Desire. Fulfilled Desires are named the Intentions of the robot, and the Strategies selection process is managed by the Intention Translator (IT). The Intentions consists of instructions to configure perception (PC) behavior producing (BH) modules and route the proper information messages between them.

6.4 ROS Implementation

All the applications of the interface were implemented using Web technologies: HTML, CSS, JavaScript (i.e., JQuery). The connection between the web-interface and the rest of the system is established through websockets and the standard ROS JavaScript library (roslib.js). The interface publishes a ROS message every time an application is loaded or closed. The main parameters of the games are implemented as ROS parameters which can be easily customized and adapted for each end user.

As the system was tested in multiple countries in Europe, the entire interface had to be customized for each country. More specifically, the language of the interface has to be easily configurable. For this purpose, all the messages displayed on the touchscreen are stored in different JSON files for each language. A ROS parameter is implemented that simply selects the requested language. Currently, the system is available in English, Polish, Greek, Dutch, and Italian.

HBBA is implemented as multiple open-sourced ROS nodes.⁶ Each Motivation module has its own process, which sends Desires to the IW. The IW Translator uses the Google OR-Tools⁷ library to solve a constraints optimization problem when resolving conflicts and selecting Strategies. The routing of messages between modules is managed by perceptual filters, which can progressively throttle the message publication rate to reduce computational load [26]. Configuration of these filters, perception and behavior-producing modules are Javascript programs interpreted by a Google V8⁸-based virtual machine embedded in the IW Translator.

7 Experiments

7.1 Human Perception

The human detection and tracking modules are the first source of information for the ENRICHME robot, necessary to find and approach the user. A detailed performance analysis of our laser and RGB-D based people tracking is presented in [44]. As noted before, during the testing in real home environments, we noticed that some common objects, such as chairs and bookcases, can sometime be confused with humans by the leg and the upper-body detectors. As demonstrated in Fig. 22, where some examples of false positive recorded in an elderly flat are shown, human detection in cluttered homes is a very challenging task. However, by using additional thermal information, as explained in Sect. 4.1, we have eliminated many false positives and improved the accuracy of the human detection, as shown for the case of the elderly user in Fig. 23.

Fig. 22

Some examples of false human detection with the laser-based leg detector (a) and the RGB-D based upper-body detector (b) in a UK home

Fig. 23

Human detection improved by thermal information

In order to have quantitative results, we also tested our detection module in a laboratory environment similar to a flat that includes typical furniture such as sofas, book shelves. We recorded over 4 min of data that contains multiple people walking, sitting on a chair. Table 2 presents results for our human detection algorithm together with an existing detection algorithm in [20]. We can see that our algorithm decreases most of the false positives and doubles the precision rate compared to the existing algorithm. This is achieved by using additional thermal information while detecting people. The true positives are low for both approaches because of the noisy depth information when people are away from the robot. Even in this case, our approaches achieves higher $F_1$-score than the existing approach.

After detection and tracking, our human perception includes a re-identification module. Thanks to volumetric features, this can recognize the user under different poses and occlusions. Figure 24 shows some examples of re-identification results recorded in two different flats of the ENRICHME users. It can be seen that the system works successfully even when the person is not facing the robot, is occluded by a walker or in case there are multiple people in the camera’s field of view. A thorough evaluation of our re-identification module is also available in [14] (Fig. 25).

In addition, we also tested our re-identification module on a public dataset⁹ recorded in a laboratory at the University of Lincoln. The dataset contains 15 people standing and turning around a spot for six times. The data simulates cases in which people are observed from various view points by the robot (e.g., facing backwards). The depth data in the dataset is used to evaluate our re-identification module. The training is done using 4 out of 6 turning around instances. The rest is used for testing. Our algorithm achieves 40.03% recall rate and 47.89% precision rate. Figure 26 shows the confusion matrix of our re-identification results. We can see that, in a 15 subject dataset, our algorithm might fail when people have similar body shapes or volumetric features. However, considering in a flat where the robot will come across with a few people, our algorithm will be sufficient to distinguish people and recognize the user.

Table 2 The results of a standard human detection system and our human detection module

Fig. 24

The ENRICHME robot can successfully recognize the correct user in challenging situations such as non-frontal views (top row), occlusions (middle row) and multiple people presence (bottom row)

Fig. 25

Examples of physiological monitoring with the ENRICHME robot while the user is reading news on its monitor

To show the discriminative property of our algorithm, we evaluated the re-identification performance on k-combinations of 15 subjects, where $k=\{2, \ldots , 14\}$. The average recall rates versus different combinations are presented in Fig. 27. It can be seen that our algorithm can accurately distinguish the user among 4 people, achieving a recall rate higher than 70%.

Fig. 26

The confusion matrix obtained for our re-identification module tested on a public dataset

Fig. 27

The average recall rates for different combinations of people

The physiological monitoring module relies on a Fourier analysis of the temperature on forehead and nose regions. The face of the user must be therefore fully visible on the thermal camera. For this reason, the physiological monitoring starts when the user interacts with the robot, e.g. while reading news or playing games (see Sect. 6). A detailed evaluation of our physiological monitoring is provided by [15]. Figure 25 illustrates representative results obtained with an elderly person in his flat. We can see that the module for physiological monitoring successfully measures temperature, respiration and heartbeat levels of the user while he is reading the news on the robot’s monitor. The figure shows also a case in which the physiological monitoring fails because the user’s face is not fully visible (last image of Fig. 25). Since the face cannot be properly detected, the algorithm cannot also estimate the heartbeat and respiration rates correctly.

7.2 Object Localization

A detailed analysis of the RFID object localization, evaluated in an office environment, was presented in [10]. In ENRICHME, our solution was further validated in real-world elderly homes, including the flat depicted in Fig. 28. This is a dynamic environment, where objects can be located at and moved between different places. New obstacles may also appear, and people moving around makes the test scenario even more challenging

Fig. 28

Layout of a validation site in Poland

Cheap RFID tags (Confidex EPC Class1 Gen2 UHF) were attached to five common objects—remote, keyring, wallet, pillbox and glasses – that could be located in two main regions, living-room or bedroom. The bedroom was further divided into three sub-regions (TV, dressing and sofa), which increased the spatial resolution of the detector. In total, there were therefore four possible locations for the five tagged objects.

In this experiment, each object was located in a distinct sub-region, except for the remote and the keyring that were both in the living-room. The bedroom area was particularly cluttered, while most of the living room was free from obstacles. A grid map was created by manually moving the robot (with a joystick) once at the beginning, starting from the dressing sub-region an heading South. After that, the robot “patrolled” these regions for 5 min, moving repeatedly between them and following the path shown in Fig. 28.

Table 3 summarizes the results of this experiment. Each Region’s confidence was calculated as explained in Sect. 5.5. In practice, the higher the confidence, the higher the probability for an object to be located in such region. In the table, the Average Region Confidence (ARC) is the confidence of the object’s actual location. The Accuracy represents the probability of the highest confidence being assigned to the correct object. Finally, the last column in the table shows the Number of Tag Detections (NTD), that is, how many detections were integrated in each object’s grid map.

Table 3 RFID-based object localization statistics

From the results, we can see how the remote was always detected in the correct region. This is expected, since this object was in the far corner of the living-room, away from other regions. However, the region confidence is not very high because of the relatively low number of detections. The latter indeed were also rather weak, due to the path of the robot which never got too close to the remote.

For similar reasons, the keyring was always missed. The glasses were also difficult to detect, since placed in a region that was mostly inaccessible by the robot. Moreover, they were located between two different sub-regions (sofa and dressing), making the accuracy drop below 50%. The most probable region (incorrectly) estimated by the localization algorithm was the dressing area, where the robot detected the tag most of the time.

Finally, the pillbox was detected and correctly localized with relatively high accuracy, while the wallet provided the best results overall. Indeed, most of the time it was accurately reported in the TV area with a large region confidence. In this case, the spatial resolution of the localization system was improved by a larger number of detections, thanks to the local robot’s path and orientation in proximity of the wallet.

7.3 Use Case Experiences

In total, six ENRICHME systems were deployed in elderly residences of the pilot sites: two in Greece, two in UK, and two in Poland. Each site was managed by an ENRICHME partner of the respective country.

The UK testing sites, managed by LACE Housing, were three “housing with extra care” schemes with more than thirty two-bedroom self-contained flats. The ENRICHME robot was tested in two different buildings by 3 participants in total, each one using the system for 10 weeks (see Fig. 29a). The participants were all male and aged 70, 71, and 90, respectively. Since the beginning, the users were very enthusiastic about the robot and keen to learn what it could do and how to use it. Only one of the users was already familiar with computer-based technology, and two of them had some vision impairment. However, all the users were quickly able to understand and operate the screen interface.

Fig. 29

ENRICHME robot operating in UK at LACE (a), Greece at AKTIOS (b), and Poland at PUMS (c)

In Greece, the ENRICHME system was tested in one of the facilities operated by the AKTIOS Group. This organization provides a colorful and vivid environment for elderly care with more than two-hundred beds in multiple units, where the emphasis is given on hospitality services, nursing care, stroke and bone fracture rehabilitation. They also provide specialized services for people with dementia and Alzheimer disease. AKTIOS hosted two ENRICHME systems for a period of 20 weeks. There were two runs of 10 weeks, including 4 participants in total. The system was installed in their room and the activities were scheduled according to user needs (see Fig. 29b). Inside the room, there was a bathroom, one bed, a television, a wardrobe and a coffee table with chairs. There were no carpet/mats and no other furniture. The participants were introduced to the use of the system and were closely monitored during the first days of its installation. There were 4 women aged from 73 to 89 years, who quickly familiarized with the use of the robot and the its interface.

The Polish pilot site was managed by the Poznan University of Medical Science (PUMS), which includes a multi-professional team (geriatrician, nurse, physiotherapist, occupational therapist, dietitian, social workers) specialized in care for older individuals. They are actively involved in various stages of geriatric care in Poland, e.g., to set up standards for geriatric and long-term care. Here, 4 participants tested the ENRICHME system in their own houses or apartments (Fig. 29c). There were 3 females and 1 male, with ages between 66 and 83. The validation period for each participant lasted approximately 10 weeks.

Table 4 Number of times each GUI application was used by the participants

Table 5 Number of times each cognitive game was used by the participants

7.3.1 Interaction and Behavior Coordination

For each testing site, the interaction data was recorded in a Mongo database with the following information: type of activity (e.g., load page, close page), the user ID, the application (e.g., cognitive games, puzzle, environmental), the date and the time. We were interested in finding out which applications were most and least used by the end users. We only considered for this analysis two participants that interacted with the same robot in one of the testing sites. Each participant interacted with the robot for a duration of 8 weeks. The robot was located in the first participant’s house between the 2nd of November and the 28th of December in 2017, and in the second participant’s house between the 15th of January and the 11th of March in 2018.

Table 4 shows the number of times each application was used by each participant in the 8 weeks of interaction. From the table, it can be seen that the most used application by the two participants is the Cognitive Games, followed by the Environmental data. The least used application by the first participant is the News application, while for the second one is the Find object application.

Next, we wanted to find out which was the preferred cognitive game. Table 5 shows how many times each games was played by each participant.

Fig. 30

An example of HBBA coordinating the physical activities application requested by a Polish user

For the first participant, the most played game is represented by both the Puzzle and the Hangman game, with a total of 35 games for each. The second most played game is the Memory game, played 28 times in 8 weeks. The two most played games by the second participant are the Hangman games and the Digit Cancellation, played 1255 and 274 times, respectively. As it can be seen, both participants really liked to play the Hangman game. The least played game by both participant is the Decimal Matrix task.

Next, we wanted to see in how many days the applications were used, the average time of interaction in each day, the duration of the longest interaction, as well as the days of the week when the robot was most and least used.

For the first participant, the GUI applications were used in 21 days out of the total 56 days. The average interaction time was of 45 min with a maximum of 166 min. The preferred day of the week for the interaction was Wednesday (5 times in 8 weeks), followed by Monday and Saturday (4 times in 8 weeks) and Tuesday and Thursday (3 times in 8 weeks).

The second participant used the GUI applications in 44 days. The average interaction time was of 32 min with a maximum of 197 min. There are no clear preferred days of the week for the interaction. For 7 times in 8 weeks, the end user interacted with the robot on Mondays, Wednesdays, Thursdays and Sundays. For 6 times it was on Tuesdays and Fridays, while the user interacted with the robot on Saturday only for 4 times.

The recorded data shows that the two participants used all 9 GUI applications designed and developed for the ENRICHME system. The data also shows that the users liked interacting with the robot all days in a week, spending at least half an hour for each interaction on average. The application most used by both end users is the Cognitive Games. Next, we plan on extracting and analyzing the data from the other sites as well so as to have a more complete picture of the applications used by all the end users of the ENRICHME system. However, the preliminary results are promising as they show that the end users used all the developed applications.

As described in Sect. 6, the robot behavior is coordinated using HBBA. The implemented coordination module is responsible of operating all nine interaction applications together with tasks for reminding appointments and charging the battery of the robot. In real-world experiments, our coordination module has successfully operated all tasks implemented in the ENRICHME robot. Fig. 30 shows an example of how HBBA works to complete the physical activities application started by an elderly user in Poland.

7.3.2 User Feedback

After the validation experiments in the pilot sites, the participants were asked to express their opinions about the ENRICHME system. In general, the participants were happy with the robot customization and interaction capabilities. The acceptance of the ENRICHME robot was emphasized in unstructured interviews throughout the validation phase [21]. The participants pointed out that, when the robot was present, there was “someone” to talk to, waiting for them at home, and making the environment more friendly. This made easier for them to cope with loneliness. At the end of the validation phase, indeed, some participants expressed regret that they could not longer use the robot system.

Two of the participants in Greece chose to name the robots after Greek gods. One was named Ermis, while the second was named Apollo. Also in Poland the participants in the pilot sites chose to personalize the robots, giving them popular male Polish names: Kuba and Stach. One participant claimed that “Kuba brings a lot of variety. Everyone is interested, more willing to visit [...] the games are very interesting, I really like the puzzle one. The atmosphere at home is more revived by this robot”. Another participant, at the end of the study, said that “the robot helped me open to something more, to something new”.

Robot Design The participants approached the robot with great confidence, believing that it looked safe and that they could handle it with the support of the ENRICHME staff. All the participants of the validation phase, after familiarizing themselves with the robot, highly appreciated its appearance and functionalities. Among the three most important satisfaction items, “dimensions” was often mentioned. Other favourite features were “safe and secure”, and “easy to use”. This confirmed that the customized platform was very well accepted by the users.

nGUI Applications and Cognitive Games The participants were very interested and engaged by the various functions of the system. All four participants enjoyed having the robot in their rooms, playing the cognitive games, using the weather forecast application, and reading some news. Their favorite applications were the cognitive games, which some of them used almost daily. All the participants had their own favorite game: some enjoyed the Hangman and the Memory game, while others preferred the Puzzle or the games with numbers. They also found the possibility to check the home air quality with the robot on-board sensors particularly useful. Indeed, they were regularly checking it (sometimes more than once a day), following the robot’s suggestion to open a window if the air quality was bad.

Diet Suggestions and Reminders It was finally found that the robot was very helpful to improve the users’ diet by providing reminders for snacks or meals during the day, and by recommending the type of food that the user was required to eat. In addition, the guidelines provided by the healthy-tips application were found very useful. After living and interacting with the robot, the nutritional status of several participants improved (i.e. their muscle tissue and daily energy intake increased).

8 Conclusions

This paper described the robotic system implemented by the ENRICHME project to enable health monitoring, complementary care and social support for elderly people at home. The system was designed based on user requirements and tested in real elderly homes, providing mobile interaction services and cognitive stimulation for people with MCI. Extensive research has been carried out in the area of mobile autonomous robot companions operating in domestic environments. As we have shown in the paper, our system builds on top of previous systems by improving the robot perception of humans and integrating a new RFID-based system to locate everyday objects in a domestic environment. The system incorporates other advanced features, such as non-intrusive physiological monitoring, cognitive games and other interaction tools. The latter have been shown to be particularly appreciated by our elderly users.

The mobile robot in ENRICHME is particularly useful for two tasks: RFID object mapping and user monitoring. Indeed, the range of RFID antennas small enough to be embedded in mobile devices, such as smartphones or tablets, would not be sufficient to detect all the objects in a typical home environment. Furthermore, without a robot, the location of the tagged objects could not be dynamically mapped, since mobile devices do not integrate enough sensors (e.g. laser scanners) and cannot be reasonably used to continuously model the environment through their cameras.

For user monitoring, while it could also be argued that an ambient intelligent solution can be integrated into a simpler mobile device and a network of fixed sensors, it would not be very effective in detecting or identifying people in cluttered environments, and it would not be as flexible as a mobile robot in terms of installation and configuration.

Although the emphasis of this paper is on the enabling technological innovations of our assistive robot, future work should look at opportune methodologies [11, 12] to evaluate the original expectations and the actual experiences of our elderly users. Future technical improvements should also consider fast and safe human–robot physical interaction (e.g. find, pick-up and bring lost objects to the user), possibly exploiting recent advances in soft robotics, as well as developing more expressive and empathetic interaction modalities that adapt to the user’s mood.