Keywords

1 Introduction

There is a growing need to address the worldwide-observed phenomena of population aging and the numerous implications it can create in the existing socioeconomic structures and healthcare frameworks of the affected countries. According to the 2015 UN report [1] on world population ageing, it is estimated that globally the number of older persons (60+) is expected to grow by 56%, by the year 2050. And older persons are projected to exceed the number of children for the first time in 2047. In Europe alone the number of senior citizens (ages from 65 to 80) will rise by nearly 40%. Australia, Japan, and the United States are also experiencing this trend. Moreover, according to the same report, the older population is itself ageing, meaning that the number of people 80 years or over, the “oldest-old” persons, is growing even faster than the number of older persons overall. As pointed out in [13], there will be a tremendous shortage on staff and qualified healthcare personnel in the near future to cater to the needs of the graying population.

Advancements in the Information and Communication Technologies (ICTs), electronics, and robotics, have allowed fields such as tele-healthcare and robotics to explore technological solutions for the support of the elderly, such as tele-monitoring platforms, ambient assisted living environments, robotic assistants, and social assistive robots (SARs). All initiatives seem to share a common goal, namely to find ways to support and elongate the time people can live independently at their homes, while maintaining a good quality of life. Furthermore, solutions are being sought to deal with the usual effects associated with aging such as physical and mental impairments, social isolation, and loneliness.

In the past decade a number of technological prototypes have been developed and evaluated. More often than not, however, these solutions are evaluated in laboratory settings or in the premises of institutions and public spaces. Very few have actually been installed in actual home environments. A long term home trial of an autonomously moving social robot was conducted in the context of the 7th Framework Programme European funded project HOBBIT-the Mutual Care RobotFootnote 1. A total of three countries, Austria, Sweden, and Greece, were involved in the HOBBIT project. The main goal of the project was to develop a low cost robotic assistant for the elderly that can move autonomously in a home environment and perform tasks such emergency detection and handling, fall prevention, and providing a “feeling of being safe and supported” [2]. In addition, Mutual Care was one of the core interaction paradigms that were studied in HOBBIT. The first prototype (PT1) of the robot was evaluated by a total of 49 users in laboratory conditions in three countries, Austria, Sweden, and Greece. The results from that first evaluation lead to the development of the final robot prototype (PT2), which was then evaluated in the actual homes of a total of 18 elderly users in these three countries for a total of 3 weeks per user 4. This paper will not focus on the methodologies used for the trials, or on the actual collective results of the trials, which are reported elsewhere [3]. Rather, it will present some of the challenges and lessons learned in organizing and running home trials with four elderly users in the island of Crete in Greece, with the goal of providing practical insights to researchers who plan similar home trials of complex technologies in the future.

2 Background

The likelihood for home robots to be adopted by older adults depends greatly on two main factors. Firstly, on how successful these systems are in meeting older adults’ needs, and secondly, on how amenable respective users are to robot assistance [21]. Borrowing user requirements techniques from the HCI field (i.e., focus groups, questionnaires, structured interviews, and exploratory studies) has allowed many research teams to investigate the above two factors and on coming up with design guidelines for such robotic systems (i.e., [7, 10, 18–20]). As a result, based on the acquired knowledge on what older users need, a number of robotic prototypes have been developed and showcased in the past years. In their review on social assistive robots in elderly care, Broekens et al. [13] give a number of examples of assistive robots such as Nursebot [12], Care-o-Bot 3 [14], Pearl [15], and PARO [16]. Other more recent examples include Robear [5], a prototype robot that helps in lifting patients from bed and supporting them in walking which was designed by Riken a Japanese research institute.

However, even though we are seeing numerous prototypes of autonomously moving robotic assistants, not a lot of them have been extensively evaluated under real conditions and outside the controlled environment of a laboratory or an institution (i.e., nursing homes, long term care institutions, schools, etc.). For example, Prakash et al. [7] studied older adults’ reactions to a robot appearance (PR2 robot) in the Aware Home Research facility of Georgia Tech, a home-based laboratory that serves as a platform for testing home-based technologies. In a different study with the same robot and the same laboratory, Prakash, Beer et al. studied how robots can assist older people in their medication management [8]. In the context of the EU project SERA, von der Pütten et al. [9] presented the results of the home trials of a stationary robotic companion for the elderly with real users and stated that despite the challenges of conducting home trials, they gained useful insights that traditional laboratory testing wouldn’t have been able to give them.

Often, traditional controlled lab settings fail to capture the complexities and richness of the real living environments in which these systems are placed 5. Thus, there is a big gap in the research as to how such complex systems can behave under real conditions where user scenarios don’t follow a specific sequence or script and important parameters can shift on a daily basis, and how users interact with them in the long run. Evaluating robotic systems that provide interactive services autonomously in real environments for an extended period of time has always being a challenge in HRI and assistive robots [17]. Apart from the high requirements on the stability and reliability of the system [22], carrying out long-term home trials of complex technological systems with real users is a rather time and resource intense endeavor. There is an ongoing debate on whether field trials in general are worth the effort, with advocates arguing on both sides on the subject; some claim that the added value is questionable [24], while others stress over the fact that field trials can unveil problems impossible to observe under the ideal conditions of a lab setting (i.e., lighting, background noise level, and perfect wireless connectivity, etc.) [4]. Even though our sample of four participants was small, the experience gained from conducting the home trials of HOBBIT in Crete gave us valuable lessons on the challenges and the factors that should be taken into consideration when planning for similar trials in the future. These challenges and considerations are presented in the following sections.

3 Recruiting Participants Challenges

Finding elderly candidates that would be willing to participate in long-term evaluations of complex systems in real home environments is a challenging task for multiple reasons. Usually, depending on the type of the study and the user profile it targets, there is a list of inclusion criteria based upon which the participant selection is made. In our case, for example, the inclusion criteria included that the candidate had to be 75 years or older, single-living at home, not suffer from any major illnesses, disabilities or cognitive impairments, and have at least one self-reported (i.e., screening questionnaire) moderate or severe vision, hearing, or mobility related impairment, and be available for the duration of the three-week long evaluation without any extended absences. In addition, the home or flat of the candidate had to be spacious enough to host the robot, have no steps, stairs or other type of levels, and wide enough door passages and corridors for the robot to cross.

As a result, there were many situations where a prospective participant that would fit perfectly the user profile, had to eventually be excluded because his or her home environment was not conducive for the trial. Such constrains may make the recruitment efforts difficult, thus it is strongly recommended for the experiment team to acquire a large number of contacts of prospective candidates initially, even if the evaluation involves a small number of users.

In choosing the best technique to recruit prospective participants, the experiment team should also take into consideration the geographic location where the experiment is going to take place and how this may affect the access to pools of elderly users. In big urban areas it may be easier to find independent living facilities for the elderly and other type of institutions and organizations for the elderly, or make use of publications for pensioners to advertise the experiment. In smaller or more remote geographic areas, such resources may be more limited and thus other venues should be examined at a local level, such as community centers, or personal contacts and referrals. For our home trials in Crete, we collaborated with a local Community Care & Active Ageing Center (Municipality of Heraklion). This care center promotes active ageing to its members by involving them in various social and recreational activities and projects. Our contact there was very helpful in finding us potential candidates that fit the general inclusion criteria and arranged to bring them to our laboratory for a presentation and demonstration of the system.

Giving presentations of the system at community centers, lodges, senior centers, or inviting groups of prospective candidates to the laboratory for a presentation of the system and the goals of the experiment offer a great opportunity for collecting contact information. Respectively, a fun evening at the laboratory, where the experiment can be discussed over coffee, tea, and cookies is a nice ice-breaker and a good opportunity for the prospective participants to get to know the research team and vice versa. Even if the presentation attendees are not interested in participating in the experiment, they may know of others who would be interested [23]. Regardless of whether the presentation takes place at the elderly center or at the laboratory, there are a few factors that the research team should take into account in order to make the most of such events. In the following paragraphs, the considerations on organizing a presentation at the laboratory are discussed.

Create a Welcoming and Comforting Atmosphere.

The organizer of the event should make sure that the guests seat comfortably during the presentation, and at the right distance from the screen if a slide presentation is used. Snacks and refreshments such as tea, coffee, juice, and water should be offered throughout the presentation.

Introduction.

After introducing the research team, it is beneficial to give a brief background on the purpose of the entire initiative. This will help the elderly candidates to get a broader perspective on the problem the scientific community is trying to address and how the current project is contributing in moving a step forward towards finding a possible solution. Often elderly people are reluctant to discuss technology matters because their lack of previous experiences with interactive technologies makes them think that they do not have anything of valuable to share. Therefore, it is important to emphasize to them how their insight and opinions, regardless of their exposure to such technologies, can help the researchers in further improvements of such systems for future generations.

Presentation Material.

The presentation material should be straightforward and the presenter must use non-technical language when describing the system and its capabilities. Dickinson et al. [6] state that using technical words that technology specialists use everyday such as “monitor” or “functionality”, can be at best confusing and at worst distressingly unfamiliar, causing the technology to become even more foreign to the potential elderly users and the users to become less confident in participating in the discussion about it. We found that the key is to focus on describing how the robot can help by giving real life examples relevant to everyday common needs of the elderly, rather than explaining the mechanisms that allow it to perform the tasks.

There should be a good balance of keeping the presentation fun but informative, without tiring or overwhelming the elderly audience with too much information. After the presentation ends, the team should allocate some time for an open discussion where answers to any questions about the system or the experiment itself can be given.

Showcase the System if Possible.

If the event takes place at the laboratory or building where the prototype is housed, it is very helpful to arrange a demonstration of the system. In our case, the majority of the elderly had not seen a robot in real life before and any notion they had was acquired from the TV or science fiction movies, in which the depiction of robots is often unrealistic or even frightening. Having no previous real experience of a robotic system would make anyone reluctant to agree to participate in any type of study, let alone agreeing to host an autonomously moving robot in their homes for an extended period of time. Being able to demonstrate the robotic prototype diminishes the fear of the unknown and closes the gap between expectations and reality. After the demonstration, it is a good idea to encourage the guests to come close and try out the system for themselves. This helps alleviate the pre-established notion that new technologies are difficult to learn or use. After we demonstrated the robot, we asked the guests to come close and try interacting with it. At the beginning they all gathered around it, but were reluctant to engage in any type of interaction. They were unsure and shy about trying it, mostly because they were afraid that they wouldn’t manage to operate it or that they would do something wrong and embarrass themselves in front of the others. But after the encouragement from the technician, the most adventurous from the group started interacting with the interface. When the rest of the guests saw their peer interacting with it, they too wanted to try it out. The uncertainty slowly dissipated as the users realized that you don’t have to be a scientist or a specialist to be able to operate it and it is indeed designed for the everyday elderly person who may not have any computer experience. By the end of the event, we could hear discussions among the guests as to who would be the best candidate to take the robot home for the trial.

This event helped us communicate the goal of the experiment and gather initial contact information from potential candidates that showed a sincere interest in participating in the home trials of the prototype.

If the prototype of the system is not available for demonstration, then a video showcasing the system or still pictures of it performing various tasks can be shown.

Prepare Information Leaflets to Give Away at the end of the Presentation.

Printed material with information on the project and its purpose and on details about the overall process of the experiment (i.e., inclusion criteria, duration, type of data that will be collected, etc.), and the team’s contact information for interested parties, should be given to each attendee on their way out.

Communicate the Terms and Conditions of the Experiment.

The communication of the terms and conditions of participating in a home trial experiment to a candidate that has shown interest in participating in the home trial is also a very important stage and should be done with great care by the experiment team, in order to ensure that the candidate has clearly understood all the details about the process of the actual trial before agreeing to participate or signing a consent form. Details about the duration of the experiment, the type of information that is going to be collected (i.e., questionnaires, data logging, interviews, etc.), the measures the team will take to safeguard the privacy of the data collected, the kind of support that is going to be offered during the trials for technical or other type of issues that may arise during the experiment, and the overall process that is going to be followed, should be clarified. It is very important at this stage to also stress that during the trials the participants will be fully supported by the experiment team and that should they decide to stop the trial at any given point for whatever reason, there will be no obligations or repercussions.

The key point is that apart from the fact that it is an ethical obligation of the experiment team to inform the participant about all aspects of the experiment, it is imperative to do so in order to avoid the creation of false expectations by the participant that could lead to serious disruptions during the experiment. Another important point to stress to the prospective participant is that the system that will be installed in his or her home environment is an experimental prototype, which means that technical issues may arise from time to time during the trial. The participant should be given a contact list of the experiment team to contact in case of such occurrences.

4 Scheduling of the Experiment

In their paper, Rogers et al. [5] stress that traditional evaluation methods and metrics (derived from laboratory settings) often fail in capturing the complexities of the real world in which systems or technologies are placed and used. In their field trial of a mobile system they concluded that even something as simple as the changing nature of the physical environment, such as the time of year, can have quite significant impact on the user experience. In planning the home trials of the robotic assistant we found this to be very true. The actual timing of the experiment affected not only our recruitment efforts, but in some cases it also affected the users’ interaction time with the system.

The season during which the experiment is going to take place can increase the rate of difficulty for the evaluation team. Obviously, a lot of times the selection of the time period that the experiment will take place is determined by factors beyond the team’s control, i.e., obligations towards the overall project timeline and deadlines, production, development and deployment timeframes, etc. In our case, the home trials were eventually scheduled for the beginning of summer, with some cascading to the middle of the summer. In the island of Crete, during the summer a lot of elderly people move away from the cities to their villages. This factor affected our recruiting efforts a great deal, simply because a lot of our initial contacts were no longer available for the experiments. As a result, we had to recruit anew, and find candidates that had not participated in any stage of the project before. This doubled the time and effort of the evaluation team, who had to reach new candidates by phone interviews, house visits, face-to-face interviews, etc.

The weather in the summer also seemed to affect the amount of time spent interacting with the robot during the home trials. One of the participants, for example, told us that she would have interacted more with the robot if we had brought it during the winter. She further explained that in the summer, she prefers to sit in her yard where it is cooler and enjoy the good weather. In winter time, she explained, she stays mostly indoors because of the rain and the cold and so she believed that the robot would have been a better distraction for her then. Another user ended up turning off the system for a few days because it emitted extra heat warming up even more the house and making it uncomfortable.

The weather factor had inflicted worries to the team at the beginning of the project, but mostly about the experiments having to take place in the winter time. The research team was worried that asking potential candidates if they would be willing to remove their carpets in the areas the robot would circulate during the cold months of winter would have received a negative response, as area rugs and carpets is a very common way of insulating the home against the cold in Greece.

Apart from the weather, another factor that may affect the scheduling of the experiment is local holidays, religious or non-religious ones, or other local special events during which the prospective users may be absent or have family visiting.

Both temporal conditions and local traditions may affect the scheduling aspects of the trials and the research team must be aware of that in order to take the appropriate measures.

5 Considerations When Conducting the Home Trial

In the handbook of usability testing [11], the author describes the characteristics of a good test monitor or facilitator. Some of these characteristics include: being able to form instant rapport with the participant by sizing up each participant’s personality and making the person feel comfortable and secure, being a good listener by laying aside personal biases and strong opinions about what she or he is seeing and hearing, being empathic and able to relate to the participant’s views and experiences, being flexible and able to understand when to deviate from the test plan if needed. These characteristics are essential for any type of user-based experiment, but even more so for long-term experiments with vulnerable groups such as the elderly. In the following section we discuss issues to take into consideration when conducting home trials.

Introducing the System to the Elderly Participant.

The introduction of the system to the elderly user is one of the most important phases of the experiment, in which the facilitator has to ensure that the user will feel comfortable and confident in operating the system when left alone with it. In order for this to be achieved, certain factors need to be taken into consideration. The first factor is that each user is unique in how fast he or she can learn a new system, how much new information they can retain in one session, and how much training they need before starting to use it. Before introducing the main operations of the system, the facilitator has to investigate the user’s overall experience with interaction methods. For example, all our participants were novice users with no former experience with any kind of interactive systems. Thus, the experiment facilitator had to start from the very basics, such as explaining how to select an option on the “touch screen” mounted on the robot, i.e., how much pressure to apply and how to navigate through the available menu and sub-menus before proceeding to explain more advance operations. Throughout this phase, it is important for the experiment facilitator to be able to recognize signs of fatigue or anxiety by the user, and adjust the training pace accordingly, either breaking down the introduction in smaller sessions, or incrementally introducing each function over the first few days. In addition, having the participant perform the basic functions in the presence of the facilitator ensures to the latter that the user has indeed understood the operation and is comfortable in using it on his or her own. The main point here is for the facilitator to allocate the appropriate time to explain the system to the user in a calm, patient, and straightforward manner.

Respecting Local Customs.

In their study on the in-home requirements gathering with frail older people, Dickinson et al. [6] stress that it is important to be conscious of the expectations that an older person may have of a guest entering their home as the ritual of hospitality is often integral to a sense of self. For example, even an apparently insignificant action like refusing a cup of tea can make a tremendous difference to the comfort of your host. The hospitality parameter proved to be a very important one during the home trials in Crete, where it is very common for example, even during a short visit to a house for the host to offer refreshments and snacks or to even invite the guest to stay for lunch or dinner depending on the time of the visit. Thus, it was very important for our experiment team when visiting to apply technical support or for the facilitator to conduct any scheduled interview to allocate extra time for such social interactions. Even though such interactions are time consuming, conversations during them proved to be very insightful in understanding the background of the participant and their routines. Furthermore, it was found that during those less structured discussions the participants often offered deeper opinions about modern technologies and their limitations or advantages. For example, during a coffee session with one of the participants, she started talking about how much harder yet simpler life was in the old days compared to modern times, and how she found it disconcerting that the younger generations have become so attached to their phones and other similar technologies often preferring to sit for hours in front of screens instead of enjoying the outdoors.

Defining the Role of the Researcher.

At the beginning of any type of evaluation, it is essential for the participant to understand that it is the system that is being evaluated and not their performance and that the role of the facilitator of the experiment is just to collect the data. At the same time, it is also important to stress that it is valuable for the experiment team to hear not only the positive aspects of the interaction with the system, but also the negative ones which will lead to further improvements of the system down the line [25]. Whereas in a short evaluation such clarification is usually given once at the beginning, in a long-term evaluation, it may need to be repeated periodically. As the trials progressed and the relationship between the experiment team and the participants grew on a more personable manner, it was noticed on multiple occasions that some participants became worried that if they reported the negative aspects of the robot that it would somehow reflect negatively on us personally or professionally. For example, one of the participants during an interview mentioned something negative and quickly added that perhaps this information should not go on the official report. When the facilitator asked why the participant thought so, the participant said that this information could hurt the performance rating of the team. In this instance, the facilitator had to reassure the participant that the experiment team did not have any personal stake in the system and to reinforce the importance of reporting all aspects of the interaction with the robot in order to draw concrete conclusions on how it can be improved in the future. Facilitators of long-term evaluations should be aware of the possibility of the participants developing similar sentiments and address them appropriately when needed.

6 Conclusions

Long-term home trials of complex assistive systems can provide tremendous insights on how users experience living with such systems in their own home environment and they are important for future developments of prototypes. However, the complexity of home trials is often much higher than laboratory-based trials and can lead even experienced HCI researchers into common pitfalls. In this paper, we presented some considerations and suggestions regarding the planning of the execution of such trials, based on our experience from conducting home trials with four elderly participants in Greece. Even though some of the considerations may seem obvious to the reader, they can be easily overlooked as more urgent issues may take precedence in the planning of the evaluations, such as testing and ensuring the prototype system to be evaluated is ready for the trials, choosing or coming up with the appropriate methodologies for collecting quantitative and qualitative data, constructing structured interviews protocols for all evaluators to follow, drawing common guidelines, preparing help manuals for the system, etc. The main conclusion is that no matter how careful the planning is, there is a higher than usual probability of unexpected factors and events to take place during such evaluations and thus the team has to be flexible enough to tackle such events without compromising the integrity of the evaluation.