1 Introduction

Population aging occurs worldwide [1]. Due to increased aging of the population, eHealth systems are increasingly used by elderly people to enable them to live more independently [2, 3]. There are eHealth systems used by the elderly that promote physical activity [4, 5], mental health care [6, 7], ambulatory care [8, 9], or disease-related therapy (such as for diabetes) [10, 11]. eHealth systems designed for elderly users can be implemented for example as mobile Health (mHealth) systems [12, 13] or web-based eHealth systems [14, 15]. mHealth systems and web-based eHealth systems differ in terms such as realization (e.g., screen size) [16], application (e.g., enhancing educational skills) [17], or accessibility [18].

Accessibility, acceptability, and usability are vital factors to be considered when designing eHealth systems because eHealth interventions are produced very rapidly [19]. Accessibility means that prospective users can interact with a system (e.g., web-based eHealth system) [20, 21] irrespective of physical and physiological disabilities [22, 23]. Acceptability means that a newly developed system meets prospective users’ needs [24]. Usability means that the quality of a system is assessed with regards to the ease of use of user interfaces [25]. Due to the aging of population, the demand for eHealth systems designed, especially for the elderly is increasing. As a result, the usability of eHealth systems for elderly users is becoming increasingly important [26].

Incorporating the needs of the elderly into the development process of eHealth systems is possible by applying the User-Centered Design approach. User-Centered Design involves prospective users through an iterative development process [27]. By conducting eHealth usability evaluations during User-Centered Design, prospective users’ feedback is iteratively collected and considered to evolve eHealth systems. eHealth usability evaluations can be carried out using Usability Testing, typically in a usability laboratory, where prospective users are observed while performing tasks to gain user feedback [28, 29]. This traditional form of eHealth usability evaluation is difficult to reconcile through an iterative development process, so integrating agile, easily applicable, and useful eHealth usability evaluations into User-Centered Design is gaining importance.

Agile User-Centered Design involves usability evaluation methods into the iterative development process resulting in improved usability [30, 31]. Conducting agile, easily applicable, and useful eHealth usability evaluations to gain rapid user feedback during agile User-Centered Design is necessary to increase the usability of eHealth systems for elderly. To identify whether eHealth systems are designed appropriately for elderly, eHealth usability evaluations are conducted. To conduct eHealth usability evaluations user-based eHealth usability evaluation methods (e.g., Usability Testing) and expert-based eHealth usability evaluation methods (e.g., Cognitive Walkthrough) can be applied. In a Cognitive Walkthrough, an expert mentally walks through the steps required to complete tasks that prospective users would perform with an eHealth system [29].

Conducting a usability test with elderly users is challenging. Elderly users need more time to perform tasks, to read instructions for the tasks, and they often forget the meaning of the task to be performed [32]. Furthermore, there are elderly who do not want to join a usability evaluation in a usability laboratory due to illness [33]. Challenges arise during eHealth usability evaluation, however, also prior to and after eHealth usability evaluations are conducted. Prior to eHealth usability evaluations, rapid prototype testing can be conducted in which elderly users may have difficulties to participate because of cognitive or physical decline. For instance, after conducting eHealth usability evaluations user errors can occur with elderly using eHealth systems, which may also be difficult to manage for elderly users due to their age-related declines. These are some reasons why eHealth usability evaluation methods such as Usability Testing are not easily applicable to evaluate eHealth systems with elderly users [2]. Well-established eHealth usability evaluation methods such as Think Aloud or Cognitive Walkthrough are not readily applicable with elderly users who suffer from motivational, cognitive, or physical decline [34,35,36].

The aforementioned reasons demonstrate why it is essential that an appropriate eHealth usability evaluation method is chosen or modified to accomplish eHealth usability evaluations with elderly users.

We aimed to examine the following research questions: (1) Which eHealth usability evaluation methods are applicable to evaluate eHealth interventions with elderly users?, (2) What challenges might be encountered when conducting an agile, easily applicable, and useful eHealth usability evaluation with elderly users?, and (3) How to deal with and overcome these challenges that arise prior to, during, and after carrying out the eHealth usability evaluation conducted with elderly users?

2 Method

We conducted an explorative case-study supplemented with a systematic literature review to learn about challenges in applying eHealth usability evaluation methods with elderly users and how these challenges can be overcome (see Fig. 1). We implemented the systematic literature review subsequently to the case study to systematically identify and classify challenges corresponding to age-related barriers, such as cognition, motivation, perception, and physical abilities. We chose these four age-related barriers because they affect the usability of eHealth systems, particularly for elderly users [37]. We structured these challenges and developed recommendations according to the stages of software lifecycle (requirements engineering, design, and evaluation), taking into account the stage that follows the eHealth usability evaluation.

Fig. 1
figure 1

Methodology to examine the challenges in applying eHealth usability evaluation methods with elderly users and how these challenges can be overcome

Full size image

As a case study, we applied an eHealth usability evaluation of an eHealth intervention. This eHealth intervention is a web-based application that allows patients to decentrally retrieve diagnostic reports or prescriptions. Prospective users of the web-based eHealth intervention are elderly persons who suffer from diseases frequently or are chronically ill.

We combined these results of the case study with the systematic literature review to develop recommendations useful for conducting eHealth usability evaluations with elderly users.

2.1 Case study: set-up and procedure of eHealth usability evaluation

We started the case study by conducting a pre-test with one elderly person (March, 2021). The pre-test was made to rehearse the procedure and technical equipment used during the eHealth usability evaluation. After finishing the pre-test, we conducted the eHealth usability evaluation with six other elderly participants (see Table 1). We accomplished the pre-test as well as the case study in March, 2021.

Table 1 Demographic data of elderly participants involved in the eHealth usability evaluation indicating the duration of each conducted session of the eHealth usability evaluation
Full size table

The median age of the elderly was 70 years and each session of the eHealth usability evaluation lasted a median of 27 minutes.

2.1.1 Phase 1: Set-up of eHealth usability evaluation

To conduct the eHealth usability evaluation we (1) selected appropriate eHealth usability evaluation methods and, (2) defined inclusion and exclusion criteria to recruit elderly participants.

Selection of appropriate eHealth usability evaluation methods

We selected three eHealth usability evaluation methods: (1) Co-Discovery Learning, (2) Cooperative Usability Testing, and (3) Remote User Testing combined with Think Aloud. We chose the eHealth usability evaluation methods from a previously developed toolbox for eHealth usability evaluations, called ToUsE [38]. ToUsE consists of rapidly applicable as well as potentially useful eHealth usability evaluation methods suitable for implementing eHealth usability evaluations [39].

Co-Discovery Evaluation was selected because we aimed to give the elderly the opportunity to work on tasks during the eHealth usability evaluation together. Co-Discovery Evaluation means that two (or more) participants perform tasks simultaneously [35]. Research has shown that elderly prefer working on tasks together or with caregivers [40].

Cooperative Usability Testing was selected to combine domain expertise with human-computer interaction expertise due to the collaboration of the elderly and the researcher (First Author). Cooperative Usability Testing means that a video recording is made during the eHealth usability evaluation that is watched together with the elderly after the eHealth usability evaluation is conducted [41].

Remote User Testing combined with Think Aloud was selected because we aimed to give the elderly the opportunity to accomplish the eHealth usability evaluation at their homes. Recent research showed that no fully conducted Remote User Testing with elderly users can be found in the literature [42]. Generally, Remote User Testing means that the evaluator and participants are located at different physical locations [35]. During Think Aloud participants express their thoughts out loud when performing tasks [29].

We intended to apply all three chosen eHealth usability evaluation methods with all elderly participants (\(n=7\)). Therefore we invited all seven elderly patients to attend the eHealth usability evaluation with each chosen eHealth usability evaluation method.

Elderly participants

The study team consisted of a researcher (First Author), two software developers of the web-based eHealth intervention, and elderly patients from Austria as participants.

We selected seven elderly participants to conduct the eHealth usability evaluation because having five to seven representative users can identify 80% of the most important usability problems [43, 44]. After conducting the eHealth usability evaluation with the fourth participant, we noticed saturation. Saturation occurred because the indicated usability problems by the elderly repeated.

We restricted the residence of the elderly to Austria because we decided to implement the eHealth usability evaluation at participants’ homes to possibility of limited mobility among the elderly. Conducting the Usability Testing in the elderly’s personal environment is described as the gold standard in the literature [40].

Inclusion criteria for the elderly participants were as follows:

  • age equal to 60 years or above;

  • living independently;

  • chronically ill or suffering from a disease;

  • availability of e-mail account;

  • having basic knowledge of how to use a computer;

  • having an own computer or technical equipment; and

  • ability to understand and sign the written informed consent.

The exclusion criteria were as follows:

  • no e-mail account;

  • residing outside of Austria (Europe);

  • no ability to understand and express themselves adequately in the German language; and

  • lack of signed written informed consent.

German language skills were required to accomplish the selected eHealth usability evaluation methods. Elderly persons that met the inclusion and exclusion criteria were recruited through the researchers’ personal and professional surroundings.

2.1.2 Phase 2: Procedure of eHealth usability evaluation

We explained the procedure of the eHealth usability evaluation prior to the start of the evaluation to the elderly and asked them to sign a written informed consent. We explained to them that we were evaluating the web-based eHealth intervention and not the performance of the elderly users themselves.

We performed the eHealth usability evaluation in March, 2021. We attuned the whole set up of the eHealth usability evaluation to the physical and cognitive capacities of the elderly participants. We asked the elderly prior to the conduction of the eHealth usability evaluation whether they would prefer to use their own computer or a computer provided by the researchers (First Author).

We developed brief tasks to help the elderly maintain their concentration during the eHealth usability evaluation. To accomplish the development of the tasks, we defined a fictitious scenario. We planned sufficient time to explain the tasks to the elderly prior to the eHealth usability evaluation.

Scenario and tasks to accomplish eHealth usability evaluation

Table 2 shows the developed fictitious scenario and the brief tasks to accomplish the eHealth usability evaluation. We planned to apply the three chosen eHealth usability evaluation methods—Remote User Testing combined with Think Aloud, Cooperative Usability Testing, and Co-Discovery Learning—in different order for each participant to reduce the risk of influencing the results of successively performed eHealth usability evaluations.

Table 2 Scenario and tasks of the eHealth usability evaluation
Full size table

We formulated a scenario and derived three tasks from the scenario. The scenario presumes that the elderly patient consented to receive the prescription via e-mail and therefore provided the physician with personal information (e-mail address and phone number). After receiving the e-mail notification, the prescription could be opened via the web-based eHealth intervention.

Figures 2, 3, 4 and 5 show the prototype of the web-based eHealth intervention. At the beginning, a Transaction Authentication Number (TAN) was sent to the phone number of the elderly patient as participant. In a first step, the elderly had to enter the received TAN in the input field of the front end of the web-based eHealth intervention (Fig. 2). After verification of the TAN, the prescription could be downloaded by the elderly in a second step (Fig. 3). After receipt of the prescription, there were two options in a third step: (1) Managing of the personal information to delete the personal information directly, otherwise the personal information will be deleted automatically after 10 days or (2) Closing of the front end of the web-based eHealth intervention (Fig. 4). In a fourth step, a final optional evaluation of the transmission of the prescription is possible (Fig. 5).

Fig. 2
figure 2

Step 1: TAN entry

Full size image
Fig. 3
figure 3

Step 2: Receipt of prescription

Full size image
Fig. 4
figure 4

Step 3: Option to manage personal information

Full size image
Fig. 5
figure 5

Step 4: Information that prescription was successfully deleted

Full size image

Data analysis of eHealth usability evaluation

We counted the number of usability problems that were described and indicated by the elderly. Some usability problems were indicated multiple times by the elderly. For example, some of the elderly stated that they had difficulties in distinguishing the color of the buttons. The usability problem “Difficulty in distinguishing color of buttons” was indicated in the course of two sessions of the eHealth usability evaluation, each carried out with one elderly participant. Therefore we counted this usability problem twice.

2.2 Systematic review: identifying challenges of eHealth usability evaluations with elderly

We performed the systematic literature review to (1) identify papers that report on applied eHealth usability evaluation methods to elderly users and (2) identify challenges of conducting agile, easily applicable, and useful eHealth usability evaluations with elderly users. The systematic literature review was conducted in March, 2022.

We searched four databases for relevant papers: ACM Digital Library, Cochrane Library, IEEE Xplore, and Medline (via PubMed). We used the following search terms related to the four thematic areas of this study (usability, eHealth, elderly, and challenges): eHealth, mHealth, “medical informatics applications”, elderly, elder*, senior, old, usability, “usability testing”, challenges, limitations, feasibility, acceptability, and applicability. For the search in Medline (via PubMed), we also encountered Medical Subject Headings terms, such as telemedicine, aged, and user-centered design. To ensure that we did not miss any relevant literature, we applied different combinations of search terms depending on the selected database.

We included papers that (1) were published in the last 10 years (2012–2022), (2) focus on eHealth interventions, and (3) report on challenges of eHealth usability evaluations with elderly users (see Table 3). In the interest of receiving a large number of relevant papers, we extended our search to papers reporting not only on challenges but also reporting about acceptability, applicability, or feasibility of eHealth usability evaluations with elderly. We also included findings on reported challenges with disabled participants because we did not want to exclude possible further relevant information from this related field.

Table 3 Inclusion and exclusion criteria
Full size table

We excluded (1) non-peer-reviewed papers (except conference papers), (2) papers that do not focus on eHealth interventions, and (3) papers not written in English. We did not restrict our search to papers that report only on user-based eHealth usability evaluations because we were also interested in reported challenges of eHealth usability evaluations where the usability was assessed by experts (expert-based eHealth usability evaluations) on behalf of elderly users.

We imported all found papers into JabRef reference manager and removed duplicates (\(n=21\)) (see Fig. 6). We screened title and abstract of the papers based upon their relevance to our defined inclusion and exclusion criteria (\(n=279\)). We excluded 243 papers based on title and abstract review and included the remaining papers (\(n=36\)) in the full text review.

Fig. 6
figure 6

PRISMA statement of the systematic literature review. Resources: ACM Digital Library, Cochrane Library, IEEE Xplore, and Medline (ordered alphabetically)

Full size image

From these 36 papers, we identified 20 papers that reported on applied eHealth usability evaluation methods to elderly users. We also identified 3 further papers by manual search that reported on challenges of eHealth usability evaluation with elderly users. We extracted the following information from these papers:

  • Applied eHealth usability evaluation methods to elderly users (for all 20 papers);

  • Challenges of conducting eHealth usability evaluations with elderly users (if reported in the papers: \(n=6\))

3 Results

We first present the results from the case study, followed by the results of the systematic review. We then derive from both the case study and the systematic review 24 recommendations on how to overcome challenges prior to, during, and after carrying out the eHealth usability evaluation with the elderly.

3.1 Case study

Conduction of the pre-test

Before actual eHealth usability evaluation, a pre-test was conducted with one elderly participant. Implementing a pre-test includes understanding our prospective users and potential challenges arising, while evaluating the usability of eHealth systems. Based on the findings and feedback of the pre-test, the formulated scenario and derived tasks were slightly modified. During the pre-test, we realized that some elderly might not be equipped with a printer at home. Therefore, we modified Task 2 to state that the patient should save the prescription on the computer before sending it to the printer, for instance.

Set-up of eHealth usability evaluation

Conducting the eHealth usability evaluation at the elderly’s homes proved to be useful because elderly felt comfortable in their familiar environment. Elderly users preferred working on their own computers because they are familiar with their own technical equipment. We also recognized that conducting the evaluation “live” is useful, because supervising the elderly with regard to their behavior during the eHealth usability evaluation was necessary (e.g., verbalizing their thoughts, comprehension of tasks). We noted that the elderly are affected by cognitive declines, such as speed of comprehension, which impacted the comprehension of conducting the eHealth usability evaluation. To overcome this challenge, we convinced the elderly and explained to them that they are a great help in finding usability problems.

Procedure of eHealth usability evaluation

It became clear that discussing the procedure of the eHealth usability evaluation with the elderly prior to the start of the eHealth usability evaluation was important because the elderly needed time to adjust to the new situation of conducting the eHealth usability evaluation that was unfamiliar to them. We noted that allocating time for answering questions that the elderly might have was important because the elderly showed interest in the procedure of the eHealth usability evaluation. For instance, the elderly asked questions to clarify concerns, such as privacy concerns. The elderly also showed interest in how personal data was processed during the eHealth usability evaluation.

We recognized that the elderly had difficulties speaking their thoughts out loud because their attention quickly decreased during the evaluation. For this reason, we encouraged the elderly to express their thoughts out loud and asked comprehension questions during the evaluation. We applied brief tasks and recognized that three brief tasks were sufficient because the elderly got tired quickly. Furthermore, we read the tasks out loud because we noticed that this promoted the attention and speed of comprehension of the elderly as to which tasks they should work on. We made no video or audio recordings because the elderly objected the use of recordings due to privacy concerns.

Co-Discovery Evaluation

We invited all seven elderly participants to conduct a Co-Discovery Evaluation as well. The elderly refused to conduct a Co-Discovery Evaluation for a number of reasons. They preferred conducting tasks on their own and emphasized this with statements like “I am old enough to accomplish tasks alone”, “I am an independent person and prefer to struggle through tasks on my own”, or “No thanks. I do not need any help from others”. Furthermore, the elderly confirmed their ability to cope with modern technologies. For example, one elderly emphasized: “Why do I need supervision? I am able to handle a computer myself”. Due to these concerns, we were not able to conduct a Co-Discovery Evaluation with the elderly participants.

Cooperative Usability Testing

We invited all seven elderly participants to conduct Cooperative Usability Testing. All participants refused to conduct Cooperative Usability Testing. They were concerned about the usefulness of video recordings and asked questions like “Why should the evaluation be recorded by video when it is also possible to take notes?” The elderly also had concerns like “I don’t want to see and hear myself twice” or “I do not agree with video recordings because I don’t know where the data really ends up”. Privacy concerns occurred despite precise explanation that personal information would be deleted after completion of the eHealth usability evaluation. Due to these concerns, we were not able to conduct Cooperative Usability Testing with the elderly.

Remote User Testing combined with Think Aloud

All seven participants agreed to attend Remote User Testing combined with Think Aloud. We observed a variety of notable differences in how the elderly interacted with the web-based eHealth intervention. Some participants had problems saving the prescription locally on the computer (see also Table 2). This was not due to poor usability of the web-based eHealth intervention but occurred because the elderly were not familiar enough with the functions of their own computer.

We found usability problems, such as problems concerning the readability of the font size and textual or visual appearance. The most common usability problem was the lack of clarity when entering the received TAN, as it was already pre-filled in light grey font in the input field (Table 4). After the TAN was verified, the description could be downloaded by the elderly (step two, see also Fig. 3). However, the second most common usability problem was that the elderly tried to complete the already pre-filled prescription themselves. The elderly did not understand that the prescription had already been completed by the physician, which made them feel impatient and nervous. The third most common usability problem was that the elderly had difficulties in distinguishing the colorized buttons.

Table 4 Identified usability problems \(n=16\) during eHealth usability evaluation
Full size table

3.2 Systematic review

Applied eHealth usability evaluation methods to elderly

We identified 20 studies that report on applied eHealth usability evaluation methods with elderly participants (Table 5). The majority of the studies applied user-based eHealth usability evaluation methods (18/20), such as Questionnaires, Think Aloud, or Usability Testing. Most of the studies applied Questionnaires as a single or additional eHealth usability evaluation method (14/20). From these mentioned studies applying Questionnaires (\(n=14\)), 11 studies used the System Usability Scale. The second most applied eHealth usability evaluation method was Think Aloud (5/20), followed by Usability Testing (3/20). The minority of the studies applied expert-based eHealth usability evaluation methods, such as Heuristic Evaluation and Cognitive Walkthrough (2/20).

Table 5 eHealth usability evaluation methods applied to elderly participants
Full size table

Equal numbers of studies evaluated both eHealth systems (e.g., for health prevention, self care, or activity tracking) (9/20) and mHealth systems (9/20), followed by web-based eHealth systems (4/20). There are studies that conducted an eHealth usability evaluation with two systems, such as mHealth system/eHealth system (1/20) as well as mHealth system/web-based eHealth system (1/20).

Challenges of conducting agile, easily applicable, and useful eHealth usability evaluations with elderly participants

We identified six studies that report on challenges of conducting eHealth usability evaluations with elderly or disabled participants. Table 6 describes four studies that indicate challenges of conducting eHealth usability evaluations with elderly and Table 7 describes two studies that indicate challenges of conducting eHealth usability evaluations with disabled participants. All six studies report on challenges due to cognition, such as that during eHealth usability evaluations elderly or disabled participants easily lose their attention and their concentration, or did not sufficiently speak their thoughts out loud. One study reports on challenges due to motivation, such as that the elderly learn a new system more easily if they are supported by a mentor. Challenges due to perception were also reported in one study, such as that the elderly had difficulties in recognizing the font size of a text needed to accomplish the tasks during an eHealth usability evaluation. Finally, one study reported on challenges due to physical abilities, such as difficulties with the finger-based input necessary to accomplish the tasks during eHealth usability evaluations.

Table 6 Reported challenges of eHealth usability evaluations with elderly participants
Full size table
Table 7 Reported challenges of eHealth usability evaluations with disabled participants
Full size table

3.3 Recommendations derived from both the case study and systematic review

Recommendations on how to deal with challenges prior to, during, and after carrying out the eHealth usability evaluation with elderly participants

From the case study and the systematic review, we formulated 24 recommendations to address challenges prior to, during, and after carrying out the eHealth usability evaluation with elderly participants (see Table 8). We categorize the challenges according to the stage of software lifecycle (requirements engineering, design, and evaluation), taking into account the stage that follows the eHealth usability evaluation. We then related the recommendations to age-related barriers (cognition, motivation, perception, and physical abilities). The recommendations originate equally from the results of the case study and systematic review (in both cases 15). Some of these recommendations derived from the case study are also supported by the literature (6/15). For example, our recommendation to encourage elderly participants to speak their thoughts out loud originates from our case study that is also confirmed by the literature. Two recommendations for the requirements engineering stage of the software lifecycle were derived (2/24). We formulated three recommendations in both cases—for the design stage of software lifecycle and for the stage that follows the eHealth usability evaluation—that are supported by literature (3/24). In total, 16 recommendations could be derived specifically for the evaluation stage of software lifecycle. From these 16 recommendations, we defined the most recommendations for motivation (8/16), the second most recommendations for cognition (5/16), the third most recommendations for physical abilities (2/16), and the fewest recommendations for perception (1/16).

Table 8 Recommendations formulated from case study and systematic literature review on how to deal with challenges prior to, during, and after carrying out the eHealth usability evaluation conducted with elderly participants
Full size table

4 Discussion

We aimed to examine the following research questions: (1) Which eHealth usability evaluation methods are applicable to evaluate eHealth interventions with elderly users?, (2) What challenges might be encountered when conducting an agile, easily applicable, and useful eHealth usability evaluation with elderly users?, and (3) How to deal with and overcome these challenges that arise prior to, during, and after carrying out the eHealth usability evaluation conducted with elderly users?

Our first research question aimed to identify which eHealth usability evaluation methods are applicable to evaluate eHealth interventions with elderly users. Our findings show that Remote User Testing combined with Think Aloud could be successfully applied to evaluate the web-based eHealth intervention with elderly users. Co-Discovery Learning and Cooperative Usability Testing were found not to be suitable for conducting the eHealth usability evaluation with elderly users. We found that there are eHealth usability evaluation methods that are not applicable in their current version with elderly users. These include Co-Discovery Evaluation, Cooperative Usability Testing, Retrospective eHealth usability evaluation methods, and Think Aloud. Some of these eHealth usability evaluation methods can be useful if they are evolved and adapted to meet the challenges of conducting eHealth usability evaluations with elderly participants.

Research has proposed the Peer Discovery usability evaluation method as a modified version for user-based eHealth usability evaluations, taking into consideration Think Aloud, which allows the elderly to complete the tasks of the eHealth usability evaluation together with a caregiver or a family member taking part in the eHealth usability evaluation to make the elderly participant feel more comfortable [40]. Research further suggests the eHealth usability evaluation method Peer Community Sessions, where eHealth interventions are evaluated by a group of elderly participants [40]. We cannot confirm these suggestions on conducting an eHealth usability evaluation with a caregiver or family member or on elderly participant working in groups because the elderly in our study refused to work with peers or to involve a family member. Our results showed that the elderly are willing and able to perform tasks on their own.

In the course of our research, Remote User Testing combined with Think Aloud proved to be suitable with elderly, although research showed that there is a lack of existing guidelines available for Remote User Testing with elderly users [42]. In our study, we conducted the Remote User Testing in a modified version bringing together the elderly at their homes with the software developers via video conference in the eHealth usability evaluation. The application of a modified version is confirmed in the literature, which states that Remote User Testing must be adapted to and tested with elderly participants [42]. We observed and thus recommend that carrying out the eHealth usability evaluation with elderly users “live” at the their homes using brief tasks is one possibility to overcome challenges.

Our second research question addressed what challenges might be encountered when conducting an agile, easily applicable, and useful eHealth usability evaluation with elderly users. The results show that challenges due to motivational, cognitive, or physical decline of elderly users were identified. For instance the identified challenges due to cognitive decline, can result in limited verbalization skills, reduced attention, or speed of comprehension. Technology should be accessible and usable [73] which confirms our findings that declines due to cognition, motivation, perception, and physical abilities of elderly users need to be considered. Our results show further that the majority of the studies applied Questionnaires as a single or additional eHealth usability evaluation method with elderly users. Half of the studies applied the System Usability Scale as Questionnaire. However, recent research has shown that the System Usability Scale is an inadequate stand-alone eHealth usability evaluation method [19]. Further, Questionnaires were criticized due to their complex scoring system [39] and their non-suitability for elderly [11]. No particularities were found with regard to the setting of the studies because research on applying eHealth usability evaluations with elderly is conducted equally often in Europe and outside of Europe.

During the eHealth usability evaluation, we recognized that the elderly struggled to speak their thoughts out loud and to keep their focus on accomplishing the tasks. We observed different concerns of the elderly that encompassed, for example, privacy concerns and the understanding (concerning Co-Discovery Learning) of why a joint evaluation with another participant should be beneficial. We focused our study on identifying challenges for conducting eHealth usability evaluation with the elderly because their ability to use modern technologies is important so that they do not lose touch with society [74]. This coincides with the concept of universal usability in the course of which the design of technologies encompasses the needs of different prospective users [75]. Universal design aims at considering special needs of the elderly and disabled users [76, 77]. This has resonance with research that suggests the need of inclusive design for universal access [78].

Our third research question addressed ways to deal with and overcome these challenges that arise prior to, during, and after carrying out the eHealth usability evaluation conducted with elderly users. We formulated 24 recommendations to address and overcome these challenges that arise prior to, during, and after carrying out the eHealth usability evaluation with elderly users. Our recommendations facilitate accessibility, acceptability, and usability of information technology, such as eHealth systems, for elderly users. We related our recommendations to the stages of the software lifecycle (requirements engineering, design, and evaluation), as not only acceptability has become one of the most important aspects of healthcare interventions design, evaluation, and implementation [79]. Indeed, in line with [80], our results confirm that accessibility as well as usability must be considered in early stages of software lifecycle, such as the design stage.

We structured the recommendations also according to age-related barriers into cognition, motivation, perception, and physical abilities to address age-related declines of the elderly. Research has shown that for universal participation, considering the two aspects of accessibility and usability is crucial [81]. In terms of accessibility of technology, more generally universal access to technology, we must consider that the interaction between human and computer is characterized by signal as well as sign processes. Considering these sign processes, the technical semiotics, is fundamental to the understanding of Human-Computer Interaction [82].

Recent research has shown that most systematic literature reviews address usability as it relates to elderly users [83]. This illustrates the need to conduct a case study and likewise the fact that most of our recommendations are based on the case study findings. Although the idea of universal usability has been addressed by various studies [75, 84], it is obvious that there is still a lack of contributions implementing case studies to investigate challenges of elderly users in making eHealth systems accessible, acceptable, and usable for the information society.

Limitations

We originally intended to apply all three chosen eHealth usability evaluation methods with the elderly. However, we observed that the elderly refused to participate in Co-Discovery Evaluation and Cooperative Usability Testing. Nevertheless, we were able to propose 24 recommendations from the case study together with the results of the systematic review. Conducting the case study with one eHealth usability evaluation method, Remote User Testing with Think Aloud, presents a disadvantage that indicates a difficulty in generalizing the results. We invited all seven elderly patients to attend the eHealth usability evaluation with each chosen eHealth usability evaluation method. Despite the small number of participants (\(n=7\)), we reached a thematic saturation after conducting the eHealth usability evaluation with four elderly users because the usability problems found were repeated. However, we cannot assume that we have found all possible usability problems. To generalize our results, it is essential to validate these findings with more elderly participants and a different eHealth intervention. During eHealth usability evaluation, the elderly objected to the use of video recordings due to privacy concerns. However, additional video recordings would have been helpful to facilitate the comprehension as to why some usability problems may have occurred.

To accomplish the systematic literature review, we selected four databases (ACM Digital Library, Cochrane Library, IEEE Xplore, and Medline) that were relevant to the thematic field of our systematic review. A disadvantage of selecting these four databases that catalog peer-reviewed literature (except conference papers) is that we may have missed relevant papers within gray literature. We included papers that report on experiences such as challenges, applicability, acceptability, or feasibility of applied eHealth usability evaluation methods to elderly. Since experiences on challenges, applicability, acceptability, or feasibility of applied eHealth usability evaluation methods for elderly users may be mentioned as a side result in research, we cannot assume to have found all possible relevant experiences. We excluded papers that were published before 2012 because we focused on the most recent results. Our results showed that half of the identified studies on applied eHealth usability evaluations with elderly were published in the last three years. However, it may be possible that we have missed relevant papers from the years before 2012. We also included findings on reported challenges with disabled participants because we did not want to exclude possible further relevant information from this related field. However, it is not possible to generalize the results, as only a small number of studies (\(n=2\)) are involved.

From both the case study and the systematic review, we derived recommendations on how to overcome challenges prior to, during, and after carrying out the eHealth usability evaluations with elderly participants. No contradictions could be identified between the results of the case study and the systematic review; however, contradictions cannot completely be excluded.

Future research

We found that established eHealth usability evaluation methods such as Co-Discovery Evaluation, Cooperative Usability Testing, and Think Aloud cannot be used in their original version. These eHealth usability evaluation methods need to be modified to be useful for evaluating eHealth systems with elderly participants. This illustrates that there is a need to explore and propose evolved versions of well-established eHealth usability evaluation methods that had been adapted to accomplish agile, easily applicable, and useful eHealth usability evaluations with elderly users. It is important to further explore how elderly persons who suffer from age-related barriers such as cognition, motivation, perception, or physical abilities behave during eHealth usability evaluation.

Furthermore, future research is needed to facilitate the selection of an appropriate eHealth usability evaluation method. In a subsequent study, the authors aim to develop a decision aid valuable for guiding the choice of a suitable eHealth usability evaluation method.

5 Conclusion

A systematic literature review conducted subsequently to the case study unveiled that there are eHealth usability evaluation methods that are unsuitable for implementing an agile, easily applicable, and useful eHealth usability evaluation with elderly participants. Based on our findings from the case study and the systematic review, we derived recommendations on how to deal with challenges prior to, during, and after applying eHealth usability evaluation methods with elderly users and how these challenges can be overcome. The majority of our proposed recommendations to facilitate eHealth usability evaluation relate to challenges concerning cognition and motivation. We found that established eHealth usability evaluation methods will require modification to be useful for evaluating eHealth systems with elderly participants. Our insights may encourage future studies on eHealth usability evaluation with elderly users. The recommendations contribute to improving the accessibility, acceptability, and usability of eHealth interventions by the elderly.