1 Introduction

Globally, increasing life expectancy is leading to growing numbers of older people with some form of cognitive impairment [1]. The majority of older people with cognitive impairment want to remain at home and live independently for as long as possible [2]. This means supporting people to function within and outside the home with activities of daily living as well as socialisation, hobbies and travel. Over time, people living with progressive cognitive impairments, such as dementia, require more help, with families across the world providing the bulk of care [1]. However, for some older adults, their needs will increase to the point where they need 24-h support which for some will be provided in a care home.

Alongside this emerging societal challenge, technological development is advancing rapidly to offer potential solutions [3]. Across Europe, a growing number of Information and Communication Technology (ICT)-based services targeting older adults with cognitive impairment have been developed, but these have frequently comprised small feasibility or pilot studies. Integrating these various solutions and making them available at scale could provide a major solution to this growing societal need, by supporting older adults with cognitive impairment to remain independent for longer. Given that the majority of older people with cognitive impairment are cared for by their families, it is also important to gain an understanding of their views in relation to services and support. Additionally, the views of formal caregivers working in services for older adults with cognitive impairment are also vital for informing future strategies for care services.

The Independent Living Functions for Elderly (IN LIFE) project was established to integrate a collection of ICT services within an interoperable, open, personalised platform to empower older adults with cognitive impairment to maintain independence [4]. Recognising the breadth of activities older adults wish to carry out, the IN LIFE platform addresses a wide range of activities including those within the home, travel outside the home, communication, health maintenance, mobility and socialisation alongside caregiving. The completed IN LIFE platform comprises 17 services presented in four categories: Independent Living Support, Travel Support, Socialisation and Communication Support and Caregiving Support (Table 1 contains the full list of services and where they were evaluated). Ten of these services, online exercising (IL1;[5], daily functions assistant (IL2; [6]), falls detection (IL4: [7]), Guardian Angel (IL5; [8]), mental training (IL6; OASIS project), physical activity monitoring (IL7), car driving ability assessment (TS1; [9]), public transport support (My Routes; TS2: ASK-IT project), socialisation and communication support (SC1; CIRCA [10]) and teleconsultation (CS3; [11]), were established services developed and tested separately in previous projects. The other seven services were extensions of pilot or feasibility prototypes that were completed for the IN LIFE project. The breadth of services required an evaluation strategy that included participants with varying levels of cognitive ability who were residing in a range of living situations.

Table 1 IN LIFE servicesa evaluated at each pilot site
Full size table

2 Evaluating IN LIFE

The IN LIFE evaluation strategy was informed by the need to assess each of the 17 services with the intended users—older adults living with cognitive impairment, family caregivers and formal caregivers—as well as the overall IN LIFE platform. While most of the services were designed for people living at home to support their daily activities (e.g. My Routes and daily functions assistant), the Socialisation and Communication support services—CIRCA and CIRCUS—were designed for promoting interactions between caregivers and people living with a wide range of cognitive impairments. These diverse services were included to address the breadth of activities and priorities of people living with cognitive impairment [12] and to evaluate the effectiveness of the platform when people experience more severe cognitive impairment. This created a need to identify evaluation methods to cover a broad range of services for users with different levels of cognitive challenges, in a range of environments. The measures also needed to be available in multiple languages and suitable for use at scale with the large number of participants required to provide evaluation of the 17 services and platform. The selection of outcome variables and appropriate measures for the target populations was informed by previous projects of the IN LIFE consortium and existing evaluation frameworks and tools.

To capture the “intricate nuances” of the introduction of ICTs into the lives of older adults requires a multifaceted approach [13]. Gomersall and colleagues [14] proposed that network-based approaches, which acknowledge the networks of individuals and organisations into which technologies are being introduced, are a useful starting place for developing technology evaluations. The IN LIFE evaluation strategy identified the networks around older adults as including informal caregivers, healthcare and formal caregivers, and a range of stakeholder organisations that provide services to older adults. The IN LIFE evaluation strategy was developed to ensure that each of the 17 individual services was examined by the intended users—older adults, healthcare professionals or informal caregivers—and, where relevant, the interactions between them. Additionally, the evaluation was designed for these groups plus representatives of stakeholder organisations, including social services, hospitals, home care, assisted living and long-term care providers to evaluate the whole IN LIFE platform.

The number of tools, frameworks and methodologies for evaluating the effectiveness and impact of ICT has increased as the range of technological solutions has expanded. Many of the existing tools have been developed in the context of Assistive Technologies, which are devices and services, such as fall alarms or medication reminders, prescribed or provided by health or social care providers. A number of evaluation approaches focus on the assistive technology or services. For example, the 12-item Quebec User Evaluation and Satisfaction with Assistive Technology devices and services (QUEST 2.0; [15]) considers characteristics of the device (including size, safety) and the services (including delivery and professionalism). Similarly, the Post-Installation Technique [16] was developed for health or social care service providers to conduct a risk assessment for equipment introduced into the homes of older adults. This looks at equipment failure or other problems, the potential impact on end users and solutions for addressing these problems.

Other approaches consider variables relating to the user. For instance, the matching person and technology (MPT; [17]) model looks at three factors related to the personal motivation to use assistive technology: environmental factors, user needs and preferences, and functions and features of the appropriate available, technology. Similarly, the Everyday Technology Use Questionnaire (ETUQ; [18]) uses a detailed interview to explore an individual current or previous usage of everyday technologies (e.g. smartphones, automated banking) with an emphasis on the relevance of the technology to them.

In a detailed examination of the product development lifecycle of assistive devices based around the experience of individuals living with disabilities, Hersh (2010) produced a three-part series covering the 1. Design [19]. 2. Evaluation [20] and 3. Outcomes [21]. She argued that evaluation needs to consider both the functioning of the technology and the impact on the user, throughout the whole process [20]. Hersh identified four factors to consider in evaluation: compliance and good practice (e.g. health and safety, ethical issues), technical issues (e.g. data security, reliability, robustness and dependability), end-user issues (e.g. usability, accessibility and acceptability) and resource and financial issues (e.g. direct and hidden costs, feasibility and costs of upgrading). In addition, the importance of understanding reasons for abandonment of technology is highlighted in Part 3 on outcomes of assistive devices [21].

Contemporaneously the European Intelligent System for Independent living and SElf-care of seniors with cognitive problems or Mild Dementia (ISISEMD) project produced a technology evaluation framework [22] focused on telemedicine. The ISISEMD framework identified quality of life of older adults and socio-economic impact on the health or social care systems as key outcome variables. Their assessment included user acceptance and satisfaction with the technology assessed through questionnaires and interviews and caregiver questions about willingness to pay for the services. Along with Hersh [21], the ISISEMD team [23] highlighted the importance and complexity of assessing the impact of a technology intervention on the quality of life of the user in context.

Building on these existing models, the IN LIFE evaluation examined the impact on the quality of life of individuals living with cognitive impairment in different settings, with different levels of cognitive impairment, different support systems and in different healthcare systems. The evaluation also captured user factors such as preferences, digital literacy, socio-economic status, affordability, plus information about usability and usefulness, technology accessibility and perceived usefulness for family caregivers, formal caregivers and stakeholder organisations. Specifically, the IN LIFE evaluation set out to (i) examine the impact on quality of life of older adults with cognitive impairment, (ii) examine the user experience and usability of the IN LIFE services and platform and (iii) assess the economic case for IN LIFE.

3 Method

3.1 Design

The IN LIFE evaluation comprised an assessment of the situation prior to availability of the IN LIFE platform (baseline) and an assessment of the impact of the IN LIFE platform (pilot). Six pilot sites, Greece [GR], Netherlands [NL], Slovenia [SL], Spain [SP], Sweden [SW] and United Kingdom [UK], were identified to conduct the IN LIFE evaluation. To ensure access to the four groups of identified participants, each pilot site included one user organisation as a partner, alongside a research organisation. The partner organisations included two acute hospitals (GR, SW), two long-term care organisations (NL, UK), one home care health provider (SL) and one ageing research organisation (SP).

Data were collected at the six pilot sites using a mixed methods approach. To maximise the evaluation, it was designed such that each of the 17 services was examined in two ways. First, each service was identified as a primary service in at least one country as detailed in Table 1. This was to ensure that each service in the IN LIFE platform underwent evaluation. In addition, each user was able to access each of the 17 services through the IN LIFE portal during their evaluation of the platform. Some of the services, such as the driving assessment, were single use, while others were designed to be used over time, e.g. Guardian Angel, Route Planning, Socialisation and Communication support (CIRCA). Each pilot site identified the service(s) that would be tested at their location (Table 1). To accommodate this variability between the 17 IN LIFE services, each pilot site developed a localised recruitment and testing plan, with common measures but different lengths of participant involvement. The evaluation took place between March 2016 and September 2017.

3.2 Participants

Four groups of participants were identified for IN LIFE: older adults living with cognitive impairment, informal caregivers, health and social care staff (healthcare professionals) and representatives of stakeholder organisations. Each national site had a target number of participants to recruit from each participant group (Table 2 shows anticipated and actual numbers of participants). Local processes were followed at each national site for securing ethical approval. Recruitment locations varied across the sites and included general hospital outpatients, older adult day centres, community programs, assisted living facilities, Alzheimer’s societies, senior’s groups and long-term care homes. Each site recruited representatives of each of the four participant groups (Table 2).

Table 2 Use group representation per country pilot site—predicted and actual
Full size table

3.3 IN LIFE services

The 17 services in the IN LIFE platform were available for participants across the six pilot sites to evaluate (see Table 1). Details of each service are provided in Appendix 1.

4 Materials

The IN LIFE project integrated different technologies to the platform, so different devices were tested in each pilot, according to the needs of each person. These were mainly mobile devices, telephones, laptops and tablets, as well as wearable monitoring devices and specific devices for driving. Likewise, pen and paper solutions and computerised assessment protocols were used to carry out the assessments. Primary outcomes for older people with cognitive impairment included quality of life and cognitive function. The Swedish pilot site focused on assessing communication and social interaction and used measures designed to assess these functions.

Each pilot site elected to use the same cognitive assessment as used by the local memory assessment services to ensure consistency. This resulted in four sites (Greece, Netherlands, Slovenia, Spain) using the Mini-Mental State Examination (MMSE: [24]) and one, (UK) using the Addenbrookes Cognitive Examination—3rd edition (ACE-III; [25]). The Swedish pilot site did not administer a cognitive measure to their older adult participant.

4.1 Mini-mental state examination [24]

The MMSE is a standardised measure that provides a brief assessment of cognitive state. It is particularly useful for patients with dementia as administration takes about ten minutes. Questions cover orientation to date and place, registration, attention and calculation, recall and language (including writing). Total score possible is 30, with scores below 27 indicating cognitive impairment.

4.2 Addenbrookes Cognitive Examination-3rd edition [ACE-III; [25]].

The ACE-III focuses on five cognitive areas–attention/orientation, memory, language, verbal fluency, visuospatial skills. It is scored out of 100, with a cut-off score for dementia between 82 and 88/100, where lower scores indicate greater cognitive impairment. The ACE-III requires training for accurate administration and scoring and takes approximately 15–20 min to administer and score. The ACE-III has high internal validity as measured by Cronbach’s a coefficient (a = 0.88; [26]), along with high levels of sensitivity and validity ([25]).

4.3 Quality of life

The EQ-5D (5L version) [27]is a generic instrument for describing and valuing health-based quality of life. It is based on a descriptive system that defines health in terms of 5 dimensions: Mobility, Self-Care, Usual Activities, Pain/Discomfort and Anxiety/Depression. For each dimension respondents rate themselves as having ‘no problems’, ‘slight problems’, ‘moderate problems’, ‘severe problems’, or ‘extreme problems’. In addition, respondents also rate their overall health on the day of the interview on a 0–100 hash-marked, vertical visual analogue scale (EQ-VAS). The EQ-5D has been widely tested and used in both the general population and patient samples and translated into over 130 different language versions (www.euroqol.org). Analysis was performed according to the EQ-5D official guidelines (https://euroqol.org/wp-content/uploads/2016/09/EQ-5D- 5L_UserGuide_2015.pdf).

4.4 IN LIFE questionnaires

4.4.1 Several survey tools were created specifically for IN LIFE

4.4.1.1 Demographics questionnaire

This was designed to collect demographic and socio-economic data. This included questions about age, gender, education, current living arrangements, employment status, healthcare provision and total annual household income.

4.4.1.2 Technology familiarity

This questionnaire was developed to provide an assessment of technology usage and familiarity at entry into the project. Questions asked about access to a PC or laptop, tablet, smartphone and the Internet.

4.4.1.3 IN LIFE evaluation

This tool was completed after using the IN LIFE platform and included questions about ease of use, confidence in daily activities, independence, future use of the platform, if free of charge or not, how people would be prepared to pay for access to the services, and factors influencing their decisions. For the services aimed at caregivers, it is important that they are evaluated by both an older adult with cognitive impairment and at least one caregiver, formal or informal, to examine the usability and usefulness. If either party does not find a service easy to use or do not see a need, then it will not be adopted.

Additional measures were used at each site—details can be found in the IN LIFE final report.

4.5 Procedure

Each pilot site identified the primary services it would evaluate in addition to the whole IN LIFE platform. The choice of services informed the recruitment strategies employed at each site to reach the target number of participants in each group. The choice of services also informed the selection of measures administered at each site (Appendix 3; Table 5). The procedure followed at each pilot site is described in Appendix 2. The results of the measures conducted at each site are reported here. Full results from each site can be found in the IN LIFE final report, with additional publications of the detailed results from the UK site (Astell, Smith, Potter & Preston-Jones, 2018) and the Sweden site (Ferm, Ekstrom, Larsson & Samuelsson, 2020).

5 Results

5.1 Demographic information

A total of 2090 participants were recruited across the six pilot sites over an 18-month period (Table 2). As intended, the largest group of participants were older adults (N = 1205; Fig. 1), with the number recruited exceeding the target amount (Table 2). Additionally, more health and social care staff were recruited than anticipated but fewer family caregivers and stakeholders (Table 2). The mean age of the participants with cognitive impairment who provided their age or date of birth (N = 1069) was 75.62 years (53–104), of whom 62% were female.

Fig. 1
figure 1

Percentage of participants in each group

Full size image

Nine hundred and fifty-six older adults provided information about where they live, which revealed 35% lived with a spouse or partner, 30% lived alone, 9% with family, and 25% in other settings. Of these 234 other settings, the largest proportion were residential care (55%), followed by retirement communities (21%) and sheltered housing (11%). Eight hundred and forty-five reported their highest educational level, with 23% completing higher education, 23% completing secondary education and 42% completing primary education. Regarding employment status, of the 892 older adults who responded, predictably the majority (88%) were retired from paid work. However, 10.4% of the remaining 12% were still working full-time, while the rest worked part-time. Six hundred and ninety-four older adults provided information about their healthcare provision with 84% funded by government, 5% private insurance, 6.8% self-pay and the rest (4.2%) some combination of these three. Finally, 317 older adults responded to the question about annual household income with the majority choosing not to answer or not knowing the answer. Those who did respond were distributed as follows: less than €5,000 = 30, €5,000–€19,999 = 217, €20,000–€49,999 = 68, €50,000–€99,999 = 2. In response to the technology use questionnaire, around 50% were using a mobile phone, of which 29% were smartphones and 15% had used a tablet before.

The 360 informal caregivers who provided their age or date of birth had a mean age of 59.4 years, 66% were female and 82% cared for a family member. Three-quarters of the informal caregivers have access to a personal computer and/or laptop either at work or at home, and around 80% of all caregivers have access to the Internet. Three hundred and ninety-one health and social care staff provided their age or date of birth. Their average age was 42 years, and they were 95% female. They were working in a range of healthcare jobs including nursing, activity coordinator and personal support in public or private residential homes, and public hospitals. Forty-one percent of the healthcare professionals had more than 11 years of experience working with older adults with cognitive impairment, interacting with an average of 5 people per day. Staff reported what they like most about their current use of technology for their work is the ease of use, how work gets carried out easier and direct access to information from anywhere. What they really do not like is that in most cases no Internet exists at their workplace, especially within public hospitals, as well as the cost of applications and services, plus when access to the Internet is slow. They think that the interoperability of applications, medical image transfer technologies, free access and Internet existence at their workplace are still missing.

The majority of representatives of stakeholder organisations who evaluated IN LIFE services were female (58%), 46% were aged less than 60 years old and 57% responded that they currently use technology (although 38% did not answer this question). The stakeholders were involved with older adults with cognitive impairment mostly through their work and less through family and friends. The majority of the stakeholders had over 15 years of work experience, and their specialisms included psychologist, psychiatrist, physiotherapist, politician, government worker and IT. They believed the most important issues for this population were feelings of anxiety, insecurity, sadness, memory loss, lack of concentration and difficulties with orientation. What stakeholders liked most in the technologies they use at work is direct access to information they seek from anywhere. What they like least is that data could get lost, access to the Internet is usually slow, and they find it difficult to keep paper copies. What they believed is still missing is a country-wide patient database, the interoperability of applications, as well as new computer power at their workplace.

5.2 IN LIFE services

The evaluation strategy was designed for each relevant user group to evaluate the individual services and the combination of services available through the IN LIFE platform. The distribution of evaluations varied across the four themes (Fig. 2). Independent Living and Socialisation and Communication services accounted for almost 80% of the evaluations (Fig. 3).

Fig. 2
figure 2

Percentage usage of services by themes

Full size image
Fig. 3
figure 3

Mean EQ-5D index per condition (overall)

Full size image

Each service was evaluated by older adults with cognitive impairment (Table 4). The impact of IN LIFE on the quality of life of older adults with cognitive impairment was examined with the EQ-5D, the health-related QoL index used across all sites that measured QoL. Initially, a data transformation was selected to include all data, but variance distribution increased, and it was decided to rely only on common instruments used at all sites in order to yield any inferences for the impact these analyses might have to European population. Consolidated EQ-5D data were used to calculate the overall QoL index of IN LIFE population. A nonparametric alternative was selected because normality assumption was highly violated (p = 0.005) (Fig. 4).

Fig. 4
figure 4

Daily activities in treatment phase

Full size image

We examined the participant’s scores on the EQ5D to look at the impact of using the IN LIFE platform. We found that the EQ5D scores declined in the baseline phase, although this was not significant (p > 0.05; see Table 3). In the pilot phase, the scores increased significantly (p = 0.001). We also examined the cognitive scores of the participants in the baseline and pilot conditions. For comparison purposes, the scores on the ACE-III were converted to MMSE equivalents using the formula of Giebel and Challis (2017). A significant overall negative correlation was found between age (years) and MMSE (0–30 scores) (r = − 0.191, p < 0.001). Comparison of the MMSE scores over the baseline phase found a decline in scores (mean difference − 2.97), but this was not significant (p > 0.05; Table 3). MMSE scores did not change during the pilot phase (p > 0.05: Table 3) with a small mean difference (− 0.312) suggesting cognitive maintenance.

Table 3 Mean (SD) baseline and pilot scores for cognition and quality of life
Full size table

Health and social care staff provided just under one quarter of the evaluations with informal caregivers providing 12% (Table 4). In regard to the informal caregivers, it was noted that IN LIFE can improve their quality of life, help with communication and assist in monitoring their family member’s everyday safety and activities, empowering their family member to feel more in control. It can also assist healthcare professionals by helping them clarify and better organise and maintain workload, communicate better with their patients and provide integrated care and continuous monitoring throughout their daily activities. Stakeholders examined the overall IN LIFE platform as well as some individual services. They believed the use of IN LIFE could improve quality of life and help maintain abilities and cognitive function of older adults with cognitive impairment. In addition, it could inspire older people to use new technologies.

Table 4 Evaluation of each IN LIFE category by the four participant groups
Full size table

In respect of the IN LIFE questionnaires, the majority of users considered the Independent Living support services as very useful or useful (59.33% of elderly, 73.63 of informal caregivers and 44.19% of healthcare professional), very reliable or reliable (51.87% of people with cognitive impairment and 53.84 of informal caregivers), were not concerned over their data privacy while using the services (37.92% of elderly users and 71.43% of informal caregivers) and a high proportion felt the use of the services improved their overall health and wellbeing (Table 4). Despite these ratings, older adults and informal caregivers do not feel prepared to pay for the services.

A high percentage of informal and formal caregivers considered the travel support services as very useful or useful ( > 76% and > 83% respectively). Informal caregivers had, however, no concern while using the services, considered it an improvement on their overall health and wellbeing while caring for their older relative with cognitive impairment, but would still not pay for using the services (Table 4). Regarding the car driving ability assessment, the specific IN LIFE service is targeted towards healthcare professionals. However, the driving assessment results (from the 156 older drivers) were also used to improve and refine the existing thresholds of the driving assessment instruments.

The Socialisation and Communication support modules were the most accessed services. Reliability of the Socialisation and Communication support modules was not included in the questionnaire administered to the users, but the majority of caregivers did, however, consider the services very useful or useful (54.54% of informal and 63.36% of formal caregivers).

The carers’ support services received a high rating on usefulness and reliability (reliability was only assessed by the elderly users) by all users participating in the pilots. Nearly half of informal caregivers, however, felt concerned over the privacy of the personal data while using the services and consequently we have 64.74% of them stating there was no effect of improved overall health and wellbeing.

6 Discussion

A pragmatic evaluation strategy succeeded in the recruitment of a large number of participants across six sites to evaluate the IN LIFE platform. This was in part attributable to the pairing at each pilot site of a user organisation and a research organisation. The partner organisations included two acute hospitals, two long-term care organisations, one home care health provider and one ageing research organisation. This range of user organisations ensured access to a wide range of older adults living with cognitive impairment, from those attending outpatient services and memory assessment clinics to people living in supported housing and care homes. This breadth of participants (in both cognitive ability and living situation) was sought to provide realistic evaluation of the range of services, in the IN LIFE platform which had different intended users.

Guided by previous approaches to evaluation, e.g. [21, 28], the IN LIFE evaluation aimed to examine the impact on quality of life of older adults with cognitive impairment and the user experience and usability of the system. In respect of the quality of life of older adults with cognitive impairment, we found a significant impact in the pilot phase relative to the baseline phase. The results indicated that ICT services targeting older adults with cognitive impairment should be developed with consideration for cognitive level of functioning (and level of deterioration), users’ level of operation of digital technologies (use, ease of use and frequency of use) as well as educational level. This is an important consideration when planning out multicomponent or long-term technological services for people living with cognitive impairment which is expected to change over time.

Following the recommendations of [17, 20], a large part of the evaluation was concerned with the user experience and usability of the individual services as well as the overall platform. The evaluation strategy was designed to ensure that each service was at least examined by the intended users. In this respect, each of the 17 services was evaluated by older adults with cognitive impairment, including the services designed to provide Caregiving support. An important element of the IN LIFE evaluation was examining how the four groups of users felt about the IN LIFE platform as a whole. This included understanding each of the four groups’ attitudes towards who should provide IN LIFE and who should pay for it. On the questions relating to finance, only one-quarter of the older adults provided information about their annual household income. This suggests a reticence on the part of some and ignorance on the part of others, regarding financial decisions. This is important to consider when thinking about the customer base for IN LIFE and similar systems.

In respect of the evaluations of the four categories of services, older adults who responded felt that Independent Living support and Travel Support were useful. Although Socialisation and Communication support were the most evaluated services, two-thirds of older adults who used them did not answer any of the evaluation questions. This most likely reflects the majority of these older adults were living in care homes and had difficulty responding to questionnaires. The majority of informal caregivers thought all four categories of services were useful, and a substantial group felt the Travel Support in particular could improve health and wellbeing. This may reflect concerns they experience in respect of their family member with cognitive impairment travelling alone.

Health and social care staff also thought the Travel Support was useful and would improve health and wellbeing, perhaps for similar reasons to family caregivers. Two thirds of health and social care staff also thought Socialisation and Communication Support was useful, which could reflect the high number of care staff who evaluated CIRCA in care homes where they have limited digital services available. In respect of concerns about the privacy of their personal information, two thirds of older adults who evaluated Independent Living services were concerned, whereas three quarters of informal caregivers had no concerns. The majority of healthcare professionals who evaluated the Travel Support, Socialisation and Communication and Caregiver support services had no concerns about the privacy of their data. In spite of positive evaluations of the services, the majority of participants would not continue using the services if they were not free, with the exception of 70% of stakeholders who would keep using the Independent Living services. These findings have important implications for understand needs and practical barriers to scaling up findings from evaluation studies into real-world services.

Limitations of the evaluation include the short time frame for identifying, recruiting and implementing the IN LIFE platform with such a large number of people in such a wide range of settings. This meant that some services received fewer evaluations than anticipated, while others, mostly those that were pre-existing, received more. There was also no opportunity to set up processes to follow the impact of introducing the IN LIFE platform into the various networks of individuals and organisations over time.

A number of methodological challenges were encountered that should be borne in mind in future large-scale deployments. The first relates to the eHealth literacy of older adults with cognitive impairment and the need for an adaptive implementation strategy. While it can be predicted this need will shrink with future cohorts, the diversity of experience with digital technologies and the Internet in this sample across six European countries indicates that accessibility and availability will continue to be unequal for some time. Touch screen technology (in smartwatches and tablets) was unfamiliar to the majority of older adult participants and presented some level of challenge at the beginning, although they quickly became accustomed to them. Using these devices was hampered for some by physical limitations such as hand tremor, poor eyesight or hearing, which necessitated very thorough training in order for them to start using a new system confidently, with several repetitions in some cases. It was also noted that participants responded differently to different devices and services. For example, in the Slovenian site, the smartwatch was really well accepted in the segment of people living independently at home, while the e-doorman was found to be too complex and too specific. Lack of Internet in various care settings was also a real obstacle at several sites. Additionally, administering standardised instruments to such a wide range of participants with varying degrees of cognitive ability was found to be challenging. Not all of the older adults, particularly those residing in care homes, were able to complete these and as such there were missing data, especially in the evaluation questionnaires of care home services.

7 Lessons learnt

Technology implementation in the real-world is absolutely essential but is messy and complicated. Evaluating that implementation is equally complex and challenging. Adopting a pragmatic approach whereby each site selected services to evaluate and recruited the participants best suited to evaluate these services, permitted recruitment of a large cohort of participants and evaluation of each of the 17 services in the IN LIFE platform. Key to this success was pairing a user organisation with a research organisation at each site. The process of conducting the IN LIFE evaluation highlighted the multiple interactions between individuals, their home situation, relationships and the importance of understanding these. Such an approach is consistent with the “need [for] an evaluation approach capable of unpicking the real-life, emergent impact of technological interventions” (page 199, [14]). To achieve this requires (i) understanding of the network into which technologies are being introduced, (ii) capacity to identify the impact of introducing technology into this network and (iii) the ability to monitor how the network changes over time.