Developing a pragmatic evaluation of ICTs for older adults with cognitive impairment at scale: the IN LIFE experience

Implementing information and communications technology (ICT) at scale requires evaluation processes to capture the impact on users as well as the infrastructure into which it is being introduced. For older adults living with cognitive impairment, this requires evaluation that can accommodate different levels of cognitive impairment, alongside input from family and formal caregivers, plus stakeholder organisations. The European Horizon 2020 project INdependent LIving support Functions for the Elderly (IN LIFE) set out to integrate 17 technologies into a single digital platform for older people living with cognitive impairment plus their families, care providers and stakeholders. The IN LIFE evaluation took place across six national pilot sites to examine a number of variables including impact on the users, user acceptance of the individual services and the overall platform, plus the economic case for the IN LIFE platform. The results confirmed the interest and need among older adults, family caregivers, formal caregivers and stakeholders, for information and communications technology (ICT). Relative to the baseline, quality of life improved and cognition stabilised; however, there was an overall reluctance to pay for the platform. The findings provide insights into existing barriers and challenges for adoption of ICT for older people living with cognitive impairment.


Introduction
Globally, increasing life expectancy is leading to growing numbers of older people with some form of cognitive impairment [1]. The majority of older people with cognitive impairment want to remain at home and live independently for as long as possible [2]. This means supporting people to function within and outside the home with activities of daily living as well as socialisation, hobbies and travel. Over time, people living with progressive cognitive impairments, such as dementia, require more help, with families across the world providing the bulk of care [1]. However, for some older adults, their needs will increase to the point where they need 24-h support which for some will be provided in a care home.
Alongside this emerging societal challenge, technological development is advancing rapidly to offer potential solutions [3]. Across Europe, a growing number of Information and Communication Technology (ICT)-based services targeting older adults with cognitive impairment have been developed, but these have frequently comprised small feasibility or pilot studies. Integrating these various solutions and making them available at scale could provide a major solution to this growing societal need, by supporting older adults with cognitive impairment to remain independent for longer. Given that the majority of older people with cognitive impairment are cared for by their families, it is also important to gain an understanding of their views in relation to services and support. Additionally, the views of formal caregivers working in services for older adults with cognitive impairment are also vital for informing future strategies for care services.
The Independent Living Functions for Elderly (IN LIFE) project was established to integrate a collection of ICT services within an interoperable, open, personalised platform to empower older adults with cognitive impairment to maintain independence [4]. Recognising the breadth of activities older adults wish to carry out, the IN LIFE platform addresses a wide range of activities including those within the home, travel outside the home, communication, health maintenance, mobility and socialisation alongside caregiving. The completed IN LIFE platform comprises 17 services presented in four categories: Independent Living Support, Travel Support, Socialisation and Communication Support and Caregiving Support (Table 1 contains the full list of services and where they were evaluated). Ten of these services, online exercising (IL1; [5], daily functions assistant (IL2; [6]), falls detection (IL4: [7]), Guardian Angel (IL5; [8]), mental training (IL6; OASIS project), physical activity monitoring (IL7), car driving ability assessment (TS1; [9]), public transport support (My Routes; TS2: ASK-IT project), socialisation and communication support (SC1; CIRCA [10]) and teleconsultation (CS3; [11]), were established Table 1 IN LIFE services a evaluated at each pilot site Key: Greece (GR), The Netherlands (NL), Slovenia (SL), Spain (SP), Sweden (SW), United Kingdom (UK) a Caregivers' scheduling and reminding & Caregivers' monitoring and supervision portals are assessed together and referred to collectively as "Carer Support Portals", whereas the Interaction strategies tool documented in the DoA is an integrated part of Leisure support (also known as CIRCA) and Socialisation and communication support ( services developed and tested separately in previous projects. The other seven services were extensions of pilot or feasibility prototypes that were completed for the IN LIFE project. The breadth of services required an evaluation strategy that included participants with varying levels of cognitive ability who were residing in a range of living situations.

Evaluating IN LIFE
The IN LIFE evaluation strategy was informed by the need to assess each of the 17 services with the intended usersolder adults living with cognitive impairment, family caregivers and formal caregivers-as well as the overall IN LIFE platform. While most of the services were designed for people living at home to support their daily activities (e.g. My Routes and daily functions assistant), the Socialisation and Communication support services-CIRCA and CIR-CUS-were designed for promoting interactions between caregivers and people living with a wide range of cognitive impairments. These diverse services were included to address the breadth of activities and priorities of people living with cognitive impairment [12] and to evaluate the effectiveness of the platform when people experience more severe cognitive impairment. This created a need to identify evaluation methods to cover a broad range of services for users with different levels of cognitive challenges, in a range of environments. The measures also needed to be available in multiple languages and suitable for use at scale with the large number of participants required to provide evaluation of the 17 services and platform. The selection of outcome variables and appropriate measures for the target populations was informed by previous projects of the IN LIFE consortium and existing evaluation frameworks and tools. To capture the "intricate nuances" of the introduction of ICTs into the lives of older adults requires a multifaceted approach [13]. Gomersall and colleagues [14] proposed that network-based approaches, which acknowledge the networks of individuals and organisations into which technologies are being introduced, are a useful starting place for developing technology evaluations. The IN LIFE evaluation strategy identified the networks around older adults as including informal caregivers, healthcare and formal caregivers, and a range of stakeholder organisations that provide services to older adults. The IN LIFE evaluation strategy was developed to ensure that each of the 17 individual services was examined by the intended users-older adults, healthcare professionals or informal caregivers-and, where relevant, the interactions between them. Additionally, the evaluation was designed for these groups plus representatives of stakeholder organisations, including social services, hospitals, home care, assisted living and long-term care providers to evaluate the whole IN LIFE platform.
The number of tools, frameworks and methodologies for evaluating the effectiveness and impact of ICT has increased as the range of technological solutions has expanded. Many of the existing tools have been developed in the context of Assistive Technologies, which are devices and services, such as fall alarms or medication reminders, prescribed or provided by health or social care providers. A number of evaluation approaches focus on the assistive technology or services. For example, the 12-item Quebec User Evaluation and Satisfaction with Assistive Technology devices and services (QUEST 2.0; [15]) considers characteristics of the device (including size, safety) and the services (including delivery and professionalism). Similarly, the Post-Installation Technique [16] was developed for health or social care service providers to conduct a risk assessment for equipment introduced into the homes of older adults. This looks at equipment failure or other problems, the potential impact on end users and solutions for addressing these problems.
Other approaches consider variables relating to the user. For instance, the matching person and technology (MPT; [17]) model looks at three factors related to the personal motivation to use assistive technology: environmental factors, user needs and preferences, and functions and features of the appropriate available, technology. Similarly, the Everyday Technology Use Questionnaire (ETUQ; [18]) uses a detailed interview to explore an individual current or previous usage of everyday technologies (e.g. smartphones, automated banking) with an emphasis on the relevance of the technology to them.
In a detailed examination of the product development lifecycle of assistive devices based around the experience of individuals living with disabilities,  produced a three-part series covering the 1. Design [19]. 2. Evaluation [20] and 3. Outcomes [21]. She argued that evaluation needs to consider both the functioning of the technology and the impact on the user, throughout the whole process [20]. Hersh identified four factors to consider in evaluation: compliance and good practice (e.g. health and safety, ethical issues), technical issues (e.g. data security, reliability, robustness and dependability), end-user issues (e.g. usability, accessibility and acceptability) and resource and financial issues (e.g. direct and hidden costs, feasibility and costs of upgrading). In addition, the importance of understanding reasons for abandonment of technology is highlighted in Part 3 on outcomes of assistive devices [21].
Contemporaneously the European Intelligent System for Independent living and SElf-care of seniors with cognitive problems or Mild Dementia (ISISEMD) project produced a technology evaluation framework [22] focused on telemedicine. The ISISEMD framework identified quality of life of older adults and socio-economic impact on the health or social care systems as key outcome variables. Their assessment included user acceptance and satisfaction with the technology assessed through questionnaires and interviews and caregiver questions about willingness to pay for the services. Along with Hersh [21], the ISISEMD team [23] highlighted the importance and complexity of assessing the impact of a technology intervention on the quality of life of the user in context.
Building on these existing models, the IN LIFE evaluation examined the impact on the quality of life of individuals living with cognitive impairment in different settings, with different levels of cognitive impairment, different support systems and in different healthcare systems. The evaluation also captured user factors such as preferences, digital literacy, socio-economic status, affordability, plus information about usability and usefulness, technology accessibility and perceived usefulness for family caregivers, formal caregivers and stakeholder organisations. Specifically, the IN LIFE evaluation set out to (i) examine the impact on quality of life of older adults with cognitive impairment, (ii) examine the user experience and usability of the IN LIFE services and platform and (iii) assess the economic case for IN LIFE.

Design
The IN LIFE evaluation comprised an assessment of the situation prior to availability of the IN LIFE platform (baseline) and an assessment of the impact of the IN LIFE platform (pilot To ensure access to the four groups of identified participants, each pilot site included one user organisation as a partner, alongside a research organisation. The partner organisations included two acute hospitals (GR, SW), two long-term care organisations (NL, UK), one home care health provider (SL) and one ageing research organisation (SP).
Data were collected at the six pilot sites using a mixed methods approach. To maximise the evaluation, it was designed such that each of the 17 services was examined in two ways. First, each service was identified as a primary service in at least one country as detailed in Table 1. This was to ensure that each service in the IN LIFE platform underwent evaluation. In addition, each user was able to access each of the 17 services through the IN LIFE portal during their evaluation of the platform. Some of the services, such as the driving assessment, were single use, while others were designed to be used over time, e.g. Guardian Angel, Route Planning, Socialisation and Communication support (CIRCA). Each pilot site identified the service(s) that would be tested at their location (Table 1). To accommodate this variability between the 17 IN LIFE services, each pilot site developed a localised recruitment and testing plan, with common measures but different lengths of participant involvement. The evaluation took place between March 2016 and September 2017.

Participants
Four groups of participants were identified for IN LIFE: older adults living with cognitive impairment, informal caregivers, health and social care staff (healthcare professionals) and representatives of stakeholder organisations. Each national site had a target number of participants to recruit from each participant group (Table 2 shows anticipated and actual numbers of participants). Local processes were followed at each national site for securing ethical approval. Recruitment locations varied across the sites and included general hospital outpatients, older adult day centres, community programs, assisted living facilities, Alzheimer's societies, senior's groups and long-term care homes. Each site recruited representatives of each of the four participant groups (Table 2).

IN LIFE services
The 17 services in the IN LIFE platform were available for participants across the six pilot sites to evaluate (see Table 1). Details of each service are provided in Appendix 1.

Materials
The IN LIFE project integrated different technologies to the platform, so different devices were tested in each pilot, according to the needs of each person. These were mainly mobile devices, telephones, laptops and tablets, as well as wearable monitoring devices and specific devices for driving. Likewise, pen and paper solutions and computerised assessment protocols were used to carry out the assessments. Primary outcomes for older people with cognitive impairment included quality of life and cognitive function. The Swedish pilot site focused on assessing communication and social interaction and used measures designed to assess these functions. Each pilot site elected to use the same cognitive assessment as used by the local memory assessment services to ensure consistency. This resulted in four sites (Greece, Netherlands, Slovenia, Spain) using the Mini-Mental State Examination (MMSE: [24]) and one, (UK) using the Addenbrookes Cognitive Examination-3rd edition (ACE-III; [25]). The Swedish pilot site did not administer a cognitive measure to their older adult participant.

Mini-mental state examination [24]
The MMSE is a standardised measure that provides a brief assessment of cognitive state. It is particularly useful for patients with dementia as administration takes about ten minutes. Questions cover orientation to date and place, registration, attention and calculation, recall and language (including writing). Total score possible is 30, with scores below 27 indicating cognitive impairment.

Addenbrookes Cognitive Examination-3rd edition [ACE-III; [25]].
The ACE-III focuses on five cognitive areas-attention/orientation, memory, language, verbal fluency, visuospatial skills. It is scored out of 100, with a cut-off score for dementia between 82 and 88/100, where lower scores indicate greater cognitive impairment. The ACE-III requires training for accurate administration and scoring and takes approximately 15-20 min to administer and score. The ACE-III has high internal validity as measured by Cronbach's a coefficient (a = 0.88; [26]), along with high levels of sensitivity and validity ([25]).

Quality of life
The EQ-5D (5L version) [27]is a generic instrument for describing and valuing health-based quality of life. It is based on a descriptive system that defines health in terms of 5 dimensions: Mobility, Self-Care, Usual Activities, Pain/Discomfort and Anxiety/Depression. For each dimension respondents rate themselves as having 'no problems', 'slight problems', 'moderate problems', 'severe problems', or 'extreme problems'. In addition, respondents also rate their overall health on the day of the interview on a 0-100 hash-marked, vertical visual analogue scale (EQ-VAS). The EQ-5D has been widely tested and used in both the general population and patient samples and translated into over 130 different language versions (www. euroq ol. org). Analysis was performed according to the EQ-5D official guidelines (https:// euroq ol. org/ wp-conte nt/ uploa ds/ 2016/ 09/ EQ-5D-5L_UserGuide_2015.pdf).

Demographics questionnaire
This was designed to collect demographic and socio-economic data. This included questions about age, gender, education, current living arrangements, employment status, healthcare provision and total annual household income.

Technology familiarity
This questionnaire was developed to provide an assessment of technology usage and familiarity at entry into the project. Questions asked about access to a PC or laptop, tablet, smartphone and the Internet.

IN LIFE evaluation
This tool was completed after using the IN LIFE platform and included questions about ease of use, confidence in daily activities, independence, future use of the platform, if free of charge or not, how people would be prepared to pay for access to the services, and factors influencing their decisions. For the services aimed at caregivers, it is important that they are evaluated by both an older adult with cognitive impairment and at least one caregiver, formal or informal, to examine the usability and usefulness. If either party does not find a service easy to use or do not see a need, then it will not be adopted. Additional measures were used at each site-details can be found in the IN LIFE final report.

Procedure
Each pilot site identified the primary services it would evaluate in addition to the whole IN LIFE platform. The choice of services informed the recruitment strategies employed at each site to reach the target number of participants in each group. The choice of services also informed the selection of measures administered at each site (Appendix 3; Table 5).
The procedure followed at each pilot site is described in Appendix 2. The results of the measures conducted at each site are reported here. Full results from each site can be found in the IN LIFE final report, with additional publications of the detailed results from the UK site (Astell, Smith, Potter & Preston-Jones, 2018) and the Sweden site (Ferm, Ekstrom, Larsson & Samuelsson, 2020).

Demographic information
A total of 2090 participants were recruited across the six pilot sites over an 18-month period ( Table 2). As intended, the largest group of participants were older adults (N = 1205; Fig. 1), with the number recruited exceeding the target amount (Table 2). Additionally, more health and social care staff were recruited than anticipated but fewer family caregivers and stakeholders ( Table 2). The mean age of the participants with cognitive impairment who provided their age or date of birth (N = 1069) was 75.62 years (53-104), of whom 62% were female.
Nine hundred and fifty-six older adults provided information about where they live, which revealed 35% lived with a spouse or partner, 30% lived alone, 9% with family, and 25% in other settings. Of these 234 other settings, the largest proportion were residential care (55%), followed by retirement communities (21%) and sheltered housing (11%). Eight hundred and forty-five reported their highest educational level, with 23% completing higher education, 23% completing secondary education and 42% completing primary education. Regarding employment status, of the 892 older adults who responded, predictably the majority (88%) were retired from paid work. However, 10.4% of the remaining 12% were still working full-time, while the rest worked part-time. Six hundred and ninety-four older adults provided information about their healthcare provision with 84% funded by government, 5% private insurance, 6.8% self-pay and the rest (4.2%) some combination of these three. Finally, 317 older adults responded to the question about annual household income with the majority choosing not to answer or not knowing the answer. Those who did respond were distributed as follows: less than €5,000 = 30, €5,000-€19,999 = 217, €20,000-€49,999 = 68, €50,000-€99,999 = 2. In response to the technology use questionnaire, around 50% were using a mobile phone, of which 29% were smartphones and 15% had used a tablet before.
The 360 informal caregivers who provided their age or date of birth had a mean age of 59.4 years, 66% were female and 82% cared for a family member. Three-quarters of the informal caregivers have access to a personal computer and/ or laptop either at work or at home, and around 80% of all caregivers have access to the Internet. Three hundred and ninety-one health and social care staff provided their age or date of birth. Their average age was 42 years, and they were 95% female. They were working in a range of healthcare jobs including nursing, activity coordinator and personal support in public or private residential homes, and public hospitals. Forty-one percent of the healthcare professionals had more than 11 years of experience working with older adults with cognitive impairment, interacting with an average of 5 people per day. Staff reported what they like most about their current use of technology for their work is the ease of use, how work gets carried out easier and direct access to information from anywhere. What they really do not like is that in most cases no Internet exists at their workplace, especially within public hospitals, as well as the cost of applications and services, plus when access to the Internet is slow. They think that the interoperability of applications, medical image transfer technologies, free access and Internet existence at their workplace are still missing.
The majority of representatives of stakeholder organisations who evaluated IN LIFE services were female (58%), 46% were aged less than 60 years old and 57% responded that they currently use technology (although 38% did not answer this question). The stakeholders were involved with older adults with cognitive impairment mostly through their work and less through family and friends. The majority of the stakeholders had over 15 years of work experience, and their specialisms included psychologist, psychiatrist, physiotherapist, politician, government worker and IT. They believed the most important issues for this population were feelings of anxiety, insecurity, sadness, memory loss, lack of concentration and difficulties with orientation. What stakeholders liked most in the technologies they use at work is direct access to information they seek from anywhere. What they like least is that data could get lost, access to the Internet is usually slow, and they find it difficult to keep paper copies. What they believed is still missing is a country-wide patient database, the interoperability of applications, as well as new computer power at their workplace.

IN LIFE services
The evaluation strategy was designed for each relevant user group to evaluate the individual services and the combination of services available through the IN LIFE platform. The distribution of evaluations varied across the four themes (Fig. 2). Independent Living and Socialisation and Communication services accounted for almost 80% of the evaluations (Fig. 3).
Each service was evaluated by older adults with cognitive impairment ( Table 4). The impact of IN LIFE on the quality of life of older adults with cognitive impairment was examined with the EQ-5D, the health-related QoL index used across all sites that measured QoL. Initially, a data transformation was selected to include all data, but variance distribution increased, and it was decided to rely only on common instruments used at all sites in order to yield any inferences for the impact these analyses might have to European population. Consolidated EQ-5D data were used to calculate the overall QoL index of IN LIFE population. A nonparametric alternative was selected because normality assumption was highly violated (p = 0.005) (Fig. 4).
We examined the participant's scores on the EQ5D to look at the impact of using the IN LIFE platform. We found that the EQ5D scores declined in the baseline phase, although this was not significant (p > 0.05; see Table 3). In We also examined the cognitive scores of the participants in the baseline and pilot conditions. For comparison purposes, the scores on the ACE-III were converted to MMSE equivalents using the formula of Giebel and Challis (2017). A significant overall negative correlation was found between age (years) and MMSE (0-30 scores) (r = − 0.191, p < 0.001). Comparison of the MMSE scores over the baseline phase found a decline in scores (mean difference − 2.97), but this was not significant (p > 0.05; Table 3). MMSE scores did not change during the pilot phase (p > 0.05: Table 3) with a small mean difference (− 0.312) suggesting cognitive maintenance.
Health and social care staff provided just under one quarter of the evaluations with informal caregivers providing 12% (Table 4). In regard to the informal caregivers, it was noted that IN LIFE can improve their quality of life, help with communication and assist in monitoring their family member's everyday safety and activities, empowering their family member to feel more in control. It can also assist healthcare professionals by helping them clarify and better organise and maintain workload, communicate better with their patients and provide integrated care and continuous monitoring throughout their daily activities. Stakeholders examined the overall IN LIFE platform as well as some individual services. They believed the use of IN LIFE could improve quality of life and help maintain abilities and cognitive function of older adults with cognitive impairment. In addition, it could inspire older people to use new technologies.
In respect of the IN LIFE questionnaires, the majority of users considered the Independent Living support services as very useful or useful (59.33% of elderly, 73.63 of informal caregivers and 44.19% of healthcare professional), very reliable or reliable (51.87% of people with cognitive impairment and 53.84 of informal caregivers), were not concerned over their data privacy while using the services (37.92% of elderly users and 71.43% of informal caregivers) and a high proportion felt the use of the services improved their overall health and wellbeing (Table 4). Despite these ratings, older adults and informal caregivers do not feel prepared to pay for the services.
A high percentage of informal and formal caregivers considered the travel support services as very useful or useful ( > 76% and > 83% respectively). Informal caregivers had, however, no concern while using the services, considered it an improvement on their overall health and wellbeing while caring for their older relative with cognitive impairment, but would still not pay for using the services (Table 4). Regarding the car driving ability assessment, the specific IN LIFE service is targeted towards healthcare professionals. However, the driving assessment results (from the 156 older drivers) were also used to improve and refine the existing thresholds of the driving assessment instruments.
The Socialisation and Communication support modules were the most accessed services. Reliability of the Socialisation and Communication support modules was not included in the questionnaire administered to the users, but the majority of caregivers did, however, consider the services very useful or useful (54.54% of informal and 63.36% of formal caregivers).
The carers' support services received a high rating on usefulness and reliability (reliability was only assessed by the elderly users) by all users participating in the pilots. Nearly half of informal caregivers, however, felt concerned over the privacy of the personal data while using the services and consequently we have 64.74% of them stating there was no effect of improved overall health and wellbeing.

Discussion
A pragmatic evaluation strategy succeeded in the recruitment of a large number of participants across six sites to evaluate the IN LIFE platform. This was in part attributable to the pairing at each pilot site of a user organisation and a research organisation. The partner organisations included two acute hospitals, two long-term care organisations, one home care health provider and one ageing research organisation. This range of user organisations ensured access to a wide range of older adults living with cognitive impairment, from those attending outpatient services and memory assessment clinics to people living in supported housing and care homes. This breadth of participants (in both cognitive ability and living situation) was sought to provide realistic evaluation of the range of services, in the IN LIFE platform which had different intended users.
Guided by previous approaches to evaluation, e.g. [21,28], the IN LIFE evaluation aimed to examine the impact on quality of life of older adults with cognitive impairment and the user experience and usability of the system. In respect of the quality of life of older adults with cognitive impairment, we found a significant impact in the pilot phase relative to the baseline phase. The results indicated that ICT services targeting older adults with cognitive impairment should be developed with consideration for cognitive level of functioning (and level of deterioration), users' level of operation of 1 3 digital technologies (use, ease of use and frequency of use) as well as educational level. This is an important consideration when planning out multicomponent or long-term technological services for people living with cognitive impairment which is expected to change over time.
Following the recommendations of [17,20], a large part of the evaluation was concerned with the user experience and usability of the individual services as well as the overall platform. The evaluation strategy was designed to ensure that each service was at least examined by the intended users. In this respect, each of the 17 services was evaluated by older adults with cognitive impairment, including the services designed to provide Caregiving support. An important element of the IN LIFE evaluation was examining how the four groups of users felt about the IN LIFE platform as a whole. This included understanding each of the four groups' attitudes towards who should provide IN LIFE and who should pay for it. On the questions relating to finance, only one-quarter of the older adults provided information about their annual household income. This suggests a reticence on the part of some and ignorance on the part of others, regarding financial decisions. This is important to consider when thinking about the customer base for IN LIFE and similar systems.
In respect of the evaluations of the four categories of services, older adults who responded felt that Independent Living support and Travel Support were useful. Although Socialisation and Communication support were the most evaluated services, two-thirds of older adults who used them did not answer any of the evaluation questions. This most likely reflects the majority of these older adults were living in care homes and had difficulty responding to questionnaires. The majority of informal caregivers thought all four categories of services were useful, and a substantial group felt the Travel Support in particular could improve health and wellbeing. This may reflect concerns they experience in respect of their family member with cognitive impairment travelling alone.
Health and social care staff also thought the Travel Support was useful and would improve health and wellbeing, perhaps for similar reasons to family caregivers. Two thirds of health and social care staff also thought Socialisation and Communication Support was useful, which could reflect the high number of care staff who evaluated CIRCA in care homes where they have limited digital services available. In respect of concerns about the privacy of their personal information, two thirds of older adults who evaluated Independent Living services were concerned, whereas three quarters of informal caregivers had no concerns. The majority of healthcare professionals who evaluated the Travel Support, Socialisation and Communication and Caregiver support services had no concerns about the privacy of their data. In spite of positive evaluations of the services, the majority of participants would not continue using the services if they were not free, with the exception of 70% of stakeholders who would keep using the Independent Living services. These findings have important implications for understand needs and practical barriers to scaling up findings from evaluation studies into real-world services.
Limitations of the evaluation include the short time frame for identifying, recruiting and implementing the IN LIFE platform with such a large number of people in such a wide range of settings. This meant that some services received fewer evaluations than anticipated, while others, mostly those that were pre-existing, received more. There was also no opportunity to set up processes to follow the impact of introducing the IN LIFE platform into the various networks of individuals and organisations over time.
A number of methodological challenges were encountered that should be borne in mind in future large-scale deployments. The first relates to the eHealth literacy of older adults with cognitive impairment and the need for an adaptive implementation strategy. While it can be predicted this need will shrink with future cohorts, the diversity of experience with digital technologies and the Internet in this sample across six European countries indicates that accessibility and availability will continue to be unequal for some time. Touch screen technology (in smartwatches and tablets) was unfamiliar to the majority of older adult participants and presented some level of challenge at the beginning, although they quickly became accustomed to them. Using these devices was hampered for some by physical limitations such as hand tremor, poor eyesight or hearing, which necessitated very thorough training in order for them to start using a new system confidently, with several repetitions in some cases. It was also noted that participants responded differently to different devices and services. For example, in the Slovenian site, the smartwatch was really well accepted in the segment of people living independently at home, while the e-doorman was found to be too complex and too specific. Lack of Internet in various care settings was also a real obstacle at several sites. Additionally, administering standardised instruments to such a wide range of participants with varying degrees of cognitive ability was found to be challenging. Not all of the older adults, particularly those residing in care homes, were able to complete these and as such there were missing data, especially in the evaluation questionnaires of care home services.

Lessons learnt
Technology implementation in the real-world is absolutely essential but is messy and complicated. Evaluating that implementation is equally complex and challenging. Adopting a pragmatic approach whereby each site selected services to evaluate and recruited the participants best suited to evaluate these services, permitted recruitment of a large cohort of participants and evaluation of each of the 17 services in the IN LIFE platform. Key to this success was pairing a user organisation with a research organisation at each site. The process of conducting the IN LIFE evaluation highlighted the multiple interactions between individuals, their home situation, relationships and the importance of understanding these. Such an approach is consistent with the "need [for] an evaluation approach capable of unpicking the real-life, emergent impact of technological interventions" (page 199, [14]). To achieve this requires (i) understanding of the network into which technologies are being introduced, (ii) capacity to identify the impact of introducing technology into this network and (iii) the ability to monitor how the network changes over time.

IL1 Online exercising
An online training module [29] which enables older adults to train independently in their home environment, in three different categories: strength, balance and flexibility. The online exercise scheme was issued by the therapist for each individual participant, and the service is designed to raise awareness and motivation for physical activity. Adaptations for older adults with cognitive impairment include light physical exercises adapted to various older user groups. The service was first offered in a group setting where the clients can start using the exercise programme under supervision of a formal caregiver. After a while, the client can decide to continue using it independently at home when they feel sufficiently familiar with the online exercise program. Offered in IN LIFE to users also outside of the Netherlands.

IL2 Daily functions assistant
Developed as an augmentative and alternative communication (AAC) means for pre-programming daily functions using pictograms. Extensions made specifically for older adults with cognitive impairment include controlling home appliances, management of daily agenda, reminders, alarms, timers and personalisation of messages.

IL3 E-doorman (home security)
Tracking entry and exits from home and detection of safety breaches. Set to contact a predefined person in case of emergency. Intended to adapt to user's habits and profile and interface with navigation and routing tool. Connected to caregiver's interface (CS1) through caregiver's mobile device.

IL4 Fall detection and behavioural monitoring (smart watch)
Wearable designed to notify a carer if the user falls. Connected to caregiver's interface (CS1) through caregiver's mobile device.

IL5 Guardian angel
Guardian Angel is a tele-monitoring service that consist of a mobile application for monitoring users on the move throughout their daily activities and a web-based review and analysis application for a meaningful presentation of the monitored data by the respective caregivers. Guardian Angel provides almost real-time monitoring of vital signs (e.g. heart rate), personalised health indicators (daily steps, hours of sleep), as well as alerts and warnings to the user (and/or to the informal caregiver) enriched with the user location. The service currently supports a wrist wearable device.

IL6 Mental training
Mental training comprises cognitive exercises, based on frequent functions associated with activities of daily living the user needs to remember. These include a memory and attention game that uses spot-the-differences with objects used in daily activities, e.g. utensils, house, dressing, etc. Extensions were made to the existing suite of games to offer personalised games that could follow older adult's evolving cognitive state.

IL7 Physical activity monitoring
Physical activity monitoring is a rule-based module that complements the Guardian Angel module. In addition to providing almost real-time monitoring of vital signs (heart rate, blood pressure, spO2), and personalised health indicators (daily steps, hours of sleep), it also incorporates per case assessment tools (standardised questionnaires for COPD, depression etc.). Through a smart decision support system, it provides alerts, warnings and motivation messages to the user (and/or to the informal caregiver) enriched with current user location while it assists the formal caregiver with more detailed and information regarding the changes or identified risks regarding the health status of the patient. The service currently supports a wrist wearable device plus Bluetooth portable devices for spO2 and blood pressure measurement.

3
Travel support (TS) TS1 Car driving ability assessment AGILE Questionnaire-based pre-screening tool combined with on the road driving ability screening test. Based on extended TRIP protocol criteria, to assess the driving ability of an older person. This was extended IN LIFE to include appropriate aids and training courses for older adults with cognitive impairment, correlated with specific driving behaviour problems.

S2 Public transport support (My Routes)
Public transport designed for older and disabled citizens. MY ROUTES is an on-route assistant composed of a Webbased interface plus an Android mobile application for guiding persons with reduced mobility due to cognitive impairments related to age, illnesses, or other accessibility issue such as not speaking the local language, while travelling by public transport, specifically inside the Madrid bus network [30].

TS3 Trip planning and routing support (MLS Destinator)
Route planning and guidance navigator with synthetic voice capabilities for pedestrian and car driver support. Emergency button operated by user with simplified and optimised user interface for older people with cognitive impairments. Extensions to the existing service include implementation of personalised and intuitive macros for content and functions grouping plus find and rescue localisation operation by a nominated caregiver. Capacity for automatic emergency detection.

SC1 Socialisation and communication support (CIRCA)
Multimedia conversation support for older adults with cognitive impairment and caregivers. Photographs, music and short video clips provide a shared activity that can be carried out in individual's homes or in care homes. IN LIFE adaptations include tools for authoring national, cultural and linguistic CIRCA experiences.

SC2 Multilingual and multicultural support (MMS)
Language technology and ontology tools available and clustered into various libraries. Free multilingual and multimodal lexical resources and software for communication and language support in interactions with people with cognitive impairment.

SC3 Leisure support (CIRCUS)
CIRCUS has a personalised content with pictures and videos related to a specific individual's or group's past and current everyday life. CIRCUS has a simple uploading function, and users can upload any kind of material they choose with labelling in their preferred language.

CS1 Carer support (Caregiving monitoring & supervision and Caregiver scheduling and reminding)
Part of the e-Doorman system (IL3), to track entries to and exits from care-recipient's home. Set up to contact a predefined person in case of emergency. IN LIFE extension to enable recording of formal caregiver visits, detecting and reporting formal carer inability to provide assistance on time, and contacting informal caregivers for help in case formal caregiver services cannot be provided. Scheduling and reminding functions provide personalised notifications to the user giving the ability and support to older users and caregivers to perform personalised scheduling. System can manage the combined activities of the older adults and their caregivers.

CS2 Health monitoring application (HELMA)
A health monitoring application [5] to monitor the health and wellbeing of people with cognitive impairments in the long term by using short frequent online monitoring questions, which can be filled in by (1) The older adults; (2) Informal caregivers and (3) Formal caregivers. A decision tree was implemented to avoid overloading the person with cognitive impairment with questions each time they accessed the service. HELMA informs caregivers about the following health and wellbeing domains: physical; mental; social; and environmental. Herewith, HELMA aims to give peace of mind to the informal caregiver and improve the quality of life for both the informal caregiver and the person with cognitive impairment. HELMA is an extension of the existing health monitoring module in an ICT-supported rehabilitation service [5].

CS3 Teleconsultation
A teleconsultation service which connects older adults remotely with (in) formal caregivers using video calls to communicate to each other visually, an update of the existing ICT-supported rehabilitation service [5].

CS4 Virtual gaming
Environment for training courses. IN LIFE adaptations to meet the needs of older adults with cognitive impairments as well as their virtual interaction with formal and informal carers.

Quality of life in dementia [QoL-AD, [31]]
The QOL-AD is a brief 13-item measure, developed for individuals with dementia and caregivers to complete. QoL-AD assesses relationships with friends and family, concerns about financial matters, physical condition, mood and an overall assessment of quality of life. The measure is administered to people with dementia in interview format, lasting 10-15 min, assessing their own Quality of Life. It also has an informant component completed in questionnaire format by the caregivers about their relative's Quality of Life, which requires about 5 min to complete. Each item is scored on a four-point scale giving a total between 13 and 52 where the higher the score, the better quality of life the participant has. The QOL-AD has been shown to have excellent internal consistency for both patient and caregiver reports (α = 0.84 and 0.86, respectively) at all levels of cognitive functioning and good validity as indicated by correlations with measures of depression, day-to-day functioning and pleasant events frequency ([31]). Thorgrimsen and colleagues [32] reported the QOL-AD to have good content validity, construct validity, interrater reliability (all Cohen's kappa values > 0.70), test-retest reliability and internal consistency (Cronbach α coefficient of 0.82). QOL-AD is recommended by the European consensus as an outcome measure for psychosocial interventions in dementia [33].

Index of independence in activities of daily living (Katz Index of ADL; [34]
The Katz Index of ADL is an informant measure covering six areas of functioning: bathing, dressing, going to the toilet, transferring (chair, bed, etc.), continence and eating. Respondents select one of three options in relation to each function, e.g. Transfer: 'moves in and out of bed without assistance', 'moves with assistance', or 'doesn't get out of bed'. Level of independence is rated from A-G, where G is 'dependent in all functions'.

Instrumental activities of daily living scale [35]
This is an informant report measure assessing eight domains of daily activity that are judged more complex than the basic activities assessed by the Katz Index of ADL: using the telephone, shopping, food preparation, housekeeping, laundry, transport, medications and finance. Respondents assign '1' or '0' and the individual is scored according to their highest level of functioning in each category. Scores range from 0 low function, dependent) to 8 (high function, independent).

The Barthel ADL index, [36]
The Barthel index, originally developed for people who have had a stroke, is a widely used measure of ability based on ADL. It consists of ten items about bathing, transfer, dressing, eating, mobility, stairs, toilet use (including incontinence) and grooming. Total scores range from 0 to 20, with higher scores indicating greater independence.

SF12
The SF-12 is a multipurpose short-form survey with 12 questions, all selected from the SF-36 Health Survey [37]. The SF-12 is a self-report measure that assesses healthrelated quality of life in eight domains of physical functioning: physical: bodily pain, general health, energy/fatigue, social functioning; emotional and mental health. The 12 questions comprise one question on general health scored from 'excellent' to 'poor' plus 11 questions examining how much various factors including pain and emotional distress have interfered with work and daily activities. Physical and Mental Health Composite Scores (PCS & MCS) are computed using the scores of twelve questions and range from 0 to 100, where a zero score indicates the lowest level of health measured by the scales and 100 indicates the highest level of health.

Positive and negative affect scale (PANAS; [38])
The PANAS is a 20-item self-report scale measuring positive and negative affect. The scale comprises ten items measuring positive affect, such as 'excited', 'proud', and ten measuring negative affect, including 'hostile' and 'ashamed'.
Respondents indicate to what extent they feel right now or over the past week on a five-point scale where 1 is 'slightly or not at all' and 5 is 'extremely'. Scores on both scales range between 10 and 50. The PANAS has been shown to be a reliable and valid measure in a large sample [39].

Shortened Zarit Burden interview (SZBI; [40])
The Shortened ZBI is a 12-item self-report measure for caregivers to complete in relation to their experience of burden as a caregiver of a person with cognitive impairment. Items cover time for self, stress, anger, strain, guilt, impact on relationships, own health, social life, etc. Respondents are asked to rate how often they feel each of the 12 items using a fivepoint scale from 'never' (0) to 'nearly always' (4). Higher scores indicate greater experience of burden.

Quality of carer-patient relationship (QCPR, [41])
The QCPR is a semi-structured questionnaire completed by caregivers, which measures the quality of the relationship between the caregiver and the individual with dementia. The QCPR has two dimensions, one measuring the level of criticism and one measuring the level of warmth. QCPR can be administered either as a face-to-face interview or as a self-completion questionnaire and each item is scored on a 5-point Likert scale, with a good internal consistency (Cronbach's alpha = 0.82; [42]). The QCPR comprises six negative items on the criticism dimension (/30) and eight on the warmth dimension (/40) giving a total score out of 70.

Greece
The Greek site focused on examining the impact of IN LIFE service on independence and travel. Five services were selected: three Independent Living Support services-Guardian Angel (IL5), Mental training (IL6), Physical Activity Monitoring (IL7) and two Travel Support: Car Driving Ability assessment (TS1) and Trip Planning and Routing (TS3). Recruitment at the Greek site was conducted at a general hospital and a connected psychiatric hospital where people were attending for cognitive assessments, led by a team at the hospital. This strategy was adopted to identify participants who were still driving and planning travel and most likely to be able to learn to use the technology to support independent living. A total of 275 older adults were recruited, plus 92 informal caregivers, 47, health and social care staff and five stakeholder representatives ( Table 2). All older adults recruited to evaluate the Guardian Angel, Physical Activity Monitoring, MLS Destinator and Mental Training IN LIFE tools, received an Android based tablet PC and a short-term Internet connection (if they did not have one already). In addition, participants recruited to assess the Guardian Angel received a wrist wearable device to monitor personalised health indicators (heart rate, daily steps and hours of sleep). Participants assessing the Physical Activity Monitoring were also given Bluetooth portable devices for blood pressure and spO2 measurement. In the baseline phase, the Car Driving Assessment was completed at the Greek research institute and in the pilot phase via the IN LIFE platform. Additional measures completed by the older adults examined mood and activities of daily living (Table 5), while caregivers completed PANAS and the Shortened Zarit Burden Inventory.

Netherlands
The primary aim of the Netherlands site was to evaluate services to support independent living of older people with cognitive impairment in their own home. These comprised Online exercising (IL1), Socialisation and Communication support (CIRCA; SC1), Health Monitoring Application (HELMA; CS2) and Teleconsultation (CS3). Two hundred and one older adult participants were recruited through a healthcare organisation specialising in older people living with physical and/or cognitive impairments. (Table 2). In addition, 20 informal caregivers, 19 health and social care staff and five stakeholders were recruited. In the Netherlands, the partner user organisation was responsible for identification and recruitment of the participants. The online exercising was offered in two ways: (1) as part of day care where it was offered by activity counsellors. The participants could exercise in a group with other clients under the guidance of the activity counsellor to get acquainted with the technology; (2) at home once familiar with the service. The HELMA service is a program that contains a questionnaire about general health, physical, mental, social and environmental aspects of the older adult completed at least once a week by the formal caregiver and on a daily basis by the informal caregiver and older adult to monitor health status. HELMA is web-based; therefore, it is accessible through various devices: PC, laptop, tablet and smartphone. Older adults were able to complete the questionnaire together with QoL-AD EQ-5D UK ACE-III QoL-AD EQ-5D a caregiver or elect for the caregiver to complete the questionnaire. The Teleconsultation service connects older adults remotely with (in) formal caregivers using video calls. The CIRCA system provides a shared social activity that was implemented in two ways: (1) as part of day care offered by activity counsellors to a group of participants and (2) at home where formal or informal caregivers used CIRCA at home to have a one-to-one conversation. Additional measures were used at the Dutch site to assess activities of daily living (Table 5).

Slovenia
The main focus of the Slovenian pilot site was to increase the feeling of safety, higher independent living and to prevent complications of falling. The services tested were: E-Doorman (IL3), Fall detection and behavioural monitoring (smart watch: IL4) and Caregiver Support interfaces (CS1). One hundred and fifty older adults drawn from three populations were recruited: (a) living at home independently, (b) living in sheltered housing and (c) living in residential institutions. Additionally, 100 informal and 10 formal caregivers plus 6 relevant stakeholders were included. At the Slovenian site, the user partner organisation lead the recruitment of all participants. In addition to measures of cognition, quality of life and general health completed by the older adults, they also provided demographic information and information about familiarity, accessibility and usage of technology (Table 5). These latter two IN LIFE questionnaires were also completed by the informal caregivers and health and social care staff. Older adults were provided with a smart watch and tablet to use at home in partnership with a caregiver.

Spain
The Spanish site tested the widest range of services with at least one from each of the four themes: Web-Based physical exercises (IL1), Daily Functions Assistant (IL2), Public Transport support (TS2), Socialisation and Communication support (CIRCA: SC1) and Virtual Gaming (CS3). To recruit a range of older adults with differing degrees of cognitive impairment who might benefit from the different services, a broad recruitment strategy was adopted. One hundred and ninety-seven older adults were recruited from gerontology centres, patients and retired associations, Cruz Roja (Red Cross), nursing homes and other older adult communities (Table 2). In addition, 101 informal caregivers, 32 health and social care staff and 10 stakeholders were also recruited. In keeping with the range of services being evaluated, a number of additional measures were used assessing activities of daily living, mood and questions about Mobility, Communication, Leisure, Safety and Physical Exercise ( Table 5). As not all of the services being tested were designed for all profiles, the specific package evaluated by each participant was agreed between the individuals, their families and the interviewer (based on the users ́ health and motivation).

Sweden
The Swedish site focused on communication (SC1), socialisation (SC2) and leisure activities (SC3). Participants were recruited in groups of 8-10 users with cognitive impairments and corresponding caregivers with a testing period for approximately 2-3 months. Older adult participants were recruited from care homes and day centres and the health and social care participants were staff in these facilities. A total of 168 older adults, 253 health and social care staff, 11 informal caregivers and 8 stakeholder representatives were recruited. Data were collected using IN LIFE demographic surveys, a 7-item questionnaire specifically about communication, interviews, participant observations and video recordings. Demographic information and baseline questionnaires were administered before participants used the services for between 4 and 12 weeks, after which the post-test questionnaires were administered (Table 5).

United Kingdom
The UK site focused on communication and socialisation (SC1). The evaluation comprised two components: Group activity sessions in care homes and dyadic interactions in people's homes. The baseline utilised the existing CIRCA, a preloaded touchscreen software application to support communication between people with dementia and caregivers. The pilot phase utilised CIRCA-WB, a web-based version accessed through the IN LIFE platform. The ACE-III, QoL-AD and EQ-5D were administered to the older adults before using CIRCA, after using CIRCA and three-months later [43]. Caregivers in the dyads using CIRCA at home were asked to complete the caregiver section of the QoL-AD for their family member, as well as the QCPR, ZBI and the EQ-5D of their own health (9N).
otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.