MobileCogniTracker

The World Health Organization has recently concluded that circa 15% of adults over 60 suffer from a cognitive disorder (World Health Organization 2016) and 47.5 million people are affected by dementia, with 7.7 million new cases every year (World Health Organization 2017). Mild cognitive impairment (MCI) is an intermediate stage between the expected cognitive decline of normal ageing and the more-serious disorder of dementia. MCI is quite a relevant stage as it involves cognitive changes that are grave enough to be noticed by the person experiencing them or to related people, but not severe enough to interfere with daily life activities or independent function. As a consequence, most applications for treatment and diagnosis target MCI or an early stage of dementia since first symptoms can generally be more easily spotted and treatment in these cases can often reduce the speed of cognitive decline, thus allowing patients to retain control over their lives as long as possible (Spitzer and Williams 1998).

According to (Bermúdez 2014), “the guiding idea of cognitive science is that mental operations involve processing information, and hence that we can study how the mind works by studying how information is processed”. Thus, the analysis of the information processing can be used to identify anomalies in the regular cognitive functioning of an individual, for example, by observing that it takes them longer than usual to think of a word or to recall a person’s name. This principle is leveraged in cognitive screening tests such as the Mini Mental State Examination (MMSE), a popular questionnaire-type test extensively used in clinical and research settings to diagnose cognitive disorders based on the observation of an individual’s mental performance. The original MMSE, as defined by ( Folstein et al. 1975), consists of a number of tasks including questions and problems addressing the time and place of the test, repeating lists of words, arithmetic operations, language use and comprehension, and basic motor skills. The test starts with the orientation section, awarded with a maximum score of ten points, which assesses spatial and temporal orientation. Here the individual should correctly identify the current time (year, season, date, day, and month) and location (state, county, town, hospital, and floor). This section is followed by a registration task, where subjects are asked to repeat the name of three unrelated objects, for a maximum score of three points. Thereafter, attention and calculation are assessed by asking participants to subtract seven from hundred five times, or alternatively to spell the word “world” backwards for a maximum of five points. Subjects are then asked to recall the three objects introduced during the registration section, for a maximum of three points. Next, the individual is asked to name two given objects or to write a sentence of their choosing to assess language aspects, which is scored with up to two points. This task is followed by a repetition section where subjects are first requested to read and follow a specific command. Finally, some complex commands are asked to the user, for example, copying and drawing two intersecting pentagons, for a total of nine points. Based on the answers given to the test, the examiner can in principle assess the level of consciousness of the subject from alert to coma (Folstein et al. 1975). The maximum achievable score is 30 points, but a score of 24 or higher indicates normal cognitive functioning (Copeland et al. 2002; Folstein et al. 1983). Below this, scores can indicate severe (\(\le 9\) points), moderate (10–18 points) or mild (19–23 points) cognitive impairment (Mungas 1991).

Cognitive tests like the MMSE are typically conducted in the presence of a specialist. The role of the specialist, prior and during the test, normally limits to explaining the structure of the test to the participant and guiding them through the examination. Processing and assessment of the answers to the test is performed posteriorly. These tests require no specialized equipment or training for administration, thus making them easy to use (Harrell et al. 2000). However, the need for a person to introduce the test and collect the answers fairly constrains the frequency of administration of these tests. As a consequence, cognitive assessments are performed a few points in time and mostly upon clinical prescription when first symptoms appear. Moreover, the presence of an expert can affect the normal cognitive responses of the individual, hence introducing some level of bias in the results. In the light of these limitations, we propose a new tool to support the realisation of cognitive tests in an immediate, continuous and ubiquitous manner. The tool, recently introduced in (Wohlfahrt-Laymann et al. 2018), implements digital cognitive tests in the form of a multimedia experience sampling method (ESM), which can run on a smartphone and can be configured and scheduled remotely. The tool further integrates the digital cognitive experience sampling with multiple smartphone sensor data streams that can be used to study the interplay of cognition and physical, social and emotional behaviours. All the collected data is securely communicated via the internet and made available to the expert through a server for its post-processing and analysis, thus avoiding the need for the individual to visit the clinician’s office or any dedicated facilities.

Diverse digital mobile tools have been proposed in the past to measure and treat cognitive disorders. A major part of these tools refers to commercial apps aimed at assessing and treating cognitive disorders through so-called “brain games”. For example, Sea Hero Quest (T-mobile 2017) is a mobile serious game that assesses cognitive functioning by analysing the user’s spatial navigation skills. The player is invited to navigate a ship within the game world through sea mazes, direct flares and photograph sea monsters challenging memory use, spatial recognition, and orientation (Aškić et al. 2016; Morgan 2016). Lumosity (Lumos Labs 2017) is an app consisting of several mobile games assessing the functioning of different cognitive abilities. The app elaborates on various developmental aspects, namely targeting (specific cognitive abilities involved in daily tasks), adaptivity (tasks of varying difficulty), novelty (non-over-learned exercises to drive nervous system remodelling), engagement (positive encouragement to stimulate brain learning and processing), and completeness (full spectrum of cognitive abilities used in daily activities). Some of the games implemented by Lumosity include tracking and remembering different fishes moving across the screen to train visual attention and working memory, comparing visual and verbal information from familiar faces to exercise associative memory, or identifying hidden rules in a card game to train mental flexibility and working memory (Hardy and Scanlon 2009). MindMate (NHS 2017) is another mobile gaming application that aims to empower not only people with dementia but also families and carers. This app provides a set of interactive games to stimulate the user’s cognitive abilities through problem-solving and memory-training activities. MindMate also explores the use of self-reporting methods to keep track of some daily activities. Tapbrain (Kang et al. 2016) is a serious game consisting of thirteen mini-games to stimulate brain exercise and four mini-games to induce physical activity. The objective of Tapbrain is to stimulate cognitive brain functions by targeting triggers that stimulate the brain and also that induce physical movements. Although most of these applications claim to help improve some cognitive abilities, through the general assumption that mental exercising can improve cognitive functioning, only a few are backed by clinical evidence. As a result, some of these apps have been shown to be deceptive and thus involved in some legal issues (Federal Trade Commission 2016).

Smartphone passive sensing has been explored in research to measure daily behaviours, which could in principle give insight into cognitive disorders. For example, various works have exploited the smartphone’s inertial sensors, GPS and microphones for detecting indoor and outdoor physical activities (Ouchi and Doi 2012; Banos et al. 2015; Hur et al. 2017), which may relate to the cognitive state of a person (Hayes et al. 2008; Hagler et al. 2010). In a similar fashion, Bluetooth scans, photo captures and ambient audio recordings are used to measure levels of sociability (Lane et al. 2011; Vu et al. 2015), that may be linked to cognitive functioning (Akl et al. 2016). Some sophisticated approaches even measure physiological parameters such as heart rate and breathing rate through the smartphone’s accelerometer using ballistocardiography (Hernandez et al. 2015), which may be used to analyse cognitive stress (McDuff et al. 2016). The straight application of smartphones for the measurement of cognitive functioning is, however, a fairly uncharted area. There exist very few studies and they all mainly focus on the measurement of attention. In Brown et al. (2014) the authors recorded mobile phone usage including messages, social media and internet navigation for fifteen users during approximately three months. The analysis of this data allowed them to identify potential fill or kill times or even breaks, normally related to boredom situations. Mobile phone interaction (e.g., amount and types of apps used), context (e.g., light levels) and demographics are also combined in Pielot et al. (2015) with machine learning techniques to automatically spot these boredom situations. The results of this work demonstrate that the recency of communication, usage intensity, time of day, and demographics are the best categories of features to fairly identify situations where attention is scarce. Subjective and objective smartphone-based assessments of alertness and fatigue are compared to the influence of chronotype and time-of-day performance in Abdullah et al. (2016). Subjective assessments include questions on alertness, fatigue, as well as recent activities influencing levels of alertness and fatigue, such as the consumption of caffeine, exercising, or napping. Alertness is measured on the smartphones using an independent application implementing the so-called “Psychomotor Vigilance Task”. This work shows that alertness can oscillate approximately 30% depending on time and circadian rhythms. The authors also showed that daylight saving time, hours slept, and stimulant intake can influence alertness. When more alert, participants checked their phones more frequently but for shorter lengths of time, while during low alertness, participants engaged in more sustained use.

Despite the relevant progress made by above and related works, current unobtrusive measurements focus on a single cognitive ability and are thus not able to provide a comprehensive overview of the cognitive state comparable to clinical tests. Hence, these techniques cannot be used in isolation yet for diagnosing cognitive impairment. It is also clear that traditional clinical tests do not provide a way of continuous assessment of cognitive state in the way this could be achieved by monitoring sensor data through unobtrusive measurements. Therefore, a technical approach combining clinical tests with unobtrusive measurements of behaviour is considered to be necessary in order to find and validate new measurements for more reliable cognitive assessment.

Developing new cognitive assessment methods based on unobtrusive mobile sensing data requires reliable digital tests exploiting the diagnosis capabilities of well-established clinical procedures. In that vein, we propose MobileCogniTracker, a digital tool that complements and extends the potential of existing mobile passive sensing platforms for the measurement of people’s cognitive functioning. MobileCogniTracker develops an innovative experience sampling tool that helps automate and objectify the measurement of clinical-grade cognitive data. In the following, we describe the requirements, design choices and implementation of the proposed mobile cognitive tracking tool.

3.2 MobileCogniTracker architecture

The architecture technical diagram of MobileCogniTracker is shown in Fig. 1. The mobile device is the core entity, which communicates with the other main entities, i.e., the user and the server (expert). The expert sets, through the server, the study properties both in terms of tests to be realised by the user and the schedule for their administration. These properties are stored in a configuration file that is then communicated to the mobile application, which automatically updates the local configuration as to ensure proper operation in the absence of internet connection. At the scheduled time, the app pushes a notification awaiting for the reaction of the user. Once the user clicks on the notification, the corresponding cognitive test, i.e. question(s) and/or task(s), is prompted to the user for its realisation. Questions and tasks can be read on the screen or spoken out for the user convenience through text-to-speech functionalities natively supported by the mobile operating system. The user’s answers, which can come in different modalities, namely text, voice and drawings, are stored temporarily on the mobile device. This, and possibly other mobile sensor data, is periodically synced with the server as to make it available to the experts for further analysis.

Open image in new window

3.3 Cognitive experience sampling methods

This section presents the set of experience sampling methods we have developed for the realisation and collection of the cognitive tests. According to the requirements identified above, different methods for giving instructions and capturing answers must be provided. Plain text (Fig. 2a) is suggested for defining the scope of a given task, describing the instructions to be followed by the user and noting the start or finalisation of the test. This view is also useful in some other cases where users are requested to remember the instructions, e.g., some given names, as part of the current or a future task. Text (Fig. 2b) and numerical (Fig. 2c) inputs are considered for answering questions such as those asking for the current date or location. These types of input are commonly used in mobile devices and they can be easily adapted to each user as to maximise accessibility, e.g. by enlarging the font or display size through magnification. Some cognitive tests require copying given objects (Fig. 2e) or sketching concepts (Fig. 2d), which in turn involve drawing. Fairly ample canvases are considered for such drawings, thus allowing individuals to use their fingertips as a sort of pen. Digital pens, sometimes available for some brands, can also be used. Some tasks involve the repetition of a given piece of text (Fig. 2f). The voice is used in such case as input, which in combination with the text-to-speech functionality helps to automatically transcribe the answer of the user. This approach makes it possible to capture, in the form of text, any voice from supported languages, and virtually the automatic translation to any other. Both text and voice can be used to name a given object that is presented to the user through an embedded picture (Fig. 2g). This type of view is particularly regarded for tasks involving the recognition of items. Finally, we also consider the development of an experience sampling method to realise n-step command type tasks. These commands tend to involve manual handling, which poses special challenges to be implemented on mobile devices. For example, in the paper-and-pencil version of the MMSE, the participants have to take a sheet of paper in their right hands, fold it in half, and place it on the floor. The relevance of this task is not on the physical aspect but on remembering three different instructions and executing them. Therefore, similar tasks involving alike steps can in principle be developed. Users are presented first with the instructions, e.g. arranging some circles in a specific order depending on their colour, which are followed by the interaction space where the user can perform the task (Fig. 2h).

Open image in new window

Open image in new window

3.4 Implementation

MobileCogniTracker has been developed using Android Studio Version 2.3, and it has been tested on Android versions 6.0-6.0.1 (Marshmallow), 7.0-7.1.2 (Nougat) and 8.0 (Oreo). The tool largely builds on AWARE (version 4.0.700.selfie), an Android-based open-source mobile instrumentation framework (Ferreira et al. 2015). The motivation for choosing this framework is twofold: (1) it provides a client-server mobile framework that supports the collection of unobtrusive passive sensor data; and (2) it is licensed under the Apache Software License 2.0 so it allows for changes and extensions to the core code. For serialisation of XML files the Simple-XML serialisation framework for Java version 2.7.1 has been used.

The tool uses a server-client approach, which is enabled through AWARE. Experts can easily set up a study on the AWARE server through a web-based dashboard. Here, for example, the specialist can define the type of mobile data to be recorded on the user device, e.g., acceleration, battery usage or phone call logs, to name a few. Users can then join a study by simply scanning a QR code through the AWARE mobile app. Once it is running, the app sends periodically the collected data to the server over WiFi or 4G. We refer the reader to Ferreira et al. (2015) for additional details on the characteristics of the AWARE framework.

AWARE also supports basic ESM, which is a sort of plugin or extension to the default set of available sensor types. These ESM questionnaires are executed remotely and can be scheduled using the web dashboard or from within a plugin. AWARE provides some ESM types, including free text, radio buttons, checkbox, Likert scale, quick answer, scale, and numeric types. The ESM consist of a title, the instruction text, the submit button text, and the user answer which is encoded into a string. Additionally, it is possible to specify for how long the notification should be active for and how much time the user has to answer the question. The development of MobileCogniTracker thus consisted of changes to the core of the AWARE framework, as well as the development of a new Cognitive Experience Sampling plugin.

MobileCogniTracker extends AWARE to support the scheduling and construction of ESM-based tests provided in XML form. This is one main advantage of our tool with respect to similar approaches since the tests can be fully customised both in terms of contents and schedule, without requiring to recompile the application at all. MobileCogniTracker can create an ESM questionnaire and set a schedule based on a definition in an XML file that follows the XML schema. The schema is defined as follows. Each test is defined through one or more components, and each component can consist of one or more tasks and/or questions respectively. Namely, a component consists of a name (<name>) and the task to be performed (<task>). The task is composed of the question(s) to be asked to the user (<question>), an optional score given to that question (<score>), the type(s) of experience sampling elements (<ESM_Type>) and the specific instructions given to the user (<instructions>).

A simplified example of a possible schema is given in Listing 1. This example shows the XML file for the clock drawing task, a classical clinical cognitive assessment test (Royall et al. 1998). In this example: the name and a short description of the role of the test is provided; the text-to-speech functionality is enabled to read the instructions out; the question or instruction is defined as well as the experience sampling type, here similar to the one shown in Fig. 2; finally, the ESM is set to activate on Mondays at 11:10. The question and score of the task are only relevant for reference to the pencil-and-paper version of the test.

Open image in new window

The responses to the cognitive ESM are stored in a central SQL database. The structure is shown in Table 2. The database contains among others: the ‘device id’ which unequivocally and anonymously identifies each device partaking in the study; the ‘esm json’ field which shows the specific question from the cognitive tasks that was executed; the ‘esm user answer’ where the answer is collected. For non-text-based answers, such as drawing, copying, and rearranging circles, the data are first converted into strings before sent to the server. Namely, in the case of the drawing and copying tasks, the user-drawn image is encoded into a base64 string, which allows it to be easily stored in the database. The voice-enabled keyboard, i.e. speech-to-text functionality available on most Android phones, is used as an alternate option for users to input their answers. It should be noted that at this stage the system does not perform any analysis, identification, or categorisation of user input whatsoever. Therefore, as it is the case for the pencil-and-paper version, a clinical expert is required to analyse the results a posteriori.

Table 2

Cognitive ESM SQL table structure

Open image in new window

The usability evaluation showed, in general, a good level of satisfaction on most aspects. A majority of users reported MobileCogniTracker to be simple to use, easy to learn and coherent.

Age-wise, we did not find significant differences to report. Yet, to our surprise, (young) adults scored the tool below older adults rates. The histogram for the group of (young) adults (Fig. 11) actually shows a large difference in the scores for that group. From the test, we got the impression that some people in that group were evaluating the app based on if they would be interested in using it or not. Since they were all cognitively fit, and did not believe that they would have a cognitive impairment in the near future, this group seemed less interested in the app as compared to the older adults group. Conversely, in the older adults group, many reported to like the app because they believed it could bring great benefits for them. We also had the impression that some people in this group were more likely to compare it to a face-to-face examination of cognitive impairment, whereas the (young) adults group was rather inclined to compare the app to their experience with other apps.

A few participants would use the help of a technical person to initiate the tests. This has mainly to do with the fact that some people were either not familiarised with Android phones (i.e., iPhone users) or newer versions of the operating system. In those few cases, users were previously assisted as to be able to realise the test.

Participants also found the different experience sampling approaches to be well integrated and straightforward to use. However, some users experienced some difficulties with drawing and copying tasks. These tasks have been tested using the finger as input, whereas in the original clinical questionnaires they are performed using a pencil. The results from the evaluation test showed that it is sometimes difficult, even for non-impaired subjects, to achieve precise results. Therefore, a stylus should be used as the preferred input method for this task, whether available. On a similar note, the use of tablets or phablets could help facilitating the realisation of the drawing tasks. Although users did not experience difficulties while using the provided smartphones, which are of a generous yet standard size, they anticipated potential difficulties while using MobileCogniTracker on smaller devices. These findings are fairly in line with the requirements and design principles considered in this work.

Participants perceived the amount of questions to be fair enough. MobileCogniTracker is intended for long-term monitoring, and as such, the frequency with which questions and tasks are administered plays an important role in the acceptance and engagement with the tool. However, such frequency fairly depends on how rapidly the cognitive ability may vary as well as the prominence of the task. MobileCogniTracker facilitates the scheduling of questions and tasks, which can be planned separately and spread over the course of a day, week and/or month. User preferences could be combined with requirements posed by the clinical tests as to maximise the efficacy of the test. Hence, some questionnaires or tasks could be triggered once the user is available or the test be shortened so that users can answer in a minimally-interruptive manner.

During the evaluation, some participants mentioned that they perceived the questions of the MMSE as “too easy” and expected more challenging questions to measure cognitive performance. Users may thus require being challenged according to their specific age and cognitive state, so personalised testing may result in a more enjoyable and engaging experience. MobileCogniTracker was not aimed at replacing existing clinically-validated cognitive procedures but to enable them digitally in order to facilitate their continuous, opportunistic and ubiquitous administration. This, however, opens up an interesting research area building at the intersection of cognitive assessment, personalisation and context-awareness, in which tests and contents are not simply digitalised but also tailored to preferences or life-events relevant to each individual.

Participants suggested that this kind of tool could be used to improve their abilities through regular cognitive training. Although MobileCogniTracker was originally devised as a tool to observe and measure cognitive functioning, it could be also used to administer specific tasks intended to exercise some cognitive abilities. Thus, for example, users could be asked to perform more complex memory or attention training tasks, which could at the same time be logged for tracking the user’s cognitive functioning. Gamification techniques could also help in this regard to increase the appeal of the tasks, thus motivating user’s participation and engagement.

The feedback obtained from the clinical expert is also of much relevance to comprehend the advantages and limitations of the current version of MobileCogniTracker. While it seems clear that question-typing tests do not present important differences with respect to the pencil-and-paper version, some built-in functions of smartphones could influence the answers of the users. The autocomplete function is used in most phones to accelerate and facilitate the typing process, which is not really tested in questionnaires like MMSE. However, the fact that words are suggested can certainly bias the response of the user. An approach to deal with this issue would simply consist in disabling such functionality for MobileCogniTracker, as it was set here for the experiments with subjects. The expert also raised another very important point that concerns the persuasiveness of the tool when asking users to execute the tasks. Along this line, the role of context and not being in the presence of the clinical expert is highlighted. This aspect has not been studied in this work but it is something definitely worth studying. MobileCogniTracker can for sure be used at the clinic, as a sort of replacement for the pencil-and-paper version of cognitive tests; however, this tool is envisaged as particularly powerful when brought to the out-of-clinic settings. Our hypothesis is that users will feel more relaxed and calm while performing this type of tests at home. This could thus in turn lead to more reliable measurements of the patient daily cognitive functioning. On the other hand, patients can also get distracted and it may be more difficult to detect misuse of the tool. In that sense, as for any other remote monitoring and/or treatment, the cooperation of the user is particularly valued.

The results of the usability and validity study show that the tool is indeed usable by healthy subjects, even if they are not very familiar with the use of smartphones. The evaluation thus shows its potential use for tracking at least very early cognitive impairment, especially when starting with non-impaired subjects. It is unclear though how much cognitive impairment will influence the perceived usability of the application, and how much the application is capable to indicate the performance differences between cognitively impaired and non-impaired subjects. Future work should perform a more thorough evaluation considering screened cognitive impaired and non-impaired users, also comparing MobileCogniTracker to the paper-and-pencil version of cognitive assessment tests such as the MMSE. In this regard, the effect of external daily stimuli, the difficulty in translation from in-person to smartphone-based assessments, and the difficulty for patients to use the technology should be fairly explored.

This paper describes MobileCogniTracker, a mobile experience sampling tool that allows for the creation, administration and remote execution of digitised cognitive assessment tests. The tool provides multiple means to realise, on a user’s regular mobile device, typical questions and tasks used in clinical practice to assess cognitive functioning. Several input types are supported for the realisation of the tests, including plain text, text-to-speech, speech-to-text and free drawing. As for standard experience sampling methods, MobileCogniTracker allows the specialist to schedule the time when a test should be administered. The specialist can also configure whether the test sections should be executed consecutively or at different times.

MobileCogniTracker builds on top of an existing mobile instrumentation framework in order to facilitate the tracking of not only cognitive experience sampling but other types of passive sensor data. This combination is seen to facilitate the realisation of future research studying the interplay of cognitive and physical, social and emotional behaviours at both individual and population levels. In that view, MobileCogniTracker could help to support not only the timely assessment of cognitive impairment but also lay the ground for future semi-obtrusive detection of cognitive decline, as well as the recognition of accelerated decline, which could be linked to complex mental disorders such as dementia or the like.

A preliminary usability evaluation has been performed in order to determine how users perceive the proposed tool. To that end, a digital implementation of the popular MMSE clinical test has been particularly considered. Results show that users are generally satisfied with the tool, which they find simple and easy to use. Tasks involving drawing on the screen can nevertheless be enhanced by using more accurate means than the fingertip plus adaptive user interfaces (Hussain et al. 2018). The performed evaluation is limited to healthy people with no recognised cognitive disorder. In addition to that, an expert-based validation of the tool has been performed in contrast to the pencil-and-paper version. According to the expert’s opinion, the tool fairly compares to the pencil-and-paper version, although the drawing tasks are more difficult to judge than when performed on paper. Additional studies on the impact of technology-literacy and contextual factors in the realisation of the test should be also performed. Future work thus includes a longitudinal validation with cognitively impaired and non-impaired people as to ascertain the extent to which the tool is usable, especially in those cases with severe cognitive disorders. Our long-term research will also aim to integrate the proposed tool into innovative e-coaching solutions (Banos and Nugent 2018; op den Akker et al. 2018) as to facilitate not only the autonomous tracking but also the intervention of cognitive impairments.