Developing new cognitive assessment methods based on unobtrusive mobile sensing data requires reliable digital tests exploiting the diagnosis capabilities of well-established clinical procedures. In that vein, we propose MobileCogniTracker, a digital tool that complements and extends the potential of existing mobile passive sensing platforms for the measurement of people’s cognitive functioning. MobileCogniTracker develops an innovative experience sampling tool that helps automate and objectify the measurement of clinical-grade cognitive data. In the following, we describe the requirements, design choices and implementation of the proposed mobile cognitive tracking tool.
We use the MoSCoW prioritization technique (Clegg and Barker 1994) for the elicitation of the requirements of MobileCogniTracker. The tool must work on modern mobile devices. Tablets would perhaps be the preferred choice given the typically large size of the screen, which resembles in a way the format of typical clinical assessment questionnaire-type handouts. However, the tool must also support the realisation of the cognitive assessment tests on smartphones since they are both more available to users and more frequently used for passive sensing. It must be possible to create different sections, as it is normally developed in clinical cognitive tests, which also contain separate tasks. These sections and tasks are typically organised around a specific cognitive ability, thus their separation facilitates proper administration, reusability and shareability among tests. The tool must present the information in the form of schedulable experience sampling methods. The specialist must be able to remotely specify the time when the application should ask the test questions to the user. The expert must be also able to specify separate schedules for the different test sections. In this way, the tests can be partitioned into various parts, possibly measuring different cognitive abilities, which are administered at different times according to the study or user preferences. This requirement is also of much relevance when it comes to the study of both temporal and contextual effects on the realisation of the tests. The data must be stored on a secure server for further analysis. This is a crucial requirement as to avoid any hazardous situation or malicious use of the collected data by third unauthorised parties. Finally, the system must allow for future extensibility, specifically the integration with new unobtrusive sensors and experience sampling modalities.
In clinical cognitive tests, the majority of answers involve a user talking to the specialist. For other questions, users are asked to write or draw their answers. Therefore, different input methods should be supported, allowing users to write, draw or speak their answers freely depending on the question. Voice input can be achieved through speech-to-text functionality, which would allow several questions to closely resemble the traditional testing scenario and avoid typing issues for users with reduced motor coordination. Furthermore, it should also include a text-to-speech functionality, meaning the instructions are read-out-loud to the user in order to avoid misinterpretation and facilitate accessibility to people with mild visual impairment. It could be possible to allow the specialists to change the order in which the test sections and tasks are presented so that the test subjects will be less likely to remember which sections or tasks are to appear next. Together with the capability to schedule specific test components, this feature could allow sections or tasks to be easily replaced by other similar ones, thereby avoiding a learning effect on the subjects after executing the test several times.
The architecture technical diagram of MobileCogniTracker is shown in Fig. 1. The mobile device is the core entity, which communicates with the other main entities, i.e., the user and the server (expert). The expert sets, through the server, the study properties both in terms of tests to be realised by the user and the schedule for their administration. These properties are stored in a configuration file that is then communicated to the mobile application, which automatically updates the local configuration as to ensure proper operation in the absence of internet connection. At the scheduled time, the app pushes a notification awaiting for the reaction of the user. Once the user clicks on the notification, the corresponding cognitive test, i.e. question(s) and/or task(s), is prompted to the user for its realisation. Questions and tasks can be read on the screen or spoken out for the user convenience through text-to-speech functionalities natively supported by the mobile operating system. The user’s answers, which can come in different modalities, namely text, voice and drawings, are stored temporarily on the mobile device. This, and possibly other mobile sensor data, is periodically synced with the server as to make it available to the experts for further analysis.
Cognitive experience sampling methods
This section presents the set of experience sampling methods we have developed for the realisation and collection of the cognitive tests. According to the requirements identified above, different methods for giving instructions and capturing answers must be provided. Plain text (Fig. 2a) is suggested for defining the scope of a given task, describing the instructions to be followed by the user and noting the start or finalisation of the test. This view is also useful in some other cases where users are requested to remember the instructions, e.g., some given names, as part of the current or a future task. Text (Fig. 2b) and numerical (Fig. 2c) inputs are considered for answering questions such as those asking for the current date or location. These types of input are commonly used in mobile devices and they can be easily adapted to each user as to maximise accessibility, e.g. by enlarging the font or display size through magnification. Some cognitive tests require copying given objects (Fig. 2e) or sketching concepts (Fig. 2d), which in turn involve drawing. Fairly ample canvases are considered for such drawings, thus allowing individuals to use their fingertips as a sort of pen. Digital pens, sometimes available for some brands, can also be used. Some tasks involve the repetition of a given piece of text (Fig. 2f). The voice is used in such case as input, which in combination with the text-to-speech functionality helps to automatically transcribe the answer of the user. This approach makes it possible to capture, in the form of text, any voice from supported languages, and virtually the automatic translation to any other. Both text and voice can be used to name a given object that is presented to the user through an embedded picture (Fig. 2g). This type of view is particularly regarded for tasks involving the recognition of items. Finally, we also consider the development of an experience sampling method to realise n-step command type tasks. These commands tend to involve manual handling, which poses special challenges to be implemented on mobile devices. For example, in the paper-and-pencil version of the MMSE, the participants have to take a sheet of paper in their right hands, fold it in half, and place it on the floor. The relevance of this task is not on the physical aspect but on remembering three different instructions and executing them. Therefore, similar tasks involving alike steps can in principle be developed. Users are presented first with the instructions, e.g. arranging some circles in a specific order depending on their colour, which are followed by the interaction space where the user can perform the task (Fig. 2h).
MobileCogniTracker has been developed using Android Studio Version 2.3, and it has been tested on Android versions 6.0-6.0.1 (Marshmallow), 7.0-7.1.2 (Nougat) and 8.0 (Oreo). The tool largely builds on AWARE (version 4.0.700.selfie), an Android-based open-source mobile instrumentation framework (Ferreira et al. 2015). The motivation for choosing this framework is twofold: (1) it provides a client-server mobile framework that supports the collection of unobtrusive passive sensor data; and (2) it is licensed under the Apache Software License 2.0 so it allows for changes and extensions to the core code. For serialisation of XML files the Simple-XML serialisation framework for Java version 2.7.1 has been used.
The tool uses a server-client approach, which is enabled through AWARE. Experts can easily set up a study on the AWARE server through a web-based dashboard. Here, for example, the specialist can define the type of mobile data to be recorded on the user device, e.g., acceleration, battery usage or phone call logs, to name a few. Users can then join a study by simply scanning a QR code through the AWARE mobile app. Once it is running, the app sends periodically the collected data to the server over WiFi or 4G. We refer the reader to Ferreira et al. (2015) for additional details on the characteristics of the AWARE framework.
AWARE also supports basic ESM, which is a sort of plugin or extension to the default set of available sensor types. These ESM questionnaires are executed remotely and can be scheduled using the web dashboard or from within a plugin. AWARE provides some ESM types, including free text, radio buttons, checkbox, Likert scale, quick answer, scale, and numeric types. The ESM consist of a title, the instruction text, the submit button text, and the user answer which is encoded into a string. Additionally, it is possible to specify for how long the notification should be active for and how much time the user has to answer the question. The development of MobileCogniTracker thus consisted of changes to the core of the AWARE framework, as well as the development of a new Cognitive Experience Sampling plugin.
MobileCogniTracker extends AWARE to support the scheduling and construction of ESM-based tests provided in XML form. This is one main advantage of our tool with respect to similar approaches since the tests can be fully customised both in terms of contents and schedule, without requiring to recompile the application at all. MobileCogniTracker can create an ESM questionnaire and set a schedule based on a definition in an XML file that follows the XML schema. The schema is defined as follows. Each test is defined through one or more components, and each component can consist of one or more tasks and/or questions respectively. Namely, a component consists of a name (<name>) and the task to be performed (<task>). The task is composed of the question(s) to be asked to the user (<question>), an optional score given to that question (<score>), the type(s) of experience sampling elements (<ESM_Type>) and the specific instructions given to the user (<instructions>).
A simplified example of a possible schema is given in Listing 1. This example shows the XML file for the clock drawing task, a classical clinical cognitive assessment test (Royall et al. 1998). In this example: the name and a short description of the role of the test is provided; the text-to-speech functionality is enabled to read the instructions out; the question or instruction is defined as well as the experience sampling type, here similar to the one shown in Fig. 2; finally, the ESM is set to activate on Mondays at 11:10. The question and score of the task are only relevant for reference to the pencil-and-paper version of the test.
The responses to the cognitive ESM are stored in a central SQL database. The structure is shown in Table 2. The database contains among others: the ‘device id’ which unequivocally and anonymously identifies each device partaking in the study; the ‘esm json’ field which shows the specific question from the cognitive tasks that was executed; the ‘esm user answer’ where the answer is collected. For non-text-based answers, such as drawing, copying, and rearranging circles, the data are first converted into strings before sent to the server. Namely, in the case of the drawing and copying tasks, the user-drawn image is encoded into a base64 string, which allows it to be easily stored in the database. The voice-enabled keyboard, i.e. speech-to-text functionality available on most Android phones, is used as an alternate option for users to input their answers. It should be noted that at this stage the system does not perform any analysis, identification, or categorisation of user input whatsoever. Therefore, as it is the case for the pencil-and-paper version, a clinical expert is required to analyse the results a posteriori.