Keywords

1 Introduction

A key issue in the therapy of patients with psychotic disorders (i.e., bipolar disorder and schizophrenia) is the ability to constant monitor their physiological and mental status, even 24/7 if that is possible. In this research work, we present an innovative and advanced computer-based platform that facilitates the effective monitoring and can support the relapse prevention in such patients. This research work is done within the e-Prevention research and development project that is coordinated by the National Technical University of Athens (https://eprevention.gr/) and it is co-funded by the European Commission and Greek National funds.

The innovative key offerings of the envisioned system, are as follows: 1) long-term continuous recordings of biometric signals and motion data through simple wearable sensors (i.e., smartwatch), 2) a portable device (tablet) installed in the patient’s residence, which records short-term audio-visual videos of the patient while communicating with the doctor; and by using affective computing methodologies is able to understand the emotional status of the patient, and 3) automatic and systematic storing and management of the captured data in a Cloud Infrastructure. The stored data are then processed through machine learning and signal processing techniques in order to detect changes and patterns, to facilitate the prediction of clinical symptoms and side effects of the patients medication.

Fig. 1.
figure 1

The overall architecture of the proposed system.

Full size image

The final user of the platform, who is the attending physician, is able to continuously monitor and optionally also annotate the data and receive information about the patient’s daily mental and physiological status by observing the data from the wearable sensors and the emotion analysis of the short videos. Additionally, the computational intelligence algorithms incorporated in the platform enable big data processing and statistical analysis and prediction of salient events regarding the patient’s daily routine along with the corresponding visualizations. The overall architecture of the proposed e-Prevention system is illustrated in Fig. 1.

2 Related Work

The evolution and rapid dissemination of mobile and wearable consumer technology, such as smartwatches and fitness trackers, has created unique opportunities for personalized data collection in an unobtrusive and even affordable way. Furthermore, the enormous technological advances have enabled the reliable recording and quantification of a large number of behavioral and biometric indexes through their sensors [23, 25]. Such sensors usually include accelerometers, gyroscopes and heart rate monitors among others for measuring and detecting the user’s motion/kinetic activity, sleep patterns and autonomic function [9, 26], opening this way the possibility for non-intrusive acquisition of activity data.

Over the last years, wearable technology due to the inclusion of these sensors has in fact become more proximal to human activity, having as an eminent result to become increasingly acceptable even in healthcare. As early as 2006, the Institute of Medicine recognized the potential to transform mental health services by providing more continuous and precise information on patient-specific behavior, symptoms, and medication side effects. In order to do that, novel methods are demanded to be developed, to transform raw data into knowledge and then use this in turn to support personalized interventions [6] so as to transform hospital-centered healthcare practice to proactive, individualized care. Behavioral and biometric indexes have been already used in general medicine and sports and nowadays the evidence shows that they could be introduced into clinical psychiatry [6], as well. Specifically, psychosis is a spectrum of disorders that underlie different etiopathogenic mechanisms acting on the Central Nervous System (CNS), leading to common symptoms [31]. Despite extensive research over the last 60 years in neurobiology and neurophysiology of psychotic disorders, their cause remains unclear and reliable biometric indexes for the diagnosis and prediction of the course of the psychotic symptomatology have not yet been found. Based on the fact that the process of psychosis is continuous and relapse is a “biological" process that evolves over time [13, 21, 32], it would be expected to observe changes in such indexes that are related and likely precede the onset and/or worsening of psychotic conditions. To mention a few, some of the typical early warning signs of mental illnesses include parkinsonian type symptoms, i.e., rigidity, tremor, jerking arm movements, or involuntary movements of the limps, awkward gait, unusual gestures or postures, decreased physical activity and social interactions, abnormal sleep patterns [1, 5, 20]. The continuous monitoring assessed through such wearables, opens the way for more precise and personalized digital interventions, and complement by enabling detection of early signs of illness relapse, medication adherence or even treatment efficacy and thus may help increase the number of positive clinical outcomes in mental healthcare [10, 28].

Major research areas in psychiatry [7, 8, 17, 22] have suggested employing technological advancement for accurate and continuous monitoring of patients to reduce the impact of mental illness on a patient’s daily activities, for early diagnosis and prevention of psychotic relapses so as to increase the effectiveness of treatments. For instance, sensor technology, has been used in order to understand patterns of daily behavior that may be indicative of trends or changes in factors related to mental health, including sleep, mood, and stress [15, 24], or to characterize a diverse array of psychopathology, from schizophrenia and depression to general mental health [11]. In [30] a combination of accelerometer and location sensors could detect relapse in depressive symptoms before patients reported such changes. It has also been found that such sensor technology could predict changes in affect and behavior (for a review, please see [15]).

3 System Architecture

The availability of commodity consumer devices (i.e., smartwatches) has eased motion data collection but in the context of the e-Prevention project the requirement to collect and transmit constantly large amounts of raw biosignal data did not fit the normal use of available consumer devices. We evaluated a number of available devices and selected the Samsung Gear S3 Frontier as the user data collection device. Compared to other smartwatches that were tested, the Samsung Gear has the ability to store and send data from acceleration, angular velocity and pulse sensors, providing also the heartbeat interval period value (R-R interval), while it has a large storage capacity (4 GB) capable of storing data for a few days and a WiFi interface removing the need for a smartphone for connectivity.

In addition, the smartwatch has the ability to run specific applications to collect data and upload them to a server. An application was developed in the Tizen Studio environment using technologies to implement Tizen applications (Tizen Hybrid Application)  [2]. Hybrid Application applications consist of sections implemented either with native C code to achieve efficient operation or with javascript code to implement interfaces with available Tizen Advanced UI (TAU) libraries. The application implements smartwatch data collection functions and transmission to a cloud infrastructure. Due to the large volume of data produced, various compression techniques without loss of information were considered taking into account the limited computational and energy resources of the smartwatch. Analytics were also incorporated to gather critical smartwatch operating parameters such as power levels, free storage and network availability.

3.1 Architecture of the Smartwatch Application

The architecture of the smartwatch application is illustrated in Fig. 2 and consists of the following modules:

  1. 1.

    The Sensor Service module that interacts with the smartwatch sensors to collect data and store them temporarily on the device. Due to performance requirements it is implemented as a Tizen native service.

  2. 2.

    The Data Transmission Module (Network Service) that compresses and transfers the data to the server; also implemented as a Tizen native service.

  3. 3.

    The Control Service that coordinates the data collection and transmits units taking into account the state of the internet connection and the smartwatch energy status. It also aggregates the smartwatch operating parameters (analytics) for sending to the server.

  4. 4.

    The Tizen Web App interface that provides the user with basic information on the functionality of the application. The implementation utilizes advanced web design technologies based on TAU wearable interface libraries.

3.2 Biosignal Data Collection and Energy Consumption of Smartwatch

The sensor data are provided by the Tizen operating system in various ways dependent on the characteristics of the measured parameters. Extensive testing during the pilot development of the system leaded to the optimum data sampling rate in relation to the optimum time between successive battery charges.

Table 1 shows the sampling frequencies that were selected according to the requirements of the project’s medical team to ensure continuous recordings between charges of approximately 24 h.

Fig. 2.
figure 2

Smartwatch application architecture.

Full size image
Table 1. Biosignal collection parameters.
Full size table

Step data and sleep duration are also stored as calculated by the smartwatch, while GPS tracking was not used as it significantly increased power consumption and reduced the time required between successive loads. The actual output data amounts to about 300 MB/user/day, where the day of use includes approximately 22 h of continuous recording. The data is compressed at about 100–120 MB/user/day. About 20 users require about 2.5 GB of storage per day. In total, 24 months of compressed logging require approximately 1.7 TB of storage.

3.3 Data Collection Module

The requirements for collecting large volumes of data from smartwatch sensors require an appropriate software architecture based on efficient event-driven sensor management. Tizen uses an asynchronous programming interface (API) for sensors based on function callbacks that allow data transfer when available. The data collection module initializes the sensors required and attaches appropriate management routines to transfer the data and store it along with a timestamp in the smartwatch storage.

Data are organized into separate folders that contain files depending on the sensor type as well as timing information. Each data structure corresponds to a smartwatch usage period between two charging events. The above organization allows smartwatch data to be cached if there is no network connection or data upload problems on the server. Also independent storage in separate structures reduces the risk of data loss.

Fig. 3.
figure 3

Flowchart of data collection and transmission.

Full size image

3.4 Data Transmission Module

The watch has advanced connectivity to local wireless WiFi networks but the volume of data requires efficient management of the data upload. The upload process is demanding on watch resources especially concerning energy, so it only takes place when the smartwatch is in the charging state.

Due to the high volume, the data collection module ensures efficient storage with minimal processing (Fig. 3). The data are stored in the smartwatch’s storage in chronological order and are organized into groups corresponding to the intervals between device charging events. The available smartwatch storage capacity (approx. 4 GB) allows continuous recording and storage of measurements corresponding to 8 days. If the smartwatch is connected to a network, then as described below the data are transferred to the cloud and this time is renewed.

Transmission is organized into packets and compressed with the DEFLATE algorithm as implemented by the zlib library  [3]. This library is a reliable solution with efficient use of the limited smartwatch computing resources and it is supported by a middleware on the node.js server platform. Running the decompression processes on the server concurrently allows data to be received from multiple smartwatches simultaneously.

3.5 Architecture of the Biosignal Collection Server

Fig. 4.
figure 4

Biosignal Collection server architecture.

Full size image

The server uses the ‘\(\sim \)okeanos’ national public infrastructure-as-a-service (IaaS), see Fig. 4 for the server architecture. The application is implemented with node.js technologies and uses the NoSQL MongoDB database. The application runs in an isolated Docker container and is structured as follows:

  1. 1.

    Authentication Controller that implements system resources access policies according to user roles.

  2. 2.

    Device Controller that implements the process of registering new devices in the system and gathering watch information on the functionality of the connected devices.

  3. 3.

    Biosignals Controller that manages the efficient transfer of biosignal data.

  4. 4.

    Admin Controller that implements the system management and monitoring.

  5. 5.

    The WEB UI module that constitutes the interface of the administrators with the application described in Sect. 4.1.

  6. 6.

    Web REST interface (REST interface) that routes REST calls from the devices and Web UI application to the appropriate module.

4 The System in Practice

4.1 Web UI for System Administrators

The Administrator Web UI interface (Fig. 5) allows devices to monitor and access downloaded data. It consists of a client subsystem implemented with front-end responsive frameworks such as Bootstrap and Vue.js and a server subsystem implemented with node.js. The portal features are: i) access to user log data, ii) new device registration, iii) e-mail notification system to administrators when a device has not uploaded data for more than 1 day, and iv) watch analytics.

Fig. 5.
figure 5

Administrator Web UI.

Full size image

The alert system uses information from the Device Controller module concerning device connectivity to generate alerts to administrators. The administrator interface allows access to functionality concerning activity alerts, user data and smartwatch analytics. It also provides access to data collected from the accompanying Android application used to collect activity data, which are manually annotated by the users.

4.2 Data Management

As mentioned above, for efficient use of limited smartwatch computing resources, data is stored as provided by the available sensors. Motion and pulse tracking sensors produce data at regular intervals depending on the sampling rate, while step and sleep data are provided by the smartwatch operating system comprising a very small amount of data compared to the original sensor data.

Various methodologies for storing the original sensor data were tested, but prior to the development of the system, priority was given to recording all available information in a way that allowed for thorough processing to eliminate errors caused by sensor operating problems and smartwatch software. In addition, in the pilot phase, the storage of the .csv format was chosen to save the processing time so as not to limit the processing capabilities.

The sheer volume of data produced and the requirement for flexible ways of securely accessing data create high storage requirements; thus was mitigated by the use of proven data compression techniques. Structuring the application of the server as a set of “micro services” for data transfer, without loss of information, entails the transfer of biosignals from the efficient processing and storage process. As part of the pilot process, the data is stored either uncompressed in the form of .csv files for immediate local processing or as compressed packets for a specific data collection time period ready for off-server transfer.

The need to store a large volume of data led to the creation of a new virtual machine in the \(\sim \)okeanos infrastructure for the ultimate storage of the compressed data, and it is available for further processing through an appropriate sftp interface.

Fig. 6.
figure 6

Visualization of user movement and heart rate data, i.e., accelaration data (top row), gyroscope data (middle row) for all three axis (x, y, z-axis) and data collected by the heart rate monitor (i.e., heart rate and RR-interval data). (Color figure online)

Full size image

5 Initial Data Processing and Results

5.1 Qualitative Analysis

We performed an exploratory data analysis (i.e., kinetic and heart rate data), employing both traditional signal processing techniques, such as short-time analysis, as well as more sophisticated non-linear methods; i.e., multi-scale fractal and non-linear dynamics analysis, in order to extract descriptors that efficiently convey behavioral and biometric information. Such non-linear descriptors have been shown to be of importance for the modeling of a series of other 1D signals, see as for instance [16, 19, 27, 33]). Our first preliminary results validate the efficiency of the features, thus they could be employed for building tools that can be used for more complex pattern extraction, i.e., behavioral changes, that can be correlated with mental health issues or relapses. Figure 6 shows examples of the raw data from the accelererometer (acc), gyroscope (gyr) and heart rate monitor (hrm) collected form a control user. All data is shown for a 24 h interval (where the horizontal blue line indicates when the user was asleep), so obvious differences can be observed.

Fig. 7.
figure 7

Short Time Energy of the acc and gyr signals (x-axis) for two different states (walking and other, e.g., resting, standing), using a 10 s sliding window.

Full size image

Short-Time Energy Analysis: We extracted the short-time energy, to explore differences when the user is walking, sleeping, or doing other activities. Figure 7 presents the calculated short-time energy, on the acc and gyr signals (x-axis), showing two 10-min. intervals corresponding to the different states of walking vs. other activity. The short-time energy is computed using a 400-sample window applied on the signal and provides insights for the signal content. We can observe that the short-time energy presents a multitude of peaks in the acc and gyr data during walking, and fewer peaks with higher values of energy when doing other activities. On the contrary, during sleep, the energy was very low, mainly including the energy of the noise produced by the sensors, thus we omitted showing the respective figures. Concluding, the results are intuitive, and show good discriminative abilities for the different states.

Fig. 8.
figure 8

Raw signals and mean and std (error bars) of the multiscale fractal dimension distribution of the acc (x-axis) and the hrm data for walking, other and sleep (the MFDs are shown for 60-s analysis windows, updated every 30 s).

Full size image
Fig. 9.
figure 9

Reconstructed attractors for the three states. Each additional dimension above 3 and up to 6 is shown using different colors. D2 is the correlation dimension, and SE the sample entropy.

Full size image

Multi-scale Fractals: Regarding non-linear analysis, we explored an efficient algorithm [18] that measures the short-time fractal dimension, based on the Minkowski-Bouligand dimension [12]. Fractal dimension D is between 1 and 2 for 1D signals; and the larger the D is, the larger the amount of geometrical fragmentation of the signal. We conducted our analysis at multiple time scales, since real-world signals do not have the same structure over all scales [18], and we measured the multiscale fractal dimension (MFD) profile as a descriptor for 10-min. intervals of continuous data. Figure 8 shows MFD profiles measured for the three different states, i.e., walking, other and sleeping for the acc and the hrm data. The various profiles show quite distinctive patterns for every state and each type of data. Specifically, we note high fractal dimension up to \(D=1.2\) and \(D=1.4\) for the acc data during walking and other, while it gets up to \(D=1.6\) for the acc-sleep data. Regarding the hrm data, the higher fractal dimension is noticed for sleeping (up to \(D=1.4\)), while walking is around \(D=1\) getting higher and up to \(D=1.3\) for larger scales s.

Nonlinear Dynamics and Attractor Reconstruction: Next, we considered that the human movements, as well as the human heart rate system, constitute separate nonlinear dynamics systems. The two variables that must be calculated in order to unfold the attractor is the time delay \(T_d\) and the dimension \(D_e\). The most common methods for calculating the lag \(T_d\) constitute of either selecting the first local minimum of the mutual information between the original signal s(n) and its time-delayed version \(s(n+T_d)\) or the first zero of the autocorrelation function. The dimension \(D_e\) is increased until the false neighbours vanish, in which case we consider the attractor has unfolded [4].

Figure 9 shows the reconstructed attractors for short 1-min. intervals. During walking we observe that the reconstructed phase space exhibits a quite clear topology, due to the fact the user performs specific movements. During sleeping, due to the low amount of movement, the sensor noise is prevalent, with few random movements, while during other activities the phase space is dominated by seemingly random movements without a pattern. We generally notice that a nonlinear dynamical systems approach could be proven to be useful for deducing activities and other behavior throughout everyday life. The reconstruction of the attractor is a necessary step towards computing the nonlinear dynamics of the system. Such values are for example the correlation dimension of the reconstructed attractor  [14], and the sample entropy  [29]. In Fig. 9 we also show those values; D2 is the correlation dimension and SE the sample entropy.

5.2 Quantitative Analysis

Database Description: In order to perform the quantitative analysis, we created a small dataset by randomly sampling 10 volunteers. We then selected from each volunteer 120 time intervals of 10-min. length, i.e., 40 intervals during which the subject was sleeping, 40 intervals during walking/running, and 40 intervals during other activities (rest, standing, working). The selected intervals were random, and we only made sure that the data collected during these intervals were valid, since in some cases, sensor malfunctions resulted in data loss, or the heart rate monitor was unable to accurately detect the heart rate. The analysis that follows use this dataset and attempts to identify useful cues/features, that can be used for accurately detecting the state of a person (sleeping, walking, running), so that later on can generalize to other physiological states. It should be finally mentioned that according to the the act for data protection a written consent was collected for all participating users providing the motion data that are describe and analyzed in Sect. 5.

Experimental Evaluation: In order to validate the performance of the features explored in Sect. 5.1, we employed a Support Vector Machine (SVM) with an RBF kernel, using 5-fold cross validation. Table 2 presents the results for different subset combinations.

Table 2. Results on the collected data for various linear and non-linear features.
Full size table

With STE we denote the statistics (mean and std) of short-time energy analysis performed on the: 3 accelerometer (x, y, z-axis), 3 gyroscope (x, y, z-axis), and 1 heart rate signals. By taking the mean and std of the short-time energy calculated over each interval we have a total of 14 features for each interval. CH denotes the correlation dimension and the sample entropy calculated by performing delay embedding with \(D_e = 4\) for the kinetic data and \(D_e=2\) for the heart rate data. MFD-G, MFD-A, and MFD-H denote features from the multi-scale fractal analysis of the gyroscope (x-axis), the accelerometer (x-axis), and the heart rate, respectively. In order to reduce the number of features extracted by the MFD analysis (900 dimensions for kinetic data and 300 for heart rate), we perform linear sampling, picking a total of 30 points for the heart rate and 32 points for the accelerometer and gyroscope MFD analysis.

As we can see from the accuracy (%) results in Table 2, both the traditional speech processing methods, such as the STE analysis, as well as the multi-scale fractal analysis can be used in order to discriminate between the different states. The same can be observed using the CH features which combined with STE, increase the accuracy of the system. Our results show that the extracted descriptors examined in this paper can provide useful insights and can be used in order to build a tool able to assess the state of the user.

6 Conclusions

Concluding, the novelty of this work is the implementation of a cloud based platform for the extraction of key markers and indicators that can possibly assist the medical personnel in monitoring patients with mental disorders and predict psychotic symptom’s relapses and adverse medicine side effects. This is done initially by long-term recordings of biosignal and motion related indexes and further processing. As this project evolves the collected data will be combined with the short-time videos and their affective analysis as well as with responses of patients to standard questionnaires. Nevertheless, the results so far prove the feasibility of our research effort in the continuous collection of biometric and motion data along with their prognostic value.