Noninvasive stress recognition considering the current activity

Abstract

In this study, we aim to determine the stress level in a non-invasive way to the maximum extent possible by analyzing behavioral and contextual data received from the only source being a smartphone containing the data gathered in real-life situations. The information collected includes audio, gyroscope and accelerometer features, light condition, screen mode (on/off), current stress level self-assessment, and the current activity type. Three stress analysis models have been built: two with the consideration of current activities of a participant and one without those. Classification of low- and high-stress conditions, which was executed for a separate model for a certain kind of activity only, enabled us to achieve 3.9 % higher accuracy than that under the conditions when those activities were neglected. Also, the Android application was developed as a means for the current activity-type identification.

Introduction

Problems related to mental health are gradually moving to priority positions in the structure of public health of today’s world. Stress is one of the main reasons for this, and it causes activation of the sympathetic division of the autonomic nervous and the hypothalamo–hypophyseal portal systems. The result of this reaction is secretion of hormones, including cortisol which is responsible for neurotoxic damages, emotional and vegetative reactions, and ultimately, behavioral and mental disorders and somatic diseases [1–3].

Stress can be both a brief reaction to some events, and it can also have a prolonged/chronic effect on the body. Such prolonged impact on the human body is very dangerous, and it can cause various diseases. According to the report [4], stress as a basic reason can lead to more than 60 % of all human illnesses. Depression, anxiety, antisocial behavior, fatigue cognitive impairments, sleep and immunological disorders, and neurodegenerative diseases can be a result of a long-term stress impact [5].

The most common stress recognition methods are connected with determination of physical and physiological responses of the body to stress. According to researchers [6], the following body features can be used for these purposes: heart rate (HR), heart rate variability (HRV), electrodermal activity (EDA), electrocardiogram (ECG), electromyogram (EMG), skin temperature (ST), pupil dilation (PD), blood volume pulse (BVP), respiration, voice features, facial expression, eye gaze, and blink rates. Despite the progress and the potential of physiological and physical stress recognition, these methods also have its disadvantages: In order to gather data on body (physiological data) parameters, it is necessary to wear various wearable sensors (chest belts, wrist bands, head wearing devices, skin patches, and others).

Rapid growth in smartphone use, development of its technical capabilities, and increasing number of sensors built into them allow us to analyze various stress situations using the data received from smartphones as the only and sufficient source (behavioral pattern and contextual data). A smartphone is a device that we always carry with us. Smartphone data analysis method for stress recognition can replace the physical and physiological data analysis method and consequently reach maximal noninvasiveness and unobtrusiveness in the process of stress recognition.

Nevertheless, there are several critical tasks that have to be taken into account in the data analysis. Physiological and physical reactions of the human body vary depending on the type of activity; our behavioral pattern, and physical and mental sensibility to stress also vary depending on the current activity type. This research applies innovative technologies to stress recognition by analyzing behavioral and contextual data received from smartphones as the only and sufficient source. Additionally, we determined current activity types in this study by using an Android application with the purpose to evaluate the accuracy of the classifiers both with the consideration of current activities of the participant and without those.

The remainder of the paper is organized as follows: In Sect. 2, related work is presented. Section 3 describes the proposed approach to stress determination. The results are presented in Sect. 4. Finally, conclusions are made in Sect. 5.

Related work

Stress determination

Due to increased public interest to mental health, numerous researches have been recently published regarding the field of stress recognition. This fact approves the importance of the mentioned topic among researchers. For example, more than 1000 scientific articles were published in the first quarter of 2015 as compared to only 616 articles in the field of stress analysis for the entire year of 2000. Figure 1 demonstrates the number of publications available through the Google Scholar service, depending on the year. To find relevant studies, we performed a search query on the Web site at http://scholar.google.ru and sorted the data by year. In this search query, there were also negative keywords added from those areas that are not associated with stress or emotion recognition.

Fig. 1

Number of articles available via Google Scholar service depending on the year

Authors of many studies evaluated changes in physical and physiological parameters of a human body depending on the stress impact. ECG, EMG, skin conduction (SC), and respiration data were used for determination of the driver stress level [7]. The stress detection system based on BVP, PD, GSR, and ST data was developed [8] to recognize the stress level among computer users. It should be noted that high accuracy of classifications was achieved in these studies which amounts to 97.4 % in the first and 90.01 % in the second study, respectively.

When smartphones gained popularity in the world, there arose an issue of whether stress leads to changes in our behavior that are possible to be detected by means of a mobile phone, and whether there is a correlation between the stress level and contextual information. The authors [9] classified whether the participants were stressed or not with 75 % accuracy through the application of data combination received from wearable sensors (accelerometer (ACC) and SC), and from a smartphone [calls, short message service, location, and screen mode (on/off)]. The other study offered a solution for stress assessment of people based on the data derived from a chest belt (collected HRV data during sleep), and a smartphone (collected audio, physical activity, and communication data during a working day) with 61 % accuracy [10]. Nevertheless, it should be noted that continuous stress monitoring means the same continuous wearing/carrying wearable sensors, which can be uncomfortable for participants. So, for example, according to the report [11], one-third of wearable fitness tracker owners stop wearing them after about 6 months. However, there is one device we cannot do without in modern life. The device we always take with us is a mobile phone. Through the analysis of behavioral and contextual data received from a smartphone, it becomes possible to receive noninvasive and unobtrusive stress recognition methods when necessary. In the last few years, researchers have put a lot of effort to achieve successful results by means of the above-mentioned methods.

Using a mobile phone for stress recognition

Changes in our behavior caused by stress can be determined through the analysis of the data received from a smartphone. The authors [12] detected behavioral changes among students during an examination period (considered as a stress-full time) and after an examination period (considered as a stress-less time) by using a mobile phone. They were able to detect an average behavior modification equal to 53 % between examination and non-examination periods. Experimental results in another study where researchers achieved 72.9 % accuracy for a two-class classification showed that human behavior metrics derived from the mobile phone activity can reliably recognize a daily stress [13]. A short comparison of various stress recognition techniques applied in the use of a smartphone for stress recognition purposes is presented in Table 1.

Table 1 Overview of stress recognition solutions using a mobile phone

The proposed approach to stress determination

Smartphone Nexus 5 was applied (behavioral and contextual data collection, provision of current stress level self-assessments) for the purpose of developing a solution for stress determination in an indirect way (noninvasive). No additional wearable sensors were applied. The following sections describe the proposed approach for data collection and analysis.

Behavioral and contextual data

Application based on framework Funf [18] gathered behavioral and context data. Then, these data were uploaded to a server in SQLite format upon availability of a WiFi connection. Once all the data were collected, it was imported to the database MySQL in separate tables, and the titles of the tables corresponded to the means of the collected information. The field “time stamp” was a primary key for every table, unique within a single table and corresponded to the time of occurrence of the event in the Unix format in 1-s increments. The schema of the main database storing information obtained from a variety of built-in smartphone sensors is depicted in Fig. 2.

Fig. 2

XML schema of the MySQL database

Brief description and collected data properties are listed below.

• (A) Assess: current self-assessment of stress level based on a seven-scale questionnaire Nasa-TLX [19]. An Android application “Stress Tracker” was developed for this purpose, the interface of which is depicted in Fig. 3. This application displayed a request to evaluate the current stress level of a participant from 09:00 to 19:00 every hour at a random time. The time interval between the requests could not be less than 30 min.

  Fig. 3

  

  Android application used for current stress level evaluation: main screen (left side), log screen (right side)

  We used a binary classification; consequently, evaluation marked with “1, 2, 3” was classified as “low stress,” while the levels “4, 5, 6, 7” were classified as “high stress.” Self-reports about current stress level were used as ground truth to label the collected data into two above-mentioned classes. After the labeling process, the data from the “assess” table were submitted to the main database table “base_weka.” Following the initial preprocessing, the data from other tables were also submitted to the database table “base_weka.”

• (B) Audio: Ambient noise data were collected every 300 s. Time interval between measuring of ambient noise (and other parameters below) was chose with the condition that a smartphone could hold a charge for at least one full day, as the smartphone was planned to be simultaneously used for usual daily activities. The data included (1) MFCC (mel-frequency cepstral coefficients) which characterize the signal based on its spectrum and amplitude; (2) l1norm (least absolute deviations) and l2norm (least squares); (3) frequencybands, i.e., power spectral density across frequency bands; and (4) diffSecs, i.e., difference in seconds between records up to the twelfth decimal place; this parameter was not involved in the computation, but it was used to test the accuracy of event detection. After completion of the data collection, average values for 3-, 5-, and 10-min intervals were calculated for all the audio features in order to consider the short-term and medium-term impact of changes in audio features on the stress level.

• (C) Gyro: Gyroscope readings on X-, Y-, and Z-axes were recorded every 120 s. Then, average values for 10- and 15-min intervals were calculated.

• (D) Screen: Every case of changing the screen mode (on/off) was counted. Information on the number of the phone screen switching on and off for the last 10 and 20 min was submitted to the main database table.

• (E) Light: The values of the ambient light were detected every 300 s. The light values for the last 10, 20, and 30 min were submitted to the main database table.

• (F) Probe: Based on the data received from ACC, the activity level (none, low, or high) was determined. Values of the activity level for the last 30, 60, and 120 min were submitted to the main database table.

• (G) Activity: We developed an Android application for classification of the current activity (walking, standing still, on car, tilting), which is compatible with Google activity recognition API [20]. The “tilting” activity corresponds to a sedentary or standing state of the user. This application shows the type of activity, confidence, date, and time in which the event was detected. The application interface is presented in Fig. 4. The current activity was recorded every minute for the purpose of receiving more detailed information. The detected activity was recognized as true if the confidence was more than 50 %. The data were further exported to.csv format, and then, it was submitted to the MySQL database.

  Fig. 4

  

  Android application for activity recognition: log screen (left side), setting screen (right side)

The above data and its properties were selected for the stress analysis based on previous experience (according to Table 1).

Activity recognition

We decided to develop an application based on the Google activity recognition (GAR) for several reasons. Firstly, there is no need in Internet and GPS access for determination of the current activity, so such condition significantly reduces battery consumption. Secondly, this is a non-obtrusive method which does not require wearing any additional sensors by the user or any other interference due to the fact that only the data from smartphone ACC are used for the activity recognition. Thirdly, preliminary tests showed that the accuracy of determination of the mentioned activity types by this method was sufficiently high.

Nevertheless, it was necessary to interpret the resulting data on daily activities in the right way. It has been observed that in the process of data collection, there can appear several activities different from the activities block of one and the same type, not being a result of an error of the activity determination algorithm. For example, if a group of “car” blocks is followed by 1–2 units of “standing,” and then, the group of “car” blocks resumes the chain, it can be a result of stopping due to a traffic light or a traffic jam. Consequently, we could replace those 1–2 units of “standing” activity with “car” activity units, because while waiting for the enabling signal to turn on, the person is still in the car, and then continues movement. An example of such a situation is illustrated in Fig. 5 in row 999, where the fragment of the activity recognition application log is presented.

Fig. 5

Imported.csv file from the activity recognition application

Thus, we will assume that the “car” activity ends if there are at least 3 units with activities other than “car” at the moment of its last detection. However, “standing”, “tilting” (the device angle relative to gravity changed significantly. This often occurs when a device is picked up from a desk or a user who is sitting stands up [21]), “walking” activities may have blocks of 1 and 2 units of an activity. An example of this can be a case of working on the computer followed by a short coffee break. To avoid motion artifacts that affect the data collected, it was important to understand whether the user is in a fixed position, which corresponds to the “standing” type of activity.

According to [22], to use stress models with physiological responses of body in practice, we must be able to correctly detect stress even during other activities that affect one’s physiology. In this study, we were interested in understanding the possibility of application of the behavioral pattern and contextual data to accurately detect stress even during various daily activities. And there also was a question to be answered, “Will information on the current user activity help build a more accurate classifier?” The question is reasonable because according to Table 1, the field of stress determination using the data received from a smartphone as the only and sufficient source with consideration of different types of activities is not fully understood yet.

A collection of algorithms for machine learning WEKA [23] was applied for data analysis. After collection and import of all the data to the software package WEKA, three models of classification were built:

• 1. Initial model based on the analysis of behavioral and contextual data derived from a smartphone.

• 2. Second model: The data on the activity type at a certain point of time were added to the initial model.

• 3. Finally, a separate model of classification for the “standing” activity type was built. We wanted to check that can we get advantages due to building separate models for certain type of daily activity.

Results

The following four algorithms were applied for data analysis: random forest, simple logic, and J48 and REPTree (both are decision tree). Tenfold cross-validation was used for all algorithms. All data were divided into 10 different folds: one fold was used for testing and remaining nine were used for training. This procedure repeated while every fold was used for a testing. The average classification accuracy of four algorithms was calculated for each model (Table 2) for evaluation of the effectiveness of each model.

Table 2 Average accuracy for various models

Figure 6 shows the results of the classification with the use of the above three models and various analysis algorithms.

Fig. 6

Classification results with the use of various models and algorithms

Application of the second model of classification considering the current activity type made it possible to improve the accuracy of high- and low-stress state classifications by 2.3 % compared to the initial model. At the same time, when building a separate model for “standing” activity only, the classification accuracy was improved by 3.8 % compared to the initial model.

The results confirm our expectations that our behavioral pattern will vary for various activities. With due consideration to the current type of human daily activity and especially by building separate analysis models for each type of activity, we can create a more precise stress recognition system.

It must also be noted that when using the data received from a smartphone ACC only, the accuracy of 82.5 % was achieved for the second model considering daily activity, and the accuracy of 90.32 % was the result of a separate model for “standing” activity only.

Conclusion and future work

In this study, we presented preliminary results of an ongoing project on how to use smartphone sensors to detect stressful situations. For classification of high- and low-stress states, we used the data received in real life from a smartphone as the only and sufficient source. Our results showed the average accuracy of 73 % in a binary classification of high- and low-stress conditions using only behavioral and contextual data received from a mobile phone. And by application of a separate model for the “standing” activity only, we achieved the accuracy of 75.81 %. Thus, the information on daily activity can be used to build more precise separate classifiers for each type of activity.

Although these results are preliminary, they showed that the data received by smartphones can be applied for stress recognition. This method can replace widespread stress analysis methods based on physical and physiological responses of the human body to stress in cases where maximal noninvasiveness must be reached.

Our future plans include collection of more data for analysis, to execute extensions and optimizations of the algorithms, to build separate stress analysis models for various daily activities which can be determined by the application developed in this study.