1 Introduction

In modern society, stress is one of the major problems in many countries around the world. A small amount of stress in work or study can be act as positive. However, high level stress for long-term can be chronic and it can cause many chronic disease. Chronic stress can result in serious health conditions including anxiety, insomnia, muscle pain, high blood pressure and a weakened immune system.

It is important to keep monitoring mental stress level for prevention of chronic disease and early treatment. Several technologies have been developed to recognize stress level; some methods are based on physiological signals: blood pressure [24], heart rate [24], heart rate variability (HRV) [4, 24], and skin conductance [8, 20]. Activity-aware mental stress detection is also available using Bayesian network and SVM.

Continuous monitoring of an individual’s stress levels is essential for understanding and managing personal stress. A number of physiological markers are widely used for stress assessment, including: galvanic skin response, several features of heart beat patterns [4], blood pressure [4, 24], and respiration activity. Fortunately, miniaturized wireless devices are available to monitor these physiological markers. By using these devices, individuals can closely track changes in their bio-signals in order to maintain better health condition.

The technology of monitoring human physiological changes for the detection of human mental stress level has been developed rapidly. Tiny multimodal sensors can measure various vital signs, such as body temperature, pulse rate, heat rate, respiration, blood pressure, electroencephalography (EEG), electrocardiogram (ECG), and electromyography (EMG). These sensors are wearable or implanted in the body, or installed in patients’ homes and workplaces. The purpose of these biosensors is to help patients to monitor their health state by themselves and their caregivers to update patient health status in real-time using intelligent environment [13].

In this paper, we introduce a multi-level assessment model for monitoring a health/mental condition that can be used for the prediction of abnormal mental status using multi-modal bio-sensors in dynamic situation. This monitoring model considers biometrics data as well as the environmental sensor data based on our previous studies [26, 27]. The measurements are used for the bio-index generation for prediction of mental stress level. This health/mental monitoring model has four modules as follows: (1) monitoring biometrics data (body temperature, EEG, ECG, respiration rates, SpO2 and blood pressure) and environmental sensor data (location, time, weather, temperature and humidity) (2) index assessment from measurements using fuzzy logic and SVM, (3) risk assessment and rule generation using decision tree, (4) activity assessment using maximized mental stress ratio based on Expectation Maximization (EM) algorithm.

The remainder of this paper is organized as follows. The section 2 reviews the recent technology for monitoring mental/physical health status. Section 3 introduces the overview of multi-level assessment model for recommendation of wellness service. Section 4 provides the monitoring step for sensor data, and section 5 provides index assessment, section 6 addresses risk assessment. Section 7 describes activity assessment algorithm based on EM. Finally, section 8 provides results from multimodal biosensor and section 9 summarize our proposal and direction for future work.

2 Related work

2.1 Mental stress and physiological response

Stress is a physiological response to the mental, emotional or physical challenges that we experience. Small amounts of stress may be desired, beneficial, and even healthy. But excessive stress may lead to bodily harm and increase the risk of disease [18]. High level of stress can be a risk factor for hypertension, heart attacks, ulcers, mental illness, and depression [19, 23]. Continuous monitoring of an individual’s stress level is important for understanding object emotional status and managing personal stress. A number of health parameters using physiological changes are widely used for stress assessment, including: heart beat patterns [1], blood pressure [9], and respiration activity [3].

2.2 The relationship between human physiological response and human mental stress level

Because the physical health and mental health are closely connected [19], continuous monitoring of physiological changes using bio-sensors can reflect human mental stress level. The detection of human mental stress level is not easy because each person is different on their physiological response to physical, mental, and emotional stress. For the detection of mental stress level, choi et al. [3] monitored heart rate variability using nonlinear system identification. Silberstein [21] have focused on external effects on human emotions using EEG measurements for determining the level of attention of a subject to a visual stimulus. Murugappan et al. [17] classified dynamic emotional content into five discrete emotions (disgust, happy, surprise, fear and neutral) based on Electroencephalography (EEG) signal. Recently, EEG-based emotion recognition using deep learning network with principal component improved classification accuracy compared to SVM and naïve Bayes classifier [12].

2.3 Characteristics of bio-sensor data (EEG, ECG and blood pressure)

In ubiquitous healthcare age, large number of bio-sensors including smartphone is used to u-healthcare services by monitoring physical conditions [5]. With the development of wearable smart device, u-healthcare is available to everyone, everywhere in anytime [28]. Because human emotion is complex, it is difficult to detect human mental stress level. This requires updated knowledge on the human bio-signals for accurate detection of human physiological response. These bio-signals include blood glucose level, respiration rate, blood pressure, skin temperature, electroencephalogram (EEG), electrocardiography (ECG), and electromyography (EMG). However, these measurements have varied primarily due to the environmental conditions (e.g. characteristics and positioning of electrodes, nature and characteristics of equipment, anatomical minor differences, presence of glands and blood vessels, different tissue fat levels, etc.) under which they are obtained, there are commonalities among them.

3 The overview of multi-level assessment for monitoring mental wellness in dynamic situation

The work-flow of multi-level assessment for monitoring health/mental status is shown in Fig. 1, based on our previous study [26, 27]. This health condition monitoring model has four major steps, (1) data collection from biosensor and environmental sensor, (2) sensor data filtering and index assessment based on SVM (3) risk assessment for prediction of mental stress level using decision tree, and (4) decision making for wellness contents recommendation based on EM algorithm.

Fig. 1
figure 1

The workflow of health and mental status monitoring system based on EM for wellness contents recommendation system (1) Data from multimodal bio-sensors and environmental sensors are collected. (2) Biometrics data combined with environmental information are analyzed for index assessment using SVM. (3) Risk assessment step classify emotional status into positive and negative (4) EM-based decision making and recommendation of proper activity considering user preferences

Full size image

The first step is the collection of bio-sensor dataset, which extracts features from input variables for prediction of mental stress level. In this step, environmental factors, such as location, time, and weather are also considered for situation-awareness.

Index assessment step is a filtering sensor datasets and recognize a pattern of the filtered data using predefined health parameters (Table 1). In risk assessment step, bio-index for emotion sensing can classify emotional status into two statuses (positive and negative). Depending on emotional status and user preferences, proper activities (indoor activity, outdoor activity, customized diet, and exercise therapy) are recommended based on EM. For recommendation of proper activity, user preferences and profiles (age, sex, and diet) are considered.

Table 1 Bio-emotional index assessment for mental wellness using health parameters
Full size table

4 Monitoring vital sign and environmental status

In monitoring step, three health parameters (blood pressure level, heart rate, and respiration rate), which known to be related with mental stress are measured using biosensor (BMS-AE-DK). Embedded software in biosensor is used for noise filtering as shown in Fig. 2. Based on our previous study [26, 27], these data are measured repeatedly and continuously from each person. All measurements are categorized into 3 levels (high, normal, and low) based on previous studies [16] using neuro-fuzzy logic and SVM.

Fig. 2
figure 2

Window interface of multimodal bio-sensor for monitoring health condition using Bio Medical System Analog & Embedded Development Kit (BMS-AE-DK). This kit has five modules such as ECG, NIBP, respiration rate, SpO2, and bio-impedance

Full size image

4.1 Sensing emotion from measurements

In order to simplify input variables from biosensor, we use fuzzy logic which categorizes the input variables as linguistic variables. For the filtering of the patterns, the monitoring step defines the criteria for filtering using SVM algorithm. This step categorizes input variables into 3 levels (high, normal, and low) as below. Three health parameters (BP, HR, and RR) known to increase with anxiety or stress are selected based in previous studies [2, 10, 11, 22, 25].

  1. (1)

    Electroencephalogram (EEG) the recording of electrical activity on the scalp. [11, 25]

    These frequency bands from low to high frequencies, respectively, are called Delta (1–3 Hz), Theta (4–7 Hz), Alpha (8–13 Hz), Beta (14–30 Hz), and Gamma (31–50 Hz).

  2. (2)

    Blood pressure (BP) is maximum value of blood pressure during one breath. BP is known to increase with anger, fear and anxiety [10].

    • low (<90 bpm), normal (90–120 bpm) and high (>120 bpm)

  3. (3)

    Heart rate (HR) is the speed of the heartbeat and HR is known to increase with anxiety or stress [22].

    • : low (20–70 bpm), normal (50–100 bpm) and high (90–120 bpm)

  4. (4)

    Respiration rate (RR) is the number of breaths taken per time interval. RR is known to increase with anxiety. [2]

    • : low (5–10 bpm), normal (7–20 bpm), and high (15–24 bpm) : low (100–121 mm Hg) normal (110–134 mm Hg) and high (120–147 mm Hg)

4.2 Environmental sensing for situation awareness

To obtain the situational information, monitoring step collects from environmental sensors in smart device using mobile applications [6]. For accurate situation awareness, we used mobile application, which supports light sensor, proximity sensor, gyroscope sensor, accelerometer sensor, orientation sensor, sound sensor and temperature sensor. Using environmental sensors in mobile device, we inference situational information, such as weather, time, location, and temperature. For understanding user-context, knowledge from situational information is combined with physiological changes and used for bio-emotional index assessment.

5 Index assessment for the prediction of human mental stress level

In Table 1, categorized health parameters are matched to mental stress level by analyzing patterns (increase or decrease) of input variables using predefined information [7]. Based on our previous study on senior health risk [27], the health parameters are matched to mental stress level for bio-emotional index assessment. The EEG sensor represents 4 types as follows; Alpha, Beta, Theta, and Alpha/Beta [11, 25]. It is known that Alpha wave is related with relaxed state and Beta wave is closely linked with tension state. As shown in Table 1, health parameters are classified into 3 mental stress levels (tension, relax, excited, and normal). For example, when brain wave ranges in 7.5–13 Hz from EEG sensor called Beta wave and it represents a person is in tension or anxiety state. When a systolic blood pressure is high (120–147 mmHg), the mental stress level is in tension.

6 Risk assessment considering biosensor datasets and situational information

The risk assessment step defines the mental stress ratio using association rules and decision tree as shown in Fig. 3. In this step, four features (EEG, BP, HR, and RR) known to be associated with mental stress level are selected and the rules for sensing mental stress level are generated [15]. The accumulated mental stress ratio is classified into four levels, such as normal, low, middle, and high using decision tree in Fig. 3. For example, the mental stress ratio (MSR) is high as following conditions: EEG sensor represents Beta waveform and systolic blood pressure, heart rate and respiration rate are in ‘tension’ categories.

Fig. 3
figure 3

Decision tree for maximization of accumulated mental stress ratio using association rule and decision tree

Full size image

6.1 Rule for sensing mental stress ratio

For the risk assessment step, we define association rules between health parameters and mental stress level by monitoring physiological changes as below. The correlation between health parameters (input) and the mental stress level (output) is done with set of fuzzy rules. Each rule uses AND/OR connectors to connect various input factors with mental stress level. As Beta wave in EEG sensor is related with mental stress, anxiety and tension as we mentioned in Table 1, we assumed (EEG is Beta). Then, we checked the status of systolic blood pressure, heart rate, and respiration rate.

  • If (EEG is Beta) and (BP is Normal) and (HR is Normal) and (RR is Normal) then (MSR is normal)

  • If (EEG is Beta) and (BP is Normal) and (HR is High) and (RR is Normal) then (MSR is low)

  • If (EEG is Beta) and (BP is High) and (HR is High) and (RR is Normal) then (MSR is middle)

  • If (EEG is Beta) and (BP is High) and (HR is High) and (RR is High) then (MSR is high)

6.2 Evaluation of mental stress ratio using decision tree

Figure 3 represents a decision tree for the determination of mental stress ratio (high, middle, and low) from association rule. Based on our previous study [27], we have enhanced the decision tree model for measuring accumulated mental stress level. To evaluate the mental stress ratio, four levels decision-tree is constructed as shown in Fig. 3. The four features (EEG, BP, HR, and RR) have the three levels of mental stress level (such as relax, normal, and tension) as predefined in Table 1. As these four features are known to increase with mental stress, we matched high level into tension and low level into relaxed state.

In EEG level, root level, data from EEG sensor can be classified into Alpha, Beta, Theta, and Delta waveforms depending on users’ emotional status. Especially, Beta wave from EEG sensor is known to be associated with ‘anxiety’ or ‘tension’ status. In second level, when BP is higher than 120 mmHg, MSR becomes low. In third level, the heart rate is checked. When the HR is higher than 84 bpm, MSR becomes middle. In Last level, respiration rate is checked if higher than 15/s. In this case, MSR evaluate high.

7 Activity assessment algorithm based on EM

To recommend the proper activity, we propose an activity assessment model based on Expectation Maximization (EM) algorithm to suggest a proper activity for users’ mental wellness. The EM is a method for finding maximum likelihood estimates of parameters from unobserved latent variables [14]. The latent variables (maximized mental stress ratios) are variables that are not directly observed. But these variables are rather inferred from other variables such as objects’ physiological changes, time, weather, and location type. For this purpose, the activity assessment step defines a mental stress ratio EM algorithm that uses an accumulated mental stress ratio. This algorithm has extended the senior health risk ratio EM algorithm on previous study [27]. As described in Algorithm 1, the mental stress ratio EM model consists of two steps such as expectation step (E-step) and maximized step (M step).

Algorithm1 Mental stress ratio EM for decision making based on mental stress level

Input:

  a sequence of MentalStressRatio (Normal, Low, Middle, and High)

    currentLL ← ∞

repeat prevMSR ← currentMentalStressRatio

    calculate MSR

 until

        check max MSR < ε

        If (EEG is not Beta) then (Normal)

        elseIf (BP is tension) then (MSR is Low)

        elseIf (HR is tension) then (MSR is Middle)

        else (RR is tension) then (MSR is High)

 Output:

        Recommend activity based on MSR level

        If (MSR is Normal) then (recommend indoor activity)

        If (MSR is Low) then (recommend outdoor activity)

        If (MSR is Middle) then (recommend customized diet)

        If (MSR is High) then (recommend exercise therapy)

In E-step, the mental stress ratio is repeatedly updated into input variables to understand the user’s mental stress level in dynamic situation. The maximization for mental stress ratio is calculated with cumulative mental stress ratio based on number of ‘tension’ in index assessment step in Algorithm 1.

In M-step, by using the MSR, a proper activity is recommended. For activity recommendation, a mapping table is defined as shown Table 2. So, activity recommendation is accomplished by using the mental stress ratio (MSR) level.

Table 2 Activity recommendation depending on MSR
Full size table

8 Results

Using multimodal biosensor, blood pressure, heart rate (ECG), respiration rate, and heart rate are measured repeatedly and data are collected from four persons as Table 3 and Table 4. Sensor data is analysed with embedded software and pre-processed for multi-level assessment for wellness service.

Table 3 Profiles of subjects for measuring
Full size table
Table 4 Measurements from multimodal biosensor and activity recommendation based on MSR
Full size table

Non-invasive blood pressures (NIBP) from four persons were measured as Fig. 4. Person D shows increased blood pressure as BP shows 213 mm Hg and DBP shows 170 mmHg. MAP is the mean arterial pressure.

Fig. 4
figure 4

Non-invasive blood pressure in systolic, mean arterial pressure (MAP), and diastolic

Full size image

The heart rates from ECG sensor are measured from four persons as shown in Fig. 5. Person D shows increased heart rates compared to other persons. If person D represents Beta wave in EEG sensor, person D has middle level of mental stress ratio based on decision tree (Fig. 3). Because person D shows high blood pressure and increased heart rate with Beta wave in EEG sensor.

Fig. 5
figure 5

Heart rates are measured from ECG sensor

Full size image

The respiration rates are measured from four people as shown in Fig. 6. Person A showed increased respiration rate and person B, C, and D show normal respiration rate. As person A shows increased respiration rate, normal range of heart rate and respiration rate. Person A is classified into low level of mental stress ratio based on Table 3.

Fig. 6
figure 6

Respiration rate are measured from four person

Full size image

As a conclusion, the mental stress ratio (MSR) in four persons is described as Table 3. Depending on the MSR, a proper activity is recommended to user for wellness service. Because person B and C showed normal range of NIBP, HR, and RR, MSR is assigned as ‘Normal’ and ‘Outdoor activity’ is recommended. As person A shows low level of MSR, ‘Indoor activity’ is recommended for user. Person D shows high level of NIBP and HR, MSR is assigned ‘Middle’ and ‘Customized diet’ is recommended to user.

9 Conclusion

This paper outlines multi-level assessment (risk, index, and activity) for the prediction of mental stress level by monitoring health parameters such as EEG, BP, HR, and RR in dynamic situation. In index assessment step, SVM algorithm was used for categorization of input variables into relax, normal and tension status. In risk assessment step, decision tree was used for classification of bio-emotional index assessment and prediction of mental stress level. In activity assessment step, EM algorithm was used for decision making based on mental stress ratio (MSR) and used for recommendation of proper activity. This paper classifies mental stress ratio (MSR) into 4 levels, such as normal, low, middle and high. For further studies, more accurate sensing with advanced biosensors is required for classification of complex emotional sensing. Customized healthcare service for mental wellness is also required using smart healthcare system.