1 Introduction

The mental workload (MWL) of employees using computers is significantly increasing as our information-based society progresses. Thus, it is very important for workers to evaluate their MWL to prevent and reduce human errors and their associated health hazards [1,2,3,4,5].

The autonomic nervous system is active under various degrees of MWL. In previous studies, temperature differences between the nasal and forehead regions of the face were used for MWL evaluation and estimation [6,7,8,9,10,11,12]. Thus, nasal skin temperature (NST) is said to be a reliable indicator of autonomic nerve activity. Moreover, autonomic nerve activity has little effect on forehead temperature; therefore, temperature differences between the nasal and forehead regions of the face have been shown to be a good indicator of autonomic nerve activity (along with other physiological indicators such as EEG and heart rate) [13,14,15,16,17,18,19,20,21,22].

The above approaches did consider temperature changes in other parts of the face. Thus, further research sought to estimate and evaluate autonomic nerve activity using thermal images of facial skin, measured by infrared thermography, to comprehensively evaluate the examined facial areas. Results demonstrated that in addition to the nasal region, the area around the lips displayed temperature changes under various MWL conditions. Moreover, large variations were not observed in the average temperature of the entire face, owing to the influence of MWL when comprehensively measuring particular areas of the face [23].

Thus, it was assumed that the total amount of blood flowing to entire face to remain unchanged; however, flow to the nose decreased, and flow to lips is increased, depending on changes in MWL. This indicates that the average change in overall face temperature was minimal, even as temperature changes in specific regions (such as the nose and lips) were significant. In other words, the deviation between temperature change of local region of face and average temperature at face area overall is large. Hence, we propose a novel method using variances in overall facial temperature in this paper. This method enables temperature change information from areas other than the nose to be captured, thereby increasing evaluation and estimation accuracy at higher sensitivity levels than conventional methods. In addition, we examined whether further high-precision evaluation and estimation was feasible.

2 Autonomic nerve activity estimation algorithm

In this paper, we aim for absolute MWL evaluation by obtaining a single thermal facial image. This study also proposes an algorithm to reproduce autonomic nerve activity, with the aim of absolute MWL evaluation, using a single thermal image of the face. As our study is based on the average temperature of an entire region (that undergoes little change owing to MWL), our proposed algorithm will estimate autonomic nerve activity.

2.1 Method of calculating the estimated values

Skin temperature gains are represented as per-pixel values by plotting the thermograph on a digital image. The average temperature of the entire face is calculated based on the pixels configured to form it. Likewise, the temperatures of localized regions are calculated based on predetermined regional pixel configurations, and the sums of the differences between the average and the local values are then calculated. Temperature shifts caused by changes in autonomic nerve activity were examined to determine whether they can be measured. Finally, the calculation formula for the average temperature and temperature differences for each region is obtained using the following equation:

$$F(x,y)=\frac{1}{n}\mathop \sum \limits_{1}^{n} {(T - {X_i})^2}$$
(1)

where T is the average temperature, Xi is a temperature value for each pixel in the region, and n is the number of pixels in the area. By squaring the reference and the difference of the temperature, the effect of the sign is taken into account.

Figure 1 shows a model of temperature changes for the nasal and oral regions of the face before and after MWL stimuli. As noted in [23], the temperature around the nasal region decreases, and the temperature around oral region increases, under the effects of MWL changes. We evaluated the MWL using the estimates obtained for each of these two regions, as shown in Fig. 1 (variances near the nasal and oral regions).

Fig. 1
figure 1

The whole face of the variance

Full size image

2.2 Evaluation method

2.2.1 Method 1

We investigated the general relationship between mental state and each proposed physiological index by comparing the psychological indicators and physiological indices that underwent changes as a result of the imposed calculation problem tasks to determine the MWL. The correlations between the visual analogue scale (VAS) [24] and each of the physiological indices were determined using Pearson’s correlation coefficient and the mean values of all subjects.

2.2.2 Method 2

Next, we investigated the relationship between mental state and each proposed physiological index within the subject using multiple regression to determine the variations between psychological indicators and physiological indices. We used VAS as the outcome variable, and each physiological index and subject as predictor variables. Subjects were treated as categorical factors using a dummy variable with eight degrees of freedom. The value from the t test for the regression slope of VAS was used to determine the probability of the analysis. The magnitude of each correlation coefficient between VAS and each physiological index within subjects was calculated as the square root of (sum of squares for VAS)/(sum of squares for VAS + residual sum of squares). The sign of the correlation coefficient was given by the sign of the regression coefficient for VAS.

3 Experiments

Experiments were carried out to acquire facial thermal images (FTIs) under different MWL conditions; subjects were required to carry out a MWL task.

3.1 Experimental protocol

Thermal images were taken when a subject solved a mental arithmetic calculation. Figure 2 shows the experimental system. An infrared thermography imager (View Ohre IMAGING XA0350) was placed at a distance of 1 m horizontally from the nose of the subject. The thermal image size was 320 × 240 pixels and the sampling period was 1 s. A PC display and numeric keypad were placed upon the desk.

Fig. 2
figure 2

Experimental system

Full size image

According to previous studies [3], accurate temperature measurement from thermographic images becomes difficult as the edge of the object to be photographed is prone to noise. Therefore, we performed preprocessing for MWL evaluation. To reduce edges, the images in Fig. 3a were omitted, and the images of the ocular, nasal, and oral regions in Fig. 3b were used for analysis.

Fig. 3
figure 3

Analysis region

Full size image

Figure 4 shows the experimental protocol. Subjects rested in a sitting position for 3 min. After this initial rest period, subjects began the mental arithmetic calculation task and continued it for 10 min (as the MWL). After completing the task, subjects again rested for 3 min, thereby completing the experiment. The calculation involved the addition of two integers, each of which was between 10 and 99. The subject input the answer of the calculation displayed on the PC using the numeric keypad. The calculation was displayed for 3 s, after which, the following calculation will display on the PC, regardless of whether the subject answers the calculation within 3 s. Seven facial thermal images were taken at 0, 2, 4, 6, 8, and 10 min after resting and used as physiological indices.

Fig. 4
figure 4

Experimental protocol

Full size image

In addition, the profile of mood states (POMS: Japanese version) [25] and VAS were used as psychological indices. VAS can measure subjective senses and feelings by marking positions on a 10-cm-long scale characterized by a pair of opposite words or phrases at the both ends. This method operates with few individual differences in the understanding of the description and produces measurements in a relatively short time frame. In this experiment, four pairings of “unpleasant–pleasant” were employed for VAS. POMS is a self-assessment questionnaire method of evaluating emotional states. Questionnaires are presented with 65 items that express feeling and emotion, and answers are made with five evaluation criteria. Based on the answers, evaluations fall within six mood scales: tension–anxiety, depression–dejection, anger–hostility, vigor, fatigue, confusion.

The subjects were eight healthy adults whose age ranged from 22 to 27 years; these subjects were well rested the night before the experiment. For each subject, POMS was written before and after the experiment; VAS was written at 0, 2, 4, 6, 8, and 10 min after the start of the experiment and after the break, respectively.

3.2 Results of measurement

First, we examined the estimated possible face area of the autonomic nerve activity. The thermal images before and after the mental arithmetic calculation task are shown in Fig. 5, moving from the upper image to the lower image, for each subject. The design of the task accounted for the fact that the temperature in the vicinity of the nose and lips changes significantly.

Fig. 5
figure 5

Thermal image before (upper image in each subject) and after (under image in each subject) MWL (before preprocessing)

Full size image

Following this, an analysis region was extracted from seven different thermal images of the MWL state, which were acquired every 2 min and were investigated using the average of the regions (b) to determine the change. Figures 6 and 7 shows the average value and standard deviation of the time series data for the average temperature of each subject. In Fig. 6, we can see differences in the average temperature of each subject; however, the standard deviation was within 1 °C. As shown in Fig. 7, large variations were observed in the average temperature of the entire area owing to the influence of the MWL. Blood flow is expected to flow to the nose and in the vicinity of the lips (depending on the MWL); thus, we considered the blood flowing to the entire face to remain unchanged. As a result, the average change in temperature was smaller than expected.

Fig. 6
figure 6

Average value of face area temperature

Full size image
Fig. 7
figure 7

Standard deviation of each local region

Full size image

The results of POMS before and after the experiment are shown in Fig. 8. Per the POMS results, the calculation of the Wilcoxon signed-rank test showed that the scale of depression (D) was significantly decreased compared with pre-experiment data (P < 0.01). Thus, we can conclude that the calculation task used this experiment functioned as an ideal MWL task.

Fig. 8
figure 8

Score of POMS and SD

Full size image

3.3 Results of evaluation and discussion

Our experimental results show that temperature decreased around nose and rose at around the lips. Therefore, we believe that blood that was supposed to flow to the nose flowed to the periphery of the lips owing to sympathetic nerve activity. If so, blood flow returned around the oral region to the nasal region owing to parasympathetic nerve activity, meaning that the opposite temperature movement occurred in direct opposition to this result. Until now, NST has been regarded as an indicator that reflected sympathetic nerve activity within autonomic nerve activity. However, our algorithm can also measure the influence of the parasympathetic nerve as temperature changes around lips are taken into account. Thus, it is possible to perform a more in-depth evaluation of all autonomic nerve activity using our proposed method.

As shown in Table 1, Evaluation Method 1 found significant correlation between some VAS data and each physiological index. In particular, the correlation of “unpleasant–pleasant” was observed. These results show that the NST obtained using our proposed method represents a high hedonic of the general relationship in psychology. In addition, Analysis Method 2 also showed a significant correlation of “unpleasant–pleasant”. Other statistically significant differences, including the pleasant–unpleasant correlation of VAS items, were also observed. This indicated that the P value was lower than the NST. Thus, for each subject, the correlation that was statistically determined using the proposed method was higher than that obtained with the evaluation method and NST from previous studies.

Table 1 Result of analysis among each condition
Full size table

4 Conclusion

Our study aims to perform a complete MWL evaluation using a single thermal image of the face. Thus, we proposed a novel algorithm to evaluate autonomic nerve activity as a whole by measuring not only sympathetic nerve activity but also parasympathetic nerve activity. The results showed our proposed method to be highly accurate when compared with the NST evaluation method (which only measured sympathetic nerve activity) proposed in previous studies.