A Study of Human Behavior and Mental Workload Based on Neural Network

  • Lan Xiao
  • Jing QiuEmail author
  • Jun Lu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9754)


Human mental activities could be displayed by human behavior, which are observable directly in work environment. In current study, a method based on human behavior not directly related to task execution in work is proposed to assess the workload in mental work situations. Ten subjects were recruited and asks to perform various levels of a mental task. The link between human behavior and mental workload for four mental tasks completed on a computer were studied based on Neural Network. The result indicates that the relationship between human behavior and mental workload could be well described in a non-linear model.


Human behavior Mental workload Neural Network 

1 Introduction

Mental workload is a crucial issue of the ergonomic research for human mental work. In general, the wide-accepted assessment and measurement of the mental workload mainly include the following 4 methods: subjective questionnaires, physiological measures, performance or errors, and task-performing-related body actions measure. However, each of them could only reflect one aspect of the human mental states. In some respects, there are a number of limitations for the intrusiveness and discontinuity in these methods (O’Donnell and Eggemeier 1986; Wierwille and Eggemeier 1993; Meshkati et al. 1995; Farmer and Brownson 2003; Cain 2007). The results of some previous studies indicated that the specific facial muscle activity are correlated to mental status (Veldhuizen et al. 2003; Capa et al. 2008). Based on the performance and error for workload measurement, the performance of both primary and secondary tasks has been analysed. The performance of primary and secondary tasks is dependent on the strategy of task performance. O’Donnell and Eggemeier (1986) pointed out that underload may enhance performance and overload may result in a floor effect. Task-performing body behaviour only reflects the input to and the output from the person, rather than the entirety of internal activities in the brain. Moreover, the effect of the workload on an operator in a mental work setting is generally interpreted by the integration of all of the measurements used.

With gradually increasing automation in the human-machine system, the role of an operator has been moved to supervisory controller from manual worker (Sheridan 1987). Hence, motor actions of the operator are being gradually decreasing and may only be episodic, while sensory-perceptual and mental activities are increasing (Bedny and Karwowski 2006). Thus, it can be difficult to observe task-performing body behaviors in such work settings due to their infrequent occurrence. Hence, the total activity (both mental and external activities) of the operator would not be entirely revealed if only their sensory-motor actions were analysed. Currently, the analysis of actions directly involved in the test tasks, such as eye movements and gross motor actions, have been widely and comprehensively accepted for workload assessment. Non-task-performing and nonverbal body behavior (e.g. face behavior) occur to accompany and support task-performing body behavior. Although human-machine interaction is different from human-human communication, nonverbal behavior can be elicited by machines during human-machine interaction in the close-loop human-machine system, which is similar to human-human communication (King 1997). Hence, the same behavior in terms of rates for tasks and types can occur as in human-human communication, where facial expression is the most innate (Argyle 1998, cited from King 1997). Furthermore, as Feyer (2007) concluded that both task performing and non-task performing body behaviours (activities) makes an influence on the interactions and human performance. Also, observable body behaviour can reflect mental processes (Keijze 2005). To date, face behavior, as one part of body behavior, has not gained much interest in workload assessment in mental work settings. The studies of King (1996; 1997) are relative early for the effect of cognitive activity on facial expression. King (1997) made a survey of facial expression in human-computer systems. The results showed that rates of facial expressions were higher for computer-based tasks. Based on his study, King (1996) argued that facial expressions were related to cognitive activities based his study. In the model of Guhe et al. (2006), mouth openness is a parameter for the predictor of mental workload. Recently, a study by Stone and Wei (2011) indicated that facial expression can be a effective index of workload for arithmetic tasks. They used FACS (Facial Action Coding System) approach by Ekman and Friesen (1978) to code face behavior. However, this face coding approach needs a well-trained coder and is very time-consuming. Meanwhile, the relationships between the facial muscle activity and workload were investigated in previous studies. The results of these studies indicated that the specific facial muscle activity are correlated to mental status (Veldhuizen et al. 2003; Capa et al. 2008). They showed indirectly the correlation between face behavior and workload in the mental settings.

In current study, the range of the human activity had been expanded to contain the head position and the upper body skeleton. Human behavior (head position, face behavior and the upper body skeleton), as an activity which is easy to observe, was used to be a parameter to verify the correlation between the human behavior and the mental workload.

The previous studies have used the linear model or the simple non-linear model to describe the link of the human behavior and the mental workload, which is not entirely linear. Hence, the purpose of this study was to figure out the relationship between human behavior and mental workload using the Neural Network algorithm.

2 Method

2.1 Subjects

A total of 10 subjects (male, mean age: 22.4 years, SD of age: 1.1 years) were recruited for the study from our university setting. They were all right-handed and have various experience on computer games. And all of them reported that they had no mental or physical health problems or diseases.

2.2 Experimental Design

Subjects would be given 4 typical psychological tests which were selected to simulate various mental activities. The 4 tests were used to represent 4 four stages of information processing (information acquisition, information analysis, decision selection, and action implementation). So the amounts to the whole workload would be affected when the major factors based on information processing changed.
  • Digit span task (DIG). DIG is a simulation of information acquisition. A series of numbers are sequentially displayed on the screen. Participants were required to duplicate the number sequence in a limited amount of time (10 s) after all the numbers were displayed. The length of the next number sequence was increased by one digit upon completion of the current task. On the other hand, the length of the next number sequence was reduced by one digit when the study participant failed in the current task. The length of the number sequence was 3–6 digits in level 1 (easy task) and 7–10 digits in level 2 (complicated task). The interface of the DIG task is shown in Fig. 1(a).
    Fig. 1.

    Snapshots of each task. (a) DIG. (b) TIME. (c) TOL. (d) DEXT

  • Time wall task (TIME). TIME is a simulation of information analysis. A dot was shown moving downward from the top of the screen to the bottom at a constant speed. The screen was divided into two parts; the dot in upper two-thirds of screen was visible. However, the lower one-third screen was covered by “red wall”, which renders the dot invisible. The study participants were asked to observe and analyze the speed of the dot as it moves downward, estimate the exact time that the dot reach the bottom, then press the space bar in the keyboard. The speed of the moving dot was randomly generated. Furthermore, the speed of the moving dot was higher in level 2 (complicated task) than that in level 1 (easy task). The interface of the TIME task is shown in Fig. 1(b).

  • The Tower of London test (TOL). The TOL simulates decision selection. Various disks of different colors that represent target styles and muddled styles were shown on the screen. The study participants were asked to put a disc across the muddled style to target another disk as it moves. However, only one disk was moved each time by clicking a mouse, and the task should be completed within a given time. The number of disks was 3–4 in level 1 (easy task) and 5–6 in level 2 (complicated task), and these randomly appeared at each task. The interface of the TOL task is shown in Fig. 1(c).

  • Dexterity task (DEXT). The DEXT simulates decision selection. A jumping green dot and a big circle were displayed on the screen. The study participants were asked to try their best to control and place the small jumping dot at the center of a big circle by using a mouse. The jumping range of the dot was determined by an elasticity function. The jumping range was wider in level 2 (complicated task) than that in level 1 (easy task). The interface of the DEXT task is shown in Fig. 1(d).

2.3 Experimental Equipment and Material

The data of the face behavior and the upper body skeleton is captured by Kinect, a low-cost sensor could be a tool with continuous, and supplementary approach function. There were two Kinect sensors were used to capture body behavior, one for face behavior and the other for upper body behavior. The data includes the rotational position of the head, 6 Animation Units (AUs), 11 Shape Units (SUs), the distance of the palpebral superior and the palpebral inferior and the angle of the upper body and the level ground with respect to the gravitational field.

The top view of laboratory structure and a snapshot of the lab structure as shown in Fig. 2, In order to collect the subject’s face behavior and upper body skeleton, one of the device placed in front of the subject. Another device placed the left side of the lab to capture the subject’s body behavior, it can analysis the change of human body behavior through infrared. It also can convert each joint data into space coordinates and record the subject’s body behavior in real-time.
Fig. 2.

Lab structure

2.4 Procedure

The experimental procedure started with an explanation of the tasks and the procedure to each subject. First, the subject was required to fill in a biographical questionnaire. Then, the subject was seated on the chair and asked to find his/her normal seating position for working in front of the display by moving the table and adjusting chair or display. The subjects practiced the four tasks until achieving a constant performance. After the training period, the subjects were asked to fill out a questionnaire regarding their state of mind using the positive affect/negative affect scale. Each task lasted 4 min. The 4 tasks were performed by the subjects. The order of tasks was randomized for each of the subjects. At the end of the experiment, the subjects were asked to assess their state of mind again.
Fig. 3.

Kinect coordinate system

2.5 Data Collection and Analysis

As mentioned earlier, a Kinect sensor is very low-cost compared with other commercial tools for face behavior analysis. The Kinect sensor can track the head pose and face behavior. A right-handed coordinate system is used to quantize tracking results1 in the Kinect. The origin of the coordinate system is located at the Kinects optical center. Z axis is pointing towards a user and Y axis is pointing up, as Fig. 3 shows. The X, Y, and Z position (rotation and translation) of the users’ head are captured as well based on the Kinect coordinate system. 87 two dimensions and 13 additional points on the face can be tracked. A Kinect sensor can also provide a subset defined in the Candide3 model2, which are weights of six Animation Units (AUs) (Fig. 4). The means and SDs of AUs of each task levels were used for statistical analysis.
Fig. 4.

Definition of AUs in Kinect

The distance between upper and lower lids and the angle of spine has been calculated in current study as two parameters.

Neural Network Toolbox provides functions and apps for modeling complex nonlinear systems that are not easily modeled with a closed-form equation. In the current study, we use the neural network pattern recognition tool in Matlab software and the neural net fitting tool to analyze data, and the statistical analysis was performed using SPSS 21 software with the significance level set to 0.05. The Repeated Measurement Test was used to analyze the recognition rate on subjects.

3 Results

Using the neural net fitting, the relationship between the facial behavior and the workload calculated based on the NASA-TLX be fitted perfectly, as Fig. 5 shows.
Fig. 5.

Regression of the workload

The link between human behavior and mental workload for four mental tasks completed on a computer were studied. In the neural net pattern recognition, according to the confusion matrix, the correct recognition rate of the task difficulty with the face expression could be 90 % above (shown as Fig. 5). All of recognition rates were over 90 %. From the Fig. 5, it can be observed that average rate in DEXT task was greater than those of the other tasks. Analyzed by Repeated Measurements, there is no statistically significant difference among the tasks for the recognition rate. However, recognition rates were significant differences amongst individuals. (F = 24202, p = 0.0000).

And the recognition rates of the task difficulty with the upper body skeleton, is shown as Fig. 6. From the Fig. 6, the average recognition rate in the TOL task was greater than those of the other tasks. For the recognition rates, there is no statistically significant difference among the tasks, but significant differences between individuals (F = 294.2, p = 0.0000) (Fig. 7).
Fig. 6.

Recognition rates of the task difficulty for the face expression

Fig. 7.

Recognition rates of the task difficulty for the upper body skeleton

Although previous literature of mental workload indicated that workload measurement could be described with directly and contiguously observable non-task-performing behaviors, the workload measurement still has a few challenges. In the present study, the relationships between task difficulties and human behavior were studies during simulated mental tasks. Except for the upper body behavior, the body behavior should be studied more in its entirety to measure workload, e.g. the subjects’ shoulder breadth while they were working will be a feasible parameter. The results of the present study show that the differences in human behavior were well discernible for midrange workload levels. It shows the possibility that non-task-performing behavior is nonlinearly correlated with workload as Bedny et al. (2000) indicated. To rich the method for measuring workload through body behavior analysis, further research is required.

4 Conclusion

In this study, the relationships between human behavior and task demand were investigated using the Kinect-base tracking method. Facial expression and upper body skeleton were quantitatively analyzed using a video-based analysis system. In the neural net pattern recognition, according to the confusion matrix, the correct recognition rate of the task difficulty, could contain a high level, and the relationship between the facial behavior and the workload calculated based on the NASA-TLX be fitted accurately by the neural net fitting. The result indicates that the relationship between human behavior and mental workload could be described well in a non-linear model. Based on these findings of the current study, human behavior may be used as an indicator of the workload in mental work settings. It is recommended that different levels of the same type of mental task be used in further research to develop a supplementary method for workload assessment based on human behavior.


  1. Argyle, M.: Bodily Communication C, 2nd edn. Methuen & Co., Ltd., London (1998)Google Scholar
  2. Cain, B.: A Review of the Mental Workload Literature. Technical reports, RTO-TR-HFM-121-Part-II, Defence Research and Development Toronto, Canada (2007)Google Scholar
  3. Capa, R.L., Audiffren, M., Ragot, S.: The interactive effect of achievement motivation and task difficulty on mental effort. Int. J. Psychophysilogy 70, 144–150 (2008)CrossRefGoogle Scholar
  4. Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  5. Famer, E., Brownson, A.: Review of workload measurement, analysis and interpretation methods. European Organisation for the Safety of Air Navigation 33 (2003)Google Scholar
  6. Feyen, R.G.: Bridging the gap: exploring interactions between digital human models and cognitive models. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 382–391. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  7. Keijzer, F.: Theoretical behaviorism meets embodied cognition: twotheoretical analyses of behavior. Philosophical Psychology 18, 123–143 (2005)CrossRefGoogle Scholar
  8. King, W.: Reflections of thought: cognitive facial expressions in the human interface. In: Proceedings of the 5th IEEE International Workshop on Robot and Human Communication, Tsukuba, Japan, 11–14 November, 1996Google Scholar
  9. King, W.: Human - computer dyads? A survey of nonverbal behavior in human-computer systems. In: Proceedings of Workshop on Perceptural User Interfaces (PUI 1997), Banff, Cananda, Los Alamitos, CA, 19–21 October, 1997Google Scholar
  10. Meshkati, N., Hancock, P.A., Rahimi, M., Dawes, S.M.: Techniques in mental workload assessment. In: Wilson, J., Corlett, E. (eds.) Evaluation of Human Work: A Practical Ergonomics Methodology, pp. 749–782. Taylor and Francis Ltd., Bristol (1995)Google Scholar
  11. O’Donnell, R., Eggemeier, F.T.: Workload assessment methodology. In: Boff, K., Kaufman, L., Thomas, J., Boff, K., Kaufman, L., Thomas, J. (eds.) Handbook of Perception and Human Performance, pp. 41–49. Wiley, New York (1986)Google Scholar
  12. Sheridan, T.: Supervisory control. In: Salvendy, G. (ed.) Handbook of Human Factors, pp. 1243–1268. Wiley, New York (1987)Google Scholar
  13. Stone, R.T., Wei, C.-S.: Exploring the linkage between facial expression and mental workload for arithmetic tasks. In: Proceedings of 55th Annual Meeting of the Human Factors and Ergonomics Society, Las Vegas, USA, 19–23 September, 2011Google Scholar
  14. Veldhuizen, I.J.T., Gaillard, T., De Vries, A.W.K.J.: The influence of mental fatigue on facial EMG activity during a simulated workday. Biol. Psychol. 63, 59–78 (2003)CrossRefGoogle Scholar
  15. Wierwille, W.W., Eggemeier, F.T.: Recommendations for mental workload measurement in a test and evaluation environment. Hum. Factors 35, 263–285 (1993)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of Mechatronics EngineeringUniversity of Electronic Science and Technology of ChinaChengduPeople’s Republic of China

Personalised recommendations