1 Introduction

The ‘Smart Virtual Worker’ project has the goal to simulate work tasks in order to either help to improve existing workflows or by supporting the planning stages during the conceptualization of upcoming task sequences. This on one hand to improve the working conditions in general. But since in almost all the industrialized nations the workforce is ageing, companies will have to find ways of keeping their existing, highly skilled and qualified employees who might not be able to perform as well due to their age or medical conditions. Therefore, the SVW simulator is an easy to use software which is capable of replicating established workflows and to find alternative routes or task sequences. Although there are solutions which compute ergonomic parameters or environmental information, the emotional tendency of the workers is often overlooked. Therefore we implemented a dynamically adaptable emotion model (see Fig. 1), which is capable of differentiating between 27 distinct virtual worker types and attributes. Based on established psychological models [1–3] the computation is not based on fixed tables of corresponding emotions but allows for a unique and timely calculation of a psychological state [4]. Due to its modular programming, the emotion module itself is capable of working as a stand-alone version and is actively pursued as a way of enhancing human-robotic-interactions due to facilitating an emotional insight of a human’s behavior for the robotic counterpart [5]. One of the main criticism regarding computational emotional architectures is the missing evaluation or at least real-world analysis of the computed data [6]. The presented paper describes our performed and ongoing process of analyzing and evaluating the model with real life counterparts.

Fig. 1.
figure 1

The flow-chart of the SVW emotion module [4]

Full size image

2 The Emotion Model

Fitted into the overarching architecture of the Smart Virtual Worker Simulation, the emotion model itself is a standalone module, capable of operating, to the greatest possible extent, on its own. Within it carries a rudimentary virtual agent which is characterized by three individualizing variables: Constitution [C], Sensitivity [S] and Experience [E]. These three allow for 27 distinct agents to perform within the simulation. A low constitution, e.g. a much older simulated worker, might be able to perform very well still, due to a higher experience.

From left to right, the emotion model takes in the proposed body movement sequence by the motion generation module. This motion has itself already been evaluated by the ergonomic module which in turn rates it as being feasible, precarious or, in extreme circumstances, as alarming. These input variables are henceforth computed by the module, while incorporating the agent’s characteristics (C, S and E), and three scales are adjusted. One for the sympathetic arousal (serving both as energy for the upcoming emotion and to facilitate an emotional transfer), another for the exhaustion and the emotional valence. As a result, the valence is either positive or negative which, together with the exhaustion factor, is handed over to the artificial intelligence, which, due to its reinforcement learning architecture, makes a decision about upcoming movements and task related actions of the simulated worker.

Over the course of last year two preliminary experiments (N1 = 2, N2 = 6) have been conducted leading up to an evaluation (Neval = 8) in November of 2014. The goal of the first two experiments was twofold: First, to gather analytical data of subjects’ heart-rate, endurance, speed and emotional state in order to calibrate the emotion module. And we conducted a second experiment to test the then calibrated module and to continually fine-tune its computational routines.

3 Pre-tests

For the analytical part, a two-day experimental setup has been developed. During day one, the test subjects were analyzed regarding their personal constitution and state of mind. This was done by having them answer a questionnaire to determine their experience in carrying heavy items, their usual workout routine, previous employments and questions about their assumed behavior during a hypothetical work related scenario. Afterwards, physiological measurements were gathered, namely running (starting with 100 m, followed by a five minute break, ending with running a 400 m round course) and lifting as well as continuously holding up a 5 kg weight as long as possible. Furthermore, their weight, height and further socio-demographic data was collected. During day two, the subjects had to carry differently weighed boxes (Experiment 1: 5 kg, 10 kg, 15 kg) over a distance of 30 m, regaining some strength by walking 30 m back without a box and continuously repeating these steps, until 10 boxes of each weight (30 boxes total) were carried across (see Fig. 2).

Fig. 2.
figure 2

The camera recording setup of the experiments

Full size image

During the experiment, the multi-dimensional-mental-state-questionnaire (MDBF [10]) was used to determine the subject’s emotional valence. Furthermore, their individually felt exhaustion was checked on a colored scale, ranging from green over yellow to red. In addition, their heart-rate, number of steps, speed and time was monitored electronically. To be able to check for specific emotionally stressful moments during the work task, and to be able to check for ergonomically critical movements, the whole experimental setup was recorded by five cameras. Two were monitoring the box pick-up and put-down spot from the side to check for precarious degrees between the upper and lower body. Two more were facing the subjects while they were carrying the boxes to their destination, which would allow for an analysis of their facial muscle activation, using the Facial Action Coding System by Eckman [7]. The fifth camera was used to record an overview of the experimental situation.

Afterwards, the data was analyzed and the individual test-subject’s endurance-index [8] computed. Correlations between the heart-rate and the individual’s exhaustion were calculated and presented a strong (rVPN1 = .5) and medium (rVPN2 = .37) coefficient. This allowed us to transfer the specific differences of the exhaustion scale during carrying and recuperation over into the exhaustion model of the virtual worker.

The individual’s decisions were henceforth simulated within the SVW Framework and consequently matched to the observed biophysiological changes. In addition, the analyzed emotional valences of the multi-dimensional-state-questionnaire were calculated and the model’s computational routines were adapted until the emotional valence and the measured data correlated on an extremely high level (rVPN1 = .93, R2 = .87; rVPN2 = .96, R2 = .92) (see Fig. 3).

Fig. 3.
figure 3

The computed and reported strain while carrying the boxes

Full size image

In order to validate these initial results we performed a second run of pre-tests (N = 6) with a change in the weight of the boxes. Since the first test showed only a marginal strain on the observed exhaustion when carrying 5 kg weighed boxes the available weights were now 10 kg, 15 kg and 20 kg. Otherwise the experimental setup remained the same with a two-day format. Due to this we were able to observe a much higher workload during the 20 kg episodes.

4 Evaluation of the Box Carrying

Based on the results from the pre-tests, the evaluation took place inside an empty factory building and the participants (N = 8) were compensated for their participation. The work space allowed for a distance of 20 m to carry the boxes from the pre-filled cupboards on the right side to the empty one on the left side. We used sand-filled plastic boxes which were weighed in differentiating increments, ranging from 10,2 kg up to 21,9 kg. Each cupboard held six boxes so 18 boxes had to be carried (see Fig. 4) back and the same 18 had to be returned (resulting in a total of 36 carried boxes, each weight twice).

Fig. 4.
figure 4

The experimental setup of the evaluation with cameras

Full size image

Due to the nature of the experiment the participants were tested sequentially. Each iteration began with an instruction of the participants, especially regarding their physical well-being. Afterwards the biophysiological equipment (Heart-Rate, Pedometer, Skin Conductance and EMG) was applied to individual parts of their body and initial calibrations of the equipment performed. Then they filled out the questionnaires already used during the pre-tests, allowing us to rate them regarding their constitution, sensitivity and experience of the upcoming work task.

Afterwards, they were led to the pre-filled cupboard. As in the pre-tests, they were free in choosing which weights of a cupboard to carry first, but that they would have to carry them from top left to bottom right and sort them back in their original order on the other side. Once a box has been placed there, they would have to walk back to the filled cupboard to fetch the next box. Before they would pick up one of the two boxes on the shelf, a confederate asked them to rate their exhaustion on the Borg-scale [9] and they would have to fill in the short MDBF Test [10] in order to assess their current level of emotional valence.

4.1 Results

Based on the measured Constitution, Sensitivity and Experience regarding carrying boxes as well as the individual choices which box and weight to carry first, the Smart Virtual Worker Simulation was run and computed corresponding emotional valences. Since a violation of normality was observed, the graphs were compared using the non-parametric correlation coefficient (Spearman-Rho) (Table 1).

Table 1. Spearman-Rho results of computed and measured emotional valence (*one-sided)
Full size table

4.2 Discussion

The data shows an 88 % capability of computing an adequate representation of the emotional valence. Furthermore, the mean correlation is quite strong (rm = .46). Looking specifically at the data from participant seven, we see a negative correlated emotional valence, leading to the conclusion that there is something off regarding the computation of these specific physiological and psychological variables. The correlations of participants two, three and four are not significant but they still show a positive correlation between the measured and computed data streams. Furthermore, they show a low and two medium coefficients which is why in this specific context, the probability of error should be interpreted as a measure for the conformity of the measured and computed data rather than a rejection of the measurement alltogether.

5 General Discussion

Due to these steps, the calibrated emotion module is now capable of adequately predicting the emotional valence over the course of a specific work task. In combination with the results from the pre-test experiments and the evaluation, the emotion module is one of the first computed psychological models, which are not just working on a binary basis, but are positively evaluated against psychological empirical criteria. The next steps will be to match the existing computational routines to once more match the measured data as it was done during the pre-tests. Afterwards, to conduct other work sequences and to compare the model to those outcomes. Especially tasks with a much higher coordinative involvement with only low levels of strain, like putting together furniture, will become increasingly important for the continued refinement of the emotion model. Furthermore, besides looking for opportunities and applications as a robotic enhancement [11] we are currently exploring the practical applications of the emotion module without physical exhaustion due to performing heavy work, but psychological and cognitive fatigue. Therefore, beginning in 2015, the emotion module will be adapted to compute emotional valences during sitting but highly cognitive stressful tasks, as in air traffic controllers.