The Effect of Gamification on Emotions - The Potential of Facial Recognition in Work Environments
- 3.5k Downloads
Gamification means using video game elements to improve user experience and user engagement in non-game services and applications. This article describes the effects when gamification is used in work contexts. Here we focus on industrial production. We describe how facial recognition can be employed to measure and quantify the effect of gamification on the users’ emotions.
The quantitative results show that gamification significantly reduces both task completion time and error rate. However, the results concerning the effect on emotions are surprising. Without gamification there are not only more unhappy expressions (as to expect) but surprisingly also more happy expressions. Both findings are statistically highly significant.
We think that in redundant production work there are generally more (negative) emotions involved. When there is no gamification happy and unhappy balance each other. In contrast gamification seems to shift the spectrum of moods towards “relaxed”. Especially for work environments such a calm attitude is a desirable effect on the users. Thus our findings support the use of gamification.
KeywordsGamification Assistive technology Facial recognition Affective computing Computer-assisted instruction Augmented reality Human machine interaction
Gamification is a delightful concept: it is a creditable idea to use “video game elements to improve user experience and user engagement in non-game services and applications” . The link between emotions and motivation is important, especially in context of employee motivation: after all, increased engagement should have numerous benefits like improved performance and greater user satisfaction .
This article describes how gamification can be used in work contexts like industrial production. Since industrial production already involves processes with measurable physical outcomes (e.g. the number of parts produced per hour), gaming elements like scores and leaderboards could be implemented with comparatively little effort. Ideally the system should also be able to measure the effect that gamified work processes have on speed and quality.
However, if gamification is to succeed in such environments, a system employing it will need a back channel to determine if a specific intervention has increased or decreased a user’s mood. A preferred approach to measuring the user’s emotional state in work environments are facial expressions since they can be measured without physical contact to the worker.
2 State of the Art
The integration of gamification into business contexts has first been described by Reeves & Read . However, they have been focusing on office contexts. A concept for the integration of gamification into production environments has first been described by Korn . While this concept already included real-time feedback based on motion tracking, it lacked a feedback channel which allows monitoring the effects of gamification.
This model also establishes the notion of a flow state as a desirable outcome of gamification. This is an emotional state in-between arousal and control where high skills and high demands converge – as described by Csíkszentmihályi in the late seventies .
However, in the context of this work the more important aspect is the integration of emotion recognition in the environmental interface. Within the CAAS-model emotion recognition is considered a pre-requisite for determining a user’s current flow state and thus adapting the interventions generated by the system. A good example is the need to determine if a worker reduces work speed because of boredom or because of exhaustion. While boredom would require an energizing stimulus (or even an increase in production speed if the system allows for such adaptations), exhaustion would require just the opposite.
Some assistive systems in production environments already feature cameras for motion recognition . Thus the preferred approach for measuring the user’s emotional state is using these cameras to analyze the facial expressions. By using these cameras no additional system is required and also no physical contact to the worker – in contrast to sensors for galvanic skin response or most heart rate sensors.
A result of this research was the facial coding system (FACS). It focuses on special muscle groups around the mouth and the eyes. These are crucial in the activity of expressing emotions and thus are called “Action Units”.
For the purpose of the pioneering study, determining the states “happy” versus “not happy” was considered a sufficient granularity to measure the effects of gamification – other emotions like anger and fear or even disgust were considered as unlikely to appear (or at least to be shown) in a production work setting.
The emotion recognition system has to ensure a high performance because real-time evaluation is required. A feedback given a few seconds after the event is already considered a disturbance. To achieve performance in real-time, the image size of the captured faces has to be reduced. Also it is required to use simple methods of pattern recognition which can be computed fast.
3.1 Face Detection
At first the image region is reduced, since for analyzing facial expressions only the face is of interest. This minimizes the memory significantly. We implemented the Viola-Jones-Algorithm which uses three simple methods that can be computed fast.
3.2 Feature Extraction and Bag of Features
As mentioned before every emotion has characteristic features, i.e. characteristic feature points (key points). These key points were extracted from detected faces using the Speeded Up Robust Features algorithm (SURF) which is invariant towards scale and rotation . Therefore it was considered to be well-suited for working processes where rotations and various illumination conditions are to be expected.
Similar to face detection it is based on the Haar-like features. The Fast-Hessian-Method is used to smooth the input image by convolution with Gaussian. Afterwards the algorithm detects key points. For every detected key point a feature vector with 64 entries is computed. Since every facial expression consists of various key points there are several 64D feature vectors for every emotion. Although every emotion causes numerous feature vectors based on unique key points, some of these points are “useless” as they do not describe an important point within the emotion. By using k-means clustering we created a “Bag of Features” for all important feature vectors of the emotions, i.e. their remarkable features. Thus similar key points were clustered and an emotion was recognized by its typical key points only.
3.3 Limitations of Facial Recognition
After the system was implemented we tested it both in the lab and in the wild. The emotional recognition (in our work-oriented scenario mainly the discrimination between happy and unhappy) was stable in the lab.
However, we found that in work environments the method is not reliable. The illumination changes quickly because of suboptimal lighting conditions in combination with the worker’s movements. Although the SURF algorithm is supposedly invariant towards scale and rotation this caused problems: shadows on the face distort the expression and emotions were not recognized correctly. The changing illumination also increased latency. To counterbalance such effects and still avoid body-attached sensors, additional cameras and lighting would be required in work environments.
Technically this results in a Wizard-of-Oz study.
providing real-time multimodal feedback (audio and visual)
displaying performance (required time and committed errors)
using scores to increase motivation and establish flow
The gamification system allocates 10 points per step. Each mistake or slow step results in a score reduction of 10 points. Thus in the 15-step scenario the score was gradually reduced from a maximum of 150 points down to zero.
Reaching zero could either be a result of 15 mistakes or of 15 slow steps or a mixture of both. Although theoretically 15 slow and wrong steps would result in a negative score (here: −150) the score was capped at zero.
The probands received real-time feedback on both the time and the error rate. However, negative feedback was phrased positively to avoid demotivation, e.g. “you can do better”.
The study was conducted with ten probands age 19 to 38. We used a repeated measures approach with 10 task repetitions without gamification and 10 task repetitions with gamification. The sequence of tasks with and without gamification was randomized.
Each test subject completed a two-part questionnaire with an emotional self-evaluation. The first part was used before the assembly and the second part after the assembly. The questionnaire evaluated the effects recognized by the test persons.
The camera was placed in the center in front of the monitor and was re-adjusted for each proband to maximize the quality of the facial recognition.
Several test persons were generally not comfortable with having to build the model several times. However, this was an intended effect which shows that repetitive work is not appreciated.
5 Discussion and Conclusion
Generally the test subjects stated that they felt the gamification elements very motivating. These assertions are confirmed by the quantitative results with respect to task completion time and error rate: gamification reduces both significantly.
the emotion detection method is not reliable: this is unlikely due to the additional human control
the participants deliberately or subconsciously conceal expressions of happiness in the gamification setup: this is possible but unlikely
when there is no gamification there are generally more emotions involved and happy and unhappy balance each other
In contrast gamification seems to shift the spectrum of moods towards “relaxed” (i.e. no measurable expression). Especially for work environments such a calm attitude is a desirable effect on the users. Thus our findings support the use of gamification.
6 Future Work
With only 10 probands this work can only be a first step into measuring the impact of gamification on emotions in work environments. Still the findings are unexpected and have to be validated by extensive studies with more test subjects. Also a more reliable facial recognition is needed which (for this purpose) could be supported by body-attached sensors.
- 2.Csíkszentmihályi, M., Nakamura, J.: The concept of flow. In: Snyder, C.R., Lopez, S.J. (eds.) The Handbook of Positive Psychology, pp. 89–92. Oxford University Press, Oxford (2002)Google Scholar
- 3.Deterding, S., Sicart, M., Nacke, L., O’Hara, K., Dixon, D.: Gamification. using game-design elements in non-gaming contexts. In: Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2425–2428. ACM (2011)Google Scholar
- 6.Korn, O.: Industrial playgrounds: how gamification helps to enrich work for elderly or impaired persons in production. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2012, pp. 313–316. ACM (2012)Google Scholar
- 7.Korn, O., Funk, M., Abele, S., Hörz, T., Schmidt, A.: Context-aware assistive systems at the workplace: analyzing the effects of projection and gamification. In: Proceedings of the 7th International Conference on Pervasive Technologies Related to Assistive Environments, PETRA 2014, pp. 38:1–38:8. ACM (2014)Google Scholar
- 8.Korn, O., Schmidt, A., Hörz, T.: Assistive systems in production environments: exploring motion recognition and gamification. In: Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2012, pp. 9:1–9:5. ACM (2012)Google Scholar
- 10.McGonigal, J.: Reality is Broken: Why Games Make us Better and How they can Change the World. Penguin Books, New York (2011)Google Scholar
- 11.Reeves, B., Read, J.L.: Total Engagement: Using Games and Virtual Worlds to Change the Way People Work and Businesses Compete. Harvard Business Press, Boston (2009)Google Scholar