Advertisement

International Conference on Human-Computer Interaction

HCI 2015: HCI International 2015 - Posters’ Extended Abstracts pp 712-717 | Cite as

Is the Mood Really in the Eye of the Beholder?

  • Mojgan HashemianEmail author
  • Hadi Moradi
  • Maryam S. Mirian
  • Mehdi Tehrani-Doost
  • Rabab K. Ward
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 528)

Abstract

A great deal of scientific evidence suggests that there is a close relationship between mood and cognitive processes of human in everyday tasks. In this study, we have investigated the feasibility of determining mood from gaze, which is one of the human cognitive processes that can be recorded during interaction with computers. To do so, we have designed a feature vector composed of typical gaze patterns, and piloted the approach on the dataset which we gathered. It consists of 145 samples of 30 people. A supervised machine learning technique was employed for classification and recognition of mood. The results of this pilot test suggests that even during these initial steps, the approach is quite promising and opens other research paths for improvement through multi-modal recognition and information fusion. Multi-modal approach would employ the added information provided by our previously developed mood extraction approach using camera and/or the information gained by the use of EEG signals. Further analysis will be performed in feature extraction process to enhance the model accuracy by enriching the feature-set of each modality.

Keywords

Mood Gaze Cognition Affective computing Human computer interaction 

1 Introduction

Mood, our background feeling which lasts for longer time in comparison to emotions, has been investigated for decades in the field of psychology [1]. In the literatures in this field, the relative importance of mood on our individual and social functions has been subject to considerable discussion. The evidence of this can be clearly seen in memory-dependent literatures [2], learning performance [3], creativity [4], etc. Another worth-mentioning role of mood is its application as a general indicator of wellness, in other words it is considered as one of the most associated evident with health and illness [5]. On the other hand, the importance of this subtle affective state in computer user satisfaction, has aroused attention of many researchers in the field of Human-Computer-Interaction (HCI) [6]. A large and growing body of literature has investigated different approaches to provide effective and desirable HCI by designing adaptable systems based on users’ affective state to make the gap between human and the machine more narrowed [7]. With this aim, the very first step would be recognition of the user’s affective state in non-intrusive ways [8]. By this time, more recent attention has been focused on human behavior or his physiological conditions in this field; however, little attention has been paid to the role of cognitive dimension.

Before proceeding to examine the effect of mood on gaze, it will be necessary to review evidences and hypothesis on this context. A great deal of scientific evidence reported that our mood interacts with cognitive influences, and there is a close relationship between this affective state and cognitive processes [1, 18]. To commence, the nervous system is organized in such a way that there is a linkage between various mood states and cognitive material that has been associated with those states. As an example, a relation between mood and memory, as a cognitive function, has been examined and investigated extensively for decades [2, 9]. Furthermore, a strong relationship between mood and attention has been reported for long time [1]. In addition another factor affecting mood is the occurrence of external events, while the interpretation of such events is highly affected by cognition [5]. These three factors each of which are considered as the most important cognitive processes proved to have direct synergy with our mood. On the other hand, there are more evidences that highlight this synergistic relation which are discussed in the following.

Furthermore, it has been proven that our energetic and tension arousal are two main physiological associations affecting our moods. On the other hand, various cognitive factors interact with energy and tension both directly and indirectly [5]. Although the evidence is limited, there is a little doubt that cognitive stimuli affect energetic arousal, i.e. cognitive-affective state association relates to naturally occurring mood [10]. Therefore, observing and examining cognitive processes may give us some clues on the ongoing mood states.

On the other hand, motor behavior is another human characteristic which is proved to be affected by mood [1110, 11]. One of the evidences of human motor behavior is “Gaze” which signifies cognition. Knowing this fact that mood is affected by our cognitive processes, we can recognize it by tracking the influence of these types of processes, for example by examining eye movements. So in this paper, we are going to propose a method in order to investigate the relationship between individuals’ mood and gaze.

2 Method

In recent years, there has been an increasing interest in using Gaze data as a source of information in fields such as cognitive psychology, human computer interaction (HCI), neuroscience and healthcare system. Thanks to the recent progress in technology, the recording of gaze data has been made easy and totally non-intrusive. The emergence of wearable wireless eye-trackers, [12], provides an opportunity to build body-network-areas with social purposes. This opportunity offers a great incentive for scientists in different areas of research, especially those in the field of HCI or ubiquitous computing, to use this information in isolated and social-oriented tasks.

In this paper, we propose an approach for recognizing the mood states of a user via a non-intrusive yet easy to implement method. Based on this method, the user’s mood state is detected by using typical features extracted from the gaze data. In this study, we propose a statistical model, which is capable of determining the user’s mood state over a short duration of time. The minimum required time is not discussed in this paper and it should be evaluated in our future work, via a different set of experiments for the specific settings discussed below.

In Table 1, fixation refers to the pause of the eye-movement on a specific area of the visual field while during this pause brain starts to process the visual information received from the eyes; this length varies from about 100 to 600 ms. Saccade refers to quick, simultaneous movement of both eyes between two consecutive fixations [20]. We consider 7 different categories of features, shown in Table 1. The first category, “counts”, stands for the total number of fixations followed by saccades during the interaction of the subject with the system. The second category refers to the rate of occurrence of each fixation/saccade throughout this time of interaction. The next four categories refer to descriptive statistics of fixation and saccade. They describe the main features in quantitative terms. Finally, the category seven is the time taken to the first fixation. We have considered these features since we believed that each of these features might provide additional information about the pattern of gaze in different mood states. All the considered features should be normalized based on the duration of interaction.
Table 1.

Gaze features extracted for mood detection

 

1

2

3

4

5

6

7

Fixation

Counts

Rate in second

Mean of duration

Min of duration

Max of duration

Variance of duration

First Fixation in each Phase

Saccade

To the best of our knowledge, no standard dataset of gaze records tagged by users’ mood state has been published yet. Thus we gathered a (large) dataset which is discussed in Sect. 4 in detail. In order to develop a data-gathering setting, we have designed a web-based platform that records the mood of the subject through a questionnaire. In each setting, a set of short video clips (minimum length of 38 s, and maximum of 2 min and 12 s) are shown to the subjects so he/she can select one of them. This platform has been already used for a camera based mood extraction [13, 14].

Since gaze is highly affected by the structure of the given task environment [10], we designed a four-phase-experiment to cover these differences as much as possible, in order to prove the generality of the work. The structure of the platform and description of each phases are discussed in Sect. 4.

3 User Study

As mentioned earlier, one of the contributions of this work is that of obtaining a dataset of gaze data labeled with the ongoing mood state of subjects performing different tasks. To gather the data, we ran an experiment using the developed web-based platform. The experiment was conducted on 30 people, including 8 females and 22 males aged between 19 and 32 (mean 25 years old). A total of 145 video samples were recorded during a two-week period by an unobtrusive eye-tracker (Tobii X2-30) with an accuracy ranging from 0.4º−1º, and a precision between 0.29º−0.80º [15]. Data Preparation and Processing.

As discussed in Sect. 2, the experiment is designed in four separate phases with different structures. The first phase is designed to record the normal behavior of the participants while working on a routine computer task. In this phase the participants are asked to sign in to the web-based platform and read the instruction manual.

Next, the subjects are asked to fill in a mood questionnaire. This will be considered as the ground-truth or gold standard data indicating the real mood state of the subjects. Then a list of diverse video clips is offered to the subjects. Then each person has to choose one clip to continue to the next phase. These two consecutive steps, in which participants are supposed to choose one option among different alternatives, resemble decision making tasks in cognitive science. Decision making is considered one of the basic cognitive processes of human behavior. When people are faced by a problem with multiple, and possibly conflicting, alternatives and they are supposed to choose one of them based on their preferences [16]. Thus during the phase of filling the questionnaire and choosing the clip, the gaze pattern was also recorded while the decision making was going on.

A relationship between scene viewing and cognitive behavior is reported in [11], thus the third phase is devoted to the subjects, while watching a clip selected in the previous phase. It is worth mentioning that all the video clips are displayed in a specified and a predefined size that are fixed and unchangeable. In this phase, the subjects’ gaze pattern during clip-viewing is recorded.

In the last phase, a participant fills in a survey that asks him/her about his/her opinion about the clip. This phase resembles the second phase. The only difference is the influence of emotion induction, induced by the watched clip. This phase provides us the opportunity to compare the two phases result. All the provided clips were believed to be “funny” and could induce happiness emotion.

To classify the data, we composed a feature vector including the features listed in Table 1 for each phase. After labeling the data by the information obtained from the self-reports, we employed LibSVM toolbox and trained by 10-fold cross-validation using Weka [17]. The classification outcomes are one of 3 moods: good, bad and neutral mood. The system yielded a 49.3 % (\( \pm \)1.1) classification accuracy in discriminating between the 3 moods. This is better than the 33.3 % accuracy which a random classifier would yield in this case.

4 Conclusion and Future Works

To the best of our knowledge, this is the first attempt to recognize mood based on gaze data in a non-intrusive fashion. As pointed out in the abstract, the aim of this paper is mainly to a report on our work in progress. The results so far have showed that this approach appears to be promising in recognizing mood states in a non-intrusive way and surpasses a random classifier (which leads to 33 % accuracy as we have considered 3 types of moods).

The results could lead to higher accuracy by performing many possible improvements. The first and foremost possibility in enriching the feature-set, is by considering a broader range of general eye-tracking features (in both the time domain and the frequency domain) such as eye-movement speed, or more complex features such as relative saccade angle. A personalized analysis may lead to higher accuracy since people usually have different page navigation habits and these habits may sometimes lead to unique gaze patterns. For instance, some people use mouse-wheel to scroll, while others prefer to use the scroll bar and fixate on it for a while before scrolling. In addition, calculating the features within a specific region of interest (ROI) may lead to more accurate results. As an example, in the third phase, i.e. scene viewing, the clip constitutes a portion of the screen and the likelihood of looking at other parts of the screen is less in this phase. Considering this fact, may lead to more accurate results.

Moreover, as mentioned in the previous section, the second and fourth phases resemble decision making tasks and provide an opportunity to investigate the effects of induced emotion on gaze. It should be mentioned here that the relation between emotion and cognition has been investigated for decades and there is a large number of published studies that have investigated the relation between emotion and gaze [19]. On the other hand, in our previous study [13], we investigated the influence of mood on emotions induced by an emotion induction technique. Hence, by comparing the patterns in the second and the fourth phase of this experiment, the effect of mood on emotion could be more highlighted considering changes in gaze pattern and vice versa. These further analyses have been considered for future

Another step that is worth-performing is that of calculating the minimum required time to be able to recognize the mood states from the gaze patterns. This would make this approach usable in many applications such as in medical and health applications, especially in applications that require real-time or near real-time constraints. By taking advantage of wireless eye-tracker, our proposed approach could also be employed in mobile Health systems aimed at developing efficient disease management. Thus we plan to benefit from the use of a multi-modal approach in which several sensory information and modalities are employed to improve the mood recognition. The initial step would involve incorporating our camera-based mood detection approach [13], and the use of EEG signals as well. Finally, we would investigate frameworks that use for example the Dempster-Shafer Theory of combination for integrating the knowledge gained from the different modalities employed.

Notes

Acknowledgements

The first author would like to thank members of Advanced Robotics and Intelligent Systems lab, Social Networks lab, and Mobile Robot lab for their endless helps and participation in the experiment. This work was supported by NPRP grant 09- 310-1– 058, and grant 7-684-1-127 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibly of the authors.

References

  1. 1.
    Thayer, R.E.: The Biopsychology of Mood and Arousal. Oxford University Press, New York (1989)Google Scholar
  2. 2.
    Lewis, P.A., Critchley, H.D.: Mood-dependent memory. Trends Cogn. Sci. 7(10), 431–433 (2003)CrossRefGoogle Scholar
  3. 3.
    Bryan, T., Mathur, S., Sullivan, K.: The impact of positive mood on learning. Learn. Disabil. Q. 19(3), 153–162 (1996)CrossRefGoogle Scholar
  4. 4.
    Suzanne, K., Vosburg, G.K.: ‘Paradoxical’ mood effects on creative problem-solving. Cogn. Emot. 11(2), 151–170 (1997)CrossRefGoogle Scholar
  5. 5.
    Thayer, R.E.: The Origin of Everyday Moods. Oxford University Press, New York (1996)Google Scholar
  6. 6.
    Hashemian, M., Nikoukaran, A., Moradi, H., Mirian, M.S., Tehrani-doost, M.: Determining mood using emotional features. In: 2014 7th International Symposium on Telecommunications (IST), pp. 418–423. IEEE (2014)Google Scholar
  7. 7.
    Hudlicka, E.: To feel or not to feel: the role of affect in human–computer interaction. Int. J. Hum. Comput. Stud. 59(1), 1–32 (2003)CrossRefGoogle Scholar
  8. 8.
    Picard, R.W.: Toward computers that recognize and respond to user emotion. IBM Syst. J. 39(3–4), 705–719 (2000)CrossRefGoogle Scholar
  9. 9.
    Bower, G.H.: Mood and memory. Am. Psychol. 36(2), 129 (1981)CrossRefGoogle Scholar
  10. 10.
    Zimmermann, P., Guttormsen, S., Danuser, B., Gomez, P.: Affective computing–a rationale for measuring mood with mouse and keyboard. Int. J. Occup. Saf. Ergon. 9(4), 539–551 (2003)CrossRefGoogle Scholar
  11. 11.
    Henderson, J.M., Hollingworth, A.: Eye movements during scene viewing: an overview. In: Eye Guidance in Reading and Scene Perception, vol. 11, pp. 269−293 (1998)Google Scholar
  12. 12.
  13. 13.
    Hashemian, M., Moradi, H., Mirian, M.S., Tehrani-doost, M.: Determining mood via emotions observed in face by induction. In: 2014 Second RSI/ISM International Conference on Robotics and Mechatronics (ICRoM), pp. 717–722. IEEE (2014)Google Scholar
  14. 14.
    Hashemian, M., Moradi, H., Mirian, M.S., Tehrani-doost, M., Mahmoudyar, N.: Recognizing Mood using Facial Emotional Features. IEEE Transaction on Affective Computing (Under Review)Google Scholar
  15. 15.
    Tobii Technology AB. Accuracy and precision Test report X2-30 fw 1.0.1. 28 May 2013Google Scholar
  16. 16.
    Frederick, S.: Cognitive reflection and decision making. J. Econ. Perspect. 19, 25–42 (2005)CrossRefGoogle Scholar
  17. 17.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  18. 18.
    Martin, E., Kerns, J.: The influence of positive mood on different aspects of cognitive control. Cogn. Emot. 25(2), 265–279 (2011)CrossRefGoogle Scholar
  19. 19.
    Subramanian, R., Shankar, D., Sebe, N., Melcher, D.: Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. J. Vis. 14(3), 31 (2014)CrossRefGoogle Scholar
  20. 20.

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Mojgan Hashemian
    • 1
    Email author
  • Hadi Moradi
    • 1
  • Maryam S. Mirian
    • 1
    • 2
  • Mehdi Tehrani-Doost
    • 3
  • Rabab K. Ward
    • 4
  1. 1.Advanced Robotics and Intelligent Systems LabUniversity of TehranTehranIran
  2. 2.Department of Computer ScienceUBCVancouverCanada
  3. 3.Tehran University of Medical SciencesTehranIran
  4. 4.Department of Electrical and Computer EngineeringUBCVancouverCanada

Personalised recommendations