Neural Computing and Applications

, Volume 26, Issue 6, pp 1277–1296 | Cite as

Hybrid affective computing—keyboard, mouse and touch screen: from review to experiment

  • Kaveh BakhtiyariEmail author
  • Mona Taghavi
  • Hafizah Husain
Original Article


Emotions play an important role in human interactions. They can be integrated into the computer system to make human–computer interaction (HCI) more effective. Affective computing is an innovative computational modeling and detecting user’s emotions to optimize system responses in HCI. However, there is a trade-off between recognition accuracy and real-time performance in some of the methods such as processing the facial expressions, human voice and body gestures. Other methods lack efficiency and usability in real-world applications such as natural language processing and electroencephalography signals. To accomplish a reliable, usable and high-performance system, this paper proposes an intelligent hybrid approach to recognize users’ emotions by using easily accessible and low computational cost input devices including keyboard, mouse (touch pad: single touch) and touch screen display (single touch). Using the proposed approach, the system is developed and trained in a supervised mode by artificial neural network and support vector machine (SVM) techniques. The result shows an increase in accuracy of 6 % (93.20 %) by SVM in comparison with the currently existing methods. It is a significant contribution to show new directions of future research in emotion recognition, user modeling and emotional intelligence.


Affective computing Human emotion recognition Keyboard keystroke dynamics Mouse touch pad movement Touch screen monitor Human–computer interaction (HCI) 



Thanks to the students of Universiti Kebangsaan Malaysia (The National University of Malaysia), University of Duisburg-Essen, Germany, and members of AIESEC UKM and AIESEC ESSEN, the largest nonprofit organization run by students, who participated in the prototype system evaluation.


  1. 1.
    Leon E, Clarke G, Callaghan V, Sepulveda F (2007) A user-independent real-time emotion recognition system for software agents in domestic environments. Eng Appl Artif Intell 20(3):337–345. doi: 10.1016/j.engappai.2006.06.001 CrossRefGoogle Scholar
  2. 2.
    Harris M, Ishii H, Chung C, Dodsworth C, Buxton B (1999) Natural and invisible human interfaces. In: International conference on computer graphics and interactive techniques (SIGGRAPH). doi: 10.1145/311625.311922
  3. 3.
    Palm G, Glodek M (2013) Towards emotion recognition in human computer interaction. In: Neural nets and surroundings, vol 19. Springer, pp 323–336. doi: 10.1007/978-3-642-35467-0_32
  4. 4.
    Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human–computer interaction. Sig Process Mag IEEE 18(1):32–80CrossRefGoogle Scholar
  5. 5.
    Picard RW (1998) Affective computing. MIT Press, CambridgeGoogle Scholar
  6. 6.
    Strongman KT (2003) The psychology of emotion. Department of Psychology, University of Canterbury, 5th edn. Wiley, ChristchurchGoogle Scholar
  7. 7.
    Lewis M, Haviland-Jones JM, Barrett LF (2010) The cultural psychology of the emotions—ancient and renewed. In: Richard A, Shweder JH, Horton R, Joseph C (eds) Handbook of emotions, 3rd edn. The Guilford Press, New York, p 19Google Scholar
  8. 8.
    Fredrickson BL (2001) The role of positive emotions in positive psychology—the broaden-and-build theory of positive emotions. Am Psychol 56(3):9. doi: 10.1037/0003-066X.56.3.218 CrossRefGoogle Scholar
  9. 9.
    SRamaraj V, Ravindran A, Thirumurugan A (2013) Emotion recognition from human eye expression. IJRCCT 2(4):158–164Google Scholar
  10. 10.
    Schmidt K, Cohn J (2002) Human facial expressions as adaptations: evolutionary questions in facial expression. Am J Phys Anthropol 33:3–24Google Scholar
  11. 11.
    Baker C, Cokely D (1980) American sign language: a teacher’s resource text on grammar and culture. T.J. Publishers, Silver SpringGoogle Scholar
  12. 12.
    Jamshidnejad A (2009) Facial emotion recognition for human computer interaction using a fuzzy model in the e-business. In: Conference on innovative technologies in intelligent systems and industrial applications (CITISIA), pp 202–204Google Scholar
  13. 13.
    Tsihrintzis GA, Virvou M, Alepis E, Stathopoulou IO (2008) Towards improving visual-facial emotion recognition through use of complementary keyboard-stroke pattern information. In: Fifth international conference on information technology: new generations, pp 32–37Google Scholar
  14. 14.
    Valstar M, Patras I, Pantic M (2004) Facial action unit recognition using temporal templates. In: IEEE international workshop on robot and human interactive communication, pp 253–258Google Scholar
  15. 15.
    Gu W, Xiang C, Venkatesh Y, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45(1):80–91CrossRefGoogle Scholar
  16. 16.
    Konar A, Chakraborty A, Halder A, Mandal R, Janarthanan R (2012) Interval type-2 fuzzy model for emotion recognition from facial expression. In: Kundu MK, Mitra S, Mazumdar D, Pal SK (eds) Perception and machine intelligence. Lecture notes in computer science, vol 7143. Springer, Heidelberg, pp 114–121. doi: 10.1007/978-3-642-27387-2_15
  17. 17.
    Kao CY, Fahn CS (2012) A design of face detection and facial expression recognition techniques based on boosting schema. Appl Mech Mater 121:617–621Google Scholar
  18. 18.
    Ilbeygi M, Shah-Hosseini H (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25(1):130–146. doi: 10.1016/j.engappai.2011.07.004 CrossRefGoogle Scholar
  19. 19.
    Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92. doi: 10.1109/T-AFFC.2012.33 CrossRefGoogle Scholar
  20. 20.
    Beaudrya O, Roy-Charlanda A, Perrona M, Cormierb I, Tappa R (2014) Featural processing in recognition of emotional facial expressions. Cogn Emot 28(3):416–432. doi: 10.1080/02699931.2013.833500 CrossRefGoogle Scholar
  21. 21.
    Kleinsmith A, Bianchi-Berthouze N (2013) Affective body expression perception and recognition: a survey. IEEE Trans Affect Comput 4(1):15–33. doi: 10.1109/T-AFFC.2012.16 CrossRefGoogle Scholar
  22. 22.
    Lee JS, Shin D-H (2013) A study on the interaction between human and smart devices based on emotion recognition. In: HCI international 2013-posters’ extended abstracts. Springer, pp 352–356Google Scholar
  23. 23.
    Bulwer J (1644) Chirologia: or the natural language of the hand. London. doi: 10.1037/11828-001
  24. 24.
    de Jorio A (1832) Gesture in naples and gesture in classical antiquity. Indiana University Press, BloomingtonGoogle Scholar
  25. 25.
    McNeill D (2005) Gesture and thought. Chicago University Press, ChicagoCrossRefGoogle Scholar
  26. 26.
    Goldin-Meadow S (2003) Hearing gesture: how our hands help us think. Harvard University Press, CambridgeGoogle Scholar
  27. 27.
    Gunes H, Piccardi M (2005) Fusing face and body gesture for machine recognition of emotions. In: IEEE international workshop on robots and human interactive communication, pp 306–311Google Scholar
  28. 28.
    Glowinski D, Camurri A, Volpe G, Dael N, Scherer K (2008) Technique for automatic emotion recognition by body gesture analysis. In: IEEE computer society conference on computer vision and pattern recognition workshopsGoogle Scholar
  29. 29.
    Chena S, Tiana Y, Liub Q, Metaxasc DN (2013) Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. Image Vis Comput 31(2):175–185. doi: 10.1016/j.imavis.2012.06.014 CrossRefGoogle Scholar
  30. 30.
    Steunebrink BR, Dastani M, Meyer JJC (2009) The OCC model revisited. Utrecht University, The NetherlandsGoogle Scholar
  31. 31.
    Li H, Pang N, Guo S, Wang H (2008) Research on textual emotion recognition incorporating personality factor. In: IEEE international conference on robotics and biomimetics, pp 2222–2227Google Scholar
  32. 32.
    Gil GB, Jesús ABd, Lopéz JMM (2013) combining machine learning techniques and natural language processing to infer emotions using Spanish twitter corpus. In: International workshops of PAAMS 365:149–167. doi: 10.1007/978-3-642-38061-7_15
  33. 33.
    Calvo RA, Kim SM (2013) Emotions in text: dimensional and categorical models. Comput Intell 29(3):527–543. doi: 10.1111/j.1467-8640.2012.00456.x MathSciNetCrossRefGoogle Scholar
  34. 34.
    Shahin I (2013) Speaker identification in emotional talking environments based on CSPHMM2s. Eng Appl Artif Intell 26(7):1652–1659. doi: 10.1016/j.engappai.2013.03.013 CrossRefGoogle Scholar
  35. 35.
    Klecková J (2009) Important nonverbal attributes for spontaneous speech recognition. In: Fourth international conference on systems, pp 13–16Google Scholar
  36. 36.
    Wang Y, Guan L (2008) Recognizing human emotional state from audiovisual signals*. Multimed IEEE Trans 10(5):936–946CrossRefGoogle Scholar
  37. 37.
    Merrett T (2008) Query by humming. McGill University, MontrealGoogle Scholar
  38. 38.
    Unal E, Chew E, Georgiou PG, Narayanan SS (2008) Challenging uncertainty in query by humming systems: a fingerprinting approach. Audio Speech Lang Process IEEE Trans 16(2):359–371CrossRefGoogle Scholar
  39. 39.
    Amarakeerthi S, Ranaweera R, Cohen M (2010) Speech-based emotion characterization using postures and gestures in CVEs. In: International conference on cyberworlds, pp 72–76Google Scholar
  40. 40.
    Pittermann J, Schmitt A, El Sayed NF (2008) Integrating linguistic cues into speech-based emotion recognition. In: 4th international conference on intelligent environments, pp 1–4Google Scholar
  41. 41.
    Huang T (2008) Audio-visual human computer interface. In: IEEE international symposium on consumer electronics, pp 1–1Google Scholar
  42. 42.
    Kotti M, Kotropoulos C (2008) Gender classification in two emotional speech databases. In: 19th international conference on pattern recognition, pp 1–4Google Scholar
  43. 43.
    Krishnan A, Fernandez M (2014) System and method for recognizing emotional state from a speech signal. Google PatentsGoogle Scholar
  44. 44.
    Liu Y, Sourina O, Nguyen MK (2010) Real-time EEG-based human emotion recognition and visualization. In: International conference on cyberworlds, pp 262–269Google Scholar
  45. 45.
    Schaaff K, Schultz T (2009) Towards emotion recognition from electroencephalographic signals. In: Affective computing and intelligent interaction and workshops (ACII), pp 1–6Google Scholar
  46. 46.
    Guangying Y, Shanxiao Y (2010) Emotion recognition of electromyography based on support vector machine. In: Third international symposium on intelligent information technology and security informatics, pp 298–301Google Scholar
  47. 47.
    Murugappan M, Murugappan S (2013) Human emotion recognition through short time electroencephalogram (EEG) signals using fast fourier transform (FFT). In: IEEE 9th international colloquium on signal processing and its applications (CSPA), pp 289–294. doi: 10.1109/CSPA.2013.6530058
  48. 48.
    Schuller B, Rigoll G, Lang M (2004) Emotion recognition in the manual interaction with graphical user interfaces. In: IEEE international conference on multimedia and expo 2. pp 1215–1218Google Scholar
  49. 49.
    Milanova M, Sirakov N (2008) Recognition of emotional states in natural human–computer interaction. In: IEEE international symposium on signal processing and information technology (ISSPIT), pp 186–191Google Scholar
  50. 50.
    Kaklauskas A, Zavadskas EK, Seniut M, Dzemyda G, Stankevic V, Simkevičius C, Stankevic T, Paliskiene R, Matuliauskaite A, Kildiene S, Bartkiene L, Ivanikovas S, Gribniak V (2011) Web-based biometric computer mouse advisory system to analyze a user’s emotions and work productivity. Eng Appl Artif Intell 24(6):928–945. doi: 10.1016/j.engappai.2011.04.006 CrossRefGoogle Scholar
  51. 51.
    Monrose F, Rubin AD (2000) Keystroke dynamics as a biometric for authentication. Future Gener Comput Syst 16(4):351–359CrossRefGoogle Scholar
  52. 52.
    Wang R, Fang B (2008) Affective computing and biometrics based HCI surveillance system. In: International symposium on information science and engineering, pp 192–195Google Scholar
  53. 53.
    Epp C, Lippold M, Mandryk RL (2011) Identifying emotional states using keystroke dynamics. In: Proceedings of the 2011 annual conference on human factors in computing systems, pp 715–724. doi: 10.1145/1978942.1979046
  54. 54.
    Waikato Uo (2010) Weka. Accessed 01 July 2010
  55. 55.
    Bashir MG, Nagarajan R, Hazry D (2010) Facial emotion detection using GPSO and Lucas–Kanade algorithms. In: International conference on computer and communication engineering (ICCCE), pp 1–6. doi: 10.1109/ICCCE.2010.5556754
  56. 56.
    Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, CambridgeGoogle Scholar
  57. 57.
    Kao ECC, Liu CC, Yang TH, Hsieh CT, Soo VW (2009) Towards text-based emotion detection. In: International conference on information management and engineering (ICIME), pp 70–74Google Scholar
  58. 58.
    Yang H, Willis A, De Roeck A, Nuseibeh B (2012) A hybrid model for automatic emotion recognition in suicide notes. Biomed Inform Insights 5(Supp 1):17–30. doi: 10.4137/BII.S8948 CrossRefGoogle Scholar
  59. 59.
    Böhlen M (2009) Second order ambient intelligence. J Ambient Intell Smart Environ 1(1):63–67Google Scholar
  60. 60.
    Aarts E, de Ruyter B (2009) New research perspectives on ambient intelligence. J Ambient Intell Smart Environ 1(1):5–14Google Scholar
  61. 61.
    Sinek S (2011) First why and then trust. TEDx. Accessed 06 Jan 2012
  62. 62.
    Larson R, Csikszentmihalyi M (1983) The experience sampling method. In: New directions for methodology of social & behavioral science, vol 15, pp 41–46Google Scholar
  63. 63.
    Hertenstein MJ, Keltner D, App B, Bulleit BA, Jaskolka AR (2006) Touch communicates distinct emotions. Emotion 6(3):528CrossRefGoogle Scholar
  64. 64.
    Rezaee Jordehi A (2014) A chaotic-based big bang–big crunch algorithm for solving global optimisation problems. Neural Comput Appl 25(6):1329–1335. doi: 10.1007/s00521-014-1613-1 CrossRefGoogle Scholar
  65. 65.
    Rezaee Jordehi A (2014) Particle swarm optimisation for dynamic optimisation problems: a review. Neural Comput Appl 25(7–8):1507–1516. doi: 10.1007/s00521-014-1661-6 CrossRefGoogle Scholar
  66. 66.
    Han J, Kamber M, Pei J (2012) Data mining concepts and techniques, 3rd edn. Elsevier, USAGoogle Scholar
  67. 67.
    Bakhtiyari K, Taghavi M, Husain H (2014) Implementation of emotional-aware computer systems using typical input devices. In: Nguyen NT, Attachoo B, Trawiński B, Somboonviwat K (eds) Intelligent Information and Database Systems, vol 8397. Springer International Publishing, Bangkok, pp 364–374. doi: 10.1007/978-3-319-05476-6_37 CrossRefGoogle Scholar
  68. 68.
    Kemp F (2003) Applied multiple regression/correlation analysis for the behavioral sciences. J R Stat Soc Ser D (Stat) 52(4):691CrossRefGoogle Scholar
  69. 69.
    Cohen J (2003) Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum Associates, PublishersGoogle Scholar
  70. 70.
    Hauske G (2003) Statistische Signaltheorie. In: Skriptum zur Vorlesung. Technische Universität München, München, DeutschlandGoogle Scholar
  71. 71.
    Xiao Z, Dellandrea E, Dou W, Chen L (2007) Automatic hierarchical classification of emotional speech. In: Ninth IEEE international symposium on multimedia workshops, pp 291–296Google Scholar
  72. 72.
    Bakhtiyari K, Husain H (2013) Fuzzy model on human emotions recognition. In: 12th international conference on applications of computer engineering, pp 77–82Google Scholar
  73. 73.
    Bakhtiyari K, Husain H (2014) Fuzzy model of dominance emotions in affective computing. Neural Comput Appl 25(6):1467–1477. doi: 10.1007/s00521-014-1637-6 CrossRefGoogle Scholar
  74. 74.
    Taghavi M, Bakhtiyari K, Scavino E (2013) Agent-based computational investing recommender system. In: Proceedings of the 7th ACM conference on recommender systems, pp 455–458. doi: 10.1145/2507157.2508072
  75. 75.
    Taghavi M, Bakhtiyari K, Taghavi H, Olyaee V, Hussain A (2014) Planning for sustainable development in the emerging information societies. J Sci Technol Policy Manag 5(3):178–211. doi: 10.1108/JSTPM-04-2014-0013 CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2014

Authors and Affiliations

  • Kaveh Bakhtiyari
    • 1
    • 2
    Email author
  • Mona Taghavi
    • 2
  • Hafizah Husain
    • 2
  1. 1.Interactive Systems, Department of Computer and Cognitive Science, Faculty of EngineeringUniversity of Duisburg-EssenDuisburgGermany
  2. 2.Department of Electrical, Electronics and Systems Engineering, Faculty of Engineering and Built EnvironmentUniversiti Kebangsaan Malaysia, UKM (The National University of Malaysia)BangiMalaysia

Personalised recommendations