Pattern Analysis and Applications

, Volume 9, Issue 1, pp 58–69 | Cite as

An empirical study of machine learning techniques for affect recognition in human–robot interaction

  • Pramila RaniEmail author
  • Changchun Liu
  • Nilanjan Sarkar
  • Eric Vanman
Theoretical Advances


Given the importance of implicit communication in human interactions, it would be valuable to have this capability in robotic systems wherein a robot can detect the motivations and emotions of the person it is working with. Recognizing affective states from physiological cues is an effective way of implementing implicit human–robot interaction. Several machine learning techniques have been successfully employed in affect-recognition to predict the affective state of an individual given a set of physiological features. However, a systematic comparison of the strengths and weaknesses of these methods has not yet been done. In this paper, we present a comparative study of four machine learning methods—K-Nearest Neighbor, Regression Tree (RT), Bayesian Network and Support Vector Machine (SVM) as applied to the domain of affect recognition using physiological signals. The results showed that SVM gave the best classification accuracy even though all the methods performed competitively. RT gave the next best classification accuracy and was the most space and time efficient.


Affect recognition Machine learning Psychophysiology Emotional robotics 


  1. 1.
    World Robotics (2004) Statistics, market analysis, forecasts, case studies and profitability of robot investment. Sales No. GV.E.04.0.20 or ISBN No. 92-1-101084-5Google Scholar
  2. 2.
    Reeves B, Nass C (1996) The media equation: how people treat computers, televisions and new media like real people and places. Cambridge University Press, New YorkGoogle Scholar
  3. 3.
    Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human–computer interaction. IEEE Signal Process Mag 18(2):32–80CrossRefGoogle Scholar
  4. 4.
    Picard R (1997) Affective computing. The MIT Press, CambridgeGoogle Scholar
  5. 5.
    Walter WG (1963) The living brain. W. W. Norton, New YorkGoogle Scholar
  6. 6.
    Breazeal C, Aryananda L (2002) Recognizing affective intent in robot directed speech. Auton Robots 12(1):83–104CrossRefzbMATHGoogle Scholar
  7. 7.
    Littlewort GC, Bartlett MS, Chenu J, Fasel I, Kanda T, Ishiguro H, Movellan JR (2004) Towards social robots: Automatic evaluation of human–robot interaction by face detection and expression classification. Adv Neural Inform Process Syst 16:1563–1570Google Scholar
  8. 8.
    Pantic M, Rothkrantz LJM (2003) Towards an affect-sensitive multimodal human–computer interaction. Proc IEEE 91(9):1370–1390CrossRefGoogle Scholar
  9. 9.
    Conati C, Zhou X (2002) Modeling students’ emotions from cognitive appraisal in educational games. In: Proceedings of 6th international conference on intelligent tutoring systems, FranceGoogle Scholar
  10. 10.
    Backs RW, Lenneman JK, Wetzel JM, Green P (2003) Cardiac measures of driver workload during simulated driving with and without visual occlusion. Hum Factors 45(4):525–539CrossRefPubMedGoogle Scholar
  11. 11.
    Hudlicka E, McNeese MD (2002) Assessment of user affective and belief states for inference adaptation: application to an air force pilot task. User Model User Adapt Interact 12:1–47CrossRefzbMATHGoogle Scholar
  12. 12.
    Hayakawa Y, Sugano S (1998) Real time simple measurement of mental strain in machine operation. ISCIE 1998 Japan–USA symposium on Flexible Automation, Otsu, Japan, pp 35–42Google Scholar
  13. 13.
    Dana Kulic, Croft E (2003) Estimating Intent for Human–robot Interaction. In: Proceedings of IEEE international conference on advanced robotics, pp 810–815Google Scholar
  14. 14.
    Tsapatsoulis N, Karpouzis K, Stamou G, Piat F, Kollias S (2000) A fuzzy system for emotion classification based on the MPEG-4 facial definition parameter set. In: Proceedings of EUSIPCO-2000, FinlandGoogle Scholar
  15. 15.
    Petrushin VA (2000) Emotion recognition agents in real world. AAAI fall symposium on socially intelligent agents: human in the loopGoogle Scholar
  16. 16.
    Moriyama T, Saito H, Ozawa S (1999) Evaluation of the relation between emotional concepts and emotional parameters on speech. IEICE J J82-DII(10):1710–1720Google Scholar
  17. 17.
    Ark W, Dryer D, Lu D (1999) The emotion mouse human–computer interaction: ergonomics and user interfaces. In: Bullinger HJ, Ziegler J (eds) Lawrence Erlbaum Assoc, London, pp 818–823Google Scholar
  18. 18.
    Picard RW, Vyzas E, Healy J (2001) Toward machine emotional intelligence: analysis of affective psychological states. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191CrossRefGoogle Scholar
  19. 19.
    Zhao J, Kearney G (1996) Classifying facial movement by backpropagation neural networks with fuzzy inputs. In: Proceedings of international conference on neural information processing, pp 454–457Google Scholar
  20. 20.
    Qi Y, Picard RW (2002) Context-sensitive Bayesian classifiers and application to mouse pressure pattern classification. In: Proceedings of international conference on pattern recognition, CanadaGoogle Scholar
  21. 21.
    Cohen I, Garg A, Huang TS (2000) Emotion recognition using multilevel HMM. In: Proceedings of NIPS workshop on affective computing, ColoradoGoogle Scholar
  22. 22.
    Conati C (2002) Probabilistic assessment of user’s emotions in educational games. J Appl Artif Intell, special issue on “Merging Cognition and Affect in HCI 16:555–575Google Scholar
  23. 23.
    Nasoz F, Alvarez K, Lisetti C, Finkelstein N (2003) Emotion recognition from physiological signals for presence technologies. Int J Cogn Technol Work Spec Issue Presence 6:1Google Scholar
  24. 24.
    Wilson GF, Russell CA (2003) Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum Factors 45(4):635–643CrossRefPubMedGoogle Scholar
  25. 25.
    Takahashi K (2004) Remarks on emotion recognition from bio-potential signals. In: Proceedings of 2nd international conference on autonomous robots and agents, New ZealandGoogle Scholar
  26. 26.
    Kim KH, Bang SW, Kim SR (2004) Emotion recognition system using short-term monitoring of physiological signals. Med Biol Eng Comput 42:419–427CrossRefPubMedGoogle Scholar
  27. 27.
    Bradley MM (2000) Emotion and motivation. In: Cacioppo JT, Tassinary LG, Berntson G (eds) Handbook of Psychophysiology. Cambridge University Press, New York, pp 602–642Google Scholar
  28. 28.
    Rani P, Sarkar N, Smith C, Kirby L (2004) Anxiety detecting robotic systems—towards implicit human–robot collaboration. Robotica 22(1):85–95CrossRefGoogle Scholar
  29. 29.
    Pecchinenda A, Smith CA (1996) The affective significance of skin conductance activity during a difficult problem-solving task. Cogn Emotion 10(50):481–504CrossRefGoogle Scholar
  30. 30.
    Brown RM, Hall LR, Holtzer R, Brown SL, Brown NL (1997) Gender and video game performance. Sex Roles 36(11–12):793–812CrossRefGoogle Scholar
  31. 31.
    Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth & Brooks/Cole Advanced Books & Software, Pacific GrovezbMATHGoogle Scholar
  32. 32.
    Kokol P, Mernik M, Završnik J, Kancler K, Malèiæ I (1994) Decision trees and automatic learning and their use in cardiology. J Med Syst 9(4):201–206CrossRefGoogle Scholar
  33. 33.
    Downey S, Russell JM (1992) A decision tree approach to task independent speech recognition. In: Proceedings of inst acoustics autumn conf on speech and hearing, vol 14(6), pp 181–188Google Scholar
  34. 34.
    Heckerman D (1999) A tutorial on learning with Bayesian networks. In: Jordan M (ed) Learning in graphical models. MIT Press, CambridgeGoogle Scholar
  35. 35.
    Brown LE, Tsamardinos I, Aliferis CF (2004) A novel algorithm for scalable and accurate Bayesian network learning. In Proceedings of the 11th world congress on medical informatics (MEDINFO), California, September 2004Google Scholar
  36. 36.
    Catlett J (1991) On changing continuous attributes into ordered discrete attributes. In: Proceedings of Fifth European working session on learning. Springer, Berlin Heidelberg New York, pp 164–177Google Scholar
  37. 37.
    Vapnik V (1998) Statistical learning theory. Wiley, New YorkzbMATHGoogle Scholar
  38. 38.
    Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: Proceedings of ECML-98, 10th European conference on machine learning. DE, Heidelberg, pp 137–142Google Scholar
  39. 39.
    Burges C (2000) A tutorial on support vector machines for pattern recognition. In: Fayyad U (ed) Knowledge discovery and data mining. Kluwer, Norwell, pp 1–43Google Scholar
  40. 40.
    Hsu CW, Lin CJ (2002) A comparison of methods for multi-class support vector machines. IEEE Trans Neural Netw 13:415–425CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2006

Authors and Affiliations

  • Pramila Rani
    • 1
    Email author
  • Changchun Liu
    • 1
  • Nilanjan Sarkar
    • 2
  • Eric Vanman
    • 3
  1. 1.Department of Electrical EngineeringVanderbilt UniversityNashvilleUSA
  2. 2.Department of Mechanical EngineeringVanderbilt UniversityNashvilleUSA
  3. 3.Department of PsychologyGeorgia State UniversityAtlantaUSA

Personalised recommendations