Skip to main content

Recognition of Emotions in Gait Patterns by Means of Artificial Neural Nets

Abstract

This paper describes an application of emotion recognition in human gait by means of kinetic and kinematic data using artificial neural nets. Two experiments were undertaken, one attempting to identify participants’ emotional states from gait patterns, and the second analyzing effects on gait patterns of listening to music while walking. In the first experiment gait was analyzed as participants attempted to simulate four distinct emotional states (normal, happy, sad, angry). In the second experiment, participants were asked to listen to different types of music (excitatory, calming, no music) before and during gait analysis. Derived data were fed into different types of artificial neural nets. Results showed not only a clear distinction between individuals, but also revealed clear indications of emotion recognition in nets.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  • Barton, G., Lees, A., Lisboa, P., & Attfield, S. (2006). Visualisation of gait data with Kohonen self-organising neural maps. Gait & Posture, 24, 46–53.

    Article  Google Scholar 

  • Bauer, H. U., & Schöllhorn, W. (1997). Self-organizing maps for the analysis of complex movement patterns. Neural Processing Letters, 5, 193–199.

    Article  Google Scholar 

  • Becker, N., Brett, S., Chambliss, C., Crowers, K., Haring, P., Marsh, C., et al. (1994). Mellow and frenetic antecedent music during athletic performance of children, adults, and seniors. Perceptual and Motor Skills, 79, 1043–1046.

    PubMed  Google Scholar 

  • Benabdelkader, C., Cutler, R. G., & Davis, L. S. (2004). Gait recognition using image self-similarity. Eurasip Journal on Applied Signal Processing, 2004, 572–585.

    Article  Google Scholar 

  • Bichot, N. P., & Desimone, R. (2006). Finding a face in the crowd: Parallel and serial neural mechanisms of visual selection. In S. Martinez-Conde, S. L. Macknik, L. M. Martinez, J.-M. Alonso, & P. U. Tse (Eds.), Progress in brain research; visual perception – fundamentals of awareness: Multi-sensory integration and high-order perception (pp. 147–156). Elsevier.

  • Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processing. Emotion, 1, 276–298.

    PubMed  Article  Google Scholar 

  • Camurri, A., Castellano, G., Ricchetti, M., & Volpe, G. (2006). Subject interfaces: Measuring bodily activation during an emotional experience of music. Gesture in Human-Computer Interaction and Simulation, 3881, 268–279.

    Article  Google Scholar 

  • Camurri, A, Mazzarino, B., & Volpe, G. (2004). Expressive interfaces. Cognition, Technology & Work, 6, 15–22.

    Article  Google Scholar 

  • Camurri, A., Lagerlof, I., & Volpe, G. (2003). Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques. International Journal of Human-Computer Studies, 59, 213–225.

    Article  Google Scholar 

  • Clarke, T. J., Bradshaw, M. F., Field, D. T., Hampson, S. E., & Rose, D. (2005). The perception of emotion from body movement in point-light displays of interpersonal dialogue. Perception, 34, 1171–1180.

    PubMed  Article  Google Scholar 

  • Coombes, S. A., Cauraugh, J. H., & Janelle, C. M. (2006). Emotion and movement: Activation of defensive circuitry alters the magnitude of a sustained muscle contraction. Neuroscience Letters, 396, 192–196.

    PubMed  Article  Google Scholar 

  • Coombes, S. A., Janelle, C. M., & Duley, A. R. (2005). Emotion and motor control: Movement attributes following affective picture processing. Journal of Motor Behavior, 37, 425–436.

    PubMed  Article  Google Scholar 

  • Coulson, M. (2004). Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28, 117–139.

    Article  Google Scholar 

  • Cutting, J. E., & Kozlowski, L. T. (1977). Recognizing friends by their walk – gait perception without familiarity cues. Bulletin of the Psychonomic Society, 9, 353–356.

    Google Scholar 

  • De Meijer, M. (1989). The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior, 13, 247–268.

    Article  Google Scholar 

  • Dittrich, W. H., Troscianko, T., Lea, S. E., & Morgan, D. (1996). Perception of emotion from dynamic point-light displays represented in dance. Perception, 25, 727–738.

    PubMed  Article  Google Scholar 

  • Ekman, P., & Friesen, W. (1978). The facial action coding system. Palo Alto: Consulting Psychologists Press.

    Google Scholar 

  • Ferguson, A. R., Carbonneau, M. R., & Chambliss, C. (1994). Effects of positive and negative music on performance of a karate drill. Perceptual and Motor Skills, 78, 1217–1218.

    PubMed  Google Scholar 

  • Fragopanagos, N., & Taylor, J. G. (2005). Emotion recognition in human–computer interaction. Neural Networks, 18, 389–405.

    PubMed  Article  Google Scholar 

  • Gunes, H., & Piccardi, M. (2005). Fusing face and body display for bi-modal emotion recognition: Single frame analysis and multi-frame post integration. Affective Computing and Intelligent Interaction, Proceedings, 3784, 102–111.

    Article  Google Scholar 

  • Guzzetta, C. E. (1989). Effects of relaxation and music-therapy on patients in a coronary-care unit with presumptive acute myocardial-infarction. Heart & Lung, 18, 609–616.

    Google Scholar 

  • Haykin, S. (1998). Neural networks: A comprehensive foundation. Prentice Hall PTR.

  • Ichimura, T., Ishida, H., Terauchi, M., Takahama, T., & Isomichi, Y. (2001). Extraction of emotion from facial expression by parallel sand glass type neural networks. In Proceedings of KES 2001, 5th International Conference on Knowledge Based Intelligent Information Engineering Systems and Allied Technology, Osaka, Japan, 6–8 Sept. 2001 (Vol. 2, pp. 988–992). Amsterdam: IOS Press.

  • Ioannou, S. V., Raouzaiou, A. T., Tzouvaras, V. A., Mailis, T. P., Karpouzis, K. C., & Kollias, S. D. (2005). Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Networks, 18, 423–435.

    PubMed  Article  Google Scholar 

  • Kaiser, S., & Wehrle, T. (1992). Automated coding of facial behavior in human–computer interactions with FACS. Journal of Nonverbal Behavior, 16, 67–84.

    Article  Google Scholar 

  • Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69.

    Article  Google Scholar 

  • Kohonen, T., Hynninen, J., Kangas, J., & Laaksonen, J. (1995). The self-organizing map program package, version 3.1. SOM Programming Team of the Helsinki University of Technology, Laboratory of Computer and Information Science, Espoo.

  • Møller, M. F. (1993). A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks, 6, 525–533.

    Article  Google Scholar 

  • Montepare, J., Koff, E., Zaitchik, D., & Albert, M. (1999). The use of body movements and gestures as cues to emotions in younger and older adults. Journal of Nonverbal Behavior, 23, 133–152.

    Article  Google Scholar 

  • Montepare, J. M., Goldstein, S. B., & Clausen, A. (1987). The identification of emotions from gait information. Journal of Nonverbal Behavior, 11, 33–42.

    Article  Google Scholar 

  • Nguyen, D., & Widrow, B. (1990). Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. Proceedings of the International Joint Conference on Neural Networks, 3, 21–26.

    Article  Google Scholar 

  • Nicholson, J., Takahashi, K., & Nakatsu, R. (2000). Emotion recognition in speech using neural networks. Neural Computing & Applications, 9, 290–296.

    Article  Google Scholar 

  • Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003). Speech emotion recognition using Hidden Markov models. Speech Communication, 41, 603–623.

    Article  Google Scholar 

  • Park, C. H., Byun, K. S., & Sim, K. B. (2005). The implementation of the emotion recognition from speech and facial expression system. Lecture Notes in Computer Science, 3611, 85–88.

    Article  Google Scholar 

  • Richardson, M. J., & Johnston, L. (2005). Person recognition from dynamic events: The kinematic specification of individual identity in walking style. Journal of Nonverbal Behavior, 29, 25–44.

    Article  Google Scholar 

  • Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.

    Article  Google Scholar 

  • Sawada, M., Suda, K., & Ishii, M. (2003). Expression of emotions in dance: Relation between arm movement characteristics and emotion. Perceptual and Motor Skills, 97, 697–708.

    PubMed  Article  Google Scholar 

  • Schöllhorn, W. I. (2004). Applications of artificial neural nets in clinical biomechanics. Clinical Biomechanics, 19, 876–898.

    PubMed  Article  Google Scholar 

  • Schöllhorn, W. I., Jäger, J. M., & Janssen, D. (2008). Artificial neural network models of sports motions. In Y.B. Hong & R. Bartlett (Eds.), Routledge handbook of biomechanics and human movement science. Routledge: London.

  • Schöllhorn, W. I., Nigg, B. M., Stefanyshyn, D. J., & Liu, W. (2002). Identification of individual walking patterns using time discrete and time continuous data sets. Gait & Posture, 15, 180–186.

    Article  Google Scholar 

  • Sloman, L., Berridge, M., Homatidis, S., Hunter, D., & Duck, T. (1982). Gait patterns of depressed patients and normal subjects. American Journal of Psychiatry, 139, 94–97.

    PubMed  Google Scholar 

  • Sloman, L., Pierrynowski, M., Berridge, M., Tupling, S., & Flowers, J. (1987). Mood, depressive illness and gait patterns. Canadian Journal of Psychiatry, 32, 190–193.

    Google Scholar 

  • Tenenbaum, G., Lidor, R., Lavyan, N., Morrow, K., Tonnel, S., Gershgoren, A., et al. (2004). The effect of music type on running perseverance and coping with effort sensations. Psychology of Sport and Exercise, 5, 89–109.

    Article  Google Scholar 

  • Vesanto, J., Himberg, J., Alhoniemi, E., & Parhankangas, J. (2000). SOM Toolbox for Matlab 5. Report A57. Helsinki University of Technology, Neural Networks Research Centre, Espoo.

  • Walk, R. D., & Homan, C. P. (1984). Emotion and dance in dynamic light displays. Bulletin of the Psychonomic Society, 22, 437–440.

    Google Scholar 

  • Wallbott, H. G., & Scherer, K. R. (1986). Cues and channels in emotion recognition. Journal of Personality and Social Psychology, 51, 690–699.

    Article  Google Scholar 

  • Yamamoto, T., Ohkuwa, T., Itoh, H., Kitoh, M., Terasawa, J., Tsuda, T., et al. (2003). Effects of pre-exercise listening to slow and fast rhythm music on supramaximal cycle performance and selected metabolic variables. Archives of Physiology and Biochemistry, 111, 211–214.

    PubMed  Article  Google Scholar 

Download references

Acknowledgments

We wish to thank Larry Katz and Veronica Everton-Williams for their useful comments.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Daniel Janssen.

Appendix

Appendix

The Nets

The processing was implemented using Matlab R2006a, the Neural Network Toolbox V5.0 and the SOM Toolbox (Vesanto et al. 2000). Before the data were fed into the networks, a further normalization to the interval [−1 to 1] was completed in order to prepare the data for the nets. For the kinetic data, the MLP consisted of three layers with 200 neurons (50 x-, 50 y-, and 100 z-data-points) in the first layer, about (n+c)/2 neurons in the hidden layer, where n is the number of input neurons and c is the number of desired classes and also the number of output neurons. The MLPs for the kinematic data were built analogically except for the first layer which contained 168 neurons (21 data points · 8 angles and angular velocities). As an activation function, the tangens hyperbolicus was chosen in all layers. The nets were initialized with the Nguyen–Widrow function (Nguyen and Widrow 1990) and trained with the scaled conjugate gradient algorithm (Møller 1993). Training lengths were set to 500 epochs in general and 600 epochs for the classification with all data from 38 participants respectively. Recognition rates were calculated counting the misclassifications and expressing them as a percentage using cross-validation.

The SOM algorithms were used in two different ways. The normal SOM was used to classify the data as usual; the 2SOM implied two connected SOMs (see Fig. 4). Within this architecture, the first SOM (SOM A) served as data reduction; the second SOM (SOM B) took note of classification tasks. To achieve this, the data vectors were presented as feature-vectors. One single vector then included the values of all angles and angular velocities for a particular point in time, what Bauer and Schöllhorn (1997) call a more ‘coordination-oriented’ approach. In this way, the process information of the movement was represented by the trajectory built by the successively activated neurons in a two-dimensional space. The trajectory of activated neurons was converted to a new data vector by using the x- and y-coordinates from each neuron as new data points. Finally, the second SOM classified the actual measurement. All used SOMs were two-dimensional maps with a hexagonal lattice and a rectangular shape, as recommended by Kohonen et al. (1995). A comparatively small lattice of 5 × 3 neurons was preferred for the data. As a training method batchtrain was chosen. The learning rate was initially set to 0.5 for the rough training phase and then reduced to 0.05 for the fine tuning phase. Training length was set to 10 epochs in rough training and 40 epochs in fine tuning. Further training was not necessary and yielded no better results. The calculation of recognition rates was achieved as follows: During repeated presentation of training data, the distances from each classified gait pattern to the ‘emotion clusters’ (not the clustering on the SOM, but the collection of gait patterns from one simulated emotion) of all emotions were calculated. If the distance to the emotion-cluster, to which the gait pattern belonged to, was not the shortest, the classification was counted as a misclassification.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Janssen, D., Schöllhorn, W.I., Lubienetzki, J. et al. Recognition of Emotions in Gait Patterns by Means of Artificial Neural Nets. J Nonverbal Behav 32, 79–92 (2008). https://doi.org/10.1007/s10919-007-0045-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10919-007-0045-3

Keywords

  • Emotion
  • Gait
  • Music
  • Neural network
  • Pattern recognition