Analysis of Articulated Motion for Social Signal Processing

  • Georg LayherEmail author
  • Michael Glodek
  • Heiko Neumann
Part of the Cognitive Technologies book series (COGTECH)


Companion technologies aim at developing sustained long-term relationships by employing non-verbal communication (NVC) skills. Visual NVC signals can be conveyed over a variety of non-verbal channels, such as facial expressions, gestures, or spatio-temporal behavior. It remains a challenge to equip technical systems with human-like abilities to reliably and effortlessly detect and analyze such social signals. In this proposal, we focus our investigation on the modeling of visual mechanisms for the processing and analysis of human-articulated motion and posture information from spatially intermediate to remote distances. From a modeling perspective, we investigate how visual features and their integration over several stages in a processing hierarchy take part in the establishment of articulated motion representations. We build upon known structures and mechanisms in cortical networks of primates and emphasize how generic processing principles might realize the building blocks for such network-based distributed processing through learning. We demonstrate how feature representations in segregated pathways and their convergence lead to integrated form and motion representations using artificially generated articulated motion sequences.



This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Argyle, M.: Bodily Communication. Methuen & Co Ltd, London (1988)Google Scholar
  2. 2.
    Baker, C., Keysers, C., Jellema, T., Wicker, B., Perrett, D.: Neuronal representation of disappearing and hidden objects in temporal cortex of the macaque. Exp. Brain Res. 140(3), 375–381 (2001)CrossRefGoogle Scholar
  3. 3.
    Barraclough, N.E., Xiao, D., Oram, M.W., Perrett, D.: The sensitivity of primate STS neurons to walking sequences and to the degree of articulation in static images. Prog. Brain Res. 154, 135–148 (2006)CrossRefGoogle Scholar
  4. 4.
    Bayerl, P., Neumann, H.: Disambiguating visual motion through contextual feedback modulation. Neural Comput. 16(10), 2041–2066 (2004)CrossRefzbMATHGoogle Scholar
  5. 5.
    Beauchamp, M.S., Lee, K.E., Haxby, J.V., Martin, A.: FMRI responses to video and point-light displays of moving humans and manipulable objects. J. Cogn. Neurosci. 15(7), 991–1001 (2003)CrossRefGoogle Scholar
  6. 6.
    Benyon, D., Mival, O.: Landscaping personification technologies: from interactions to relationships. In: Proceedings of the CHI ’08, Extended Abstracts on Human Factors in Computing Systems, CHI EA ’08, pp. 3657–3662. ACM, New York (2008)Google Scholar
  7. 7.
    Benyon, D., Mival, O.: Scenarios for companions. In: Your Virtual Butler. Lecture Notes in Computer Science, vol. 7407, pp. 79–96. Springer, Berlin (2013)Google Scholar
  8. 8.
    Bickmore, T.W., Picard, R.W.: Establishing and maintaining long-term human-computer relationships. ACM Trans. Comput.-Hum. Interaction 12, 293–327 (2005)CrossRefGoogle Scholar
  9. 9.
    Blakemore, S.J., Decety, J.: From the perception of action to the understanding of intention. Nat. Rev. Neurosci. 2(8), 561–567 (2001)Google Scholar
  10. 10.
    Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Trans. Pattern Anal. Mach. Intell. 23(3), 257–267 (2001)CrossRefGoogle Scholar
  11. 11.
    Bouecke, J.D., Tlapale, E., Kornprobst, P., Neumann, H.: Neural mechanisms of motion detection, integration, and segregation: from biology to artificial image processing systems. EURASIP J. Adv. Signal Process. 2011(1), 781561 (2010)CrossRefGoogle Scholar
  12. 12.
    Carandini, M., Heeger, D.J., Movshon, J.A.: Linearity and gain control in V1 simple cells. Cereb. Cortex (13), 401–444 (1999)CrossRefGoogle Scholar
  13. 13.
    Carpenter, G.A.: Neural network models for pattern recognition and associative memory. Neural Netw. 2(4), 243–257 (1989)CrossRefGoogle Scholar
  14. 14.
    Casile, A., Giese, M.A.: Critical features for the recognition of biological motion. J. Vis. 5(4), 6 (2005)CrossRefGoogle Scholar
  15. 15.
    Castellano, G., McOwan, P.W.: Towards affect sensitive and socially perceptive companions. In: Your Virtual Butler. Lecture Notes in Computer Science, vol. 7407, pp. 42–53. Springer, Berlin (2013)Google Scholar
  16. 16.
    Dollár, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. In: 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, 2005, pp. 65–72. IEEE, New York (2005)Google Scholar
  17. 17.
    Escobar, M.J., Kornprobst, P.: Action recognition via bio-inspired features: the richness of center–surround interaction. Comput. Vis. Image Underst. 116(5), 593–605 (2012)CrossRefGoogle Scholar
  18. 18.
    Escobar, M.J., Masson, G.S., Vieville, T., Kornprobst, P.: Action recognition using a bio-inspired feedforward spiking network. Int. J. Comput. Vis. 82(3), 284–301 (2009)CrossRefGoogle Scholar
  19. 19.
    Frith, C.D., Wolpert, D.M.: The Neuroscience of Social Interaction: Decoding, Imitating, and Influencing the Actions of Others. Oxford University Press, Oxford (2004)Google Scholar
  20. 20.
    Giese, M.A., Poggio, T.: Neural mechanisms for the recognition of biological movements. Nat. Rev. Neurosci. 4(3), 179–192 (2003)CrossRefGoogle Scholar
  21. 21.
    Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2247–2253 (2007)CrossRefGoogle Scholar
  22. 22.
    Grüsser, O.J.: Grundlagen der neuronalen Informationsverarbeitung in den Sinnesorganen und im Gehirn. In: GI - 8. Jahrestagung, pp. 234–273. Springer, Berlin (1978)Google Scholar
  23. 23.
    Hansen, T., Neumann, H.: A recurrent model of contour integration in primary visual cortex. J. Vis. 8(8), 1–25 (2008)CrossRefGoogle Scholar
  24. 24.
    Jellema, T., Perrett, D.I.: Cells in monkey STS responsive to articulated body motions and consequent static posture: a case of implied motion? Neuropsychologia 41(13), 1728–1737 (2003)CrossRefGoogle Scholar
  25. 25.
    Jellema, T., Maassen, G., Perrett, D.I.: Single cell integration of animate form, motion and location in the superior temporal cortex of the macaque monkey. Cereb. Cortex 14(7), 781–790 (2004)CrossRefGoogle Scholar
  26. 26.
    Jhuang, H., Serre, T., Wolf, L., Poggio, T.: A biologically inspired system for action recognition. In: Proceedings of the 11th IEEE International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  27. 27.
    Johansson, G.: Visual perception of biological motion and a model for its analysis. Percept. Psychophys. 14(2), 201–211 (1973)CrossRefGoogle Scholar
  28. 28.
    Kourtzi, Z., Kanwisher, N.: Activation in human MT/MST by static images with implied motion. J. Cogn. Neurosci. 12(1), 48–55 (2000)CrossRefGoogle Scholar
  29. 29.
    Lange, J., Lappe, M.: A model of biological motion perception from configural form cues. J. Neurosci. 26(11), 2894–2906 (2006)CrossRefGoogle Scholar
  30. 30.
    Lappe, M.: Perception of biological motion as motion-from-form. e-Neuroforum 3(3), 67–73 (2012)Google Scholar
  31. 31.
    Laptev, I.: On space-time interest points. Int. J. Comput. Vis. 64(2-3), 107–123 (2005)CrossRefGoogle Scholar
  32. 32.
    Laptev, I., Caputo, B., Schüldt, C., Lindeberg, T.: Local velocity-adapted motion events for spatio-temporal recognition. Comput. Vis. Image Underst. 108(3), 207–229 (2007)CrossRefGoogle Scholar
  33. 33.
    Layher, G., Giese, M.A., Neumann, H.: Learning representations of animated motion sequences - a neural model. Top. Cogn. Sci. 6(1), 170–182 (2014)CrossRefGoogle Scholar
  34. 34.
    Oja, E.: Simplified neuron model as a principal component analyzer. J. Math. Biol. 15(3), 267–273 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Pentland, A.: Social Signal Processing. IEEE Signal Process. Mag. 24(4), 108–111 (2007)CrossRefGoogle Scholar
  36. 36.
    Raudies, F., Mingolla, E., Neumann, H.: A model of motion transparency processing with local center-surround interactions and feedback. Neural Comput. 1–45 (2011)Google Scholar
  37. 37.
    Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat. Neurosci. 2, 1019–1025 (1999)CrossRefGoogle Scholar
  38. 38.
    Rittscher, J., Blake, A., Hoogs, A., Stein, G.: Mathematical modelling of animate and intentional motion. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 358(1431), 475–490 (2003)CrossRefGoogle Scholar
  39. 39.
    Schindler, K., Van Gool, L.: Action snippets: how many frames does human action recognition require? In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE Computer Society, New York (2008)Google Scholar
  40. 40.
    Senior, C., Barnes, J., Giampietroc, V., Simmons, A., Bullmore, E.T., Brammer, M., David, A.S.: The functional neuroanatomy of implicit-motion perception or ‘representational momentum’. Curr. Biol. 10(1), 16–22 (2000)CrossRefGoogle Scholar
  41. 41.
    Thirkettle, M., Benton, C.P., Scott-Samuel, N.E.: Contributions of form, motion and task to biological motion perception. J. Vis. 9(3), 28 (2009)CrossRefGoogle Scholar
  42. 42.
    Thompson, J.C., Clarke, M., Stewart, T., Puce, A.: Configural processing of biological motion in human superior temporal sulcus. J. Neurosci. 25(39), 9059–9066 (2005)CrossRefGoogle Scholar
  43. 43.
    Turaga, P., Chellappa, R., Subrahmanian, V.S., Udrea, O.: Machine recognition of human activities: a survey. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1473–1488 (2008)CrossRefGoogle Scholar
  44. 44.
    Ungerleider, L.G., Pasternak, T.: Ventral and dorsal cortical processing streams. Vis. Neurosci. 1(34), 541–562 (2004)Google Scholar
  45. 45.
    Wallis, G., Rolls, E.: Invariant face and object recognition in the visual system. Prog. Neurobiol. 51(2), 167–194 (1997)CrossRefGoogle Scholar
  46. 46.
    Weidenbacher, U., Neumann, H.: Extraction of surface-related features in a recurrent model of V1-V2 interactions. PloS ONE 4(6), e5909 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Institute of Neural Information ProcessingUlm University89069 UlmGermany

Personalised recommendations