A Game-Theoretic Probabilistic Approach for Detecting Conversational Groups

  • Sebastiano VasconEmail author
  • Eyasu Zemene Mequanint
  • Marco Cristani
  • Hayley Hung
  • Marcello Pelillo
  • Vittorio Murino
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9007)


A standing conversational group (also known as F-formation) occurs when two or more people sustain a social interaction, such as chatting at a cocktail party. Detecting such interactions in images or videos is of fundamental importance in many contexts, like surveillance, social signal processing, social robotics or activity classification. This paper presents an approach to this problem by modeling the socio-psychological concept of an F-formation and the biological constraints of social attention. Essentially, an F-formation defines some constraints on how subjects have to be mutually located and oriented while the biological constraints defines the plausible zone in which persons can interact. We develop a game-theoretic framework embedding these constraints, which is supported by a statistical modeling of the uncertainty associated with the position and orientation of people. First, we use a novel representation of the affinity between pairs of people expressed as a distance between distributions over the most plausible oriented region of attention.Additionally, we integrate temporal information over multiple frames to smooth noisy head orientation and pose estimates, solve ambiguous situations and establish a more precise social context. We do this in a principled way by using recent notions from multi-payoff evolutionary game theory. Experiments on several benchmark datasets consistently show the superiority of the proposed approach over state of the art and its robustness under severe noise conditions.


Nash Equilibrium Mixed Strategy Pure Strategy Replicator Dynamic Evolutionary Stable Strategy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

Supplementary material (mp4 20,977 KB)


  1. 1.
    Groh, G., Lehmann, A., Reimers, J., Frieß, M.R., Schwarz, L.: Detecting social situations from interaction geometry. In: 2010 IEEE Second International Conference on Social Computing (SocialCom), pp. 1–8. IEEE (2010)Google Scholar
  2. 2.
    Li, R., Porfilio, P., Zickler, T.: Finding group interactions in social clutter. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  3. 3.
    Gan, T., Wong, Y., Zhang, D., Kankanhalli, M.S.: Temporal encoded F-formation system for social interaction detection. In: Proceedings of the 21st ACM International Conference on Multimedia, MM 2013, pp. 937–946. ACM, New York, NY, USA (2013)Google Scholar
  4. 4.
    Marin-Jimenez, M., Zisserman, A., Ferrari, V.: Here’s looking at you, kid. Detecting people looking at each other in videos. In: British Machine Vision Conference (2011)Google Scholar
  5. 5.
    Zen, G., Lepri, B., Ricci, E., Lanz, O.: Space speaks: towards socially and personality aware visual surveillance. In: 1st ACM International Workshop on Multimodal Pervasive Video Analysis, pp. 37–42 (2010)Google Scholar
  6. 6.
    Hung, H., Kröse, B.: Detecting F-formations as dominant sets. In: ICMI (2011)Google Scholar
  7. 7.
    Cristani, M., Bazzani, L., Paggetti, G., Fossati, A., Tosato, D., Del Bue, A., Menegaz, G., Murino, V.: Social interaction discovery by statistical analysis of F-formations. In: Proceedings of the BMVC, pp. 23.1–23.12. BMVA Press (2011)Google Scholar
  8. 8.
    Lan, T., Wang, Y., Yang, W., Robinovitch, S.N., Mori, G.: Discriminative latent models for recognizing contextual group activities. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1549–1562 (2012)CrossRefGoogle Scholar
  9. 9.
    Yu, T., Lim, S., Patwardhan, K.A., Krahnstoever, N.: Monitoring, recognizing and discovering social networks. In: CVPR (2009)Google Scholar
  10. 10.
    Tran, K., Gala, A., Kakadiaris, I., Shah, S.: Activity analysis in crowded environments using social cues for group discovery and human interaction modeling. Pattern Recogn. Lett. 44, 49–57 (2013)CrossRefGoogle Scholar
  11. 11.
    Kendon, A.: Conducting Interaction: Patterns of Behavior in Focused Encounters (Studies in Interactional Sociolinguistics). Cambridge University Press, Cambridge (1990)Google Scholar
  12. 12.
    Hüttenrauch, H., Eklundh, K.S., Green, A., Topp, E.A.: Investigating spatial relationships in human-robot interaction. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5052–5059. IEEE (2006)Google Scholar
  13. 13.
    Hall, E.T.: The Hidden Dimension. Anchor, New York (1990)Google Scholar
  14. 14.
    Goffman, E.: Behavior in Public Places: Notes on the Social Organization of Gatherings. Free Press, New York (1966)Google Scholar
  15. 15.
    Ciolek, T.M., Kendon, A.: Environment and the spatial arrangement of conversational encounters. Sociol. Inq. 50, 237–271 (1980)CrossRefGoogle Scholar
  16. 16.
    Chen, C., Odobez, J.: We are not contortionists: coupled adaptive learning for head and body orientation estimation in surveillance video. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1544–1551. IEEE (2012)Google Scholar
  17. 17.
    Jain, V., Crowley, J.L.: Head pose estimation using multi-scale gaussian derivatives. In: Kämäräinen, J.-K., Koskela, M. (eds.) SCIA 2013. LNCS, vol. 7944, pp. 319–328. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  18. 18.
    Setti, F., Hung, H., Cristani, M.: Group detection in still images by F-formation modeling: a comparative study. In: WIAMIS (2013)Google Scholar
  19. 19.
    Torsello, A., Rota Bulò, S., Pelillo, M.: Grouping with asymmetric affinities: a game-theoretic perspective. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 292–299 (2006)Google Scholar
  20. 20.
    Rota Buló, S., Pelillo, M.: A game-theoretic approach to hypergraph clustering. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1312–1327 (2013)CrossRefGoogle Scholar
  21. 21.
    Somasundaram, K., Baras, J.S.: Achieving symmetric Pareto Nash equilibria using biased replicator dynamics. In: 48th IEEE Conference Decision Control, pp. 7000–7005 (2009)Google Scholar
  22. 22.
    Pellegrini, S., Ess, A., Van Gool, L.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 452–465. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  23. 23.
    Yamaguchi, K., Berg, A., Ortiz, L., Berg, T.: Who are you with and where are you going? In: IEEE Conference on Computer Vision and Patter Recognition (CVPR) (2011)Google Scholar
  24. 24.
    Ge, W., Collins, R.T., Ruback, R.B.: Vision-based analysis of small groups in pedestrian crowds. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1003–1016 (2012)CrossRefGoogle Scholar
  25. 25.
    Qin, Z., Shelton, C.R.: Improving multi-target tracking via social grouping. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  26. 26.
    Chang, M., Krahnstoever, N., Ge, W.: Probabilistic group-level motion analysis and scenario recognition. In: IEEE ICCV (2011)Google Scholar
  27. 27.
    Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B.: Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker. In: IEEE International Conference on Computer Vision Workshops (ICCVW). 1st Workshop on Modeling, Simulation and Visual Analysis of Large Crowds (2011)Google Scholar
  28. 28.
    Mckenna, S.J., Jabri, S., Duric, Z., Wechsler, H., Rosenfeld, A.: Tracking groups of people. Comput. Vis. Image Underst. 80, 42–56 (2000)CrossRefzbMATHGoogle Scholar
  29. 29.
    Cupillard, F., Bremond, F., Thonnat, M.: Tracking groups of people for video surveillance. In: Remagnino, P., Jones, G.A., Paragios, N., Regazzoni, C.S. (eds.) Video-Based Surveillance Systems, pp. 89–100. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  30. 30.
    Marques, J.S., Jorge, P.M., Abrantes, A.J., Lemos, J.M.: Tracking groups of pedestrians in video sequences. In: IEEE Conference on Computer Vision and Patter Recognition Workshops (CVPR Workshops), vol. 9, pp. 101–101 (2003)Google Scholar
  31. 31.
    Pellegrini, S., Ess, A., Schindler, K., Gool, L.J.V.: You’ll never walk alone: modeling social behavior for multi-target tracking. In: ICCV 2009, pp. 261–268 (2009)Google Scholar
  32. 32.
    Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51, 4282 (1995)CrossRefGoogle Scholar
  33. 33.
    Smith, K., Ba, S.O., Odobez, J.M., Gatica-Perez, D.: Tracking the visual focus of attention for a varying number of wandering people. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1212–1229 (2008)CrossRefGoogle Scholar
  34. 34.
    Adams, R.B.: The Science of Social Vision, vol. 7. Oxford University Press, New York (2011)Google Scholar
  35. 35.
    Kendon, A.: Some functions of gaze-direction in social interaction. Acta Psychol (Amst) 26, 22–63 (1967)CrossRefGoogle Scholar
  36. 36.
    Jovanovic, N., op den Akker, R.: Towards automatic addressee identification in multi-party dialogues. In: Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pp. 89–92, Pennsylvania, USA. Association for Computational Linguistics (2004). Imported from HMIGoogle Scholar
  37. 37.
    Duncan, S.: Some signals and rules for taking speaking turns in conversations. J. Pers. Soc. Psychol. 23, 283–292 (1972)CrossRefGoogle Scholar
  38. 38.
    Ba, S.O., Odobez, J.: Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Trans. Pattern Anal. Mach. Intell. 33, 101–116 (2011)CrossRefGoogle Scholar
  39. 39.
    Subramanian, R., Staiano, J., Kalimeri, K., Sebe, N., Pianesi, F.: Putting the pieces together: multimodal analysis of social attention in meetings. In: Proceedings of the International Conference on Multimedia, MM 2010, pp. 659–662. ACM, New York, NY, USA (2010)Google Scholar
  40. 40.
    Pavan, M., Pelillo, M.: Dominant sets and pairwise clustering. IEEE Trans. Pattern Anal. Mach. Intell. 29, 167–172 (2007)CrossRefGoogle Scholar
  41. 41.
    Setti, F., Lanz, O., Ferrario, R., Murino, V., Cristani, M.: Multi-scale F-formation discovery for group detection. In: International Conference on Image Processing (ICIP) (2013)Google Scholar
  42. 42.
    Weibull, J.W.: Evolutionary Game Theory. MIT Press, Cambridge (2005)Google Scholar
  43. 43.
    Blackwell, D.: An analog of the minimax theorem for vector payoffs. Pacific J. Math. 6, 1–8 (1956)CrossRefzbMATHMathSciNetGoogle Scholar
  44. 44.
    Shapley, L.S.: Equilibrium points in games with vector payoffs. Naval Res. Logist. Q. 6, 57–61 (1959)CrossRefMathSciNetGoogle Scholar
  45. 45.
    Contini, B.M.: A decision model under uncertainty with multiple objectives. In: Mensch, A. (ed.) Theory of Games: Techniques and Applications. American Elsevier, New York (1966)Google Scholar
  46. 46.
    Zeleny, M.: Games with multiple payoffs. Int. J. Game Theory 4, 179–191 (1975)CrossRefzbMATHMathSciNetGoogle Scholar
  47. 47.
    Kawamura, T., Kanazawa, T., Ushio, T.: Evolutionarily and neutrally stable strategies in multicriteria games. IEICE Trans. Fundam. Electr. Commun. Comp. Sci. E96–A, 814–820 (2013)CrossRefGoogle Scholar
  48. 48.
    Ehrgott, M.: Multicriteria Optimization, 2nd edn. Springer, Berlin (2005)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Sebastiano Vascon
    • 1
    Email author
  • Eyasu Zemene Mequanint
    • 2
  • Marco Cristani
    • 1
    • 3
  • Hayley Hung
    • 4
  • Marcello Pelillo
    • 2
  • Vittorio Murino
    • 1
    • 3
  1. 1.Department of Pattern Analysis and Computer Vision (PAVIS)Istituto Italiano di TecnologiaGenovaItaly
  2. 2.Department of Environmental Sciences, Informatics and StatisticsUniversity Ca’ Foscari of VeniceVeneziaItaly
  3. 3.Department of Computer ScienceUniversity of VeronaVeronaItaly
  4. 4.Faculty of Electrical Engineering, Mathematics and Computer ScienceTechnical University of DelftDelftThe Netherlands

Personalised recommendations