Seeing People in Social Context: Recognizing People and Social Relationships

  • Gang Wang
  • Andrew Gallagher
  • Jiebo Luo
  • David Forsyth
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6315)


The people in an image are generally not strangers, but instead often share social relationships such as husband-wife, siblings, grandparent-child, father-child, or mother-child. Further, the social relationship between a pair of people influences the relative position and appearance of the people in the image. This paper explores using familial social relationships as context for recognizing people and for recognizing the social relationships between pairs of people. We introduce a model for representing the interaction between social relationship, facial appearance, and identity. We show that the family relationship a pair of people share influences the relative pairwise features between them. The experiments on a set of personal collections show significant improvement in people recognition is achieved by modeling social relationships, even in a weak label setting that is attractive in practical applications. Furthermore, we show the social relationships are effectively recognized in images from a separate test image collection.


  1. 1.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. JMLR 3, 1107–1135 (2003)zbMATHCrossRefGoogle Scholar
  2. 2.
    Berg, T., Berg, A., Edwards, J., Maire, M., White, R., Teh, Y.W., Learned-Miller, E., Forsyth, D.: Names and faces in the news. In: Proc. CVPR (2004)Google Scholar
  3. 3.
    Chen, L., Hu, B., Zhang, L., Li, M., Zhang, H.J.: Face annotation for family photo album management. IJIG 3(1), 81–94 (2003)Google Scholar
  4. 4.
    Divvala, S.K., Hoiem, D., Hays, J., Efros, A., Hebert, M.: An empirical study of context in object detection. In: Proc. CVPR (2009)Google Scholar
  5. 5.
    Gallagher, A., Chen, T.: Estimating age, gender, and identity using first name priors. In: Proc. CVPR (2008)Google Scholar
  6. 6.
    Gallagher, A., Chen, T.: Understanding Images of Groups of People. In: Proc. CVPR (2009)Google Scholar
  7. 7.
    Guo, G., Fu, Y., Dyer, C., Huang, T.: Image-based human age estimation by manifold learning and locally adjusted robust regression. In: IEEE Trans. on Image Proc. (2008)Google Scholar
  8. 8.
    Gupta, A., Davis, L.S.: Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 16–29. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Hoiem, D., Efros, A.A., Hebert, M.: Putting objects in perspective. IJCV 80(1), 3–15 (2008)CrossRefGoogle Scholar
  10. 10.
    Kapoor, A., Hua, G., Akbarzadeh, A., Baker, S.: Which Faces to Tag: AddingPrior Constraints into Active Learning. In: Proc. ICCV (2009)Google Scholar
  11. 11.
    Kumar, N., Berg, A., Belhumeur, P., Nayar, S.: Attribute and simile classifiers for face verification. In: Proc. ICCV (2009)Google Scholar
  12. 12.
    Naaman, M., Yeh, R., Garcia-Molina, H., Paepcke, A.: Leveraging context to resolve identity in photo albums. In: Proc. JCDL (2005)Google Scholar
  13. 13.
    National Center for Health Statistics. CDC growth charts, United States (2007),
  14. 14.
    O’Hare, N., Smeaton, A.: Context-aware person identification in personal photo collections. IEEE Trans. MM (2009)Google Scholar
  15. 15.
    Parikh, D., Zitnick, L., Chen, T.: From appearance to context-based recognition: Dense labeling in small images. In: Proc. CVPR (2008)Google Scholar
  16. 16.
    Pellegrini, S., Ess, A., Schindler, K., van Gool, L.: You’ll never walk alone: Modeling social behavior for multi-target tracking. In: Proc. ICCV (2009)Google Scholar
  17. 17.
    Song, Y., Leung, T.: Context-aided human recognition- clustering. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 382–395. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Stone, Z., Zickler, T., Darrell, T.: Autotagging Facebook: Social network context improves photo annotation. In: Proc. CVPR Internet Vision Workshop (2008)Google Scholar
  19. 19.
    Tian, Y., Liu, W., Xian, R., Wen, F., Tang, X.: A face annotation framework with partial clustering and interactive labeling. In: Proc. CVPR (2007)Google Scholar
  20. 20.
    Zhao, M., Teo, Y.W., Liu, S., Chua, T., Jain, R.: Automatic person annotation of family photo album. In: Sundaram, H., Naphade, M., Smith, J.R., Rui, Y. (eds.) CIVR 2006. LNCS, vol. 4071, pp. 163–172. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Gang Wang
    • 1
  • Andrew Gallagher
    • 2
  • Jiebo Luo
    • 2
  • David Forsyth
    • 1
  1. 1.University of Illinois at Urbana-ChampaignUrbana
  2. 2.Kodak Research LaboratoriesRochester

Personalised recommendations