Graph-Based Social Relation Reasoning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12360)


Human beings are fundamentally sociable—that we generally organize our social lives in terms of relations with other people. Understanding social relations from an image has great potential for intelligent systems such as social chatbots and personal assistants. In this paper, we propose a simpler, faster, and more accurate method named graph relational reasoning network (GR\(^2\)N) for social relation recognition. Different from existing methods which process all social relations on an image independently, our method considers the paradigm of jointly inferring the relations by constructing a social relation graph. Furthermore, the proposed GR\(^2\)N constructs several virtual relation graphs to explicitly grasp the strong logical constraints among different types of social relations. Experimental results illustrate that our method generates a reasonable and consistent social relation graph and improves the performance in both accuracy and efficiency.


Social relation reasoning Paradigm shift Graph neural networks Social relation graph 



This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 61822603, Grant U1813218, Grant U1713214, and Grant 61672306, in part by Beijing Natural Science Foundation under Grant No. L172051, in part by Beijing Academy of Artificial Intelligence (BAAI), in part by a grant from the Institute for Guo Qiang, Tsinghua University, in part by the Shenzhen Fundamental Research Fund (Subject Arrangement) under Grant JCYJ20170412170602564, and in part by Tsinghua University Initiative Scientific Research Program.

Supplementary material

504470_1_En_2_MOESM1_ESM.pdf (405 kb)
Supplementary material 1 (pdf 404 KB)


  1. 1.
    Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013)
  2. 2.
    Bugental, D.B.: Acquisition of the algorithms of social life: a domain-based approach. Psychol. Bull. 126(2), 187 (2000)CrossRefGoogle Scholar
  3. 3.
    Chen, S., Abhinav, S., Carl, V., Rahul, S., Kevin, M., Cordelia, S.: Relational action forecasting. In: CVPR (2019)Google Scholar
  4. 4.
    Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., Kalantidis, Y.: Graph-based global reasoning networks. In: CVPR, pp. 433–442 (2019)Google Scholar
  5. 5.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP (2014)Google Scholar
  6. 6.
    Conte, H.R., Plutchik, R.: A circumplex model for interpersonal personality traits. J. Pers. Soc. Psychol. 40(4), 701 (1981)CrossRefGoogle Scholar
  7. 7.
    Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: NeurIPS, pp. 3844–3852 (2016)Google Scholar
  8. 8.
    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition. In: CVPR, pp. 4690–4699 (2019)Google Scholar
  9. 9.
    Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure inference machines: recurrent neural networks for analyzing relations in group activity recognition. In: CVPR, pp. 4772–4781 (2016)Google Scholar
  10. 10.
    Fiske, A.P.: The four elementary forms of sociality: framework for a unified theory of social relations. Psychol. Rev. 99(4), 689 (1992)CrossRefGoogle Scholar
  11. 11.
    Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: ICML, pp. 1263–1272 (2017)Google Scholar
  12. 12.
    Girshick, R.: Fast R-CNN. In: ICCV, pp. 1440–1448 (2015)Google Scholar
  13. 13.
    Goel, A., Ma, K.T., Tan, C.: An end-to-end network for generating social relationship graphs. In: CVPR, pp. 11186–11195 (2019)Google Scholar
  14. 14.
    Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: IJCNN, pp. 729–734 (2005)Google Scholar
  15. 15.
    Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: NeurIPS, pp. 1024–1034 (2017)Google Scholar
  16. 16.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  17. 17.
    Kipf, T., Fetaya, E., Wang, K.C., Welling, M., Zemel, R.: Neural relational inference for interacting systems. In: ICML (2018)Google Scholar
  18. 18.
    Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)Google Scholar
  19. 19.
    Lan, T., Sigal, L., Mori, G.: Social roles in hierarchical models for human activity recognition. In: CVPR, pp. 1354–1361 (2012)Google Scholar
  20. 20.
    Li, J., Wong, Y., Zhao, Q., Kankanhalli, M.S.: Dual-glance model for deciphering social relationships. In: ICCV, pp. 2650–2659 (2017)Google Scholar
  21. 21.
    Li, Q., Han, Z., Wu, X.M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: AAAI (2018)Google Scholar
  22. 22.
    Li, W., Lu, J., Feng, J., Xu, C., Zhou, J., Tian, Q.: Bridgenet: a continuity-aware probabilistic network for age estimation. In: CVPR, pp. 1145–1154 (2019)Google Scholar
  23. 23.
    Li, W., Zhang, Y., Lv, K., Lu, J., Feng, J., Zhou, J.: Graph-based kinship reasoning network. In: ICME, pp. 1–6 (2020)Google Scholar
  24. 24.
    Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. In: ICLR (2016)Google Scholar
  25. 25.
    Lu, J., Hu, J., Tan, Y.P.: Discriminative deep metric learning for face and kinship verification. TIP 26(9), 4269–4282 (2017)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Lu, J., Tan, Y.P.: Cost-sensitive subspace analysis and extensions for face recognition. TIFS 8(3), 510–519 (2013)Google Scholar
  27. 27.
    Lu, J., Wang, G., Deng, W., Jia, K.: Reconstruction-based metric learning for unconstrained face verification. TIFS 10(1), 79–89 (2014)Google Scholar
  28. 28.
    Lu, J., Zhou, X., Tan, Y.P., Shang, Y., Zhou, J.: Neighborhood repulsed metric learning for kinship verification. TPAMI 36(2), 331–345 (2013)Google Scholar
  29. 29.
    Ramanathan, V., Yao, B., Fei-Fei, L.: Social role discovery in human events. In: CVPR, pp. 2475–2482 (2013)Google Scholar
  30. 30.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR, pp. 779–788 (2016)Google Scholar
  31. 31.
    Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. In: CVPR, pp. 7263–7271 (2017)Google Scholar
  32. 32.
    Reis, H.T., Collins, W.A., Berscheid, E.: The relationship context of human behavior and development. Psychol. Bull. 126(6), 844 (2000)CrossRefGoogle Scholar
  33. 33.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: NeurIPS, pp. 91–99 (2015)Google Scholar
  34. 34.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. TNNLS 20(1), 61–80 (2008)Google Scholar
  36. 36.
    Schlichtkrull, M., Kipf, T.N., Bloem, P., Van Den Berg, R., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. In: ESWC, pp. 593–607 (2018)Google Scholar
  37. 37.
    Shao, M., Li, L., Fu, Y.: What do you do? occupation recognition in a photo via social context. In: ICCV, pp. 3631–3638 (2013)Google Scholar
  38. 38.
    Shu, T., Xie, D., Rothrock, B., Todorovic, S., Chun Zhu, S.: Joint inference of groups, events and human roles in aerial videos. In: CVPR, pp. 4576–4584 (2015)Google Scholar
  39. 39.
    Song, Z., Wang, M., Hua, X.s., Yan, S.: Predicting occupation via human clothing and contexts. In: ICCV, pp. 1084–1091 (2011)Google Scholar
  40. 40.
    Sun, C., Karlsson, P., Wu, J., Tenenbaum, J.B., Murphy, K.: Stochastic prediction of multi-agent interactions from partial observations. In: ICLR (2019)Google Scholar
  41. 41.
    Sun, Q., Schiele, B., Fritz, M.: A domain based approach to social relation recognition. In: CVPR, pp. 3481–3490 (2017)Google Scholar
  42. 42.
    Tacchetti, A., et al.: Relational forward models for multi-agent learning. In: ICLR (2019)Google Scholar
  43. 43.
    Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: ICLR (2018)Google Scholar
  44. 44.
    Wang, G., Gallagher, A., Luo, J., Forsyth, D.: Seeing people in social context: recognizing people and social relationships. In: ECCV, pp. 169–182 (2010)Google Scholar
  45. 45.
    Wang, T., Gong, S., Zhu, X., Wang, S.: Person re-identification by video ranking. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 688–703. Springer, Cham (2014). Scholar
  46. 46.
    Wang, X., Gupta, A.: Videos as space-time region graphs. In: ECCV, pp. 399–417 (2018)Google Scholar
  47. 47.
    Xu, D., Zhu, Y., Choy, C.B., Fei-Fei, L.: Scene graph generation by iterative message passing. In: CVPR, pp. 5410–5419 (2017)Google Scholar
  48. 48.
    Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? In: ICLR (2019)Google Scholar
  49. 49.
    Zhang, M., Liu, X., Liu, W., Zhou, A., Ma, H., Mei, T.: Multi-granularity reasoning for social relation recognition from images. In: ICME, pp. 1618–1623 (2019)Google Scholar
  50. 50.
    Zhang, N., Paluri, M., Taigman, Y., Fergus, R., Bourdev, L.: Beyond frontal faces: improving person recognition using multiple cues. In: CVPR, pp. 4804–4813 (2015)Google Scholar
  51. 51.
    Zhang, Z., Luo, P., Loy, C.C., Tang, X.: From facial expression recognition to interpersonal relation prediction. IJCV 126(5), 550–569 (2018)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Zhouxia, W., Tianshui, C., Jimmy, R., Weihao, Y., Hui, C., Liang, L.: Deep reasoning with knowledge graph for social relationship understanding. In: IJCAI (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of AutomationTsinghua UniversityBeijingChina
  2. 2.State Key Lab of Intelligent Technologies and SystemsBeijingChina
  3. 3.Beijing National Research Center for Information Science and TechnologyBeijingChina
  4. 4.Tsinghua Shenzhen International Graduate SchoolTsinghua UniversityBeijingChina
  5. 5.Stanford UniversityStanfordUSA

Personalised recommendations