Machine Vision and Applications

, Volume 29, Issue 5, pp 873–890 | Cite as

Kinship verification from facial images and videos: human versus machine

  • Miguel Bordallo LopezEmail author
  • Abdenour Hadid
  • Elhocine Boutellaa
  • Jorge Goncalves
  • Vassilis Kostakos
  • Simo Hosio
Original Paper


Automatic kinship verification from facial images is a relatively new and challenging research problem in computer vision. It consists in automatically determining whether two persons have a biological kin relation by examining their facial attributes. In this work, we compare the performance of humans and machines in kinship verification tasks. We investigate the state-of-the-art methods in automatic kinship verification from facial images, comparing their performance with the one obtained by asking humans to complete an equivalent task using a crowdsourcing system. Our results show that machines can consistently beat humans in kinship classification tasks in both images and videos. In addition, we study the limitations of currently available kinship databases and analyzing their possible impact in kinship verification experiment and this type of comparison.


Kinship verification Face analysis Biometrics Crowdsourcing 



The support of the Academy of Finland is fully acknowledged.

Supplementary material

138_2018_943_MOESM1_ESM.pdf (287 kb)
Supplementary material 1 (pdf 287 KB)


  1. 1.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)CrossRefzbMATHGoogle Scholar
  2. 2.
    Ahonen, T., Rahtu, E., Ojansivu, V., Heikkila, J.: Recognition of blurred faces using local phase quantization. In: International Conference on Pattern Recognition, pp. 1–4 (2008)Google Scholar
  3. 3.
    Alirezazadeh, P., Fathi, A., Abdali-Mohammadi, F.: A genetic algorithm-based feature selection for kinship verification. Signal Process. Lett. IEEE 22(12), 2459–2463 (2015). CrossRefGoogle Scholar
  4. 4.
    Alvergne, A., Faurie, C., Raymond, M.: Differential facial resemblance of young children to their parents: who do children look like more? Evol. Hum. Behav. 28(2), 135–144 (2007)CrossRefGoogle Scholar
  5. 5.
    Alvergne, A., Oda, R., Faurie, C., Matsumoto-Oda, A., Durand, V., Raymond, M.: Cross-cultural perceptions of facial resemblance between kin. J. Vis. 9(6), 23–23 (2009)CrossRefGoogle Scholar
  6. 6.
    Alvergne, A., Perreau, F., Mazur, A., Mueller, U., Raymond, M.: Identification of visual paternity cues in humans. Biol. Lett. 10(4), 20140063 (2014)CrossRefGoogle Scholar
  7. 7.
    Amazon: Amazon mechanical turk. (2016).
  8. 8.
    Anastasi, J.S., Rhodes, M.G.: An own-age bias in face recognition for children and older adults. Psychon. Bull. Rev. 12(6), 1043–1047 (2005). CrossRefGoogle Scholar
  9. 9.
    Barkan, O., Weill, J., Wolf, L., Aronowitz, H.: Fast high dimensional vector multiplication face recognition. In: IEEE International Conference on Computer Vision, pp. 1960–1967 (2013)Google Scholar
  10. 10.
    Bordallo López, M., Boutellaa, E., Hadid, A.: Comments on the “kinship face in the wild” data sets. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1–1 (2016). Google Scholar
  11. 11.
    Bottino, A.G., De Simone, M., Laurentini, A., Vieira, T.: A new problem in face image analysis: finding kinship clues for siblings pairs (2012)Google Scholar
  12. 12.
    Bottino, A.G., Ul Islam, I., Vieira, T.: A multi-perspective holistic approach to kinship verification in the wild (2015)Google Scholar
  13. 13.
    Boutellaa, E., López, B., M. Ait-Aoudia, S., Feng, X., Hadid, A.: Kinship verification from videos using texture spatio-temporal features and deep learning features. In: International Conference on Biometrics (ICB’16) (2016)Google Scholar
  14. 14.
    Brédart, S., French, R.M.: Do babies resemble their fathers more than their mothers? a failure to replicate christenfeld and hill (1995). Evol. Hum. Behav. 20(2), 129–135 (1999)CrossRefGoogle Scholar
  15. 15.
    Bressan, P., Grassi, M.: Parental resemblance in 1-year-olds and the gaussian curve. Evol. Hum. Behav. 25(3), 133–141 (2004)CrossRefGoogle Scholar
  16. 16.
    Burch, R.L., Gallup, G.G.: Perceptions of paternal resemblance predict family violence. Evol. Hum. Behav. 21(6), 429–435 (2000)CrossRefGoogle Scholar
  17. 17.
    Chan, C.H., Tahir, M., Kittler, J., Pietikainen, M.: Multiscale local phase quantization for robust component-based face recognition using kernel fusion of multiple descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1164–1177 (2013)CrossRefGoogle Scholar
  18. 18.
    Chen, J., Shan, S., He, C., Zhao, G., Pietikainen, M., Chen, X., Gao, W.: Wld: a robust local image descriptor. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1705–1720 (2010)CrossRefGoogle Scholar
  19. 19.
    Chen, X., An, L., Yang, S., Wu, W.: Kinship verification in multi-linear coherent spaces. Multimed. Tools Appl. 76, 1–18 (2015)Google Scholar
  20. 20.
    Christenfeld, N., Hill, E.A., et al.: Whose baby are you. Nature 378(6558), 669–669 (1995)CrossRefGoogle Scholar
  21. 21.
    Dal Martello, M.F., Maloney, L.T.: Where are kin recognition signals in the human face? J. Vis. 6(12), 2–2 (2006)CrossRefGoogle Scholar
  22. 22.
    Dal Martello, M.F., Maloney, L.T.: Lateralization of kin recognition signals in the human face. J. Vis. 10(8), 9 (2010)CrossRefGoogle Scholar
  23. 23.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)Google Scholar
  24. 24.
    DeBruine, L.M., Smith, F.G., Jones, B.C., Roberts, S.C., Petrie, M., Spector, T.D.: Kin recognition signals in adult faces. Vis. Res. 49(1), 38–43 (2009)CrossRefGoogle Scholar
  25. 25.
    Dehghan, A., Ortiz, E., Villegas, R., Shah, M.: Who do i look like? determining parent-offspring resemblance via gated autoencoders. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1757–1764 (2014).
  26. 26.
    Dibeklioglu, H.: Visual transformation aided contrastive learning for video-based kinship verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2459–2468 (2017)Google Scholar
  27. 27.
    Dibeklioglu, H., Salah, A., Gevers, T.: Are you really smiling at me? spontaneous versus posed enjoyment smiles. In: Computer Vision ECCV 2012, vol. 7574, pp. 525–538. Springer, Berlin (2012).
  28. 28.
    Dibeklioglu, H., Salah, A., Gevers, T.: Like father, like son: Facial expression dynamics for kinship verification. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1497–1504 (2013)Google Scholar
  29. 29.
    Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2399–2402. ACM (2010)Google Scholar
  30. 30.
    Fang, R., Tang, K.D., Snavely, N., Chen, T.: Towards computational models of kinship verification. In: 2010 17th IEEE International Conference on Image Processing (ICIP), pp. 1577–1580. IEEE (2010)Google Scholar
  31. 31.
    Ghahramani, M., Yau, W.Y., Teoh, E.K.: Family verification based on similarity of individual family member’s facial segments. Mach. Vis. Appl. 25(4), 919–930 (2014)CrossRefGoogle Scholar
  32. 32.
    Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., Kostakos, V.: Motivating participation and improving quality of contribution in ubiquitous crowdsourcing. Comput. Netw. 90, 34–48 (2015)CrossRefGoogle Scholar
  33. 33.
    Guo, G., Wang, X.: Kinship measurement on salient facial features. IEEE Trans. Instrum. Meas. 61(8), 2322–2325 (2012). CrossRefGoogle Scholar
  34. 34.
    Hamilton, W.: The genetical evolution of social behaviour (1964)Google Scholar
  35. 35.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645. Springer, Berlin (2016)Google Scholar
  36. 36.
    Hills, P.J., Lewis, M.B.: The own-age face recognition bias in children and adults. Q. J. Exp. Psychol. 64(1), 17–23 (2011)CrossRefGoogle Scholar
  37. 37.
    Hu, J., Lu, J., Tan, Y.P., Yuan, J., Zhou, J.: Local large-margin multi-metric learning for face and kinship verification. IEEE Trans. Circuits Syst. Video Technol. (2017)Google Scholar
  38. 38.
    Hu, J., Lu, J., Yuan, J., Tan, Y.P.: Large margin multi-metric learning for face and kinship verification in the wild. In: Computer Vision–ACCV 2014, pp. 252–267. Springer, Berlin (2015)Google Scholar
  39. 39.
    Kaminski, G., Gentaz, E., Mazens, K.: Development of children’s ability to detect kinship through facial resemblance. Anim. Cognit. 15(3), 421–427 (2012)CrossRefGoogle Scholar
  40. 40.
    Kaminski, G., Méary, D., Mermillod, M., Gentaz, E.: Perceptual factors affecting the ability to assess facial resemblance between parents and newborns in humans. Perception 39(6), 807–818 (2010)CrossRefGoogle Scholar
  41. 41.
    Kan, M., Shan, S., Xu, D., Chen, X.: Side-information based linear discriminant analysis for face recognition. In: British Machine Vision Conference, pp. 1–12 (2011)Google Scholar
  42. 42.
    Kannala, J., Rahtu, E.: BSIF: Binarized statistical image features. In: International Conference on Pattern Recognition (ICPR), pp. 1363–1366 (2012)Google Scholar
  43. 43.
    Kohli, N., Singh, R., Vatsa, M.: Self-similarity representation of weber faces for kinship classification. In: 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 245–250 (2012)Google Scholar
  44. 44.
    Kou, L., Zhou, X., Xu, M., Shang, Y.: Learning a genetic measure for kinship verification using facial images. Math. Probl. Eng. 2015 (2015)Google Scholar
  45. 45.
    Lu, J., Hu, J., Liong, V.E., Zhou, X., Bottino, A., Islam, I.U., Vieira, T.F., Qin, X., Tan, X., Keller, Y., et al.: The FG 2015 kinship verification in the wild evaluation. Face and Gesture (FG’15) (2015)Google Scholar
  46. 46.
    Lu, J., Hu, J., Tan, Y.P.: Discriminative deep metric learning for face and kinship verification. IEEE Trans. Image Process. 26(9), 4269–4282 (2017)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Lu, J., Hu, J., Zhou, X., Zhou, J., Castrillón-Santana, M., Lorenzo-Navarro, J., Kou, L., Shang, Y., Bottino, A., Vieira, T.F.: Kinship verification in the wild: The first kinship verification competition. In: 2014 IEEE International Joint Conference on Biometrics (IJCB), pp. 1–6. IEEE (2014)Google Scholar
  48. 48.
    Lu, J., Zhou, X., Tan, Y.P., Shang, Y., Zhou, J.: Neighborhood repulsed metric learning for kinship verification. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 331–345 (2014)CrossRefGoogle Scholar
  49. 49.
    Maloney, L.T., Dal Martello, M.F.: Kin recognition and the perceived facial similarity of children. J. Vis. 6(10), 4–4 (2006)CrossRefGoogle Scholar
  50. 50.
    Mateo, J.M.: Perspectives: Hamilton’s legacy: mechanisms of kin recognition in humans. Ethology 121(5), 419–427 (2015)CrossRefGoogle Scholar
  51. 51.
    Mathews, S., Kambhamettu, C., Barner, K.: Am I your sibling? inferring kinship cues from facial image pairs. In: 2015 49th Annual Conference on Information Sciences and Systems (CISS), pp. 1–5 (2015).
  52. 52.
    McLain, D.K., Setters, D., Moulton, M.P., Pratt, A.E.: Ascription of resemblance of newborns by parents and nonrelatives. Evol. Hum. Behav. 21(1), 11–23 (2000)CrossRefGoogle Scholar
  53. 53.
    Meissner, C.A., Brigham, J.C.: Thirty years of investigating the own-race bias in memory for faces: a meta-analytic review. Psychol. Public Policy Law 7(1), 3 (2001)CrossRefGoogle Scholar
  54. 54.
    Nesse, R.M., Silverman, A., Bortz, A.: Sex differences in ability to recognize family resemblance. Ethol. Sociobiol. 11(1), 11–21 (1990). CrossRefGoogle Scholar
  55. 55.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)Google Scholar
  56. 56.
    Peleg, G., Katzir, G., Peleg, O., Kamara, M., Brodsky, L., Hel-Or, H., Keren, D., Nevo, E.: Hereditary family signature of facial expression. Proc. Natl. Acad. Sci. 103(43), 15921–15926 (2006)CrossRefGoogle Scholar
  57. 57.
    Platek, S.M., Burch, R.L., Panyavin, I.S., Wasserman, B.H., Gallup, G.G.: Reactions to children’s faces: resemblance affects males more than females. Evol. Hum. Behav. 23(3), 159–166 (2002)CrossRefGoogle Scholar
  58. 58.
    Platek, S.M., Keenan, J.P., Mohamed, F.B.: Sex differences in the neural correlates of child facial resemblance: an event-related fmri study. NeuroImage 25(4), 1336–1344 (2005)CrossRefGoogle Scholar
  59. 59.
    Platek, S.M., Kemp, S.M.: Is family special to the brain? an event-related fmri study of familiar, familial, and self-face recognition. Neuropsychologia 47(3), 849–858 (2009)CrossRefGoogle Scholar
  60. 60.
    Porter, R.H., Cernoch, J.M., Balogh, R.D.: Recognition of neonates by facial-visual characteristics. Pediatrics 74(4), 501–504 (1984)Google Scholar
  61. 61.
    Qin, X., Tan, X., Chen, S.: Tri-subject kinship verification: understanding the core of a family. IEEE Trans. Multimed. 17, 1855–1867 (2015)CrossRefGoogle Scholar
  62. 62.
    Robinson, J.P., Shao, M., Wu, Y., Fu, Y.: Families in the wild (fiw): large-scale kinship image database and benchmarks. In: Proceedings of the 2016 ACM on Multimedia Conference, MM ’16, pp. 242–246. ACM, New York, NY, USA (2016)Google Scholar
  63. 63.
    Robinson, J.P., Shao, M., Zhao, H., Wu, Y., Gillis, T., Fu, Y.: Recognizing families in the wild (rfiw): data challenge workshop in conjunction with acm mm 2017. In: Proceedings of the 2017 Workshop on Recognizing Families In the Wild, RFIW ’17, pp. 5–12. ACM, New York, NY, USA (2017)Google Scholar
  64. 64.
    Simonyan, K., Parkhi, O., Vedaldi, A., Zisserman, A.: Fisher vector faces in the wild. In: Proceedings of the British Machine Vision Conference. BMVA Press (2013)Google Scholar
  65. 65.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).
  66. 66.
    Somanath, G., Kambhamettu, C.: Can faces verify blood-relations? In: International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 105–112. IEEE (2012)Google Scholar
  67. 67.
    Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10,000 classes. In: Computer Vision and Pattern Recognition, pp. 1891–1898. IEEE, Washington, DC, USA (2014).
  68. 68.
    Wang, S., Robinson, J.P., Fu, Y.: Kinship verification on families in the wild with marginalized denoising metric learning. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), pp. 216–221 (2017)Google Scholar
  69. 69.
    Wang, X., Kambhamettu, C.: Leveraging appearance and geometry for kinship verification. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 5017–5021. IEEE (2014)Google Scholar
  70. 70.
    Wolf, L., Hassner, T., Taigman, Y.: Descriptor based methods in the wild. In: Real-Life Images workshop at the European Conference on Computer Vision (2008)Google Scholar
  71. 71.
    Wu, H., Yang, S., Sun, S., Liu, C., Luo, Y.J.: The male advantage in child facial resemblance detection: behavioral and erp evidence. Soc. Neurosci. 8(6), 555–567 (2013)CrossRefGoogle Scholar
  72. 72.
    Wu, X., Boutellaa, E., Lpez, M.B., Feng, X., Hadid, A.: On the usefulness of color for kinship verification from face images. In: 2016 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6 (2016)Google Scholar
  73. 73.
    Xia, S., Shao, M., Fu, Y.: Toward kinship verification using visual attributes. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp. 549–552. IEEE (2012)Google Scholar
  74. 74.
    Xia, S., Shao, M., Luo, J., Fu, Y.: Understanding kin relationships in a photo. IEEE Trans. Multimed. 14(4), 1046–1056 (2012)CrossRefGoogle Scholar
  75. 75.
    Yan, H., Hu, J.: Video-based kinship verification using distance metric learning. Pattern Recognit. 75, 15–24 (2018)CrossRefGoogle Scholar
  76. 76.
    Yan, H., Lu, J.: Video-based facial kinship verification. In: Facial Kinship Verification, pp. 63–80. Springer, Singapore (2017)Google Scholar
  77. 77.
    Yan, H., Lu, J., Deng, W., Zhou, X.: Discriminative multimetric learning for kinship verification. IEEE Trans. Inf. Forensics Secur. 9(7), 1169–1178 (2014)CrossRefGoogle Scholar
  78. 78.
    Yan, H.C., Lu, J., Zhou, X.: Prototype-based discriminative feature learning for kinship verification (2014)Google Scholar
  79. 79.
    Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv:1411.7923 (2014)
  80. 80.
    Zhang, J., Xia, S., Pan, H., Qin, A.: A genetics-motivated unsupervised model for tri-subject kinship verification. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2916–2920. IEEE (2016)Google Scholar
  81. 81.
    Zhang, K., Huang, Y., Song, C., Wu, H., Wang, L.: Kinship verification with deep convolutional neural networks. In: British Machine Vision Conference (BMVC), pp. 148.1–148.12 (2015)Google Scholar
  82. 82.
    Zheng, L., Idrissi, K., Garcia, C., Duffner, S., Baskurt, A. (eds.): Triangular similarity metric learning for face verification. IEEE (2015)Google Scholar
  83. 83.
    Zhou, X., Hu, J., Lu, J., Shang, Y., Guan, Y.: Kinship verification from facial images under uncontrolled conditions. In: International Conference on Multimedia, MM ’11, pp. 953–956. ACM, New York, NY, USA (2011)Google Scholar
  84. 84.
    Zhou, X., Lu, J., Hu, J., Shang, Y.: Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 725–728 (2012)Google Scholar
  85. 85.
    Zhou, X., Shang, Y., Yan, H., Guo, G.: Ensemble similarity learning for kinship verification from facial images in the wild. Inf. Fus. 32, 40–48 (2015)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Center for Machine Vision and Signal AnalysisUniversity of OuluOuluFinland
  2. 2.School of Computing and Information SystemsUniversity of MelbourneMelbourneAustralia
  3. 3.Center for Ubiquitous ComputingUniversity of OuluOuluFinland

Personalised recommendations