Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)


Recently, video streams have occupied a large proportion of Internet traffic, most of which contain human faces. Hence, it is necessary to predict saliency on multiple-face videos, which can provide attention cues for many content based applications. However, most of multiple-face saliency prediction works only consider visual information and ignore audio, which is not consistent with the naturalistic scenarios. Several behavioral studies have established that sound influences human attention, especially during the speech turn-taking in multiple-face videos. In this paper, we thoroughly investigate such influences by establishing a large-scale eye-tracking database of Multiple-face Video in Visual-Audio condition (MVVA). Inspired by the findings of our investigation, we propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face. The visual branch takes the RGB frames as the input and encodes them into visual feature maps. The audio and face branches encode the audio signal and multiple cropped faces, respectively. A fusion module is introduced to integrate the information from three modalities, and to generate the final saliency map. Experimental results show that the proposed method outperforms 11 state-of-the-art saliency prediction works. It performs closer to human multi-modal attention.


Visual-audio Saliency prediction Multiple-face video. 



This work is supported by Beijing Natural Science Foundation (Grant No. L172051, JQ18018), the Natural Science Foundation of China (Grant No. 61902401, 61972071, 61751212, 61721004, 61876013, 61922009, 61573037 and U1803119), the NSFC-general technology collaborative Fund for basic research (Grant No. U1636218, U1936204), CAS Key Research Program of Frontier Sciences (Grant No. QYZDJ-SSW-JSC040), CAS External cooperation key project, and NSF of Guangdong (No. 2018B030311046). Bing Li is also supported by CAS Youth Innovation Promotion Association.

Supplementary material (35 mb)
Supplementary material 1 (zip 35830 KB)


  1. 1.
    Bak, C., Kocak, A., Erdem, E., Erdem, A.: Spatio-temporal saliency networks for dynamic saliency prediction. IEEE Trans. Multimedia 20(7), 1688–1698 (2017)CrossRefGoogle Scholar
  2. 2.
    Borji, A.: Saliency prediction in the deep learning era: an empirical investigation. arXiv preprint arXiv:1810.03716 (2018)
  3. 3.
    Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., Durand, F.: What do different evaluation metrics tell us about saliency models? IEEE Trans. Pattern Anal. Mach. Intell. (2018)Google Scholar
  4. 4.
    Cornia, M., Baraldi, L., Serra, G., Cucchiara, R.: Predicting human eye fixations via an LSTM-based saliency attentive model. IEEE Trans. Image Process. 27(10), 5142–5154 (2018)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Coutrot, A., Guyader, N.: Toward the introduction of auditory information in dynamic visual attention models. In: International Workshop on Image Analysis for Multimedia Interactive Services, pp. 1–4. IEEE (2013)Google Scholar
  6. 6.
    Coutrot, A., Guyader, N.: An audiovisual attention model for natural conversation scenes. In: IEEE International Conference on Image Processing, pp. 1100–1104. IEEE (2014)Google Scholar
  7. 7.
    Coutrot, A., Guyader, N.: How saliency, faces, and sound influence gaze in dynamic social scenes. J. Vis. 14(8), 5 (2014)CrossRefGoogle Scholar
  8. 8.
    Coutrot, A., Guyader, N.: An efficient audiovisual saliency model to predict eye positions when looking at conversations. In: European Signal Processing Conference, pp. 1531–1535. IEEE (2015)Google Scholar
  9. 9.
    Coutrot, A., Guyader, N., Ionescu, G., Caplier, A.: Influence of soundtrack on eye movements during video exploration. J. Eye Mov. Res. 5(4), 2 (2012)Google Scholar
  10. 10.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)Google Scholar
  11. 11.
    Hershey, S., et al.: CNN architectures for large-scale audio classification. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 131–135. IEEE (2017)Google Scholar
  12. 12.
    Hossein Khatoonabadi, S., Vasconcelos, N., Bajic, I.V., Shan, Y.: How many bits does it take for a stimulus to be salient? In: IEEE Conference on Computer Vision and Pattern (2015)Google Scholar
  13. 13.
    Huang, X., Shen, C., Boix, X., Zhao, Q.: SALICON: reducing the semantic gap in saliency prediction by adapting deep neural networks. In: IEEE International Conference on Computer Vision (2015)Google Scholar
  14. 14.
    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 11, 1254–1259 (1998)CrossRefGoogle Scholar
  15. 15.
    Jiang, L., Xu, M., Liu, T., Qiao, M., Wang, Z.: DeepVS: a deep learning based video saliency prediction approach. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 625–642. Springer, Cham (2018). Scholar
  16. 16.
    Kayser, C., Petkov, C.I., Lippert, M., Logothetis, N.K.: Mechanisms for allocating auditory attention: an auditory saliency map. Curr. Biol. 15(21), 1943–1947 (2005)CrossRefGoogle Scholar
  17. 17.
    Li, C., Xu, M., Du, X., Wang, Z.: Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model. In: ACM International Conference on Multimedia, pp. 932–940 (2018)Google Scholar
  18. 18.
    Liu, Y., Zhang, S., Xu, M., He, X.: Predicting salient face in multiple-face videos. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4420–4428 (2017)Google Scholar
  19. 19.
    Marighetto, P., et al.: Audio-visual attention: eye-tracking dataset and analysis toolbox. In: IEEE International Conference on Image Processing, pp. 1802–1806. IEEE (2017)Google Scholar
  20. 20.
    Min, K., Corso, J.J.: TASED-Net: temporally-aggregating spatial encoder-decoder network for video saliency detection (2019)Google Scholar
  21. 21.
    Pan, J., Ferrer, C.C., McGuinness, K., O’Connor, N.E., Torres, J., Sayrol, E., Giro-iNieto, X.: Salgan: Visual saliency prediction with generative adversarial networks. arXiv preprint arXiv:1701.01081 (2017)
  22. 22.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)Google Scholar
  23. 23.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  24. 24.
  25. 25.
    Tavakoli, H.R., Borji, A., Rahtu, E., Kannala, J.: DAVE: a deep audio-visual embedding for dynamic saliency prediction. arXiv preprint arXiv:1905.10693 (2019)
  26. 26.
    Tsiami, A., Katsamanis, A., Maragos, P., Vatakis, A.: Towards a behaviorally-validated computational audiovisual saliency model. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2847–2851. IEEE (2016)Google Scholar
  27. 27.
    Wang, W., Shen, J.: Deep visual attention prediction. IEEE Trans. Image Process. 27(5), 2368–2378 (2017)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Wang, W., Shen, J., Guo, F., Cheng, M.M., Borji, A.: Revisiting video saliency: a large-scale benchmark and a new model. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4894–4903 (2018)Google Scholar
  29. 29.
    Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)Google Scholar
  30. 30.
    Xu, M., Liu, Y., Hu, R., He, F.: Find who to look at: turning from action to saliency. IEEE Trans. Image Process. 27(9), 4529–4544 (2018)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Xu, M., Song, Y., Wang, J., Qiao, M., Huo, L., Wang, Z.: Predicting head movement in panoramic video: a deep reinforcement learning approach. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2693–2708 (2019)CrossRefGoogle Scholar
  32. 32.
    Zanca, D., Melacci, S., Gori, M.: Gravitational laws of focus of attention. IEEE (2019)Google Scholar
  33. 33.
    Zhang, J., Sclaroff, S.: Exploiting surroundedness for saliency detection: a boolean map approach. IEEE Trans. Pattern Anal. Mach. Intell., 889–902 (2016)Google Scholar
  34. 34.
    Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)CrossRefGoogle Scholar
  35. 35.
    Zoom: Zoom announces lineup of global technology and thought leaders for zoomtopia 2018.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.National Laboratory of Pattern RecognitionCASIABeijingChina
  2. 2.AI SchoolUniversity of Chinese Academy of SciencesBeijingChina
  3. 3.CEBSITBeijingChina
  4. 4.The School of Electronic and Information Engineering and Hangzhou Innovation InstituteBeihang UniversityBeijingChina
  5. 5.MarkableAI Inc.New YorkUSA

Personalised recommendations