Abstract
Recently, video streams have occupied a large proportion of Internet traffic, most of which contain human faces. Hence, it is necessary to predict saliency on multiple-face videos, which can provide attention cues for many content based applications. However, most of multiple-face saliency prediction works only consider visual information and ignore audio, which is not consistent with the naturalistic scenarios. Several behavioral studies have established that sound influences human attention, especially during the speech turn-taking in multiple-face videos. In this paper, we thoroughly investigate such influences by establishing a large-scale eye-tracking database of Multiple-face Video in Visual-Audio condition (MVVA). Inspired by the findings of our investigation, we propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face. The visual branch takes the RGB frames as the input and encodes them into visual feature maps. The audio and face branches encode the audio signal and multiple cropped faces, respectively. A fusion module is introduced to integrate the information from three modalities, and to generate the final saliency map. Experimental results show that the proposed method outperforms 11 state-of-the-art saliency prediction works. It performs closer to human multi-modal attention.
Y. Liu, B. Li, W. Hu are with National Laboratory of Pattern Recognition, Institution of Automation, Chinese Academy of Sciences (CASIA), the School of Artificial Intelligence (AI), University of Chinese Academy of Sciences (UCAS) and CAS Center for Excellence in Brain Science and Intelligence Technology (CEBSIT).
Y. Liu and M. Qiao—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
\(\mathcal {N}_{n,t}(\mathbf {x})=\mathrm {exp}\{-\frac{1}{2}(\mathbf {x}-\mu _{n,t})^T\Sigma _{n,t}^{-1}(\mathbf {x}-\mu _{n,t})\}\).
References
Bak, C., Kocak, A., Erdem, E., Erdem, A.: Spatio-temporal saliency networks for dynamic saliency prediction. IEEE Trans. Multimedia 20(7), 1688–1698 (2017)
Borji, A.: Saliency prediction in the deep learning era: an empirical investigation. arXiv preprint arXiv:1810.03716 (2018)
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., Durand, F.: What do different evaluation metrics tell us about saliency models? IEEE Trans. Pattern Anal. Mach. Intell. (2018)
Cornia, M., Baraldi, L., Serra, G., Cucchiara, R.: Predicting human eye fixations via an LSTM-based saliency attentive model. IEEE Trans. Image Process. 27(10), 5142–5154 (2018)
Coutrot, A., Guyader, N.: Toward the introduction of auditory information in dynamic visual attention models. In: International Workshop on Image Analysis for Multimedia Interactive Services, pp. 1–4. IEEE (2013)
Coutrot, A., Guyader, N.: An audiovisual attention model for natural conversation scenes. In: IEEE International Conference on Image Processing, pp. 1100–1104. IEEE (2014)
Coutrot, A., Guyader, N.: How saliency, faces, and sound influence gaze in dynamic social scenes. J. Vis. 14(8), 5 (2014)
Coutrot, A., Guyader, N.: An efficient audiovisual saliency model to predict eye positions when looking at conversations. In: European Signal Processing Conference, pp. 1531–1535. IEEE (2015)
Coutrot, A., Guyader, N., Ionescu, G., Caplier, A.: Influence of soundtrack on eye movements during video exploration. J. Eye Mov. Res. 5(4), 2 (2012)
Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)
Hershey, S., et al.: CNN architectures for large-scale audio classification. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 131–135. IEEE (2017)
Hossein Khatoonabadi, S., Vasconcelos, N., Bajic, I.V., Shan, Y.: How many bits does it take for a stimulus to be salient? In: IEEE Conference on Computer Vision and Pattern (2015)
Huang, X., Shen, C., Boix, X., Zhao, Q.: SALICON: reducing the semantic gap in saliency prediction by adapting deep neural networks. In: IEEE International Conference on Computer Vision (2015)
Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 11, 1254–1259 (1998)
Jiang, L., Xu, M., Liu, T., Qiao, M., Wang, Z.: DeepVS: a deep learning based video saliency prediction approach. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 625–642. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_37
Kayser, C., Petkov, C.I., Lippert, M., Logothetis, N.K.: Mechanisms for allocating auditory attention: an auditory saliency map. Curr. Biol. 15(21), 1943–1947 (2005)
Li, C., Xu, M., Du, X., Wang, Z.: Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model. In: ACM International Conference on Multimedia, pp. 932–940 (2018)
Liu, Y., Zhang, S., Xu, M., He, X.: Predicting salient face in multiple-face videos. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4420–4428 (2017)
Marighetto, P., et al.: Audio-visual attention: eye-tracking dataset and analysis toolbox. In: IEEE International Conference on Image Processing, pp. 1802–1806. IEEE (2017)
Min, K., Corso, J.J.: TASED-Net: temporally-aggregating spatial encoder-decoder network for video saliency detection (2019)
Pan, J., Ferrer, C.C., McGuinness, K., O’Connor, N.E., Torres, J., Sayrol, E., Giro-iNieto, X.: Salgan: Visual saliency prediction with generative adversarial networks. arXiv preprint arXiv:1701.01081 (2017)
Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
SR-Research: Eyelink 1000 plus. https://www.sr-research.com/products/eyelink-1000-plus/
Tavakoli, H.R., Borji, A., Rahtu, E., Kannala, J.: DAVE: a deep audio-visual embedding for dynamic saliency prediction. arXiv preprint arXiv:1905.10693 (2019)
Tsiami, A., Katsamanis, A., Maragos, P., Vatakis, A.: Towards a behaviorally-validated computational audiovisual saliency model. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2847–2851. IEEE (2016)
Wang, W., Shen, J.: Deep visual attention prediction. IEEE Trans. Image Process. 27(5), 2368–2378 (2017)
Wang, W., Shen, J., Guo, F., Cheng, M.M., Borji, A.: Revisiting video saliency: a large-scale benchmark and a new model. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4894–4903 (2018)
Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)
Xu, M., Liu, Y., Hu, R., He, F.: Find who to look at: turning from action to saliency. IEEE Trans. Image Process. 27(9), 4529–4544 (2018)
Xu, M., Song, Y., Wang, J., Qiao, M., Huo, L., Wang, Z.: Predicting head movement in panoramic video: a deep reinforcement learning approach. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2693–2708 (2019)
Zanca, D., Melacci, S., Gori, M.: Gravitational laws of focus of attention. IEEE (2019)
Zhang, J., Sclaroff, S.: Exploiting surroundedness for saliency detection: a boolean map approach. IEEE Trans. Pattern Anal. Mach. Intell., 889–902 (2016)
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)
Zoom: Zoom announces lineup of global technology and thought leaders for zoomtopia 2018. https://blog.zoom.us/wordpress/2018/07/11/zoom-announces-lineup-of-global-technology-and-thought-leaders-for-zoomtopia-2018/
Acknowledgement
This work is supported by Beijing Natural Science Foundation (Grant No. L172051, JQ18018), the Natural Science Foundation of China (Grant No. 61902401, 61972071, 61751212, 61721004, 61876013, 61922009, 61573037 and U1803119), the NSFC-general technology collaborative Fund for basic research (Grant No. U1636218, U1936204), CAS Key Research Program of Frontier Sciences (Grant No. QYZDJ-SSW-JSC040), CAS External cooperation key project, and NSF of Guangdong (No. 2018B030311046). Bing Li is also supported by CAS Youth Innovation Promotion Association.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Y., Qiao, M., Xu, M., Li, B., Hu, W., Borji, A. (2020). Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12365. Springer, Cham. https://doi.org/10.1007/978-3-030-58565-5_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-58565-5_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58564-8
Online ISBN: 978-3-030-58565-5
eBook Packages: Computer ScienceComputer Science (R0)