Skip to main content
SpringerLink
Log in

Attention-embedding mesh saliency

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Recently, the learning method is gradually penetrating into the field of 3D saliency, but the ground truth annotation is too insufficient to directly train a 3D saliency network. Here, we propose a novel attention-embedding strategy for 3D saliency estimation by directly applying the attention embedding scheme to 3D mesh. With this method, the network is trained in a weakly supervised manner, requiring no saliency annotations but generalizing well on different categories of objects, such as animals, furniture, cars and people. Experimental results show that our approach is comparable with existing state-of-the-art methods. We also apply saliency results to mesh simplification. Evaluations on simplified models show that the visually significant parts can be retained during saliency-aware simplification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ba, J., Mnih, V., Kavukcuoglu, K.: Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755 (2014)

  2. Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond Euclidean data. IEEE Signal Process. Mag. 34(4), 18–42 (2017)

    Article  Google Scholar 

  3. Castellani, U., Cristani, M., Fantoni, S., Murino, V.: Sparse points matching by combining 3d mesh saliency with statistical descriptors. In: Computer Graphics Forum, vol. 27, pp. 643–652. Wiley Online Library (2008)

  4. Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. Computer Science, pp. 1–14 (2014)

  5. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T.S.: SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5659–5667 (2017)

  6. Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 234–250 (2018)

  7. Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1907–1915 (2017)

  8. Chen, X., Saparov, A., Pang, B., Funkhouser, T.: Schelling points on 3d surface meshes. ACM Trans. Graph. 31(4CD), 29.1-29.12 (2012)

    Google Scholar 

  9. Cignoni, P., Rocchini, C., Scopigno, R.: Metro: measuring error on simplified surfaces. Comput. Graph. Forum 17(2), 167–174 (2010)

    Article  Google Scholar 

  10. Cong, R., Lei, J., Zhang, C., Huang, Q., Cao, X., Hou, C.: Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Process. Lett. 23(6), 819–823 (2016)

    Article  Google Scholar 

  11. Ding, X., Lin, W., Chen, Z., Zhang, X.: Point cloud saliency detection by local and global feature fusion. IEEE Trans. Image Process. 28(11), 5379–5393 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  12. Dutagaci, H., Cheung, C.P., Godil, A.: Evaluation of 3d interest point detection techniques via human-generated ground truth. Vis. Comput. 28(9), 901–917 (2012)

    Article  Google Scholar 

  13. Engelmann, F., Kontogianni, T., Leibe, B.: Dilated point convolutions: on the receptive field size of point convolutions on 3d point clouds. In: International Conference on Robotics and Automation (ICRA), vol. 1 (2020)

  14. Gal, R., Cohen-Or, D.: Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. (TOG) 25(1), 130–150 (2006)

    Article  Google Scholar 

  15. Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. ACM Siggraph Comput. Graph. 1997, 209–216 (1997)

    Google Scholar 

  16. Guo, F., Shen, J., Li, X.: Learning to detect stereo saliency. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2014)

  17. Hamann, B.: A data reduction scheme for triangulated surfaces. Comput. Aided Geom. Des. 11(2), 197–214 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  18. Hanocka, R., Hertz, A., Fish, N., Giryes, R., Fleishman, S., Cohen-Or, D.: Meshcnn: a network with an edge. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  19. Hoppe, H.: Mesh optimization. In: Conference on Computer Graphics & Interactive Techniques (1993)

  20. Hou, T., Qin, H.: Admissible diffusion wavelets and their applications in space-frequency processing. IEEE Trans. Vis. Comput. Graph. 19(1), 3–15 (2012)

    Google Scholar 

  21. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  22. Hu, S., Liang, X., Shum, H.P., Li, F.W., Aslam, N.: Sparse metric-based mesh saliency. Neurocomputing 400, 11–23 (2020)

    Article  Google Scholar 

  23. Hua, B.S., Tran, M.K., Yeung, S.K.: Pointwise convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 984–993 (2018)

  24. Huang, J., You, S.: Point cloud labeling using 3d convolutional neural network. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2670–2675. IEEE (2016)

  25. Jeong, S.W., Sim, J.Y.: Saliency detection for 3d surface geometry using semi-regular meshes. IEEE Trans. Multimed. 19(12), 2692–2705 (2017)

    Article  Google Scholar 

  26. Koch, C., Poggio, T.: Predicting the visual world: silence is golden. Nat. Neurosci. 2(1), 9–10 (1999)

    Article  Google Scholar 

  27. Komarichev, A., Zhong, Z., Hua, J.: A-CNN: annularly convolutional neural networks on point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7421–7430 (2019)

  28. Lahav, A., Tal, A.: Meshwalker: deep mesh understanding by random walks. arXiv preprint arXiv:2006.05353 (2020)

  29. Lan, S., Yu, R., Yu, G., Davis, L.S.: Modeling local geometric structure of 3d point clouds using geo-CNN. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  30. Lee, C.H., Varshney, A., Jacobs, D.W.: Mesh saliency. In: ACM SIGGRAPH 2005 Papers, pp. 659–666 (2005)

  31. Leifman, G., Shtrom, E., Tal, A.: Surface regions of interest for viewpoint selection. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (2012)

  32. Limper, M., Kuijper, A., Fellner, D.W.: Mesh saliency analysis via local curvature entropy. In: Eurographics (Short Papers), pp. 13–16 (2016)

  33. Liu, F., Wen, Y., Zhang, D., Jiang, X., Xing, X., Meng, D.: Log2vec: a heterogeneous graph embedding based approach for detecting cyber threats within enterprise. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1777–1794 (2019)

  34. Low, K.L., Tan, T.S.: Model simplification using vertex-clustering. In: Symposium on Interactive 3d Graphics (1997)

  35. Mnih, V., Heess, N., Graves, A., Kavukcuoglu, K.: Recurrent models of visual attention. arXiv preprint arXiv:1406.6247 (2014)

  36. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: ICML (2010)

  37. Nousias, S., Arvanitis, G., Lalos, A.S., Moustakas, K.: Mesh saliency detection using convolutional neural networks. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2020)

  38. Papon, J., Abramov, A., Schoeler, M., Worgotter, F.: Voxel cloud connectivity segmentation-supervoxels for point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2027–2034 (2013)

  39. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

  40. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)

  41. Schroeder, W.J., Zarge, J.A., Lorensen, W.E.: Decimation of triangle meshes. ACM Siggraph Comput. Graph. 26(2), 65–70 (1992)

    Article  Google Scholar 

  42. Sibson, R.: A brief description of natural neighbor interpolation. In: Barnett, V. (ed.) Interpreting Multivariate Data. Wiley, New York, pp. 21–36 (1981)

  43. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: Workshop at International Conference on Learning Representations. Citeseer (2014)

  44. Song, R., Liu, Y., Martin, R.R., Echavarria, K.R.: Local-to-global mesh saliency. Vis. Comput. 34(3), 323–336 (2018)

    Article  Google Scholar 

  45. Song, R., Liu, Y., Martin, R.R., Rosin, P.L.: Mesh saliency via spectral processing. ACM Trans. Graph. (TOG) 33(1), 1–17 (2014)

    Article  MATH  Google Scholar 

  46. Song, R., Liu, Y., Rosin, P.: Mesh saliency via weakly supervised classification-for-saliency CNN. IEEE Trans. Vis. Comput. Graph. 27(1), 151–164 (2019)

    Article  Google Scholar 

  47. Tao, P., Cao, J., Li, S., Liu, X., Liu, L.: Mesh saliency via ranking unsalient patches in a descriptor space. Comput. Graph. 46, 264–274 (2015)

    Article  Google Scholar 

  48. Tatarchenko, M., Park, J., Koltun, V., Zhou, Q.Y.: Tangent convolutions for dense prediction in 3d. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3887–3896 (2018)

  49. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: flexible and deformable convolution for point clouds. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6411–6420 (2019)

  50. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

  51. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (TOG) 38(5), 1–12 (2019)

    Article  Google Scholar 

  52. Wei, N., Gao, K., Ji, R., Chen, P.: Surface saliency detection based on curvature co-occurrence histograms. IEEE Access 6, 54536–54541 (2018)

    Article  Google Scholar 

  53. Wolfe, J.M.: Guided search 2.0 a revised model of visual search. Psychon. Bull. Rev. 1(2), 202–238 (1994)

    Article  Google Scholar 

  54. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  55. Wu, J., Shen, X., Zhu, W., Liu, L.: Mesh saliency with global rarity. Graph. Models 75(5), 255–264 (2013)

    Article  Google Scholar 

  56. Wu, W., Qi, Z., Fuxin, L.: Pointconv: deep convolutional networks on 3d point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9621–9630 (2019)

  57. Xi, W., Koch, S., Holmqvist, K., Alexa, M.: Tracking the gaze on objects in 3d: how do people really look at the bunny? In: SIGGRAPH Asia 2018 Technical Papers (2018)

  58. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057. PMLR (2015)

  59. Zhao, H., Jiang, L., Fu, C.W., Jia, J.: Pointweb: enhancing local neighborhood features for point cloud processing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5565–5573 (2019)

  60. Zheng, T., Chen, C., Yuan, J., Li, B., Ren, K.: Pointcloud saliency maps. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1598–1606 (2019)

Download references

Acknowledgements

We thank all the anonymous reviewers for their valuable comments. This work was supported by the National Key Research and Development Program of China (2020YFB1712401, 2018YFC0824402).

Author information

Authors and Affiliations

  1. School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou, 450002, China

    Cheng-ming Liu, Wan-na Luan, Rong-hua Fu, Hai-bo Pang & Ying-hao Li

Authors
  1. Cheng-ming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wan-na Luan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rong-hua Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hai-bo Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying-hao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hai-bo Pang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Cm., Luan, Wn., Fu, Rh. et al. Attention-embedding mesh saliency. Vis Comput 39, 1783–1795 (2023). https://doi.org/10.1007/s00371-022-02444-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02444-y

Keywords

Search

Navigation