Skip to main content
SpringerLink
Log in

Latent-MVCNN: 3D Shape Recognition Using Multiple Views from Pre-defined or Random Viewpoints

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

The Multi-view Convolution Neural Network (MVCNN) has achieved considerable success in 3D shape recognition. However, 3D shape recognition using view-images from random viewpoints has not been yet exploited in depth. In addition, 3D shape recognition using a small number of view-images remains difficult. To tackle these challenges, we developed a novel Multi-view Convolution Neural Network, “Latent-MVCNN” (LMVCNN), that recognizes 3D shapes using multiple view-images from pre-defined or random viewpoints. The LMVCNN consists of three types of sub Convolution Neural Networks. For each view-image, the first type of CNN outputs multiple category probability distributions and the second type of CNN outputs a latent vector to help the first type of CNN choose the decent distribution. The third type of CNN outputs the transition probabilities from the category probability distributions of one view to the category probability distributions of another view, which further helps the LMVCNN to find the decent category probability distributions for each pair of view-images. The three CNNs cooperate with each other to the obtain satisfactory classification scores. Our experimental results show that the LMVCNN achieves competitive performance in 3D shape recognition on ModelNet10 and ModelNet40 for both the pre-defined and the random viewpoints and exhibits promising performance when the number of view-images is quite small.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Bai S, Bai X, Zhou Z, Zhang Z, Latecki LJ (2016) Gift: a real-time and scalable 3d shape search engine. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 5023–5032

  2. Bronstein MM, Bruna J, Lecun Y, Szlam A, Vandergheynst P (2017) Geometric deep learning: going beyond euclidean data. IEEE Signal Process Mag 34(4):18–42

    Google Scholar 

  3. Bruna J, Zaremba W, Szlam A, Lecun Y (2014) Spectral networks and locally connected networks on graphs. In: international conference on learning representations

  4. Bu S, Wang L, Han P, Liu Z, Lib K (2017) 3d shape recognition and retrieval based on multi-modality deep learning. Neurocomputing 259:183–193

    Google Scholar 

  5. Charles RQ, Su H, Mo K, Guibas LJ (2016) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 77–85

  6. Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: delving deep into convolutional nets. In: British Machine Vision Conference

  7. Chen X, Chen Y, Gupta K, Zhou J, Najjaran H (2018) Slicenet: a proficient model for real-time 3d shape-based recognition. Neurocomputing 316:144–155

    Google Scholar 

  8. Cohen TS, Geiger M, Koehler J, Welling M (2018) Spherical cnns. In: Proceedings of international conference on learning representations

  9. Feng Y, Zhang Z, Zhao X, Ji R, Gao Y (2018) Gvcnn: Group-view convolutional neural networks for 3d shape recognition. In: Proceedings of IEEE international conference on computer vision, pp 264–272

  10. Ghodrati H, Luciano L, Hamza AB (2019) Convolutional shape-aware representation for 3d object classification. Neural Process Lett 49(2):797–817

    Google Scholar 

  11. Hamza AB (2016) A graph-theoretic approach to 3d shape classification. Neurocomputing 211:11–21

    Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 770–778

  13. He X, Yang Z, Zhou Z, Song B, Xiang B (2018) Triplet-center loss for multi-view 3d object retrieval. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  14. Sfikas K, Pratikakis TTI (2017) Exploiting the panorama representation for convolutional neural network classification and retrieval. In: 3DOR2017

  15. Kanezaki A, Matsushita Y, Nishida Y (2018) Rotationnet: joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In: Proceedings of IEEE international conference on computer vision

  16. Klokov R, Lempitsky V (2017) Escape from cells: deep kd-networks for the recognition of 3d point cloud models. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 863–872

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: International conference on neural information processing systems, pp 1097–1105

  18. Monti F, Boscaini D, Masci J, Rodolà E, Svoboda J, Bronstein MM (2017) Geometric deep learning on graphs and manifolds using mixture model cnns. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 5425–5434

  19. Nie W, Liu A, Hao Y, Su Y (2018) View-based 3d model retrieval via multi-graph matching. Neural Process Lett 48(3):1395–1404

    Google Scholar 

  20. Papadakis P, Pratikakis I, Theoharis T, Perantonis S (2010) Panorama: a 3d shape descriptor based on panoramic views for unsupervised 3d object retrieval. Int J Comput Vision 89(2–3):177–192

    Google Scholar 

  21. Qi CR, Su H, Nießner M, Dai A, Yan M, Guibas LJ (2016) Volumetric and multi-view cnns for object classification on 3d data. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 5648–5656

  22. Qi CR, Yi L, Su H, Guibas LJ (2017) Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Proceedings of neural information processing systems

  23. Shi B, Song B, Zhou Z, Xiang B (2015) Deeppano: deep panoramic representation for 3-d shape recognition. IEEE Signal Process Lett 22(12):2339–2343

    Google Scholar 

  24. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Proceedings of international conference on learning representations

  25. Sinha A, Bai J, Ramani K (2016) Deep learning 3d shape surfaces using geometry images. In: proceedings of European conference on computer vision, pp 223–240,

  26. Su H, Jampani V, Sun D, Maji S, Kalogerakis E, Yang MH, Kautz J (2018) Splatnet: sparse lattice networks for point cloud processing. In: Proceedings of IEEE international conference on computer vision

  27. Su H, Maji S, Kalogerakis E (2015) Learned-Miller, E.: Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of IEEE international conference on computer vision, pp 945–953

  28. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of IEEE conference on computer vision and pattern recognition

  29. Wang C, Cheng M, Sohelb F, Bennamounc M, Li J (2019) Normalnet: a voxel-based cnn for 3d object classification and retrieval. Neurocomputing 323:139–147

    Google Scholar 

  30. Wu J, Zhang C, Xue T, Freeman WT, Tenenbaum JB (2016) Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In: Proceedings of neural information processing systems

  31. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3d shapenets: a deep representation for volumetric shapes. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1912–1920

  32. Xie J, Zheng Z, Gao R, Wang W, Zhu SC, Wu YN (2018) Learning descriptor networks for 3d shape synthesis and analysis. In: Proceedings of IEEE conference on computer vision and pattern recognition

  33. Yan Z, Zeng F (2017) 2d compressive sensing and multi-feature fusion for effective 3d shape retrieval. Inf Sci 409–410:101–120

    Google Scholar 

  34. Yi L, Su H, Guo X, Guibas L (2017) Syncspeccnn: synchronized spectral cnn for 3d shape segmentation. In: Proceedings of IEEE international conference on computer vision, pp 6584–6592

  35. Yu T, Meng J, Yuan J (2018) Multi-view harmonized bilinear network for 3d object recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 186–194

  36. Zhou Y, Zeng F, Qian J, Han X (2019) 3d shape classification and retrieval based on polar view. Inf Sci 474:205–220

    MathSciNet  Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their helpful suggestions. This work was supported by Natural Science Fund for Colleges and Universities in Jiangsu Province (Grant No. 18KJB520013), the Dual Creative Doctors of Jiangsu Province, National Nature Science Foundation of China (Grant Nos. 61902159 and 61771146), Zhejiang Provincial Natural Science Foundation of China (LQ19F020003) and Qing Lan Project of Jiangsu Province.

Author information

Authors and Affiliations

  1. School of Computer Engineering, Jiangsu University of Technology, No. 1801 Zhongwu Road, Changzhou, 213001, China

    Qian Yu & Honghui Fan

  2. School of Information, Zhejiang University of Finance and Economics, No. 18 Xueyuan Road, Hangzhou, 310018, China

    Chengzhuan Yang

  3. School of Computer Science, Fudan University, No. 825 Zhangheng Road, Shanghai, 201203, China

    Hui Wei

Authors
  1. Qian Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengzhuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Honghui Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qian Yu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Q., Yang, C., Fan, H. et al. Latent-MVCNN: 3D Shape Recognition Using Multiple Views from Pre-defined or Random Viewpoints. Neural Process Lett 52, 581–602 (2020). https://doi.org/10.1007/s11063-020-10268-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10268-x

Keywords

Search

Navigation