Image-based 3D model retrieval using manifold learning

  • Pan-pan MuEmail author
  • San-yuan Zhang
  • Yin Zhang
  • Xiu-zi Ye
  • Xiang Pan


We propose a new framework for image-based three-dimensional (3D) model retrieval. We first model the query image as a Euclidean point. Then we model all projected views of a 3D model as a symmetric positive definite (SPD) matrix, which is a point on a Riemannian manifold. Thus, the image-based 3D model retrieval is reduced to a problem of Euclid-to-Riemann metric learning. To solve this heterogeneous matching problem, we map the Euclidean space and SPD Riemannian manifold to the same high-dimensional Hilbert space, thus shrinking the great gap between them. Finally, we design an optimization algorithm to learn a metric in this Hilbert space using a kernel trick. Any new image descriptors, such as the features from deep learning, can be easily embedded in our framework. Experimental results show the advantages of our approach over the state-of-the-art methods for image-based 3D model retrieval.

Key words

Model retrieval Euclidean space Riemannian manifold Hilbert space Metric learning 

CLC number



Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bai S, Bai X, Zhou Z, et al., 2016. GIFT: a real-time and scalable 3D shape search engine. 16th IEEE Conf on Computer Vision and Pattern Recognition, p.5023–5032. Google Scholar
  2. Bai X, Bai S, Zhu Z, et al., 2015. 3D shape matching via two layer coding. IEEE Trans Patt Anal Mach Intell, 37(12): 2361–2373. MathSciNetCrossRefGoogle Scholar
  3. Cevikalp H, Triggs B, 2010. Face recognition based on image sets. IEEE Society Conf on Computer Vision and Pattern Recognition, p.2567–2573. Google Scholar
  4. Chatfield K, Simonyan K, Vedaldi A, et al., 2014. Return of the devil in the details: delving deep into convolutional nets. p.1–11. Google Scholar
  5. Chen DY, Tian XP, Shen YT, et al., 2003. On visual similarity based 3D model retrieval. Comput Graph Forum, 22(3): 223–232. CrossRefGoogle Scholar
  6. Chien JT, Wu CC, 2002. Discriminant waveletfaces and nearest feature classifiers for face recognition. IEEE Trans Patt Anal Mach Intell, 24(12):1644–1649. CrossRefGoogle Scholar
  7. Eitz M, Richter R, Boubekeur T, et al., 2012. Sketch-based shape retrieval. ACM Trans Graph, 31(4):31–40. Google Scholar
  8. Furuya T, Ohbuchi R, 2013. Ranking on cross-domain manifold for sketch-based 3D model retrieval. Int Conf on Cyberworlds, p.274–281. Google Scholar
  9. Hamm J, Lee DD, 2008. Grassmann discriminant analysis: a unifying view on subspace-based learning. Proc 25th Int Conf on Machine Learning, p.376–383. Google Scholar
  10. Hamm J, Lee DD, 2009. Extended Grassmann kernels for subspace-based learning. Advances in Neural Information Processing Systems, p.601–608.Google Scholar
  11. Huang Z, Wang R, Shan S, et al., 2014. Learning Euclideanto-Riemannian metric for point-to-set classification. IEEE Conf on Computer Vision and Pattern Recognition, p.1677–1684. Google Scholar
  12. Jayasumana S, Hartley R, Salzmann M, et al., 2013. Kernel methods on the Riemannian manifold of symmetric positive definite matrices. IEEE Conf on Computer Vision and Pattern Recognition, p.73–80. Google Scholar
  13. Kazhdan M, Funkhouser T, Rusinkiewicz S, 2003. Rotation invariant spherical harmonic representation of 3D shape descriptors. Proc Eurographics/ACM SIGGRAPH Symp on Geometry Processing, p.156–164.Google Scholar
  14. Kim T, Kittler J, Cipolla R, 2007. Discriminative learning and recognition of image set classes using canonical correlations. IEEE Trans Patt Anal Mach Intell, 29(6): 1005–1018. CrossRefGoogle Scholar
  15. Li B, Lu Y, Godil A, et al., 2014. A comparison of methods for sketch-based 3D shape retrieval. Comput Vis Image Underst, 119:57–80. CrossRefGoogle Scholar
  16. Lian Z, Godil A, Sun X, et al., 2013. CM-BOF: visual similarity-based 3D shape retrieval using clock matching and bag-of-features. Mach Vis Appl, 24(8):1685–1704. CrossRefGoogle Scholar
  17. Mu P, Zhang S, Ye X, 2017. A metric learning method for image-based 3D shape retrieval. Proc Int Conf on Data Mining, Communications and Information Technology, Article 17. Google Scholar
  18. Ohbuchi R, Osada K, Furuya T, et al., 2008. Salient local visual features for shape-based 3D model retrieval. IEEE Int Conf on Shape Modeling and Applications, p.93–102. Google Scholar
  19. Papadakis P, Pratikakis I, Theoharis T, et al., 2010. Panorama: a 3D shape descriptor based on panoramic views for unsupervised 3D object retrieval. Int J Comput Vis, 89(2-3):177–192. CrossRefGoogle Scholar
  20. Saavedra JM, Bustos B, Schreck T, et al., 2012. Sketch-based 3D model retrieval using keyshapes for global and local representation. Proc 5th Eurographics Conf on 3D Object Retrieval, p.47–50. Google Scholar
  21. Shilane P, Min P, Kazhdan M, et al., 2004. The Princeton Shape Benchmark. Proc Shape Modeling Applications, p.167–178. Google Scholar
  22. Sousa P, Fonseca MJ, 2010. Sketch-based retrieval of drawings using spatial proximity. J Vis Lang Comput, 21(2):69–80. CrossRefGoogle Scholar
  23. Su H, Maji S, Kalogerakis E, et al., 2015. Multi-view convolutional neural networks for 3D shape recognition. IEEE Int Conf on Computer Vision, p.945–953. Google Scholar
  24. Tabia H, Laga H, Picard D, et al., 2014. Covariance descriptors for 3D shape matching and retrieval. IEEE Conf on Computer Vision and Pattern Recognition, p.4185–4192. Google Scholar
  25. Vemulapalli R, Pillai JK, Chellappa R, 2013. Kernel learning for extrinsic classification of manifold features. IEEE Conf on Computer Vision and Pattern Recognition, p.1782–1789. Google Scholar
  26. Vincent P, Bengio Y, 2001. K-local hyperplane and convex distance nearest neighbor algorithms. Proc 14th Int Conf on Neural Information Processing Systems: Natural and Synthetic, p.985–992.Google Scholar
  27. Wang F, Kang L, Li Y, 2015. Sketch-based 3D shape retrieval using convolutional neural networks. IEEE Conf on Computer Vision and Pattern Recognition, p.1875–1883. Google Scholar
  28. Wang R, Guo H, Davis LS, et al., 2012. Covariance discriminative learning: a natural and efficient approach to image set classification. IEEE Conf on Computer Vision and Pattern Recognition, p.2496–2503. Google Scholar
  29. Wen Y, Zhang K, Li Z, et al., 2016. A discriminative feature learning approach for deep face recognition. European Conf on Computer Vision, p.499–515. Google Scholar
  30. Wu Z, Song S, Khosla A, et al., 2015. 3D shapenets: a deep representation for volumetric shapes. IEEE Conf on Computer Vision and Pattern Recognition, p.1912–1920. Google Scholar
  31. Yamaguchi O, Fukui K, Maeda K, 1998. Face recognition using temporal image sequence. Proc 3rd IEEE Int Conf on Automatic Face and Gesture Recognition, p.318–323. CrossRefGoogle Scholar
  32. Zhu P, Zhang L, Zuo W, et al., 2013. From point to set: extend the learning of distance metrics. IEEE Int Conf on Computer Vision, p.2664–2671. Google Scholar

Copyright information

© Editorial Office of Journal of Zhejiang University Science and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyZhejiang UniversityHangzhouChina
  2. 2.College of Mathematics and Information ScienceWenzhou UniversityWenzhouChina
  3. 3.College of Computer Science and TechnologyZhejiang University of TechnologyHangzhouChina

Personalised recommendations