Skip to main content
SpringerLink
Log in

Similarity metric learning for sketch-based 3D object retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketch-Based 3D Object Retrieval (SB3DOR) algorithms retrieve 3D models similar to hand-drawn sketch queries. It is one of the most effective modalities to query 3D models by their shape. However, comparison of a sketch, which is a 2D image, with a 3D model is not straightforward. Most of the SB3DOR algorithms compare a sketch image with images of 3D models rendered from multiple viewpoints. However, retrieval accuracies of state-of-the-art SB3DOR algorithms are still not satisfactory, due, in part, to stylistic variation, semantic influence, abstraction, and drawing error found in sketches. To improve retrieval accuracy of SB3DOR systems, we propose a manifold-based similarity metric learning algorithm that relates two kinds of features, that are, of sketches and 3D models. Features in a high dimensional space often lie on a lower dimensional subspace, or manifold. Feature similarity may be computed more accurately on the manifold than in the original high dimensional space. Our Cross-Domain Manifold Ranking (CDMR) algorithm tries to keep the two distinct feature manifolds, one for sketch features and the other for 3D model features, intact. These two manifolds are interlinked by using inter-feature similarity to form a Cross-Domain Manifold (CDM). If available, semantic labels may also be used in forming the CDM. Relevance diffusion is used to compute similarities between a sketch and 3D models in a database. Experimental evaluation showed that the CDMR algorithm produces higher retrieval accuracy than the algorithms we compared against.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Akaho S (2001) A kernel method for canonical correlation analysis, proc. the international meeting of the psychometric society.

  2. Choo J, Bohn S, Nakamura G. C, White A. M, and Park H (2012) Heterogeneous data fusion via space alignment using nonmetric multidimensional scaling, Proc. SIAM International Conference on Data Mining, p 177–188

  3. Csurka G, Dance C.R, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints, proc. European conference on computer vision ’04 workshop on statistical learning in computer vision, p 59–74

  4. CULA, http://www.culatools.com/

  5. DeCarlo D, Finkelstein A, Rusinkiewicz S, Santella A (2003) Suggestive contours for conveying shape. ACM TOG 22(3):848–855

    Article  Google Scholar 

  6. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?, ACM TOG, 31(4), p 44:1–44:10

  7. Eitz M., Richter R, Boubekeur T, Hildebrand K, and Alexa M (2012) Sketch-Based Shape Retrieval, ACM TOG, 31(4) pp 31:1–31:10

  8. ElNaghy H, Hamad S, Khalifa ME (2013) Taxonomy for 3D content-based object retrieval methods. Int J Res Rev Appl Sci (IJRRAS) 14(2):412–446

    Google Scholar 

  9. Furuya T, Ohbuchi R (2013) Ranking on cross-domain manifold for sketch-based 3D model retrieval, regular paper, proc. CyberWorlds 2013:274–281

    Google Scholar 

  10. Furuya T, Ohbuchi R (2014) Visual saliency weighting and cross-domain manifold ranking for sketch-based image retrieval, proc. Multimedia modeling 2014 (MMM 2014), p 37–49

  11. Furuya T, Ohbuchi R (2009) Dense sampling and fast encoding for 3D model retrieval using bag-of-visual features, proc. ACM CIVR 2009, Article No. 26

  12. Geurts P, Ernst D, Wehenkel L (2006) Extremely randomized trees. Mach Learn J 63(1):3–42

    Article  MATH  Google Scholar 

  13. Goldfeder C, Allen P (2008) Autotagging to improve text search for 3D models, proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries 2008 (JCDL’08), p 355–358

  14. Han J, Ma K-K (2007) Rotation-invariant and scale-invariant gabor features for texture image retrieval. Image Vis Comput 25(9):1474–1481

    Article  MathSciNet  Google Scholar 

  15. Hotelling H (1936) Relations between two sets of variates. Biometrika 28:321–377

    Article  MATH  Google Scholar 

  16. Kanai S (2008) Content-based 3D mesh model retrieval from hand-written sketch. Int J Interact Des Manuf (IJIDeM) 2(2):87–98

    Article  Google Scholar 

  17. Li B, Godil A, Aono M, Bai X, Furuya T, Li L, Lopez-Sastre R, Johan H, Ohbuchi R, Redondo-Cabrera C, Tatsuma A, Yanagimachi T, Zhang S (2012) SHREC’12 track: generic 3D shape retrieval, proc. EG 3DOR 2012, p 119–126

  18. Li B, Johan H (2013) Sketch-based 3D model retrieval by incorporating 2D-3D alignment. Multimedia Tools Appl 65(3):363–385

    Article  Google Scholar 

  19. Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra JM, Tashiro S (2013) SHREC’13 track: large scale sketch-based 3D shape retrieval, EG 3DOR 2013, p 1–9

  20. Li B, Lu Y, Godil A, Schreck T, Bustos B, Ferreira A, Furuya T, Fonseca MJ, Johan H, Matsuda T, Ohbuchi R, Pascoal PB, Saavedra JM (2014) A comparison of methods for sketch-based 3D shape retrieval. Comp Vision Image Underst (CVIU) 119:57–80

    Article  Google Scholar 

  21. Li B, Lu Y, Fares R (2013) Semantic sketch-based 3D model retrieval, Multimedia and Expo Workshops (ICMEW), proc. IEEE International Conference on Multimedia and Expo 2013, p 1–3

  22. Li B, Lu Y, Johan H (2013) Sketch-based 3D model retrieval by viewpoint entropy-based adaptive view clustering, proc. eurographics workshop on 3D object retrieval 2013 (3DOR 2013), p 49–56

  23. Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra JM, Scherer M, Yanagimachi T, Yoon G, Yoon SM (2012) SHREC’12 track: sketch-based 3D shape retrieval, prof. EG 3DOR 2012, p 109–118

  24. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  25. Ng C.B.R, Lu G, Zhang D (2005) Performance study of gabor filters and rotation invariant gabor filters, proc. Multimedia modelling 2005 (MMM 2005), p 158–162

  26. Ohbuchi R, Kawamura S (2009) Shape-based autotagging of 3D models for retrieval, proc. 4th international conference on semantic and digital media technologies (SAMT 2009), p 137–148

  27. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  28. Rasiwasia N, Pereira JC, Coviello E, Doyle G, Lanckriet GRG, Levy R, Vasconcelos N (2010) A new approach to cross-modal multimedia retrieval, proc. ACM Multimed 2010:251–260

    Google Scholar 

  29. Saavedra JM, Bustos B, Scherer M, Schreck T (2011) STELA: sketch-based 3D model retrieval using a structure-based local approach, proc. ACM Int’l Conf. on Multimedia Retrieval (ICMR’11), 26, p 1–8

  30. Saavedra JM, Bustos B, Schreck T, Yoon SM, Scherer M (2012) Sketch-based 3D model retrieval using keyshapes for global and local representation, proc. 5th Eurographics Conference on 3D Object Retrieval (EG 3DOR) 2012, p 47–50

  31. Shao T, Xu W, Yin K, Wang J, Zhou K, Guo B (2011) Discriminative sketch-based 3D model retrieval via robust shape matching. Comput Graph Forum 30(7):2011–2020

    Article  Google Scholar 

  32. Shilane P, Min P, Kazhdan M, Funkhouser T (2004) The princeton shape benchmark, proc. SMI ‘04, p 167–178 http://shape.cs.princeton.edu/benchmark/

  33. SHREC (2013) Large scale sketch-based 3D shape retrieval, http://www.itl.nist.gov/iad/vug/sharp/contest/2013/SBR/

  34. SHREC home page, http://www.aimatshape.net/event/SHREC/

  35. Sun L, Ji S, Ye J LSCCA Package: Least Squares Canonical Correlation Analysis, http://www.public.asu.edu/~jye02/Software/CCA/

  36. Tangelder JWH, Veltkamp RC (2008) A survey of content based 3D shape retrieval methods. Multimed Tools Appl 39(3):441–471

    Article  Google Scholar 

  37. Trimble 3D warehouse, http://sketchup.google.com/3dwarehouse/

  38. Wang X, Ma W, Xue G, Li X (2004) Multi-model similarity propagation and its application for web image retrieval, proc. ACM Multimed 2004:944–951

    Google Scholar 

  39. Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. Proc. ACM Multimed 2010:193–200

    Google Scholar 

  40. Zhai X, Peng Y, Xiao J (2012) Cross-modality correlation propagation for cross-media retrieval. Proc. IEEE Int’l conf. on Acoustics, Speech and Signal Processing (ICASSP), pp 2337–2340

    Google Scholar 

  41. Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B (2003) Learning with local and global consistency, proc. advances in Neural Information Processing Systems (NIPS) 2003, p 321–328

Download references

Author information

Authors and Affiliations

  1. Graduate School of Medicine and Engineering, University of Yamanashi, 4-3-11 Takeda, Kofu-shi, Yamanashi-ken, Japan

    Takahiko Furuya & Ryutarou Ohbuchi

Authors
  1. Takahiko Furuya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryutarou Ohbuchi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Takahiko Furuya.

Additional information

The description of the CDMR algorithm in similar form has previously appeared in [9]. However, all the experiments to evaluate retrieval accuracy of the CDMR algorithm is redone by using a new image rotaion alignment algorithm.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Furuya, T., Ohbuchi, R. Similarity metric learning for sketch-based 3D object retrieval. Multimed Tools Appl 74, 10367–10392 (2015). https://doi.org/10.1007/s11042-014-2171-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2171-3

Keywords

Search

Navigation