Abstract
Searching for relevant 3D models based on hand-drawn sketches is both intuitive and important for many applications, such as sketch-based 3D modeling and recognition, human computer interaction, 3D animation, game design, and etc. In this paper, our target is to significantly improve the current sketch-based 3D retrieval performance in terms of both accuracy and efficiency. We propose a new sketch-based 3D model retrieval framework by utilizing adaptive view clustering and semantic information. It first utilizes a proposed viewpoint entropy-based 3D information complexity measurement to guide adaptive view clustering of a 3D model to shortlist a set of representative sample views for 2D-3D comparison. To bridge the gap between the query sketches and the target models, we then incorporate a novel semantic sketch-based search approach to further improve the retrieval performance. Experimental results on several latest benchmarks have evidently demonstrated our significant improvement in retrieval performance.
Similar content being viewed by others
References
AIM@SHAPE: SHREC Contest. Home Page (2010). http://www.aimatshape.net/event/SHREC/
Ansary TF, Daoudi M, Vandeborre JP (2007) A Bayesian 3-D search engine using adaptive views clustering. IEEE Trans Multimedia 9(1):78–88
Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522
Bezdek JC (1981) Pattern Recognition with Fuzzy Objective Function Algorithms. Kluwer Academic Publishers, Norwell, MA, USA
Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?. ACM Trans Graph 31(4):44:1–44:10
Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636
Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph 31(4):31:1–31:10
Fang Y, Xie J, Dai G, Wang M, Zhu F, Xu T, Wong EK (2015) 3D deep shape descriptor. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 2319–2328. IEEE Computer Society. doi:10.1109/CVPR.2015.7298845
Gong B, Liu J, Wang X, Tang X (2013) Learning semantic signatures for 3D object retrieval. IEEE Trans Multimedia 15(2):369–377
Hou S, Lou K, Ramani K (2005) SVM-based Semantic Clustering and Retrieval of A 3D Model Database. Comput-Aided Des Applic 2:155–164
Jayanti S, Kalyanaraman Y, Iyer N, Ramani K (2006) Developing an engineering shape benchmark for CAD models. Comput Aided Des 38(9):939–953
Jonker R, Volgenant A (1987) A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing 38(4):325–340
Li B, Johan H (2012) Sketch-based 3D model retrieval by incorporating 2D-3D alignment. Multimedia Tools Appl:1–23
Li B (2013) Johan, H.: 3D model retrieval using hybrid features and class information. Multimedia Tools Appl 62(3):821–846
Li B, Lu Y, Fares R (2013) Semantic sketch-based 3D model retrieval. In: ICME, pp 1–4
Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra JM, Tashiro S (2013) SHREC’13 track: Large scale sketch-based 3D shape retrieval. In: 3DOR, pp 89–96
Li B, Lu Y, Godil A, Schreck T, Bustos B, Ferreira A, Furuya T, Fonseca MJ, Johan H, Matsuda T, Ohbuchi R, Pascoal PB, Saavedra JM (2014) A comparison of methods for sketch-based 3D shape retrieval. Comput Vis Image Underst 119:57–80
Li B, Lu Y, Johan H (2013) Sketch-based 3D model retrieval by viewpoint entropy-based adaptive view clustering. In: Eurographics Workshop on 3D Object Retrieval, Girona, Spain, 2013. Proceedings, pp 49–56
Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Chen Q, Chowdhury NK, Fang B, Fu H, Furuya T, Li H, Liu J, Johan H, Kosaka R, Koyanagi H, Ohbuchi R, Tatsuma A, Wan Y, Zhang C, Zou C (2015) A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Comput Vis Image Underst 131:1–27. doi:10.1016/j.cviu.2014.10.006
Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Fu H, Furuya T, Johan H, Liu J, Ohbuchi R, Tatsuma A, Zou C (2014) SHREC’14: Extended Large Scale Sketch-Based 3D Shape Retrieval. In: Eurographics Workshop on 3D Object Retrieval, Strasbourg, France, 2014. Proceedings, pp 121–130
Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra JM, Scherer M, Yanagimachi T, Yoon G, Yoon SM (2012) SHREC’12 track: Sketch-based 3D shape retrieval. In: Eurographics Workshop on 3D Object Retrieval 2012, Cagliari, Italy, May 13, 2012. Proceedings, pp 109– 118
Mokhtarian F, Abbasi S (2005) Robust automatic selection of optimal views in multi-view free-form object recognition. Pattern Recogn 38(7):1021–1031
Nie L, Yan S, Wang M, Hong R, Chua T (2012) Harvesting visual concepts for image search with complex queries. In: Babaguchi N, Aizawa K, Smith J.R., Satoh S., Plagemann T., Hua X., Yan R. (eds) Proceedings of the 20th ACM Multimedia Conference, MM ’12, Nara, Japan, October 29 - November 02, 2012, pp 59–68. ACM. doi:10.1145/2393347.2393363
Ohbuchi R, Yamamoto A, Kobayashi J (2007) Learning semantic categories for 3D model retrieval. In: Multimedia Information Retrieval, pp 31–40
Page DL, Koschan A, Sukumar SR, Roui-Abidi B, Abidi MA (2003) Shape analysis algorithm based on information theory. In: ICIP (1), pp 229–232
Powers DMW (2007) Evaluation: From Precision, Recall and F-measure to ROC, Informedness, Markedness & Correlation. J Mach Learn Technol 2(1):37–63
Rossignac J (2005) Shape complexity. Vis Comput 21(12):985–996
Saleem W, Belyaev AG, Wang D, Seidel HP (2011) On visual complexity of 3D shapes. Comput Graph 35(3):580–585
(2016) ShapeNet: http://shapenet.cs.stanford.edu/
Shilane P, Min P, Kazhdan M, Funkhouser T (2004) The Princeton shape benchmark. In: SMA, pp 167–178
Takahashi S, Fujishiro I, Takeshima Y, Nishita T (2005) A feature-driven approach to locating optimal viewpoints for volume visualization. In: IEEE Visualization, pp 495–502
Vázquez P.P., Feixas M, Sbert M, Heidrich W (2003) Automatic view selection using viewpoint entropy and its applications to image-based modelling. Comput Graph Forum 22(4):689–700
Vranic D (2004) 3D Model Retrieval. University of Leipzig, PhD thesis
Wang F, Kang L, Li Y (2015) Sketch-based 3D shape retrieval using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 1875–1883
Wessel R, Blümel I., Klein R (2009) A 3D shape benchmark for retrieval and automatic classification of architectural data. In: Eurographics Workshop on 3D Object Retrieval, 3DOR 2009, pp 53–56
Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In: ACM Multimedia, pp 193–200
Zhu F, Xie J, Fang Y (2016) Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pp 3683–3689
Zhu J, Zhu F, Wong EK, Fang Y (2015) Learning pairwise neural network encoder for depth image-based 3d model retrieval. In: Zhou X, Smeaton AF, Tian Q, Bulterman DCA, Shen HT, Mayer-Patel K, Yan S (eds) Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, MM ’15, Brisbane, Australia, October 26 - 30, 2015, pp 1227–1230. ACM. doi:10.1145/2733373.2806323
Acknowledgments
The work of Bo Li and Yijuan Lu is supported by the Texas State University Research Enhancement Program (REP), Army Research Office grant W911NF-12-1-0057, and NSF CNS 1305302 to Dr. Yijuan Lu.
The research done by Henry Johan in Fraunhofer IDM@NTU is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centres in Singapore Funding Initiative.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Li, B., Lu, Y., Johan, H. et al. Sketch-based 3D model retrieval utilizing adaptive view clustering and semantic information. Multimed Tools Appl 76, 26603–26631 (2017). https://doi.org/10.1007/s11042-016-4187-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-4187-3