Deep Cross-Modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-Based 3D Shape Retrieval

  • Jiaxin Chen
  • Yi FangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11217)


Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches.


Sketch-based 3D shape retrieval Cross-modality transformation Adversarial learning Importance-aware metric learning 


  1. 1.
    Chen, J., Wang, Y., Qin, J., Liu, L., Shao, L.: Fast person re-identification via cross-camera semantic binary transformation. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  2. 2.
    Dai, G., Xie, J., Zhu, F., Fang, Y.: Deep correlated metric learning for sketch-based 3D shape retrieval. In: AAAI, pp. 4002–4008 (2017)Google Scholar
  3. 3.
    Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., Alexa, M.: Sketch-based shape retrieval. ACM Trans. Graph. 31(4), 1–10 (2012)Google Scholar
  4. 4.
    Furuya, T., Ohbuchi, R.: Ranking on cross-domain manifold for sketch-based 3D model retrieval. In: 2013 International Conference on Cyberworlds (CW), pp. 274–281. IEEE (2013)Google Scholar
  5. 5.
    Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2096–2030 (2016)Google Scholar
  6. 6.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  8. 8.
    Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017)
  9. 9.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)Google Scholar
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  11. 11.
    Li, B., et al.: Shrec’12 track: generic 3D shape retrieval. In: 3DOR, vol. 6 (2012)Google Scholar
  12. 12.
    Li, B., et al.: SHREC’13 track: large scale sketch-based 3D shape retrieval (2013)Google Scholar
  13. 13.
    Li, B., et al.: A comparison of methods for sketch-based 3D shape retrieval. Comput. Vis. Image Underst. 119, 57–80 (2014)CrossRefGoogle Scholar
  14. 14.
    Li, B., Lu, Y., Johan, H., Fares, R.: Sketch-based 3D model retrieval utilizing adaptive view clustering and semantic information. Multimed. Tools Appl. 76(24), 26603–26631 (2017)CrossRefGoogle Scholar
  15. 15.
    Li, B., et al.: A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Comput. Vis. Image Underst. 131, 1–27 (2015)CrossRefGoogle Scholar
  16. 16.
    Li, B., et al.: SHREC’14 track: extended large scale sketch-based 3D shape retrieval. In: Eurographics Workshop on 3D Object Retrieval, vol. 2014 (2014)Google Scholar
  17. 17.
    Liu, Z., Qin, J., Li, A., Wang, Y., Van Gool, L.: Adversarial binary coding for efficient person re-identification. arXiv preprint arXiv:1803.10914 (2018)
  18. 18.
    Motiian, S., Jones, Q., Iranmanesh, S., Doretto, G.: Few-shot adversarial domain adaptation. In: Advances in Neural Information Processing Systems, pp. 6673–6683 (2017)Google Scholar
  19. 19.
    Phong, B.T.: Illumination for computer generated pictures. Commun. ACM 18(6), 311–317 (1975)CrossRefGoogle Scholar
  20. 20.
    Qin, J., Liu, L., Yu, M., Wang, Y., Shao, L.: Fast action retrieval from videos via feature disaggregation. Comput. Vis. Image Underst. 156, 104–116 (2017)CrossRefGoogle Scholar
  21. 21.
    Shilane, P., Min, P., Kazhdan, M., Funkhouser, T.: The princeton shape benchmark. In: Proceedings of Shape Modeling Applications 2004, pp. 167–178. IEEE (2004)Google Scholar
  22. 22.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  23. 23.
    Sousa, P., Fonseca, M.J.: Sketch-based retrieval of drawings using spatial proximity. J. Vis. Lang. Comput. 21(2), 69–80 (2010)CrossRefGoogle Scholar
  24. 24.
    Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 945–953 (2015)Google Scholar
  25. 25.
    Tabia, H., Laga, H.: Learning shape retrieval from different modalities. Neurocomputing 253, 24–33 (2017)CrossRefGoogle Scholar
  26. 26.
    Tatsuma, A., Koyanagi, H., Aono, M.: A large-scale shape benchmark for 3D object retrieval: Toyohashi shape benchmark. In: Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, pp. 1–10. IEEE (2012)Google Scholar
  27. 27.
    Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Computer Vision and Pattern Recognition (CVPR), vol. 1, p. 4 (2017)Google Scholar
  28. 28.
    Wang, F., Kang, L., Li, Y.: Sketch-based 3D shape retrieval using convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1875–1883. IEEE (2015)Google Scholar
  29. 29.
    Wang, P.S., Liu, Y., Guo, Y.X., Sun, C.Y., Tong, X.: O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. (TOG) 36(4), 72 (2017)Google Scholar
  30. 30.
    Xie, J., Dai, G., Zhu, F., Fang, Y.: Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3615–3623. IEEE (2017)Google Scholar
  31. 31.
    Xie, J., Dai, G., Zhu, F., Wong, E.K., Fang, Y.: Deepshape: deep-learned shape descriptor for 3D shape retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1335–1345 (2017)CrossRefGoogle Scholar
  32. 32.
    Yasseen, Z., Verroust-Blondet, A., Nasri, A.: View selection for sketch-based 3D model retrieval using visual part shape description. Vis. Comput. 33(5), 565–583 (2017)CrossRefGoogle Scholar
  33. 33.
    Yoon, G.J., Yoon, S.M.: Sketch-based 3D object recognition from locally optimized sparse features. Neurocomputing 267, 556–563 (2017)CrossRefGoogle Scholar
  34. 34.
    Zhang, Y., Barzilay, R., Jaakkola, T.: Aspect-augmented adversarial networks for domain adaptation. arXiv preprint arXiv:1701.00188 (2017)
  35. 35.
    Zhu, F., Xie, J., Fang, Y.: Heat diffusion long-short term memory learning for 3D shape analysis. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 305–321. Springer, Cham (2016). Scholar
  36. 36.
    Zhu, F., Xie, J., Fang, Y.: Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: AAAI, pp. 3683–3689 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.NYU Multimedia and Visual Computing Lab, Department of Electrical and Computer EngineeringNew York University Abu DhabiAbu DhabiUAE
  2. 2.Department of Electrical and Computer EngineeringNYU Tandon School of EngineeringNew YorkUSA

Personalised recommendations