The Visual Computer

, Volume 35, Issue 6–8, pp 909–920 | Cite as

Image-driven unsupervised 3D model co-segmentation

  • Juncheng Liu
  • Paul L. Rosin
  • Xianfang Sun
  • Jianguo Xiao
  • Zhouhui LianEmail author
Original Article


Segmentation of 3D models is a fundamental task in computer graphics and vision. Geometric methods usually lead to non-semantic and fragmentary segmentations. Learning techniques require a large amount of labeled training data. In this paper, we explore the feasibility of 3D model segmentation by taking advantage of the huge number of easy-to-obtain 2D realistic images available on the Internet. The regional color exhibited in images provides information that is valuable for segmentation. To transfer the segmentations, we first filter out inappropriate images with several criteria. The views of these images are estimated by our proposed texture-invariant view estimation Siamese network. The training samples are generated by rendering-based synthesis without laborious labeling. Subsequently, we transfer and merge the segmentations produced by each individual image by applying registration and a graph-based aggregation strategy. The final result is obtained by combining all segmentations within the 3D model set. Our qualitative and quantitative experimental results on several model categories validate effectiveness of our proposed method.


3D segmentation Image-driven View estimation 



This work was supported by National Natural Science Foundation of China (Grant Nos.: 61672043 and 61672056), National Key Research and Development Program of China (2017YFB1002601) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). The support provided by China Scholarship Council (CSC) during a visit of Juncheng Liu to Cardiff University is acknowledged.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Lai, Y.-K., Hu, S.-M., Martin, R.R., Rosin, P.L.: Fast mesh segmentation using random walks. In: Proceedings of the 2008 ACM symposium on Solid and Physical Modeling, pp. 183–191. ACM, New York (2008)Google Scholar
  2. 2.
    Sidi, O., van Kaick, O., Kleiman, Y., Zhang, H., Cohen-Or, D.: Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering, vol. 30. ACM, New York (2011)Google Scholar
  3. 3.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  4. 4.
    Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3828–3836 (2015)Google Scholar
  5. 5.
    Kalogerakis, E., Hertzmann, A., Singh, K.: Learning 3d mesh segmentation and labeling. In: ACM SIGGRAPH 2010 Papers, SIGGRAPH ’10, pp. 102:1–102:12. ACM, New York, NY (2010)Google Scholar
  6. 6.
    Guo, K., Zou, D., Chen, X.: 3d mesh labeling via deep convolutional neural networks. ACM Trans. Graph. (TOG) 35(1), 3 (2015)Google Scholar
  7. 7.
    Shu, Z., Qi, C., Xin, S., Hu, C., Wang, L., Zhang, Y., Liu, L.: Unsupervised 3d shape segmentation and co-segmentation via deep learning. Comput. Aided Geom. Des. 43, 39–52 (2016)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Kalogerakis, E., Averkiou, M., Maji, S., Chaudhuri, S.: 3d shape segmentation with projective convolutional networks. Proc. CVPR 1, 8 (2017)Google Scholar
  9. 9.
    Wang, Y., Asafi, S., Van Kaick, O., Zhang, H., Cohen-Or, D., Chen, B.: Active co-analysis of a set of shapes. ACM Trans. Graph. (TOG) 31(6), 165 (2012)Google Scholar
  10. 10.
    Wang, Y., Gong, M., Wang, T., Cohen-Or, D., Zhang, H., Chen, B.: Projective analysis for 3d shape segmentation. ACM Trans. Graph. (TOG) 32(6), 192 (2013)MathSciNetGoogle Scholar
  11. 11.
    Huang, Q., Wang, H., Koltun, V.: Single-view reconstruction via joint analysis of image and shape collections. ACM Trans. Graph. (TOG) 34(4), 87 (2015)Google Scholar
  12. 12.
    Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 978–994 (2011)Google Scholar
  13. 13.
    Huang, Q., Koltun, V., Guibas, L.: Joint shape segmentation with linear programming. In: ACM Transactions on Graphics (TOG), vol. 30, p. 125. ACM, New York (2011)Google Scholar
  14. 14.
    Fish, N., van Kaick, O., Bermano, A., Cohen-Or, D.: Structure-oriented networks of shape collections. ACM Trans. Graph. (TOG) 35(6), 171 (2016)Google Scholar
  15. 15.
    Su, H., Qi, C.R., Li, Y., Guibas, L.J.: Render for CNN: viewpoint estimation in images using CNNs trained with rendered 3d model views. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2686–2694 (2015)Google Scholar
  16. 16.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)Google Scholar
  17. 17.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)Google Scholar
  18. 18.
    Shapira, L., Shamir, A., Cohen-Or, D.: Consistent mesh partitioning and skeletonisation using the shape diameter function. Vis. Comput. 24(4), 249 (2008)Google Scholar
  19. 19.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)Google Scholar
  20. 20.
    Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. In: Advances in Neural Information Processing Systems, pp. 737–744 (1994)Google Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS’12, vol 1, pp. 1097–1105. Curran Associates Inc., Red Hook (2012)Google Scholar
  22. 22.
    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  23. 23.
    Andriy, M., Xubo, S.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–75 (2010)Google Scholar
  24. 24.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)zbMATHGoogle Scholar
  25. 25.
    Chen, X., Golovinskiy, A., Funkhouser, T.: A benchmark for 3D mesh segmentation. In: ACM Transactions on Graphics (Proceedings of SIGGRAPH), August, vol. 28, no. (3) (2009)Google Scholar
  26. 26.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  27. 27.
    Kim, V.G., Li, W., Mitra, N.J., Chaudhuri, S., DiVerdi, S., Funkhouser, T.: Learning part-based templates from large collections of 3d shapes. ACM Trans. Graph. (TOG) 32(4), 70 (2013)zbMATHGoogle Scholar
  28. 28.
    Van Kaick, O., Fish, N., Kleiman, Y., Asafi, S., Cohen-Or, D.: Shape segmentation by approximate convexity analysis. ACM Trans. Graph. 34(1), 1–11 (2014)Google Scholar
  29. 29.
    Xie, Z., Xu, K., Liu, K., Xiong, Y.: 3d shape segmentation and labeling via extreme learning machine. In: Computer Graphics Forum, vol. 33, pp. 85–95. Wiley, New York (2014)Google Scholar
  30. 30.
    Xu, W., Shi, Z., Xu, M., Zhou, K., Wang, J., Zhou, B., Wang, J., Yuan, Z.: Transductive 3d shape segmentation using sparse reconstruction. In: Computer Graphics Forum, vol. 33, pp. 107–115. Wiley, New York (2014)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Juncheng Liu
    • 1
    • 2
  • Paul L. Rosin
    • 2
  • Xianfang Sun
    • 2
  • Jianguo Xiao
    • 1
  • Zhouhui Lian
    • 1
    Email author
  1. 1.Institute of Computer Science and TechnologyPeking UniversityBeijingChina
  2. 2.School of Computer Science and InformaticsCardiff UniversityCardiffUK

Personalised recommendations