3D Pick & Mix: Object Part Blending in Joint Shape and Image Manifolds

  • Adrian Penate-SanchezEmail author
  • Lourdes Agapito
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11361)


We present 3D Pick & Mix, a new 3D shape retrieval system that provides users with a new level of freedom to explore 3D shape and Internet image collections by introducing the ability to reason about objects at the level of their constituent parts. While classic retrieval systems can only formulate simple searches such as “find the 3D model that is most similar to the input image” our new approach can formulate advanced and semantically meaningful search queries such as: “find me the 3D model that best combines the design of the legs of the chair in image 1 but with no armrests, like the chair in image 2”. Many applications could benefit from such rich queries, users could browse through catalogues of furniture and pick and mix parts, combining for example the legs of a chair from one shop and the armrests from another shop.


Shape blending Image embedding Shape retrieval 

Supplementary material

Supplementary material 1 (mp4 27271 KB)


  1. 1.
    Alhashim, I., Li, H., Xu, K., Cao, J., Ma, R., Zhang, H.: Topology-varying 3D shape creation via structural blending. ACM Trans. Graph. 33(4) (2014). Article No. 158CrossRefGoogle Scholar
  2. 2.
    Aubry, M., Maturana, D., Efros, A., Russell, B., Sivic, J.: Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of cad models. In: Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  3. 3.
    Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. Technical report arXiv:1512.03012 [cs.GR], Stanford University – Princeton University – Toyota Technological Institute at Chicago (2015)
  4. 4.
    Chen, D.Y., Tian, X.P., Shen, Y.T., Ouhyoung, M.: On visual similarity based 3D model retrieval. Comput. Graph. Forum 22(3), 223–232 (2003)CrossRefGoogle Scholar
  5. 5.
    Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016). Scholar
  6. 6.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition (CVPR) (2005)Google Scholar
  7. 7.
    Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  8. 8.
    Fish, N., van Kaick, O., Bermano, A., Cohen-Or, D.: Structure-oriented networks of shape collections. ACM Trans. Graph. 35(6), 171 (2016)CrossRefGoogle Scholar
  9. 9.
    Girdhar, R., Fouhey, D.F., Rodriguez, M., Gupta, A.: Learning a predictable and generative vector representation for objects. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 484–499. Springer, Cham (2016). Scholar
  10. 10.
    Hueting, M., Ovsjanikov, M., Mitra, N.: Crosslink: joint understanding of image and 3D model collections through shape and camera pose variations. ACM Trans. Graph. 34(6), 233 (2015). Proc. SIGGRAPH AsiaCrossRefGoogle Scholar
  11. 11.
    Kalogerakis, E., Chaudhuri, S., Koller, D., Koltun, V.: A probabilistic model for component-based shape synthesis. ACM Trans. Graph. 31(4), 55 (2012)CrossRefGoogle Scholar
  12. 12.
    Kar, A., Tulsiani, S., Carreira, J., Malik, J.: Category-specific object reconstruction from a single image. In: Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  13. 13.
    Kokkinos, I.: UberNet: training a ‘universal’ convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  14. 14.
    Kruskal, J.B.: Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 29(1), 1–27 (1964)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Li, Y., Su, H., Qi, C.R., Fish, N., Cohen-Or, D., Guibas, L.J.: Joint embeddings of shapes and images via CNN image purification. ACM Trans. Graph. 34(6), 234 (2015)Google Scholar
  16. 16.
    Lim, I., Gehre, A., Kobbelt, L.: Identifying style of 3D shapes using deep metric learning. Comput. Graph. Forum 35(5), 207–215 (2016)CrossRefGoogle Scholar
  17. 17.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  18. 18.
    Qi, C.R., Su, H., Niessner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3D data. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  19. 19.
    Sammon, J.W.: A nonlinear mapping for data structure analysis. IEEE Trans. Comput. 100(5), 401–409 (1969)CrossRefGoogle Scholar
  20. 20.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)Google Scholar
  21. 21.
    Su, H., Huang, Q., Mitra, N.J., Li, Y., Guibas, L.: Estimating image depth using shape collections. ACM Trans. Graph. 33(4), 37 (2014)Google Scholar
  22. 22.
    Su, H., Qi, C.R., Li, Y., Guibas, L.J.: Render for CNN: viewpoint estimation in images using CNNs trained with rendered 3D model views. In: International Conference on Computer Vision (ICCV) (2015)Google Scholar
  23. 23.
    Tasse, F.P., Dodgson, N.: Shape2vec: semantic-based descriptors for 3D shapes, sketches and images. ACM Trans. Graph. 35(6) (2016). Article No. 208CrossRefGoogle Scholar
  24. 24.
    Tulsiani, S., Su, H., Guibas, L.J., Efros, A.A., Malik, J.: Learning shape abstractions by assembling volumetric primitives. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  25. 25.
    Xiang, Y., Mottaghi, R., Savarese, S.: Beyond Pascal: a benchmark for 3D object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)Google Scholar
  26. 26.
    Xie, X., et al.: Sketch-to-design: context-based part assembly. Comput. Graph. Forum 32(8), 233–245 (2013)CrossRefGoogle Scholar
  27. 27.
    Xu, K., Zhang, H., Cohen-Or, D., Chen, B.: Fit and diverse: set evolution for inspiring 3D shape galleries. ACM Trans. Graph. 31(4), 57 (2012)CrossRefGoogle Scholar
  28. 28.
    Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., Xiong, Y.: Photo-inspired model-driven 3d object modeling. ACM Trans. Graph. 30(4) (2011). Article No. 80Google Scholar
  29. 29.
    Yi, L., et al.: A scalable active framework for region annotation in 3D shape collections. In: SIGGRAPH Asia (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity College LondonLondonUK
  2. 2.Oxford Robotics InstituteUniversity of OxfordOxfordUK

Personalised recommendations