Furniture style compatibility recommendation with cross-class triplet loss

  • Tse-Yu Pan
  • Yi-Zhu Dai
  • Min-Chun Hu
  • Wen-Huang Cheng
Article
  • 26 Downloads

Abstract

Harmonizing the style of all the furniture placed within a constrained space/scene is an important principle for interior design. In this paper, we propose a furniture style compatibility recommendation approach for users to create a harmonic 3D virtual scene based on 2D furniture photos. Most previous works of 3D model style analysis measure the style similarity or compatibility based on predefined geometric features extracted from 3D models. However, “style” is a high-level semantic concept, which is difficult to be described explicitly by hand-crafted geometric features. Moreover, analyzing the style compatibility between two or more furniture belonging to different classes (e.g., table and lamp) is much more challenging since the given furniture may have very distinctive structures or geometric elements. Recently, deep neural network has been claimed to have more powerful ability to mimic the perception of human visual cortex, and therefore we propose to analyze style compatibility between 3D furniture models of different classes based on a Cross-Class Triplet Convolutional Neural Network (CNN). We conducted experiments based on a collected dataset containing 420 textured 3D furniture models. A group of raters were recruited from Amazon Mechanical Turk (AMT) to evaluate the comparative suitability of paired models within the dataset. The experimental results reveal that the proposed furniture style compatibility method based on deep learning performs better than the state-of-the-art method and can be used to efficiently generate harmonic virtual scenes.

Keywords

Style compatibility Deep metric learning Triplet convolutional neural network 3D furniture model recommendation 

References

  1. 1.
    Balntas V, Johns E, Tang L, Mikolajczyk K (2016) PN-Net: conjoined triple deep network for learning local image descriptors, arXiv:1601.05030
  2. 2.
    Bansal A, Russell B, Gupta A (2016) Marr revisited: 2D-3D alignment via surface normal prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5965–5974Google Scholar
  3. 3.
    Bell S, Bala K (2015) Learning visual similarity for product design with convolutional neural networks. ACM Trans Graph (TOG) 34(4):98CrossRefGoogle Scholar
  4. 4.
    Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H et al (2015) ShapeNet: an information-rich 3D model repository, arXiv:1512.03012
  5. 5.
    Chang Y-J, Lo H-Y, Huang M-S, Hu M-C (2015) Representative photo selection for restaurants in food blogs. In: Proceedings of the IEEE international conference on multimedia & expo workshops, pp 1–6Google Scholar
  6. 6.
    Chen K, Xu K, Yu Y, Wang T-Y, Hu S-M (2015) Magic decorator: automatic material suggestion for indoor digital scenes. ACM Trans Graph (TOG) 34(6):232Google Scholar
  7. 7.
    Chen X, Li J, Li Q, Gao B, Zou D, Zhao Q (2015) Image2scene: transforming style of 3D room. In: Proceedings of ACM multimedia conferenceGoogle Scholar
  8. 8.
    Cheng Z, Shen J, Mei T (2014) Just-for-me: an adaptive personalization system for location-aware social music recommendation. In: Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval, pp 1267–1268Google Scholar
  9. 9.
    Dev K, Kim K, Villar N, Lau M (2016) Improving style similarity metrics of 3D shapes. In: Proceedings of the 42nd graphics interface conference. Canadian information processing society, pp 175–182Google Scholar
  10. 10.
    Edwards R, Holland J (2013) What is qalitative interviewing? Bloomsbury Publishing, LondonGoogle Scholar
  11. 11.
    Fisher M, Ritchie D, Savva M, Funkhouser T, Hanrahan P (2016) Example-based synthesis of 3D object arrangements. ACM Trans Graph (TOG) 31(6):135Google Scholar
  12. 12.
    Fukano K, Mochizuki Y, Iizuka S (2016) Room reconstruction from a single spherical image by higher-order energy minimization. In: Proceedings of the IEEE international conference on pattern recognitionGoogle Scholar
  13. 13.
    Garces E, Agarwala A, Gutierrez D, Hertzmann A (2014) A similarity measure for illustration style. ACM Trans Graph (TOG) 33(4):93CrossRefGoogle Scholar
  14. 14.
    Garces E, Agarwala A, Hertzmann A, Gutierrez D (2017) Style-based exploration of illustration datasets. Multimedia Tools and Applications 76(11):13067–13086CrossRefGoogle Scholar
  15. 15.
    Guo R, Zou C, Hoiem D (2015) Predicting Complete 3D Models of Indoor Scenes, arXiv:1504.02437
  16. 16.
    Guo H, Wang J, Gao Y, Li J, Lu H (2016) Multi-view 3D object retrieval with deep embedding network. IEEE Trans Image Process 25(12):5526–5537MathSciNetCrossRefGoogle Scholar
  17. 17.
    Hoffer E, Ailon N (2015) Deep metric learning using triplet network. In: International workshop on similarity-based pattern recognition. Springer, Berlin, pp 84–92Google Scholar
  18. 18.
    Hu M-C, Pan T-Y, Lo L-Y, Lo H-Y (2017) Framework and method for creating virtual model of three-dimensional space, Google Patents US Patent 9547943. https://www.google.com/patents/US9547943
  19. 19.
    Huang Q-X, Su H, Guibas L (2013) Fine-grained semi-supervised labeling of large shape collections. ACM Trans Graph (TOG) 32(6):190Google Scholar
  20. 20.
  21. 21.
    Isaak L, Anne G, Leif K (2016) Identifying style of 3D shapes using deep metric learning. Comput Graphics Forum 35(5):207–215CrossRefGoogle Scholar
  22. 22.
    Izadinia H, Shan Q, Seitz SM (2017) IM2CAD. In: Proceedings of the IEEE international conference on pattern recognitionGoogle Scholar
  23. 23.
    Jia Y et al (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia, pp 675–678Google Scholar
  24. 24.
    Kulis B et al (2013) Metric learning: a survey. Foundations and Trends® in Machine Learning 5(4):287–364MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Li Z, Tang J (2015) Weakly supervised deep metric learning for Community-Contributed image retrieval. IEEE Trans Multimedia 17(11):1989–1999CrossRefGoogle Scholar
  26. 26.
    Li Z, Tang J (2017) Weakly supervised deep matrix factorization for social image understanding. IEEE Trans Image Process 26(1):276–288MathSciNetCrossRefGoogle Scholar
  27. 27.
    Li Z, Liu J, Tang J, Lu H (2015) Robust structured subspace learning for data representation. IEEE Trans Pattern Anal Mach Intell 37(10):2085–2098CrossRefGoogle Scholar
  28. 28.
    Liu T, Hertzmann A, Li W, Funkhouser T (2015) Style compatibility for 3D furniture models. ACM Trans Graph (TOG) 34(4):85CrossRefGoogle Scholar
  29. 29.
    López GL, Negrón APP, De Antonio Jiménez A, Rodríguez JR, Paredes RI (2016) Comparative analysis of shape descriptors for 3D objects. Multimedia Tools and Applications 76(5):6993–7040Google Scholar
  30. 30.
    Lun Z, Kalogerakis E, Sheffer A (2015) Elements of style: learning perceptual shape style similarity. ACM Trans Graph (TOG) 34(4):84CrossRefGoogle Scholar
  31. 31.
    Luo Y, Liu T, Tao D, Xu C (2014) Decomposition-based transfer distance metric learning for image classification. IEEE Trans Image Process 23(9):3789–3801MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Mallya A, Lazebnik S (2015) Learning informative edge maps for indoor scene layout prediction. In: Proceedings of the IEEE international conference on computer visionGoogle Scholar
  33. 33.
    Mei T, Yang B, Hua X-S, Li S (2011) Contextual video recommendation by multimodal relevance and user feedback. ACM Trans Inf Syst 29(2):10CrossRefGoogle Scholar
  34. 34.
    O’Donovan P, Lı̄beks J, Agarwala A, Hertzmann A (2014) Exploratory font selection using crowdsourced attributes. ACM Trans Graph (TOG) 33(4):92Google Scholar
  35. 35.
    Ren S, He K, Girshick R, Sun J (2017) Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149CrossRefGoogle Scholar
  36. 36.
    Saleh B, Dontcheva M, Hertzmann A, Liu Z (2015) Learning style similarity for searching infographics. In: Proceedings of the 41st graphics interface conference. Canadian information processing society, pp 59–64Google Scholar
  37. 37.
    Wang Y, Wang H, Li X (2013) An intelligent recommendation system model based on style for virtual home furnishing in Three-Dimensional scene. In: Proceedings of the IEEE international symposium on computational and business intelligence, pp 213–216Google Scholar
  38. 38.
    Wilber MJ, Kwak IS, Belongie SJ (2014) Cost-effective hits for relative similarity comparisons. In: Second AAAI conference on human computation and crowdsourcingGoogle Scholar
  39. 39.
    Xie J, Dai G, Zhu F, Wong E, Fang Y (2017) Deepshape: Deep-learned shape descriptor for 3D shape retrieval. IEEE Trans Pattern Anal Mach Intell 39(7):1335–1345CrossRefGoogle Scholar
  40. 40.
    Xu K, Li H, Zhang H, Cohen-Or D, Xiong Y, Cheng Z-Q (2010) Stylecontent separation by anisotropic part scales. ACM Trans Graph (TOG) 29(6):184CrossRefGoogle Scholar
  41. 41.
    Yu L-F, Yeung S-K, Tang C-K, Terzopoulos D, Chan TF, Osher SJ (2011) Make it home: automatic optimization of furniture arrangement. ACM Trans Graph (TOG) 30(4):86CrossRefGoogle Scholar
  42. 42.
    Yumer ME, Chaudhuri S, Hodgins JK, Kara LB (2015) Semantic shape editing using deformation handles. ACM Trans Graph (TOG) 34(4):86CrossRefGoogle Scholar
  43. 43.
    Zhang Y, Song S, Tan P, Xiao J (2015) Panocontext: a whole-room 3D context model for panoramic scene understanding. In: Proceedings of european conference on computer visionGoogle Scholar
  44. 44.
    Zhu J, Guo Y, Ma H (2017) A data-driven approach for furniture and indoor scene colorization. IEEE Trans Vis Comput Graph PP(99):1–1CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science and Information EngineeringNational Cheng Kung UniversityTainanTaiwan
  2. 2.Research Center for Information Technology Innovation (CITI)Academia SinicaTaipeiTaiwan

Personalised recommendations