Computational Visual Media

, Volume 2, Issue 1, pp 87–96 | Cite as

An interactive approach for functional prototype recovery from a single RGBD image

  • Yuliang Rong
  • Youyi Zheng
  • Tianjia ShaoEmail author
  • Yin Yang
  • Kun Zhou
Open Access
Research Article


Inferring the functionality of an object from a single RGBD image is difficult for two reasons: lack of semantic information about the object, and missing data due to occlusion. In this paper, we present an interactive framework to recover a 3D functional prototype from a single RGBD image. Instead of precisely reconstructing the object geometry for the prototype, we mainly focus on recovering the object’s functionality along with its geometry. Our system allows users to scribble on the image to create initial rough proxies for the parts. After user annotation of high-level relations between parts, our system automatically jointly optimizes detailed joint parameters (axis and position) and part geometry parameters (size, orientation, and position). Such prototype recovery enables a better understanding of the underlying image geometry and allows for further physically plausible manipulation. We demonstrate our framework on various indoor objects with simple or hybrid functions.


functionality cuboid proxy prototype part relations shape analysis 

Supplementary material

41095_2016_32_MOESM1_ESM.mp4 (40.7 mb)
Supplementary material, approximately 41628 KB.


  1. [1]
    Han, Y.; Lee, J.-Y.; Kweon, I. S. High quality shape from a single RGB-D image under uncalibrated natural illumination. In: Proceedings of IEEE International Conference on Computer Vision, 1617–1624, 2013.Google Scholar
  2. [2]
    Izadi, S.; Kim, D.; Hilliges, O.; Molyneaux, D.; Newcombe, R.; Kohli, P.; Shotton, J.; Hodges, S.; Freeman, D.; Davison, A.; Fitzgibbon, A. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 559–568, 2011.Google Scholar
  3. [3]
    Shao, T.; Monszpart, A.; Zheng, Y.; Koo, B.; Xu, W.; Zhou, K.; Mitra, N. J. Imagining the unseen: Stabilitybased cuboid arrangements for scene understanding. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 209, 2014.CrossRefGoogle Scholar
  4. [4]
    Shen, C.-H.; Fu, H.; Chen, K.; Hu, S.-M. Structure recovery by part assembly. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 180, 2012.CrossRefGoogle Scholar
  5. [5]
    Zheng, Y.; Chen, X.; Cheng, M.-M.; Zhou, K.; Hu, S.-M.; Mitra, N. J. Interactive images: Cuboid proxies for smart image manipulation. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 99, 2012.Google Scholar
  6. [6]
    Sullivan, L. H. The tall office building artistically considered. Lippincott’s Magazine 57, 1896.Google Scholar
  7. [7]
    Koo, B.; Li, W.; Yao, J.; Agrawala, M.; Mitra, N. J. Creating works-like prototypes of mechanical objects. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 217, 2014.CrossRefGoogle Scholar
  8. [8]
    Li, Y.; Wu, X.; Chrysanthou, Y.; Sharf, A.; Cohen-Or, D.; Mitra, N. J. GlobFit: Consistently fitting primitives by discovering global relations. ACM Transactions on Graphics Vol. 30, No. 4, Article No. 52, 2011.Google Scholar
  9. [9]
    Lafarge, F.; Alliez, P. Surface reconstruction through point set structuring. Computer Graphics Forum Vol. 32, No. 2pt2, 225–234, 2013.CrossRefGoogle Scholar
  10. [10]
    Arikan, M.; Schwärzler, M.; Flöry, S.; Wimmer, M.; Maierhofer, S. O-snap: Optimization-based snapping for modeling architecture. ACM Transactions on Graphics Vol. 32, No. 1, Article No. 6, 2013.CrossRefzbMATHGoogle Scholar
  11. [11]
    Gupta, A.; Efros, A. A.; Hebert, M. Blocks world revisited: Image understanding using qualitative geometry and mechanics. In: Lecture Notes in Computer Science, Vol. 6314. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer Berlin Heidelberg, 482–496, 2010.CrossRefGoogle Scholar
  12. [12]
    Gupta, A.; Hebert, M.; Kanade, T.; Blei, D. M. Estimating spatial layout of rooms using Volumetric reasoning about objects and surfaces. In: Advances in Neural Information Processing Systems 23. Lafferty, J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; Culotta, A. Eds. Curran Associates, Inc., 1288–1296, 2010.Google Scholar
  13. [13]
    Del Pero, L.; Bowdish, J.; Fried, D.; Kermgard, B.; Hartley, E.; Barnard, K. Bayesian geometric modeling of indoor scenes. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2719–2726, 2012.Google Scholar
  14. [14]
    Jia, Z.; Gallagher, A.; Saxena, A.; Chen, T. 3Dbased reasoning with blocks, support, and stability. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2013.Google Scholar
  15. [15]
    Jiang, H.; Xiao, J. A linear approach to matching cuboids in RGBD images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2171–2178, 2013.Google Scholar
  16. [16]
    Zheng, B.; Zhao, Y.; Yu, J. C.; Ikeuchi, K.; Zhu, S.-C. Beyond point clouds: Scene understanding by reasoning geometry and physics. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 3127–3134, 2013.Google Scholar
  17. [17]
    Umetani, N.; Igarashi, T.; Mitra, N. J. Guided exploration of physically valid shapes for furniture design. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 86, 2012.CrossRefGoogle Scholar
  18. [18]
    Shao, T.; Li, W.; Zhou, K.; Xu, W.; Guo, B.; Mitra, N. J. Interpreting concept sketches. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 56, 2013.CrossRefGoogle Scholar
  19. [19]
    Bokeloh, M.; Wand, M.; Seidel, H.-P.; Koltun, V. An algebraic model for parameterized shape editing. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 78, 2012.CrossRefGoogle Scholar
  20. [20]
    Gal, R.; Sorkine, O.; Mitra, N. J.; Cohen-Or, D. iWIRES: An analyze-and-edit approach to shape manipulation. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 33, 2009.CrossRefGoogle Scholar
  21. [21]
    Xu, W.; Wang, J.; Yin, K.; Zhou, K.; van de Panne, M.; Chen, F.; Guo, B. Joint-aware manipulation of deformable models. ACM Transactions on Graphic Vol. 28, No. 3, Article No. 35, 2009.CrossRefGoogle Scholar
  22. [22]
    Daniel, M.; Lucas, M. Towards declarative geometric modelling in mechanics. In: Integrated Design and Manufacturing in Mechanical Engineering. Chedmail, P.; Bocquet, J.-C.; Dornfeld, D. Eds. Springer Netherlands, 427–436, 1997.CrossRefGoogle Scholar
  23. [23]
    Yvars, P.-A. Using constraint satisfaction for designing mechanical systems. International Journal on Interactive Design and Manufacturing Vol. 2, No. 3, 161–167, 2008.CrossRefGoogle Scholar
  24. [24]
    Zhang, Q.; Ye, M.; Yang, R.; Matsushita, Y.; Wilburn, B.; Yu, H. Edge-preserving photometric stereo via depth fusion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2472–2479, 2012.Google Scholar
  25. [25]
    Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.CrossRefGoogle Scholar
  26. [26]
    Schnabel, R.; Wahl, R.; Klein, R. Efficient RANSAC for point-cloud shape detection. Computer Graphics Forum Vol. 26, No. 2, 214–226, 2007.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2016

Authors and Affiliations

  • Yuliang Rong
    • 1
  • Youyi Zheng
    • 2
  • Tianjia Shao
    • 1
    Email author
  • Yin Yang
    • 3
  • Kun Zhou
    • 1
  1. 1.State Key Lab of CAD&CGZhejiang UniversityHangzhouChina
  2. 2.ShanghaiTech UniversityShanghaiChina
  3. 3.The University of New MexicoAlbuquerqueUSA

Personalised recommendations