Abstract
Inferring the functionality of an object from a single RGBD image is difficult for two reasons: lack of semantic information about the object, and missing data due to occlusion. In this paper, we present an interactive framework to recover a 3D functional prototype from a single RGBD image. Instead of precisely reconstructing the object geometry for the prototype, we mainly focus on recovering the object’s functionality along with its geometry. Our system allows users to scribble on the image to create initial rough proxies for the parts. After user annotation of high-level relations between parts, our system automatically jointly optimizes detailed joint parameters (axis and position) and part geometry parameters (size, orientation, and position). Such prototype recovery enables a better understanding of the underlying image geometry and allows for further physically plausible manipulation. We demonstrate our framework on various indoor objects with simple or hybrid functions.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Han, Y.; Lee, J.-Y.; Kweon, I. S. High quality shape from a single RGB-D image under uncalibrated natural illumination. In: Proceedings of IEEE International Conference on Computer Vision, 1617–1624, 2013.
Izadi, S.; Kim, D.; Hilliges, O.; Molyneaux, D.; Newcombe, R.; Kohli, P.; Shotton, J.; Hodges, S.; Freeman, D.; Davison, A.; Fitzgibbon, A. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 559–568, 2011.
Shao, T.; Monszpart, A.; Zheng, Y.; Koo, B.; Xu, W.; Zhou, K.; Mitra, N. J. Imagining the unseen: Stabilitybased cuboid arrangements for scene understanding. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 209, 2014.
Shen, C.-H.; Fu, H.; Chen, K.; Hu, S.-M. Structure recovery by part assembly. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 180, 2012.
Zheng, Y.; Chen, X.; Cheng, M.-M.; Zhou, K.; Hu, S.-M.; Mitra, N. J. Interactive images: Cuboid proxies for smart image manipulation. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 99, 2012.
Sullivan, L. H. The tall office building artistically considered. Lippincott’s Magazine 57, 1896.
Koo, B.; Li, W.; Yao, J.; Agrawala, M.; Mitra, N. J. Creating works-like prototypes of mechanical objects. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 217, 2014.
Li, Y.; Wu, X.; Chrysanthou, Y.; Sharf, A.; Cohen-Or, D.; Mitra, N. J. GlobFit: Consistently fitting primitives by discovering global relations. ACM Transactions on Graphics Vol. 30, No. 4, Article No. 52, 2011.
Lafarge, F.; Alliez, P. Surface reconstruction through point set structuring. Computer Graphics Forum Vol. 32, No. 2pt2, 225–234, 2013.
Arikan, M.; Schwärzler, M.; Flöry, S.; Wimmer, M.; Maierhofer, S. O-snap: Optimization-based snapping for modeling architecture. ACM Transactions on Graphics Vol. 32, No. 1, Article No. 6, 2013.
Gupta, A.; Efros, A. A.; Hebert, M. Blocks world revisited: Image understanding using qualitative geometry and mechanics. In: Lecture Notes in Computer Science, Vol. 6314. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer Berlin Heidelberg, 482–496, 2010.
Gupta, A.; Hebert, M.; Kanade, T.; Blei, D. M. Estimating spatial layout of rooms using Volumetric reasoning about objects and surfaces. In: Advances in Neural Information Processing Systems 23. Lafferty, J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; Culotta, A. Eds. Curran Associates, Inc., 1288–1296, 2010.
Del Pero, L.; Bowdish, J.; Fried, D.; Kermgard, B.; Hartley, E.; Barnard, K. Bayesian geometric modeling of indoor scenes. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2719–2726, 2012.
Jia, Z.; Gallagher, A.; Saxena, A.; Chen, T. 3Dbased reasoning with blocks, support, and stability. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2013.
Jiang, H.; Xiao, J. A linear approach to matching cuboids in RGBD images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2171–2178, 2013.
Zheng, B.; Zhao, Y.; Yu, J. C.; Ikeuchi, K.; Zhu, S.-C. Beyond point clouds: Scene understanding by reasoning geometry and physics. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 3127–3134, 2013.
Umetani, N.; Igarashi, T.; Mitra, N. J. Guided exploration of physically valid shapes for furniture design. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 86, 2012.
Shao, T.; Li, W.; Zhou, K.; Xu, W.; Guo, B.; Mitra, N. J. Interpreting concept sketches. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 56, 2013.
Bokeloh, M.; Wand, M.; Seidel, H.-P.; Koltun, V. An algebraic model for parameterized shape editing. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 78, 2012.
Gal, R.; Sorkine, O.; Mitra, N. J.; Cohen-Or, D. iWIRES: An analyze-and-edit approach to shape manipulation. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 33, 2009.
Xu, W.; Wang, J.; Yin, K.; Zhou, K.; van de Panne, M.; Chen, F.; Guo, B. Joint-aware manipulation of deformable models. ACM Transactions on Graphic Vol. 28, No. 3, Article No. 35, 2009.
Daniel, M.; Lucas, M. Towards declarative geometric modelling in mechanics. In: Integrated Design and Manufacturing in Mechanical Engineering. Chedmail, P.; Bocquet, J.-C.; Dornfeld, D. Eds. Springer Netherlands, 427–436, 1997.
Yvars, P.-A. Using constraint satisfaction for designing mechanical systems. International Journal on Interactive Design and Manufacturing Vol. 2, No. 3, 161–167, 2008.
Zhang, Q.; Ye, M.; Yang, R.; Matsushita, Y.; Wilburn, B.; Yu, H. Edge-preserving photometric stereo via depth fusion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2472–2479, 2012.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.
Schnabel, R.; Wahl, R.; Klein, R. Efficient RANSAC for point-cloud shape detection. Computer Graphics Forum Vol. 26, No. 2, 214–226, 2007.
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is published with open access at Springerlink.com
Yuliang Rong is currently a senior undergraduate student majoring in computer science at Zhejiang University. He will work towards a master degree at the State Key Lab of CAD&CG in Zhejiang University after graduation. His research interests include image analysis and geometric modeling.
Youyi Zheng is currently an assistant professor at the School of Information Science and Technology, ShanghaiTech University. He obtained his Ph.D. degree from the Department of Computer Science and Engineering at Hong Kong University of Science & Technology, and his M.S. and B.S. degrees from the Department of Mathematics, Zhejiang University. His research interests include geometric modeling, imaging, and human–computer interaction.
Tianjia Shao is currently an assistant researcher at the State Key Lab of CAD&CG, Zhejiang University. He received his Ph.D. degree in computer science from the Institute for Advanced Study, and his B.S. degree from the Department of Automation, both at Tsinghua University. His research interests include RGBD image processing, indoor scene modeling, structure analysis, and 3D model retrieval.
Yin Yang received his Ph.D. degree in computer science from the University of Texas at Dallas in 2013. He is an assistant professor in the Electrical Communication Engineering Department, University of New Mexico, Albuquerque, USA. His research interests include physics-based animation and simulation, visualization, and medical imaging analysis.
Kun Zhou is a Cheung Kong professor in the Computer Science Department of Zhejiang University, and the director of the State Key Lab of CAD&CG. Prior to joining Zhejiang University in 2008, he was a lead researcher in the Internet Graphics Group at Microsoft Research Asia. He received his B.S. and Ph.D. degrees in computer science from Zhejiang University in 1997 and 2002, respectively. His research interests are visual computing, parallel computing, human–computer interaction, and virtual reality. He currently serves on the editorial/advisory boards of ACM Transactions on Graphics and IEEE Spectrum. He is a Fellow of the IEEE.
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Electronic supplementary material
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Rong, Y., Zheng, Y., Shao, T. et al. An interactive approach for functional prototype recovery from a single RGBD image. Comp. Visual Media 2, 87–96 (2016). https://doi.org/10.1007/s41095-016-0032-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-016-0032-x