Abstract
Reconstruction of man-made scenes from multi-view images is an important problem in computer vision and computer graphics. Observing that man-made scenes are usually composed of planar surfaces, we encode plane shape prior in reconstructing man-made scenes. Recent approaches for single-view reconstruction employ multi-branch neural networks to simultaneously segment planes and recover 3D plane parameters. However, the scale of available annotated data heavily limits the generalizability and accuracy of these supervised methods. In this paper, we propose multi-view regularization to enhance the capability of piecewise planar reconstruction during the training phase, without demanding extra annotated data. Our multi-view regularization enables the consistency among multiple views by making the feature embedding more robust against view change and lighting variations. Thus, the neural network trained by multi-view regularization performs better on a wide range of views and lightings in the test phase. Based on more consistent prediction results, we merge the recovered models from multiple views to reconstruct scenes. Our approach achieves state-of-the-art reconstruction performance compared to previous approaches on the public ScanNet dataset.
References
Gallup, D.; Frahm, J.-M.; Mordohai, P.; Yang, Q.; Pollefeys, M. Real-time plane-sweeping stereo with multiple sweeping directions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2007.
Hirschmuller, H. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 30, No. 2, 328–341, 2008.
Yao, Y.; Luo, Z. X.; Li, S. W.; Fang, T.; Quan, L. MVSNet: Depth inference for unstructured multi-view stereo. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11212. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer International Publishing, 785–801, 2018.
Yao, Y.; Luo, Z.; Li, S.; Shen, T.; Fang, T.; Quan, L. Recurrent MVSNet for high-resolution multiview stereo depth inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5525–5534, 2019.
Chen, R.; Han, S.; Xu, J.; Su, H. Point-based multiview stereo network. In: Proceedings of the IEEE International Conference on Computer Vision, 1538–1547, 2019.
Luo, K.; Guan, T.; Ju, L.; Huang, H.; Luo, Y. PMVSNet: Learning patch-wise matching confidence aggregation for multi-view stereo. In: Proceedings of the IEEE International Conference on Computer Vision, 10452–10461, 2019.
Yang, R.; Pollefeys, M. Multi-resolution real-time stereo on commodity graphics hardware. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003.
Monszpart, A.; Mellado, N.; Brostow, G. J.; Mitra, N. J. RAPter: Rebuilding man-made scenes with regular arrangements of planes. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 103, 2015.
Liu, C.; Yang, J.; Ceylan, D.; Yumer, E.; Furukawa, Y. PlaneNet: Piece-wise planar reconstruction from a single RGB image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2579–2588, 2018.
Yang, F. T.; Zhou, Z. H. Recovering 3D planes from a single image via convolutional neural networks. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11214- Ferrari, V.; Hebert, M.; Sminchisescu, C; Weiss, Y. Eds. Springer Cham, 87–103, 2018.
[11] Liu, C.; Kim, K.; Gu, J.; Furukawa, Y.; Kautz, J. PlaneRCNN: 3D plane detection and reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4450–4459, 2019.
Yu, Z.; Zheng, J.; Lian, D.; Zhou, Z.; Gao, S. Single-image piece-wise planar 3D reconstruction via associative embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1029–1037, 2019.
Zhang, Y. Z.; Xu, W. W.; Tong, Y. Y.; Zhou, K. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics Vol. 34, No. 5, Article No. 159, 2015.
Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; Niessner, M. ScanNet: Richlyannotated 3D reconstructions of indoor scenes. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 5828–5839, 2017.
Furukawa, Y.; Ponce, J. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 32, No. 8, 1362–1376, 2010.
Schonberger, J. L.; Zheng, E. L.; Frahm, J. M.; Pollefeys, M. Pixelwise view selection for unstructured multi-view stereo. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer International Publishing, 501–518, 2016.
Jensen, R.; Dahl, A.; Vogiatzis, G.; Tola, E.; Aanaes, H. Large scale multi-view stereopsis evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 406–413, 2014.
Knapitsch, A.; Park, J.; Zhou, Q.-Y.; Koltun, V. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 78, 2017.
Delage, E.; Lee, H.; Ng, A. Y. Automatic single-image 3d reconstructions of indoor manhattan world scenes. In: Robotics Research. Springer Tracts in Advanced Robotics, Vol. 28. Thrun, S.; Brooks, R.; Durrant-Whyte, H. Eds. Springer Berlin Heidelberg, 305–321, 2007.
Barinova, O.; Konushin, V.; Yakubenko, A.; Lee, K.; Lim, H.; Konushin, A. Fast automatic single-view 3-d reconstruction of urban scenes. In: Computer Vision -ECCV 2008. Lecture Notes in Computer Science, Vol. 5303. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin Heidelberg, 100–113, 2008.
Saxena, A.; Chung, S. H.; Ng, A. Y. Learning depth from single monocular images. In: Proceedings of the 18th International Conference on Neural Information Processing Systems, 1161–1168, 2005.
De Brabandere, B.; Neven, D.; Van Gool, L. Semantic instance segmentation for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 7–9, 2017.
Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; Lerer, A. Automatic differentiation in PyTorch. In: Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
Zhang, T. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of the 21st International Conference on Machine Learning, 2004.
Silberman, N.; Hoiem, D.; Kohli, P.; Fergus, R. Indoor segmentation and support inference from RGBD images. In: Computer Vision - ECCV 2012. Lecture Notes in Computer Science, Vol. 7576. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 746–760, 2012.
Acknowledgements
This work was supported by the National Key R&D Program of China under Grant 2017YFB1002202, the National Natural Science Foundation of China (NSFC) under Grant 61632006, as well as the Fundamental Research Funds for the Central Universities under Grants WK3490000003 and WK2100100030.
Author information
Authors and Affiliations
Corresponding author
Additional information
Weijie Xi is a master candidate in the Department of Electronic Engineering and Information Science, University of Science and Technology of China. His research interests focus on geometry in computer vision. Weijie Xi obtained his B.S. degree from Chongqing University in 2018. He started his master in University of Science and Technology of China in 2018.
Xuejin Chen is an associate professor of the University of Science and Technology of China. She received her B.S. degree in 2003 and Ph.D. degree in 2008 from the University of Science and Technology of China (USTC). She conducted research as a postdoctoral scholar in the Computer Graphics Lab at Yale University from 2008 to 2010. She visited Stanford University from Feb. to Aug. 2017. Her research interests include 3D modeling, geometry processing, sketch-based content generation, and scene understanding.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Xi, W., Chen, X. Reconstructing piecewise planar scenes with multi-view regularization. Comp. Visual Media 5, 337–345 (2019). https://doi.org/10.1007/s41095-019-0159-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-019-0159-7
Keywords
- scene modeling
- multi-view
- regularization
- neural network