Advertisement

Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry

Conference paper
  • 1.7k Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12346)

Abstract

Deep implicit field regression methods are effective for 3D reconstruction from single-view images. However, the impact of different sampling patterns on the reconstruction quality is not well-understood. In this work, we first study the effect of point set discrepancy on the network training. Based on Farthest Point Sampling algorithm, we propose a sampling scheme that theoretically encourages better generalization performance, and results in fast convergence for SGD-based optimization algorithms. Secondly, based on the reflective symmetry of an object, we propose a feature fusion method that alleviates issues due to self-occlusions which makes it difficult to utilize local image features. Our proposed system Ladybird is able to create high quality 3D object reconstructions from a single input image. We evaluate Ladybird on a large scale 3D dataset (ShapeNet) demonstrating highly competitive results in terms of Chamfer distance, Earth Mover’s distance and Intersection Over Union (IoU).

Keywords

3D reconstruction Deep learning Sampling Symmetry 

Notes

Acknowledgement

We would like to thank the anonymous reviewers for their helpful feedback and suggestions. We would like to thank Zilei Huang for his help in accelerating the data processing and debugging.

Supplementary material

500725_1_En_15_MOESM1_ESM.pdf (2.9 mb)
Supplementary material 1 (pdf 2991 KB)

References

  1. 1.
    Uni(corn—form) tool kit. https://utk-team.github.io/utk/
  2. 2.
    Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 16), pp. 265–283 (2016)Google Scholar
  3. 3.
    Chang, A.X., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  4. 4.
    Chen, Z., Tagliasacchi, A., Zhang, H.: BSP-Net: generating compact meshes via binary space partitioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45–54 (2020)Google Scholar
  5. 5.
    Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5939–5948 (2019)Google Scholar
  6. 6.
    Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_38CrossRefGoogle Scholar
  7. 7.
    Eldar, Y., Lindenbaum, M., Porat, M., Zeevi, Y.Y.: The farthest point strategy for progressive image sampling. IEEE Trans. Image Process. 6(9), 1305–1315 (1997)CrossRefGoogle Scholar
  8. 8.
    Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605–613 (2017)Google Scholar
  9. 9.
    Gao, L., et al.: SDM-NET: deep generative network for structured deformable mesh. ACM Trans. Graph. (TOG) 38(6), 1–15 (2019)Google Scholar
  10. 10.
    Gkioxari, G., Malik, J., Johnson, J.: Mesh R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9785–9795 (2019)Google Scholar
  11. 11.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: Atlasnet: Apapier-m\(\backslash \hat{}\) ach\(\backslash \)’e approach to learning 3d surfacegeneration. arXiv preprint arXiv:1802.05384 (2018)
  12. 12.
    Halton, J.H.: Algorithm 247: radical-inverse quasi-random point sequence. Commun. ACM 7(12), 701–702 (1964)CrossRefGoogle Scholar
  13. 13.
    Joe, S., Kuo, F.Y.: Constructing Sobol sequences with better two-dimensional projections. SIAM J. Sci. Comput. 30(5), 2635–2654 (2008)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arxiv:1412.6980 (2014)
  15. 15.
    Kuipers, L., Niederreiter, H.: Uniform Distribution of Sequences. Courier Corporation, North Chelmsford (2012)zbMATHGoogle Scholar
  16. 16.
    Li, K., Pham, T., Zhan, H., Reid, I.: Efficient dense point cloud object reconstruction using deformation vector fields. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 497–513 (2018)Google Scholar
  17. 17.
    Lin, C.H., Kong, C., Lucey, S.: Learning efficient point cloud generation for dense 3D object reconstruction. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  18. 18.
    Liu, S., Zhang, Y., Peng, S., Shi, B., Pollefeys, M., Cui, Z.: Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. arXiv preprint arXiv:1911.13225 (2019)
  19. 19.
    Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3D reasoning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7708–7717 (2019)Google Scholar
  20. 20.
    Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3D reconstruction in function space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460–4470 (2019)Google Scholar
  21. 21.
    Mitchell, E., Engin, S., Isler, V., Lee, D.D.: Higher-order function networks for learning composable 3d object representations. arXiv preprint arXiv:1907.10388 (2019)
  22. 22.
    Niederreiter, H.: Low-discrepancy and low-dispersion sequences. J. Number Theor. 30(1), 51–70 (1988)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Pan, J., Han, X., Chen, W., Tang, J., Jia, K.: Deep mesh reconstruction from single RGB images via topology modification networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9964–9973 (2019)Google Scholar
  24. 24.
    Pilleboue, A., Singh, G., Coeurjolly, D., Kazhdan, M., Ostromoukhov, V.: Variance analysis for Monte Carlo integration. ACM Trans. Graph. (Proc. SIGGRAPH) 34(4), 124:1–124:14 (2015)CrossRefGoogle Scholar
  25. 25.
    Singh, G., et al.: Analysis of sample correlations for Monte Carlo rendering. Comput. Graph. Forum 38(2), 473–491 (2019)CrossRefGoogle Scholar
  26. 26.
    Sun, X., et al.: Pix3D: Dataset and methods for single-image 3D shape modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2974–2983 (2018)Google Scholar
  27. 27.
    Villani, C.: Optimal Transport: Old and New, vol. 338. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-71050-9CrossRefzbMATHGoogle Scholar
  28. 28.
    Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: generating 3D mesh models from single RGB images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 52–67 (2018)Google Scholar
  29. 29.
    Wang, W., Ceylan, D., Mech, R., Neumann, U.: 3DN: 3D deformation network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1038–1046 (2019)Google Scholar
  30. 30.
    Xie, H., Yao, H., Sun, X., Zhou, S., Zhang, S.: Pix2Vox: context-aware 3D reconstruction from single and multi-view images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2690–2698 (2019)Google Scholar
  31. 31.
    Xu, H., Barbič, J.: Signed distance fields for polygon soup meshes. Proc. Graph. Interface 2014, 35–41 (2014)Google Scholar
  32. 32.
    Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711 (2019)
  33. 33.
    Yao, Y., Schertler, N., Rosales, E., Rhodin, H., Sigal, L., Sheffer, A.: Front2back: single view 3d shape reconstruction via front to back prediction. arXiv preprint arXiv:1912.10589 (2019)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Netease Fuxi AI LabHangzhouChina
  2. 2.Saarland Informatics CampusSaarbrueckenGermany
  3. 3.MPI for InformaticsSaarbrueckenGermany

Personalised recommendations