Skip to main content
SpringerLink
Log in

Fast incremental structure from motion based on parallel bundle adjustment

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Structure from motion has attracted a lot of research in recent years, with new state-of-the-art approaches coming almost every year. One of its advantages over 3D reconstruction is that it can be used for any cameras (UAVs, depth sensor, light field) and produces relatively accurate point clouds and camera parameters. One of its disadvantages compared to other approaches is that it is computationally expensive. In this paper, we design a novel structure-from-motion framework to reduce the computational cost and implement a parallel bundle adjustment on GPU device for large-scale optimization. In our framework, the local bundle adjustment is added into the architecture of the incremental structure from motion; namely, the point clouds and camera’s parameters are optimized when an additional number of images was added. Then, the purpose is not only to improve the quality of the produced point clouds but also to reduce computation time via parallel bundle adjustment. We conduct extensively experiments on several challenging datasets and make comparison with the state-of-the-art methods. Experimental results show that the proposed method has the best performance in terms of accuracy and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. In: ACM Transactions on Graphics (TOG) 2006, vol. 25, pp. 835–846. ACM

  2. Cui, Q., Fragoso, V., Sweeney, C., Sen, P.: GraphMatch: Efficient Large-Scale Graph Construction for Structure from Motion. arXiv preprint arXiv:1710.01602 (2017)

  3. Cao, M.W., Jia, W., Zhao, Y., Li, S.J., Liu, X.P.: Fast and robust absolute camera pose estimation with known focal length. Neural Comput. Appl. 29(5), 1383–1398 (2017). https://doi.org/10.1007/s00521-017-3032-6

    Article  Google Scholar 

  4. Chatterjee, A., Govindu, V.: Robust relative rotation averaging. IEEE Trans. Pattern Anal. Mach. Intell. (2017). https://doi.org/10.1109/tpami.2017.2693984

    Article  Google Scholar 

  5. Kang, L., Wu, L., Yang, Y.-H.: Robust multi-view L2 triangulation via optimal inlier selection and 3D structure refinement. Pattern Recogn. 47(9), 2974–2992 (2014)

    Article  Google Scholar 

  6. Cao, M.W., Li, S.J., Jia, W., Li, S.L., Liu, X.P.: Robust bundle adjustment for large-scale structure from motion. Multimed. Tools Appl. 76(21), 21843–21867 (2017). https://doi.org/10.1007/s11042-017-4581-5

    Article  Google Scholar 

  7. Locher, A., Perdoch, M., Gool, L.V.: Progressive prioritized multi-view stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3244–3252

  8. Ummenhofer, B., Brox, T.: Global, dense multiscale reconstruction for a billion points. Int. J. Comput. Vis. 125(1), 82–94 (2017). https://doi.org/10.1007/s11263-017-1017-7

    Article  MathSciNet  Google Scholar 

  9. Colbert, M., et al.: Building indoor multi-panorama experiences at scale. In: ACM Siggraph 2012 Talks2012, p. 24. ACM

  10. Michael, N., Drakou, M., Lanitis, A.: Model-based generation of personalized full-body 3D avatars from uncalibrated multi-view photographs. Multimed. Tools Appl. (2016). https://doi.org/10.1007/s11042-016-3808-1

    Article  Google Scholar 

  11. Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, 2007. ISMAR 2007, pp. 225–234. IEEE (2007)

  12. Kelly, T., Femiani, J., Wonka, P., Mitra, N.J.: BigSUR: large-scale structured urban reconstruction. ACM Trans. Gr. 36(6), 204 (2017). https://doi.org/10.1145/3130800.3130823

    Article  Google Scholar 

  13. Song, S., Chandraker, M.: Robust scale estimation in real-time monocular SFM for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1566–1573

  14. Schönberger, J.L., Frahm, J.-M.: Structure-from-motion revisited. In: Proceedings of CVPR IEEE, 27–30 June 2016, pp. 4104-4113. CVPR (2016)

  15. Wu, C.: Towards linear-time incremental structure from motion. In: International Conference on 3D Vision-3DV, pp. 127–134. IEEE (2013)

  16. Wu, C.: SiftGPU: A GPU Implementation of Scale Invariant Feature Transform. http://cs.unc.edu/~ccwu/siftgpu (2011)

  17. Ni, K., Dellaert, F.: HyperSfM. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 144–151. IEEE (2012)

  18. Guofeng, Z., Haomin, L., Zilong, D., Jiaya, J., Tien-Tsin, W., Hujun, B.: Efficient non-consecutive feature tracking for robust structure-from-motion. IEEE Trans. Image Process. 25(12), 5957–5970 (2016). https://doi.org/10.1109/TIP.2016.2607425

    Article  MathSciNet  MATH  Google Scholar 

  19. Crandall, D.J., Owens, A., Snavely, N., Huttenlocher, D.P.: SfM with MRFs: discrete-continuous optimization for large-scale structure from motion. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2841–2853 (2013). https://doi.org/10.1109/TPAMI.2012.218

    Article  Google Scholar 

  20. Micusik, B., Wildenauer, H.: Structure from motion with line segments under relaxed endpoint constraints. Int. J. Comput. Vis. (2016). https://doi.org/10.1007/s11263-016-0971-9

    Article  Google Scholar 

  21. Zhu, S., et al.: Very large-scale global SfM by distributed motion averaging. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18–23 June 2018, pp. 4568–4577 (2018)

  22. Dai, A., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM Trans. Gr. 36(4), 76 (2017)

    Article  Google Scholar 

  23. Sinha, S.N., Frahm, J.-M., Pollefeys, M., Genc, Y.: GPU-based video feature tracking and matching. In: EDGE, Workshop on Edge Computing Using New Commodity Architectures, 2006, vol. 278, pp. 189–196

  24. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment—a modern synthesis. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice, pp. 298–372. Springer, Berlin (1999)

    Google Scholar 

  25. Granshaw, S.I.: Bundle adjustment methods in engineering photogrammetry. Photogramm. Rec. 10(56), 181–207 (1980)

    Article  Google Scholar 

  26. Garro, V., Crosilla, F., Fusiello, A.: Solving the PNP problem with anisotropic orthogonal procrustes analysis. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, pp. 262–269. IEEE (2012)

  27. Mishkin, D., Matas, J., Perdoch, M.: MODS: fast and robust method for two-view matching. Comput. Vis. Image Underst. 141, 81–93 (2015). https://doi.org/10.1016/j.cviu.2015.08.005

    Article  Google Scholar 

  28. Wilson, K., Snavely, N.: Network principles for SfM: disambiguating repeated structures with local context. In: IEEE International Conference on Computer Vision, pp. 513–520 (2013)

  29. Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., Cheng, M.M.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2828–2837 (2017)

  30. Zhao, J., Ma, J., Tian, J., Ma, J., Zhang, D.: A robust method for vector field learning with application to mismatch removing. In: CVPR, vol. 32, no. 14, pp. 2977–2984 (2011)

  31. Jia, K., et al.: ROML: a robust feature correspondence approach for matching objects in a set of images. Int. J. Comput. Vis. 117(2), 173–197 (2016). https://doi.org/10.1007/s11263-015-0858-1

    Article  MathSciNet  MATH  Google Scholar 

  32. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  33. Wu, F.C., Zhang, Q., Hu, Z.Y.: Efficient suboptimal solutions to the optimal triangulation. Int. J. Comput. Vis. 91(1), 77–106 (2011)

    Article  MathSciNet  Google Scholar 

  34. Hartley, R.I., Sturm, P.: Triangulation. Comput. Vis. Image Underst. 68(2), 146–157 (1997)

    Article  Google Scholar 

  35. Agarwal, S., Snavely, N., Seitz, S.M.: Fast algorithms for L∞ problems in multiview geometry. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8. IEEE (2008)

  36. Huber, P.J.: Robust estimation of a location parameter. In: Kotz, S., Johnson, N.L. (eds.) Breakthroughs in Statistics, pp. 492–518. Springer, Berlin (1992)

    Chapter  Google Scholar 

  37. Fletcher, R., Powell, M.J.D.: On the modification of LDLT factorizations. Math. Comput. 28(128), 1067–1087 (1974)

    MATH  Google Scholar 

  38. Lourakis, M.I., Argyros, A.A.: SBA: a software package for generic sparse bundle adjustment. ACM Trans. Math. Softw. 36(1), 1–30 (2009)

    Article  MathSciNet  Google Scholar 

  39. Byröd, M., Åström, K.: Conjugate gradient bundle adjustment. In: Computer Vision, ECCV 2010, pp. 114–127. Springer (2010)

  40. Zach, C.: Robust bundle adjustment revisited. In: European Conference on Computer Vision, pp. 772–787. Springer (2014)

  41. Kümmerle, R., Grisetti, G., Strasdat, H., Konolige, K., Burgard, W.: g2o: a general framework for graph optimization. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 3607–3613. IEEE (2011)

  42. Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3057–3064. IEEE (2011)

  43. Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In: Computer Vision, ECCV 2010, pp. 29–42. Springer (2010)

Download references

Acknowledgements

The authors gratefully acknowledge the support of the National Key Research and Development Plan (Grant No. 2016YFC0800100), the National Natural Science Foundation (Grant Nos.: 61972128, 61802103, 61877016, 61602146 and 61673157), China Postdoctoral Science Foundation (Grant No.: 2018M632522), Fundamental Research Funds for the Central Universities (Grant Nos.: PA2019GDPK0071, JZ2018HGBH0280 and PA2018GDQT0014), and Key Research and Development Program in Anhui Province (Grant No.: 1804a09020036).

Author information

Authors and Affiliations

  1. Key Laboratory of Knowledge Engineering with Big Data, Ministry of Education, Hefei University of Technology, Hefei, 230009, China

    Mingwei Cao, Liping Zheng, Wei Jia & Xiaoping Liu

  2. Anhui Province Key Laboratory of Industry Safety and Emergency Technology, Hefei University of Technology, Hefei, 230009, China

    Mingwei Cao, Liping Zheng, Wei Jia & Xiaoping Liu

  3. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230009, China

    Mingwei Cao, Liping Zheng, Wei Jia & Xiaoping Liu

Authors
  1. Mingwei Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liping Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingwei Cao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, M., Zheng, L., Jia, W. et al. Fast incremental structure from motion based on parallel bundle adjustment. J Real-Time Image Proc 18, 379–392 (2021). https://doi.org/10.1007/s11554-020-00970-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00970-3

Keywords

Search

Navigation