Abstract
Bundle adjustment is a fundamental problem in computer vision, with important applications such as 3D structure reconstruction from 2D images. This paper focuses on large-scale bundle adjustment tasks, e.g., city-wide 3D reconstruction, which require highly efficient solutions. For this purpose, it is common to apply the Levenberg-Marquardt algorithm, whose bottleneck lies in solving normal equations. The majority of recent methods focus on achieving scalability through modern hardware such as GPUs and distributed systems. On the other hand, the core of the solution, i.e., the math underlying the optimizer for the normal equations, remains largely unimproved since the proposal of the classic parallel bundle adjustment (PBA) algorithm, which increasingly becomes a major limiting factor for the scalability of bundle adjustment.
This paper proposes parallel preconditioned conjugate gradient (PPCG) method, a novel parallel method for bundle adjustment based on preconditioned conjugate gradient, which achieves significantly higher efficiency and scalability than existing methods on the algorithmic level. The main idea is to exploit the sparsity of the Hessian matrix and reduce its structure parameters through an effective parallel Schur complement method; the result of this step is then fed into our carefully designed PPCG method which reduces matrix operations that are either expensive (e.g., large matrix reverse or multiplications) or scales poorly to multi-processors (e.g., parallel Reduce operators). Extensive experiments demonstrate that PPCG outperforms existing optimizers by large margins, on a wide range of datasets.
Similar content being viewed by others
References
Triggs B, McLauchlan P, Hartley R, Fitzgibbon A (2000) Bundle adjustment - a modern synthesis. In: Triggs B, Zisserman A, Szeliski R (eds) Vision algorithms: theory and practice. IWVA 1999 lecture notes in computer science, vol 1883. Springer, Berlin
Agarwal S, Snavely N, Seitz SM, Szeliski R (2010) Bundle Adjustment in the Large. In: European conference on computer vision. Springer, pp 29–42
Byröd M, Åström K (2010) Conjugate gradient bundle adjustment. In: European conference on computer vision. Springer, pp 114–127
Ma T, Kuang P, Tian W (2020) An improved recurrent neural networks for 3d object reconstruction. Appl Intell 50:905
Lhuillier M, Quan L (2005) A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Trans Pattern Anal Mach Intell 27(3):418
Klingner B, Martin D, Roseborough J (2013) Street view motion-from-structure-from-motion. In: Proceedings of the IEEE international conference on computer vision (ICCV). IEEE, pp 953–960
Engel J, Schöps T, Cremers D (2014) LSD-SLAM: Large-scale direct monocular SLAM. In: European conference on computer vision. Springer, pp 834–849
Vijayanarasimhan S, Ricco S, Schmid C, Sukthankar R, Fragkiadaki K (2017) Sfm-net: Learning of structure and motion from video
Zhou T, Brown M, Snavely N, Lowe DG (2017) Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 1851–1858
Wang C, Miguel Buenaposada J, Zhu R, Lucey S (2018) Learning depth from monocular videos using direct methods. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 2022–2030
Clark R, Bloesch M, Czarnowski J, Leutenegger S, Davison AJ (2018) Learning to solve nonlinear least squares for monocular stereo. In: Proceedings of the European conference on computer vision (ECCV). Springer, pp 284–299
Wu C, Agarwal S, Curless B, Seitz SM (2011) Multicore Bundle Adjustment. In: Proceedings of the 2011 IEEE conference on computer vision and pattern recognition. IEEE, pp 3057–3064
Agarwal S, Snavely N, Simon I, Seitz SM, Szeliski R (2011) Building Rome in a Day. In: Proceedings of the 12th International Conference on Computer Vision. IEEE, pp 72–79
Ni K, Steedly D, Dellaert F (2007) Out-of-Core bundle adjustment for large-scale 3D reconstruction. In: Proceedings of the IEEE international conference on computer vision (ICCV). IEEE, pp 1–8
Ni K, Dellaert F (2012) HyperSfM. In: 2012, Second international conference on 3D imaging, modeling, processing, visualization and transmission. IEEE, pp 144–151
Eriksson A, Bastian J, Chin TJ, Isaksson M (2016) A consensus-based framework for distributed bundle adjustment. In: The IEEE Conference on computer vision and pattern recognition (CVPR). IEEE, pp 1754–1762
Zheng S, Kwok JT (2016) Fast-and-light stochastic ADMM. In: Proceedings of the twenty-fifth international joint conference on artificial intelligece (IJCAI-16). Morgan Kaufmann, pp 2407–2413
Huang F, Gao S, Chen S, Huang H (2019) Zeroth-order stochastic alternating direction method of multipliers for nonconvex nonsmooth optimization. In: Proceedings of the twenty-eighth international joint conference on artificial intelligece (IJCAI-19). Morgan Kaufmann, pp 2549–2555
Yu Y, Huang L (2017) Fast stochastic variance reduced ADMM for stochastic composition optimization. In: Proceedings of the twenty-sixth international joint conference on artificial Intelligece (IJCAI-17). Morgan Kaufmann, pp 3364–3370
Mayer H (2019) RPBA - robust parallel bundle adjustment based on covariance information
Lourakis MIA, Argyros AA (2009) SBA: A software package for generic sparse bundle adjustment. ACM Trans Math Softw 36(1):2:1
Jeong Y, Nistér D, Steedly D, Szeliski R, Kweons IS (2012) Pushing the envelope of modern methods for bundle adjustment. IEEE Trans Pattern Anal Mach Intell 34(8):1605
Hennessy JL, Patterson DA (2012) Computer architecture a quantitative approach (5th edition). Elsevier, New York
Acknowledgements
This research was supported by the National Key Research and Development Program of China (2017YFB0202104).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Peng, J., Liu, J. & Wei, H. A scalable parallel preconditioned conjugate gradient method for bundle adjustment. Appl Intell 52, 753–765 (2022). https://doi.org/10.1007/s10489-021-02349-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-021-02349-8