International Journal of Computer Vision

, Volume 121, Issue 2, pp 234–252 | Cite as

MultiCol Bundle Adjustment: A Generic Method for Pose Estimation, Simultaneous Self-Calibration and Reconstruction for Arbitrary Multi-Camera Systems

  • Steffen Urban
  • Sven Wursthorn
  • Jens Leitloff
  • Stefan Hinz
Article

Abstract

In this paper, we present a generic, modular bundle adjustment method for pose estimation, simultaneous self-calibration and reconstruction for multi-camera systems. In contrast to other approaches that use bearing vectors (camera rays) as observations, we extend the common collinearity equations with a general camera model and include the relative orientation of each camera w.r.t to the fixed multi-camera system frame yielding the extended collinearity equations that directly express all image observations as functions of all unknowns. Hence, we can either calibrate the camera system, the cameras, reconstruct the observed scene, and/or simply estimate the pose of the system by including the corresponding parameter block into the Jacobian matrix. Apart from evaluating the implementation with comprehensive simulations, we benchmark our method against recently published methods for pose estimation and bundle adjustment for multi-camera systems. Finally, all methods are evaluated using a 6 degree of freedom ground truth data set, that was recorded with a lasertracker.

Keywords

Camera Calibration Accuracy Precision Geometric Robotics Bundle 

References

  1. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., et al. (2011). Building Rome in a day. Communications of the ACM, 54(10), 105–112.CrossRefGoogle Scholar
  2. Agarwal, S., & Mierle, K. (2012). Ceres solver: Tutorial & reference. Mountain View: Google Inc.Google Scholar
  3. Baker, P., & Aloimonos, Y. (2000). Complete calibration of a multi-camera network. In Proceedings IEEE Workshop on Omnidirectional Vision (pp. 134–141). IEEE.Google Scholar
  4. Barreto, J., & Daniilidis, K. (2004). Wide area multiple camera calibration and estimation of radial distortion. In Omnivis-2004, ECCV-2004 Workshop.Google Scholar
  5. Blanco-Claraco, J. L., Moreno-Dueñas, F. Á., & González-Jiménez, J. (2014). The Málaga urban dataset: High-rate stereo and LiDAR in a realistic urban scenario. The International Journal of Robotics Research, 33(2), 207–214.CrossRefGoogle Scholar
  6. Bouguet, J. (2014). Camera calibration toolbox for Matlab. Retrieved February, 2014, from http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
  7. Brown, D. C. (1966). Decentering distortion of lenses. Photometric Engineering, 32(3), 444–462.Google Scholar
  8. Brückner, M., Bajramovic, F., & Denzler, J. (2014). Intrinsic and extrinsic active self-calibration of multi-camera systems. Machine Vision and Applications, 25(2), 389–403.CrossRefGoogle Scholar
  9. Carrera, G., Angeli, A., & Davison, A.J. (2011). SLAM-based automatic extrinsic calibration of a multi-camera rig. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 2652–2659). IEEE.Google Scholar
  10. Ceriani, S., Fontana, G., Giusti, A., Marzorati, D., Matteucci, M., Migliore, D., et al. (2009). Rawseeds ground truth collection systems for indoor self-localization and mapping. Autonomous Robots, 27(4), 353–371.CrossRefGoogle Scholar
  11. Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., & Hartley, R. (2008). Robust 6dof motion estimation for non-overlapping, multi-camera systems. In IEEE Workshop on Applications of Computer Vision. WACV 2008 (pp. 1–8). IEEE.Google Scholar
  12. Davison, A.J., Cid, Y.G., Kita, N. (2004). Real-time 3d SLAM with wide-angle vision. In Proceedings of the IFAC/EURON Symposium on Intelligent Autonomous Vehicles.Google Scholar
  13. Duane, C. B. (1971). Close-range camera calibration. Photogrammetric Engineering, 37, 855–866.Google Scholar
  14. Ess, A., Neubeck, A., Van Gool, L.J. (2007). Generalised linear pose estimation. In BMVC (pp. 1–10).Google Scholar
  15. Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.MathSciNetCrossRefGoogle Scholar
  16. Förstner, W. (2010). Minimal representations for uncertainty and estimation in projective spaces. In Computer Vision–ACCV 2010 (pp. 619–632). Springer, Berlin.Google Scholar
  17. Frahm, J.M., Köser, K., & Koch, R. (2004). Pose estimation for multi-camera systems. In Pattern Recognition (pp. 286–293). Springer, Berlin.Google Scholar
  18. Frank, O., Katz, R., Tisse, C.L., & Durrant-Whyte, H.F. (2007). Camera calibration for miniature, low-cost, wide-angle imaging systems. In BMVC (pp. 1–10). Citeseer.Google Scholar
  19. Gao, X. S., Hou, X. R., Tang, J., & Cheng, H. F. (2003). Complete solution classification for the perspective-three-point problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8), 930–943.CrossRefGoogle Scholar
  20. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012 (pp. 3354–3361). IEEEGoogle Scholar
  21. Geyer, C., Daniilidis, K. (2000). A unifying theory for central panoramic systems and practical implications. In Computer VisionECCV 2000 (pp. 445–461). Springer, Berlin.Google Scholar
  22. Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: Cambridge university press.MATHGoogle Scholar
  23. Hesch, J.A., Roumeliotis, S.I. (2011). A direct least-squares (DLS) method for PnP. In IEEE International Conference on Computer Vision (ICCV) (pp. 383–390). IEEE.Google Scholar
  24. Kannala, J., & Brandt, S. S. (2006). A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1335–1340.CrossRefGoogle Scholar
  25. Kneip, L., & Furgale, P. (2014). OpenGV: A unified and generalized approach to real-time calibrated geometric vision. InIEEE International Conference on Robotics and Automation (ICRA) (pp. 1–8). IEEE.Google Scholar
  26. Kneip, L., Furgale, P., & Siegwart, R. (2013) Using multi-camera systems in robotics: Efficient solutions to the NPnP problem. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3770–3776). IEEE.Google Scholar
  27. Kneip, L., Li, H., & Seo, Y. (2014). UPnP: An optimal O(n) solution to the absolute pose problem with universal applicability. In Computer Vision–ECCV (pp. 127–142). Springer, Berlin.Google Scholar
  28. Kneip, L., Scaramuzza, D., & Siegwart, R. (2011). A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2969–2976). IEEE.Google Scholar
  29. Kumar, R.K., Ilie, A., Frahm. J.M., & Pollefeys, M. (2008). Simple calibration of non-overlapping cameras with a mirror. In IEEE Conference on, Computer Vision and Pattern Recognition. CVPR 2008 (pp. 1–7). IEEEGoogle Scholar
  30. Kummerle, R., Grisetti, G., Strasdat, H., Konolige, K., & Burgard, W. (2011). \(g^2o\): A general framework for graph optimization. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3607–3613). IEEEGoogle Scholar
  31. Kurillo, G., Li, Z., & Bajcsy, R. (2008). Wide-area external multi-camera calibration using vision graphs and virtual calibration object. In Second ACM/IEEE International Conference on In: Distributed Smart Cameras, ICDSC 2008 (pp. 1–9). IEEEGoogle Scholar
  32. Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 81(2), 155–166.CrossRefGoogle Scholar
  33. Li, B., Heng, L., Koser, K., & Pollefeys M. (2013). A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern. iN IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013 (pp. 1301–1307). IEEEGoogle Scholar
  34. Lourakis, M. I., & Argyros, A. A. (2009). SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software (TOMS), 36(1), 2.MathSciNetCrossRefGoogle Scholar
  35. Ly, D. S., Demonceaux, C., Vasseur, P., & Pégard, C. (2014). Extrinsic calibration of heterogeneous cameras by line images. Machine Vision and Applications, 25(6), 1601–1614.CrossRefGoogle Scholar
  36. Maas, H. G. (1999). Image sequence based automatic multi-camera system calibration techniques. ISPRS Journal of Photogrammetry and Remote Sensing, 54(5), 352–359.CrossRefGoogle Scholar
  37. Mei, C., & Rives, P. (2007). Single view point omnidirectional camera calibration from planar grids. In IEEE International Conference on Robotics and Automation Google Scholar
  38. Micušık, B. (2004). Two-view geometry of omnidirectional cameras. Ph.D. Thesis, Czech Technical University.Google Scholar
  39. Muhle, D., Abraham, S., Heipke, C., & Wiggenhagen, M. (2011). Estimating the mutual orientation in a multi-camera system with a non overlapping field of view. In Photogrammetric Image Analysis (pp. 13–24). Springer, BerlinGoogle Scholar
  40. Puig, L., Bermúdez, J., Sturm, P., & Guerrero, J. (2012). Calibration of omnidirectional cameras in practice: A comparison of methods. Computer Vision and Image Understanding, 116(1), 120–137. doi:10.1016/j.cviu.2011.08.003. http://www.sciencedirect.com/science/article/pii/S1077314211001858.
  41. Scaramuzza, D. (2014). OCamCalib: Omnidirectional camera calibration toolbox for matlab. Retrieved February, 2014, from https://sites.google.com/site/scarabotix/ocamcalib-toolbox
  42. Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006a). A flexible technique for accurate omnidirectional camera calibration and structure from motion. In IEEE International Conference on Computer Vision Systems, 2006 ICVS’06 (pp. 45–45). IEEEGoogle Scholar
  43. Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006b). A toolbox for easily calibrating omnidirectional cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5695–5701). IEEEGoogle Scholar
  44. Schönbein, M., & Geiger, A. (2014) Omnidirectional 3d reconstruction in augmented manhattan worlds. In International Conference on Intelligent Robots and Systems (IROS).Google Scholar
  45. Schneider, J., & Förstner, W. (2013). Bundle adjustment and system calibration with points at infinity for omnidirectional camera systems. Z. f. Photogrammetrie, Fernerkundung und Geoinformation, 4, 309–321. doi:10.1127/1432-8364/2013/0179.CrossRefGoogle Scholar
  46. Schneider, J., Schindler, F., Läbe, T., & Förstner, W. (2012). Bundle adjustment for multi-camera systems with points at infinity. In XXII ISPRS congress–technical commission III. ISPRS annals of the photogrammetry, remote sensing and spatial information sciences, vol. I-3 (pp. 75–80). Melbourne.Google Scholar
  47. Schönbein, M., Straus, T., & Geiger, A. (2014). Calibrating and centering quasi-central catadioptric cameras. In IEEE International Conference on Robotics and Automation (ICRA), 2014 (pp. 4443–4450). IEEE.Google Scholar
  48. Schweighofer, G., & Pinz, A. (2008). Globally optimal O(n) solution to the PnP problem for general camera models. In: BMVC (pp. 1–10).Google Scholar
  49. Sturm, J., Engelhard, N., Endres, F., Burgard, W., & Cremers, D. (2012). A benchmark for the evaluation of RGB-D SLAM systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012 (pp. 573–580). IEEE.Google Scholar
  50. Svoboda, T. (2003). Quick guide to multi-camera self-calibration. ETH, Swiss Federal Institute of Technology, Zurich. Technical Report BiWi-TR-263, http://www.vision.ee.ethz.ch/svoboda/SelfCal.
  51. Triggs, B., McLauchlan, P.F., Hartley, R.I., & Fitzgibbon, A.W. (2000). Bundle adjustment-a modern synthesis. In: Vision Algorithms: Theory and Practice (pp. 298–372). Springer, Berlin.Google Scholar
  52. Urban, S., Leitloff, J., & Hinz, S. (2015). Improved wide-angle, fisheye and omnidirectional camera calibration. ISPRS Journal of Photogrammetry and Remote Sensing, 108, 72–79.CrossRefGoogle Scholar
  53. Wu, C. (2011). VisualSFM: A visual structure from motion system (Vol. 9). http://homes.cs.washington.edu/~ccwu/vsfm.
  54. Zhang, Z. (1999). Flexible camera calibration by viewing a plane from unknown orientations. In The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1 (pp. 666–673). IEEE.Google Scholar
  55. Zheng, Y., Kuang, Y., Sugimoto, S., Astrom, K., & Okutomi, M. (2013). Revisiting the PnP problem: A fast, general and optimal solution. In IEEE International Conference on Computer Vision (ICCV) (pp. 2344–2351). IEEE.Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Steffen Urban
    • 1
  • Sven Wursthorn
    • 1
  • Jens Leitloff
    • 1
  • Stefan Hinz
    • 1
  1. 1.KIT - Karlsruhe Institute of Technology, Institute of Photogrammetry and Remote SensingKarlsruheGermany

Personalised recommendations