Relative Pose Estimation and Fusion of Omnidirectional and Lidar Cameras

  • Levente Tamas
  • Robert Frohlich
  • Zoltan KatoEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8926)


This paper presents a novel approach for the extrinsic parameter estimation of omnidirectional cameras with respect to a 3D Lidar coordinate frame. The method works without specific setup and calibration targets, using only a pair of 2D-3D data. Pose estimation is formulated as a 2D-3D nonlinear shape registration task which is solved without point correspondences or complex similarity metrics. It relies on a set of corresponding regions, and pose parameters are obtained by solving a small system of nonlinear equations. The efficiency and robustness of the proposed method was confirmed on both synthetic and real data in urban environment.


Omnidirectional camera Lidar Pose estimation Fusion 


  1. 1.
    Baker, S., Nayar, S.K.: A Theory of Single-Viewpoint Catadioptric Image Formation. International Journal of Computer Vision 35(2), 175–196 (1999)CrossRefGoogle Scholar
  2. 2.
    Domokos, C., Nemeth, J., Kato, Z.: Nonlinear Shape Registration without Correspondences. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(5), 943–958 (2012)CrossRefGoogle Scholar
  3. 3.
    Frohlich, R., Tamas, L., Kato, Z.: Homography Estimation between Omnidirectional Cameras without Point Correspondences. In: ICRA Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras. Hong Kong, June 2014Google Scholar
  4. 4.
    Furgale, P.T., Schwesinger, U., Rufli, M., Derendarz, W., Grimmett, H., Muehlfellner, P., Wonneberger, S., Timpner, J., Rottmann, S., Li, B., Schmidt, B., Nguyen, T.N., Cardarelli, E., Cattani, S., Bruning, S., Horstmann, S., Stellmacher, M., Mielenz, H., Köser, K., Beermann, M., Hane, C., Heng, L., Lee, G.H., Fraundorfer, F., Iser, R., Triebel, R., Posner, I., Newman, P., Wolf, L.C., Pollefeys, M., Brosig, S., Effertz, J., Pradalier, C., Siegwart, R.: Toward automated driving in cities using close-to-market sensors: An overview of the V-Charge Project. In: Intelligent Vehicles Symposium, Gold Coast City, Australia, pp. 809–816, June 2013Google Scholar
  5. 5.
    Geiger, A., Lauer, M., Wojek, C., Stiller, C., Urtasun, R.: 3D Traffic Scene Understanding From Movable Platforms. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(5), 1012–1025 (2014)CrossRefGoogle Scholar
  6. 6.
    Geyer, C., Daniilidis, K.: A unifying theory for central panoramic systems. In: European Conference on Computer Vision, Dublin, Ireland, pp. 445–462, June 2000Google Scholar
  7. 7.
    Goron, L., Tamas, L., Reti, I., Lazea, G.: 3D laser scanning system and 3D segmentation of urban scenes. In: Automation Quality and Testing Robotics Conference, vol. 1, pp. 81–85. IEEE, Cluj-Napoca, May 2010Google Scholar
  8. 8.
    Ioannou, Y., Taati, B., Harrap, R., Greenspan, M.: Difference of Normals as a Multi-scale Operator in Unorganized Point Clouds. International Conference on 3D Imaging. Modeling, Processing, Visualization and Transmission, pp. 501–508. IEEE Computer Society, Los Alamitos, CA, USA (September (2012)Google Scholar
  9. 9.
    Józsa, O., Börcs, A., Benedek, C.: Towards 4D virtual city reconstruction from Lidar point cloud sequences. In: ISPRS Workshop on 3D Virtual City Modeling, ISPRS Annals of Photogrammetry, Remote Sensing and the Spatial Information Sciences, vol. II-3/W1, pp. 15–20. Regina, Canada (2013)Google Scholar
  10. 10.
    Kannala, J., Brandt, S.S.: A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(8), 1335–1340 (2006)CrossRefGoogle Scholar
  11. 11.
    Lin, D., Fidler, S., Urtasun, R.: Holistic Scene Understanding for 3D Object Detection with RGBD Cameras. In: International Conference on Computer Vision. pp. 1417–1424. IEEE Computer Society, Sydney, December 2013Google Scholar
  12. 12.
    Mei, C., Rives, P.: Single View Point Omnidirectional Camera Calibration from Planar Grids. In: International Conference on Robotics and Automation, Roma, Italy, pp. 3945–3950, April 2007Google Scholar
  13. 13.
    Mičušík, B.: Two-View Geometry of Omnidirectional Cameras. Phd thesis, Department of Cybernetics, Faculty of Electrical Engineering, Czech Technical University, Prague, Czech Republic, June 2004Google Scholar
  14. 14.
    Mičušík, B., Pajdla, T.: Para-catadioptric Camera Auto-calibration from Epipolar Geometry. In: Asian Conference on Computer Vision, Seoul, Korea South, vol. 2, pp. 748–753, January 2004Google Scholar
  15. 15.
    Mirzaei, F.M., Kottas, D.G., Roumeliotis, S.I.: 3D LIDAR-camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization. International Journal of Robotics Research 31(4), 452–467 (2012)CrossRefGoogle Scholar
  16. 16.
    Mitra, J., Kato, Z., Marti, R., Oliver, A., Lladó, X., Sidibé, D., Ghose, S., Vilanova, J.C., Comet, J., Meriaudeau, F.: A Spline-based Non-linear Diffeomorphism for Multimodal Prostate Registration. Medical Image Analysis 16(6), 1259–1279 (2012)CrossRefGoogle Scholar
  17. 17.
    Nayar, S.K.: Catadioptric Omnidirectional Camera. In: Conference on Computer Vision and Pattern Recognition. pp. 482–488. Washington, USA, June 1997Google Scholar
  18. 18.
    Nurunnabi, A., Belton, D., West, G.: Robust Segmentation in Laser Scanning 3D Point Cloud Data. In: Digital Image Computing Techniques and Applications, Fremantle, Australia, pp. 1–8, December 2012Google Scholar
  19. 19.
    Pandey, G., McBride, J.R., Savarese, S., Eustice, R.M.: Automatic Targetless Extrinsic Calibration of a 3D Lidar and Camera by Maximizing Mutual Information. In: AAAI National Conference on Artificial Intelligence, Toronto, Canada, pp. 2053–2059, July 2012Google Scholar
  20. 20.
    Preetha, M., Suresh, L., Bosco, M.: Image Segmentation using Seeded Region Growing. Computing. In: Electronics and Electrical Technologies, Nagercoil, India, pp. 576–583, March 2012Google Scholar
  21. 21.
    Santa, Z., Kato, Z.: Correspondence-Less Non-rigid Registration of Triangular Surface Meshes. In: Conference on Computer Vision and Pattern Recognition, pp. 2275–2282. IEEE Computer Society, Portland, June 2013Google Scholar
  22. 22.
    Scaramuzza, D., Harati, A., Siegwart, R.: Extrinsic Self Calibration of a Camera and a 3D Laser Range Finder from Natural Scenes. In: International Conference on Intelligent Robots and Systems, San Diego, USA, pp. 4164–4169, October 2007Google Scholar
  23. 23.
    Scaramuzza, D., Martinelli, A., Siegwart, R.: A Flexible Technique for Accurate Omnidirectional Camera Calibration and Structure from Motion. In: International Conference on Computer Vision Systems, Washington, USA, pp. 45–51, January 2006Google Scholar
  24. 24.
    Scaramuzza, D., Martinelli, A., Siegwart, R.: A Toolbox for Easily Calibrating Omnidirectional Cameras. In: International Conference on Intelligent Robots and Systems, Bejing, China, pp. 5695–5701, October 2006Google Scholar
  25. 25.
    Schoenbein, M., Strauss, T., Geiger, A.: Calibrating and Centering Quasi-Central Catadioptric Cameras. In: International Conference on Robotics and Automation, Hong-Kong, China, pp. 1253–1256, June 2014Google Scholar
  26. 26.
    Tamas, L., Kato, Z.: Targetless Calibration of a Lidar - Perspective Camera Pair. International Conference on Computer Vision. Bigdata3dcv Workshops, Australia, Sydney, pp. 668–675, December (2013)Google Scholar
  27. 27.
    Tamas, L., Majdik, A.: Heterogeneous Feature Based Correspondence Estimation. In: Multisensor Fusion and Integration for Intelligent Systems, pp. 89–94. IEEE, Hamburg, June 2012Google Scholar
  28. 28.
    Taylor, Z., Nieto, J.: A Mutual Information Approach to Automatic Calibration of Camera and Lidar in Natural Environments. In: Australian Conference on Robotics and Automation, Wellington, Australia, pp. 3–8, December 2012Google Scholar
  29. 29.
    Vantaram, S.R., Saber, E.: Survey of Contemporary Trends in Color Image Segmentation. Journal of Electronic Imaging 21(4), 1–28 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Robotics Research GroupTechnical University of Cluj-NapocaCluj-NapocaRomania
  2. 2.Institute of InformaticsUniversity of SzegedSzegedHungary

Personalised recommendations