Skip to main content
SpringerLink
Log in

Automatic targetless LiDAR–camera calibration: a survey

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

The recent trend of fusing complementary data from LiDARs and cameras for more accurate perception has made the extrinsic calibration between the two sensors critically important. Indeed, to align the sensors spatially for proper data fusion, the calibration process usually involves estimating the extrinsic parameters between them. Traditional LiDAR–camera calibration methods often depend on explicit targets or human intervention, which can be prohibitively expensive and cumbersome. Recognizing these weaknesses, recent methods usually adopt the autonomic targetless calibration approach, which can be conducted at a much lower cost. This paper presents a thorough review of these automatic targetless LiDAR–camera calibration methods. Specifically, based on how the potential cues in the environment are retrieved and utilized in the calibration process, we divide the methods into four categories: information theory based, feature based, ego-motion based, and learning based methods. For each category, we provide an in-depth overview with insights we have gathered, hoping to serve as a potential guidance for researchers in the related fields.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Alba M, Barazzetti L, Scaioni M, Remondino F (2012) Automatic registration of multiple laser scans using panoramic RGB and intensity images. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVIII–5/W12:49–54. https://doi.org/10.5194/isprsarchives-xxxviii-5-w12-49-2011

  • Bai Z, Jiang G, Xu A (2020) LiDAR-camera calibration using line correspondences. Sensors 20(21):6319. https://doi.org/10.3390/s20216319

    Article  Google Scholar 

  • Banerjee K, Notz D, Windelen J, Gavarraju S, He M (2018) Online camera LiDAR fusion and object detection on hybrid data for autonomous driving. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2018.8500699

  • Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148. https://doi.org/10.1093/imanum/8.1.141

    Article  MathSciNet  MATH  Google Scholar 

  • Bay H, Tuytelaars T, Gool LV (2006) SURF: speeded up robust features. In: Computer vision – ECCV 2006. Springer, Berlin, pp 404–417. https://doi.org/10.1007/11744023_32

  • Belghazi MI, Baratin A, Rajeshwar S, Ozair S, Bengio Y, Courville A, Hjelm D (2018) Mutual information neural estimation. In: International conference on machine learning. PMLR, pp 531–540

  • Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Schenker PS (ed) Sensor fusion IV: control paradigms and data structures. SPIE. https://doi.org/10.1117/12.57955

  • Bileschi S (2009) Fully automatic calibration of LIDAR and video streams from a vehicle. In: 2009 IEEE 12th international conference on computer vision workshops, ICCV workshops. IEEE. https://doi.org/10.1109/iccvw.2009.5457439

  • Blaga B-C-Z, Nedevschi S (2017) Online cross-calibration of camera and LIDAR. In: 2017 13th IEEE international conference on intelligent computer communication and processing (ICCP). IEEE. https://doi.org/10.1109/iccp.2017.8117020

  • Böhm J, Becker S (2007) Automatic marker-free registration of terrestrial laser scans using reflectance. In: Proceedings of the 8th conference on optical 3D measurement techniques, Zurich, Switzerland, pp 9–12

  • CANNY J (1987) A computational approach to edge detection, pp 184–203. https://doi.org/10.1016/b978-0-08-051581-6.50024-6

  • Castorena J, Kamilov US, Boufounos PT (2016) Autocalibration of lidar and optical cameras via edge alignment. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. https://doi.org/10.1109/icassp.2016.7472200

  • Castorena J, Puskorius GV, Pandey G (2020) Motion guided LiDAR-camera self-calibration and accelerated depth upsampling for autonomous vehicles. J Intell Robot Syst 100(3–4):1129–1138. https://doi.org/10.1007/s10846-020-01233-w

    Article  Google Scholar 

  • Chen X, Ma H, Wan J, Li B, Xia T (2017) Multi-view 3d object detection network for autonomous driving. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2017.691

  • Chien H-J, Klette R, Schneider N, Franke U (2016) Visual odometry driven online calibration for monocular LiDAR-camera systems. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE. https://doi.org/10.1109/icpr.2016.7900068

  • Chou JCK, Kamel M (1991) Finding the position and orientation of a sensor on a robot manipulator using quaternions. Int J Robot Res 10(3):240–254. https://doi.org/10.1177/027836499101000305

    Article  Google Scholar 

  • Corsini M, Dellepiane M, Ganovelli F, Gherardi R, Fusiello A, Scopigno R (2012) Fully automatic registration of image sets on approximate geometry. Int J Comput Vis 102(1–3):91–111. https://doi.org/10.1007/s11263-012-0552-5

    Article  Google Scholar 

  • Cortinhal T, Tzelepis G, Aksoy EE (2020) SalsaNext: fast, uncertainty-aware semantic segmentation of LiDAR point clouds. In: Advances in visual computing. Springer, pp 207–222. https://doi.org/10.1007/978-3-030-64559-5_16

  • Cui Y, Chen R, Chu W, Chen L, Tian D, Li Y, Cao D (2022) Deep learning for image and point cloud fusion in autonomous driving: a review. IEEE Trans Intell Transp Syst 23(2):722–739. https://doi.org/10.1109/tits.2020.3023541

    Article  Google Scholar 

  • Dhall A, Chelani K, Radhakrishnan V, Krishna KM (2017) Lidar-camera calibration using 3d-3d point correspondences. arXiv preprint arXiv:1705.09785

  • Edelsbrunner H, Kirkpatrick D, Seidel R (1983) On the shape of a set of points in the plane. IEEE Trans Inf Theory 29(4):551–559. https://doi.org/10.1109/tit.1983.1056714

    Article  MathSciNet  MATH  Google Scholar 

  • Feng D, Haase-Schutz C, Rosenbaum L, Hertlein H, Glaser C, Timm F, Wiesbeck W, Dietmayer K (2021) Deep multi-modal object detection and semantic segmentation for autonomous driving: datasets, methods, and challenges. IEEE Trans Intell Transp Syst 22(3):1341–1360. https://doi.org/10.1109/tits.2020.2972974

    Article  Google Scholar 

  • Förstner W, Gülch E (1987) A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proc. ISPRS intercommission conference on fast processing of photogrammetric data, vol 6. Interlaken, pp 281–305

  • Geiger A, Moosmann F, Car O, Schuster B (2012) Automatic camera and range sensor calibration using a single shot. In: 2012 IEEE international conference on robotics and automation. IEEE. https://doi.org/10.1109/icra.2012.6224570

  • Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the KITTI dataset. Int J Robot Res 32(11):1231–1237. https://doi.org/10.1177/0278364913491297

    Article  Google Scholar 

  • González-Aguilera D, Rodríguez-Gonzálvez P, Gómez-Lahoz J (2009) An automatic procedure for co-registration of terrestrial laser scanners and digital cameras. ISPRS J Photogramm Remote Sens 64(3):308–316. https://doi.org/10.1016/j.isprsjprs.2008.10.002

    Article  Google Scholar 

  • Graeter J, Wilczynski A, Lauer M (2018) LIMO: Lidar-monocular visual odometry. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8594394

  • Guindel C, Beltran J, Martin D, Garcia F (2017) Automatic extrinsic calibration for lidar-stereo vehicle sensor setups. In: 2017 IEEE 20th international conference on intelligent transportation systems (ITSC). IEEE. https://doi.org/10.1109/itsc.2017.8317829

  • Guislain M, Digne J, Chaine R, Monnier G (2017) Fine scale image registration in large-scale urban LIDAR point sets. Comput Vis Image Underst 157:90–102. https://doi.org/10.1016/j.cviu.2016.12.004

    Article  Google Scholar 

  • Hassanein M, Moussa A, El-Sheimy N (2016) A new automatic system calibration of multi-cameras and lidar sensors. ISPRS XLI–B1:589–594. https://doi.org/10.5194/isprs-archives-xli-b1-589-2016

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2016.90

  • Hofmann S, Eggert D, Brenner C (2014) Skyline matching based camera orientation from images and mobile mapping point clouds. ISPRS Ann Photogramm Remote Sens Spat Inf Sci II–5:181–188. https://doi.org/10.5194/isprsannals-ii-5-181-2014

  • Horn M, Wodtko T, Buchholz M, Dietmayer K (2021) Online extrinsic calibration based on per-sensor ego-motion using dual quaternions. IEEE Robot Autom Lett 6(2):982–989. https://doi.org/10.1109/lra.2021.3056352

    Article  Google Scholar 

  • Hsu C-M, Wang H-T, Tsai A, Lee C-Y (2018) Online recalibration of a camera and lidar system. In: 2018 IEEE international conference on systems, man, and cybernetics (SMC). IEEE. https://doi.org/10.1109/smc.2018.00687

  • Huang K, Stachniss C (2017) Extrinsic multi-sensor calibration for mobile robots using the gauss-helmert model. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2017.8205952

  • Hussein A, Marin-Plaza P, Martin D, de la Escalera A, Armingol JM (2016) Autonomous off-road navigation using stereo-vision and laser-rangefinder fusion for outdoor obstacles detection. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2016.7535372

  • Igelbrink F, Wiemann T, Pütz S, Hertzberg J (2018) Markerless ad-hoc calibration of a hyperspectral camera and a 3d laser scanner. In: Intelligent autonomous systems, vol 15. Springer, pp 748–759. https://doi.org/10.1007/978-3-030-01370-7_58

  • Irie K, Sugiyama M, Tomono M (2016) Target-less camera-LiDAR extrinsic calibration using a bagged dependence estimator. In: 2016 IEEE international conference on automation science and engineering (CASE). IEEE. https://doi.org/10.1109/coase.2016.7743564

  • Ishikawa R, Oishi T, Ikeuchi K (2018) LiDAR and camera calibration using motions estimated by sensor fusion odometry. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8593360

  • Iyer G, Ram RK, Murthy JK, Krishna KM (2018) CalibNet: geometrically supervised extrinsic calibration using 3d spatial transformer networks. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros.2018.8593693

  • Jiang J, Xue P, Chen S, Liu Z, Zhang X, Zheng N (2018) Line feature based extrinsic calibration of LiDAR and camera. In: 2018 IEEE international conference on vehicular electronics and safety (ICVES). IEEE. https://doi.org/10.1109/icves.2018.8519493

  • Jiang P, Osteen P, Saripalli S (2021) SemCal: semantic LiDAR-camera calibration using neural mutual information estimator. In: 2021 IEEE international conference on multisensor fusion and integration for intelligent systems (MFI). IEEE. https://doi.org/10.1109/mfi52462.2021.9591203

  • Jing X, Ding X, Xiong R, Deng H, Wang Y (2022) DXQ-Net: differentiable lidar-camera extrinsic calibration using quality-aware flow. arXiv preprint arXiv:2203.09385

  • Kelley CT (1999) Iterative methods for optimization. SIAM

  • Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of ICNN’95-international conference on neural networks, vol 4. IEEE, pp 1942–1948

  • Khurana A, Nagla KS (2021) Extrinsic calibration methods for laser range finder and camera: a systematic review. Mapan 36(3):669–690. https://doi.org/10.1007/s12647-021-00500-x

    Article  Google Scholar 

  • Kim A, Osep A, Leal-Taixe L (2021) EagerMOT: 3d multi-object tracking via sensor fusion. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra48506.2021.9562072

  • Krähenbühl P, Koltun V (2011) Efficient inference in fully connected crfs with Gaussian edge potentials. Advances in neural information processing systems 24

  • Levenberg K (1944) A method for the solution of certain non-linear problems in least squares. Q Appl Math 2(2):164–168. https://doi.org/10.1090/qam/10666

    Article  MathSciNet  MATH  Google Scholar 

  • Levinson J, Thrun S (2013) Automatic online calibration of cameras and lasers. In: Robotics: science and systems IX. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2013.ix.029

  • Li M, Chen X, Li X, Ma B, Vitányi PM (2004) The similarity metric. IEEE Trans Inf Theory 50(12):3250–3264

    Article  MathSciNet  MATH  Google Scholar 

  • Li T, Fang J, Zhong Y, Wang D, Xue J (2017) Online high-accurate calibration of rgb+ 3d-lidar for autonomous driving. In: Lecture notes in computer science. Springer, pp 254–263. https://doi.org/10.1007/978-3-319-71598-8_23

  • Li J, Yang B, Chen C, Huang R, Dong Z, Xiao W (2018) Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features. ISPRS J Photogramm Remote Sens 136:41–57. https://doi.org/10.1016/j.isprsjprs.2017.12.005

    Article  Google Scholar 

  • Liao Q, Liu M (2019) Extrinsic calibration of 3d range finder and camera without auxiliary object or human intervention. In: 2019 IEEE international conference on real-time computing and robotics (RCAR). IEEE. https://doi.org/10.1109/rcar47638.2019.9044146

  • Li-Chee-Ming J, Armenakis C, Fusion of optical and terrestrial laser scanner data. In: The (2010) Canadian geomatics conference and symposium of commission I. ISPRS Convergence in Geomatics-Shaping Canada’s Competitive Landscape, Citeseer, p 2010

  • Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400

  • Liu X, Deng Z, Yang Y (2018) Recent progress in semantic image segmentation. Artif Intell Rev 52(2):1089–1106. https://doi.org/10.1007/s10462-018-9641-3

    Article  Google Scholar 

  • Liu H, Liu Y, Gu X, Wu Y, Qu F, Huang L (2018) A deep-learning based multi-modality sensor calibration method for USV. In: 2018 IEEE fourth international conference on multimedia big data (BigMM). IEEE. https://doi.org/10.1109/bigmm.2018.8499349

  • Liu X, Yuan C, Zhang F (2021) Fast and accurate extrinsic calibration for multiple lidars and cameras. arXiv preprint arXiv:2109.06550

  • Lowe D (1999) Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision. IEEE. https://doi.org/10.1109/iccv.1999.790410

  • Lu X, Liu Y, Li K (2019) Fast 3d line segment detection from unorganized point cloud. arXiv preprint arXiv:1901.02532

  • Lv X, Wang S, Ye D (2021a) CFNet: LiDAR-camera registration using calibration flow network. Sensors 21(23):8112. https://doi.org/10.3390/s21238112

    Article  Google Scholar 

  • Lv X, Wang B, Dou Z, Ye D, Wang S (2021b) LCCNet: LiDAR and camera self-calibration using cost volume network. In: 2021 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE. https://doi.org/10.1109/cvprw53098.2021.00324

  • Ma H, Liu K, Liu J, Qiu H, Xu D, Wang Z, Gong X, Yang S (2021a) Simple and efficient registration of 3d point cloud and image data for an indoor mobile mapping system. JOSA A 38(4):579–586. https://doi.org/10.1364/josaa.414042

    Article  Google Scholar 

  • Ma T, Liu Z, Yan G, Li Y (2021b) Crlf: automatic calibration and refinement based on line feature for lidar and camera in road scenes. arXiv preprint arXiv:2103.04558

  • Miled M, Soheilian B, Habets E, Vallet B (2016) Hybrid online mobile laser scanner calibration through image alignment by mutual information. ISPRS Ann Photogramm Remote Sens Spat Inf Sci III–1:25–31. https://doi.org/10.5194/isprsannals-iii-1-25-2016

  • Morel J-M, Yu G (2009) ASIFT: a new framework for fully affine invariant image comparison. SIAM J Imag Sci 2(2):438–469. https://doi.org/10.1137/080732730

    Article  MathSciNet  MATH  Google Scholar 

  • Moussa W, Abdel-Wahab M, Fritsch D (2012) Automatic fusion of digital images and laser scanner data for heritage preservation. In: Progress in cultural heritage preservation. Springer, Berlin, pp 76–85. https://doi.org/10.1007/978-3-642-34234-9_8

  • Munoz-Banon MA, Candelas FA, Torres F (2020) Targetless camera-LiDAR calibration in unstructured environments. IEEE Access 8:143692–143705. https://doi.org/10.1109/access.2020.3014121

    Article  Google Scholar 

  • Mur-Artal R, Montiel JMM, Tardos JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Rob 31(5):1147–1163. https://doi.org/10.1109/tro.2015.2463671

    Article  Google Scholar 

  • Nagy B, Benedek C (2020) On-the-fly camera and lidar calibration. Remote Sens 12(7):1137. https://doi.org/10.3390/rs12071137

    Article  Google Scholar 

  • Nagy B, Kovacs L, Benedek C (2019a) Online targetless end-to-end camera-LIDAR self-calibration. In: 2019 16th international conference on machine vision applications (MVA). IEEE. https://doi.org/10.23919/mva.2019.8757887

  • Nagy B, Kovacs L, Benedek C (2019b) SFM and semantic information based online targetless camera-LIDAR self-calibration. In: 2019 IEEE international conference on image processing (ICIP). IEEE. https://doi.org/10.1109/icip.2019.8804299

  • Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7(4):308–313. https://doi.org/10.1093/comjnl/7.4.308

    Article  MathSciNet  MATH  Google Scholar 

  • Nie J, Pan F, Xue D, Luo L (2021) A survey of extrinsic parameters calibration techniques for autonomous devices. In: 2021 33rd Chinese control and decision conference (CCDC). IEEE. https://doi.org/10.1109/ccdc52312.2021.9602601

  • Nieto JI, Monteiro ST, Viejo D (2010) 3d geological modelling using laser and hyperspectral data. In: 2010 IEEE international geoscience and remote sensing symposium. IEEE. https://doi.org/10.1109/igarss.2010.5651553

  • Nurunnabi A, Belton D, West G (2012) Robust segmentation in laser scanning 3d point cloud data. In: 2012 international conference on digital image computing techniques and applications (DICTA). IEEE. https://doi.org/10.1109/dicta.2012.6411672

  • Oishi T, Nakazawa A, Kurazume R, Ikeuchi K (2005) Fast simultaneous alignment of multiple range images using index images. In: Fifth international conference on 3-D digital imaging and modeling (3DIM’05). IEEE. https://doi.org/10.1109/3dim.2005.41

  • Pandey G, McBride JR, Savarese S, Eustice RM (2012) Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information. In: Twenty-sixth AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v26i1.8379

  • Pandey G, McBride JR, Savarese S, Eustice RM (2014) Automatic extrinsic calibration of vision and lidar by maximizing mutual information. J Field Robot 32(5):696–722. https://doi.org/10.1002/rob.21542

    Article  Google Scholar 

  • Park F, Martin B (1994) Robot sensor calibration: solving AX=XB on the euclidean group. IEEE Trans Robot Autom 10(5):717–721. https://doi.org/10.1109/70.326576

    Article  Google Scholar 

  • Park C, Moghadam P, Kim S, Sridharan S, Fookes C (2020) Spatiotemporal camera-LiDAR calibration: a targetless and structureless approach. IEEE Robot Autom Lett 5(2):1556–1563. https://doi.org/10.1109/lra.2020.2969164

    Article  Google Scholar 

  • Parmehr EG, Fraser CS, Zhang C, Leach J (2014) Automatic registration of optical imagery with 3d lidar data using statistical similarity. ISPRS J Photogramm Remote Sens 88:28–40

    Article  Google Scholar 

  • Pascoe G, Maddern W, Newman P (2015) Direct visual localisation and calibration for road vehicles in changing city environments. In: 2015 IEEE international conference on computer vision workshop (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2015.23

  • Peršić J, Petrović L, Marković I, Petrović I (2020) Online multi-sensor calibration based on moving object tracking. Adv Robot 35(3–4):130–140. https://doi.org/10.1080/01691864.2020.1819874

    Article  Google Scholar 

  • Pomerleau F, Colas F, Siegwart R, Magnenat S (2013) Comparing ICP variants on real-world data sets. Auton Robot 34(3):133–148. https://doi.org/10.1007/s10514-013-9327-2

    Article  Google Scholar 

  • Powell MJ (2009) The bobyqa algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, 26

  • Pusztai Z, Hajder L (2017) Accurate calibration of lidar-camera systems using ordinary boxes. In: 2017 IEEE international conference on computer vision workshops (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2017.53

  • Quan L, Lan Z (1999) Linear n-point camera pose determination. IEEE Trans Pattern Anal Mach Intell 21(8):774–780. https://doi.org/10.1109/34.784291

    Article  Google Scholar 

  • Scaramuzza D, Harati A, Siegwart R (2007) Extrinsic self calibration of a camera and a 3d laser range finder from natural scenes. In: 2007 IEEE/RSJ international conference on intelligent robots and systems. IEEE. https://doi.org/10.1109/iros.2007.4399276

  • Schneider N, Piewak F, Stiller C, Franke U (2017) RegNet: multimodal sensor registration using deep neural networks. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2017.7995968

  • Scott DW (1992) Multivariate density estimation: theory, practice and visualisation. Wiley, New York

    Book  Google Scholar 

  • Shannon CE (2001) A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev 5(1):3–55

    Article  MathSciNet  Google Scholar 

  • Shi C, Huang K, Yu Q, Xiao J, Lu H, Xie C (2019a) Extrinsic calibration and odometry for camera-LiDAR systems. IEEE Access 7:120106–120116. https://doi.org/10.1109/access.2019.2937909

    Article  Google Scholar 

  • Shi S, Wang X, Li H (2019b) PointRCNN: 3d object proposal generation and detection from point cloud. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2019.00086

  • Shi J, Zhu Z, Zhang J, Liu R, Wang Z, Chen S, Liu H (2020) CalibRCNN: calibrating camera and LiDAR by recurrent convolutional neural network and geometric constraints. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros45743.2020.9341147

  • Shiu Y, Ahmad S (1989) Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. IEEE Trans Robot Autom 5(1):16–29. https://doi.org/10.1109/70.88014

    Article  Google Scholar 

  • Sobel I, Duda R, Hart P Sobel-feldman operator

  • Studholme C, Hill D, Hawkes D (1999) An overlap invariant entropy measure of 3d medical image alignment. Pattern Recogn 32(1):71–86. https://doi.org/10.1016/s0031-3203(98)00091-0

    Article  Google Scholar 

  • Swart A, Broere J, Veltkamp R, Tan R (2011) Refined non-rigid registration of a panoramic image sequence to a LiDAR point cloud. In: Photogrammetric image analysis. Springer, Berlin, pp 73–84. https://doi.org/10.1007/978-3-642-24393-6_7

  • Takikawa T, Acuna D, Jampani V, Fidler S (2019) Gated-SCNN: gated shape CNNs for semantic segmentation. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00533

  • Taylor Z, Nieto J (2012) A mutual information approach to automatic calibration of camera and lidar in natural environments. In: Australian conference on robotics and automation, pp 3–5

  • Taylor Z, Nieto J (2013) Automatic calibration of lidar and camera images using normalized mutual information. In: 2013 IEEE international conference on robotics and automation (ICRA). Citeseer

  • Taylor Z, Nieto J (2014) Parameterless automatic extrinsic calibration of vehicle mounted lidar-camera systems. In: International conference on robotics and automation: long term autonomy workshop, number October, pp 3–6

  • Taylor Z, Nieto J (2015) Motion-based calibration of multimodal sensor arrays. In: 2015 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra.2015.7139872

  • Taylor Z, Nieto J (2016) Motion-based calibration of multimodal sensor extrinsics and timing offset estimation. IEEE Trans Rob 32(5):1215–1229. https://doi.org/10.1109/tro.2016.2596771

    Article  Google Scholar 

  • Taylor Z, Nieto J, Johnson D (2013) Automatic calibration of multi-modal sensor systems using a gradient orientation measure. In: 2013 IEEE/RSJ international conference on intelligent robots and systems, pp 1293–1300. IEEE

  • Taylor Z, Nieto J, Johnson D (2014) Multi-modal sensor calibration using a gradient orientation measure. J Field Robot 32(5):675–695. https://doi.org/10.1002/rob.21523

    Article  Google Scholar 

  • Teed Z, Deng J (2021) RAFT: recurrent all-pairs field transforms for optical flow (extended abstract) . In: Proceedings of the thirtieth international joint conference on artificial intelligence. International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2021/662

  • Toth T, Pusztai Z, Hajder L (2020) Automatic LiDAR-camera calibration of extrinsic parameters using a spherical target. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra40945.2020.9197316

  • Ullman S (1979) The interpretation of structure from motion. Proc R Soc Lond B 203(1153):405–426. https://doi.org/10.7551/mitpress/3877.003.0009

    Article  Google Scholar 

  • Unnikrishnan R, Hebert M (2005) Fast extrinsic calibration of a laser rangefinder to a camera. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-05-09

  • Vaida A-S, Nedevschi S (2019) Automatic extrinsic calibration of LIDAR and monocular camera images. In: 2019 IEEE 15th international conference on intelligent computer communication and processing (ICCP). IEEE. https://doi.org/10.1109/iccp48234.2019.8959801

  • Vel’as M, Španěl M, Materna Z, Herout A (2014) Calibration of rgb camera with velodyne lidar

  • Vo A-V, Truong-Hong L, Laefer DF, Bertolotto M (2015) Octree-based region growing for point cloud segmentation. ISPRS J Photogramm Remote Sens 104:88–100. https://doi.org/10.1016/j.isprsjprs.2015.01.011

    Article  Google Scholar 

  • von Gioi RG, Jakubowicz J, Morel J-M, Randall G (2012) LSD: a line segment detector. Image Process On Line 2:35–55. https://doi.org/10.5201/ipol.2012.gjmr-lsd

    Article  Google Scholar 

  • Vora S, Lang AH, Helou B, Beijbom O (2020) PointPainting: sequential fusion for 3d object detection. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr42600.2020.00466

  • Wang R, Ferrie FP, Macfarlane J (2012) Automatic registration of mobile LiDAR and spherical panoramas. In: 2012 IEEE Computer Society conference on computer vision and pattern recognition workshops. IEEE. https://doi.org/10.1109/cvprw.2012.6238912

  • Wang L, Xiao Z, Zhao D, Wu T, Dai B (2018) Automatic extrinsic calibration of monocular camera and LIDAR in natural scenes. In: 2018 IEEE international conference on information and automation (ICIA). IEEE. https://doi.org/10.1109/icinfa.2018.8812555

  • Wang Z, Wu Y, Niu Q (2020a) Multi-sensor fusion in automated driving: a survey. IEEE Access 8:2847–2868. https://doi.org/10.1109/access.2019.2962554

    Article  Google Scholar 

  • Wang W, Nobuhara S, Nakamura R, Sakurada K (2020b) Soic: semantic online initialization and calibration for lidar and camera. arXiv preprint arXiv:2003.04260

  • Wang Y, Li J, Sun Y, Shi M (2021) A survey of extrinsic calibration of lidar and camera. In: International conference on autonomous unmanned systems. Springer, pp 933–944

  • Willis A, Sui Y (2009) An algebraic model for fast corner detection. In: 2009 IEEE 12th International Conference on Computer Vision. IEEE. https://doi.org/10.1109/iccv.2009.5459443

  • Xiao Z, Li H, Zhou D, Dai Y, Dai B (2017) Accurate extrinsic calibration between monocular camera and sparse 3d lidar points without markers. In: IEEE intelligent vehicles symposium (IV). IEEE. https://doi.org/10.1109/ivs.2017.7995755

  • Xu B, Jiang W, Shan J, Zhang J, Li L (2015) Investigation on the weighted RANSAC approaches for building roof plane segmentation from LiDAR point clouds. Remote Sens 8(1):5. https://doi.org/10.3390/rs8010005

    Article  Google Scholar 

  • Xu H, Lan G, Wu S, Hao Q (2019) Online intelligent calibration of cameras and LiDARs for autonomous driving systems. In: 2019 IEEE intelligent transportation systems conference (ITSC). IEEE. https://doi.org/10.1109/itsc.2019.8916872

  • Yaopeng L, Xiaojun G, Shaojing S, Bei S (2021) Review of a 3d lidar combined with single vision calibration. In: 2021 IEEE international conference on data science and computer application (ICDSCA). IEEE. https://doi.org/10.1109/icdsca53499.2021.9649726

  • Ye C, Pan H, Gao H (2022) Keypoint-based LiDAR-camera online calibration with robust geometric network. IEEE Trans Instrum Meas 71:1–11. https://doi.org/10.1109/tim.2021.3129882

    Article  Google Scholar 

  • Yoo J-S, Kim D-H, Kim G-W (2018) Improved lidar-camera calibration using marker detection based on 3d plane extraction. J Electr Eng Technol 13(6):2530–2544

    Google Scholar 

  • Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018) BiSeNet: bilateral segmentation network for real-time semantic segmentation. In: Computer vision – ECCV 2018. Springer, pp 334–349. https://doi.org/10.1007/978-3-030-01261-8_20

  • Yu H, Zhen W, Yang W, Scherer S (2020) Line-based 2-d-3-d registration and camera localization in structured environments. IEEE Trans Instrum Meas 69(11):8962–8972. https://doi.org/10.1109/tim.2020.2999137

    Article  Google Scholar 

  • Yuan K, Guo Z, Wang ZJ (2020) RGGNet: tolerance aware LiDAR-camera online calibration with geometric deep learning and generative model. IEEE Robot Autom Lett 5(4):6956–6963. https://doi.org/10.1109/lra.2020.3026958

    Article  Google Scholar 

  • Yuan C, Liu X, Hong X, Zhang F (2021) Pixel-level extrinsic self calibration of high resolution LiDAR and camera in targetless environments. IEEE Robot Autom Lett 6(4):7517–7524. https://doi.org/10.1109/lra.2021.3098923

    Article  Google Scholar 

  • Zhang Q, Pless R (2004) Extrinsic calibration of a camera and laser range finder (improves camera calibration) . In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS) (IEEE Cat. No.04CH37566). IEEE. https://doi.org/10.1109/iros.2004.1389752

  • Zhang J, Singh S (2014) LOAM: Lidar odometry and mapping in real-time. In: Robotics: science and systems X. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2014.x.007

  • Zhang X, Zhang A, Meng X (2015) Automatic fusion of hyperspectral images and laser scans using feature points. J Sens 1–9:2015. https://doi.org/10.1155/2015/415361

    Article  Google Scholar 

  • Zhang W, Zhou H, Sun S, Wang Z, Shi J, Loy CC (2019) Robust multi-modality multi-object tracking. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00245

  • Zhang X, Zhu S, Guo S, Li J, Liu H (2021) Line-based automatic extrinsic calibration of LiDAR and camera. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra48506.2021.9561216

  • Zhao Y, Wang Y, Tsai Y (2016) 2d-image to 3d-range registration in urban environments via scene categorization and combination of similarity measurements. In: 2016 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra.2016.7487332

  • Zhao G, Hu J, You S, Kuo CCJ (2021) CalibDNN: multimodal sensor calibration for perception using deep neural networks. In: Grewe LL, Blasch EP, Kadar I (eds) Signal processing, sensor/information fusion, and target recognition XXX. SPIE. https://doi.org/10.1117/12.2587994

  • Zhou L, Deng Z (2012) A new algorithm for computing the projection matrix between a LIDAR and a camera based on line correspondences. In: 2012 IV international congress on ultra modern telecommunications and control systems. IEEE. https://doi.org/10.1109/icumt.2012.6459706

  • Zhou Y, Qi H, Ma Y (2019) End-to-end wireframe parsing. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00105

  • Zhu N, Jia Y, Ji S (2018) Registration of panoramic/fish-eye image sequence and LiDAR points using skyline features. Sensors 18(5):1651. https://doi.org/10.3390/s18051651

    Article  Google Scholar 

  • Zhu Y, Sapra K, Reda FA, Shih KJ, Newsam S, Tao A, Catanzaro B (2019) Improving semantic segmentation via video propagation and label relaxation. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2019.00906

  • Zhu Y, Li C, Zhang Y (2020) Online camera-LiDAR calibration with sensor semantic information. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE. https://doi.org/10.1109/icra40945.2020.9196627

  • Zuniga-Noel D, Ruiz-Sarmiento J-R, Gomez-Ojeda R, Gonzalez-Jimenez J (2019) Automatic multi-sensor extrinsic calibration for mobile robots. IEEE Robot Autom Lett 4(3):2862–2869. https://doi.org/10.1109/lra.2019.2922618

    Article  Google Scholar 

  • Zuo X, Geneva P, Lee W, Liu Y, Huang G (2019) LIC-fusion: LiDAR-Inertial-Camera Odometry. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE. https://doi.org/10.1109/iros40897.2019.8967746

Download references

Acknowledgements

The work is partially supported by the 2030 National Key AI Program of China 2018AAA0100500, Guangdong Province R&D Program 2020B0909050001, Anhui Province Development and Reform Commission 2020 New Energy Vehicle Industry Innovation Development Project and 2021 New Energy and Intelligent Connected Vehicle Innovation Project, CAAI-Huawei MindSpore Open Fund, Shenzhen Yijiahe Technology R&D Co., Ltd., and Huawei Cloud Computing Technologies Co., Ltd.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Science and Technology of China, Jinzhai Road, Hefei, 230026, Anhui, China

    Xingchen Li, Yuxuan Xiao, Haojie Ren, Yanyong Zhang & Jianmin Ji

  2. Department of Fundamental Research, Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, West Wangjiang Road, Hefei, 230088, Anhui, China

    Beibei Wang

Authors
  1. Xingchen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxuan Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Beibei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haojie Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanyong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jianmin Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Had the idea for the article: XL, JJ; Performed the literature search and data analysis: XL, YX; Drafted the work: XL, YX, BW, HR; Critically revised the work: BW, YZ, JJ.

Corresponding author

Correspondence to Jianmin Ji.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Xiao, Y., Wang, B. et al. Automatic targetless LiDAR–camera calibration: a survey. Artif Intell Rev 56, 9949–9987 (2023). https://doi.org/10.1007/s10462-022-10317-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-022-10317-y

Keywords

Search

Navigation