Skip to main content

Rethinking Generic Camera Models for Deep Single Image Camera Calibration to Recover Rotation and Fisheye Distortion

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

Although recent learning-based calibration methods can predict extrinsic and intrinsic camera parameters from a single image, the accuracy of these methods is degraded in fisheye images. This degradation is caused by mismatching between the actual projection and expected projection. To address this problem, we propose a generic camera model that has the potential to address various types of distortion. Our generic camera model is utilized for learning-based methods through a closed-form numerical calculation of the camera projection. Simultaneously to recover rotation and fisheye distortion, we propose a learning-based calibration method that uses the camera model. Furthermore, we propose a loss function that alleviates the bias of the magnitude of errors for four extrinsic and intrinsic camera parameters. Extensive experiments demonstrated that our proposed method outperformed conventional methods on two large-scale datasets and images captured by off-the-shelf fisheye cameras. Moreover, we are the first researchers to analyze the performance of learning-based methods using various types of projection for off-the-shelf cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD), pp. 2623–2631 (2019). https://doi.org/10.1145/3292500.3330701

  2. Alemán-Flores, M., Alvarez, L., Gomez, L., Santana-Cedrés, D.: Automatic lens distortion correction using one-parameter division models. Image Process. On Line (IPOL) 4, 327–343 (2014). https://doi.org/10.5201/ipol.2014.106

    Article  Google Scholar 

  3. Alhaija, H.A., Mustikovela, S.K., Mescheder, L., Geiger, A.: Augmented reality meets computer vision: efficient data generation for urban driving scenes. Int. J. Comput. Vision (IJCV) 126(9), 961–972 (2018). https://doi.org/10.1007/s11263-018-1070-x

    Article  Google Scholar 

  4. Alvarez, L., Gómez, L., Sendra, J.: An algebraic approach to lens distortion by line rectification. J. Math. Imaging Vision (JMIV) 35(1), 36–50 (2009). https://doi.org/10.1007/s10851-009-0153-2

    Article  MathSciNet  MATH  Google Scholar 

  5. Barreto, J.: A unifying geometric representation for central projection systems. Comput. Vision Image Understanding (CVIU) 103(3), 208–217 (2006). https://doi.org/10.1016/j.cviu.2006.06.003

    Article  Google Scholar 

  6. Basu, A., Licardie, S.: Modeling fish-eye lenses. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). vol. 3, pp. 1822–1828 (1993). https://doi.org/10.1109/IROS.1993.583883

  7. Benligiray, B., Topal, C.: Blind rectification of radial distortion by line straightness. In: Proceedings of European Signal Processing Conference (EUSIPCO), pp. 938–942 (2016). https://doi.org/10.1109/EUSIPCO.2016.7760386

  8. Bogdan, O., Eckstein, V., Rameau, F., Bazin, J.C.: DeepCalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras. In: Proceedings of SIGGRAPH European Conference on Visual Media Production (CVMP) (2018). https://doi.org/10.1145/3278471.3278479

  9. Brown, D.: Close-range camera calibration. Photogram. Eng. 37(8), 855–866 (1971)

    Google Scholar 

  10. Bukhari, F., Dailey, M.: Automatic radial distortion estimation from a single image. J. Math. Imaging Vision (JMIV) 45, 31–45 (2013). https://doi.org/10.1007/s10851-012-0342-2

    Article  MathSciNet  MATH  Google Scholar 

  11. Cao, Z., Simon, T., Wei, S., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302–1310 (2017). https://doi.org/10.1109/CVPR.2017.143

  12. Chang, S., et al.: Generating 360 outdoor panorama dataset with reliable sun position estimation. In: Proceedings of SIGGRAPH Asia, pp. 1–2 (2018). https://doi.org/10.1145/3283289.3283348

  13. Chao, C., Hsu, P., Lee, H., Wang, Y.: Self-supervised deep learning for fisheye image rectification. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 2248–2252 (2020). https://doi.org/10.1109/ICASSP40776.2020.9054191

  14. Chen, Y., Schmid, C., Sminchisescu, C.: Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 7062–7071 (2019). https://doi.org/10.1109/ICCV.2019.00716

  15. Dauphin, Y.N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., Bengio, Y.: Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 2, pp. 2933–2941 (2018). arXiv:1406.2572

  16. Davidson, B., Alvi, M., Henriques, J.F.: \(360^{\circ }\) Camera alignment via segmentation. In: Proceedings of European Conference on Computer Vision (ECCV), pp. 579–595 (2020). https://doi.org/10.1007/978-3-030-58604-1_35

  17. Fitzgibbon, A.: Simultaneous linear estimation of multiple view geometry and lens distortion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). vol. 1, pp. I-I (2001). https://doi.org/10.1109/CVPR.2001.990465

  18. Fu, Z., Liu, Q., Fu, Z., Wang, Y.: Template-free visual tracking with space-time memory networks. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 13774–13783 (2021). https://doi.org/10.1109/CVPR46437.2021.01356

  19. González-Aguilera, D., Gómez-Lahoz, J., Rodríguez-Gonzálvez, P.: An automatic approach for radial lens distortion correction from a single image. IEEE Sensors J. (JSEN) 11(4), 956–965 (2011). https://doi.org/10.1109/JSEN.2010.2076403

    Article  Google Scholar 

  20. Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 2, pp. 2672–2680 (2014). arXiv:1406.2661

  21. Goodfellow, I.J., Vinyals, O., Saxe, A.M.: Qualitatively characterizing neural network. In: Proceedings of International Conference on Learning Representations (ICLR), pp. 1–20 (2015). arXiv:1412.6544

  22. Gordon, A., Li, H., Jonschkowski, R., Angelova, A.: Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8976–8985 (2019). https://doi.org/10.1109/ICCV.2019.00907

  23. Groenendijk, R., Karaoglu, S., Gevers, T., Mensink, T.: Multi-loss weighting with coefficient of variations. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1468–1477 (2021). https://doi.org/10.1109/WACV48630.2021.00151

  24. Hold-Geoffroy, Y., et al.: A perceptual measure for deep single image camera calibration. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2354–2363 (2018). https://doi.org/10.1109/CVPR.2018.00250

  25. Hosono, M., Simo-Serra, E., Sonoda, T.: Self-supervised deep fisheye image rectification approach using coordinate relations. In: Proceedings of IEEE International Conference on Machine Vision and Applications (MVA), pp. 1–5 (2021). https://doi.org/10.23919/MVA51890.2021.9511349

  26. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017). https://doi.org/10.1109/CVPR.2017.243

  27. Huang, Z., Xu, Y., Shi, J., Zhou, X., Bao, H., Zhang, G.: Prior guided dropout for robust visual localization in dynamic environments. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2791–2800 (2019). https://doi.org/10.1109/ICCV.2019.00288

  28. Huber, P.J.: Robust estimation of a location parameter. Annals Math. Stat. 35(1), 73–101 (1964). https://doi.org/10.1214/aoms/1177703732

    Article  MathSciNet  MATH  Google Scholar 

  29. Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of International Conference on Machine Learning (ICML). vol. 37, pp. 448–456 (2015). arXiv:1502.03167

  30. Kannala, J., Brandt, S.S.: A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Trans. Pattern Analysis Mach. Intell. (PAMI) 28(8), 1335–1340 (2006). https://doi.org/10.1109/TPAMI.2006.153

    Article  Google Scholar 

  31. Lee, J., Go, H., Lee, H., Cho, S., Sung, M., Kim, J.: CTRL-C: Camera calibration TRansformer with line-classification. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16228–16237 (2021). https://doi.org/10.1109/ICCV48922.2021.01592

  32. Lee, J., Sung, M., Lee, H., Kim, J.: Neural geometric parser for single image camera calibration. In: Proceedings of European Conference on Computer Vision (ECCV), pp. 541–557 (2020). https://doi.org/10.1007/978-3-030-58610-2_32

  33. Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T.: Visualizing the loss landscape of neural nets. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 6391–6401 (2018). arXiv:1712.09913

  34. Li, X., Zhang, B., Sander, P.V., Liao, J.: Blind geometric distortion correction on images through deep learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4855–4864 (2019). https://doi.org/10.1109/CVPR.2019.00499

  35. Liao, K., Chunyu, L., Liao, L., Zhao, Y., Lin, W.: Multi level curriculum for training a distortion-aware barrel distortion rectification model. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4369–4378 (2021). https://doi.org/10.1109/ICCV48922.2021.00435

  36. Liao, K., Lin, C., Wei, Y., Li, F., Yang, S., Zhao, Y.: Towards complete scene and regular shape for distortion rectification. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14549–14558 (2021). https://doi.org/10.1109/ICCV48922.2021.01430

  37. Liao, K., Lin, C., Zhao, Y.: A deep ordinal distortion estimation approach for distortion rectification. IEEE Trans. Image Process. (TIP) 30, 3362–3375 (2021). https://doi.org/10.1109/TIP.2021.3061283

    Article  Google Scholar 

  38. Liao, K., Lin, C., Zhao, Y., Gabbouj, M.: DR-GAN: automatic radial distortion rectification using conditional GAN in real-time. IEEE Trans. Circ. Syst. Video Technol. (TCSVT) 30(3), 725–733 (2020). https://doi.org/10.1109/TCSVT.2019.2897984

    Article  Google Scholar 

  39. Liao, K., Lin, C., Zhao, Y., Xu, M.: Model-free distortion rectification framework bridged by distortion distribution map. IEEE Trans. Image Process. (TIP) 29, 3707–3718 (2020). https://doi.org/10.1109/TIP.2020.2964523

    Article  MATH  Google Scholar 

  40. Lin, M., Chen, Q., Yan, S.: Network in network. In: Proceedings of International Conference on Learning Representations (ICLR), pp. 1–10 (2014). arXiv:1312.4400

  41. Liu, C., et al.: Auto-DeepLab: Hierarchical neural architecture search for semantic image segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 82–92 (2019). https://doi.org/10.1109/CVPR.2019.00017

  42. Liu, L., et al.: On the variance of the adaptive learning rate and beyond. In: Proceedings of International Conference on Learning Representations (ICLR), pp. 1–14 (2020). arXiv:1908.03265

  43. Locher, A., Perdoch, M., Gool, L.V.: Progressive prioritized multi-view stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3244–3252 (2016). https://doi.org/10.1109/CVPR.2016.353

  44. Lochman, Y., Dobosevych, O., Hryniv, R., Pritts, J.: Minimal solvers for single-view lens-distorted camera auto-calibration. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 2886–2895 (2021). https://doi.org/10.1109/WACV48630.2021.00293

  45. López-Antequera, M., Marí, R., Gargallo, P., Kuang, Y., Gonzalez-Jimenez, J., Haro, G.: Deep single image camera calibration with radial distortion. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11809–11817 (2019). https://doi.org/10.1109/CVPR.2019.01209

  46. Mirowski, P., et al.: The StreetLearn environment and dataset. arXiv preprint arXiv:1903.01292 (2019)

  47. Misra, D.: Mish: A self regularized non-monotonic neural activation function. In: Proceedings of British Machine Vision Conference (BMVC), pp. 1–14 (2020). arXiv:1908.08681

  48. Nie, Y., Han, X., Guo, S., Zheng, Y., Chang, J., Zhang, J.J.: Total3DUnderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 52–61 (2020). https://doi.org/10.1109/CVPR42600.2020.00013.

  49. Pritts, J., Kukelova, Z., Larsson, V., Chum, O.: Radially-distorted conjugate translations. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1993–2001 (2018). https://doi.org/10.1109/CVPR.2018.00213

  50. Puskorius, G., Feldkamp, L.: Camera calibration methodology based on a linear perspective transformation error model. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). vol. 3, pp. 1858–1860 (1988). https://doi.org/10.1109/ROBOT.1988.12340

  51. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You Only Look Once: Unified, real-time object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788 (2016). https://doi.org/10.1109/CVPR.2016.91

  52. Ren, L., Song, Y., Lu, J., Zhou, J.: Spatial geometric reasoning for room layout estimation via deep reinforcement learning. In: Proceedings of European Conference on Computer Vision (ECCV), pp. 550–565 (2020). https://doi.org/10.1007/978-3-030-58583-9_33

  53. Rong, J., Huang, S., Shang, Z., Ying, X.: Radial lens distortion correction using convolutional neural networks trained with synthesized images. In: Proceedings of Asian Conference on Computer Vision (ACCV). vol. 10113, pp. 35–49 (2017). https://doi.org/10.1007/978-3-319-54187-7_3

  54. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  55. Santana-Cedrés, D., et al.: An iterative optimization algorithm for lens distortion correction using two-parameter models. Image Processing OnLine (IPOL) 6, 326–364 (2016). https://doi.org/10.5201/ipol.2016.130

    Article  Google Scholar 

  56. Saputra, M.R.U., Gusmao, P., Almalioglu, Y., Markham, A., Trigoni, N.: Distilling knowledge from a deep pose regressor network. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 263–272 (2019). https://doi.org/10.1109/ICCV.2019.00035

  57. Sha, L., Hobbs, J., Felsen, P., Wei, X., Lucey, P., Ganguly, S.: End-to-End camera calibration for broadcast videos. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13624–13633 (2020). https://doi.org/10.1109/CVPR42600.2020.01364

  58. Shah, S., Aggarwal, J.K.: A simple calibration procedure for fish-eye (high distortion) lens camera. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). vol. 4, pp. 3422–3427 (1994). https://doi.org/10.1109/ROBOT.1994.351044

  59. Tsai, R.Y.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. (JRA) 3(4), 323–344 (1987). https://doi.org/10.1109/JRA.1987.1087109

    Article  Google Scholar 

  60. Wakai, N., Yamashita, T.: Deep single fisheye image camera calibration for over 180-degree projection of field of view. In: Proceedings of IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 1174–1183 (2021). https://doi.org/10.1109/ICCVW54120.2021.00137

  61. Wang, Z., Bovik, A.C.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. (TIP) 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  62. Xian, W., Li, Z., Snavely, N., Fisher, M., Eisenman, J., Shechtman, E.: UprightNet: Geometry-aware camera orientation estimation from single images. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9973–9982 (2019). https://doi.org/10.1109/ICCV.2019.01007

  63. Xue, F., Wang, X., Yan, Z., Wang, Q., Wang, J., Zha, H.: Local supports global: Deep camera relocalization with sequence enhancement. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2841–2850 (2019). https://doi.org/10.1109/ICCV.2019.00293

  64. Xue, Z., Xue, N., Xia, G., Shen, W.: Learning to calibrate straight lines for fisheye image rectification. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1643–1651 (2019). https://doi.org/10.1109/CVPR.2019.00174

  65. Yang, S., Lin, C., Liao, K., Zhang, C., Zhao, Y.: Progressively complementary network for fisheye image rectification using appearance flow. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6344–6353 (2021). https://doi.org/10.1109/CVPR46437.2021.00628

  66. Yin, X., Wang, X., Yu, J., Zhang, M., Fua, P., Tao, D.: FishEyeRecNet: a multi-context collaborative deep network for fisheye image rectification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 475–490. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_29

    Chapter  Google Scholar 

  67. Zhang, M., Yao, J., Xia, M., Li, K., Zhang, Y., Liu, Y.: Line-based multi-label energy optimization for fisheye image rectification and calibration. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4137–4145 (2015). https://doi.org/10.1109/CVPR.2015.7299041

  68. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Analysis Mach. Intell. (PAMI) 22(11), 1330–1334 (2000). https://doi.org/10.1109/34.888718

    Article  Google Scholar 

  69. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ADE20K dataset. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130 (2017). https://doi.org/10.1109/CVPR.2017.544

  70. Żołądek, H.: The topological proof of Abel-Ruffini theorem. Topological Methods Nonlinear Analysis 16, 253–265 (2000)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nobuhiko Wakai .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 15015 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wakai, N., Sato, S., Ishii, Y., Yamashita, T. (2022). Rethinking Generic Camera Models for Deep Single Image Camera Calibration to Recover Rotation and Fisheye Distortion. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics