Modeling the Effects of Windshield Refraction for Camera Calibration

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)


In this paper, we study the effects of windshield refraction for autonomous driving applications. These distortion effects are surprisingly large and can not be explained by traditional camera models. Instead of using a generalized camera approach, we propose a novel approach to jointly optimize a traditional camera model, and a mathematical representation of the windshield’s surface. First, using the laws of geometric optics, the refraction is modeled using a local spherical approximation to the windshield’s geometry. Next, a spline-based model is proposed as a refinement to better adapt to deviations from the ideal shape and manufacturing variations. By jointly optimizing refraction and camera parameters, the projection error can be significantly reduced. The proposed models are validated on real windshield observations and custom setups to compare recordings with and without windshield, with accurate laser scan measurements as 3D ground truth.



This work was supported by the TRACE project with Toyota Motors Europe (TME).

Supplementary material

Supplementary material 1 (mp4 528 KB)

504443_1_En_24_MOESM2_ESM.pdf (4.8 mb)
Supplementary material 2 (pdf 4953 KB)


  1. 1.
    Agarwal, S., Mierle, K., Others: Ceres solver.
  2. 2.
    Agrawal, A., Ramalingam, S., Taguchi, Y., Chari, V.: A theory of multilayer flat refractive geometry. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3346–3353. IEEE (2012) (2012)Google Scholar
  3. 3.
    Chari, V., Sturm, P.: Multiple-view geometry of the refractive plane. In: 20th British Machine Vision Conference, pp. 1–11 (2009). In BMVAGoogle Scholar
  4. 4.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Gregson, J., Krimerman, M., Hullin, M.B., Heidrich, W.: Stochastic tomography and its applications in 3D imaging of mixing fluids. ACM Trans. Graph. (Proc. SIGGRAPH 2012) 31(4), 52:1–52:10 (2012)Google Scholar
  6. 6.
    Grossberg, M.D., Nayar, S.K.: The raxel imaging model and ray-based calibration. Int. J. Comput. Vis. 61(2), 119–137 (2005)CrossRefGoogle Scholar
  7. 7.
    Hanel, A., Hoegner, L., Stilla, U.: Towards the influence of a car windshield on depth calculation with a stereo camera system. In: ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B5, pp. 461–468 (2016)Google Scholar
  8. 8.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, Second edn. Cambridge University Press, Cambridge (2004)CrossRefGoogle Scholar
  9. 9.
    Huber, P.J.: Robust Statistic. In: Lovric, M. (ed.) International Encyclopedia of Statistical Science, pp. 1248-1251. Springer, Heidelberg (2011).
  10. 10.
    Kukelova, Z., Bujnak, M., Pajdla, T.: Real-time solution to the absolute pose problem with unknown radial distortion and focal length. In: Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV 2013), Washington, DC, USA, pp. 2816–2823 (2013)Google Scholar
  11. 11.
    Kunz, C., Singh, H.: Hemispherical refraction and camera calibration in underwater vision. In: OCEANS2008, pp. 1–7 (2008)Google Scholar
  12. 12.
    Miraldo, P., Araujo, H.: Calibration of smooth camera models. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2091–2103 (2013)CrossRefGoogle Scholar
  13. 13.
    Narasimhan, S., Nayar, S., Sun, B., Koppal, S.: Structured light in scattering media. In: Proceedings of ICCV, pp. 420–427 (2005)Google Scholar
  14. 14.
    Nishimura, M., Nobuhara, S., Matsuyama, T., Shimizu, S., Fujii, K.: A linear generalized camera calibration from three intersecting reference planes. In: Proceedings of ICCV, pp. 2354–2362 (2015)Google Scholar
  15. 15.
    Pável, S., Sándor, C., Csató, L.: Distortion estimation through explicit modeling of the refractive surface. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11729, pp. 17–28. Springer, Cham (2019). Scholar
  16. 16.
    Pedersen, M., Bengtson, S.H., Gade, R., Madsen, N., Moeslund, T.B.: Camera calibration for underwater 3D reconstruction based on ray tracing using Snell’s law. In: CVPR Workshops (CVPRW), pp. 1410–1417 (2018)Google Scholar
  17. 17.
    Ramalingam, S., Lodha, S., Sturm, P.: A generic structure-from-motion framework. Comput. Vis. Image Understand. 103, 218–228 (2006)CrossRefGoogle Scholar
  18. 18.
    Ramalingam, S., Sturm, P., Lodha, S.: Towards complete generic camera calibration. IEEE Conference on Computer Vision and Pattern Recognition, San Diego, USA, pp. 1093–1098 (2005)Google Scholar
  19. 19.
    Rousseeuw, P.J., Croux, C.: Alternatives to the median absolute deviation. J. Am. Stat. Assoc. 88(424), 1273–1283 (1993)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Sturm, P., Ramalingam, S.: A generic concept for camera calibration. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3022, pp. 1–13. Springer, Heidelberg (2004). Scholar
  21. 21.
    Sturm, P., Ramalingam, S., Tardif, J.P., Gasparini, S., Barreto, J.: Camera models and fundamental concepts used in geometric computer vision. Found. Trends ®Comput. Graph. Vis. 6(1–2), 1–183 (2011)Google Scholar
  22. 22.
    Trifonov, B., Bradley, D., Heidrich, W.: Tomographic reconstruction of transparent objects. In: Eurographics Conference on Rendering Techniques (2006)Google Scholar
  23. 23.
    Wikipedia: False position method – Wikipedia, the free encyclopedia (2018).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Center for Processing Speech and Images, ESAT-PSIKU LeuvenLeuvenBelgium
  2. 2.Computer Vision Lab, D-ITETETH ZürichZürichSwitzerland

Personalised recommendations