Semantic Segmentation of Fisheye Images

  • Gregor BlottEmail author
  • Masato Takami
  • Christian Heipke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11129)


Semantic segmentation of fisheye images (e.g., from action-cameras or smartphones) requires different training approaches and data than those of rectilinear images obtained using central projection. The shape of objects is distorted depending on the distance between the principal point and the object position in the image. Therefore, classical semantic segmentation approaches fall short in terms of performance compared to rectilinear data. A potential solution to this problem is the recording and annotation of a new dataset, however this is expensive and tedious. In this study, an alternative approach that modifies the augmentation stage of deep learning training to re-use rectilinear training data is presented. In this way we obtain a considerably higher semantic segmentation performance on the fisheye images: +18.3% intersection over union (IoU) for action-camera test images, +8.3% IoU for artificially generated fisheye data, and +18.0% IoU for challenging security scenes acquired in bird’s eye view.


Semantic segmentation Fisheye images Deep learning 


  1. 1.
    Barreto, J.P., Araujo, H.: Issues on the geometry of central catadioptric image formation. In: CVPR, pp. 422–427. IEEE (2001)Google Scholar
  2. 2.
    Brown, D.C.: Decentering distortion of lenses. Photogram. Eng. 130, 444–462 (1966).
  3. 3.
    Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: CVPR, pp. 3213–3223. IEEE (2016)Google Scholar
  4. 4.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255. IEEE (2009)Google Scholar
  5. 5.
    Deng, L., Yang, M., Qian, Y., Wang, C., Wang, B.: CNN based semantic segmentation for urban traffic scenes using fisheye camera. In: Intelligent Vehicles Symposium (IV), pp. 231–236. IEEE (2017)Google Scholar
  6. 6.
    Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Rodríguez, J.G.: A review on deep learning techniques applied to semantic segmentation. CoRR abs/1704.06857 (2017)Google Scholar
  7. 7.
    Geyer, C., Daniilidis, K.: A unifying theory for central panoramic systems and practical implications. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 445–461. Springer, Heidelberg (2000). Scholar
  8. 8.
    Goodfellow, I.J., et al.: Generative adversarial networks. CoRR abs/1406.2661 (2014)Google Scholar
  9. 9.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  10. 10.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  11. 11.
    Mei, C.: Couplage vision omnidirectionnelle et télémétrie laser pour la navigation en robotique/laser-augmented omnidirectional vision for 3D localisation and mapping. Ph.D. thesis, INRIA Sophia Antipolis, Project-team ARobAS (2007)Google Scholar
  12. 12.
    Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. CoRR abs/1605.06211 (2016)Google Scholar
  13. 13.
    Shou, Z., Chan, J., Zareian, A., Miyazawa, K., Chang, S.: CDC: convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. CoRR abs/1703.01515 (2017)Google Scholar
  14. 14.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  15. 15.
    Strauß, T., Ziegler, J., Beck, J.: Calibrating multiple cameras with non-overlapping views using coded checkerboard targets. In: ITSC, pp. 2623–2628. IEEE (2014)Google Scholar
  16. 16.
    Su, T.-F., Chen, Y.-L., Lai, S.-H.: Over-segmentation based background modeling and foreground detection with shadow removal by using hierarchical MRFs. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6494, pp. 535–546. Springer, Heidelberg (2011). Scholar
  17. 17.
    Thoma, M.: A survey of semantic segmentation. CoRR abs/1602.06541 (2016)Google Scholar
  18. 18.
    Wei, X., Guo, Y., Gao, X., Yan, M., Sun, X.: A new semantic segmentation model for remote sensing images. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 1776–1779. IEEE (2017)Google Scholar
  19. 19.
    Yang, Y., Yang, J., Yan, J., Liao, S., Yi, D., Li, S.Z.: Salient color names for person re-identification. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 536–551. Springer, Cham (2014). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Computer Vision Research LabRobert Bosch GmbHHildesheimGermany
  2. 2.Institute of Photogrammetry and GeoInformationLeibniz Universität HannoverHannoverGermany

Personalised recommendations