Skip to main content
SpringerLink
Log in

Enhancing Low-light Images for Monocular Visual Odometry in Challenging Lighting Conditions

  • Special Issue on the 38th ICROS Annual Conference (ICROS 2023)
  • Published:
International Journal of Control, Automation and Systems Aims and scope Submit manuscript

Abstract

Visual odometry (VO) estimates the robot’s current position based on feature matching or brightness variation between images, making it primarily suitable for well-lit environments with good image quality. Consequently, existing visual odometry methods exhibit degraded performance in low-light or highly dynamic environments, limiting their operational efficiency in outdoor settings. To overcome these challenges, research has been conducted to enhance low-light images to improve odometry performance. Recent advancements in deep learning have facilitated extensive research on image enhancement, including low-light conditions. Utilizing generative adversarial networks (GANs) and techniques like CycleGAN, researchers have achieved robust improvements in various lighting conditions and enhanced odometry performance in low-light environments. However, these methods are typically trained on single images, compromising the structural consistency between consecutive images. In this paper, we propose learning-based low-light image enhancement and the preservation of structural consistency between consecutive images for monocular visual odometry. The proposed model utilizes the CycleGAN approach for domain transformation between different illumination levels, effectively avoiding the failure of visual odometry in low-light environments. To handle diverse lighting conditions within images, a local discriminator is employed to enhance local brightness. Additionally, a structure loss is introduced using sequence images to ensure structural consistency between the original and generated images. This method simultaneously improves low-light conditions and preserves structural consistency, leading to enhanced visual odometry performance in low-light environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus, “LIO-SAM: Tightly-coupled lidar inertial odometry via smoothing and mapping,” Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5135–5142, IEEE, 2020.

  2. R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras,” IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.

    Article  Google Scholar 

  3. J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” IEEE IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 3, pp. 611–625, 2018.

    Article  Google Scholar 

  4. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient alternative to SIFT or SURF,” Proc of International Conference on Computer Vision, pp. 2564–2571, Ieee, 2011.

  5. P. C. Ng and S. Henikoff, “SIFT: Predicting amino acid changes that affect protein function,” Nucleic Acids Research, vol. 31, no. 13, pp. 3812–3814, 2003.

    Article  Google Scholar 

  6. J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “Semanticfusion: Dense 3D semantic mapping with convolutional neural networks,” Proc. of IEEE International Conference on Robotics and automation (ICRA), pp. 4628–4635, IEEE, 2017.

  7. A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone, “3D dynamic scene graphs: Actionable spatial perception with places, objects, and humans,” arXiv preprint arXiv:2002.06289, 2020.

  8. L. Hao, H. Li, Q. Zhang, X. Hu, and J. Cheng, “LMVI-SLAM: Robust low-light monocular visual-inertial simultaneous localization and mapping,” Proc. of IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 272–277, IEEE, 2019.

  9. S. Zhang, Y. Zhi, S. Lu, Z. Lin, and R. He, “Monocular vision SLAM research for parking environment with low light,” International Journal of Automotive Technology, vol. 23, no. 3, pp. 693–703, 2022.

    Article  Google Scholar 

  10. J. Wang, R. Wang, and A. Wu, “Improved gamma correction for visual SLAM in low-light scenes,” Proc. of IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp. 1159–1163, IEEE, 2019.

  11. X. Guo, Y. Li, and H. Ling, “LIME: Low-light image enhancement via illumination map estimation,” IEEE Transactions on Image Processing, vol. 26, no. 2, pp. 982–993, 2016.

    Article  MathSciNet  MATH  Google Scholar 

  12. C. Li, C. Guo, L. Han, J. Jiang, M.-M. Cheng, J. Gu, and C. C. Loy, “Low-light image and video enhancement using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 9396–9416, 2021.

    Article  Google Scholar 

  13. Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, “EnlightenGAN: Deep light enhancement without paired supervision,” IEEE Transactions on Image Processing, vol. 30, pp. 2340–2349, 2021.

    Article  Google Scholar 

  14. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.

  15. Q. Zhang, L. Hao, H. Li, Z. Ren, and J. Cheng, “GANSLAM: GAN based monocular visual-inertial simultaneous localization and mapping in dark environments,” Proc. of 5th International Symposium on Autonomous Systems (ISAS), pp. 1–6, IEEE, 2022.

  16. D. You, J. Jung, W. Lee, and J. Oh, “Low-light image enhancement for visual odometry usingcyclegan and ssim-loss,” Proc. of the 38th ICROS Annual Conference (ICROS 2023), pp. 903–904, 2023.

  17. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” Proc. of the IEEE International Conference on Computer Vision, pp. 2223–2232, 2017.

  18. E. Jung, N. Yang, and D. Cremers, “Multi-frame GAN: Image enhancement for stereo visual odometry in low light,” Proc. of Conference on Robot Learning, pp. 651–660, PMLR, 2020.

  19. C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM,” IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021.

    Article  Google Scholar 

  20. A. J. Lee, Y. Cho, Y.-s. Shin, A. Kim, and H. Myung, “ViViD++: Vision for visibility dataset,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6282–6289, 2022.

    Article  Google Scholar 

  21. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.

  22. A. Savinykh, M. Kurenkov, E. Kruzhkov, E. Yudin, A. Potapov, P. Karpyshev, and D. Tsetserukou, “DarkSLAM: GAN-assisted visual SLAM for reliable operation in low-light conditions,” Proc. of IEEE 95th Vehicular Technology Conference (VTC2022-Spring), pp. 1–6, IEEE, 2022.

  23. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

    Article  Google Scholar 

  24. M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” Advances in Neural Information Processing Systems, vol. 30, 2017.

  25. X. Wang, “Laplacian operator-based edge detectors,” IEEE IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 5, pp. 886–890, 2007.

    Article  Google Scholar 

  26. W. Rong, Z. Li, W. Zhang, and L. Sun, “An improved Canny edge detection algorithm,” Proc. of IEEE International Conference on Mechatronics and Automation, pp. 577–582, IEEE, 2014.

  27. Y. Liu, M.-M. Cheng, X. Hu, K. Wang, and X. Bai, “Richer convolutional features for edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 8, pp. 1939–1946, 2019.

    Article  Google Scholar 

  28. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462–2470, 2017.

  29. L.-T. Hsu, F. Huang, H.-F. Ng, G. Zhang, Y. Zhong, X. Bai, and W. Wen, “Hong Kong UrbanNav: An open-source multisensory dataset for benchmarking urban navigation algorithms,” Navigation: Journal of the Institute of Navigation, vol. 70, no. 4, navi.602, 2023.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Robotics Engineering, Kwangwoon University, 21, Gwangun-ro, Nowongu, Seoul, Korea

    Donggil You, Jihoon Jung & Junghyun Oh

Authors
  1. Donggil You
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jihoon Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junghyun Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junghyun Oh.

Ethics declarations

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work has supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(No.2022R1A4A3033961), and in part by the Research Grant of Kwangwoon University, in 2022.

Donggil You received his B.S. degree in robotics from Kwangwoon University in 2022. His research interests include visual SLAM, place recognition, deep learning, and image translation.

Jihoon Jung received his B.S. degree in mechatronics from Sahmyook University in 2022. His research interests include visual SLAM, place recognition, and deep learning.

Junghyun Oh received his B.S., M.S., and Ph.D. degrees in electrical engineering from Seoul National University, Seoul, Korea, in 2012, 2014, and 2018, respectively. From 2018 to 2019, he worked as a senior engineer at Samsung Research of Samsung Electronics Co., Ltd., Seoul, Korea. Since 2019, he has been an Assistant Professor at Department of Robotics, Kwangwoon University, Seoul, Korea. His research interests include long-term robot autonomy, SLAM, and artificial intelligence for robotics.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

You, D., Jung, J. & Oh, J. Enhancing Low-light Images for Monocular Visual Odometry in Challenging Lighting Conditions. Int. J. Control Autom. Syst. 21, 3528–3539 (2023). https://doi.org/10.1007/s12555-023-0378-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12555-023-0378-7

Keywords

Search

Navigation