Advertisement

Event Enhanced High-Quality Image Recovery

Conference paper
  • 433 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)

Abstract

With extremely high temporal resolution, event cameras have a large potential for robotics and computer vision. However, their asynchronous imaging mechanism often aggravates the measurement sensitivity to noises and brings a physical burden to increase the image spatial resolution. To recover high-quality intensity images, one should address both denoising and super-resolution problems for event cameras. Since events depict brightness changes, with the enhanced degeneration model by the events, the clear and sharp high-resolution latent images can be recovered from the noisy, blurry and low-resolution intensity observations. Exploiting the framework of sparse learning, the events and the low-resolution intensity observations can be jointly considered. Based on this, we propose an explainable network, an event-enhanced sparse learning network (eSL-Net), to recover the high-quality images from event cameras. After training with a synthetic dataset, the proposed eSL-Net can largely improve the performance of the state-of-the-art by 7–12 dB. Furthermore, without additional training process, the proposed eSL-Net can be easily extended to generate continuous frames with frame-rate as high as the events.

Keywords

Event camera Intensity reconstruction Denoising Deblurring Super resolution Sparse learning 

Notes

Acknowledgement

The research was partially supported by the National Natural Science Foundation of China under Grants 61871297, and the Fundamental Research Funds for the Central Universities under Grants 2042020kf0019.

Supplementary material

504454_1_En_10_MOESM1_ESM.pdf (14.1 mb)
Supplementary material 1 (pdf 14412 KB)

References

  1. 1.
    Almatrafi, M.M., Hirakawa, K.: DAViS camera optical flow. IEEE Trans. Comput. Imag. 6, 396–407 (2020)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 884–892 (2016)Google Scholar
  3. 3.
    Barua, S., Miyatani, Y., Veeraraghavan, A.: Direct face detection and video reconstruction from event cameras. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9. IEEE (2016)Google Scholar
  4. 4.
    Brandli, C., Berner, R., Yang, M., Liu, S.C., Delbruck, T.: A 240\(\times \)180 130 db 3 \(\mu \)s latency global shutter spatiotemporal vision sensor. IEEE J. Solid State Circ. 49(10), 2333–2341 (2014)CrossRefGoogle Scholar
  5. 5.
    Choi, J., Yoon, K.J., et al.: Learning to super resolve intensity images from events. In: CVPR 2020. arXiv preprint arXiv:1912.01196 (2019)
  6. 6.
    Cook, M., Gugelmann, L., Jug, F., Krautz, C., Steger, A.: Interacting maps for fast visual interpretation. In: The 2011 International Joint Conference on Neural Networks, pp. 770–776. IEEE (2011)Google Scholar
  7. 7.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Dirac, P.A.M.: The Principles of Quantum Mechanics. No. 27. Oxford University Press, London (1981)Google Scholar
  9. 9.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)CrossRefGoogle Scholar
  10. 10.
    Donoho, D.L., et al.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gallego, G., et al.: Event-based vision: a survey. arXiv preprint arXiv:1904.08405 (2019)
  13. 13.
    Gehrig, D., Loquercio, A., Derpanis, K.G., Scaramuzza, D.: End-to-end learning of representations for asynchronous event-based data. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5633–5643 (2019)Google Scholar
  14. 14.
    Gregor, K., LeCun, Y.: Learning fast approximations of sparse coding. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399–406 (2010)Google Scholar
  15. 15.
    Kim, H., Handa, A., Benosman, R., Ieng, S., Davison, A.: Simultaneous mosaicing and tracking with an event camera. In: BMVC 2014-Proceedings of the British Machine Vision Conference (2014)Google Scholar
  16. 16.
    Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3D reconstruction and 6-DoF tracking with an event camera. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 349–364. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46466-4_21CrossRefGoogle Scholar
  17. 17.
    Lichtsteiner, P., Posch, C., Delbruck, T.: A 128 \(\times \) 128 120 db 15 \(\mu \)s latency asynchronous temporal contrast vision sensor. IEEE J. Solid State Circ. 43(2), 566–576 (2008)CrossRefGoogle Scholar
  18. 18.
    Liu, S.C., Delbruck, T.: Neuromorphic sensory systems. Curr. Opin. Neurol. 20(3), 288–295 (2010)CrossRefGoogle Scholar
  19. 19.
    Munda, G., Reinbacher, C., Pock, T.: Real-time intensity-image reconstruction for event cameras using manifold regularisation. Int. J. Comput. Vis. 126(12), 1381–1393 (2018)CrossRefGoogle Scholar
  20. 20.
    Nah, S., et al.: NTIRE 2019 challenge on video deblurring and super-resolution: dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)Google Scholar
  21. 21.
    Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)Google Scholar
  22. 22.
    Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 261–270 (2017)Google Scholar
  23. 23.
    Pan, J., Sun, D., Pfister, H., Yang, M.H.: Blind image deblurring using dark channel prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1628–1636 (2016)Google Scholar
  24. 24.
    Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., Dai, Y.: High frame rate video reconstruction based on an event camera. arXiv preprint arXiv:1903.06531 (2019)
  25. 25.
    Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., Dai, Y.: Bringing a blurry frame alive at high frame-rate with an event camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6820–6829 (2019)Google Scholar
  26. 26.
    Papyan, V., Romano, Y., Elad, M.: Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. 18(1), 2887–2938 (2017)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Posch, C., Matolin, D., Wohlgenannt, R.: A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS. IEEE J. Solid State Circ. 46(1), 259–275 (2010)CrossRefGoogle Scholar
  28. 28.
    Rebecq, H., Gehrig, D., Scaramuzza, D.: ESIM: an open event camera simulator. In: Conference on Robot Learning, pp. 969–982 (2018)Google Scholar
  29. 29.
    Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: Events-to-video: Bringing modern computer vision to event cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3857–3866 (2019)Google Scholar
  30. 30.
    Scheerlinck, C., Barnes, N., Mahony, R.: Continuous-time intensity estimation using event cameras. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11365, pp. 308–324. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-20873-8_20CrossRefGoogle Scholar
  31. 31.
    Son, B., et al.: A 640\(\times \) 480 dynamic vision sensor with a 9\(\mu \)m pixel and 300meps address-event representation. In: 2017 IEEE International Solid-State Circuits Conference (ISSCC), pp. 66–67. IEEE (2017)Google Scholar
  32. 32.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Royal Stat. Soc. Ser. B Method. 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Vidal, A.R., Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Ultimate SLAM? Combining events, images, and IMU for robust visual SLAM in HDR and high-speed scenarios. IEEE Robot. Autom. Lett. 3(2), 994–1001 (2018)CrossRefGoogle Scholar
  34. 34.
    Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398–1402. IEEE (2003)Google Scholar
  35. 35.
    Wang, Z.W., Jiang, W., He, K., Shi, B., Katsaggelos, A., Cossairt, O.: Event-driven video frame synthesis. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)Google Scholar
  36. 36.
    Wieschollek, P., Hirsch, M., Scholkopf, B., Lensch, H.: Learning blind motion deblurring. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 231–240 (2017)Google Scholar
  37. 37.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_18CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Wuhan UniversityWuhanChina

Personalised recommendations