Advertisement

Dynamic Low-Light Imaging with Quanta Image Sensors

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12366)

Abstract

Imaging in low light is difficult because the number of photons arriving at the sensor is low. Imaging dynamic scenes in low-light environments is even more difficult because as the scene moves, pixels in adjacent frames need to be aligned before they can be denoised. Conventional CMOS image sensors (CIS) are at a particular disadvantage in dynamic low-light settings because the exposure cannot be too short lest the read noise overwhelms the signal. We propose a solution using Quanta Image Sensors (QIS) and present a new image reconstruction algorithm. QIS are single-photon image sensors with photon counting capabilities. Studies over the past decade have confirmed the effectiveness of QIS for low-light imaging but reconstruction algorithms for dynamic scenes in low light remain an open problem. We fill the gap by proposing a student-teacher training protocol that transfers knowledge from a motion teacher and a denoising teacher to a student network. We show that dynamic scenes can be reconstructed from a burst of frames at a photon level of 1 photon per pixel per frame. Experimental results confirm the advantages of the proposed method compared to existing methods.

Keywords

Quanta Image Sensors Low light Burst photography 

Notes

Acknowledgement

This work is supported in part by the US National Science Foundation under grant CCF-1718007.

References

  1. 1.
    Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
  2. 2.
    Aittala, M., Durand, F.: Burst image deblurring using permutation invariant convolutional neural networks. In: ECCV (2018)Google Scholar
  3. 3.
    Buades, A., Coll, B., Morel, J.M.: Denoising image sequences does not require motion estimation. In: IEEE Conference Advanced Video and Signal Based Surveillance, pp. 70–74 (2005)Google Scholar
  4. 4.
    Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. SIAM Multiscale Modeling Simul. 4(2), 490–530 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Burri, S., Maruyama, Y., Michalet, X., Regazzoni, F., Bruschini, C., Charbon, E.: Architecture and applications of a high resolution gated SPAD image sensor. Optics Express 22(14), 17573–17589 (2014)CrossRefGoogle Scholar
  6. 6.
    Callenberg, C., Lyons, A., den Brok, D., Henderson, R., Hullin, M.B., Faccio, D.: EMCCD-SPAD camera data fusion for high spatial resolution time-of-flight imaging. In: Computational Optical Sensing and Imaging. Optical Society of America (2019)Google Scholar
  7. 7.
    Chan, S.H., Elgendy, O.A., Wang, X.: Images from bits: non-iterative imagereconstruction for Quanta Image Sensors. Sensors 16(11) (2016)Google Scholar
  8. 8.
    Chan, S.H., Lu, Y.M.: Efficient image reconstruction for gigapixel Quantum Image Sensors. In: IEEE Global Conference Signal and Information Processing (2014)Google Scholar
  9. 9.
    Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3(1), 84–98 (2017)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Chandramouli, P., Burri, S., Bruschini, C., Charbon, E., Kolb, A.: A bit too much?. In: ICCP, High Speed Imaging from Sparse Photon Counts (2019)Google Scholar
  11. 11.
    Chen, C., Chen, Q., Do, M.N., Koltun, V.: Seeing motion in the dark. In: ICCV (2019)Google Scholar
  12. 12.
    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR (2018)Google Scholar
  13. 13.
    Choi, J.H., Elgendy, O.A., Chan, S.H.: Image reconstruction for Quanta Image Sensors using deep neural networks. In: ICASSP (2018)Google Scholar
  14. 14.
    Chollet, F., et al.: Keras (2015). https://www.keras.io
  15. 15.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Davy, A., et al.: A non-local CNN for video denoising. In: ICIP (2019)Google Scholar
  17. 17.
    Dutton, N.A., et al.: A SPAD-based QVGA image sensor for single-photon counting and quanta imaging. IEEE Trans. Electron Devices 63(1), 189–196 (2015)CrossRefGoogle Scholar
  18. 18.
    Dutton, N.A., Parmesan, L., Holmes, A.J., Grant, L.A., Henderson, R.K.: 320\( \times \)240 oversampled digital single photon counting image sensor. In: Symposium on VLSI Circuits Digest of Technical Papers (2014)Google Scholar
  19. 19.
    Elgendy, O.A., Chan, S.H.: Optimal threshold design for Quanta Image Sensor. IEEE Trans. Comput. Imaging 4(1), 99–111 (2017)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Elgendy, O.A., Chan, S.H.: Color Filter Arrays for Quanta Image Sensors. arXiv preprint arXiv:1903.09823 (2019)
  21. 21.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  22. 22.
    Fossum, E.R.: Gigapixel digital film Sensor (DFS) proposal. In: Nanospace Manipulation of Photons and Electrons for Nanovision Systems (2005)Google Scholar
  23. 23.
    Fossum, E.R.: Some thoughts on future digital still cameras. In: Image Sensors and Signal Processing for Digital Still Cameras (2006)Google Scholar
  24. 24.
    Fossum, E.R.: Modeling the performance of single-bit and multi-bit quanta image sensors. IEEE J. Electron Devices Soc. 1(9), 166–174 (2013)CrossRefGoogle Scholar
  25. 25.
    Fu, Q., Jung, C., Xu, K.: Retinex-based perceptual contrast enhancement in images using luminance adaptation. IEEE Access 6, 61277–61286 (2018)CrossRefGoogle Scholar
  26. 26.
    Gariepy, G., et al.: Single-photon sensitive light-in-fight imaging. Nat. Commun. 6(1), 1–7 (2015)MathSciNetGoogle Scholar
  27. 27.
    Gnanasambandam, A., Elgendy, O., Ma, J., Chan, S.H.: Megapixel photon-counting color imaging using Quanta Image Sensor. Optics Express 27(12), 17298–17310 (2019)CrossRefGoogle Scholar
  28. 28.
    Gnanasambandam, A., Ma, J., Chan, S.H.: High dynamic range imaging using Quanta Image Sensors. In: International Image Sensors Workshop (2019)Google Scholar
  29. 29.
    Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: ECCV (2018)Google Scholar
  30. 30.
    Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009)Google Scholar
  31. 31.
    Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Gupta, A., Ingle, A., Gupta, M.: Asynchronous single-photon 3D imaging. In: ICCV (2019)Google Scholar
  33. 33.
    Gyongy, I., Dutton, N., Henderson, R.: Single-photon tracking for high-speed vision. Sensors 18(2), 323 (2018)CrossRefGoogle Scholar
  34. 34.
    Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6) (2016)Google Scholar
  35. 35.
    Horn, B.K., Schunck, B.G.: Determining optical flow. In: Techniques and Applications of Image Understanding, vol. 281. International Society Optics and Photonics (1981)Google Scholar
  36. 36.
    Hu, Z., Cho, S., Wang, J., Yang, M.H.: Deblurring low-light images with light streaks. In: CVPR (2014)Google Scholar
  37. 37.
    Ji, H., Liu, C., Shen, Z., Xu, Y.: Robust video denoising using low rank matrix completion. In: CVPR (2010)Google Scholar
  38. 38.
    Joshi, N., Cohen, M.: Seeing Mt. Rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. In: ICCP (2010)Google Scholar
  39. 39.
    Kokkinos, F., Lefkimmiatis, S.: Iterative residual CNNs for burst photography applications. In: CVPR (2019)Google Scholar
  40. 40.
    Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deepsensor fusion. ACM Trans. Graph. 37(4) (2018)Google Scholar
  41. 41.
    Liu, C., Freeman, W.: A high-quality video denoising algorithm based on reliable motion estimation. In: ECCV (2010)Google Scholar
  42. 42.
    Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graph., 33(6) (2014)Google Scholar
  43. 43.
    Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn. 61, 650–662 (2017)CrossRefGoogle Scholar
  44. 44.
    Ma, J., Fossum, E.: A pump-gate jot device with high conversion gain for a Quanta Image Sensor. IEEE J. Electron Devices Soc. 3(2), 73–77 (2015)CrossRefGoogle Scholar
  45. 45.
    Ma, J., Masoodian, S., Starkey, D., Fossum, E.R.: Photon-number-resolving megapixel image sensor at room temperature without avalanche gain. Optica 4(12), 1474–1481 (2017)CrossRefGoogle Scholar
  46. 46.
    Ma, S., Gupta, S., Ulku, A.C., Brushini, C., Charbon, E., Gupta, M.: Quanta burst photography. ACM Trans. Graph. (TOG), 39(4) (2020)Google Scholar
  47. 47.
    Maggioni, M., Katkovnik, V., Egiazarian, K., Foi, A.: Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Trans. Image Process. 22(1), 119–133 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Makitalo, M., Foi, A.: Optimal inversion of the Anscombe transformation in low-count Poisson image denoising. IEEE Trans. Image Process. 20(1), 99–109 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  49. 49.
    Malm, H., Oskarsson, M., Warrant, E., et al.: Adaptive enhancement and noise reduction in very low light-level video. In: ICCV (2007)Google Scholar
  50. 50.
    Mao, X.J., Shen, C., Yang, Y.B.: Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921 (2016)
  51. 51.
    Mildenhall, B., et al.: Burst denoising with kernel prediction networks. In: CVPR (2018)Google Scholar
  52. 52.
    O’Toole, M., Heide, F., Lindell, D.B., Zang, K., Diamond, S., Wetzstein, G.: Reconstructing transient images from single-photon sensors. In: CVPR (2017)Google Scholar
  53. 53.
    Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: CVPR (2017)Google Scholar
  54. 54.
    Protter, M., Elad, M.: Image sequence denoising via sparse and redundant representations. IEEE Trans. Image Process. 18(1), 27–35 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  55. 55.
    Remez, T., Litany, O., Bronstein, A.: A picture is worth a billion bits: real-time image reconstruction from dense binary threshold pixels. In: ICCP (2016)Google Scholar
  56. 56.
    Remez, T., Litany, O., Giryes, R., Bronstein, A.: Deep convolutional denoising of low-light images. arXiv preprint arXiv:1701.01687 (2017)
  57. 57.
    Sutour, C., Deledalle, C.A., Aujol, J.F.: Adaptive regularization of the NL-means: application to image and video denoising. IEEE Trans. Image Process. 23(8), 3506–3521 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Werlberger, M., Pock, T., Unger, M., Bischof, H.: Optical flow guided TV-L 1 video interpolation and restoration. In: International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition (2011)Google Scholar
  59. 59.
    Xia, Z., Perazzi, F., Gharbi, M., Sunkavalli, K., Chakrabarti, A.: Basis prediction networks for effective burst denoising with large kernels. arXiv preprint arXiv:1912.04421 (2019)
  60. 60.
    Xu, J., Li, H., Liang, Z., et al.: Real-world noisy image denoising: a new benchmark. arXiv preprint arXiv:1804.02603 (2018)
  61. 61.
    Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: An optimal algorithm for reconstructing images from binary measurements. In: Proceedings SPIE, vol. 7533 (2010)Google Scholar
  62. 62.
    Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: Bits from photons: oversampled image acquisition using binary poisson statistics. IEEE Trans. Image Process. 21(4), 1421–1436 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    Yang, F., Sbaiz, L., Charbon, E., Süsstrunk, S., Vetterli, M.: Image reconstruction in the gigavision camera. In: ICCV Workshops (2009)Google Scholar
  64. 64.
    Zhang, K., Zuo, W., Chen, Y., et al.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7) (2017)Google Scholar
  65. 65.
    Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9) (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of ECEPurdue UniversityWest LafayetteUSA
  2. 2.Intel LabsSanta ClaraUSA

Personalised recommendations