Advertisement

Image Classification in the Dark Using Quanta Image Sensors

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12353)

Abstract

State-of-the-art image classifiers are trained and tested using well-illuminated images. These images are typically captured by CMOS image sensors with at least tens of photons per pixel. However, in dark environments when the photon flux is low, image classification becomes difficult because the measured signal is suppressed by noise. In this paper, we present a new low-light image classification solution using Quanta Image Sensors (QIS). QIS are a new type of image sensors that possess photon-counting ability without compromising on pixel size and spatial resolution. Numerous studies over the past decade have demonstrated the feasibility of QIS for low-light imaging, but their usage for image classification has not been studied. This paper fills the gap by presenting a student-teacher learning scheme which allows us to classify the noisy QIS raw data. We show that with student-teacher learning, we can achieve image classification at a photon level of one photon per pixel or lower. Experimental results verify the effectiveness of the proposed method compared to existing solutions.

Keywords

Quanta image sensors Low light Classification 

Notes

Acknowledgement

This work is supported, in part, by the US National Science Foundation under grant CCF-1718007.

Supplementary material

504445_1_En_29_MOESM1_ESM.pdf (1.2 mb)
Supplementary material 1 (pdf 1206 KB)

References

  1. 1.
    Ba, J., Caruana, R.: Do deep nets really need to be deep? In: NeurIPS (2014)Google Scholar
  2. 2.
    Callenberg, C., Lyons, A., den Brok, D., Henderson, R., Hullin, M.B., Faccio, D.: EMCCD-SPAD camera data fusion for high spatial resolution time-of-flight imaging. In: Computational Optical Sensing and Imaging (2019)Google Scholar
  3. 3.
    Chan, S.H., Elgendy, O.A., Wang, X.: Images from bits: non-iterative image reconstruction for quanta image sensors. Sensors 16(11), 1961 (2016)Google Scholar
  4. 4.
    Chan, S.H., Lu, Y.M.: Efficient image reconstruction for gigapixel quantum image sensors. In: IEEE Global Conference on Signal and Information Processing (2014)Google Scholar
  5. 5.
    Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3(1), 84–98 (2016)MathSciNetGoogle Scholar
  6. 6.
    Chandramouli, P., Burri, S., Bruschini, C., Charbon, E., Kolb, A.: A bit too much? High speed imaging from sparse photon counts. In: ICCP (2019)Google Scholar
  7. 7.
    Chen, B., Perona, P.: Seeing into darkness: scotopic visual recognition. In: CVPR (2017)Google Scholar
  8. 8.
    Chen, C., Chen, Q., Do, M.N., Koltun, V.: Seeing motion in the dark. In: ICCV (2019)Google Scholar
  9. 9.
    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR (2018)Google Scholar
  10. 10.
    Chen, G., Li, Y., Srihari, S.N.: Joint visual denoising and classification using deep learning. In: ICIP (2016)Google Scholar
  11. 11.
    Choi, J.H., Elgendy, O.A., Chan, S.H.: Image reconstruction for quanta image sensors using deep neural networks. In: ICASSP (2018)Google Scholar
  12. 12.
    Davy, A., Ehret, T., Morel, J.M., Arias, P., Facciolo, G.: A non-local CNN for video denoising. In: ICIP (2019)Google Scholar
  13. 13.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  14. 14.
    Diamond, S., Sitzmann, V., Boyd, S., Wetzstein, G., Heide, F.: Dirty pixels: optimizing image classification architectures for raw sensor data. arXiv preprint arXiv:1701.06487 (2017)
  15. 15.
    Dodge, S., Karam, L.: Quality resilient deep neural networks. arXiv preprint arXiv:1703.08119 (2017)
  16. 16.
    Elgendy, O.A., Chan, S.H.: Optimal threshold design for quanta image sensor. IEEE Trans. Comput. Imaging 4(1), 99–111 (2017)MathSciNetGoogle Scholar
  17. 17.
    Elgendy, O.A., Chan, S.H.: Color filter arrays for quanta image sensors. arXiv preprint arXiv:1903.09823 (2019)
  18. 18.
    Fossum, E.R.: Gigapixel digital film sensor (DFS) proposal. In: Nanospace Manipulation of Photons and Electrons for Nanovision Systems (2005)Google Scholar
  19. 19.
    Fossum, E.R.: Some thoughts on future digital still cameras. In: Image Sensors and Signal Processing for Digital Still Cameras (2006)Google Scholar
  20. 20.
    Fossum, E.R.: Modeling the performance of single-bit and multi-bit quanta image sensors. IEEE J. Electron Dev. Soc. 1(9), 166–174 (2013)Google Scholar
  21. 21.
    Fossum, E.R., Ma, J., Masoodian, S., Anzagira, L., Zizza, R.: The quanta image sensor: every photon counts. Sensors 16(8), 1260 (2016)Google Scholar
  22. 22.
    Fowler, B., McGrath, D., Bartkovjak, P.: Read noise distribution modeling for CMOS image sensors. In: International Image Sensor Workshop (2013)Google Scholar
  23. 23.
    Fu, Q., Jung, C., Xu, K.: Retinex-based perceptual contrast enhancement in images using luminance adaptation. IEEE Access 6, 61277–61286 (2018)Google Scholar
  24. 24.
    Gariepy, G., et al.: Single-photon sensitive light-in-fight imaging. Nat. Commun. 6(1), 1–7 (2015)MathSciNetGoogle Scholar
  25. 25.
    Gatys, L., Ecker, A., Bethge, M.: A neural algorithm of artistic style. Nat. Commun. (2015) Google Scholar
  26. 26.
    Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. In: NeurIPS (2015)Google Scholar
  27. 27.
    Gnanasambandam, A., Elgendy, O., Ma, J., Chan, S.H.: Megapixel photon-counting color imaging using quanta image sensor. Opt. Express 27(12), 17298–17310 (2019)Google Scholar
  28. 28.
    Gnanasambandam, A., Ma, J., Chan, S.H.: High dynamic range imaging using quanta image sensors. In: International Image Sensors Workshop (2019)Google Scholar
  29. 29.
    Guo, T., Xu, C., He, S., Shi, B., Xu, C., Tao, D.: Robust student network learning. IEEE Trans. Neural Netw. Learn. Syst. 31(7), 2455–2468 (2020)Google Scholar
  30. 30.
    Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Gupta, A., Ingle, A., Gupta, M.: Asynchronous single-photon 3D imaging. In: ICCV, October 2019Google Scholar
  32. 32.
    Gyongy, I., Dutton, N., Henderson, R.: Single-photon tracking for high-speed vision. Sensors 18(2), 323 (2018)Google Scholar
  33. 33.
    Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6), 192 (2016)Google Scholar
  34. 34.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NeurIPS Deep Learning and Representation Learning Workshop (2015)Google Scholar
  35. 35.
    Hossain, M.T., Teng, S.W., Zhang, D., Lim, S., Lu, G.: Distortion robust image classification using deep convolutional neural network with discrete cosine transform. In: ICIP (2019)Google Scholar
  36. 36.
    Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  37. 37.
    Hu, Z., Cho, S., Wang, J., Yang, M.H.: Deblurring low-light images with light streaks. In: CVPR (2014)Google Scholar
  38. 38.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  39. 39.
    Joshi, N., Cohen, M.: Seeing Mt. Rainier: lucky imaging for multi-image denoising, sharpening, and haze removal. In: ICCP (2010)Google Scholar
  40. 40.
    Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fine-grained image categorization: stanford dogs. In: CVPR Workshop on Fine-Grained Visual Categorization (2011)Google Scholar
  41. 41.
    Kokkinos, F., Lefkimmiatis, S.: Iterative residual CNNs for burst photography applications. In: CVPR (2019)Google Scholar
  42. 42.
    Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37(4), 1–12 (2018)Google Scholar
  43. 43.
    Liu, C., Freeman, W.T.: A high-quality video denoising algorithm based on reliable motion estimation. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 706–719. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15558-1_51CrossRefGoogle Scholar
  44. 44.
    Liu, D., Wen, B., Jiao, J., Liu, X., Wang, Z., Huang, T.S.: Connecting image denoising and high-level vision tasks via deep learning. IEEE Trans. Image Process. 29, 3695–3706 (2020)Google Scholar
  45. 45.
    Liu, Z., Zhou, T., Shen, Z., Kang, B., Darrell, T.: Transferable recognition-aware image processing. arXiv preprint arXiv:1910.09185 (2019)
  46. 46.
    Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graph. 33(6), 232 (2014)Google Scholar
  47. 47.
    Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn. 61, 650–662 (2017)Google Scholar
  48. 48.
    Ma, J., Fossum, E.: A pump-gate jot device with high conversion gain for a quanta image sensor. IEEE J. Electron Dev. Soc. 3(2), 73–77 (2015)Google Scholar
  49. 49.
    Ma, J., Masoodian, S., Starkey, D., Fossum, E.R.: Photon-number-resolving megapixel image sensor at room temperature without avalanche gain. Optica 4(12), 1474–1481 (2017)Google Scholar
  50. 50.
    Ma, S., Gupta, S., Ulku, A.C., Brushini, C., Charbon, E., Gupta, M.: Quanta burst photography. ACM Trans. Graph. (TOG) 39(4) (2020) Google Scholar
  51. 51.
    Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: CVPR (2015)Google Scholar
  52. 52.
    Makitalo, M., Foi, A.: Optimal inversion of the Anscombe transformation in low-count Poisson image denoising. IEEE Trans. Image Process. 20(1), 99–109 (2010)MathSciNetzbMATHGoogle Scholar
  53. 53.
    Malm, H., Oskarsson, M., Warrant, E., Clarberg, P., Hasselgren, J., Lejdfors, C.: Adaptive enhancement and noise reduction in very low light-level video. In: ICCV (2007)Google Scholar
  54. 54.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: CVPR (2018)Google Scholar
  55. 55.
    O’Toole, M., Heide, F., Lindell, D.B., Zang, K., Diamond, S., Wetzstein, G.: Reconstructing transient images from single-photon sensors. In: CVPR (2017)Google Scholar
  56. 56.
    Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: CVPR (2017)Google Scholar
  57. 57.
    Remez, T., Litany, O., Giryes, R., Bronstein, A.: Deep convolutional denoising of low-light images. arXiv preprint arXiv:1701.01687 (2017)
  58. 58.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. ICLR (2014)Google Scholar
  59. 59.
  60. 60.
    Talebi, H., Milanfar, P.: Learned perceptual image enhancement. In: ICCP (2018)Google Scholar
  61. 61.
    Wu, J., Timofte, R., Huang, Z., Van Gool, L.: On the relation between color image denoising and classification. arXiv preprint arXiv:1704.01372 (2017)
  62. 62.
    Xu, J., Li, H., Liang, Z., Zhang, D., Zhang, L.: Real-world noisy image denoising: a new benchmark. arXiv preprint arXiv:1804.02603 (2018)
  63. 63.
    Yang, C., Xie, L., Qiao, S., Yuille, A.: Training deep neural networks in generations: a more tolerant teacher educates better students. In: AAAI Conference on Artificial Intelligence, July 2019Google Scholar
  64. 64.
    Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: An optimal algorithm for reconstructing images from binary measurements. In: Proceedings of SPIE, vol. 7533, pp. 158–169 (2010)Google Scholar
  65. 65.
    Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: Bits from photons: Oversampled image acquisition using binary Poisson statistics. IEEE Trans. Image Process. 21(4), 1421–1436 (2011)MathSciNetzbMATHGoogle Scholar
  66. 66.
    Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. In: ICML Deep Learning Workshop (2015)Google Scholar
  67. 67.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetzbMATHGoogle Scholar
  68. 68.
    Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)MathSciNetGoogle Scholar
  69. 69.
    Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: CVPR (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Purdue UniversityWest LafayetteUSA

Personalised recommendations