Photon-Efficient 3D Imaging with A Non-local Neural Network

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)


Photon-efficient imaging has enabled a number of applications relying on single-photon sensors that can capture a 3D image with as few as one photon per pixel. In practice, however, measurements of low photon counts are often mixed with heavy background noise, which poses a great challenge for existing computational reconstruction algorithms. In this paper, we first analyze the long-range correlations in both spatial and temporal dimensions of the measurements. Then we propose a non-local neural network for depth reconstruction by exploiting the long-range correlations. The proposed network achieves decent reconstruction fidelity even under photon counts (and signal-to-background ratio, SBR) as low as 1 photon/pixel (and 0.01 SBR), which significantly surpasses the state-of-the-art. Moreover, our non-local network trained on simulated data can be well generalized to different real-world imaging systems, which could extend the application scope of photon-efficient imaging in challenging scenarios with a strict limit on optical flux. Code is available at


Photon-efficient imaging Long-range correlation Non-local network Depth reconstruction 



We acknowledge funding from National Key R&D Program of China under Grants 2017YFA0700800 and 2018YFB0504300, National Natural Science Foundation of China under Grants 61671419 and 61771443, the Shanghai Municipal Science and Technology Major Project (2019SHZDZX01), the Shanghai Science and Technology Development Funds (18JC1414700), and the Fundamental Research Funds for the Central Universities (WK2340000083).

Supplementary material

504443_1_En_14_MOESM1_ESM.pdf (482 kb)
Supplementary material 1 (pdf 481 KB)


  1. 1.
    Abreu, E., Lightstone, M., Mitra, S.K., Arakawa, K.: A new efficient approach for the removal of impulse noise from highly corrupted images. IEEE Trans. Image Process. 5(6), 1012–1025 (1996)CrossRefGoogle Scholar
  2. 2.
    Altmann, Y., Ren, X., Mccarthy, A., Buller, G., Mclaughlin, S.: Lidar waveform based analysis of depth images constructed using sparse single-photon data. IEEE Trans. Comput. Imaging 25(5), 1935–1946 (2016)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Altmann, Y., McLaughlin, S., Padgett, M.J., Goyal, V.K., Hero, A.O., Faccio, D.: Quantum-inspired computational imaging. Science 361(6403), 2298 (2018)CrossRefGoogle Scholar
  4. 4.
    Bar-David, I.: Communication under the poisson regime. IEEE Trans. Inf. Theory 15(1), 31–37 (1969)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Barbastathis, G., Ozcan, A., Situ, G.: On the use of deep learning for computational imaging. Optica 6(8), 921–943 (2019)CrossRefGoogle Scholar
  6. 6.
    Buller, G.S., Wallace, A.M., Mccarthy, A., Lamb, R.A.: Ranging and three-dimensional imaging using time-correlated single-photon counting. IEEE J. Sel. Top. Quantum Electron. 13(4), 1006–1015 (2007)CrossRefGoogle Scholar
  7. 7.
    Chan, S., et al.: Long-range depth imaging using a single-photon detector array and non-local data fusion. Sci. Rep. 9(1), 8075 (2019)CrossRefGoogle Scholar
  8. 8.
    Chen, C., Xiong, Z., Tian, X., Wu, F.: Deep boosting for image denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, Part XI. LNCS, vol. 11215, pp. 3–19. Springer, Cham (2018). Scholar
  9. 9.
    Chen, C., Xiong, Z., Tian, X., Zha, Z.J., Wu, F.: Real-world image denoising with deep boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019) Google Scholar
  10. 10.
    Cheng, Z., Xiong, Z., Liu, D.: Light field super-resolution by jointly exploiting internal and external similarities. IEEE Trans. Circuits Syst. Video Technol. 30(8), 2604–2616 (2019) CrossRefGoogle Scholar
  11. 11.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)Google Scholar
  13. 13.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)CrossRefGoogle Scholar
  14. 14.
    Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  15. 15.
    Gupta, A., Ingle, A., Gupta, M.: Asynchronous single-photon 3D imaging. In: IEEE International Conference on Computer Vision, pp. 7909–7918 (2019)Google Scholar
  16. 16.
    Gupta, A., Ingle, A., Velten, A., Gupta, M.: Photon-flooded single-photon 3D cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6770–6779 (2019)Google Scholar
  17. 17.
    Hadfield, R.H.: Single-photon detectors for optical quantum information applications. Nat. Photonics 3(12), 696 (2009)CrossRefGoogle Scholar
  18. 18.
    Holst, G.C.: CCD Arrays, Cameras, and Displays. SPIE Optical Engineering, Bellingham (1998)Google Scholar
  19. 19.
    Ingle, A., Velten, A., Gupta, M.: High flux passive imaging with single-photon sensors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6760–6769 (2019)Google Scholar
  20. 20.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  21. 21.
    Kirmani, A., et al.: First-photon imaging. Science 343(6166), 58–61 (2014)CrossRefGoogle Scholar
  22. 22.
    Köllner, M., Wolfrum, J.: How many photons are necessary for fluorescence-lifetime measurements? Chem. Phys. Lett. 200(1–2), 199–204 (1992)CrossRefGoogle Scholar
  23. 23.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  24. 24.
    Li, Z.P., et al.: Single-photon computational 3D imaging at 45 km. Photon. Res. 8(9), 1532–1540 (2020)CrossRefGoogle Scholar
  25. 25.
    Li, Z.P., et al.: All-time single-photon 3D imaging over 21 km. In: Conference on Lasers and Electro-Optics, p. SM1N.1 (2019)Google Scholar
  26. 26.
    Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37(4), 113 (2018)CrossRefGoogle Scholar
  27. 27.
    Liu, P., Chang, S., Huang, X., Tang, J., Cheung, J.C.K.: Contextualized non-local neural networks for sequence learning. In: Association for the Advancement of Artificial Intelligence, pp. 6762–6769 (2019)Google Scholar
  28. 28.
    Liu, X., et al.: Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572(7771), 620–623 (2019)CrossRefGoogle Scholar
  29. 29.
    O’Toole, M., Heide, F., Lindell, D.B., Zang, K., Diamond, S., Wetzstein, G.: Reconstructing transient images from single-photon sensors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1539–1547 (2017)Google Scholar
  30. 30.
    O’Toole, M., Lindell, D.B., Wetzstein, G.: Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555(7696), 338 (2018)CrossRefGoogle Scholar
  31. 31.
    Pawlikowska, A.M., Halimi, A., Lamb, R.A., Buller, G.S.: Single-photon three-dimensional imaging at up to 10 kilometers range. Opt. Express 25(10), 11919–11931 (2017)CrossRefGoogle Scholar
  32. 32.
    Pediredla, A.K., Sankaranarayanan, A.C., Buttafava, M., Tosi, A., Veeraraghavan, A.: Signal processing based pile-up compensation for gated single-photon avalanche diodes. arXiv preprint arXiv:1806.07437 (2018)
  33. 33.
    Peng, J., Xiong, Z., Liu, D., Chen, X.: Unsupervised depth estimation from light field using a convolutional neural network. In: International Conference on 3D Vision, pp. 295–303 (2018)Google Scholar
  34. 34.
    Peng, J., Xiong, Z., Wang, Y., Zhang, Y., Liu, D.: Zero-shot depth estimation from light field using a convolutional neural network. IEEE Trans. Comput. Imaging 6, 682–696 (2020)CrossRefGoogle Scholar
  35. 35.
    Rapp, J., Goyal, V.K.: A few photons among many: unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Comput. Imaging 3(3), 445–459 (2017)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Ren, X., et al.: High-resolution depth profiling using a range-gated CMOS SPAD quanta image sensor. Opt. Express 26(5), 5541–5557 (2018)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Renker, D.: Geiger-mode avalanche photodiodes, history, properties and problems. Nucl. Instrum. Methods Phys. Res. 567(1), 48–56 (2006)CrossRefGoogle Scholar
  38. 38.
    Richardson, J.A., Grant, L.A., Henderson, R.K.: Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology. IEEE Photon. Technol. Lett. 21(14), 1020–1022 (2009)CrossRefGoogle Scholar
  39. 39.
    Saunders, C., Murray-Bruce, J., Goyal, V.K.: Computational periscopy with an ordinary digital camera. Nature 565(7740), 472 (2019)CrossRefGoogle Scholar
  40. 40.
    Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  41. 41.
    Schwartz, D.E., Charbon, E., Shepard, K.L.: A single-photon avalanche diode array for fluorescence lifetime imaging microscopy. IEEE J. Solid-State Circuits 43(11), 2546–2557 (2008)CrossRefGoogle Scholar
  42. 42.
    Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2018)Google Scholar
  43. 43.
    Shin, D., Kirmani, A., Goyal, V.K., Shapiro, J.H.: Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors. IEEE Trans. Comput. Imaging 1(2), 112–125 (2015)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Shin, D., et al.: Photon-efficient imaging with a single-photon camera. Nat. Commun. 7, 12046 (2016)CrossRefGoogle Scholar
  45. 45.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). Scholar
  46. 46.
    Villa, F., et al.: CMOS imager with 1024 SPADs and TDCs for single-photon timing and 3D time-of-flight. IEEE J. Sel. Top. Quantum Electron. 20(6), 364–373 (2014)CrossRefGoogle Scholar
  47. 47.
    Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)Google Scholar
  48. 48.
    Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., Wu, F.: HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2017)Google Scholar
  49. 49.
    Yao, M., Xiong, Z., Wang, L., Liu, D., Chen, X.: Spectral-depth imaging with deep learning based reconstruction. Opt. Express 27(26), 38312–38325 (2019)CrossRefGoogle Scholar
  50. 50.
    Yue, K., Sun, M., Yuan, Y., Zhou, F., Ding, E., Xu, F.: Compact generalized non-local network. In: International Conference on Neural Information Processing Systems, pp. 6510–6519 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Science and Technology of ChinaHefeiChina

Personalised recommendations