Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and Its Application for Human Detection

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)


We introduce a lensless imaging framework for contemporary computer vision applications in long-wavelength infrared (LWIR). The framework consists of two parts: a novel lensless imaging method that utilizes the idea of local directional focusing for optimal binary sparse coding, and lensless imaging simulator based on Fresnel-Kirchhoff diffraction approximation. Our lensless imaging approach, besides being computationally efficient, is calibration-free and allows for wide FOV imaging. We employ our lensless imaging simulation software for optimizing reconstruction parameters and for synthetic image generation for CNN training. We demonstrate the advantages of our framework on a dual-camera system (RGB-LWIR lensless), where we perform CNN-based human detection using the fused RGB-LWIR data.


Lensless imaging Long-wave infrared (LWIR) imaging Diffractive optics Image reconstruction Diffraction simulation Pedestrian detection Human detection Visible-infrared image fusion Faster R-CNNs 

Supplementary material

504475_1_En_15_MOESM1_ESM.pdf (307 kb)
Supplementary material 1 (pdf 306 KB)

Supplementary material 2 (mp4 27273 KB)


  1. 1.
    Antipa, N., et al.: DiffuserCam: lensless single-exposure 3D imaging. Optica 5(1), 1 (2018)CrossRefGoogle Scholar
  2. 2.
    Asif, M.S., Ayremlou, A., Sankaranarayanan, A., Veeraraghavan, A., Baraniuk, R.G.: FlatCam: thin, lensless cameras using coded aperture and computation. IEEE Trans. Comput. Imaging 3(3), 384–397 (2016)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Barrera Campo, F., Lumbreras Ruiz, F., Sappa, A.D.: Multimodal stereo vision system: 3D data extraction and algorithm evaluation. IEEE J. Sel. Top. Signal Process. 6(5), 437–446 (2012)CrossRefGoogle Scholar
  4. 4.
    Barrett, H.H.: Fresnel zone plate imaging in nuclear medicine. J. Nucl. Med. 13(6), 382–385 (1972)Google Scholar
  5. 5.
    Benenson, R., Omran, M., Hosang, J., Schiele, B.: Ten years of pedestrian detection, what have we learned? In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8926, pp. 613–627. Springer, Cham (2015). Scholar
  6. 6.
    Born, M., Wolf, E.: Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light, 7th edn. Cambridge University Press, Cambridge (1999)zbMATHCrossRefGoogle Scholar
  7. 7.
    Busboom, A., Elders-Boll, H., Schotten, H.: Uniformly redundant arrays. Exp. Astron. 8, 97–123 (1998). Scholar
  8. 8.
    Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: Advances in Neural Information Processing Systems, pp. 3089–3097 (2016)Google Scholar
  9. 9.
    Chang, J., Wetzstein, G.: Deep optics for monocular depth estimation and 3D object detection. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 10192–10201 (2019)Google Scholar
  10. 10.
    Chu, Y.S., et al.: Hard-X-ray microscopy with Fresnel zone plates reaches 40 nm Rayleigh resolution. Appl. Phys. Lett. 92(10), 103119 (2008)CrossRefGoogle Scholar
  11. 11.
    Cieślak, M.J., Gamage, K.A., Glover, R.: Coded-aperture imaging systems: past, present and future development - a review. Radiat. Meas. 92, 59–71 (2016)CrossRefGoogle Scholar
  12. 12.
    DeWeert, M.J., Farm, B.P.: Lensless coded-aperture imaging with separable Doubly-Toeplitz masks. Opt. Eng. 54(2), 023102 (2015)CrossRefGoogle Scholar
  13. 13.
    Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2012)CrossRefGoogle Scholar
  14. 14.
    Fenimore, E.E., Cannon, T.M.: Coded aperture imaging with uniformly redundant arrays. Appl. Opt. 17(3), 337 (1978)CrossRefGoogle Scholar
  15. 15.
    Fenimore, E.E., Cannon, T.M.: Uniformly redundant arrays: digital reconstruction methods. Appl. Opt. 20(10), 1858 (1981)CrossRefGoogle Scholar
  16. 16.
    Fergus, R., Torralba, A., Freeman, W.T.: Random Lens Imaging. MIT-CSAIL-TR-2006-058, September 2006Google Scholar
  17. 17.
    Gade, R., Moeslund, T.B.: Thermal cameras and applications: a survey. Mach. Vision Appl. 25(1), 245–262 (2014). Scholar
  18. 18.
    Gill, P.R., et al.: Thermal Escher sensors: pixel-efficient lensless imagers based on tiled optics. In: Optics InfoBase Conference Papers, vol. Part F46-COSI 2017, p. CTu3B.3. OSA - The Optical Society, June 2017Google Scholar
  19. 19.
    Goodman, J.W.: Introduction to Fourier Optics, 3rd edn. Roberts, Greenwood Village (2005)Google Scholar
  20. 20.
    Grulois, T., Druart, G., Guérineau, N., Crastes, A., Sauer, H., Chavel, P.: Extra-thin infrared camera for low-cost surveillance applications. Opt. Lett. 39(11), 3169 (2014)CrossRefGoogle Scholar
  21. 21.
    Hwang, S., Park, J., Kim, N., Choi, Y., Kweon, I.S.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), 07–12 June 2015, pp. 1037–1045. IEEE Computer Society, October 2015Google Scholar
  22. 22.
    in’t Zand, J.: A coded-mask imager as monitor of Galactic X-ray sources. Ph.D. thesis, Space Research Organization Netherlands, Sorbonnelaan 2, 3584 CA Utrecht, The Netherlands (1992)Google Scholar
  23. 23.
    Karasawa, T., Watanabe, K., Ha, Q., Tejero-De-Pablos, A., Ushiku, Y., Harada, T.: Multispectral object detection for autonomous vehicles. In: Proceedings of Thematic Workshops of ACM Multimedia, pp. 35–43. Association for Computing Machinery Inc, New York, October 2017Google Scholar
  24. 24.
    Khan, S.S., Adarsh, V.R., Boominathan, V., Tan, J., Veeraraghavan, A., Mitra, K.: Towards photorealistic reconstruction of highly multiplexed lensless images. In: Proceddings of International Conference on Computer Vision (ICCV), pp. 7859–7868. IEEE, October 2019Google Scholar
  25. 25.
    Kim, G., Isaacson, K., Palmer, R., Menon, R.: Lensless photography with only an image sensor. Appl. Opt. 56(23), 6450–6456 (2017)CrossRefGoogle Scholar
  26. 26.
    Kirz, J.: Phase zone plates for X rays and the extreme UV. J. Opt. Soc. Am. 64(3), 301–309 (1974)CrossRefGoogle Scholar
  27. 27.
    Konig, D., Adam, M., Jarvers, C., Layher, G., Neumann, H., Teutsch, M.: Fully convolutional region proposal networks for multispectral person detection. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), July 2017, pp. 243–250. IEEE Computer Society, August 2017Google Scholar
  28. 28.
    Leykin, A., Ran, Y., Hammoud, R.: Thermal-visible video fusion for moving target tracking and pedestrian classification. In: Proceedings of Computer Vision and Pattern Recognition (CVPR) (2007)Google Scholar
  29. 29.
    Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networks for pedestrian detection. In: British Machine Vision Conference (BMVC), September 2016, pp. 73.1–73.13 (2016)Google Scholar
  30. 30.
    Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), August 2020Google Scholar
  31. 31.
    Miezianko, R., Pokrajac, D.: People detection in low resolution infrared videos. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2008)Google Scholar
  32. 32.
    Mudau, A.E., Willers, C.J., Griffith, D., Le Roux, F.P.: Non-uniformity correction and bad pixel replacement on LWIR and MWIR images. In: Saudi International Electronics, Communications and Photonics Conference (SIECPC) (2011)Google Scholar
  33. 33.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)CrossRefGoogle Scholar
  34. 34.
    Sitzmann, V., et al.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37(4), 1–13 (2018) CrossRefGoogle Scholar
  35. 35.
    Stylianou, A., Pless, R.: SparkleGeometry: glitter imaging for 3D point tracking. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 919–926. IEEE Computer Society, December 2016Google Scholar
  36. 36.
    Sun, Q., Zhang, J., Dun, X., Ghanem, B., Peng, Y., Heidrich, W.: End-to-end learned, optically coded super-resolution SPAD camera. ACM Trans. Graph. 39(2), 1–14 (2020)CrossRefGoogle Scholar
  37. 37.
    Tanida, J., et al.: Thin observation module by bound optics (TOMBO): concept and experimental verification. Appl. Opt. 40(11), 1806 (2001)CrossRefGoogle Scholar
  38. 38.
    Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. W.H. Winston, Washington (1977) zbMATHGoogle Scholar
  39. 39.
    Wagner, J., Fischer, V., Herman, M., Behnke, S.: Multispectral pedestrian detection using deep fusion convolutional neural networks. In: Proceedings of European Symposium on Artificial Neural Networks (ESANN) (2016)Google Scholar
  40. 40.
    Wang, Z., Simoncelli, E.P.: Translation insensitive image similarity in complex wavelet domain. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. II (2005)Google Scholar
  41. 41.
    Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A., Veeraraghavan, A.: PhaseCam3D - learning phase masks for passive single view depth estimation (2019)Google Scholar
  42. 42.
    Xu, D., Ouyang, W., Ricci, E., Wang, X., Sebe, N.: Learning cross-modal deep representations for robust pedestrian detection. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), January 2017, pp. 4236–4244. Institute of Electrical and Electronics Engineers Inc., November 2017Google Scholar
  43. 43.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Sony CorporationTokyoJapan

Personalised recommendations