Advertisement

Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery

  • Ying Fu
  • Tao Zhang
  • Yinqiang Zheng
  • Debing Zhang
  • Hua HuangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)

Abstract

Hyperspectral image (HSI) recovery from a single RGB image has attracted much attention, whose performance has recently been shown to be sensitive to the camera spectral sensitivity (CSS). In this paper, we present an efficient convolutional neural network (CNN) based method, which can jointly select the optimal CSS from a candidate dataset and learn a mapping to recover HSI from a single RGB image captured with this algorithmically selected camera. Given a specific CSS, we first present a HSI recovery network, which accounts for the underlying characteristics of the HSI, including spectral nonlinear mapping and spatial similarity. Later, we append a CSS selection layer onto the recovery network, and the optimal CSS can thus be automatically determined from the network weights under the nonnegative sparse constraint. Experimental results show that our HSI recovery network outperforms state-of-the-art methods in terms of both quantitative metrics and perceptive quality, and the selection layer always returns a CSS consistent to the best one determined by exhaustive search.

Keywords

Camera spectral sensitivity selection Hyperspectral image recovery Spectral nonlinear mapping Spatial similarity 

Notes

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants No. 61425013 and No. 61672096.

References

  1. 1.
    Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 63–78. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_5CrossRefGoogle Scholar
  2. 2.
    Aly, H.A., Sharma, G.: A regularized model-based optimization framework for pan-sharpening. IEEE Trans. Image Process. 23(6), 2596–2608 (2014)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Arad, B., Ben-Shahar, O.: Sparse recovery of hyperspectral signal from natural RGB images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 19–34. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46478-7_2CrossRefGoogle Scholar
  4. 4.
    Arad, B., Ben-Shahar, O.: Filter selection for hyperspectral estimation. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 3172–3180, October 2017Google Scholar
  5. 5.
    Basedow, R.W., Carmer, D.C., Anderson, M.E.: HYDICE system: implementation and performance. In: SPIE’s Symposium on OE/Aerospace Sensing and Dual Use Photonics, pp. 258–267 (1995)Google Scholar
  6. 6.
    Bioucas-Dias, J.M., Plaza, A., Camps-Valls, G., Scheunders, P., Nasrabadi, N.M., Chanussot, J.: Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci. Remote Sens. Mag. 1(2), 6–36 (2013)CrossRefGoogle Scholar
  7. 7.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 60–65, June 2005Google Scholar
  8. 8.
    Cao, X., Du, H., Tong, X., Dai, Q., Lin, S.: A prism-based system for multispectral bideo acquisition. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 33(12), 2423–2435 (2011)CrossRefGoogle Scholar
  9. 9.
    Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 193–200, June 2011Google Scholar
  10. 10.
    Chi, C., Yoo, H., Ben-Ezra, M.: Multi-spectral imaging by optimized wide band illumination. Int. J. Comput. Vis. (IJCV) 86(2–3), 140–151 (2010)CrossRefGoogle Scholar
  11. 11.
    Dian, R., Fang, L., Li, S.: Hyperspectral image super-resolution via non-local sparse tensor factorization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5344–5353, June 2017Google Scholar
  12. 12.
    Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Dong, W., Zhang, L., Shi, G.: Centralized sparse representation for image restoration. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 1259–1266, November 2011Google Scholar
  14. 14.
    Ford, B.K., Descour, M.R., Lynch, R.M.: Large-image-format computed tomography imaging spectrometer for fluorescence microscopy. Opt. Express 9(9), 444–453 (2001)CrossRefGoogle Scholar
  15. 15.
    Gao, L., Kester, R.T., Hagen, N., Tkaczyk, T.S.: Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy. Opt. Express 18(14), 14330–14344 (2010)CrossRefGoogle Scholar
  16. 16.
    Gat, N., Scriven, G., Garman, J., Li, M.D., Zhang, J.: Development of four-dimensional imaging spectrometers (4D-IS). In: Proceeding of SPIE Optics + Photonics, vol. 6302, pp. 63020M–63020M-11 (2006)Google Scholar
  17. 17.
    Gehm, M.E., John, R., Brady, D.J., Willett, R.M., Schulz, T.J.: Single-shot compressive spectral imaging with a dual-disperser architecture. Opt. Express 15(21), 14013–27 (2007)CrossRefGoogle Scholar
  18. 18.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 249–256, May 2010Google Scholar
  19. 19.
    Han, S., Sato, I., Okabe, T., Sato, Y.: Fast spectral reflectance recovery using DLP projector. Int. J. Comput. Vis. (IJCV) 110(2), 172–184 (2014)CrossRefGoogle Scholar
  20. 20.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_38CrossRefGoogle Scholar
  21. 21.
    Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269, July 2017Google Scholar
  22. 22.
    Jia, Y., et al.: From RGB to spectrum for natural scenes via manifold-based mapping. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 4715–4723, October 2017Google Scholar
  23. 23.
    Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of ACM Multimedia Conference (MM), pp. 675–678, November 2014Google Scholar
  24. 24.
    Jiang, J., Liu, D., Gu, J., Ssstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: IEEE Workshop on Applications of Computer Vision (WACV), pp. 168–179 (2013)Google Scholar
  25. 25.
    Kawakami, R., Wright, J., Tai, Y.W., Matsushita, Y., Ben-Ezra, M., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2329–2336, June 2011Google Scholar
  26. 26.
    Kawakami, R., Zhao, H., Tan, R.T., Ikeuchi, K.: Camera spectral sensitivity and white balance estimation from sky images. Int. J. Comput. Vis. (IJCV) 105(3), 187–204 (2013)CrossRefGoogle Scholar
  27. 27.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654, June 2016Google Scholar
  28. 28.
    Kim, S.J., Deng, F., Brown, M.S.: Visual enhancement of old documents with hyperspectral imaging. Pattern Recogn. 44(7), 1461–1469 (2011)CrossRefGoogle Scholar
  29. 29.
    Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: Proceedings of International Conference on Learning Representations (ICLR), May 2015Google Scholar
  30. 30.
    Kruse, F.A., et al.: The spectral image processing system (SIPS)-interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44(2–3), 145–163 (1993)CrossRefGoogle Scholar
  31. 31.
    Kwon, H., Tai, Y.W.: RGB-guided hyperspectral image upsampling. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 307–315, December 2015Google Scholar
  32. 32.
    Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 3586–3594, December 2015Google Scholar
  33. 33.
    Lin, X., Liu, Y., Wu, J., Dai, Q.: Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph. 33(6), 233:1–233:11 (2014). (Proceedings of SIGGRAPH Asia)CrossRefGoogle Scholar
  34. 34.
    Loffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of International Conference on Machine Learning (ICML), pp. 448–456, June 2015Google Scholar
  35. 35.
    Ma, C., Cao, X., Tong, X., Dai, Q., Lin, S.: Acquisition of high spatial and spectral resolution video with a hybrid camera system. Int. J. Comput. Vis. (IJCV) 110(2), 141–155 (2014)CrossRefGoogle Scholar
  36. 36.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of International Conference on Machine Learning (ICML), pp. 807–814, June 2010Google Scholar
  37. 37.
    Nguyen, H.V., Banerjee, A., Chellappa, R.: Tracking via object reflectance using a hyperspectral video camera. In: IEEE Conference on Computer Vision and Pattern Recognition - Workshops, pp. 44–51, June 2010Google Scholar
  38. 38.
    Nguyen, R.M.H., Prasad, D.K., Brown, M.S.: Training-based spectral reconstruction from a single RGB image. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 186–201. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_13CrossRefGoogle Scholar
  39. 39.
    Pan, Z., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 25(12), 1552–1560 (2003)CrossRefGoogle Scholar
  40. 40.
    Park, J.I., Lee, M.H., Grossberg, M.D., Nayar, S.K.: Multispectral imaging using multiplexed illumination. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 1–8, October 2007Google Scholar
  41. 41.
    Porter, W.M., Enmark, H.T.: A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In: Annual Technical Symposium, pp. 22–31 (1987)Google Scholar
  42. 42.
    Robles-Kelly, A.: Single image spectral reconstruction for multimedia applications. In: Proceedings of ACM Multimedia Conference (MM), pp. 251–260, October 2015Google Scholar
  43. 43.
    Tarabalka, Y., Chanussot, J., Benediktsson, J.A.: Segmentation and classification of hyperspectral images using watershed transformation. Pattern Recogn. 43(7), 2367–2379 (2010)CrossRefGoogle Scholar
  44. 44.
    Wagadarikar, A., John, R., Willett, R., Brady, D.: Single disperser design for coded aperture snapshot spectral imaging. Appl. Opt. 47(10), 44–51 (2008)CrossRefGoogle Scholar
  45. 45.
    Wang, L., Xiong, Z., Gao, D., Shi, G., Wu, F.: Dual-camera design for coded aperture snapshot spectral imaging. Appl. Opt. 54(4), 848–858 (2015)CrossRefGoogle Scholar
  46. 46.
    Wang, L., Xiong, Z., Shi, G., Wu, F., Zeng, W.: Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 39(10), 2104–2111 (2017)CrossRefGoogle Scholar
  47. 47.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  48. 48.
    Wu, D., Sun, D.W.: Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: a reviewpart I: fundamentals. Innov. Food Sci. Emerg. Technol. 19, 1–14 (2013)CrossRefGoogle Scholar
  49. 49.
    Xu, X., Li, J., Huang, X., Dalla Mura, M., Plaza, A.: Multiple morphological component analysis based decomposition for remote sensing image classification. IEEE Trans. Geosci. Remote Sens. 54(5), 3083–3102 (2016)CrossRefGoogle Scholar
  50. 50.
    Xu, X., Wu, Z., Li, J., Plaza, A., Wei, Z.: Anomaly detection in hyperspectral images based on low-rank and sparse representation. IEEE Trans. Geosci. Remote Sens. 54(4), 1990–2000 (2016)CrossRefGoogle Scholar
  51. 51.
    Yamaguchi, M., et al.: High-fidelity video and still-image communication based on spectral information: natural vision system and its applications. In: Electronic Imaging, pp. 60620G–60620G-12 (2006)Google Scholar
  52. 52.
    Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., John, P.: PanNet: a deep network architecture for pan-sharpening. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 1753–1761, October 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Beijing Laboratory of Intelligent Information Technology, School of Computer Science and TechnologyBeijing Institute of TechnologyBeijingChina
  2. 2.National Institute of InformaticsTokyoJapan
  3. 3.DeepGlintBeijingChina

Personalised recommendations