Advertisement

Accurate Spectral Super-Resolution from Single RGB Image Using Multi-scale CNN

  • Yiqi Yan
  • Lei Zhang
  • Jun Li
  • Wei Wei
  • Yanning Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11257)

Abstract

Different from traditional hyperspectral super-resolution approaches that focus on improving the spatial resolution, spectral super-resolution aims at producing a high-resolution hyperspectral image from the RGB observation with super-resolution in spectral domain. However, it is challenging to accurately reconstruct a high-dimensional continuous spectrum from three discrete intensity values at each pixel, since too much information is lost during the procedure where the latent hyperspectral image is downsampled (e.g., with \(\times \)10 scaling factor) in spectral domain to produce an RGB observation. To address this problem, we present a multi-scale deep convolutional neural network (CNN) to explicitly map the input RGB image into a hyperspectral image. Through symmetrically downsampling and upsampling the intermediate feature maps in a cascading paradigm, the local and non-local image information can be jointly encoded for spectral representation, ultimately improving the spectral reconstruction accuracy. Extensive experiments on a large hyperspectral dataset demonstrate the effectiveness of the proposed method.

Keywords

Hyperspectral imaging Spectral super-resolution Multi-scale analysis Convolutional neural networks 

Notes

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China (No. 61671385, 61571354), Natural Science Basis Research Plan in Shaanxi Province of China (No. 2017JM6021, 2017JM6001) and China Postdoctoral Science Foundation under Grant (No. 158201).

References

  1. 1.
    NTIRE 2018 challenge on spectral reconstruction from RGB images. http://www.vision.ee.ethz.ch/ntire18/
  2. 2.
    Aeschbacher, J., Wu, J., Timofte, R.: In defense of shallow learned spectral reconstruction from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 471–479 (2017)Google Scholar
  3. 3.
    Aharon, M., Elad, M., Bruckstein, A.: \( rm k \)-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)CrossRefGoogle Scholar
  4. 4.
    Arad, B., Ben-Shahar, O.: Sparse recovery of hyperspectral signal from natural RGB images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 19–34. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46478-7_2CrossRefGoogle Scholar
  5. 5.
    Cheng, G., Yang, C., Yao, X., Guo, L., Han, J.: When deep learning meets metric learning: remote sensing image scene classification via learning discriminative CNNs. IEEE Trans. Geosci. Remote. Sens. 56, 2811–2821 (2018)CrossRefGoogle Scholar
  6. 6.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  7. 7.
    Galliani, S., Lanaras, C., Marmanis, D., Baltsavias, E., Schindler, K.: Learned spectral super-resolution. CoRR abs/1703.09470 (2017). http://arxiv.org/abs/1703.09470
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  9. 9.
    Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1175–1183. IEEE (2017)Google Scholar
  10. 10.
    Kang, X., Zhang, X., Li, S., Li, K., Li, J., Benediktsson, J.A.: Hyperspectral anomaly detection with attribute and edge-preserving filters. IEEE Trans. Geosci. Remote. Sens. 55(10), 5600–5611 (2017)CrossRefGoogle Scholar
  11. 11.
    Loncan, L., et al.: Hyperspectral pansharpening: a review. IEEE Geosci. Remote. Sens. Mag. 3(3), 27–46 (2015)CrossRefGoogle Scholar
  12. 12.
    Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., Du, Q.: Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens. 9(11), 1139 (2017)CrossRefGoogle Scholar
  13. 13.
    Nguyen, R.M.H., Prasad, D.K., Brown, M.S.: Training-based spectral reconstruction from a single RGB image. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 186–201. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_13CrossRefGoogle Scholar
  14. 14.
    Pan, Z., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003)CrossRefGoogle Scholar
  15. 15.
    Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: 1993 Conference Record of The Twenty-Seventh Asilomar Conference on Signals, Systems and Computers, pp. 40–44. IEEE (1993)Google Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  17. 17.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)Google Scholar
  18. 18.
    Tarabalka, Y., Chanussot, J., Benediktsson, J.A.: Segmentation and classification of hyperspectral images using watershed transformation. Pattern Recogn. 43(7), 2367–2379 (2010)CrossRefGoogle Scholar
  19. 19.
    Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-16817-3_8CrossRefGoogle Scholar
  20. 20.
    Van Nguyen, H., Banerjee, A., Chellappa, R.: Tracking via object reflectance using a hyperspectral video camera. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 44–51. IEEE (2010)Google Scholar
  21. 21.
    Yan, Q., Sun, J., Li, H., Zhu, Y., Zhang, Y.: High dynamic range imaging by sparse representation. Neurocomputing 269, 160–169 (2017)CrossRefGoogle Scholar
  22. 22.
    Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote. Sens. Mag. 5(2), 29–56 (2017)CrossRefGoogle Scholar
  23. 23.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Zhang, L., et al.: Adaptive importance learning for improving lightweight image super-resolution network. arXiv preprint arXiv:1806.01576 (2018)
  25. 25.
    Zhang, L., Wei, W., Bai, C., Gao, Y., Zhang, Y.: Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Trans. Image Process. 27, 5969–5982 (2018)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Zhang, L., Wei, W., Shi, Q., Shen, C., Hengel, A.v.d., Zhang, Y.: Beyond low rank: a data-adaptive tensor completion method. arXiv preprint arXiv:1708.01008 (2017)
  27. 27.
    Zhang, L., Wei, W., Zhang, Y., Shen, C., van den Hengel, A., Shi, Q.: Cluster sparsity field: an internal hyperspectral imagery prior for reconstruction. Int. J. Comput. Vis. 1–25 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Yiqi Yan
    • 1
  • Lei Zhang
    • 2
  • Jun Li
    • 4
  • Wei Wei
    • 2
    • 3
  • Yanning Zhang
    • 2
    • 3
  1. 1.School of Electronics and InformationNorthwestern Polytechnical UniversityXi’anChina
  2. 2.School of Computer ScienceNorthwestern Polytechnical UniversityXi’anChina
  3. 3.National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application TechnologyXi’anChina
  4. 4.Guangdong Provincial Key Laboratory of Urbanization and Geo-simulation, School of Geography and PlanningSun Yat-Sen UniversityGuangzhouChina

Personalised recommendations