Advertisement

Deep Residual Attention Network for Spectral Image Super-Resolution

  • Zhan Shi
  • Chang Chen
  • Zhiwei XiongEmail author
  • Dong Liu
  • Zheng-Jun Zha
  • Feng Wu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11133)

Abstract

Spectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be retained. In this paper, we propose a novel deep residual attention network for the spatial super-resolution (SR) of spectral images. The proposed method extends the classic residual network by (1) directly using the 3D low-resolution (LR) spectral image as input instead of upsampling the 2D bandwise images separately, and (2) integrating the channel attention mechanism into the residual network. These two operations fully exploit the correlations across both the spectral and spatial dimensions and greatly promote the performance of spectral image SR. In addition, for the scenario when stereo pairs of LR spectral and high-resolution (HR) RGB measurements are available, we design a fusion framework based on the proposed network. The spatial resolution of the spectral input is enhanced in one branch, while the spectral resolution of the RGB input is enhanced in the other. These two branches are then fused together through the attention mechanism again to reconstruct the final HR spectral image, which achieves further improvement compared to using the single LR spectral input. Experimental results demonstrate the superiority of the proposed method over plain residual networks, and our method is one of the winning solutions in the PIRM 2018 Spectral Super-resolution Challenge.

Keywords

Spectral image Super-resolution Channel attention 

Notes

Acknowledgments

We acknowledge funding from National Key R&D Program of China under Grant 2017YFA0700800, and Natural Science Foundation of China under Grants 61671419 and 61425026.

References

  1. 1.
    Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 63–78. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_5CrossRefGoogle Scholar
  2. 2.
    Akhtar, N., Shafait, F., Mian, A.: Hierarchical beta process with gaussian process prior for hyperspectral image super resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 103–120. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_7CrossRefGoogle Scholar
  3. 3.
    Basedow, R.W., Carmer, D.C., Anderson, M.E.: HYDICE system: implementation and performance. In: Proceedings of SPIE (1995)Google Scholar
  4. 4.
    Bluche, T.: Joint line segmentation and transcription for end-to-end handwritten paragraph recognition. In: NIPS (2016)Google Scholar
  5. 5.
    Brady, D.J.: Optical Imaging and Spectroscopy. Wiley, Hoboken (2009)CrossRefGoogle Scholar
  6. 6.
    Cao, C., et al.: Look and think twice: capturing top-down visual attention with feedback convolutional neural networks. In: ICCV (2015)Google Scholar
  7. 7.
    Cao, X., Du, H., Tong, X., Dai, Q., Lin, S.: A prism-mask system for multispectral video acquisition. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2423–35 (2011)CrossRefGoogle Scholar
  8. 8.
    Chang, C.I.: Spectral information divergence for hyperspectral image analysis. In: IGARSS (1999)Google Scholar
  9. 9.
    Chen, C., Tian, X., Xiong, Z., Wu, F.: UDNET: up-down network for compact and efficient feature representation in image super-resolution. In: ICCVW (2017)Google Scholar
  10. 10.
    Descour, M., Dereniak, E.: Computed-tomography imaging spectrometer: experimental calibration and reconstruction results. Appl. Opt. 34(22), 4817–4826 (1995)CrossRefGoogle Scholar
  11. 11.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  12. 12.
    Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Fang, L., Zhuo, H., Li, S.: Super-resolution of hyperspectral image via superpixel-based sparse representation. Neurocomputing 273, 171–177 (2018)CrossRefGoogle Scholar
  14. 14.
    Gat, N.: Imaging spectroscopy using tunable filters: a review. In: Proceedings of SPIE (2000)Google Scholar
  15. 15.
    Goel, M., et al.: HyperCam: hyperspectral imaging for ubiquitous computing applications. In: UbiComp (2015)Google Scholar
  16. 16.
    Goetz, A.F., Vane, G., Solomon, J.E., Rock, B.N.: Imaging spectrometry for earth remote sensing. Science 228(4704), 1147–1153 (1985)CrossRefGoogle Scholar
  17. 17.
    Gowen, A., O’Donnell, C., Cullen, P., Downey, G., Frias, J.: Hyperspectral imaging–an emerging process analytical tool for food quality and safety control. Trends Food Sci. Technol. 18(12), 590–598 (2007)CrossRefGoogle Scholar
  18. 18.
    Haboudane, D., Miller, J.R., Pattey, E., Zarco-Tejada, P.J., Strachan, I.B.: Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: modeling and validation in the context of precision agriculture. Remote Sens. Environ. 90(3), 337–352 (2004)CrossRefGoogle Scholar
  19. 19.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  20. 20.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR (2018)Google Scholar
  21. 21.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)Google Scholar
  22. 22.
    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems (2015)Google Scholar
  23. 23.
    Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.W., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: CVPR (2011)Google Scholar
  24. 24.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)Google Scholar
  25. 25.
    Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: ICCV (2015)Google Scholar
  26. 26.
    Li, S., Yang, B.: A new pan-sharpening method using a compressed sensing technique. IEEE Trans. Geosci. Remote Sens. 49(2), 738–746 (2011)CrossRefGoogle Scholar
  27. 27.
    Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266, 29–41 (2017)CrossRefGoogle Scholar
  28. 28.
    Liebel, L., Körner, M.: Single-image super resolution for multispectral remote sensing data using convolutional neural networks. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 41, 883–890 (2016)CrossRefGoogle Scholar
  29. 29.
    Lin, X., Liu, Y., Wu, J., Dai, Q.: Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph 33(6), 233 (2014)CrossRefGoogle Scholar
  30. 30.
    Ma, C., Cao, X., Tong, X., Dai, Q., Lin, S.: Acquisition of high spatial and spectral resolution video with a hybrid camera system. Int. J. Comput. Vision 110(2), 141–155 (2014)CrossRefGoogle Scholar
  31. 31.
    Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., Du, Q.: Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens. 9(11), 1139 (2017)CrossRefGoogle Scholar
  32. 32.
    Miech, A., Laptev, I., Sivic, J.: Learnable pooling with context gating for video classification. arXiv preprint arXiv:1706.06905 (2017)
  33. 33.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: ICML (2010)Google Scholar
  34. 34.
    Pan, Z., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003)CrossRefGoogle Scholar
  35. 35.
    Porter, W.M., Enmark, H.T.: A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In: Proceedings of SPIE (1987)Google Scholar
  36. 36.
    Rahmani, S., Strait, M., Merkurjev, D., Moeller, M., Wittman, T.: An adaptive IHS pan-sharpening method. IEEE Geosci. Remote Sens. Lett. 7(4), 746–750 (2010)CrossRefGoogle Scholar
  37. 37.
    Schechner, Y., Nayar, S.: Generalized mosaicing: wide field of view multispectral imaging. IEEE Trans. Pattern Anal. Mach. Intell. 24(10), 1334–1348 (2002)CrossRefGoogle Scholar
  38. 38.
    Shah, V.P., Younan, N.H., King, R.L.: An efficient pan-sharpening method via a combined adaptive pca approach and contourlets. IEEE Trans. Geosci. Remote Sens. 46(5), 1323–1335 (2008)CrossRefGoogle Scholar
  39. 39.
    Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In: CVPRW (2018)Google Scholar
  40. 40.
    Shoeiby, M., et al.: PIRM2018 challenge on spectral image super-resolution: methods and results. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018 Workshops. LNCS, vol. 11133, pp. 356–371. Springer, Cham (2018)Google Scholar
  41. 41.
    Shoeiby, M., Robles-Kelly, A., Wei, R., Timofte, R.: PIRM2018 challenge on spectral image super-resolution: dataset and study. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018 Workshops. LNCS, vol. 11133, pp. 276–287. Springer, Cham (2018)Google Scholar
  42. 42.
    Simões, M., Bioucas-Dias, J., Almeida, L.B., Chanussot, J.: A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans. Geosci. Remote Sens. 53(6), 3373–3388 (2015)CrossRefGoogle Scholar
  43. 43.
    Tarabalka, Y., Chanussot, J., Benediktsson, J.A.: Segmentation and classification of hyperspectral images using watershed transformation. Pattern Recognit. 43(7), 2367–2379 (2010)CrossRefGoogle Scholar
  44. 44.
    Van Nguyen, H., Banerjee, A., Chellappa, R.: Tracking via object reflectance using a hyperspectral video camera. In: CVPRW (2010)Google Scholar
  45. 45.
    Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)Google Scholar
  46. 46.
    Veganzones, M.A., Simoes, M., Licciardi, G., Yokoya, N., Bioucas-Dias, J.M., Chanussot, J.: Hyperspectral super-resolution of locally low rank images from complementary multisource data. IEEE Trans. Image Process. 25(1), 274–288 (2016)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Wagadarikar, A., John, R., Willett, R., Brady, D.: Single disperser design for coded aperture snapshot spectral imaging. Appl. Opt. 47(10), B44–B51 (2008)CrossRefGoogle Scholar
  48. 48.
    Wang, L., Xiong, Z., Gao, D., Shi, G., Zeng, W., Wu, F.: High-speed hyperspectral video acquisition with a dual-camera architecture. In: CVPR (2015)Google Scholar
  49. 49.
    Wang, L., Xiong, Z., Shi, G., Wu, F., Zeng, W.: Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans. Pattern Anal. Mach. Intell. 39(10), 2104–2011 (2017)CrossRefGoogle Scholar
  50. 50.
    Wang, L., Xiong, Z., Gao, D., Shi, G., Wu, F.: Dual-camera design for coded aperture snapshot spectral imaging. Appl. Opt. 54(4), 848–858 (2015)CrossRefGoogle Scholar
  51. 51.
    Wug Oh, S., Brown, M.S., Pollefeys, M., Joo Kim, S.: Do it yourself hyperspectral imaging with everyday digital cameras. In: CVPR (2016)Google Scholar
  52. 52.
    Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., Wu, F.: HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In: ICCVW (2017)Google Scholar
  53. 53.
    Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2012)CrossRefGoogle Scholar
  54. 54.
    Yuan, Y., Zheng, X., Lu, X.: Hyperspectral image superresolution by transfer learning. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 10(5), 1963–1974 (2017)CrossRefGoogle Scholar
  55. 55.
    Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_18CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Zhan Shi
    • 1
  • Chang Chen
    • 1
  • Zhiwei Xiong
    • 1
    Email author
  • Dong Liu
    • 1
  • Zheng-Jun Zha
    • 1
  • Feng Wu
    • 1
  1. 1.University of Science and Technology of ChinaHefeiChina

Personalised recommendations