Skip to main content

Hyperspectral Demosaicing of Snapshot Camera Images Using Deep Learning

  • Conference paper
  • First Online:
Pattern Recognition (DAGM GCPR 2022)

Abstract

Spectral imaging technologies have rapidly evolved during the past decades. The recent development of single-camera-one-shot techniques for hyperspectral imaging allows multiple spectral bands to be captured simultaneously (\(3\times 3\), \(4\times 4\) or \(5\times 5\) mosaic), opening up a wide range of applications. Examples include intraoperative imaging, agricultural field inspection and food quality assessment. To capture images across a wide spectrum range, i.e. to achieve high spectral resolution, the sensor design sacrifices spatial resolution. With increasing mosaic size, this effect becomes increasingly detrimental. Furthermore, demosaicing is challenging. Without incorporating edge, shape, and object information during interpolation, chromatic artifacts are likely to appear in the obtained images. Recent approaches use neural networks for demosaicing, enabling direct information extraction from image data. However, obtaining training data for these approaches poses a challenge as well. This work proposes a parallel neural network based demosaicing procedure trained on a new ground truth dataset captured in a controlled environment by a hyperspectral snapshot camera with a \(4\times 4\) mosaic pattern. The dataset is a combination of real captured scenes with images from publicly available data adapted to the \(4\times 4\) mosaic pattern. To obtain real world ground-truth data, we performed multiple camera captures with 1-pixel shifts in order to compose the entire data cube. Experiments show that the proposed network outperforms state-of-art networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bayer, B.: Color imaging array us patent 3 971 065 (1976)

    Google Scholar 

  2. Bendoumi, M.A., He, M., Mei, S.: Hyperspectral image resolution enhancement using high-resolution multispectral image based on spectral unmixing. IEEE Trans. Geosci. Remote Sens. 52(10), 6574–6583 (2014)

    Article  Google Scholar 

  3. Calin, M.A., Parasca, S.V., Savastru, D., Manea, D.: Hyperspectral imaging in the medical field: present and future. Appl. Spectrosc. Rev. 49(6), 435–447 (2014)

    Article  Google Scholar 

  4. Daudkhane, C.: Hyperspectral demosaicing of snapshot camera images using deep learning. Master’s thesis, Technical University Berlin (2021)

    Google Scholar 

  5. Dijkstra, K., van de Loosdrecht, J., Schomaker, L.R.B., Wiering, M.A.: Hyperspectral demosaicking and crosstalk correction using deep learning. Mach. Vision Appl. 30(1), 1–21 (2018). https://doi.org/10.1007/s00138-018-0965-4

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  7. Eismann, M.T., Hardie, R.C.: Application of the stochastic mixing model to hyperspectral resolution enhancement. IEEE Trans. Geosci. Remote Sens. 42(9), 1924–1933 (2004)

    Article  Google Scholar 

  8. Eismann, M.T., Hardie, R.C.: Hyperspectral resolution enhancement using high-resolution multispectral imagery with arbitrary response functions. IEEE Trans. Geosci. Remote Sens. 43(3), 455–465 (2005)

    Article  Google Scholar 

  9. Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (TOG) 35(6), 1–12 (2016)

    Article  Google Scholar 

  10. Habtegebrial, T.A., Reis, G., Stricker, D.: Deep convolutional networks for snapshot hypercpectral demosaicking. In: 2019 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), pp. 1–5. IEEE (2019)

    Google Scholar 

  11. Hardie, R.C., Eismann, M.T., Wilson, G.L.: Map estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Trans. Image Process. 13(9), 1174–1184 (2004)

    Article  Google Scholar 

  12. Hershey, J., Zhang, Z.: Multispectral digital camera employing both visible light and non-visible light sensing on a single image sensor. uS Patent 7,460,160 (2008)

    Google Scholar 

  13. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertainty Fuzziness Knowl.-Based Syst. 6(02), 107–116 (1998)

    Article  Google Scholar 

  14. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)

    Google Scholar 

  15. Jung, A., Kardeván, P., Tőkei, L.: Hyperspectral technology in vegetation analysis. Progr. Agric. Eng. Sci. 2(1), 95–117 (2006)

    Google Scholar 

  16. Khan, H.A., Mihoubi, S., Mathon, B., Thomas, J.B., Hardeberg, J.Y.: Hytexila: high resolution visible and near infrared hyperspectral texture images. Sensors 18(7), 2045 (2018)

    Article  Google Scholar 

  17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  18. Kossack, B., Wisotzky, E., Eisert, P., Schraven, S.P., Globke, B., Hilsmann, A.: Perfusion assessment via local remote photoplethysmography (rppg). In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2192–2201 (2022)

    Google Scholar 

  19. Li, Q., Wang, Q., Li, X.: Mixed 2D/3D convolutional network for hyperspectral image super-resolution. Remote Sens. 12(10), 1660 (2020)

    Article  Google Scholar 

  20. Li, Y., Zhang, L., Dingl, C., Wei, W., Zhang, Y.: Single hyperspectral image super-resolution with grouped deep recursive residual network. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pp. 1–4. IEEE (2018)

    Google Scholar 

  21. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  22. Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19(1), 10901 (2014). https://doi.org/10.1117/1.JBO.19.1.010901

    Article  Google Scholar 

  23. Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., Du, Q.: Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens. 9(11), 1139 (2017)

    Article  Google Scholar 

  24. Mihoubi, S., Losson, O., Mathon, B., Macaire, L.: Multispectral demosaicing using intensity-based spectral correlation. In: International Conference on Image Processing Theory, Tools and Applications (IPTA), vol. 5, pp. 461–466. IEEE (2015). https://doi.org/10.1109/IPTA.2015.7367188, http://ieeexplore.ieee.org/document/7367188/

  25. Moghadam, P., Ward, D., Goan, E., Jayawardena, S., Sikka, P., Hernandez, E.: Plant disease detection using hyperspectral imaging. In: 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8. IEEE (2017)

    Google Scholar 

  26. Monno, Y., Tanaka, M., Okutomi, M.: Multispectral demosaicking using guided filter. In: Digital Photography VIII, vol. 8299, p. 82990O. International Society for Optics and Photonics (2012)

    Google Scholar 

  27. Mühle, R., Markgraf, W., Hilsmann, A., Malberg, H., Eisert, P., Wisotzky, E.L.: Comparison of different spectral cameras for image-guided organ transplantation. J. Biomed. Opt. 26(7), 076007 (2021). https://doi.org/10.1117/1.JBO.26.7.076007

  28. Ogawa, S., et al.: Demosaicking method for multispectral images based on spatial gradient and inter-channel correlation. In: Mansouri, A., Nouboud, F., Chalifour, A., Mammass, D., Meunier, J., ElMoataz, A. (eds.) ICISP 2016. LNCS, vol. 9680, pp. 157–166. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33618-3_17

    Chapter  Google Scholar 

  29. Shafri, H.Z., Taherzadeh, E., Mansor, S., Ashurov, R.: Hyperspectral remote sensing of urban areas: an overview of techniques and applications. Res. J. Appl. Sci. Eng. Technol. 4(11), 1557–1565 (2012)

    Google Scholar 

  30. Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: Hscnn+: advanced CNN-based hyperspectral recovery from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 939–947 (2018)

    Google Scholar 

  31. Stockman, A.: Cone fundamentals and CIE standards. Curr. Opin. Behav. Sci. 30, 87–93 (2019)

    Article  Google Scholar 

  32. Wang, Y.Q.: A multilayer neural network for image demosaicking. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 1852–1856. IEEE (2014)

    Google Scholar 

  33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  34. Wisotzky, E.L., et al.: Validation of two techniques for intraoperative hyperspectral human tissue determination. In: Proceedings of SPIE, vol. 10951, p. 109511Z (2019). https://doi.org/10.1117/12.2512811

  35. Wisotzky, E.L., Kossack, B., Uecker, F.C., Arens, P., Hilsmann, A., Eisert, P.: Validation of two techniques for intraoperative hyperspectral human tissue determination. J. Med. Imaging 7(6) (2020). https://doi.org/10.1117/1.JMI.7.6.065001

  36. Wisotzky, E.L., Uecker, F.C., Arens, P., Dommerich, S., Hilsmann, A., Eisert, P.: Intraoperative hyperspectral determination of human tissue properties. J. Biomed. Opt. 23(9), 1–8 (2018). https://doi.org/10.1117/1.JBO.23.9.091409

    Article  Google Scholar 

  37. Yamanaka, J., Kuwashima, S., Kurita, T.: Fast and accurate image super resolution by deep cnn with skip connection and network in network. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.S. (eds.) ICONIP 2017, vol. 10635, pp. 217–225. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-70096-0_23

    Chapter  Google Scholar 

  38. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010)

    Article  MathSciNet  Google Scholar 

  39. Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2011)

    Article  Google Scholar 

  40. Zhang, Y.: Spatial resolution enhancement of hyperspectral image based on the combination of spectral mixing model and observation model. In: Image and Signal Processing for Remote Sensing XX, vol. 9244, p. 924405. International Society for Optics and Photonics (2014)

    Google Scholar 

Download references

Acknowledgment

This work was funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 16SV8061 (MultiARC) and the German Federal Ministry for Economic Affairs and Climate Action (BMWi) under Grant No. 01MK21003 (NaLamKI). Only tissue that has been exposed during normal surgical treatment has been scanned additionally with our described camera. This procedure has been approved by Charité–Universitätsmedizin Berlin, Germany.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Eric L. Wisotzky .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wisotzky, E.L., Daudkane, C., Hilsmann, A., Eisert, P. (2022). Hyperspectral Demosaicing of Snapshot Camera Images Using Deep Learning. In: Andres, B., Bernard, F., Cremers, D., Frintrop, S., Goldlücke, B., Ihrke, I. (eds) Pattern Recognition. DAGM GCPR 2022. Lecture Notes in Computer Science, vol 13485. Springer, Cham. https://doi.org/10.1007/978-3-031-16788-1_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16788-1_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16787-4

  • Online ISBN: 978-3-031-16788-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics