Self-supervised Bayesian Deep Learning for Image Recovery with Applications to Compressive Sensing

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12356)


In recent years, deep learning emerges as one promising technique for solving many ill-posed inverse problems in image recovery, and most deep-learning-based solutions are based on supervised learning. Motivated by the practical value of reducing the cost and complexity of constructing labeled training datasets, this paper proposed a self-supervised deep learning approach for image recovery, which is dataset-free. Built upon Bayesian deep network, the proposed method trains a network with random weights that predicts the target image for recovery with uncertainty. Such uncertainty enables the prediction of the target image with small mean squared error by averaging multiple predictions. The proposed method is applied for image reconstruction in compressive sensing (CS), i.e., reconstructing an image from few measurements. The experiments showed that the proposed dataset-free deep learning method not only significantly outperforms traditional non-learning methods, but also is very competitive to the state-of-the-art supervised deep learning methods, especially when the measurements are few and noisy.


Self-supervised learning Bayesian neural network Compressive sensing Image recovery 



Tongyao Pang and Hui Ji would like to acknowledge the support from Singapore MOE Academic Research Fund (AcRF) Tier 2 research project (MOE2017-T2-2-156), and Yuhui Quan would like to acknowledge the support of National Natural Science Foundation of China (61872151, U1611461).

Supplementary material

504452_1_En_28_MOESM1_ESM.pdf (483 kb)
Supplementary material 1 (pdf 483 KB)


  1. 1.
    Arce, G., Brady, D., Carin, L., Arguello, H., Kittle, D.: Compressive coded aperture spectral imaging: an introduction. IEEE Signal Process. Mag. 31(1), 105–115 (2013)CrossRefGoogle Scholar
  2. 2.
    Baldi, P., Sadowski, P.J.: Understanding dropout. In: NeurIPS, pp. 2814–2822 (2013)Google Scholar
  3. 3.
    Barber, D., Bishop, C.M.: Ensemble learning in Bayesian neural networks. Nato ASI Ser. F Comput. Syst. Sci. 168, 215–238 (1998)zbMATHGoogle Scholar
  4. 4.
    Bernstein, M.A., Fain, S.B., Riederer, S.J.: Effect of windowing and zero-filled reconstruction of MRI data on spatial resolution and acquisition strategy. J. Magn. Reson. Imaging 14(3), 270–280 (2001)CrossRefGoogle Scholar
  5. 5.
    Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural network. In: ICML, pp. 1613–1622 (2015)Google Scholar
  6. 6.
    Cai, J., Ji, H., Liu, C., Shen, Z.: Blind motion deblurring from a single image using sparse approximation. In: CVPR, pp. 104–111 (2009)Google Scholar
  7. 7.
    Candes, E.J., Tao, T.: Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inf. Theor. 52(12), 5406–5425 (2006)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chen, G., Tang, J., Leng, S.: Prior image constrained compressed sensing (PICCS): a method to accurately reconstruct dynamic CT images from highly undersampled projection data sets. Med. Phys. 35(2), 660–663 (2008)CrossRefGoogle Scholar
  9. 9.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Danielyan, A., Katkovnik, V., Egiazarian, K.: Bm3d frames and variational image deblurring. IEEE Trans. Image Process. 21(4), 1715–1728 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Ding, Q., Chen, G., Zhang, X., Huang, Q., Ji, H., Gao, H.: Low-dose CT with deep learning regularization via proximal forward backward splitting. Phys. Med. Biol. 65(12), 125009 (2020)CrossRefGoogle Scholar
  12. 12.
    Dong, W., Shi, G., Li, X., Ma, Y., Huang, F.: Compressive sensing via nonlocal low-rank regularization. IEEE Trans. Image Process. 23(8), 3618–3632 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Duarte, M., et al.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: ICML, pp. 1050–1059 (2016)Google Scholar
  16. 16.
    Gamper, U., Boesiger, P., Kozerke, S.: Compressed sensing in dynamic MRI. Magn. Reson. Med. 59(2), 365–373 (2008)CrossRefGoogle Scholar
  17. 17.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  18. 18.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV, pp. 1026–1034 (2015)Google Scholar
  19. 19.
    Heckel, R., Hand, P.: Deep decoder: Concise image representations from untrained non-convolutional networks. arXiv preprint arXiv:1810.03982 (2018)
  20. 20.
    Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: NeurIPS, pp. 5574–5584 (2017)Google Scholar
  21. 21.
    Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., Ashok, A.: Reconnet: Non-iterative reconstruction of images from compressively sensed measurements. In: CVPR, pp. 449–458 (2016)Google Scholar
  22. 22.
    Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NeurIPS, pp. 6402–6413 (2017)Google Scholar
  23. 23.
    Li, C., Yin, W., Jiang, H., Zhang, Y.: An efficient augmented lagrangian method with applications to total variation minimization. Comput. Optim. Appl. 56(3), 507–530 (2013)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Li, M., Fan, Z., Ji, H., Shen, Z.: Wavelet frame based algorithm for 3d reconstruction in electron microscopy. SIAM J. Sci. Comput. 36(1), B45–B69 (2014)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Liu, J., Chen, N., Ji, H.: Learnable Douglas-Rachford iteration and its applications in dot imaging. Inverse Prob. Imaging 14(4), 683 (2020)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Liu, J., Kuang, T., Zhang, X.: Image reconstruction by splitting deep learning regularization from iterative inversion. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 224–231. Springer, Cham (2018). Scholar
  27. 27.
    Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007)CrossRefGoogle Scholar
  28. 28.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423. IEEE (2001)Google Scholar
  29. 29.
    Meinhardt, T., Moller, M., Hazirbas, C., Cremers, D.: Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In: ICCV, pp. 1781–1790 (2017)Google Scholar
  30. 30.
    Metzler, C.A., Maleki, A., Baraniuk, R.: Bm3d-amp: a new image recovery algorithm based on bm3d denoising. In: ICIP, pp. 3116–3120. IEEE (2015)Google Scholar
  31. 31.
    Metzler, C.A., Maleki, A., Baraniuk, R.: From denoising to compressed sensing. IEEE Trans. Inf. Theory 62(9), 5117–5144 (2016)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Mousavi, A., Patel, A., Baraniuk, R.: A deep learning approach to structured signal recovery. In: Allerton, pp. 1336–1343. IEEE (2015)Google Scholar
  33. 33.
    Nan, Y., Quan, Y., Ji, H.: Variational-EM-based deep learning for noise-blind image deblurring. In: CVPR, pp. 3626–3635 (June 2020)Google Scholar
  34. 34.
    Nan, Y., Ji, H.: Deep learning for handling kernel/model uncertainty in image deconvolution. In: CVPR, pp. 2388–2397 (June 2020)Google Scholar
  35. 35.
    Quan, Y., Chen, M., Pang, T., Ji, H.: Self2self with dropout: Learning self-supervised denoising from single image. In: CVPR, pp. 1890–1898 (2020)Google Scholar
  36. 36.
    Quan, Y., Ji, H., Shen, Z.: Data-driven multi-scale non-local wavelet frame construction and image recovery. J. Sci. Comput. 63(2), 307–329 (2015). Scholar
  37. 37.
    Schuler, C., B., C., Harmeling, S., Scholkopf, B.: A machine learning approach for non-blind image deconvolution. In: CVPR, pp. 1067–1074 (2013)Google Scholar
  38. 38.
    Shi, W., Jiang, F., Liu, S., Zhao, D.: Scalable convolutional neural network for image compressed sensing. In: CVPR, pp. 12290–12299 (2019)Google Scholar
  39. 39.
    Soltanayev, S., Chun, S.: Training deep learning based denoisers without ground truth data. In: NeurIPS, pp. 3257–3267 (2018)Google Scholar
  40. 40.
    Tang, J., Nett, B.E., Chen, G.: Performance comparison between total variation (TV)-based compressed sensing and statistical iterative reconstruction algorithms. Phys. Med. Biol. 54(19), 5781 (2009)CrossRefGoogle Scholar
  41. 41.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: CVPR, pp. 9446–9454 (2018)Google Scholar
  42. 42.
    Xu, K., Zhang, Z., Ren, F.: Lapran: A scalable laplacian pyramid reconstructive adversarial network for flexible compressive sensing reconstruction. In: ECCV, pp. 485–500 (2018)Google Scholar
  43. 43.
    Xu, L., Ren, J.S., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: NIPS, pp. 1790–1798 (2014)Google Scholar
  44. 44.
    Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-Net for compressive sensing MRI. In: NeurIPS, pp. 10–18 (2016)Google Scholar
  45. 45.
    Zhang, J., Ghanem, B.: Ista-net: Interpretable optimization-inspired deep network for image compressive sensing. In: CVPR, pp. 1828–1837 (2018)Google Scholar
  46. 46.
    Zhang, J., Pan, J., Lai, W.S., Lau, R.W., Yang, M.H.: Learning fully convolutional networks for iterative non-blind deconvolution. In: CVPR, pp. 3817–3825 (2017)Google Scholar
  47. 47.
    Zhussip, M., Soltanayev, S., Chun, S.: Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior. In: CVPR, pp. 10255–10264 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of MathematicsNational University of SingaporeSingaporeSingapore
  2. 2.School of Computer Science and EngineeringSouth China University of TechnologyGuangzhouChina

Personalised recommendations