Skip to main content

Advertisement

SpringerLink
Log in

Inpainting non-anatomical objects in brain imaging using enhanced deep convolutional autoencoder network

  • Published:
Sādhanā Aims and scope Submit manuscript

Abstract

Medical diagnosis can be severely hindered by distorted medical images, especially in the analysis of Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) images. Therefore, enhancing the accuracy of diagnostic imaging and inpainting damaged areas are essential for medical diagnosis. Over the past decade, image inpainting techniques have advanced due to deep learning and multimedia information. In this paper, we proposed a deep convolutional autoencoder network with improved parameters as a robust method for inpainting non-anatomical objects in MRI and CT images. Traditional approaches based on the exemplar methods are much less effective than deep learning methods in capturing high-level features. However, the inpainted regions would appear blurr and with global inconsistency. To handle the fuzzy problem, we enhanced the network model by introducing skip connections between mirrored layers in the encoder and decoder stacks. This allowed the generative process of the inpainting region to directly use the low-level feature information of the processed image. To provide both pixel-accurate and local-global contents consistency, the proposed model is trained with a combination of the typical pixel-wise reconstruction loss and two adversarial losses, which makes the inpainted output seem more realistic and consistent with its surrounding contexts. As a result, the proposed approach is much faster than existing methods while providing unprecedented qualitative and quantitative evaluation with a high inpainting inception score of 10.58, peak signal-to-noise ratio (PSNR) 52.44, structural similarity index (SSIM) 0.95, universal image quality index (UQI) 0.96, and mean squared error (MSE) 40.73 for CT and MRI images. This offers a promising avenue for enhancing image fidelity, potentially advancing clinical decision-making and patient care in neuroimaging practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9

Similar content being viewed by others

Data availability

The datasets analyzed during the current study are not publicly available due to individual privacy but are available from the corresponding author on reasonable request.

Notes

  1. https://www.nlm.nih.gov/research/visible/getting_data.html

References

  1. Jing Y, Zheng H, Zheng W and Dong K 2022 A pixel-wise foreign object debris detection method based on multi-scale feature inpainting. Aerospace 9(9): 480.

    Article  Google Scholar 

  2. Jingwen Su, Boyan Xu and Yin Hujun 2022 A survey of deep learning approaches to image restoration. Neurocomputing 487: 46–65.

    Article  Google Scholar 

  3. Yu J, Xu X, Gao F, Shi S, Wang M,Tao D and Huang Q et al. 2021 Toward realistic face photo-sketch synthesis via composition-aided gans. IEEE Transactions on Cybernetics 51(9): 4350–4362.

    Article  Google Scholar 

  4. Dogan Y and Keles H Y 2022 Iterative facial image inpainting based on an encoder-generator architecture. Neural Computing & Application 34(12): 10001–10021.

    Article  Google Scholar 

  5. Luo H and Zheng Y 2022 Semantic residual pyramid network for image inpainting. Information 13(2): 7.

    Article  MathSciNet  Google Scholar 

  6. Feng S, Zhao H, Shi F, Cheng X, Wang M, Ma Y, Xiang D, Zhu W, and Chen X et al. 2020 CPFNet: context pyramid fusion network for medical image segmentation. IEEE Trans. Med. Imaging 39: 3008–3018.

    Article  Google Scholar 

  7. Yu J, Li K and Peng J 2022 Reference-guided face inpainting with reference attention network. Neural Comput & Application 34: 9717–9731.

    Article  Google Scholar 

  8. Fair D A, Miranda-Dominguez O, Snyder A Z, Perrone A, Earl E A, Van A N, Koller J M, Feczko E, Tisdall M D, Kouwe A, Klein R L, Mirro A E, Hampton J M, Adeyemo B, Laumann T O, Gratton C, Greene D J, Schlaggar B L, Hagler D J, Watts R, Garavan H, Barch D M, Nigg J T, Petersen S E, Dale A M, Feldstein-Ewing S W, Nagel B N, and Dosenbach N U F et al. 2020 Correction of respiratory artifacts in MRI head motion estimates. Neuroimage 208: 116400.

    Article  Google Scholar 

  9. Criminisi A, Perez P and Toyama K 2003 Object removal by exemplar-based inpainting. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2: 1–8.

    Google Scholar 

  10. Citko W and Sienko W 2022 Inpainted image reconstruction using an extended hopfield neural network based machine learning system. Sensors 22(3): 813.

    Article  Google Scholar 

  11. Zhao W, Li D, Niu K, Qin W, Peng H and Niu T 2018 Robust beam hardening artifacts reduction for computed tomography using spectrum modeling. IEEE Trans. Comput. Imaging. 5(2): 333–342.

    Article  Google Scholar 

  12. Alsalamah M and Amin S 2016 Medical image inpainting with RBF interpolation technique. Int. J. Adv. Comput. Sci. Appl. 7: 91–99.

    Google Scholar 

  13. Yang C, Lu X, Lin Z, Shechtman E, Wang O and Li H 2017 High-resolution image inpainting using multi-scale neural patch synthesis. In IEEE Conference on Computer Vision and Pattern Recognition. 6721–6729

  14. Gatys L A, Ecker A S and Bethge M 2016 Image style transfer using convolutional neural networks. Computer Vision and Pattern Recognition, 2414–2423

  15. Li J, Wang N, Zhang L, Du B and Tao D 2020 Recurrent Feature Reasoning for Image Inpainting. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7760–7768

  16. Feng, Z, Chi, S, Yin, J, Zhao, D and Liu, X 2007 A Variational Approach to Medical Image Inpainting Based on Mumford-Shah Model. International Conference on Service Systems and Service Management. 1–5

  17. Tran M T, Kim S H, Yang H J and Lee G S 2020 Medical Image Inpainting with Deep Neural Network. In: Proceedings of the Smart Media Spring Conference. 22–23

  18. Tran M T, Kim S H, Yang H J and Lee G S 2020 Deep Learning-Based Inpainting for Chest X-Ray Image. In: Proceedings of the International Conference on Smart Media and Applications (SMA). 17–19

  19. Kang S K, Shin S A, Seo S, Byun M S, Lee D Y and Kim Y K et al. 2021 Deep learning-based 3d inpainting of brain mr images. Scientific Reports. 11: 1673.

    Article  Google Scholar 

  20. Li M, Lin Z, Mech R, Yumer E and Ramanan D 2019 Photo Sketching: Inferring Contour Drawings from Images. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). 1403–1412

  21. Kang S K, Shin S A, Seo S, Byun M S, Lee D Y, Kim Y K, Lee D S, Lee J S, Olaf R, Philipp F and Thomas B 2015 U-net: Convolutional networks for biomedical image segmentation. Proc. Of the International Conference on Medical Image Computing and Computer-assisted Intervention. 234–241

  22. Vijay B, Ankur H and Roberto C 2015 Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling. arXiv preprint. 1505.07293

  23. Li Z D, Cheng J X and Liu J W 2020 MRF image inpainting algorithm based on structure offsets statistic and multidirection features. Acta Electronica Sinica 48(5): 985–989.

    Google Scholar 

  24. Ngo D K, Tran M T, Kim S H, Yang H J and Lee G S 2020 Multi-task learning for small brain tumor segmentation from MRI. Appl. Sci. 10: 7790.

    Article  Google Scholar 

  25. Dong C, Loy CC and Tang X. 2016 Accelerating the Super-Resolution Convolutional Neural Network. In: Proceedings of the European Conference on Computer Vision. 391–407

  26. Ting Xu, Ting Z H, Liang J D, Xi L Z and Jin F H 2022 Exemplar-based image inpainting using adaptive two-stage structure-tensor based priority function and nonlocal filtering. Journal of Visual Communication and Image Representation 83: 103430.

    Article  Google Scholar 

  27. Pavel R, Thomas P and Arjan K 2020 Style-transfer GANs for bridging the domain gap in synthetic pose estimator training. IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). 188-195

  28. Johnson J, Alahi A and Fei-Fei L 2016 Perceptual Losses Forreal-Time Style Transfer and Super-Resolution. In: Proceedings of the European Conference on Computer Vision. 694–711

  29. Liu G, Reda F A, Shih K J, Wang T C, Tao A and Catanzaro B 2018 Image Inpainting for Irregular Holes Using Partial Convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). 85–100

  30. Yu J, Lin Z, Yang J, Shen X, Lu X and Huang T S Free-Form Image Inpainting with Gated Convolution. In: Proceedings of the IEEE International Conference on Computer Vision. 4471–4480

  31. Comelli A, Dahiya N, Stefano A, Vernuccio F, Portoghese M, Cutaia G et al. 2018 Deep learning-based methods for prostate segmentation in magnetic resonance imaging. Appl Sci (Basel). 11(2): 782, 1-23

  32. Comelli A, Dahiya N, Stefano A, Vernuccio F, Portoghese M, Cutaia G, Bruno A, Salvaggio G, Yezzi A, Zang D, Zhao X and Qiao Y 2023 Enhanced brain parcellation via abnormality inpainting for neuroimage-based consciousness evaluation of hydrocephalus patients by lumbar drainage. Brain Informatics. 10(3): 1–15.

    Google Scholar 

  33. Amber L S, Michela A, Spyridon B, Michel B, Keyvan F, Bram V G, Annette K, Bennett A L, Geert L, Bjoern M, Olaf R, Ronald M S, Patrick B, Patrick F C, Richard K G, Marc G, Jennifer G, Stephan H, William R, Maureen K M, Sandy N, Eugene V, Lena M, and Jorge et al. 2019 A Large Annotated Medical Image Dataset for the Development and Evaluation of Segmentation Algorithms. arXiv:1902.09063

  34. Pathak D P, Krahenbuhl J, Donahue, Darrell T and Efros A A 2016 Context encoders: Feature learning by inpainting. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition. 2536–2544

  35. Sergey I and Christian S 2015 Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint. 1502.03167

  36. Tran M T, Kim S H, Yang H J and Lee G S 2021 Multi-task learning for medical image inpainting based on organ boundary awareness. Applied Sciences. 11(9): 4247.

    Article  Google Scholar 

  37. Karen S and Andrew Z 2014 Very deep convolutional networks for large-scale image recognition. arXiv preprint. 1409.1556

  38. Goodfellow I J, Pouget A and Mirza M 2014 Generative adversarial nets. Advances in Neural Information Processing Systems. 27: 1–9.

    Google Scholar 

  39. Shao H, Wang Y, Fu Y and Yin Z 2020 Generative image inpainting via edge structure and color aware fusion. Signal Process. Image Communication. 87: 115929.

    Article  Google Scholar 

  40. Liu G, Reda F A, Shih K J, Wang T C, Tao A and Catanzaro B 2018 Image Inpainting for Irregular Holes Using Partial Convolutions. In: Proceedings of the European Conference on ComputerVision (ECCV). 85–100

  41. Xie C, Liu S, Li C, Cheng M M, Zuo W, Liu X, Wen S and Ding E et al. 2019 Image Inpainting with Learnable Bidirectional Attention Maps. In: Proceedings of the IEEE International Conference on Computer Vision. 8858–8867

  42. Liu G, Reda F A, Shih K J, Wang T C and Tao A 2018 Catanzaro, Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision. 85–100

  43. Nguyen B, Feldman A, Bethapudi S, Jennings A and Willcocks C G 2021 Unsupervised Region-Based Anomaly Detection in Brain MRI with Adversarial Image Inpainting. In: IEEE 18th International Symposium on Biomedical Imaging (ISBI). 1127-1131

  44. Barnes C, Shechtman E, Finkelstein A and Goldman D B 2009 Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3): 1–11.

    Article  Google Scholar 

  45. Iizuka S, Simo S E and Ishikawa H 2017 Globally and locally consistent image completion. ACM Transactions on Graphics (ToG). 36(4): 1–14.

    Article  Google Scholar 

  46. Liu G, Reda F A, Shih K J, Wang T C, Tao A and Catanzaro B 2018 Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). 85-100

  47. Liu X, Xing F, Yang C, Kuo CJ, Fakhri G E and Woo J 2021 Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes (Workshop), 12658: 80

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence and Machine Learning, School of Engineering, Malla Reddy University, Hyderabad, India

    Puranam Revanth Kumar & Thayyaba Khatoon Mohammed

  2. Department of Computer Science and Engineering, AVN Institute of Engineering and Technology, Hyderabad, India

    B Shilpa

  3. Department of Electronics and Communication Engineering, Faculty of Science and Technology (IcfaiTech), ICFAI Foundation for Higher Education, Hyderabad, India

    Rajesh Kumar Jha

  4. Department of DS & AI, Faculty of Science and Technology (IcfaiTech), ICFAI Foundation for Higher Education, Hyderabad, India

    B Deevena Raju

Authors
  1. Puranam Revanth Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B Shilpa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajesh Kumar Jha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. B Deevena Raju
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thayyaba Khatoon Mohammed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Puranam Revanth Kumar or Rajesh Kumar Jha.

Ethics declarations

Competing interests

The authors declare that there are no competing interests.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, P.R., Shilpa, B., Jha, R.K. et al. Inpainting non-anatomical objects in brain imaging using enhanced deep convolutional autoencoder network. Sādhanā 49, 181 (2024). https://doi.org/10.1007/s12046-024-02536-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12046-024-02536-6

Keywords

Search

Navigation