Skip to main content

Image-to-Images Translation for Multiple Virtual Histological Staining of Unlabeled Human Carotid Atherosclerotic Tissue

Molecular Imaging and Biology volume 24, pages 31–41 (2022)Cite this article

Abstract

Purpose

Histological analysis of human carotid atherosclerotic plaques is critical in understanding atherosclerosis biology and developing effective plaque prevention and treatment for ischemic stroke. However, the histological staining process is laborious, tedious, variable, and destructive to the highly valuable atheroma tissue obtained from patients.

Procedures

We proposed a deep learning-based method to simultaneously transfer bright-field microscopic images of unlabeled tissue sections into equivalent multiple sections of the same samples that are virtually stained. Using a pix2pix model, we trained a generative adversarial neural network to achieve image-to-images translation of multiple stains, including hematoxylin and eosin (H&E), picrosirius red (PSR), and Verhoeff van Gieson (EVG) stains.

Results

The quantification of evaluation metrics indicated that the proposed approach achieved the best performance in comparison with other state-of-the-art methods. Further blind evaluation by board-certified pathologists demonstrated that the multiple virtual stains have high consistency with standard histological stains. The proposed approach also indicated that the generated histopathological features of atherosclerotic plaques, such as the necrotic core, neovascularization, cholesterol crystals, collagen, and elastic fibers, are optimally matched with those of standard histological stains.

Conclusions

The proposed approach allows for the virtual staining of unlabeled human carotid plaque tissue images with multiple types of stains. In addition, it identifies the histopathological features of atherosclerotic plaques in the same tissue sample, which could facilitate the development of personalized prevention and other interventional treatments for carotid atherosclerosis.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Code Availability

https://github.com/PangziZhang523/multi-image2images.

References

  1. WHO The top 10 causes of death, Fact Sheet. https://www.who.int/en/news-room/fact-sheets/detail/the-top-10-causes-of-death (accessed 9 December 2020).

  2. Pelisek J, Well G, Reeps C et al (2012) Neovascularization and angiogenic factors in advanced human carotid artery stenosis. Circ J 76:1274–1282

    CAS  Article  Google Scholar 

  3. Pelisek J, Pongratz J, Deutsch L, Reeps C, Stadlbauer T, Eckstein HH (2012) Expression and cellular localization of metalloproteases ADAMs in high graded carotid artery lesions. Scand J Clin Lab Inv 72:648–656

    CAS  Article  Google Scholar 

  4. Zhong XY, Ma ZC, Su YS et al (2020) Flavin adenine dinucleotide ameliorates hypertensive vascular remodeling via activating short chain acyl-CoA dehydrogenase. Life Sci 258:118156

    CAS  Article  Google Scholar 

  5. Rivenson Y, de Haan K, Wallace WD, Ozcan A (2020) Emerging advances to transform histopathology using virtual staining. BME Frontiers 2020:1–11

    Article  Google Scholar 

  6. Croce AC, Bottiroli G (2014) Autofluorescence spectroscopy and imaging: a tool for biomedical research and diagnosis. Eur J Histochem 58:2461

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Jamme F, Kascakova S, Villette S et al (2013) Deep UV autofluorescence microscopy for cell biology and tissue histology. Biol Cell 105:277–288

    CAS  Article  Google Scholar 

  8. Le TT, Langohr IM, Locker MJ, Sturek M, Cheng JX (2007) Label-free molecular imaging of atherosclerotic lesions using multimodal nonlinear optical microscopy. J Biomed Opt 12:54007

    Article  Google Scholar 

  9. Zoumi A, Yeh A, Tromberg BJ (2002) Imaging cells and extracellular matrix In vivo by using second-harmonic generation and two-photon excited fluorescence. P Natl Acad Sci USA 99:11014–11019

    CAS  Article  Google Scholar 

  10. Witte S, Negrean A, Lodder JC et al (2011) Label-free live brain imaging and targeted patching with third-harmonic generation microscopy. Proc Natl Acad Sci U S A 108:5970–5975

    CAS  Article  Google Scholar 

  11. Ji M, Orringer DA, Freudiger CW et al (2013) Rapid, label-free detection of brain tumors with stimulated Raman scattering microscopy. Sci Transl Med 5:201ra119

    Article  Google Scholar 

  12. Orringer DA, Pandian B, Niknafs YS et al (2017) Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy. Nat Biomed Eng 1:27

    Article  Google Scholar 

  13. Seeger M, Karlas A, Soliman D, Pelisek J, Ntziachristos V (2016) Multimodal optoacoustic and multiphoton microscopy of human carotid atheroma. Photoacoustics 4:102–111

    Article  Google Scholar 

  14. Bayramoglu N, Kaakinen M, Eklund L, Heikkila J (2017) Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks. Ieee Int Conf Comp V:64–71.

  15. Rivenson Y, Wang H, Wei Z et al (2019) Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat Biomed Eng 3:466–477

    CAS  Article  Google Scholar 

  16. Rivenson Y, Liu TR, Wei ZS, Zhang Y, de Haan K, Ozcan A (2019) PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light-Sci Appl 8:23

    Article  Google Scholar 

  17. Christiansen EM, Yang SJ, Ando DM et al (2018) In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173:792

    CAS  Article  Google Scholar 

  18. Liu Y, Yuan H, Wang ZY, Ji SW (2020) Global pixel transformers for virtual staining of microscopy images. Ieee T Med Imaging 39:2256–2266

    Article  Google Scholar 

  19. Li D, Hui H, Zhang YQ et al (2020) Deep learning for virtual histological staining of bright-field microscopic images of unlabeled carotid artery tissue. Mol Imaging Biol 22:1301–1309

    CAS  Article  Google Scholar 

  20. Zhang Y, de Haan K, Rivenson Y, Li J, Delis A, Ozcan A (2020) Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light Sci Appl 9:78

    CAS  Article  Google Scholar 

  21. Zhou NY, Cai D, Han X, Yao JH (2019) Enhanced cycle-consistent generative adversarial network for color normalization of H&E stained images. Lect Notes Comput Sc 11764:694–702

    Article  Google Scholar 

  22. Gupta L, Klinkhammer BM, Boor P, Merhof D, Gadermayr M (2019) GAN-based image enrichment in digital pathology boosts segmentation accuracy. Lect Notes Comput Sc 11764:631–639

    Article  Google Scholar 

  23. Isola P, Zhu JY, Zhou TH, Efros AA (2017) Image-to-image translation with conditional adversarial networks. Proc Cvpr Ieee:5967–5976.

  24. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. Med Imag Comput Comput Assist Interv Pt Iii 9351:234–241

    Google Scholar 

  25. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. Ieee T Image Process 13:600–612

    Article  Google Scholar 

  26. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. 2018 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (Cvpr):586–595.

  27. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun Acm 60:84–90

    Article  Google Scholar 

  28. Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. 2018 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (Cvpr):8789–8797.

  29. He ZL, Zuo WM, Kan MN, Shan SG, Chen XL (2019) AttGAN: facial attribute editing by only changing what you want. Ieee T Image Process 28:5464–5478

    Article  Google Scholar 

  30. Liu M, Ding YK, Xia M, et al. (2019) STGAN: a unified selective transfer network for arbitrary image attribute editing. 2019 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (Cvpr 2019):3668–3677.

  31. Wang W, Zhang Y, Hui H et al (2021) The effect of endothelial progenitor cell transplantation on neointimal hyperplasia and reendothelialisation after balloon catheter injury in rat carotid arteries. Stem Cell Res Ther 12:99

    CAS  Article  Google Scholar 

  32. Tong W, Hui H, Shang W et al (2021) Highly sensitive magnetic particle imaging of vulnerable atherosclerotic plaque with active myeloperoxidase-targeted nanoparticles. Theranostics 11:506–521

    CAS  Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the instrumental and technical support of multimodal biomedical imaging experimental platform, Institute of Automation, Chinese Academy of Sciences.

Funding

This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700401, 2016YFC0103803; the National Natural Science Foundation of China under Grant 81730050, 81827808, 62027901, 81671851, 81527805; CAS Youth Innovation Promotion Association under Grant 2018167; CAS Scientific Instrument R&D Program under Grant YJKYYQ20170075; CAS Key Technology Talent Program; and the Project of High-Level Talents Team Introduction in Zhuhai City (Zhuhai HLHPTP201703).

Author information

Author notes

  1. Guanghao Zhang, Bin Ning and Hui Hui are contributed equally to this work.

Authors and Affiliations

  1. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China

    Guanghao Zhang

  2. CAS Key Laboratory of Molecular Imaging, Institute of Automation, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Guanghao Zhang, Hui Hui, Xin Yang & Jie Tian

  3. Department of Ultrasound, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China

    Bin Ning, Tengfei Yu, Hongxia Zhang & Wen He

  4. University of Chinese Academy of Sciences, Beijing, China

    Hui Hui & Xin Yang

  5. Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine, Beihang University, Beijing, 100083, China

    Jie Tian

  6. Zhuhai Precision Medical Center, Zhuhai People’s Hospital, Affiliated With Jinan University, Zhuhai, 519000, China

    Jie Tian

Authors
  1. Guanghao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Hui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tengfei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongxia Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jie Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wen He
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Guanghao Zhang: Conceptualization, Methodology, Software, Writing Original draft.

Bin Ning: Conceptualization, Data acquisition, Writing—original draft.

Hui Hui: Conceptualization, Methodology, Supervision, Writing—review & editing.

Tengfei Yu: Investigation, Data interpretation.

Xin Yang: Conceptualization, Methodology, Data analysis.

Hongxia Zhang: Investigation, Data interpretation.

Jie Tian: Conceptualization, Project administration, Investigation, Supervision.

Wen He: Conceptualization, Project administration, Investigation, Supervision.

Corresponding authors

Correspondence to Jie Tian or Wen He.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1938 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Ning, B., Hui, H. et al. Image-to-Images Translation for Multiple Virtual Histological Staining of Unlabeled Human Carotid Atherosclerotic Tissue. Mol Imaging Biol 24, 31–41 (2022). https://doi.org/10.1007/s11307-021-01641-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11307-021-01641-w

Key words

  • Multiple virtual histological staining
  • Pix2pix network
  • Human carotid atheroma
  • Blind evaluation
  • Bright-field microscopic imaging