Skip to main content
SpringerLink
Log in

A novel encoder–decoder wavelet model for multifocal region segmentation of TAO facial images

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Thyroid-associated ophthalmopathy (TAO) is an organ-specific autoimmune disease that has a significant impact on patients` life and health of. Clinically, the Clinical Activity Score (CAS) is one of the crucial methods for the early diagnosis of TAO. However, due to the diversity of TAO symptoms, utilizing medical expertise to artificially obtain CAS scores is challenging and highly dependent on personal subjectivity. Therefore, accurate identification of TAO regions segmented by scientific techniques is one of the essential prerequisites for the objective acquisition of the CAS scores. In this study, an encoder–decoder wavelet model (EDWM) with multiple-scale cascaded attention mechanism (MCAM) and residual deformable convolution (RDC) was proposed for multifocal region segmentation of TAO from facial images. The proposed method employs the discrete wavelet transform (DWT) to construct an encoder structure for the coarse feature extraction of the diseased regions. The inverse wavelet transform (IWT) is designed to build a decoder structure for resolution recovery. Meanwhile, the MCAM is developed to extract finer features of adjacent wavelet scales in the encoder structure by suppressing the background and focusing on the coarse segmentation of the diseased regions. The RDC is ultimately utilized for enlargement of arbitrary receptive fields and the accurate multi-segmentation task in different regions. In comparison with other selected benchmark models, the EDWM has, respectively, achieved state-of-the-art segmentation performance with 93.12% and 0.804 of the precision and the MIoU when tested on the images of 600 TAO patients. Since the EDWM is characterized by compact structure, interpretability, and strong feature extraction capability, it can provide a much more reliable and scientific basis for the early detection and diagnosis of TAO, reducing reliance on subjective experience in obtaining CAS scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability statement

Due to the restrictions related to privacy and third-party partner center, the TAO data is only available to collaborating scientists from the TAO research team. Some relevant research teams may provide data upon request, but not all researchers have access to this data due to relevant data protection laws.

References

  1. Bahn RS (2010) Graves’ ophthalmopathy. N Engl J Med 362(8):726–738

    Article  Google Scholar 

  2. Bartalena L, Baldeschi L, Boboridis K et al (2016) The 2016 European thyroid association/European group on graves’Orbitopathy guidelines for the Management of Graves’Orbitopathy. Eur Thyroid J 5(1):9–26. https://doi.org/10.1159/000443828

    Article  Google Scholar 

  3. Bartalena L (2012) Prevention of Graves’ ophthalmopathy. Best Pract Res Clin Endocrinol Metab 26(3):371–379. https://doi.org/10.1016/j.beem.2011.09.004

    Article  Google Scholar 

  4. Mourits MPH, Koornneef L, Wiersinga WM et al (1989) Clinical criteria for the assessment of disease activity in Graves’ ophthalmopathy: a novel approach. Br J Ophthalmol 73(8):639–644. https://doi.org/10.1136/bjo.73.8.639

    Article  Google Scholar 

  5. Higashiyama T, Nishida Y, Ohji M (2015) Changes of orbital tissue volumes and proptosis in patients with thyroid extraocular muscle swelling after methylprednisolone pulse therapy. Jpn J Ophthalmol 59(6):430–435. https://doi.org/10.1007/s10384-015-0410-4

    Article  Google Scholar 

  6. Mourits MP, Pmmmel MF, wiersinga wM, et al (1997) Clinical activity score as a guide in the management 0f patients with Graves ophthalmopathy. Clin Endocrinol(0xf) 47(1):9–14. https://doi.org/10.1046/j.1365-2265.1997.2331047.x

    Article  Google Scholar 

  7. Le Moli R, Pluchino A, Muscia V et al (2012) Graves’ orbitopathy: extraocular muscle/total orbit area ratio is positively related to the clinical activity score. Eur J Ophthalmol 22(3):301–308. https://doi.org/10.5301/ejo.5000018

    Article  Google Scholar 

  8. Song X, Liu Z, Li L et al (2021) Artificial intelligence CT screening model for thyroid-associated ophthalmopathy and tests under clinical conditions. Int J Comput Assist Radiol Surg 16(2):323–330. https://doi.org/10.1007/s11548-020-02281-1

    Article  Google Scholar 

  9. Zhu F, Gao Z, Zhao C et al (2021) Semantic segmentation using deep learning to extract total extraocular muscles and optic nerve from orbital computed tomography images. Optik 244:167551. https://doi.org/10.1016/j.ijleo.2021.167551

    Article  Google Scholar 

  10. Dolman PJ (2018) Grading severity and activity in thyroid eye disease. Ophthalmic Plast Reconstr Surg 34(4S):S34–S40. https://doi.org/10.1097/IOP.0000000000001150

    Article  Google Scholar 

  11. Tortora F, Cirillo M, Ferrara M et al (2013) Disease activity in graves’ ophthalmopathy: diagnosis with orbital MR imaging and correlation with clinical score. Neuroradiol J 26(5):555–564. https://doi.org/10.1177/197140091302600509

    Article  Google Scholar 

  12. Yang H, Qu X (2005) Overview of image segmentation methods. Comput Dev Appl 18(3):21–23

    Google Scholar 

  13. Parker JR (1997) Algorithms for image processing and computer vision. Wiley, New York

    Google Scholar 

  14. Lemieux L, Hagemann G, Krakow K et al (1999) Fast, accurate, and reproducible automatic segmentation of the brain in T1-weighted volume MRI data. Magn Reson Med 42(1):127–135. https://doi.org/10.1002/(SICI)1522-2594(199907)42:13.3.CO;2-F

    Article  Google Scholar 

  15. Pohle R, Toennies KD (2001) Segmentation of medical images using adaptive region growing[C]//Medical Imaging 2001: image processing. Int Soc Optics Photonics 4322:1337–1346. https://doi.org/10.1117/12.431013

    Article  Google Scholar 

  16. Kannan SR (2008) A new segmentation system for brain MR images based on fuzzy techniques. Appl Soft Comput 8(4):1599–1606. https://doi.org/10.1016/j.asoc.2007.10.025

    Article  Google Scholar 

  17. AlZubi S, Islam N, Abbod M (2011) Multiresolution analysis using wavelet, ridgelet, and curvelet transforms for medical image segmentation. Int J Biomed Imaging. https://doi.org/10.1155/2011/136034

    Article  Google Scholar 

  18. Mulcahy C (1997) Image compression using the Haar wavelet transform. Spelman Sci Math J 1(1):22–31

    MathSciNet  Google Scholar 

  19. Fourati W, Kammoun F, Bouhlel MS (2005) Medical image denoising using wavelet thresholding. J Test Eval 33(5):364–369. https://doi.org/10.1520/JTE12481

    Article  Google Scholar 

  20. Kara B, Watsuji N (2003) Using wavelets for texture classification[C]//IJCI. In: proceedings of international conference on signal processing 1(2)

  21. Do MN, Vetterli M (2003) The finite ridgelet transform for image representation. IEEE Trans Image Process 12(1):16–28. https://doi.org/10.1109/TIP.2002.806252

    Article  MathSciNet  MATH  Google Scholar 

  22. Iscan Z, Yüksel A, Dokur Z et al (2009) Medical image segmentation with transform and moment based features and incremental supervised neural network. Digit Signal Process 19(5):890–901. https://doi.org/10.1016/j.dsp.2009.03.001

    Article  Google Scholar 

  23. Leandro JJG, Cesar JR, Jelinek HF (2001) Blood vessels segmentation in retina: preliminary assessment of the mathematical morphology and of the wavelet transform techniques[C]//. In: Proceedings XIV Brazilian Symposium on Computer Graphics and Image Processing. IEEE 84–90. https://doi.org/10.1109/SIBGRAPI.2001.963041

  24. Akram MU, Atzaz A, Aneeque SF et al. (2009) Blood vessel enhancement and segmentation using wavelet transform[C]//. In: 2009 International Conference on Digital Image Processing. IEEE 34–38. https://doi.org/10.1109/ICDIP.2009.70

  25. Hoover A, STARE database [Online]. Available: http://www.ces.clemson.edu/ ahoover/stare

  26. Song J, Chi Z, Liu J (2006) A robust eye detection method using combined binary edge and intensity information. Pattern Recogn 39(6):1110–1125. https://doi.org/10.1016/j.patcog.2005.11.015

    Article  MATH  Google Scholar 

  27. Quellec G, Lamard M, Josselin PM et al. (2006) Detection of lesions in retina photographs based on the wavelet transform[C]//. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE 2618–2621. https://doi.org/10.1109/IEMBS.2006.260220

  28. Jiang K, Zhou Z, Geng X et al (2015) Isotropic undecimated wavelet transform fuzzy algorithm for retinal blood vessel segmentation. J Med Imag Health Inform 5(7):1524–1527. https://doi.org/10.1166/jmihi.2015.1561

    Article  Google Scholar 

  29. Stankiewicz A, Marciniak T, Dąbrowski A et al (2017) Denoising methods for improving automatic segmentation in OCT images of human eye. Bullet Polish Acad Sci Tech Sci. https://doi.org/10.1515/bpasts-2017-0009

    Article  Google Scholar 

  30. Mahesh Kumar SV (2018) Computer-aided diagnosis of anterior segment eye abnormalities using visible wavelength image analysis based machine learning. J Med Syst 42(7):1–12. https://doi.org/10.1007/s10916-018-0980-z

    Article  Google Scholar 

  31. Biswal B, Vyshnavi E, Sairam MVS et al (2020) Robust retinal optic disc and optic cup segmentation via stationary wavelet transform and maximum vessel pixel sum. IET Image Process 14(4):592–602. https://doi.org/10.1049/iet-ipr.2019.0845

    Article  Google Scholar 

  32. You S, Lei B, Wang S et al (2022) Fine perceptive gans for brain mr image super-resolution in wavelet domain. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3153088

    Article  Google Scholar 

  33. Zaidi H, El Naqa I (2010) PET-guided delineation of radiation therapy treatment volumes: a survey of image segmentation techniques. Eur J Nucl Med Mol Imaging 37(11):2165–2187. https://doi.org/10.1007/s00259-010-1423-3

    Article  Google Scholar 

  34. Ko JP, Naidich DP (2004) Computer-aided diagnosis and the evaluation of lung disease. J Thorac Imaging 19(3):136–155. https://doi.org/10.1097/01.rti.0000135973.65163.69

    Article  Google Scholar 

  35. Hiromatsu Y, KojimAa K, Ishisaka N et al (1992) Role of magnetic resonance imaging in thyroid-associated ophthalmopathy: its predictive value for therapeutic outcome of immunosuppressive therapy. Thyroid 2(4):299–305. https://doi.org/10.1089/thy.1992.2.299

    Article  Google Scholar 

  36. Yokoyama N, Nagataki S, Uetani M et al (2002) Role of magnetic resonance imaging in the assessment of disease activity in thyroid-associated ophthalmopathy. Thyroid 12(3):223–227. https://doi.org/10.1089/105072502753600179

    Article  Google Scholar 

  37. Kim HC, Yoon SW, Lew H (2015) Usefulness of the ratio of orbital fat to total orbit area in mild-to-moderate thyroid-associated ophthalmopathy. Br J Radiol 88(1053):20150164. https://doi.org/10.1259/bjr.20150164

    Article  Google Scholar 

  38. Byun JS, Moon NJ, Lee JK (2017) Quantitative analysis of orbital soft tissues on computed tomography to assess the activity of thyroid-associated orbitopathy. Graefes Arch Clin Exp Ophthalmol 255(2):413–420. https://doi.org/10.1007/s00417-016-3538-0

    Article  Google Scholar 

  39. Van Der Heijden AA, Abramoff MD, Verbraak F et al (2018) Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmol 96(1):63–68. https://doi.org/10.1111/aos.13613

    Article  Google Scholar 

  40. Lin H, Lin D, Liu Z et al (2016) A novel congenital cataract category system based on lens opacity locations and relevant anterior segment characteristics. Invest Ophthalmol Vis Sci 57(14):6389–6395. https://doi.org/10.1167/iovs.16-20280

    Article  Google Scholar 

  41. Lin C, Song X, Li L et al (2021) Detection of active and inactive phases of thyroid-associated ophthalmopathy using deep convolutional neural network. BMC Ophthalmol 21(1):1–9. https://doi.org/10.1186/s12886-020-01783-5

    Article  Google Scholar 

  42. Wang M, Zhang H, Dong L, et al (2021) Using the random forest algorithm to detect the activity of thyroid-associated ophthalmopathy. PREPRINT available at Research Square. https://doi.org/10.21203/rs.3.rs-787674/v1

  43. Wu C, Jin J (2018) Application of deep learning in the identification of TAO[C]//MIPPR 2017: parallel processing of images and optimization techniques; and medical imaging. Int Soc Optics Photonics 10610:106100E. https://doi.org/10.1117/12.2305837

    Article  Google Scholar 

  44. Wu C, Zou Y (2018) Application of transfer learning in the recognition of TAO[C]//. In: 2018 13th International Conference on Computer Science & Education (ICCSE). IEEE, 2018; 1–6. https://doi.org/10.1109/ICCSE.2018.8468803

  45. Nageswari CS, Kumar MNV, Raveena C et al (2021) An identification and classification of thyroid diseases using deep learning methodology. Rev Geintec-Gestao Inov E Tecnol 11(2):2004–2015. https://doi.org/10.47059/revistageintec.v11i2.1820

    Article  Google Scholar 

  46. Wu C, Zhan J, Zou Y, et al (2019) Ocular rectus muscle segmentation based on improved U-net[C]//. In: IOP Conference Series: Materials Science and Engineering. IOP Publishing, 533(1): 012058. https://doi.org/10.1088/1757-899X/533/1/012058

  47. Kingsbury NG (1998) The dual-tree complex wavelet transform: a new technique for shift invariance and directional filters[C]//IEEE digital signal processing workshop. Citeseer 86:120–131

    Google Scholar 

  48. Starck JL, Candès EJ (2002) Donoho D L (2002) The curvelet transform for image denoising[J]. IEEE Trans Image Process 11(6):670–684. https://doi.org/10.1109/TIP.2002.1014998

    Article  MathSciNet  MATH  Google Scholar 

  49. He H, Chen S (2021) Identification of facial expression using a multiple impression feedback recognition model. Appl Soft Comput 113:107930. https://doi.org/10.1016/j.asoc.2021.107930

    Article  Google Scholar 

  50. Gonzalez RC, Woods RE (1992) Digital image processing. Addison-Wesley, Reading

    Google Scholar 

  51. Dai J, Qi H, Xiong Y, et al (2017) Deformable convolutional networks[C]//. In: Proceedings of the IEEE international conference on computer vision pp 764–773. https://doi.org/10.48550/arXiv.1703.06211

  52. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26(3):297–302. https://doi.org/10.2307/1932409

    Article  Google Scholar 

  53. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation[C]//. In: International Conference on Medical image computing and computer-assisted intervention. Springer, Cham pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

  54. Long J, Shelhamer E, Darrell T (2015) Fully convolutional network for semantic segmentation[C]. In: Proceedings of the IEEE conference on computer vision and pattern recognition pp 3431–3440. https://doi.org/10.1109/TPAMI.2016.2572683

  55. Iandola FN, Han S, Moskewicz MW et al (2016) SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. arXiv preprint arXiv:1602.07360. https://doi.org/10.48550/arXiv.1602.07360

  56. Molchanov P, Tyree S, Karras T et al (2016) Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016. https://doi.org/10.48550/arXiv.1611.06440

  57. He H, Liu X, Hao Y (2021) A progressive deep wavelet cascade classification model for epilepsy detection. Artif Intell Med 118:102117. https://doi.org/10.1016/j.artmed.2021.102117

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Project of Ministry of Science and Technology of People’s Republic of China (No.G2021013008), the National Natural Science Foundation of China (No.61906121), the Project of the Science and Technology Commission of Shanghai Municipality (No. 18070503000), Innovative Research Team of High-Level Local Universities in Shanghai (SHSMU-ZDCX20210901), Key Project of Crossing Innovation of Medicine and Engineering, University of Shanghai for Science and Technology (No. 1020308405, 1022308502).

Funding

Ministry of Science and Technology of the People’s Republic of China, G2021013008, Hong He, National Natural Science Foundation of China, 61906121, Lei Zhou, Science and Technology Commission of Shanghai Municipality, 18070503000, Hong He, Innovative Research Team of High-Level Local Universities in Shanghai, SHSMU-ZDCX20210901, Huifang Zhou, Key Project of Crossing Innovation of Medicine and Engineering, University of Shanghai for Science and Technology, 1020308405, Hong He, 1022308502, Hong He.

Author information

Author notes

  1. Haipeng Zhu and Huifang Zhou have contributed equally to this work.

Authors and Affiliations

  1. School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China

    Haipeng Zhu, Hong He, Jiayu Chen, Kunhao Li & Lei Zhou

  2. Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    Huifang Zhou & Xuefei Song

  3. Shanghai Key Laboratory of Orbital Diseases and Ocular Oncology, Shanghai, China

    Huifang Zhou & Xuefei Song

Authors
  1. Haipeng Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huifang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiayu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xuefei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kunhao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong He.

Ethics declarations

Conflict of interest

The authors declare that the publication of this paper has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, H., Zhou, H., He, H. et al. A novel encoder–decoder wavelet model for multifocal region segmentation of TAO facial images. Neural Comput & Applic 35, 19145–19167 (2023). https://doi.org/10.1007/s00521-023-08727-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08727-2

Keywords

Search

Navigation