Abstract
Diabetic foot ulcers are a common manifestation of lesions on the diabetic foot, a syndrome acquired as a long-term complication of diabetes mellitus. Accompanying neuropathy and vascular damage promote acquisition of pressure injuries and tissue death due to ischaemia. Affected areas are prone to infections, hindering the healing progress. The research at hand investigates an approach on classification of infection and ischaemia, conducted as part of the Diabetic Foot Ulcer Challenge (DFUC) 2021. Different models of the EfficientNet family are utilized in ensembles. An extension strategy for the training data is applied, involving pseudo-labeling for unlabeled images, and extensive generation of synthetic images via pix2pixHD to cope with severe class imbalances. The resulting extended training dataset features 8.68 times the size of the baseline and shows a real to synthetic image ratio of 1:3. Performances of models and ensembles trained on the baseline and extended training dataset are compared. Synthetic images featured a broad qualitative variety. Results show that models trained on the extended training dataset as well as their ensemble benefit from the large extension. F1-Scores for rare classes receive outstanding boosts, while those for common classes are either not harmed or boosted moderately. A critical discussion concretizes benefits and identifies limitations, suggesting improvements. The work concludes that classification performance of individual models as well as that of ensembles can be boosted utilizing synthetic images. Especially performance for rare classes benefits notably.
L. Bloch and R. Brüngel—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Lancashire Teaching Hospitals: https://www.lancsteachinghospitals.nhs.uk/, access 2021-09-22.
- 2.
EfficientNet: https://github.com/mingxingtan/efficientnet, access 2021-10-03.
- 3.
pix2pixHD: https://github.com/NVIDIA/pix2pixHD, access 2021-09-12.
- 4.
V100: https://www.nvidia.com/en-us/data-center/v100/, access 2021-09-13.
- 5.
DGX-1: https://www.nvidia.com/en-us/data-center/dgx-1/, access 2021-09-13.
- 6.
Ubuntu Linux: https://ubuntu.com/, access 2021-07-10.
- 7.
NVIDIA®-Docker: https://github.com/NVIDIA/nvidia-docker, access 2021-07-10.
- 8.
Docker: https://www.docker.com/, access 2021-07-10.
- 9.
Deepo: https://github.com/ufoym/deepo, access 2021-09-22.
- 10.
ImageMagick: https://github.com/ImageMagick/ImageMagick, access 2021-09-22.
- 11.
pix2pixHD artifacts: https://github.com/NVIDIA/pix2pixHD/issues/46, access 2021-09-11.
- 12.
LIME: https://github.com/marcotcr/lime, access 2021-11-12.
- 13.
StyleGAN2+ADA: https://github.com/NVlabs/stylegan2-ada, access 2021-09-22.
References
Alzubaidi, L., Fadhel, M.A., Oleiwi, S.R., Al-Shamma, O., Zhang, J.: DFU_QUTNet: diabetic foot ulcer classification using novel deep convolutional neural network. Multimed. Tools Appl. 79(21), 15655–15677 (2019). https://doi.org/10.1007/s11042-019-07820-w
Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2), 125 (2020). https://doi.org/10.3390/info11020125
Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986). https://doi.org/10.1109/tpami.1986.4767851
Cassidy, B., et al.: Diabetic foot ulcer grand challenge 2021: evaluation and summary. arXiv preprint arXiv:2111.10376 (2021)
Das, S.K., Roy, P., Mishra, A.K.: DFU_SPNet: a stacked parallel convolution layers based CNN to improve diabetic foot ulcer classification. ICT Express (2021). https://doi.org/10.1016/j.icte.2021.08.022
Das, S.K., Roy, P., Mishra, A.K.: Recognition of ischaemia and infection in diabetic foot ulcer: a deep convolutional neural network based approach. Int. J. Imaging Syst. Technol. (2021). https://doi.org/10.1002/ima.22598
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009), pp. 248–255. IEEE (2009). https://doi.org/10.1109/cvpr.2009.5206848
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: Proceedings of the 9th International Conference on Learning Representations (ICLR 2021) (2021)
Falanga, V.: Wound healing and its impairment in the diabetic foot. The Lancet 366(9498), 1736–1743 (2005). https://doi.org/10.1016/s0140-6736(05)67700-8
Galdran, A., Carneiro, G., Ballester, M.A.G.: Convolutional nets versus vision transformers for diabetic foot ulcer classification. arXiv preprint arXiv:2111.06894 (2021)
Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems (NIPS 2017), vol. 27. Curran Associates, Inc. (2014)
Goyal, M., Reeves, N.D., Davison, A.K., Rajbhandari, S., Spragg, J., Yap, M.H.: DFUNet: convolutional neural networks for diabetic foot ulcer classification. IEEE Trans. Emerg. Top. Comput. Intell. 4(5), 728–739 (2020). https://doi.org/10.1109/tetci.2018.2866254
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hinton, G., Srivastava, N., Swersky, K.: Lecture 6e rmsprop: divide the gradient by a running average of its recent magnitude (2012). https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). IEEE (2017). https://doi.org/10.1109/cvpr.2017.632
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems (NeurIPS 2020), vol. 33, pp. 12104–12114. Curran Associates, Inc. (2020)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems (NIPS 2012), vol. 25. Curran Associates, Inc. (2012)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). https://doi.org/10.1023/b:visi.0000029664.99615.94
Merkel, D.: Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014(239), 2 (2014)
Micikevicius, P., et al.: Mixed precision training. In: Proceedings of the 6th International Conference on Learning Representations (ICLR 2018) (2018)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems (NeuriPS 2019), vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD 2016), pp. 1135–1144 (2016). https://doi.org/10.1145/2939672.2939778
Saeedi, P., et al.: Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: results from the international diabetes federation diabetes atlas, 9th edition. Diabetes Res. Clin. Pract. 157, 107843 (2019). https://doi.org/10.1016/j.diabres.2019.107843
Sarp, S., Kuzlu, M., Wilson, E., Guler, O.: WG\(^2\)AN: synthetic wound image generation using generative adversarial network. J. Eng. 2021(5), 286–294 (2021). https://doi.org/10.1049/tje2.12033
Siddiqui, A.R., Bernstein, J.M.: Chronic wound infection: facts and controversies. Clin. Dermatol. 28(5), 519–526 (2010). https://doi.org/10.1016/j.clindermatol.2010.03.009
Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research (PMLR 2019), vol. 97, pp. 6105–6114. PMLR (2019)
Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: K. Chaudhuri, R. Salakhutdinov (eds.) Proceedings of the 36th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research (PLMR 2019), vol. 97, pp. 6105–6114. PMLR (2019)
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H.: Training data-efficient image transformers distillation through attention. In: Proceedings of the International Conference on Machine Learning (ICML 2021), vol. 139, pp. 10347–10357 (2021)
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going deeper with image transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2021), pp. 32–42 (2021)
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018). IEEE (2018). https://doi.org/10.1109/cvpr.2018.00917
Wightman, R.: PyTorch image models (2019). https://doi.org/10.5281/zenodo.4414861. https://github.com/rwightman/pytorch-image-models
Yap, M.H., Cassidy, B., Pappachan, J.M., O’Shea, C., Gillespie, D., Reeves, N.D.: Analysis towards classification of infection and ischaemia of diabetic foot ulcers. In: Proceedings of the IEEE EMBS International Conference on Biomedical and Health Informatics (BHI 2021), pp. 1–4 (2021). https://doi.org/10.1109/BHI50953.2021.9508563
Yap, M.H., et al.: Deep learning in diabetic foot ulcers detection: a comprehensive evaluation. Comput. Biol. Med. 135, 104596 (2021). https://doi.org/10.1016/j.compbiomed.2021.104596
Yap, M.H., et al.: Diabetic Foot Ulcers Grand Challenge 2020. https://doi.org/10.5281/zenodo.3731068
Yap, M.H., et al.: Diabetic Foot Ulcers Grand Challenge 2021. https://doi.org/10.5281/zenodo.3715020
Yap, M.H., et al.: Diabetic Foot Ulcers Grand Challenge 2022. https://doi.org/10.5281/zenodo.4575228
Zhang, H., et al.: ResNeSt: split-attention networks. arXiv preprint arXiv:2004.08955 (2020)
Zhang, J., Zhu, E., Guo, X., Chen, H., Yin, J.: Chronic wounds image generator based on deep convolutional generative adversarial networks. In: Li, L., Lu, P., He, K. (eds.) Theoretical Computer Science. NCTCS 2018. CCIS, vol. 882, pp. 150–158. Springer, Singapore (2018). https://doi.org/10.1007/978-981-13-2712-4_11
Zhang, P., Lu, J., Jing, Y., Tang, S., Zhu, D., Bi, Y.: Global epidemiology of diabetic foot ulceration: a systematic review and meta-analysis. Ann. Med. 49(2), 106–116 (2016). https://doi.org/10.1080/07853890.2016.1231932
Acknowledgments
Louise Bloch and Raphael Brüngel were partially funded by PhD grants from University of Applied Sciences and Arts Dortmund, Dortmund, Germany. The authors thank Henryk Birkhölzer for advice on pix2pixHD.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Bloch, L., Brüngel, R., Friedrich, C.M. (2022). Boosting EfficientNets Ensemble Performance via Pseudo-Labels and Synthetic Images by pix2pixHD for Infection and Ischaemia Classification in Diabetic Foot Ulcers. In: Yap, M.H., Cassidy, B., Kendrick, C. (eds) Diabetic Foot Ulcers Grand Challenge. DFUC 2021. Lecture Notes in Computer Science(), vol 13183. Springer, Cham. https://doi.org/10.1007/978-3-030-94907-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-94907-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94906-8
Online ISBN: 978-3-030-94907-5
eBook Packages: Computer ScienceComputer Science (R0)
